The Business of Software Wiki: “The point of this wiki is to bring under one roof as much useful, quality information as possible about the business of software, whether it's microISVs selling desktop software, Web 2.0 sites or even the big enterprise kind of outfits.”
Hmm, gee, that phrasing is a little bit awkward.
“The point of this wiki is to bring under one roof as much high quality, useful
, qualityinformation as possible about the business of software, whether it's microISVs selling desktop software, Web 2.0 sites or even the big enterprise kind of outfits.”
I’m happy to announce that FogBugz On Demand is now available. This is a professionally-hosted version of FogBugz 5.0, previously only available as a download.
Selling web software has always been a slightly strange aspect of the way Fog Creek operates. Since FogBugz is a web-based project management tool, why should customers have to download it and install it on their own servers?
In the past, we did it that way because we’re a small company. With just a handful of full time employees, we didn’t really have the resources to be a reliable service provider that customers could trust with their mission-critical data.
To prepare for FogBugz On Demand, we’ve done a lot of hard work over the past year.
First, we hired a professional system administrator. He has pored over every inch of our hosting infrastructure, patching and testing and improving reliability everywhere. He upgraded all the NetBSD servers to Linux, installed a bunch of new hardware, and added lots and lots of automated monitoring.
In general, we decided to use high-end components for our hosting architecture: Dell PowerEdge 2950 Servers with SCSI RAID, Windows Server 2003, and SQL Server 2005. Yep, that’s an expensive way to do it. Since FogBugz runs fine on LAMP (Linux / Apache / MySQL / PHP), we could have gotten a bunch of cheap boxes, used all free software, and saved some money in exchange for some level of headaches. Indeed, most hosted services really should be built on LAMP. In our case, though, the cost of those Microsoft licenses and those extremely reliable Dell servers can be spread out over quite a few paying customers, so for us the cost difference per customer is really inconsequential. And we’ve been running IIS and Microsoft SQL Server here for six years without data loss, so that’s what we know and trust. But to be honest, if we ever get to the point where we’re racking up 10 new servers at a time, we’ll almost certainly switch to LAMP. I’m still gonna buy Dell servers and SCSI hard drives, because frankly, the small extra cost over cheapo white boxes is well worth it in reliability.
We made changes to the FogBugz code base itself to make it work better in a multi-hosted environment. The biggest surprise was how much work it took so that every user sees things in their own time zone. We also put a lot of work into the accounting and billing system (FogBugz On Demand is $21 a month, with no commitment). We implemented a database-backed DNS system so that each On Demand customer gets their own domain (you.fogbugz.com).
The biggest change was bringing up a second data center. I can’t tell you how scary it is to be responsible for our customers’ mission-critical data, so I didn’t want to have any single point of failure, no matter how fortified it is.
Our first data center has been with Peer 1 Network in New York’s financial district. Peer 1 is a Canadian backbone provider where we’ve been since the beginning of 2003. To take advantage of their backbone, we put our second data center in their new Los Angeles facility. This new data center is pretty much an exact replica of what we have in New York.
To some extent, by using Peer 1 for our second facility we are, technically, putting all our eggs on one backbone. But it’s a pretty darn reliable backbone and an excellent company. We actually investigated a couple of other colo providers and even went so far as to build out a facility in Chicago (with an unnamed provider). But shortly before we launched, they had a six hour outage, and in the aftermath of that, we discovered that their network connectivity was inadequate and their concept of building reliable systems did not use the same definition of “reliable” as we do. So we gave up on them, shipped all the servers from Chicago to LA, and went with the tried and true Peer 1.
Rather than setting up Los Angeles as a mere backup, we decided it would be completely live. Half our customers will be hosted from Los Angeles, and half from New York. That way we know at any time that both data centers are working and set up correctly, and we don’t have to wait until a massive failure to discover the problems with the backup data center.
Copies of the database backups are maintained in both cities, and each city serves as a warm backup for the other. If the New York data center goes completely south, we’ll wait a while to make sure it’s not coming back up, and then we’ll start changing the DNS records and start bringing up our customers on the warm backup in Los Angeles. It’s not an instantaneous failover, since customers will have to wait for two things: we’ll have to decide that a data center is really gone, not just temporarily offline, and they’ll have to wait up to 15 minutes for the DNS changes to propagate. Still, this is for the once-in-a-lifetime case of an entire data center blowing up, not just for occasional outages: each data center already has incredible backbone connectivity, UPSs, backup diesel generators, and so forth (Peer 1 survived that huge blackout during the summer of 2003 while many of their competitors were winking out).
To implement this warm backup feature, I wrote a SQL mirroring application that implements transaction log shipping: basically, it does an incremental backup in one city, compresses that backup, ships it to the other city, uncompresses it, and applies it to the warm backup database. Right now, we’re log shipping twice a day, so you might lose a day of work if an entire city blew up, but in a couple of weeks, we’ll implement a system that does more continuous backups, and we expect that the warm backups will never get more than 15 minutes behind.
FogBugz On Demand has actually been in beta since April, and in fact we have been hosting FogBugz trials on line since 2000, without ever losing anyone’s data. (The first FogBugz 1.0 trial server, believe it or not, was a Thinkpad laptop with a broken screen plugged into our office T1!) So I’m pretty confident now that our little company can do a pretty good job of hosting FogBugz for you.
Mike P. asks:
“On your recent post you mention that FogBugz is now going to be hosted and that you hired a sysadmin who converted your systems from NetBSD to Linux. Then you go on to say that you bought a Dell server, W2K3 and W2K5 SQL Server. I don't understand. Are you or are you not runing FBOD with the LAMP stack?”
I was not very clear. We have a lot of servers. Some of them were running NetBSD, and are now running Linux. Others are running Windows Server 2003. The Linux servers tend to be used for routers and assorted network services like DNS. The Windows servers are either working as web front ends (IIS) or database back ends (MS SQL 2005).
FogBugz On Demand is, indeed, on the Windows servers. We try very hard to stick to a single hardware architecture so any server can be swapped with any other in case of problems, and parts can be moved hither and yon as needed. We weren't really able to do this perfectly because Dell switched from SCSI to SAS hard drives when they went from the 8th generation to the 9th generation servers, but mostly we use a common platform so it's easy to keep spare parts on hand. (In fact, one backup server even has an extra NIC in it so it can replace the router if it comes to that).
Maxim Shemanarev: “The amazing thing is there is no rocket science! Nothing to patent! All information is publicly available and/or deducible from what we see. You only need to use a bit of your engineering intuition plus common sense. So, it goes. You can download an application with full sources in the end of the article and play with it, but not now, please. Now be patient to read rather a long story.” Wow!
We do it every summer... an open house at the Fog Creek offices in New York City.
Come on by next Thursday, July 19, 2007, between 5:00 pm and 7:00 pm to see our offices and meet the Fog Creek team including this years' interns, a.k.a. the caribou, the software management trainees, and me. It's always a fun chance to meet other software developers in the New York area and geek out about impedence mismatch. Bring your boss to prove how private offices, 30" LCDs, and Aeron chairs are possible. Drink free wine. Show off your new iPhone. No RSVP necessary.
Fog Creek Software is at 535 8th Ave., on the 18th Floor.
Our systems administrator Michael Gorsuch explains it: “So, yes, even though the SQL Server Mirroring technology sounds like an ideal fit at first, it is easy to see how it doesn’t really suit our needs.”
Back in 2001, I wrote: “When you're working on a really, really good team with great programmers, everybody else's code, frankly, is bug-infested garbage, and nobody else knows how to ship on time. When you're a cordon bleu chef and you need fresh lavender, you grow it yourself instead of buying it in the farmers' market, because sometimes they don't have fresh lavender or they have old lavender which they pass off as fresh.”
The Business of Software conference coming up at the end of October is new this year, but it's got a pretty phenomenal line-up of speakers:
Also speaking: Dan Nunan, Jennifer Aaker, Jeffrey Pfeffer, Bill Buxton, and me. Register here.
Steve from Richmond, VA, wrote in to ask a couple of questions:
“With software development, is it better to get something out there with customers and then continually improve or build the best wiz-bang software and then start marketing?”
That depends. You want to avoid the Marimba Phenomenon, where you get so much publicity in the early days that everybody checks out your underwhelming offering and decides that you're never going to have something worth looking at. (I should rename this the live.com phenomenon, in honor of Microsoft's horrible launch of live.com. Or the Zune phenomenon.) On the other hand, small startups are unlikely to have the problem of too much attention, so most companies with a 1.0 product can certainly get real customers with their earliest usable versions and build from there.
Steve is starting a company with a software developer.
“Assuming the software developer is getting paid at a reduced rate, but concurrently with his development. If you were giving also giving him some equity in your company, would you make that equity contingent on phases of the software getting done, the entire software getting done or vesting over time.”
The standard solution is to vest over time -- anywhere from four to seven years -- with unvested shares being forfeited if he leaves for any reason. If the software doesn't get done, you fire him and he loses the unvested shares -- it's not necessary to make the vesting contingent specifically on finishing the software (besides, "finishing" software would be too hard to define in a contract).
You can set it up either as normal vesting (where he gets, say, 20% of the shares every year) or reverse vesting (where he gets the shares up front, but you have the right to repurchase them for a penny, and this right evaporates by 20% every year). Reverse vesting is preferable for tax reasons, because at the time you give him the shares, they're worth a lot less, so there's less income and more capital gains, which are taxed at a lower rate.
I'm still working ferociously on getting ready for the FogBugz World Tour. We won't have the final list of cities until we can get hotel meeting rooms confirmed in each city, but it looks like the first round will hit about 20 cities in the US and Canada plus a few stops in New Zealand, Australia, and Europe. In any case, I'm looking at about six weeks on the road.
Fog Creek's office manager Liz is planning all the logistics for the event. Today she faxed around about 80 RFPs to hotels to get meeting rooms lined up. (There's a huge business opportunity there--getting hotel event coordinators onto the Internet). I have to buy a bunch of gear:
Is there such a thing as a video projector which connects to the PC using Cat-5 instead of video cable, so I don't have to carry a 50 foot video cable around?
What's a good subnotebook? Does anyone have any experience (good or bad) with the new flash-drive based notebooks?
What about the portable PA systems? Is there something self-contained that fits in the overhead bin on a plane?
Reminder! The annual Fog Creek open house is today (Thursday), so if you're in New York, I'd love to meet you in person and show you around.
Come on by between 5:00 pm and 7:00 pm to see our offices and meet the Fog Creek team including this years' interns, a.k.a. the caribou, the software management trainees, and me. Bring your boss to prove how private offices, 30" LCDs, and Aeron chairs are possible. See the fish. Drink free wine. Show off your new iPhone. No RSVP necessary.
Fog Creek Software is at 535 8th Ave., on the 18th Floor.
Even if you never read a single thing Dave Winer wrote in his 439 years of blogging, it's worth taking time to study his ideas about comments on blogs (he doesn't allow them).
"...to the extent that comments interfere with the natural expression of the unedited voice of an individual, comments may act to make something not a blog.... The cool thing about blogs is that while they may be quiet, and it may be hard to find what you're looking for, at least you can say what you think without being shouted down. This makes it possible for unpopular ideas to be expressed. And if you know history, the most important ideas often are the unpopular ones.... That's what's important about blogs, not that people can comment on your ideas. As long as they can start their own blog, there will be no shortage of places to comment."
The important thing to notice here is that Dave does not see blog comments as productive to the free exchange of ideas. They are a part of the problem, not the solution. You don't have a right to post your thoughts at the bottom of someone else's thoughts. That's not freedom of expression, that's an infringement on their freedom of expression. Get your own space, write compelling things, and if your ideas are smart, they'll be linked to, and Google will notice, and you'll move up in PageRank, and you'll have influence and your ideas will have power.
When a blog allows comments right below the writer's post, what you get is a bunch of interesting ideas, carefully constructed, followed by a long spew of noise, filth, and anonymous rubbish that nobody ... nobody ... would say out loud if they had to take ownership of their words. Look at this innocent post on a real estate blog. By comment #6 you're already seeing complete noise. By #13 you have someone cursing and saying "go kill yourself." On a real estate blog. #18 and #23 have launched into a middle eastern nuclear conflageration which continues for 100 posts. They're proving John Gabriel's Greater Internet Fuckwad Theory every day. Pathetic. On a real estate blog. Lockhart Steele, is this what you want Curbed to look like? Really? It's not fun, freewheeling freedom of expression, yay first amendment!. It's mostly anonymous hate speech.
OK, that's an extreme example... or is it? I don't know how many times I've read a brilliant article someone wrote on a blog. By the end of the article, I'm excited, I'm impressed, it was a great article. And then you get the dribble of morbid, meaningless, thoughtless comments. If the article, for example, mentions anything in anyway related to Microsoft, you get some kind of open source nuclear war. If the article mentions web browsing in any way, there's always some person without an outbound filter who feels compelled to tell you about how he uses Opera, so he doesn't have this problem, although, frankly, I could care less what Anonymous uses. He's not even human to me, he's anonymous. What web browser he uses doesn't amount to a hill of beans. It's not a single bean. It's not even the memory of last week's huevos rancheros. It's just noise. Useless noise. Thoughtless drivel written by some anonymous non-entity who really didn't read the article very carefully and didn't come close to understanding it and who has no ability whatsoever to control his typing diarrhea if the site's software doesn't physically prevent him from posting.
Dave is absolutely right. The way to give people freedom of expression is to give them a quiet place to post their ideas. If other people disagree, they're welcome to do so... on their own blogs, where they have to take ownership of their words.
I'm really losing patience with anonymous posts, "anon", "anon for this one," people who don't even have the energy to sign their messages with a made up name and leave the whole signature blank. Frankly if every anonymous post disappeared from the Joel on Software discussion group, the overall quality of the conversation would go up, way up, and the discussion would be way more interesting. Try this as an experiment: read through the last few dozen topics on the discussion group, and imagine that all the "anonymous" and signed-blank posts just disappeared. Would the quality of conversation be higher? Would that be a place you'd be more likely to want to spend time in?
Clay: “...the sites that suffer most from anonymous postings and drivel are the ones operating at large scale. If you are operating below that scale, comments can be quite good, in a way not replicable in any ‘everyone post to their own blog’”.
Dave: “...he says that I don't allow comments on Scripting News. That's not exactly true, there are comments here, but you have to look carefully to find them. I find this ups the quality enormously -- people don't generally comment here to embarass anyone or to provoke a fight -- there isn't enough traffic to interest those people. But the people who want to add information to a thread here on Scripting News, and have been reading the site long enough to know what it's about, they find their way to the comments and add something to the mix.”
Both excellent comments. Thank you. This is an example of people posting their replies on their own site. There is a lot more value to them than the comments about this post on my own discussion forum. Why? Because I know who Clay is, I've met him, he wrote A Group Is Its Own Worst Enemy, which, to this date, is the most important, insightful, and brilliant understanding of group dynamics in online communities. Dave has something to add to the conversation; some thoughts he's had since the article I quoted him on. Great content, in their own spaces.
And you're not only hearing their replies because I'm telling you about them. You might be hearing about it from Techmeme, which has really cool algorithms to figure out where the conversation is, and put it back together. Look at Techmeme right now:
They reconstructed the conversation magically for you. Of course, Techmeme only works for the biggest, noisiest bloggers. But Bloglines will do it for anyone.
PS to Clay: I'm still in awe at:
Now, there's a large body of literature saying "We built this software, a group came and used it, and they began to exhibit behaviors that surprised us enormously, so we've gone and documented these behaviors." Over and over and over again this pattern comes up. (I hear Stewart [Brand, of the WELL] laughing.) The WELL is one of those places where this pattern came up over and over again.
I apologize for falling into such a common pattern!
Liz and I have been working to get ready for the FogBugz 6.0 World Tour, in which I travel from city to city (accompanied by a programmer) giving demos of all the cool features in the upcoming version. This is basically about the same logistical complexity as planning a family trip to DisneyLand, times 32.
Very little has been finalized, but it looks like the first round will hit Vancouver, Seattle, Chicago, New York, Princeton, Philadelphia, Boston, Toronto, Waterloo, Washington, Atlanta, Dallas, Austin, Denver, San Francisco, Berkeley, Newark (CA), Mountain View, Santa Monica, Irvine, and San Diego. The second round will hit Auckland, Christchurch, Brisbane, Melbourne, Sydney, Munich, Copenhagen, Amsterdam, London, Cambridge, and Dublin. Whenever my squirt nephew decides to have his bar mitzvah, we'll come to Tel Aviv.
Given the cost of flights, hotels, rental cars, meals, catering (we'll serve refreshments), printing (we'll pass out brochures), FogBugz polo shirts, etc., it looks like the cost for this trip is going to come to about $64 per attendee... that is, it's costing us $64 to put on a live demo of FogBugz for each person who attends. Phew. No wonder people do those crappy webinar things instead. I still think it's worth it, though.
1111 posts over 13 years. Everything I’ve ever published is right here.
There’s a software company in New York City dedicated to doing things the right way and proving that it can be done profitably and successfully.