We've just spent the last week getting the Entlib 4 project going again, and I'm amazed as usual how much stuff just got broken in the build system.

 At p&p, we run our continuous integration servers as VPC's on a fairly big multi-processor box. It generally works out pretty well because none of the projects are typically building at the same time. And the nice thing for us is we can (in theory) archive build servers when a project goes into "hibernation"; just shut down the VPC, copy the file off to backup, and bring it back when you're ready to go.

Of course, in practice stuff like that never just works. Entlib 3.1 wrapped in May of 07, and the build server's been shut down since then. Around Microsoft, if a machine isn't used for more than a month, it gets kicked off the domain, so we had to fix that. Then the account credentials the server was running under had expired. Ok, next one, hey it's building ... what do you mean out of disk space? Turned out the new account didn't have access to the TFS project, so the build server happily output 20 gigs of log files saying "can't connect to source control." And then there was the Oracle server that wasn't there anymore, and the permission changes on the event log that made 200 tests fail, and on and on and on...

ARGH! All this breakage, and there wasn't even a single checkin between May and the start of the project. It's all straightened out, but I have to say, oy, what a pain!

Luckily, I've got a fantastic team that volunteered to shoulder most of the burden. Fernando, thanks a million!