Why IT's So Hard

Why is providing reliable IT infrastructure so hard? Here's a good example.

There was a fire in downtown Vancouver this week that knocked out power to a good part of downtown for up to three days. Angela noted that the Internet was slow the day of the fire. I know there's a major network hub in the area of the fire at Harbour Centre, and I suspected that something had gone wrong there, despite all the precautions that would have been taken. Now I have proof.

The fire knocked out power to the network hub, and the generator kicked in as planned, but the Vancouver Fire and Rescue Services were sucking so much water to fight the fire that the generator had to shut down because it wasn't getting enough cooling water. Not only was that hard to predict, it would have been really hard to test -- I suppose the Fire Department would have loved an excuse to play with their hoses, but I'm not sure the City would have wanted them to run a test that tried to use up all the water in downtown Vancouver.