drama, drama everywhere

We had an actual Major Incident in work today. Because what afternoon would be complete without noxious gases and risk of explosion, a full-scale evacuation and a phalanx of fire officers in breathing gear?!

A weird smell was reported to me pretty much the moment I stepped through the door this morning. Now, I have pretty much no sense of smell whatsoever, so couldn't really comment, but apparently it was pretty bad and was being investigated throughout the day. The source was traced to the new UPS (Uninterruptible Power Supply - basically a giant battery) in the data centre. The dratted thing was commissioned exactly one week ago! It's brand new! But it was overheating badly and was bulging and swelling and emitting noxious fumes and mid-afternoon my boss L made the decision to hit the fire alarm, get everyone out of the building and call the fire brigade.

I heard later that if action hadn't been taken, the UPS would have exploded within about an hour.

In total we had four appliances in attendance, plus a fire officer in a marked car, plus police to direct the traffic since the fire engines were parked in the road. Our business continuity planning had to swing into operation, with an emergency management team convening to make decisions and disseminate information and whatnot - I spent the afternoon running back and fore supporting them, found an empty desk in an adjacent building to log in and print off documentation and whatever was needed, posted 'do not enter' signs on the back door, etc. It was all go.

About half the staff who were evacuated hadn't stopped to pick anything up, because you aren't supposed to, so had no money or house keys or anything, so couldn't get home, so we had to make plans for what to do about that if we weren't allowed back into the building. The Chief Operating Officer and Vice Chancellor both wandered down the road to see what was happening and rally the troops, or something along those lines. There were big discussions over whether cutting power to the building would help the situation or simply make things worse - and cutting the power to the building would be a Very Big Deal because our machine room is the primary data centre for the entire University. About 80% of the services should, in theory, fail over to the secondary data centre in the event of an emergency, but that wouldn't happen for all of them, and just cutting everything off without powering down properly would cause all kinds of damage.

After about two hours, we were allowed back into the building in small groups, escorted by fire officers, to collect personal belongings. Then we were sent home - but as we left, the Emergency Management Team were just convening for phase two of the crisis. It remains to be seen if we're able to return to work in the morning!

drama, drama everywhere

We had an actual Major Incident in work today. Because what afternoon would be complete without noxious gases and risk of explosion, a full-scale evacuation and a phalanx of fire officers in breathing gear?!

A weird smell was reported to me pretty much the moment I stepped through the door this morning. Now, I have pretty much no sense of smell whatsoever, so couldn't really comment, but apparently it was pretty bad and was being investigated throughout the day. The source was traced to the new UPS (Uninterruptible Power Supply - basically a giant battery) in the data centre. The dratted thing was commissioned exactly one week ago! It's brand new! But it was overheating badly and was bulging and swelling and emitting noxious fumes and mid-afternoon my boss L made the decision to hit the fire alarm, get everyone out of the building and call the fire brigade.

I heard later that if action hadn't been taken, the UPS would have exploded within about an hour.

In total we had four appliances in attendance, plus a fire officer in a marked car, plus police to direct the traffic since the fire engines were parked in the road. Our business continuity planning had to swing into operation, with an emergency management team convening to make decisions and disseminate information and whatnot - I spent the afternoon running back and fore supporting them, found an empty desk in an adjacent building to log in and print off documentation and whatever was needed, posted 'do not enter' signs on the back door, etc. It was all go.

About half the staff who were evacuated hadn't stopped to pick anything up, because you aren't supposed to, so had no money or house keys or anything, so couldn't get home, so we had to make plans for what to do about that if we weren't allowed back into the building. The Chief Operating Officer and Vice Chancellor both wandered down the road to see what was happening and rally the troops, or something along those lines. There were big discussions over whether cutting power to the building would help the situation or simply make things worse - and cutting the power to the building would be a Very Big Deal because our machine room is the primary data centre for the entire University. About 80% of the services should, in theory, fail over to the secondary data centre in the event of an emergency, but that wouldn't happen for all of them, and just cutting everything off without powering down properly would cause all kinds of damage.

After about two hours, we were allowed back into the building in small groups, escorted by fire officers, to collect personal belongings. Then we were sent home - but as we left, the Emergency Management Team were just convening for phase two of the crisis. It remains to be seen if we're able to return to work in the morning!