[rescue] FW: WTT: 1.5G of PC2700 for 1G of PC100

[Michael Parson]

On Thu, 1 May 2008, Bill Bradford wrote:

> On Thu, May 01, 2008 at 01:03:01PM -0500, Steve Hatle wrote:
>> Why should that take 30 hours? Hit the Big Red Button and BOOM data center
>> is shut down.
>> Should only take a minute or two at most...
>
> My first day at $JOB-1 was like that.  I was sitting at my boss' desk,
> filling out paperwork, when there suddenly came a quiet in the room
> (the DC was on the other side of the wall).
>
> People started popping up from cubes like a game of whack-a-mole, and
> then everyone ran into the DC.
>
> I walked in just in time to see the last bit of equipment with power
> in the room, some EMC Symmetrix frames, shut down and go off their
> internal emergency UPS backup.  It was eerily quiet.
>
> What happened was that someone (who was new - but it WASNT me) had
> gone into the other side of the DC and not shut the door properly, and
> it was buzzing at her.  Someone else was waving at her to shut the
> door to make the noise stop - so she figured that the big red button
> next to the door was the "alarm stop" switch, right?
>
> The following week EVERYONE in the building had to go through a short
> course of "This is the PROTECTED[0] Big Red Switch.  This is what
> it does, this is when you should and should not touch it, and if
> you touch it without authorization you will be escorted out of the
> building and your employment terminated." [1]

Yes, this is one of my favorite stories.  I was a customer of said DC
when that happened, but I had worked for the company that owned the DC
previously and was actually invovled in the design of it.  I had told
them to put break-glass over the Big Red Button, but they wouldn't do
it, said it might violate code (no one bothered to check).

The sad thing is, this was the SECOND time someone had hit the Big Red
Button thinking it was the alarm killer button.  The question then
becomes, why would someone think this?  Well, in EVERY OTHER FACILITY
this company ran out of, they had a big red alarm stop button next to
the doors.  You were supposed to prox-card in and out of every door, but
if you didn't there was a handy alarm-kill button next to the door you
could hit.  Nice, eh?

This building was the anomoly.

The first time it happened, the new company I was working for was
demoing our new product to the Board of Directors, showing them that
their investment dollars had actually started to produce something
(other than an office full of raging alcoholics).   They got through
about two pages when it quit responding.

Seems that a telco worker was leaving and tried to go through the exit
door w/o badging first, the alarm went off, so he stood there with his
foot in the door, punching buttons on the wall next to the exit.  The
FM 200 release was the first button he hit, but thankfully, it had
a pull-pin to prevent accidental discharge, then he hit the Big Red
Button.

So, what did the brilliant DC manager do to prevent this from happening
again?  He put a clear plastic cover over the button on a hinge.  After
the second incident that Bill told us about, they put break-glass covers
over the Big Red Buttons.

> Bill
>
> [0] Previously, it had not had any sort of protective cover, labeling,
> etc - everyone just knew what it was.  Now it had a flip-up cover like
> a fire alarm, along with descripive labeling.
>
> [1] There was no corrective action taken against the perpetrator, as
> she had not been properly educated on what the Big Red Button was,
> and so forth.  The ribbing she got from her coworkers sufficed.

-- 
Michael Parson
mparson at bl.org