|
|
Enron Mail |
Cc: mike.mcconnell@enron.com, jeffrey.shankman@enron.com,
jeffrey.mcmahon@enron.com, raymond.bowen@enron.com, john.lavorato@enron.com, louise.kitchen@enron.com, rebecca.mcdonald@enron.com, kevin.hannon@enron.com, ken.rice@enron.com Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Bcc: mike.mcconnell@enron.com, jeffrey.shankman@enron.com, jeffrey.mcmahon@enron.com, raymond.bowen@enron.com, john.lavorato@enron.com, louise.kitchen@enron.com, rebecca.mcdonald@enron.com, kevin.hannon@enron.com, ken.rice@enron.com X-From: Philippe A Bibi X-To: Mark Frevert, Greg Whalley X-cc: Mike McConnell, Jeffrey A Shankman, Jeffrey McMahon, Raymond Bowen, John J Lavorato, Louise Kitchen, Rebecca McDonald, Kevin Hannon, Ken Rice X-bcc: X-Folder: \Lawrence_Whalley_Nov2001\Notes Folders\All documents X-Origin: WHALLEY-G X-FileName: gwhalley.nsf Mark/Greg, Please find attached a post-mortem of our weekend problems... Summary is last paragraph in red. Regards -----Original Message----- From: Terech, Henry Sent: Sunday, June 10, 2001 11:23 PM To: Bibi, Philippe A.; Donovan, Bill Cc: Kean, Steven Subject: Enron Center North/Enron Center South Damage & Restoration Overview Bill/Philippe- Below is a summary of the major problems which occurred the past 96 hours and the steps taken to mitigate the damage and risk. Enron Center North (1400 Smith) As a result of the flooding downtown, approximately six skyscrapers had their electrical vaults flooded and began to cause cascading problemd in the electrical grid downtown. The net result was the loss of one of the two 35kv circuits which serve 1400 Smith. We (engineers, EES Maint, electrical subs) believe at the time the circuits blew we shifted from three phase to single phase power, which caused an imbalance in the load to the building. Consequently, UPS "C" which serves a portion of the Lv 34 Data Ctr, the Rolm phone switch, and a protion of the Traders floors lost a circuitboard, and blew two internal fuses. This power degradation caused PDU "S" on Lv 34 to trip its main breaker and shutdown power load to servers and equipment on this PDU. This occurred shortly after Noon on Saturday. Response/Restoration < UPS "D" transferred load from UPS "C" as the N+1 back up and carried the full load of the UPS loss as designed < EPSC/electrical sub (Henderson Electric); EES Maint; IT worked together to power down systems to begin repair work by 4:00pm Saturday < EPSC/EES Maint coordinated with Pillar/Henderson Elec to obtain replacement parts and begin the repairs. By 9:30pm all repairs to UPS"C" had been made and the system bought back on line. As a measure of extra protection the back up genset for Enron Ctr North was in operation < Reliant Energy restored the circuit at approximately 10:30pm Saturday evening < EPSC/EES Maint and subs conducted a thorough review of all electrical and back up systems Saturday evening < Enron Network IT Operations began restoration of all affected systems Saturday evening < EPSC/EES Maint obtained diesel fuel and topped off all gensets Saturday in case of additional problems and due to fuel consumption during the outage Other Building damage was limited to flooding in the tunnels, and water damage/flooding to portions of the Body Shop on Lv B-1 Enron Center South (1500 Louisiana) Impact/Damage Summary < Levels 8 & 9 received very limited amounts of water from the core area elevator shafts and exterior curtainwall. There was no exposure to water on any systems or network cabling during the entire period. (Response) Crews contained and cleaned all water as it traced into the building. < ALL IDF closets on Lvs 3-5 Traders Floors remained dry throughout the duration of the rains/flooding on Friday & Saturday < Lv 6 received some water from Lv 7. (Response) The area was contained and cleaned. No additional damage beyond Tuesday evenings flooding occurred < Due to extreme street flooding and water pressure, a 10" sanitary sewer line tied into ECS burst on Friday evening causing extensive flooding of Lv B-1 area (approximately 4-6 inches of water in the basement. (Response) Clark & Way Engr was able to install a temporary plug in the line to stop the on going flood waters from penetrating the basement. Clean up has continued throughout the weekend < Due to the extreme flooding and water pressure, the water seal of the Reliant Vault burst early Saturday and flooded the electrical vault room with 24-30" of water. This resulted in a complete loss of power at approximately 1:30pm on Staurday afternoon. The back up systems engaged and operated per design to support the Data Ctr and Traders Floors. The temporary cooling tower went off line. Prior to the flooding Clark had been in the process of tieing in the temporary feed into the back up system by 7-1-01 in prep for the migration (Response) * Clark used back up water pumps to pump out and clean the Reliant electrical vault, as well as maintained operation of the back up systems * The Data Ctr AHU's were kept on line with the dehumidifiers/blowers to modulate the airflow in the Data Ctr. EPSC had Way Engr techs use temp/humidity probes to monitor the areas. The recordings indicated temps on Lvs 8 & 9 did not exceed 78 degrees and relative humidity did not exceed 68% during the outage * Clark/Fisk Elec/EPSC coordinated desing efforts and obtained gear on an emergency basis and completed the tie in off the temporary coolong tower to the generators by 10:30pm * Reliant Energy is still cleaning and replacing gear in th vault room. Presntly (AT 11:00PM Sunday) we have one circuit operational in the building and 50% of the Reliant gear on line). We are still operating all systems safely on the back up sytems for the past 36 hours. It is expected Relaint will be complete by 7:00am. Clark; the engineers and MEP subs shall be conducting a thorough system review and repirs as required of all systems throughout the building during the coming week. In summary, the back up systems in both facilities did engage and operate as designed; however, due to the extreme conditons and stress placed upon the systems the facility held up extremely well. I would also like to note throughout the ordeal beginning Friday evening through Sunday, everyone (Enron NetWorks;EES FacilitiesClark Const;Way Engring;Fisk Electric;Henderson Electric; KW-Pillar;Hines) did an outstanding job under enourmous pressure and the most severe conditions I have seen in downtown Houston. Communication, around the clock work and cool heads managed to assess damage and develop action/restoration plans to facilitate repairs for Monday business Let me know if you need further details on specific issues or items. regards- Henry
|