One early Autumn morning, American’s perception of its place in the world was irrevocably altered by four hijacked airplanes; while the devastation and emotional toll was immediate, they have had a profound and lasting impact on the psychology of the nation.
Speakeasy’s headquarters were located across the nation, but we had thousands of customers on the East Coast, many of whom were served via our backhaul located at 25 Broadway, just blocks away from the World Trade Center in New York City.
Due to the proximity, we experienced a major outage that impacted our customers and their families; while the first responders were searching the rubble for survivors, Speakeasy’s crew was working hard to maintain and then reestablish connectivity in order to enable our customers to communicate with and stay up to date on the news as it unfolded.
Shortly after September 11th, the fallout began to hit the central office near the World Trade Center towers. This is how we handled it:
- The central office near the towers went down and we lost one of our NY circuits
- We migrated all the customers on that circuit to the national circuit and rate limited the connection
- The power substation for Telehouse was abandoned by Verizon and failed; it was flooded with firefighting water
- Telehouse had to switch onto generator with roughly 70 hours of fuel
- We scrambled to get additional DS3 capacity at our facilities in Boston & Washington DC
- Generators overheated at Telehouse; Con Edison brings in a backup, and we keep on stumbling along
- On Friday night, everything finally fell apart, Internap died, and multiple DS3 circuits failed
- We initiated an emergency migration; Tim Hurley saved the day with his mighty automated migration tools
- The long cleanup began
— Daniel Pickford
Here are emails showing the initial response right after the attacks, and how the services degraded over the subsequent days, due largely to the debris from the fallen towers.
On Wed, 12 Sep 2001, Danny Pickford wrote: We are on Generator, with an ETR of this weekend at best. Here it is from the Horses mouth: > "Primary concern to all is certainly the restoration of commercial power > and our generators and fuel supply. It is our understanding that the > substation serving 25 Broadway failed when debris from WTC 7 fell on top > of it. It appears likely that there will be no commercial power before > this weekend. At the present rate of use, we expect our fuel supply to > last at least until Friday noon. We have arranged for additional fuel > delivery from a supplier BUT need approval for the actual delivery from > the police and emergency teams. We are actively pursuing this through > several channels but do not anticipate quick approval. We will keep you > informed of our progress." Once the CO that serves our second DS3 is stable we will migrate the customers back to that circuit, waiting for good info. That CO has power now, at least. ---------- Forwarded message ---------- Date: Wed, 12 Sep 2001 09:22:11 -0400 From: E. Mule <email@example.com> To: firstname.lastname@example.org Subject: Important Update for Telehouse 25 Broadway Clients To our 25 Broadway clients: Here's an update with regard to power at 25 Broadway... The commercial power supply to 25 Broadway was lost at about 16:35 EDT yesterday, Sept. 11. Telehouse emergency generators picked up the load and as far as we have been able to determine, there was no impact on our customers' loads. As of 07:35 EDT this morning, our emergency power system has furnished 15 hours of uninterrupted backup power. It continues to operate normally, providing support for our clients' equipment. We are conservatively stating that we have sufficient fuel for 24 hours or more, but actual fuel usage seems to indicate that could possibly continue for as long as 70 hours. Unfortunately, if the commercial power outage continues for a longer period, we may not be able to replenish the fuel given the state of the streets, etc. in the disaster area. We'll watch this situation closely. Further complicating issues, governmental authorities have evacuated the entire area of lower Manhattan (in which the 25 Broadway facility is located) and are barring anyone from entering that part of the city. We have kept essential staff on-site who are continuing to monitor the plant and its performance. We must advise you, however, that authorities may eventually force our staff to leave for safety reasons. Should an evacuation be enforced, our ability to continue to provide emergency power would be impacted. To repeat, we are presently operating normally, and expect to continue. We will notify you as soon as possible when commercial power is restored, or if there is any significant change from the current status. Our Customer Assistance Center (CAC) is staffed 24/7. Please feel free to contact them at email@example.com, (718) 355-2525 or (718) 370-9663 at any time if you have questions. Danny Pickford Speakeasy Network Janitor P:206.971.5142 C:206.396.5926 From firstname.lastname@example.org Thu Sep 13 14:38:20 2001 -0700 Date: Thu, 13 Sep 2001 14:38:20 -0700 (PDT) From: Danny Pickford <email@example.com> To: Steven Shults - Speakeasy Domain Hosting <firstname.lastname@example.org> cc: root <email@example.com>, <firstname.lastname@example.org> Subject: Re: [spkleads] NYC In-Reply-To: <Pine.LNX.4.33L2.email@example.com> Message-ID: <Pine.LNX.4.33L2.firstname.lastname@example.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Status: O X-Status: X-Keywords: X-UID: 2567 Yes New York is going down, the national and local circuits. Awaiting Status report on ETR, will take a full day to move the customers over Covads ATM backhaul, thus waiting to see how damaged the generator is before performing an emergency migration. On Thu, 13 Sep 2001, Steven Shults - Speakeasy Domain Hosting wrote: > so the nyc national backhaul pop and all circuits on it are going down > indefinately? > > (going to be updating vmail system status, need more specifics, > thanks...) > > > -=-=- > Steven Shults - Speakeasy Network Inc - Domain Name Services > email@example.com > www.speakeasy.net/hosting > > "There is no way to Peace. Peace is the Way." - Traditional > -=-=- > spksls > On Thu, 13 Sep 2001, Danny Pickford wrote: > > > Date: Thu, 13 Sep 2001 14:23:43 -0700 (PDT) > > From: Danny Pickford <firstname.lastname@example.org > > > To: email@example.com, root <firstname.lastname@example.org > > > Cc: email@example.com > > Subject: [spkleads] NYC > > > > > > The generator in NY is overheating. > > > > New York is going down in about 10 minutes. > > > > All connectivity and services in New York will be discontinued for the > > present. > > > > Danny Pickford > > Speakeasy Network Janitor > > P:206.971.5142 > > C:206.396.5926 From firstname.lastname@example.org Fri Sep 14 08:09:42 2001 -0700 Date: Fri, 14 Sep 2001 08:09:42 -0700 (PDT) From: Danny Pickford <email@example.com> To: <firstname.lastname@example.org> cc: <email@example.com> Subject: New York City. Message-ID: <Pine.LNX.4.33L2.firstname.lastname@example.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Status: O X-Status: X-Keywords: X-UID: 2593 Currently we have an estimated time to repair of between 4-6 hours. New Yorks Edison Electric has brought a generator on site in front of the building and is attempting to wire it directly into the facilities main transformer. This is estimated to take another 4-6 hours. Also, the manufacturer of the original generator is going to deliver a new water pump to the facility to repair the failed part, this will be ready tomorrow morning as a backup plan. The reason we are suffering this outage is as follows. Although we have capacity and could move the customers in about 10 minutes across our network (with the help of Internap), it would take Covads provisioning system about 24 hours to move the customers to the new POPs. There is no manual recourse for Covad. It would then take 24 hours to move them back. The hope is that Telehouse will restore power quicker that the outages involved in moving the customers. Covad also does not have enough capacity for all of the New York customers in the facilities where we can handle the additional customers. I appreciate everyones patience, and we are all hoping that power is restored soon. Danny Pickford Speakeasy Network Janitor P:206.971.5142 C:206.396.5926 From email@example.com Sat Sep 15 10:55:55 2001 -0700 Date: Sat, 15 Sep 2001 10:55:55 -0700 (PDT) From: Danny Pickford <firstname.lastname@example.org> To: <email@example.com> cc: <firstname.lastname@example.org>, <email@example.com> Subject: Latest NYC Update. Message-ID: <Pine.LNX.4.33L2.firstname.lastname@example.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Status: O X-Status: X-Keywords: X-UID: 2651 Currently all the New York National Customers are being routed to the second DS3 in WDC. This is happening because once power was restored 2 of our 3 DS3's were not available. This is supposedly because one of the CO's the circuits were routed through was damaged in recovery efforts. All of the Speakeasy Equipment is up and operational, however it doesn't do us much good when connectivity to Covad is down. The second NY redback is operating normally with no problems. All the customers are currently built on the WDC Redback; however only about 1/5 of the customers are migrated across Covads ATM Network. To further complicate things, Internap is still in the process of performing an emergency routing switch. This is not completed yet, but it is in progress and will be completed relatively soon. This means there is no IP layer connectivity to the endusers and thus no Internet. The migration should be completed in 8 hours or so, if all goes well. Once Internap is up the migrated customers will just start working. Moses is not updated with the information yet, so automated build tools will not help the customers affected by this transition. The data in Moses will be updated once all emergency migration details are finalized. Finally, we are still severely straining the resources available on the DS3 capacity; and we all customer who are not local to WDC (the New York National Customers, and the NYC-2 local customers) are currently rate limited to guarantee connectivity for all users. There is no easy solution to this issues, as we have no spare sms-1800's with spare circuits available. Danny Pickford Speakeasy Network Janitor P:206.971.5142 C:206.396.5926 From email@example.com Sat Sep 15 12:25:53 2001 -0700 Date: Sat, 15 Sep 2001 12:25:53 -0700 (PDT) From: Danny Pickford <firstname.lastname@example.org> To: Tim Hurley <email@example.com> Subject: Re: Latest NYC Update. In-Reply-To: <Pine.LNX.4.33L2.firstname.lastname@example.org> Message-ID: <Pine.LNX.4.33L2.email@example.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Status: O X-Status: X-Keywords: X-UID: 2654 Please!!!! I'm dying over here! Call, I need some serious help. On Sat, 15 Sep 2001, Tim Hurley wrote: > > sorry, just woke up. need any help? > > On Sat, 15 Sep 2001, Danny Pickford wrote: From firstname.lastname@example.org Sat Sep 15 15:59:50 2001 -0700 Date: Sat, 15 Sep 2001 15:59:50 -0700 (PDT) From: Danny Pickford <email@example.com> To: <firstname.lastname@example.org> cc: Mike <email@example.com> Subject: Final Steps. Message-ID: <Pine.LNX.4.33L2.firstname.lastname@example.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Status: O X-Status: X-Keywords: X-UID: 2668 Covad has completed the moving of all the National customers to the WDC-2, and is now moving the NYC-2 customers over to WDC-2. This will take another 3-4 hours for the 2503 customers. Once this is completed we only have one step remaining; moving the customers from NYC-3 to the BOS-1 circuit. This final move will take another 3-4 hours to complete, and in order to minimize the impact on our customers, we will coordinate the Internap/Covad changeover to happen late tonight, probably at about 10pm our time. The reason for this delay is that Telehouse is up again, and they probably have enough fuel to last a little while, and rather than take the customers down in the evening peak, it will be much better to take care of them later tonight and move them to a stable environment. Tim, can you take care of the Redback and checking the customers are bound once the migration is completed? Covad is prepped for the move, worth calling the NOC and introducing yourself and giving them your contact number. Jack is currently taking care of this on Covads end. Chris, same again with the Internap side of things? All Internap needs to do is move the 2 routes 18.104.22.168/19 and 64.81.192/19 to spk-1-bos, and the appropriate checking to see that all the routes are in place. Internap is also aware that they need to do this, and Chris Ely has been handling the situation. Prolly should touch bases with him on timing. It is important that these 2 events happen as close as possible to each other. Let me know if any issues, and I will be home (though sleeping). Danny Pickford Speakeasy Network Janitor P:206.971.5142 C:206.396.5926 --------- Forwarded message ---------- Date: Sat, 15 Sep 2001 16:24:40 -0700 (PDT) From: Danny Pickford <email@example.com> To: Mike <firstname.lastname@example.org> Subject: New York, Part XII Dear Speakeasy Customers: After a series of valiant yet failed attempts by many tireless individuals representing dozens of major companies, the New York Telehouse facility and location of our New York Data Center (aka POP or "Point of Presence"), continues to demonstrate major instability. For this reason, Speakeasy has decided to engage in a full contingency plan to move customers to alternative facilities effective immediately. After suffering multiple generator failures, multiple CO failures, and the loss of two-thirds of our backhaul capacity in this region, it has become clear in our view that it is time to move our customers to alternate facilities. Until now, in our view, the odds of restoring service in New York relative to the down-time customers may have incurred in the event of a full re-routing, were in the favor of remaining in New York. The tide has now turned and re-routing has already begun. In an effort to guarantee our customers the best possible service, we are moving all the customers who were routed through New York, to the closest next available POP's, Washington DC and Boston. Customers who were on our first New York DSL Router are in the process of being moved to the Washington DC POP, this circuit will remain heavily utilized until we are able to bring up the Philadelphia POP, and at that time we project a full restoration of the best possible performance for all of these customers. All customers who were on the second New York DSL Router are being moved to Boston, and will experience no performance impediments apart from a very slight increase in latency (transparent to the majority of services such as Email, Web browsing and most file transfer programs). For your convenience, we have gone to great lengths to ensure all customers IP addresses and DNS settings will remain the same. In short, your service will be restored "transparently" apart from the downtime during the transition. The transition for the remaining customers will begin at 2am EST and is estimated to be completed by 5am EST. All updates will be posted to our real-time network status page located at http://www.speakeasy.net/systemstatus. We have no good estimates on what it will take to bring our New York City POP back in operation, but an optimistic expectation is that we will resume normal service in a month or so. The local Power station is considered irreparable, and the Central Office that contains 2 of our circuits, is buried in rubble, and filled with water and chemicals as a result of the disaster and the ensuing events. We deeply regret the tragedy that has undoubtedly touched all of your lives and especially those impacted more personally by the heavy toll paid by civilians, military and emergency workers. The loss of property, time or services clearly pales in comparison, however, as your Internet Service Provider in the midst of this crisis, we will do everything we can personally and exhaust all viable avenues to ensure you receive the best services we can provide. We know that many of you have been relying on Internet service to stay informed and communicate with loved-ones. Thank you for your patience and understanding as we work diligently to restore your service and in cooperation with our partners, establish a reliable network infrastructure platform for use over the next several weeks. Sincerely, Danny Pickford Director of Network Operations Speakeasy Network Michael Apgar President/CEO Speakeasy Network
My most memorable experience is one that I wish I could forget — the fallout from 9/11 and all the phone calls we got from New York City. Customers were begging to get their DSL back up so they could look for friends and relatives that were missing that day. It was very sad and I’ll never forget it.
— Kevin Albers