September 11th

The Tribute in Light on September 11, 2014, on the thirteenth anniversary of the attacks, seen from Bayonne, New Jersey. The tallest building in the picture is the new One World Trade Center.
The Tribute in Light on September 11, 2014.

One early Autumn morning, American’s perception of its place in the world was irrevocably altered by four hijacked airplanes; while the devastation and emotional toll was immediate, they have had a profound and lasting impact on the psychology of the nation.

Speakeasy’s headquarters were located across the nation, but we had thousands of customers on the East Coast, many of whom were served via our backhaul located at 25 Broadway, just blocks away from the World Trade Center in New York City.

Due to the proximity, we experienced a major outage that impacted our customers and their families; while the first responders were searching the rubble for survivors, Speakeasy’s crew was working hard to maintain and then reestablish connectivity in order to enable our customers to communicate with and stay up to date on the news as it unfolded.

Shortly after September 11th, the fallout began to hit the central office near the World Trade Center towers. This is how we handled it:

  1. The central office near the towers went down and we lost one of our NY circuits
  2. We migrated all the customers on that circuit to the national circuit and rate limited the connection
  3. The power substation for Telehouse was abandoned by Verizon and failed; it was flooded with firefighting water
  4. Telehouse had to switch onto generator with roughly 70 hours of fuel
  5. We scrambled to get additional DS3 capacity at our facilities in Boston & Washington DC
  6. Generators overheated at Telehouse; Con Edison brings in a backup, and we keep on stumbling along
  7. On Friday night, everything finally fell apart, Internap died, and multiple DS3 circuits failed
  8. We initiated an emergency migration; Tim Hurley saved the day with his mighty automated migration tools
  9. The long cleanup began

— Daniel Pickford

Here are emails showing the initial response right after the attacks, and how the services degraded over the subsequent days, due largely to the debris from the fallen towers.

On Wed, 12 Sep 2001, Danny Pickford wrote:
We are on Generator, with an ETR of this weekend at best.
Here it is from the Horses mouth:

> "Primary concern to all is certainly the restoration of commercial power
> and our generators and fuel supply. It is our understanding that the
> substation serving 25 Broadway failed when debris from WTC 7 fell on top
> of it. It appears likely that there will be no commercial power before
> this weekend. At the present rate of use, we expect our fuel supply to
> last at least until Friday noon. We have arranged for additional fuel
> delivery from a supplier BUT need approval for the actual delivery from
> the police and emergency teams. We are actively pursuing this through
> several channels but do not anticipate quick approval. We will keep you
> informed of our progress."

Once the CO that serves our second DS3 is stable we will migrate the
customers back to that circuit, waiting for good info. That CO has power
now, at least.

---------- Forwarded message ----------
Date: Wed, 12 Sep 2001 09:22:11 -0400
From: E. Mule <mule@telehouse.com>
To: th-all@mail1.telehouse.com
Subject: Important Update for Telehouse 25 Broadway Clients

To our 25 Broadway clients:

Here's an update with regard to power at 25 Broadway...

The commercial power supply to 25 Broadway was lost at about 16:35 EDT
yesterday, Sept. 11. Telehouse emergency generators picked up the load and
as far as we have been able to determine, there was no impact on our
customers' loads. As of 07:35 EDT this morning, our emergency power system
has furnished 15 hours of uninterrupted backup power. It continues to
operate normally, providing support for our clients' equipment.

We are conservatively stating that we have sufficient fuel for 24 hours or
more, but actual fuel usage seems to indicate that could possibly continue
for as long as 70 hours. Unfortunately, if the commercial power outage
continues for a longer period, we may not be able to replenish the fuel
given the state of the streets, etc. in the disaster area. We'll watch this
situation closely.

Further complicating issues, governmental authorities have evacuated the
entire area of lower Manhattan (in which the 25 Broadway facility is
located) and are barring anyone from entering that part of the city. We
have kept essential staff on-site who are continuing to monitor the plant
and its performance. We must advise you, however, that authorities may
eventually force our staff to leave for safety reasons. Should an
evacuation be enforced, our ability to continue to provide emergency power
would be impacted.

To repeat, we are presently operating normally, and expect to continue. We
will notify you as soon as possible when commercial power is restored, or if
there is any significant change from the current status.

Our Customer Assistance Center (CAC) is staffed 24/7. Please feel free to
contact them at cac@telehouse.com, (718) 355-2525 or (718) 370-9663 at any
time if you have questions.

Danny Pickford
Speakeasy Network Janitor
P:206.971.5142
C:206.396.5926

From danny@speakeasy.net Thu Sep 13 14:38:20 2001 -0700
Date: Thu, 13 Sep 2001 14:38:20 -0700 (PDT)
From: Danny Pickford <danny@speakeasy.net>
To: Steven Shults - Speakeasy Domain Hosting <steven@speakeasy.net>
cc: root <root@speakeasy.net>, <spkleads@speakeasy.net>
Subject: Re: [spkleads] NYC
In-Reply-To: <Pine.LNX.4.33L2.0109131427410.25867-100000@web0.speakeasy.net> 
Message-ID: <Pine.LNX.4.33L2.0109131437060.3150-100000@web0.speakeasy.net> 
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII
Status: O
X-Status:
X-Keywords:
X-UID: 2567

Yes New York is going down, the national and local circuits.

Awaiting Status report on ETR, will take a full day to move the customers
over Covads ATM backhaul, thus waiting to see how damaged the generator
is before performing an emergency migration.

On Thu, 13 Sep 2001, Steven Shults - Speakeasy Domain Hosting wrote:

> so the nyc national backhaul pop and all circuits on it are going down
> indefinately?
> > (going to be updating vmail system status, need more specifics, > thanks...)
> > > -=-=-
> Steven Shults - Speakeasy Network Inc - Domain Name Services
> steven@speakeasy.net
> www.speakeasy.net/hosting > > "There is no way to Peace. Peace is the Way." - Traditional
> -=-=-
> spksls
> On Thu, 13 Sep 2001, Danny Pickford wrote:
> > > Date: Thu, 13 Sep 2001 14:23:43 -0700 (PDT)
> > From: Danny Pickford <danny@speakeasy.net
> > > To: sst@speakeasy.net, root <root@speakeasy.net
> > > Cc: spkleads@speakeasy.net
> > Subject: [spkleads] NYC
> > > > > > The generator in NY is overheating.
> > > > New York is going down in about 10 minutes.
> > > > All connectivity and services in New York will be discontinued for the
> > present.
> > > > Danny Pickford
> > Speakeasy Network Janitor
> > P:206.971.5142
> > C:206.396.5926

From danny@speakeasy.net Fri Sep 14 08:09:42 2001 -0700
Date: Fri, 14 Sep 2001 08:09:42 -0700 (PDT)
From: Danny Pickford <danny@speakeasy.net>
To: <sst@speakeasy.net> cc: <spkleads@speakeasy.net>
Subject: New York City.
Message-ID: <Pine.LNX.4.33L2.0109140801540.31264-100000@web0.speakeasy.net>
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII
Status: O
X-Status:
X-Keywords:
X-UID: 2593

Currently we have an estimated time to repair of between 4-6 hours.
New Yorks Edison Electric has brought a generator on site in front
of the building and is attempting to wire it directly into the
facilities main transformer. This is estimated to take another
4-6 hours.

Also, the manufacturer of the original generator is going to deliver
a new water pump to the facility to repair the failed part, this will
be ready tomorrow morning as a backup plan. The reason we are suffering
this outage is as follows. Although we have capacity and could move the
customers in about 10 minutes across our network (with the help of Internap),
it would take Covads provisioning system about 24 hours to move the customers
to the new POPs.

There is no manual recourse for Covad. It would then take 24 hours to move
them back. The hope is that Telehouse will restore power quicker that the
outages involved in moving the customers. Covad also does not have enough
capacity for all of the New York customers in the facilities where we can
handle the additional customers.

I appreciate everyones patience, and we are all hoping that power is
restored soon.

Danny Pickford
Speakeasy Network Janitor
P:206.971.5142
C:206.396.5926

From danny@speakeasy.net Sat Sep 15 10:55:55 2001 -0700
Date: Sat, 15 Sep 2001 10:55:55 -0700 (PDT)
From: Danny Pickford <danny@speakeasy.net>
To: <sst@speakeasy.net> cc: <spkleads@speakeasy.net>, <not@speakeasy.net>
Subject: Latest NYC Update.
Message-ID: <Pine.LNX.4.33L2.0109151024250.12829-100000@web0.speakeasy.net>
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII
Status: O
X-Status:
X-Keywords:
X-UID: 2651

Currently all the New York National Customers are being routed
to the second DS3 in WDC. This is happening because once power
was restored 2 of our 3 DS3's were not available.

This is supposedly because one of the CO's the circuits were routed
through was damaged in recovery efforts. All of the Speakeasy Equipment
is up and operational, however it doesn't do us much good when connectivity
to Covad is down. The second NY redback is operating normally with no problems.

All the customers are currently built on the WDC Redback; however only about
1/5 of the customers are migrated across Covads ATM Network. To further complicate
things, Internap is still in the process of performing an emergency routing switch.
This is not completed yet, but it is in progress and will be completed relatively
soon. This means there is no IP layer connectivity to the endusers and thus no Internet.

The migration should be completed in 8 hours or so, if all goes well. Once Internap
is up the migrated customers will just start working. Moses is not updated with
the information yet, so automated build tools will not help the customers
affected by this transition. The data in Moses will be updated once all emergency
migration details are finalized.

Finally, we are still severely straining the resources available on the DS3 capacity;
and we all customer who are not local to WDC (the New York National Customers,
and the NYC-2 local customers) are currently rate limited to guarantee
connectivity for all users. There is no easy solution to this issues, as we have
no spare sms-1800's with spare circuits available.

Danny Pickford
Speakeasy Network Janitor
P:206.971.5142
C:206.396.5926

From danny@speakeasy.net Sat Sep 15 12:25:53 2001 -0700
Date: Sat, 15 Sep 2001 12:25:53 -0700 (PDT)
From: Danny Pickford <danny@speakeasy.net>
To: Tim Hurley <timofh@speakeasy.net>
Subject: Re: Latest NYC Update.
In-Reply-To: <Pine.LNX.4.33L2.0109151222250.18389-100000@web0.speakeasy.net>
Message-ID: <Pine.LNX.4.33L2.0109151224020.12829-100000@web0.speakeasy.net>
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII
Status: O
X-Status:
X-Keywords:
X-UID: 2654

Please!!!! I'm dying over here!

Call, I need some serious help.

On Sat, 15 Sep 2001, Tim Hurley wrote:

> > sorry, just woke up. need any help?

> > On Sat, 15 Sep 2001, Danny Pickford wrote:

From danny@speakeasy.net Sat Sep 15 15:59:50 2001 -0700 Date:
Sat, 15 Sep 2001 15:59:50 -0700 (PDT)
From: Danny Pickford <danny@speakeasy.net> To: <not@speakeasy.net> cc: Mike <mike@speakeasy.net>
Subject: Final Steps.
Message-ID: <Pine.LNX.4.33L2.0109151547120.12829-100000@web0.speakeasy.net>
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII
Status: O
X-Status:
X-Keywords:
X-UID: 2668

Covad has completed the moving of all the National customers to the WDC-2,
and is now moving the NYC-2 customers over to WDC-2. This will take another
3-4 hours for the 2503 customers.

Once this is completed we only have one step remaining; moving the customers
from NYC-3 to the BOS-1 circuit. This final move will take another 3-4 hours
to complete, and in order to minimize the impact on our customers, we will
coordinate the Internap/Covad changeover to happen late tonight, probably
at about 10pm our time.

The reason for this delay is that Telehouse is up again, and they probably
have enough fuel to last a little while, and rather than take the customers
down in the evening peak, it will be much better to take care of them later
tonight and move them to a stable environment. Tim, can you take care of the
Redback and checking the customers are bound once the migration is completed?

Covad is prepped for the move, worth calling the NOC and introducing yourself
and giving them your contact number. Jack is currently taking care of this
on Covads end.

Chris, same again with the Internap side of things? All Internap needs to do is
move the 2 routes 66.92.96.0/19 and 64.81.192/19 to spk-1-bos, and the appropriate
checking to see that all the routes are in place. Internap is also aware that they
need to do this, and Chris Ely has been handling the situation. Prolly should touch
bases with him on timing. It is important that these 2 events happen as close as
possible to each other. Let me know if any issues, and I will be home (though sleeping).

Danny Pickford
Speakeasy Network Janitor
P:206.971.5142
C:206.396.5926

--------- Forwarded message ----------
Date: Sat, 15 Sep 2001 16:24:40 -0700 (PDT)
From: Danny Pickford <danny@speakeasy.net>
To: Mike <mike@speakeasy.net>
Subject: New York, Part XII

Dear Speakeasy Customers:

After a series of valiant yet failed attempts by many tireless individuals
representing dozens of major companies, the New York Telehouse facility
and location of our New York Data Center (aka POP or "Point of Presence"),
continues to demonstrate major instability.

For this reason, Speakeasy has decided to engage in a full contingency plan
to move customers to alternative facilities effective immediately. After suffering
multiple generator failures, multiple CO failures, and the loss of two-thirds
of our backhaul capacity in this region, it has become clear in our view that it is
time to move our customers to alternate facilities. Until now, in our view, the odds
of restoring service in New York relative to the down-time customers may have incurred
in the event of a full re-routing, were in the favor of remaining in New York.

The tide has now turned and re-routing has already begun. In an effort to guarantee our
customers the best possible service, we are moving all the customers who were routed
through New York, to the closest next available POP's, Washington DC and Boston.

Customers who were on our first New York DSL Router are in the process of being
moved to the Washington DC POP, this circuit will remain heavily utilized until
we are able to bring up the Philadelphia POP, and at that time we project a full
restoration of the best possible performance for all of these customers. All customers
who were on the second New York DSL Router are being moved to Boston, and will
experience no performance impediments apart from a very slight increase in latency
(transparent to the majority of services such as Email, Web browsing and most
file transfer programs).

For your convenience, we have gone to great lengths to ensure all customers
IP addresses and DNS settings will remain the same. In short, your service will
be restored "transparently" apart from the downtime during the transition.

The transition for the remaining customers will begin at 2am EST and is estimated
to be completed by 5am EST. All updates will be posted to our real-time network
status page located at http://www.speakeasy.net/systemstatus.

We have no good estimates on what it will take to bring our New York City POP
back in operation, but an optimistic expectation is that we will resume normal
service in a month or so. The local Power station is considered irreparable,
and the Central Office that contains 2 of our circuits, is buried in rubble,
and filled with water and chemicals as a result of the disaster and the ensuing
events.

We deeply regret the tragedy that has undoubtedly touched all of your lives and
especially those impacted more personally by the heavy toll paid by civilians,
military and emergency workers. The loss of property, time or services clearly
pales in comparison, however, as your Internet Service Provider in the midst of
this crisis, we will do everything we can personally and exhaust all viable
avenues to ensure you receive the best services we can provide.

We know that many of you have been relying on Internet service to stay informed
and communicate with loved-ones. Thank you for your patience and understanding
as we work diligently to restore your service and in cooperation with our partners,
establish a reliable network infrastructure platform for use over the next several weeks.

Sincerely,

Danny Pickford
Director of Network Operations
Speakeasy Network

Michael Apgar
President/CEO
Speakeasy Network

My most memorable experience is one that I wish I could forget — the fallout from 9/11 and all the phone calls we got from New York City. Customers were begging to get their DSL back up so they could look for friends and relatives that were missing that day. It was very sad and I’ll never forget it.

— Kevin Albers