Anti-Metallica DDoS – A Tribute to Speakeasy

After their infamous battle with Napster, Metallica decided to take the reigns of how their music was distributed digitally by partnering with Speakeasy to release it on the internet.

We built demand and generated excitement about the release of Metallica’s St. Anger album via Speakeasy-hosted website, counting down to the release day.

Of course, given that they’d turned off a contingent of technically-skilled music fans, the only response that could have happened is what did: A massive Distributed Denial of Service (DDoS) attack against Speakeasy the very moment that the album was released.

Here’s how we handled the DDOS Attack we experienced upon launching Metallica’s album St. Anger:

Big countdown on a Speakeasy-hosted St. Anger website

The moment the countdown gets to zero, a DDoS is triggered

It ultimately gets up to about 10gbps or so (which was huge back then)

AT&T and Internap’s border routers are all overloaded

The platform, which managed the secure download of media, was knocked out

We had a firewalls etc. which caught all the DDoS traffic, but because our little OC3 was filled up with the DDoS traffic, it didn’t matter as we couldn’t send anything out

We eventually gave all the filter rules to the upstream providers so they could block the traffic.

— Daniel Pickford

The below emails detail our preparations and

From danny@speakeasy.net Wed Jun 4 19:09:52 2003 -0700
Date: Wed, 4 Jun 2003 19:09:52 -0700 (PDT)
From: Danny <danny@speakeasy.net>
To: Sandra Eynon <sandra@speakeasy.net>
cc: metallica@speakeasy.net, Kat Oak <bastet@speakeasy.net>
Subject: Re: [metallica] Public Messaging on Vault launch
In-Reply-To: <C8F3842A2BBC994391A829261890F66D850BBA@exchange.speakeasy.net>
Message-ID: <Pine.LNX.4.44.0306041858050.27199-100000@web0.speakeasy.net> MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII
Status: O
X-Status:
X-Keywords:
X-UID: 20

Sandra,

Here is a list of scenarios you should be aware of in regards to events
that we may need to explain to the Media.

Most of these outages will be for short periods of time as we react to the
various attacks, and add capacity/rewrite code as needed, with the
exception of option 6.

We have options in all of these scenarios.

Scenarios (in order of probability):

1. Website is attacked and made inaccessible, because of Denial of
Service (large amounts of traffic flooding the site), resource-allocation
attacks (false web requests render site unusable), or other malicious
activity.
2. Akamai/Platform security compromised, content made available to
small group of clever unauthorized users.
3. Demand is so large that we and/or the platform cannot handle the
load.
a. Akamai Problem.
b. Platform Problem.
c. Speakeasy Problem.
4. -or- Variant #3, load is so high we have intermittent Service
problems at peak traffic times.
5. Website gets hacked, Metallica website turned into Metallica
hatesite and/or content made available to unauthorized users.
6. Coordinated Multiple Attacks from multiple sources with the design
of keeping the website complete unusable for an extended period of time.

On Wed, 4 Jun 2003, Sandra Eynon wrote:

> Here is what we are saying about the launch timeline for the Vault.
>
> Q: When will MetallicaVault.com launch?
> A: Thursday, June 5; 6:00 pm, Pacific Daylight Time
>
> Q: Why isn't the site up and running now?
> A: As you know, the band decided just a few days ago to significantly adjust the release schedule of its CD, from June 10 up to today, June 5. We've been working together to present an excellent online product that reflects this adjusted schedule, ready for fans to enjoy this evening.
>
> Any issues, please let us know.
>
> Thanks,
> Sandra
>
> Sandra Eynon
> Sr. Marcom Manager
> 206.971.5103
> Speakeasy Inc.
> www.speakeasy.net
>
>
>

From danny@speakeasy.net Wed Jun 4 21:51:41 2003 -0700
Date: Wed, 4 Jun 2003 21:51:41 -0700 (PDT)
From: Danny <danny@speakeasy.net>
To: Edward Bender <edward@speakeasy.net>
Subject: RE: update
In-Reply-To: <C8F3842A2BBC994391A829261890F66D89B81A@exchange.speakeasy.net>
Message-ID: <Pine.LNX.4.44.0306042151120.27199-100000@web0.speakeasy.net> MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII
Status: O
X-Status:
X-Keywords:
X-UID: 37

Oh, and Ed. Congrat's on landing a big fish. Looks like we're going to
get this one home fer dinner.

On Wed, 4 Jun 2003, Edward Bender wrote:

> Whats the URL for the test/beta ?
>
>
> -----Original Message-----
> From: Danny Pickford
> Sent: Wednesday, June 04, 2003 9:01 PM
> To: Edward Bender
> Cc: Sean Paul
>
>
> Once the Site is done, and we are satisfied we have it working, thats
> Alpha!
>
> Beta's when other folks get to tell us what we missed!
>
> On Wed, 4 Jun 2003, Edward Bender wrote:
>
> > Alpha tonight rite?
> >
> >
> > -----Original Message-----
> > From: Danny Pickford
> > Sent: Wednesday, June 04, 2003 8:59 PM
> > To: Tim Sale
> > Cc: Stef; Edward Bender; richardb@speakeasy.net; mike; Alan Ramaley;
> > Nick Rossi; Thuong Nguyen; Andy Sodt; Ed Kuerner; Ian Blaine; Sean
> > Paul
> >
> >
> > If your server is live, the content is primed, and the basic system
> > functionality is tested, we should be good to go.
> >
> > On our side are now finishing up the Site content, performing the
> > final hardware/software upgrades at the maintenance window, and at 8am
>
> > we will be doing a final sanity check before getting the Metallica Fan
>
> > Club to beta test the site for us.
> >
> > Once that is done, we'll fix the minor bugs, perform a final set of
> > performance tests, and live at 6pm tommorrow...
> >
> > On Wed, 4 Jun 2003, Tim Sale wrote:
> >
> > > We've verified that LDAP works as expected. We're running some
> > > performance tests now, though more to do with our stuff then yours.
> > > Are you guys waiting on us for anything? When do we
> > alpha/bravo/charlie?
> > >
> > > --tim
> > >
> > >
> > >
> >
> > --
> > Danny Pickford
> > Director of Network Operations
> > Speakeasy Network
> > P:206.971.5142
> > C:206.396.5926
> >
> >
> >
> >
>
> --
> Danny Pickford
> Director of Network Operations
> Speakeasy Network
> P:206.971.5142
> C:206.396.5926
>
>
>
>

--
Danny Pickford
Director of Network Operations
Speakeasy Network
P:206.971.5142
C:206.396.5926

On Fri, 6 Jun 2003, Gretchen wrote:

>
> Hi all,
>
> Again given the launch situation with www.metallilcavault.com we
> are in a position where we all need to be very clear on the
> facts.
>
> Some things that will help us understand the impact as well as
> message to the outside world are clear metrics around what's
> happening.
>
> That said, two priorities for today are:
>
> 1) Sales Mapped against Site Traffic: We need to map CD
> sales data against metallicavault traffic,
> registratations, etc. Reporting on this data now will
> help us understand the trends as issues get resolved.

This is available; What Site Traffic?

1. Speakeasy Website.
2. Metallica Website.
3. Platform Portal.
4. Akamai Downloads.

> 2) Replicating Visitors Experiences: We need to
> understand who's getting to the site and who's not. The
> most comprehensive list we can gather of what it's like
> for people on different Tier 1 providers (i.e. Sprint,
> Level Three, At&T, Comcast, Qwest, etc.). This may also
> help us understand some of the reported bugs.

1. Not Accessible: Sprint, UUnet, ATT, Cable & Wireless, Global
Crossing.

2. Accessible: L3, Verio, Cable & Wireless, Genuity.

> 3) Other Reports, etc that give us insight and metrics on
> this situation.

Can we designate a report owner, as was discussed on the Operational
action items document about a week ago?

> Thanks Gretchen

--
Danny Pickford
Director of Network Operations
Speakeasy Network
P:206.971.5142
C:206.396.5926

From danny@speakeasy.net Fri Jun 6 00:55:44 2003 -0700
Date: Fri, 6 Jun 2003 00:55:44 -0700 (PDT)
From: Danny <danny@speakeasy.net>
To: Mike <mike@speakeasy.net>
Subject: Status.
Message-ID: <Pine.LNX.4.44.0306060047240.27199-100000@web0.speakeasy.net> MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII
Status: O
X-Status:
X-Keywords:
X-UID: 40

1. Site is up and fully functional.

a. Our web system was running at 7% load with about 400
users, pretty damn good. Confident it will scale.
b. Platform; network loading issues resolved, feel confident
that this will scale with us up to about 60% of our current
capacity. No need to shift to CDN at this time.
c. Security/Haxor's; am concerned about what is next. Will
be beefing up sensors and hardening when tools are configured and
available. In pretty good shape.
d. Need to undo some changes made to the system/network architecture to
troubleshoot the platform. Nothing too major; will
coordinate events.

2. Internap is still blocking significant amounts of traffic. 4 out
of 7 providers are null routing us; will modify our outbound traffic not
to use them after I receive an update from the status of the DOS
from Internap in the morning (if necessary).

3. Rebuilding system; have a few ideas, need to redo large components
of the site. Have some decent options. Will discuss further.
Not any one great option; but will do the best we can do with the current
tools.

It was live before midnight!

--
Danny Pickford
Director of Eng & Net Ops
Speakeasy Network
P:206.971.5142
C:206.396.5926