After their infamous battle with Napster, Metallica decided to take the reigns of how their music was distributed digitally by partnering with Speakeasy to release it on the internet.
We built demand and generated excitement about the release of Metallica’s St. Anger album via Speakeasy-hosted website, counting down to the release day.
Of course, given that they’d turned off a contingent of technically-skilled music fans, the only response that could have happened is what did: A massive Distributed Denial of Service (DDoS) attack against Speakeasy the very moment that the album was released.
Here’s how we handled the DDOS Attack we experienced upon launching Metallica’s album St. Anger:
- Big countdown on a Speakeasy-hosted St. Anger website
- The moment the countdown gets to zero, a DDoS is triggered
- It ultimately gets up to about 10gbps or so (which was huge back then)
- AT&T and Internap’s border routers are all overloaded
- The platform, which managed the secure download of media, was knocked out
- We had a firewalls etc. which caught all the DDoS traffic, but because our little OC3 was filled up with the DDoS traffic, it didn’t matter as we couldn’t send anything out
- We eventually gave all the filter rules to the upstream providers so they could block the traffic.
— Daniel Pickford
The below emails detail our preparations and
From email@example.com Wed Jun 4 19:09:52 2003 -0700 Date: Wed, 4 Jun 2003 19:09:52 -0700 (PDT) From: Danny <firstname.lastname@example.org> To: Sandra Eynon <email@example.com> cc: firstname.lastname@example.org, Kat Oak <email@example.com> Subject: Re: [metallica] Public Messaging on Vault launch In-Reply-To: <C8F3842A2BBC994391A829261890F66D850BBA@exchange.speakeasy.net> Message-ID: <Pine.LNX.firstname.lastname@example.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Status: O X-Status: X-Keywords: X-UID: 20 Sandra, Here is a list of scenarios you should be aware of in regards to events that we may need to explain to the Media. Most of these outages will be for short periods of time as we react to the various attacks, and add capacity/rewrite code as needed, with the exception of option 6. We have options in all of these scenarios. Scenarios (in order of probability): 1. Website is attacked and made inaccessible, because of Denial of Service (large amounts of traffic flooding the site), resource-allocation attacks (false web requests render site unusable), or other malicious activity. 2. Akamai/Platform security compromised, content made available to small group of clever unauthorized users. 3. Demand is so large that we and/or the platform cannot handle the load. a. Akamai Problem. b. Platform Problem. c. Speakeasy Problem. 4. -or- Variant #3, load is so high we have intermittent Service problems at peak traffic times. 5. Website gets hacked, Metallica website turned into Metallica hatesite and/or content made available to unauthorized users. 6. Coordinated Multiple Attacks from multiple sources with the design of keeping the website complete unusable for an extended period of time. On Wed, 4 Jun 2003, Sandra Eynon wrote: > Here is what we are saying about the launch timeline for the Vault. > > Q: When will MetallicaVault.com launch? > A: Thursday, June 5; 6:00 pm, Pacific Daylight Time > > Q: Why isn't the site up and running now? > A: As you know, the band decided just a few days ago to significantly adjust the release schedule of its CD, from June 10 up to today, June 5. We've been working together to present an excellent online product that reflects this adjusted schedule, ready for fans to enjoy this evening. > > Any issues, please let us know. > > Thanks, > Sandra > > Sandra Eynon > Sr. Marcom Manager > 206.971.5103 > Speakeasy Inc. > www.speakeasy.net > > > From email@example.com Wed Jun 4 21:51:41 2003 -0700 Date: Wed, 4 Jun 2003 21:51:41 -0700 (PDT) From: Danny <firstname.lastname@example.org> To: Edward Bender <email@example.com> Subject: RE: update In-Reply-To: <C8F3842A2BBC994391A829261890F66D89B81A@exchange.speakeasy.net> Message-ID: <Pine.LNX.firstname.lastname@example.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Status: O X-Status: X-Keywords: X-UID: 37 Oh, and Ed. Congrat's on landing a big fish. Looks like we're going to get this one home fer dinner. On Wed, 4 Jun 2003, Edward Bender wrote: > Whats the URL for the test/beta ? > > > -----Original Message----- > From: Danny Pickford > Sent: Wednesday, June 04, 2003 9:01 PM > To: Edward Bender > Cc: Sean Paul > > > Once the Site is done, and we are satisfied we have it working, thats > Alpha! > > Beta's when other folks get to tell us what we missed! > > On Wed, 4 Jun 2003, Edward Bender wrote: > > > Alpha tonight rite? > > > > > > -----Original Message----- > > From: Danny Pickford > > Sent: Wednesday, June 04, 2003 8:59 PM > > To: Tim Sale > > Cc: Stef; Edward Bender; email@example.com; mike; Alan Ramaley; > > Nick Rossi; Thuong Nguyen; Andy Sodt; Ed Kuerner; Ian Blaine; Sean > > Paul > > > > > > If your server is live, the content is primed, and the basic system > > functionality is tested, we should be good to go. > > > > On our side are now finishing up the Site content, performing the > > final hardware/software upgrades at the maintenance window, and at 8am > > > we will be doing a final sanity check before getting the Metallica Fan > > > Club to beta test the site for us. > > > > Once that is done, we'll fix the minor bugs, perform a final set of > > performance tests, and live at 6pm tommorrow... > > > > On Wed, 4 Jun 2003, Tim Sale wrote: > > > > > We've verified that LDAP works as expected. We're running some > > > performance tests now, though more to do with our stuff then yours. > > > Are you guys waiting on us for anything? When do we > > alpha/bravo/charlie? > > > > > > --tim > > > > > > > > > > > > > -- > > Danny Pickford > > Director of Network Operations > > Speakeasy Network > > P:206.971.5142 > > C:206.396.5926 > > > > > > > > > > -- > Danny Pickford > Director of Network Operations > Speakeasy Network > P:206.971.5142 > C:206.396.5926 > > > > -- Danny Pickford Director of Network Operations Speakeasy Network P:206.971.5142 C:206.396.5926 On Fri, 6 Jun 2003, Gretchen wrote: > > Hi all, > > Again given the launch situation with www.metallilcavault.com we > are in a position where we all need to be very clear on the > facts. > > Some things that will help us understand the impact as well as > message to the outside world are clear metrics around what's > happening. > > That said, two priorities for today are: > > 1) Sales Mapped against Site Traffic: We need to map CD > sales data against metallicavault traffic, > registratations, etc. Reporting on this data now will > help us understand the trends as issues get resolved. This is available; What Site Traffic? 1. Speakeasy Website. 2. Metallica Website. 3. Platform Portal. 4. Akamai Downloads. > 2) Replicating Visitors Experiences: We need to > understand who's getting to the site and who's not. The > most comprehensive list we can gather of what it's like > for people on different Tier 1 providers (i.e. Sprint, > Level Three, At&T, Comcast, Qwest, etc.). This may also > help us understand some of the reported bugs. 1. Not Accessible: Sprint, UUnet, ATT, Cable & Wireless, Global Crossing. 2. Accessible: L3, Verio, Cable & Wireless, Genuity. > 3) Other Reports, etc that give us insight and metrics on > this situation. Can we designate a report owner, as was discussed on the Operational action items document about a week ago? > Thanks Gretchen -- Danny Pickford Director of Network Operations Speakeasy Network P:206.971.5142 C:206.396.5926 From firstname.lastname@example.org Fri Jun 6 00:55:44 2003 -0700 Date: Fri, 6 Jun 2003 00:55:44 -0700 (PDT) From: Danny <email@example.com> To: Mike <firstname.lastname@example.org> Subject: Status. Message-ID: <Pine.LNX.email@example.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Status: O X-Status: X-Keywords: X-UID: 40 1. Site is up and fully functional. a. Our web system was running at 7% load with about 400 users, pretty damn good. Confident it will scale. b. Platform; network loading issues resolved, feel confident that this will scale with us up to about 60% of our current capacity. No need to shift to CDN at this time. c. Security/Haxor's; am concerned about what is next. Will be beefing up sensors and hardening when tools are configured and available. In pretty good shape. d. Need to undo some changes made to the system/network architecture to troubleshoot the platform. Nothing too major; will coordinate events. 2. Internap is still blocking significant amounts of traffic. 4 out of 7 providers are null routing us; will modify our outbound traffic not to use them after I receive an update from the status of the DOS from Internap in the morning (if necessary). 3. Rebuilding system; have a few ideas, need to redo large components of the site. Have some decent options. Will discuss further. Not any one great option; but will do the best we can do with the current tools. It was live before midnight! -- Danny Pickford Director of Eng & Net Ops Speakeasy Network P:206.971.5142 C:206.396.5926