On Wednesday, at about 12:15 pm EST, 1.35 terabits per second of traffic hit the developer platform GitHub all at once. It was the most powerful distributed denial of service attack recorded to date—and it used an increasingly popular DDoS method, no botnet required.
GitHub briefly struggled with intermittent outages as a digital system assessed the situation. Within 10 minutes it had automatically called for help from its DDoS mitigation service, Akamai Prolexic. Prolexic took over as an intermediary, routing all the traffic coming into and out of GitHub, and sent the data through its scrubbing centers to weed out and block malicious packets. After eight minutes, attackers relented and the assault dropped off.
The scale of the attack has few parallels, but a massive DDoS that struck the internet infrastructure company Dyn in late 2016 comes close. That barrage peaked at 1.2 terabits per second and caused connectivity issues across the US as Dyn fought to get the situation under control.
“We modeled our capacity based on fives times the biggest attack that the internet has ever seen,” Josh Shaul, vice president of web security at Akamai told WIRED hours after the GitHub attack ended. “So I would have been certain that we could handle 1.3 Tbps, but at the same time we never had a terabit and a half come in all at once. It’s one thing to have the confidence. It’s another thing to see it actually play out how you’d hope.”
Akamai defended against the attack in a number of ways. In addition to Prolexic’s general DDoS defense infrastructure, the firm had also recently implemented specific mitigations for a type of DDoS attack stemming from so-called memcached servers. These database caching systems work to speed networks and websites, but they aren’t meant to be exposed on the public internet; anyone can query them, and they’ll likewise respond to anyone. About 100,000 memcached servers, mostly owned by businesses and other institutions, currently sit exposed online with no authentication protection, meaning an attacker can access them and send them a special command packet that the server will respond to with a much larger reply.
Unlike the formal botnet attacks used in large DDoS efforts, like against Dyn and the French telecom OVH, memcached DDoS attacks don’t require a malware-driven botnet. Attackers simply spoof the IP address of their victim and send small queries to multiple memcached servers—about 10 per second per server—that are designed to elicit a much larger response. The memcached systems then return 50 times the data of the requests back to the victim.
Known as an amplification attack, this type of DDoS has shown up before. But as internet service and infrastructure providers have seen memcached DDoS attacks ramp up over the last week or so, they’ve moved swiftly to implement defenses to block traffic coming from memcached servers.
“Large DDoS attacks such as those made possible by abusing memcached are of concern to network operators,” says Roland Dobbins, a principal engineer at the DDoS and network-security firm Arbor Networks who has been tracking the memcached attack trend. “Their sheer volume can have a negative impact on the ability of networks to handle customer internet traffic.”
The infrastructure community has also started attempting to address the underlying problem, by asking the owners of exposed memcached servers to take them off the internet, keeping them safely behind firewalls on internal networks. Groups like Prolexic that defend against active DDoS attacks have already added or are scrambling to add filters that immediately start blocking memcached traffic if they detect a suspicious amount of it. And if internet backbone companies can ascertain the attack command used in a memcached DDoS, they can get ahead of malicious traffic by blocking any memcached packets of that length.
“We are going to filter that actual command out so no one can even launch the attack,” says Dale Drew, chief security strategist at the internet service provider CenturyLink. And companies need to work quickly to establish these defenses. “We’ve seen about 300 individual scanners that are searching for memcached boxes, so there are at least 300 bad guys looking for exposed servers,” Drew adds.
Most of the memcached DDoS attacks CenturyLink has seen top out at about 40 to 50 gigabits per second, but the industry had been increasingly noticing bigger attacks up to 500 gbps and beyond. On Monday, Prolexic defended against a 200 gbps memcached DDoS attack launched against a target in Munich.
Wednesday’s onslaught wasn’t the first time a major DDoS attack targeted GitHub. The platform faced a six-day barrage in March 2015, possibly perpetrated by Chinese state-sponsored hackers. The attack was impressive for 2015, but DDoS techniques and platforms—particularly Internet of Things–powered botnets—have evolved and grown increasingly powerful when they’re at their peak. To attackers, though, the beauty of memcached DDoS attacks is there’s no malware to distribute, and no botnet to maintain.
The web monitoring and network intelligence firm ThousandEyes observed the GitHub attack on Wednesday. “This was a successful mitigation. Everything transpired in 15 to 20 minutes,” says Alex Henthorne-Iwane, vice president of product marketing at ThousandEyes. “If you look at the stats you’ll find that globally speaking DDoS attack detection alone generally takes about an hour plus, which usually means there’s a human involved looking and kind of scratching their head. When it all happens within 20 minutes you know that this is driven primarily by software. It’s nice to see a picture of success.”
GitHub continued routing its traffic through Prolexic for a few hours to ensure that the situation was resolved. Akamai’s Shaul says he suspects that attackers targeted GitHub simply because it is a high-profile service that would be impressive to take down. The attackers also may have been hoping to extract a ransom. “The duration of this attack was fairly short,” he says. “I think it didn’t have any impact so they just said that’s not worth our time anymore.”
Until memcached servers get off the public internet, though, it seems likely that attackers will give a DDoS of this scale another shot.
