DDoS Defense Guide
How to Detect an Attack
If suddenly you see many users disconnect from a server...
[12:31:23] *** Quits: Lucifer_|des (JohnReb@AEJva.DesireNET.Org) (Ping timeout)
[12:31:51] *** Quits: depeche|nat (depeche@depeche.users.nationchat.org) (Ping timeout)
[12:32:36] *** Quits: iulian7502|des (iulian@ADx0-.DesireNET.Org) (Ping timeout)
[12:34:07] *** Quits: Counter|under (CPT@CPT.fig.ircnow.org) (Ping timeout)
[12:34:12] *** Quits: katrok|quake (katrok@katrok.bnc1.ircnow.org) (Ping timeout)
[12:34:19] *** Quits: Lucifer|des (Lucifer@ACXJz.DesireNET.Org) (Ping timeout)
[12:35:29] *** Quits: depeche|quake (depeche@depeche.users.quakenet.org) (Ping timeout)
[12:35:38] *** Quits: Elafi|under (Elafi@Elafi.fig.ircnow.org) (Ping timeout)
[12:35:52] *** Quits: edu|dal (ed@fig.ircnow.org) (Ping timeout)
[12:36:45] *** Quits: Soportes (Soportes@Soportes.fig.ircnow.org) ("IRCNow and Forever!")
...your server may be under attack!
Different Attack Types
If you see ping timeouts like above, your server's bandwidth is clogged with so many junk packets that it cannot respond to real traffic. This could be the result of a SSDP attack or a TCP ack flood. Logging and analyzing incoming packets is how we identify the attack type.
If you see many EOF from client, Read error: Input/output error, or Client closed connection quit messages, this may be due to a TCP reset flood?:
[02:02:42] *** Quits: jrmu|dal (jrmu@jrmu.plum.ircnow.org) (Read error: Input/output error) [02:02:42] *** Quits: semut_|dal (semut@semut.lu2.ircnow.org) (Client closed connection) [02:02:42] *** Quits: starr|dal (starr@starr.lu2.ircnow.org) (Read error: Input/output error) [02:02:42] *** Quits: Gisa|dal (thekingofb@thekingofbandit.lu2.ircnow.org) (Read error: Input/output error) [02:02:42] *** Quits: Gisa|quake (thekingofb@lu2.ircnow.org) (EOF from client) [02:02:42] *** Quits: Freak|quake (Freak@Freak.lu2.ircnow.org) (EOF from client) [02:02:42] *** Quits: IRCuser|quake (unknwon@IRCuser.users.quakenet.org) (EOF from client) [02:02:42] *** Quits: ramadi|quake (ramadi@ramadi.lu2.ircnow.org) (EOF from client) [02:02:43] *** Quits: Fat1 (Fatfem@Fatfem.lu2.ircnow.org) (Client closed connection) [02:02:43] *** Quits: Freak (Freak@Freak.lu2.ircnow.org) (Client closed connection)
Confirming an Attack
To see if there is a ddos attack, first run:
$ ping fruit.ircnow.org
Replace fruit.ircnow.org with your actual server's hostname. If you get more than 30% packet loss, this could be a sign you are being flooded with fake packets. During a DDoS, an attacker is flooding your internet pipe with junk packets so that your server is unable to respond to any real network traffic.
Check the serial console (using cu? if on VMM, VNC if on BuyVM). Try to log in and see if the system is still responsive. If the server is offline or has errors, it may be due to hardware/software issues rather than a DDoS attack.
During a DDoS attack, your system should feel slightly sluggish. This is due to the massive number of packets that OpenBSD must process. Run top. If you see the system/interrupt CPU usage at 50% or more, then you either have a process using too much CPU or you may be under an attack. If the CPU usage is low, under 5%, it might not be a DDoS attack. It could be a networking or hardware issue, or an attack of another nature.
If you suspect an attack, you should log the packets that are coming in:
$ doas tcpdump -w YYYYMMDDSS.pcap
This will log the packets onto the file YYYYMMDDSS.pcap, where YYYY is the year, MM is the month, DD is the day, and SS is the serial number.
You can fine-tune this:
$ doas tcpdump -i vio0 -s 1500 -A -v -n -w YYYYMMDDSS.pcap
In the above command, the argument -i specifies the interface vio0, -s 1500 analyzes the first 1500 bytes, -A prints each packet in ASCII, -v provides more verbose output, and -n avoids address conversion. Make sure to read up on tcpdump.
To stop collection, type [ctrl]+c, ^C. If you log for a few seconds and only receive a few hundred packets, perhaps you are not under attack. If, however, you see millions of packets arriving within a few seconds, you are certainly under attack. Save the pcap file because it is very useful for forensics. We will use it for reporting the attacker and for blocking his attacks.
To read the pcap, you can transfer the pcap to your desktop using scp? or sftp. Or, more quickly, you can analyze it on the server itself using tcpdump. Use the same options but replace -w with -r:
$ doas tcpdump -r YYYYMMDDSS.pcap
or
$ doas tcpdump -i vio0 -s 1500 -A -v -n -r YYYYMMDDSS.pcap
Identifying Attack Type
| Name | Packet Type | Description |
|---|---|---|
| SYN Flood? | SYN packets | Your server sends ACK packets and wastes resources on useless TCP connections |
| HTTP Flood? | HTTP packets | Your server wastes resources responding to fake HTTP requests |
| NTP Amplification? | NTP packets | Your server is flooded with fake NTP packets |
| UDP Flood | UDP packets | Your server receives lots of fake UDP packets at random ports |
| SSDP Attacks | SSDP packets | Your server is flooded with packets from Universal Plug and Play (UPnP) devices |
| TCP ACK Flood | ACK packets | Your server is flooded with useless ACK packets |
| TCP RST Flood | RST packets | Your system daemons are tricked into disconnecting by fake RST (reset) packets |
Reporting Criminals
To stop abuse, it is necessary to report attackers. Often, a DDoSer can be quickly identified because an attack took place during an argument. A DDoSer may also be a competitor or have some other financial motive. If the pcap shows a specific IPv6 address was targeted, contact the user with that unique IPv6 address and ask him for clues. Attackers sometimes will show up on your channel to insult or mock you during an attack. This type of evidence, although not 100% certain, can provide clues as to who is behind an attack.
Hints provide a very useful starting point, but you will still need to do the necessary investigation in order to get enough evidence to report. Please see the police guide.
Keep in mind that criminals often spoof IP addresses (lie about the source IP address) or use proxies (other insecure machines) to amplify their attacks. Logs of packets alone often do not provide enough evidence to report a criminal. Attackers often amplify their attacks using IoT devices, universities, and large corporations to mask their true origin. The attacking IP might be the stolen server, home computer, or electronic device of some innocent bystander.
DDoS Filtering
If you are hosting a public service, sooner or later you will get hit with DDoS attacks. The Internet is full of criminals, and the anonymity of IRC makes it hard to catch them. Strong and determined attackers can sustain DDoS attacks of more than 500Gbps.
A software firewall on a single server alone cannot stop all DDoS attacks because of physical limitations. Once the maximum bandwidth of your server is reached, it is unable to accept any further traffic. If you get attacked but you are not using a filtered IP, your provider will null route your IP (take it offline). So long as the attack continues, your service will be completely offline, and your users will blame you.
As a result, we are forced to purchase DDoS filtering or acquire more bandwidth. Both solutions cost money, which is why it is necessary to work as a team to purchase more bandwidth at a cheaper rate. The larger our network, the more money we can afford to spend on DDoS defenses. It is wise to avoid depending entirely upon one single internet provider like Cloudflare or Voxility. Using diverse providers decreases the risk of failure.
Troubleshooting
Cloudflare/Voxility's DDoS protection can mangle SSL certs when you make requests from some servers. For example, I noticed HTTPS requests to Let's Encrypt's servers were being mangled:
$ openssl s_client -connect 172.65.32.248:443
It just hung there with no certificate being issued. If this happens, you will need to report this to the Internet Provider (BuyVM).
DDoS filtered IPs have been known to mangle SMTP, NTP, and DNS packets. So if networking fails for some inexplicable reason, check using openssl to see if SSL certs are being mangled.