The most effective way to prevent bots from spamming your server is to drop them at the firewall. This is generally achieved using tools like Denyhosts or fail2ban, which monitor your logs, identify suspicious activity, and block the offending IP addresses before they cause harm.

Denyhosts works at the application level by adding entries to /etc/hosts.deny, whereas fail2ban operates at the firewall level using iptables, which makes it far more efficient.

However, on resource-constrained machines, fail2ban can still be taxing. A few years ago, we shared a demo of a lightweight log parser called banbylog, tailored specifically for our needs (SSH and WordPress activity monitoring) at a much lower resource cost. If that sounds like a good fit, feel free to check it out, but keep in mind that it’s not production-ready!

While the consensus is to parse logs and block hostile IPs at the firewall, it will not work under Cloudflare umbrella!

Ideally the attack would stop here, but it won't work with a full CDN like Cloudflare

Why can't we flag offending IPs with Cloudflare?

Because the IPs that are "attacking" your server are not the actual offending IPs, but Clouflare machines that are proxying the request to your server. Take a look at the diagram below:

Cloudflare acts as a middleman between your server and the users. The only IP addresses visible to your firewall are from Cloudflare, not from the original user.

In fact, you should explicitly whitelist Cloudflare IP range. If you happen to block an IP that belongs to Cloudflare, legitimate users will see your site as down.

😱
While this article's focus is on securing servers while working under a Content Distribution Network, Cloudflare offers a variety of tools - free and paid - you may leverage to achieve these same results.

If the IP we see doesn't belong to the user making the request, what can we look for?

Every request Cloudflare sends to your server has an attached header that carries the user original IP under CF-Connecting-IP. And that's what we can leverage to get them! Unfortunately, reading http headers is too upstream for iptables, thus HAProxy to the rescue!

While iptables operates at Layer 4, HAProxy can operate at OSI Layer 7.

πŸ’‘
Remember the OSI model:

Layer 7 - Application; protocols (HTTP, ...)
Layer 6 - Presentation; character encoding (ASCII, UTF8, ...)
Layer 5 - Session; stick client to server
Layer 4 - Transport; protocols (TCP, UDP, ...)
Layer 3 - Network; routing protocols (IP)
Layer 2 - Data Link; physical to network (ARP, Ethernet)
Layer 1 - Physical; cabling, Wi-Fi

The easiest way to test HAProxy configurations is to boot a Docker instance of HAProxy:

docker run --name haproxy -p 888:80 -v ${pwd}:/usr/local/etc/haproxy --sysctl net.ipv4.ip_unprivileged_port_start=0 haproxy:2.3

Start HAProxy listening on host port 888

The above assumes you're using Powershell and are in the directory that contains haproxy.cfg. Change ${pwd} accordingly if not.

docker kill -s HUP haproxy

This command forces HAProxy restart, which is useful to iterate different configs

Below is a straightforward haproxy.cfg that will put HAProxy listening on port 80, log to stdout so you can get an instant glimpse on what's going on, and also print the CF-Connecting-IP header.

global
    # output logs straight to stdout
    log stdout format raw daemon debug

defaults
    # put HAProxy in Level 7 mode allowing to inspect http protocol
    log     global
    mode    http

frontend  main
    bind *:80
	
    # capture original IP from header and put it in variable txn.cf_conn_ip
    http-request set-var(txn.cf_conn_ip) hdr(CF-Connecting-IP)

    # get the original IP on logs (just to check if things are working)
    log-format "%ci\ %hr\ %ft\ %b/%s\ %Tw/%Tc/%Tt\ %B\ %ts\ %r\ %ST\ %Tr CF-IP:%{+Q}[var(txn.cf_conn_ip)]"

    # use backend ok, which always returns a 200, for debug
    use_backend ok

backend ok
	http-request return status 200 content-type "text/plain" lf-string "ok"

This should be enough to test HAProxy

Let's make a test request to HAProxy.

I'm partial to Bruno, a portable and offline alternative to Postman. Download it, add the CF-Connecting-IP header and make a POST request to http://localhost:888/wp-login.php to test if everything's working.

Nothing to install, a POST request in under a minute!

If things went as planned, your HAProxy should have printed this:

172.17.0.1 main ok/<NOSRV> -1/-1/0 78 LR POST /wp-login.php HTTP/1.1 200 -1 CF-IP:"222.222.222.222"

Now that we have the offending IP (222.222.222.222), we can block it!

On our particular case, bots are hitting wp-login.php, xmlrpc.php and xmrlpc.php (last one is a typo, but we've had more than 100k hits in the last 24h!). We also know that they're flooding the server with POST requests trying to brute-force passwords.

frontend main
    # requests that will be monitored and blocked if abused
    acl is_wp_login path_end -i /wp-login.php /xmlrpc.php /xmrlpc.php
    acl is_post method POST

Add this to end of the frontend main block

We now have a couple of options:

a) Now that we have the offending IPs in the log, we could change banbylog to write them to a file, and have HAProxy deny those requests. While this would work, HAProxy would need to be constantly reloaded.

b) Or we may simply leverage HAProxy stick tables and do a rate-limiting on offending requests.

While "a" would allow us to ban the offending IP for an indefinite amount of time, "b" has less moving pieces.

frontend main
    # requests that will be monitored and blocked if abused
    acl is_wp_login path_end -i /wp-login.php /xmlrpc.php /xmrlpc.php
    acl is_post method POST

    # table than can store 100k IPs, entries expire after 1 minute
	stick-table type ip size 100k expire 1m  store http_req_rate(1m)

    # we'll track (save to table) the original IP only if the request hits
    # one of the monitored paths with a POST request
    http-request track-sc0 hdr(CF-Connecting-IP) if is_wp_login is_post

    # we now query the stick-table and if the IP has made more than 
    # 5 requests of the offending type in the last minute, 
    # current request is denied
    http-request deny if is_wp_login is_post { sc_http_req_rate(0) gt 5 }

HAProxy has multiple deny options, tarpit, silent drop, reject or shadowban. A tarpit deny would be something like this:

# make request hang for 20 seconds before replying back with a 403
timeout tarpit 20s
http-request tarpit deny_status 403 if is_wp_login is_post { sc_http_req_rate(0) gt 2 }

Notice that this puts extra stress on both HAProxy and iptables as they must keep the connection open

Just for reference, here's the full (overly simplified) HAProxy config file that blocks requests if they hit one of the monitored URL's with more than 5 hits in less than a minute:

global
    log stdout format raw daemon debug
	
defaults
    log     global
	mode    http

frontend main
    bind *:80
	
    # capture original IP from header and put it in variable txn.cf_conn_ip
    http-request set-var(txn.cf_conn_ip) hdr(CF-Connecting-IP)

    # get the original IP on logs (just to check if things are working)
    log-format "%ci\ %hr\ %ft\ %b/%s\ %Tw/%Tc/%Tt\ %B\ %ts\ %r\ %ST\ %Tr CF-IP:%{+Q}[var(txn.cf_conn_ip)]"

    acl is_wp_login path_end -i /wp-login.php /xmlrpc.php /xmrlpc.php
    acl is_post method POST
	stick-table type ip size 100k expire 1m  store http_req_rate(1m)
	http-request track-sc0 var(txn.cf_conn_ip) if is_wp_login is_post
	http-request deny if is_wp_login is_post { sc_http_req_rate(0) gt 5 }

    # obviously change this to whatever backend you're using
    use_backend ok

backend ok
	http-request return status 200 content-type "text/plain" lf-string "ok"

And there you have it, using your proxy as a gatekeeper!

The CPU toll of a wp-login.php brute-force attack and then HAProxy acting as a bouncer.

Re-Captcha, obviously!

While this article focus on the server side of things, the first thing you should obviously do, is to foolproof forms with Re-Captcha, which you can easily do with a plugin such as Advanced Google reCAPTCHA by WebFactory.


EDIT: A user queried: Β«I have some WordPress installations under Cloudflare and some exposing the server directly. Can it work on both?Β»

You can. It's as simple as doing something like this:

# match IP against Cloudflare list
acl from_cf src -f /usr/local/etc/haproxy/cloudflare_ips.lst

# if IP originates from cloudflare check CF-Connecting-IP header, otherwise use src
http-request set-var(txn.client_ip) hdr(CF-Connecting-IP) if from_cf
http-request set-var(txn.client_ip) src if !{ var(txn.client_ip) -m found }

On previous examples, all IPs were from Cloudflare. Here it's obligatory to check if IP originates from Cloudflare. Otherwise, a spammer could forge CF-Connecting-IP and trick the simple if/else.


This week, I assisted a friend in upgrading to professional TP-Link access points. I'm a strong advocate for devices that excel at a single task, and these EAP610 access points do just that. Highly recommended!