Setting up a custom fail2ban jail

> For a long while I've had a system setup that logs people probing servers for vulnerabilities. I describe it to customers like someone walking down your street trying all the doors and windows, if you saw someone doing that you might call the police. When someone or some robot does a similar thing on your website there isn't an authority to call.
I have had systems that read those malicious actor logs and then later blocks those IP addresses at the application level. There is a bit of a gap though as those jobs might only read the logs and block daily. Which gives the person a number of free goes before they are stopped. Most of the requests I'm blocking are robotic automatic drive by attacks rather than a human being.
I have fail2ban running on several servers but there didn't seem to be a jail that enabled blocking on a response code. When one of these requests happens the site returns a 403 response code which ends up in the apache logs so it seems like that is the thing to target.
After a bit of googling at the moment I have added my own custom jail to /etc/fail2ban/jail.local

[403]
enabled = true
logpath = /var/log/httpd/apache.log
maxretry = 5
findtime = 3600
bantime = 7200
action = %(action_)s

So their IP address will get banned if they make more than 5 requests resulting in 403 response codes in an hour (3600 seconds) for two hours. I created my custom filter at /etc/fail2ban/filter.d/403.conf which provide the regex to find 403 responses in the apache log file.

[INCLUDES]
before = common.conf
[Definition]
failregex = ^.*"(GET|POST|PUT|DELETE).*" (403) .*$
ignoreregex = /media/

Its a bit cryptic but you can test it like so before you activate it to make sure your only catching the requests you want.

fail2ban-regex --print-all-matched /var/log/httpd/apache.log 
/etc/fail2ban/filter.d/403.conf

I found my regex accidentally caught some requests which were not malicious but returned a 403 code hence the ignoreregex. For someone else on a different website the regex might need to be different. I expect it will get tweaked in the future. At the moment its catching the requests I want it to catch. There are a lot of different filters that come with fail2ban by default which you can look at.
You can check if your jail is working over time by looking at the fail2ban.log you should see lines like this

INFO    [403] Found 144.217.180.194 - 2024-08-09 05:31:04
NOTICE  [403] Ban 144.217.180.194
INFO    [403] Found 144.217.180.194 - 2024-08-09 05:31:07
...
NOTICE  [403] Unban 144.217.180.194

The next stage is to add reporting to AbuseIPDB which hopefully gives the bots bad behaviour some consequences as they get reported and blocked hopefully other people can read from AbuseIPDB and protect themselves. This should be a simple as adding another line to the actions in the jail.

%(action_abuseipdb)s[abuseipdb_apikey="my-api-key", abuseipdb_category="10"]

You can register for an API key and find out the correct categories on the abuseipdb.com website.