Menü schliessen
Created: September 16th 2019
Last updated: December 10th 2021
Categories: Linux
Author: Marcus Fleuti

blocklist.de iptables / ipset update script - How to automatically update your firewall with the IP set from blocklist.de

Tags:  blocklist.de,  firewall,  grep,  ipset,  iptables,  mail,  Postfix,  sort,  wget
Donation Section: Background
Monero Badge: QR-Code
Monero Badge: Logo Icon Donate with Monero Badge: Logo Text
82uymVXLkvVbB4c4JpTd1tYm1yj1cKPKR2wqmw3XF8YXKTmY7JrTriP4pVwp2EJYBnCFdXhLq4zfFA6ic7VAWCFX5wfQbCC

How to automatically update your iptables firewall with the IP set from blocklist.de

The goal

We want to download the blocklist.de IP blocklist and import it into our iptables firewall.

The problems we occured

First we were using the blocklist.de update script from here: https://github.com/vlcty/Blocklist.de-Sync/blob/master/blocklist-update.sh. This script imports the full IP list from blocklist.de into a dedicated iptables chain. Which one might think is exactly what is intended. Yet, unfortunately, we've experienced the problem, that this massively pulled down the whole server performance. As it seems iptables does not like huge IP lists. Also the script took like 3 hours to update all the iptables entries, which was pretty annoying. During that time the whole iptables system required like 80% of 1 CPU core in ressources and slowed down all network traffic (also the internal traffic).

The solution

Next to iptables there's a tool called ipset. With ipset managing huge IP lists is totally simple and fast. Hence we've rewritten the above linked blocklist-update.sh script to work with iptables/ipset. You can download our script here: blocklist-update.txt (this is a Linux Bash Shellscript) or copy/paste it from below:

#!/bin/bash

# The download path to the file which contains all the IP addresses
TO_DOWNLOAD="http://lists.blocklist.de/lists/all.txt"

# Other settings; Edit if necesarry
CHAINNAME="blocklist-de"
ACTION="REJECT" # Can be DROP or REJECT
IPTABLES_PATH="/usr/sbin/iptables"
IPSET_PATH="/sbin/ipset"
SORT_PATH="/usr/bin/sort"
MAIL_PATH="/usr/bin/mail"
GREP_PATH="/bin/grep"

if [ -z $IPTABLES_PATH ]; then echo "Cannot find [ iptables ]. Is it installed? Exiting"; exit 1; fi;
if [ -z $IPSET_PATH ]; then echo "Cannot find [ ipset ]. Is it installed? Exiting"; exit 1; fi;
if [ -z $SORT_PATH ]; then echo "Cannot find [ sort ]. Is it installed? Exiting"; exit 1; fi;
if [ -z $MAIL_PATH ]; then echo "Cannot find [ mail ]. Is it installed? Exiting"; exit 1; fi;
if [ -z $GREP_PATH ]; then echo "Cannot find [ grep ]. Is it installed? Exiting"; exit 1; fi;

# E-Mail variables
MAILLOG="/var/log/blocklist-update.log"
MAIL_SENDER="mailinfo" #this defines a system-user without a shell or password. It's used as the e-mail sender name. You can create one like this: useradd -M -N -s /usr/sbin/nologin myuser && passwd -d myuser
MAIL_SUBJECT="ERROR - IP blocklist script failed to download the IP set"
MAIL_RECIPIENTS="youradmin@email.tld" #send mail to multiple receipients by overgiving a space-seperated address list

BLOCKLIST_FILE="/tmp/ip-blocklist.txt"
BLOCKLIST_TMP_FILE="/tmp/ip-blocklist.txt.tmp"
WHITELIST="/home/srvdata/blocklist-de-update.whitelist"

TIME_START=$(date +"%s")

# Delete existing blocklist files (if any)
rm -f ${BLOCKLIST_FILE}
rm -f ${BLOCKLIST_TMP_FILE}

# Create a new MAILLOG from scratch. Do it the very simplest way possible
rm -f $MAILLOG
touch $MAILLOG

echo "" >> $MAILLOG
echo "Starting the process on `date +%d.%B.%Y,%T`." >> $MAILLOG
echo "" >> $MAILLOG

echo "" >>$MAILLOG
echo "Downloading the most recent IP list from $TO_DOWNLOAD ..." >>$MAILLOG
wgetOK=$(wget -qO - $TO_DOWNLOAD > $BLOCKLIST_FILE) >>$MAILLOG 2>&1
if [ $? -ne 0 ]; then
        echo "Most recent IP blocklist could not be downloaded from $TO_DOWNLOAD" >>$MAILLOG
        echo "Please check manually. The script calling this function: $0" >>$MAILLOG
        echo "You can download and import the IP list manually like this:" >>$MAILLOG

        ### Sending warning e-mail and cancelling the update process
        sudo -u $MAIL_SENDER /usr/bin/mail -s "$MAIL_SUBJECT" $MAIL_RECIPIENTS < $MAILLOG ### Exit with error in this case exit 1 fi echo "" >>$MAILLOG
echo "Parsing the downloaded file and filter out only IPv4 addresses (because blocklist.de does not provide IPv6 for the time being but contains some malformed IPv4 addresses sometimes..." >>$MAILLOG
grep -E -o "(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)" $BLOCKLIST_FILE > ${BLOCKLIST_TMP_FILE}

echo "" >>$MAILLOG
echo "Removing whitelisted IPs from the downloaded IP blacklist..."  >>$MAILLOG
IPWHITELIST=`cat $WHITELIST`
IPWHITELISTREGEX=""
while IFS= read -r WHITELISTEDIP
        do
        IPWHITELISTREGEX+="(${WHITELISTEDIP})|"
done <<< ${IPWHITELIST}
## Clean the bounce variable (remove all line-breaks)
IPWHITELISTREGEX="${IPWHITELISTREGEX//$'\n'/ }"
IPWHITELISTREGEX=$(perl -pe "s/(.*)\|/\1/gms" <<< ${IPWHITELISTREGEX}) ## Remove all IPs listed in the whitelist file grep -v -E ${IPWHITELISTREGEX} ${BLOCKLIST_TMP_FILE} > ${BLOCKLIST_FILE}
cp -f ${BLOCKLIST_FILE} ${BLOCKLIST_TMP_FILE}

echo "" >>$MAILLOG
echo "Removing duplicate IPs from the list ..." >>$MAILLOG
sort -u $BLOCKLIST_TMP_FILE -o $BLOCKLIST_FILE >>$MAILLOG 2>&1
rm $BLOCKLIST_TMP_FILE

echo "" >>$MAILLOG
echo "Setting up the ipset configuration by creating the '$CHAINNAME' IP set ..." >>$MAILLOG
if [ `$IPSET_PATH list | grep "Name: $CHAINNAME" | wc -l` -eq 0 ]
then
        # Create the new ipset set
        $IPSET_PATH create $CHAINNAME hash:ip maxelem 16777216 >>$MAILLOG 2>&1
else
        echo "ipset configuration already exists - Flushing and recreating the iptables/ipset configuration ..." >>$MAILLOG
        # Reason: The kernel sometimes did not properly flush the ipset list which caused errors. Thus we remove the whole list and recreate it from scatch
        $IPTABLES_PATH --flush $CHAINNAME >>$MAILLOG 2>&1
        $IPSET_PATH flush $CHAINNAME >>$MAILLOG 2>&1
        $IPSET_PATH destroy $CHAINNAME >>$MAILLOG 2>&1
        $IPSET_PATH create $CHAINNAME hash:ip maxelem 16777216 >>$MAILLOG 2>&1
fi

echo "" >>$MAILLOG
echo "Setting up the $CHAINNAME chain on iptables, if required..." >>$MAILLOG
if [ `$IPTABLES_PATH -L -n | grep "Chain $CHAINNAME" | wc -l` -eq 0 ]
then
        # Create the iptables chain
        $IPTABLES_PATH --new-chain $CHAINNAME >>$MAILLOG 2>&1
fi

echo "" >>$MAILLOG
echo "Inserting the new chain $CHAINNAME into iptables INPUT, if required" >>$MAILLOG
# Insert rule (if necesarry) into the INPUT chain so the chain above will also be used
if [ `$IPTABLES_PATH -L INPUT | grep $CHAINNAME | wc -l` -eq 0 ]
then
        # Insert rule because it is not present
        $IPTABLES_PATH -I INPUT -j $CHAINNAME >>$MAILLOG 2>&1
fi

# Create rule (if necesarry) into the $CHAINNAME
echo "" >>$MAILLOG
echo "Creating the firewall rule, if required..." >>$MAILLOG
if [ `$IPTABLES_PATH -L $CHAINNAME | grep REJECT | wc -l` -eq 0 ]
then
        # Create the one and only firewall rule
        $IPTABLES_PATH -I $CHAINNAME -m set --match-set $CHAINNAME src -j $ACTION >>$MAILLOG 2>&1
fi

## Read all IPs from the downloaded IP list and fill up the ipset filter set
echo "" >>$MAILLOG
echo "Importing the IP list into the IP set..." >>$MAILLOG
for i in $( cat $BLOCKLIST_FILE ); do $IPSET_PATH add $CHAINNAME $i >>$MAILLOG 2>&1; done

echo "" >>$MAILLOG
echo "Done." >>$MAILLOG
TIME_DIFF=$(($(date +"%s")-${TIME_START}))
echo "" >>$MAILLOG
echo "" >>$MAILLOG
echo "Process finished in $((${TIME_DIFF} / 60)) Minutes and $((${TIME_DIFF} % 60)) Seconds." >>$MAILLOG
sudo -u $MAIL_SENDER $MAIL_PATH -s "SUCCESS - IP blocklist script has updated the IP set with the newest IP list" $MAIL_RECIPIENTS < $MAILLOG

What do you need?

In order for the script to work you need to install the following packages (example: Debian/Ubuntu):

  1. wget
  2. iptables
  3. ipset
  4. mail
  5. sort
  6. grep
  7. postfix (or any other MTA) - optional
apt install wget iptables ipset bsd-mailx coreutils grep

The easiest way to make the script work is like this:

  1. Download the blocklist script file to your Linux system like this:
    wget -O /opt/blocklist-update.sh -U "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)" https://www.lexo.ch/wp-content/uploads/2019/09/blocklist-update.txt
  2. Assign execute permissions to the file
    chmod 755 /opt/blocklist-update.sh
  3. Edit the file and change parameters according to your desire:
    nano /opt/blocklist-update.sh
  4. Save the file with CTRL+X --> 'y'

Postfix / MTA configuration to send e-mails

If you want the script to send e-mails you need to set up a local MTA (Mail Transfer Agent) like postfix. If you have not set up an MTA yet let me explain quickly how to set up a simple postfix relay MTA which simply allows you to send out e-mails from you Linux system:

Install Postfix

apt install postfix

During the setup process tell the system to install Postfix with no pre-set configuration whatsoever. It's easier to set it up from scratch.

Configure Postfix

mv /etc/postfix/main.cf /etc/postfix/main.cf.old
nano /etc/postfix/main.cf

Paste the following into the /etc/postfix/main.cf

smtpd_banner = $myhostname ESMTP $mail_name (Debian/GNU)
biff = no
# appending .domain is the MUA's job.
append_dot_mydomain = no
# Uncomment the next line to generate "delayed mail" warnings
#delay_warning_time = 4h
readme_directory = no
# TLS parameters
smtpd_tls_cert_file=/etc/ssl/certs/ssl-cert-snakeoil.pem
smtpd_tls_key_file=/etc/ssl/private/ssl-cert-snakeoil.key
smtpd_use_tls=yes
smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
# See /usr/share/doc/postfix/TLS_README.gz in the postfix-doc package for
# information on enabling SSL in the smtp client.
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
myorigin = /etc/mailname
mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128
mailbox_command = procmail -a "$EXTENSION"
mailbox_size_limit = 0
recipient_delimiter = +
inet_interfaces = 127.0.0.1
smtp_use_tls = yes
smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = hash:/etc/postfix/saslpass
smtp_sasl_security_options = noanonymous
relayhost = mail.anymailserveryoulike.com:587
myhostname = yourserver.yourdomain.com
mydomain = yourdomain.com
mydestination = yourserver.yourdomain.com, localhost.yourdomain.com, localhost
  • Change the last 3 parameters (myhostname, mydomain, mydestination) according to your domain/servername.

Create the file /etc/postfix/saslpass like this:

nano /etc/postfix/saslpass

The file simply contains one line. That line contains the SMTP credentials for your outgoing e-mail server: So add the following line, set the servername, username and password as required by your e-mail service and save the file:
mail.anymailserveryoulike.com                     username:password
In order for the Postfix service to accept the configuration you need to do the following:

postmap /etc/postfix/saslpass
/etc/init.d/postfix resetart

Try to send an e-mail to yourself like this:

mail -s "test e-mail" your-email-address@domain.tld

This command will start a new e-mail on the Linux console. That might be a bit confusing because it will just display a blank line waiting for input. What you need to do now:

  1. The system is now waiting for you to enter the message. Enter anything you want, like "Lorem Ipsum" and press Enter
  2. Press CTRL+d
  3. The system will ask you for a Cc: address - just press Enter again
  4. The e-mail will be sent. If it worked you should've already received it. If you did not receive anything you can check out the mail log to see what went wrong:
    tail -n100 /var/log/mail.log

If everything went fine the mail.log will contain several lines like this

Sep 20 10:46:28 yourserver postfix/qmgr[32756]: D122E815A6: from=<root@yourserver.yourdomain.com>, size=413, nrcpt=1 (queue active)
Sep 20 10:46:29 yourserver postfix/smtp[32488]: D122E815A6: to=<your-email-address@domain.tld>, relay=mail.anymailserveryoulike.com[123.123.123.123]:587, delay=0.51, delays=0.03/0.2/0.27/0, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as 533AB116007)
Sep 20 10:46:29 yourserver postfix/qmgr[32756]: D122E815A6: removed

Set up a mail user

In our script you can see this line of code which is responsible for sending out the e-mail:

sudo -u $MAIL_SENDER $MAIL_PATH -s "SUCCESS - IP blocklist script has updated the IP set with the newest IP list" $MAIL_RECIPIENTS < $MAILLOG

This means, that the system will try to send the email with the user $MAIL_SENDER. The user is by default set to websupport like this (you can change that of course):

MAIL_SENDER="websupport"

This means that there has to be a user called websupport in order for this command to work. To make the mail sending work you'll need to create this user without a password and with no login permissions like this:

useradd --no-create-home --no-user-group -s /bin/false websupport
passwd --delete websupport

This user should now be able to send out e-mails in its name.

How does the script work / What does it do exactly?

  1. It downloads the most recent IP list from http://lists.blocklist.de/lists/all.txt
  2. It creates a list into which the IP addresses will be imported with ipset (by default the list is called blocklist-de)
  3. It creates a new iptables chain (by default the chain is called blocklist-de)
  4. It creates a new rule within the new iptables chain to REJECT (or optinally DROP) all requests from IPs in the ipset list.
  5. It imports all IP addresses from the previously downloaded IP list
  6. It creates an ERROR/SUCCESS report and sends it via e-mail to a recipient

Known issues / bugs

There's a few flaws that we know and that could be improved:

  1. The mail logging could be improved a bit by giving more detailed errors. But for us this is OK. We use REGEX filter rules on our e-mail server which automatically filter out the success e-mails from the error emails storing them in different folders. In 99.9% of the time the success messages can just be ignored and in case there's an error one would need to check manually anyway.
  2. There might be some typos in the messages. We've written them rather carelessly. Please feel free to change them however you like. Let's say that this is our impressionistic way of programming 😉
  3. We've experienced once that IPSET caused an error when importing the IP addresses but the script reported SUCCESS anyway whilst the error log (e-mail body) displayed the error. We could not reproduce the error ever since. Sometimes the blocklist.de server does not respond and the list cannot be downloaded. In that case the system correctly reported errors and stopped at the right moment keeping the latest list intact. The script has been working stable for almost 1 year now.