In this tutorial, we will look at how to implement the concept of a large dynamic blacklist or whitelist using an Nginx web server, a Redis server, and an Nginx extension script implemented in Lua.
In this tutorial, all software will be installed on Ubuntu 16.04 or 18.04. With little or no change, these solutions can be applied to other Debian-based distributions.
Nginx has a built-in blacklisting or whitelisting mechanism that is suitable for small lists. However, this mechanism is inconvenient when you need to work with large lists of addresses, when the lists are generated dynamically.
Fortunately, Nginx supports Lua plugins that, together with the Redis in-memory cache, allow you to implement the required whitelisting and blacklisting functionality.
Installing Nginx and Redis on Ubuntu Linux
All components will be installed using APT:
$ sudo apt update
$ sudo apt install redis-server nginx nginx-extras lua-nginx-redisWe check the availability of Nginx by opening the page on the server ip in the browser:
Let's check the availability of Redis:
$ redis-cli
127.0.0.1:6379> ping
PONG
127.0.0.1:6379> quit
$If you're setting up a server for production use, be sure to set up Redis security with our guide .
Let's move on to setting up a white or black list.
Setting up a whitelist
We'll be using a recipe for a whitelist script that will lead to GitHub . To use it, let's add several directives to Nginx:
Add the following line to the /etc/nginx/nginx.conf file:
http {
##
# Basic Settings
##
lua_shared_dict ip_whitelist 1m;and in the server or location section, the whitelisted handler itself. In our example, add to the /etc/nginx/sites-enabled/default file:
# Default server configuration
#
server {
listen 80 default_server;
listen [::]:80 default_server;
access_by_lua_file /etc/nginx/lua/ip_whitelist.lua;
....Download the Lua script itself:
$ sudo mkdir /etc/nginx/lua
$ cd /etc/nginx/lua
$ sudo wget https://gist.githubusercontent.com/itbdw/bc6c03f754cc30f66b824f379f3da30f/raw/a655713ecbe676558244d44dc78c875d7a84f8d1/ip_whitelist.luaLet's check the Nginx configuration and restart it:
$ sudo nginx -t && sudo service nginx restartIf you now try to open the site, you will get a page with an error like:
Access denied
Now add your IP to Redis using the Redis console:
$ redis-cli SADD ip_whitelist A.B.C.Dand check the site access again, the Nginx page we saw before should open:
Again, remove the entry for your IP using the Redis console:
$ redis-cli SREM ip_whitelist 148.77.35.49The result of accessing the site will again be an error.
Thus, we have successfully set up a whitelist using Nginx, Redis, and a Lua script that is suitable for dynamically managing entries and working with large lists. The list is implemented using high-performance Redis sets, the access speed to which obeys the O (1) labor law, that is, it works quickly if in Russian.
Blacklist setup
The black list is configured in the same way, with an accuracy of replacing white with black :
- lua_shared_dict ip_blacklist 1m;
- access_by_lua_file
/etc/nginx/lua/ip_blacklist.lua; - script .
Modify the script to use the correct Redis server and the correct Lua module for it (the author uses resty.redis for this script, not nginx.redis):
$ sudo sed -i 's|your.redis.server.here|127.0.0.1|' ip_blacklist.lua
$ sudo sed -i 's| \= require "resty.redis"| \= require "nginx.redis"|' ip_blacklist.lua Adding and removing entries to the blacklist is carried out by the following commands:
$ redis-cli SADD ip_blacklist A.B.C.D
$ redis-cli SREM ip_blacklist A.B.C.DConclusion
Dynamic black and white lists are a more versatile solution when you need to use large lists with the need to change records on the fly, which may be useful if you have intruder diagnostics.
Storing lists in Redis allows you to provide the highest solution performance that does not degrade as the list of addresses in the list grows. API availability of Redis is suitable for integration with third-party list record providers.
In the event that it is necessary to preserve the lists between Redis reboots, then the line in the /etc/redis/redis.conf configuration file must be changed:
appendonly no
on
appendonly yesThe Lua script is quite simple and can be easily adapted for more specific tasks.
