Limit requests per IP on Nginx using HttpLimitZoneModule and HttpLimitReqModule except whitelist

- Make sure to check Nginx, PHP posts for information on Nginx and PHP setup and configuration.

Nginx offers two modules, HttpLimitReqModule and HttpLimitZoneModule, to limit simultaneous connections for the assigned session and the number of requests for a given session from one IP address. Basically these modules are built to protect the web server from possible DDos attacks; For example, this configuration limits remote clients to no more than 20 concurrently “open” connections per remote ip address:

http{
    limit_zone limit_per_ip $binary_remote_addr 5m;
    limit_conn  limit_per_ip  20;



Additionally, this configuration creates zone “allips” allocating 5MB for this session then limits queries for remote ip address to 200 request per second

http{
    limit_req_zone $binary_remote_addr zone=allips:5m rate=200r/s;
    limit_req zone=allips burst=200 nodelay;

As a result, additional requests or connections from the same remote ip address to Nginx will be dropped with 403 error. But what if we want to create a whitelist of IPs; for instance, Search bots should not obey these limits. That can be accomplished by HttpGeoModule.

Firstly, a list of search bots CIDR is added to the zone “notabot”, so that any IP that is not on list will return “1″ else it returns “0″, as following:
- The example contains real CIDR list of search bot.

http{
     geo $notabot {
        default 1;
        #google 
        64.233.160.0/19 0;
        65.52.0.0/14 0;
        66.102.0.0/20 0;
        66.249.64.0/19 0;
        72.14.192.0/18 0;
        74.125.0.0/16 0;
        209.85.128.0/17 0;
        216.239.32.0/19 0;
        #M$
        64.4.0.0/18 0;
        157.60.0.0/16 0;
        157.54.0.0/15 0;
        157.56.0.0/14 0;
        207.46.0.0/16 0;
        207.68.192.0/20 0;
        207.68.128.0/18 0;
        #yahoo
        8.12.144.0/24 0;
        66.196.64.0/18 0;
        66.228.160.0/19 0;
        67.195.0.0/16 0;
        74.6.0.0/16 0;
        68.142.192.0/18 0;
        72.30.0.0/16 0;
        209.191.64.0/18 0;
        #My IPs
        127.0.0.1/32 0;
        123.456.0.0/28 0; #example for your server CIDR
    }

Secondly, zone “notabot” is added to limit_req to limit requests from other IPs that are not on the whilelist:

http{
    limit_req_zone $binary_remote_addr zone=notabot:5m rate=200r/s;
    limit_req zone=notabot burst=200 nodelay;

Finally, test these configuration with ab against PHP-FPM:
- example PHP file:

<?php
for( $i=0; $i < 1000; $i++)
	echo 'Hello World';
?>

- It is important to check the IP you are connecting with to the server before you start.

#benchmark using ab with compression enabled
ab -n2000 -c400  -H 'Accept-Encoding: gzip,deflate'  http://www.example.com/helloworld.php
 
#results with server CIDR added to whitelist
Concurrency Level:      400
Time taken for tests:   0.426980 seconds
Complete requests:      2000
Failed requests:        275             <-------------------------
   (Connect: 0, Length: 275, Exceptions: 0)
Requests per second:    4684.06 [#/sec] (mean)
Time per request:       85.396 [ms] (mean)
Time per request:       0.213 [ms] (mean, across all concurrent requests)
 
#results after commenting server CIDR from the whitelist
Concurrency Level:      400
Time taken for tests:   0.665429 seconds
Complete requests:      2000
Failed requests:        1807             <-------------------------
   (Connect: 0, Length: 1807, Exceptions: 0)
Write errors:           0
Requests per second:    3005.58 [#/sec] (mean)
Time per request:       133.086 [ms] (mean)
Time per request:       0.333 [ms] (mean, across all concurrent requests)

In conclusion, requests or connections from IPs on the whitelist will have no limits, whereas IPs outside the whitelist will be limited to a number of connections and requests, consequently, they will be dropped or fail as showen in the above ab example.

Recommended posts:

  • Jonathan Matthews

    When you say “zone ‘notabot’ is added to limit_req”, the next config example doesn’t mention “notabot”.
    Is this a mistake?

    • http://gadelkareem.com/ Waleed Gadelkareem

      Yes a mistake..thx for the correction

  • Mb

    Failed requests:  275  even thought the IP is on the whitelist ? How is that correct ?

    • http://gadelkareem.com/ Waleed Gadelkareem

      This  is normal due to the  ”length failure” 

  • certabiz

    Hi, this doesn’t work for me. The limit_req_zone indiscriminately limits requests for everything, even though I set the zone to the name of the geo.

    • http://gadelkareem.com/ Waleed Gadelkareem

      try to include your IP range to notabot list as mentioned and test again. Also make sure your have Nginx 1.0.12 or later version.

  • Anony

    The CIDR number you give for Google include 65.52.0.0/14 0 which is a microsoft range and not google

    • http://gadelkareem.com/ Waleed Gadelkareem

      Thank you! I will check again.

  • Afdah

    instead of limit requests, i tried to limit connection using the above method. doesn’t work. any ideas?

  • SplitIce

    Do you know if this works with other methods of setting variables or just geo? I wasnt able to find the C code that implements this feature in the source.