Topics

bot

Limit requests per IP on Nginx using HttpLimitZoneModule and HttpLimitReqModule except whitelist

– Make sure to check Nginx, PHP posts for information on Nginx and PHP setup and configuration.

Nginx offers two modules, HttpLimitReqModule and HttpLimitZoneModule, to limit simultaneous connections for the assigned session and the number of requests for a given session from one IP address. Basically these modules are built to protect the web server from possible DDos attacks; For example, this configuration limits remote clients to no more than 20 concurrently “open” connections per remote ip address:

http{
    limit_conn_zone  $binary_remote_addr zone=concurrent:10m;
    limit_conn_log_level warn;
    limit_conn  concurrent  20;

Sitemap Creator 0.2 beta

New Version is available. Click here

Sitemap Creator crawls/spiders your website creating XML sitemaps compatible with the standard sitemaps.org protocol supported by Google, Yahoo!, MSN and MoreOver. The script pings Google, Yahoo!, MSN and MoreOver bots to download the sitemap file, then tracks the bot and sends you an email on every scan to your Sitemap and gives you a full report of the Search Engine respond.
Sitemaps are created from a CSV file which could easily be edited using any text editor before creating the sitemap.

Sitemap Creator 0.2a: Create sitemaps 09 valid for Google, Yahoo, MSN, Ask.com and moreover sitemaps

New Sitemap Creator Beta available
Sitemap Creator 0.2a is different from version 0.1, The script now is able to crawl/spider your website, create your sitemaps, ping Google, Yahoo, MSN, Ask.com, moreover.com with the location of your sitemaps and send you alerts by email when sitemaps are created or crawled by the search bot. The crawler saves sitemaps data into an easy to edit CSV file.

Download (build 20070109) :
sitemap_creator.tar.gzsitemap_creator.zip

Using iptables to block ips that spam or attack your server

Check Configuring iptables on CentOS post.

Why should you do that while APF or CSF can do it automatically?
Because APF/CSF could block an important bot testing your server to add to search index. So reviewing every ip would be a daily task!
Ok, so how?
using RHEL4, centos4

Sitemap Creator 0.1 : Create Sitemaps 0.9 valid for Google, Yahoo! and MSN Sitemaps

New Sitemap Creator 0.2a available

Sitemap Creator is a php script making use of the
announced new standard SiteMaps protocol supported by Google, Yahoo and MSN.