GadElKareem

Dynamic Content Caching using Lighty + mod_magnet + lua

*Updated 11 Jan 2007 to include luazlib, fix lighty config file and add zlib decoding to cache.lua for older browsers
Since I read the documentations for mod_cml, I was very excited to use this module since caching using PHP running as FastCGI is not helping much during server peak load. However, mod_cml was replaced by mod_magnet which is more flexible and gives more control over request handling in Lighttpd. This Article will focus on caching your PHP scripts using Lua and mod_magnet under Lighttpd, You should read “Compressing your HTML, CSS and Javascript using simple PHP Code” as I will use some functions from there

Installing Lua 5.1.2 + md5 lib

wget http://www.lua.org/ftp/lua-5.1.2.tar.gz
tar xfz lua-5.1.2.tar.gz
cd lua-5.1.2
yum install readline-devel
make linux install
cd ..
wget http://luaforge.net/frs/download.php/2384/md5-1.0.2.tar.gz
tar xfz md5-1.0.2.tar.gz
cd md5-1.0.2
make
make install
cd ..
wget http://luaforge.net/frs/download.php/1678/luazlib-0.0.1.rar
#if only you have to unrar on linux
#get your compatible unrar binary from
#http://www.rarlab.com/rar_add.htm
wget wget http://www.rarlab.com/rar/unrar-3.7.7-centos.gz
gunzip unrar-3.7.7-centos.gz
chmod +x unrar-3.7.7-centos

./unrar-3.7.7-centos x luazlib-0.0.1.rar
cd luazlib-0.0.1
make
make install

Installing Lighttpd 1.4.18 with mod_magnet support

wget http://www.lighttpd.net/download/lighttpd-1.4.18.tar.bz2
tar xfj lighttpd-1.4.18.tar.bz2
cd lighttpd-1.4.18
export LUA_CFLAGS="-I/usr/local/include" 
export LUA_LIBS="-L/usr/local/lib -llua"
./configure --with-lua
make
make install

Enable mod_magnet on your lighttpd.conf

server.modules += ( "mod_magnet" )
#use cache.lua script only on php script from domain gadelkareem.com
$HTTP["host"] == "gadelkareem.com" {
	$HTTP["url"] =~ "(^/(\?.*)?$|\.php)" {
		magnet.attract-physical-path-to = ( "/myserver/cache.lua" )
	}
}

Now every time a request to .php file will be handled by cache.lua script to see if we already have a cached copy of that script.
Here is cache.lua

require"md5"


--is request empty
if ( lighty.env["request.uri"] == nil ) then 
	lighty.env["request.uri"] = "/"
end

local file = "/server/cache/" .. md5.sumhexa( lighty.env["request.uri"] )


if lighty.stat(file) then
	--is encoding empty
	if ( lighty.request["accept-encoding"] == nil ) then 
		 lighty.request["accept-encoding"] = ""
	end
	if( string.find( lighty.request["accept-encoding"] , "gzip") ) then
		lighty.content = { { filename = file } }
		lighty.header["Content-Encoding"] = "gzip"
	else
		require"zlib"
		local ht = io.open(file, "r")
		local t = ht:read("*a")
		ht:close()
		lighty.content = { zlib.gzuncompress(t) .. ""}
	end
	lighty.header["Content-Type"] = "text/html"
	return 200
end

Now a simple “hello world” page with cache functions

<?php
function html_compress($html){
#for compression code read previous post 
#http://gadelkareem.com/2007/06/23/compressing-your-html-css-and-javascript-using-simple-php-code/

	#caching
	if(!empty($GLOBALS['cachethis'])){ 
		if($GLOBALS['cachethis'] == 1){
				$filename = '/myserver/cache/'.md5($_SERVER['REQUEST_URI']);
				$gz = gzopen($filename, "w9");
				gzwrite($gz, $html);
				gzclose($gz);
				#_echo('No Cache');
		}else #you can add special cookie or keyword that you generate during the script and store in memory as alternative cache
			apc_store($_SERVER['REQUEST_URI'].$GLOBALS['cachethis'],$html,86400); 
	}
	return $html;
}

#enable caching for this script
$cachethis = 1;

#alternative cache in memory using APC
if($cachethis == 2 && $html=apc_fetch($_SERVER['REQUEST_URI'].'2')){
	echo $html;
	exit;
}


#add page content to $html just dont echo anything
$html = "";
for($i=0;$i<10000;$i++)
	$html .= "Hello World<br>";


#displaying page content and caching it
echo html_compress($html);

?>

Server info :

~: cat /proc/cpuinfo
vendor_id       : GenuineIntel
model name      : Intel(R) Celeron(R) CPU 2.66GHz
cpu MHz         : 2668.432
cache size      : 256 KB

~: cat /proc/meminfo
MemTotal:       239792 kB
MemFree:         12336 kB

let’s check the difference with ab


#benchmark using ab with compression enabled, as most of today browsers do have gzip
ab -n2000 -c400  -H 'Accept-Encoding: gzip,deflate' http://gadelkareem.com/helloworld.php

#with $cachethis = 0; -- No Caching (no mod_magnet)
Requests per second:    99.55 [#/sec] (mean)
Time per request:       4018.271 [ms] (mean)
Time per request:       10.046 [ms] (mean, across all concurrent requests)


#with $cachethis = 2; -- Caching with APC (no mod_magnet)
Requests per second:    131.93 [#/sec] (mean)
Time per request:       3031.818 [ms] (mean)
Time per request:       7.580 [ms] (mean, across all concurrent requests)

#with $cachethis = 1; -- Caching with mod_magnet
Requests per second:    899.37 [#/sec] (mean)                                      <----------------------------|-
Time per request:       444.758 [ms] (mean)                                        <----------------------------|-
Time per request:       1.112 [ms] (mean, across all concurrent requests)          <----------------------------|-


#what about old browsers, will lua still be faster after decompressing data
ab -n2000 -c400 http://gadelkareem.com/helloworld.php

#with $cachethis = 0; -- No Caching (no mod_magnet)
Requests per second:    110.72 [#/sec] (mean)
Time per request:       5419.094 [ms] (mean)
Time per request:       9.032 [ms] (mean, across all concurrent requests)



#with $cachethis = 2; -- Caching with APC (no mod_magnet)
Requests per second:    156.24 [#/sec] (mean)
Time per request:       2560.163 [ms] (mean)
Time per request:       6.400 [ms] (mean, across all concurrent requests)



#with $cachethis = 1; -- Caching with mod_magnet
Requests per second:    112.84 [#/sec] (mean)
Time per request:       3544.811 [ms] (mean)
Time per request:       8.862 [ms] (mean, across all concurrent requests)

make sure to clear your cache folder before running ab.

I have to say I was amazed by the results myself, you're giving users static cached pages while you are enjoying your dynamic code, if your system is built with that on mind you should have no problem caching all your pages. Additionally, you can add cookies or browser language to the requested URI using Lua before searching for its Id in the cache. I would recommend using Javascript to read cookies instead or disable cache while users are logged-in.

Clearing cache
Since we did not check the mtime of cached files we should create a cron job to clear old files, let's say over 10 days old

0  7  */3  *  *  find /myserver/cache -type f -mtime +10 | xargs -r rm;