Dynamic Content Caching using Lighty + mod_magnet + lua

*Updated 11 Jan 2007 to include luazlib, fix lighty config and add zlib decoding to cache.lua for older browsers
Since I read the documentations for mod_cml, I was very excited to use this module since caching PHP running as FastCGI is not helping much during peak load. However, mod_cml was replaced by mod_magnet which is more flexible and gives more control over request handling in Lighttpd. This Article will focus on caching your scripts using Lua and mod_magnet under Lighttpd, You should read “Compressing your HTML, CSS and Javascript using simple PHP Code” as I will use some functions from there

Installing Lua 5.1.2 + md5 lib
tar xfz lua-5.1.2.tar.gz
cd lua-5.1.2
yum install readline-devel
make linux install
cd ..
tar xfz md5-1.0.2.tar.gz
cd md5-1.0.2
make install
cd ..
#if only you have to unrar on linux
#get your compatible unrar binary from
wget wget http://www.rarlab./rar/unrar-3.7.7-centos.gz
gunzip unrar-3.7.7-centos.gz
chmod +x unrar-3.7.7-centos

./unrar-3.7.7-centos x luazlib-0.0.1.rar
cd luazlib-0.0.1
make install

Installing Lighttpd 1.4.18 with mod_magnet support

tar xfj lighttpd-1.4.18.tar.bz2
cd lighttpd-1.4.18
export LUA_CFLAGS="-I/usr/local/include" 
export LUA_LIBS="-L/usr/local/lib -llua"
./configure --with-lua
make install

Enable mod_magnet on your lighttpd.

server.modules += ( "mod_magnet" )
#use cache.lua script only on php script from domain
$HTTP["host"] == "" {
	$HTTP["url"] =~ "(^/(\?.*)?$|\.php)" {
		magnet.attract-physical-path-to = ( "/myserver/cache.lua" )

Now every time a request to .php file will be handled by cache.lua script to see if we already have a cached copy of that script.
Here is cache.lua


--is request empty
if ( lighty.env["request.uri"] == nil ) then 
	lighty.env["request.uri"] = "/"

local file = "/server/cache/" .. md5.sumhexa( lighty.env["request.uri"] )

if lighty.stat(file) then
	--is encoding empty
	if ( lighty.request["accept-encoding"] == nil ) then 
		 lighty.request["accept-encoding"] = ""
	if( string.find( lighty.request["accept-encoding"] , "gzip") ) then
		lighty.content = { { filename = file } }
		lighty.header["Content-Encoding"] = "gzip"
		local ht = .open(file, "r")
		local t = ht:read("*a")
		lighty.content = { zlib.gzuncompress(t) .. ""}
	lighty.header["Content-Type"] = "text/html"
	return 200

Now a simple “hello world” page with cache functions

function html_compress($html){
#for compression  read previous post 

		if($GLOBALS['cachethis'] == 1){
				$filename = '/myserver/cache/'.md5($_SERVER['REQUEST_URI']);
				$gz = gzopen($filename, "w9");
				gzwrite($gz, $html);
				#_echo('No Cache');
		}else #you can add special cookie or keyword that you generate during the script and store in memory as alternative cache
	return $html;

#enable caching for this script
$cachethis = 1;

#alternative cache in memory using APC
if($cachethis == 2 && $html=apc_fetch($_SERVER['REQUEST_URI'].'2')){
	echo $html;

#add page content to $html just dont echo anything
$html = "";
	$html .= "Hello World<br>";

#displaying page content and caching it
echo html_compress($html);


Server info :

~: cat /proc/cpuinfo
vendor_id       : GenuineIntel
model name      : Intel(R) Celeron(R)  2.66GHz
 MHz         : 2668.432
cache size      : 256 KB

~: cat /proc/meminfo
MemTotal:       239792 kB
MemFree:         12336 kB

let’s check the difference with ab

# using ab with compression enabled, as most of today browsers do have gzip
ab -n2000 -c400  -H 'Accept-Encoding: gzip,deflate'

#with $cachethis = 0; -- No Caching (no mod_magnet)
Requests per second:    99.55 [#/sec] (mean)
Time per request:       4018.271 [ms] (mean)
Time per request:       10.046 [ms] (mean, across all concurrent requests)

#with $cachethis = 2; -- Caching with APC (no mod_magnet)
Requests per second:    131.93 [#/sec] (mean)
Time per request:       3031.818 [ms] (mean)
Time per request:       7.580 [ms] (mean, across all concurrent requests)

#with $cachethis = 1; -- Caching with mod_magnet
Requests per second:    899.37 [#/sec] (mean)                                      <----------------------------|-
Time per request:       444.758 [ms] (mean)                                        <----------------------------|-
Time per request:       1.112 [ms] (mean, across all concurrent requests)          <----------------------------|-

#what about old browsers, will lua still be faster after decompressing data
ab -n2000 -c400

#with $cachethis = 0; -- No Caching (no mod_magnet)
Requests per second:    110.72 [#/sec] (mean)
Time per request:       5419.094 [ms] (mean)
Time per request:       9.032 [ms] (mean, across all concurrent requests)

#with $cachethis = 2; -- Caching with APC (no mod_magnet)
Requests per second:    156.24 [#/sec] (mean)
Time per request:       2560.163 [ms] (mean)
Time per request:       6.400 [ms] (mean, across all concurrent requests)

#with $cachethis = 1; -- Caching with mod_magnet
Requests per second:    112.84 [#/sec] (mean)
Time per request:       3544.811 [ms] (mean)
Time per request:       8.862 [ms] (mean, across all concurrent requests)

make sure to clear your cache folder before running ab.

I have to say I was amazed by the results myself, you're giving users static cached pages while you are enjoying your dynamic code, if your system is built with that on mind you should have no problem caching all your pages. Additionally, you can add cookies or browser language to the requested URI using Lua before searching for its Id in the cache. I would recommend using Javascript to read cookies instead or disable cache while users are logged-in.

Clearing cache
Since we did not check the mtime of cached files we should create a cron job to clear old files, let's say over 10 days old

0  7  */3  *  *  find /myserver/cache -type f -mtime +10 | xargs -r rm;
  • Hi!
    This is a very nice idea!

    But, considering $html as the web page content, i need to rewrite all my scripts…
    How can I cache the page without putting the content to $html var?

    Any idea?

  • @Tiago Fischer
    Yes, you can use Output Control Functions to store the output buffer. Please review the examples on that page and ob_get_contents function page.

  • Vic

    Your solution might work if cache files aren’t constantly being updated or used, but what about open file pointers? Unless I’m missing something, with large files and open file pointer conflicts, you could be generating a lot of lost+found files only retrievable by a disk repair when clearing the cache that way, because lighttpd or the fastcgi processes may have them open for reading/writing. So best case you avoid any conflicts, worst case you interrupt files being served and lose files in the filesystem.

  • Pingback: Lighttpd + mod_magnet on CentOS (64 bit) | BlueScripts()

  • wesnur

    It’s really very interesting. There is no doubt that dynamic caching is the ultimate solution for the issues like Scalability, Reliability and Performance. State of the growth unfolds many opportunities but is also holds challenges as a result of change. Not only does performance affect your operational cost, it also reflects directly on end-users experience. I would like to share another very informative read with you people. Benefits of using NCache Dynamic Clustering Capabilities not only describes the benefits of cache but it also highlights the some other important aspects like different topologies InProc Mode, OutProc Mode etc.