Dynamic Content Caching using Lighty + mod_magnet + lua

*Updated 11 Jan 2007 to include luazlib, fix lighty config file and add zlib decoding to cache.lua for older browsers
Since I read the documentations for mod_cml, I was very excited to use this module since caching using PHP running as FastCGI is not helping much during server peak load. However, mod_cml was replaced by mod_magnet which is more flexible and gives more control over request handling in Lighttpd. This Article will focus on caching your PHP scripts using Lua and mod_magnet under Lighttpd, You should read “Compressing your HTML, CSS and Javascript using simple PHP Code" as I will use some functions from there

Installing Lua 5.1.2 + md5 lib

  1. wget http://www.lua.org/ftp/lua-5.1.2.tar.gz
  2. tar xfz lua-5.1.2.tar.gz
  3. cd lua-5.1.2
  4. yum install readline-devel
  5. make linux install
  6. cd ..
  7. wget http://luaforge.net/frs/download.php/2384/md5-1.0.2.tar.gz
  8. tar xfz md5-1.0.2.tar.gz
  9. cd md5-1.0.2
  10. make
  11. make install
  12. cd ..
  13. wget http://luaforge.net/frs/download.php/1678/luazlib-0.0.1.rar
  14. #if only you have to unrar on linux
  15. #get your compatible unrar binary from
  16. #http://www.rarlab.com/rar_add.htm
  17. wget wget http://www.rarlab.com/rar/unrar-3.7.7-centos.gz
  18. gunzip unrar-3.7.7-centos.gz
  19. chmod +x unrar-3.7.7-centos
  20.  
  21. ./unrar-3.7.7-centos x luazlib-0.0.1.rar
  22. cd luazlib-0.0.1
  23. make
  24. make install
  25.  

Installing Lighttpd 1.4.18 with mod_magnet support

  1. wget http://www.lighttpd.net/download/lighttpd-1.4.18.tar.bz2
  2. tar xfj lighttpd-1.4.18.tar.bz2
  3. cd lighttpd-1.4.18
  4. export LUA_CFLAGS="-I/usr/local/include"
  5. export LUA_LIBS="-L/usr/local/lib -llua"
  6. ./configure –with-lua
  7. make
  8. make install

Enable mod_magnet on your lighttpd.conf

  1. server.modules += ( "mod_magnet" )
  2. #use cache.lua script only on php script from domain gadelkareem.com
  3. $HTTP["host"] == "gadelkareem.com" {
  4.         $HTTP["url"] =~ "(^/(\?.*)?$|\.php)" {
  5.                 magnet.attract-physical-path-to = ( "/myserver/cache.lua" )
  6.         }
  7. }

Now every time a request to .php file will be handled by cache.lua script to see if we already have a cached copy of that script.
Here is cache.lua

  1.  
  2. require"md5"
  3.  
  4.  
  5. –is request empty
  6. if ( lighty.env["request.uri"] == nil ) then
  7.         lighty.env["request.uri"] = "/"
  8. end
  9.  
  10. local file = "/server/cache/" .. md5.sumhexa( lighty.env["request.uri"] )
  11.  
  12.  
  13. if lighty.stat(file) then
  14.         –is encoding empty
  15.         if ( lighty.request["accept-encoding"] == nil ) then
  16.                  lighty.request["accept-encoding"] = ""
  17.         end
  18.         if( string.find( lighty.request["accept-encoding"] , "gzip") ) then
  19.                 lighty.content = { { filename = file } }
  20.                 lighty.header["Content-Encoding"] = "gzip"
  21.         else
  22.                 require"zlib"
  23.                 local ht = io.open(file, "r")
  24.                 local t = ht:read("*a")
  25.                 ht:close()
  26.                 lighty.content = { zlib.gzuncompress(t) .. ""}
  27.         end
  28.         lighty.header["Content-Type"] = "text/html"
  29.         return 200
  30. end
  31.  

Now a simple “hello world" page with cache functions

  1. <?php
  2. function html_compress($html){
  3. #for compression code read previous post
  4. #http://gadelkareem.com/2007/06/23/compressing-your-html-css-and-javascript-using-simple-php-code/
  5.  
  6.         #caching
  7.         if(!empty($GLOBALS['cachethis'])){
  8.                 if($GLOBALS['cachethis'] == 1){
  9.                                 $filename = '/myserver/cache/'.md5($_SERVER['REQUEST_URI']);
  10.                                 $gz = gzopen($filename, "w9");
  11.                                 gzwrite($gz, $html);
  12.                                 gzclose($gz);
  13.                                 #_echo('No Cache');
  14.                 }else #you can add special cookie or keyword that you generate during the script and store in memory as alternative cache
  15.                         apc_store($_SERVER['REQUEST_URI'].$GLOBALS['cachethis'],$html,86400);
  16.         }
  17.         return $html;
  18. }
  19.  
  20. #enable caching for this script
  21. $cachethis = 1;
  22.  
  23. #alternative cache in memory using APC
  24. if($cachethis == 2 && $html=apc_fetch($_SERVER['REQUEST_URI'].'2′)){
  25.         echo $html;
  26.         exit;
  27. }
  28.  
  29.  
  30. #add page content to $html just dont echo anything
  31. $html = "";
  32. for($i=0;$i<10000;$i++)
  33.         $html .= "Hello World<br>";
  34.  
  35.  
  36. #displaying page content and caching it
  37. echo html_compress($html);
  38.  
  39. ?>

Server info :

  1.  
  2. ~: cat /proc/cpuinfo
  3. vendor_id       : GenuineIntel
  4. model name      : Intel(R) Celeron(R) CPU 2.66GHz
  5. cpu MHz         : 2668.432
  6. cache size      : 256 KB
  7.  
  8. ~: cat /proc/meminfo
  9. MemTotal:       239792 kB
  10. MemFree:         12336 kB
  11.  
  12.  

let’s check the difference with ab

  1.  
  2.  
  3. #benchmark using ab with compression enabled, as most of today browsers do have gzip
  4. ab -n2000 -c400  -H 'Accept-Encoding: gzip,deflate' http://gadelkareem.com/helloworld.php
  5.  
  6. #with $cachethis = 0; — No Caching (no mod_magnet)
  7. Requests per second:    99.55 [#/sec] (mean)
  8. Time per request:       4018.271 [ms] (mean)
  9. Time per request:       10.046 [ms] (mean, across all concurrent requests)
  10.  
  11.  
  12. #with $cachethis = 2; — Caching with APC (no mod_magnet)
  13. Requests per second:    131.93 [#/sec] (mean)
  14. Time per request:       3031.818 [ms] (mean)
  15. Time per request:       7.580 [ms] (mean, across all concurrent requests)
  16.  
  17. #with $cachethis = 1; — Caching with mod_magnet
  18. Requests per second:    899.37 [#/sec] (mean)  <—————————-|-
  19. Time per request:       444.758 [ms] (mean)
  20. Time per request:       1.112 [ms] (mean, across all concurrent requests)
  21.  
  22.  
  23. #what about old browsers, will lua still be faster after decompressing data
  24. ab -n2000 -c400 http://gadelkareem.com/helloworld.php
  25.  
  26. #with $cachethis = 0; — No Caching (no mod_magnet)
  27. Requests per second:    110.72 [#/sec] (mean)
  28. Time per request:       5419.094 [ms] (mean)
  29. Time per request:       9.032 [ms] (mean, across all concurrent requests)
  30.  
  31.  
  32.  
  33. #with $cachethis = 2; — Caching with APC (no mod_magnet)
  34. Requests per second:    156.24 [#/sec] (mean)
  35. Time per request:       2560.163 [ms] (mean)
  36. Time per request:       6.400 [ms] (mean, across all concurrent requests)
  37.  
  38.  
  39.  
  40. #with $cachethis = 1; — Caching with mod_magnet
  41. Requests per second:    112.84 [#/sec] (mean)
  42. Time per request:       3544.811 [ms] (mean)
  43. Time per request:       8.862 [ms] (mean, across all concurrent requests)
  44.  
  45.  

make sure to clear your cache folder before running ab.

I have to say I was amazed by the results myself, you're giving users static cached pages while you are enjoying your dynamic code, if your system is built with that on mind you should have no problem caching all your pages. Additionally, you can add cookies or browser language to the requested URI using Lua before searching for its Id in the cache. I would recommend using Javascript to read cookies instead or disable cache while users are logged-in.

Clearing cache
Since we did not check the mtime of cached files we should create a cron job to clear old files, let’s say over 10 days old

  1. 0  7  */3  *  *  find /myserver/cache -type f -mtime +10 | xargs -r rm;

Tags :

This entry was posted on Monday, September 17th, 2007 at 8:19 am and is filed under Blog, Solutions. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

5 Responses to “Dynamic Content Caching using Lighty + mod_magnet + lua”

  1. Tiago Fischer Says:
    February 3rd, 2008 at 9:17 pm

    Hi!
    This is a very nice idea!

    But, considering $html as the web page content, i need to rewrite all my scripts…
    How can I cache the page without putting the content to $html var?

    Any idea?
    Thanks!

  2. wkarim Says:
    February 4th, 2008 at 3:01 am

    @Tiago Fischer
    Yes, you can use Output Control Functions to store the output buffer. Please review the examples on that page and ob_get_contents function page.

  3. Vic Says:
    June 8th, 2009 at 9:21 am

    Your solution might work if cache files aren't constantly being updated or used, but what about open file pointers? Unless I'm missing something, with large files and open file pointer conflicts, you could be generating a lot of lost+found files only retrievable by a disk repair when clearing the cache that way, because lighttpd or the fastcgi processes may have them open for reading/writing. So best case you avoid any conflicts, worst case you interrupt files being served and lose files in the filesystem.

  4. Lighttpd + mod_magnet on CentOS (64 bit) | BlueScripts Says:
    August 18th, 2009 at 5:20 pm

    [...] to: http://gadelkareem.com/2007/09/17/dynamic-content-caching-using-lighty-mod_magnet-lua/ && [...]

  5. wesnur Says:
    December 15th, 2009 at 11:58 am

    It’s really very interesting. There is no doubt that dynamic caching is the ultimate solution for the issues like Scalability, Reliability and Performance. State of the growth unfolds many opportunities but is also holds challenges as a result of change. Not only does performance affect your operational cost, it also reflects directly on end-users experience. I would like to share another very informative read with you people. Benefits of using NCache Dynamic Clustering Capabilities not only describes the benefits of cache but it also highlights the some other important aspects like different topologies InProc Mode, OutProc Mode etc.

 

Leave a Reply


 Top