Creating Dynamic Websites with Lisp and Apache

Creating Dynamic Websites with Lisp and Apache

apache + mod_lisp + cmucl + lml + cl-ppcre + clsql + mysql

Introduction

Common Lisp is a
standard

programming language with a superlative development environment and
excellent documentation.
apache is a reliable, highly configurable
web server. You can use an apache module, mod_lisp, to allow a
lisp process to communicate with the apache web server. With this
approach, you can dynamically generate web pages and provide web
services with lisp programs. Below, I describe the way I created
this site, which includes dynamically created content, served
from a mysql database. The approach I describe is by no means the only
way to use lisp programs to serve web pages. There are complete
http servers written in lisp:
CL-HTTP,
araneida,
AllegroServe,
portable allegroserve,
and others.
Le Sursis uses lisp
to handle CGI requests, and employs a markup language similar to lml.
IMHO

uses apache with mod_lisp or mod_webapp.

Requirements

Here's what I used:

  • linux version 2.2.19,
    slackware distribution running on x86,
    although debian or gentoo et al. are fine too.
  • cmucl lisp distribution,
    version 18d in this
    case, although sbcl
    should also work without modifying anything else here.
  • asdf lisp library
    management system
  • cl-ppcre lisp regular expressions library
  • clsql lisp sql database library
  • uffi lisp foreign function interface library
  • lml html lisp markup library
  • mysql database, version 3.23.49 in this case
    , although I could
    have used
    postgresql
    just as easily with clsql
  • emacs text editor with
    ilisp

    ,although other editors are fine, too.

  • gcc C compiler
  • apache web server,
    version 1.3.27, in this case.
  • mod_lisp apache module for
    communication with lisp, version 2.2 in this case.
  • detachtty, a C program
    which enables you to run your lisp application as a unix daemon
    process and to connect remotely to your running lisp process for
    maintenance.
  • webdemo.cl lisp library of functions.

The software programs listed above are generously licensed under the
GPL, LLGPL, BSD license, modified BSD, or are public domain.

apache + mod_lisp communication block diagram

Preparation

You still have some work to do once you download the software to your host
machine. You may also spread out the applications among as many as three
machines, one each for the lisp process, the apache server, and the mysql

database.

Preparing apache with mod_lisp

You must make the mod_lisp.c source code accessible to apache source tree,
and then set
the appropriate flag at configure and compile time to compile the loadable
source library,
mod_lisp.so. You will need to add something like this to your configure
invocation when you are building apache:

bash% ./configure \
  --add-module=/usr/local/src/mod_lisp/mod_lisp.c \
  --enable-shared=lisp

After compiling, follow the procedure in the
mod_lisp documentation to properly
edit httpd.conf to load the library and call the lisp handler.
In the case of this site, the handler location is lispy:

   LoadModule   mod_lisp    libexec/mod_lisp.so
    ...
    ...
  AddModule mod_lisp.c
    ...
    ...
  LispServer IP.IP.IP.IP  3000 "lisp"
    ...
    ...
  < Location /lispy >  
      SetHandler   lisp-handler 
  </Location > 

Preparing the lisp system

Install cmucl,
install ilisp in emacs, if that's your editor. Download asdf.lisp
into your working directory, create a ~/systems/ directory where you
will keep your library registry. Consider editing asdf.lisp to include
this directory path in the *central-registry* list. Download and install
the lisp libraries,
clsql, uffi, lml, and cl-ppcre. You have to compile
some C libraries in clsql
and uffi, so be sure to check the documentation
in each of these libraries. After you have prepared them, enter the ~/systems
directory and create links to the xxxx.asd files in each of the libraries:

    base64.asd -> ../cl-base64-3.0.2/base64.asd
   cl-ppcre.asd -> ../cl-ppcre-0.3.1/cl-ppcre.asd
   clsql-base.asd -> ../clsql-1.3.0/clsql-base.asd
   clsql-mysql.asd -> ../clsql-1.3.0/clsql-mysql.asd
   clsql-uffi.asd -> ../clsql-1.3.0/clsql-uffi.asd
   clsql.asd -> ../clsql-1.3.0/clsql.asd
   lml.asd -> ../lml-2.2.0/lml.asd
   uffi.asd -> ../../../usr/local/src/uffi-1.1.0/uffi.asd

Now when you execute the following code in your lisp session, the libraries
are loaded:

    (load "/home/yourdir/asdf.lisp")
   (asdf:oos `asdf:load-op :uffi)
   (asdf:oos `asdf:load-op :clsql)
   (asdf:oos `asdf:load-op :clsql-base)
   (asdf:oos `asdf:load-op :clsql-uffi)
   (asdf:oos `asdf:load-op :clsql-mysql)
   (asdf:oos `asdf:load-op :lml)
   (asdf:oos `asdf:load-op :base64)
   (asdf:oos `asdf:load-op :cl-ppcre)

Alternatively, you can put this in your .cmucl-init.lisp file.

(start-apache-listener) to communicate between apache and lisp

If you have never run mod_lisp
before, it is probably better
to load modlisp-cmucl.lisp,
available at Marc Battyani's web site

and ensure that the lisp listener is working with apache. The lisp
listener is launched with the (start-apache-listener) command and then
listens on port 3000 (a value set in httpd.conf and
in +apache-port+ in the lisp file) and communicates with the
mod_lisp module
when apache is running and configured properly.

Preparing your database

Follow the mysql documentation to install your database and initialize grant
tables. If you like,
add a user and password
especially for your clsql user.
Use command line invocations to create the database and create the table
described by:

   CREATE TABLE pagecomment (
 page varchar(255),
 name varchar(255),
 date date DEFAULT '0000-00-00',
 email varchar(20),
 title varchar(20),
 comments text,
 rating int(1) DEFAULT '0',
 id int(1) DEFAULT '0' NOT NULL auto_increment,
 PRIMARY KEY (id) ) 

A typical command line invocation sequence would be something like:

 bash%  mysqladmin create dbname
 bash% mysql -u dbuser --password=dbpasswd <  sqlfile

where sqlfile is a text file with the SQL statement above. This
creates the table which stores comments submitted to the website. The
rating field is set to 1 to allow publication.

Then edit the function dblist in the webdemo.cl
source file to reflect your host,
name of the database, the username, and the password.

Running webdemo.cl

When you've confirmed that mod_lisp
and the lisp apache listener are working together,
it's time to try the webdemo.cl application.
Stop any current lisp sessions interacting with mod_lisp, start a new one, and
load the webdemo.cl library into the cmucl session. Then start the apache
listener process, (start-apache-listener). Then, try the URL <yourdomain>

/lispy/fixed to test to see if apache and the listener are communicating.
When you have
established your mysql or postgresql database, edit the database connection
parameters to reflect your host, username, database name and password. Then try
out the yourdomain/lispy/home URL.

Once you are happy running interactively, you will want to run your lisp
web site process as a daemon, perhaps started during bootup. Use detachtty
for this, with an init script like this:

if [ -x /path/to/lispy/webdemo.cl ]; then
  echo "starting lisp webdemo..."
  rm /var/run/lisp.pid
  rm /var/run/lisp-socket
  rm /var/run/lisp-dribble
  rm /var/run/detachtty.log
  /usr/local/bin/detachtty --dribble-file /var/run/lisp-dribble \
     --log-file /var/run/detachtty.log --pid-file /var/run/lisp.pid \
     /var/run/lisp-socket  \
     /usr/bin/lisp -noinit \
     -eval '(load "/path/to/lispy/webdemo.cl")'
fi

Use the attachtty
command to connect to the lisp process to do maintenance
and updates. I cannot emphasize enough how productive the lisp development environment
is. Functions can be edited, updated, and compiled interactively without
interrupting the lisp process. In contrast to php, perl or java, you can
run your website application compiled to native x86 instructions. Use the
disassemble
function to generate an assembly code listing:

;;; [8] (OR (SECOND X) )

     D6C:       MOV   EBX, [EBP-16]
     D6F:       MOV   EDI, [EBP-12]
     D72:       MOV   BYTE PTR [#x280001D4], 0 
     D79:       MOV   BYTE PTR [#x280001BC], 4 
     D80:       MOV   EAX, 8
     D85:       ADD   EAX, [#x80697A4]       
     D8B:       CMP   EAX, [#x8069778]       
     D91:       JBE   L6
      .          .     .       .         
      .          .     .       .

This Demo Site

This site is just about the simplest, non-trivial, web site possible.
People routinely develop sites like this with scripting languages like
php,
asp,
perl,
eperl,
mod_perl, and java servelets.
I've used php in the past to create simple, dynamic sites about putting
linux on a toshiba 2100cdt laptop, putting
linux on a Compaq LTE5200 laptop, and
using
bbdb and mew together in emacs.
On this site, you can view dynamically generated content, part of which is
queried from an SQL database, and you can interactively
submit comments
to the website's database. In addition, even the
static content is generated by dynamic
rendering of content written in lml markup.
The source code,
webdemo.cl, is
written in lisp, and consists of library calls and function definitions.

For some,
separation of presentation, content, and logic
seems to be the holy grail of web
development tools.
mason,
turbine, and
smarty are some attempts to
achieve this.
It turns out that it is just remarkably hard to avoid mixing the markup,
presentation and control code in site files.
Since you can't avoid mixing without a lot of trouble, why not
consider finding a way to mix the content and code with a single style?
I'm delighted with this web site's application files
which are well structured and consistently highlighted in a
good text editor like emacs.
mod_lisp+ clsql+ lml... provides that
because the only syntax is the syntax of common lisp. Here's what
I mean:

(html 
 (head 
   (title "hello-world"))
     (body (h1 "hello-world")
       (p "hello-world greetings in several languages")
       (if (newyear) 
           (gen-table-newyear)
          (gen-table-hello))))

There's basically one syntax there for html, some control, and functions
which generate tables.
The only
other website language whose code comes close to this in terms of
readability is brl, which is built in
scheme.
Lisp source files, like webdemo.cl , can be
rendered in html with the wonderful emacs package,
htmlize.el.
Of course, that's the format which you see when you follow any of the
webdemo.cl links on this site.

Performance

For the following crude tests, the apache server and lisp image were
running on an AMD-K6(tm) 3D processor, 350 MHz , with a 64KB cache, and
290MB RAM.
I'll just let you judge for yourself about the performance of the
server. Here are the results of the apache
ab
program calling a page written
in lml which also has a read query of the database:

Server Software: Apache/1.3.22
Server Hostname: lisp.t2100cdt.kippona.net
Server Port: 80
Document Path: /lispy/home
Document Length: 15705 bytes
Concurrency Level: 1
Time taken for tests: 168.458 seconds
Complete requests: 1000
Failed requests: 0
Total transferred: 15881000 bytes
HTML transferred: 15705000 bytes
Requests per second: 5.94
Transfer rate: 94.27 kb/s received
Connnection Times (ms)
  min avg max
Connect: 0 0 0
Processing: 77 167 7977
Total: 77 167 7977

Following is the result for a similar test on a page
written solely in lml without any query to the database:

Server Software: Apache/1.3.22
Server Hostname: lisp.t2100cdt.kippona.net
Server Port: 80
Document Path: /lispy/thiscode
Document Length: 60852 bytes
Concurrency Level: 1
Time taken for tests: 27.787 seconds
Complete requests: 1000
Failed requests: 0
Total transferred: 61028000 bytes
HTML transferred: 60852000 bytes
Requests per second: 35.99
Transfer rate: 2196.28 kb/s received
Connnection Times (ms)
  min avg max
Connect: 0 0 2
Processing: 9 27 2189
Total: 9 27 2191

Keep in mind that the very popular apache webserver version 1.3.x is
not designed with native
threads,
but rather relies on child processes which are preforked or forked on demand.
CMUCL has only user threads

which I do not use in this application.
Dan Barlow and the team designing
SBCL
are implementing native threads in SBCL. In addition, a commercial lisp
derived from CMUCL,
Scieneer Common Lisp (SCL),
incorporates native
threads.
Native threads capability will improve the performance of this apache ab
benchmark. The database queries through clsql do not use pooled
connections but at least reuse open connections to mysql.

Conclusions

CMU Common Lisp works well with apache and mod_lisp. The
lisp development
environment is extremely useful for quick prototyping, troubleshooting,
and maintenance of
a simple database backed web site. The web site is written in a single
syntax, lisp syntax, and can be efficiently edited in emacs. The toolset
provides a straightforward migration path from a php + apache, or a
mod_perl + apache, or a
java + mod_jk + apache database backed website.

On the other hand, I am not taking
advantage of pooled or persistent database connections between the lisp process
and mysql, which would improve performance. Somehow the inclusion
of document type definition statements causes rendering by css generated
by a lisp function to fail. It does work if the css is called from an external
static css page on the server. The
SQL query statements are strings rather than lisp syntax. Although I claim
the syntax of my source files is all lisp, I
effortlessly violate
most common lisp style rules

and
macrology guidelines.

Acknowledgements

Thanks to Marc Battyani, author of mod_lisp. Thanks to Kevin Rosenberg, author of
clsql, lml, and uffi. Thanks to Edi Weitz, author of cl-ppcre.
Thanks to Dan Barlow, author of the
CMUCL on Linux Howto and
detachtty and a visionary
blazing the trail to make
common lisp play nicely with unix.
Thanks to Shiro Kawai whose articles

Shooting a Moving Target and
Tracking Assets
opened my eyes to using common lisp
to get real work done on Unix boxes. These articles are also
the finest project post-mortem documents I have ever read.

Submit Comments

Here is where you can
submit
comments on this website. New comments appear after I review them.

Comments



2003-02-04

first one

Comments are welcome

chrisb chrisb@kippona.com



2003-02-18

lispnyc can use this

cool. i'm a web no-nothing, but those
in lispnyc that know more are interested
in having our web site be lisp-based.
This could be a big help.

Kenny Tilton ktilton@nyc.rr.com



2003-02-18

Very nice

Very nice. All ingredients are here and their relation are explained.

Although it is a little frightning to see how much you need:

apache + mod_lisp + cmucl + lml + cl-ppcre + clsql + mysql

Jens Axel S�gaard jensaxel@soegaard.net



2003-02-18

Throw in UncommonSQL

Nice!

If you throw in UncommonSQL
you can use Lispy syntax to do
the SQL, e.g.,
(setf blob
(car (select 'blob-table
:where [

Ng Pheng Siong ngps@netmemetic.com



2003-02-19

Very impressive

Easy to read, easy to learn and complex enough at the same time.

Vladimir Sekissov svg@surnet.ru



2003-02-19

ah ha!

just what the doctor ordered... beware now php!!

quasi quasiabhi at yahoo.com



2003-02-23

Very Cool

Yes, very cool and useful, I have put links to this page on the mod_lisp pages.

Marc Battyani marc.battyani@fractalconcept.com



2003-02-23

A good read - but is the .css GPLed too?

Yay! The clear description and understandable source code make this is a very useful place to point frustrated php coders to.

But one question remains: Is the .css file you used for this web site also covered by the GPL? (-;

Andreas Fuchs asf@void.at



2003-03-03

Cool Site

This is very cool. Thanks for the contribution.

Reggie reggie@lambdawerks.org



2003-03-06

Nice Site! :-)

The GPL'd database for Loads of Linux Links now includes this site.
See
http://loll.sourceforge.net/linux/links/index.html
for the details.

Barbara I. Irwin beirwin@shaw.ca



2003-03-07

Followups

Thanks for the comments. I'll offer some followups:

  • Jens - Maybe some of the libraries can be bundled (clsql, uffi, lml) to make installation less imposing
  • Ng - I agree, lispy SQL queries are superior to CLSQL's quoted string queries. CLSQL is GPL, so the rest is up to us!
  • Andreas - I inserted the GPL into the .css file after your inquiry.

Chris Beggy chrisb@kippona.com



2003-03-07

Thanks

Thanks to some webloggers who found out about this site and reviewed it:

Chris chrisb@kippona.com



2003-08-11

creating Dynamic Websites

This is a very useful 'getting started' document. A few comments:
about your use of asdf:

  1. it probably runs faster if you load the compiled version
  2. rather than editing asdf.lisp directly to add ~/systems to it,
    consider loading it from .cmucl-init.lisp and then doing

    (pushnew "/home/yourdir/systems/" asdf:*central-registry*)
  3. instead of loading each of those systems by hand, write a system
    of your own that depends on them all. Call this webdemo.asd and
    put it with the other files

    (defpackage :webdemo-system (:use "CL" "ASDF"))
          (in-package :webdemo-system)
          (defsystem webdemo
                 :components (("webdemo" :pathname "webdemo.cl"))
                 :depends-on (uffi clsql clsql-base clsql-uffi clsql-mysql
                              lml base64 cl-ppcre))

    (untested, may need frobbing)

  4. unsolicited SBCL plug: as of SBCL 0.8.3 you'll be able to download
    and install many of these libararies automatically with a single
    command. See
    http://www.cliki.net/asdf-install :-)

Dan Barlow dan@telent.net


Valid XHTML 1.0! Valid CSS! Powered by LML! Copyright &copy 2003 Chris Beggy