Creating Dynamic Websites with Lisp and Apache
Creating Dynamic Websites with Lisp and Apache
apache + mod_lisp + cmucl + lml + cl-ppcre + clsql + mysql
programming language with a superlative development environment and
apache is a reliable, highly configurable
web server. You can use an apache module, mod_lisp, to allow a
lisp process to communicate with the apache web server. With this
approach, you can dynamically generate web pages and provide web
services with lisp programs. Below, I describe the way I created
this site, which includes dynamically created content, served
from a mysql database. The approach I describe is by no means the only
way to use lisp programs to serve web pages. There are complete
http servers written in lisp:
Le Sursis uses lisp
to handle CGI requests, and employs a markup language similar to lml.
Here's what I used:
- linux version 2.2.19,
slackware distribution running on x86,
although debian or gentoo et al. are fine too.
- cmucl lisp distribution,
version 18d in this
case, although sbcl
should also work without modifying anything else here.
- asdf lisp library
- cl-ppcre lisp regular expressions library
- clsql lisp sql database library
- uffi lisp foreign function interface library
- lml html lisp markup library
- mysql database, version 3.23.49 in this case
, although I could
just as easily with clsql
- emacs text editor with
,although other editors are fine, too.
- gcc C compiler
- apache web server,
version 1.3.27, in this case.
- mod_lisp apache module for
communication with lisp, version 2.2 in this case.
- detachtty, a C program
which enables you to run your lisp application as a unix daemon
process and to connect remotely to your running lisp process for
- webdemo.cl lisp library of functions.
You still have some work to do once you download the software to your host
machine. You may also spread out the applications among as many as three
machines, one each for the lisp process, the apache server, and the mysql
Preparing apache with mod_lisp
You must make the mod_lisp.c source code accessible to apache source tree,
and then set
the appropriate flag at configure and compile time to compile the loadable
mod_lisp.so. You will need to add something like this to your configure
invocation when you are building apache:
bash% ./configure \ --add-module=/usr/local/src/mod_lisp/mod_lisp.c \ --enable-shared=lisp
After compiling, follow the procedure in the
mod_lisp documentation to properly
edit httpd.conf to load the library and call the lisp handler.
In the case of this site, the handler location is lispy:
LoadModule mod_lisp libexec/mod_lisp.so ... ... AddModule mod_lisp.c ... ... LispServer IP.IP.IP.IP 3000 "lisp" ... ... < Location /lispy > SetHandler lisp-handler </Location >
Preparing the lisp system
install ilisp in emacs, if that's your editor. Download asdf.lisp
into your working directory, create a ~/systems/ directory where you
will keep your library registry. Consider editing asdf.lisp to include
this directory path in the *central-registry* list. Download and install
the lisp libraries,
clsql, uffi, lml, and cl-ppcre. You have to compile
some C libraries in clsql
and uffi, so be sure to check the documentation
in each of these libraries. After you have prepared them, enter the ~/systems
directory and create links to the xxxx.asd files in each of the libraries:
base64.asd -> ../cl-base64-3.0.2/base64.asd cl-ppcre.asd -> ../cl-ppcre-0.3.1/cl-ppcre.asd clsql-base.asd -> ../clsql-1.3.0/clsql-base.asd clsql-mysql.asd -> ../clsql-1.3.0/clsql-mysql.asd clsql-uffi.asd -> ../clsql-1.3.0/clsql-uffi.asd clsql.asd -> ../clsql-1.3.0/clsql.asd lml.asd -> ../lml-2.2.0/lml.asd uffi.asd -> ../../../usr/local/src/uffi-1.1.0/uffi.asd
Now when you execute the following code in your lisp session, the libraries
(load "/home/yourdir/asdf.lisp") (asdf:oos `asdf:load-op :uffi) (asdf:oos `asdf:load-op :clsql) (asdf:oos `asdf:load-op :clsql-base) (asdf:oos `asdf:load-op :clsql-uffi) (asdf:oos `asdf:load-op :clsql-mysql) (asdf:oos `asdf:load-op :lml) (asdf:oos `asdf:load-op :base64) (asdf:oos `asdf:load-op :cl-ppcre)
Alternatively, you can put this in your .cmucl-init.lisp file.
(start-apache-listener) to communicate between apache and lisp
and ensure that the lisp listener is working with apache. The lisp
listener is launched with the (start-apache-listener) command and then
listens on port 3000 (a value set in httpd.conf and
in +apache-port+ in the lisp file) and communicates with the
when apache is running and configured properly.
Preparing your database
Follow the mysql documentation to install your database and initialize grant
tables. If you like,
add a user and password
especially for your clsql user.
Use command line invocations to create the database and create the table
CREATE TABLE pagecomment ( page varchar(255), name varchar(255), date date DEFAULT '0000-00-00', email varchar(20), title varchar(20), comments text, rating int(1) DEFAULT '0', id int(1) DEFAULT '0' NOT NULL auto_increment, PRIMARY KEY (id) )
A typical command line invocation sequence would be something like:
bash% mysqladmin create dbname bash% mysql -u dbuser --password=dbpasswd < sqlfile
where sqlfile is a text file with the SQL statement above. This
creates the table which stores comments submitted to the website. The
rating field is set to 1 to allow publication.
Then edit the function dblist in the webdemo.cl
source file to reflect your host,
name of the database, the username, and the password.
When you've confirmed that mod_lisp
and the lisp apache listener are working together,
it's time to try the webdemo.cl application.
Stop any current lisp sessions interacting with mod_lisp, start a new one, and
load the webdemo.cl library into the cmucl session. Then start the apache
listener process, (start-apache-listener). Then, try the URL <yourdomain>
/lispy/fixed to test to see if apache and the listener are communicating.
When you have
established your mysql or postgresql database, edit the database connection
parameters to reflect your host, username, database name and password. Then try
out the yourdomain/lispy/home URL.
Once you are happy running interactively, you will want to run your lisp
web site process as a daemon, perhaps started during bootup. Use detachtty
for this, with an init script like this:
if [ -x /path/to/lispy/webdemo.cl ]; then echo "starting lisp webdemo..." rm /var/run/lisp.pid rm /var/run/lisp-socket rm /var/run/lisp-dribble rm /var/run/detachtty.log /usr/local/bin/detachtty --dribble-file /var/run/lisp-dribble \ --log-file /var/run/detachtty.log --pid-file /var/run/lisp.pid \ /var/run/lisp-socket \ /usr/bin/lisp -noinit \ -eval '(load "/path/to/lispy/webdemo.cl")' fi
Use the attachtty
command to connect to the lisp process to do maintenance
and updates. I cannot emphasize enough how productive the lisp development environment
is. Functions can be edited, updated, and compiled interactively without
interrupting the lisp process. In contrast to php, perl or java, you can
run your website application compiled to native x86 instructions. Use the
function to generate an assembly code listing:
;;;  (OR (SECOND X) ) D6C: MOV EBX, [EBP-16] D6F: MOV EDI, [EBP-12] D72: MOV BYTE PTR [#x280001D4], 0 D79: MOV BYTE PTR [#x280001BC], 4 D80: MOV EAX, 8 D85: ADD EAX, [#x80697A4] D8B: CMP EAX, [#x8069778] D91: JBE L6 . . . . . . . .
This site is just about the simplest, non-trivial, web site possible.
People routinely develop sites like this with scripting languages like
mod_perl, and java servelets.
I've used php in the past to create simple, dynamic sites about putting
linux on a toshiba 2100cdt laptop, putting
linux on a Compaq LTE5200 laptop, and
bbdb and mew together in emacs.
On this site, you can view dynamically generated content, part of which is
queried from an SQL database, and you can interactively
to the website's database. In addition, even the
static content is generated by dynamic
rendering of content written in lml markup.
The source code,
written in lisp, and consists of library calls and function definitions.
separation of presentation, content, and logic
seems to be the holy grail of web
smarty are some attempts to
It turns out that it is just remarkably hard to avoid mixing the markup,
presentation and control code in site files.
Since you can't avoid mixing without a lot of trouble, why not
consider finding a way to mix the content and code with a single style?
I'm delighted with this web site's application files
which are well structured and consistently highlighted in a
good text editor like emacs.
mod_lisp+ clsql+ lml... provides that
because the only syntax is the syntax of common lisp. Here's what
(html (head (title "hello-world")) (body (h1 "hello-world") (p "hello-world greetings in several languages") (if (newyear) (gen-table-newyear) (gen-table-hello))))
There's basically one syntax there for html, some control, and functions
which generate tables.
other website language whose code comes close to this in terms of
readability is brl, which is built in
Lisp source files, like webdemo.cl , can be
rendered in html with the wonderful emacs package,
Of course, that's the format which you see when you follow any of the
webdemo.cl links on this site.
For the following crude tests, the apache server and lisp image were
running on an AMD-K6(tm) 3D processor, 350 MHz , with a 64KB cache, and
I'll just let you judge for yourself about the performance of the
server. Here are the results of the apache
program calling a page written
in lml which also has a read query of the database:
|Document Length:||15705 bytes|
|Time taken for tests:||168.458 seconds|
|Total transferred:||15881000 bytes|
|HTML transferred:||15705000 bytes|
|Requests per second:||5.94|
|Transfer rate:||94.27 kb/s received|
|Connnection Times (ms)|
|Document Length:||60852 bytes|
|Time taken for tests:||27.787 seconds|
|Total transferred:||61028000 bytes|
|HTML transferred:||60852000 bytes|
|Requests per second:||35.99|
|Transfer rate:||2196.28 kb/s received|
|Connnection Times (ms)|
Keep in mind that the very popular apache webserver version 1.3.x is
not designed with native
but rather relies on child processes which are preforked or forked on demand.
CMUCL has only user threads
which I do not use in this application.
Dan Barlow and the team designing
are implementing native threads in SBCL. In addition, a commercial lisp
derived from CMUCL,
Scieneer Common Lisp (SCL),
Native threads capability will improve the performance of this apache ab
benchmark. The database queries through clsql do not use pooled
connections but at least reuse open connections to mysql.
CMU Common Lisp works well with apache and mod_lisp. The
environment is extremely useful for quick prototyping, troubleshooting,
and maintenance of
a simple database backed web site. The web site is written in a single
syntax, lisp syntax, and can be efficiently edited in emacs. The toolset
provides a straightforward migration path from a php + apache, or a
mod_perl + apache, or a
java + mod_jk + apache database backed website.
On the other hand, I am not taking
advantage of pooled or persistent database connections between the lisp process
and mysql, which would improve performance. Somehow the inclusion
of document type definition statements causes rendering by css generated
by a lisp function to fail. It does work if the css is called from an external
static css page on the server. The
SQL query statements are strings rather than lisp syntax. Although I claim
the syntax of my source files is all lisp, I
most common lisp style rules
Thanks to Marc Battyani, author of mod_lisp. Thanks to Kevin Rosenberg, author of
clsql, lml, and uffi. Thanks to Edi Weitz, author of cl-ppcre.
Thanks to Dan Barlow, author of the
CMUCL on Linux Howto and
detachtty and a visionary
blazing the trail to make
common lisp play nicely with unix.
Thanks to Shiro Kawai whose articles
Shooting a Moving Target and
Tracking Assets opened my eyes to using common lisp
to get real work done on Unix boxes. These articles are also
the finest project post-mortem documents I have ever read.
Here is where you can
comments on this website. New comments appear after I review them.
Comments are welcome
lispnyc can use this
cool. i'm a web no-nothing, but those
in lispnyc that know more are interested
in having our web site be lisp-based.
This could be a big help.
Kenny Tilton email@example.com
Very nice. All ingredients are here and their relation are explained.
Although it is a little frightning to see how much you need:
apache + mod_lisp + cmucl + lml + cl-ppcre + clsql + mysql
Jens Axel S�gaard firstname.lastname@example.org
Throw in UncommonSQL
If you throw in UncommonSQL
you can use Lispy syntax to do
the SQL, e.g.,
(car (select 'blob-table
Ng Pheng Siong email@example.com
Easy to read, easy to learn and complex enough at the same time.
Vladimir Sekissov firstname.lastname@example.org
just what the doctor ordered... beware now php!!
quasi quasiabhi at yahoo.com
Yes, very cool and useful, I have put links to this page on the mod_lisp pages.
Marc Battyani email@example.com
A good read - but is the .css GPLed too?
Yay! The clear description and understandable source code make this is a very useful place to point frustrated php coders to.
But one question remains: Is the .css file you used for this web site also covered by the GPL? (-;
Andreas Fuchs firstname.lastname@example.org
This is very cool. Thanks for the contribution.
Nice Site! :-)
The GPL'd database for Loads of Linux Links now includes this site.
http://loll.sourceforge.net/linux/links/index.htmlfor the details.
Barbara I. Irwin email@example.com
Thanks for the comments. I'll offer some followups:
- Jens - Maybe some of the libraries can be bundled (clsql, uffi, lml) to make installation less imposing
- Ng - I agree, lispy SQL queries are superior to CLSQL's quoted string queries. CLSQL is GPL, so the rest is up to us!
- Andreas - I inserted the GPL into the .css file after your inquiry.
Chris Beggy firstname.lastname@example.org
Thanks to some webloggers who found out about this site and reviewed it:
- Chris Double's Radio Weblog
- Ted Leung's Blog
- Kenneth Hunt's weblog
creating Dynamic Websites
This is a very useful 'getting started' document. A few comments:
about your use of asdf:
- it probably runs faster if you load the compiled version
- rather than editing asdf.lisp directly to add ~/systems to it,
consider loading it from .cmucl-init.lisp and then doing
(pushnew "/home/yourdir/systems/" asdf:*central-registry*)
- instead of loading each of those systems by hand, write a system
of your own that depends on them all. Call this webdemo.asd and
put it with the other files
(defpackage :webdemo-system (:use "CL" "ASDF")) (in-package :webdemo-system) (defsystem webdemo :components (("webdemo" :pathname "webdemo.cl")) :depends-on (uffi clsql clsql-base clsql-uffi clsql-mysql lml base64 cl-ppcre))
(untested, may need frobbing)
- unsolicited SBCL plug: as of SBCL 0.8.3 you'll be able to download
and install many of these libararies automatically with a single
Dan Barlow email@example.com