		 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
		 %	   WEB PROTOCOLS	%
		 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

% Towards ECLiPSe Agents on the INTERNET
% http://clip.dia.fi.upm.es/lpnet/lp-internet/eclipse/ea.html L. Leth
% Ph. Bonnet, S. Bressan and B. Thomsen. Towards ECLiPSe Agents on the
% INTERNET. In Proceedings of the 1st Workshop on Logic Programming
% Tools for INTERNET Applications, JICSLP''96, Bonn, September 1996.
% Available from http://clement.info.umoncton.ca/~lpnet/lpnet2.html.

\section{Supporting HTTP}
\label{sec:http}

HTTP, or HyperText Transfer Protocol, is the key W3C standard protocol
for exchanging Web documents. All browsers and Web servers implement it.
The initial version of the protocol was very simple. The client request
consists of a single line of the format \mbox{\bnfmeta{action} \bnfmeta{path}}, the
server replies with the requested document and closes the connection.
Version 1.1 of the protocol is more complicated, providing additional
name-value pairs in the request as well as the reply, features to
request status such as modification time, transfer partial documents,
etcetera.

Adding HTTP support in Prolog, we must consider both the client- and
server-side. In both cases our choice is between doing it in Prolog or
re-using an existing application or library by providing an interface for
it. We compare our work with PiLLoW \cite{journals/corr/cs-DC-0312031} and
the ECLiPSe HTTP services \cite{Eclipse-http}.

Given a basic TCP/IP socket library, writing an HTTP client is trivial
(our client is 258 lines of code). Both PiLLoW and ECLiPSe include
a client written in Prolog.   More issues complicate the choice for a
pure Prolog based server.  

\begin{itemize}
    \item
The server is more complex, which implies there is more to gain by
re-using external code.  Our core server library counts 1,784 lines.

    \item
A single computer can only host one server at port 80 used by default
for public HTTP. Using an alternate port for middleware and storage tier
components is no problem, but use as a public server often conflicts
with firewall or proxy settings. This can be solved using a proxy
server such as the Apache
\emph{mod_proxy}\fnurl{http://httpd.apache.org/docs/1.3/mod/mod_proxy.html}
configured as \jargon{reverse proxy}.

    \item
Servers by definition introduce security risks.  Administrators are
reluctant to see non-proven software in the role of a public server.
Using a proxy as above also reduces this risk, especially if the 
proxy blocks malformed requests.
\end{itemize}

Dispite these observations, we consider, like the ECLiPSe team, a pure
Prolog based server worthwhile. As argued in \secref{mt}, many Prolog
Web applications profit from using state stored in the server. Large
resources such as WordNet \cite{Miller:95a} cause long startup times. In
such cases the use of CGI (Common Gateway Interface) is not appropriate
as a new copy of the application is started for each request. PiLLoW
resolves this issue using \jargon{Active Modules}, where a small CGI
application talks to a continuously running Prolog server using a
private protocol. Using a Prolog HTTP server and optionally a reverse
proxy has the same benefits, but based on a standard protocol, it is
much more flexible.

Another approach is embedding Prolog in another server framework such as
the Java based Tomcat server. Although feasible, embedding non-Java
based Prolog systems in Java is complicated. Embedding through
\emph{jni} introduces platform and Java version dependent problems.
Connecting Prolog and Java concurrency models and garbage collection is
difficult and the resulting system is much harder to manage by the user
than a pure Prolog based application.

\begin{figure}
    \centerline{\includegraphics[width=0.7\linewidth]{http}}
    \caption{Module dependencies of the HTTP library}
    \label{fig:http}
\end{figure}

In the following sections we describe our HTTP client and server
libraries.  An overall overview of the modules and their dependencies is
given in \figref{http}.


\subsection{HTTP client libraries}
\label{sec:httpclient}

We support two clients. The first is a lightweight client that only
supports the HTTP GET method by means of \term{http_open}{+URL, -Stream,
+Options}. \arg{Options} allows for setting a timeout or proxy as well
as getting information from the reply-header such as the size of the
document. The \index{http_open/3}\predref{http_open}{3} predicate internally handles HTTP 3xx
(redirect) replies. Other non-ok replies are mapped to a Prolog
exception. After reading the document the user must close the returned
stream-handle using the standard Prolog \index{close/1}\predref{close}{1} predicate. This
predicate makes accessing an HTTP resource as simple as accessing a
local file. The second library, called \file{http_client.pl}, provides
support for HTTP POST and a plugin interface that allows for installing
handlers for documents of specified MIME-types. It shares
\file{http_header.pl} with the server libraries for DCG based creation
and parsing of HTTP headers. Currently provided plugins include
\file{http_mime_plugin.pl} to handle multipart MIME messages and
\file{http_sgml_plugin.pl} for automatically parsing HTML, XML and SGML
documents. \Figref{httpclient} shows the code for fetching a URL and
parsing the returned HTML document it into a Prolog term as described in
\secref{markupdom}.

\begin{figure}
\figrule
\begin{verbatim}
?- use_module(library('http/http_client')).
?- use_module(library('http/http_sgml_plugin')).

?- http_get('http://www.swi-prolog.org/', DOM, []).
DOM = [element(html, [version='-//W3C//DTD HTML 4.0 Transitional//EN'],
                     [element(head, [],
                              [element(title, [],
                                       ['SWI-Prolog\'s Home']), ...
\end{verbatim}

\noindent
    \caption{Fetching an HTML document}
    \label{fig:httpclient}
\figrule
\end{figure}

Both the PiLLoW and ECLiPSe approach return the documents content as a
string. Our interface is stream-based (\index{http_open/3}\predref{http_open}{3}) or allows for
plugin-based processing of the stream (\index{http_get/3}\predref{http_get}{3}, \index{http_post/4}\predref{http_post}{4}). This
interface avoids potentially large intermediate data-structures and
allows for processing unbounded documents.

\subsection{The HTTP server library}
\label{sec:httpd}

Both to simplify re-use of application code and to make it possible to
use the server without committing to a large infrastructure we adopted
the reply-strategy of the CGI protocol, where the handler writes a page
consisting of an HTTP header followed by the document content.
\Figref{httpreply} provides a simple example that returns the
request-data to the client. By importing \file{thread_http.pl} we
implicitly selected the multi-threaded server model. Other models
provided are \file{inetd_http}, causing the (Unix) inet daemon to start
a server for each request and \file{xpce_http} which uses I/O
multiplexing realising multiple clients without using Prolog threads.
The logic of handling a single HTTP request given a predicate realising
the handler, an input and output stream is implemented by
\file{http_wrapper}.

\begin{figure}
\hfill
\begin{minipage}[c]{.45\textwidth}
\footnotesize
\begin{verbatim}
:- use_module(
   library('http/thread_httpd')).

start_server(Port) :-
  http_server(reply, [port(Port)]).

reply(Request) :-
  format('Content-type: text/plain~n~n'),
  writeln(Request).
\end{verbatim}

\noindent
\end{minipage}
\hfill
\begin{minipage}[c]{.50\textwidth}
\includegraphics[width=\linewidth]{mozilla}
\end{minipage}
\hfill

    \caption{A simple HTTP server.  The right window shows the client
	     and the format of the parsed request.}
    \label{fig:httpreply}
\end{figure}

Replies other than ``200 OK'' are generated using a Prolog exception.
Recognised replies are defined by the predicate
\term{http_reply}{+Reply, +Stream, +Header}. For example to indicate that the
user has no access to a page we must use the following call. 

\begin{verbatim}
        throw(http_reply(forbidden(URL))).
\end{verbatim}

\noindent
Failure of the handler raises a ``404 existence error'' reply, while
exceptions other than the ones described above raise a ``500 Server
error'' reply.

\subsubsection{Form parameters}
\label{sec:httpparam}

The library \file{http_parameters.pl} defines
\term{http_parameters}{+Request, ?Parameters} to fetch and type-check
parameters transparently for both GET and POST requests.
\Figref{httpparms} illustrates the functionality. Parameter values are
returned as atoms. If large documents are transferred using a POST
request the user may wish to revert to \term{http_read_data}{+Request,
-Data, +Options} underlying \index{http_get/3}\predref{http_get}{3} to process arguments using
plugins.

\begin{figure}
\figrule
\begin{verbatim}
reply(Request) :-
        http_parameters(Request,
                        [ title(Title, [optional(true)]),
                          name(Name,   [length >= 2]),
                          age(Age,     [integer])
                        ]), ...
\end{verbatim}

\noindent
    \caption{Fetching HTTP form data}
    \label{fig:httpparms}
\figrule
\end{figure}



\subsubsection{Session management}
\label{sec:httpsession}

The library \file{http_session.pl} provides session over the stateless
HTTP protocol. It does so by adding a cookie using a randomly generated
code if no valid session id is found in the current request. The
interface to the user consists of a predicate to set options (timeout,
cookie-name and path) and a set of wrappers around \index{assert/1}\predref{assert}{1} and
\index{retract/1}\predref{retract}{1}, the most important of which are
\term{http_session_assert}{+Data}, \term{http_session_retract}{?Data}
and \term{http_session_data}{?Data}.  In the current version the data
associated with sessions that have timed out is simply discarded.  
Session-data does not survive the server.

Note that a session generally consists of a number of HTTP requests and
replies. Each \emph{request} is scheduled over the available worker
threads and requests belonging to the same session are therefore
normally not handled by the same thread. This implies no session state
can be stored in global variables or in the control-structure of a
thread. If such style of programming is wanted the user must create a
thread that represents the session and setup communication from the
HTTP-worker thread to the session thread. \Figref{httpsession}
illustrates the idea.

\begin{figure}
\figrule
\begin{verbatim}
reply(Request) :-                               % HTTP worker
        (   http_session_data(thread(Thread))
        ->  true
        ;   thread_create(session_loop([]), Thread, [detached(true)]),
            http_session_assert(thread(Thread))
        ),
        current_output(CGIOut),
        thread_self(Me),
        thread_send_message(Thread, handle(Request, Me, CGIOut)),
        thread_get_message(_Done).

session_loop(State) :-                          % Session thread
        thread_get_message(handle(Request, Sender, CGIOut)),
        next_state(Request, State, NewState, CGIOut).
        thread_send_message(Sender, done).
\end{verbatim}

\noindent
    \caption{Managing a session in a thread. The \index{reply/1}\predref{reply}{1} predicate is
	     part of the HTTP worker pool, while \index{session_loop/1}\predref{session_loop}{1} is
	     executed in the thread handling the session.  We omitted
	     error handling for readability of the example.}
    \label{fig:httpsession} \figrule
\end{figure}


\subsubsection{Evaluation}
\label{sec:httpevaluation}

The presented server infrastructure is currently used by many internal
and external projects. Coding a server is very similar to writing CGI
handlers and running in the interactive Prolog process it is much easier
to debug. As Prolog is capable of reloading source files in the running
system, handlers can be updated while the server is running. Handlers
running during the update are likely to die on an exception though. We
plan to resolve this issue by introducing read/write locks. The protocol
overhead of the multi-threaded server is illustrated in
\tabref{httpperf}.

\begin{table}
    \caption{HTTP performance executing a trivial query 10,000 times.
	     Times are in seconds. Localhost, dual AMD 1600+ running
	     SuSE Linux 10.0}
    \label{tab:httpperf}
\begin{tabular}{lrrr}
\hline\hline
\bf Connection & \bf Elapsed & \bf Server CPU & \bf Client CPU \\
\hline
Close 	       & 20.84	    & 11.70	     & 7.48 \\
Keep-Alive     & 16.23      & 8.69           & 6.73 \\
\hline\hline
\end{tabular}
\end{table}
