		 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
		 %	   WEB DOCUMENTS	%
		 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Parsing and representing XML and HTML documents}
\label{sec:markupdom}

% element(Name, Attrs, Content) (Acknowledge Anjo)
% Parser (Acknowledge Richard O'Keefe)
% Also in use by XSB (more?)

The core of the Web is formed by document standards and exchange
protocols. Here we describe tree-structured documents transferred as
SGML or XML. HTML, an SGML application, is the most commonly used
document format on the Web. HTML represents documents as a tree using a
fixed set of \jargon{elements} (tags), where the SGML DTD (Document Type
Declaration) puts constraints on how elements can be nested. Each node
in the hierarchy has a name (the element-name), a set of name-value
pairs known as its attributes and \jargon{content}, a sequence of
sub-elements and text (data).

XML is a rationalisation of SGML using the same tree-model, but removing
many rarely used features as well as abbreviations that were introduced
in SGML to make the markup easier to type and read by humans. XML
documents are used to represent text using custom application-oriented
tags as well as a serialization format for arbitrary data exchange
between computers. XHTML is HTML based on XML rather than SGML.

The first SGML parser for SWI-Prolog was created by Anjo Anjewierden
based on the SP parser\footnote{\url{http://www.jclark.com/sp/}}. A
stable Prolog term-representation for SGML/XML trees plays a similar
role as the DOM (\jargon{Document Object Model}) representation in
use in the object-oriented world. The term-structure we use is described
in \figref{dom}. Some issues have been subject to debate.

\begin{figure}
\figrule
\begin{tabular}{lrl}
\bnfmeta{document}	\isa list-of \bnfmeta{content} \\
\bnfmeta{content}	\isa \bnfmeta{element} $\mid$ \bnfmeta{pi} $\mid$ \bnfmeta{cdata} $\mid$ \bnfmeta{sdata} $\mid$ \bnfmeta{ndata} \\
\bnfmeta{element}	\isa element(\bnfmeta{tag}, list-of \bnfmeta{attribute}, list-of \bnfmeta{content}) \\
\bnfmeta{attribute}	\isa \bnfmeta{name} = \bnfmeta{value} \\
\bnfmeta{pi}		\isa pi(\bnfmeta{atom})  \\
\bnfmeta{sdata}		\isa sdata(\bnfmeta{atom})  \\
\bnfmeta{ndata}		\isa ndata(\bnfmeta{atom}) \\
\bnfmeta{cdata}, \bnfmeta{name}	\isa \bnfmeta{atom} \\
\bnfmeta{value}		\isa \bnfmeta{svalue} $\mid$ list-of \bnfmeta{svalue} \\
\bnfmeta{svalue}	\isa \bnfmeta{atom} $\mid$ \bnfmeta{number}
\end{tabular}
    \caption{SGML/XML tree representation in Prolog.   The notation
	     list-of \bnfmeta{x} describes a Prolog list of terms of type \bnfmeta{x}.}
    \label{fig:dom}
\figrule
\end{figure}

\begin{itemize}
    \item Representation of text by a Prolog atom is biased by the use of
          SWI-Prolog which has no length-limit on atoms and atoms that
	  can represent Unicode text as motivated in
	  \secref{unicode}. At the same time SWI-Prolog stacks are
	  limited to 128MB each. Using atoms only the structure of the
	  tree is represented on the stack, while the bulk of the data
	  is stored on the unlimited heap. Using lists of character
	  codes is another possibility adopted by both PiLLoW
	  \cite{PiLLoW} and ECLiPSe \cite{Eclipse-http}. Two
	  observations make lists less attractive: lists use two cells
	  per character while practical experience shows text is
	  frequently processed as a unit only. For (HTML) text-documents
	  we profit from the compact representation of atoms. For XML
	  documents representing serialized data-structures we profit
	  from frequent repetition of the same value.

    \item Attribute values of multi-value attributes (e.g.\ 
 	  \tag{NAMES}) are returned as a Prolog list.   This implies
	  the DTD must be available to get unambiguous results.  With
	  SGML this is always true, but not with XML.

    \item Optionally attribute values of type \tag{NUMBER} or \tag{NUMBERS}
	  are mapped to Prolog numbers.  In addition to the DTD issues
	  mentioned above, this conversion also suffers from possible
	  loss of information. Leading zeros and different floating
	  point number notations used are lost after conversion. Prolog
	  systems with bounded arithmetic may also not be able to
	  represent all values. Still, automatic conversion is useful in
	  many applications, especially involving serialized
	  data-structures.

    \item Attribute values are represented as \arg{Name}=\arg{Value}.  Using
    	  \mbox{\arg{Name}({Value})} is an alternative. The
	  \arg{Name}=\arg{Value} representation was chosen for its
	  similarity to the SGML notation and because it avoids the
	  need for univ (\verb|=..|) for processing argument-lists.
\end{itemize}

\paragraph{Implementation}

The SWI-Prolog SGML/XML parser is implemented as a C-library that has
been built from scratch to reach at a lightweight parser. Total source
is 11,835 lines. The parser provides two interfaces. Most natural to
Prolog is \term{load_structure}{+Src, -DOM, +Options} which parses a
Prolog stream into a term as described above. Alternatively,
\index{sgml_parse/2}\predref{sgml_parse}{2} provides an \jargon{event-based} parser making
call-backs on Prolog for the SGML events. The call-back mode can deal
with unbounded documents in streaming mode. It can be mixed with the
term-creation mode, where the handler for \emph{begin} calls the parser
to create a term-representation for the content of the element. This
feature is used to process long files with a repetitive record structure
in limited memory. \Secref{rdfio} describes how this is used to process
RDF documents.

Full documentation is available from
\url{http://www.swi-prolog.org/packages/sgml2pl.html}
The SWI-Prolog SGML parser has been adopted by XSB Prolog.


\section{Generating Web documents}
\label{sec:genml}

There are many approaches to generating Web pages from programs in
general and Prolog in particular. We believe the preferred choice
depends on various aspects.

\begin{itemize}
    \item
How much of the document is generated from dynamic data and how much is
static?  Pages that are static except for a few strings are best
generated from a template using variable substitution. Pages
consisting of a table generated from dynamic data are best entirely
generated from the program.

    \item
For program generated pages we can choose between direct printing and
generating using a language-native syntax, for example
\verb|format('<b|\verb|>bold</b>')| or \verb|print_html(b(bold))|. The
second approach can guarantee well-formed output, but the first requires
the programmer to learn about \index{format/3}\predref{format}{3} only.

    \item
Documents that contain a significant static part are best represented in
the markup language where special constructs insert program-generated
parts. A popular approach implemented by PHP\fnurl{www.php.net} and
ASP\fnurl{www.microsoft.com} is to add a reserved element such as
\bnfmeta{script} and use the SGML/XML \jargon{programming instruction} written
as \verb"<?...?>". The obvious name PSP (Prolog Server Pages) is in use
by various projects taking this approach.%
 \footnote{\url{http://www.prologonlinereference.org/psp.psp},\\
	   \url{http://www.benjaminjohnston.com.au/template.prolog?t=psp},\\
	   \url{http://www.ifcomputer.com/inap/inap2001/program/inap_bartenstein.ps}}
Another approach is
PWP\fnurl{http://www.cs.otago.ac.nz/staffpriv/ok/pwp.pl} (Prolog
Well-formed Pages).  It is based on the principle that the source
is well-formed XML and interacts with Prolog through additional
attributes.  Output is guaranteed to be well-formed XML.
Our infrastructure does not yet include any of these approaches.

    \item
Page \jargon{transformation} is realised by parsing the original
document into its tree representation, managing the tree and writing a
new document from the tree. Managing the source-text directly is not
reliable as due to character encoding choice, entity usage and SGML
abbreviations there are many different source-texts that represent the
same tree. The \index{load_structure/3}\predref{load_structure}{3} predicate described in
\secref{markupdom} together with output primitives from the library
\file{sgml_write.pl} provide this functionality.  The XDIG case study
described in \secref{xdig} follows this approach.
\end{itemize}


\subsection{Generating documents using DCG}
\label{sec:htmlwrite}

The traditional method for creating Web documents is using print
routines such as \index{write/1}\predref{write}{1} or \index{format/2}\predref{format}{2}. Although simple and easily
explained to novices, the approach has serious drawbacks from a software
engineering point of view. In particular the user is responsible for
HTML quoting, character encoding issues and proper nesting of HTML
elements. Automated validation is virtually impossible using this
approach.

Alternatively we can produce a DOM term as described in
\secref{markupdom} and use the library \file{sgml_write.pl} to create
the HTML or XML document. Such documents are guaranteed to use proper
nesting of elements, escape sequences and character encoding. The terms
however are big, deeply nested and hard to read and write. Prolog allows
them to be built from skeletons containing variables. This approach is
taken by PiLLoW (\secref{pillow}) to control the complexity. In our
opinion, the result is not optimal due to the unnatural order of
statements as illustrated in \figref{pillowwrite}. PiLLoW has partly
overcome this shortcoming by defining a large number of `utility terms'
that are translated in a special way, as discussed in section 6.2 of
\cite{PiLLoW}.

\begin{figure}
\figrule
\begin{verbatim}
        ...,
        mkthumbnail(URL, Caption, ThumbNail),
        output_html([ h1("Photo gallery"),
                      ThumbNail
                    ]).

mkthumbnail(URL, Caption, Term) :-
        Term = table([ tr(td([halign=center], img([src=URL],[]))),
                       tr(td([halign=center], Caption))
                     ])
\end{verbatim}

\noindent
    \caption{Building PiLLoW terms}
    \label{fig:pillowwrite}
\figrule
\end{figure}

We introduced a DCG rule html//1.%
    \footnote{The notation \bnfmeta{name}//\bnfmeta{arity} refers to the grammar rule
	      \bnfmeta{name} with the given \bnfmeta{arity}, and consequently the
	      predicate \bnfmeta{name} with arity \bnfmeta{arity}$+2$.}
This rule translates proper trees into a list of high-level HTML/XML
commands that are handed to \index{html_print/1}\predref{html_print}{1} to realise proper quoting,
character encoding and layout. The intermediate format is of no concern
to the user and similar in structure to the PiLLoW representation
without using environments. Generated from the tree representation
however, consistent opening and closing of elements is guaranteed. In
addition to variable substitution which is provided by Prolog we allow
calling rules. Rules are invoked by a term \verb|\|\arg{Rule} embedded
in the argument of html//1. \Figref{affiliation} illustrates our
approach. Note that any reusable part of the page generation can easily
be translated into a DCG rule and the difference between direct
translation of terms to HTML and rule-invocation is eminent.

\begin{figure}
\figrule
\begin{verbatim}
affiliation_table :-
        findall(Name-Aff, affiliation(Name, Aff), Pairs0),
        keysort(Pairs0, Pairs),
        reply_page(table([border(2),align(center)],
                         [ tr([th('Name'), th('Affiliation')])
                         | \affiliations(Pairs)
                         ])).

affiliations([]) --> [].
affiliations([H|T]) --> affiliation(H), affiliations(T).

affiliation(Name-Aff) --> html(tr(td(Name), td(Aff))).

% database
affiliation(wielemaker, uva).
affiliation(huang, vu).
affiliation('van der meij', vu).

% Page template
reply_page(Term) :-
        format('Content-type: text/html~n~n'),
        phrase(html(Term), Tokens),
        print_html(Tokens).
\end{verbatim}

\noindent
    \caption{Library html_write.pl in action}
    \label{fig:affiliation}
\figrule
\end{figure}

In our current implementation rules are called using meta-calling from
html//1. Using \index{term_expansion/2}\predref{term_expansion}{2} it is straightforward to move the rule
invocation out of the term, using variable substitution similar to
PiLLoW. It is also possible to recursively expand the generated tree
and validate it to the HTML DTD at compile-time and even insert omitted
tags at compile-time to generate valid XHMTL from an incomplete
specification.  An overview of the argument to html//1 is
given in \figref{html1}.

\begin{figure}
\figrule
\begin{tabular}{lrl}
\bnfmeta{html}		\isa list-of \bnfmeta{content} $\mid$ \bnfmeta{content} \\
\bnfmeta{content}	\isa \bnfmeta{atom} \\
		\ora \bnfmeta{tag}(list-of \bnfmeta{attribute}, \bnfmeta{html}) \\
		\ora \bnfmeta{tag}(\bnfmeta{html}) \\
		\ora \verb|\|\bnfmeta{rule} \\
\bnfmeta{attribute}	\isa \bnfmeta{name}(\bnfmeta{value}) \\
\bnfmeta{tag}, \bnfmeta{entity}	\isa \bnfmeta{atom} \\
\bnfmeta{value}		\isa \bnfmeta{atom} $\mid$ \bnfmeta{number} \\
\bnfmeta{rule}		\isa \bnfmeta{callable}
\end{tabular}
    \caption{The html//1 argument specification}
    \label{fig:html1}
\figrule
\end{figure}


\subsection{Comparison with PiLLoW}
\label{sec:pillow}

The PiLLoW library \cite{PiLLoW} is a well established framework
for Web programming based on Prolog. PiLLoW defines \index{html2terms/2}\predref{html2terms}{2},
converting between an HTML string and a document represented as a
Herbrand term. There are fundamental differences between PiLLoW and the
primitives described here.

\begin{itemize}
    \item
PiLLoW creates an HTML document from a Herbrand term that is passed
to \index{html2terms/2}\predref{html2terms}{2}. Complex terms are composed of partial terms passed as
Prolog variables. Frequent HTML constructs are supported using reserved
terms using dedicated processing. We use DCGs and the \verb|\|\arg{Rule}
construct, which makes it eminent which terms directly refer to HTML
elements and which function as a `macro'. In addition, the user can
define application-specific reusable fragments in a uniform way.

    \item
The PiLLoW parser does not create the SGML document tree. It does not
insert omitted tags, default attributes, etcetera. As a result, HTML
documents that differ only in omitted tags and whether or not default
attributes are included in the source produce different terms. In our
approach the term representation is equivalent, regardless of the input
document. This is illustrated in \figref{tabledom}. Having a canonical
DOM representation greatly simplifies processing parsed HTML documents.
\end{itemize}

\begin{figure}
\figrule
\begin{verbatim}
[env(table, [], [tr$[], td$[], "Hello"])]
\end{verbatim}

\noindent
\begin{verbatim}
[element(table, [],
         [ element(tbody, [],
                   [ element(tr, [],
                            [ element(td, [rowspan='1', colspan='1'],
                                      ['Hello'])])])])]
\end{verbatim}

\noindent
    \caption{Term representations for
	     \texttt{\string<table\mbox{}><tr\mbox{}><td\mbox{}>Hello</table>} in
	     PiLLoW (top) and our parser (bottom).  Our parser completes
	     the \tag{tr} and \tag{td} environments, inserts the omitted
	     \tag{tbody} element and inserts the defaults for the
	     \tag{rowspan} and \tag{colspan} attributes}
    \label{fig:tabledom}
\figrule
\end{figure}


\section{RDF documents}
\label{sec:rdfdoc}

Where the datamodel of both HTML and XML is a tree-structure with
attributes, the datamodel of the Semantic Web (SW)
RDF\fnurl{http://www.w3.org/RDF/} language consists of
\mbox{\{\arg{Subject}, \arg{Predicate}, \arg{Object}\}}
\jargon{triples}. Both \arg{Subject} and \arg{Predicate} are a
\jargon{URI}.% 
    \footnote{URI: \jargon{Uniform Resource Identifier} is
	      like a URL, but need not refer to an existing
	      resource on the Web.}
\arg{Object} is either a URI or a \jargon{Literal}. As the \arg{Object}
of one triple can be the \arg{Subject} of another, a set of triples
forms a graph, where each edge is labelled with a URI (the
\arg{Predicate}) and each vertex is either a URI or a literal. Literals
have no out-going edges. \Figref{triple} illustrates this.

\begin{figure}
    \centerline{\includegraphics[width=0.7\linewidth]{triple}}
    \caption{Sample RDF graph.  Ellipses are vertices representing
	     URIs.  Quoted text is a literal.  Edges are labelled with
	     URIs.}
    \label{fig:triple}
\end{figure}

A number of languages are layered on top of the RDF triple model. RDFS
provides a frame-based representation. The
OWL-dialects\fnurl{http://www.w3.org/2004/OWL/} provide three
increasingly complex Description Logic (DL) languages.
SWRL\fnurl{http://www.w3.org/Submission/SWRL/} is a proposal for a rule
language. The W3C standard for exchanging these triple models is an XML
application known as RDF/XML.

As there are multiple XML tree representations for the same triple-set,
RDF documents cannot be processed at the level of the XML-DOM as
described in \secref{markupdom}. A triple- or graph-based structure is
the most natural choice for representating an RDF document in Prolog.
First we must decide on the representation of URIs and literals. As a
URI is a string and the only operation defined on URIs by SW languages
is equivalence test, using a Prolog atom is an obvious choice. One may
consider using a term \mbox{\bnfmeta{namespace}:\bnfmeta{localname}}, but given that
decomposing a URI into its namespace and localname is only relevant
during I/O we consider this an inferior choice. The RDF library comes
with a compile-time rewrite mechanism based on \index{goal_expansion/2}\predref{goal_expansion}{2} that
allows for writing resources in Prolog sourcetext as
\mbox{\bnfmeta{ns}:\bnfmeta{local}}. Literals are expressed as \term{literal}{Value}.
The full type description is in \figref{rdftypes}.

\begin{figure}
\figrule
\begin{tabular}{lrl}
\bnfmeta{subject}, \bnfmeta{predicate}	\isa \bnfmeta{URI} \\
\bnfmeta{object}	\isa \bnfmeta{URI} \\
		\ora literal(\bnfmeta{lit_value}) \\
\bnfmeta{lit_value}	\isa \bnfmeta{text} \\
		\ora lang(\bnfmeta{langid}, \bnfmeta{text}) \\
		\ora type(\bnfmeta{URI}, \bnfmeta{text}) \\
\bnfmeta{URI}, \bnfmeta{text}	\isa \bnfmeta{atom} \\
\bnfmeta{langid}	\isa \bnfmeta{atom} (ISO639)
\end{tabular}
    \caption{RDF types in Prolog.}
    \label{fig:rdftypes}
\figrule
\end{figure}

The typical SW use-scenario is to `harvest' triples from multiple
sources and collect them in a database before reasoning with them.
Prolog can represent data as a Herbrand term on the stack or as
predicates in the database. Given the relatively static nature of the
RDF data as well desired access from multiple threads, using the Prolog
database is the most obvious choice. Here we have two options. One is
the predicate \term{rdf}{Subject, Predicate, Object} using the argument
types described above. The alternative is to map each RDF predicate on a
Prolog predicate \mbox{\arg{Predicate}({\arg{Subject}, \arg{Object}})}.
We have chosen for \index{rdf/3}\predref{rdf}{3} because it supports queries with uninstantiated
predicates better and a single predicate is easier to manage than an
unbounded set of predicates with unknown names.


\subsection{Input and output of RDF documents}
\label{sec:rdfio}

The RDF/XML parser is realised as a Prolog library on top of the XML
parser described in \secref{markupdom}. Similar to the XML parser it has
two interfaces. The predicate \term{load_rdf}{+Src, -Triples, +Options}
parses a document and returns a Prolog list of \term{rdf}{S,P,O}
triples. Note that despite harvesting to the database is the typical
use-case scenario, the parser delivers a list of triples for maximal
flexibility. The predicate \term{process_rdf}{+Src, :Action, +Options}
exploits the mixed call-back/convert mode of the XML parser to process
the RDF file one \jargon{description} (record) at a time, calling
\arg{Action} with a list of triples extracted from the description.
\Figref{rdfload} illustrates how this is used by the storage module to
load unbounded files with limited stack usage. Source location as
\bnfmeta{file}:\bnfmeta{line} is passed to the \arg{Src} argument of \index{assert_triples/2}\predref{assert_triples}{2}.

\begin{figure}
\figrule
\begin{verbatim}
load_triples(File, Options) :-
        process_rdf(File, assert_triples, Options).

assert_triples([], _).
assert_triples([rdf(S,P,O)|T], Src) :-
        rdf_assert(S, P, O, Src),
        assert_triples(T, Src).
\end{verbatim}

\noindent
    \caption{Loading triples using \index{process_rdf/3}\predref{process_rdf}{3}}
    \label{fig:rdfload}
\figrule
\end{figure}

In addition to named URIs, RDF resources can be \jargon{blank-nodes}. A
blank-node (short \jargon{bnode}) is an anonymous resource that is
created from an in-lined description. \Figref{bnode} describes the
dimensions of a painting as a compound instance of class \arg{Dimension}
with width and height properties. The \arg{Dimension} instance has no
URI. Our parser generates an identifier that starts with a double
underscore, followed by the source and a number. The double underscore
is used to identify bnodes. Source and number are needed to guarantee
the bnode is unique.

\begin{figure}
\figrule
\begin{verbatim}
<Painting rdf:about="...">
  <dimension>
     <Dimension width="45" height="50"/>
  </dimension>
</Painting>

\end{verbatim}

\noindent
    \caption{Blank node to express the compound dimension property}
    \label{fig:bnode}
\figrule
\end{figure}

The parser from XML and RDF triples covers the full RDF specification,
including Unicode handling, RDF datatypes and RDF language tags. The
Prolog source is 1,788 lines. It processes approximately 9,000 triples
per second on an AMD 1600+ based computer. Implementation details and
evaluation of the parser are described in \cite{Wielemaker:03a}.%
	\footnote{The parser described there did not yet support RDF
		  datatypes and language tags, nor Unicode.}

We have two libraries for writing RDF/XML. One,
\term{rdf_write_xml}{+Stream, +Triples}, provides the inverse of
\index{load_rdf/2}\predref{load_rdf}{2}, writing an XML document from a list of \term{rdf}{S,P,O}
terms. The other, called \index{rdf_save/2}\predref{rdf_save}{2} is part of the RDF storage module
described in \secref{rdfdb} and writes a database directly to a file or
stream. The first (\index{rdf_write_xml/2}\predref{rdf_write_xml}{2}) is used to exchange computed graphs
to external programs using network communication, while the second
(\index{rdf_save/2}\predref{rdf_save}{2}) is used to save modified graphs back to file. The resulting
code duplication is unfortunate, but unavoidable. Creating a temporary
graph in a database requires potentially much memory, and harms
concurrency, while graphs fetched from the database into a list may not
fit in the Prolog stacks and is also considerably slower than a direct
write.


\subsection{Storage of RDF}
\label{sec:rdfdb}

% As list of rdf(S,P,O) terms
% As predicate rdf(S,P,O) in rdf_db.pl

Assuming the `harvesting' use-case, we need to implement a
predicate \term{rdf}{?S,?P,?O}. Indexing the database is crucial for
good performance. \Tabref{rdfindex} illustrates the calling pattern from
a real-world application counting 4 million triples. In addition, we
know our data is described by \figref{rdftypes}. The RDF store was
developed in the context of projects which formulated the following
requirements.

\begin{itemize}
    \item Upto at least 10 million triples on 32-bit hardware.
    \item Fast graph traversal using any instantiation pattern.
    \item Case-insensitive search on literals.
    \item Prefix search on literals for completion in the UI.
    \item Searching for words that appear in literals.
    \item Multi-threaded access based on read/write locks.
    \item Transaction management and persistent store.
    \item Maintain source information, so we can update, save or
          remove data based on its source.
    \item Fast load/save of current state.
\end{itemize}

\begin{table}
    \caption{Call-statistics on a real-world system}
    \label{tab:rdfindex}
\begin{tabular}{cccr}
\hline\hline
\multicolumn{3}{c}{\bf Index pattern} & \bf Calls \\
\hline
-   &	-   &	-   &	58 \\
+   &	-   &	-   &	253,554 \\
-   &	+   &	-   &	62 \\
+   &	+   &	-   &	23,292,353 \\
-   &	-   &	+   &	633,733 \\
-   &	+   &	+   &	7,807,846 \\
+   &	+   &	+   &	26,969,003 \\
\hline\hline
\end{tabular}
\end{table}

Our first version of the database used the Prolog database with
secondary tables to improve indexing. As requirements pushed us against
the limits of what is achievable in a 32-bit address-space we decided to
implement the low level store in C. Profiting from the known uniform
structure of the data we realised about two times more compact storage
with better indexing than using a pure Prolog approach. We took the
following design decisions for the C-based storage module:

\begin{itemize}
    \item The RDF \emph{predicates} are represented as unique entities and
          organised according to the rdfs:subPropertyOf relation in
	  multiple hierarchies. The root of each hierarchy is used to
	  compute the hash for the triple. If there is no unique root
	  due to a cycle an arbitrary predicate is assigned to be the
	  root.

    \item Literals are kept in an AVL tree, sorted case-insensitive
	  and case-preserving (e.g. AaBb\ldots).  Numeric literals
	  preceed all non-numeric and are kept sorted on their
	  numeric value.   Storing literals in a separate sorted table
	  allows for indexed search for prefixes and numeric values.  It
	  also allows for monitoring creation and destruction of
	  literals to maintain derived tables such as stemming or
	  double methaphone \cite{Philips:2000:DMS} based on
	  \index{rdf_monitor/3}\predref{rdf_monitor}{3} described below. The space overhead of
	  maintaining the table is roughly cancelled by avoiding
	  duplicates. Experience on real data ranges between -5\% and
	  +10\%.

    \item Resources are represented by Prolog atom-handles.  The hash
	  is computed from the handle-value.  Note that avoiding the
	  translation between Prolog atom and text both avoids
	  duplication of data and table-lookup. We consider
	  this a crucial aspect.

    \item Each triple is represented by the atom-handle for the subject,
	  predicate-pointer, atom-handle or literal pointer for object,
	  a pointer to the source, a line number,
	  a general bit-flag field and 6 `hash-next' pointers covering
	  all indexing patterns except for +,+,+ and +,-,+.  Queries
	  using the pattern +,-,+ are rare.  Fully instantiated
	  queries internally use the pattern +,+,-, assuming few
	  values on the same property.  Considering experience with
	  real data we will probably add a +,+,+ index in the future.
	  The un-indexed table is a simple linked list. The others are
	  hash-tables that are automatically resized if they become too
	  populated.
\end{itemize}

The store itself does not allow for writes while there are active reads
in progress. If another thread is reading the write operation will stall
until all threads have finished reading. If the thread itself has an
open choicepoint a permission error exception is raised. To arrive at
meaningful update semantics we introduced \jargon{transactions}. The
thread starting a transaction obtains a write-lock, initially allowing
readers to proceed. During the transaction all changes are recorded in a
linked list of actions. If the transaction is ready for commit, the
thread denies access for new readers and waits for all readers to vanish
before updating the database. Transactions are realised by
\term{rdf_transaction}{:Goal}. If \arg{Goal} succeeds, its choicepoints
are discarded and the transaction is committed. If \arg{Goal} fails or
raises an exception the transaction is discarded and \index{rdf_transaction/1}\predref{rdf_transaction}{1}
returns failure or exception. Transactions can be nested. Nesting a
transaction places a transaction-mark in the list of actions of the
current transaction. Committing implies removing this mark from the
list. Discarding removes all action cells following the mark as well as
the mark itself.

It is possible to monitor the database using \term{rdf_monitor}{:Goal,
+Events}.  Whenever one of the monitored events happens \arg{Goal} is
called.  Modifying actions inside a transaction are called during the
commit. Modifications by the monitors are collected in a new transaction
which is committed immediately after completing the preceeding commit.
Monitor events are assert, retract, update, new_literal, old_literal,
transaction begin/end and file-load.  \arg{Goal} is called in the
modifying thread.  As this thread is holding the database write lock
all invocations of monitor calls are fully serialized.

Although the 9,000 triples per second of the RDF/XML parser ranks it
among the fast parsers, loading 10 million triples takes nearly 20
minutes. For this reason we developed a binary format. The format is
described in \cite{Wielemaker:03a} and loads approximately 10 times
faster than RDF/XML, while using about the same space. The format is
independent from byte-order and word-length, supporting both 32- and
64-bit hardware.

Persistency is achieved through the library \file{rdf_persistency.pl},
which uses \index{rdf_monitor/3}\predref{rdf_monitor}{3} to maintain a set of files in a directory. Each
source known to the database is represented by two files, one file
representing the initial state using the quick-load binary format and
one file containing Prolog terms representing changes, called the
\jargon{journal}. 


\subsection{Reasoning with RDF documents}
\label{sec:rdfreasoning}

We have identified two approaches for reasoning on top of the plain RDF
predicate for more high-level languages such as RDFS or OWL. One
approach is taken by the SeRQL query system described in \secref{serql}.
It is based on the observation that these languages provide rules to
deduce new triples from the set of known triples. The API for high level
languages is now simply the \index{rdf/3}\predref{rdf}{3} predicate, where \term{rdf}{S,P,O} is
true for any triple in the deductive closure of the original triple set
under the given language. The deductive closure can be realised using
full forward reasoning, deducing new triples until this is no longer
possible or by a combination of backward reasoning and forward
reasoning. An alternative approach is to consider RDFS or OWL at the
conceptual level and introduce a set of predicates that are inspired on
this level. This approach is taken by our library \file{rdfs.pl},
defining predicates such as \term{rdfs_individual_of}{?Resource,
?Class}, \term{rdfs_subclass_of}{?Sub, ?Super}. \Figref{rdfentail}
illustrates the difference in these approaches.

\begin{figure}
\figrule
\hfill
\begin{minipage}[t]{.32\textwidth}
\begin{verbatim}
% triples
mary type woman .
women type Class .
women subClassOf human .
human type Class .
\end{verbatim}

\noindent
\end{minipage}
\hfill
\begin{minipage}[t]{.30\textwidth}
\begin{verbatim}
% entailment interface
?- rdf(mary, type, X).
X = woman ;
X = human ;
No
\end{verbatim}

\noindent
\end{minipage}
\hfill
\begin{minipage}[t]{.35\textwidth}
\begin{verbatim}
% RDFS interface
?- rdfs_individual_of(mary, X).
X = woman ;
X = human ;
No
\end{verbatim}

\noindent
\end{minipage}
\hfill
    \caption{Different interface styles for RDFS}
    \label{fig:rdfentail}
\figrule
\end{figure}

\subsection{Experience}

The RDF infrastructure is used in two from the three case-studies
described at the end of this paper. RDF has a natural representation in
Prolog, either as a list of terms or as a pure predicate. Prolog
non-determinism greatly simplifies querying the database. Transitivity
is easily expressed using recursion. However, as cycles in SW graphs are
allowed and frequent, such algorithms must be protected against
them. Cycle-detection complicates the code and harms performance. We
plan to investigate tabling \cite{iclp95*697} to improve on this
situation.

Although designed as RDF store for SW-based projects, the infrastructure
is also commonly used to create RDF documents from other sources as well
as for filtering and reorganizing RDF documents. In the e-culture
project\fnurl{e-culture.multimedian.nl} it has been used to convert
WordNet \cite{Miller:95a} from its Prolog representation and the Getty
thesauri\fnurl{http://www.getty.edu/research/conducting_research/vocabularies/}
from XML (4GB data) into RDF.
