%%
%%

\section*{Bacula Regression Testing}
\label{_ChapterStart8}
\index{Testing!Bacula Regression }
\index{Bacula Regression Testing }
\addcontentsline{toc}{section}{Bacula Regression Testing}

\subsection*{General}
\index{General }
\addcontentsline{toc}{subsection}{General}

This document is intended mostly for developers who wish to ensure that their
changes to Bacula don't introduce bugs in the base code. 

You can find the existing regression script in the Bacula CVS on the
SourceForge CVS in the project tree named {\bf regress}. 

There are two different aspects of regression testing that this document will
discuss: 1. Running the Regression Script, 2. Writing a Regression test. 

\subsection*{Running the Regression Script}
\index{Running the Regression Script }
\index{Script!Running the Regression }
\addcontentsline{toc}{subsection}{Running the Regression Script}

There are a number of different tests that may be run, such as: the standard
set that uses disk Volumes and runs under any userid; a small set of tests
that write to tape; another set of tests where you must be root to run them.
To date, each subset of tests runs no more than about 15 minutes. 

\subsubsection*{Setting the Configuration Parameters}
\index{Setting the Configuration Parameters }
\index{Parameters!Setting the Configuration }
\addcontentsline{toc}{subsubsection}{Setting the Configuration Parameters}

Once you have the regression directory loaded, you will first need to create a
custom xxx.conf file for your system. You can either edit {\bf prototype.conf}
directly or copy it to a new file and edit it. To see a real example of a
configuration file, look at {\bf kern.conf}. The variables you need to modify
are: 

\footnotesize
\begin{verbatim}
                                                                                        
# Where to get the source to be tested
BACULA_SOURCE="${HOME}/bacula/k"
                                                                                        
# Where to send email   !!!! Change me !!!!!!!
EMAIL=your-email@domain.com
                                                                                        
# Full path where to find sqlite
DEPKGS="${HOME}/bacula/depkgs/sqlite"
                                                                                        
TAPE_DRIVE="/dev/nst0"
# if you don't have an autochanger set
#   AUTOCHANGER to /dev/null
AUTOCHANGER="/dev/sg0"
# This must be the path to the autochanger
#  including its name
AUTOCHANGER_PATH="/bin/mtx"
                                                                                        
\end{verbatim}
\normalsize

\begin{itemize}
\item {\bf BACULA\_SOURCE} should be the full path to the Bacula source code 
   that you wish to test. 
\item {\bf EMAIL} should be your email addres. Please remember  to change this
   or I will get a flood of unwanted  messages. You may or may not want to see
   these emails. In  my case, I don't need them so I direct it to the bit bucket.

\item {\bf SQLITE\_DIR} should be the full path to the sqlite package,  must
   be build before running a Bacula regression, if you are  using SQLite. This
   variable is ignored if you are using  MySQL or PostgreSQL. To use PostgreSQL,
edit the Makefile  and change (or add) WHICHDB?=``\verb{--{with-postgresql''.  For
MySQL use ``WHICHDB?=''\verb{--{with-mysql``. 
\item {\bf TAPE\_DRIVE} is the full path to your tape drive.  The base set of
   regression tests do not use a tape, so  this is only important if you want to
   run the full tests. 
\item {\bf AUTOCHANGER} is the name of your autochanger device.  Set this to
   /dev/null if you do not have one. 
\item {\bf AUTOCHANGER\_PATH} is the full path including the  program name for
   your autochanger program (normally  {\bf mtx}. Leave the default value if you
   do not have one. 
\end{itemize}

\subsubsection*{Building the Test Bacula}
\index{Building the Test Bacula }
\index{Bacula!Building the Test }
\addcontentsline{toc}{subsubsection}{Building the Test Bacula}

Once the above variables are set, you can build Bacula by entering: 

\footnotesize
\begin{verbatim}
./config xxx.conf
make setup
\end{verbatim}
\normalsize

Where xxx.conf is the name of the conf file containing your system parameters.
This will build a Makefile from Makefile.in, then copy the source code within
the regression tree (in directory regress/build), configure it, and build it.
There should be no errors. If there are, please correct them before
continuing. 

\subsubsection*{Running the Disk Only Regression}
\index{Regression!Running the Disk Only }
\index{Running the Disk Only Regression }
\addcontentsline{toc}{subsubsection}{Running the Disk Only Regression}

Once Bacula is built, you can run the basic disk only non-root regression test
by entering: 

\footnotesize
\begin{verbatim}
make test
\end{verbatim}
\normalsize

This will run the base set of tests using disk Volumes, currently (19 Dec
2003), there are current 18 separate tests that run. If you are testing on a
non-Linux machine two of the tests will not be run. In any case, as we add new
tests, the number will vary. It will take about 5 or 10 minutes if you have a
fast (2 GHz) machine, and you don't need to be root to run these tests (I run
under my regular userid). The result should be something similar to: 

\footnotesize
\begin{verbatim}
Test results
  
  ===== Backup Bacula Test OK =====
  ===== Verify Volume Test OK =====
  ===== sparse-test OK =====
  ===== compressed-test OK =====
  ===== sparse-compressed-test OK =====
  ===== Weird files test OK =====
  ===== two-jobs-test OK =====
  ===== two-vol-test OK =====
  ===== six-vol-test OK =====
  ===== bscan-test OK =====
  ===== Weird files2 test OK =====
  ===== concurrent-jobs-test OK =====
  ===== four-concurrent-jobs-test OK =====
  ===== bsr-opt-test OK =====
  ===== bextract-test OK =====
  ===== recycle-test OK =====
  ===== span-vol-test OK =====
  ===== restore-by-file-test OK =====
  ===== restore2-by-file-test OK =====
  ===== four-jobs-test OK =====
  ===== incremental-test OK =====
\end{verbatim}
\normalsize

and the working tape tests are: 

\footnotesize
\begin{verbatim}
Test results
  
  ===== Bacula tape test OK =====
  ===== Small File Size test OK =====
  ===== restore-by-file-tape test OK =====
  ===== incremental-tape test OK =====
  ===== four-concurrent-jobs-tape OK =====
  ===== four-jobs-tape OK =====
\end{verbatim}
\normalsize

Each separate test is self contained in that it initializes to run Bacula from
scratch (i.e. newly created database). It will also kill any Bacula session
that is currently running. In addition, it uses ports 8101, 8102, and 8103 so
that it does not intefere with a production system. 

\subsubsection*{Other Tests}
\index{Other Tests }
\index{Tests!Other }
\addcontentsline{toc}{subsubsection}{Other Tests}

There are a number of other tests that can be run as well. All the tests are a
simply shell script keep in the regress directory. For example the ''make
test`` simply executes {\bf ./all-non-root-tests}. The other tests are: 

\begin{description}

\item [all\_non-root-tests]
   \index{all\_non-root-tests }
   All non-tape tests not requiring root.  This is the standard set of tests,
that in general, backup some  data, then restore it, and finally compares the
restored data  with the original data.  

\item [all-root-tests]
   \index{all-root-tests }
   All non-tape tests requiring root permission.  These are a relatively small
number of tests that require running  as root. The amount of data backed up
can be quite large. For  example, one test backs up /usr, another backs up
/etc. One  or more of these tests reports an error -- I'll fix it one  day. 

\item [all-non-root-tape-tests]
   \index{all-non-root-tape-tests }
   All tape test not requiring root.  There are currently three tests, all run
without being root,  and backup to a tape. The first two tests use one volume,
and the third test requires an autochanger, and uses two  volumes. If you
don't have an autochanger, then this script  will probably produce an error. 

\item [all-tape-and-file-tests]
   \index{all-tape-and-file-tests }
   All tape and file tests not requiring  root. This includes just about
everything, and I don't run it  very often. 
\end{description}

\subsubsection*{If a Test Fails}
\index{Fails!If a Test }
\index{If a Test Fails }
\addcontentsline{toc}{subsubsection}{If a Test Fails}

If you one or more tests fail, the line output will be similar to: 

\footnotesize
\begin{verbatim}
  !!!!! concurrent-jobs-test failed!!! !!!!!
\end{verbatim}
\normalsize

If you want to determine why the test failed, you will need to modify the
script so that it prints. Do so by finding the file in {\bf regress/tests}
that corresponds to the name printed. For example, the script for the above
error message is in: {\bf regress/tests/concurrent-jobs-test}. 

In order to see the output produced by Bacula, you need only change the lines
that start with {\bf @output} to {\bf @tee}, then rerun the test by hand. it
is very important to start the test from the {\bf regress} directory. 

To modify the test mentioned above so that you can see the output, change
every occurrence of {\bf @output} to {\bf @tee} in the file. In rare cases you
might need to remove the {\bf 2\gt{}\&1 \gt{}/dev/null} from the end of the
{\bf bacula}, {\bf bconsole}, or {\bf diff} lines, but this is rare. 

\subsection*{Writing a Regression Test}
\index{Test!Writing a Regression }
\index{Writing a Regression Test }
\addcontentsline{toc}{subsection}{Writing a Regression Test}

Any developer, who implements a major new feature, should write a regression
test that exercises and validates the new feature. Each regression test is a
complete test by itself. It terminates any running Bacula, initializes the
database, starts Bacula, then runs the test by using the console program. 

\subsubsection*{Running the Tests by Hand}
\index{Hand!Running the Tests by }
\index{Running the Tests by Hand }
\addcontentsline{toc}{subsubsection}{Running the Tests by Hand}

You can run any individual test by hand by cd'ing to the {\bf regress}
directory and entering: 

\footnotesize
\begin{verbatim}
tests/<test-name>
\end{verbatim}
\normalsize

\subsubsection*{Directory Structure}
\index{Structure!Directory }
\index{Directory Structure }
\addcontentsline{toc}{subsubsection}{Directory Structure}

The directory structure of the regression tests is: 

\footnotesize
\begin{verbatim}
  regress                - Makefile, scripts to start tests
    |------ scripts      - Scripts and conf files
    |-------tests        - All test scripts are here
    |
    |------------------ -- All directories below this point are used
    |                       for testing, but are created from the
    |                       above directories and are removed with
    |                       "make distclean"
    |
    |------ bin          - This is the install directory for
    |                        Bacula to be used testing
    |------ build        - Where the Bacula source build tree is
    |------ tmp          - Most temp files go here
    |------ working      - Bacula working directory
    |------ weird-files  - Weird files used in two of the tests.
\end{verbatim}
\normalsize

\subsubsection*{Adding a New Test}
\index{Adding a New Test }
\index{Test!Adding a New }
\addcontentsline{toc}{subsubsection}{Adding a New Test}

If you want to write a new regression test, it is best to start with one of
the existing test scripts, and modify it to do the new test. 

When adding a new test, be extremely careful about adding anything to any of
the daemons' configuration files. The reason is that it may change the prompts
that are sent to the console. For example, adding a Pool means that the
current scripts, which assume that Bacula automatically selects a Pool, will
now be presented with a new prompt, so the test will fail. If you need to
enhance the configuration files, consider making your own versions. 
