\chapter{Plugins}
\def\kubernetes{Kubernetes~}
\section{Kubernetes Plugin}
\label{kubernetesplugin}
\subsection{Features Summary}
\begin{bitemize}
	\item \kubernetes cluster resources configuration backup
	\item Ability to restore single \kubernetes configuration resource
	\item Ability to restore \kubernetes resource configuration to local directory
	\item \kubernetes Persistent Volumes data backup and restore
	\item Ability to restore \kubernetes Persistent Volumes data to local directory
	\item Ability to use new \kubernetes CSI driver snapshot features to perform Persistent Volume data backup using Cloned or Snapshot techniques
	\item Ability to execute user defined commands on required Pod containers to prepare and clean data backup
	\item Configure \kubernetes workload backup requirements directly with Pod annotations
\end{bitemize}


\subsection{Glossary}
\begin{description}
	\item[Docker] is a computer program that performs operating-system-level
		virtualization, also known as "containerization".
	\item[Kubernetes] is an open-source system for automating deployment, scaling, and
		management of containerized applications. It groups containers that make up an application
		into logical units for easy management and discovery.
	\item[Container] is a runnable instance of an image and refers to an operating system
		feature in which the kernel allows the existence of multiple isolated user-space instances.
	\item[Kubernetes Resource] is a configuration entity which describe how a particular component
		should be executed and maintained. It includes core components like Pod, Service,
		Persistent Volume Claim or extended components like Replica Set, Deployment or
		Stateful Set.
	\item[K8S] is short for Kubernetes and is used interchangeably.
	\item[Persistent Volume] (PV) is a piece of storage in the cluster that has been
		dynamically provisioned or provisioned by an administrator who's life cycle is independent of
		any individual Pod that uses the PV.
	\item[Persistent Volume Claim] (PVC) is a request for storage by a user which could be
		consumed by Pod. This resource associates a PV with a Pod.
	\item[Bacula Backup Proxy Pod] is a Kubernetes service dedicated to PVC Data archive backup
		and restore operations which acts as a proxy application between Persistent Volume data and
		a Bacula File Daemon with the Kubernetes Plugin. This application requires a dedicated
		bacula-backup docker image to be deployed.
	\item[CSI] is the Container Storage Interface. CSI is a standard for exposing arbitrary block and 
		file storage systems to containerized workloads on Container Orchestration Systems (COs) 
		like Kubernetes. Using CSI third-party storage providers can write and deploy plugins exposing 
		new storage systems in Kubernetes without ever having to touch the core Kubernetes code.
\end{description}

Red Hat, Red Hat Enterprise Linux are trademarks
of Red Hat, Inc., registered in the United States and other countries.

\subsection{\kubernetes Backup Overview}

Containers are very light system level virtualization with less overhead
because programs in virtual partitions use the operating system's normal
system call interface and do not need to be subjected to emulation or
run in an intermediate virtual machine. \kubernetes manages a set of
containers to create a flexible execution environment for applications and
services.

\smallskip
The Bacula \kubernetes plugin will save all the important
Kubernetes resources which make up the application or service. This includes the
following namespaced objects:

\label{kubernetesresources}
\begin{bitemize}
	\item{Config Map}
	\item{Daemon Set}
	\item{Deployment}
	\item{Endpoint}
	\item{Limit Range}
	\item{Pod}
	\item{Persistent Volume Claim}
	\item{Pod Template}
	\item{Replica Set}
	\item{Replication Controller}
	\item{Resource Quota}
	\item{Secret}
	\item{Service}
	\item{Service Account}
	\item{Stateful Set}
	\item{PVC Data Archive}\footnote{The PVC Data is not exact Kubernetes Object
		but represents archive of real data existed on selected PVC.}
\end{bitemize}

and non namespaced objects:

\begin{bitemize}
	\item{Namespace}
	\item{Persistent Volume}
	\item{Storage Class}
\end{bitemize}

All namespaced objects which belong to the particular namespace are grouped
together for easy backup data browsing and recovery.

\smallskip
Users and service accounts can be authorized to access the server API. This process goes
through authentication, authorization and admission control. To be able to successfully backup
the Kubernetes resources, it is required to have a user or service account with the correct permissions
and rights to be successfully authenticated and authorized to access the API server and resources to
be backed up.

\smallskip
For resource configuration backups, the user or service account must be able to read
and list the resources. In the case of PVC Data backup, it is also required that the user
or service account can create and delete pods because the plugin will need to create and delete
the Bacula Backup Proxy Pod during the backup.

\smallskip
Please see the Kubernetes documentation for more details.

\subsection{\kubernetes Persistent Volume Claim Backup}

All Pods in Kubernetes are ephemeral and may be destroyed manually or
by operations from controllers. Pods do not store data locally because 
stored data would be destroyed with a pod's life cycle management, so data
is saved on Persistent Volumes using Persistent Volume Claim objects to
control Volume Space availability.

\smallskip
This brings a new challenge to data backup. Fortunately most of the challenges
found here are similar to standard bare metal or virtualized environments.
As with bare metal and virtual machine environments,  data stored in databases
should be protected with dedicated Bacula plugins that take advantage
of the database backup routines.

\bigskip
\textit{Please refer to the appropriate Bacula plugin whitepapers for more details
on database backups.}
\bigskip

On the other hand, most non-database applications store data as simple
flat files we can backup as-is without forcing complicated transactions or data
consistency procedures. This use case is handled directly with the \kubernetes plugin
using a dedicated Bacula Backup Proxy Pod executed in the cluster.

\smallskip
If the container application is more complex, it is possible to execute
commands inside the container to quiesce the application.
\begin{bitemize}
	\item before the snapshot
	\item after the snapshot
	\item after the backup of the container
\end{bitemize}

A problem with command execution can abort the backup of the container with the
\texttt{run.*.failonerror} annotation. You can find detailed description of this feature
at \vref{csisnapshot}.

\smallskip
A Bacula Backup Proxy Pod is a service executed automatically by the \kubernetes plugin
which manages secure access to Persistent Volume data for the plugin. It is
executed on the \kubernetes cluster infrastructure and requires a network connection to
the \kubernetes plugin for data exchange on backup and restore operations. No external
cluster service like \texttt{NodePort}, \texttt{LoadBalancer}, \texttt{Ingress} or
\texttt{Host Based Networking} configuration is required to use this feature.

It is also not required to permanently deploy and run this service on the cluster itself
as it is executed on demand. The Bacula Backup Proxy Pod does not consume any
valuable compute resources outside of the backup window. You can even operate your
\kubernetes backup solution (Bacula Edition service with \kubernetes plugin)
directly from your on-premise backup infrastructure to backup a public Kubernetes
cluster (it requires a simple port forwarding firewall rule) or use public backup
infrastructure to backup on-premise Kubernetes cluster(s).
Support for these varied architecture modes is built into the \kubernetes plugin.
It is designed to be a \textit{One-Click} solution for \kubernetes backups.

\smallskip

It is possible to backup and restore any PVC data including PVCs not attached
to any running Kubernetes Pod.

\subsection{CSI Snapshot Support}
\label{csisnapshot}

Kubernetes CSI Snapshot functionality support together with a bunch of other features
was added. Starting from this version you can use CSI Snapshots to acquire a consistent
data view of selected Volume. Additionally, you can configure remote command execution on
a selected Container of the Pod. You can configure command execution just before or after 
a Pod backup and just after snapshot creation.

Our plugin uses the volume clone or the snapshot api when doing a volume
snapshot. CSI drivers may or may not have implemented the volume cloning or
snapshot functionalities.

\subsection{Kubernetes Pod Annotations}

The \textbf{CSI Snapshot Support} feature described in \vref{csisnapshot} comes with
a configuration of Volume data backup using Kubernetes Pod annotations. This feature allows
you to define what volumes (PVC Data) to backup, where and what commands to execute, and 
how to react to some failures to achieve the best results from data snapshot functionality. 
You can select which volumes mounted at the Pod you want to backup, the 
preferred backup mode for the Pod, and what commands you want to execute.

\smallskip
The supported annotations are:

\begin{description}
\item[\texttt{bacula/backup.mode:[snapshot|standard]}] defines how to handle
	PVC Data backups. If not defined, the default is \texttt{snapshot}.

\item[\texttt{bacula/backup.volumes:<pvcname[,pvcname2...]>}] defines what
	volumes will be backed up from this Pod. Multiple PVC names may be selected 
	as a comma separated list. This annotation is required if you want to backup
        volumes on this Pod. Any other annotations are optional.

\item[\texttt{bacula/run.before.job.container.command:<container>/<command>}] 
	defines what command to execute on which container of the Pod before any snapshot
	and data backup of this Pod occurs. An asterisk (\texttt{*}) as \texttt{<container>} name means 
	to execute \texttt{<command>} on all containers.

\item[\texttt{bacula/run.before.job.failjobonerror:[yes|no]}] defines whether or not to fail the job
	when the exit code of the executed command is not zero. The default is yes.

\item[\texttt{bacula/run.after.job.container.command:<container>/<command>}]
	defines what command to execute on which container of the Pod after all snapshot
	and data backup of this Pod. An asterisk (\texttt{*}) as \texttt{<container>} name means 
	to execute \texttt{<command>} on all containers.

\item[\texttt{bacula/run.after.job.failjobonerror:[yes|no]}] defines whether or not to fail the job
        when exit code of the executed command is not zero. The default is no.

\item[\texttt{bacula/run.after.snapshot.container.command:<container>/<command>}]
	defines what command to execute on which container of the Pod after all snapshot
	creations and just before any data backup. 
	An asterisk (\texttt{*}) as \texttt{<container>} name means to execute \texttt{<command>} 
	on all containers.

\item[\texttt{bacula/run.after.snapshot.failjobonerror:[yes|no]}] defines to fail the job
	when exit code of the executed command is not zero. The default is no.
\end{description}

Pod annotations is an extension to the current PVC Data backup feature available
with the \texttt{pvcdata=\ldots} plugin parameter as described in \vref{backupparameters}.
This is an independent function which may be used together with the functionality 
described above, especially since both use the same data archive stream handling with 
Bacula Backup Pod.

\smallskip
All you need to use a new feature is to configure selected Pod annotations and make sure
that the backup for a required Kubernetes namespace is properly configured. There is no 
need for any plugin configuration modifications. A Pod's volumes will be backed up automatically.

\subsection{Examples}

Below you can find some examples how to configure Bacula annotations in Kubernetes Pods.

\smallskip
In the example below you will use a simple Linux command \texttt{sync} to synchronize 
cached writes to persistent storage before volume snapshot.

\begin{verbatim}
apiVersion: v1
  kind: Pod
  metadata:
    name: app1
    namespace: default
    annotations:
      bacula/backup.mode: snapshot
      bacula/run.before.job.container.command: "*/sync -f /data1; sync -f /data2"
      bacula/run.before.job.failjobonerror: "no"
      bacula/backup.volumes: "pvc1,  pvc2"
  spec:
    containers:
    - image: ubuntu:latest
      name: test-container
      volumeMounts:
      - name: pvc1
        mountPath: /data1
      - name: pvc2
        mountPath: /data2
      volumes:
      - name: pvc1
        persistentVolumeClaim:
          claimName: pvc1
      - name: pvc2
        persistentVolumeClaim:
          claimName: pvc2
\end{verbatim}

In the example below you will use PostgreSQL's database data files quiesce feature to perform consistent 
backup with snapshot\footnote{The final PostgreSQL backup solution requires more configuration and preparation
which was skipped in this example to make it clear}.

The first command (\textit{run.before.job.container.command}) freezes writes to database files 
and the second (\textit{run.after.snapshot.container.command}) resumes standard database operation
as soon as PVC snapshot becomes ready.

\begin{verbatim}
apiVersion: v1
kind: Pod
metadata:
  name: postgresql13
  namespace: default
  annotations:
    bacula/backup.mode: standard
    bacula/run.before.job.container.command: "*//bin/startpgsqlbackup.sh"
    bacula/run.after.snapshot.container.command: "*//bin/stoppgsqlbackup.sh"
    bacula/run.after.snapshot.failjobonerror: "yes"
    bacula/backup.volumes: "pgsql"
  spec:
    containers:
    - image: postgresql:13
      name: postgresql-server
      volumeMounts:
      - name: pgsql
        mountPath: /var/lib/pgsql
      volumes:
      - name: pgsql
        persistentVolumeClaim:
          claimName: pgsql-volume
\end{verbatim}

\subsection{Run Container Command annotation}

All flavors of the Run Container Command parameters are remotely executed using the Kubernetes Pod 
remote execution API. Every command is prepared to execute with a standard Linux shell \texttt{/bin/sh}.
This requires that a container image has to have the specified shell available. Using command shell 
execution gives flexibility to command execution or even allows for preparation of small scripts without
additional container image customization.

\subsection{Build}

To enable the build of the Kubernetes plugin, add the following options to
your \texttt{./configure} command line:

\begin{bVerbatim}
./configure --enable-kubernetes-plugin [... other options]
\end{bVerbatim}

The Kubernetes plugin is written in Python, to ensure the stability of the Python environment,
the default installation process will use Cython to convert the Python program to an executable.

\subsection{Installation}

The Bacula File Daemon and the Kubernetes plugin can be installed outside of the
Kubernetes cluster on a server which has access to the Kubernetes API, or inside a protected
cluster in a Container / Pod. The Kubernetes plugin can be installed on different
operating systems and distributions, so the Bacula File Daemon
for the correct operating system and platform has to be used.

There is no need, or in some solutions (when \texttt{K8S} is a cloud service
like \texttt{GKE} or \texttt{EKS}), is it even a possible to install a Bacula File Daemon
and the Kubernetes Plugin on a Kubernetes Master Server (\texttt{etcd}, \texttt{control pane}).

The \texttt{--enable-kubernetes-plugin} command line option of the \texttt{./configure} script
will allow to install the kubernetes plugin with \texttt{make install}

The following packages are required to build and use the kubernetes plugin:
\begin{itemize}
\item Cython (to bundel python dependencies)
\end{itemize}

Note that it should be possible to use directly python files rather than Cython.

\subsection{File Daemon Configuration}

The \texttt{Plugin Directory} directive of \texttt{File Daemon}
resource in \texttt{/opt/bacula/etc/bacula-fd.conf} must
point to where the \texttt{kubernetes-fd.so} plugin file is installed. The
standard Bacula plugin directory is \texttt{/opt/bacula/plugins}

\begin{verbatim}
FileDaemon {
  Name = bacula-fd
  Plugin Directory = /opt/bacula/plugins
  ...
}
\end{verbatim}

\subsection{Bacula Backup Proxy Pod Image}
\label{baculatarimage}
For Kubernetes PVC Data backup functionality, the Bacula Kubernetes plugin
requires a dedicated Bacula Backup Proxy Pod which is automatically deployed
using an image that is available in the bacula/scripts/kubernetes-bacula-backup
directory.

This image should be installed manually on your local Docker images
registry service which is available on your Kubernetes cluster as a source
for application images.

Installation of the image can be performed with the following example commands:

\begin{verbatim}
# cd bacula-13.0.0/scripts/kubernetes-bacula-backup
# make
# docker load -i bacula-backup-<timestamp>.tar
# docker image tag bacula-backup:<timestamp> <registry>/bacula-backup:<timestamp>
# docker push <registry>/bacula-backup:<timestamp>
\end{verbatim}

Where \texttt{<timestamp>} is the image version generated with the
above package and \texttt{<registry>} is the location of your Docker images registry
service. The exact procedure depends on your Kubernetes cluster deployment, so please
verify the above steps before attempting to run the docker commands.

You can use any registry service available for
your cluster, public or private, i.e. \texttt{gcr.io/}.

\smallskip
Depending on your cluster configuration it may be necessary to set the
\texttt{baculaimage=<name>} plugin parameter (described at \vref{baculaimageparam})
to define which repository and container image to use. The default for this parameter
is \texttt{bacula-backup:<timestamp>} which may not be correct for your deployment.

Another example where you will need to modify the Bacula Backup Proxy Pod
Image is in the case where your registry requires authentication. Please see the
section \vref{advancedpoddeployment}  for more details.

\section{Backup and Restore Operations}

\subsection{Backup}

The plugin can backup a number of Kubernetes Resources including: Deployments,
Pods, Services or Persistent Volume Claims, check chapter~\vref{kubernetesresources}
for a complete list.

\smallskip
The backup will create a single (\texttt{.yaml}) file for any Kubernetes Resource which
is saved. For PVC Data backup functionality the \kubernetes plugin generates a data archive
as a single \texttt{<pvcname>.tar} archive file.
The resources are organized inside the Bacula catalog to facilitate browsing
and restore operations. In the Bacula catalog, Kubernetes resources are represented as follows:

\begin{bitemize}
	\item \texttt{/@kubernetes/namespaces/<namespace>.yaml}  - Namespace definition
	\item \texttt{/@kubernetes/namespaces/<namespace>/<resource>/<name>.yaml}
	 - Resource definitions in the namespace
	\item \texttt{/@kubernetes/namespaces/<namespace>/persistentvolumeclaim/<pvcname>.tar}
		- PVC Data backup in the selected namespace
	\item \texttt{/@kubernetes/persistentvolumes/<pvname>.yaml} - Persistent Volume definition
	\item \texttt{/@kubernetes/storageclass/<scname>.yaml} - Storage Class definition
\end{bitemize}

All supported Kubernetes Resources will be saved if no filter options are set.
You may limit which resources are saved using filtering options described in
chapter~\vref{backupparameters}.
By default, if no filter options are set, all supported Kubernetes Resources will be saved.
To see the Kubernetes Resources that may be filtered, a listing mode is available. This mode is described in chapter~\vref{listing}.

\subsection{Restore}

The \kubernetes plugin provides two targets for restore operations:
\begin{bitemize}
	\item Restore to a Kubernetes cluster
	\item Restore to a local directory
\end{bitemize}


\subsubsection{Restore to a \kubernetes Cluster}

To use this restore method, the \texttt{where=/}
parameter of a Bacula \texttt{restore} command is used. You can select
any supported Kubernetes Resource to restore, or batch restore the whole
namespace or even multiple namespaces. If you select a single resource to restore
it will be restored as is without any dependent objects. In most cases, for
(\texttt{Config Maps}, \texttt{Secrets}, \texttt{Services}, etc.) this is fine and
restore will always succeed.
On the other hand, compound objects (\texttt{Pods}, \texttt{Deployments}, etc.)
won't be ready unless all dependencies are resolved during the restore.
In this case you should make sure that you select all required resources to restore.

\smallskip
In Kubernetes, a successful resource restore doesn't necessarily result in the service
successfully coming online. In some cases further monitoring and investigation
will be required. For example:

\begin{bitemize}
	\item \textit{Container image is unavailable}.
	\item \textit{Volume Claim cannot provision new volume}.
	\item \textit{Untrusted Image Repository}.
	\item \textit{Infrastructure limits exceeded, i.e. a number of Pods or Memory allocations}.
	\item \textit{etc\ldots}
\end{bitemize}

All example cases above must be resolved by the \kubernetes administrator.
When all issues are resolved, the resource should automatically come online. If not, it may
be necessary to repeat a restore to redeploy the Resource configuration.

\smallskip
The \kubernetes plugin does not wait for a Resource to come online during restore.
It checks the Resource creation or replace operation status and reports any errors
in the job log. The only exception to this is PVC Data restore, when the \kubernetes
plugin will wait for a successful archive data restore. This operation is always
executed at the end of the namespace recovery (when pvcdata is restored with other
k8s objects) and should wait for proper PVC mount.

\subsubsection{Restore to a Local Directory}

To use this mode, the \texttt{where=/some/path} Bacula \texttt{restore}
parameter is set to a full path on a server where the Bacula File Daemon and \kubernetes plugin
are installed. If the path does not exist, it will be created by the \kubernetes
plugin. With this restore mode you can restore any saved Kubernetes Resource
including PVC Data archive file to a location on disk.

\subsection{Plugin Configuration}

The plugin is configured using \texttt{Plugin Parameters} defined
in a \texttt{FileSets} -> \texttt{Include} section of the Bacula Enterprise
Edition Director configuration.

\subsection{Generic Plugin Parameters}
\label{genericparameters}

The following \kubernetes plugin parameters affect any type of Job
(Backup, Estimate, or Restore).

\begin{description}
\item[\texttt{abort\_on\_error[=<0 or 1>]}] specifies whether or
	not the plugin should abort execution (and the Bacula Job) if a
	fatal error occurs during a Backup, Estimate, or Restore
	operation.

	This parameter is optional.

\item[\texttt{config=</path/to/file>}] points to the location of
	the config file which defines how to connect to the Kubernetes cluster.
	You can read more about this \textit{kubeconfig} file at:
	\url{https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig/}
	If this directive is omitted and no other access method is configured then
	a default config file location will be used - \texttt{\$HOME/.kube/config}.

	This parameter is optional.

\item[\texttt{incluster}] if this directive is defined then a standard in-cluster
	access to the Kubernetes API will be used. This option should be used only
	when the Bacula File Daemon and \kubernetes plugin are deployed as a Kubernetes
	cluster service in a Pod. In this case, Kubernetes itself will provide the
	required access credentials. This option won't work when executed outside of a Kubernetes
	cluster and services.

	This parameter is optional.

\item[\texttt{host=<url-k8s-api>}] defines a Kubernetes API url. This option
	is used only with \texttt{token} parameter described below. If this option is specified,
	then both parameters are required.

	This parameter is optional.

\item[\texttt{token=<bearer-token>}] defines a \texttt{Bearertoken} used
	for authorization to the Kubernetes API available at \texttt{host=<url-k8s-api>}.
	This option is used only with \texttt{host} parameter described above.
	You can read more about this type of authentication at:
	\url{https://swagger.io/docs/specification/authentication/bearer-authentication/}

	This parameter is optional.

\item[\texttt{verify\_ssl[=<0 or 1>]}] specifies whether or not the plugin
	should verify a Kubernetes API certificate when the connection uses SSL/TLS.
	If set to \texttt{verify\_ssl=0} then verification will be disabled.
	This is useful when connecting to a Kubernetes API server with a self-signed
	certificate. The default behavior if this parameter if not set
	is to perform proper certificate validation.

	This parameter is optional.

\item[\texttt{ssl\_ca\_cert=</path/to/file>}] specifies a file with the CA certificate
	used to customize the Kubernetes API server identity to verify the peer.

	This parameter is optional.

\item[\texttt{timeout=<seconds>}] specifies the number of seconds for various network
	operations. Examples include: Waiting for Bacula Backup Proxy Pod connection
	or Kubernetes API operations, waiting for Pod execution or removal. The default is
	600 seconds. The minimum timeout you may set is 1 second. When an invalid value is set
	the default will be used.

	This parameter is optional.

\item[\texttt{debug[=1]}] specifies that the plugin backend will generate an execution
	debug file at location \texttt{/bacula/working/backendplugin/}. This file can help with troubleshooting
	the job execution if something goes wrong. If not defined then no debug file will be generated.

	This parameter is optional.
\end{description}

\subsection{Estimate and Backup Plugin Parameters}
\label{backupparameters}
These plugin parameters are relevant only for Backup and Estimate jobs:

\begin{description}
\item[\texttt{namespace=<name>}] specifies a Kubernetes namespace
	name which you want to backup. Multiple \texttt{namespace=<name>}
	parameters are allowed if you want to backup multiple namespaces.
	If a namespace with \texttt{name} can not be found its backup will be silently
	ignored. If this parameter is not set, all namespaces will be saved.

	This parameter is optional.

\item[\texttt{persistentvolume=<name>}] specifies a Kubernetes persistent
	volume configuration you want to backup. Multiple \texttt{persistentvolume=<name>}
	parameters are allowed if you want to backup multiple volumes.
	You can use standard shell wildcard pattern matching to select multiple volumes
	using a single \texttt{persistentvolume} parameter.
	If a persistent volume with \texttt{name} can not be found its backup will be silently
	ignored. If this parameter is not set, all persistent volume configurations will be saved
	unless \texttt{pvconfig=0} is set as described below.

	This parameter is optional.

\item[\texttt{pvconfig=[0|1]}] this option is used to disable persistent volume
	configuration backups when \texttt{pvconfig=0} is set. The default is to
	backup persistent volume configurations.

	This parameter is optional.

\item[\texttt{storageclass=<name>}] specifies a Kubernetes Storage Class
configuration to be backed up. Multiple \texttt{storageclass=<name>}
parameters are allowed if you want to backup multiple resources.
You can use standard shell wildcard pattern matching to select multiple volumes
using a single \texttt{storageclass} parameter.
If a storage class with \texttt{name} can not be found, its backup will be silently
ignored. If this parameter is not set, all storage class configuration will be saved
unless \texttt{scconfig=0} is set as described below.

This parameter is optional.

\item[\texttt{scconfig=[0|1]}] this option is used to disable Storage Class
configuration backups when \texttt{scconfig=0} is set. The default is to
backup Storage Class resource configurations.

This parameter is optional.

\item[\texttt{pvcdata[=<pvcname>]}] specifies a Kubernetes Persistent
	Volume Claim name you want to make a PVC Data archive for. As all PVCs are
	namespaced objects, to use this option you should select a required
	namespace with \texttt{namespace=<name>} parameter. If you define a simple
	\texttt{pvcdata} parameter without the equals sign (\textit{"="}) and subsequent
	value, all detected persistent volume claims will be selected for the PVC Data
	archive backup.

	This parameter is optional.
\end{description}

If none of the parameters above are specified, then all available
Namespaces and Persistent Volume Configurations will be backed up.
However, the plugin does not force a PVC Data archive backup in this case.

\subsection{Backup and Restore Plugin Parameters}
\label{rbackupparameters}
These plugin parameters are relevant only for Backup and Restore
jobs when the PVC Data archive functionality is used:

\begin{description}
	\item[\texttt{fdaddress=<IP or name>}] is the IP address or host name where the \kubernetes
		plugin should listen for incoming connections from a Bacula Backup Proxy Pod. This
		parameter, combined with \texttt{fdport=<number>} below will define a socket to listen on.
		The default is to listen on all available interfaces (\texttt{0.0.0.0}) which should be
		fine in most cases.

		This parameter is optional if \texttt{pluginhost=<IP or name>} is defined.

	\item[\texttt{fdport=<number>}] is a port number on which \kubernetes plugin should
		listen for incoming connections from a Bacula Backup Proxy Pod. This parameter, combined with
		\texttt{fdaddress=<IP or name>} above will define a socket to listen on.
		 The default is to listen on port \texttt{9104}.

		This parameter is optional.

	\item[\texttt{pluginhost=<IP or name>}] is the entry point address where a Bacula Backup
		Proxy Pod should connect during backup or restore operations. The name should be
		resolvable on the \kubernetes cluster, otherwise an IP address must be used here.
		This with \texttt{pluginport=<number>} parameter below will define the exact service entry point.
		The default is to use the value from \texttt{fdaddress=<IP or name>} parameter above.

		This parameter is required when \texttt{fdaddress=<IP or name>} is not defined. Otherwise
		it is optional.

	\item[\texttt{pluginport=<number>}] is the port number for service entry point address
		where the Bacula Backup Proxy Pod should connect during backup or restore operations.
		This, combined  with the \texttt{pluginhost=<IP or name>} parameter above define the
		exact service entry point.
		The default is to use the value from the \texttt{fdaddress=<IP or name>} parameter above.
		When neither is defined the default \texttt{9104} port number will be used.

		This parameter is optional.

	\item[\texttt{fdcertfile=<path>}] is the SSL certificate file location for
		the \kubernetes plugin endpoint service data connection for a Bacula Backup Proxy Pod.
		The certificate and key (see \texttt{fdkeyfile=<path>} below) files are required for
		proper Bacula Backup Proxy Pod operations. You can create and use any valid SSL
		certificate including a self-signed one. You can even just  use the certificate generated
	  during the \kubernetes plugin installation located at \texttt{/opt/bacula/etc/bacula-backup.cert}
		The default is to use the certificate generated during plugin installation.

		This parameter is optional.

	\item[\texttt{fdkeyfile=<path>}] is an SSL private key file location for the SSL
		certificate defined by \texttt{fdcertfile=<path>} above. An unencrypted private key
		must be used for this purpose. The default is to use a private key
		file created during plugin installation at \texttt{/opt/bacula/etc/bacula-backup.key}
		when \texttt{fdcertfile=<path>} above is not defined (the default). Otherwise the plugin
		will refer to the same certificate file from \texttt{fdcertfile=<path>} and use it as
		a \texttt{.pem} container.

		This parameter is optional.

	\label{baculaimageparam}
	\item[\texttt{baculaimage=<name>}] is a Bacula Backup Proxy Pod container
		image registry location for your cluster as described in the image installation chapter
		\vref{baculatarimage}. In most cases this parameter will consist of your
		Docker images registry location with the tag of the required image.
		The default for this parameter is \texttt{bacula-backup:<timestamp>} which
		may not match your cluster configuration.

		This parameter is optional.

	\item[\texttt{imagepullpolicy=<Always|Never|ifNotPresent>}] sets the Kubernetes
		Pod image pull policy during Bacula Backup Proxy Pod container deployment.
		You can configure Kubernetes to \texttt{Always} or \texttt{Never} pull the image
		from the repository, or pull it only when not available with \texttt{ifNotPresent}.
		The option value is case insensitive. If this parameter is not defined, the default
		\texttt{ifNotPresent} will be used.

		This parameter is optional.
\end{description}

\subsection{Restore Plugin Parameters}

During restore, the \kubernetes plugin will use the same parameters which were
set for the backup job and saved in the catalog. During restore, you may change any of the
parameters described in chapter \vref{genericparameters} and \vref{rbackupparameters}.
In addition to the options used for backups, the \texttt{outputformat} option can be
used during restore. This option specifies the file format when restoring to a local filesystem.
You can choose the restore output in \texttt{JSON} or \texttt{YAML}.
If not defined the restored files will be saved in \texttt{YAML}.

\begin{description}

\item[\texttt{outputformat: <JSON or YAML>}] specifies the
	file format when restoring to a local filesystem as described above.

	This parameter is optional.
\end{description}

\subsection{FileSet Examples}

In the example below, all Kubernetes Namespaces, Resources and Persistent
Volume Configurations will be backed up using the default Kubernetes API
access credentials.

\begin{verbatim}
FileSet {
  Name = FS_Kubernetes_All
  Include {
    Plugin = "kubernetes:"
  }
}
\end{verbatim}

In this example, we will backup a single Kubernetes Namespace using the Bearer
Token authorization method.

\begin{verbatim}
FileSet {
  Name = FS_Kubernetes_plugintest
  Include {
    Plugin = "kubernetes: host=http://10.0.0.1/k8s/clusters/test \
        token=kubeconfig-user:cbhssdxq8vv8hrcw8jdxs2 namespace=plugintest"
  }
}
\end{verbatim}

The same example as above, but with a Persistent Volume:

\begin{verbatim}
FileSet {
  Name = FS_Kubernetes_mcache1
  Include {
    Plugin = "kubernetes: host=http://10.0.0.1/k8s/clusters/test \
        token=kubeconfig-user:cbhssdxq8vv8hrcw8jdxs2 \
        namespace=plugintest persistentvolume=myvol"
  }
}
\end{verbatim}

This example backs up a single Namespace and all detected PVCs in
this Namespace using a defined listening and entry point address and the default connection port:

\begin{verbatim}
FileSet {
  Name = FS_Kubernetes_test_namespace
  Include {
    Plugin = "kubernetes: namespace=test pvcdata fdaddress=10.0.10.10"
  }
}
\end{verbatim}

The same example as above, but using different listening and entry point addresses as
may be found when the service is behind a firewall using port forwarding features:

\begin{verbatim}
FileSet {
  Name = FS_Kubernetes_test_namespace_through_firewall
  Include {
    Plugin = "kubernetes: namespace=test pvcdata=plugin-storage fdaddress=10.0.10.10 \
        pluginhost=backup.example.com pluginport=8080"
  }
}
\end{verbatim}

This example shows PVC Data archive backup with the Bacula File Daemon inside a
Kubernetes cluster:

\begin{verbatim}
FileSet {
  Name = FS_Kubernetes_incluster
  Include {
    Plugin = "kubernetes: incluster namespace=test pvcdata \
        pluginhost=backup.bacula.svc.cluster.local"
  }
}
\end{verbatim}

The configuration above is designed for use in situations where the Bacula server components
are located on-premise and behind a firewall with no external ports allowed in, but must back
up data on an external \kubernetes cluster.

\newpage
\section{Restore examples}

\subsection{Restore to \kubernetes Cluster}

To restore Kubernetes resources to a Kubernetes cluster, the administrator
should execute the restore command and specify the \texttt{where}
parameter as in this example:
\begin{verbatim}
* restore where=/
\end{verbatim}

\noindent and then set any other required restore plugin parameters
for the restore.

\begin{verbatim}
* restore where=/
...
$ cd /@kubernetes/namespaces/plugintest/configmaps/
cwd is: /@kubernetes/namespaces/plugintest/configmaps/
$ ls
plugintest-configmap.yaml
$ add *
1 file marked.
$ done
Bootstrap records written to /opt/bacula/working/bacula-devel-dir.restore.1.bsr

The Job will require the following (*=>InChanger):
Volume(s)                 Storage(s)                SD Device(s)
===========================================================================

Vol005                    File1                     FileChgr1

Volumes marked with "*" are in the Autochanger.


1 file selected to be restored.

Run Restore job
JobName:         RestoreFiles
Bootstrap:       /opt/bacula/working/bacula-devel-dir.restore.1.bsr
Where:           /
Replace:         Always
FileSet:         Full Set
Backup Client:   bacula-devel-fd
Restore Client:  bacula-devel-fd
Storage:         File1
When:            2019-09-30 12:39:13
Catalog:         MyCatalog
Priority:        10
Plugin Options:  *None*
OK to run? (yes/mod/no): mod
Parameters to modify:
	1: Level
	2: Storage
	3: Job
	4: FileSet
	5: Restore Client
	6: When
	7: Priority
	8: Bootstrap
	9: Where
	10: File Relocation
	11: Replace
	12: JobId
	13: Plugin Options
Select parameter to modify (1-13): 13
Automatically selected : kubernetes: config=/home/radekk/.kube/config
Plugin Restore Options
config:              radekk/.kube/config  (*None*)
host:                *None*               (*None*)
token:               *None*               (*None*)
username:            *None*               (*None*)
password:            *None*               (*None*)
verify_ssl:          *None*               (True)
ssl_ca_cert:         *None*               (*None*)
outputformat:        *None*               (RAW)
Use above plugin configuration? (yes/mod/no): mod
You have the following choices:
	1: config (K8S config file)
	2: host (K8S API server URL/Host)
	3: token (K8S Bearertoken)
	4: verify_ssl (K8S API server cert verification)
	5: ssl_ca_cert (Custom CA Certs file to use)
	6: outputformat (Output format when saving to file (JSON, YAML))
	7: fdaddress (The address for listen to incoming backup pod data)
	8: fdport (The port for opening socket for listen)
	9: pluginhost (The endpoint address for backup pod to connect)
	10: pluginport (The endpoint port to connect)
Select parameter to modify (1-8): 1
Please enter a value for config: /root/.kube/config
Plugin Restore Options
config:              /root/.kube/config   (*None*)
host:                *None*               (*None*)
token:               *None*               (*None*)
verify_ssl:          *None*               (True)
ssl_ca_cert:         *None*               (*None*)
outputformat:        *None*               (RAW)
fdaddress:           *None*               (*FDAddress*)
fdport:              *None*               (9104)
pluginhost:          *None*               (*FDAddress*)
pluginport:          *None*               (9104)
Use above plugin configuration? (yes/mod/no): yes
Job queued. JobId=1084
\end{verbatimx}

The plugin does not wait for Kubernetes Resources to become ready and online in the same way
as the \texttt{kubectl} or the \texttt{oc} commands.

\subsection{Restore to a Local Directory}
\label{localdirrestore}
It is possible to restore any Kubernetes Resource(s) to file without
loading them into a cluster. To do so, the \texttt{where} restore option should point
to the local directory:

\begin{verbatim}
* restore where=/tmp/bacula/restores
...
$ cd /@kubernetes/namespaces/
cwd is: /@kubernetes/namespaces/
$ ls
bacula/
cattle-system/
default/
graphite/
ingress/
plugintest/
$ add plugintest
25 files marked.
$ done
Bootstrap records written to /opt/bacula/working/bacula-devel-dir.restore.2.bsr

The Job will require the following (*=>InChanger):
Volume(s)                 Storage(s)                SD Device(s)
===========================================================================

Vol005                    File1                     FileChgr1

Volumes marked with "*" are in the Autochanger.


25 files selected to be restored.

Run Restore job
JobName:         RestoreFiles
Bootstrap:       /opt/bacula/working/bacula-devel-dir.restore.2.bsr
Where:           /tmp/bacula/restores
Replace:         Always
FileSet:         Full Set
Backup Client:   bacula-devel-fd
Restore Client:  bacula-devel-fd
Storage:         File1
When:            2019-09-30 12:58:16
Catalog:         MyCatalog
Priority:        10
Plugin Options:  *None*
OK to run? (yes/mod/no): mod
Parameters to modify:
	1: Level
	2: Storage
	3: Job
	4: FileSet
	5: Restore Client
	6: When
	7: Priority
	8: Bootstrap
	9: Where
	10: File Relocation
	11: Replace
	12: JobId
	13: Plugin Options
Select parameter to modify (1-13): 13
Automatically selected : kubernetes: config=/home/radekk/.kube/config debug=1
Plugin Restore Options
config:              *None*               (*None*)
host:                *None*               (*None*)
token:               *None*               (*None*)
verify_ssl:          *None*               (True)
ssl_ca_cert:         *None*               (*None*)
outputformat:        *None*               (RAW)
fdaddress:           *None*               (*FDAddress*)
fdport:              *None*               (9104)
pluginhost:          *None*               (*FDAddress*)
pluginport:          *None*               (9104)
Use above plugin configuration? (yes/mod/no): mod
You have the following choices:
	1: config (K8S config file)
	2: host (K8S API server URL/Host)
	3: token (K8S Bearertoken)
	4: verify_ssl (K8S API server cert verification)
	5: ssl_ca_cert (Custom CA Certs file to use)
	6: outputformat (Output format when saving to file (JSON, YAML))
	7: fdaddress (The address for listen to incoming backup pod data)
	8: fdport (The port for opening socket for listen)
	9: pluginhost (The endpoint address for backup pod to connect)
	10: pluginport (The endpoint port to connect)
Select parameter to modify (1-8): 8
Please enter a value for outputformat: JSON
Plugin Restore Options
config:              *None*               (*None*)
host:                *None*               (*None*)
token:               *None*               (*None*)
verify_ssl:          *None*               (True)
ssl_ca_cert:         *None*               (*None*)
outputformat:        *None*               (RAW)
fdaddress:           *None*               (*FDAddress*)
fdport:              *None*               (9104)
pluginhost:          *None*               (*FDAddress*)
pluginport:          JSON                 (9104)
Use above plugin configuration? (yes/mod/no): yes
Run Restore job
JobName:         RestoreFiles
Bootstrap:       /opt/bacula/working/bacula-devel-dir.restore.2.bsr
Where:           /tmp/bacula/restores
Replace:         Always
FileSet:         Full Set
Backup Client:   bacula-devel-fd
Restore Client:  bacula-devel-fd
Storage:         File1
When:            2019-09-30 12:58:16
Catalog:         MyCatalog
Priority:        10
Plugin Options:  User specified
OK to run? (yes/mod/no):
Job queued. JobId=1085
\end{verbatim}

Output format conversion at restore time will format all data in a human readable format.
You can find an example of this restore below.

\begin{verbatim}
# cat /tmp/bacula/restores/namespaces/plugintest/plugintest.json
{
	"apiVersion": "v1",
	"kind": "Namespace",
	"metadata": {
		"annotations": {
			"field.cattle.io/projectId": "c-hb9ls:p-bm6cw",
			"lifecycle.cattle.io/create.namespace-auth": "true"
		},
		"cluster_name": null,
		"creation_timestamp": "2019-09-25T16:31:03",
		"deletion_grace_period_seconds": null,
		"deletion_timestamp": null,
		"finalizers": [
		"controller.cattle.io/namespace-auth"
		],
		"generate_name": null,
		"generation": null,
		"initializers": null,
		"labels": {
			"field.cattle.io/projectId": "p-bm6cw"
		},
		"name": "plugintest",
		"namespace": null,
		"owner_references": null,
		"resource_version": "11622",
		"self_link": "/api/v1/namespaces/plugintest",
		"uid": "dd873930-dfb1-11e9-aad0-022014368e80"
	},
	"spec": {
		"finalizers": [
		"kubernetes"
		]
	},
	"status": {
		"phase": "Active"
	}
}
\end{verbatim}

The supported output transformations are: \texttt{JSON} and \texttt{YAML}.

\subsection{Restore PVC Data Archive}

Here we describe functionalities and requirements related to pvcdata restore.

\subsubsection{Local Directory Restore}

The procedure to restore a PVC Data archive file to a local directory is basically the same as restoring
the Kubernetes Resource configuration file as described in \vref{localdirrestore}. However,
output transformation is unavailable and ignored when restoring PVC data. Restore of this data will create a
\texttt{tar} archive file you can manually inspect and use.

\subsubsection{Restore to PVC}

This procedure is similar to the one described in PVC Data backup and uses the same Bacula Backup
Proxy Pod image. During restore, the plugin uses the same endpoint configuration parameters
so it is not necessary to setup it again. If your endpoint parameters have changed you
can update them using Bacula plugin restore options modification as in example below:

\begin{verbatim}
*restore select all done where=/
(...)
OK to run? (yes/mod/no): mod
Parameters to modify:
	1: Level
	2: Storage
	3: Job
	4: FileSet
	5: Restore Client
	6: When
	7: Priority
	8: Bootstrap
	9: Where
	10: File Relocation
	11: Replace
	12: JobId
	13: Plugin Options
Select parameter to modify (1-13): 13
Automatically selected : kubernetes: namespace=plugintest pvcdata pluginhost=example.com
Plugin Restore Options
config:              *None*               (*None*)
host:                *None*               (*None*)
token:               *None*               (*None*)
verify_ssl:          *None*               (True)
ssl_ca_cert:         *None*               (*None*)
outputformat:        *None*               (RAW)
fdaddress:           *None*               (*FDAddress*)
fdport:              *None*               (9104)
pluginhost:          *None*               (*FDAddress*)
pluginport:          *None*               (9104)
Use above plugin configuration? (yes/mod/no): mod
You have the following choices:
	1: config (K8S config file)
	2: host (K8S API server URL/Host)
	3: token (K8S Bearertoken)
	4: verify_ssl (K8S API server cert verification)
	5: ssl_ca_cert (Custom CA Certs file to use)
	6: outputformat (Output format when saving to file (JSON, YAML))
	7: fdaddress (The address for listen to incoming backup pod data)
	8: fdport (The port for opening socket for listen)
	9: pluginhost (The endpoint address for backup pod to connect)
	10: pluginport (The endpoint port to connect)
Select parameter to modify (1-10): 9
Please enter a value for pluginhost: newbackup.example.com
Plugin Restore Options
config:              *None*               (*None*)
host:                *None*               (*None*)
token:               *None*               (*None*)
verify_ssl:          *None*               (True)
ssl_ca_cert:         *None*               (*None*)
outputformat:        *None*               (RAW)
fdaddress:           *None*               (*FDAddress*)
fdport:              *None*               (9104)
pluginhost:          newbackup.example.com (*FDAddress*)
pluginport:          *None*               (9104)
Use above plugin configuration? (yes/mod/no): yes
\end{verbatim}

You can restore all data available from the backup archive for a selected Persistent Volume Claim
and all data will be overwritten, ignoring the \texttt{Replace} job parameter. Please take note of this
behavior, which may change in the future.


\subsection{Other}

\subsubsection{Resource listing}
\label{listing}

The Bacula Enterprise \kubernetes plugin supports the "plugin listing"
feature of Bacula Enterprise Edition 8.x or newer. This mode allows the
plugin to display some useful information about available Kubernetes
resources such as:

\begin{itemize}
	\item List of Kubernetes namespaces
	\item List of Kubernetes persistent volumes
	\item List of Kubernetes storage class resources
\end{itemize}

The feature uses the special \texttt{.ls} command with a
\texttt{plugin=<plugin>} parameter.

\bigskip
The command requires the following parameters to be set:
\begin{description}
	\item[\texttt{client=<client>}] A Bacula Client name with the \kubernetes plugin installed.
	\item[\texttt{plugin=<plugin>}] A plugin name, which would be
		\texttt{kubernetes:} in this case, with optional plugin parameters
		as described in section~\vref{genericparameters}.
	\item[\texttt{path=<path>}] An object path to display.
\end{description}

\bigskip
The supported values for a \texttt{path=<path>} parameter are:
\begin{description}
	\item[\texttt{/}] to display Object types available to list
	\item[\texttt{namespaces}] to display a list of Namespaces
	\item[\texttt{persistentvolumes}] to display a list of Persistent Volumes
	\item[\texttt{storageclass}] to display a list of Storage Class Resources
	\item[\texttt{namespaces/<name>/pvcdata}] to display all available Persistent
		Volume Claims available in Namespace \texttt{<name>}
\end{description}

\bigskip
To display available Object types, follow the following command
example:

\begin{verbatim}
*.ls plugin=kubernetes: client=kubernetes-fd path=/
Connecting to Client kubernetes-fd at localhost:9102
drwxr-x---   1 root     root 2018-09-28 14:32:20  /namespaces
drwxr-x---   1 root     root 2018-09-28 14:32:20  /persistentvolumes
drwxr-x---   1 root     root 2018-09-28 14:32:20  /storageclass
2000 OK estimate files=2 bytes=0
\end{verbatim}

To display the list of all available Kubernetes namespaces, the following command
example can be used:

\begin{verbatim}
*.ls plugin=kubernetes: client=kubernetes-fd path=namespaces
Connecting to Client kubernetes-fd at localhost:9102
drwxr-xr-x   1 root     root 2019-09-25 16:39:56  /namespaces/default
drwxr-xr-x   1 root     root 2019-09-25 16:39:56  /namespaces/kube-public
drwxr-xr-x   1 root     root 2019-09-25 16:39:56  /namespaces/kube-system
drwxr-xr-x   1 root     root 2019-09-25 16:46:19  /namespaces/cattle-system
drwxr-xr-x   1 root     root 2019-09-27 13:04:01  /namespaces/plugintest
2000 OK estimate files=5 bytes=0
\end{verbatim}

To display the list of available Persistent Volume Claims which could be used for PVC Data
archive feature selection, you can use the following example command for the \texttt{mysql}
namespace:

\begin{verbatim}
*.ls client=bacula-devel-fd plugin="kubernetes:" path=/namespaces/mysql/pvcdata
Connecting to Client kubernetes-fd at localhost:9102
-rw-r-----   1 root     root 2019-10-16 14:29:38  /namespaces/mysql/pvcdata/mysql-mysql
2000 OK estimate files=1 bytes=0
\end{verbatim}

To display the list of all available Persistent Volumes, the following command
example can be used:

\begin{verbatim}
*.ls plugin=kubernetes: client=kubernetes-fd path=persistentvolumes
Connecting to Client kubernetes-fd at localhost:9102
-rw-r-----  1073741824 2019-09-25 /persistentvolumes/pvc-bfaebd0d-dfad-11e9-a2cc-42010a8e0174
-rw-r-----  1073741824 2019-09-25 /persistentvolumes/pvc-b1a49497-dfad-11e9-a2cc-42010a8e0174
-rw-r-----  1073741824 2019-09-25 /persistentvolumes/pvc-949cb638-dfad-11e9-a2cc-42010a8e0174
-rw-r-----  1073741824 2019-09-25 /persistentvolumes/pvc-9313388c-dfad-11e9-a2cc-42010a8e0174
-rw-r----- 10737418240 2019-09-24 /persistentvolumes/myvolume
2000 OK estimate files=5 bytes=15,032,385,536
\end{verbatim}

The volume lists display a Volume Storage size which does not reflect
the actual configuration size during backup.

To display the list of all defined Storage Class Resources, the following command
example can be used:

\begin{verbatim}
*.ls plugin=kubernetes: client=kubernetes-fd path=storageclass
Connecting to Client kubernetes-fd at localhost:9102
-rw-r----- 1024 2020-07-27 13:39:48  /storageclass/local-storage
-rw-r----- 1024 2020-07-23 16:14:13  /storageclass/default-postgresql-1
-rw-r----- 1024 2020-07-24 11:47:02  /storageclass/local-storage-default
-rw-r----- 1024 2020-07-23 12:00:02  /storageclass/standard
2000 OK estimate files=4 bytes=4,096
\end{verbatim}


% disabled until prepared
%\subsection{Best practice}

\subsection{Advanced Bacula Backup Proxy Pod Deployment}
\label{advancedpoddeployment}

\textit{WARNING: This is an advanced topic related to Kubernetes clusters. You should !!NOT!!
try to implement or customize the Bacula \kubernetes plugin behavior unless you REALLY
know what you are doing.}
\bigskip

You can customize the service parameters used for deploying Bacula backup Pods
dedicated to Persistent Volume Claim data backups to suit your needs. The plugin uses
the following Pod service deployment YAML template to execute the proxy operation
pod on the cluster.

\begin{verbatim}
apiVersion: v1
kind: Pod
metadata:
  name: {podname}
  namespace: {namespace}
  labels:
    app: {podname}
spec:
  hostname: {podname}
  {nodenameparam}
  containers:
  - name: {podname}
    resources:
      limits:
        cpu: "1"
        memory: "64Mi"
      requests:
        cpu: "100m"
        memory: "16Mi"
    image: {image}
    env:
    - name: PLUGINMODE
      value: "{mode}"
    - name: PLUGINHOST
      value: "{host}"
    - name: PLUGINPORT
      value: "{port}"
    - name: PLUGINTOKEN
      value: "{token}"
    imagePullPolicy: {imagepullpolicy}
    volumeMounts:
      - name: {podname}-storage
        mountPath: /{mode}
  restartPolicy: Never
  volumes:
    - name: {podname}-storage
      persistentVolumeClaim:
        claimName: {pvcname}
\end{verbatim}

The above template uses a number of predefined placeholders which will be replaced
by corresponding variables during Pod execution preparation. To customize proxy
Pod deployment you can change or tune template variables or the template body.
Below is a list of all supported variables with short descriptions and requirement
conditions.

\begin{description}
	\item[\texttt{podname}] - This is the predefined Pod name used by a plugin.
		This variable is required and cannot be customized.
	\item[\texttt{namespace}] - This is a Namespace name for which the PVC
		Data backup is performed.
		This variable is required and cannot be customized.
	\item[\texttt{nodenameparam}] - This is a placeholder for Cluster node name 
		parameter (\texttt{nodeName: \ldots}) used to mount an existing Volume Claim 
		for backup or restore if required for the selected PVC. In most cases this variable is 
		required and cannot be customized.
	\item[\texttt{image}] - This is a Pod container image name to execute.
		You can customize or omit this variable as long as you provide
		a container image name required by the cluster.
	\item[\texttt{mode}] - This is an operation mode for the Proxy Pod.
		The supported values are: \texttt{backup} and \texttt{restore}.
		This variable is required and cannot be customized.
	\item[\texttt{host}] - This is an endpoint address which corresponds to the
	\texttt{pluginport=\ldots} Kubernetes plugin parameter.
		This variable is required. You can customize or omit this variable as long
		as you provide a value for the \texttt{PLUGINHOST} container environment.
	\item[\texttt{port}] - This is an endpoint address which corresponds to the
	\texttt{pluginport=\ldots} Kubernetes plugin parameter.
		This variable is required. You can customize or omit this variable as long
		as you provide a value for the \texttt{PLUGINPORT} container environment.
	\item[\texttt{token}] - This is an Authorization Token (randomly generated).
		This variable is required and cannot be customized.
	\item[\texttt{pvcname}] - This is the name of the PVC for backup or restore
		operations.
		This variable is required and cannot be customized.
\end{description}
You can create the required file: \texttt{/opt/bacula/scripts/bacula-backup.yaml} or point
to the custom one using \texttt{\$DEFAULTPODYAML} environment variable.

\subsection{Limitations}

\begin{bitemize}
	\item Only full level backups are possible. This is a Kubernetes limitation.
	\item You can perform a single PVC Data backup or restore with a single Bacula File
		Daemon installation associated with single \texttt{fdaddress=<name>}.
		This limitation may be removed in a future release of the \kubernetes plugin.
\end{bitemize}

\subsection{Common Problems}

In this section we describe the common problems you may encounter when you first deploy the
Bacula \kubernetes plugin or when you are not very familiar with this plugin.

\begin{bitemize}
	\item You have set the \texttt{incluster} parameter in your Job but you have the following error:
\begin{verbatim}
Error: kubernetes: incluster error: Service host/port is not set.
\end{verbatim}
	This means you are running the Bacula File Daemon and \kubernetes plugin not in a Pod, or
	Kubernetes does not provide default service access in your installation.
	\smallskip
	In the latter case you should use a standard Kubernetes access method in a prepared kubeconfig file.
\end{bitemize}
