[Music]
good afternoon and everybody welcome
back to the final session for the day
and the final session for lca in this
room uh we have yemel joining us from
sunny london town where it's a beautiful
5 degrees outside and 6 45 a.m so all of
us who are having coffee to get through
the rest of the day yaml's probably
starting his to get him going
so yaml is a long-term
debian developer working from london but
working out of norway so i feel his
sympathies with him working for an
international company the clock is
always wrong no matter where you are in
the world somewhere that you're working
with and having a meeting with so yemel
over to you and looking forward to hear
all about debbie and janitor
all right uh yeah thanks uh thanks
michael um yeah so today i'm going to
talk about various ways of automating
debian packaging and this is a
continuation of work that i've been
doing for the last couple of years
um
and what i'm going to talk about today
um
is something that's part of the the
debian projects
but a lot of the lessons and the tools
apply outside of debian as well to other
distributions but also to to upstream
projects
um so let me start by
giving a little bit of an introduction
um to to debian uh you're probably
familiar with it um but maybe it's good
to to sort of uh
take a closer look at actually what what
that distribution is trying to achieve
um so linux distributions like debian
fulfill an important function in the fos
ecosystem uh there are system
integrators that take existing free and
open source software projects
and adapt them where necessary to work
well together
they also make it possible for users to
install more software in an easy and
consistent way
and with some degree of quality control
and review
and of course you could you could argue
over how much review and quality control
happens for some packages in debian
but at least there is there's some
minimal bar
so packages have been reviewed for their
license they have to meet certain parts
of the
um
of debian's rules debian's policy
and
there are mechanisms
to encourage conformance where the
policy is not
absolute
and in the same way
debian makes some promises around how
long security updates are provided for
example
now if you're installing software
directly from an upstream project
um
you would only be able to um
get these sorts of guarantees for a
certain number of upstream projects uh
but you wouldn't be able to get that
that consistency across across all of
the things you installed
um and there is some variety between
linux distributions so that being falls
squarely on the side of trying to
integrate the different upstream
projects more tightly
and generally has much more complicated
patches and and packaging and other
distributions fall bore on the other
side and they have much smaller
uh shim around the the upstream project
like arch for example
um
where um the packaging is is much
smaller it's it's much easier to make
packages um but there's also uh less
integration work that has happened
now if you've probably heard of debian
even if you haven't used it debian is
one of the older and larger
distributions
debian has about a thousand uploading
developers and many more people who
contribute changes on a regular basis
there are about 30 000 source packages
in debian and that basically means that
there are about 30 000 upstream projects
that get integrated into the
distribution
um
so that's that's quite a lot um and
there are many more
binary packages that get created from
those 30 files and source packages
and then debian has a large policy that
specifies what packages binary packages
in particular
should look like
uh and and comply with
uh the development of the policy is is
mostly consensus driven
uh but the policy also tries to be uh
generally flexible about how um it is
met so
maybe the shape of a binary package is
is
particularly well documented
you can use many different
tools on the build sites
to achieve that so there's no
um
it's not very well dictated exactly how
you should do that implementation
and then of course there are many many
tools available that can
can can do the same thing
um
and then each package has a single owner
so called maintainer
and with team ownership uh
now being more common than individual
maintainership so
generally there's a set of people who
maintain
a usually a large number of packages
and keep those packages in a good shape
that's different from some other
distributions where the ownership model
for each package is often a lot weaker
and where it's um
more common for other contributors to to
step in
it's it's not hard and fast in debian
either but
traditionally it's been very
much that there was
a maintainer who took care of a package
and and nobody else would
would really touch it
um
and then
source packages in debian are
essentially just turbos with some some
debian specific metadata
and uploading a new version of a package
generally involves copying some pgp
signed turbos around
um
so that that's um how debian um
started and and uh
how the basics work um but
things have moved on since the
distribution was originally created
um
when debian started store starburst with
a way of delivering software either on a
cd or a floppy or maybe even over this
this shiny internet thing
and that was true for for both upstream
projects as well as debian itself
nowadays git is fairly ubiquitous and
most upstream projects and most packages
in debian use it
and there are still exceptions of course
that don't use any version control
system at all
or maybe that use a different version
control system
but those exceptions are are getting
more and more rare
as a as a version control geek i'm i'm a
little bit sad that
we have this this monoculture now uh but
git is a really great tool and it's it's
nice to not have to deal with half a
dozen different systems like we had
like we had to back in 2009
so today in debian there isn't really a
predictable location for where git
repositories with debian packages live
however we've worked around that by
adding a header to the debian control
file in the source package that points
at the git repository
or technically
could also be the mercurial
repository location the bizarre branch
location the subversion repository
location the arch location cvs location
or the monoto monotone repo location
but as you can see from the the graph on
the right
um
basically all of the packages nowadays
use git
um and there are some holdouts that you
don't use any version control system at
all
um these headers have been around for
quite some time
um
but they're awesome and there are tools
around them to automatically
for example
check out
the git repository for a package
and there's a service that will monitor
all of the packages that have these
header sets
and that will notify you whenever for
example a repository goes away or when
it breaks or when it doesn't match
what's in the archive
and from for most packages this this
header is pretty accurate which is great
right
and then there's another control file in
debian packages
um
called debian upstream metadata and
that's a standard that defines um
a format based on yaml that allows the
maintainer to specify
various bits of metadata about the
packages upstream project
so for example where does the upstream's
documentation live
where is its wiki where can you report
bugs
but also
where does the upstream project's
version control system live
there's a little bit of
overlap here with other control fields
in debian
but that's something that's being worked
on
um so this format is a draft but it's
been widely adopted um there's about 10
000 packages or so that have this this
file present
and about 8 000 of those
have the repository header set so for
about a third of all of the
30 000 source packages in debian
we know where we can find the upstream
source repository
and if you're a maintainer you can you
can use the
guest upstream metadata command to
automatically
generate one of these files and that
command will actually
do things like introspect setup.pre-wi
if you're a python package
or look at the make file or in in some
cases it'll even just
um scrape the readme file for anything
that looks like a git repository and and
it tries to see whether it matches
what's in the package
another innovation that's happened uh in
the last couple of years is auto package
test um so other package tests defines a
framework for running tests um
on the debian binary package
um
so uh when you run them you can make you
can be sure that the package has been
built correctly
and functions as a user would expect
um
so about half of the archive now um
has sd set um
and that is great because it allows you
to
verify when you make a change
that you haven't broken anything
significant
there's also a service ci debian.net
that regularly runs these packages runs
these tests against packages
as well as their dependencies uh
whenever one of them changes
and then finally
uh one of the uh the recent developments
or um
or rather one of the one of the things
that has been evolving is um
a gradual adoption of of just a single
tool for building packages um so in the
past there were many different ways of
building a debian package
you could pick a random package from the
archive
and depending on the maintainer's
preference you would find one of
about five different tools
um and you would have to be familiar
with all five different tools if you
wanted to
contribute to a random package
um
one of the uh or the tool that we've
sort of settled on is
called deb helper
and the dev helper dh interface in
particular one of the other things
that's nice about the dh interface is
that
deb helper can increasingly figure out
how to just do the right thing rather
than requiring a lot of configuration
so that makes the packaging less effort
but it also means that it's less likely
that for example when you import a new
upstream version
um that you will need to make extensive
changes to the packaging
so that's sort of
a lot of background on on what has
happened in
the debian ecosystem over the last
um
maybe five to ten years
um that has made
it's
has improved the um
the way we can make automated changes
um
so let's now have a look at making
large-scale changes in the debian
ecosystem because there are some unique
challenges because
every package is basically maintained as
a
project by itself
um so if you're ever if you've ever
touched the debian package you're
probably familiar with linden so linting
is a it's a linting tool that can scan
debian packages
and report common packaging issues
so it categorizes issues by so-called
tags
and
you see the documentation for one of
those stacks here
so this stack in particular uh carriage
return line feed
gets reported whenever
a debian control file contains a
turned characters
or sorry carries turn line feed
characters and
um
suggests that maintainers basically
strip the um
the line feeds from there uh many tools
in debian will actually many build tools
and debian will actually run lynchian
straight after building
um and the hope is that like if we
report these
tags uh that maintainers will go and and
fix them
um and so the traditional way of making
changes across debian packages is to
somehow make developers aware of an
issue
for example through lynchian tag or bug
reports
or maybe some other form of
communication
and then you get everybody to pick up on
the problem and fix it
so you get them to care and make the
appropriate changes and upload new
packages to the archive
and then after you've done that and
you've waited
enough years
you may end up with a significant chunk
of the archive fixed
um
generally though there will be a long
tail of packages that aren't updated um
there are a bunch of packages in debian
that haven't been touched in years like
maybe they've been rebuilt
but if they um if they've just just kept
building um then there's there's
generally no need to change them
um
but as you can see this is a very slow
process it also requires a lot of um
cumulative time across all debian
developers so not only do they all need
to be aware of the details of the
problem they also only to allocate time
to
um
to actually uh make the the changes and
to upload a new package
and frankly for for a lot of problems
that that's probably not worth the
effort
um so let's let's go back to that
lynchian output
um
you you'll notice that lynching actually
tells you the command to run to fix the
issue
um
so
what if rather than requiring the user
to run lynchian and then copy and paste
the commands
we just render command for them
so that's that's basically the idea
behind a tool called lynching brush
so lynchian brush is a is a set of
more than 100 different scripts that can
automatically fix common linkedin issues
there's also a framework that allows you
to write those scripts easily and to run
them so the framework provides all of
the interaction
with the git repository for you so you
don't have to worry about that
it'll also take care of updating the
debian change log file for you
if the package doesn't use
fancy tools for for generating the
change lock
and it's got all kinds of cleverness
underneath so
um
when you run the script it uses inotify
to figure out when you make changes to
the package it manages to get adds and
removes
it creates the commits if there are
multiple
scripts that run
it cleans up if this if a script fails
to run
and then as mentioned it will update the
changelog file appropriately
um most of the things of the fixtures do
things that would take developers a
couple of minutes each
and this this basically just makes it
at most a couple of seconds
per package
this kind of automation doesn't really
make that much of a difference
like
you take it down from maybe a couple of
minutes to a couple of seconds
um but you can just fire and forget and
you can also
if you look at the entire archive the
the impact is more significant
and if there's nothing for the tool to
do
it'll it'll usually
run
in less than a second
um and this is a an example for what one
of those scripts looks like
uh so this is writing in in python
most of the heavy lifting is done by the
framework
but in this case
it's it's fixing um a up a particular
field in the control file
where
previously debian supported a priority
extra and that's been removed and should
now be replaced with priority optional
um
and
the way that the framework is written
makes sure that
if there are um
formatting
specialties um they are preserved
and if the script can can preserve
the previous formatting it'll just raise
an exception
that exception isn't handled here the
script will fill
and
its changes will be discarded
however um
that
having one of those tools and um
available
doesn't mean that everybody
automatically starts running it
so this is a graph of the total number
of installations of linton brush from
debian's popularity contest program
lynching brush has been around since mid
2018.
um ideally we'd like to see it run
um over all 30 000 packages in debian
on a regular basis
um
but that's
not something that can easily happen
um and so one of the things that
we would really like to do is
um
to run it across the archive without
having to
get every single developer to every
month run it
on each one of their packages
and
the way to do that is
to
basically bring the changes to them
rather than requiring them to to run the
tool
um so we make it easy for people to
apply these changes by
creating merge proposals with the
changes
and there's a tool called silver bladder
that automates all of that so it
iterates over a set of git repositories
um
it applies a recipe which is generally
just a command to run
uh in the git repository
um and then if there were changes
created by the recipe it proposes them
in a pull request
and so this means that
a single person can come up with a
recipe
can run it against all of the archive
and the only thing that maintainers have
to do is
look at the merge request and hit the
merge button if they think the changes
are reasonable
and the tool can also directly push to a
git repository so if the user running it
has appropriate permissions
it can just directly push to the
repository
and then this is what it concretely
looks like
um so there's a yaml file with um
the urls for the repositories to process
and those can also be just names of
debian packages
in which case um
the tool introspects the the headers
that i mentioned earlier
uh and then there's a recipe
which basically specifies what command
to run
um and a template for the pull request
that needs to be created
um and then uh you can you can invoke
the commands uh on those two files
um ideally starting off with
a dry run to see if the changes are
reasonable
and then you can run it
and it just iterates over all of the
repositories
it can also
build the debian package
to verify that you haven't accidentally
broken anything
um
so that's all well and good but debian
has about 30 000 source packages
and um using a command line tool to dry
run review and upload 30 000
uh changes is gonna be quite challenging
um
so um
there's another project called the
debian janitor which sits atop
silver platter
and provides a web interface
so it
looks at all of the packages in debian
and regularly processes them
to um
make sure that's
linked your brush for example but other
tools as well get run regularly
so the debian janitor is basically it's
a website
uh as well as a uh
a scheduling interface
that constantly watches the debian
archive
processes the entire archive regularly
as well as
packages that have been changed recently
and an that they should in addition to
that
um it provides an interface where you
can you can actually look at these
changes
um and you can do um
qa and review
uh as well as um
managing all of the merge requests that
are open
um
and one of the the more challenging
parts of it is
um
how to do this in such a way
that it delights users rather than um
annoying them so
there are some risks here
um
the goal behind all of these tools is to
save developers time and effort by
automating away things that can be done
by a robot so that
the the developers can focus on things
that require human judgment
um but there's no time saved and in fact
quite the opposite
if if the pull request is incorrect
so
the developer ends up spending time
looking at the merge request
and the package in the end isn't
improved
oh and then
when the
developer gets another merge request
they're probably gonna be less likely to
review it
um and they're they're also gonna be
less likely to
just give the bots blanket access to the
repository
um so therefore the the actions of the
bot are restricted to a very limited set
of problems
for which there is an obviously correct
action that can be taken
and it's not meant to automate all
packaging or even to cover
automating all instances of the issues
that it knows about
um so so basically
if if it's unsure about something it
just does nothing
there's nothing really lost there
some packages use very complicated
packaging methods that lintone brush
just doesn't understand
um for example they're control files
that are generated in complicated ways
um
i'm looking at you people who use m4 to
generate your control files
and lynching brush just ignores those
and then
when we process
the archive with with the debian janitor
we only allow
changes that we have a very high
certainty about
after changes have been made we do a
builds of the package
uh with and without the changes no
matter how trivial those changes were
uh if that fails for whatever reason we
discard the changes
um
then we run the the other package test
as well if one of those fills we discard
the changes
and then finally there's a qa step where
a human looks at the source package diff
and sometimes that also
helps us catch issues
um and then finally we we only open a
single merge request per tool per
package
so if we process the package again
we'll just update the existing merge
request
rather than creating a new one and
having you look at that again
um and then per maintainer there is a
rate limits
on the number of open merch requests so
we don't really want to
spam you with um
100 merge requests if you have a hundred
packages that you maintain
so what we do is
um
basically tcp slow start so
we'll send you one merge request if you
ignore that
we won't send you any more
um if you merge it we'll send you two
more
if you merge those we'll send you four
etc until basically until we reach a
limit of 20.
um
and then in addition to that
uh the bot
tries very hard to identify itself as a
bot so that you know that the merge
request is automated
um
and can prioritize
its merge requests appropriately
but otherwise you can just interact with
it
as you would with a
merger request that came from a human
so basically if you comment on one of
the merge requests
one of the developers from the janitor
will look at
your comments
so
um yeah then how much does it take to
keep keep this beast running
one of the key things
that we look at regularly is the
rejections of merge proposals so if
people close merge proposals without
merging them
they'll usually leave a comment
and and we go and we look at why they
close those
so in some cases they've already made
the change themselves
um or maybe they introduced the merge
conflict before they noticed
the pull request for the package
in other cases um sometimes they
disagree with the change
um so sometimes we
end up discussing the chains with them
um and occasionally that leads to
bug reports for one of the tools
involved in the interchange
um
if we find a bug that might affect other
pull requests as well
we go and proactively find all of those
pool requests
and we reprocess them
and then if we if we can't fix things
immediately
uh we'll just disable the the tools in
question
um until we we fix the issue so that
um there are no developers that end up
looking at changes that we know are
wrong
um
and um of course reviewing all of these
changes takes quite a bit of time as
well
so uh every single change is still
reviewed by uh one of the the janitor
developers before it goes
out in a pull request
uh but that's something that i'm hoping
to
automate away in the near near term
future
so at the moment this is like hundreds
of diffs a day
and that's that's quite a lot of manual
effort
um
so that's that's what's been happening
for uh about a year and a half
um
what have we achieved so far
so
unfortunately we're in the winter
process of
uh gluing the history from our old and
new database together
so i've only really got good graphs from
up until last decembe last september
but this is the graph with the number of
changes we've
directly pushed to packaging
repositories up until that point
so this is for repositories where the
maintainer has given right access to the
bot
and as you can see we've almost hit the
10 000 gauges mark back in september and
in fact uh we've we've since gone over
that
um and then there's the number of uh
merger
merge proposals or pool requests
depending on the terminology you want to
use
so we've had about 7 000 of our merchant
fossils merged so far
and we've got about a thousand more open
at the moment
[Music]
we have
many more changes queued up to propose
uh but because of the restrictions that
i talked about earlier
uh we haven't created those pull
requests yet so that's that's all
because
um
we we will rate limit the number of pull
requests that's open for an individual
maintainer
um
there's some interesting artifacts on
this graph
that's mostly related to
either the tooling being paused when we
discovered bugs
um or related to the
freeze for the debian release early last
year
where we stopped
submitting pull requests
um and then this is how the um
the merge proposals are distributed over
the different hosting sites
um so it's in the earlier snapshot so
the numbers don't quite add up to those
in the previous graph
but as you can tell um all of the
changes happen
or most of the changes happened almost
exclusively on uh salsa on debian's
own instance of gitlab
and the rejection rate is is fairly uh
low so
um close basically means
um that the maintainer closed
the the merge request and didn't accept
it
um
in any way
um
there are a couple of
challenges still associated with this
um so
if you get your changes accepted um and
merged into a git repository that
doesn't mean that they end up in the
archive um so
um
it might take uh years before the
changes that were merged into a git
repository
are actually part of an upload to the
debian archive
um
and um last week we checked about 52
of changes haven't actually been
uploaded yet
um there is a some work on going to make
it easier for
people to upload changes that are in a
git repository to the archive
uh but those haven't haven't learned yet
um and then there's also some
challenging some challenges around
the the long tail packages that haven't
switched over to a version control
system yet
or that still claim that they are hosted
on um elif which is
the previous hosting platform that
debian used
and that was deprecated
i think over four years ago
um so maybe uh we should consider
alternatives like
uh maybe not creating pull requests but
maybe sending um
bug reports with with diffs attached
and then finally um
one of the challenges that we have with
um with gitlab is the fact that
uh for a lot of repositories there's
nobody who gets notified when there are
pull requests so the pull requests just
sit there
and that's not because people don't care
it's just because they don't know about
the pull request
so i talked about lintium brush earlier
that fixes um
smaller issues in debian packages but
there are some very other some other
very common operations that packagers do
that could benefit from automation
and with the recent improvements to the
egg system that i was talking about
earlier
it actually becomes feasible in a lot of
situations to update the debian package
to a new upstream version automatically
of course that requires that
the
packaging gets repository is available
and that the upstream git repository is
available
but for a lot of packages we actually
have that information
um and um
if we can make this work for some
fraction of packages even if it's like
ten percent
uh that could still be very variable
because it's about um three thousand
packages
um
yeah and one of the consequences of
debian's model is that uh the
distribution packages often lag behind
upstream releases
um so that's that's especially true for
um distributions where
uh it takes more work to integrate a new
release
um
and and there are uh
users who
would prefer
more of a rolling distribution
um
so
yeah what does that the process look
like for updating a package to a new
version today
um there's a there's a lot of individual
uh steps that a maintainer needs to take
and um
some of these steps are now somewhat
automated um but um
until recently there wasn't really a
single command that could replace all of
these
um as part of the the janitor work we've
developed a new command
that you can basically just run in any
debian git repository and they will
attempt to pull in
a new upstream release or a new upstream
git snapshot
and it takes care of all of the
work that needs to happen along with
that such as updating the
changelog to some extent updating the
packaging as well
running building and run and running the
test for the package
etc
this is uh all of the steps that it goes
through i won't uh go into detail
but i'm happy to
talk about this afterwards
um
so this this importing of new upstream
releases has been happening for
uh the last year and a bit
um
and we've attempted to import a new
upstream git snapshot and a new upstream
release for every single package in the
archive
where the required information was
available
and
this is
what we
what we ended up doing
um so for
about 20 of packages we managed to
successfully import a new release
and that's about 2200 packages which i'm
pretty happy with
um there's also a whole bunch of
packages that were already up to date
many more than i expected so about 35
was already at the latest release
and then if you look at the failures for
the other packages
there are a variety of reasons so some
of them are just random build failures
which is to be expected
some are related to
us not being able to find the
new upstream release
because we don't we don't know where
it's hosted
um
some are due to just
uh errors importing tarballs
um and in a lot of cases the debian
specific patches
no longer apply when we update the
upstream release
and then similarly if we look at
importing git snapshots
you can see that um
we um
are successful uh about 20 of the time
as well
um but this accounts for about 5500
packages
um
and um
in a lot of cases we don't actually know
where the uh upstream repository lives
and then some of the other errors are
very similar to
those when we were importing um releases
so
merge conflicts
or patches that no longer apply
or tireballs that don't import
um and the
packages that get built as part of
this process are actually available
um so if you'd like a debian release
with
very experimental packages with
newly imported upstreams
you can go to that url and find the
instructions on
how to
add those app repositories
um and then yeah what's what's next
um so there are a bunch of improvements
that we still need to make
um in order to
get more of the um
upstream imports to succeed
um and then
in the future we'd also like to look at
automating backboards so
in particular automating the cherry
picking of security fixes to stable
packages
as well as automating back porting of
unstable packages to
a stable
and there's a whole bunch of work that
needs to happen for
[Music]
debian patches to be updated
appropriately
there's some harder problems that we
haven't really looked at such as
transitions so packages that need to be
updated in lockstep for example
and and one of the things that
i would like to look at is
how we can apply some of these tools
outside of debian so there are some
existing projects that do things like
modify repositories to for example
reformat things
but that generally happens on a
[Music]
uh
as part of um existing changes uh rather
than proactively
um
github has a uh bots called the
pandabots that proactively submits
changes to um repositories
but it's proprietary and it's specific
to github
and there are some unique challenges if
we start doing this outside of debian
such as who would we which project will
we do it for um
what are the standards that we should
try and um aim for
um
and
how actually do we build a random
upstream project that we find somewhere
on getup
all right um that's that's all i wanted
to talk about today uh thank you very
much um for listening
um and also uh thanks for
thanks to um all of the people in debian
who've sort of worked on
the infrastructure that made this sort
of work possible
um
so we just bare bare turbos um none of
this would have been possible
um
from
metadata and and standardization
for the repository fields to
the uh ultimate debian database
lincion the ci systems as well as
jenkins which runs all of the workers
for the gender
and this is where you can find more
information
about the projects
thank you very much
fell for it again my apologies so yelmir
thank you so much for your talk um and
teaching us more about debian which i
didn't know a lot about and let alone
the janitor heard of merging i'm really
new to this world so thank you so much
uh we've really only got a few seconds
left in this room um the conference is
not over but this stream is over in this
room so if every quality please now move
back to the main room which is the ko
room and we'll have the conference close
out she'll be wrapping up here in a few
seconds and we'll over to the k a room
back in the main portal thanks everybody
for joining lca have a fantastic rest of
your sunday
coming up in london and for those of us
who are ending our sunday may your week
go well thanks for time and putting up
with my oopses of being on mute when i
shouldn't have been bye everyone