[Music]
good afternoon welcome back
um
russell curry is a linux and open source
hacker and microsoft powerpoint 2003
aficionado um who works at ibm oz labs
uh on the
power platform focusing on linux kernel
hardening and continuous integration uh
he'll be presenting to us today on
using or abusing github actions for
building and testing kernels thank you
russell
thanks a lot andrew uh it's good to be
back
at the colonel miniconf um
i apologize
slightly for the eye searing madness you
are currently witnessing on your screens
and i'm not just referring to my face
in this talk i'm going to talk about
github actions for for kernel
development and what's why that might be
relevant to you um you might be
wondering why should i care about a
proprietary
centralized uh
vendor lock-in-ish platform
that
we don't even use
for kernel development you might also be
wondering uh why you're looking at word
art
i hope to answer one of these questions
appropriately by the end of the talk so
so let's get into it let's go over
what github actions is and then we can
look at why it might be a value to us
github actions is a continuous
integration
service provided by
by github
if you think about how a typical
software project works on github if it
uses github for development you know
people send pull requests and when they
send the pull request that code will get
built and tested and
if there's something wrong with it it'll
flag that on the pull request there'll
be like an inline if it can narrow down
the problem to a specific line of code
it'll like be there visually in the
code review
stuff
for release management and builds and
publishing you can use github actions if
you maintain software packages to build
for a variety of different platforms and
and publish that to
docker hub or whatever um we don't care
about any of that uh
that's not what we're here for um
there's one main reason why i think
github actions is quite valuable to us
in the kernel development world and it
can be summarized in in three magic
words and those three magic words are
unlimited free compute now
i'm not sure about you but when i see
these three words uh my brain produces
the the happy chemicals my brain gets
excited when i see this i start thinking
of the many possibilities i could take
advantage of with with this resource
and so it doesn't really matter well it
doesn't matter but it's not a deal
breaker that the platform is is
proprietary and centralized when we can
use it for
unlimited free compute to make our
kernel development better so that's what
i'm going to get into today
uh so to explain a little bit more about
why that is
github makes the service available for
free to public repositories
not strictly open source because you can
have a public repository with a
proprietary license and that would still
have this but you know the idea from
github is that this is a give back to
the open source community and then you
know
you might use this for your personal
project think it's good and then
convince your employer to pay for it and
so i guess that's the
the business model um so let's get into
what it can do
and and what it can do for us so if you
think about what it can do
and how that might be relevant to the
kernel well the answer is you can do it
kind of do anything it's just a server
you get given a server on which you can
execute
whatever you want um
except they don't have nested kvm so
that's somewhat of a limiter but aside
from that anything you can do within a
virtual machine running linux you can do
the the the limitation is that you're
running inside a relatively crummy vm
that you know you're not going to be
able to make that j200 and have it
complete in a minute
you're going to be constrained by that
but there's still a lot you can get done
within those constraints
and that can definitely be a value so
let's let's get a bit more into it let's
build a kernel
the bare minimum requisite to get some
value out of the service is to have it
build kernels for us so let's figure out
what we need to be able to do that
so
you need a github repository so you need
a github account you need a reaper you
may need to go into the settings of that
repo and manually enable actions if you
haven't done so before you need the
source tree
you could
within your action get the source tree
from a different repo but for
simplicity's sake we'll say it's in the
same one
and you need a workflow file this is the
the special source that actually tells
github what to do this is a file checked
into the repository it's in dot github
workflows so it needs to be committed to
the repository none of this is
configured externally to the repository
all lives in in the repo so you could
check this in
into git
uh good luck up streaming that but you
know depending on your workflow
so what's a workflow file how do we how
do we get one how do we how do we get
this going uh the workflow
is
described in yaml which is a
another markup language
they're all the same they all have
different syntax they're all
kind of annoying
you need to define within that file what
triggers the workflow so
what are the conditions for it to run
you need to describe what the workflow
will run on so
what hardware essentially what you are
asking github for and then you know the
meat and potatoes of what does the
action actually do
so quickly looking at triggers this
isn't particularly interesting to us
we're pretty much just going to be using
push triggers for the workflows i have
in mind but um there's a lot of stuff
here and if you go to the documentation
there's a whole bunch of stuff that's
probably largely irrelevant if you don't
have a very github-centric
you know workflow already
you can have stuff trigger whenever you
push to any branch whenever you push to
specific branches on pull requests which
we probably won't use
manual triggers so that's either you
know going in and clicking a button in
the ui or making an api request
and there's also a bunch of filtering
you can do based on the actual contents
of the commit or the the body of work
a common example is that you have
actions that are going to run tests and
those tests are only going to run if
the subsystem that they're testing has
actually changed so if there's no code
changes in the subsystem you probably
don't need to waste time testing it on
this given commit so
there's a bunch of fanciness you can do
there
now let's talk about
where it's actually going to run
we're going to be talking about github's
freely provided hosted runners now you
can provide your own you can you can
bring your own
and give github an ssh key and then run
their software on your machine do i
think it's only x86 and maybe arm you
know but then you're buying into a
proprietary ecosystem
uh without using their computers for
free so that's a bit less fun so i'm not
going to talk about that
instead we're going to talk about you
know the two viable options for us which
is ubuntu and older ubuntu
there's also other operating systems
that you might want for other projects
but we're looking at two ubuntu's if
you're not a fan of ubuntu you can you
know there's
container runtimes you can do whatever
you want to get into a build environment
you prefer but you start with ubuntu
these are hosted on microsoft azure
github is owned by microsoft
github actions is derivative from azure
pipelines i believe if you've done azure
stuff you might recognize this class of
instances
uh but they're not too hot right you get
a two core cpu from a xeon from
somewhere in the last several years um
from
you know lower end zeons from 2021 back
to haswell which is like 24 nanometer i
think um
there's seven gigs of ram and 14 gigs of
disk space per
instance
so that is not what you get total across
your repository that is what you get for
one configuration of one job at once so
you can have a lot of these running at
the same time
it's not like your entire workload must
be constrained to this single
vm and finally we have the you know
the workflow file we have
putting it all together to build a
kernel this is everything you need to
build a kernel well there's another
slide these two slides are everything
you need to be able to be able to build
a kernel through the system
so
let's go over it you know we've got a
cosmetic name there we're calling this
this workflow kernel build we're
triggering it on push so this will run
whenever any push is made to any branch
are
defining a job called kernel and saying
that it runs on
ubuntu latest which is ubuntu latest lts
so that will you know go out to their
gigantic farm of vms and find them on
the title and make it run your stuff the
stuff that's going to run gets defined
in the steps of that job so
first up we've got a users which
essentially
instead of defining our own commands to
run we are going to use a predefined
action in this case it comes from github
but you can get actions from
anywhere essentially it like there's a
marketplace thing
you know people in the community sharing
their own their own pre-made actions to
save you from doing some stuff yourself
so you can look through and find stuff
probably of limited relevance to the
kernel but there's certainly stuff out
there
that could be useful
um so in this case it's just going to
check out our repo on disk so uh 3 is
going to be it slash home slash runner
linux or something
next we have a a run step that we are
describing as install dependencies where
we install the dependencies there is a
decent amount of stuff pre-installed i
believe to build the kernel i just had
to install
lib ssl dev and lib elf dev like it had
making stuff so
i should probably manually specify
everything just in case and then we
build the kernel and we're done that's
all you need put that in a github repo
and magic will eventually occur
so it it works that's that's all you
need someone else's computer has run
make for us which is
pretty good it took 14 minutes and 27
seconds in this case
not super fast i could have done that in
two minutes on my computer
on my power bill so still neat right you
can have this triggering all the time
it's not your computer you don't have to
maintain it this is all this is all
pretty good um but it's not really
getting us anywhere interesting so let's
look at scaling this up and having a
look at what else we can do to get some
more get some more value here well
this is where we get some value
the kernel
has
probably infinite scaling configurations
of
architectures endian's uh
32-bit 64-bit def configs um just
configs in general platforms within
those architectures you know ways of
running bare metal
kvm whatever
matrix is really really big
and
you can scale that up using github
actions whatever your matrix is that you
don't have time to to manually test or
to maintain the infrastructure for
you can just
get github to do it for you
as much as you can within a virtual
machine so you can
run queue on different architectures you
can use different compilers you can you
know cross-compile stuff you can use gcc
clang different versions of those you
can scale this up usually and
some of them will run at the same time
you probably won't get an entire data
center to yourself but some of them will
run at the same time based on what
github feels like chucking you at the
time
so you can scale this up pretty nicely
and define that through your matrix
there's some quite complicated stuff you
can do in there to make sure that you're
getting exactly the stuff you want
tested in the way you want it um
if a you know n by m size matrix of
stuff doesn't actually help you
in this case we've just defined one
variable that we're going to be
switching through which is just the def
config we're still just dealing with x86
here we're just going to change the def
config so we've got four instead of one
and when we we run our make instead of
explicitly referring to the def config
we're going to refer to the variable
with some funky variable reference
syntax
so
instead of building one thing slowly
we've built four things slowly but then
not too much more time so this is where
you start creeping in with the the value
you can get out of github actions it's
infrastructure you don't manage you
don't pay for that can do a lot of stuff
for you without you telling it to um
i'll get into it later but i run some
continuous integration using github
actions and
i'm routinely surprised that it's just
been running posting results not
exploding for months without me checking
it so
that's some good time investment i think
so here you can see that i i pushed the
commit it took 17 minutes that's to
complete all possible all the
configurations to find all four of the
jobs
and you can see from the yellow icon
thing those were spinning so they were
all running at the same time we weren't
resource constrained waiting for more
instances all four of them got given one
of those vms at the same time um so
that's scaling
just briefly mentioning before we get
into some of the fancier stuff you can
do with this
um there's first class docker support so
if you have a docker
container that you build stuff in with
all the nice things that you like um
you can give that to github actions and
then it won't need to recreate it every
time um it knows how to do documentation
stuff so um there isn't that same kind
of first class support for some other
container runtimes but there is
people in the community who have done it
before so there is prior art there
um
caching so cash um
exists and is cool and is complicated
and hard to get a good hit rate on
sometimes
and i'm not an expert in getting the
perfect special source to make it work
in github actions but my builds do run
faster most of the time from having it
so i'm going to call that a win uh
github actions has a cache feature your
repository has five gigs of storage for
you to chuck your cache in there
the way it works is that you give it
what you want to chuck in there so in
this case the c cache directory and you
give it a key to refer to it later so if
you want you know your cache to be hot
you can you know make it so that the key
is this specific config and then you'll
use that specific c cache when you you
build that specific config again later
saving you some time
so um you can go up to five gigs if
something in the cache isn't hit in a
week it gets evicted or if you are
running up to that five gigs it will
evict the oldest thing to make room so
that's how it works it's not perfect but
it's it's pretty good and not too hard
to get going if you can um
you know if you're using docker maybe
it's hard to get the c cache in and out
you know
um
problem matches so this is essentially a
way to teach github actions what your
output means uh you know you're running
gcc or clang it's spitting out a bunch
of stuff some of it's exploding lots of
it is unimportant some of it is
important how do you get github to know
what those things mean um
if you're just using this to run stuff
for you you kind of don't really care
but if you're trying to build something
on top of it whether that's you're using
the api or you want that kind of
interactive feedback that you would get
from submitting a pull request where you
know you introduce a warning on a
specific line it puts that warning on
the line in the code review thing then
you can use this so um truncated a lot
here but essentially you chuck in the
regex and tell it what the groups refer
to and then it will put that into into
the github api and present it
so something you can do
uh this is an example of that so we've
got a spas warning here um
that's what the text looks like when
it's just vomited out onto the
um onto the console output and with our
regex that knows how to pause it um
github turns that into a into an
annotation so it has the job that was
run it has the uh matrix configuration
so that's going to be rpc64
little endian
uh with pvc64 l8 def config running in
an environment that was
ubuntu 2204 or built on ubuntu 104.
um and when you click on that it takes
you to like you know it knows the line
of code that generated the sparse
warning
so you can build stuff on this
uh
quickly going to talk about getting
stuff in and out of jobs so
much like something like jenkins you can
archive artifacts and artifacts are
essentially the part of your build that
is preserved so you know you might want
to build your tree for a bunch of
different architectures well you can you
know
archive the vm linux and then people can
run your kernel
in this example we're just taking uh a
build log we're saying that okay when we
build our kernel it's gonna leave its
login build.log and then we're gonna
archive it but with a reference to the
the def config we used so at the end of
our thing when we've run all these jobs
we'll have you know separate archived
artifacts referencing each configuration
then we can refer to them later so
in this case we are accessing the the
artifacts from a previous run and the
specific case i use this for is that um
i will have the maintainers tree i'll
have like the base that i apply patches
on top of
running these actions as well and so
when i i run jobs on incoming patches i
have a base to compare it to so i can
tell is the spurs warning an introduced
regression or is it was it already there
upstream and we shouldn't
mark that against this incoming patch
series
so uh this one is a it's using a
community action not from github um
to download another fact we say this is
the workflow we care about we only want
to look at successes so if the build
failed we don't want to use the log from
that we specify the branch and we
specify the uh the name to put it at
and
this lets us do direct comparisons with
whatever we previously want to compare
to which is quite important if you're
trying to
find newly introduced things you do need
to have some other logic that you know
knows how to diff
at your output
but this is a way to do inline
comparison so you're only flagging stuff
that's newly introduced
um
after you've built your kernel you can
boot it and test it um i'm not going to
cover that too much but it's just the
same as doing it in a shell script right
if you already have a shell script just
tell github actions to run your shell
script
you
just
run it you take your thing you boot the
thing you probably write a script using
expect you
you know
check if it worked or not you
reconsider your career options
and you carry on so
anything like that is possible you know
you can because you can build you can
take cross compilers you can build other
architectures you can run stuff inside
that you know you can
run yourself tests inside queue
for the thing you just built so
again lots of possibilities within a
restricted environment
now i'm going to give an example of
putting it all together into into
something quite unquote real this is
what i worked on a couple months of last
year
doing
so
i'm using github actions right now to
run some some continuous integration for
arch power pc based on um the actions
that the powerpc maintainer michael
hellman
had written
um
because
we were running rci using jenkins and
jenkins
is jenkins so
we have the standard stuff for a
kernel
development list we've got our list
that all development happens on we don't
use github or anything
i've got my github repository with
actions in it based on based on mpus
um we've got a patchwork server so i'll
show patchwork in a sec but essentially
we use it
for an api to get the patch series from
and then there's an api to push the the
final results too so that no one really
has to care that this is running on on
github that can be kind of transparent
and then there's snow patch which is a
tool i wrote um that does a little
middle manning it's you know taking
stuff pushing stuff putting gluing all
the pieces together is what snow patch
does
so
this is patchwork if you haven't seen it
it essentially just gives you a the
state of a mailing list right now
only
useful for that purpose if the
maintainer actually uses it but if they
do use it then you know we can see here
that
it's only showing us patches that
you know are currently for consideration
it's not showing stuff that's been
accepted already or superseded
on the right side here we can see some
green
is successes warnings and failures and
this is the result of the tests running
on on github so the github actions are
being plumbed through here to produce
test results
so
um
the process of doing that goes through
snowpatch
it's just it's a daemon that
watches patchwork for new stuff applies
that stuff to a tree pushes the tree to
github
waits for the actions that get triggered
from that push to complete and then
sends the results back to patchwork
and it does that a lot
concurrently
so
the actual workflows themselves are
doing quite a bit i didn't write
these sprinkled some stuff in that's of
lower quality um
so
power pc if you know anything about it
has quite a wide matrix itself 32-bit
64-bit um
big endian little endian lots of
different platforms
we have 64-bit and better than 64-bit
server lots of different depth configs
still support for the wii and gamecube
you know it's it's a bit it's a big
matrix so there's a lot of value here in
spreading some of that out you know
building and testing these lesser uh
lower priority or you know just less
lower user base platforms i guess um
it builds stuff it boots stuff in kwemu
uses gcn clang
um we use a tool called smart spars diff
to
you know intelligently split new stuff
in a spa static analysis output from
what upstream was showing and then
register a problem manager on that diff
to add annotations just for newly
introduced sparse diff so we can use
that to report test results
and yeah those actions were primarily
written by michael allman i just made
them worse
and this is the end result so if you go
to patchwork and click on the series
this is the kind of thing you see right
you see
is the the workflows that got run
a bunch of these are running a bunch of
different configurations so you know you
can see eight jobs 24 jobs from the the
matrix that ran on this specific series
and in this case
that warning is coming from an
annotation
on
on github actions from that problem
matcher so that's how it kind of all
goes full circle and you can see that it
knows the specific config that produced
the um the warning and that comes from
that strategy matrix so that's how it
kind of all comes full circle after a
lot of github api wrestling
so some other potential use cases for
this aside from this specific one i
think you know can certainly be of use
to subsystem maintainers because you
have a matrix of stuff that you're not
constantly testing you don't always have
the hardware to test it you can you know
keep some of that more
more niche stuff
rolling um
developer pre-submits is another one a
lot of discussion about that in the last
few years in general but um one of the
hardest things for especially new
developers is getting your build
environment going and then you know then
being asked to consider all these other
platforms that you don't have to develop
on yourself it's quite daunting um so
you know if you pre-construct something
and can just say to a new developer hey
clone this repo or you know open a pull
request against it and it'll test all
this stuff before someone on the mailing
list tells you you broke 32-bit again um
then that can be that could be quite
useful so it could be a very easy
jumping off point to cover a lot of
stuff that people probably won't test
manually especially if they're not aware
of it
similarly with other architectures
because you can build
all the architectures and at least you
know emulate all the architectures in
claim you always for a smoke test that
doesn't boot
also quite useful when you don't have
access to specific hardware and you know
if one person does it once everyone can
copy it and everyone doesn't have to
individually figure out the magic when
you incantations to to boot some
architecture you're not familiar with um
and yeah anything with a big matrix that
can scale out um you can set github
actions to work doing um so it's it's
worth considering for that again it's
it's not great investing in a
proprietary platform you know vendor
lock-in etc but um
it really is just kind of run it and go
um and someone else's energy bill goes
up as a result so i'm a big fan of that
personally
quickly covering limitations uh
you know
if you are using this on a moderate
volume mailing list to test every single
patch it can keep up just fine and run
things in a reasonable amount of time if
you
ingest
a couple months of that mailing list all
creating new workflow triggers at the
same time
github will kind of just silently
realize that you're a jerk and give you
very little so if you're flooding it
with a ton of stuff you can get rate
limited um
if you have like a matrix of 24 things
on one job it's unlikely you'll get 24
vms at the same time so
it's never going to be super time
sensitive typically like you know we'll
do all these tests and they'll come back
in like half an hour to an hour so it's
not glacial but it's not instant as is
the nature of
infrastructure you don't own or pay for
um
no guarantee you'll get you know you
might be like i need this to finish
right now right now give me a give me a
machine you have no ability to influence
that
sometimes it'll just die your job will
just fail and action will fail it'll
just error
happens infrequently but it does happen
and finally
if for example you had
a github repository
with thousands of branches
that you were using to like track every
single series on the mailing list for
a couple years and then
you merged into that someone's branch
that had actions in it and you hadn't
disabled the email that triggers when
i got ddos basically from from github to
the point where i couldn't receive more
mail for the day because my mail gateway
had just given up it had hit its limit
and it took half an hour to delete 250
000 emails from the the fast mail web ui
so uh
unlikely anyone will ever hit this but i
felt like i had to talk about it because
it was just baffling
um
and yeah that's all that's all i had so
happy to take any questions if we we
still have any time and um
thanks a lot for having me especially
big thanks to to andrew for organizing
the colonel mini comp yet again he does
a fantastic job so thanks everyone
okay thank you very much russell
um unfortunately we have uh reached the
uh end of the time slot so we don't
really have time for questions but uh
you can see russell will be around the
conference and he's got his twitter
email there um and i'm sure he'll be
happy to
take further questions
um
thanks a lot angie