[Music]
welcome back
um
please welcome david uh who is a
software engineer at google uh working
on k unit a unit testing framework for
the linux kernel and he'll be speaking
to us today on kernel testing with k
unit bridging the gap
thanks
uh
no good afternoon everyone uh as you
already know
i'm david gal and i am going to be uh
talking about kernel testing with k unit
bridging the gap
um
so
who am i as mentioned i'm david um i've
been using linux basically full time
since 2006 and have been working on the
k unit kernel testing framework at
google for the last three years
and in this talk we're basically going
to cover
you know what why and how you should
test the kernel
and particularly look at k unit which is
what i work on and k self-test which is
the other main entry kernel testing
framework
um
in particular we're going to cover
what's been changing and what's improved
both in k unit and in how it interacts
with case self-test and how in general
these testing frameworks work over the
last year and where we're looking to go
in the future
uh what we're not going to be talking
about uh
is actually a step-by-step guide and how
to write tests with k unit um there are
lots of other talks about that uh out
there
um but this should provide a good
overview of of what k unit is um and as
a preview we'll be bragging about how
much better our documentation is so
there's there's plenty you can find out
about that
um
so let's get started by talking about
testing the kernel and why you should do
it and the reason you should test the
kernel is quite obviously because you
want the kernel to work
in fact you really want the kernel to
work because uh security and reliability
bugs are bad
in the kernel they're really bad
the kernel has access to literally
everything
and if it crashes your entire system
goes down so
there's a pretty good incentive not to
break the kernel um
similarly there have been been around
for a long time there have been a lot of
bugs in it over time we don't want to
have a regression where a bug we thought
was fixed came back and automated
testing as has sort of been well
described by uh
most people in the software engineering
field over the last
uh several decades is a really good way
of doing that
so there's a little bit of history um
kernel testing has come quite a long way
since linus basically said
hey
if i can get the kernel to build it's
good and if it then boots it's perfect
these days we we have a lot of tools for
testing the kernel and these largely
started as people working on on some
feature
saying hey i can write a script or i can
write a test module once modules existed
and just execute some code check the
results see if it makes any sense
the downside is
lots of people writing their own tests
in their own way ends up with something
of a mess there are lots of tests that
you know each do their own things so
case self test and and later k to unit
are both sort of attempts to standardize
these tests
um
and they're both
as part of being standard in tree
they're part of the linux kernel source
tree you go to kernel.org you download
the kernel source both k unit and k
self-test and all of the k unit and k
self-test tests are there in the table
you downloaded um
as part of being said they're also
automatically being run on a few
different ci systems um and
generally the community is uh running
these as these are the tests that you
run as a kernel developer
um there are other kernel testing
systems that are not sort of internal um
these are also very useful but we're
going to focus on the internal ones
today
so
i've mentioned case self-test and k-unit
and sort of the big question that no
one's had a a really easy answer to is
when should i use one and when should i
use the other
um
so
basically that's the question i want to
start off by trying to answer
case self test is basically a bunch of
scripts that run in user space each test
is a script
that
exercises some kernel functionality uh
just by
executing whatever's accessible so it
might be
um you know a syscall um it might be
reading something from csfs it might be
you know say you wanted to test ext4 as
a file system uh with k self-test the
way you would do that would be have a
test that mounts a file system image
that you've got and does some checks on
that from user space
you're getting sort of a whole view of
the system but not much view of exactly
internally how it works
k self test is really convenient but you
can't access anything that is
inside the kernel like you can't just
say i want to call this kernel function
directly
or i want to read the contents of this
variable in my tests um if you want to
do that you need some kernel code and
there are some case self-test tests that
have more kernel modules that they build
and run to expose these things but the
better way to handle that sort of thing
where you want to test individual kernel
functions or individual kernel variables
have the correct value
is to use k unit
and k unit is basically designed as a
unit testing framework so you're going
to be testing like individual c
functions uh within the kernel
um
and it has the advantage that all of the
test code is part of the kernel it's
built into your vm linux.so
your vm linux file or it can be loaded
as a kernel module itself directly
you can you want to call a kernel
function you can just call it
um
this does make it a bit trickier to
write those big integration tests that
that cover the whole journey from you
run mount to the file has the contents
you want but it means if you want to
test something like you know the code
that passes timestamps in the directory
entry or something you can have a test
that tests just that function and passes
all the possible inputs that you know
would be awkward to have in like a test
file system and verifies them one by one
this means that things have to be
written exactly in c like other kernel
things or i guess rust in the future or
assembly if you really really hate
yourself but um
it also means that
you know
if you can use the kernel it works there
are no extra dependencies
now so that's really when you should use
case self-testing k unit are you trying
to check
sort of the external behavior of the
system or are you trying to attack
individual functions
there are some other testing tools um
that the kernel provides that
are useful to know about but don't
actually do the same sort of thing so
either a bunch of dynamic analysis tools
address sanitizers concurrency
sanitizers memory leak checkers etc um
but these are to some extent orthogonal
to k unit and k self-test instead of
testing that a particular bit of code
works and gives the correct answer
they're just looking for common classes
of failure
like you know memory corruption
accessing things you shouldn't um
deadlocks etc um
this means that so the way i like to
think about it is a test like k unit and
k self test is something that can pass
as well as fail
whereas these um sanitizers
you know they're not testing a
particular bit of code they'll just run
the entire length of time that the
kernel is running and they might fail if
something goes wrong but you can never
use them to say
this is passed i know it's all good
similarly there are code coverage tools
i suspect everyone's got a pretty good
idea what code coverage is
uh and as with many things there are two
different ones you can use with the
kernel um
you know again these are a useful tool
to have alongside k unit okay self-test
tests that's an awful lot of different
information about sort of what um
tools there exists so
it's understandably confusing and one of
the the things that we've done on the k
unit side in the last year has actually
written all of this down and put it in
the kernel documentation
so the first sort of great achievement
of the last year is actually this is
written there's now the kernel testing
guide in the kernel documentation with a
happy link there that goes through what
all of these things are and when you
should use each in a little bit more
detail than we can squeeze into half an
hour here
um and has all of the appropriate links
uh spread through so
uh if there's one thing you should learn
from this talk it's really
if you want to know how to test the
kernel there's now a document telling
you how to do it
so we've got these two
kernel testing systems in case self test
and k unit um
and
trying to get people to use the right
one for the right purpose and there have
been some challenges over the last year
and doing that and
the first thing really is around
integration um
cayenne and self-test are in a lot of
ways very similar and serve very similar
purposes
and
the same people probably should be using
both uh we don't want to be in the
situation where someone says well i like
k self-test better i'm going to use it
for everything instead of k unit or vice
versa um because you know they're
different enough they don't want to need
to learn how to use both so one of our
goals has been to basically bring these
closer together in the areas where it
doesn't really matter
you know where the differences don't
count so
one of these is you want to be able to
put a test from one to the other and get
results in a similar format um you want
the same ci systems and same pieces of
tooling to be able to handle uh results
from case self-test and k-unit the same
way
the other thing is there were some
feature gaps that k unit and in
particular k-units tooling has had
compared to k-self-test
and the main one here is that k-units
tooling was entirely based around
this thing called user mode linux or uml
and uml is basically a custom
kernel architecture
which compiles the entire kernel as a
user mode
normal x86 linux executable which is
really neat and works really fast so you
can run your tests with k unit extremely
quickly
but totally lets you down if you want to
test something that's architecture
specific or that
involves some sort of hardware that uml
can't expose
and while uml is improving a lot in what
sorts of hardware it can expose and
emulate
this was really a gap between k unit
particularly its tooling and k self-test
so that's something else we wanted to
address in the last year
so let's start with the first one and
talk about k-tap
so
when you have a bunch of tests you need
to output their results sometimes
whether they passed or failed um and
before the days of a case self-test and
the like this was done basically by
people's scripts doing whatever the
person who wrote the script thought made
sense at the time
so some tests would print out
all was well some would return a
particular error code some would print
out nothing on success and just error
messages on failure
and unless you knew what that specific
uh test script did you know it'd be
different from everything else you'd
have no clue um
so it was agreed to standardize all of
the test results with k self-test and
other related testing tools on
tap
now tap is the test anything protocol
and is this very simple
uh output format for test results that i
think originally came from perl or
something
it's very simple it's basically just a
small header with tap version whatever
um
a list of how many tests and a bunch of
lines saying okay
test one
test name not okay test two just okay or
not okay for each test um and a couple
of slightly more interesting things for
a couple of other cases
but in general extremely simple and
worked pretty well
doesn't upset anyone's human readable
except it's a bit too simple um
k self test and later k unit both wanted
to do things and the big one is have
nested tests where you can have tests
that have sub tests
um and so you can have a single result
file with all of your tests that are
nicely categorized
um and
tap had no support whatsoever
for
having nested tests
so
every test and every test system tried
to hack nested tests into tap in their
own way and we ended up basically back
where we started
in k units case we found a draft version
14 of the tap spec that had nested test
support and implemented something that
was almost but not quite that
and tried to convince everyone to use
them and it turns out nobody liked tap
14 even the people who made it um
it was abandoned at some point um
the actual
spec text was under a slightly weird
license
uh it was not a great uh
base to to build off
um
so it was decided and uh tim bird from
sony sort of really got the ball rolling
here that we should come up with our own
specification based on tap
that would have everything that the
colonel specifically needed in it
and there were some discussions around
that uh
last year and a bit the year before
what we'd do
and we picked it up again particularly
one of our interns at google raymore to
finish off and write a new full
specification for this output format uh
which
rather than being something completely
new would rather basically just
standardize what okay self-test and
k-unit tests are mostly already doing
and that tools mostly can already parse
and just shave off a few of the the
sharp edges
so we've got something that mostly works
already and is a reasonably solid base
to build off once those uh
few things that need changing have been
changed to match it uh it also got rid
of some unnecessary stuff like nobody
wanted embedded yaml or json in the
kernel even though the tap spec
technically supported it
so this is what a tap
ktab output looks like it's very basic
i'm not going to go into in detail as
we've all got better things to do but
it's basically
um
you've got a header line a plan line and
you can nest test results you've got an
entire nested
set of test results in this case there
are 36 tests which all have interesting
list test names
and they all passed
and
the bottom one also passed
um which is the sort of aggregate of all
of these
all the subtests passed therefore the
test as a whole past a more complicated
example
is
this one which has some tests which are
skipped we'll we'll get to skip tests
later um
but you can you can mark tests as having
been skipped tests that have failed uh
comments which are called uh
um
you know these uh diagnostic lines um
some statistics about how many subtests
passed and failed uh and because
something failed the whole thing failed
and this is reasonably nice and human
readable
but
kunit already had a bunch of tooling for
parsing its own variant of tap
and we want to expand that so that it
can be used in k self-test as well uh
and so there's this script it's part of
uh k unit and therefore part of the
kernel tree uh tools testing k unit k
dot pi
run the pars option pass that in and you
get a nice prettily printed uh
summary of the results
you know it'll it'll print these
diagnostic comments but only for tests
that have failed it will give the
statistics even if the test didn't print
any statistic lines itself um
it's pretty useful we're really glad
that um k self-test will be able to use
this uh in the future and again that uh
uh ray moore who is our intern rewrote
all of the parser for this to make it
robust against things okay self-test did
but k unit didn't
the other big feature is qe mu support
uh in k unit tool
so
k unit itself as i mentioned works on
all architectures the kernel supports
it's just more c kernel code
but this k unit tool
which will automatically build the
kernel for you and run it and pass the
output and print those nice uh summaries
was specifically based on this user mode
linux architecture
so we've fixed that k unit tool now can
cross compile
to other architectures there's a cross
compile flag you can just pass a uh
gnu
a triplet in
and that will let you pick a compiler
there's also an arch option which you
pass an architecture in and there are a
bunch of pre-written configs that will
cross-compile the kernel for you using
that particular architecture
and then run it under qemu
and extract the test results
parse them print a nice summary so that
means basically you can create your
tests and then oh i want to test these
on s390 oh i want to test these on x86
oh i want to test these on
you know a big indian system a powerpc
one or something and it's it's possible
very easily
at the moment we support quite a few
architectures all of the the really
popular ones i think um
x86 um alpha power pc
risc-5 s390 um there are actually some
s390 specific k unit tests um
and it's quite easy to add new
architectures um
there's just a python script containing
uh a whole bunch of config options for
the k config options the arguments that
need to be passed to qemu what all of
the compiler names are by default uh etc
um
and this you know
i i kind of say how useful it is to be
able to just quickly test on another
architecture in particular a real one
rather than the uml um
sort of virtualized run as a user mode
process one
which is fast but but does behave a
little bit differently
there are also a bunch of other um
new features in k unit um
that some are more related to tap and
some aren't um
so since last year's lca
there are a bunch of of these new things
the first one is that we've got a
release notes page that lists them um
so
if you've been following k unit you
should definitely check in on that
to see what's been introduced in what
new versions
um
it still i think says 516 has isn't out
yet as we need to update it but it's you
know got all of the the actual features
there
um
and those include tests can now be
skipped we talked about this last year
it exists uh now it's actually upstream
you can use it um the test statistics
stuff i pointed out print statistics
there even when you're not using k unit
tool and there were a few people who had
the need to print those sort of
statistics out as a blocker for uh using
k unit instead of whatever custom script
they'd done before
uh integration with the under heart iron
behavior sanitizer
the in documentation i i've mentioned a
couple times has been drastically
improved in the last year uh it's been
totally reorganized if you go into
documentation dev tools k unit in the
the kernel source
um
there's now a better architecture
overview it's structured a bit better it
should be much easier to get started
with k unit um and a bit easier to find
specifics on how do i write tests versus
how do i run them versus how do i use
the k unit tool wrapper or how do i do
it manually
and and bug and usability fixes
there are also
a number of better ways of selecting
which tests get run
if you're a maintainer you can now put a
dot k unit config file in a directory of
the kernel source code uh for your
subsystem and list all of the options
that you want k unit to use when testing
that subsystem and people can pass
the path to that k unit config fragment
in the path to that subsystem in um and
the kernel will run
be built with all of those features and
tests included
and will be run so that's a great way of
saying run with this
um
and you can
execute things um
see all of the test results this is what
i expect before you send me a patch
similarly if you just want to add one
extra config option to one of these
there's this k config add option simple
i want to run the ext4 test but i want
to enable ksan or something like that
it's very easy
test filtering you can test filter what
tests run at runtime as well as at
compile time
and that lets you build your kernel once
and run several different tests in
several different
environments with it which can be quite
convenient too and is a prerequisite for
hermetic testing
um we all know that tests in theory
should leave everything the way they
found them and it shouldn't matter what
order you run the tests in because test
a is not going to break test b that runs
afterwards by scribbling over memory or
whatever but equally in the real world
that doesn't always happen
so
to help debug those cases there's a run
isolated option to k unit tool which
will run each test or each test suite in
its own invocation of the kernel
so they can't stamp on each other's toes
very useful for tracking down weird bugs
where memory has been corrupted or the
like
finally there are a bunch of new tests
this is always my favorite slide um
in the last year we've doubled the
number of test suites in cayun and 511
we had 20 and 516 there are 40 um
from 204 individual tests to 324
and this is despite the fact that the
parameterized test feature that was
merged at the end of
2020
now has
basically merged a bunch of other tests
so
the number went down before it went up
but
some of these new tests are really neat
uh one of them i i won't go through the
whole list but one of them i want to
point out is these timestamp conversion
tests for
converting from unix time to civil time
there it turns out someone's written
this paper of how you can use this
frankly terrifying bit of algebra to do
this really efficiently
and in order to introduce that to the
kernel um they used test driven
development by writing k unit tests for
the existing
uh
you know naive implementation of the
conversion
then implemented their their new
algorithm and then checked that the
tests still passed
and that's really neat they've done that
for both
regular timestamp conversions and rtc
clock conversions um
k-fence is a new
memory checker
there's sound driver sd card driver
thunderbolt multipath tcp
stack unwinding for the s390
memory management daemon's really cool
uh this was the first
project that actually used k unit and
it's finally upstream so we're really
excited
basic kernel functions and c functions
like mem set mem copy mem move all
actually have some k unit tests um
the hashing for files it's all there
so
what's what's coming up in the future
um
we're going to fix all of the things to
actually match this new ktap standard
that we wrote uh we started on that but
there are still obviously a number of
things that now we know what the
standard is we can bring up to it
um
improved support for running tests as
modules is a big one
as k unit
developers we mostly build the tests
into the kernel image rather than
loading them as modules afterwards but k
unit does support this
and jeremy who as i think is here has
started uh proposing this new way of
handling modules that is a lot less
weirdly uh
hacky than our old ones so we're looking
forward to improving that
more test test examples particularly for
testing hardware
we've reduced memory usage and stack
size even further
which is is particularly useful since
linus was whinging about it uh when w
error went in and a number of uh bug
fixes and documentation changes
um
the final thing is please reach out if
you have any um questions for us i'm
very curious if people have actually
been using ku nor k self-tested
if there's anything blocking people from
doing so
whether
maintainers in particular wants to have
lists of tests they want people to run
using this k unit config mechanism or
another like an entry in maintainers
saying please run these before you send
me patches
and the contributors like the idea of
that
if you send a patch will you tell people
what you've tested in it
and how much refactoring of code i think
this is going to be the theme for the
next year
is acceptable to make something more
testable
because a lot of code in the kernel is
intertwined in a way
um that makes it difficult
okay
uh questions comments
uh i see there are a few questions here
um
so i'll get started um the first one
from andrew tap wise are we going to be
okay with reporting multiple sets of
results where kernel tests may be run at
different times like initialization
versus device pro versus module
in it um
this basically is is one of the tricky
things um
k unit uh tooling we're hoping to make
handle
okay if there are multiple sets of
results we'll be able to merge them uh
it's not quite done yet but uh you know
in theory it should be able to work
um
uh
most tests
you know when you run k unit run in it
um and there's this uh dev
debug fs way of getting the individual
results for
things that are loaded later
do any cis run
kernel tests with random enabled config
options
um i know there are some that at least
build them with random enabled config
options
um
i'm not sure off the top of my head if
any of them running there are some that
run with um
i think there are there are certainly
some that run with with pretty large uh
all yes config style setups
um
uh the hermetic testing
um
can the hermetic testing run isolated
feature uh be used for target specific
drivers tests only
eg when developing an out of tree
prototype driver against a stable kernel
at the moment run isolated as a feature
can't be used to separate one driver
from the others in the same execution
but you can combine it
or use instead the uh glob filter thing
to run just a particular driver um
with a separate invocation of k unit
tool
um
so it's definitely possible to do that
um you just you know if if the driver
was called foo you can just k unit dot
pi run
foo star and it will run all tests that
start with foo
so
if you name things well it's easy
well thank you very much david that
brings us to the end of our time flop
for this talk
and david will be uh more than happy to
take further questions or comments uh
outside of the session
um yes so thank you thank you very much
um we'll now take a another 10 minute
break and we'll be back at 5 p.m for
russell curry abusing github action for
building and testing kernels see you all
then