[Music]
hello everyone welcome back to kaya
theta where we are in the middle of a
little colonel talk party um so next up
we have keith packard uh keith packard
has been developing free software since
1986. he is currently a senior principal
engineer with amazon's device os group
he has received a usenix lifetime
achievement award an o'reilly open
source award and sits on the x.org
foundation and amateur radio digital
communications boards um
i think keith is probably a pretty
familiar face um at
linux comps
so
i think i think he knows the drill as do
many of you so uh keith will be taking
questions after the talk so if you have
any questions for keith please put them
in the little questions tab above the
chat in venulis and we'll pass them on
you can also upvote questions that you
think are great and want to be asked
okay all over to you keith
thank you so much betsy uh thank you
again for welcoming me to another
glorious lca conference i wish i could
be with you all uh maybe next year in
canberra
i'm going to be talking today about some
work that i started um and somebody else
has taken over the reins and is is
working on much more than i am now uh
talking about kernel hardening uh for
arm 32
working on some stuff that's that uh
some bugs that got filed in uh 2019
as betsy said i'm working in the device
os group at amazon we're the group
responsible for building operating
systems for all of amazon's fun devices
from tablets to tvs to echo devices
[Music]
okay the kernel self-protection project
i asked case um when he started this
project and he said he actually sent me
a link to the email message that he sent
out on the 5th of november in 2015.
if you were all here for the last
session you'll know that case cook has
been doing kernel security for a very
long time um and i'm really i i keep
being awed by the amount of work and the
amount of progress that he's made in in
making our our favorite operating system
secure uh even even though the language
that it's written in is uh is not the
best in the world
the kernel self-protection project
is all about defense in-depth or linux
you heard case talking about one of the
one of the newer projects i'm here to
talk about one of the oldest uh set of
bugs in the in that that are filed in
that project um the part the in fact the
bug that i'm uh mostly talking about
today is bug number one in the kernel
self-protection project
uh kspp is is all about eliminating
classes of bugs
uh you can you can hear
his talk about fixing overflows in mem
copy
it's worked he's worked on eliminating
variable length arrays in the kernels
static array overflows um and also
eliminating methods of exploitation of
bugs so the method that a lot of a lot
of exploits use is return oriented
programming and kspp has been working on
mitigation techniques for things like
that so both fixing the source code
classes of bugs and then making the
kernel harder uh harder to exploit once
you've actually found a way in
okay so
i'm talking about the 32-bit arm
architecture and you might be asking me
are you actually building devices
with 32-bit arm processor the answer is
well
not really
essentially all of the devices that we
build that run linux are actually modern
arm chips that could run 64-bit code
so why are we still running 32-bits and
this graph is designed to give you an
indication of why we might still be
doing that
so starting in
in
this data actually comes from jcmit.net
who has historical data back to the
1950s
but i truncated this graph to just
starting in 2000 to show you kind of the
last
of the last 15 years of that exponential
decline in memory prices
that kind of stopped somewhere between
2010 and 2015 we stopped being able to
reliably expect that memory would get
cheaper every year
and for the past 20 years or so we've
had to expect that memory prices are
pretty constant somewhere between five
and ten dollars a gigabyte uh depending
upon when you actually make your orders
and so
saving memory
is
you you can no longer expect that just
delaying your product by a year uh and
and
and will will make the memory cheap
enough for you to be able to afford to
build your product you really need to
start thinking about memories of fixed
cost instead of an ever decreasing cost
and so by using a 32-bit arm
architecture we're able to save not a
lot of memory but a bit of memory and
every bit of memory
every bit of memory saved is a bit of
memory i can do more fun fun features in
the products with
and so we're really using the 32-bit arm
kernel right now in order to save the
memory that we can
this may change in the future as arm
tries to push a 32-bit architectures off
of their roadmaps
and vendors stop selling parts that can
run 32-bit code
we may have to do something different
but for now we're still doing a lot of
32-bit work
and all for all for saving memory
alas uh arm 32 world kind of feels left
out of the kernel self protection
project
a lot of the um kernel self protection
project fixes require architecture
specific changes
um and a lot of the people working in
that area uh are working are focused on
newer and higher end architectures that
are more more interesting and fun to
play with like you know x86 and arm 64
power pc risk 5
the kinds of places where it's actually
easier to do a lot of this work
older architectures
especially especially smaller more
limited devices like arm
and mips have have unique challenges
that we'll get into in the process of
the pre of this presentation the problem
is that some of the most critical fixes
are still not available uh for kind of
these two older architectures that are
really common in consumer devices arm 32
and mips
so what are these first
first bugs you ask
the first four of these bugs
are
the thread info in the kernel stack and
that's the one we're working on today
that's where the the the a significant
amount of the per process information
um is actually stored in the same pages
in memory uh as the kernel stack
and we'll we'll find out why that's a
really terrible idea
the second second bug that we want to
resolve from kspp number two is that the
kernel stack should be protected so that
if you overflow the kernel stack instead
of smashing memory adjacent to the
kernel stack that it should probably
trap
so that you know that a kernel stack is
overflowed
one of the problems with a c language is
that it doesn't really have any guards
against stack overflow
you can kind of allocate stack memory
however you like and there's no there's
no easy way to
uh to detect in the c code that you've
overflowed the stack
so we're using a hardware protection
here uh to try to catch that
instead of instead of relying on
software
and that's with these guard pages the
way that this is done is you have
pages
you have
the kernel stack pages and surrounding
the kernel stack are are unmapped pages
in memory
and so that if you try to access those
there's no memory there
and so the kernel actually takes a
memory protection fault
in hardware and so that lets you let you
trap the kernel stack overflow
number bug uh bug number three and bug
number four are things we're not gonna
be working on today uh but i'm hoping to
get started on those in in the future um
and those are those are addressing some
more common uh common problems uh that
would be good to fix uh on the on the
arm architecture uh the kernel uh base
address offset randomization uh would
make it more difficult for uh attacks to
know where data and and code is in the
kernel
by uh by making locations of stuff in
memory random and undetectable
undiscoverable by applications
and then turning on some more
mandatory kernel memory protections
right now in the arm environment we just
don't have enough memory address space
uh to really enable a lot of the kernel
memory protections that we've used in
other environments
um so we're hoping to be able to do some
of these uh and and improve the security
for army 32 and mips devices
uh why did i get started in this um so
in last january i was talking uh talking
to you from a different company
and in may i started a new job uh at
amazon
um
and i'm a senior principal engineer
which is a senior a senior uh individual
contributor but i spend most of my time
uh mentoring other engineers uh doing
project management uh and doing and
doing uh high higher scale technical
activities and i really miss uh the
opportunity to get engaged in a
low-level uh seriously technical project
um
it's so i
so because i have so many outside
commitments i was looking for kind of a
side project related to my amazon work
that was technical uh clearly relevant
to amazon and as we use arma 32 this is
this clearly uh relates to that um and
something that is super important but is
not being worked on by other people so
that i could kind of contribute as i had
time
two other side goals of course we're
playing with my friend case he and i
both live in portland um and get
together talk about the linux kernel and
play board games and so having another
topic to chat with case about admit that
i might get to play more board games
and of course learning another area of
the linux kernel
and my risk in my last job i was
starting to get involved on in kernel
initialization uh for risk five
processors and i really got excited
about the super low level uh details
about how the kernel ran on an
individual processor and so the
opportunity to figure out how the how
linux runs on arm 32 is is really really
exciting to me
it's something i haven't spent a lot of
time working on i've done a lot of stuff
on device drivers and outside that in
kind of memory memory management
graphics and that kind of thing
and this uh this is really a very
different area and so it's always fun to
learn something new
okay so i want to get started uh
fixing these kspp
issues
so we'll start on bug number one it's
always good to start at the beginning
so this is talking the bug number one
says that we want to get the thread info
out of the kernel stack
uh the thread info is used it contains
information that is uh that is
architecture specific
um and it's used in early cis early it's
used early in syscall operations so when
you jump into the kernel
and you're just and you're going to do a
syscall there's some data in the thread
info that's needed
super early in that process mostly to do
memory bounds checking
it's super vulnerable to kernel stack
overflow it's literally sitting in the
same memory pages
and just below the kernel stack so if
you manage to overflow the kernel stack
you can smash
you can actually smash the thread info
and there's absolutely no detection of
this at all um in particular this is
some critical security bits in here that
that that limit how much that limit
which addresses the application are
considered valid for the application to
use in the syscall interface and if you
can smash that you can actually get the
kernel to access arbitrary kernel memory
uh through the syscall interface
the details about that are kind of a
little more convoluted than we have time
to go into here but it's super super
important to protect this uh protect
these elements uh from applications uh
destroying them
and the number of kernel stack overflow
exploits used to be super big because
everybody used to do this threat info
was always stored in the kernel stack
and when all the other architectures
moved it out all those exploits appeared
to go away because they weren't present
on arm64 or x86
but guess what you can still use all
those same techniques on an arm32 kernel
and the way that this is solved is by
merging this thread info
into another per task data structure
called the task struct
right now the task struct is allocated
just in regular kernel memory
and the thread info is in this magic
spot in the kernel stack
and you can just merge those together
it's a little tricky
for reasons we'll go into later uh but
essentially every other architecture
other than mips has already done this
work and so there's a lot of a lot of uh
a lot of
kind of a well-trodden path
which meant that as i was learning how
this system worked i could go back and
review the patches from other
architectures and figure out how those
how this was done there
and that made it super easy for me to
kind of follow along and figure out
how this work was uh needed to get done
so what does this work really mean uh
what is what are we going to be doing
here uh so in the current arm 32 kernel
we have these two structures we have the
thread info and we have the task struct
um and the thread info has a pointer to
the task and the task struck has a
pointer to the stack segment and of
course the very first thing in the stack
segment is the thread info so they kind
of reference one another so if you have
a task struct you can get a thread info
and if you have a thread info you can
get a task struct
the goal here is to just smash these
together and stick the thread info at
the top of the task struct and the
reason it goes the top of the task
struct is super complicated um and
there's that i actually have a slide
about the the convolutions uh within the
linux kernel uh that require to live at
the top of this later on in the talk
that was kind of a surprise to me
welcome to the c programming language
again
so that's the goal
is to take these two data structures and
smash them together
okay so why is the kernel struct uh why
is the thread info in the in the kernel
stack right now
and and the number one reason on the arm
processors is that you want to be able
to find
the thread info from your kernel stack
pointer
um
and uh and
it
when you enter the sys when you enter us
from a system call
all you've got is the cpu registers and
you need to be able to find something
that lets you know what task is
currently running
um and so on arm the way that we do that
is we just take the current stack
pointer uh mask off all the high bits
and then voila because we're now
pointing to the base of the kernel stack
setting we have a pointer to the threads
to the thread info structure
so it's architecture independent
it doesn't depend upon any other state
in the processor or in the system so
it's atomic with respect to thread
switching i mean it's super fast all you
have to do is take the stack pointer
and do a simple arithmetic operation on
it
a bunch of other per task information is
in the task struct
so we'll need to be able to find that
but fortunately the thread info has a
pointer to that and so we can go get
that when we need it
but the key here is that we need a way
to we need kind of that ground truth
once you once you are just sitting here
happily running along the kernel and you
need to find out what your thread info
is all you have is the cpu registers and
so you need to be able to take those cpu
registers and compute
thread info from them
so that's one of the main reasons
another reason is just historical
in old v7 linux
not linux in old v7 unix
most of the uh per process information
was stored um in in the uh in the
in the in the thread stack as well in
the kernel stack as well
just because
because there wasn't enough memory to
keep it in regular kernel memory um and
so the v7 unix uh saved a bunch of
memory by putting all this per thread
information
uh in the in the in the task itself and
so that when the task got swapped out to
disk
it could it could use that memory for
other things so that's where this kind
of behavior came from
kind of a classic a classic hack of
saving memory and address space
okay
so because we are about we're trying to
smash the thread info and task struct
together those are no longer going to be
in the kernel stack and that means we
need to find another way to get a hold
of the thread info and the task struct
from an arbitrary thread context that
doesn't depend upon the stack pointer
anymore
so we need to we need to find another
way
and the basic problem is is that when
the task enters the kernel we need to
get these two pointers and all we have
are the cpu registers
it can be interrupted at any point
so the the a lot of the places that we
need to get this pointer aren't in an
atomic context which means that between
any two instructions we could switch
which cpu we're running on so we can't
depend upon which cpu we're running on
we need to depend upon context in the
cpu that is that is uh thread specific
and that's why the stack pointer is so
tempting because that is by by its very
nature thread specific it points into
the thread's kernel stack
uh it turns out this is a lot harder
than i thought uh because i really
didn't have a good understanding of what
was required here
uh so i did the dumb thing uh well not
really dumb this the the kind of the
obvious thing
i tried to create a per cpu variable
that would point at the current uh the
current thread info
um per cpu variables are kind of a magic
part of the kernel uh that i got to
learn about when i did this which was
cool uh learning about new stuff is
always fun uh they're allocated uh
they're kind of allocated
at
boot time through magic
every per cpu variable has a magic
offset value
and a base address and you can find your
per cpu value by adding the per cpu
offset to the base address of the
variable it's kind of funky but it means
that you can uh that if you know your
cpu number uh or if you know your your
cpu offset you can go get these per cpu
variables
the problem is on newer arms this isn't
atomic
because you have to load the per cpu
offset from this magic register
that's holding that value on new arms
and then you have to fetch the per cpu
value from memory
the problem is as was pointed out to me
if you if you switch processors between
these two steps
then you have the wrong per cpu value in
your register and you're going to load
the wrong cpu
the law the wrong uh per cpu value from
memory um and in fact you're going to go
talk about some other thread running in
the system which is probably not going
to work out very well
an older arm 32 is even more problematic
the per cpu offset on these older arm
processors are stored in memory
and it's fetched using the cpu as an
index
oh and the only place the system stores
the cpu index for the current thread is
oh right in thread info
so that means if i want to find the per
thread
data structure this thread info data
structure i need to go get go get the
per cpu offset which is stored in an
array which is indexed by the cpu
which is in the thread info so i can't
do this at all
fortunately around the same time
case decided to
to
bring another member kernel developer
into his team
and that that person is is uh named art
um
uh
bishovel
and i asked permission to pronounce his
name in public uh and i apologize uh for
not getting it quite right um he and
case
are he's actually
dutch
and case tried to give me some pointers
on how to pronounce his name uh and so i
hope i did okay uh case brought art into
his google team uh probably about four
or five months ago
um and one of the first things he
started to do was reviewing the patches
that i'd provided um which was awesome
you know the the best part about a free
software world is that when you submit
code out you get comments back
and art's comments were super helpful
and really positive um and so one of the
comments that he made was yeah that per
cpu variable thing is probably not going
to work out very well for you
so we'll have to go fix uh go fix the
whole how do we find the thread info
thing
okay so we decided to make a second
attempt
that only worked on some arm 32
processors
newer arm 32 processors have have a
bunch of extra registers
and two of them are these tpidr prw and
tpidr uro
i have no idea what those initialisms
are supposed to stand for
but i knew i do know that the
tpi dr prw register was already used
as i as we as we saw before that's
already being used in the kernel for
these per cpu offsets
uh and the tpi dru r0r
is already being used
in the gcc abi
for the tls base register so when when
you're up running in a user space are
you using that register to find the base
of your thread local storage uh
data in your in your application
and and so those two registers which i
had available to me were both already
being used
so i put together a patch that switched
the tpi dr prw
from the per cpu offset to the thread
info
because i knew that once i had the
thread info i could go get the cpu out
of the thread info and use that to find
the the per cpu offset
using the global array just like it does
on unit processor on on older arm
processors
so that seemed like a pretty easy thing
to change
and i put together a patch and i got it
all working and i submitted it and
i got back a bunch of comments uh mostly
about um that's a performance problem we
use this per cpu offset a bunch
so now you're making that a lot more
expensive to get you're adding several
memory fetches in fact
to go get that data you have to go get
the cpu index out of the thread info
and then you have to go get the per cpu
offset out of the array
and the other thing is is that gcc
already knows about tpidr uro
because it uses it in user space for the
thread local storage pointer so we can
actually use this magic gcc built-in
function that knows all kinds of
semantics about built-in thread pointer
which means that when i optimize the
code gcc knows how that function behaves
it knows that it doesn't change a change
between
function calls it's always the same
and so gcc can actually hoist
that operation out of inner loops it can
share the share the value between
multiple statements that use this that
use the the um the thread info pointer
and so using tpi d-r-u-r-o was super
tempting uh for that reason as well
so art suggests that we try using that
instead
um the com the hard part there is that
now we need to actually save and restore
uh that value whenever we go back to
user space
but that didn't turn out to be too bad
um and kernel and we already save and
restore a bunch of registers across that
across that operation anyhow
and we often times have to restore the
tpid or uro register going back to user
space because we need to make sure that
the right
thread local storage pointer
is stored for user space so we just
needed to add a couple more checks uh
to make that happen
okay
so now we are uh at slide 15
and all that we've managed to do is get
a pointer set uh in the kernel
we haven't actually changed anything yet
uh the data structures are still all in
the same place
but now that we have this pointer we can
finally move the thread info
and so that turned out to be kind of the
easiest part of the easiest part of this
process because the kernel already knows
what to do here it already has all this
configuration infrastructure and code
that supports
thread this config option thread info
and task
and so it was super easy to enable that
and all of a sudden things were working
again
it took a few arm specific changes
uh to actually use the register in the
appropriate places um and then to kind
of clean up the the thread info to get
rid of the dregs of data that were stuck
in there
the good news is that this piece of the
patch has actually
landed in 5.16 kernel
and so
we've actually fixed bug number one
for some arm architectures in 5.16 and
we'll talk about the remaining work to
be done
later on
okay so
we managed to get that fixed
but in the process of fixing that there
was an unexpected new problem of course
always right
the thread info in task struct
i don't know why but somebody decided
that when you enabled that it should
move where the cpu field remember that
cpu field we're using to index the per
thread information
i mean the the per cpu information it's
used for other things as well
and it
and for some reason when they move the
thread info
and merged it into the task struct
all of those patches also assume that
the cpu field is now in the task struct
on all architectures
so you don't get a choice about where
that lives you didn't get a choice about
where that lives
the problem is is the task struct is
super super huge and it has
hundreds of fields
of data types from all over the kernel
architecture specific architect
architecture independent um and so the
problem is is that
i can't include the files that reference
the task struct
every place that i need to fetch the cpu
field out of the thread info
the circular it became a circular
include reference uh kind of disaster
um you you'd try to include that file
and then all of a sudden you had to
include 50 other files before it
and the patch were the patch that i
actually put together to to fix this
problem touched
three or four thousand files in the
kernel um and was super invasive um and
kind of super scary in terms of
changing how files were getting included
across large swaths of the kernel which
meant that it was really difficult for
me to validate that it was correct
and so
i looked at that and decided to kind of
walk away
and not do that fix
and so the first patch that i put
together
adopted a patch uh a terrible hack that
power had uh had implemented
where it computed the offset of the cpu
field within the task struct before the
kernel was compiled and then has this
magic function that would add that
offset to the base address of the of the
thread info pointer and know that the
thread info pointer was always embedded
in a task struct and go pull the cpu
field out of the enclosing task struct
that was really awful
but it did get rid of the circular
reference problem
and so that was kind of a piece of the
first patch that i put together
and art suggested a better solution a
clearly better solution
instead of
using this terrible clue we should go
evaluate
why the cpu field was in the task struct
and could we just move it back so
instead of being in the task struct it
could be in the thread info again
and from a memory perspective it doesn't
matter at all these the remember the
thread info is embedded in the task
struct so placing the cpu field in the
thread info didn't change the allocation
at all
we were still going to have it in the
same piece of the same allocation and in
fact when i looked at x86 it turned out
that moving the cpu field
from where it was located in the task
struct back into the thread info was
going to get rid of a couple of padding
fields
that got inserted into the task struct
and shrink it so that was kind of pretty
cool
and so we actually worked with all the
other architecture well we by we i mean
art uh worked with all of the other
architecture teams uh to get these
patches landed and he managed to land
patches that moved the cpu field back
into the thread info
one of the results of that was the power
pc uh terrible hack got removed and so
now power gets a cleaner implementation
uh for fetching the cpu field
uh x86 saves a little bit of memory um
and we are our fine arm patches can move
forward
okay so we fixed it on
this arm chips that have these magic new
registers uh so those are
arm v7 and arm v6k
it turns out that the only smp parts
supported in the kernel right now which
is to say the only kernel the only arm
chips you can run a multi-core kernel on
are either v7 or v6k and both of those
include this magic new register and all
the other arm chips that we run linux on
our uniprocessor
well it turns out that registers and
memory are really similar to unit
processor
because
the the there is only one set of
registers and there's only one set of
memory so on a unit processor part we
can actually use a global variable
for the current thread pointer
um and so
we don't need to use the old thread info
in stack
code anymore because we can just use the
we can you we can put the thread info in
the task struct i'm going to either
reference the thread info the current
thread info using the magic cpu register
or if you're on a unit processor part
you can use a global variable
so that meant that we could get rid of
the old code paths in in the arm code
and that was really the kind of the the
big concern that we had was that if we
had to leave around support for thread
info in the kernel stack then all of the
arm
assembly code and all of the arm
specific architecture code would have to
have to retain this these old code paths
and because it didn't get run very often
for testing and certainly i wasn't ever
running it and art wasn't ever running
it we were concerned that that code
would eventually stop working correctly
in some cases and so getting to a single
code paths was super important to make
these patches viable and so figuring out
a figuring out that there were no parts
that we needed to use that old code path
on was really really helpful
and now we just had to figure out what
we wanted to do uh to solve the unit
processor versus uh s p parts
um and so uh i i actually did wasn't
part of this development at all
this happened in november and december
when art went back and figured out that
what he could do
was patch the linux kernel at boot time
to switch between these two modes so you
can actually compile the kernel and say
if you run on a uniprocessor part that
doesn't have this magic register
then rewrite all the code that that
fetches the current thread info pointer
from a register rewrite that with code
that fetches it from a global variable
it took some clever code to make sure
that the
uh code sequences that fetch data from
memory didn't need any temporary
registers because we didn't have any uh
in the in the uh to to use at that point
but art found a nice clever mechanism
using a couple of a couple of magic arm
instructions in thumb mode that managed
to do that computation
so now we can compile the kernel
uh a single kernel can get compiled that
will run on all arm systems uh
uniprocessor or the magic new
multiprocessor parts with this met the
magic registers
these patches haven't quite landed
these and a couple of other things are
are
are hoping to land in 5.18
um but it's going to be super cool
because all of a sudden we're going to
have a single kernel that runs on uh
that runs on all arm parts um and fixes
this particular bug okay so after we
found after we got uh bug number one
fixed uh art has started to go off uh
and work on bug number two
we hope it doesn't take six months to
fix that one as well
and that was talking about guard pages
around the kernel stack um
we're going to put missing pages around
the kernel stack to cause a hardware
fault and that'll that'll catch both
underflow and underflow overflow and
underflow of the stack
and it means that applicate exploits
can't write read or write data adjacent
to the stack allocation
remember the current the thread info
used to be right in the stack frame i'm
going to stack overflow could smash that
but if the stack is allocated in regular
kernel memory then whatever is allocated
right next to the kernel stack is still
subject to overflow
and so we want to protect the kernel
against that as well
it requires a virtually mapped stack and
that means that the
the stack is no longer going to be part
of the
part of the kernel's linear address
space and that has a huge number of
implications across the entire kernel
uh especially for low-level boot code
and especially on architecture's old
architectures like arm32 arm32
um so
the the first thing we tried was just
turn it on and see what happens
there's a config vmap stack option and
you just turn it on and suddenly you get
virtually mapped stacks
well on arm 32 yeah not so much
there's a huge amount of code in the arm
32 architecture specific stuff that
assumes the kernel stacks are all in
this linear map
and not mapped in virtual address space
the suspend and resume code assumes that
the physical addresses will match the
kernel addresses
we also have to deal with the fact that
if you can overflow the stack
you want to be able to recover safely
from the stack overflow
so in order to do that you need to
allocate a new temporary stack in case
you overflow the current kernel stack
and so for every c every core in the
system there's an overflow stack
allocated a single page
that we can use when we're dealing with
a problem
but now all of a sudden the kernel stack
might not be contiguous in memory
and that causes all kinds of havoc
especially with kernel stack traces so
when you get a kernel stack overflow you
really want to find out where that
happened you really want to get a stack
trace printed out that's reliable to the
to the console
so you can figure out where the bug was
and that means that we had that art had
to go back and
re-engineer a ton of the stack craze
stack trace code to deal with the fact
that now we have this overflow stack and
it has pointer references back up into
the main stack and those are
not in contiguous parts of memory
and those patches still have not landed
upstream
uh but we're getting closer
okay so what is the current status
as i said above
the bug number one has been fixed uh for
for the modern arm 32 processors v6k and
v7
um we're waiting to get uh
number one for that to get merged um we
hope to get that merged in in 5.18
uh number two uh the vmap stack fix uh
the fixes for bug number two uh uh got
dropped from the 5.17 merge window
hoping to get
them into the 5.18 merge window so
things are making progress uh and i'm
hoping to hoping to help art get these
merged in the next couple of months
uh and with that at the end of my
presentation if we have questions we've
got uh about eight minutes seven and a
half minutes for questions thanks very
much for letting me come back to lca
thank you so much keith um for sharing
that story with us
um
it's
yeah it's fascinating hearing stories of
bug fixes and investigations and things
we do have a few questions for you
and we've got quite a bit of time
um and the rest of the questions are a
bit more technical i promise but the
most upvoted question here so far is
which are the best board games to play
with case while chatting colonel
casey has a huge collection of board
games which is awesome
and so one of the ones that that he
showed me that we've been playing quite
a bit is called patchwork
it's a quilt-themed two-player board
game uh we're trying to piece together
uh so it's not really a quilt themed
it's more of a patchwork theme
or you're trying to patch work together
a quilt on your game board and scoring
points as a result of that and we've
been having a lot of fun with that one
that sounds lovely that sounds
absolutely lovely
okay
next question
does the current status mean for someone
making a product with um arm 64 chip
there is choice between more security uh
running 64-bit kernel or less memory
usage running a 32-bit kernel
yes exactly oh and so right now because
there are still so many unfixed kspp
problems uh arm32 kernels are vulnerable
to exploits that arm 64 kernels simply
are not
and
in
and we're going to try to fix those as
rapidly as we can to try to try try to
get back to parity um even still arm 64
is likely to be more secure because it
has a bigger memory address space
um and so a lot of the things like uh
address-based randomization are kind of
more effective in the 64-bit world
but i'm hoping that arm32 kernels will
will at least be
kind of to parity with 32-bit x86
kernels within the next year or so
your next question
is
when will amazon be able to use these
new features in their products
i'm hoping super soon um we're we're
constantly updating which kernels we're
using
um one of the one of the uh challenges
with any with any linux based products
is that you you get uh you work with an
soc vendor to get linux kernels
tuned for a particular soc and ready to
go and integrate into your project
and so we need to work with the our soc
vendors
to figure out when they're going to be
ready to switch to a kernel that has the
stuff enabled
i would i really can't uh give any kind
of dates for that because i don't have
any idea
[Laughter]
timelines but we do have quite a bit of
leverage with soc vendors uh and and
getting them getting them to run more
recent kernels is definitely one of our
one of our big big issues
that makes sense
is
anyone working on similar issues and i'm
sorry i'm not sure if this is supposed
to be pronounced letter by letter or as
a whole mips
on mips i don't know of anybody working
on this on mips which is kind of
interesting
um
there are still a lot of products based
on mips especially in the television
space i mean it would be awesome for uh
for people who are who are working with
those parts uh to actually dig in
and figure out how to how to how to do
something similar um
uh i don't know of anything that that i
don't think there are any amazon
products that are using mips chips at
this point uh they used to be used a lot
in routers so um
maybe the maybe the uh open worked uh
crowd uh wants to dig in and find uh
find some some people interested in
working on these patches for mips
are there any plans to also apply the
thread info task struct change to power
pc
uh those are already all in power pc all
of this stuff oh i'm sorry the power pc
hack that i was talking about was was
actually a patch required because they
moved to this mechanism
so in power pc they needed to go find
that cpu value given only a thread info
pointer
and the place where they needed to use
that they couldn't include the entire
task struck include file um and so
that's the reason they used this magic
cluj they used the kluge because they've
already done this and all powerpc uses
this already
um
there's a there's a bunch of questions
that are tied for votes here um
with the thread info being architecture
specific are many attacks seen in the
wild yet against arm 32
i don't know
i would love i would love to get some
information about that but um
i'm i'm i haven't really looked to see
what kind of cves are being reported
against arm 32 products
i'm kind of terrified to go look because
i know that they're vulnerable and
surely somebody must be using these to
exploit vulnerabilities in arm32 based
products but i
i can't really see them separately
because the cves that i get that i have
visibility to are mostly on the server
side and so those are the arm 64 and x86
64 cves
um and so maybe i'll get some visibility
into arm32 cves
um
but they're they're much
there's kind of a different community of
of people working on those chips
and so i don't know how the security
vulnerabilities are reported in that
environment
i should go find out thank you that's a
good good suggestion
was most of this work performed under
emulation
yes uh essentially all the development
work was performed under emulation
um because the emulator lets you run gdb
on the target
and so you get a full debugging
environment
but not to worry um our case and case
and i have a friend who live in portland
vagrant cascadian
who had a couple of spare raspberry pi
boards
with suitable processors and so let me
see if i can watch it here
i actually have a board that he shipped
me
uh that i've been doing uh the actual
validation of the patches on real
hardware
because it's nice to know that they work
in emulation but you really need to
validate that it works on hardware
before you're sure um i need to get a
couple more boards running uh now that
we have the unit processor stuff going
uh to make sure that also works on those
so yes emulation is awesome everybody
should do all kernel development in
emulation
that makes sense if only that were
possible
okay we are out of time there's still a
couple more questions um so we will move
those transfer those questions over to
the post talk chat kaya theater channel
which is invenulous if you have you may
have to go into the browse channels um
button to find that channel if it's not
appearing in your list of channels
and keith will be there to have a bit of
a chat after
so we're out of time it is now lunch
time enjoy a little bit of a break
everyone um and a reminder that the
linux australia agm is happening now so
if you're going to that ah don't miss it
okay thanks again keith and enjoy your
lunch everyone yep