[Music]
third time's a charm um so i've caused
worse outages than myself at work so
this is fine um
and now for our final keynote speaker of
the conference liz fong jones
as a bit of an aside as follows of liz
on social media we'll know she's had
something of an odyssey over the last
couple days having to fly from sydney to
the west coast of usa to get some urgent
document documents notarized then
getting back on a plane to sydney
returning yesterday morning and it all
seems to be sorted now about liz liz is
a developer advocate labor and ethics
organizer and site reliability engineer
with 16 plus years of
experience
she's an advocate at honeycomb for sre
and the
observability communities
and previously was an sre working on
projects ranging from the google cloud
load balancer to google flights
liz will be talking about cultivating
production excellence
over to you liz
good day everyone and thank you for
joining me um i wanted to acknowledge
that i am living on the lands of the
gradual people and i also wanted to
acknowledge that there are
things happening in the world right now
and that um tonga is under threat from
climate change and from the volcano that
has just erupted
okay let us go ahead and begin
so today i'm really excited to be
telling you about all the lessons i've
learned from the past
from the past 15 years of my experience
working in the field of site reliability
engineering at companies large and small
and how we can take those lessons and
make our workplaces more humane for us
and to make our systems more operable
and reliable
so
those of us who are attending lca are
very familiar with the idea of operating
in production
but some of our colleagues may not
necessarily have that same notion so to
get everyone on the same page we have to
orient around what the objectives are of
the business what the objectives are of
the code that we are writing
and fundamentally we're writing code in
order to solve problems but
the problem is that just landing the
code in the main branch of get does not
necessarily mean that your job is done
that there are so many additional steps
that you have to complete in order to
make sure that your services are
delivering value to end users
because if your change has to be
repeatedly rolled back or it takes
months to push out you're not actually
delivering that value until months later
another challenge that we frequently
encounter is that production is
increasingly complex
that we have all of these challenges of
scale and and and uh
and and reliability needs and future
needs and we've developed these layers
of infrastructure to try to
make it easier
but we've also made it hard to fit that
all into the head of one individual
person
that with microservices the theory was
that we could decouple our services from
each other and decouple the release and
launch cycles so that each team could
ship independently
but that now means that a service three
layers away from you can break you in a
way that you didn't anticipate in a way
that was not a failure mode of a more
tightly coupled system
we also have problems with big data
where big data can result in increasing
demands upon compute and networking and
we have to be able to scale our systems
to meet and address that demand
so some of the systems that we're
creating add complexity in order to try
to solve some of these challenges
but when the systems don't work exactly
as planned for instance as the audio
feed this morning did not necessarily
work right it takes a substantial amount
of effort to try to figure out what's
going on and how do we fix it
so that's what the subject of today's
talk is how do we fix our problems with
more confidence more quickly
but let's start with defining what the
problem is
what does uptime
mean when i was a wii game developer
when i was about 16 or 17 years old
i worked at a small game studio and we
had the database and we had the game
world server and if those were up
everything was fine but if they were not
up everything was 100 down
but that's no longer the world we live
in today where you may have hundreds or
thousands of linux servers that are
running as vms on amazon's
infrastructure or in azure or on gcp
and we can't wait until our customers
complain to us and ring us up on the
phone and say i'd like to cancel my
subscription right like we would like to
be a little bit more proactive at
detecting when our customers are having
problems without necessarily getting
paged every single time a single server
flaps
there are all of these demands for
reliability
for features and for
and for investment and future
scalability
it's a lot to deal with
and honestly as someone who has been in
the role of a hero in an organization
who's the one person holding up the
reliability on her shoulders
it doesn't work not for not for you know
a decade two decades you get really
tired
so what strategies can we employ in
order to make our services more reliable
and friendlier for the people
so as miles introduced me yes hi my name
is liz and i'm a principal developer
advocate at honeycomb
and honeycomb is a company that aims to
help make developers
happier and more productive through
improving observability into production
so here are some of the lessons i've
learned from both working with
honeycombs clients as well as in my
previous life at google on the customer
reliability engineering team as well as
on various other esri teams at google
the lesson that i learned is that
heroism doesn't work and that we need
different strategies
and that those strategies cannot
necessarily just be buying tools and
fixing to fix things
and i know that this is ironic because i
do work at a company that sells
developer tooling and here i am telling
you don't buy devops right why is that
the answer is that when you have all of
these technologies that you're trying to
integrate together that you're trying to
make work together sometimes you're
adding to your workload rather than
reducing your workload
when you actually try to glom on all
these things that are supposed to make
your developers lives easier
for instance
you get told to write more tests but
what happens if you add more tests and
add more tests and add more tests and
they start flaking all the time and then
people ignore the red build because oh
that's just flaky tests i'm just going
to merge anyways right congratulations
you've just made your problem harder not
easier
or let's talk about the drive recently
to kind of encourage people to push to
production quickly recklessly almost
right what happens if you're continuous
integration and continuous delivery
that's meant to ship [ __ ] fast
ships [ __ ] fast
oops
or
those of us who have been around the
community for a long time you've
probably run rm-rf rf
a couple of times before
right now imagine doing that to your
entire production infrastructure well
congratulations you can do that now with
infrastructure as code you can delete
your entire production infrastructure
with one stray commit
whoops or let's talk about kubernetes
not everyone needs kubernetes it's a lot
of complexity and it's only worth it to
solve certain problems
but the number one thing that i see
people getting wrong
is adopting the idea of production
ownership or let's put everyone into
pagerduty
and i think that that can have some
negative side effects let's talk about
them
when you put everyone on call for
systems that they are not prepared to
run right those of us who have scar
tissue from years and years of being
around production we can handle to an
extent um being paid at 3am
but let's suppose you're a brand new
engineer and your first experience with
on-call is being paged at 1 am 2 am 3 am
multiple times per week
and
you eventually are going to say take me
off this on call rotation or i quit
right
it's not a happy situation if you have a
system that is continuously generating
noise
and even when you do try to debug it you
don't know heads from tails where do i
get started
and if you have dashboards those
dashboards are often a source of
technical that in and of themselves
because they're chasing your last outage
there are you know 20 different graphs
on 20 different pages and you're trying
to figure out which line wiggle the same
line as a southern line right like
this isn't actually helping you solve
the problem all that's happening is
you're spending time in a state of
cognitive overload while your customers
are suffering and incidents are taking
forever to fix
so finally you pick up the phone and you
call someone like me or like miles and
and we you know tell you oh you just
frob the thing restart it it'll be fine
in the morning right
except for someone like mirror miles
like we've gotten really tired of
getting woken up every single week at
least once a week even if we're not on
call for years on end
and finally
you go back to sleep you wake up in the
morning cup of coffee in the hand and
you realize that you can't actually push
fix because someone has broken the build
overnight
and even though each individual set of
unit tests passes
the integration tests don't because
there is a problem in the gaps between
our services that is causing things to
flake
so this is what we described in the
field of site reliability engineering as
a state of operational overload
where there's no time to do projects and
even if you did have time to do projects
there's really not much of a plan of how
do i get myself out of this situation
how do i spend one or two spare hours in
order to chip away at this pile of
technical and operational debt
so this can feel very very draining this
can feel stressful right like many of us
have been here
and it can feel like you're barely
hanging on to the edge of the cliff with
your fingernails
so how do we make this better what are
we missing
and why is it that tools don't help
well i think the thing that we need to
focus on is that people are the ones who
operate your systems that you cannot
have a healthy system without healthy
people standing behind it
for instance let's take me as an example
every single time i've probably given
this talk several dozen times now
every time i go through this section my
heart rate goes up right like i have all
that packed up anxiety about production
outages of getting paid at 3 am
and it gets really challenging right but
i need to take a moment and focus on me
i encourage you to focus on yourself
right like to take care of your needs in
my case i'm going to take a deep breath
and i'm going to take take a drink of
water
and that will help me deliver a better
talk in the same way you should make
sure to take a deep breath everything's
going to be fine and drink water even
when you're in the middle of an outage
focus on the people first
so much better right
so you cannot run a healthy system
without healthy people
and that means that tools cannot fix a
culture if your people are not healthy
right
that your tools can help automate
processes that you already are inclined
to do um but it can't fix a culture
where people are blamed where people are
on call all the time and stressed out
right like
tooling that generates more alerts is
not going to be helpful there
so what should we do instead
i argue that we should invest in people
culture and process
that investing in those three things is
how we get out of this mess of
operational overload and production pain
this is what i call production
excellence the idea that our system
should be more reliable and friendlier
that we shouldn't have to feed computers
with the blood and sweat of human beings
that we have to plan intentionally and
figure out how are we getting there
what milestones are we aiming for
and what key performance indicators are
we measuring
we also have to think about getting the
rights of people into the room
and this means
a lot of the people who should be in
this room are not necessarily in this
room uh even at lca
that we need to involve everyone from
the business level to the support teams
right we need to involve people who are
even in sales right like we need to
involve people all across the spectrum
in order to make sure that we're aligned
about delivering value to customers in a
sustainable fashion
so that means that we need a
psychologically safe environment that
people need to be able to contribute and
that if you cannot feel safe to
contribute none of the rest of this talk
is going to make sense
but let's talk about the four key
elements of production excellence that
will make your life easier
first of all in order to operate an
excellent system you need to know when
that system is too broken
when it's outside of the bounds of the
normal expected system behavior
secondly you need to be able to debug
those failures in order to restore them
to working order
third you have to be able to collaborate
across multiple teams
in order to solve these problems across
multiple microservices
and then fourth and finally you need to
close that feedback loop we don't live
in the world of groundhog day it's not
okay to repeat the same outages over and
over so we need that feedback loop to
solve the longer running issues that we
face as engineers
so if you do those four things you'll
have a system that is much friendlier to
your human beings who are operating the
system
so why did i say
no on our systems are failing too much
right why did they not say no when our
systems are failing at all
the answer is that
if you got alerted every single time a
packet dropped in the uh in the fiber
optic channel between here in la right
like you'd get paged all the time right
our systems are always failing in small
microscopic ways but we build redundancy
into our systems in order to try to
solve those problems
so instead of thinking about are the
systems failing at all right like is the
lawn outside on the domain outside where
i am right like is every single blade of
grass on that lawn green no
but overall does it look green enough
yes right like it's soft it's blush i
can go lie down and and have a picnic
right like that's that i think is good
enough
so instead of saying you know is the
system failing at all we need to talk
about is the system failing too much
and that means we need a quantitative
measure of that that we can use to
operate the system
this is the idea from site reliability
engineering at google the idea of the
service level indicator and its
companion the service level objective
a service level indicator and service
level objective
what they represent is a common language
between us as engineers our business
stakeholders and our customers about
what the expected level of reliability
is for our services
and we need to think about our services
in terms of the broader context
what is it that our customers are trying
to achieve
what is that critical user journey that
people are trying to do
for instance maybe it's people buying
tickets to attend the outdoor film
festival in sydney
or maybe it's someone who is trying to
place an order for uh for computer parts
from scorpiotech right
either way there's some action that a
customer is trying to achieve and we
need to make sure that we're measuring
it to ensure that it is of a
satisfactory level of performance and
quality
so we need to understand is a given
interaction between our customers and
ourselves
creating a good or bad experience for
that customer
how do you know
well this is where product managers and
user experience researchers and customer
success can really help you out to help
you understand what differentiates a
successful from a failing transaction
or if you work in a field where you are
able to experiment on your own systems
and where you use those systems
yourselves my main piece of advice is to
do chaos engineering to deliberately
slow down transactions
so you can understand where does it
start to feel sluggish and then set your
threshold just shy of there
because our goal is to give machines
rules for deciding whether customers are
having a good or bad experience
and sure we're all used to trans-pacific
latency
but we can in general say you know hey
like
a request is successful if it responds
in you know maybe not 100 milliseconds
but 300 milliseconds right that's fast
enough and then it has to return a
success it can't return a fast fail is
not a success to us as a customer
and then we can think about what's the
total denominator right how many
eligible events did we see
how many transactions were attempted
and i don't just mean like you know just
take everything comes into your load
balancer
because there's often health check
traffic there's often denial of service
attack traffic right like we need to
think only about the real customer
traffic not synthetic events
and then we can compute our actual
achieved availability the number of good
events divided by the number of eligible
events
and we can set a target that we can
measure that is our service level
objective
so the service level indicator earlier
helped separate good events from bad
events but the service level objective
helps us decide over a longer period of
time what percentage of the events in
the sli should be should be
successful so for instance
if the website of that i'm operating was
100 down today
but i came to you and said you know what
but tomorrow we're going to be 100 up
you'd laugh at me right
so it turns out that human beings have
memories longer than day so you need to
think about measuring reliability and
setting goals for it over a longer
period like 30 or 90 days
we also need to set an appropriate
target percentage
for instance i might aim to have 99.9 of
events be successful over the past 30
days where an event is successful if it
was served in less than 300 milliseconds
and with an http response code of 200.
so
why did i not say aim for
99.99999 why should i not have infinite
nines right
the answer is
that every nine is additionally costly
right
and indeed we see this in australia with
nbn right
that's sure
we could spend
trillions of dollars
putting putting you know dozens of
satellites into space
in order to fulfill
99.99999
population coverage
or we could put two satellites into
space and say you know what we're
covering four nines of the population
that's good enough right
there is a cost to each incremental
failure that you prevent
and a incremental benefit associated
with that individual failure that was
prevented
at some point those two lines cross over
so it's up to you to figure out how
critical is a service and then decide
okay what is the level of reliability
that we're aiming for and what's going
to be cost effective
now what do i do with the service level
objective right all this is abstract
right setting goals why do these goals
matter how do i make my life easier as
an operations engineer let me tell you
why
the answer is going back to that first
point i made about operationally
overloaded engineers
often when you get paged at 2 am there's
no actual end user impact it's just
something paging you because the disk
usage got to 90.1
90.01 oh no such an emergency right
were any users impacted
no was this worth waking someone up for
no so we can instead think about using
our measurements of what the reliability
level experienced by our customers is
and use that to decide is this important
enough or not to wake a human up
so for instance let's talk about
the idea of the error budget which is
the inverse of your service level
objective if i'm targeting 99.9
reliability
that means 1 in 1 000 requests are
allowed to fail
so if i'm serving 10 million requests
per month that means 10 000 requests per
month can fail
and i can have a different degree of
urgency based off of the error rate
right it's no longer an emergency if one
out of one requests fails at 2 am
right instead i ask
if i continue having this number of
errors how many hours will it be until i
run out
if i'm serving a thousand bad requests
per hour and i'm allowed to have 10 000
bad requests per month i'm going to run
out in 10 hours that's probably worth
waking someone up for
but if i'm only going to run out in days
it can wait until the next business day
it's okay
and the beauty of this is that it
catches subtle failures as well that are
not entirely up or down
here's an example of an actual failure
that we encountered at honeycomb when we
were first turning on service level
objectives for our own services
we started having an intermittent brown
out where two percent of our traffic
would fail for 20 minutes at a time
repeating every three hours
now it turns out that there is a memory
leak and all of our servers had started
at the same time and were crashing and
stampeding at the same time but we
didn't know that at the time
but what we did know was that our
service level objective fired saying
that too many users were experiencing
problems uploading their telemetry to
honeycomb
whereas our conventional black box
alerts were not able to catch the issue
because of the fact that they wait for
two consecutive
probes in a row to fail and we did not
have two consecutive probe failures in a
row
so besides making your alerting much
higher fidelity the other thing that
service level objectives do is that they
help you navigate the tension between
product development and reliability
instead of having product managers
saying ship more features i want more
features right like now you have a
framework to treat reliability as a
product feature
so for instance if you're having too
much reliability that can almost be a
bad thing because people's expectations
will ratchet up and your competitors
might be innovating faster than you if
you're focusing on delivering
reliability at the expense of product
features
so you might decide i'm going to do an
experiment and i'm going to
roll something out to one percent or two
percent or five percent of my users
knowing that even even if it fails 100
you know of five views of five percent
of users you can roll it back within
five minutes and you're not going to
damage that many people's experiences
but conversely if you had a set of
really bad outages recently
you can think instead about how do we
invest in more reliability how do we
make the business case right the answer
is we're failing our objectives
therefore our customers are at risk of
no longer trusting our service no
feature that we ship is going to matter
unless we bring reliability back up to
par
now you don't have to have super complex
slos to start with even your load
balancer logs are great because they
help you understand
what is it that your customers are
experiencing from a neutral-ish point of
view right it helps you measure what you
can
in order to deliver a better customer
experience and actually put yourselves
in the shoes of those users
and over time you can iterate to make
those slos better right like you can
start incorporating things like real
user monitoring from your client devices
right like there are a million things
you can do
but start by meeting your user needs and
then if users are having problems that
your oslo isn't firing you need to
correct your slo
so slos help you reduce alerting noise
and really help you hone in only on what
matters but that's only half of the
story to me why
because we don't just need to focus on
the issue of operator burnout we also
need to focus on restoring customer
experience as quickly as possible when
we do confirm that there's a genuine
issue
so let's talk about how that works
our outages are never exactly identical
right that we
always have if we're doing our job as
engineers we're always going to have
these new kinds of failures that are
happening that we didn't anticipate
because you wouldn't have written the
bug in the first place if you knew it
was going to break so therefore by
definition anything that goes wrong in
production is going to be something
unexpected
and not only that
it may be something that is challenging
or hard to reproduce in staging because
staging is not production
right staging doesn't have the same
scale and it's a futile waste of money
to make staging look like production
we have to be able to debug things as
they're happening live in production
rather than waiting weeks to reproduce
them
and we have to avoid creating these
silos between different tooling across
different uh problem domains services uh
environments right like we have to be
able to use the same tooling to very
quickly iterate and understand what's
happening
now the thing that i've discovered is
that in my 15 years as a site
reliability engineer
when we have an outage when something is
bumped in the middle of the night
the thing that takes the longest is
figuring out what do we think is going
wrong and how can we verify that as
quickly as possible that's what takes
the most time
or kind of going to um fighter pilot
school for a moment right like there's
this idea in fighter pilot school of uh
orient of orient observe decide act
right the ooda loop
so we have to think about how do we
accelerate that orient and observe part
as quickly as possible
how do we actually explore that data in
order to ask new questions rather than
just leafing through the existing
dashboards that showed us the questions
we thought to ask before
all this is to say that our services
must be observable
in control theory observability is the
ability to based off of the outputs of a
system in for its inner state
but i prefer to in the systems
engineering context of computer systems
think about
how do we ask and answer unknown
unknowns things that we didn't
anticipate would break in the first
place
this requires us to be able to examine
the events that are happening inside of
our system in their full context to
understand properties like
you know which request is this from
this happens to be some of the innards
of our uh query engine
and we can understand things like
for this query which service issued it
um you know was it an end user was it
our uh was our alerting service
and where do we get slow right did it
get slow waiting on aws lambda did it
get slow uh reading files off of disk um
did it gets did was it blocked in cpu
and which user is it right like not not
just kind of mushing all of our users
together but instead thinking about how
can we break apart and understand the
behavior of each individual user in
isolation
we have to be able to explain the
variance we have to be able to
understand
what separates a good user experience
right a succeeding sli from a bad user
experience a failing soi
so for instance this is one way that you
might visualize this data right to look
at how can i see what dimensions are
different between the succeeding and
failing requests
but even better yet why should we have
to do this at 2am when we're at our
worst
i do not think that we should need to be
doing debugging at 2am
it's about mitigating impact first
as long as you're collecting enough
telemetry along the way and you don't
have to catch it live in the act
then you can roll back the bad release
drain the bad data center and then you
can debug it sitting cup of coffee in
hand at 9am instead i think that's a lot
better
but to me observability is not just a
benefit to the break fix it's not just a
benefit to ops it's a benefit to
everyone
because observability helps us gain
better confidence in what our code is
doing in the first place why are tests
failing
or why is it that our deployment loop is
taking two hours to run where it
previously took 30 minutes
or what are these users actually doing
with these features that we released
last monday which users are making the
most use of those features
and do i have any single points of
failure in my system or circular
dependencies that i should be working to
fix over the longer term
those are all use cases that i think
everyone can benefit from not just the
people who are on call
the other misconception that i often
hear is that observability is logs
choices and metrics that it's kind of
these three pillars
and that's not true to me
observability as i described is a
capability it is in fact a
socio-technical capability it's one
where the humans have to be able to use
the tools appropriately to be able to
answer their own questions
and that means that it should be as easy
to add
the necessary debugging and
instrumentation
as it is to add a printf to bug line
because as much as i have done a lot of
printf debugging and got here before
like that's not a scalable method for
working in prod how do we make it as
easy to add good observability as it is
to do a printf
how do we send that data and store it in
an economical fashion and finally most
importantly can you actually query that
data if you cannot actually query that
data and answer your unknown unknown
questions
you don't have observability you just
have a very expensive dev null
so hopefully this elucidates for you why
slos and observability go together
because slos help you understand when
things have gone too wrong and
observability helps you piece together
why so you can remediate the issue
and then finally deliver a lasting fix
to it
as soon as it's convenient to you
but we need to talk about the two other
elements of production excellence
besides slos and observability we need
to talk about collaboration
because the reality is that no
individual human debugs and solves
things alone no individual human is on
call for services 24 7 365 that just is
not sustainable anymore
people deserve to go on vacations people
deserve to uh retire right like people
might might
be out on sick leave because of covid
right we have to be able to allow that
slack in our system
so how do we make it possible for
everyone to collaborate and debug
together how do we raise every human
being to the level of the best debugger
on their team
and how do we make sure that information
is not lost as we cross organizational
gaps
we have to think about a broader set of
users we need to think about the
customer support agent or the uh or the
cloud provider as equal customers to our
engineering teams as far as who is a
client of our debugging
and are we working together have we
practiced this at 3 pm and not at 3 am
are we doing game days right are we
doing disaster drills so that we work
out those kinks rather than doing it
live
and we need to make sure that we have
you know
not service selfishness right service
ownership does not mean selfishness
we have to make sure that people
understand that
we are working together and that
hoarding knowledge inside of your head
you know sure
many of us have read the uh bastard
operator from hell the bo foh comics
right
but you know that's not a model to
follow right it's not okay to have your
job security tied up and how much you
alone know
a modern system is built up of people
who are working together
my colleague jessica care calls this the
idea of semanthicy the idea of learning
systems that are learning together made
up of humans and machines they're trying
to do better and to iterate and to work
better together and build better tools
over time
we have to be able to work together and
lean on our teams right we have to be
able to hand off on-call
responsibilities we have to recognize
that on-call is a team level
responsibility not an individual level
responsibility
if someone is a new parent and they are
having sleepless nights because of their
child don't put them on call right
well i do believe that every developer
should have some exposure to production
it doesn't have to take the form of
on-call it can take the form of working
tickets during normal business hours
right it just involves some form of
feedback loops that you are exposed to
the consequences of the technical
decisions of your team
regardless of whether it's during
working hours or not
and we have to document things so that
we're leaving
as my colleague tanya riley says we're
leaving cookies for our future self not
traps
we have to keep things organized and not
just you know have misleading documents
that that lead us astray that have
gotten out of date but we have to at
least have these common patterns right
of
what is the service for how do i shut it
off how do i update it what happens if
this breaks right those are kind of the
key things that we need to remember
and we need to make sure that you know
you don't have that single person like
miles or myself you know who has been
that hero for ages and ages right we
have to share that knowledge and we have
to make sure that people are able to
collaborate and having common sources of
data
one common source of data that i want to
point out is open telemetry
which is a vendor neutral standard that
is developed by end users and multiple
vendors working together in order to
make sure that when you produce
observability data that is a single
source of truth that you can pipe to
anywhere you like
without experiencing lock-in and that's
really powerful because it means that
you no longer have data that is tied to
one specific platform that for instance
no one else besides one team can access
or that you know you lose your data if
you wind up having to uh change
providers right like
having the shared understanding and
ground truth about what's happening in
our systems is really powerful
we also have to make sure that we're
rewarding curiosity and teamwork right
that instead of saying you know uh hey
jess i can't believe you didn't know
that right like do you think just gonna
ask me another question after that no
she's not
so instead we have to say things like
you know hey
thank you for asking the question i'm
sorry that the documentation wasn't
clear let's work together to document
that better so that no one else has to
stumble into that again
so by rewarding curiosity and teamwork
that really creates a healthier culture
of collaboration
but it's not just about collaborating
with your current colleagues it's also
about collaborating with your present
past and future self
that often you know we pull up get blame
and get blame says you know who that
idiot was who wrote that code that that
crashed
yeah it was me it was me again
right so you have to leave yourself
breadcrumbs and be kind to yourself
right to make sure that you're
documenting things for future you so
that you're not cursing future you're in
the future but be kind to yourself about
it
but while i did say earlier that
outreaches are not exactly the same
there are common patterns that we often
see in outages and it behooves us as
engineers to think about closing
feedback loops and eliminating some of
the more common categories of outages
and we can think about employing risk
analysis to help us become more
proactive in approaching uh our failure
cases
so i have out my window the sydney
harbour bridge and let's suppose that um
there's been uh some lack of maintenance
and cars are falling through the road
bed and the sydney transport trains are
falling through it's no good right and
also we all know um especially in light
of the volcanic activity that at some
point you know we live along the pacific
rim the earthquake's gonna happen right
and finally it's probably long past time
that we took down the christmas lighting
on the sydney harbor bridge right
which one of those three things do we
address first
we fix the cars that are falling through
the road bed and the trains that are
traveling along the tracks they're
leaving nowhere right like we fix that
first because it's having the highest
most critical impact on our users
so we need to think about what's the
frequency and impact of a risk to users
now let's move back from the physical
world to the computer world
how many of you have that shutter down
your spine when i say
the my sequel database
right
that is a single point of failure and we
know
as systems engineers that the my sequel
database is going to fail at some point
in the next year right it's going to
happen
you don't often have control over
frequency but what you do have control
over is impact so how can we make the
mysql database failing
not take out everything well
you could shard the database right you
could um have it only take down two
percent of users data at a time if it
goes out
you could decrease the amount of time
that it takes to identify that it's a
mysql database
you could also decrease the amount of
time that it takes to fail over right
you could have a hot spare running
instead of needing to restore from a
backup right that cuts the time from you
know two hours to restore the backup to
two seconds before the uh the watchdog
notices and kicks things over to the
replica right
so think about what are the risks that
are the most significant what's
happening with the highest impact
impacting the highest percentage of
users and lasting the longest
and then trying to move that needle
trying to work on on reducing those
significant impacts that doesn't mean
you have to eliminate them entirely but
you can think about ways to mitigate the
impact that they're less bad
and how do i decide what's a bad risk
what's not acceptable well the answer is
we just defined a service level
objective right
so if you have a service level objective
in mind then that helps you figure out
okay i'm allowed to have 10 000 bed
requests per month
this one failure case for instance the
mysql database is responsible for you
know in expectation it's going to cause
5000 failed requests per month
that's too high because that doesn't
leave us enough room for our unknown
unknowns for things that we didn't think
were going to happen
so there right there that's your
business case for being proactive for
fixing those issues before they cause
you to burn through your entire error
budget
because it's not possible to operate a
sustainable error budget
if that error budget is already filled
with known things that is going that are
going to break
and finally this also helps give you a
framework for how to approach your
retrospectives from incidents
many of us when we create retrospectives
kind of create this list of 10 things
that would have prevented this specific
outage right it's not the right way to
think about it instead think about what
were the highest impact things that
resulted in
an outage or an outage similar to this
being able to happen in the future right
pick those one or two most important
things
based off of
the frequency and the blast radius
and those are the ones that you should
actually fix and you can put the other
ones on a bug bash list or on a or a
hack week list right don't waste your
precious production improvement time
doing chrome polishing
but i want to call out two things
first of all that if you have a lack of
observability that is systematic risk
that if every algae just taking half an
hour an hour two hours three hours even
for people to figure out what's even
going on what services are impacted
who's impacted like what do we think is
happening right like
that is time that your users are
suffering
and if you shrink the amount of time
that it takes to debug things
and know like you know we don't believe
in the idea of mean time to recovery but
certainly you know you can think about
the number of dropped queries right as
users are experiencing pain as you're
figuring out what's even going on right
so better observability is something
that impacts every item on that list of
risks
and similarly if you have a lack of
collaboration right if i'd talked to the
av team just now and i and i'd said
something like you know can't believe
this get your [ __ ] together how could
this possibly really
do you think that's going to make them
work better no that's going to make them
work worse right like
being friendly being supportive right
like that's how you ensure that people
report issues early that people
collaborate that they're transparent
in 2018 google cloud had a large outage
that impacted global networking for
about 45 or 50 minutes
do you know what caused that edge to be
solved in 45 50 minutes and not three
hours
the developer who had pushed the code
change
that had inadvertently tickled that
outage
they raised their hand and they said i
think it might be my change and i'm
already reverting it instead of cowering
and being in fear and not speaking up
for fear of being fired right
good collaboration decreases the amount
of time that outages
take so you don't have to be a hero in
order to have a
system that is production excellent
you just have to follow this recipe of
those four things
and i'm not just talking the abstract
i'm talking about what i've learned from
my time at honeycomb
that we're today about 40 50 engineers
but when i started we're 10 engineers
right and we're competing with companies
that are 10 times their size
and we managed to pull it off only
because of some of these factors of
production excellence of feeling like we
have the freedom to deploy on fridays to
deploy confidently
but also to have the responsibility to
look after changes after they go out
we do not push and run right like
it's not safe to push at 6 pm and walk
out the door and go get a and and go get
a beer right
no matter what day of the week it is on
the other hand if it's friday morning go
ahead and push as long as you're going
to be awake and around to look at it
so you can see here that we push up to
14 times per day
and we've done this all while traffic
has gone up three to five times in a
single year
covid has been really interesting for a
business because more things are moving
online
we have had basically a tripling of the
amount of right workload of the amount
of telemetry data coming from our
customers into honeycomb
and we've also had three times as many
people who are asking us questions
right people who are trying to
understand the behavior of their systems
so we've had to add all these features
and scale out our system and keep it
reliable at the same time
and how did we do that
well
we defined our slos
we thought about our areas of risk
and then we designed experiments to
validate that that risk was not there
and to fix those risks if we found them
right that's how we approach this
so we actually practice slos and
honeycomb and our slos reflect the user
value that we provide
remember when i said earlier that we're
a observability provider right what that
means fundamentally is we are a big data
platform right we ingest telemetry in
for instance open telemetry format
and we have to break apart that data and
provide kind of almost an index on it so
that it can be quickly retrieved and
queried later
so therefore our slos are shaped like
those user journeys that if you view the
home page 99.9 of the time it should
load quickly enough within 250
milliseconds
but that it's okay for some queries to
fail even up to one percent of arbitrary
queries that those could take longer
than 10 seconds right like because the
idea is
you might be able to hit retry right
there that's okay
but the one thing that we do not mess
around with is user data coming in
because we know we in general have like
one maybe two chances to get it right if
you happen to have a buffer and retry
right that if we drop customer data on
the floor that's going to result in a
graph in a divot in the graph of every
honeycomb customer in perpetuity
so that means that we have to think
about how do we adhere to such a high
slow right
right that's 99.99 that's 4.3 minutes of
violation a month that can be really
hairy
so how do we stay within slo
the answer is the state accelerate state
of devops metrics right which tell us
that if you deploy on demand multiple
times per day it's safer to do that
right
no amount of qa checking is going to
find these big giant issues so it's
better almost like riding a bicycle the
faster you ride to a degree the more
stable you are because you kind of have
that spinning gyroscope right
that the quicker the feedback loop is
between you pushing it to change into
main and it's landing in production the
more likely it is that you'll remember
what was going on and have context to
figure out what's happening
and that if there is a problem right
like you'll notice that the dora metrics
say that 15 of companies
or 15 of the changes pushed by elite
companies can fail that's okay the thing
that distinguishes them is that failures
are not expensive that they can roll
back in minutes or at most an hour
rather than than days
so for us
it starts with lead time we think about
how do you actually
take less than three hours from the time
that you're sitting hands on keyboard
writing the tests making sure they pass
to having it running in production
and the answer is we keep the build time
fast less than 10 minutes per build and
every time it gets higher than that we
do some debugging to figure out why
we automatically push once an hour and
the reason for that is that we are
trying to keep the number of commits per
build artifact low
and again to keep that lead time very
low
and
while it is true that you know maybe
five percent of the time our changes
don't work exactly the way that we
anticipated
we only have about one in a thousand
changes fail in a way that we cannot
quickly remediate right that require
actually doing a fixed forward or a
rollback
rather than just doing a flag flip
and that means that we've optimized
around that path of making things take
as little time as possible to repair
so what this means is that in practice
we add instrumentation right we add open
telemetry spans as we write our code so
we can better understand that behavior
both as we're coding
writing our tests and also as it reaches
production
we don't do kind of heavyweight
you know selenium tests we do dom tests
to kind of serialize and diff the dom to
make sure that it's good enough
and we make sure every major change has
the ability to turn on and off so that
we decouple a release from a deploy
and we really prioritize making those
unit tests fast as i said earlier a
build time 10 minute or bust right and
if it takes longer than that we're doing
some digging to figure out why
and also human beings prioritize doing
their reviews as quickly as possible
said people don't drop standing floor
and people are not blocking on each
other that doesn't mean you interrupt
each other all the time but like i'll
check once an hour to see whether there
are code reviews waiting for me because
i know that that's blocking someone from
getting their work into production
but what we don't do is we don't kind of
hold things in the holding pen forever
right as soon as the tests are green we
merge
and not only do we merge we push the
latest green build once an hour we
automatically push through each of the
environments in sequence
but we have the ability to stop to kind
of pull and on chord to stop releases if
we think that there's a problem so
that's kind of how we keep the assembly
line of production changes rolling out
but the most important thing that we do
is that we observe real customer
behavior and production
that we have a set of environments not
just for customers to observe their data
but for us to observe what's happening
inside production how are customers
experiencing honeycomb
and are there any problems right like
for instance how are people actually
using the feature that we built how fast
is it how performing is it
and we also have to be able to observe
dog food so of course we have a third
environment for that because it's
turtles all the way down
and that's how we have 40 engineers
deploying 18 times per day
but it's not just kind of product
deployment we also have applied that
same mentality to our to our
infrastructure
so for instance we think about using
terraform and chef and kind of all these
lovely tools to manage our linux vms
every day
and we apply these approaches of
continuous integration right
so for instance we use terraform cloud
to automatically push the latest screen
build so we can't drift out of sync
we use feature flags to handle things
like you know hey if i'm behind i can
toggle a feature flag and terraform to
stand up a catch-up fleet automatically
and i can quarantine bad traffic that's
causing crashes or that i want to
profile just with a feature flag in our
terraform code
and that really helps make life a lot
easier for us as as people who are
responsible for the platform who are
responsible for making it possible for
the product to move fast and scale on
top of our infrastructure
but we don't just design an abstract we
actually validate we actually test are
these things working correctly
so we experiment using our error budgets
now when we talk about chaos engineering
we're not just talking about enrolled
chaos
we have a goal of the experiment in mind
what are we validating
and is there a stop button is there
ability to pause or reset an experiment
if it's causing a problem
so we'll use feature flags for instance
to control this sort of thing
in the event of our persistent services
we have to do a lot of work in order to
make sure that the persistence
mechanisms work as we anticipate
because about half of our microservices
are stateless but half are stateful and
we have to treat them as separate
classes
now there's a lot been said in the past
about kind of uh you know chaos monkey
of automatically restarting stateless
servers i'm not going to hone in on that
too much what i want to hone in on is
our stateful workload
where we have to be able to tolerate
things like
a
individual kafka broker going away or
one of our indexing workers going away
we have to be able to validate that we
are able to do deploy safely that amazon
is able to terminate our instances
without causing failures to our
customers
so when we have these kind of
long-running storage instances that need
data integrity and consistency how do we
manage that
well the answer is we test these
failover dances we test the system to
make sure that in practice it works as
designed
for instance if a kafka broker is lost
are we able to successfully
start reading or writing to the new
leader
and in the background replicate in a new
kafka broker
or if we take out an indexing worker is
it able to replace successfully
based off of a snapshot stored to s3
plus replaying off of kafka
so we do experiments in production
so we restart one server one service at
a time right controlled experiments
we're not restarting all five things at
once we're just trying to test one thing
at a time to make sure
at 3 pm and not 3 a.m for two reasons
number one all hands are on deck right
bugs are more shallow with more eyes
but also number two
you know 2 p.m at least in the us is our
peak traffic time right if a failure
happens during peak traffic that's kind
of the decimal situation as far as you
know the amount of load on the system
and we want to verify we can always
catch up and make progress even under
the most load
and we are monitoring to make sure that
we are not damaging user experience
using our slos and slis
and we are using our observability
infrastructure to debug to understand if
our experiment didn't go the way that we
anticipated
why is it happening and how do we repair
it
and we also need to make sure that we
are not you know dropping telemetry
right like that if we perform one of
these experiments this actually happened
uh last week
a kafka broker failed to come back
correctly after being killed and
restarted
and we didn't have the telemetry to tell
us that the kafka broker was missing it
didn't report itself as missing and
therefore we didn't actually have
telemetry to tell us that there was a
problem so that was a useful scenario
for us to know that that that was a
problem that we needed to
fix and then you verify your fixes right
like if something doesn't go according
to plan you gotta keep doing the painful
thing until it stops being painful
so for instance taking out indexing
workers taking out kafka brokers
guess what we do that once per week
right
we try to not keep things running
forever so that we can validate that if
things do go wrong it's not been more
than a week since our most recent
failure test
same thing with zookeeper right like we
discovered oops
our zookeeper um coordinators
were we were only pulling the first
zookeeper worker
and that if you took out the zookeeper
first worker
none of our alerts would run oops so we
fixed that right
when you de-risk things with the design
and automation it makes it a lot easier
to run a sustainable system that is not
going to page you in the middle of the
night
so we just have to continuously verify
and keep on making sure that our system
is working the way that we intend
and this doesn't have benefits just for
reliability it also has benefits for
cost
why because if you're allowed to
tolerate failures more often
you can do things like adopt spot
instances or preemptable uh instances
because you know that you can tolerate
losing those stateless workers
or for instance
if you know that there's a well-managed
process for replacing worker nodes
that means you can gradually and
incrementally roll your fleet without
fear of damaging user experience
including rolling out arm 64
architecture instances
this is almost an entire talk in itself
but i firmly believe that the future of
solving climate change within the tech
industry at least
is by consuming less power per compute
and when you adopt arm instances you're
playing a direct role in fighting the
amount of carbon emissions that your
services produce
as well as of course saving about 40 40
to 60 on your bill
but not every experiment is going to
succeed and that's okay right you can
mitigate those risks and it's just
important to think about how do i bound
the risk to make it acceptable to do
these experiments how do i make it safe
to do these things and design for
reliability through my life cycle how do
i make my system excellent in production
so
feature flags can help um kind of doing
proactive risk experiments can help
but above all i think kind of you have
to ground yourself in the four things
they said originally right slos
observability collaboration and risk
management if you do those things you're
going to be in a much better place
and always be prepared always expect the
unexpected because we're always working
in the turbulent situations of
production
make experimentation routine and you can
not so jokingly talk about the idea of
oh i'm just going to set a chaos money
monkey running loose on our fleet you
don't get there no from nothing you have
to do the groundwork you have to do the
preparation in order to get there
so
we're all part of sociotechnical systems
as customers as engineers and as
stakeholders
learn from your outages invest
appropriately in observability
and have those conversations about how
we can improve our tools and improve
ourselves and improve our processes in
order to react better in the future
and yes you know tools can help buy the
right tools buy just the right amount of
tools right it's almost like uh that
author who said you know
eat food not too much mostly grains
right i think that's that's true that's
true about about our socio-technical
systems too right like you know run your
systems
by buy some tools buy the right tools
but above all else focus on your culture
first and the rest will follow
so i implore you think about how do you
measure your your reliability levels how
do you actually debug how do you
actually understand what's happening in
production
do you have the ability to collaborate
across teams
and are you actually investing time in
closing that feedback loop and not just
repeating the same outages over and over
like groundhog day
so that's all that i have for you today
and i understand that we're a little bit
behind schedule because of the av issues
so i'm not going to delay you too long
um but if you like and copy these slides
go check out honeycomb.io liz and also
um i happen to be around sydney at least
for the next couple of months so if
you'd like to go for a walk in hyde park
um just uh i'll drop a link into the
conference chat and you can uh just book
an hour with me and i'd love to go walk
around and meet some of you
thanks cheers everyone
thanks liz that was that was really good
again another great speak that uh speech
that uh was looking at the chat and then
we're saying oh this is really hitting
with a lot of people
um really good stuff um i especially
like all those little illustrations
you've got they're fantastic
yes there by the wonderful emily griffin
uh at emily with curls highly encouraged
checking her out
excellent great
anyway so we're about to finish off this
session um we're just coming up to the
first break of the day sorry about all
the delays there we've still got a fair
amount of time
remember that the main sessions start at
10 45
and then the lunch break will be around
12 25 depending on when stuff finishes
the afternoon session starts at 1 30 and
don't forget the conference close at 5
35 where you'll get to you know cry and
cheer and all that stuff so please go
and have fun and we'll see you a little
bit later on