[Music]
hi everyone welcome to the first session
of system and mini conf our first talk
is from alan
shone and it is on the importance of
visibility
so here down excellent
thank you very much simon
so hi everyone uh today i want to talk a
bit about visibility and
it's really a very uh simple
concept um for me uh it's it's a very
basic thing
and uh
like the
the sort of first steps that you take
for anything system related really
and so i've got three particular things
that i'm going to cover uh first off uh
the basics
which is really just the basics uh those
first bits and pieces uh
a bit about
what i'm actually trying to talk about
uh not necessarily what
you might expect
and a bit about why
the second will be a particular
situation a bit of a story
to go along with this
something that has
a bit of background and
hopefully either some lessons or
takeaways
for everyone that's watching
and the third uh incidence which really
is another situation
but a little bit different and hopefully
a bit more
color on the the actual topic of
visibility in general
so to start
we have visibility
what is it
depending on
who you talk to or what you're thinking
about
it's a buzz word really uh
there's a lot of
different interpretations of the one
word and
it doesn't necessarily mean
what
either i'm trying to talk about or what
you might think it means uh
so it's it's not
uh necessarily
what you might see talked about in an
open source or or a public space uh from
from companies and and uh tooling
options
what it should be though is is the bare
minimum it's it's that first step
that
just
knowing that something is there or
knowing
something is happening
depending on the systems that you use as
well it's it's often already available
you might see the aws console for
instance or other cloud providers
you can see every service that's up and
running and so that's
that's
visibility in in a very basic sense so
for for what i'm trying to talk about is
just
that bare definition
the the fact state or degree of being
visible
and this was
purely from the dictionary uh
and it really highlights
just
what visibility
is at least to me
and the way that i think about it
um
distinctly from from other terms
and other concepts
which then means that
it's not observability uh
it's
the monitoring side of things nor the
tooling side of things
depending on exactly what you're
thinking about or
what you're talking about or trying to
do it might encompass
some of these things but visibility in a
strict sense is not these things
like again things like the the aws
console that's that's a tool of some
sort
which gives that visibility but it
doesn't mean that the tool is visibility
uh
and really it's
it comes back to visibility being
just the ability to know that something
is there and that
it's theoretically doing
what it should be doing
at a very bare minimum level
and so with that
we have our situation um
and i i picked this picture here because
visibility is like an onion and
in that same sense visibility is a bit
like an ogre uh
there's a lot of layers and
once you get beyond those first steps
and start digging in deep but that's
where you get into those other terms the
the observability the
the monitoring and all the rest of it
and then you get into a lot more
complexity
but really
being the first step visibility just
gives those those very base
uh insights and understandings that
something is available and is doing
at least
at face value what it's supposed to be
doing
and so for this situation we had a
problem uh
we were already lacking
uh certain
observances and and
visible aspects to
some of our systems
and
what was available
for us to readily implement
was
incredibly expensive
and even though
we could have just implemented it
uh it would have been
something that would have turned a lot
of heads and
probably gotten quite a few people into
a lot of hot water if it had been
enabled straight away
but what we did have
is
a set of scenarios that
we knew we needed to be able to
to make visible so
we knew
certain things
were potentially going to happen if they
weren't already happening and we just
needed to make them visible
and it was really that that thing
of
we know that these are possible but we
don't know that they're not happening
and that was our visibility dilemma of
the time
and so
we we had a solution um
we can build our own tool
which is
always the best idea um
but also not please don't uh
we could make it composable though
where
we
have our certain set of rules that we
already know
but we might have more in the future for
instance um especially as
the makeup of our platform changes
we we need to
make different components more visible
and
by making this particular tool
composable we can split it up and
i guess try a different paradigm for how
we run
tooling
so this particular situation uses a
series of lambda functions
that interact with different
components to
retrieve data massage it and then look
for certain things
at different points in time based on the
different rules for each from each of
the different functions
and this also meant that we could
integrate it with all of our existing
workflows all of our existing tools and
within the rest of our
uh
infrastructure in general
and
what it looked like then was a slack
message
so we would have this
uh set of lambda functions that would do
all of this work and
if anything was found a message would be
dropped into this particular channel
so that way one of the team were able to
investigate it
and so one of the
interesting components was we we kept
track of uh known ip addresses
and these were from infrastructure these
were from
internal usage so like my ip address for
instance
as a member of the team
and we we stored all of these
uh via one of these particular lambda
functions so that way
when we were looking at different
activity
from one of the other functions it
it could request
uh whether or not
a an ip address was known and it would
use this other other function to do that
and so this sort of composable
architecture
it makes sense uh because we could reuse
these things in different ways and
for very different purposes than than
what they were built for
and this was good it worked for us
it
allowed us to have our own rules we
could
work our own way
and it also
meant that we only implemented the
things that we needed
we we didn't end up with
all of this extra stuff
that we just weren't going to use
it wasn't
it was more of a a toolbox of sorts
rather than a swiss army knife for
instance
um
like i've never used a saw on a swiss
army knife
because i'm not sure what kind of wood
it's going to cut but
it's probably not going to be useful for
anything that i would try to try to use
it for
um
there were some additional problems with
this though um
which is always the way when you build
things yourself um how do we know it's
working um
when it comes to things like
external
or external to the business
third-party integrations these sorts of
sas type solutions
there's usually
feedback loops and
there's there's ways to get information
about the current state
of of the uh platform
and so we added some very basic
visibility uh
which
as as you might guess was another lambda
function um or at least a couple of
lambda functions
and these
all
uh
worked within the the aws environment
that we we managed all of this
and would follow the same steps as the
actual tool itself
it would reuse the
the function that sent the message into
slack it would
look at the
known
uh cloudwatch information that we would
generate and do all the processing um ip
addresses and and all that sort of stuff
as well
and so it was great it
was was an afterthought for sure
but
it was done and so that was fine
[Music]
except
we didn't know when things weren't
working
we we knew when things were working and
we could test that we could
take our own actions
to essentially canary
the the processing pipeline and we could
do things that we know would definitely
trigger
something happening
and
yeah we
oh like this this visibility was very
basic uh it it didn't really
test everything um but it worked for
just letting us know
um
and
that was that was all good
except um
yeah we
we didn't test things like the different
integrations themselves
so
as as
this image shows um
we didn't test to make sure that the
slack integration worked fine
and
we didn't know if it ever stopped
working because the only messages we got
from the system were interslack
which is also a problem if slack is
unavailable and that sort of thing but
if the integration itself breaks um
we don't know about that
and so one day
uh one of the team members
uh
noticed this
like hey
why haven't we had anything for a while
like
surely
nothing is running so good that there
are no problems uh that's
usually a good sign that something is
not actually
working
and
that was a problem uh because you don't
really think about it unless there's an
alert
when you look in a channel or
when you're used to having notifications
about things
the absence of a notification doesn't
really
trigger anything you don't really think
about it until this sort of time where
i think it had been about a month a
month and a half
since we had actually
seen anything come from this
and so we did what we did before we we
ran through
uh one of our canary tests and sure
enough
nothing happened
and so
that was
the problem
so
we started digging in we started looking
around we went back through
how everything was set up in the first
place
and we we looked
at all the functions we looked at the
code for the functions we looked at the
infrastructure and
everything was there
the config
hadn't drifted
we we managed everything with terraform
and all of the code was in git along
with that terraform
there hadn't been changes for months uh
everything looks fine we could see that
the functions were running
we could validate that
they executed without errors
and
it looked weird
and
what we
sort of got to the point of was that
well
we know that all of this is there
it's all very disparate but
we know that things are working fine
just that messages aren't coming through
and so
we checked okay well if that's the only
thing that we're not seeing then
maybe we need to update how that
integration is working
and suddenly
it works fine again um
we
had to do all this digging but
we just didn't know what we didn't know
and
a lot of this was was built in isolation
uh away from the rest of the team so we
had to dig we we didn't have
any way to just know where certain
things are or be able to see
how things are progressing
and updating the details was was all we
needed to do
and
then we were back on track
so
the problem was that the the slack uh
integration which was just a web hook to
send the message uh it had been disabled
uh
which is a a wonderful feature it may
have changed actually but it was a
wonderful feature from slack that
whenever a user
adds an integration or sets up an
application or anything like this it's
owned by that user themself
and that also means if that user is
disabled or removed
or otherwise
unavailable to manage the integration
anymore then
that integration is disabled
and so
this web hook had
become unavailable to be used and that
meant that no messages were ever sent
but yeah once we created a new one and
we used
a headless user to do so
so that way there was
a much lower risk of it being removed in
the future
we could update the function and it
would work fine again
and then retesting
and it's all good
i think this is this is another a good
thing to think about as well
when it comes to
how these sorts of things fit uh
how they fit together and i'm sure that
if that original
broken
web hook
had been attempted to be used
there would have been an error message
of some sort and
that would have been a good thing to
return from that lambda function
if we had been able to see that there
was that error message we would have
known very quickly and it would have
been something that we could have done
something about
and we
wouldn't have missed whatever messages
we had missed along the way up until
that point
but
after a period of a couple of days we
were all back on track again
and so the third topic i wanted to cover
was around incidents and
the incidents uh the the sort of typical
things from
production infrastructure or just
infrastructure in general the the things
where you say hey there's
like something is down or unavailable
uh those those types of things and
the the analogy here is
if a tree falls in the woods
and no one's around to hear it doesn't
make a sound
uh and
that's where incidents come in
for for us
um
so a couple of years ago um
all
sort of interruptions and incidents were
very chaotic they were
all over the place they were confusing
and
they typically happened in private
channels
they were
typically between the same couple of
people
and
often afterwards
there weren't clear actions to come out
from from handling that incident so
the sorts of things like
when there's a problem
and we know that there is a fundamental
uh issue
whether it's design or architecture or
even just
bugs in code
there was
not
always a very clear
outcome to say okay we need to
improve or shift
these things so that way we can prevent
this in the future
and
it was because of those things that we
we decided okay we needed to do
something about this we needed to
make it visible
and so we created our own
incident management framework um and
when i say created our own i don't mean
we
took a blank blank slate and started
from scratch um we borrowed a lot from a
lot of the common uh publicly known
frameworks such as
uh the one from etsy and what a lot of
places like i think dropbox and netflix
and some of the other uh big software
companies
their frameworks that they've published
and talked about
the intention here was to to streamline
handling uh those
interruptions
um
but also distribute the workload so make
sure that it's not always just those
same handful of people
make it more visible as well uh
for two reasons i guess um
one was
to sort of
show just how much of an impact it makes
to the team when there are these sorts
of interruptions where
groups of people have to drop whatever
work they're doing to to look at a
critical problem
because if they don't then
there's going to be customer problems or
that sort of thing
but the other thing
that was a really great outcome from
this was was the visibility and
visibility in this case is
uh both an internal to the company but
an external thing to customers
so not only making
say a product manager aware that the
reason
their squad of engineers hasn't been
able to progress is because of the
incidents that have been happening it's
also to communicate with customers to
make them aware of
like the reasons they're unable to use
the platform right now or because of
these issues happening
and
then giving more of an eta and keeping
them in the loop on how everything's
going and that visibility was was really
one of the biggest impactful changes to
come from the incident management
framework
and so the framework itself
um
we
we have it um
in a way that we give a lot of structure
to
i guess live problem solving
so we we have a set of roles and
responsibilities for those roles uh we
have a hierarchy of um
of the way that those roles interact
with each other and we train people
for each of those roles in particular
because
they have different requirements they
they have a different skill set
and
what might work for one role and the way
that you approach
uh working in that role may not work at
all in another role so we we provide
that training
for those individuals when they would
like to or depending on role
uh some roles are more suited to certain
uh
some roles in the business are more
suited to different roles inside the
framework
and that structure is is really
important because it means when there's
like a critical issue where the entire
platform is unavailable
we don't have
a series of executives or or managers or
a group of people
uh
worrying that the world is on fire
we have a structure that gives people a
grounding to
keep them calmer and actually focus on
solving the problem
without
worrying as much
um
and it's a great way to facilitate the
communication it's it's the back and
forth the
making sure that people are aware making
sure that stakeholders know that
this problem we know about it and we're
actively working on it um and
they can get updates uh through through
that process and
uh it's same for customers like uh for
our incident management we publish to
status page
and people can subscribe to email or
rss updates that sort of stuff
and the framework also means that we
involve the right people so because we
have those roles we
have people who are trained in those
orals we can
have very particular say page duty
groups that we can page people
for those roles in particular
and
we also capture actions as a part of the
process and as a part of the
retrospective afterwards we look at
what the problems were that caused the
incident and then we bring those into
our development workflows and we make
sure that we resolve those long term so
that way we don't have those incidents
again in the future
and so
the visibility aspect of this was all
from
the actual implementation
when we first started the incident
management framework it was mayhem and
there was just a lot of confusion
for about three weeks
mostly in the systems team or actually
in a lot of the engineering teams
there was not really a lot of planned
work getting done and that was because
we would
get an incident called we would all jump
in
as we needed to
and as we resolved it there would be
another incident called for something
completely different
and
when you factor in that and writing the
documentation afterwards and then
meeting to discuss
how it went um and then continuing to
train everybody it was it was very
taxing in the beginning um and it also
gave
uh quite a bit of a a hit to the
confidence across the leadership group
because it was just hey
what's going on there's all these
incidents um it looks really really bad
uh and that was purely because it's
visible now um
originally these things were happening
anyway it's just that nobody knew about
it
and so once we knew about it but we
actually had a plan to do something
about it we could handle those things
and then we we get better we continually
improve
we also update the the framework if we
need to if we find
that we have different types of
incidents that don't quite lend
themselves to the way that we're doing
it
we can change that and that's fine
and it works really well now um it's
not really mayhem at all anymore
and
we have a really good feel for the way
that the different roles interact
and uh we have a pool of people that we
can lean on to be able to help
if and when anything goes wrong
and so i want to
talk a little bit about some takeaways
then from these three uh particular
topics in general
um
the first side is the the technical side
of things um
the
the key point of visibility is just that
it's being visible uh and it should be
very basic it shouldn't be complex there
shouldn't be
any
rules engine or anything like that
it's just the bare minimum
making sure that something is running
like having a health check on a service
whatever it is wherever it's running it
doesn't really matter but
having a service be able to
to
alert
when it's not able to do what it's
supposed to do
and you can see like a dashboard for
instance to say like the service is
healthy it's it's working fine
the next thing is to think about the way
that things can fail
so
with our
situation we we thought about okay how
do we know that something's running and
we added some some basic visibility for
that and that was fine but
what we didn't consider was what things
could go wrong
and that was where we missed that aspect
of
the actual message that gets sent to let
us know that something is wrong we
didn't know when that broke
so
being able to send the message was sort
of
the most fundamental part of all of it
if
if the messages can't be sent then
the whole rest of the pipeline doesn't
matter because we will never get any
output from it
so thinking about that is is something
that's
really good at informing
how the rest of the visibility and
beyond journey goes
it's also good to have redundancy
in the visibility side of things so
adding the observability in the metrics
means that you not only say get the
health check for the visibility but then
you can say okay
the services is reporting that it's
working fine
but now
let's check the the business logic to
make sure that it's still
when it takes an action it's actually
doing the action we expect it to do
and
cost isn't always about money
so
while the tool that we didn't use was
prohibitively expensive
that doesn't mean that building our own
was necessarily a good idea uh when you
factor in things like uh the hours it
takes for people to maintain or the
hours it took to write in the first
place um and then the hours for
the few of us that dug into why it
wasn't working
they all add up and
those hours and
the time can be converted into some
monetary value that would be comparable
to the actual tool cost itself
and and the final technical point is
that sometimes standards can be good
we
have ways of running services and
processes across the platform and they
weren't used for this which meant that
the team
that already had a good understanding of
how things function
were left a bit
uh in the dark
really and generally for visibility
uh
it might
be horrifying to make something visible
and
sometimes that's exactly what you need
so with the incidents
making them visible was really important
but it meant that we could actually do
something about it
and with the data that you get from
these types of things
you can make improvements things can get
better
and it's good to understand who the
target audience is and
you can use that visibility then to
build trust
and and get better
cool that so that was
uh
visibility i'm al
thank you
okay thank you very much helen um we
don't have any questions but there's a
few people sharing your stories in the
chat
all right thanks
thank you