[Music]
hello everyone and welcome back from
your lunch i hope you had a great time
uh and hopefully had some good food
uh next up in in the kia ora theater at
linux conference 2022 is mike cohen a
renowned digital forensic engineer and
senior software engineer who described
himself as a digital paleontologist
mike is the founder and creator of
philosoraptor which is an advanced open
source digital forensic and incident
response framework supporting
linux mac os and windows there's so many
thousand digital forensics
uh
today mike is talking through hunting
for threats on a linux host using
velociraptor and its query language vql
take it off
thanks very much thank you so um i'm
really glad to be here today and talk to
you about velociraptor uh and today
we're going to cover some of the
linux aspects of typical investigation
that we see
when we're doing linux incident response
or
forensic investigations to give you guys
a little bit of a taste to
how to do forensic response at scale and
how velociraptor can make that easier so
specifically
velociraptor has a lot of capabilities
we're really not going to even touch on
many of the capabilities that it has
uh and we're going to cover a lot of the
the things in very briefly but because
this is a linux conference um then we're
going to really look at
typical linux use cases
and
since it's an open source conference so
i'm hoping to give you guys a bit of an
idea as to how to join the open source
uh project and contribute and uh and and
use that so um
what is velociraptor and i've spoken
about uh velociraptor in linux conf um i
think last year there was um
a workshop about it so you know we we
covered it in a lot of depth uh but it
is an open source tool uh that's really
designed for um for digital forensic and
instant response
and uh and also alerting and detection
and essentially it's a way of making it
easy for us to manage uh or investigate
uh at scale
so
the the thing that makes velociraptor
really cool is that it has a query
language called vql and vql is really
kind of at the core of velociraptor it
makes it
do everything and we're going to cover
some of the um
some of the things that we can do with
vql today and how we can use that in the
real world um so
you know it's it's just a bit of a taste
um oops
all right so let's have a look at
generally what velociraptor looks like
so we have a velociraptor server usually
we deploy it in the cloud
and it basically it connects with uh
assets which could be laptops or
servers or essentially any kind of
system that runs the agent so we have
support for windows mac os and linux
agents
today we'll talk about linux
but the agents are connected consider
persistently to the server so that means
that we can investigate each of these
agents
with you know within seconds we don't
need to wait for them to pull or
anything like that we can immediately
get results from them and then we have
the admin ui which
we use that to manage the deployment so
i actually have a bit of a demo today so
i'm just going to show you guys what the
admin ui looks like and as you can see
um
we have just the the welcome screen
there is a dashboard over here that uh
you know just tells us uh some
information about this deployment
uh like how much disk space there is and
things like that and uh today in this
demonstration i have um i have about a
thousand clients connected so a thousand
endpoints connected and uh and the
server is you know kind of waiting for
us we're gonna do some some interesting
work on that so if i just um
search for my clients i can see these
are all my clients here
and
you know i can look at each of them
randomly and see some information about
it including um collecting so telemetry
you know about like how how much cpu and
usage it's you know that each client is
taking each endpoint is taking
but uh that's just
um so just showing you how i can control
each
client each uh we call clients the um
the assets right so they are the clients
okay so typically um we it's very
efficient it's really fast designed to
collect a lot of data real quickly
um
because most of the work is done by
using the query language which runs on
the endpoint so you'll see that later
when we're going to do some pretty heavy
lifting and you'll see the endpoints are
doing a lot of work so even if we hunt
for it with
many many endpoints then we will we will
be able to
uh very quickly
um
see um
we're going to very quickly uh see that
it you know they'll scale really quickly
all right so um
the the idea behind vql is instead of
having specific analysis modules
um we have generic what we call vql
plugins and those plugins uh perform
some low-level forensic analysis such as
uh parsing files
um you know in in the windows world we
have you know ntfs buzzing mft and so on
uh in the linux world we have parsing
using grog sqlite and so on uh and
binary parsing we're going to look at
some of those today but instead of just
having like a module that just does you
know we're going to look at
you know browser history
we have generic pauses and then the
query uses that to uh to build a more
complicated query
parser out of that
so so this is the point of having the
query language we can string together
different basic building blocks to
create a more complex and capable
um
capability
so
because this is all about open source
and this conference you know really
focuses on a lot of the open source
aspects as well uh because it's an open
source we have a vibrant community of
people who write these vql queries for
us so
if you just wanted to know
how to do a particular forensic analysis
or particular or look for particular
threat then probably there's going to be
someone that had written a vql query
that they would share with the world and
that that allows us to kind of
crowdsource these capabilities so we
have on our website let me just
quickly point out
uh so this is our
website talks to the velociraptor.apps
there's going to be links at the you
know at the end to it but we have this
thing called uh the artifact exchange
and this is where people share
their different artifacts so you can see
there's a whole bunch of different
artifacts here
um you know for example
look for j
uh detection you know someone has a
contributed
log4j
artifact and this is the vql that runs
so we can simply share those uh easily
so let me just show you quickly
uh in velociraptor we call artifacts are
those vql libraries
that contains those queries and so these
are the ones that come you know uh built
in and you can see that you know this is
the query here
and these are all built in
right and we can actually
leverage that uh artifact exchange to
obtain all of that community sourced
artifacts so these are built-in and i
can simply
import
those artifacts
so i just
choose to run a server collection uh and
search for
import
in the artifact
and
select this
uh this artifact so it's it's like
there's a built-in artifact that a
built-in query that populates the server
with the community queries basically so
when we when we collect that from the
server
um then it will go off and fetch
uh you know and fetch the um
all the other artifacts and insert them
into the server we're going to use those
today so that's why i need to do that
first
and you can see that
uh now when i look at all of our so
these are like our artifacts which are
saved queries essentially there are the
built-in ones from before but then there
are ones with the
little user icon these are the
contributed artifacts that came from the
artifact exchange so we can see these
are all the ones and we're going to use
some of those today so so now we have
those loaded so we can use them
um so let's
uh so the artifact exchange again is a
place for exchanging uh your uh these
community contributed artifacts queries
uh and we just imported it from the um
artifact exchange by just going to new
collection from the server
so
um so let's have a look at some actual
example like how do we how can we
actually use this vql
to
um to create some actual
um something useful right
so
let's uh and i'm gonna go through a bit
of the process of creating your own
content
to try and give you guys the idea
of
uh how you can use vql creatively to
make some new content to to make new
detections and new new ideas
so
the first example we're going to look
for
is detecting ssh logging events and
because you know linux typically
uh a lot of the investigations that you
know we do on linux are around ssh
compromise
lateral movement happens by
compromising ssh keys
and you know and then sometimes we have
to go through and recover um
you know
who who logged into this machine where
did they come from these kind of things
so ssh is a big part of linux
investigations not the only part but
we're going to we're going to look at
that as an example today
so we're going to look at
how do we
leverage ssh logs to try and understand
how this kind of attack chain
occurs
so let's take a look at
what does an ssh log look like and
you've all seen i'm sure
ssh logs uh typically they are logged
through syslog
uh and there's a file in syslog var log
off log and it contains or any on
different systems it's in a different
location perhaps uh but essentially
syslog is uh is the defect the default
logging system on linux so i think
pretty much all linux systems use syslog
but syslog is not
uh especially
easy to work with the
the difficulty with this log is that it
it consists of line based unstructured
logs so it's essentially just
you know like a print you know statement
essentially you're printing a a line
and that means something right but from
a um a dfir perspective or you know an
investigation of forensics it's it's
unstructured so it's very hard to
to uh to associate it with anything you
know like to make queries on it because
it's uh it's unstructured
so typically
this is what it looks like uh this is a
line
and it has all the key pieces of
information in it that we want but
they're kind of like all over the place
right so it has the date and as you can
see in syslog even it doesn't have the
year which is
terrible
and then it has the host name it has the
servers the service and then and then it
has some key pieces of information like
whether the key was accepted the
connection was accepted or rejected so
we have the word accepted here
and then we have
what kind of authentication it was from
here and then who's the user
and ip addresses and so on
and this is really bad this is really
hard to to
query against right so when we do an
investigation usually what we need to do
is convert these unstructured
you know essentially text soup i would
say into structured logs that we can
query you know in a proper way and
usually the way this works um well you
know i mean you can write like regular
expressions to try and find little bits
and pieces from that and you know
essentially
the way uh that the industry is kind of
settled on solving this problem is using
something called grog
uh grok is just like a way of
expressing very complicated regular
expressions in a little bit simpler way
so these end up essentially being very
large regular expressions still
and you're kind of matching that against
what the log supposed to look like and
sometimes it sort of works
so
that's kind of i guess that's that's the
state
of that's the state of
of logging on linux is not great so um
so this is the best we can do so let's
just have a look at how we can use vql
to
get some structured information from
these syslogs and i'm going to show you
how to quickly write a vql query so the
first thing that we do is we have this
thing called a notebook and a notebook
is like something
that we can use to build up uh to build
vql and run it interactively sort of
like
if you've ever used a jupiter notebook
so it's sort of similar to that so i'm
going to open this notebook here that
i've prepared earlier just for the sake
of time
uh and going through the example of
parsing ssh logs and so i'm going to
give some i'm going to talk about some
of the vql and point out how it's used
to parse these logs so just for uh to
get better real estate on the screen i'm
just going to change it into full screen
so it's a little bit easier to see so a
notebook consists of
these are called cells and they're kind
of invisible initially but if you click
on it then you know they become obvious
and then we can edit each cell so each
cell is like it's kind of quick it has a
query and then it runs that query so you
can see here this is the vql query
uh here
and so the first query we're just going
to grab the files out of the off logs
right so
sorry grab the lines out of the oslo so
you know as i said syslog is just a line
based format so it's just they're just
straightforward lines and so you can see
that this query what it does
uh there is this thing called a plugin
called pars
uh
uh parse lines and then parselines
basically grabs each line and puts it
out into a variable called line so this
is a query and as a
because it's a query it returns a series
of rows
and columns right so queries always
always return rows and columns and so we
can have this is actually a
whole bunch of rows
and that's the column called line right
so
so this is how we would uh now in vql in
in here uh you know we can we can use
command line completion and things like
that so we can see like what parameters
does you know this plugin use
uh you know or we could do like you know
select
start from
and then we can see these are all the
plugins that we could use
um
you know pause and we can search for it
so so this is the uh preferred interface
to write your query because it really
helps you with writing the query once
you write the query and you click save
then it recalculates it so in this
particular case we are pulling 50 lines
out of the first log so that's the first
step is to just get the lines out but
again they're not structured at this
point so what i want to do is i want to
convert them into something structured
and i use the grok expression that's the
big expression that you see before and
these things are available on the on the
nets
um and there's libraries of them so it's
not like you have to come up with them
yourself but it's basically expands into
a big regular expression that matches
that line like i mentioned before and it
converts it into a structured uh format
and you can see that's that's the
structured format it's like it creates
uh uh the whole thing basically uh
splits into a dictionary and then you
know it has these different fields so it
pulls out specific things and you'll see
that the the sad thing is that the time
stamp again is has no year in it so it's
like it's not easy to parse
um but you know
we've got all the key pieces of
information whether it was accepted you
know what kind of thing it was public
key private key etc program etc so so we
can use that to essentially pull out
these structured information so let me
just um
so now once once we have that we want to
actually create something called an
artifact because we don't want people to
have to type
all of this vql into the gui each time
right it's kind of a pain and error
prone so what we want to do is have have
it somehow encapsulate so we can publish
it in an artifact
so luckily
uh that let me just get out of full
screen mode and go back to our
uh
artifact library here and if i search
for ssh then luckily uh oh no this is
this is this uh there is a built-in one
which is actually exactly the same as
before
uh
it's just now it's just kind of like
encapsulated inside of this thing called
artifact which is we can just use
so we don't need to type any of these
queries in we could just use them and
you can edit it and customize it you
know so this is the query that we've had
before it's a little bit more
complicated now because it's going to
look for different files in different
places because it could be a number of
auth logs and it could be zipped up and
etc right so but you know this is a very
simple thing
um and uh let's just uh let's just find
my favorite machine
uh let's uh pick up uh this one
one of my recent hosts
and uh and this is this is uh i've got a
tag on it called mike right so i've got
a label on that machine so i can go to
it straight away quickly
uh and let's have a look at all the
artifacts that we've collected before so
i've collected some other ones before
right for example i grabbed like uh
different files and so on uh but let me
just add uh
let me just search for this ssh
login
okay so uh in this case what i want to
do is search for that uh ssh login again
it just it just goes through and it
takes parameters here
uh so this is just the defaults
so then the next step i'll configure the
parameters for this artifact
uh and you'll notice that i mean the the
vicuoil is in there but i don't really
need to know anything about it so i
don't need to really pause it or
anything um i've got some defaults that
i can change like maybe if my logs are
in a different place i can look for them
uh and this is the grok expression that
i can maybe tweak a little bit maybe
it's a non-conventional version of ssh
and the logs are a little bit different
yeah that happens
um but anyway the defaults are usually
fine we'll just launch it and uh and go
off and collect it and you'll see that
it's you know it's finished in 0.15
seconds it just got essentially as soon
as i tasked this endpoint it went off
and collected that thing and paused it
right and then if i look at my results
then i've got all the ssh logos all the
ssh logs
um and you can see clearly that this is
a problematic machine right right away
why because we're seeing all these
failed password logins so somehow this
machine is getting uh if you've ever run
you know linux machines on the internet
of course they're going to get brute
force all the time so these failed
passwords you know you're going to see
them a lot if your machine is available
on the internet
you'll see the accepted public key which
is the legitimate users using keys and
that's
prob that's fine probably
but then you have a whole bunch of
passwords here
but what would be really bad what would
be really bad if this machine had a
successful attempt with a password
because that is not good right like so
normally you're supposed to use keys and
if someone's brute forcing the password
and got in then you know then that's not
good right so what we can do is we can
do this thing called post processing of
the data so we've collected all the data
with the artifact from this machine
and i can open up a notebook just to
post-process that one collection
and uh and it's the same thing but what
i'm going to do is i'm going to change
this query and i'm just going to add
conditions so it returns all these rows
but i just want to see the rows
but the result
right matches
accepted
all right because so that uh equal tilde
is the regular expression match operator
uh and
uh methods
matches
password okay so if someone got in with
a password you know that would be super
bad right
and so immediately that thing pops up to
me it's like hey that is not cool right
someone use the password now it could
well be configured to do that maybe it's
okay but usually um that requires
further in
inspection
so
so let's just go back to the slides
and recap so we don't go too far ahead
of the of the slides so we used vql we
could use vql to pass each line out of
the file
and then we applied a grok expression to
create a structure out of the
text soup of the syslog right and then
um and then we
wrapped it in something called an
artifact which basically is a yaml file
with metadata so it has a name and it
has parameters that are declared as part
of the the artifact
and you can see that here if i um
simply
find it again so this it's built in
right but in this case but um
you can click edit and then you can see
this is what an artifact looks like
right it has different parts the name
description references and then it has
these parameters section and those are
the things that we can change you know
when we run it so essentially that query
you don't need to really kind of i mean
you can look at it right but you don't
need to really type it each time once
that artifact is created then it's just
ready to be used by anyone right
um and so it can be easily discovered we
just searched for it and ran it and it
was done right so all we did is we can
search for it in our artifact library
which is that um third
thing here
you can
view artifact screen right
uh and then uh and then we selected it
we can look at it we can customize it
and we can collect it so we've collected
it on a system this was the system that
we were looking at so the one that shows
up up the top here so you know in this
case it was this machine here
right and then we went to collected
artifacts and we collected that artifact
and you know
um so that's what we're doing with that
machine just specifically
but now uh we would really like to be
able to do it like everywhere like you
know look we have a thousand machines
right so we want to know
you know did anyone you know pass would
put force our password in any of our you
know machines so going from
investigating one machine to
investigating thunder machines is easy
it's called hunt so we just go and hit
hunt manager create a new hunt
look
for password logins description
uh and this tells me like how many
machines it's expecting that it will
apply to and if i say around everywhere
that's all my deployment
i can match it by label and you know
there's only one machine that has that
label so i can just target it just with
that uh label or if i just match it by
um all my linux machines in this case
they're all linux anyway so so that you
know that's all of them
um and then if i just simply click uh
search for ssh
uh we're gonna do the same thing but on
all our thousand machines and and then
just go right so
uh
when we you know see how it's in the
post state so we can just start it
all right and you can see that as soon
as i click start it's starting to
schedule it
and it goes off
uh you know scheduling it for all the
machines right 200 300 it's going to go
off and collect the results from every
single one now because each one of them
is doing it in parallel they are all
coming back pretty quick um and so you
know we can
we can see the um the results as they
come in so let me just uh let me just
see where are we
okay so that's hunting and processing so
the same thing is happening but on
multiple machines
and you can do the post processing on
the hunt as well and
and we've seen that let me just move on
to the next example real quick
um and so in this example we're talking
about unsecured search keys so again ssh
is our theme today so let's look at um
how
ssh keys should be protected now we all
know that we need to protect our ssh
keys with at least the passphrase
because if we don't then someone can
that can break into that machine they
can just use those ssh keys the
unprotected ones to uh laterally move
from that machine to all the other
machines on the environment right
without having any impediments right so
essentially that key
becomes you know a liability so we need
to uh protect them with a password but
you know like on many environments for
instance in aws when you get a key pair
they're not encrypted or not protected
and a lot of people just look at them
and they're like okay cool and they use
them but they don't realize they need to
go through that extra step of actually
encrypting them uh and so they have a
lot of these keys lying around the
environment that are not protected so
what we want to do here is we want to
find you know all the keys in the
environments that are not properly
encrypted so let's take a look at how we
can parse these these files this private
key format so let me just show you like
how you come up with this id this query
for something quite new so again i've
got an example here
of um
of you know a new notebook i'll just
make it full screen again
and we can see that the first query the
first thing i'm going to do is i'm just
going to read that file right this is my
private key
i'm just going to read it and
i have a read file function in vql and i
can see that it looks like you know it
has this uh header a private key and
then it has a whole bunch of what looks
to be base64 encoded something and then
there's a tail on the end so that's
that's cool so then clearly the thing in
between the the thing between here and
there is looks to be base64
uh encrypted uh encoded right so let's
um let's take a look at
how we decode it
and um
what i'm going to do the first thing is
i'm going to use a regular expression to
pull out
the data between the key
the start the the header and the end
right and that will give me the first
part which is just that base 64-bit
right and then in the second part i'm
going to decode it with base64 decoding
and so i can see straight away this is
what you know so there's a whole bunch
of binary data but straight away you can
see that there is something here right
it says open ssh key
and it has none and none and if you look
and there's a whole bunch of information
here that could be useful as well like
inside the key so that where it was made
and things like that
and you know the the type of key that it
is and so on so there's some information
here that is quite useful but uh but all
this thing is you know binary so now we
have this problem of like okay we have
all this binary data and i can see stuff
in there but i have no idea how to pass
it out
so let's uh
so let's
do some research on google what is this
binary blob uh what is the structure
so we have this there is a someone has
done some research which is great
and describing the uh the format so
let's go to that site here
okay so this is just a page on the
internet that explains the format and
you can see here uh the format is a
binary format and there is like there is
a description of you know so there's
like the length and and some um
uh it explains how the binary data is
you know structured right so we can take
this information and we can build a
binary parser to extract this info this
uh fields out of the binary data
so uh like you know you can write a
python script or something to get it out
but
in vql we actually have built-in binary
parser
and i just want to quickly show you that
i'm not going to go into details about
the parser
uh but in this private keys one
i'm just going to show you
what puzzle looks like so it's kind of
like a descriptive thing there is a
profile that we can use to describe
how each field is laid out so you know
that's that's the header there's a magic
string at offset zero and then the
length of the cipher is offset 15 and
it's a uni-32 big endian
and then the cipher itself which is the
string that describes you know what
cipher is used to encrypt it uh is at
offset 19 and you know the length is
given by you know that other field so so
we can have this so this is this is this
is a description of uh of the binary
format and we can use that to pass the
keys out so again the same thing we're
going to go out to our machine here
and we're going to add another
collection and let's look for our
private keys
okay so here's our private key on a real
machine
and we're going to collect this artifact
again we don't need to necessarily
understand the vql we just need to know
how to use it so if we go over here we
can change the parameters and this one
basically there's a bit more complexity
in the
uh in this artifact because it can
search for keys everywhere and we want
to make sure that it doesn't go into
proc and you know and then get lost in
there right so uh so we can so there's a
few more functions of functionality than
you know we've just described uh but
basically we go off and and collect this
thing
and it comes back in you know a matter
of seconds and we can say oh look you
know this user has a key and it's
protected so that's great so so this is
good right we checked it and and that's
good
so um but you know maybe that key has
other that user has other keys lying
around right so we only by default
search for the keys if you look at the
parameter uh or the the default
parameter uh only uses it in slash home
slash ubuntu sshd which is the location
where normally the keys sit right but
let's uh let's um search we copy that
artifact and we we can tell it to search
you know everywhere
so
that's the default search pattern which
is wildcard
right and we can just make it if if we
do star star that's like a recursive
search for this the system we look for
pam id rsa or idsa these are the three
types of names that we're gonna search
for and uh and you know we go ahead and
we do that and uh and it's gonna take a
little bit longer because it's gonna
search through the whole system so i'm
going to leave it for a couple of
seconds and we'll come back to it later
but you can see that just recapping uh
we read the file we basically fold the
coded we noticed some binary data
and we created a parser for it now
because the parser is in vql we don't
really need to rebuild or recompile or
redeploy anything right we just we just
you know write the vql it's descriptive
and it the vql can go ahead and uh
dissect that data out of the endpoint
right so so then you know we wrote it
into an artifact and then we collected
it
from the artifact uh
um repository here
and uh let me just see if it's finished
yeah it's it's it's only taken uh 16
seconds to go over the file system it's
only a cloud vm so it's quite small it
could take longer on other systems but
we can see
uh that this user has some aws keys
here and they don't have any cyphers so
this immediately because that's the
default way that aws creates those keys
this user didn't
you know go to the extra step of
re-securing their keys after they
downloaded them from the aws console so
this is this is really problematic and
this is a really big deal we can see
lateral movement through these keys all
the time right because people don't do
that
so this is now we can go ahead and uh
and you know tell the user hey you know
you've done the wrong thing let's fix it
but let's just think about what actually
happened here um is that we could
actually do that we could actually do
that as a hand on all the systems um
maybe we can do that real quick
um you know
i've shown you how to do that before but
like you know uh search
search or tam
uh and again we do the same thing
for the private key so it's the same
process right but we're just gonna do it
you know everywhere instead of on one
machine
and then go for it and what's going to
happen now is that all of our machines
are going to go like all thousands of
them and it could be more right they're
going to go and search for that on their
own system but because each one is doing
it sort of in parallel then it still
doesn't take very long to uh to do that
so the goal come back and getting the
the results you know so like just like
before uh it's very calculated so before
we found all the logins right the same
thing so they all came back from all the
machines right then we could still do
the post processing then uh in this case
we uh we're doing the the same thing uh
see if the results are here yet no
they're still coming um and
uh and then we can we can find that oh
here we go so we've got some data there
and uh
yeah so we can we can then see you know
everybody's you know keys and someone
that that are
uh
this this all these machines thousand
machines are kind of virtual all the
same
machines so we're going to get the same
data but
you get the idea of hunting so this is
cool the other thing that's cool about
it so this is how we created a new hunt
we configured it
and then we ran it now the interesting
thing about it is that we haven't
actually downloaded anyone's keys right
so it's not like we went out
grabbed all the keys and ran a python
script locally to check are they
encrypted because obviously that would
be like really bad right because we
don't
copy everybody's private keys right so
having it done by the end point uh means
means that we can we we don't have to
get the data essentially all right last
uh last example recovering deleted log
so we looked at uh how you know the logs
look at syslog
but let's say in a lot of cases you know
people delete the logs or uh compromises
happened so long ago that the logs got
rotated uh maybe a few weeks you know
before normally it's like four weeks
after and then they get rotated out
depending on the rotation policy uh
those logs can be aggressively rotated
so in that case we really we really need
to go back in time and try and find
forensic evidence of
these compromises from the logs uh and
we try to recover deleted logs now if we
if we are back to that you know this is
is not good right it's not a good
outcome it's better to have vlogs right
but
if you are struggling
and we don't have logs then we can use a
technique called carving and carving is
a very simple technique where we
basically look for patterns in
unstructured data
and the idea is that when someone
deletes those logs then the data is
still on the disk so we we might be able
to find it you know from just the disk
unstructured data so let me just show
you an example of how that's how we how
we do that
so in this
particular example let me just make it
full screen again so the idea is to try
and look for
uh patterns that look like a syslog
message right so uh we've seen before
the syslog starts with uh jan feb
martial look at month name and then it
has the dates of the month and then it
has you know the the time and then we
know that you know it's a line so we're
going to take all the characters until
the next new line so that's one line so
when we do this we can so we're going to
do this query here
we use a tool called yara which is uh
used for essentially applying regular
expressions at scale it's very fast and
efficient and so on uh and so we can do
that on
ideally we want to do it on a device in
the end but for testing we're just going
to do it on the real file to find the
right
regular expression and so on and so you
know we go ahead we we grab that file
and we run the cr expression on it and
it's supposed to
uh hit on all of the
all of the lines that sort of look like
that right that sort of look like maybe
a c slog line right so when when i run
this the first query so that's this one
uh you can see that there is you know
some hex data here and it matches you
know that kind of pattern right so we
got the gen 16 and then it goes all the
way to and again no no year right but um
you know so this basically pulls out our
log lines or things that look sort of
like a log line so you know uh then what
we're going to do is we're going to uh
extract the actual hit and look for only
things that sort of look like maybe ssh
logins right so it has to have the word
either accepted or failed that's really
all we care about in this case and so
you know that's the second query here
and so you can see the hit
is uh essentially
you know uh the ssh
keys right so it's accepted a login
accepted login and so on right so so
basically all we do is we write this
thing and then we just
and then again we do the same thing with
rocket and all the rest right and so we
can apply that and get
uh and then carve out
uh
the
ssh logs that could be deleted so let's
have a look at this one comes from the
exchange so again it's a probably
probably contributed content and again
this is the query this is the regular
expression that finds out the error rule
and then this is the grok expression
okay so let's go over here and carve it
now carving takes a long time because
you are really looking for the raw disk
right so if we actually look at
uh
it's it's it's looking at the raw device
and then just looking for patterns that
sort of look like ssh um
messages right so it could take a while
to do it's going to scan all the disk
this is a cloud machine so it's not that
big but it could take a while so let's
just leave it
and we'll come back to that
okay so just recapping
and then we're going to see some hits
over here um we're going to see that
later so in the last five minutes i just
want to show you guys another cool
feature in velociraptor which is about
monitoring events from the endpoints so
normally when we run a query we talked
about vql and you can see it's really
quick and it finishes and gives you
a table but it doesn't have to finish
right so the query actually returns data
as soon as it's available so that means
that if we can write a query that runs
sort of forever right
then as soon as something happens it
will return stream data back so vql can
support streaming queries and that is
where event queries
go in so you can write
a query that is running all the time but
it's constantly streaming back
uh
you know rows and then that row those
rows can simply be forwarded to the
server and then we are collecting those
as events so we can use that for a
number of things we can use it for
monitoring and also we can use it for
response for creating uh for automating
response so we can go and do stuff based
on those those queries so here's an
example of i just wanted to show you
guys the query here how do we turn that
other query that we did before which was
remember we had
passed lines which just goes off and
reads the lines but there is a similar
event version of that query called watch
syslog and that is watching the the line
so it's essentially like a tail uh uh a
tail that shaft or you know something
like that um
or or a
less with a tail following or whatever
right so it looks at the end of the file
it has watches for new lines to appear
and then it releases each line into the
query so it never terminates but once we
run this query it will always work and
and then just grok the lines as they
come and then filter them out and if
they're ssh then it will say hey that's
that's you know an interesting one and
it will pass it on and so we can use
that to um to monitor for ssh logins so
there is this artifact here which is
windows event ssh login i'll just
quickly show you that
so again we're going to look for ssh and
this one is an event version of that and
we can see that it's a client it's a
slightly different type but it's still
the same kind of general structure it's
still an artifact and but to install it
we have to go into this screen here
which shows us
uh the event monitoring on the client so
we can target it specifically to a label
group say mike
and you know and then otherwise it's
kind of the same um ui right we just
select which ones we want and then we
can configure them and so on right
and
and then once we do that then the event
starts streaming in
so we can see that so in this case for
instance uh you can see that there is
one event that came in
i had them on before so i can show you
how it looks like
when someone logs in then immediately
that event is streamed to the server so
it's not log folding it's not just
forwarding all the logs indiscriminately
it's doing the querying and processing
on the endpoint directly and then just
for forwarding back
uh just
those ones that are relevant to the
query right so we can do we can do both
we can
uh forward all the events or we can do
the post uh process the pre-filtering
and the processing on the endpoint and
just forward back those really
high-valued i mean this is a really
high-valued event uh you know and it
could be sitting between thousands of
syslog lines right we don't care about
those we just care about this one so
they all go in the same place right uh
then the
let me just finally the last uh thing
that i wanted to show you guys
uh so this is how we collect the events
right we just run and we see those uh
things the last thing that i wanted to
show uh to talk about is sysmon and
cismo is really exciting it's sysmon is
like the default um i guess kernel
events monitoring tool for windows so
it's been around for a long time on
windows and just recently they've
released a sysmon for linux based on
ebpf
and we've talked a lot about ebpf in
this conference especially in the kernel
hacking uh minicom early on uh early on
in conference so ebpf is a method for us
to be able to get information from the
kernel about things like process
execution network connections all that
really good stuff from the detection
perspective
and sysmon is now a nice easy way of
getting into that um it's still immature
you know it's still a little bit buggy
but it has a lot of interest from the
community everybody's excited about it
um it still has some shortfalls
but i'll just show you how how it looks
like that this is the sysmon uh the
synthetic version itself just writes the
syslog which is
terrible because then you have to apply
these regular expressions to get the
data out
um so i've
written a patch to fix it to
write it to unix domain socket so it's a
lot more efficient and json encoded
and so we can use this plugin called
netcat which connects to the unix domain
socket and reads all the lines out uh
and but otherwise it's exactly the same
path that jason didn't show it so
let me just quickly show you what that
looks like
uh so all we do is we just collect that
from our endpoint
uh and you can see that it's basically
uh it's it's giving us this structured
information about
you know process execution like here's a
ps command line that ran
uh you know where it ran from and so on
a lot of these fields are they kind of
all only make sense on windows but maybe
there isn't really an equivalent
you know source data source for it on
linux but
um but you can use them to just like
filter and say oh you know when this
process ran what was the parent process
what did it do and then you can write
detections based on that so again
so much mike we've run up against time
and we need to keep on schedule no
worries uh well so just last last slide
um just references check out the uh the
the website the github uh and the
discord and um thank you very much for
your time
thank you very much um i hope that you
can drop some of those links and answer
the questions in the text chat in
venulis
uh up next at three at uh 2 25 p.m adt
is j
rosen rosenbaum with rolfer initiative
how to make the world of ai a more
ethical place
thank you very much mike
thanks