[Music]
hi everybody we're back uh so next talk
is karen jacobson talking about avoiding
dns pain
hi everyone and welcome to avoiding dns
pain with me karen jacobson
before i start the presentation i'd like
to take a moment to acknowledge the
traditional owners of the country
throughout australia and recognize their
continuing connection to land waters and
culture i pay my respects to their
elders past present and emerging
so as i said earlier my name is karen
jacobson i recently joined focus
software as their head of business
systems i'm leading a team that's
supporting our internal i.t systems our
infrastructure and a lot of our business
processes and
working through that to help drive the
business to further growth
i've been a microsoft mvp for the last
five years and i'm also now a git
cracker ambassador but enough about me
let's start talking about dns
so it's my firm belief that dns is one
of these systems that's really
under-appreciated and it runs
uh and powers all of our networks and
our you know the internet as we know it
well it's under-appreciated until it
becomes unavailable and that's when we
find ourselves not just as
individuals and organizations but
globally in a world of pain
2021 last year was actually pretty
significant in terms of dns outages
there was pretty much every major player
every cloud provider was impacted at
some point in time by a dns related
change
or issue
in april microsoft and azure and a lot
of dependent services and customers were
impacted by a dns outage that was caused
by a coding issue
inside azure's dns services
in july customers of akamai
including oracle cloud ups fedex steam
lastpass and sony playstation were
impacted by an outage in akamai's edge
dns service
september saw an outage in with slack
and their dns
services
but these were all overshadowed by what
was really an internet meltdown that was
when facebook instagram messenger and
whatsapp went down for over six hours in
october
this outage wasn't actually caused by
dns or dns change or server issue
directly but networking issues that made
those services unavailable
but once again that helps highlight how
a dns resolution issue can quickly start
to you know cause absolute chaos
um you know the estimate is that mark
zuckerberg lost you know millions of
dollars and it's hundreds of millions of
dollars as
meta's stock market value just dropped
over that period of time
and of course you know not to be left
out of the club aws had a dns related
outage as well in december
this one
actually had a lot of downstream impact
there was a lot of other i.t service
providers and organizations
um that rely on aws that had had outages
as well caused by this issue
and all of this keeps building up to
this really big challenge that
organizations are starting to face
around ensuring that
you know websites and web applications
are still resolvable in the event of
issues
but there's also been a lot of dna
security concerns as well it's not just
outages we need to think about when we
look at dns
attacks against dns have been on the
increase year on year
organizations have been you know
repeatedly warned to be on the lookout
for attacks against dns infrastructure
over these past three years
three years ago
cesar the american
government security agency they released
their very first emergency directive and
that was actually warning organizations
and government agencies to take steps to
prevent attacks against critical dns
infrastructure
that warning has
in a way fallen on deaf ears mostly
security threats to dns come in a couple
of different forms the most common still
being
you know a tax fire compromising dns
entries but the other one that we're
starting to see a lot more of is what's
called dangling dns entries
so let's take this situation
we're going to provision a web app on
azure and it's got the fqdn of
app.contogreatdev001
dot azurewebsites.net we're going to
assign greatapp.contoso.com
to route traffic to that azure resource
now after a while we decide we don't
need that application anymore somebody's
gone into the azure dns
azure portal and they've deleted that
web application
at this point we've still got that cname
record that's pointing uh so we've got
the cnn record greatapp.contoso.com
pointing to appcontogreatdev001
but the
underlying application isn't there
anymore
and this is a dangling dns record this
record is now vulnerable to being taken
over by some other resource on azure
some other subscription it doesn't even
have to be the same subscription
could provision
a
a web service and start to get traffic
routed to them and that's what a lot of
threat actors are now looking for is
that they're scanning through dns site
zones and looking for these entries
and so let's just say in this example
some malicious you know actor some
threat actor discovers this entry they
go into azure they recreate app dash
contour great dash dev zero zero one
and now
any traffic that's going to grade
up.contoso.com is now landing on a
website they control
these dangling dns entries pose a pretty
significant risk threat actors can take
over this over a dns name and they can
host a malicious website or a malicious
web service
using your organization's brand
and it's really important that when we
start talking about domain names
in a business sense that they are part
of an organization's brand misuse of a
domain is misuse of an organizational
organization's brand and that's linked
to the value of that brand and that
organization
with a malicious site on your on on a
domain like this they could launch
fitting attacks against your customers
cookie harvesting attacks cross-site
scripting cause bypass attacks
a lot of these things could really
significantly damage your brand's
reputation
and big and small companies over the
internet have all been impacted by these
attacks
um several years ago microsoft actually
had a dangling
dot microsoft.com
domain and that was a you know a pretty
pretty significant issue that they had
that they had and others have seen it as
well
it's also worth pointing out it's not
just
web applications it's not just c names
it can be vulnerable
there's actually a lot of
domains out there that have got dangling
mx records where you could actually go
and provision
uh
you know services on some other domain
and suddenly have mail flowing through
from other domains so it's not just the
sea names you need to be worried about
you need to look at some of the other
resource types as well
another big challenge in organizations
is the speed at which dns changes happen
now this image here i usually have in
talks about devops and devsecops where i
talk about the relationship between
development teams and operations teams
um
dns is is one of these things that's
often really overlooked where we have a
you know an agile or a devops
transformation
so we end up with environments where
there's often strict or very rigid
change control processes especially for
dns changes they're rarely seen as like
a standard pre-approved change
and that's partially because we're sort
of
subconsciously aware that these things
are sensitive but at the same time
not really thinking them through
and what's the result of all this change
control making a simple dns change can
often take weeks in some environments
i've seen environments where change
control for dns change can be several
months to make an
update to an entry and that's just not
feasible as organizations continue to
grow and develop
so when i joined an organization and i i
joined focus in march
and april sorry april uh
last year one of the things that i
really find interesting to do is to
start looking at dns zones and start to
play a game i called what's this dns
entry
you might have 50 or 100 or maybe a
thousand dns entries in a public facing
dns zone
but do you have clear documentation and
traceability about what each one is for
who requested it when did they request
it why did they request it
are you regularly as an organization
reviewing all of your dns entries
and seeing if they're still accurate how
would you even start to work out if
maybe somebody's created a dns entry
maliciously in one of your zones
and often once again the fallback is
change control
let's just go back to our change
management system and we can have a look
at you know the requests and you know
see who's changed what
but as we know from the facebook outage
the teams trying to fix facebook's dns
couldn't get to their management
consoles and their management systems
because dns was down
and so there's got to be some better
ways of managing these things so we
don't end up with some of these
challenges during an incident
and thankfully there is a better way
and the solution is to start looking at
these problems
as infrastructure as code or as i like
to call it dns's code
today i want to talk about dns control
and this is a tool that was developed
and is maintained
by the stack exchange team and from a
lot of contributions from the community
at large
dns control is heavily influenced by the
experience of the stack exchange team in
managing
websites at a large scale there's a lot
of things a lot of changes and features
that are there based on their
experiences with managing dns outages
so what is dns control
it allows us to use a javascript-based
dsl
don't worry if you don't know javascript
i don't and i haven't had any problems
but it allows us to use this javascript
dsl to define what entries should be in
a dns zone
[Music]
and then from there it will actually
look at a dns zone in your provider and
say okay i need to create these entries
i need to delete these entries because
they shouldn't be there anymore and i
need to modify these entries
the syntax is really simple to look at
as you can see up on the screen right
now
you can see that i've got a domain zone
called planetpowershell.com
that's a community project that i
maintain
we can see in this dns zone that there's
some spf records dmarc
some office 365 related records we can
also see there's a production website
and development website that are hosted
on a records as well
and even without many comments it's
pretty easy to just quickly look at this
and know what's there in the in the zone
but we can also add comments to help
explain
the who what when how and why of some of
these entries
and that really starts to help explain
and
really communicate why things have been
done that way and that's really
important
so say you're going to use dns control
to manage your zone what would sort of a
standard you know let's talk about how
making a change to your zone might work
what sort of a standard approach
so in this example i want to add a text
record
hello.planetpowershell.com with the
content hello linuxconfig
so i'm going to open up my dns control
zone file in my favorite editor yours
might be something different in my case
it's vs code and i'm going to add in
this txt row at the bottom here
and it's really simple it's just txt
what that name is and then the content
i'm going to commit those changes i'm
going to create a
i'm going to push those to a branch on
github i'm going to create a pull
request
i'm here using git kraken and they have
a built-in pr
request feature so from within side the
tool you can just go ahead and create
your pr's which you know i like because
it saves a bit of time
but
we can start to see with the templates
and stuff like that how the process
starts to become a bit more clear in
terms of change control
so now we've created that that that uh
that change request that pull request
we need somebody to obviously go in and
and review that and approve that and
this is where when we combine something
like dns control with github pull
request templates we're starting to make
our our lives as an infrastructure team
and sysadmins even easier
pull request templates they're really
great because we can have a template
that's got a checklist of what needs to
happen to make sure
that a change actually meets our
internal requirements
um in this case you know there's a check
to confirm there's no dangling dns
entries that spf doesn't go over the 10
dns request limit
and some other sort of checks as well
if this was for an internal
zone inside your organization you might
want to have it with some
you know
steps around say maybe you know sending
out a notification to users or impacted
teams this change is happening
we can also use dns control to provide
automated checks dns control comes with
two commands
check which validates the syntax of your
files and make sure that they are all
correct and preview
preview is actually going to go ahead
and look at the zones as they are in
your production environment and see
what needs to be created what might need
to be modified what might need to be
deleted and it doesn't actually do it it
just tells you this is what i would do
if i was told to make the changes
i find this really great because
somebody can then propose a change and i
can actually go and double check
that what they're intending to do
actually matches what's actually going
to happen
now once all of the checks are passed
i've reviewed this change i can merge it
into my main branch
and so that's when our ci cd process can
actually go ahead in this case i'm using
azure devops and push those changes into
production
if i go and look in cloudflare which is
the dns provider i'm using for this
domain i can see that that entry has
been created
and then if i give it a bit of time for
dns replication and caching
i can see using mx toolboxes dns query
tool that that record is now available
for clients to resolve
so we've gone through a very simple
process
from somebody proposing a change all the
way through to
that change being reviewed and then the
change being made
in a pretty pretty quick amount of time
and a lot faster than most organizations
would see a dns change being made
but there's some other cool features
about dns controller i want to talk
about
there's a lot of quality of life and
security features in dns control
there's the spf builder it's often
called the spf optimizer in the
documentation this allows us to not only
sort of really cleanly define all the
parts of our spf entry but it will also
support what's called flattening of
entries
this will basically go in and actually
as it goes to create the spf
record it will remove some of the dns
queries so that we can try and
squeak through that 10 lookup limit for
spf records
in some cases i've seen it work really
well and it's saved queries and helped
companies stay under that limit other
cases it hasn't worked quite so well
it also has
dns control also has
functionality to help us build out dmacc
policies in a way that's a lot easier to
read and understand as well as our cia
records as well
the cia builder is a little bit new
still a little bit experimental
but
once again it makes it so much easier to
read and understand
dns control also has a bunch of features
around working with azure dns
including alias records aws route 53
alias records
cloudflare
um
[Music]
there's a lot of stuff in there as well
from inside
dns control you can turn
the proxy of cloudflare on and off you
can also create
page rules from inside cloudflare so you
can now start to use dns control for all
of your sort of cloudflare management as
well bring it all into one central spot
have all that documentation that change
control within one spot which is really
really cool
dns provider resilience and migration is
probably
the main reason that stack overflow
created dns control to begin with um
when we look at sort of
what was behind dns control
and the stack overflow development it
was
their experience of having an entire
provider just becoming unavailable
and needing to be able to quickly
set up another provider and know that
all of the entries were there and ready
to go and that all of their systems
would
work
and their experience before that was
that they had one provider that was um
a gui portal another that had an api and
they were really sort of worried during
this outage of well have we created
everything
all the records there and that's where
um
the provider resilience inside dns
control really comes about
so we've got three examples here
the first example
is example.com
that will actually have eight
authoritative ns records
uh four from route 53 and four from uh
gcp or google cloud
and so in that case
you know
clients could go and resolve example.com
using any of those eight authority of ns
records
in example two essentially what we're
saying here is we want two
ns records from route 53 and two from
from google cloud we don't want the full
four we want to balance it just across
two
in the third example example3.com
essentially we're saying we want the
authority of ns records to only be from
route 53
but when dns control goes to configure
the zone and push that zone out it's
still going to go and do all those tasks
on
gcp
so
that basically means that
all of the records inside route 53 and
google cloud
should in theory all match up
um
now this is good if you want to just
have a like a sort of a hot spare dns
provider
but one of the big things that i've used
this in the past for and i know a lot of
others that do is actually migrating
from one provider to the other
you know say you're an organization and
you're moving everything from running on
aws across to gcp
then you know this is a really good
approach that you could
simply put the google cloud dns provider
in don't make it authoritative
run dns control have it set up all of
those and then be able to take your time
and review all of those entries before
starting to make it the authority of
provider
and then you could slowly move load
across to that if you wanted to as well
that's the power behind a lot of some of
this stuff
migration is one of these ones that
there's still more work happening in the
background because obviously
stack overflow team the stack exchange
team are seeing more ways to improve
that process through their own usage
um
before i finish up and open up some time
for questions
i've put together some links here that i
think will help everybody sort of start
on a dns code a dns controlled journey
the first two are blog posts that i've
written
on how you can help protect an
organization against
some of the infrastructure dns
infrastructure tampering attacks
that are out there thinking about
two-factor thinking about change control
and security controls
and the other is around dns squatting
with a particular focus on azure app
services
my next entry is a pretty mammoth blog
post that i put out a few years ago
on setting up dns control having it
hosted in
[Music]
github
and using what was then vsts or now
azure devops
to basically have a full sort of
you know ci cd process
that's you know configuring entries
inside cloudflare and then simple
[Music]
whilst a lot of that stuff in that the
content of that post might not be
applicable to you
uh if you're sort of looking at setting
it up it will give you a really good
sort of
idea and sort of point you in the right
direction for starting to come up with
your own cicd process fit that meets
your organization's needs
my next link there is what i call my
it's a dns accelerator package that i've
come up with
the idea with that is i've got a dns
control zone file and some cicd
and documentation and some little
helpers
to basically be able to buy a domain and
have it set up really quickly
and the goal is to also include things
like spf and dmarc and cia records as
well so that the domain is you know
uh on its way to being you know having
some of these sort of security related
dns entries defined in there from the
start
the next two uh links there are for
the official documentation for
dns control i really recommend that you
take a look at their getting started
guide i also recommend taking a look at
their migration guidance
they have a migration tool that will
actually allow you to get a bind file
and convert that into a dns
control zone file it works really really
well um and considering that a lot of
some of the sort of third-party you know
the different dns providers have an
export to bind option
it's probably one of your you know your
best ideas for starting to pick up and
use dns control
finally i've got a referral link there
to get kraken
basically if you use that link
both you and i will go into a draw for a
100 gift card and there's four winners
every month i believe so always a
a great option to try and get some free
amazon gift cards
thank you all for listening to my
session um i'm going to leave a
very sort of old and lame recursive dns
joke there for you all um
i'd like to thank you all for listening
to me once again if you want to reach
out to me i'm on twitter at k jacobson
and i've got my website at
poshsecurity.com
thank you all so much
okay thanks karen um we have a couple of
questions
first question is
have you tried managing dns records
using other than infrastructure as code
tools and if so is there anything you've
found it's particularly standout
functionality in dns control and
anything you've found dns control to let
compared to the rest
i think the reason that i was drawing a
dns control is i've i've tried using
terraform i've tried using azure arm
templates and powershell and things like
that
um
dns control supports
so many different services providers and
so many different domain registrars
that it just allows you sort of
pick up really quickly and just sort of
start plugging stuff in
um
you know you can
you know they have a very long list of
service providers that they support now
and so you can use the same structure
no matter where your domain's hosted so
that's why i've lent a bit more towards
dns control
[Music]
i think the biggest challenge right now
is some more of the
some of the cloud providers have very
sort of specific entries and whilst some
of that has been added to dns control
for azure and aws and cloudflare there's
still some that are missing that you
know would be really handy to see but
you know once again it's it's a lot of
stuff is community contributed so you
know it's it's up to the sort of you
know people who have time to put this
stuff in
okay
uh another question is how do you handle
dynamically named names cloud resources
eg load balancer dns names which might
change at any point
um yeah so there's a few tricks around
that
um
for
azure there's their azure alias records
i think they're called um
that's one way that you can
dns controller can actually talk to
azure and start to bring those things in
um and i think there's a similar one for
aws as well but i haven't played around
with that one yet that's on my to-do
list
okay i think that's uh all the questions
we've got okay thank you very much karen
brilliant thank you but thank you
everybody