[Music]
i'd like to introduce claire
um claire is an urban planner
and programmer and is currently
undertaking a phd in the intersection of
these two fields
they find themselves writing scripts
frequently during their research to help
them record and make sense of the large
amounts of qualitative data necessary to
answer questions about the adoption of
digital technology and the future of
urban planning work
outside of their phd claire works with
others in urban planning pro in the
urban planning profession to advocate
for the use of open technology and
standards to ensure good governance
and our cities and regions so over to
you claire
hi thanks for that
great looks like it's all working
um
let's get started
so before we begin i would like to
acknowledge the wonga people who are the
traditional owners of the land from
which i am speaking
to you from today and acknowledge their
elders past present and emerging and to
also acknowledge that sovereignty was
never seated
so hi my name is claire daniel and i'm
an urban planner and data science type
person currently undertaking a phd at
unsw and now as part of this phd i
conducted a citation network analysis of
the planning support systems literature
now as fascinating as planning support
systems are
and as interesting as this niche area of
academic endeavor is
i've made the call that is probably not
relevant to most of you watching today
however the process of undertaking a
citation network analysis may well be
quite relevant for people within the
galleries libraries and museums
community so it is that which i thought
i'd talk to you about today
so right from the start what is citation
network analysis well it is a type of
systematic literature review which is
essentially conceptually quite simple
but can be computationally attached
and um as we know in academic literature
there are very strict con conventions
saying if there's any information or
ideas that have been published elsewhere
that those are referenced within
scholarly publications and there are a
huge body of standards and syntax strict
syntax the kind of process of doing so
um and a citation network analysis
simply assumes that if one document has
cited another document that these two
documents are somehow related and what
you can then do is you can map out all
of these citation relationships using a
formal mathematical network
representation or mathematical graph
and you upon which you can then do
various quantitative statistics which
will give you insights into the
structure of your research field
so next up in our program how to
citation network analysis well according
to zaun strotman 2015 there are five
steps which i will be going through in
more detail
step one delineation of your research
field now this is difficult you do have
to put a boundary around the research
papers that you want to include you
cannot
feasibly do an analysis of everything so
like most forms of systematic review
this delineation is usually done on a
keyword search in a citation database
um
and so what are your options for
databases
well traditionally there have been only
two databases with decent coverage of
published papers that is scopus and
weber science of course google scholar
has for long
a long period of time now provided a
free means of searching the academic
literature that currently provides no
easy way
to to download data from that system in
bulk and provides no api to its
databases
so traditionally weber science and
scopus have been the only way to do
broad scale citation analysis
now my university is subscribed to
scopus and for my analysis i use the
scopus api
um but the fact that i was advised to
use the proprietary api for this
analysis
kind of seemed like another artifact of
the structural problems that we have in
academia with unjust academic paywalls
and citation data is really important so
the amount of academic literature is
growing exponentially and without access
to the citation data we kind of create
the risk of
duplication of effort creating even more
research silos
and even more importantly
ensure like not
preventing access to academic research
from
which is often publicly funded
uh making it harder for for people who
need to actually use that research to
access it
so
in preparation for this talk i have
actually done a little bit of reading
about some of the new initiatives that
are changing this status quo um and to
qualify this information i'm not
personally involved in any of the
projects that i am about to mention
so
it seems that the initiative for open
citations has been a big influence in
recent years to make citation data more
accessible
crossref of course has existed since
2000 and crossref is the registration
agency for digital object identifiers
for scholarly work and it maintains an
open infrastructure an open database
to which
various publishers submit
details of their publications and it
maintains an open api
um however in 2017 just one percent of
the eligible papers
that were listed in crossref
uh contained open citation data and this
is when the where the initiative for
open citations comes in so this
initiative was supporting publishers to
open this citation data to
allow it to be openly available in
crossref and there's been massive
success
because as of october 2021 the
percentage of
relevant articles that now provide this
data openly is has gone up to a whopping
88
separate to this um there was an
initiative by microsoft called microsoft
academic and this was doing something
similar to the way that google
automatically compiles its records in
google scholar but unlike google
microsoft academic made
its data available over
under a license like an open attribution
type of license
um and so from these initiatives you
start to see a number of open projects
for searching the literature kind of
starting to bloom um that rely on one or
the other of these two databases or a
combination of both and this is quite
good because the search functions built
into the cross cross riff infrastructure
are very rudimentarily essentially it's
designed to return the details of an
individual paper if you feed it uh
individual identification information um
like the doi or the
title and also details so on and so
forth
um however
microsoft academic has actually been
retired as of about two weeks ago
december 2021
um and i'm not quite sure what the
downstream impact this might have on
these projects
luckily however when we look at the
overall coverage of each of these
foundation databases there's been some
analysis done by martin martin at al
and cross rift ref has really shot up so
in mid 2021 they finally convinced
elsevier who i hear is one of the
largest academic publishers in the world
finally convinced them to make their
citation data openly available in
crossref
um and according to the study it is
looking like the
citation databases that have been built
from that data are now rivaling
uh the proprietary databases of scopus
that like kobus dimensions web science
in terms of
the scope of their coverage so that is
good news
right
so once we have chosen our database
there will be various python packages
that will assist with extracting data
from some of the more well-known apis
and most of these apis will return to
you a long list of data metadata about
every individual paper the title the
attract details journal details
institutional details and of course a
list of digital ids for their reference
list
onto step two
construction of the network
so first to construct a raw citation
network for
for our purposes we need to construct a
raw adjacency matrix so this is a large
and sparse matrix of ones and zeros
where one access represents the citing
papers and the other access represents
the cited papers and so you can see on
the slide if paper a size paper b
you then put a one in the corresponding
cell it is as simple as that
and so for those of you who are working
with people perhaps who aren't python
users or aren't programmers
the good news is there is plenty of free
and open source software that has
graphic user interfaces that will
construct these networks automatically
from data that is downloaded manually
from the websites of these various
citation databases and some of the
more useful ones i have put up on the
screen
for those that are python users there
may be packages that do a lot of this
but for my part i constructed a list of
the citation pairs
and then i gave this to the network x
package to construct a graph and then
generate the raw adjacency matrix for me
which was a fairly straight forward
process
so
we have our adjacency matrix we've got
our raw network set up the second step
half of step two comes with a major
methodological consideration
which is which connected is
connectedness measure to use
so essentially the raw citation network
is very sparse uh it contains a lot of
zeros it contains a lot of blank space
um and therefore it's very hard to
calculate any meaningful statistics with
this
matrix
so instead what we do is we calculate
connectedness measures to solidify this
matrix and we use the raw adjacency
matrix to calculate either a co-citation
matrix or a bibliographic coupling
matrix these things are both useful
but they measure slightly different
things so it's important to know the
difference between these two things when
you interpret your results
so to try to explain a co-citation
matrix represents how many times paper a
and paper b appear in the same reference
list of another article
um and a bibliographic coupling matrix
on the other hand measures how many
references paper a and paper b have in
common
and so
as you imagine the co-citation matrix is
most useful in the identification of
groups of influential papers important
in defining the past direction and
structure of your research field while
the bibliographic cutting matrix is a
little more useful for the
identification of clauses or papers that
draw on similar ideas and it's a
slightly more useful measure for
classifying recent papers that have yet
to be cited by others
so in python all you need to do here is
transpose the raw adjacency matrix and
multiply it by the original adjacency
matrix and further solidify i've applied
a cosine similarity function
on to step three which is multivariate
statistical analysis
so the standard method of citation
analysis is then to perform multivariate
statistical analysis or factor analysis
on our matrix this is a linear
math statistical method and it's a great
way to identify a smaller number of
factors when you have a large
uh but in the relationships between a
large number of underlying variables and
in our analysis we have a lot of
underlying
underlying variables because every
individual paper in the group of papers
that we are studying our research corpus
is a is an individual variable um so
what we really need to do is we need to
group these papers together we need to
identify a few underlying research
themes
um and these under the these underlying
factors will will uh these underlying
research themes might represent
different topics or different lines of
inquiry within your research field
um and this type of statistical analysis
has proved to be quite a robust way of
doing this kind of characterization of
your research field
so principal component analysis is the
most common type of factor analysis used
and there are various statistical
program packages in python that will
help you do this
so four network analysis and
visualization so in addition to the
factory analysis there are various other
quantitative measures you can use to
analyze your network um you have your
algorithms that will do your network
petitioning although that's not as good
as your factor analysis because network
partitioning algorithms will force you
to classify paper in one theme or the
other when in reality they could be
related to to more than one different
theme
um
but
the other useful thing that you can do
with your network is to calculate
various measures of centrality so you
have your things like your degree
centrality which is the number of times
a paper is cited by others in that
network um all the way through to
something like betweenness centrality
which means that your paper kind of
forms a node in a very in a frequent
path a path that's frequently um
used if you were to run an algorithm
going from one
side of your network to the other for
instance and that could indicate a high
between a centrality could indicate that
that paper has some kind of boundary
spanning properties um and is is
expanding across different research
ideas or different research silos
there are lots of python packages out
there that will help you do network
analysis i used igraph because it is
what i'm most familiar with but there's
the aforementioned network x and at
python this year i went to a talk by
someone who is
helping out with kg lab which is a
python project kind of looking to
integrate
all of these things
um and whilst these python packages will
draw decent represent visual
representations of the raw networks i do
recommend using software the graphic
user interface because that will make it
easier for you to play with your visual
properties if you are doing your network
diagrams
um doing graphics programmatically can
be a faff
so
step five interpretation of results
and validation
um so the important thing here is to go
back and look at each of the subgroups
of papers that you have identified so
your factor analysis will give every
individual paper a score for each factor
and you look at the groups of paper
which uh have a high score within each
factor usually above 0.3 or 0.5
depending on what tolerance you choose
and so you look at all those high
scoring papers in each under each factor
as a group
um and you can do things like
descriptive statistics looking at the
type the the universities and the
countries that are contributing research
in that subgroup um but most importantly
is to do your qualitative analysis and
now while this might seem quite daunting
when you have thousands of research
papers it is actually quite gratifying
at least in my experience
when even just skimming through the
titles of those subgroups just how
quickly those research themes emerge and
how easy it is to delineate those
different groups of papers within your
network um and to give you a better idea
of the type of insights that are
possible for from citation network
analysis i would like to run through
some of the key figures
from my upcoming paper
um so this is just a raw citation
network showing a number of different
keyword searches
this shows the kind of research output
that was um
downloaded for each of those keyword
searches over time and you can see the
trends in those different fields
over time
with my
fields kind of modest
modest increment kind of being dwarfed
by exponential increases in the interest
in things like smart cities and urban
science and urban analytics
um
this is a diagram that shows kind of the
first order relationships between
various different keyword searches
um and how closely though those
different areas are related
and this is my
research corpus itself so this is the
planning support systems literature and
it's been colored by the highest scoring
factor for each paper and gratifyingly
you can already see those clusters kind
of
within the
raw
um citation network as well
and the size of each of the little
bubbles is corresponds to the number of
citations
and like
you
and similar to the having those large
those few papers with a large number of
citations that's quite
quite normal in the citation network so
you'll have
um
a it's kind of like any social network
it follows a power law you'll find that
there are papers that act as hubs within
the networks which have exponentially
more citations
um than
the majority of
papers within your network which might
only have one or two
uh you can do these kinds of descriptive
statistics on the metadata like the
countries the institutions um the most
prolific authors
here's another representation that i did
of my different research streams kind of
showing the overlap between the papers
within those different research streams
and again those bubbles are
proportionate to the number
of papers within the research streams
uh here's another showing each of my
individual factors or research streams
to change changing those different
groups over time
so
um that's some of the things that you
can do with it and in conclusion
performing a citation network analysis
of course doesn't replace a traditional
literature review
doing a citation network analysis
doesn't tell you
um anything about the quality of the
research itself or even that much about
its findings but it is a really good
shortcut to kind of that overall view of
your research field and identifying
those key players
and those key lines of inquiry
um one word of warning for anyone who is
going to attempt this or
advising anyone to attempt this that
there are hundreds of different
statistics that can be calculated from
these citation data sets uh which was a
little overwhelming and that they're for
and then it's kind of a challenge for
the researcher to condense all of these
statistics into some kind of meaningful
and useful story
finally though to reiterate academic
literature is growing exponentially and
opening this data and these tools will
become more and more important both as a
means of just retaining
the ability to keep track of research
findings
um which will prevent duplication of
effort but also make research more
accessible to everyone
here is the main how-to text that i use
when i perform my own analysis and it's
a really useful textbook that is free to
download
um
also if you're in a specific field i
would recommend that you look for
previously published papers that utilize
this method to get an
idea
of how how it could be used in your own
speciality
um the full paper of my analysis will be
published very shortly it's been
accepted and i've submitted the final
documents when it is published i will
make sure the pre-print will go up on my
personal website and the code will be
available on my github account
so
thank you everyone and
happy to take any discussion or any
questions
wonderful thank you so much claire
um we currently have two questions uh
first one is i'm on the periphery of the
research world this seems to be useful
to use uh connections between papers
rather than the content of the paper is
that another area of research
connections between papers rather than
content so
okay so yes
um
the citation network analysis is good
for looking at getting that overall
structure
of the field um
and those identifying those
relationships and those different
research streams
you might identify kind of silos of
thought or or
kind of different groups of universities
that might be collaborating really
closely together
um
it's a really good way of making sure
you don't miss
those really key pieces of research that
have been done before that have been
cited thousands of times and because
they
i know i've done that before and i've
always felt stupid when i'm like oh i
should have known about this
um
but yes it doesn't it doesn't help you
really evaluate the quality of the
research as such so you still need to do
your traditional literature review or
you can do another type of systematic
literature review where you do
go in and read the papers exhaustively
and that kind of thing um yeah so this
is just another
another useful tool and those tools are
being um
with the with this data becoming so much
more open and a number of those open
source projects both with the apis but
also um
the free open source software which
anyone can can download and use
without necessarily
a statistics degree or a computer
programming degree um
it's kind of something that i would
recommend most people give a go before
they start a major research project
yeah interestingly um because i work in
the library sector we um try and link up
between universities and their
repositories as to who's got the same
papers and if there's duplicates do we
really need to store the duplicates so
that would be an interesting study as
well
um another question for you did claire
consider creating an intermediary mind
map of what she had found or taxonomy so
adding a halfway step that is more an
art than science
right
um
i suppose
what i did to make sense of it so when
you get your
results from your factor analysis
basically it will have
it will give you an eigenvalue
for each factor and that kind of
represents
it kind of gives it each factor scored
and
which tells you
the um how well that factor
is describing what percentage of
variation in your network that factor is
describing how important is it to the
entire structure of the network and it
will you'll usually have a few that have
really high and then it will
will drop off
kind of like that
and you kind of go one two three four
five so the numbers on my um
[Music]
let me see if i can bring up my
screen again
am i still showing my screen yeah oops
let's see how we go
so if i skip through
and look at
that's not that's not helping
stop sharing let's try again share share
screen
share screen uh screen too
and then people can probably see that
i'll just bring out my powerpoint that
might be easier okay
sorry back on track
all right so each of those numbers
represents the kind of ranked order of
those factors and what i've done
is kind of grouped them
in um
brought four broader things
um and i've done that by analyzing how
much they overlap in a more quantitative
way
um but there's also a little bit of
qualitative kind of
yeah assigning one
assigning a bubble to a category
particularly on those edges um was
was a qualitative decision so
yeah
wonderful
next question as you say there are many
different analysis options for your
network as well as
all the choices for how you construct
your network can you tell us more about
how you determined what story you wanted
to tell and which technical choices
would be useful for that
right
okay so the first major decision was
co-citation of google electric coupling
um i went with bibliographic coding
because i knew there would be a lot of
useful papers that probably haven't had
many citations
um it's still limited citation network
analysis is better at looking at what's
happened in the past and what is
happening right now or what might happen
in the future
um so i went bibliographic coupling and
then
then yeah it took it took a while i just
generated big dashboards for every
single
factor and you just go through it and
you're like okay this this this thing is
different about this factor suddenly
twice as many researchers from china are
kind of in this bubble why is that
um so going through those like you can
get python to print out however many
charts of your most frequent words or
your institutions and countries and all
that kind of thing
um
yeah and just combing through and being
like okay that's that's different that's
different why is that different
um but it is it is a time consuming
process to make it
to to kind of go through all of that
information even if it is technically
especially if you may be using a piece
of software you might be able to bring
up your network in a matter of minutes
but um yeah
making sense of it can take a while
uh i recommend looking for those
outliers yeah
one last question before we go to the
break
have you been able to determine if a
citation provider has an impact on
articles that are unearthed
how's the impact of articles that are on
earth i have not personally done that
research i am certain
that there are
papers out there with people who have
looked at that
um so
the person who
did this kind of analysis
is probably a good place to start
in terms of
that kind of research so they've been
really interested in the coverage of all
these different citation databases
um there was another study at flight
past that looked at the quality of the
search functions um and there's probably
various people who could do statistical
analysis looking at
um the correlation between citations and
it's it's sourced within a database or
it's accessibility within any of these
these kind of tools
wonderful well thank you so much if
anyone has any other questions for
claire claire will be in the
go glam community chat
channel and
we will be back with bonnie at 11 30 a.m
and we'll see you there thanks