[Music]
welcome back everyone um thank you
for
bearing with us until the end of today
um
next up we have
daniel giving us a talk about
musicbrainz.org and wikidata.org
um daniel is a frequent lca attendee and
i remember him coming to our first ever
go glam which was fantastic so it's good
to see familiar faces giving talks i've
been going to as many as lcas as you
have
oh geez
it's been a long time then um he's also
a music enthusiast and i will hand
straight over to daniel thanks
okay
hey everyone um
i'm gonna
give an introduction to two projects
that i've been editing in
um for many many years
um
and
hopefully
you learned something
um
so about me um
i'm daniel sobi i'm from adelaide
i attend the conf um sort of in my
spare time sitting in front of the tv
i occasionally edit a website called
musicbrainz.org
um
i have 31 000 edits and i'm not one of
the software developers or
there's things called auto editors that
can auto approve things i'm just a
normal
person that's just been stuck around for
ages
um
so the two projects
that i'm going to talk about is a
musicbrainz.org
it's a website
it's got music data it's got people
things about artists labels
recordings it's
not necessarily all music
there are things like audio books and
other things that are sort of music
related
we don't really care if it's a
commercial cd or
someone's soundcloud that they've
uploaded something it just
has to exist
and uh the other project um
i'm going to talk about is uh
wikidata.org
it's sort of a
parallel database used by wikipedia
to um
to link
all the sites together and to store
factual information
so there's going to be things like the
height of
everest and the height of all the large
mountains
presidents
all sorts of things
films you name it there's probably an
entry for it
and
it's a
another database
that anyone can edit and they sort of
have two different approaches to how
they're structured a bit so it's sort of
related sort of do things differently
and
hopefully if i get to it i'll
explain some of that on the way
so
it's database of musicbrainz it's
database of audio recordings
um each entity has a
long string a uuid
that's
really important because
there's an awful lot of name collision
and that's sort of thing so there are
two bands called pendulum
they both have rear awards
you have to be able to say which one it
is
when dealing with
such a big database
it's old user pretty much all user
edited
there's pretty much no bots that edit
um it's
um
hosted
and run by a
non-profit organization called
metabrains.org
they do get funding
the data is all public
um
mostly creative com zero license
uh they get funding from uh things like
google and spotify and
other
big music companies
um
[Music]
so we provide the raw data and
they'll use the data to help fix up
their system and that's sort of
um been enough to help
fund the organization and keep a few
key employees keeping the lights on
fixing bugs adding new features
um
there's a public api
anyone can do um use it
uh the thing that we need
from that if you're gonna use the public
api is
in your http headers you need to include
a string to say who you are so we can
let you know if you're doing something
really bad
that is occasionally being needed and
if you have a qnap
nas
they would be
um
hitting the api every nas that
was in existence was hitting the api so
much
that eventually they decided to block
them and if you need to
use the api you need to say can you
please unblock me until they updated the
software to not pit than that every
single every five minutes
um
each entry hasn't edited history
so you can easily say who was the idiot
that
made this change and
okay hopefully it's not you that made it
stuff things up
so
the basic data structure is we've got
artists
we've got release groups
which i'll get to in a minute but that's
sort of the
overall concept of an album so
there is a release which is a specific
one so you might have a cd with
13 tracks and a special edition with 18
tracks
and another edition with
two cds
so
to
fit
all the concept of these different
editions this the release group so
anything that fits in the
overall
properties
the data stores the release group
anything that's specific to a
a
particular cd goes on the release
there is medium so
there's
we can have
data about things like
records and
wax cylinders and all sorts of other
medium types
uh
you can add
of course i've got recordings
we'll get to that bit later
there's works
which is sort of the writing credits and
that sort of thing as opposed to
who sung or not
um there's labels
um the series
uh relationships between the two and
then there's a few other things like
um we've got a definitive list of
all possible instruments
and things like that that are maintained
by the
the editors and developers
so
here's
an example of what you'd go to see daft
punk
um
one thing that you might want to
take notice of
for the from the glam perspective is
uh we
try to link to other databases and have
identifiers as much as we can so
ipi and isis are
um
therefore
right societies and that sort of thing
to track a particular person or a group
um
there's attributes to say
who are the members of the group they'll
be another
artist entry so it
so you can have
further detail
um
we link to official home pages um
youtube
pages
twitter accounts that sort of thing
so if you've got a sound cloud
we want that sort of information so that
makes it easier for
if someone finds an artist they can
find more information about them
so
yeah lots of
lots of urls lots of
extra data
so
uh release groups
sort of the overall thing so
one thing that i'd like
to
to include is if you
look down a bit further that
um you can have um
link to
singles linked to other releases and
there's a whole bunch of relationships
of this relates to this other thing
so everything's built
with relationships
that you as an editor can add and
expanding
um the knowledge so
so if you look at
associated singles from
random access memories they've got get
lucky instant crush etc
so getting back down to the
further level of a release
you can have things like barcode
there's
cover art isn't hosted by
musicbrainz.org
that's hosted by archive.org for on our
behalf
so if you
upload cover uh
archive.org will take
we'll host it for us and
if there's a copyright issue
they can deal with it as another
organization
um
you've got a list of tracks
there
there'll be catalog numbers and that
sort of thing
um
so mediums
um
there'll be a type so it'll be cd or dvd
or
just digital to download
um
one of the edge cases that i took or two
educations that i'd like to
point out is
a thing called a pre-gap track
um
some cds when people were experimenting
um a cd is not a doesn't have a table of
con a proper table of contents it's not
a data track
it's just pcm audio
and
it's got a
when you put in a cd it reads
the table of contents is start at
one minute start at three minutes
started five minutes for the next track
with a two seven gap
um there are a handful of cds that have
hidden tracks
so you put the cd in
you press rewind and there's hidden data
there
so
if you've got some of the early hilltop
hoods
they have hidden tracks
it's just them chatting but
some cool things like that
um
the other example
um
is a from
nine inch nails broken
what they did is
the maximum tracks that you can have on
a cd is 99
so they had
the first six were normal
then
a whole bunch of tracks that had
a fraction of a second as audio and then
the last two
are the tracks 98 and 99 so
if you've got a cd player
it would play skip to last two and you
can't go back really easily because
there's a whole bunch of silence
so
recordings is where you add an awful lot
of data
if you it's sort of up to the users how
um
good the data is but
uh the schema allowance for things like
who played best bass guitar who played
keyboard who who
sung so
with some of these really popular tracks
you get
extra data that you can use to tag your
music
and
the other thing is they usually if you
someone's gone to the time
they'll have works
so works is
who wrote the thing
um
it's got um
ids from right societies
allowing someone else to
double check your work
and
they're all linked to
to who actually wrote it and
yeah
um
the other thing that
uh
you want to take note of is uh classical
music
uh
in music brains is sort of
more important
to have works than it is to have
artists on track
because the classical music
they tend to deal with
um
barriers which have movements and sub
sub subsections so having that sort of
structure
as a series of works that link to other
sub works
allows
people that are interested in classical
music to say
i want this want to listen to this bits
um
so
if you've got time
feel free to look at
there's also labels
which is sort of who you are into the
copyright and who is the publisher
and uh
there's a
list of string
list of uh
series is sort of something that they've
added
sort of five years ago for things like
compilation albums which
it's going to be the ministry is down
year number
that sort of thing
so everything is a
sort of fixed structure
um creating
um
[Music]
creating new relationships that's sort
of hard coded
so
there's a lot of
asking in forums discussing tickets on
what gets added
and some things like musical instruments
there's a person that
has a list of possible music instruments
and you've got to ask them to add it to
the system in the background back end
um
there's a query api which
you send it a string and it'll return
you with the album or the barcode or
something like that
and then you probably want to look up
quit do a look up thing to say give me
info about this id
this
album
release group etc
so the other
sort of related thing
if i've got time is um
let's talk about wikidata
so that sort of built in
a different sort of thing it's
everything is item property value
so everything's built from that sort of
basic structure
so
if you go to
daf punk's wiki data entry
it's an instance of an electronic
uh electronic duo which is a
instance of a musical group
they've got start and end years
he does good for things like um
awards received
you can easily add more rewards so they
find a grammy
record that properly
and it's another thing for
good source of
including external line identifiers
so you can
from music brains you can link to
wikidata and from wikidata you can link
back to musicbrainz so
looking at one you can look at the other
um properties are added so relationships
are added
through a voting process process
you sort of
go through that process and say i want
to have
for pro something so going from a talk
earlier
if i could
add my
web id
from talk earlier
that was a
something that was proposed
and
give it a what format of your urls uh
and
pattern matching and that's how wikidata
extends there
possible things that you can
uh
use
so querying
um
is done through a thing called a graphql
it's done through a thing called a spa
through a sparkle i mean
it sort of looks like this
you
you have a
list of select
statements so you define what you want
you'll have
a list of where sort of
clauses to say
this property on an entry
look for things with this property on
the entry
where's
exist or not exist this such and such
limit
so this is sort of the
that's
thing um
so
changing gears to
if you wanted to look at
uh things with the act my id
this is the sort of query that you'd
have to say
give me the
act my id which is that there
and
optionally return genre if they've got
that on the
film
option include the country the director
and based on so
there's all sorts of things that you can
do with
with that so
i think my time is
nearly up
so
yes thank you so much and we've actually
got quite a few questions for you which
is fantastic
um first question is are there any
metadata standards for this stuff or is
music brains just the de facto
it's we've got an api and you use the
api it's yeah
it's all
um
it's available in
json.xml idf
it's sort of
it used to be
they've de-emphasized idf because
no one was really using it it was an api
that no one
really used so everyone just uses json
yeah pretty much
okay
um how does music brains deal with
erroneous or conflicting data
um
it's usually someone will edit it out so
i think the data quality is it's pretty
much
there is 90 95 accurate
looking at random cd
find the occasional typho here or there
but
um sort of
because the way that the editing system
is it enforces constraints so that
people are less likely to make idiot
moves you sort of
feeds you down the path to the most
correct
data and someone can then
um come back later and fix your mistakes
um in the streaming world how does music
brains work
it
doesn't really make a difference it's an
album it's got a list of tracks
it
yeah
it
there's a digital medium so
you might get
it that it's digital and that's it
okay
um do record labels or artists send data
to music brains in a way that can be
ingested easily or are they just
generally unhelpful
uh they're generally unhelpful there's
there's been a few proposals of back-end
systems but
they've sort of been
against automation a fair bit it's sort
of
we want quality instead of
random junk that the label sent you so
yeah um and one last question is it only
via api or are there forms that people
can fill in to
add data
so you just go the website
if you want to
find a
find dark punk
punk
so say if i wanted to
edit this and
mess with the title or something
forget it
and
that's the title ah cool
change the
article
something else
click next next
and
click accept but you need to put in that
note to say what you're doing so
yeah
sort of now that seems quite simple you
follow the bounding box and hopefully
get there
we should um we should all contribute to
this um what's relatively easy today so
um that's fantastic well thank you so
much daniel that's um
it was great to see those those two
there's two sites for us to contribute
to because i always know that i'm happy
to put in
more data where possible to help out so
thank you so much
go to musicbrainz.org
download a program called picard
that's the tagger that they work on and
there's a few other taggers so
some of the cd rippers will
you put the cd in it'll
automatically retrieve the data
wonderful
thank you
okay thank you