1
00:00:06,320 --> 00:00:11,499
[Music]

2
00:00:16,000 --> 00:00:19,199
good afternoon and everybody welcome

3
00:00:17,440 --> 00:00:20,960
back to the final session for the day

4
00:00:19,199 --> 00:00:23,439
and the final session for lca in this

5
00:00:20,960 --> 00:00:25,599
room uh we have yemel joining us from

6
00:00:23,439 --> 00:00:29,840
sunny london town where it's a beautiful

7
00:00:25,599 --> 00:00:31,279
5 degrees outside and 6 45 a.m so all of

8
00:00:29,840 --> 00:00:32,800
us who are having coffee to get through

9
00:00:31,279 --> 00:00:34,960
the rest of the day yaml's probably

10
00:00:32,800 --> 00:00:37,360
starting his to get him going

11
00:00:34,960 --> 00:00:40,160
so yaml is a long-term

12
00:00:37,360 --> 00:00:42,800
debian developer working from london but

13
00:00:40,160 --> 00:00:44,079
working out of norway so i feel his

14
00:00:42,800 --> 00:00:45,600
sympathies with him working for an

15
00:00:44,079 --> 00:00:47,039
international company the clock is

16
00:00:45,600 --> 00:00:48,480
always wrong no matter where you are in

17
00:00:47,039 --> 00:00:50,559
the world somewhere that you're working

18
00:00:48,480 --> 00:00:52,320
with and having a meeting with so yemel

19
00:00:50,559 --> 00:00:55,120
over to you and looking forward to hear

20
00:00:52,320 --> 00:00:56,719
all about debbie and janitor

21
00:00:55,120 --> 00:00:59,920
all right uh yeah thanks uh thanks

22
00:00:56,719 --> 00:01:02,399
michael um yeah so today i'm going to

23
00:00:59,920 --> 00:01:04,320
talk about various ways of automating

24
00:01:02,399 --> 00:01:05,840
debian packaging and this is a

25
00:01:04,320 --> 00:01:08,560
continuation of work that i've been

26
00:01:05,840 --> 00:01:09,600
doing for the last couple of years

27
00:01:08,560 --> 00:01:12,080
um

28
00:01:09,600 --> 00:01:13,680
and what i'm going to talk about today

29
00:01:12,080 --> 00:01:15,520
um

30
00:01:13,680 --> 00:01:17,360
is something that's part of the the

31
00:01:15,520 --> 00:01:19,600
debian projects

32
00:01:17,360 --> 00:01:21,840
but a lot of the lessons and the tools

33
00:01:19,600 --> 00:01:24,240
apply outside of debian as well to other

34
00:01:21,840 --> 00:01:26,640
distributions but also to to upstream

35
00:01:24,240 --> 00:01:29,119
projects

36
00:01:26,640 --> 00:01:30,799
um so let me start by

37
00:01:29,119 --> 00:01:33,439
giving a little bit of an introduction

38
00:01:30,799 --> 00:01:36,159
um to to debian uh you're probably

39
00:01:33,439 --> 00:01:38,000
familiar with it um but maybe it's good

40
00:01:36,159 --> 00:01:40,320
to to sort of uh

41
00:01:38,000 --> 00:01:44,399
take a closer look at actually what what

42
00:01:40,320 --> 00:01:47,200
that distribution is trying to achieve

43
00:01:44,399 --> 00:01:49,759
um so linux distributions like debian

44
00:01:47,200 --> 00:01:51,840
fulfill an important function in the fos

45
00:01:49,759 --> 00:01:53,840
ecosystem uh there are system

46
00:01:51,840 --> 00:01:55,759
integrators that take existing free and

47
00:01:53,840 --> 00:01:58,000
open source software projects

48
00:01:55,759 --> 00:01:59,840
and adapt them where necessary to work

49
00:01:58,000 --> 00:02:01,759
well together

50
00:01:59,840 --> 00:02:03,759
they also make it possible for users to

51
00:02:01,759 --> 00:02:05,200
install more software in an easy and

52
00:02:03,759 --> 00:02:07,040
consistent way

53
00:02:05,200 --> 00:02:08,560
and with some degree of quality control

54
00:02:07,040 --> 00:02:10,800
and review

55
00:02:08,560 --> 00:02:12,720
and of course you could you could argue

56
00:02:10,800 --> 00:02:15,280
over how much review and quality control

57
00:02:12,720 --> 00:02:18,160
happens for some packages in debian

58
00:02:15,280 --> 00:02:20,239
but at least there is there's some

59
00:02:18,160 --> 00:02:21,760
minimal bar

60
00:02:20,239 --> 00:02:24,000
so packages have been reviewed for their

61
00:02:21,760 --> 00:02:25,200
license they have to meet certain parts

62
00:02:24,000 --> 00:02:26,239
of the

63
00:02:25,200 --> 00:02:29,680
um

64
00:02:26,239 --> 00:02:31,280
of debian's rules debian's policy

65
00:02:29,680 --> 00:02:33,120
and

66
00:02:31,280 --> 00:02:35,760
there are mechanisms

67
00:02:33,120 --> 00:02:37,920
to encourage conformance where the

68
00:02:35,760 --> 00:02:40,480
policy is not

69
00:02:37,920 --> 00:02:40,480
absolute

70
00:02:40,959 --> 00:02:45,519
and in the same way

71
00:02:43,680 --> 00:02:47,440
debian makes some promises around how

72
00:02:45,519 --> 00:02:49,680
long security updates are provided for

73
00:02:47,440 --> 00:02:50,879
example

74
00:02:49,680 --> 00:02:53,040
now if you're installing software

75
00:02:50,879 --> 00:02:54,160
directly from an upstream project

76
00:02:53,040 --> 00:02:57,760
um

77
00:02:54,160 --> 00:02:59,680
you would only be able to um

78
00:02:57,760 --> 00:03:01,840
get these sorts of guarantees for a

79
00:02:59,680 --> 00:03:03,519
certain number of upstream projects uh

80
00:03:01,840 --> 00:03:05,840
but you wouldn't be able to get that

81
00:03:03,519 --> 00:03:07,440
that consistency across across all of

82
00:03:05,840 --> 00:03:10,000
the things you installed

83
00:03:07,440 --> 00:03:11,840
um and there is some variety between

84
00:03:10,000 --> 00:03:13,280
linux distributions so that being falls

85
00:03:11,840 --> 00:03:14,480
squarely on the side of trying to

86
00:03:13,280 --> 00:03:16,080
integrate the different upstream

87
00:03:14,480 --> 00:03:18,080
projects more tightly

88
00:03:16,080 --> 00:03:20,400
and generally has much more complicated

89
00:03:18,080 --> 00:03:22,239
patches and and packaging and other

90
00:03:20,400 --> 00:03:24,159
distributions fall bore on the other

91
00:03:22,239 --> 00:03:26,560
side and they have much smaller

92
00:03:24,159 --> 00:03:28,080
uh shim around the the upstream project

93
00:03:26,560 --> 00:03:29,040
like arch for example

94
00:03:28,080 --> 00:03:31,760
um

95
00:03:29,040 --> 00:03:33,360
where um the packaging is is much

96
00:03:31,760 --> 00:03:37,120
smaller it's it's much easier to make

97
00:03:33,360 --> 00:03:40,480
packages um but there's also uh less

98
00:03:37,120 --> 00:03:40,480
integration work that has happened

99
00:03:41,840 --> 00:03:46,480
now if you've probably heard of debian

100
00:03:43,920 --> 00:03:47,599
even if you haven't used it debian is

101
00:03:46,480 --> 00:03:49,599
one of the older and larger

102
00:03:47,599 --> 00:03:51,440
distributions

103
00:03:49,599 --> 00:03:53,519
debian has about a thousand uploading

104
00:03:51,440 --> 00:03:57,200
developers and many more people who

105
00:03:53,519 --> 00:03:59,200
contribute changes on a regular basis

106
00:03:57,200 --> 00:04:01,760
there are about 30 000 source packages

107
00:03:59,200 --> 00:04:04,319
in debian and that basically means that

108
00:04:01,760 --> 00:04:06,159
there are about 30 000 upstream projects

109
00:04:04,319 --> 00:04:07,680
that get integrated into the

110
00:04:06,159 --> 00:04:08,640
distribution

111
00:04:07,680 --> 00:04:10,480
um

112
00:04:08,640 --> 00:04:12,560
so that's that's quite a lot um and

113
00:04:10,480 --> 00:04:14,879
there are many more

114
00:04:12,560 --> 00:04:18,479
binary packages that get created from

115
00:04:14,879 --> 00:04:18,479
those 30 files and source packages

116
00:04:19,280 --> 00:04:24,479
and then debian has a large policy that

117
00:04:22,320 --> 00:04:25,840
specifies what packages binary packages

118
00:04:24,479 --> 00:04:27,120
in particular

119
00:04:25,840 --> 00:04:29,120
should look like

120
00:04:27,120 --> 00:04:31,120
uh and and comply with

121
00:04:29,120 --> 00:04:33,120
uh the development of the policy is is

122
00:04:31,120 --> 00:04:36,560
mostly consensus driven

123
00:04:33,120 --> 00:04:39,840
uh but the policy also tries to be uh

124
00:04:36,560 --> 00:04:42,479
generally flexible about how um it is

125
00:04:39,840 --> 00:04:45,199
met so

126
00:04:42,479 --> 00:04:45,919
maybe the shape of a binary package is

127
00:04:45,199 --> 00:04:48,880
is

128
00:04:45,919 --> 00:04:50,800
particularly well documented

129
00:04:48,880 --> 00:04:52,880
you can use many different

130
00:04:50,800 --> 00:04:55,440
tools on the build sites

131
00:04:52,880 --> 00:04:56,479
to achieve that so there's no

132
00:04:55,440 --> 00:04:59,120
um

133
00:04:56,479 --> 00:05:02,479
it's not very well dictated exactly how

134
00:04:59,120 --> 00:05:02,479
you should do that implementation

135
00:05:02,639 --> 00:05:07,919
and then of course there are many many

136
00:05:04,400 --> 00:05:11,520
tools available that can

137
00:05:07,919 --> 00:05:12,320
can can do the same thing

138
00:05:11,520 --> 00:05:14,479
um

139
00:05:12,320 --> 00:05:16,560
and then each package has a single owner

140
00:05:14,479 --> 00:05:18,880
so called maintainer

141
00:05:16,560 --> 00:05:21,440
and with team ownership uh

142
00:05:18,880 --> 00:05:23,520
now being more common than individual

143
00:05:21,440 --> 00:05:25,520
maintainership so

144
00:05:23,520 --> 00:05:27,440
generally there's a set of people who

145
00:05:25,520 --> 00:05:30,720
maintain

146
00:05:27,440 --> 00:05:34,400
a usually a large number of packages

147
00:05:30,720 --> 00:05:35,759
and keep those packages in a good shape

148
00:05:34,400 --> 00:05:37,440
that's different from some other

149
00:05:35,759 --> 00:05:40,000
distributions where the ownership model

150
00:05:37,440 --> 00:05:42,160
for each package is often a lot weaker

151
00:05:40,000 --> 00:05:44,400
and where it's um

152
00:05:42,160 --> 00:05:46,639
more common for other contributors to to

153
00:05:44,400 --> 00:05:48,479
step in

154
00:05:46,639 --> 00:05:49,840
it's it's not hard and fast in debian

155
00:05:48,479 --> 00:05:52,720
either but

156
00:05:49,840 --> 00:05:54,639
traditionally it's been very

157
00:05:52,720 --> 00:05:56,639
much that there was

158
00:05:54,639 --> 00:05:58,720
a maintainer who took care of a package

159
00:05:56,639 --> 00:06:01,840
and and nobody else would

160
00:05:58,720 --> 00:06:01,840
would really touch it

161
00:06:01,919 --> 00:06:04,160
um

162
00:06:02,880 --> 00:06:05,680
and then

163
00:06:04,160 --> 00:06:07,759
source packages in debian are

164
00:06:05,680 --> 00:06:10,240
essentially just turbos with some some

165
00:06:07,759 --> 00:06:12,400
debian specific metadata

166
00:06:10,240 --> 00:06:15,199
and uploading a new version of a package

167
00:06:12,400 --> 00:06:18,240
generally involves copying some pgp

168
00:06:15,199 --> 00:06:18,240
signed turbos around

169
00:06:19,759 --> 00:06:25,759
um

170
00:06:21,440 --> 00:06:28,080
so that that's um how debian um

171
00:06:25,759 --> 00:06:31,280
started and and uh

172
00:06:28,080 --> 00:06:33,039
how the basics work um but

173
00:06:31,280 --> 00:06:35,440
things have moved on since the

174
00:06:33,039 --> 00:06:37,600
distribution was originally created

175
00:06:35,440 --> 00:06:39,520
um

176
00:06:37,600 --> 00:06:42,880
when debian started store starburst with

177
00:06:39,520 --> 00:06:45,919
a way of delivering software either on a

178
00:06:42,880 --> 00:06:49,280
cd or a floppy or maybe even over this

179
00:06:45,919 --> 00:06:51,919
this shiny internet thing

180
00:06:49,280 --> 00:06:55,440
and that was true for for both upstream

181
00:06:51,919 --> 00:06:58,960
projects as well as debian itself

182
00:06:55,440 --> 00:07:01,039
nowadays git is fairly ubiquitous and

183
00:06:58,960 --> 00:07:03,840
most upstream projects and most packages

184
00:07:01,039 --> 00:07:05,520
in debian use it

185
00:07:03,840 --> 00:07:07,120
and there are still exceptions of course

186
00:07:05,520 --> 00:07:08,479
that don't use any version control

187
00:07:07,120 --> 00:07:11,039
system at all

188
00:07:08,479 --> 00:07:12,400
or maybe that use a different version

189
00:07:11,039 --> 00:07:14,080
control system

190
00:07:12,400 --> 00:07:17,120
but those exceptions are are getting

191
00:07:14,080 --> 00:07:19,520
more and more rare

192
00:07:17,120 --> 00:07:21,280
as a as a version control geek i'm i'm a

193
00:07:19,520 --> 00:07:24,240
little bit sad that

194
00:07:21,280 --> 00:07:26,080
we have this this monoculture now uh but

195
00:07:24,240 --> 00:07:27,520
git is a really great tool and it's it's

196
00:07:26,080 --> 00:07:30,560
nice to not have to deal with half a

197
00:07:27,520 --> 00:07:34,960
dozen different systems like we had

198
00:07:30,560 --> 00:07:34,960
like we had to back in 2009

199
00:07:38,160 --> 00:07:42,080
so today in debian there isn't really a

200
00:07:40,240 --> 00:07:45,840
predictable location for where git

201
00:07:42,080 --> 00:07:45,840
repositories with debian packages live

202
00:07:46,160 --> 00:07:49,599
however we've worked around that by

203
00:07:47,919 --> 00:07:51,919
adding a header to the debian control

204
00:07:49,599 --> 00:07:53,759
file in the source package that points

205
00:07:51,919 --> 00:07:56,000
at the git repository

206
00:07:53,759 --> 00:07:58,479
or technically

207
00:07:56,000 --> 00:08:00,800
could also be the mercurial

208
00:07:58,479 --> 00:08:02,800
repository location the bizarre branch

209
00:08:00,800 --> 00:08:05,759
location the subversion repository

210
00:08:02,800 --> 00:08:10,800
location the arch location cvs location

211
00:08:05,759 --> 00:08:12,639
or the monoto monotone repo location

212
00:08:10,800 --> 00:08:13,759
but as you can see from the the graph on

213
00:08:12,639 --> 00:08:15,840
the right

214
00:08:13,759 --> 00:08:18,319
um

215
00:08:15,840 --> 00:08:19,440
basically all of the packages nowadays

216
00:08:18,319 --> 00:08:21,599
use git

217
00:08:19,440 --> 00:08:23,280
um and there are some holdouts that you

218
00:08:21,599 --> 00:08:26,280
don't use any version control system at

219
00:08:23,280 --> 00:08:26,280
all

220
00:08:26,400 --> 00:08:30,720
um these headers have been around for

221
00:08:28,879 --> 00:08:32,159
quite some time

222
00:08:30,720 --> 00:08:34,000
um

223
00:08:32,159 --> 00:08:36,399
but they're awesome and there are tools

224
00:08:34,000 --> 00:08:38,000
around them to automatically

225
00:08:36,399 --> 00:08:39,120
for example

226
00:08:38,000 --> 00:08:42,640
check out

227
00:08:39,120 --> 00:08:44,320
the git repository for a package

228
00:08:42,640 --> 00:08:45,760
and there's a service that will monitor

229
00:08:44,320 --> 00:08:46,880
all of the packages that have these

230
00:08:45,760 --> 00:08:49,120
header sets

231
00:08:46,880 --> 00:08:51,519
and that will notify you whenever for

232
00:08:49,120 --> 00:08:52,959
example a repository goes away or when

233
00:08:51,519 --> 00:08:55,839
it breaks or when it doesn't match

234
00:08:52,959 --> 00:08:55,839
what's in the archive

235
00:08:57,680 --> 00:09:04,480
and from for most packages this this

236
00:09:00,080 --> 00:09:04,480
header is pretty accurate which is great

237
00:09:06,000 --> 00:09:08,640
right

238
00:09:06,880 --> 00:09:10,080
and then there's another control file in

239
00:09:08,640 --> 00:09:11,200
debian packages

240
00:09:10,080 --> 00:09:13,279
um

241
00:09:11,200 --> 00:09:16,480
called debian upstream metadata and

242
00:09:13,279 --> 00:09:18,560
that's a standard that defines um

243
00:09:16,480 --> 00:09:20,800
a format based on yaml that allows the

244
00:09:18,560 --> 00:09:22,560
maintainer to specify

245
00:09:20,800 --> 00:09:24,720
various bits of metadata about the

246
00:09:22,560 --> 00:09:27,279
packages upstream project

247
00:09:24,720 --> 00:09:28,800
so for example where does the upstream's

248
00:09:27,279 --> 00:09:30,880
documentation live

249
00:09:28,800 --> 00:09:32,240
where is its wiki where can you report

250
00:09:30,880 --> 00:09:33,760
bugs

251
00:09:32,240 --> 00:09:35,200
but also

252
00:09:33,760 --> 00:09:37,040
where does the upstream project's

253
00:09:35,200 --> 00:09:38,399
version control system live

254
00:09:37,040 --> 00:09:40,160
there's a little bit of

255
00:09:38,399 --> 00:09:41,279
overlap here with other control fields

256
00:09:40,160 --> 00:09:42,880
in debian

257
00:09:41,279 --> 00:09:45,040
but that's something that's being worked

258
00:09:42,880 --> 00:09:45,040
on

259
00:09:45,279 --> 00:09:51,120
um so this format is a draft but it's

260
00:09:48,480 --> 00:09:52,959
been widely adopted um there's about 10

261
00:09:51,120 --> 00:09:54,560
000 packages or so that have this this

262
00:09:52,959 --> 00:09:57,200
file present

263
00:09:54,560 --> 00:09:59,519
and about 8 000 of those

264
00:09:57,200 --> 00:10:01,920
have the repository header set so for

265
00:09:59,519 --> 00:10:04,640
about a third of all of the

266
00:10:01,920 --> 00:10:07,519
30 000 source packages in debian

267
00:10:04,640 --> 00:10:10,160
we know where we can find the upstream

268
00:10:07,519 --> 00:10:11,760
source repository

269
00:10:10,160 --> 00:10:13,279
and if you're a maintainer you can you

270
00:10:11,760 --> 00:10:14,959
can use the

271
00:10:13,279 --> 00:10:16,480
guest upstream metadata command to

272
00:10:14,959 --> 00:10:18,079
automatically

273
00:10:16,480 --> 00:10:19,760
generate one of these files and that

274
00:10:18,079 --> 00:10:21,839
command will actually

275
00:10:19,760 --> 00:10:23,440
do things like introspect setup.pre-wi

276
00:10:21,839 --> 00:10:25,760
if you're a python package

277
00:10:23,440 --> 00:10:27,680
or look at the make file or in in some

278
00:10:25,760 --> 00:10:30,800
cases it'll even just

279
00:10:27,680 --> 00:10:33,279
um scrape the readme file for anything

280
00:10:30,800 --> 00:10:34,959
that looks like a git repository and and

281
00:10:33,279 --> 00:10:37,760
it tries to see whether it matches

282
00:10:34,959 --> 00:10:37,760
what's in the package

283
00:10:44,320 --> 00:10:48,240
another innovation that's happened uh in

284
00:10:46,320 --> 00:10:50,640
the last couple of years is auto package

285
00:10:48,240 --> 00:10:53,120
test um so other package tests defines a

286
00:10:50,640 --> 00:10:55,120
framework for running tests um

287
00:10:53,120 --> 00:10:55,920
on the debian binary package

288
00:10:55,120 --> 00:10:58,480
um

289
00:10:55,920 --> 00:10:59,760
so uh when you run them you can make you

290
00:10:58,480 --> 00:11:01,040
can be sure that the package has been

291
00:10:59,760 --> 00:11:03,600
built correctly

292
00:11:01,040 --> 00:11:04,399
and functions as a user would expect

293
00:11:03,600 --> 00:11:07,839
um

294
00:11:04,399 --> 00:11:10,480
so about half of the archive now um

295
00:11:07,839 --> 00:11:12,880
has sd set um

296
00:11:10,480 --> 00:11:14,480
and that is great because it allows you

297
00:11:12,880 --> 00:11:16,560
to

298
00:11:14,480 --> 00:11:18,160
verify when you make a change

299
00:11:16,560 --> 00:11:20,160
that you haven't broken anything

300
00:11:18,160 --> 00:11:22,240
significant

301
00:11:20,160 --> 00:11:24,320
there's also a service ci debian.net

302
00:11:22,240 --> 00:11:26,240
that regularly runs these packages runs

303
00:11:24,320 --> 00:11:28,240
these tests against packages

304
00:11:26,240 --> 00:11:31,519
as well as their dependencies uh

305
00:11:28,240 --> 00:11:31,519
whenever one of them changes

306
00:11:33,440 --> 00:11:37,600
and then finally

307
00:11:34,880 --> 00:11:39,839
uh one of the uh the recent developments

308
00:11:37,600 --> 00:11:41,920
or um

309
00:11:39,839 --> 00:11:44,320
or rather one of the one of the things

310
00:11:41,920 --> 00:11:46,959
that has been evolving is um

311
00:11:44,320 --> 00:11:49,600
a gradual adoption of of just a single

312
00:11:46,959 --> 00:11:52,079
tool for building packages um so in the

313
00:11:49,600 --> 00:11:54,320
past there were many different ways of

314
00:11:52,079 --> 00:11:55,760
building a debian package

315
00:11:54,320 --> 00:11:56,880
you could pick a random package from the

316
00:11:55,760 --> 00:11:59,120
archive

317
00:11:56,880 --> 00:12:01,839
and depending on the maintainer's

318
00:11:59,120 --> 00:12:04,079
preference you would find one of

319
00:12:01,839 --> 00:12:05,920
about five different tools

320
00:12:04,079 --> 00:12:07,279
um and you would have to be familiar

321
00:12:05,920 --> 00:12:08,639
with all five different tools if you

322
00:12:07,279 --> 00:12:11,839
wanted to

323
00:12:08,639 --> 00:12:11,839
contribute to a random package

324
00:12:12,079 --> 00:12:14,399
um

325
00:12:16,959 --> 00:12:21,440
one of the uh or the tool that we've

326
00:12:19,600 --> 00:12:23,279
sort of settled on is

327
00:12:21,440 --> 00:12:25,360
called deb helper

328
00:12:23,279 --> 00:12:27,360
and the dev helper dh interface in

329
00:12:25,360 --> 00:12:29,839
particular one of the other things

330
00:12:27,360 --> 00:12:31,440
that's nice about the dh interface is

331
00:12:29,839 --> 00:12:34,880
that

332
00:12:31,440 --> 00:12:37,040
deb helper can increasingly figure out

333
00:12:34,880 --> 00:12:40,240
how to just do the right thing rather

334
00:12:37,040 --> 00:12:42,880
than requiring a lot of configuration

335
00:12:40,240 --> 00:12:45,200
so that makes the packaging less effort

336
00:12:42,880 --> 00:12:46,959
but it also means that it's less likely

337
00:12:45,200 --> 00:12:48,240
that for example when you import a new

338
00:12:46,959 --> 00:12:50,240
upstream version

339
00:12:48,240 --> 00:12:53,600
um that you will need to make extensive

340
00:12:50,240 --> 00:12:55,040
changes to the packaging

341
00:12:53,600 --> 00:12:56,959
so that's sort of

342
00:12:55,040 --> 00:12:58,320
a lot of background on on what has

343
00:12:56,959 --> 00:13:00,880
happened in

344
00:12:58,320 --> 00:13:01,680
the debian ecosystem over the last

345
00:13:00,880 --> 00:13:03,839
um

346
00:13:01,680 --> 00:13:06,880
maybe five to ten years

347
00:13:03,839 --> 00:13:07,839
um that has made

348
00:13:06,880 --> 00:13:10,399
it's

349
00:13:07,839 --> 00:13:14,240
has improved the um

350
00:13:10,399 --> 00:13:15,600
the way we can make automated changes

351
00:13:14,240 --> 00:13:17,440
um

352
00:13:15,600 --> 00:13:19,040
so let's now have a look at making

353
00:13:17,440 --> 00:13:21,440
large-scale changes in the debian

354
00:13:19,040 --> 00:13:23,120
ecosystem because there are some unique

355
00:13:21,440 --> 00:13:25,920
challenges because

356
00:13:23,120 --> 00:13:27,120
every package is basically maintained as

357
00:13:25,920 --> 00:13:30,079
a

358
00:13:27,120 --> 00:13:30,079
project by itself

359
00:13:30,959 --> 00:13:34,320
um so if you're ever if you've ever

360
00:13:33,120 --> 00:13:36,880
touched the debian package you're

361
00:13:34,320 --> 00:13:38,560
probably familiar with linden so linting

362
00:13:36,880 --> 00:13:39,920
is a it's a linting tool that can scan

363
00:13:38,560 --> 00:13:42,800
debian packages

364
00:13:39,920 --> 00:13:44,800
and report common packaging issues

365
00:13:42,800 --> 00:13:45,920
so it categorizes issues by so-called

366
00:13:44,800 --> 00:13:46,800
tags

367
00:13:45,920 --> 00:13:48,639
and

368
00:13:46,800 --> 00:13:50,800
you see the documentation for one of

369
00:13:48,639 --> 00:13:52,959
those stacks here

370
00:13:50,800 --> 00:13:55,839
so this stack in particular uh carriage

371
00:13:52,959 --> 00:13:55,839
return line feed

372
00:13:56,480 --> 00:14:01,600
gets reported whenever

373
00:13:58,560 --> 00:14:04,320
a debian control file contains a

374
00:14:01,600 --> 00:14:04,320
turned characters

375
00:14:08,480 --> 00:14:12,639
or sorry carries turn line feed

376
00:14:10,399 --> 00:14:15,639
characters and

377
00:14:12,639 --> 00:14:15,639
um

378
00:14:16,639 --> 00:14:19,680
suggests that maintainers basically

379
00:14:18,240 --> 00:14:21,920
strip the um

380
00:14:19,680 --> 00:14:24,000
the line feeds from there uh many tools

381
00:14:21,920 --> 00:14:26,079
in debian will actually many build tools

382
00:14:24,000 --> 00:14:27,680
and debian will actually run lynchian

383
00:14:26,079 --> 00:14:29,760
straight after building

384
00:14:27,680 --> 00:14:30,959
um and the hope is that like if we

385
00:14:29,760 --> 00:14:33,360
report these

386
00:14:30,959 --> 00:14:35,839
tags uh that maintainers will go and and

387
00:14:33,360 --> 00:14:35,839
fix them

388
00:14:38,079 --> 00:14:42,160
um and so the traditional way of making

389
00:14:40,320 --> 00:14:43,760
changes across debian packages is to

390
00:14:42,160 --> 00:14:44,720
somehow make developers aware of an

391
00:14:43,760 --> 00:14:46,560
issue

392
00:14:44,720 --> 00:14:47,519
for example through lynchian tag or bug

393
00:14:46,560 --> 00:14:48,880
reports

394
00:14:47,519 --> 00:14:50,480
or maybe some other form of

395
00:14:48,880 --> 00:14:52,160
communication

396
00:14:50,480 --> 00:14:55,360
and then you get everybody to pick up on

397
00:14:52,160 --> 00:14:57,519
the problem and fix it

398
00:14:55,360 --> 00:14:59,440
so you get them to care and make the

399
00:14:57,519 --> 00:15:02,399
appropriate changes and upload new

400
00:14:59,440 --> 00:15:04,160
packages to the archive

401
00:15:02,399 --> 00:15:05,360
and then after you've done that and

402
00:15:04,160 --> 00:15:07,120
you've waited

403
00:15:05,360 --> 00:15:10,240
enough years

404
00:15:07,120 --> 00:15:13,199
you may end up with a significant chunk

405
00:15:10,240 --> 00:15:14,160
of the archive fixed

406
00:15:13,199 --> 00:15:15,839
um

407
00:15:14,160 --> 00:15:18,079
generally though there will be a long

408
00:15:15,839 --> 00:15:19,760
tail of packages that aren't updated um

409
00:15:18,079 --> 00:15:21,519
there are a bunch of packages in debian

410
00:15:19,760 --> 00:15:24,000
that haven't been touched in years like

411
00:15:21,519 --> 00:15:27,279
maybe they've been rebuilt

412
00:15:24,000 --> 00:15:29,360
but if they um if they've just just kept

413
00:15:27,279 --> 00:15:32,800
building um then there's there's

414
00:15:29,360 --> 00:15:33,680
generally no need to change them

415
00:15:32,800 --> 00:15:35,360
um

416
00:15:33,680 --> 00:15:39,120
but as you can see this is a very slow

417
00:15:35,360 --> 00:15:41,199
process it also requires a lot of um

418
00:15:39,120 --> 00:15:43,600
cumulative time across all debian

419
00:15:41,199 --> 00:15:45,040
developers so not only do they all need

420
00:15:43,600 --> 00:15:47,759
to be aware of the details of the

421
00:15:45,040 --> 00:15:48,720
problem they also only to allocate time

422
00:15:47,759 --> 00:15:49,680
to

423
00:15:48,720 --> 00:15:52,240
um

424
00:15:49,680 --> 00:15:54,480
to actually uh make the the changes and

425
00:15:52,240 --> 00:15:58,480
to upload a new package

426
00:15:54,480 --> 00:16:00,480
and frankly for for a lot of problems

427
00:15:58,480 --> 00:16:02,800
that that's probably not worth the

428
00:16:00,480 --> 00:16:02,800
effort

429
00:16:06,320 --> 00:16:10,160
um so let's let's go back to that

430
00:16:08,320 --> 00:16:11,279
lynchian output

431
00:16:10,160 --> 00:16:13,040
um

432
00:16:11,279 --> 00:16:14,720
you you'll notice that lynching actually

433
00:16:13,040 --> 00:16:16,240
tells you the command to run to fix the

434
00:16:14,720 --> 00:16:17,279
issue

435
00:16:16,240 --> 00:16:18,639
um

436
00:16:17,279 --> 00:16:20,560
so

437
00:16:18,639 --> 00:16:22,399
what if rather than requiring the user

438
00:16:20,560 --> 00:16:23,839
to run lynchian and then copy and paste

439
00:16:22,399 --> 00:16:26,639
the commands

440
00:16:23,839 --> 00:16:28,240
we just render command for them

441
00:16:26,639 --> 00:16:31,600
so that's that's basically the idea

442
00:16:28,240 --> 00:16:34,880
behind a tool called lynching brush

443
00:16:31,600 --> 00:16:36,880
so lynchian brush is a is a set of

444
00:16:34,880 --> 00:16:39,839
more than 100 different scripts that can

445
00:16:36,880 --> 00:16:42,079
automatically fix common linkedin issues

446
00:16:39,839 --> 00:16:44,160
there's also a framework that allows you

447
00:16:42,079 --> 00:16:46,160
to write those scripts easily and to run

448
00:16:44,160 --> 00:16:47,440
them so the framework provides all of

449
00:16:46,160 --> 00:16:49,279
the interaction

450
00:16:47,440 --> 00:16:51,279
with the git repository for you so you

451
00:16:49,279 --> 00:16:52,959
don't have to worry about that

452
00:16:51,279 --> 00:16:55,519
it'll also take care of updating the

453
00:16:52,959 --> 00:16:57,440
debian change log file for you

454
00:16:55,519 --> 00:16:59,120
if the package doesn't use

455
00:16:57,440 --> 00:17:01,680
fancy tools for for generating the

456
00:16:59,120 --> 00:17:01,680
change lock

457
00:17:01,920 --> 00:17:05,199
and it's got all kinds of cleverness

458
00:17:03,839 --> 00:17:06,720
underneath so

459
00:17:05,199 --> 00:17:08,799
um

460
00:17:06,720 --> 00:17:10,720
when you run the script it uses inotify

461
00:17:08,799 --> 00:17:13,280
to figure out when you make changes to

462
00:17:10,720 --> 00:17:14,720
the package it manages to get adds and

463
00:17:13,280 --> 00:17:16,319
removes

464
00:17:14,720 --> 00:17:17,600
it creates the commits if there are

465
00:17:16,319 --> 00:17:20,160
multiple

466
00:17:17,600 --> 00:17:22,400
scripts that run

467
00:17:20,160 --> 00:17:24,559
it cleans up if this if a script fails

468
00:17:22,400 --> 00:17:27,839
to run

469
00:17:24,559 --> 00:17:31,200
and then as mentioned it will update the

470
00:17:27,839 --> 00:17:33,360
changelog file appropriately

471
00:17:31,200 --> 00:17:34,799
um most of the things of the fixtures do

472
00:17:33,360 --> 00:17:36,720
things that would take developers a

473
00:17:34,799 --> 00:17:39,120
couple of minutes each

474
00:17:36,720 --> 00:17:42,400
and this this basically just makes it

475
00:17:39,120 --> 00:17:42,400
at most a couple of seconds

476
00:17:43,600 --> 00:17:47,280
per package

477
00:17:45,679 --> 00:17:48,960
this kind of automation doesn't really

478
00:17:47,280 --> 00:17:49,840
make that much of a difference

479
00:17:48,960 --> 00:17:52,080
like

480
00:17:49,840 --> 00:17:53,919
you take it down from maybe a couple of

481
00:17:52,080 --> 00:17:56,480
minutes to a couple of seconds

482
00:17:53,919 --> 00:17:57,600
um but you can just fire and forget and

483
00:17:56,480 --> 00:18:00,960
you can also

484
00:17:57,600 --> 00:18:04,559
if you look at the entire archive the

485
00:18:00,960 --> 00:18:04,559
the impact is more significant

486
00:18:05,600 --> 00:18:09,280
and if there's nothing for the tool to

487
00:18:08,160 --> 00:18:11,600
do

488
00:18:09,280 --> 00:18:12,480
it'll it'll usually

489
00:18:11,600 --> 00:18:15,840
run

490
00:18:12,480 --> 00:18:15,840
in less than a second

491
00:18:17,280 --> 00:18:21,440
um and this is a an example for what one

492
00:18:19,760 --> 00:18:24,320
of those scripts looks like

493
00:18:21,440 --> 00:18:26,160
uh so this is writing in in python

494
00:18:24,320 --> 00:18:27,440
most of the heavy lifting is done by the

495
00:18:26,160 --> 00:18:29,679
framework

496
00:18:27,440 --> 00:18:33,280
but in this case

497
00:18:29,679 --> 00:18:35,360
it's it's fixing um a up a particular

498
00:18:33,280 --> 00:18:37,120
field in the control file

499
00:18:35,360 --> 00:18:41,120
where

500
00:18:37,120 --> 00:18:43,520
previously debian supported a priority

501
00:18:41,120 --> 00:18:47,360
extra and that's been removed and should

502
00:18:43,520 --> 00:18:47,360
now be replaced with priority optional

503
00:18:48,480 --> 00:18:50,320
um

504
00:18:49,520 --> 00:18:52,799
and

505
00:18:50,320 --> 00:18:54,240
the way that the framework is written

506
00:18:52,799 --> 00:18:58,000
makes sure that

507
00:18:54,240 --> 00:18:58,000
if there are um

508
00:18:58,559 --> 00:19:04,480
formatting

509
00:19:00,799 --> 00:19:07,120
specialties um they are preserved

510
00:19:04,480 --> 00:19:09,280
and if the script can can preserve

511
00:19:07,120 --> 00:19:11,440
the previous formatting it'll just raise

512
00:19:09,280 --> 00:19:13,520
an exception

513
00:19:11,440 --> 00:19:15,520
that exception isn't handled here the

514
00:19:13,520 --> 00:19:17,760
script will fill

515
00:19:15,520 --> 00:19:17,760
and

516
00:19:18,240 --> 00:19:22,760
its changes will be discarded

517
00:19:24,640 --> 00:19:29,600
however um

518
00:19:27,440 --> 00:19:29,600
that

519
00:19:29,919 --> 00:19:34,000
having one of those tools and um

520
00:19:32,640 --> 00:19:35,120
available

521
00:19:34,000 --> 00:19:38,320
doesn't mean that everybody

522
00:19:35,120 --> 00:19:39,919
automatically starts running it

523
00:19:38,320 --> 00:19:42,400
so this is a graph of the total number

524
00:19:39,919 --> 00:19:46,480
of installations of linton brush from

525
00:19:42,400 --> 00:19:46,480
debian's popularity contest program

526
00:19:46,799 --> 00:19:51,360
lynching brush has been around since mid

527
00:19:49,280 --> 00:19:54,960
2018.

528
00:19:51,360 --> 00:19:58,720
um ideally we'd like to see it run

529
00:19:54,960 --> 00:20:01,039
um over all 30 000 packages in debian

530
00:19:58,720 --> 00:20:03,039
on a regular basis

531
00:20:01,039 --> 00:20:04,240
um

532
00:20:03,039 --> 00:20:07,039
but that's

533
00:20:04,240 --> 00:20:10,000
not something that can easily happen

534
00:20:07,039 --> 00:20:12,320
um and so one of the things that

535
00:20:10,000 --> 00:20:13,919
we would really like to do is

536
00:20:12,320 --> 00:20:16,320
um

537
00:20:13,919 --> 00:20:17,600
to run it across the archive without

538
00:20:16,320 --> 00:20:19,840
having to

539
00:20:17,600 --> 00:20:21,360
get every single developer to every

540
00:20:19,840 --> 00:20:24,640
month run it

541
00:20:21,360 --> 00:20:24,640
on each one of their packages

542
00:20:26,080 --> 00:20:29,039
and

543
00:20:27,200 --> 00:20:30,480
the way to do that is

544
00:20:29,039 --> 00:20:32,159
to

545
00:20:30,480 --> 00:20:34,400
basically bring the changes to them

546
00:20:32,159 --> 00:20:35,360
rather than requiring them to to run the

547
00:20:34,400 --> 00:20:37,600
tool

548
00:20:35,360 --> 00:20:39,600
um so we make it easy for people to

549
00:20:37,600 --> 00:20:41,120
apply these changes by

550
00:20:39,600 --> 00:20:43,600
creating merge proposals with the

551
00:20:41,120 --> 00:20:43,600
changes

552
00:20:44,080 --> 00:20:48,720
and there's a tool called silver bladder

553
00:20:46,799 --> 00:20:51,440
that automates all of that so it

554
00:20:48,720 --> 00:20:52,400
iterates over a set of git repositories

555
00:20:51,440 --> 00:20:54,559
um

556
00:20:52,400 --> 00:20:56,320
it applies a recipe which is generally

557
00:20:54,559 --> 00:20:58,559
just a command to run

558
00:20:56,320 --> 00:21:00,640
uh in the git repository

559
00:20:58,559 --> 00:21:04,320
um and then if there were changes

560
00:21:00,640 --> 00:21:07,280
created by the recipe it proposes them

561
00:21:04,320 --> 00:21:07,280
in a pull request

562
00:21:10,080 --> 00:21:14,240
and so this means that

563
00:21:12,480 --> 00:21:15,520
a single person can come up with a

564
00:21:14,240 --> 00:21:18,480
recipe

565
00:21:15,520 --> 00:21:20,240
can run it against all of the archive

566
00:21:18,480 --> 00:21:21,760
and the only thing that maintainers have

567
00:21:20,240 --> 00:21:23,760
to do is

568
00:21:21,760 --> 00:21:25,919
look at the merge request and hit the

569
00:21:23,760 --> 00:21:28,480
merge button if they think the changes

570
00:21:25,919 --> 00:21:28,480
are reasonable

571
00:21:29,440 --> 00:21:35,200
and the tool can also directly push to a

572
00:21:32,400 --> 00:21:37,280
git repository so if the user running it

573
00:21:35,200 --> 00:21:39,360
has appropriate permissions

574
00:21:37,280 --> 00:21:42,000
it can just directly push to the

575
00:21:39,360 --> 00:21:42,000
repository

576
00:21:43,520 --> 00:21:46,080
and then this is what it concretely

577
00:21:44,960 --> 00:21:49,120
looks like

578
00:21:46,080 --> 00:21:52,720
um so there's a yaml file with um

579
00:21:49,120 --> 00:21:54,559
the urls for the repositories to process

580
00:21:52,720 --> 00:21:56,159
and those can also be just names of

581
00:21:54,559 --> 00:21:57,840
debian packages

582
00:21:56,159 --> 00:21:59,760
in which case um

583
00:21:57,840 --> 00:22:01,760
the tool introspects the the headers

584
00:21:59,760 --> 00:22:03,600
that i mentioned earlier

585
00:22:01,760 --> 00:22:05,440
uh and then there's a recipe

586
00:22:03,600 --> 00:22:06,559
which basically specifies what command

587
00:22:05,440 --> 00:22:10,400
to run

588
00:22:06,559 --> 00:22:13,840
um and a template for the pull request

589
00:22:10,400 --> 00:22:13,840
that needs to be created

590
00:22:14,720 --> 00:22:19,919
um and then uh you can you can invoke

591
00:22:17,200 --> 00:22:22,400
the commands uh on those two files

592
00:22:19,919 --> 00:22:24,400
um ideally starting off with

593
00:22:22,400 --> 00:22:25,760
a dry run to see if the changes are

594
00:22:24,400 --> 00:22:27,600
reasonable

595
00:22:25,760 --> 00:22:29,280
and then you can run it

596
00:22:27,600 --> 00:22:31,919
and it just iterates over all of the

597
00:22:29,280 --> 00:22:31,919
repositories

598
00:22:32,000 --> 00:22:35,360
it can also

599
00:22:33,200 --> 00:22:37,840
build the debian package

600
00:22:35,360 --> 00:22:41,159
to verify that you haven't accidentally

601
00:22:37,840 --> 00:22:41,159
broken anything

602
00:22:42,320 --> 00:22:45,600
um

603
00:22:43,840 --> 00:22:48,159
so that's all well and good but debian

604
00:22:45,600 --> 00:22:51,600
has about 30 000 source packages

605
00:22:48,159 --> 00:22:54,559
and um using a command line tool to dry

606
00:22:51,600 --> 00:22:58,159
run review and upload 30 000

607
00:22:54,559 --> 00:23:00,000
uh changes is gonna be quite challenging

608
00:22:58,159 --> 00:23:01,760
um

609
00:23:00,000 --> 00:23:03,440
so um

610
00:23:01,760 --> 00:23:06,880
there's another project called the

611
00:23:03,440 --> 00:23:08,159
debian janitor which sits atop

612
00:23:06,880 --> 00:23:11,679
silver platter

613
00:23:08,159 --> 00:23:13,120
and provides a web interface

614
00:23:11,679 --> 00:23:16,000
so it

615
00:23:13,120 --> 00:23:19,919
looks at all of the packages in debian

616
00:23:16,000 --> 00:23:19,919
and regularly processes them

617
00:23:20,080 --> 00:23:24,480
to um

618
00:23:22,559 --> 00:23:26,159
make sure that's

619
00:23:24,480 --> 00:23:30,760
linked your brush for example but other

620
00:23:26,159 --> 00:23:30,760
tools as well get run regularly

621
00:23:38,159 --> 00:23:41,840
so the debian janitor is basically it's

622
00:23:40,559 --> 00:23:44,480
a website

623
00:23:41,840 --> 00:23:46,400
uh as well as a uh

624
00:23:44,480 --> 00:23:48,799
a scheduling interface

625
00:23:46,400 --> 00:23:50,400
that constantly watches the debian

626
00:23:48,799 --> 00:23:53,039
archive

627
00:23:50,400 --> 00:23:54,640
processes the entire archive regularly

628
00:23:53,039 --> 00:23:58,400
as well as

629
00:23:54,640 --> 00:23:58,400
packages that have been changed recently

630
00:24:02,559 --> 00:24:05,360
and an that they should in addition to

631
00:24:04,559 --> 00:24:07,360
that

632
00:24:05,360 --> 00:24:08,799
um it provides an interface where you

633
00:24:07,360 --> 00:24:09,919
can you can actually look at these

634
00:24:08,799 --> 00:24:12,240
changes

635
00:24:09,919 --> 00:24:14,000
um and you can do um

636
00:24:12,240 --> 00:24:16,159
qa and review

637
00:24:14,000 --> 00:24:17,679
uh as well as um

638
00:24:16,159 --> 00:24:20,080
managing all of the merge requests that

639
00:24:17,679 --> 00:24:20,080
are open

640
00:24:21,600 --> 00:24:25,200
um

641
00:24:23,760 --> 00:24:26,880
and one of the the more challenging

642
00:24:25,200 --> 00:24:28,159
parts of it is

643
00:24:26,880 --> 00:24:30,640
um

644
00:24:28,159 --> 00:24:34,400
how to do this in such a way

645
00:24:30,640 --> 00:24:36,159
that it delights users rather than um

646
00:24:34,400 --> 00:24:38,559
annoying them so

647
00:24:36,159 --> 00:24:39,760
there are some risks here

648
00:24:38,559 --> 00:24:41,679
um

649
00:24:39,760 --> 00:24:43,840
the goal behind all of these tools is to

650
00:24:41,679 --> 00:24:45,600
save developers time and effort by

651
00:24:43,840 --> 00:24:49,039
automating away things that can be done

652
00:24:45,600 --> 00:24:49,039
by a robot so that

653
00:24:49,279 --> 00:24:54,080
the the developers can focus on things

654
00:24:51,360 --> 00:24:56,720
that require human judgment

655
00:24:54,080 --> 00:24:58,080
um but there's no time saved and in fact

656
00:24:56,720 --> 00:25:00,480
quite the opposite

657
00:24:58,080 --> 00:25:01,360
if if the pull request is incorrect

658
00:25:00,480 --> 00:25:03,039
so

659
00:25:01,360 --> 00:25:04,880
the developer ends up spending time

660
00:25:03,039 --> 00:25:06,799
looking at the merge request

661
00:25:04,880 --> 00:25:08,720
and the package in the end isn't

662
00:25:06,799 --> 00:25:10,640
improved

663
00:25:08,720 --> 00:25:12,480
oh and then

664
00:25:10,640 --> 00:25:14,640
when the

665
00:25:12,480 --> 00:25:16,799
developer gets another merge request

666
00:25:14,640 --> 00:25:18,080
they're probably gonna be less likely to

667
00:25:16,799 --> 00:25:20,799
review it

668
00:25:18,080 --> 00:25:22,320
um and they're they're also gonna be

669
00:25:20,799 --> 00:25:24,640
less likely to

670
00:25:22,320 --> 00:25:27,279
just give the bots blanket access to the

671
00:25:24,640 --> 00:25:29,440
repository

672
00:25:27,279 --> 00:25:31,440
um so therefore the the actions of the

673
00:25:29,440 --> 00:25:32,559
bot are restricted to a very limited set

674
00:25:31,440 --> 00:25:34,240
of problems

675
00:25:32,559 --> 00:25:36,080
for which there is an obviously correct

676
00:25:34,240 --> 00:25:37,600
action that can be taken

677
00:25:36,080 --> 00:25:40,400
and it's not meant to automate all

678
00:25:37,600 --> 00:25:42,480
packaging or even to cover

679
00:25:40,400 --> 00:25:44,559
automating all instances of the issues

680
00:25:42,480 --> 00:25:47,840
that it knows about

681
00:25:44,559 --> 00:25:50,400
um so so basically

682
00:25:47,840 --> 00:25:52,080
if if it's unsure about something it

683
00:25:50,400 --> 00:25:54,559
just does nothing

684
00:25:52,080 --> 00:25:56,240
there's nothing really lost there

685
00:25:54,559 --> 00:25:58,240
some packages use very complicated

686
00:25:56,240 --> 00:26:00,880
packaging methods that lintone brush

687
00:25:58,240 --> 00:26:02,799
just doesn't understand

688
00:26:00,880 --> 00:26:04,880
um for example they're control files

689
00:26:02,799 --> 00:26:06,000
that are generated in complicated ways

690
00:26:04,880 --> 00:26:08,400
um

691
00:26:06,000 --> 00:26:10,960
i'm looking at you people who use m4 to

692
00:26:08,400 --> 00:26:15,360
generate your control files

693
00:26:10,960 --> 00:26:15,360
and lynching brush just ignores those

694
00:26:19,919 --> 00:26:22,320
and then

695
00:26:23,600 --> 00:26:29,120
when we process

696
00:26:25,919 --> 00:26:31,200
the archive with with the debian janitor

697
00:26:29,120 --> 00:26:33,600
we only allow

698
00:26:31,200 --> 00:26:36,400
changes that we have a very high

699
00:26:33,600 --> 00:26:36,400
certainty about

700
00:26:36,799 --> 00:26:40,720
after changes have been made we do a

701
00:26:38,799 --> 00:26:42,799
builds of the package

702
00:26:40,720 --> 00:26:45,120
uh with and without the changes no

703
00:26:42,799 --> 00:26:47,200
matter how trivial those changes were

704
00:26:45,120 --> 00:26:48,960
uh if that fails for whatever reason we

705
00:26:47,200 --> 00:26:49,919
discard the changes

706
00:26:48,960 --> 00:26:51,919
um

707
00:26:49,919 --> 00:26:54,400
then we run the the other package test

708
00:26:51,919 --> 00:26:56,080
as well if one of those fills we discard

709
00:26:54,400 --> 00:26:57,840
the changes

710
00:26:56,080 --> 00:27:00,480
and then finally there's a qa step where

711
00:26:57,840 --> 00:27:02,240
a human looks at the source package diff

712
00:27:00,480 --> 00:27:05,440
and sometimes that also

713
00:27:02,240 --> 00:27:05,440
helps us catch issues

714
00:27:05,520 --> 00:27:10,480
um and then finally we we only open a

715
00:27:08,320 --> 00:27:11,440
single merge request per tool per

716
00:27:10,480 --> 00:27:13,760
package

717
00:27:11,440 --> 00:27:15,360
so if we process the package again

718
00:27:13,760 --> 00:27:16,480
we'll just update the existing merge

719
00:27:15,360 --> 00:27:18,960
request

720
00:27:16,480 --> 00:27:22,159
rather than creating a new one and

721
00:27:18,960 --> 00:27:22,159
having you look at that again

722
00:27:23,360 --> 00:27:28,159
um and then per maintainer there is a

723
00:27:26,399 --> 00:27:30,240
rate limits

724
00:27:28,159 --> 00:27:32,480
on the number of open merch requests so

725
00:27:30,240 --> 00:27:34,559
we don't really want to

726
00:27:32,480 --> 00:27:37,279
spam you with um

727
00:27:34,559 --> 00:27:39,520
100 merge requests if you have a hundred

728
00:27:37,279 --> 00:27:41,840
packages that you maintain

729
00:27:39,520 --> 00:27:43,279
so what we do is

730
00:27:41,840 --> 00:27:46,080
um

731
00:27:43,279 --> 00:27:48,720
basically tcp slow start so

732
00:27:46,080 --> 00:27:50,000
we'll send you one merge request if you

733
00:27:48,720 --> 00:27:52,559
ignore that

734
00:27:50,000 --> 00:27:55,520
we won't send you any more

735
00:27:52,559 --> 00:27:57,120
um if you merge it we'll send you two

736
00:27:55,520 --> 00:28:00,399
more

737
00:27:57,120 --> 00:28:03,520
if you merge those we'll send you four

738
00:28:00,399 --> 00:28:06,480
etc until basically until we reach a

739
00:28:03,520 --> 00:28:06,480
limit of 20.

740
00:28:07,120 --> 00:28:10,080
um

741
00:28:08,159 --> 00:28:11,760
and then in addition to that

742
00:28:10,080 --> 00:28:14,159
uh the bot

743
00:28:11,760 --> 00:28:15,919
tries very hard to identify itself as a

744
00:28:14,159 --> 00:28:17,679
bot so that you know that the merge

745
00:28:15,919 --> 00:28:18,960
request is automated

746
00:28:17,679 --> 00:28:21,520
um

747
00:28:18,960 --> 00:28:23,760
and can prioritize

748
00:28:21,520 --> 00:28:25,840
its merge requests appropriately

749
00:28:23,760 --> 00:28:26,799
but otherwise you can just interact with

750
00:28:25,840 --> 00:28:28,640
it

751
00:28:26,799 --> 00:28:31,039
as you would with a

752
00:28:28,640 --> 00:28:32,720
merger request that came from a human

753
00:28:31,039 --> 00:28:34,240
so basically if you comment on one of

754
00:28:32,720 --> 00:28:36,960
the merge requests

755
00:28:34,240 --> 00:28:38,159
one of the developers from the janitor

756
00:28:36,960 --> 00:28:41,399
will look at

757
00:28:38,159 --> 00:28:41,399
your comments

758
00:28:50,399 --> 00:28:54,559
so

759
00:28:51,360 --> 00:28:58,240
um yeah then how much does it take to

760
00:28:54,559 --> 00:28:59,760
keep keep this beast running

761
00:28:58,240 --> 00:29:02,640
one of the key things

762
00:28:59,760 --> 00:29:04,799
that we look at regularly is the

763
00:29:02,640 --> 00:29:06,559
rejections of merge proposals so if

764
00:29:04,799 --> 00:29:08,000
people close merge proposals without

765
00:29:06,559 --> 00:29:10,080
merging them

766
00:29:08,000 --> 00:29:12,159
they'll usually leave a comment

767
00:29:10,080 --> 00:29:14,240
and and we go and we look at why they

768
00:29:12,159 --> 00:29:15,840
close those

769
00:29:14,240 --> 00:29:17,600
so in some cases they've already made

770
00:29:15,840 --> 00:29:19,360
the change themselves

771
00:29:17,600 --> 00:29:21,279
um or maybe they introduced the merge

772
00:29:19,360 --> 00:29:24,480
conflict before they noticed

773
00:29:21,279 --> 00:29:26,880
the pull request for the package

774
00:29:24,480 --> 00:29:28,960
in other cases um sometimes they

775
00:29:26,880 --> 00:29:31,279
disagree with the change

776
00:29:28,960 --> 00:29:33,679
um so sometimes we

777
00:29:31,279 --> 00:29:36,799
end up discussing the chains with them

778
00:29:33,679 --> 00:29:38,320
um and occasionally that leads to

779
00:29:36,799 --> 00:29:42,000
bug reports for one of the tools

780
00:29:38,320 --> 00:29:43,279
involved in the interchange

781
00:29:42,000 --> 00:29:46,320
um

782
00:29:43,279 --> 00:29:48,159
if we find a bug that might affect other

783
00:29:46,320 --> 00:29:50,399
pull requests as well

784
00:29:48,159 --> 00:29:52,080
we go and proactively find all of those

785
00:29:50,399 --> 00:29:55,039
pool requests

786
00:29:52,080 --> 00:29:57,200
and we reprocess them

787
00:29:55,039 --> 00:29:58,480
and then if we if we can't fix things

788
00:29:57,200 --> 00:30:00,720
immediately

789
00:29:58,480 --> 00:30:02,000
uh we'll just disable the the tools in

790
00:30:00,720 --> 00:30:05,360
question

791
00:30:02,000 --> 00:30:07,840
um until we we fix the issue so that

792
00:30:05,360 --> 00:30:09,840
um there are no developers that end up

793
00:30:07,840 --> 00:30:12,159
looking at changes that we know are

794
00:30:09,840 --> 00:30:12,159
wrong

795
00:30:13,200 --> 00:30:18,159
um

796
00:30:15,360 --> 00:30:19,600
and um of course reviewing all of these

797
00:30:18,159 --> 00:30:20,840
changes takes quite a bit of time as

798
00:30:19,600 --> 00:30:23,520
well

799
00:30:20,840 --> 00:30:26,399
so uh every single change is still

800
00:30:23,520 --> 00:30:28,799
reviewed by uh one of the the janitor

801
00:30:26,399 --> 00:30:30,640
developers before it goes

802
00:30:28,799 --> 00:30:32,480
out in a pull request

803
00:30:30,640 --> 00:30:33,440
uh but that's something that i'm hoping

804
00:30:32,480 --> 00:30:35,200
to

805
00:30:33,440 --> 00:30:36,960
automate away in the near near term

806
00:30:35,200 --> 00:30:38,880
future

807
00:30:36,960 --> 00:30:40,720
so at the moment this is like hundreds

808
00:30:38,880 --> 00:30:42,880
of diffs a day

809
00:30:40,720 --> 00:30:45,200
and that's that's quite a lot of manual

810
00:30:42,880 --> 00:30:45,200
effort

811
00:30:46,960 --> 00:30:49,360
um

812
00:30:47,760 --> 00:30:52,320
so that's that's what's been happening

813
00:30:49,360 --> 00:30:53,360
for uh about a year and a half

814
00:30:52,320 --> 00:30:55,520
um

815
00:30:53,360 --> 00:30:57,519
what have we achieved so far

816
00:30:55,520 --> 00:30:58,880
so

817
00:30:57,519 --> 00:31:00,320
unfortunately we're in the winter

818
00:30:58,880 --> 00:31:02,559
process of

819
00:31:00,320 --> 00:31:04,399
uh gluing the history from our old and

820
00:31:02,559 --> 00:31:06,640
new database together

821
00:31:04,399 --> 00:31:10,880
so i've only really got good graphs from

822
00:31:06,640 --> 00:31:10,880
up until last decembe last september

823
00:31:11,919 --> 00:31:15,200
but this is the graph with the number of

824
00:31:13,760 --> 00:31:16,559
changes we've

825
00:31:15,200 --> 00:31:19,840
directly pushed to packaging

826
00:31:16,559 --> 00:31:21,679
repositories up until that point

827
00:31:19,840 --> 00:31:23,679
so this is for repositories where the

828
00:31:21,679 --> 00:31:25,120
maintainer has given right access to the

829
00:31:23,679 --> 00:31:27,440
bot

830
00:31:25,120 --> 00:31:30,640
and as you can see we've almost hit the

831
00:31:27,440 --> 00:31:32,720
10 000 gauges mark back in september and

832
00:31:30,640 --> 00:31:34,960
in fact uh we've we've since gone over

833
00:31:32,720 --> 00:31:34,960
that

834
00:31:39,039 --> 00:31:42,240
um and then there's the number of uh

835
00:31:41,039 --> 00:31:44,399
merger

836
00:31:42,240 --> 00:31:46,640
merge proposals or pool requests

837
00:31:44,399 --> 00:31:48,000
depending on the terminology you want to

838
00:31:46,640 --> 00:31:50,720
use

839
00:31:48,000 --> 00:31:52,399
so we've had about 7 000 of our merchant

840
00:31:50,720 --> 00:31:54,399
fossils merged so far

841
00:31:52,399 --> 00:31:56,660
and we've got about a thousand more open

842
00:31:54,399 --> 00:31:58,080
at the moment

843
00:31:56,660 --> 00:31:59,360
[Music]

844
00:31:58,080 --> 00:32:02,080
we have

845
00:31:59,360 --> 00:32:03,840
many more changes queued up to propose

846
00:32:02,080 --> 00:32:05,600
uh but because of the restrictions that

847
00:32:03,840 --> 00:32:07,039
i talked about earlier

848
00:32:05,600 --> 00:32:08,880
uh we haven't created those pull

849
00:32:07,039 --> 00:32:09,919
requests yet so that's that's all

850
00:32:08,880 --> 00:32:10,880
because

851
00:32:09,919 --> 00:32:13,600
um

852
00:32:10,880 --> 00:32:16,159
we we will rate limit the number of pull

853
00:32:13,600 --> 00:32:18,799
requests that's open for an individual

854
00:32:16,159 --> 00:32:18,799
maintainer

855
00:32:20,080 --> 00:32:22,240
um

856
00:32:20,880 --> 00:32:23,360
there's some interesting artifacts on

857
00:32:22,240 --> 00:32:25,360
this graph

858
00:32:23,360 --> 00:32:27,760
that's mostly related to

859
00:32:25,360 --> 00:32:29,200
either the tooling being paused when we

860
00:32:27,760 --> 00:32:31,840
discovered bugs

861
00:32:29,200 --> 00:32:33,679
um or related to the

862
00:32:31,840 --> 00:32:34,559
freeze for the debian release early last

863
00:32:33,679 --> 00:32:36,159
year

864
00:32:34,559 --> 00:32:39,799
where we stopped

865
00:32:36,159 --> 00:32:39,799
submitting pull requests

866
00:32:42,240 --> 00:32:47,120
um and then this is how the um

867
00:32:45,440 --> 00:32:48,880
the merge proposals are distributed over

868
00:32:47,120 --> 00:32:50,799
the different hosting sites

869
00:32:48,880 --> 00:32:52,640
um so it's in the earlier snapshot so

870
00:32:50,799 --> 00:32:54,159
the numbers don't quite add up to those

871
00:32:52,640 --> 00:32:56,640
in the previous graph

872
00:32:54,159 --> 00:32:58,000
but as you can tell um all of the

873
00:32:56,640 --> 00:33:00,000
changes happen

874
00:32:58,000 --> 00:33:04,000
or most of the changes happened almost

875
00:33:00,000 --> 00:33:07,279
exclusively on uh salsa on debian's

876
00:33:04,000 --> 00:33:07,279
own instance of gitlab

877
00:33:08,159 --> 00:33:12,559
and the rejection rate is is fairly uh

878
00:33:11,440 --> 00:33:15,039
low so

879
00:33:12,559 --> 00:33:17,279
um close basically means

880
00:33:15,039 --> 00:33:19,440
um that the maintainer closed

881
00:33:17,279 --> 00:33:20,320
the the merge request and didn't accept

882
00:33:19,440 --> 00:33:21,600
it

883
00:33:20,320 --> 00:33:25,240
um

884
00:33:21,600 --> 00:33:25,240
in any way

885
00:33:27,279 --> 00:33:30,880
um

886
00:33:29,440 --> 00:33:34,080
there are a couple of

887
00:33:30,880 --> 00:33:35,919
challenges still associated with this

888
00:33:34,080 --> 00:33:38,320
um so

889
00:33:35,919 --> 00:33:40,399
if you get your changes accepted um and

890
00:33:38,320 --> 00:33:42,799
merged into a git repository that

891
00:33:40,399 --> 00:33:44,880
doesn't mean that they end up in the

892
00:33:42,799 --> 00:33:46,640
archive um so

893
00:33:44,880 --> 00:33:49,600
um

894
00:33:46,640 --> 00:33:51,519
it might take uh years before the

895
00:33:49,600 --> 00:33:52,799
changes that were merged into a git

896
00:33:51,519 --> 00:33:54,399
repository

897
00:33:52,799 --> 00:33:55,840
are actually part of an upload to the

898
00:33:54,399 --> 00:33:56,880
debian archive

899
00:33:55,840 --> 00:34:01,039
um

900
00:33:56,880 --> 00:34:02,799
and um last week we checked about 52

901
00:34:01,039 --> 00:34:06,279
of changes haven't actually been

902
00:34:02,799 --> 00:34:06,279
uploaded yet

903
00:34:08,320 --> 00:34:12,720
um there is a some work on going to make

904
00:34:11,040 --> 00:34:14,879
it easier for

905
00:34:12,720 --> 00:34:17,119
people to upload changes that are in a

906
00:34:14,879 --> 00:34:21,119
git repository to the archive

907
00:34:17,119 --> 00:34:23,280
uh but those haven't haven't learned yet

908
00:34:21,119 --> 00:34:27,440
um and then there's also some

909
00:34:23,280 --> 00:34:30,079
challenging some challenges around

910
00:34:27,440 --> 00:34:32,000
the the long tail packages that haven't

911
00:34:30,079 --> 00:34:33,599
switched over to a version control

912
00:34:32,000 --> 00:34:36,000
system yet

913
00:34:33,599 --> 00:34:38,480
or that still claim that they are hosted

914
00:34:36,000 --> 00:34:40,159
on um elif which is

915
00:34:38,480 --> 00:34:41,760
the previous hosting platform that

916
00:34:40,159 --> 00:34:44,079
debian used

917
00:34:41,760 --> 00:34:47,359
and that was deprecated

918
00:34:44,079 --> 00:34:50,399
i think over four years ago

919
00:34:47,359 --> 00:34:51,839
um so maybe uh we should consider

920
00:34:50,399 --> 00:34:54,480
alternatives like

921
00:34:51,839 --> 00:34:56,639
uh maybe not creating pull requests but

922
00:34:54,480 --> 00:35:01,320
maybe sending um

923
00:34:56,639 --> 00:35:01,320
bug reports with with diffs attached

924
00:35:02,000 --> 00:35:05,920
and then finally um

925
00:35:04,000 --> 00:35:09,040
one of the challenges that we have with

926
00:35:05,920 --> 00:35:10,880
um with gitlab is the fact that

927
00:35:09,040 --> 00:35:12,720
uh for a lot of repositories there's

928
00:35:10,880 --> 00:35:14,960
nobody who gets notified when there are

929
00:35:12,720 --> 00:35:17,119
pull requests so the pull requests just

930
00:35:14,960 --> 00:35:19,200
sit there

931
00:35:17,119 --> 00:35:21,040
and that's not because people don't care

932
00:35:19,200 --> 00:35:23,680
it's just because they don't know about

933
00:35:21,040 --> 00:35:23,680
the pull request

934
00:35:29,200 --> 00:35:33,359
so i talked about lintium brush earlier

935
00:35:31,280 --> 00:35:35,839
that fixes um

936
00:35:33,359 --> 00:35:38,240
smaller issues in debian packages but

937
00:35:35,839 --> 00:35:41,119
there are some very other some other

938
00:35:38,240 --> 00:35:44,400
very common operations that packagers do

939
00:35:41,119 --> 00:35:46,000
that could benefit from automation

940
00:35:44,400 --> 00:35:47,359
and with the recent improvements to the

941
00:35:46,000 --> 00:35:48,560
egg system that i was talking about

942
00:35:47,359 --> 00:35:50,320
earlier

943
00:35:48,560 --> 00:35:52,400
it actually becomes feasible in a lot of

944
00:35:50,320 --> 00:35:55,680
situations to update the debian package

945
00:35:52,400 --> 00:35:58,560
to a new upstream version automatically

946
00:35:55,680 --> 00:35:59,760
of course that requires that

947
00:35:58,560 --> 00:36:02,240
the

948
00:35:59,760 --> 00:36:04,400
packaging gets repository is available

949
00:36:02,240 --> 00:36:06,079
and that the upstream git repository is

950
00:36:04,400 --> 00:36:08,560
available

951
00:36:06,079 --> 00:36:11,359
but for a lot of packages we actually

952
00:36:08,560 --> 00:36:13,839
have that information

953
00:36:11,359 --> 00:36:15,280
um and um

954
00:36:13,839 --> 00:36:17,520
if we can make this work for some

955
00:36:15,280 --> 00:36:18,640
fraction of packages even if it's like

956
00:36:17,520 --> 00:36:21,280
ten percent

957
00:36:18,640 --> 00:36:23,280
uh that could still be very variable

958
00:36:21,280 --> 00:36:26,280
because it's about um three thousand

959
00:36:23,280 --> 00:36:26,280
packages

960
00:36:29,280 --> 00:36:32,960
um

961
00:36:31,440 --> 00:36:34,960
yeah and one of the consequences of

962
00:36:32,960 --> 00:36:36,960
debian's model is that uh the

963
00:36:34,960 --> 00:36:38,400
distribution packages often lag behind

964
00:36:36,960 --> 00:36:41,359
upstream releases

965
00:36:38,400 --> 00:36:43,599
um so that's that's especially true for

966
00:36:41,359 --> 00:36:45,680
um distributions where

967
00:36:43,599 --> 00:36:48,079
uh it takes more work to integrate a new

968
00:36:45,680 --> 00:36:48,079
release

969
00:36:49,280 --> 00:36:52,160
um

970
00:36:50,320 --> 00:36:53,520
and and there are uh

971
00:36:52,160 --> 00:36:54,720
users who

972
00:36:53,520 --> 00:36:58,160
would prefer

973
00:36:54,720 --> 00:36:58,160
more of a rolling distribution

974
00:36:58,400 --> 00:37:00,640
um

975
00:36:59,599 --> 00:37:02,240
so

976
00:37:00,640 --> 00:37:03,760
yeah what does that the process look

977
00:37:02,240 --> 00:37:05,119
like for updating a package to a new

978
00:37:03,760 --> 00:37:08,560
version today

979
00:37:05,119 --> 00:37:10,960
um there's a there's a lot of individual

980
00:37:08,560 --> 00:37:13,359
uh steps that a maintainer needs to take

981
00:37:10,960 --> 00:37:15,599
and um

982
00:37:13,359 --> 00:37:18,560
some of these steps are now somewhat

983
00:37:15,599 --> 00:37:20,560
automated um but um

984
00:37:18,560 --> 00:37:22,320
until recently there wasn't really a

985
00:37:20,560 --> 00:37:24,560
single command that could replace all of

986
00:37:22,320 --> 00:37:24,560
these

987
00:37:24,720 --> 00:37:29,839
um as part of the the janitor work we've

988
00:37:27,599 --> 00:37:31,920
developed a new command

989
00:37:29,839 --> 00:37:34,480
that you can basically just run in any

990
00:37:31,920 --> 00:37:36,640
debian git repository and they will

991
00:37:34,480 --> 00:37:39,040
attempt to pull in

992
00:37:36,640 --> 00:37:41,680
a new upstream release or a new upstream

993
00:37:39,040 --> 00:37:45,280
git snapshot

994
00:37:41,680 --> 00:37:46,720
and it takes care of all of the

995
00:37:45,280 --> 00:37:48,839
work that needs to happen along with

996
00:37:46,720 --> 00:37:51,280
that such as updating the

997
00:37:48,839 --> 00:37:53,920
changelog to some extent updating the

998
00:37:51,280 --> 00:37:56,000
packaging as well

999
00:37:53,920 --> 00:37:58,000
running building and run and running the

1000
00:37:56,000 --> 00:38:00,560
test for the package

1001
00:37:58,000 --> 00:38:00,560
etc

1002
00:38:03,280 --> 00:38:09,280
this is uh all of the steps that it goes

1003
00:38:05,760 --> 00:38:10,640
through i won't uh go into detail

1004
00:38:09,280 --> 00:38:13,760
but i'm happy to

1005
00:38:10,640 --> 00:38:13,760
talk about this afterwards

1006
00:38:15,680 --> 00:38:20,000
um

1007
00:38:17,440 --> 00:38:22,320
so this this importing of new upstream

1008
00:38:20,000 --> 00:38:24,400
releases has been happening for

1009
00:38:22,320 --> 00:38:25,359
uh the last year and a bit

1010
00:38:24,400 --> 00:38:28,079
um

1011
00:38:25,359 --> 00:38:30,079
and we've attempted to import a new

1012
00:38:28,079 --> 00:38:32,160
upstream git snapshot and a new upstream

1013
00:38:30,079 --> 00:38:33,599
release for every single package in the

1014
00:38:32,160 --> 00:38:36,560
archive

1015
00:38:33,599 --> 00:38:38,800
where the required information was

1016
00:38:36,560 --> 00:38:41,040
available

1017
00:38:38,800 --> 00:38:41,040
and

1018
00:38:41,920 --> 00:38:44,400
this is

1019
00:38:44,560 --> 00:38:48,079
what we

1020
00:38:46,320 --> 00:38:50,079
what we ended up doing

1021
00:38:48,079 --> 00:38:52,320
um so for

1022
00:38:50,079 --> 00:38:54,480
about 20 of packages we managed to

1023
00:38:52,320 --> 00:38:57,280
successfully import a new release

1024
00:38:54,480 --> 00:38:58,960
and that's about 2200 packages which i'm

1025
00:38:57,280 --> 00:39:00,880
pretty happy with

1026
00:38:58,960 --> 00:39:03,200
um there's also a whole bunch of

1027
00:39:00,880 --> 00:39:06,560
packages that were already up to date

1028
00:39:03,200 --> 00:39:09,760
many more than i expected so about 35

1029
00:39:06,560 --> 00:39:11,520
was already at the latest release

1030
00:39:09,760 --> 00:39:13,200
and then if you look at the failures for

1031
00:39:11,520 --> 00:39:15,680
the other packages

1032
00:39:13,200 --> 00:39:17,520
there are a variety of reasons so some

1033
00:39:15,680 --> 00:39:19,599
of them are just random build failures

1034
00:39:17,520 --> 00:39:21,520
which is to be expected

1035
00:39:19,599 --> 00:39:23,680
some are related to

1036
00:39:21,520 --> 00:39:26,320
us not being able to find the

1037
00:39:23,680 --> 00:39:27,920
new upstream release

1038
00:39:26,320 --> 00:39:29,440
because we don't we don't know where

1039
00:39:27,920 --> 00:39:30,400
it's hosted

1040
00:39:29,440 --> 00:39:32,320
um

1041
00:39:30,400 --> 00:39:34,480
some are due to just

1042
00:39:32,320 --> 00:39:38,000
uh errors importing tarballs

1043
00:39:34,480 --> 00:39:39,440
um and in a lot of cases the debian

1044
00:39:38,000 --> 00:39:44,079
specific patches

1045
00:39:39,440 --> 00:39:46,960
no longer apply when we update the

1046
00:39:44,079 --> 00:39:46,960
upstream release

1047
00:39:50,160 --> 00:39:54,480
and then similarly if we look at

1048
00:39:52,160 --> 00:39:57,040
importing git snapshots

1049
00:39:54,480 --> 00:39:59,680
you can see that um

1050
00:39:57,040 --> 00:40:02,240
we um

1051
00:39:59,680 --> 00:40:03,119
are successful uh about 20 of the time

1052
00:40:02,240 --> 00:40:05,839
as well

1053
00:40:03,119 --> 00:40:07,520
um but this accounts for about 5500

1054
00:40:05,839 --> 00:40:09,040
packages

1055
00:40:07,520 --> 00:40:11,359
um

1056
00:40:09,040 --> 00:40:13,359
and um

1057
00:40:11,359 --> 00:40:18,160
in a lot of cases we don't actually know

1058
00:40:13,359 --> 00:40:18,160
where the uh upstream repository lives

1059
00:40:18,560 --> 00:40:21,440
and then some of the other errors are

1060
00:40:20,000 --> 00:40:24,800
very similar to

1061
00:40:21,440 --> 00:40:26,000
those when we were importing um releases

1062
00:40:24,800 --> 00:40:27,839
so

1063
00:40:26,000 --> 00:40:31,440
merge conflicts

1064
00:40:27,839 --> 00:40:31,440
or patches that no longer apply

1065
00:40:32,400 --> 00:40:37,160
or tireballs that don't import

1066
00:40:38,480 --> 00:40:42,720
um and the

1067
00:40:39,920 --> 00:40:45,040
packages that get built as part of

1068
00:40:42,720 --> 00:40:47,520
this process are actually available

1069
00:40:45,040 --> 00:40:48,400
um so if you'd like a debian release

1070
00:40:47,520 --> 00:40:51,280
with

1071
00:40:48,400 --> 00:40:53,839
very experimental packages with

1072
00:40:51,280 --> 00:40:55,680
newly imported upstreams

1073
00:40:53,839 --> 00:40:57,599
you can go to that url and find the

1074
00:40:55,680 --> 00:40:58,640
instructions on

1075
00:40:57,599 --> 00:41:02,280
how to

1076
00:40:58,640 --> 00:41:02,280
add those app repositories

1077
00:41:06,319 --> 00:41:12,000
um and then yeah what's what's next

1078
00:41:09,920 --> 00:41:13,760
um so there are a bunch of improvements

1079
00:41:12,000 --> 00:41:16,079
that we still need to make

1080
00:41:13,760 --> 00:41:18,720
um in order to

1081
00:41:16,079 --> 00:41:22,160
get more of the um

1082
00:41:18,720 --> 00:41:22,160
upstream imports to succeed

1083
00:41:24,560 --> 00:41:29,680
um and then

1084
00:41:26,480 --> 00:41:31,839
in the future we'd also like to look at

1085
00:41:29,680 --> 00:41:33,920
automating backboards so

1086
00:41:31,839 --> 00:41:35,680
in particular automating the cherry

1087
00:41:33,920 --> 00:41:36,800
picking of security fixes to stable

1088
00:41:35,680 --> 00:41:39,520
packages

1089
00:41:36,800 --> 00:41:41,280
as well as automating back porting of

1090
00:41:39,520 --> 00:41:43,920
unstable packages to

1091
00:41:41,280 --> 00:41:43,920
a stable

1092
00:41:44,720 --> 00:41:48,960
and there's a whole bunch of work that

1093
00:41:46,560 --> 00:41:50,319
needs to happen for

1094
00:41:48,960 --> 00:41:52,319
[Music]

1095
00:41:50,319 --> 00:41:54,960
debian patches to be updated

1096
00:41:52,319 --> 00:41:54,960
appropriately

1097
00:41:57,359 --> 00:42:00,560
there's some harder problems that we

1098
00:41:58,720 --> 00:42:03,040
haven't really looked at such as

1099
00:42:00,560 --> 00:42:06,560
transitions so packages that need to be

1100
00:42:03,040 --> 00:42:06,560
updated in lockstep for example

1101
00:42:08,160 --> 00:42:13,280
and and one of the things that

1102
00:42:10,880 --> 00:42:14,880
i would like to look at is

1103
00:42:13,280 --> 00:42:17,440
how we can apply some of these tools

1104
00:42:14,880 --> 00:42:20,800
outside of debian so there are some

1105
00:42:17,440 --> 00:42:23,680
existing projects that do things like

1106
00:42:20,800 --> 00:42:25,680
modify repositories to for example

1107
00:42:23,680 --> 00:42:27,700
reformat things

1108
00:42:25,680 --> 00:42:29,280
but that generally happens on a

1109
00:42:27,700 --> 00:42:30,800
[Music]

1110
00:42:29,280 --> 00:42:34,560
uh

1111
00:42:30,800 --> 00:42:36,800
as part of um existing changes uh rather

1112
00:42:34,560 --> 00:42:37,839
than proactively

1113
00:42:36,800 --> 00:42:40,400
um

1114
00:42:37,839 --> 00:42:42,400
github has a uh bots called the

1115
00:42:40,400 --> 00:42:45,680
pandabots that proactively submits

1116
00:42:42,400 --> 00:42:47,839
changes to um repositories

1117
00:42:45,680 --> 00:42:50,400
but it's proprietary and it's specific

1118
00:42:47,839 --> 00:42:50,400
to github

1119
00:42:50,960 --> 00:42:54,800
and there are some unique challenges if

1120
00:42:52,480 --> 00:42:56,960
we start doing this outside of debian

1121
00:42:54,800 --> 00:42:59,359
such as who would we which project will

1122
00:42:56,960 --> 00:43:00,960
we do it for um

1123
00:42:59,359 --> 00:43:03,839
what are the standards that we should

1124
00:43:00,960 --> 00:43:04,800
try and um aim for

1125
00:43:03,839 --> 00:43:05,680
um

1126
00:43:04,800 --> 00:43:07,599
and

1127
00:43:05,680 --> 00:43:09,280
how actually do we build a random

1128
00:43:07,599 --> 00:43:12,359
upstream project that we find somewhere

1129
00:43:09,280 --> 00:43:12,359
on getup

1130
00:43:12,560 --> 00:43:16,480
all right um that's that's all i wanted

1131
00:43:14,640 --> 00:43:18,960
to talk about today uh thank you very

1132
00:43:16,480 --> 00:43:22,240
much um for listening

1133
00:43:18,960 --> 00:43:24,560
um and also uh thanks for

1134
00:43:22,240 --> 00:43:27,200
thanks to um all of the people in debian

1135
00:43:24,560 --> 00:43:29,280
who've sort of worked on

1136
00:43:27,200 --> 00:43:30,640
the infrastructure that made this sort

1137
00:43:29,280 --> 00:43:31,599
of work possible

1138
00:43:30,640 --> 00:43:34,800
um

1139
00:43:31,599 --> 00:43:36,480
so we just bare bare turbos um none of

1140
00:43:34,800 --> 00:43:37,839
this would have been possible

1141
00:43:36,480 --> 00:43:38,720
um

1142
00:43:37,839 --> 00:43:42,079
from

1143
00:43:38,720 --> 00:43:45,680
metadata and and standardization

1144
00:43:42,079 --> 00:43:48,880
for the repository fields to

1145
00:43:45,680 --> 00:43:52,160
the uh ultimate debian database

1146
00:43:48,880 --> 00:43:53,839
lincion the ci systems as well as

1147
00:43:52,160 --> 00:43:56,560
jenkins which runs all of the workers

1148
00:43:53,839 --> 00:43:56,560
for the gender

1149
00:43:59,760 --> 00:44:02,319
and this is where you can find more

1150
00:44:01,119 --> 00:44:05,359
information

1151
00:44:02,319 --> 00:44:05,359
about the projects

1152
00:44:06,720 --> 00:44:09,839
thank you very much

1153
00:44:26,800 --> 00:44:31,920
fell for it again my apologies so yelmir

1154
00:44:29,280 --> 00:44:33,440
thank you so much for your talk um and

1155
00:44:31,920 --> 00:44:34,720
teaching us more about debian which i

1156
00:44:33,440 --> 00:44:37,119
didn't know a lot about and let alone

1157
00:44:34,720 --> 00:44:39,359
the janitor heard of merging i'm really

1158
00:44:37,119 --> 00:44:41,200
new to this world so thank you so much

1159
00:44:39,359 --> 00:44:43,119
uh we've really only got a few seconds

1160
00:44:41,200 --> 00:44:45,040
left in this room um the conference is

1161
00:44:43,119 --> 00:44:47,359
not over but this stream is over in this

1162
00:44:45,040 --> 00:44:48,960
room so if every quality please now move

1163
00:44:47,359 --> 00:44:50,880
back to the main room which is the ko

1164
00:44:48,960 --> 00:44:52,720
room and we'll have the conference close

1165
00:44:50,880 --> 00:44:55,280
out she'll be wrapping up here in a few

1166
00:44:52,720 --> 00:44:57,200
seconds and we'll over to the k a room

1167
00:44:55,280 --> 00:45:00,079
back in the main portal thanks everybody

1168
00:44:57,200 --> 00:45:01,520
for joining lca have a fantastic rest of

1169
00:45:00,079 --> 00:45:03,599
your sunday

1170
00:45:01,520 --> 00:45:05,440
coming up in london and for those of us

1171
00:45:03,599 --> 00:45:07,119
who are ending our sunday may your week

1172
00:45:05,440 --> 00:45:08,720
go well thanks for time and putting up

1173
00:45:07,119 --> 00:45:12,520
with my oopses of being on mute when i

1174
00:45:08,720 --> 00:45:12,520
shouldn't have been bye everyone