1
00:00:06,320 --> 00:00:11,499
[Music]

2
00:00:15,759 --> 00:00:20,480
welcome back everyone and to our uh

3
00:00:18,080 --> 00:00:22,000
second talk of the day at the lca

4
00:00:20,480 --> 00:00:24,640
colonel miniconf

5
00:00:22,000 --> 00:00:26,960
uh next up we've got paul mckinney

6
00:00:24,640 --> 00:00:29,439
who has been a programmer for a very

7
00:00:26,960 --> 00:00:31,279
long time and has been working on

8
00:00:29,439 --> 00:00:32,640
parallel hardware for approximately my

9
00:00:31,279 --> 00:00:35,600
entire life

10
00:00:32,640 --> 00:00:36,719
um he's a software engineer at facebook

11
00:00:35,600 --> 00:00:39,520
where he

12
00:00:36,719 --> 00:00:41,280
works as the maintainer of the rcu or

13
00:00:39,520 --> 00:00:43,120
read copy update

14
00:00:41,280 --> 00:00:45,200
implementation uh

15
00:00:43,120 --> 00:00:46,879
synchronization primitive um within the

16
00:00:45,200 --> 00:00:49,360
linux kernel

17
00:00:46,879 --> 00:00:52,320
and today he'll be talking to us uh

18
00:00:49,360 --> 00:00:55,280
about the rcu torture test suite

19
00:00:52,320 --> 00:00:57,120
um paul has asked uh if uh will be he'll

20
00:00:55,280 --> 00:00:58,640
be taking questions during the talk so

21
00:00:57,120 --> 00:01:02,160
if you want to um get your questions in

22
00:00:58,640 --> 00:01:04,640
via the questions tab in venulis

23
00:01:02,160 --> 00:01:06,400
as is going along we'll um we'll do our

24
00:01:04,640 --> 00:01:07,760
best to

25
00:01:06,400 --> 00:01:09,439
get them to paul

26
00:01:07,760 --> 00:01:11,600
during the course of the talk and

27
00:01:09,439 --> 00:01:12,400
there'll be a q a session at the end as

28
00:01:11,600 --> 00:01:14,799
well

29
00:01:12,400 --> 00:01:17,119
please welcome paul

30
00:01:14,799 --> 00:01:19,119
thank you andrew

31
00:01:17,119 --> 00:01:20,560
so why would anyone want to torture rcu

32
00:01:19,119 --> 00:01:22,400
i suppose is a

33
00:01:20,560 --> 00:01:23,920
starting question and let's face it

34
00:01:22,400 --> 00:01:25,920
synchronization primitives can be

35
00:01:23,920 --> 00:01:27,759
somewhat obnoxious to use rsu is a

36
00:01:25,920 --> 00:01:29,360
synchronization primitive when you're

37
00:01:27,759 --> 00:01:31,520
using them sometimes it feels like

38
00:01:29,360 --> 00:01:32,640
they're torturing you so why not get

39
00:01:31,520 --> 00:01:35,200
back

40
00:01:32,640 --> 00:01:37,360
but before we start torturing rcu i

41
00:01:35,200 --> 00:01:39,920
would like to call for a round of

42
00:01:37,360 --> 00:01:41,360
appreciation for these projects and the

43
00:01:39,920 --> 00:01:43,119
people behind them

44
00:01:41,360 --> 00:01:45,520
these guys don't always get the respect

45
00:01:43,119 --> 00:01:47,600
and appreciation they deserve but uh

46
00:01:45,520 --> 00:01:50,320
their efforts are really important

47
00:01:47,600 --> 00:01:52,640
to making all our software work as

48
00:01:50,320 --> 00:01:55,840
robustly reliably as it does so

49
00:01:52,640 --> 00:01:57,439
big thanks to them okay so some of you

50
00:01:55,840 --> 00:01:59,280
are probably thinking hey paul shut up

51
00:01:57,439 --> 00:02:01,439
and let me start torturing

52
00:01:59,280 --> 00:02:04,320
uh before we get to that point though uh

53
00:02:01,439 --> 00:02:06,880
the url at the bottom is uh

54
00:02:04,320 --> 00:02:08,800
an overview of a bunch of blog posts

55
00:02:06,880 --> 00:02:10,319
covering this material and there'll be

56
00:02:08,800 --> 00:02:11,680
urls at the bottom some of the slides

57
00:02:10,319 --> 00:02:13,840
that will have more information on the

58
00:02:11,680 --> 00:02:15,440
topic in that slide

59
00:02:13,840 --> 00:02:17,200
for the people who want me to shut up so

60
00:02:15,440 --> 00:02:19,200
they start torturing i'm sorry but i'm

61
00:02:17,200 --> 00:02:20,640
not going to shut up immediately maybe

62
00:02:19,200 --> 00:02:22,239
never

63
00:02:20,640 --> 00:02:24,720
but if you want to start torturing this

64
00:02:22,239 --> 00:02:27,680
slide is for you all right if you have a

65
00:02:24,720 --> 00:02:29,599
system that can run qmu and kvm and any

66
00:02:27,680 --> 00:02:31,760
modern x86 system will do just fine and

67
00:02:29,599 --> 00:02:34,160
there's a bunch of other ones as well

68
00:02:31,760 --> 00:02:36,239
uh you set your path up as shown there

69
00:02:34,160 --> 00:02:39,360
and if you have a lot of time all you

70
00:02:36,239 --> 00:02:40,879
have to do is just type kvm.sh enter

71
00:02:39,360 --> 00:02:42,879
what that's going to do

72
00:02:40,879 --> 00:02:44,640
is it's going to torture rsu with 19

73
00:02:42,879 --> 00:02:46,480
different scenarios

74
00:02:44,640 --> 00:02:47,840
how that's going to work is it's going

75
00:02:46,480 --> 00:02:51,040
to build a kernel for one of the

76
00:02:47,840 --> 00:02:52,959
scenarios it'll then do a 30 minute run

77
00:02:51,040 --> 00:02:55,440
by creating a guest os and then running

78
00:02:52,959 --> 00:02:57,760
rcu torture within that guest os

79
00:02:55,440 --> 00:03:00,000
when he gets done with that it'll build

80
00:02:57,760 --> 00:03:01,599
the next kernel and so on through all 19

81
00:03:00,000 --> 00:03:03,280
scenarios

82
00:03:01,599 --> 00:03:05,120
as a result this is going to take like

83
00:03:03,280 --> 00:03:08,080
10 or maybe even 11 hours depending on

84
00:03:05,120 --> 00:03:09,920
how quickly your system builds kernels

85
00:03:08,080 --> 00:03:11,680
if you have a system with a lot of cpus

86
00:03:09,920 --> 00:03:13,280
this is a really wasteful process

87
00:03:11,680 --> 00:03:16,640
because a lot of those scenarios use

88
00:03:13,280 --> 00:03:19,280
only one or two or four cpus

89
00:03:16,640 --> 00:03:24,000
so if you have 64 cpus the next line is

90
00:03:19,280 --> 00:03:25,840
more appropriate kvm.s h cpus 64.

91
00:03:24,000 --> 00:03:27,120
and what that's going to do

92
00:03:25,840 --> 00:03:28,799
is going to run the scenarios in two

93
00:03:27,120 --> 00:03:30,560
batches so it's going to build a whole

94
00:03:28,799 --> 00:03:32,239
pile of kernels one at a time build them

95
00:03:30,560 --> 00:03:34,400
and then when it's got them all built

96
00:03:32,239 --> 00:03:36,239
it's going to create a guest awaits for

97
00:03:34,400 --> 00:03:38,799
each of the kernels that just built and

98
00:03:36,239 --> 00:03:41,120
run them concurrently for 30 minutes

99
00:03:38,799 --> 00:03:43,040
when it gets done with that it'll build

100
00:03:41,120 --> 00:03:46,640
the rest of the kernels one at a time

101
00:03:43,040 --> 00:03:49,280
and then it'll run all of the

102
00:03:46,640 --> 00:03:51,760
kernels produced as guest os's again

103
00:03:49,280 --> 00:03:53,360
then it'll summarize the results

104
00:03:51,760 --> 00:03:54,799
so that allows you to get done a lot

105
00:03:53,360 --> 00:03:55,599
faster that's going to be oh i don't

106
00:03:54,799 --> 00:03:57,760
know

107
00:03:55,599 --> 00:03:59,439
maybe one two hours depending on how

108
00:03:57,760 --> 00:04:01,519
fast again your current

109
00:03:59,439 --> 00:04:03,920
kernels are built on your system

110
00:04:01,519 --> 00:04:05,920
but if you're running different systems

111
00:04:03,920 --> 00:04:07,760
keeping track of which cpu has how many

112
00:04:05,920 --> 00:04:10,000
put how much system has so many cpus for

113
00:04:07,760 --> 00:04:12,239
that matter can be a little obnoxious so

114
00:04:10,000 --> 00:04:13,599
you can just say kvm

115
00:04:12,239 --> 00:04:15,120
kvm.sh

116
00:04:13,599 --> 00:04:16,400
all cpus

117
00:04:15,120 --> 00:04:18,479
and that'll just look and see how many

118
00:04:16,400 --> 00:04:21,199
cpu is there and use all of them which

119
00:04:18,479 --> 00:04:22,639
makes everybody happy well aside from

120
00:04:21,199 --> 00:04:24,400
anybody who might be trying to use that

121
00:04:22,639 --> 00:04:25,680
system for something else but yeah you

122
00:04:24,400 --> 00:04:27,600
can't have everything

123
00:04:25,680 --> 00:04:29,600
and you can also tell it how long to run

124
00:04:27,600 --> 00:04:32,479
so dash dash duration

125
00:04:29,600 --> 00:04:34,320
1d says one day 24 hours

126
00:04:32,479 --> 00:04:36,800
what this will do then

127
00:04:34,320 --> 00:04:39,120
is it if you have 64 cpus it'll act just

128
00:04:36,800 --> 00:04:41,280
as it did before but it'll take

129
00:04:39,120 --> 00:04:43,199
two days kind of a weekend run to run

130
00:04:41,280 --> 00:04:45,840
through all of the scenarios if you have

131
00:04:43,199 --> 00:04:47,440
128 cpus or more you can run all the

132
00:04:45,840 --> 00:04:49,360
scenarios in one batch build all the

133
00:04:47,440 --> 00:04:50,320
kernels run everything in parallel be

134
00:04:49,360 --> 00:04:51,520
happy

135
00:04:50,320 --> 00:04:53,520
in this case we're setting the duration

136
00:04:51,520 --> 00:04:56,639
to 12 hours

137
00:04:53,520 --> 00:04:57,840
one important safety tip

138
00:04:56,639 --> 00:04:59,440
you've got your kernel source tree

139
00:04:57,840 --> 00:05:02,080
you're doing this in

140
00:04:59,440 --> 00:05:03,840
if it has things like tags files or c

141
00:05:02,080 --> 00:05:05,600
scopes databases you'll want this trust

142
00:05:03,840 --> 00:05:07,520
make thing that's going to do two things

143
00:05:05,600 --> 00:05:10,800
for you well one

144
00:05:07,520 --> 00:05:12,639
it's not going to do a make mr proper

145
00:05:10,800 --> 00:05:15,440
make this clean at the beginning of the

146
00:05:12,639 --> 00:05:16,800
bill each build

147
00:05:15,440 --> 00:05:18,479
and that means that you'll be able to

148
00:05:16,800 --> 00:05:19,919
reuse build products of course there's

149
00:05:18,479 --> 00:05:21,600
some risk in there you may have a

150
00:05:19,919 --> 00:05:22,960
problem that's caused by an earlier

151
00:05:21,600 --> 00:05:25,280
build

152
00:05:22,960 --> 00:05:26,880
but the nice thing about trust make is

153
00:05:25,280 --> 00:05:30,240
that it's not going to destroy all of

154
00:05:26,880 --> 00:05:32,560
your tags files or c scopes databases

155
00:05:30,240 --> 00:05:35,039
if you just want to torture one aspect

156
00:05:32,560 --> 00:05:36,960
of rcu for example i modified srcu some

157
00:05:35,039 --> 00:05:38,320
last week so i'd want to torture srcu

158
00:05:36,960 --> 00:05:39,199
and this is one way to do it on the next

159
00:05:38,320 --> 00:05:43,120
line

160
00:05:39,199 --> 00:05:45,440
given 28 cpus if i say configs and then

161
00:05:43,120 --> 00:05:47,520
list all of the srcu scenarios there are

162
00:05:45,440 --> 00:05:49,199
four of them there i can also put two

163
00:05:47,520 --> 00:05:50,800
star in front of it to say run each of

164
00:05:49,199 --> 00:05:52,000
them twice

165
00:05:50,800 --> 00:05:53,840
so what's going to happen with this

166
00:05:52,000 --> 00:05:55,280
thing is it's going to do four kernel

167
00:05:53,840 --> 00:05:57,199
builds

168
00:05:55,280 --> 00:05:59,600
and then it's gonna

169
00:05:57,199 --> 00:06:01,039
run eight

170
00:05:59,600 --> 00:06:03,199
guest os's

171
00:06:01,039 --> 00:06:05,199
two for each kernel and do that by

172
00:06:03,199 --> 00:06:07,120
default we haven't said so do it for 30

173
00:06:05,199 --> 00:06:09,280
minutes

174
00:06:07,120 --> 00:06:11,039
if you have a boot bug

175
00:06:09,280 --> 00:06:12,400
that happens rarely

176
00:06:11,039 --> 00:06:14,400
it's going to be really annoying waiting

177
00:06:12,400 --> 00:06:17,360
for the kernel build each time

178
00:06:14,400 --> 00:06:19,919
and so kvm again allows you to reuse the

179
00:06:17,360 --> 00:06:22,080
build products from a previous run so if

180
00:06:19,919 --> 00:06:23,600
you just run kvm.sh like the previous

181
00:06:22,080 --> 00:06:25,120
five lines

182
00:06:23,600 --> 00:06:27,199
in this output there'll be a results

183
00:06:25,120 --> 00:06:28,720
directory and a big path name you take

184
00:06:27,199 --> 00:06:31,919
that path name and drop it in as a

185
00:06:28,720 --> 00:06:33,520
second argument to kvm.again.sh

186
00:06:31,919 --> 00:06:36,000
and in this case we can set the duration

187
00:06:33,520 --> 00:06:38,319
of the runs to 45 seconds 45 s 45

188
00:06:36,000 --> 00:06:40,400
seconds and that allows us to do a huge

189
00:06:38,319 --> 00:06:41,840
number of boots really quickly

190
00:06:40,400 --> 00:06:44,080
and hopefully be able to reproduce the

191
00:06:41,840 --> 00:06:46,720
bug we are chasing

192
00:06:44,080 --> 00:06:48,880
if you have a large number of systems

193
00:06:46,720 --> 00:06:50,639
then kvm remote is for you you can give

194
00:06:48,880 --> 00:06:52,560
it a list of systems the names as you

195
00:06:50,639 --> 00:06:54,479
used ssh into them

196
00:06:52,560 --> 00:06:56,880
and uh you'll usually need to give it

197
00:06:54,479 --> 00:06:59,440
the configs the cpu's 80 each system has

198
00:06:56,880 --> 00:07:02,479
at least 80 cpus configs you want to

199
00:06:59,440 --> 00:07:04,960
usually run a lot of them cf list is a

200
00:07:02,479 --> 00:07:06,880
keyword that says all of the 19 normal

201
00:07:04,960 --> 00:07:09,360
scenarios so we're going to run 19 times

202
00:07:06,880 --> 00:07:11,039
15 scenarios plus 310 so it's going to

203
00:07:09,360 --> 00:07:13,039
do a lot of torturing in a short period

204
00:07:11,039 --> 00:07:15,840
of time that's something i normally do

205
00:07:13,039 --> 00:07:19,280
something like that for my testing

206
00:07:15,840 --> 00:07:20,639
okay so that's how we start testing

207
00:07:19,280 --> 00:07:22,240
and

208
00:07:20,639 --> 00:07:25,440
so one question is what success look

209
00:07:22,240 --> 00:07:27,360
like and if we just run the stock

210
00:07:25,440 --> 00:07:31,280
thing without specifying configurations

211
00:07:27,360 --> 00:07:33,440
we get those 19 scenarios

212
00:07:31,280 --> 00:07:34,639
and the important thing is that all of

213
00:07:33,440 --> 00:07:36,240
these scripts all three of them will

214
00:07:34,639 --> 00:07:38,479
give you an x code of zero if there are

215
00:07:36,240 --> 00:07:39,840
no problems detected by the test and

216
00:07:38,479 --> 00:07:41,199
this can be really useful you can take

217
00:07:39,840 --> 00:07:43,199
any of those scripts and hand it to get

218
00:07:41,199 --> 00:07:46,000
bisect run for example

219
00:07:43,199 --> 00:07:47,919
and yes i really have done get bisect

220
00:07:46,000 --> 00:07:49,840
run with kvm remote

221
00:07:47,919 --> 00:07:54,560
it helps you

222
00:07:49,840 --> 00:07:56,240
reproduce and test things involving uh

223
00:07:54,560 --> 00:07:58,400
difficult to reduce bugs much more

224
00:07:56,240 --> 00:08:00,800
readily

225
00:07:58,400 --> 00:08:02,000
okay uh there are other kvm.sage

226
00:08:00,800 --> 00:08:04,479
parameters i'm not going to go through

227
00:08:02,000 --> 00:08:06,479
these in violent detail k config allows

228
00:08:04,479 --> 00:08:08,400
you to vacation arguments you can

229
00:08:06,479 --> 00:08:10,160
specify kernel boot arguments as well

230
00:08:08,400 --> 00:08:11,919
you can tell it to run either ka san or

231
00:08:10,160 --> 00:08:12,720
kcsan you'll get to run both at the same

232
00:08:11,919 --> 00:08:14,720
time

233
00:08:12,720 --> 00:08:16,400
both of these guys generate big binaries

234
00:08:14,720 --> 00:08:18,479
big vm linux files so be a little

235
00:08:16,400 --> 00:08:20,639
careful of your disk space

236
00:08:18,479 --> 00:08:22,879
and kcsan requires recent compilers

237
00:08:20,639 --> 00:08:24,160
clang 11 works for me but you'll need to

238
00:08:22,879 --> 00:08:25,599
check for you

239
00:08:24,160 --> 00:08:26,720
you can torture different things if you

240
00:08:25,599 --> 00:08:28,000
don't want if you're not upset enough

241
00:08:26,720 --> 00:08:29,280
that are seating torture locks for

242
00:08:28,000 --> 00:08:31,360
example

243
00:08:29,280 --> 00:08:32,719
uh and uh if you're

244
00:08:31,360 --> 00:08:34,800
running a special purpose thing if you

245
00:08:32,719 --> 00:08:37,039
have a lot of cpus you can do dash dash

246
00:08:34,800 --> 00:08:39,039
dry runs scenario and uh

247
00:08:37,039 --> 00:08:40,959
tell us scenarios and what that'll do is

248
00:08:39,039 --> 00:08:42,719
it'll won't actually run the tests but

249
00:08:40,959 --> 00:08:43,680
it'll tell you what the batches are

250
00:08:42,719 --> 00:08:45,279
going to be in other words what

251
00:08:43,680 --> 00:08:46,880
scenarios can run in what order and that

252
00:08:45,279 --> 00:08:49,600
can allow you to

253
00:08:46,880 --> 00:08:50,560
choose the dash configs or the cpus

254
00:08:49,600 --> 00:08:51,760
argument

255
00:08:50,560 --> 00:08:53,680
appropriate for whatever you're trying

256
00:08:51,760 --> 00:08:55,760
to test

257
00:08:53,680 --> 00:08:57,040
all right uh maybe you can't decide what

258
00:08:55,760 --> 00:08:59,519
you want or torture well there's the

259
00:08:57,040 --> 00:09:00,959
torture.sh script and it tortures a

260
00:08:59,519 --> 00:09:02,640
little of everything i use it as kind of

261
00:09:00,959 --> 00:09:04,480
an acceptance test before i do a pull

262
00:09:02,640 --> 00:09:06,880
request occasionally

263
00:09:04,480 --> 00:09:09,200
i also use it for dash next testing

264
00:09:06,880 --> 00:09:11,519
now this does take some time like about

265
00:09:09,200 --> 00:09:13,519
11 hours on a heavy duty laptop it does

266
00:09:11,519 --> 00:09:15,440
not run kc sand by default if you do

267
00:09:13,519 --> 00:09:16,959
that you're looking at 14. it's got a

268
00:09:15,440 --> 00:09:19,200
lot of arguments and controls torturing

269
00:09:16,959 --> 00:09:21,680
but we'll leave those off

270
00:09:19,200 --> 00:09:23,440
due to time

271
00:09:21,680 --> 00:09:26,480
you maybe you want to run a debugger

272
00:09:23,440 --> 00:09:28,640
well dash gb does that for you you only

273
00:09:26,480 --> 00:09:30,160
get to run one scenario at a time i'm

274
00:09:28,640 --> 00:09:32,480
sorry i didn't want to mess with having

275
00:09:30,160 --> 00:09:34,000
multiple parallel debugging sessions and

276
00:09:32,480 --> 00:09:35,839
well if you want to do that send me a

277
00:09:34,000 --> 00:09:37,600
patch okay

278
00:09:35,839 --> 00:09:38,880
one

279
00:09:37,600 --> 00:09:40,720
safety tip

280
00:09:38,880 --> 00:09:42,320
you don't get to use the gdb brake

281
00:09:40,720 --> 00:09:44,000
command

282
00:09:42,320 --> 00:09:46,880
the reason is the linux kernel really

283
00:09:44,000 --> 00:09:49,040
doesn't like gdb rewriting as binary

284
00:09:46,880 --> 00:09:51,040
so use h-break which uses the hardware

285
00:09:49,040 --> 00:09:53,279
breakpoint registers use that instead

286
00:09:51,040 --> 00:09:54,720
and life will be a lot better

287
00:09:53,279 --> 00:09:56,480
you also have to be fast once you hit a

288
00:09:54,720 --> 00:09:57,680
breakpoint otherwise all sorts of things

289
00:09:56,480 --> 00:09:59,440
the linux kernel complain about the

290
00:09:57,680 --> 00:10:01,200
delay but

291
00:09:59,440 --> 00:10:03,519
either be fast or just hit the

292
00:10:01,200 --> 00:10:05,200
breakpoint look around

293
00:10:03,519 --> 00:10:06,880
so you found a bug what do you do now

294
00:10:05,200 --> 00:10:09,120
well you know if you fixed it sent me a

295
00:10:06,880 --> 00:10:10,720
patch that'd be great

296
00:10:09,120 --> 00:10:12,640
uh of course fixing it can be a little

297
00:10:10,720 --> 00:10:15,839
bit of an issue because with rcu

298
00:10:12,640 --> 00:10:18,720
heisenbugs are the common case

299
00:10:15,839 --> 00:10:20,480
and you'd like to dehyze and bug them

300
00:10:18,720 --> 00:10:21,760
and that can be hard but here's a few

301
00:10:20,480 --> 00:10:23,120
tricks

302
00:10:21,760 --> 00:10:25,519
the overall

303
00:10:23,120 --> 00:10:28,000
approach is to make risky operations

304
00:10:25,519 --> 00:10:29,680
happen more frequently in my testing cpu

305
00:10:28,000 --> 00:10:31,680
hot plug is often

306
00:10:29,680 --> 00:10:33,200
what's causing the problem or related to

307
00:10:31,680 --> 00:10:35,360
the problem and that's because it

308
00:10:33,200 --> 00:10:37,200
doesn't really get tested that much

309
00:10:35,360 --> 00:10:39,760
so i end up doing some of the testing

310
00:10:37,200 --> 00:10:41,440
early on for it and you can give a boot

311
00:10:39,760 --> 00:10:43,360
parameter to tell

312
00:10:41,440 --> 00:10:45,680
the scripts to do the hot plug more

313
00:10:43,360 --> 00:10:47,760
quickly by default you'll get a cpu hot

314
00:10:45,680 --> 00:10:50,160
plug operation every few seconds

315
00:10:47,760 --> 00:10:51,839
with this boot args in red there it's

316
00:10:50,160 --> 00:10:52,800
going to wait only 200 milliseconds

317
00:10:51,839 --> 00:10:54,480
between

318
00:10:52,800 --> 00:10:56,240
hot plug operations so you'll be getting

319
00:10:54,480 --> 00:10:57,920
the hot plug operations about an order

320
00:10:56,240 --> 00:10:59,519
of magnitude more quickly

321
00:10:57,920 --> 00:11:01,120
and thus putting a lot more stress on

322
00:10:59,519 --> 00:11:04,399
whatever might be causing the problem if

323
00:11:01,120 --> 00:11:04,399
it's related to cpu hot plug

324
00:11:04,720 --> 00:11:08,959
long-lived readers rcu torture does this

325
00:11:06,800 --> 00:11:10,640
automatically

326
00:11:08,959 --> 00:11:12,399
so you shouldn't have to worry about

327
00:11:10,640 --> 00:11:14,399
this but you can always modify the

328
00:11:12,399 --> 00:11:15,839
source code if you do

329
00:11:14,399 --> 00:11:17,279
sometimes there are bugs that happen

330
00:11:15,839 --> 00:11:19,760
when the system goes from really really

331
00:11:17,279 --> 00:11:21,760
busy down to idle or back up

332
00:11:19,760 --> 00:11:23,440
and there's a module parametered rcu

333
00:11:21,760 --> 00:11:25,839
torture called stutter that controls how

334
00:11:23,440 --> 00:11:27,760
quickly and often that happens

335
00:11:25,839 --> 00:11:29,760
some other bugs happen when you have

336
00:11:27,760 --> 00:11:31,920
huge numbers of callbacks and there's a

337
00:11:29,760 --> 00:11:34,000
fwd underbar progress

338
00:11:31,920 --> 00:11:35,920
module parameter that controls the

339
00:11:34,000 --> 00:11:37,600
intensity and how how frequently those

340
00:11:35,920 --> 00:11:39,120
happen

341
00:11:37,600 --> 00:11:41,839
another thing that can make things

342
00:11:39,120 --> 00:11:45,279
happen more reasonably more quickly is

343
00:11:41,839 --> 00:11:46,880
preemption and the neat thing is is that

344
00:11:45,279 --> 00:11:48,480
if you're running on a hypervisor which

345
00:11:46,880 --> 00:11:50,399
these scripts are

346
00:11:48,480 --> 00:11:51,760
you can preempt the guest os even if the

347
00:11:50,399 --> 00:11:53,040
guest os thinks it has interrupts

348
00:11:51,760 --> 00:11:54,800
disabled

349
00:11:53,040 --> 00:11:57,839
okay and there's a dash dash jitter

350
00:11:54,800 --> 00:12:01,279
argument to kvm.h to make that happen

351
00:11:57,839 --> 00:12:03,600
by default it just makes a jitter script

352
00:12:01,279 --> 00:12:05,040
for every cpu that it's using

353
00:12:03,600 --> 00:12:06,959
and each of these things goes and

354
00:12:05,040 --> 00:12:08,399
randomly grabs a cpu spins on it for a

355
00:12:06,959 --> 00:12:11,200
while lets go and waits for a little bit

356
00:12:08,399 --> 00:12:13,440
and then randomly grabs another cpu

357
00:12:11,200 --> 00:12:15,920
you can control how long it sleeps and

358
00:12:13,440 --> 00:12:17,279
how long it spins and you can also

359
00:12:15,920 --> 00:12:19,120
control the number you can make more or

360
00:12:17,279 --> 00:12:20,800
fewer of them

361
00:12:19,120 --> 00:12:22,480
so these can help accelerate there are

362
00:12:20,800 --> 00:12:24,560
some other things you can do as well but

363
00:12:22,480 --> 00:12:26,720
that'll do for the for right now

364
00:12:24,560 --> 00:12:28,800
another thing that it does we could try

365
00:12:26,720 --> 00:12:30,720
torturing rcu from user space you know

366
00:12:28,800 --> 00:12:32,160
we could have a user space thing a big

367
00:12:30,720 --> 00:12:34,320
test script that went and hammered

368
00:12:32,160 --> 00:12:36,320
various system calls and io ctls and

369
00:12:34,320 --> 00:12:38,720
whatever else to try to make the kernel

370
00:12:36,320 --> 00:12:40,320
do rcu work and try to force things to

371
00:12:38,720 --> 00:12:42,079
happen but

372
00:12:40,320 --> 00:12:43,440
there's a lot of overhead involved in

373
00:12:42,079 --> 00:12:44,959
all that and so you're not going to be

374
00:12:43,440 --> 00:12:47,440
putting a whole lot of stress on rcu

375
00:12:44,959 --> 00:12:50,079
when you do that so we don't do that

376
00:12:47,440 --> 00:12:52,399
what we do instead is these tests are

377
00:12:50,079 --> 00:12:55,519
kernel modules and that allows us to

378
00:12:52,399 --> 00:12:57,360
directly invoke the rc apis and thus

379
00:12:55,519 --> 00:12:59,839
allows us to put a lot more stress on

380
00:12:57,360 --> 00:12:59,839
our cu

381
00:12:59,920 --> 00:13:04,399
alright so you know i torture rsu a lot

382
00:13:02,560 --> 00:13:06,399
is it really worthwhile torturing it

383
00:13:04,399 --> 00:13:08,320
more a legitimate question to ask i

384
00:13:06,399 --> 00:13:10,720
suppose

385
00:13:08,320 --> 00:13:12,959
and uh unfortunately the answer is yeah

386
00:13:10,720 --> 00:13:13,839
there's a lot of reason to torture it

387
00:13:12,959 --> 00:13:15,600
more

388
00:13:13,839 --> 00:13:17,200
turns out rcu

389
00:13:15,600 --> 00:13:18,480
as of 5 15

390
00:13:17,200 --> 00:13:20,240
as in before

391
00:13:18,480 --> 00:13:22,079
just the beginning of this week contains

392
00:13:20,240 --> 00:13:26,160
a little over 18 000 lines of code and

393
00:13:22,079 --> 00:13:27,680
that's just wc-l lines

394
00:13:26,160 --> 00:13:29,279
there's a lot of statistics on bug

395
00:13:27,680 --> 00:13:31,040
density and usually get somewhere

396
00:13:29,279 --> 00:13:32,720
between one and three bugs per thousand

397
00:13:31,040 --> 00:13:36,639
lines of code if you work the math

398
00:13:32,720 --> 00:13:38,480
that's somewhere between 18 and 55 bugs

399
00:13:36,639 --> 00:13:40,079
i'd like to think i'm better than that

400
00:13:38,480 --> 00:13:41,120
but

401
00:13:40,079 --> 00:13:42,720
you know

402
00:13:41,120 --> 00:13:43,839
i know better i've

403
00:13:42,720 --> 00:13:45,440
been around for a while and i've seen

404
00:13:43,839 --> 00:13:47,920
what kind of code i generate

405
00:13:45,440 --> 00:13:50,000
plus the median age of a line of code in

406
00:13:47,920 --> 00:13:53,040
rc was less than four years

407
00:13:50,000 --> 00:13:54,720
okay in contrast rc was introduced in

408
00:13:53,040 --> 00:13:56,880
the kernel

409
00:13:54,720 --> 00:13:58,320
almost 20 years ago

410
00:13:56,880 --> 00:14:00,480
all right so

411
00:13:58,320 --> 00:14:02,000
there's a lot of new code in there and

412
00:14:00,480 --> 00:14:04,399
new code tends to be bigger than all the

413
00:14:02,000 --> 00:14:05,760
code so yeah there's more bugs in here

414
00:14:04,399 --> 00:14:07,519
and they need to be tortured so we can

415
00:14:05,760 --> 00:14:09,440
find them

416
00:14:07,519 --> 00:14:10,880
sad but true

417
00:14:09,440 --> 00:14:12,399
another thing working against this is

418
00:14:10,880 --> 00:14:13,920
the installed base and i'm going to

419
00:14:12,399 --> 00:14:15,600
illustrate this by kind of running

420
00:14:13,920 --> 00:14:18,160
quickly through my career

421
00:14:15,600 --> 00:14:19,920
my very first professional

422
00:14:18,160 --> 00:14:22,480
program

423
00:14:19,920 --> 00:14:25,279
was written in 1975

424
00:14:22,480 --> 00:14:27,120
pro bono for the national honor society

425
00:14:25,279 --> 00:14:28,639
it was i kid you not a computer dating

426
00:14:27,120 --> 00:14:30,399
program

427
00:14:28,639 --> 00:14:31,199
they used it as a fundraiser

428
00:14:30,399 --> 00:14:32,880
well

429
00:14:31,199 --> 00:14:34,959
if we had a million-year bug in that

430
00:14:32,880 --> 00:14:36,800
program a sequential year program

431
00:14:34,959 --> 00:14:38,320
millionaire bug but whatever

432
00:14:36,800 --> 00:14:41,199
that bug would happen once every million

433
00:14:38,320 --> 00:14:43,199
years so yeah you know murphy is

434
00:14:41,199 --> 00:14:45,680
actually kind of a nice guy sure

435
00:14:43,199 --> 00:14:47,839
everything can happen will

436
00:14:45,680 --> 00:14:49,760
eventually you know maybe in geologic

437
00:14:47,839 --> 00:14:51,680
time

438
00:14:49,760 --> 00:14:53,839
as i've moved through my career i've had

439
00:14:51,680 --> 00:14:55,839
the good fortune and privilege

440
00:14:53,839 --> 00:14:57,440
to have increasing numbers of people use

441
00:14:55,839 --> 00:14:59,680
my software

442
00:14:57,440 --> 00:15:01,600
in 2017 a gentleman from arm told me

443
00:14:59,680 --> 00:15:03,279
that there were at least 20 billion

444
00:15:01,600 --> 00:15:04,800
that's billion with a b

445
00:15:03,279 --> 00:15:07,519
you know 10 to the ninth power 20 times

446
00:15:04,800 --> 00:15:09,680
10 to the ninth power

447
00:15:07,519 --> 00:15:13,760
linux instance is running on arm at that

448
00:15:09,680 --> 00:15:16,399
time about four years ago

449
00:15:13,760 --> 00:15:17,760
and uh that's a lot of systems if we

450
00:15:16,399 --> 00:15:20,079
have a million year bug it's happening

451
00:15:17,760 --> 00:15:22,320
several times per hour across the

452
00:15:20,079 --> 00:15:23,440
installed base

453
00:15:22,320 --> 00:15:25,600
so

454
00:15:23,440 --> 00:15:26,720
you know infrequent bug

455
00:15:25,600 --> 00:15:28,639
i'm sorry it's still happening

456
00:15:26,720 --> 00:15:30,959
frequently

457
00:15:28,639 --> 00:15:32,240
and it could get a lot worse

458
00:15:30,959 --> 00:15:34,480
we got this thing called internet of

459
00:15:32,240 --> 00:15:36,639
things uh there was a fact an lca about

460
00:15:34,480 --> 00:15:38,880
a little while ago of internet of

461
00:15:36,639 --> 00:15:40,399
internet of things lca

462
00:15:38,880 --> 00:15:42,880
and we could easily end up with

463
00:15:40,399 --> 00:15:44,000
trillions even if linux doesn't get that

464
00:15:42,880 --> 00:15:45,279
much

465
00:15:44,000 --> 00:15:48,399
of a share

466
00:15:45,279 --> 00:15:48,399
of the iot devices

467
00:15:48,639 --> 00:15:52,639
and my concern

468
00:15:50,399 --> 00:15:55,120
is that over a period of

469
00:15:52,639 --> 00:15:57,040
you know 45 years

470
00:15:55,120 --> 00:15:59,920
murphy might have transitioned

471
00:15:57,040 --> 00:16:01,759
from a nice guy to a homicidal maniac

472
00:15:59,920 --> 00:16:03,600
and i don't know about you guys but i'd

473
00:16:01,759 --> 00:16:05,600
feel really really bad if my software

474
00:16:03,600 --> 00:16:08,560
happened to kill somebody so you know

475
00:16:05,600 --> 00:16:09,680
this is this is even worse so what can

476
00:16:08,560 --> 00:16:11,199
we do here

477
00:16:09,680 --> 00:16:12,720
i mean uh

478
00:16:11,199 --> 00:16:14,240
my current employer is quite generous

479
00:16:12,720 --> 00:16:16,399
with systems i really thank them for

480
00:16:14,240 --> 00:16:18,560
providing a good environment for me to

481
00:16:16,399 --> 00:16:20,639
test and develop rcu it's wonderful

482
00:16:18,560 --> 00:16:22,399
but uh they don't provide me with with

483
00:16:20,639 --> 00:16:23,680
20 billion of them let alone a trillion

484
00:16:22,399 --> 00:16:27,120
of them

485
00:16:23,680 --> 00:16:30,560
nor nor do i ever expect them to

486
00:16:27,120 --> 00:16:30,560
so what can we do about this

487
00:16:30,880 --> 00:16:33,920
well one

488
00:16:32,320 --> 00:16:36,240
one reason for hope is that there are

489
00:16:33,920 --> 00:16:38,000
living things like ourselves

490
00:16:36,240 --> 00:16:40,399
and this

491
00:16:38,000 --> 00:16:42,000
evolutionary process has generated us

492
00:16:40,399 --> 00:16:42,800
over a period of a fair amount of time

493
00:16:42,000 --> 00:16:44,320
and

494
00:16:42,800 --> 00:16:45,920
we are subject to all sorts of ills

495
00:16:44,320 --> 00:16:48,079
witness the fact that i'm presenting to

496
00:16:45,920 --> 00:16:50,079
you virtually rather than being in

497
00:16:48,079 --> 00:16:52,480
australia with you okay i'd rather be

498
00:16:50,079 --> 00:16:54,720
there but you know if if i can't this is

499
00:16:52,480 --> 00:16:56,160
at least second best so yes there are

500
00:16:54,720 --> 00:16:59,279
some problems with living things but

501
00:16:56,160 --> 00:17:01,040
still we seem to operate fairly well

502
00:16:59,279 --> 00:17:03,040
we certainly live longer than software

503
00:17:01,040 --> 00:17:06,480
does generally

504
00:17:03,040 --> 00:17:07,919
so maybe we can use natural selection

505
00:17:06,480 --> 00:17:09,280
and of course we're going to discuss

506
00:17:07,919 --> 00:17:11,360
natural selection we need a picture of

507
00:17:09,280 --> 00:17:13,120
the great man there charles jarwin who

508
00:17:11,360 --> 00:17:14,480
observed natural selection happening in

509
00:17:13,120 --> 00:17:16,640
living things

510
00:17:14,480 --> 00:17:18,720
what charles darwin didn't know

511
00:17:16,640 --> 00:17:21,120
was he was watching natural selection in

512
00:17:18,720 --> 00:17:22,319
software of a sort because we have this

513
00:17:21,120 --> 00:17:23,199
dna

514
00:17:22,319 --> 00:17:25,760
and

515
00:17:23,199 --> 00:17:28,720
the dna and codes proteins and things

516
00:17:25,760 --> 00:17:30,080
like that and it resembles a software of

517
00:17:28,720 --> 00:17:32,160
a sort

518
00:17:30,080 --> 00:17:33,919
well if we can do natural selection on

519
00:17:32,160 --> 00:17:36,559
living software we should be able to do

520
00:17:33,919 --> 00:17:38,320
that on non-living software as well

521
00:17:36,559 --> 00:17:39,919
and uh you know that shouldn't be too

522
00:17:38,320 --> 00:17:42,320
hard to do and so let's give the great

523
00:17:39,919 --> 00:17:44,240
man something to look at

524
00:17:42,320 --> 00:17:46,960
and here we've got a process for doing

525
00:17:44,240 --> 00:17:48,640
this now some of you may object to my

526
00:17:46,960 --> 00:17:50,080
bit about software being randomly

527
00:17:48,640 --> 00:17:50,880
generated

528
00:17:50,080 --> 00:17:52,799
and

529
00:17:50,880 --> 00:17:55,039
i'm sorry but

530
00:17:52,799 --> 00:17:57,039
if you want me to change my description

531
00:17:55,039 --> 00:17:58,000
of software in general in the world

532
00:17:57,039 --> 00:18:00,720
today

533
00:17:58,000 --> 00:18:03,120
change the software okay

534
00:18:00,720 --> 00:18:05,120
but that's okay it's randomly generated

535
00:18:03,120 --> 00:18:07,039
we can still do validation on it run it

536
00:18:05,120 --> 00:18:08,640
through tests formal verification where

537
00:18:07,039 --> 00:18:10,640
that's applicable whatever else we can

538
00:18:08,640 --> 00:18:12,000
do and this will act as a selection

539
00:18:10,640 --> 00:18:13,919
function

540
00:18:12,000 --> 00:18:15,600
so we'll get bugs out of that we'll fix

541
00:18:13,919 --> 00:18:17,440
those bugs hopefully injecting fewer

542
00:18:15,600 --> 00:18:19,679
bugs than we fix along the process and

543
00:18:17,440 --> 00:18:21,520
if we do that we go around this cycle

544
00:18:19,679 --> 00:18:23,600
around and around and eventually we have

545
00:18:21,520 --> 00:18:26,640
robot software dropping out the bottom

546
00:18:23,600 --> 00:18:26,640
wonderful stuff right

547
00:18:26,840 --> 00:18:32,559
well almost

548
00:18:30,080 --> 00:18:34,160
you see we're assuming here the natural

549
00:18:32,559 --> 00:18:36,400
selection applies to software with good

550
00:18:34,160 --> 00:18:38,720
reason after all living things have this

551
00:18:36,400 --> 00:18:42,160
dna based software right

552
00:18:38,720 --> 00:18:44,799
and there's a particularly annoying form

553
00:18:42,160 --> 00:18:46,640
of software that is called

554
00:18:44,799 --> 00:18:47,600
a bug

555
00:18:46,640 --> 00:18:50,000
all right

556
00:18:47,600 --> 00:18:51,520
so if we go through this cycle we're not

557
00:18:50,000 --> 00:18:52,960
going to have robust software dropping

558
00:18:51,520 --> 00:18:54,960
out the bottom unless we're luckier than

559
00:18:52,960 --> 00:18:57,440
anyone deserves to be instead we're

560
00:18:54,960 --> 00:18:59,840
going to have software and bugs that

561
00:18:57,440 --> 00:19:02,240
have adapted to the validation

562
00:18:59,840 --> 00:19:03,840
this by the way is one reason that test

563
00:19:02,240 --> 00:19:06,000
suites tend to become less effective

564
00:19:03,840 --> 00:19:07,200
over time because the bugs adapt to the

565
00:19:06,000 --> 00:19:08,799
test suite and it doesn't find them

566
00:19:07,200 --> 00:19:11,200
anymore

567
00:19:08,799 --> 00:19:13,440
what this means is that we have to do

568
00:19:11,200 --> 00:19:15,120
more okay so one additional thing would

569
00:19:13,440 --> 00:19:17,360
be put a loop on the bottom as we get

570
00:19:15,120 --> 00:19:19,200
bug reports from the field escapes

571
00:19:17,360 --> 00:19:21,200
uh yeah we fixed the bug in the software

572
00:19:19,200 --> 00:19:22,559
sure but let's also fix the validation

573
00:19:21,200 --> 00:19:24,160
where feasible

574
00:19:22,559 --> 00:19:25,440
so that will catch similar bugs in the

575
00:19:24,160 --> 00:19:28,080
future

576
00:19:25,440 --> 00:19:29,760
if we do this we should have some

577
00:19:28,080 --> 00:19:30,880
reasonable chance of getting

578
00:19:29,760 --> 00:19:32,799
better

579
00:19:30,880 --> 00:19:35,200
software at the bottom on fewer adapted

580
00:19:32,799 --> 00:19:35,200
bugs

581
00:19:35,919 --> 00:19:39,360
still you have to be a little bit

582
00:19:37,039 --> 00:19:41,520
careful

583
00:19:39,360 --> 00:19:42,640
you see normally what you do is you

584
00:19:41,520 --> 00:19:43,919
write

585
00:19:42,640 --> 00:19:46,240
if you write tests at all hopefully you

586
00:19:43,919 --> 00:19:47,440
do okay you do write tests right

587
00:19:46,240 --> 00:19:49,440
when you do that you're going to

588
00:19:47,440 --> 00:19:51,280
validate the use cases you're aware of i

589
00:19:49,440 --> 00:19:53,919
mean you write a test case for a use

590
00:19:51,280 --> 00:19:55,600
case i'm not aware of what would i write

591
00:19:53,919 --> 00:19:57,200
and then what happens is we randomly

592
00:19:55,600 --> 00:19:59,280
generate our software which has lots of

593
00:19:57,200 --> 00:20:01,360
bugs we run the tests and fix the bugs

594
00:19:59,280 --> 00:20:04,400
and if things work perfectly we get rid

595
00:20:01,360 --> 00:20:06,400
of all the bugs that the test finds

596
00:20:04,400 --> 00:20:08,720
and then we do it again right and then

597
00:20:06,400 --> 00:20:10,960
we end up something like that

598
00:20:08,720 --> 00:20:12,720
okay well you know our users aren't

599
00:20:10,960 --> 00:20:14,400
inconvenienced by the bug so what's the

600
00:20:12,720 --> 00:20:16,240
problem

601
00:20:14,400 --> 00:20:17,919
well the problem is of course that the

602
00:20:16,240 --> 00:20:19,919
world changes

603
00:20:17,919 --> 00:20:22,000
and so we're likely to get some new use

604
00:20:19,919 --> 00:20:23,120
cases and what we'll find if we've been

605
00:20:22,000 --> 00:20:24,799
doing this

606
00:20:23,120 --> 00:20:26,720
is those new use cases will have been

607
00:20:24,799 --> 00:20:28,559
protected from our software by walls of

608
00:20:26,720 --> 00:20:30,240
bugs

609
00:20:28,559 --> 00:20:32,559
and the poor guys that are trying to

610
00:20:30,240 --> 00:20:34,480
make those use cases happen might

611
00:20:32,559 --> 00:20:35,679
quite understandably decide you know

612
00:20:34,480 --> 00:20:37,120
this is a real pain i'm going to write

613
00:20:35,679 --> 00:20:38,159
my own software thank you this isn't

614
00:20:37,120 --> 00:20:40,960
working

615
00:20:38,159 --> 00:20:42,960
now don't get me wrong

616
00:20:40,960 --> 00:20:44,880
life is sometimes hard and sometimes

617
00:20:42,960 --> 00:20:46,400
hard choices have to be made and if

618
00:20:44,880 --> 00:20:47,200
you're in a situation where you can only

619
00:20:46,400 --> 00:20:49,360
fix

620
00:20:47,200 --> 00:20:50,720
one of two bugs

621
00:20:49,360 --> 00:20:52,559
yeah you're going to fix the one that's

622
00:20:50,720 --> 00:20:53,679
hurting your users right now

623
00:20:52,559 --> 00:20:57,760
but

624
00:20:53,679 --> 00:20:58,720
understand this is where that leads to

625
00:20:57,760 --> 00:21:00,480
so

626
00:20:58,720 --> 00:21:02,559
what we'd like to do then if we have the

627
00:21:00,480 --> 00:21:04,400
luxury of doing that and

628
00:21:02,559 --> 00:21:06,559
sometimes we do

629
00:21:04,400 --> 00:21:09,360
instead of just bug reports we'd like to

630
00:21:06,559 --> 00:21:11,919
add paranoia and uh make the validation

631
00:21:09,360 --> 00:21:13,840
a little bit more aggressive and try to

632
00:21:11,919 --> 00:21:15,039
clear the way for future

633
00:21:13,840 --> 00:21:16,799
use cases

634
00:21:15,039 --> 00:21:18,799
and the fuzzers actually seem to do a

635
00:21:16,799 --> 00:21:20,400
reasonable job of that at least so far

636
00:21:18,799 --> 00:21:24,240
so we are already doing this to some

637
00:21:20,400 --> 00:21:25,679
extent and hopefully that will continue

638
00:21:24,240 --> 00:21:27,120
okay

639
00:21:25,679 --> 00:21:29,440
the another thing is that natural

640
00:21:27,120 --> 00:21:31,520
selection is a euphemism for a rather

641
00:21:29,440 --> 00:21:32,720
ugly process actually

642
00:21:31,520 --> 00:21:34,159
but sticking just to software and

643
00:21:32,720 --> 00:21:35,200
avoiding living things for the rest of

644
00:21:34,159 --> 00:21:37,520
the slide

645
00:21:35,200 --> 00:21:39,200
the key point is if your tests are not

646
00:21:37,520 --> 00:21:40,799
failing they are not helping to improve

647
00:21:39,200 --> 00:21:42,480
your software

648
00:21:40,799 --> 00:21:44,320
that actually bears repeating if your

649
00:21:42,480 --> 00:21:46,480
tests are not failing at least from time

650
00:21:44,320 --> 00:21:48,000
to time they are not helping to improve

651
00:21:46,480 --> 00:21:50,400
your software

652
00:21:48,000 --> 00:21:52,080
they might be maybe prevent preventing

653
00:21:50,400 --> 00:21:55,280
bugs from coming in but they aren't

654
00:21:52,080 --> 00:21:55,280
actually improving it as it is

655
00:21:55,679 --> 00:21:57,760
so

656
00:21:56,480 --> 00:21:59,440
what's the price than a robust

657
00:21:57,760 --> 00:22:01,280
concurrent software

658
00:21:59,440 --> 00:22:03,360
as you know internal bug fixing i mean

659
00:22:01,280 --> 00:22:05,840
we all know that we've done it but also

660
00:22:03,360 --> 00:22:07,120
eternal validation development

661
00:22:05,840 --> 00:22:09,200
as

662
00:22:07,120 --> 00:22:11,120
especially in the popular things like

663
00:22:09,200 --> 00:22:13,280
the linux kernel and any number of other

664
00:22:11,120 --> 00:22:15,360
popular software projects

665
00:22:13,280 --> 00:22:19,120
new use cases appear all the time and we

666
00:22:15,360 --> 00:22:19,120
need to prepare the software for them

667
00:22:19,840 --> 00:22:24,159
so we've gone through these topics

668
00:22:21,919 --> 00:22:26,640
torturing rcu tracking down heisenbugs

669
00:22:24,159 --> 00:22:28,640
installed base causing trouble uh

670
00:22:26,640 --> 00:22:30,720
natural selection good and bad ugly and

671
00:22:28,640 --> 00:22:33,200
the price of current software

672
00:22:30,720 --> 00:22:34,960
uh this next slide uh is where you can

673
00:22:33,200 --> 00:22:36,799
find some more information as is the

674
00:22:34,960 --> 00:22:39,039
slide after that

675
00:22:36,799 --> 00:22:41,679
and with that uh if you've got questions

676
00:22:39,039 --> 00:22:43,360
i'm happy to answer them or at least say

677
00:22:41,679 --> 00:22:46,840
something in response to them

678
00:22:43,360 --> 00:22:46,840
so any questions

679
00:23:03,840 --> 00:23:08,320
hearing none uh thank you all very much

680
00:23:05,679 --> 00:23:11,360
for your time and attention

681
00:23:08,320 --> 00:23:13,360
and uh here we are

682
00:23:11,360 --> 00:23:15,760
oh we do have one okay i guess i spoke

683
00:23:13,360 --> 00:23:15,760
too soon

684
00:23:19,919 --> 00:23:25,200
okay should we view the rcu torture as a

685
00:23:23,039 --> 00:23:27,360
pass fail only or other statistics that

686
00:23:25,200 --> 00:23:29,679
can flag performance aggressions that's

687
00:23:27,360 --> 00:23:31,039
actually a good question and then that

688
00:23:29,679 --> 00:23:32,080
gives me excuse to revisit an early

689
00:23:31,039 --> 00:23:34,400
slide

690
00:23:32,080 --> 00:23:37,840
let's get back up to it

691
00:23:34,400 --> 00:23:37,840
eventually here

692
00:23:39,039 --> 00:23:44,480
yeah there we go

693
00:23:42,240 --> 00:23:46,640
okay that slide

694
00:23:44,480 --> 00:23:49,279
um there is our statistics

695
00:23:46,640 --> 00:23:52,159
uh the the uh column after all the

696
00:23:49,279 --> 00:23:54,880
dashes is the grace periods that were

697
00:23:52,159 --> 00:23:56,400
executed for the full test

698
00:23:54,880 --> 00:23:58,080
and uh

699
00:23:56,400 --> 00:23:59,520
and then that there's also per seconds

700
00:23:58,080 --> 00:24:01,760
as you can see the different types of

701
00:23:59,520 --> 00:24:03,279
rcu have rather wildly different

702
00:24:01,760 --> 00:24:04,799
uh statistics on the number of grace

703
00:24:03,279 --> 00:24:06,159
bridges per second and that's that's

704
00:24:04,799 --> 00:24:07,279
life i mean they're doing very different

705
00:24:06,159 --> 00:24:08,720
things

706
00:24:07,279 --> 00:24:11,919
um

707
00:24:08,720 --> 00:24:14,480
the type of rcub tested so tasks rude is

708
00:24:11,919 --> 00:24:17,039
uh one that uses uh

709
00:24:14,480 --> 00:24:20,000
work cues to interrupt all the cpus kind

710
00:24:17,039 --> 00:24:22,080
of the old style approach and so on

711
00:24:20,000 --> 00:24:23,919
uh and so it'll it'll give some state

712
00:24:22,080 --> 00:24:26,960
information pertaining to that type of

713
00:24:23,919 --> 00:24:28,960
rcu uh there uh the grace period number

714
00:24:26,960 --> 00:24:31,840
and the flags thing that says whether

715
00:24:28,960 --> 00:24:34,000
there's a grace period being asked for

716
00:24:31,840 --> 00:24:36,000
there are a few types of rcu that are

717
00:24:34,000 --> 00:24:38,000
robust enough to be able to stand having

718
00:24:36,000 --> 00:24:39,120
piles of callbacks jammed down their

719
00:24:38,000 --> 00:24:41,360
throats

720
00:24:39,120 --> 00:24:42,960
and for those and if you've selected the

721
00:24:41,360 --> 00:24:45,120
test that's the the

722
00:24:42,960 --> 00:24:45,120
if

723
00:24:45,440 --> 00:24:49,919
the progress uh uh

724
00:24:48,080 --> 00:24:52,240
kernel

725
00:24:49,919 --> 00:24:54,720
parameter we talked about earlier

726
00:24:52,240 --> 00:24:57,200
it'll tell you how many the maximum

727
00:24:54,720 --> 00:24:59,360
queue depth of of callbacks

728
00:24:57,200 --> 00:25:02,080
and so uh the tiny ones which are single

729
00:24:59,360 --> 00:25:04,480
cpu keep up fairly well uh the worst

730
00:25:02,080 --> 00:25:06,240
case is tree zero three which

731
00:25:04,480 --> 00:25:08,320
uh it makes things hard on its

732
00:25:06,240 --> 00:25:09,760
configuration many things hard on rcu

733
00:25:08,320 --> 00:25:12,000
and there was a point in time when there

734
00:25:09,760 --> 00:25:13,360
were in this particular run 10 million

735
00:25:12,000 --> 00:25:15,520
callbacks excuse me one million

736
00:25:13,360 --> 00:25:17,600
callbacks waiting to be

737
00:25:15,520 --> 00:25:19,279
uh invoked

738
00:25:17,600 --> 00:25:21,919
so there are some statistics let's see

739
00:25:19,279 --> 00:25:24,159
i've gotten behind on questions um

740
00:25:21,919 --> 00:25:26,880
and uh the statistics some of the six is

741
00:25:24,159 --> 00:25:28,159
really noisy for example the nmax cbs is

742
00:25:26,880 --> 00:25:32,000
quite noisy so you have to be a little

743
00:25:28,159 --> 00:25:32,000
bit careful relying too much on that one

744
00:25:33,919 --> 00:25:36,799
what's failure look like you know i

745
00:25:35,600 --> 00:25:39,120
you're right i should have a slide on

746
00:25:36,799 --> 00:25:41,840
that i don't what'll do what'll happen

747
00:25:39,120 --> 00:25:44,080
is between these you'll see these lines

748
00:25:41,840 --> 00:25:46,080
and then if there was a failure

749
00:25:44,080 --> 00:25:48,240
after the line corresponding to the test

750
00:25:46,080 --> 00:25:50,159
that failed there'd be stuff printed out

751
00:25:48,240 --> 00:25:52,240
which couldn't vary really wildly

752
00:25:50,159 --> 00:25:53,840
depending on exactly what failed

753
00:25:52,240 --> 00:25:55,600
it'll give a summary of i got this many

754
00:25:53,840 --> 00:25:57,600
warnings i got this many stalls it'll

755
00:25:55,600 --> 00:25:59,440
print out the occasional thing about

756
00:25:57,600 --> 00:26:03,039
what's going on there's quite a bit of

757
00:25:59,440 --> 00:26:03,039
variety on the failure reports

758
00:26:06,720 --> 00:26:10,400
can more testing be done with

759
00:26:08,640 --> 00:26:12,400
emulated timing hit entries case and

760
00:26:10,400 --> 00:26:14,559
currency bugs are lighting on relying on

761
00:26:12,400 --> 00:26:15,840
large numbers of machines and uh doing

762
00:26:14,559 --> 00:26:17,679
things randomly

763
00:26:15,840 --> 00:26:19,200
uh there is a little bit of that in rsu

764
00:26:17,679 --> 00:26:20,320
as it is now

765
00:26:19,200 --> 00:26:22,480
so

766
00:26:20,320 --> 00:26:25,039
there are places in the grace period

767
00:26:22,480 --> 00:26:26,559
k-thread where some of the tests will

768
00:26:25,039 --> 00:26:28,000
insert delays

769
00:26:26,559 --> 00:26:31,520
and if i remember correct that's one of

770
00:26:28,000 --> 00:26:33,760
the reasons 303 has a hard time

771
00:26:31,520 --> 00:26:33,760
and

772
00:26:34,080 --> 00:26:38,559
what will happen is that it'll randomly

773
00:26:36,320 --> 00:26:40,720
insert delays as you ask it to in

774
00:26:38,559 --> 00:26:42,720
addition uh there's random delays

775
00:26:40,720 --> 00:26:43,919
inserted in the in the readers that are

776
00:26:42,720 --> 00:26:45,679
being done

777
00:26:43,919 --> 00:26:48,080
uh you could of course do that quite a

778
00:26:45,679 --> 00:26:49,919
bit more there have been at eth zurich i

779
00:26:48,080 --> 00:26:51,120
believe it was there were about 10 years

780
00:26:49,919 --> 00:26:52,640
ago there was a project that would

781
00:26:51,120 --> 00:26:53,840
insert those automatically based on

782
00:26:52,640 --> 00:26:55,679
various things

783
00:26:53,840 --> 00:26:56,960
perhaps another sanitizer sometime in

784
00:26:55,679 --> 00:27:00,240
the kernel's future we'll do that

785
00:26:56,960 --> 00:27:00,240
automatically throughout the kernel

786
00:27:00,559 --> 00:27:04,799
how can we identify which torture is the

787
00:27:02,480 --> 00:27:07,200
most relevant to an employer's use cases

788
00:27:04,799 --> 00:27:07,200
okay

789
00:27:07,520 --> 00:27:11,679
okay let's uh

790
00:27:09,919 --> 00:27:15,440
take a look at that one one thing to do

791
00:27:11,679 --> 00:27:18,320
is to look at the kernel you build

792
00:27:15,440 --> 00:27:19,440
and see what rc related k config options

793
00:27:18,320 --> 00:27:21,120
it uses

794
00:27:19,440 --> 00:27:23,200
and so one thing you could do

795
00:27:21,120 --> 00:27:24,559
there's a directory

796
00:27:23,200 --> 00:27:27,279
what you can do in fact to find that

797
00:27:24,559 --> 00:27:29,840
directory is just for example do a find

798
00:27:27,279 --> 00:27:31,919
uh dot dash name root zero one dash

799
00:27:29,840 --> 00:27:34,159
print okay and that'll that'll show you

800
00:27:31,919 --> 00:27:36,080
the directory where the root zero one

801
00:27:34,159 --> 00:27:39,360
file exists and that root zero one file

802
00:27:36,080 --> 00:27:41,679
defines the k config options for the

803
00:27:39,360 --> 00:27:43,679
root zero one scenario there also is a

804
00:27:41,679 --> 00:27:46,480
root zero one dot boot file the boot

805
00:27:43,679 --> 00:27:48,799
file is optional and if it exists it'll

806
00:27:46,480 --> 00:27:50,960
have boot arguments that help define

807
00:27:48,799 --> 00:27:54,480
that so what you could do is just make

808
00:27:50,960 --> 00:27:56,480
your own scenario for your employer

809
00:27:54,480 --> 00:27:57,840
just go and drop the file into that

810
00:27:56,480 --> 00:28:01,679
directory

811
00:27:57,840 --> 00:28:04,159
it's tools testing self-test rcu torture

812
00:28:01,679 --> 00:28:05,760
configs rcu and then the names of the

813
00:28:04,159 --> 00:28:08,960
files

814
00:28:05,760 --> 00:28:10,080
uh and so you can put a you know

815
00:28:08,960 --> 00:28:12,240
my company

816
00:28:10,080 --> 00:28:13,919
uh all it has to be all caps you have

817
00:28:12,240 --> 00:28:15,200
dashes but it has to be all caps this is

818
00:28:13,919 --> 00:28:18,159
an old things that's why there's

819
00:28:15,200 --> 00:28:20,080
lowercase d and u there

820
00:28:18,159 --> 00:28:22,399
and put in the k config parameters that

821
00:28:20,080 --> 00:28:24,080
match the kernel you guys use and get

822
00:28:22,399 --> 00:28:25,840
exactly what you want

823
00:28:24,080 --> 00:28:27,840
and then also the boot

824
00:28:25,840 --> 00:28:29,600
dot boot file if you have boot arguments

825
00:28:27,840 --> 00:28:31,200
that you care about and then you can run

826
00:28:29,600 --> 00:28:33,200
whatever you want otherwise what you can

827
00:28:31,200 --> 00:28:34,399
do is just go through the existing files

828
00:28:33,200 --> 00:28:36,480
and see if there's one that's a close

829
00:28:34,399 --> 00:28:38,799
match and there might well be

830
00:28:36,480 --> 00:28:40,919
there is a another file in that bin

831
00:28:38,799 --> 00:28:42,880
directory called

832
00:28:40,919 --> 00:28:44,960
config.csv.sh and that will make a

833
00:28:42,880 --> 00:28:47,039
spreadsheet of the existing ones

834
00:28:44,960 --> 00:28:48,960
the ones that are run by default and you

835
00:28:47,039 --> 00:28:50,480
can look at that and and get a quicker

836
00:28:48,960 --> 00:28:51,919
view

837
00:28:50,480 --> 00:28:54,480
how often are you finding this is

838
00:28:51,919 --> 00:28:56,320
catching bugs in new patches

839
00:28:54,480 --> 00:28:58,480
uh if i write a patch i probably have an

840
00:28:56,320 --> 00:29:00,960
rc torture failure in my future

841
00:28:58,480 --> 00:29:02,480
i sometimes find uh problems in other

842
00:29:00,960 --> 00:29:04,960
parts of the colonel

843
00:29:02,480 --> 00:29:07,120
uh the there was a rsu torture failure

844
00:29:04,960 --> 00:29:09,120
that showed up just before thanksgiving

845
00:29:07,120 --> 00:29:10,960
that and i forget where it was uh

846
00:29:09,120 --> 00:29:12,240
frederick weisberg took pity on me and

847
00:29:10,960 --> 00:29:14,080
chased it down for me so i could

848
00:29:12,240 --> 00:29:15,919
actually could celebrate uh usa

849
00:29:14,080 --> 00:29:17,360
thanksgiving holiday he's in france so

850
00:29:15,919 --> 00:29:20,799
he didn't have that one

851
00:29:17,360 --> 00:29:22,159
i hope to return the favor someday

852
00:29:20,799 --> 00:29:24,000
we had found some things in the

853
00:29:22,159 --> 00:29:25,120
scheduler uh we occasionally find things

854
00:29:24,000 --> 00:29:27,520
and timers

855
00:29:25,120 --> 00:29:30,240
and so uh

856
00:29:27,520 --> 00:29:33,919
but mostly uh i'll write a patch i'll

857
00:29:30,240 --> 00:29:33,919
run oversea torture and it'll explode

858
00:29:34,640 --> 00:29:37,919
anyway i think we're at the end of the

859
00:29:36,480 --> 00:29:39,679
session thank you all very much for your

860
00:29:37,919 --> 00:29:41,600
time and attention and for all the good

861
00:29:39,679 --> 00:29:44,600
questions and have a great rest of the

862
00:29:41,600 --> 00:29:44,600
conference