thomasjudgeApr 2025Recommends
For an entertaining/terrifying perspective on firmware, obligatory Bryan Cantrill talk "Zebras All the Way Down" ▶ this video
in “Synology Lost the Plot with Hard Drive Locking Move” · View on HN
Zebras All the Way Down - Bryan Cantrill, Uptime 2017
Zebras All the Way Down - Bryan Cantrill, Uptime 2017 on HackerTube: recommended by 6 Hacker News commenters across 6 separate months, 2018–2025.
Code & Supply
Open on YouTubetalk · intermediate · evergreen
Bryan Cantrill's Uptime 2017 talk on the engineering challenges of the stateful data path, including storage, firmware and hardware bugs. It uses the 'zebras vs. horses' diagnostic heuristic to explain why rare failures keep appearing.
Why Hacker News recommends it
A perennial HN favorite whenever firmware, disks, ECC or opaque hardware come up, valued as both entertaining and sobering.
- people recommended it
- 6
- separate months
- 6
- strong recommendations
- 0
- discussed on HN
- 2018–2025
“As with all his talks, you can expect it to be quite entertaining as well as informative”
What HN commenters said
18 Hacker News comments link this video. Each one links back to its thread.
BayartJan 2021Recommends
>Why does anyone really care about non-free firmware blobs? Aside from the philosophical considerations, stuff breaking in a black-box firmware that you're lead into believing "just works" is particularly aggravating. I'll link Bryan Cantrill's talk on the subject, he's more entertaining than I am. [1] [1] ▶ this video
in “Installing Debian on Modern Hardware” · View on HN
ignoramousFeb 2020Recommends
In my time writing databases, I found that not only the mount-flags could result in data-loss despite fsync, given you use fsync as it is supposed to be used [0], but even enterprise-grade hardware would at times (at that scale [1]) drop and/or corrupt a few bits here and there. That reminds me of the case where AWS famously revealed that a single NIC in their ginaromous S3 fleet flipped a single bit once in a while and that caused an outage because their gossip-daemon responsible for fleet health-checks failed spectacularly [2]. Bryan Cantrill's talk on the realted topic of hardware/firmware bugs is pretty good [3]. [0] [link] [1] [link] [2] [link] [3] ▶ this video
in “Ask HN: What were the things you did that made the biggest impact at your work?” · View on HN
kbensonJan 2020Recommends
There's a good Bryan Cantrill talk about that.[1] The gist is that eventually, when you throw enough resources at a problem, all that's left are the really uncommon problem and bugs, and this is specifically what you get in the data path (including drive firmware) where things get harder and harder to figure out as the code gets more hidden and obscure. As with all his talks, you can expect it to be quite entertaining as well as informative and historical (if from his POV). 1: ▶ this video
in “Five Years of Btrfs” · View on HN
kbensonJul 2019Recommends
On this topic (and related in many ways) is Bryan Cantrill's talk Zebras All the way Down.[1] The gist is that in medicine there's the concept of diseases that are zebras and horses, rare and common, and the saying goes "when you hear hoof beats think horses not Zebras". Unlike medicine, in software most the easy problems have already been abstracted away or made trivial through our software stacks, so often what's left are Zebras. What used to be the exceedingly rare bugs now account for a large proportion of the odd behavior we see in very robust tech stacks, because the easy bugs have been hunted down and eliminated already. 1: ▶ this video
in “Replacing a silently failing disk in a ZFS pool” · View on HN
gt2Oct 2018Recommends
Zebras All the Way Down - Bryan Cantrill, Uptime 2017 ▶ this video Lots of great tangents that add to the subjects. I love talks like this.
in “Ask HN: What are some of the best technical talks you've heard?” · View on HN
Show 12 more comments
ar0bOct 2018Recommends
One of my favorite conference talks is semi related to problems like this. ▶ this video Brian Cantrill - Zebras all the way down
in “iPhones are allergic to helium” · View on HN
throw0101cJul 2025Cites it
> Or you can spend half a billion dollars to solve the issue in hardware. And hope that your hardware/firmware doesn't ever get bugs. Or you can do checksumming at the hardware layer and checksumming at the software/FS layer. Protection in depth. ZFS has caught issues from hardware, like when LBA 123 is requested but LBA 456 is delivered: the hardware-level checksum for LBA 456 was fine, and so it was passed up the stack, but it wasn't actually the data that was asked for. See Bryan Cantrill's talk "Zebras All the way Down": * ▶ this video And if checksums are not needed for a particular use-case, make them toggleable: even ZFS has a set checksums=off option. My problem is not having the option at all .
in “Upgrading an M4 Pro Mac mini's storage for half the price” · View on HN
ElectricalUnionDec 2023Cites it
> When we really should have been formally addressing problems from first principles. Because if our OS can't recover gracefully when the disk fills up, then we don't really have anything. And so many OSs (including our desktops) panic when the disk fills up, that we don't really have anything. At a low enough level, all of the tools we rely upon are trash. Unfortunately, the OS can't really "recover" most of the time because it is being constantly lied to by several(!!!) layers of uncooperative hardware/firmware. It's zebras all the way down: ▶ this video > In this case, by letting programs wait until the user has cleared off disk space If your system is really in a no-disk-space-left state you might not be able to login to fix it: [link] And even assuming you can login, sometimes you can't delete files either; you need to free space in another way before being able to delete files. [link] Filesystems and persistence are hard.
in “Database Fundamentals” · View on HN
tgbugsJan 2021Cites it
A relevant Bryan Cantrill talk segment on this, which heightens the paranoia around this. Namely, firmware hiding correctable errors and only reporting uncorrectable errors. ▶ this video
in “ECC matters” · View on HN
floatbothSep 2019Cites it
Well, on the other hand, better to have the ability to deal with it rather than have everything obfuscated in unfixable proprietary firmware… ▶ this video
in “Samsung Announces Key-Value SSD Prototype” · View on HN
ShroudedNightJul 2019Cites it
It's worth noting that in a computing context the reliability of this heuristic is questionable relative to what it would be in medicine: ▶ this video Zebras All the Way Down - Bryan Cantrill, Uptime 2017
in “Investigating Multi-Account IAM Issues in S3 and CloudFront” · View on HN
cypharDec 2018Cites it
> skates of magnetic readers flying on a cushion of air above a rapidly rotating disc, with the gap separating a few dozen nanometers, often smaller than the process size in the controller's silicon. Complete aside, the fly-height of a magnetic head is actually fractions of a nanometer (i.e. hundreds of picometers). EDIT: I got this from a talk by Bryan Cantrill[1]. The fly-height is allegedly 0.8 nanometers (800 picometers). [1]: ▶ this video
in “Why I'm usually unnerved when modern SSDs die on us” · View on HN
cypharDec 2018Cites it
(Sheepishly) My only reference is a talk by Bryan Cantrill[1] where he states that an exec at a hard-drive manufacturer stated it was ".8 nanometers". I will try to find a better source than that. EDIT: There is a paper from 2016 which did an analysis in the difference in the flying height of the head during different operations, and it was measured in Angstroms[2]. I couldn't find one that actually gives a precise value of the flying height. There is a 2011 paper which states that some system they were testing allowed for 4-9 nm flying heights[3] which is about half-an-order-of-magnitude larger than the claim -- but that's already 7 years old. [1]: ▶ this video [2]: [link] [3]: [link]
in “Why I'm usually unnerved when modern SSDs die on us” · View on HN
cypharDec 2018Cites it
Bryan Cantrill made this claim in a talk recently[1], and the fly-height is apparently 800 picometers. [1]: ▶ this video
in “Why I'm usually unnerved when modern SSDs die on us” · View on HN
nordsieckApr 2018Cites it
We Really Don't Know How to Compute: Gerry Sussman - [link] Zebras All the Way Down: Bryan Cantrill - ▶ this video Jonathan Blow on Deep Work: Jonathan Blow - [link] Simple Made Easy: Rich Hickey - [link] Effective Programs - 10 Years of Clojure: Rich Hickey - [link] The Last Thing D Needs: Scott Meyers - [link]
in “Ask HN: What is your favourite tech talk?” · View on HN
floatbothJan 2018Cites it
On The Turing Completeness of PowerPoint [link] Compiling C to printable x86, to make an executable research paper [link] HDR Photography in Microsoft Excel?! [link] Zebras All the Way Down ▶ this video Solving Layout Problems with CSS Grid and Friends [link] Console Security - Switch [link] Let's move SMM out of firmware and into the kernel [link] Bringing Linux back to server boot ROMs with NERF and Heads [link] Sharing the Chisel [link] (looks like the actual conference video wasn't released)
in “Ask HN: What’s your favorite talk from 2017?” · View on HN
VoultapherDec 2023Joke
I'm shocked! Firmware vendors, the last bastion of high quality memory-safe C code, solving complex problems. Code firmly audited and formally verified to the highest degree. And now you tell me these people, with the highest known software quality got "parsing image formats" wrong? It's truly zebras all the way down ▶ this video
in “Just about every Windows/Linux device vulnerable to new LogoFAIL firmware attack” · View on HN
Topics & tags
Databases & storageTesting, debugging & developer toolsCloud, DevOps & reliabilitybryan cantrillfirmwarestoragedata pathdebugging