Debugging Under Fire: Keep your Head when Systems have Lost their Mind • Bryan Cantrill • GOTO 2017

I'll give it a watch and will summarize, because while i'm not always on board with the Rust hype (and therefore want to know more, to help eliminate my biases, one way or the other). That said, i rather enjoyed Bryan Cantrill's talk from 2017, "Debugging Under Fire: Keep your Head when Systems have Lost their Mind • Bryan Cantrill • GOTO 2017": ▶ this video So i wouldn't necessarily turn away from any of his videos just because of the sometimes humorous or not awfully serious tone.

in “Is it time to rewrite the Operating System in Rust? (2018) [video]” · View on HN

Not (I think) the exacttalk/blog post gp was thinking of - but worth watching IMNHO: "Debugging Under Fire: Keep your Head when Systems have Lost their Mind • Bryan Cantrill • GOTO 2017" ▶ this video Ed: oh, here we go I think? > Running Aground: Debugging Docker in Production Bryan Cantrill19,102 views16 Jan 2018 Talk originally given at DockerCon '15, which (despite being a popular presentation and still broadly current) Docker Inc. has elected to delist. [link]

in “Harbormaster: Anti-Kubernetes for your personal server” · View on HN

> Joyent famously took down their whole region by rebooting wrong nodes. In case anyone's interested, here's a pretty funny and educational talk by Bryan Cantrill about that particular incident: GOTO 2017 • Debugging Under Fire: Keep your Head when Systems have Lost their Mind ▶ this video

in “Google admits Kubernetes container tech is too complex” · View on HN

> I run a YouTube channel that is over 10 years old , has over 770,000 subscribers , almost 600 public videos , 120,000,000 views When writing software involved in managing a live, public, massively multi-user system, the traditional unix-style commands that are immediate, often silent, and capable of damaging effects become a really easy way to shoot yourself in the foot. Worse, some commands might let you accidentally shoot everyone's foot on a typo. The traditional example is accidentally typing something like "rm -f * .bak" (note the extra space after the star). For a good discussion of this type of problem, I highly recommend Bryan Cantrill's talk[1] about the time an operator accidentally rebooted an entire datacenter with a single miss-typed command. The general solution to this is building sanity checks into the software. The user just asked to reformat 500 hosts, but almost all previous uses olf the 'reformat' command affected less than 10 hosts. Maybe we should ask for verification from an actual human if they really intended to run this unusually destructive command. Why doesn't YouTube have this kind of sanity check in their automated takedown/strike/channel-deletion tools? Google wrote automation that can decide to delete a channel with a long history and many successful videos. Why doesn't that automation have basic sanity checks that ask for operator input when asked to do an unusually destructive action like deleting a 10 year old channel with a huge history?

in “They're deleting my channel, but they don't know why? [video]” · View on HN

> In my experience customers deeply detest the idea of waiting around for a failure case to re-occur so that you can understand it better. Bryan Cantrill has a great talk[0] about dealing with fires where he says something to the effect of: > Now you will find out if you are more operations or development - developers will want to leave things be to gather data and understand, while operations will want to rollback and fix things as quickly as possible [0] Debugging Under Fire: Keep your Head when Systems have Lost their Mind - Bryan Cantrill: ▶ this video

in “Root cause analysis: significantly elevated error rates on 2019‑07‑10” · View on HN

I was! (Along with co-author Dave Pacheco.) And I still have the dream that we'll finish it one day: we had written probably a third of it, but then life intervened in various dimensions. And indeed, as part of our preparation to write our book (which we titled The Joy of Debugging ), we read Wheeler's Debugging . On the one hand, I think it's great to have anything written about debugging, as it's a subject that has not been treated with the weight that it deserves. But on the other, the "methodology" here is really more of a collection of aphorisms; if folks find it helpful, great -- but I came away from Debugging thinking that the canonical book on debugging has yet to be written. Fortunately, my efforts with Dave weren't for naught: as part of testing our own ideas on the subject, I gave a series of presentations from ~2015 to ~2017 that described our thinking. A talk that pulls many of these together is my GOTO Chicago talk in 2017, on debugging production systems.[0] That talk doesn't incorporate all of our thinking, but I think it gets to a lot of it -- and I do think it stands at a bit of contrast to Wheeler's work. [0] ▶ this video

in “Debugging: Indispensable rules for finding even the most elusive problems (2004)” · View on HN

I'm reminded of Bryan Cantrill's talk "Debugging Under Fire"[1], which includes a retrospective of sorts about an entire datacenter rebooting.[2] That is a pretty large-scale disaster, but even that is a rung below a continent-wide outage. Poor "Bill" must have saw the proverbial light when he heard some folks wanted to trash the DIASTER script. [1]: ▶ this video [2]: [link]

in “Cold restart whole system after total outage” · View on HN

If it were so obvious, I am wondering why you need to worry about doing it wrong. You talk about trade-offs as if they are fixed in time. That is not the case at all in anything but toy systems. Maybe you need to find someone who can design a quality micro-service? You have no idea what I do and what my competencies are, yet, you are assuming you know a lot more about system design than I do and I am missing something simple. Here is another take from Bryan Cantrill [ ▶ this video ]. Very likely, he does not know anything about services either, I guess.

in “Amazon confirms corporate staff cuts that could hit 10k employees” · View on HN

> All I say is that you must relay on externals if A-E are all on the same network as it may go down. Thankfully, it's not too hard to take advantage of multiple networks in a hybrid/multi-cloud setup nowadays! Though, depending on the necessary access controls and auditing, such a setup might require slightly more work. You do bring up an excellent point, though, about how it's a serious single point of failure in many systems out there, because personally I've also seen many setups like that (the majority of them, actually): I do suspect that in many cases that is indeed done for ease of use/convenience, even if it may lead to downtime. Of course, in some cases downtime is acceptable, so I cannot argue that it can also make sense to choose such a simpler setup - for example, for having your own company's applications/monitoring for development environments all on the same network. Though if this topology is retained at scale, things can get a bit interesting. On a similar note, I recall Bryan Cantrill doing an interesting presentation "Debugging Under Fire: Keep your Head when Systems have Lost their Mind" that talked about restarting their whole data center and the implications of that: ▶ this video

in “Upptime/upptime: Uptime monitor and status page powered by GitHub” · View on HN

It reminds me of this: ▶ this video -- "GOTO 2017 • Debugging Under Fire: Keep your Head when Systems have Lost their Mind • Bryan Cantrill"

in “Google outage – resolved” · View on HN

> I'm not sure you do, since you admitted to really being at least part sysadmin, earlier :) Hah! I wish I could remember which talk it was. Bryan Cantrill had a good line about DevOps in one of his talks (I think it was this one: ▶ this video ). The gist was "you can say you're DevOps, but when the shit hits the fan, you're either going to be Dev or Ops. If you're a Dev, you're going to want to debug the problem before rebooting the failed machine. If you're Ops, you're going to want to reboot the machine as fast as you can to get it back up." Through that lens, I'm definitely pretty far over on the Dev spectrum; when something goes catastrophically wrong and someone reboots a box to "solve the problem", my first reaction is "YOU FUCKER YOU BURNED THE CORPSE" It's such a tricky thing all around. Looking back at what I wrote earlier, I also realize that in some ways I'm facilitating the ignorance. I've got a really nice Consul and Nomad setup for the team, so they can pretty much just toss .wars and Docker containers at the cluster and they'll automatically get scheduled somewhere with spare capacity. The load balancer, the database cluster, all of the service discovery and job scheduling stuff... they've never had to get in and set any of that up. Maybe it's time to do more mentoring... Anyway, thanks for getting me thinking. I've been a little bit grouchy lately about all of the recent experiences of people not knowing how the stuff they build actually runs. You've been a great

in “The sad state of sysadmin in the age of containers (2015)” · View on HN