"Testing Distributed Systems w/ Deterministic Simulation" by Will Wilson

The great thing about FDB is that they have a testing infrastructure that is unlike any other database: ▶ this video If there is a Swift compiler bug that causes data loss I will bet money they will catch it (I will also bet they will find at least one Swift compiler bug)

in “Swift as C++ Successor in FoundationDB [video]” · View on HN

Tangent about FoundationDB: this is a great video that explains how the team tested it ▶ this video Spoiler: they even had their own custom power supplies used to test against power failures.

in “How FoundationDB works and why it works” · View on HN

It is a tall order, but but it's doable. The most important thing is to reduce the surface area of your application. Our business logic is almost entirely pure. The 'procs' have no side effects, except for things that happen on the other side of a clojure protocol (java interface). That means all side effects can be stubbed out during tests. Our "user" code has no access to threading libraries. Threading happens in "kernel" code. Watch ▶ this video for a good example of how this has already been done in practice.

in “Griffin - A fully-regulated, API-driven bank, with Clojure” · View on HN

There is a great talk about this testing, it goes all the way to deterministic seeds and other basics. The level of determinism is admirable. ▶ this video

in “Challenges with distributed systems” · View on HN

>>> But for the Web site for my startup, early on it was clear that I needed a good key-value store as a Web user session state store (server). So, I designed one, coded it up, and have been using it. It seems to work fine. Well depending on what error scenarios you might run into your code will work until it runs into those scenarios. Few ideas what could go wrong you can get from this video: ▶ this video

in “CrimsonDB: A Self-Designing Key-Value Store” · View on HN

I'll give you some brief thoughts from my experience working at FoundationDB, but if you really want the in-depth answer to what makes our simulation framework an enabling technology for everything we do, you should take a look at the talk my colleague Will Wilson gave at Strange Loop [0]. Everything in our core is written in Flow, our asynchronous programming environment. The architecture of our server is essentially single threaded, with (pseudo-)concurrent actors allowing a single process to satisfy multiple roles at once. The interfaces that these actors use to communicate -- with the disk, over the network, with the OS in general -- are abstracted and implemented in Flow as well. Each of these interfaces not only has at least one "real" implementation, but at least one, and sometimes more, simulated implementations. Since we run multiple actors in a single process all the time, running yet more actors, pretending to be different machines, all still in the same process, was an obvious step. These workers communicate with each other via the _same_ network interfaces that real-world workers do in real clusters. In the end we are able to become our own adversaries, pushing the limits of the system in ways that just don't happen often enough in the real world to test and debug in the wild. Our simulated implementations are allowed to present any behavior that can be observed in the real world, or justified by the spec, or implied by the contract. And they can do so much more

in “Databases at 14.4Mhz” · View on HN

> We would like to combine Jepsen’s workload generation and history checking with Antithesis’ deterministic and replayable environment to make our tests more reproducible. For those unaware, Antithesis was founded by some of the folks who worked on FoundationDB - see ▶ this video for some of their handiwork. A Jepsen + Antithesis team up is something the world needs right now, specifically on the back of the Horizon Post Office scandal. Thanks for all your work highlighting the importance of db safety Aphyr

in “Jepsen: Bufstream 0.1” · View on HN

What always fascinated me is they built the simulator for the database first(ish) and relied on it as a first class citizen while building the DB: ▶ this video > We wanted FoundationDB to survive failures of machines, networks, disks, clocks, racks, data centers, file systems, etc., so we created a simulation framework closely tied to Flow. By replacing physical interfaces with shims, replacing the main epoll-based run loop with a time-based simulation, and running multiple logical processes as concurrent Flow Actors, Simulation is able to conduct a deterministic simulation of an entire FoundationDB cluster within a single-thread! Even better, we are able to execute this simulation in a deterministic way, enabling us to reproduce problems and add instrumentation ex post facto. This incredible capability enabled us to build FoundationDB exclusively in simulation for the first 18 months and ensure exceptional fault tolerance long before it sent its first real network packet. For a database with as strong a contract as the FoundationDB, testing is crucial, and over the years we have run the equivalent of a trillion CPU-hours of simulated stress testing. [link]

in “FoundationDB: A Distributed Key-Value Store” · View on HN

AHA! I should have known that generation-reference business was intended for games all along. Reminds me of the time the FoundationDB team wrote a compiler to write a simulator to test their database: ▶ this video

in “Accidentally making a language, for an engine, for a game” · View on HN

I was watching the old 2014 video: ▶ this video I've included the timestamp of the relevant bit. He is talking about swizzling, which is like clogging++. He says they take a subset of the network connections, and have them timeout, but then bring them all back in reverse order, and that this somehow exposes far more bugs than other methods. This seems like a very interesting "well, that's funny!" moment that might lead to understanding something foundational about how distributed systems fail. Does anyone have more insight on this?

in “FoundationDB: A distributed unbundled transactional key value store” · View on HN

The FoundationDB team took a different approach, that I found pretty interesting. They created a fully deterministic (single threaded) simulation of a distributed database, and then use that simulation to test their implementation of the database under difference scenarios. They'd probably be interested in something like what you describe, as the bulk of their work seems to be rooting out and squashing bugs caused by factors out of their control (dropped connections, etc.) ▶ this video

in “Software engineering research is a train wreck” · View on HN

Neat to learn about thread sanitizer. It sounds similar to another tool from Microsoft Research called Torch ( [link] ) which automatically instruments binaries to detect data races. Coyote is similar in some ways but different in others. Coyote serializes the execution of the entire program (running one task at a time), exploring one set of interleavings and then rewinding, and then exploring another set of interleavings, hoping to hit hard-to-find safety and liveness bugs. In addition to finding concurrency bugs in one isolated process, we use it to find bugs in our distributed system by effectively running our entire distributed system in one process and having Coyote explore the various states our system can be in. It sounded mind-boggingly cool when I first came across this way of testing distributed systems through Foundation DB ( ▶ this video we're emulating this kind of testing in our distributed system through Coyote. And unlike Foundation DB which had to develop their own variant of C++ to be able to do this kind of testing (kudos to them for doing it), Coyote allows us to do it on regular C# programs written using async/await asynchrony and benefit from decades of Microsoft Research in exploring large state spaces effectively.

in “Even in Go, concurrency is still not easy” · View on HN

Their development velocity might have been due to their management or internal culture. You can get most of those results faster and on a budget. The best evidence was the startup behind FoundationDB continually improving their product in highly-competitive space while doing QA like this: ▶ this video Generally, on top of code reviews, one can get high return with minimal labor and hardware with a combo of Design-by-Contract, contract/spec/property-based testing, low-F.P. static analysis, and fuzzing with contracts in as runtime checks. That's my default recommendation. In Github's case, they also might have access to both closed and open tools that Microsoft Research makes. MSR is awesome if you've never checked them out. Two examples applicable to system reliability and security: [link] [link] Plus some of their other tools in various states of usability: [link]

in “GitHub Outage Currently Ongoing – Issues, PRs, Notifications” · View on HN

I'm very excited by this work. As they say, testing for distributed systems leaves a great deal to be desired, but there work is a great step! I was excited for FoundationDB, who put a great deal of focus on testing[1], but they were acquired into mystery by Apple. To give you an idea of how well tested they were, Aphyr of "Call Me Maybe" fame didn't test them "because their testing appears to be waaaay more rigorous than mine"[2]. FoundationDB ran Jepsen internally anyway[3]. DEMi is a really nice tool that has me excited. I've worked with similar test systems for non-blocking IO models and they're hugely useful. For non-blocking IO, you can artificially accelerate time as you can leap to the next interrupt, speeding up fuzzing substantially (see: Flow from FoundationDB in [1]). @ikneaddough, the demi-applications repo notes Spark, have you tested it so far? I quite like Spark but had some very temperamental issues at scale. [1]: ▶ this video [2]: [link] [3]: [link]

in “Fuzzing Raft for Fun and Publication” · View on HN