Zach Tellman - Everything Will Flow

Yeah this is true, typically you want to keep blocking functions outside this. By default, core.async allocates as many threads as cores in your CPU. In addition to this, I personally find core.async to deal poorly with flow control (e.g. slow subscribers slowing down the entire flow for every subscriber), and seems to not have a lot of telemetry / it’s difficult to find out what’s going on in the thread pool behind it. I’ve personally settled on Zach Tellman’s Manifold library, which provides a much more sane abstraction and is fully compatible with core.async. For some addition thought and material on these topics, I highly recommend watching this talk by Zach, “Everything Will Flow” ▶ this video

in “Fearless Concurrency: Clojure, Rust, Pony, Erlang and Dart” · View on HN

No. Check out this video by Zach Tellman. He talks about queues and how they break down under load. One of the least intuitive things he points out is that when you have more processors, the breakdown tends to be more of a step function: everything is running smoothly till the moment that it isn't. ▶ this video The point he makes arises from basic queue theory and is applicable to all kinds of systems, and how those systems react to load. It's got little to due with particular hardware and everything to do with basic math.

in “Linux Load Averages: Solving the Mystery” · View on HN

On dealing with backpressure, I would recommend what Zach Tellman writes about in his video "Everything will flow": ▶ this video Tellman also tells his own adventure about a system something like this: "a system that would latch up into an overload state because in overload it would log errors at a severity high enough to force the log file to be flushed on every line" He talks about some problems he faced when writing to Amazon S3, when S3 stopped accepting requests. It's a fascinating talk.

in “Notes on Distributed Systems for Young Bloods (2013)” · View on HN

Anything by Zach Tellman: [link] [link] In terms of his ideas, watch "Always be composing": [link] And he offers some great thoughts about queues and backpressure in "Everything will flow": ▶ this video

in “Ask HN: What are some examples of beautiful software?” · View on HN

See also a very nice video from ClojureWest about queues in system architectures: ▶ this video The queue are everywhere - your messaging queue, the threadpool, the hardware threads, and other layers of the stack and APIs you use. The video adds the interesting detail that as you add more tellers (workers) you learn of impending disaster only in the outlier p99 (or higher) latencies; by the time your p85 latency rises, you're already about to stall out.

in “What happens when you add a new teller? (2008)” · View on HN

Queue theory suggests the more nodes you add to handle your queue then the longer your queue can survive (facing rising traffic) but the faster it will collapse when it finally runs out of capacity. Zach Tellman went over this in one of his talks and he had a nice graphic to demonstrate what this looks like: ▶ this video I suppose any complex system has dynamics like that, more capacity to absorb load, but then a faster collapse when capacity is surpassed. Hemmingway's Laws Of Motion applies: ”How did you go bankrupt?” Bill asked. ”Two ways,” Mike said. ”Gradually and then suddenly.” Put a bit differently by the economist Rudiger Dornbusch: ”The crisis takes a much longer time coming than you think, and then it happens much faster than you would have thought.” lifted from: [link]

in “Complex systems collapse faster” · View on HN