15 Hacker News comments link this video. Each one links back to its thread.
chamomealAug 2025Strongly recommends
Idk I think caching is a crucial part of many well-designed systems. There’s a lot of very cache-able data out there. If invalidating events are well defined or the data is fine being stale (week/month level dashboards, for example), that’s a fantastic reason to use a cache. I’d much rather just stuff those values in a cache than figure out any other more complicated solution. I also just think it’s a necessary evil of big systems. Sometimes you need derived data. You can even think about databases as a kind of cache: the “real” data is the stream of every event that ever updated data in the database! (Yes this stretching the meaning of cache lol) However I agree that caching is often an easy bandaid for a bad architecture. This talk on Apache Samza completely changed how I think about caching and derived data in general: ▶ this video And this interview has some interesting insights on the problems that caching faces at super large scale systems (twitter specifically): [link]
in “Replacing a cache service with a database” · View on HN
chamomealAug 2025Strongly recommends
Hey OP you may have seen this already, but in case you didn’t see my other comment, you should definitely check out this talk by Martin Kleppman. ▶ this video It details Apache samza, which I didn’t totally grasp but it seems similar to what you’re talking about here. He talks about how if you could essentially use an event stream as your source of truth instead of a database, and you had a sufficiently powerful stream processor, you could define views on that data by consuming stream events. The end result is kind of like an auto-updating cache with no invalidation issues or race conditions. Need a new view on the data? Just define it and run the entire event stream through it. Once the stream is processed, that source of data is perpetually accurate and up-to-date. I’m not a database guy and most of this stuff is over my head, but I loved this talk and I think you should check it out! It’s the first thing I thought of when I read your post.
in “Replacing a cache service with a database” · View on HN
btownMay 2019Strongly recommends
The talk on "Turning the database inside-out" [0][1] by the author, Martin Kleppmann, is a fantastic intro to these dynamics, and it's something I'll always recommend to both experienced and inexperienced data modelers and backend developers. It goes pedagogically through the way things are typically done in a relational database in such a clear way that word-for-word it's one of the best tutorials I've seen... but it also weaves a narrative of "how can this be done better/more scalably/more reliably/more flexibly-to-business-needs" in pointing to a streaming/event-sourcing architecture. You may or not need the latter right away, but it's a fantastic tool to have in your toolbox to be able to say "ah, this new requirement feels like it would benefit hugely from this architecture." Especially for OP who's starting to think about the "why" of messaging queues, this could be a fantastically valuable first step. [0] ▶ this video [1] [link]
in “Ask HN: What overlooked class of tools should a self-taught programmer look into” · View on HN
lbradstreetMay 2018Strongly recommends
Yes, I like to think of it as as the transaction log without any of the logic. If you’re interested, I can highly recommend Martin Kleppmann’s talk “Turning the database inside out” ▶ this video
in “GCP and Confluent partner to deliver a managed Apache Kafka service” · View on HN
btownApr 2015Strongly recommends
Since this is taking off a bit: Here's an excellent talk that provides some context to the Confluent article, and takes us through the thought process of simplifying convoluted database-cache-user interactions by thinking in terms of streams of derived data: ▶ this video . Also highly recommended.
in “How Sift Science Rebuilt Its App on React and DropWizard” · View on HN
avinasshApr 2023Recommends
I think you would love this talk by Martin Kleppmann - Turning the database inside out with Apache Samza: ▶ this video He basically explores similar idea, you have a log to persist and materialised views to query.
in “Ask HN: Append-Only SQL Schema?” · View on HN
Show 9 more comments
dmitryminkovskySep 2021Recommends
I second this wholeheartedly because this happened to me. I'm the kind of person who learns by my making my own mistakes (as opposed to learning from others', which is what smart people do)—and boy, did I learn a lot from the mistake of trying to build a general purpose application with event sourcing. Theoretically, event sourcing is the way all applications should be built. I was first taken by this methodology by Martin Kleppmann's talk "Turning the Database Inside Out" (2014)[0]. Looking back, I think I had always subconsciously felt like there was something wrong with mutating state in place. You lose information. You lose accountability. After all, "accountants don't use erasers," and Martin's talk validated many of my obsessive-compulsive insecurities about data. As Charity Majors points out on Twitter [1]: > ...while storing raw events may be "expensive" compared to any single optimization, all optimizations can be derived from raw events. The reverse is impossible. And that's true. The table-stream duality [2][3] means that if you store events, you can later project those events into the exact perfect table for any use-case. But if you transform those events into a projection and store that projection, well, whatever table you store is data you have. You wind up having just an aggregate. Unfortunately, the technical costs you incur when you attempt to store events instead of tabular data are tremendous . The problems and questions that arise are basically endless. To
in “The Reactive Monolith – How to Move from CRUD to Event Sourcing” · View on HN
je42Dec 2020Recommends
Indeed. Known the tools and apply the one that has least cons and most pros ;). Also, a good starter to for knowing how to use a Streaming Store like Kafka as DB is the video Database inside-out. ▶ this video
in “Kafka Is Not a Database” · View on HN
z1mm32m4nJun 2016Recommends
Hearing the way Alan Kay talks about "messaging" always reminds me of two technologies: the Flux/Redux architecture and Apache Samza & Kafka. > The key in making great and growable systems is much more to design how its modules communicate rather than what their internal properties and behaviors should be. With React and Flux, this is exactly what's going on! You're defining view components, and when someone interacts with them, there's a very structured, discrete set of actions that can be fired. These actions are then dispatched and handled elsewhere, but somewhere at the core, there's a single source of truth for what happens upon any given message. And for Apache Samza, I think this video[1] does a nice job of talking about stream processing as a programming architecture. There's a connection to be made between processing streams of information and processing messages between modules, and that video goes a good way towards exposing it. [1]: ▶ this video "Turning the database inside out with Apache Samza"
in “Alan Kay on the misunderstanding of OOP (1998)” · View on HN
pramodliv1Apr 2016Recommends
If you want a more modern and concrete example, the "Turning your database inside out" talk by Martin Kleppmann is pretty good too: ▶ this video
in “GraphQL-enabled event sourcing engine” · View on HN
felipesabinoApr 2015Recommends
Samza's author really opened my mind related to how useful stream process is to high performance data processing [1] and I think Samza's only bummer (for me, personally, today) is it's lack of support for non-JMV languages [2] [1] ▶ this video [2] [link]
in “Real-Time Full-Text Search with Luwak and Samza” · View on HN
ZenoArrowNov 2016Cites it
Previous discussion: [link] IIRC this works closely with Samza. There's a decent video by the author knocking around somewhere. EDIT: This is the video I was thinking of: ▶ this video
in “Bottled Water – Real-Time Integration of PostgreSQL and Kafka” · View on HN
ZenoArrowMay 2016Cites it
> "The trouble with this is that the database is the hardest part to scale." Probably depends on the plan followed for data replication, as some approaches seem to make it easier to add new DB nodes as you scale. I've not tried it myself, but I think Apache Samza has some promising ideas in this area: ▶ this video
in “Andl, a relational language that is not SQL, is coming to Postgres” · View on HN
rapindJan 2016Cites it
The idea of changefeeds / subscription cursors has been around for a while now, especially in Meteor. I doubt it'll be that crazy to add reactivity to graphql queries when compared to what Meteor is already doing w/ subscriptions and DDP. RethinkDB for example has changefeeds built in and is tackling this problem as well. See: [link] Data as a stream is pretty interesting too: ▶ this video
in “Reactive GraphQL” · View on HN
atrilumenJul 2015Cites it
Immutability is useful everywhere in the stack, even the database. See "Turning the database inside out with Apache Samza" by Martin Kleppmann: ▶ this video
in “Mostly Adequate Guide to Functional Programming in JavaScript” · View on HN