a_wild_dandanMar 2024Strongly recommends
To those curious about the tradeoffs between transformer and state space model layers, I highly recommend Sasha Rush's video on it: ▶ this video
in “Jamba: Production-grade Mamba-based AI model” · View on HN
Do we need Attention? - Linear RNNs and State Space Models (SSMs) for NLP
Do we need Attention? - Linear RNNs and State Space Models (SSMs) for NLP on HackerTube: recommended by 3 Hacker News commenters across 3 separate months, 2024.
Sasha Rush
Open on YouTubelecture · advanced · dated
Sasha Rush surveys linear RNNs and state space models as replacements for attention in transformer-style models, in a talk for MLSys. It covers the foundational preliminaries and predates Mamba.
Why Hacker News recommends it
HN commenters recommend it for building intuition about state space models and the tradeoffs between transformer and SSM layers.
- people recommended it
- 3
- separate months
- 3
- strong recommendations
- 1
- discussed on HN
- 2024
“I highly recommend Sasha Rush's video on it”
What HN commenters said
3 Hacker News comments link this video. Each one links back to its thread.
srushFeb 2024Recommends
Hi! Blog author. This was an attempt a couple years ago to understand and write about this paper in a detailed way. Here is a video going through this topic as well: ▶ this video Section 2 of the blog post is no longer very relevant. A lot of advances (DSS, S4D) simplified that part of the process. Arguably also this all should be updated for Mamba (same authors).
in “The Annotated S4” · View on HN
anomaly72Jan 2024Recommends
Would recommend Sasha Rush's lecture ▶ this video , it provides the intuition for state space models via linear RNNs
in “MoE-Mamba: Efficient Selective State Space Models with Mixture of Experts” · View on HN