Great breakdown of Twitter’s transition from a lambda to a Kappa architecture! 🚀 The shift to a single real-time pipeline clearly improved latency, throughput, and aggregation accuracy while simplifying the overall architecture. It’s impressive to see the new system handle up to ~1GB/s throughput and achieve stable latency, a big leap from the limitations of the old setup.

The integration of Google Cloud services like PubSub and Dataflow seems to have provided much-needed scalability and deduplication capabilities. Also, the approach to monitoring deduplication and comparing with the old batch pipeline results adds a level of transparency that's crucial for validating the new system.