Welcome back to Data This Week!
This week’s lineup covers a mix of architectural shifts, real-time streaming transitions, and a heavy dose of Python DataFrame evolutions. From a compelling argument on why Pandas should be retired in favor of single-machine engines, to Shopify’s brilliant approach to compressing LLM context windows, there is a lot for senior data folks to unpack. We also look at PlanetScale’s new Postgres offering and how Sony LIV handles live sports analytics at scale.
Here are the top reads, tools, and community discussions for this week.
📚 Blogs to Read
Pandas Should Go Extinct
If you are defaulting to distributed systems like Spark when Pandas runs out of memory, you might be over-engineering. An analysis of Amazon Redshift’s fleet data reveals that over 94% of tables are under 100GB—firmly in “Medium Data” territory. This post argues that Pandas’ eager, sequential execution forces premature distributed scaling. The modern answer? Single-machine, high-performance tools like Polars (with its lazy evaluation and streaming engine) or DuckDB (SQL on everything).
Delta Lake 4.3: Selective Data Replacement
Delta Lake 4.3 brings a major quality-of-life improvement for overwriting data. The new replaceUsing and replaceOn DataFrame APIs provide a much cleaner primitive for selective data replacement—allowing you to replace rows matching a user-defined condition, or update specific columns. Additionally, all managed Delta operations for metadata-changing writes are now routed natively through the Unity Catalog Delta APIs.
Introducing Neki: Sharded Postgres from PlanetScale
PlanetScale is bringing its massive MySQL sharding expertise to Postgres. Neki is a new architecture that scales Postgres across machines while keeping real Postgres on every shard—no custom storage engines, meaning full extension and SQL support. It features intelligent connection pooling via sidecars and built-in online workflows for zero-downtime schema changes, failovers, and resharding.
Apache Polaris with PyIceberg: Iceberg from Python, No Cluster
PyIceberg allows you to read and write Iceberg tables directly from Python using Arrow, bypassing the JVM and Spark entirely. This guide walks through connecting PyIceberg to Apache Polaris over the REST API using vended credentials. It highlights exactly where PyIceberg shines (metadata inspection, scripted maintenance, and filtered reads that fit in memory) and where it hits its limits (distributed joins).
Curated List: Analytics Agent Articles
If you are tasked with building AI-powered data agents, start here. This repository is a curated collection of articles detailing how various data engineering teams have architected and deployed their own analytics agents. It is a highly practical reference for teams navigating the rapidly evolving agentic AI landscape.
Gisting: Compressing LLM Agent Context
Shopify Engineering shares a fascinating technique to reduce LLM inference costs and latency. By using “gisting”—learning embeddings for a set of special tokens via knowledge distillation—they compressed their GraphQL agent’s system prompt from ~6,000 tokens to ~1,500 tokens. This 4:1 compression preserved prediction quality while dropping the time-to-first-token by 19% and allowing them to serve traffic on 14% fewer GPUs.
How Sony LIV Built Real-Time Video Streaming Analytics
Sony LIV moved away from purely batch-based processing to monitor live sports viewership at massive scale. Their architecture leverages Amazon Kinesis Data Streams for sub-second event ingestion directly into ClickHouse for low-latency dashboards, while high-volume batch events hit S3 via Firehose and are processed into Apache Iceberg tables via EMR Spark. The result is a unified catalog powering both real-time operational KPIs and historical ML workloads.
🛠️ Tools
Pyrefly 1.3.0
What it is: The latest release of Meta’s Python type checker and schema-tracking tool brings heavy improvements for modern DataFrame pipelines. The standout feature is expanded schema support for both Polars and Pandas. Pyrefly now tracks Polars schemas through common transformations with full support for typed Series and schema annotations. For Pandas, columns= projections now correctly preserve the requested schema, bringing much-needed strictness to your data transformations.
💬 Community Sentiments
dbt Development Standards: Floor or Ceiling?
A discussion over on r/dataengineering about dbt Labs’ official development standards struck a nerve. The consensus among engineers managing sprawling monorepos (2,000+ models) is that the official best practices are the floor, not the ceiling. At scale, teams are enforcing strict CI/CD checks, mandating unit tests, and moving column definitions to separate markdown files to maintain DRY principles across the pipeline. However, the sheer administrative overhead of governing thousands of models prompted others to warn: if you need an elaborate bureaucracy to keep your models under control, you might just be using too much dbt.
That’s all for this week! See you in the next edition.