Welcome back to Data This Week!
This week’s lineup covers major milestones in query engine design, large-scale orchestrator migrations, and unified streaming-lakehouse storage. From the release of DuckDB 2.0 LTS and Datadog’s zero-copy distributed DataFusion framework to battle-tested engineering deep dives on moving 2,500 DAGs to Airflow 3 at Razorpay and unifying Kafka with Hudi via Apache Fluss at JD.com, the focus is heavily on architecture at scale. We also look at DoorDash’s agentic grocery reorder pipeline and the ongoing Python 3.10 runtime deprecation on AWS Lambda.
Here are the top reads, tools, and community discussions for senior data folks this week.
đź“š Blogs to Read
Announcing DuckDB 2.0.0 LTS “Cinnamon Teal”
While DuckDB 1.0 was all about stability, DuckDB 2.0.0 LTS is a massive feature and architectural release supported until at least October 2027. Standout additions include a polished Quack server protocol with a single-command CONNECT statement for remote databases, asynchronous I/O, first-class support for the VARIANT type, SQL triggers, and a brand-new PEG-based SQL parser replacing the legacy YACC/Bison implementation. Under the hood, it ships a revamped C API v2 exposing file systems and table functions to extensions, drops the ICU dependency for timezones and collations, and upgrades the build baseline to C++17 with Clang.
Building Ask DoorDash (Part 6): A Grocery Reorder Case Study
Reordering accounts for roughly one in three DoorDash orders, yet simply replaying order history fails because buying history doesn’t equal immediate need (e.g., weekly milk vs. a bulk pack of paper towels). DoorDash engineering walks through the 5-stage architecture behind their agentic grocery reorder flow: establishing store context, reshaping history for model reasoning, distinguishing recurring staples from one-off purchases, resolving live catalog variants, and pre-computing shoppable bundles asynchronously. By carefully drawing the boundary between ahead-of-time inference, request-time reasoning, and deterministic filtering, they cut latency from ~16 seconds to ~2 seconds and drove a ~34% increase in order rate for the flow.
Data Movement Strategies In System Design
Connecting distributed components without creating bottlenecks requires picking the right data movement paradigm for the workload. This piece breaks down the core mechanics and trade-offs across the spectrum of data movement: pull vs. push architectures, short and long polling, continuous streaming via Server-Sent Events (SSE) and WebSockets, asynchronous decoupling with message queues, and database-level Change Data Capture (CDC). A solid architectural refresher for evaluating latency, throughput, and operational coupling when designing distributed pipelines.
How Datadog Built Distributed DataFusion
As Datadog consolidated its fragmented, workload-specific query systems onto an open standard stack—Apache Arrow for in-memory data, Substrait for execution plans, and Apache DataFusion—they hit single-node compute ceilings on heavy interactive scans. Because Spark and Ballista rely on intermediate materialization and Trino proved costly to integrate with Arrow/Substrait, Datadog built and open-sourced Distributed DataFusion in Rust. It extends DataFusion with zero-copy streaming execution across multiple nodes, keeping lightweight queries on a single machine to avoid distribution overhead while accelerating heavy production queries by up to 10x at the same total cost.
Data Streaming Summit 2026: Diskless Kafka, Streamhouse, and Declarative Pipelines
StreamNative’s Data Streaming Summit in San Francisco made one theme hard to miss: streaming and lakehouse storage are collapsing into a single architecture. Their Ursa project puts object storage on the primary data path instead of local disks, then piggybacks on background compaction to push data into external systems—including Kafka Connect sinks, turning the classic Source → Kafka → Sink chain into Source → Object Storage → Sink. The Streamhouse working group pitched the same convergence as a vendor-neutral category for keeping business state continuously available to applications and agents. On the engine side, Goldsky’s Streamling (a general-purpose streaming engine built on Apache DataFusion) and Airbnb’s declarative YAML pipelines—one definition, executed as batch or streaming on Flink’s Table API—showed how far “define once, run anywhere” has come. Also worth noting: Oxia as Pulsar’s ZooKeeper replacement, and SQL notebooks in StreamNative Cloud powered by RisingWave.
How Razorpay Moved 2,500 DAGs to Airflow 3, One DAG at a Time
Razorpay recently moved 2,500 DAGs from Airflow 2.10 on Celery to Airflow 3.3 on the Kubernetes executor, a fleet that now runs 2,753 DAGs, ~11,200 task runs a day, 261 concurrent task pods at the nightly peak, and ~1,600 managed-Spark clusters a day. The biggest architectural shift in Airflow 3 is that tasks no longer connect directly to the metadata DB; all heartbeats, state changes, and XComs flow through the API server via the Task Execution API—putting the API server on the critical path of every running task. Razorpay details why they avoided an in-place upgrade in favor of a blue-green cluster setup, how Ruff’s AIR301/AIR302 rules and internal wrapper libraries turned migrations into simple import swaps, and the production gotchas they hit along the way (from dag-processor memory budgets to scheduler crash loops in 3.2.1).
How JD.com Built a Lakestream with Apache Fluss and Apache Hudi
Running an 8,000-node Kafka cluster alongside a 700+ PB Apache Hudi lakehouse created massive duplication for JD.com’s recommendation ranking pipelines—including row-based Kafka consumers pulling 195 GB/min just to extract 6 GB/min of selective columns. To solve this, JD adopted Apache Fluss to build a unified “Lakestream” architecture, contributing a native Hudi tiering plugin (FIP-24), server-side consumer offset management, and ZooKeeper-backed Coordinator HA. By streaming real-time features into columnar Fluss tables and continuously tiering them into Hudi with union reads, JD merged three separate training pipelines into one—slashing batch compute costs by over 80%, storage costs by over 50%, and improving training data freshness by over 30%.
🛠️ Tools
Streamhouse
What it is: An open, vendor-neutral architecture category—coined by Ververica and maintained by a working group of Aiven, Confluent, Redpanda, StreamNative, and Ververica—for data architectures that keep the current state of a business continuously available to production applications, analytics, and AI agents. Instead of stitching together isolated message brokers, batch refreshes, and duplicated ingestion pipelines, a Streamhouse treats capture (CDC, event streams), transport, stream processing, open table formats, catalogs, and low-latency serving as one production-native path: real-time, decentralized, and meeting data where it already lives rather than consolidating it first.
đź’¬ Community Sentiments
AWS Lambda Deprecates Python 3.10: Gotchas Moving to 3.12+
Over on r/Python, engineers are trading war stories as AWS officially deprecates the python3.10 Lambda runtime as of October 31, 2026—ending security patches and console edits now, before blocking function creation (July 29, 2027) and updates (August 31, 2027). Because the python3.11 runtime is itself deprecated on June 30, 2027, most teams are jumping straight to 3.12 or 3.13—which triggers an underlying OS jump from Amazon Linux 2 to Amazon Linux 2023. The community highlighted several silent traps to watch out for during the upgrade:
- AL2023 & Native Binaries: Compiled C-extensions and Lambda layers (
psycopg2,numpy,cryptography,pydantic-corein Pydantic v2) must be rebuilt against AL2023, and olderurllib3 1.xpins break against AL2023’s OpenSSL 3. - Removed Standard Libraries:
distutilsis gone in 3.12, and PEP 594 removescgi,crypt, andtelnetlibin 3.13. - Behavioral Changes: Python 3.11+ altered how
(str, Enum)mixins format in f-strings (emitting"Status.ACTIVE"instead of"active"unless switched toStrEnum), anddatetime.utcnow()deprecation warnings in 3.12 will break test suites configured to fail on warnings.
That’s all for this week! See you in the next edition.