Welcome back to Data This Week!
This edition dives deep into database extensions, dynamic optimization, and performance breakthroughs. We look at new ways to run DuckDB on Azure and AWS, Pydantic’s dynamic JSON shredding for large query speedups, and the much-anticipated release of Polars 2.0. We also cover MongoDB’s disaggregated architecture and community discussion around Snowflake’s Semantic Views.
Here are the top reads, tools, and community discussions for senior data folks this week.
📚 Blogs to Read
5 Jev Use Cases for Analytics
Jev is a new type of decision model that runs structured LLM extraction and returns a typed column from a closed list (like a label or a score) fast enough to run on every row. Through MotherDuck’s prompt_jev() function, you can query these models directly in SQL to run fast classification over millions of rows for pennies. The post covers real-world pipelines: grading AI agent trace logs for wasted tool calls, entity-resolved sentiment analysis on Hacker News, and classifying job postings, all executed inside the database.
Read more →
Polars 2.0 Is Here: Streaming by Default
Polars 2.0 has officially shipped, bringing first-class SQL support and making the streaming engine and out-of-core execution (spill-to-disk) the default for LazyFrame.collect(). It also introduces a native Arrow Map dtype and reports significant performance gains over DuckDB and DataFusion on TPC-H and TPC-DS benchmarks, especially on high-core-count instances.
Read more →
Python 3.15: Lazy Imports and the Tachyon Profiler
Python 3.15 has real performance implications. The new lazy import keyword defers module loading until the module is first used, which speeds up short-lived processes such as build backends. It also adds Tachyon, a near-zero-overhead statistical profiler in the standard library, and extends typing.TypedDict with closed and extra_items parameters for stricter typing.
Read more →
When Spark Silently Stops Compiling Your Query
A deep dive into Spark SQL’s whole-stage code generation and what happens when it quietly fails. The post shows that operators like from_json, window functions, and object aggregates often drop out of codegen stages without any log message. Surprisingly, for wide projections of cheap columns, lowering spark.sql.codegen.maxFields can improve performance by bypassing the JVM’s C2 compiler inlining limits.
Read more →
Dynamic JSON Shredding in Logfire for 1000x Speedups
Logfire (by Pydantic) explains how they reached sub-second query times on semi-structured OpenTelemetry data by moving from a single JSON column to dynamic shredding. During ingestion, the system extracts frequently accessed JSON attributes into strongly typed Parquet columns. This cuts I/O, improves compression and zone map pruning, and needs no manual schema definitions. Read more →
Atlas Infinite: MongoDB’s Compute-Storage Disaggregation
MongoDB 9.0 introduces Atlas Infinite, which separates compute from storage while keeping WiredTiger’s efficiency. Unlike traditional disaggregated databases that ship physical redo logs to smart storage, MongoDB uses a dual-log approach (oplog and phylog) so storage never sees unencrypted data. The design keeps performance high through checkpoint batching and provides strong tenant isolation. Read more →
The Full pg_duckdb Extension on Azure Database for PostgreSQL
Franck Pachot shows how to run the pg_duckdb extension on Azure Database for PostgreSQL Flexible Server. After configuring shared_preload_libraries, you can query Parquet files directly from Azure Blob Storage. The post also shows DuckDB execution plans appearing inside PostgreSQL’s EXPLAIN ANALYZE, which means you can run OLAP-style queries on a managed Postgres service.
Read more →
Serverless ETL with DuckDB and S3 Tables on AWS Glue
AWS Glue 6.0 now supports running DuckDB natively on Python 3.13 with an updated glibc. Data engineers can run single-node, vectorized SQL transformations. Using DuckDB’s httpfs and iceberg extensions, you can read S3 Parquet files and commit Iceberg snapshots directly to Amazon S3 Tables. For non-distributed jobs, this is a cost-effective alternative to Spark.
Read more →
Multi-Catalog and Cross-Account Queries in Amazon EMR 8.1
EMR 8.1 introduces the RedirectingSessionCatalog, an opt-in default that detects table formats (Iceberg, Delta, Hudi, Hive) from AWS Glue metadata and hands each table to the right handler. It also allows cross-account joins without copying data: you declare named catalogs that point to remote Glue Data Catalog IDs.
Read more →
🛠️ Tools
Xonai FuseCore
What it is: A Spark 3+ accelerator enabled by a JAR plugin (com.xonai.spark.SQLPlugin). FuseCore replaces standard Spark physical plans with fused vectorized loops, with claims of up to 4x faster aggregations and a 6x faster cache serializer. It runs on open-source Spark, Databricks, and EMR clusters (Intel, AMD, and ARM) and aims to cut petabyte-scale ETL compute costs without code changes.
blob-stream
What it is: A Kafka alternative built for simple, low-cost, high-volume streaming. Built by bitdrift, blob-stream looks and feels like Kafka from the user’s side but removes much of the operational complexity and infrastructure overhead. It targets teams that need a cheaper, simpler way to handle large event volumes.
💬 Community Sentiments
Snowflake Semantic Views: Just a Rebrand or Actually Useful?
On r/dataengineering, the community is debating whether Snowflake’s Semantic Views are useful. Some see a rebrand of decades-old OLAP cube ideas, which restrict SQL to predefined pivots and aggregations. Others point to their value in the AI era: rigid, predictable SQL templates keep LLM agents (like Cortex Analyst) from inventing metric definitions.
The Community Takeaway: The feature is mature, but it depends on organizational alignment. If the business can’t agree on how “revenue” is defined, the tool won’t fix that for you.
That’s all for this week! See you in the next edition.