Tag: data-engineering

All the articles with the tag “data-engineering”.

• Data This Week #36

  1 Nov, 2026 · 7 min read

  DuckDB 2.0.0 LTS, DoorDash's agentic grocery reorder pipeline, data movement strategies, Datadog's distributed DataFusion, the 2026 Data Streaming Summit, Razorpay's Airflow 3 migration, JD.com's Fluss + Hudi Lakestream, Streamhouse, and the Python 3.10 Lambda deprecation.

• Data This Week #35

  18 Oct, 2026 · 10 min read

  Why sorted Parquet files don't give DataFusion an ordered scan, point-in-time leak-free graphs on Iceberg, rank-space Hilbert clustering in DuckDB, Netflix's ontology-driven observability, Uber's dependency analysis, Apache Polaris disaster recovery, Redshift data sharing and Iceberg writes, JEV classification, duckxlsx, and debugging data mismatches.

• Data This Week #34

  11 Oct, 2026 · 5 min read

  Jev brings typed LLM extraction into SQL, Polars 2.0 makes streaming the default, Python 3.15 adds lazy imports and Tachyon, Logfire's dynamic JSON shredding, MongoDB's Atlas Infinite, pg_duckdb on Azure, DuckDB on AWS Glue, EMR 8.1 multi-catalog queries, Xonai FuseCore, and community views on Snowflake Semantic Views.

• Data This Week #33

  4 Oct, 2026 · 6 min read

  Microsoft backs Apache Ossie with a Power BI converter and DAX integration, Cloudflare launches Basin with zero egress fees, Aurora PostgreSQL queries Iceberg natively via DuckDB, Snowflake open-sources pg_lake, streaming deletes vs. Iceberg compaction, Supabase acquires Turso, and the community debates dbt 2 maturity.

• Data This Week #30

  6 Sept, 2026 · 7 min read

  Amazon acquires DuckDB Labs, AngelList's self-updating Markdown semantic layer, managing AI coding costs at Databricks, federated MCP access for AI agents, Glue 6.0 real-time mode, Polars 2.0 RC, and community advice on extracting from legacy on-prem databases.

• Data This Week #29

  23 Aug, 2026 · 5 min read

  DuckDB v2.0's new PEG-based SQL parser, Cassandra's ACCORD consensus protocol for ACID transactions, migrating multilingual full-text search to PostgreSQL, stream deduplication techniques, the hidden joins inside Snowflake MERGE, and the community debate on enterprise data catalog alternatives.

• Data This Week #28

  16 Aug, 2026 · 5 min read

  Instacart's Postgres-native search replacing Elasticsearch and FAISS, Netflix migrating batch workloads to Kueue on Kubernetes, AWS Dogwood for AI agent governance, distributed DuckDB with Quack, Debezium for CDC-driven event architecture, and the community debate on dbt Cloud pricing.

• Data This Week #27

  9 Aug, 2026 · 6 min read

  Safely migrating Iceberg catalogs without moving data, DuckDB's MySQL engine stress-tested at 500 GB, the end of the Hive Metastore era, hands-on S3 Tables evaluation, why dbt tests don't equal data trust, and an ontology+LLM approach to legacy data modernization.