DuckLake and QuestDB: the same Parquet, a different table format

Last week I wrote about Parquet and Iceberg and how QuestDB cold storage can become an Iceberg table with no copy of the data. The short version: QuestDB converts older partitions to Parquet, QuestDB Enterprise tiers them to object storage, and Iceberg registers those exact files as a table by writing metadata only.

Nothing in that story is specific to Iceberg, though. QuestDB writes plain Hive-partitioned Parquet and does not bind it to any table format. Which one you put on top is a decision you make later, and can make more than once. So this post is about DuckLake, which reached 1.0 in April 2026, takes a different design decision from Iceberg on where table metadata lives, and sits over the same untouched QuestDB Parquet.

TL;DR

• A table format is metadata over a collection of Parquet files: which files exist, what the schema is, which snapshot you're reading, what statistics they carry.
• Iceberg keeps that metadata as files in object storage plus a catalog pointer. DuckLake keeps all of it in a SQL database. That single choice explains most of the differences between them.
• ducklake_add_data_files registers existing Parquet without copying it, the same zero-copy move as Iceberg's add_files.
• Because the data is still just Parquet files in your bucket, one physical copy can be an Iceberg table, a DuckLake table, and plain files you point read_parquet() at, all at the same time.

What a table format has to do over a collection of Parquet files

Strip away the branding and every table format solves the same problem. A directory of Parquet files is not a table: nothing says which files belong to it, which of them are current, what the schema is now versus last month, or how to read a consistent view while someone else is writing. A table format is the metadata that answers those questions:

• the list of data files that make up the table right now
• the schema, and how it has changed
• snapshots, so a reader sees one consistent version
• statistics, so a planner can skip files it doesn't need

Every format on the market records those four things. Where they differ is where that record lives, and that decides most of what follows.

Where the metadata lives: Iceberg manifests vs a DuckLake SQL catalog

Iceberg writes its metadata as files next to the data: a metadata JSON file per table version, a manifest list per snapshot, and Avro manifests listing data files and their statistics. A catalog holds one pointer, the current metadata file for each table, and a commit is an atomic swap of that pointer. The upside is that the metadata is as portable as the data, sitting in the same bucket, readable by anything that speaks the spec. The cost is that planning a query means reading a chain of metadata files out of object storage, and that every commit writes more of them, so you end up with maintenance jobs to expire snapshots and rewrite manifests.

DuckLake starts from the opposite end. All of the metadata, the file list, the schema history, the snapshots, the statistics, lives in a SQL database. The catalog is not a pointer to metadata files, the catalog is the metadata. DuckDB's ducklake extension supports SQLite, PostgreSQL, or DuckDB itself as the backing store.

Three things follow from that:

• Planning is a query. "Which files do I need for this predicate on this snapshot" is a SQL statement against indexed tables, not a walk through manifest files in object storage.
• There is no metadata small-files problem. Frequent commits write rows, not Avro files, so there is no manifest debt to compact away.
• The reach is narrower. Iceberg's metadata is a file format anyone can implement, which is why the whole ecosystem reads it. DuckLake's is a database schema, so a reader needs a connection to that database as well as to the bucket.

Neither is the correct answer. If you want your cold history readable by everything in a large lakehouse, Iceberg's portability is the point. If you want a table you can stand up in one command and query at speed from a laptop or a single node, carrying a Postgres or SQLite catalog is a lot less machinery.

Registering QuestDB cold storage into DuckLake

The mechanics are short. QuestDB Enterprise tiers cold partitions to your bucket in a Hive-style layout:

fx_trades/year=2026/month=06/day=19/hour=08/data.parquet

DuckLake registers those files with ducklake_add_data_files, which records references and copies nothing:

INSTALL ducklake; LOAD ducklake;
INSTALL httpfs;   LOAD httpfs;

CREATE SECRET questdb_bucket (
    TYPE s3, PROVIDER credential_chain, REGION 'eu-west-1'
);

ATTACH 'ducklake:questdb_lake.ducklake' AS lake;

-- Leave QuestDB's Parquet untouched: turn DuckLake compaction off on the
-- catalog so its maintenance never rewrites or deletes the registered files.
CALL lake.set_option('auto_compact', false);

-- No hand-written DDL: infer the schema straight from the Parquet.
CREATE TABLE lake.fx_trades AS
    SELECT * FROM read_parquet(
        's3://your-bucket/cold_storage/fx_trades~701/**/*.parquet',
        hive_partitioning = false
    ) LIMIT 0;

CALL ducklake_add_data_files(
    'lake', 'fx_trades',
    's3://your-bucket/cold_storage/fx_trades~701/**/*.parquet'
);

After that it's an ordinary table:

SELECT symbol, count(*) AS trades, avg(price) AS avg_price
FROM lake.fx_trades
WHERE timestamp >= TIMESTAMPTZ '2026-06-19 00:00:00+00'
  AND timestamp <  TIMESTAMPTZ '2026-06-20 00:00:00+00'
GROUP BY symbol
ORDER BY trades DESC;

There's no hand-written DDL here: Parquet is self describing, so DuckDB infers the column types from the files, and the example script does the same before it registers. Registration is zero-copy: the Parquet is never rewritten, and nothing is lost from storage.

A type can still read back differently. QuestDB has both microsecond and nanosecond timestamps, and DuckDB has no nanosecond-with-time-zone type, so a nanosecond column surfaces as microseconds. The nanoseconds aren't lost though: the Parquet still holds them, and an Iceberg v3 reader or pyarrow sees full precision. Only DuckDB's view is coarser, and the scripts flag which columns it affects.

Keeping the DuckLake catalog in sync with QuestDB cold storage

The point I made about Iceberg having no partition projection applies here too, for the same underlying reason. Neither format discovers files on its own: the metadata is an explicit list, and that's what buys you snapshot isolation and planning without listing object storage on every query. So when QuestDB tiers a new partition, something has to register it, and when a partition is dropped by hand or expired by a storage policy or TTL, something has to deregister it.

Incremental registration is the same shape as with Iceberg: list what is in the bucket now, diff against what the catalog already holds, add the difference. ducklake_add_data_files is not idempotent, so re-adding a file double-counts its rows; the diff is not an optimisation, it's required.

Removals are where the designs diverge. Iceberg deregisters one file with a DeleteFiles snapshot. DuckLake has no per-file deregister; the way out is to re-register the surviving set, which is cheap because it's metadata only and only happens on runs where something actually disappeared. Nothing in the bucket is touched either way: the vanished files are already gone, and you're only fixing a dangling reference.

One setting is worth knowing while you're here. Registering a file makes DuckLake treat it as its own, so DuckLake's optional maintenance, compaction or snapshot expiry followed by cleanup, is allowed to rewrite or delete it. Since these files are QuestDB's cold storage, the example script sets auto_compact = false on the catalog. In DuckLake 1.0 that skips the table even when a maintenance call names it explicitly, and the script's sync only ever drops references to partitions that already left the bucket. If you want that to hold regardless of version, give the DuckLake credentials read-only access to the bucket.

Iceberg or DuckLake for QuestDB cold storage?

Iceberg's engine reach is the long list you'd expect: Spark, Trino, Snowflake, Databricks, Flink, Athena, PyIceberg. DuckLake's is DuckDB and MotherDuck as the mature clients, with DataFusion, Spark, Trino, and pandas implementations in the ecosystem.

They're not really competing for the same slot in a QuestDB stack. Iceberg is what you register once and point the organisation at. DuckLake is what you attach when you want to run analysis over your cold history in a DuckDB session without standing anything up.

One copy of the Parquet, several table formats over it

The reason both of these are short scripts rather than migrations is that QuestDB's cold storage is plain, self describing, Hive-partitioned Parquet with no format lock-in above it. That leaves the same bytes serving several readers at once:

• QuestDB itself, spanning hot native partitions and cold Parquet in one SQL surface, at full precision
• an Iceberg table, for reach across the lakehouse
• a DuckLake table, for low-friction querying from DuckDB
• and nothing at all, if you just want read_parquet('s3://.../**/*.parquet', hive_partitioning = true)

None of those copies the data. Delta Lake, which also stores its data as Parquet, would be another metadata layer over the same files. Once your history is written in an open file format, the table format on top becomes a per-use-case decision instead of a commitment.

The DuckLake registration and sync scripts I used here live in python/ducklake, alongside the Iceberg ones.