One table, three readers: QuestDB, Polars and DuckDB on the same Parquet

Most trading stacks have a nightly export job somewhere. The database serves the dashboards, research wants history in a notebook, and someone writes a cron job that dumps yesterday to Parquet in a bucket. It works until the morning a batch of late prints arrives after the export ran. The dashboard and the notebook now disagree about yesterday's VWAP, and the next hour goes into working out which copy is wrong.

QuestDB can skip the copy. A table can store its partitions as Parquet, either from the moment it's created or by converting older ones, and the database keeps serving them. A single SQL query spans native and Parquet partitions, and in QuestDB Enterprise, Parquet partitions on local disk and in object storage too. The query is the same whatever format or tier each partition is in. The Parquet files themselves are standard, so anything that reads Parquet can open them too.

We built this on QuestDB open source: one table of FX trades, queried with SQL and read straight from its Parquet files with Polars and DuckDB, and all three return the same numbers. QuestDB Enterprise goes further and moves the same files to object storage with standard Hive partitioning, where Databricks, BigQuery, Trino, Snowflake and DuckDB read them in place.

Run the example with Docker and Python

The companion code is in the blog-examples repository. You need Docker and Python:

docker run --rm -d --name qdb-parquet-readers -p 9000:9000 \
  -v ./qdb_root:/var/lib/questdb questdb/questdb:10.1.0
pip install -r requirements.txt
python walkthrough.py

docker run starts QuestDB 10.1.0 with its data directory mounted on your machine at ./qdb_root. That mount is what lets Python open the files QuestDB writes. --rm removes the container when you stop it, and deleting qdb_root removes the data. The script generates 1.5 million synthetic FX trades, runs the steps below, and stops at the first check that fails.

One QuestDB table for live and historical trades

The table is an ordinary QuestDB table, partitioned by day:

CREATE TABLE trades (
  ts TIMESTAMP,
  symbol SYMBOL,
  side SYMBOL,
  price DOUBLE,
  quantity DOUBLE,
  trade_id LONG
) TIMESTAMP(ts) PARTITION BY DAY WAL
DEDUP UPSERT KEYS(ts, symbol, trade_id);

Partitioning by day matters because a partition is the unit that changes format. The deduplication key matters later, when a replayed trade arrives.

The script sends five days of trades, 300,000 a day across EURUSD, GBPUSD and USDJPY, straight from a Polars DataFrame over QWP, QuestDB's binary protocol:

import questdb

with questdb.connect("ws::addr=localhost:9000;") as db:
    db.dataframe(trades, table_name="trades", symbols=["symbol", "side"], at="ts")

dataframe() returns once the server has acknowledged the data. QuestDB applies WAL transactions asynchronously, so the script polls count() until SQL sees all 1.5 million rows before going on.

One-minute bars come from a materialized view, which QuestDB refreshes incrementally as trades arrive:

CREATE MATERIALIZED VIEW trades_ohlc_1m AS (
  SELECT ts, symbol,
         first(price) AS open, max(price) AS high,
         min(price) AS low, last(price) AS close,
         sum(quantity) AS volume
  FROM trades
  SAMPLE BY 1m
) PARTITION BY DAY;

Before converting anything, the script computes each day's trade count, volume, low, high and VWAP per symbol twice: in SQL, and in Polars from the DataFrame it sent. They match, and every later step is compared against these numbers.

Turning older partitions into Parquet

QuestDB Enterprise converts partitions on its own, with a storage policy: SET STORAGE POLICY(TO PARQUET 3d) converts each partition three days after it closes. On open source you run the conversion yourself, from a nightly cron job if you like. It's one statement, and it works on every edition:

ALTER TABLE trades CONVERT PARTITION TO PARQUET WHERE ts < '2026-10-22';

Conversion runs in the background, and SHOW PARTITIONS reports when it's done:

The same 300,000 rows take 3.5 MiB as Parquet and 11.4 MiB in native format, about a third, with the default LZ4_RAW codec. That ratio comes from our synthetic data, and yours will differ with the column mix. The active partition shows 96 MiB because QuestDB allocates space up front for incoming writes, not because its 300,000 rows take that much. Once a row for the next day arrives and QuestDB opens a new partition, the previous one shrinks to its real size: in our run, 2026-10-23 went from 96 MiB to 11.4 MiB, the same as 2026-10-22.

To skip the native format entirely, CREATE TABLE ... FORMAT PARQUET stores every partition as Parquet from the start, the active one included, on every edition. It suits tables written mostly in order, such as a backfill of history, because out-of-order writes cost more in Parquet than in native partitions. Parquet covers all three options.

Reader one: SQL across both formats

Now the table is three days of Parquet and two native days. The script runs the same daily aggregation again and gets the same numbers as before. A query across both formats is plain SQL:

SELECT symbol,
       sum(price * quantity) / sum(quantity) AS vwap,
       count() AS trades
FROM trades
WHERE ts >= '2026-10-20' AND ts < '2026-10-23'
ORDER BY symbol;

QuestDB reads each partition in whatever format it's in, so the SQL doesn't change. Every query works the same over native partitions, local Parquet, remote Parquet in QuestDB Enterprise, or a time range that spans all three.

Readers two and three: Polars and DuckDB on the partition files

Each converted partition is a directory under the table's folder, holding a data.parquet file and a small QuestDB metadata file:

qdb_root/db/trades~10/2026-10-19.5/data.parquet
qdb_root/db/trades~10/2026-10-20.3/data.parquet
qdb_root/db/trades~10/2026-10-21.4/data.parquet

~10 is QuestDB's internal table ID, and the suffix after the date is a partition version. Both vary from run to run. That's the layout on the server's own disk. Partitions that QuestDB Enterprise moves to object storage use standard Hive partitioning instead, with year=, month=, day= and hour= folders going as deep as the table's partitioning, so this daily table would stop at day=.

Polars opens the files like any other Parquet:

import polars as pl

history = pl.read_parquet("qdb_root/db/trades~*/2026-10-*/data.parquet")

So does DuckDB:

SELECT symbol, count(*) AS trades,
       sum(price * quantity) / sum(quantity) AS vwap
FROM read_parquet('qdb_root/db/trades~*/2026-10-*/data.parquet')
GROUP BY symbol;

Here's the VWAP over the three Parquet days from all three readers, with QuestDB running the same query as SQL with WHERE ts < '2026-10-22':

The script goes further, per day and symbol: counts, volumes, lows and highs match exactly, and VWAP to 12 significant digits. The table rounds to 10 decimal places. At full precision the VWAPs can differ in the last digits, because each engine adds the floating point numbers in a different order. The types come through intact. Polars reads ts as Datetime(us, UTC), symbol and side as String, and the numeric columns as Float64 and Int64.

The script also queries one Parquet day over QWP into Polars and compares it with that day's file, row for row. They're identical. The only difference is that QWP hands SYMBOL columns over as Polars Categorical, while the file gives you plain strings.

So you pick per job. Over QWP, filtering, joins and SAMPLE BY run in QuestDB and only the result crosses the wire. From the files, Polars or DuckDB scan the history themselves.

The files belong to a live table

Next, the script sends a late EURUSD trade for 2026-10-20 at 15:00, a day that is already Parquet. It arrives like any other write. SQL counts one more row, the partition stays Parquet, the 15:00 bar in the materialized view picks up the extra 7 million in volume, and the file on disk contains the trade.

The file also changed size:

before: 2026-10-20.3/data.parquet (3,711,723 bytes)
after:  2026-10-20.3/data.parquet (4,949,151 bytes)

A one-row insert added 1.2 MB, because QuestDB rewrote the affected row group at the end of the file. The superseded row group becomes dead space, which a background compaction reclaims once the partition has been idle for an hour. In another run, after a few more late writes into one day, QuestDB wrote a compacted copy into a new directory, 2026-10-21.10 in place of 2026-10-21.4, and removed the old one later.

Then the script replays a trade that's already in the table, with the same timestamp, symbol and trade ID. QuestDB commits the transaction and the row count doesn't move, because the deduplication key matched. Inserts and deduplicated upserts work on Parquet partitions as they do on native ones. UPDATE is the exception: QuestDB rejects it on a table with any Parquet partitions, even when the rows it targets are in a native one. Send corrections as rows with the same deduplication key instead.

Sometimes you do want a separate Parquet file: for a system that can't reach the server's disk, or for the result of a query rather than a whole table. COPY does both. Given a table, it writes one Parquet file per partition:

COPY trades TO 'trades_export' WITH FORMAT PARQUET;

Given a query, it writes the result as a single file, here hourly VWAP bars for the three Parquet days:

COPY (
  SELECT ts, symbol,
         sum(price * quantity) / sum(quantity) AS vwap,
         sum(quantity) AS volume
  FROM trades
  WHERE ts < '2026-10-22'
  SAMPLE BY 1h
) TO 'vwap_1h' WITH FORMAT PARQUET;

QuestDB writes both under qdb_root/export/.

Where QuestDB Enterprise takes the same files

QuestDB Enterprise adds two stages to the storage policy. They move partitions to S3, Google Cloud Storage, Azure Blob Storage or a shared NFS mount, and keep them in the table:

ALTER TABLE trades SET STORAGE POLICY(
  TO PARQUET 3d,
  TO REMOTE 7d,
  DROP LOCAL 30d
);

TO REMOTE uploads a Parquet copy of the partition while the local one keeps serving queries and accepting late writes, and a late write triggers a fresh upload. DROP LOCAL seals the partition: the local data goes, queries switch to the object store, and QuestDB fetches only the row groups and columns a query needs.

What changes is who can read the history. Cold storage uses standard Hive partitioning, the layout most lakehouse engines already understand: one directory per partition, down to day= for this daily table:

s3://my-bucket/questdb/
  trades~5/
    _manifest
    year=2026/month=10/day=20/
      data.parquet
      _pm.a

Underscore-prefixed objects are QuestDB control data, which Hive-style readers skip. data.parquet is the copy QuestDB itself serves after the seal, so there's no export to fall behind. The engines your other teams already run can read it in place:

• DuckDB and Polars read it with read_parquet and scan_parquet and Hive partitioning turned on.
• Trino, Spark and Databricks read Hive-partitioned Parquet from object storage directly.
• BigQuery and Snowflake can define external tables over it.
• To give it a table format, register the same files as an Iceberg table, as in Parquet and Iceberg, or in DuckLake, as in DuckLake and QuestDB. Neither copies the data. Iceberg is the route to catalogs like Unity Catalog, BigLake and Snowflake's Iceberg tables.

Here's the DuckDB version, with the partition columns doing the pruning:

SELECT symbol,
       sum(price * quantity) / sum(quantity) AS vwap,
       count(*) AS trades
FROM read_parquet('s3://my-bucket/questdb/trades~5/*/*/*/data.parquet',
                  hive_partitioning = true)
WHERE year = 2026 AND month = 10 AND day = 20
GROUP BY symbol;

Three rules from the cold storage documentation matter before you rely on it:

• QuestDB is the only writer. Other tools may read data.parquet, but rewriting or deleting an object, or a bucket lifecycle rule that archives it, loses data.
• The seal is final. After DROP LOCAL, writes for that partition are skipped when QuestDB applies them, not rejected when you send them. Set DROP LOCAL beyond your longest window for late data, replays and corrections.
• Remote reads are slower than local ones. Every query that reaches cold data waits on the object store. Narrow time ranges and projecting only the columns you need have a direct effect.

Try it on your own tables

Any table partitioned by time works. Convert an old partition with ALTER TABLE ... CONVERT PARTITION TO PARQUET, find its folder with SELECT directoryName FROM tables(), and open the data.parquet inside with Polars or DuckDB.