Parquet

QuestDB works with Apache Parquet in several ways:

  • A table can be created directly in Parquet format, so its partitions are stored as Parquet from the start.
  • Existing partitions can be converted in place, automatically with a storage policy (QuestDB Enterprise) or manually with ALTER TABLE, while the data stays managed by QuestDB.
  • Query results and whole tables can be exported as external Parquet files via the REST /exp endpoint or COPY.

Parquet partitions managed by QuestDB remain fully queryable with SQL, exactly like native partitions. To read external Parquet files, see the read_parquet function.

Create a table as Parquet​

A partitioned WAL table can store its partitions as Parquet by default with the FORMAT PARQUET clause, rather than converting them later:

Create a table with Parquet partitions
CREATE TABLE trades (
  timestamp TIMESTAMP,
  symbol SYMBOL,
  price DOUBLE,
  amount DOUBLE
) TIMESTAMP(timestamp)
PARTITION BY DAY
FORMAT PARQUET
WAL;

See CREATE TABLE — Partition format for the clause, and ALTER TABLE SET FORMAT to switch an existing table's format. NATIVE is the default, and FORMAT PARQUET applies to partitions written after it is set; it does not convert existing partitions.

In-place conversion​

In-place conversion turns existing partitions from native format into Parquet (or back) while they remain under QuestDB's control and can still be queried as if they were in native format.

Parquet partitions behave like native partitions for most operations:

  • Inserts are supported, with and without dedup. When new or out-of-order rows land on a Parquet partition, the partition is rewritten to include them.
  • Column type changes via ALTER TABLE ALTER COLUMN TYPE are supported. On Parquet partitions the change is applied lazily; see that page for details.
  • TTL applies to Parquet partitions, so expired partitions are dropped as usual.

Partitions can be converted automatically on a schedule (QuestDB Enterprise) or manually with ALTER TABLE (all editions).

Automatic conversion with storage policies​

In QuestDB Enterprise, conversion is fully automated with storage policies. A policy converts partitions to Parquet on a schedule as they age, and can also drop the native files afterwards.

A policy can be attached when creating the table with the STORAGE POLICY clause on CREATE TABLE, or set later with ALTER TABLE SET STORAGE POLICY. For example, convert a partition to Parquet one hour after it stops being the active partition:

Convert to Parquet after 1 hour (Enterprise)
ALTER TABLE market_data SET STORAGE POLICY(TO PARQUET 1h);

The TTL unit cannot be finer than the table's partition size (market_data is partitioned by hour). See the Storage Policy concept guide for the full partition lifecycle, including DROP LOCAL.

Converting to Parquet​

On all editions, partitions can be converted manually with ALTER TABLE. You need to pass a filter specifying the partitions to convert. The filter can be either a WHERE or a LIST, in the same way it is used for the DETACH command.

tip

The active (most recent) partition will never be converted into Parquet, even if it matches the filter.

Conversion is asynchronous, and can take a while to finish, depending on the number of partitions, on the partition size, on the compression being used, and on disk performance and general load of the server.

To monitor how the conversion is going, you can issue a SHOW PARTITIONS command. Partitions in the Parquet format will have the isParquet column set to true and will show the size on the parquetFileSize column.

ALTER TABLE trades CONVERT PARTITION TO PARQUET WHERE timestamp < '2025-08-31';

Bloom filters for in-place conversion​

Bloom filters enable row group pruning for equality and IN queries on Parquet partitions. There are two ways to generate them during in-place conversion.

Per-column metadata — If a column was defined with the BLOOM_FILTER keyword in its PARQUET() clause, bloom filters are generated automatically during conversion. No additional options are needed:

Columns with BLOOM_FILTER metadata are indexed automatically
ALTER TABLE trades CONVERT PARTITION TO PARQUET WHERE timestamp < '2025-08-31';

Explicit column list — You can specify which columns to index and optionally set the false positive probability (FPP) using WITH:

Convert with explicit bloom filter columns
ALTER TABLE trades CONVERT PARTITION TO PARQUET
WHERE timestamp < '2025-08-31'
WITH (bloom_filter_columns = 'symbol,side', bloom_filter_fpp = 0.01);
note

When an explicit bloom_filter_columns list is provided, it overrides any per-column PARQUET(BLOOM_FILTER) metadata on the table. If the option is omitted, per-column metadata is used.

Converting to Native​

ALTER TABLE trades CONVERT PARTITION TO NATIVE WHERE timestamp < '2025-08-31';

Data Compression​

By default, Parquet files generated by QuestDB are compressed using lz4_raw compression. One of the key advantages of Parquet over QuestDB's native format is its built-in compression.

There are two separate configuration properties in server.conf, one for exports (REST and COPY) and one for in-place conversion:

# Export (REST /exp and COPY TO)
# Supported codecs: UNCOMPRESSED, SNAPPY, GZIP, BROTLI, ZSTD, LZ4_RAW
cairo.parquet.export.compression.codec=LZ4_RAW
cairo.parquet.export.compression.level=0

# In-place conversion (ALTER TABLE CONVERT PARTITION TO PARQUET)
# Supported codecs: UNCOMPRESSED, SNAPPY, GZIP, BROTLI, ZSTD, LZ4_RAW
cairo.partition.encoder.parquet.compression.codec=LZ4_RAW
cairo.partition.encoder.parquet.compression.level=0

When using ZSTD, the level ranges from 1 (fastest) to 22, with a default of 9.

For COPY exports, you can also override compression per-query. See Overriding compression.

Minimum compression ratio​

The cairo.partition.encoder.parquet.min.compression.ratio property controls whether compressed Parquet pages are worth keeping. After compressing a page, QuestDB checks the ratio of uncompressed_size / compressed_size. If the ratio falls below the threshold, the compressed output is discarded and the page is stored uncompressed instead.

# Default: 1.2 (keep compressed output only if it achieves ~17% size reduction)
cairo.partition.encoder.parquet.min.compression.ratio=1.2

A value of 0.0 (or any value <= 1.0) disables the check, always keeping compressed output.

The ratio check applies to both data pages and dictionary pages and works with all compression codecs. It runs after compression, so the CPU cost of compression is still incurred -- this setting only avoids the I/O and storage penalty of keeping pages that barely compress.

Per-column overrides​

Individual columns can override the global encoding and compression settings. See CREATE TABLE - Per-column Parquet encoding, compression, and bloom filters for defining overrides at table creation, or ALTER TABLE ALTER COLUMN SET PARQUET for modifying existing tables.

Bloom Filters​

Bloom filters are opt-in probabilistic indexes that enable row group pruning for equality and IN queries. When generated, they are embedded in the Parquet file metadata alongside min/max statistics.

Bloom filters can be enabled per-column via the BLOOM_FILTER keyword in CREATE TABLE or ALTER TABLE, or per-export via bloom_filter_columns in CONVERT PARTITION, COPY TO, and the REST /exp endpoint.

The false positive probability (FPP) determines the trade-off between filter size and accuracy. It is configured globally:

# In-place conversion (ALTER TABLE CONVERT PARTITION TO PARQUET)
cairo.partition.encoder.parquet.bloom.filter.fpp=0.01

# Export (REST /exp and COPY TO)
cairo.parquet.export.bloom.filter.fpp=0.01

See the Configuration reference for all Parquet-related settings.

Limitations​

  • UPDATE statements are not supported on Parquet partitions. Overwriting rows through a dedup upsert is supported, since it is an insert.
  • While a partition is being converted, writes to that partition are briefly blocked.
  • Some parallel queries are not yet optimized for Parquet.

Export​

Query results and whole tables can be exported as external Parquet files. Unlike in-place conversion, the exported files are written outside QuestDB (streamed to the client or written to the server filesystem) for use in other tools such as DuckDB, Pandas, or Polars.

Export via REST​

The /exp REST API endpoint executes a query and streams the result as a Parquet file directly to the client. This is a synchronous operation — the HTTP response completes when the file is fully transferred.

tip

See also the /exp documentation.

You can use the same parameters as when doing a CSV export, but passing parquet as the fmt parameter value.

curl -G \
  --data-urlencode "query=select * from market_data limit 3;" \
  'http://localhost:9000/exp?fmt=parquet' > ~/tmp/exp.parquet
note

For larger queries you might prefer to use the COPY method, which runs asynchronously and writes files to the server filesystem.

Once exported, you can use it from anywhere, including DuckDB, Pandas, or Polars. If you wanted to point DuckDB to the example file exported in the previous example, you could start DuckDB and execute:

select * from read_parquet('~/tmp/exp.parquet');

Export via COPY​

The COPY command writes Parquet files to the server filesystem. Unlike REST export, this is an asynchronous operation — the command returns immediately and the export runs in the background.

tip

See also the COPY-TO documentation.

You can use the COPY command from the web console, from any pgwire-compliant client, or using the exec endpoint of the REST API.

You can export a query:

COPY (select * from market_data limit 3) TO 'market_data_parquet_table' WITH FORMAT PARQUET;

Or you can export a whole table:

COPY market_data TO 'market_data_parquet_table' WITH FORMAT PARQUET;

The output files (one per partition) will be under $QUESTDB_ROOT_FOLDER/export/$TO_TABLE_NAME/.

The COPY command will return immediately, but the export happens in the background. The command will return an export id string:

id
45ba24e5ba338099

If you want to monitor the export process, you can issue a call like this:

SELECT * FROM 'sys.copy_export_log' WHERE id = '45ba24e5ba338099';

While it is running, export can be cancelled with:

COPY '45ba24e5ba338099' CANCEL;

Controlling partitioning​

COPY table_name TO ... produces one Parquet file per partition, matching the table's own partitioning scheme. COPY (SELECT ...) TO ... produces a single file by default.

To override either default, add PARTITION_BY to the export options.

Export a table into a single consolidated file:

COPY market_data TO 'market_data_single' WITH FORMAT PARQUET PARTITION_BY NONE;

Re-partition independently of the source table. For example, export a day-partitioned table into monthly files:

COPY market_data TO 'market_data_monthly' WITH FORMAT PARQUET PARTITION_BY MONTH;

Partition a query export by month:

COPY (SELECT * FROM market_data WHERE timestamp IN '2024')
TO 'market_data_2024'
WITH FORMAT PARQUET PARTITION_BY MONTH;

Partitioning requires a designated timestamp column in the source table or query result. Valid values: NONE, HOUR, DAY, WEEK, MONTH, YEAR.

For the full list of export options, see the COPY-TO documentation.

Overriding compression​

By default, exported Parquet files use lz4_raw compression. You can change the default via server.conf as shown in Data Compression, or override the compression individually for each export. For example:

COPY market_data TO 'market_data_parquet_table' WITH FORMAT PARQUET COMPRESSION_CODEC LZ4_RAW;