# Parquet

How QuestDB stores and exports data in Apache Parquet format. Covers creating tables directly as Parquet, converting partitions in place manually or automatically, and exporting query results as external Parquet files.

QuestDB works with [Apache Parquet](https://parquet.apache.org/) in several ways:

- A table can be **created directly in Parquet format**, so its partitions are
  stored as Parquet from the start.
- Existing partitions can be **converted in place**, automatically with a storage
  policy (QuestDB Enterprise) or manually with `ALTER TABLE`, while the data stays
  managed by QuestDB.
- Query results and whole tables can be **exported** as external Parquet files
  via the REST `/exp` endpoint or `COPY`.

Parquet partitions managed by QuestDB remain fully queryable with SQL, exactly
like native partitions. To read external Parquet files, see the
[`read_parquet` function](/docs/query/functions/parquet/).

## Create a table as Parquet

A partitioned [WAL](/docs/concepts/write-ahead-log/) table can store its
partitions as Parquet by default with the `FORMAT PARQUET` clause, rather than
converting them later:

```questdb-sql title="Create a table with Parquet partitions"
CREATE TABLE trades (
  timestamp TIMESTAMP,
  symbol SYMBOL,
  price DOUBLE,
  amount DOUBLE
) TIMESTAMP(timestamp)
PARTITION BY DAY
FORMAT PARQUET
WAL;
```

See [CREATE TABLE — Partition format](/docs/query/sql/create-table/#partition-format)
for the clause, and
[ALTER TABLE SET FORMAT](/docs/query/sql/alter-table-set-format/) to switch an
existing table's format. `NATIVE` is the default, and `FORMAT PARQUET` applies to
partitions written after it is set; it does not convert existing partitions.

## In-place conversion

In-place conversion turns existing partitions from native format into Parquet
(or back) while they remain under QuestDB's control and can still be queried as
if they were in native format.

Parquet partitions behave like native partitions for most operations:

* Inserts are supported, with and without dedup. When new or out-of-order rows
  land on a Parquet partition, the partition is rewritten to include them.
* Column type changes via
  [`ALTER TABLE ALTER COLUMN TYPE`](/docs/query/sql/alter-table-change-column-type/)
  are supported. On Parquet partitions the change is applied lazily; see that
  page for details.
* TTL applies to Parquet partitions, so expired partitions are dropped as usual.

Partitions can be converted automatically on a schedule (QuestDB Enterprise) or
manually with `ALTER TABLE` (all editions).

### Automatic conversion with storage policies

In QuestDB Enterprise, conversion is fully automated with
[storage policies](/docs/concepts/storage-policy/). A policy converts partitions
to Parquet on a schedule as they age, and can also drop the native files
afterwards.

A policy can be attached when creating the table with the `STORAGE POLICY`
clause on [`CREATE TABLE`](/docs/query/sql/create-table/#storage-policy), or set
later with
[`ALTER TABLE SET STORAGE POLICY`](/docs/query/sql/alter-table-set-storage-policy/).
For example, convert a partition to Parquet one hour after it stops being the
active partition:

```questdb-sql title="Convert to Parquet after 1 hour (Enterprise)"
ALTER TABLE market_data SET STORAGE POLICY(TO PARQUET 1h);
```

The TTL unit cannot be finer than the table's partition size (`market_data` is
partitioned by hour). See the [Storage Policy](/docs/concepts/storage-policy/)
concept guide for the full partition lifecycle, including `DROP LOCAL`.

### Converting to Parquet

On all editions, partitions can be converted manually with `ALTER TABLE`. You
need to pass a filter specifying the partitions to convert. The filter can be
either a `WHERE` or a `LIST`, in the same way it is used for the
[`DETACH` command](/docs/query/sql/alter-table-detach-partition/).

:::tip
The active (most recent) partition will never be converted into Parquet, even if it matches the filter.
:::

Conversion is asynchronous, and can take a while to finish, depending on the number of partitions, on the partition size,
on the compression being used, and on disk performance and general load of the server.

To monitor how the conversion is going, you can issue a [`SHOW PARTITIONS`](/docs/query/sql/show/#show-partitions)
command. Partitions in the Parquet format will have the `isParquet` column set to `true` and will show the size on the
`parquetFileSize` column.

```questdb-sql
ALTER TABLE trades CONVERT PARTITION TO PARQUET WHERE timestamp < '2025-08-31';
```

### Bloom filters for in-place conversion

Bloom filters enable row group
pruning for equality and `IN` queries on Parquet partitions. There are two ways
to generate them during in-place conversion.

**Per-column metadata** — If a column was defined with the `BLOOM_FILTER`
keyword in its
[`PARQUET()` clause](/docs/query/sql/create-table/#bloom-filters), bloom
filters are generated automatically during conversion. No additional options are
needed:

```questdb-sql title="Columns with BLOOM_FILTER metadata are indexed automatically"
ALTER TABLE trades CONVERT PARTITION TO PARQUET WHERE timestamp < '2025-08-31';
```

**Explicit column list** — You can specify which columns to index and
optionally set the false positive probability (FPP) using `WITH`:

```questdb-sql title="Convert with explicit bloom filter columns"
ALTER TABLE trades CONVERT PARTITION TO PARQUET
WHERE timestamp < '2025-08-31'
WITH (bloom_filter_columns = 'symbol,side', bloom_filter_fpp = 0.01);
```

:::note

When an explicit `bloom_filter_columns` list is provided, it overrides any
per-column `PARQUET(BLOOM_FILTER)` metadata on the table. If the option is
omitted, per-column metadata is used.

:::

### Converting to Native

```questdb-sql
ALTER TABLE trades CONVERT PARTITION TO NATIVE WHERE timestamp < '2025-08-31';
```

### Data Compression

By default, Parquet files generated by QuestDB are compressed using `lz4_raw` compression. One of the key advantages of Parquet
over QuestDB's native format is its built-in compression.

There are two separate configuration properties in `server.conf`, one for
exports (REST and COPY) and one for in-place conversion:

```ini
# Export (REST /exp and COPY TO)
# Supported codecs: UNCOMPRESSED, SNAPPY, GZIP, BROTLI, ZSTD, LZ4_RAW
cairo.parquet.export.compression.codec=LZ4_RAW
cairo.parquet.export.compression.level=0

# In-place conversion (ALTER TABLE CONVERT PARTITION TO PARQUET)
# Supported codecs: UNCOMPRESSED, SNAPPY, GZIP, BROTLI, ZSTD, LZ4_RAW
cairo.partition.encoder.parquet.compression.codec=LZ4_RAW
cairo.partition.encoder.parquet.compression.level=0
```

When using `ZSTD`, the level ranges from 1 (fastest) to 22, with a default of 9.

For COPY exports, you can also override compression per-query. See [Overriding compression](#overriding-compression).

#### Minimum compression ratio

The `cairo.partition.encoder.parquet.min.compression.ratio` property controls
whether compressed Parquet pages are worth keeping. After compressing a page,
QuestDB checks the ratio of `uncompressed_size / compressed_size`. If the ratio
falls below the threshold, the compressed output is discarded and the page is
stored uncompressed instead.

```ini
# Default: 1.2 (keep compressed output only if it achieves ~17% size reduction)
cairo.partition.encoder.parquet.min.compression.ratio=1.2
```

A value of `0.0` (or any value &lt;= 1.0) disables the check, always keeping
compressed output.

The ratio check applies to both data pages and dictionary pages and works with
all compression codecs. It runs after compression, so the CPU cost of
compression is still incurred -- this setting only avoids the I/O and storage
penalty of keeping pages that barely compress.

#### Per-column overrides

Individual columns can override the global encoding and compression settings.
See [CREATE TABLE - Per-column Parquet encoding, compression, and bloom filters](/docs/query/sql/create-table/#per-column-parquet-encoding-compression-and-bloom-filters)
for defining overrides at table creation, or
[ALTER TABLE ALTER COLUMN SET PARQUET](/docs/query/sql/alter-table-alter-column-set-parquet/)
for modifying existing tables.

### Bloom Filters

Bloom filters are opt-in
probabilistic indexes that enable row group pruning for equality and `IN`
queries. When generated, they are embedded in the Parquet file metadata
alongside min/max statistics.

Bloom filters can be enabled per-column via the `BLOOM_FILTER` keyword in
[`CREATE TABLE`](/docs/query/sql/create-table/#bloom-filters) or
[`ALTER TABLE`](/docs/query/sql/alter-table-alter-column-set-parquet/#bloom-filter),
or per-export via `bloom_filter_columns` in
[`CONVERT PARTITION`](#bloom-filters-for-in-place-conversion),
[`COPY TO`](/docs/query/sql/copy/), and the
[REST `/exp` endpoint](/docs/query/rest-api/#parquet-export-parameters).

The false positive probability (FPP) determines the trade-off between filter
size and accuracy. It is configured globally:

```ini
# In-place conversion (ALTER TABLE CONVERT PARTITION TO PARQUET)
cairo.partition.encoder.parquet.bloom.filter.fpp=0.01

# Export (REST /exp and COPY TO)
cairo.parquet.export.bloom.filter.fpp=0.01
```

See the [Configuration reference](/docs/configuration/overview/) for all
Parquet-related settings.

### Limitations

* `UPDATE` statements are not supported on Parquet partitions. Overwriting rows
  through a dedup upsert is supported, since it is an insert.
* While a partition is being converted, writes to that partition are briefly
  blocked.
* Some parallel queries are not yet optimized for Parquet.

## Export

Query results and whole tables can be exported as external Parquet files. Unlike
in-place conversion, the exported files are written outside QuestDB (streamed to
the client or written to the server filesystem) for use in other tools such as
DuckDB, Pandas, or Polars.

### Export via REST

The `/exp` REST API endpoint executes a query and streams the result as a Parquet file directly to the client. This is a synchronous operation — the HTTP response completes when the file is fully transferred.

:::tip

See also the [/exp documentation](/docs/query/rest-api/#exp---export-data).

:::

You can use the same parameters as when doing a [CSV export](/docs/query/rest-api/#exp---export-data), but passing `parquet` as the `fmt` parameter value.

```bash
curl -G \
  --data-urlencode "query=select * from market_data limit 3;" \
  'http://localhost:9000/exp?fmt=parquet' > ~/tmp/exp.parquet
```

:::note
For larger queries you might prefer to use the [`COPY` method](#export-via-copy),
which runs asynchronously and writes files to the server filesystem.
:::

Once exported, you can use it from anywhere, including DuckDB, Pandas, or Polars. If you wanted
to point DuckDB to the example file exported in the previous example, you could
start DuckDB and execute:

```sql
select * from read_parquet('~/tmp/exp.parquet');
```

### Export via COPY

The `COPY` command writes Parquet files to the server filesystem. Unlike REST export, this is an asynchronous operation — the command returns immediately and the export runs in the background.

:::tip

See also the [COPY-TO documentation](/docs/query/sql/copy).

:::

You can use the `COPY` command from the web console, from any pgwire-compliant client,
or using the [`exec` endpoint](/docs/query/rest-api/#exec---execute-queries) of the REST API.

You can export a query:

```questdb-sql
COPY (select * from market_data limit 3) TO 'market_data_parquet_table' WITH FORMAT PARQUET;
```

Or you can export a whole table:

```questdb-sql
COPY market_data TO 'market_data_parquet_table' WITH FORMAT PARQUET;
```

The output files (one per partition) will be under `$QUESTDB_ROOT_FOLDER/export/$TO_TABLE_NAME/`.

The `COPY` command will return immediately, but the export happens in the background. The command will return an export
id string:

| id               |
| ---------------- |
| 45ba24e5ba338099 |

If you want to monitor the export process, you can issue a call like this:

```questdb-sql
SELECT * FROM 'sys.copy_export_log' WHERE id = '45ba24e5ba338099';
```

While it is running, export can be cancelled with:

```questdb-sql
COPY '45ba24e5ba338099' CANCEL;
```

#### Controlling partitioning

`COPY table_name TO ...` produces one Parquet file per partition, matching the table's own partitioning scheme. `COPY (SELECT ...) TO ...` produces a single file by default.

To override either default, add `PARTITION_BY` to the export options.

Export a table into a single consolidated file:

```questdb-sql
COPY market_data TO 'market_data_single' WITH FORMAT PARQUET PARTITION_BY NONE;
```

Re-partition independently of the source table. For example, export a day-partitioned table into monthly files:

```questdb-sql
COPY market_data TO 'market_data_monthly' WITH FORMAT PARQUET PARTITION_BY MONTH;
```

Partition a query export by month:

```questdb-sql
COPY (SELECT * FROM market_data WHERE timestamp IN '2024')
TO 'market_data_2024'
WITH FORMAT PARQUET PARTITION_BY MONTH;
```

Partitioning requires a designated timestamp column in the source table or query result. Valid values: `NONE`, `HOUR`, `DAY`, `WEEK`, `MONTH`, `YEAR`.

For the full list of export options, see the [COPY-TO documentation](/docs/query/sql/copy/#options-1).

#### Overriding compression

By default, exported Parquet files use `lz4_raw` compression. You can change the default via `server.conf` as shown in [Data Compression](#data-compression),
or override the compression individually for each export. For example:

```questdb-sql
COPY market_data TO 'market_data_parquet_table' WITH FORMAT PARQUET COMPRESSION_CODEC LZ4_RAW;
```