QuestDB Enterprise 4.0: cold storage, QWP, and hot failover

QuestDB Enterprise 4.0 is out, and it brings three long-awaited features.

Cold storage tiers your history out to object storage automatically and keeps every partition queryable from the same SQL you already write. Storage gets cheaper, and a table can grow past anything a single volume could hold.

QWP is our new binary protocol, and it runs in both directions from a single client. Ingestion peaks at 19 million rows per second, up to 3.6x faster than ILP, and on the read side columns stream back and decode into Apache Arrow at 220 million rows per second, landing in a dataframe with no row-by-row conversion on the way.

High availability stops being a manual procedure. Failing over used to mean running a queue in front of the database, then promoting a replica by hand on its local filesystem and restarting it. Now the promotion is a hot switch with no restart, and the QWP clients cover the rest themselves: they buffer whatever the server hasn't confirmed and move to the new primary on their own, so no rows are lost and queries keep working throughout.

There is more in the release: Parquet tables that skip the conversion step, live views that work on a replicated cluster, and a Web Console rebuilt around notebooks, with a light mode.

Cold storage: older partitions move to object storage and stay queryable

Cold storage moves older data onto object storage on its own schedule. Object storage is cheaper than attached volumes, and it can grow bigger than any single volume, so a table can hold more history than one node could ever store. One copy serves the whole cluster instead of being duplicated on every node, and every node queries it in place, so your SQL does not change. The target can be S3, GCS, Azure Blob, or an NFS mount that every instance can see.

A storage policy is a set of TTLs attached to a table, one per stage of a partition's life. QuestDB Enterprise 4.0 enables the remote tier of that policy:

ALTER TABLE trades SET STORAGE POLICY(
    TO REMOTE 1h,
    DROP LOCAL 30d,
    DROP REMOTE 1y
);

In this example, a partition goes up to the object store an hour after it ages out, while the local copy stays behind and keeps answering queries. After thirty days the local copy is deleted and queries start reading the bucket instead. After one year the partition is deleted from the bucket too.

Once TO REMOTE fires, the partition exists in the object store, where anything else with access to the bucket can read it. Queries are unaffected at this point: the local copy is still there, so they are served from the local tiers, native or Parquet, as before.

DROP LOCAL removes that copy. The partition is gone from local disk but not from the table, and queries that reach it are served from the remote tier instead. A two-year window plans and runs the way it always did, reading native partitions, local Parquet partitions and remote ones in a single scan.

Cold storage is off by default, covers WAL tables, and needs a few lines of configuration pointing at a bucket. Which instance uploads, how to monitor what is where, and the full set of settings are all in the storage policy docs.

Standard Parquet, straight into your lakehouse

The files shipped to object storage are ordinary Parquet, carrying the metadata that comes with the format, compression and bloom filters included, so the rest of the ecosystem can read them efficiently and directly. If you run a table format like Iceberg or DuckLake, you can register those partitions in your catalog.

We will publish a couple of posts shortly on integrating cold storage with Iceberg and with DuckLake.

Parquet as a first-class citizen, now in QuestDB Enterprise

Parquet is not only the format used by cold storage. Tables are created in QuestDB's native columnar format by default, and they can now be created directly as Parquet instead, on the local volume. Parquet compresses better and gives you compatibility with any other tool that reads those files on disk. The trade-off is on the write path: native tables are faster to write, especially with deduplication or out-of-order ingestion.

CREATE TABLE trades (
    timestamp TIMESTAMP,
    symbol SYMBOL,
    price DOUBLE,
    amount DOUBLE
) TIMESTAMP(timestamp)
PARTITION BY DAY
FORMAT PARQUET;

New partitions in that table are written as Parquet directly, with no conversion step. ALTER TABLE ... SET FORMAT PARQUET does the same for a table that already exists, from its next partition onwards.

QWP: faster in, faster out, and one client for both

QWP is the QuestDB Wire Protocol, a new binary columnar protocol that runs over WebSocket.

It's faster in both directions. We measured ingestion up to 3.6x faster than ILP over a network, because a TSBS row is about 347 bytes as line protocol text and about 97 bytes as QWP. On the read side we stream columns directly, and in Rust or Python those columns decode into Apache Arrow at 220 million rows a second.

One client does both. You stop needing an ingestion library and a separate PostgreSQL driver. One dependency, one connect string, one handle that writes and reads:

with questdb.connect("ws::addr=localhost:9000;") as db:
    with db.sender() as sender:
        sender.row("trades", symbols={"symbol": "ETH-USDT"},
                   columns={"price": 2615.54}, at=TimestampNanos.now())
    with db.query("SELECT * FROM trades WHERE symbol = $1", ["ETH-USDT"]) as r:
        frame = r.to_polars()

It does not replace anything. ILP and the PostgreSQL wire protocol are both still supported, still maintained, and nothing you run today stops working when you upgrade. If you have Telegraf pointed at QuestDB or a Grafana datasource on PGWire, you don't need to change them. QWP is the better option for new work, not the successor to the other two.

High availability, built into the client

QWP clients keep a local buffer of every row the server hasn't confirmed yet, a mechanism called store-and-forward. If the connection drops, the client reconnects and sends the buffered rows again, while your code keeps calling row() without ever blocking on the network. The buffer can live on disk rather than in memory, so the rows survive the producer process crashing too. Delivery is at-least-once, and pairing it with table DEDUP keys collapses any duplicate on write, giving you exactly-once ingestion.

That is one half of high availability, the half that runs in your application. The other half is what the cluster does when the primary goes away.

Failover without a restart

QuestDB Enterprise 4.0 brings you hot replica promotion. Turning a replica into the primary needs no restart and no touching the local filesystem. You do still need to run a SQL command, from a client library, interactively in the Web Console, or over the REST API. That is what makes a watchdog possible: a process you run can detect that the primary is gone and promote a replica.

The clients find the new primary themselves. Give a client every peer in the connect string and it handles a disconnect on its own:

addr=node-a:9000,node-b:9000,node-c:9000

It tracks the health of each host and moves to the next healthy one when the node it is on stops answering. If no host is ready to take the write yet, store-and-forward covers the gap: the client keeps buffering, and those rows go out as soon as a promoted primary is there to accept them.

You can control which node a client fails over to. target says whether it wants the primary or a replica, and zone keeps queries inside one location.

Readers never wait for any of that: any healthy replica can answer a query, so the read path keeps serving while the write path moves.

See client failover for the full model.

Live views on a replicated cluster

A live view is a table holding the result of a window-function query, kept up to date as rows arrive. The window runs once per new row and the output is appended, so reading a rolling VWAP, a cumulative volume or a running rank scans precomputed rows instead of recomputing the window over millions of rows on every query. Live views are in beta, with a deliberately narrow SQL surface in this first version.

CREATE LIVE VIEW trades_ma
FLUSH EVERY 1s
IN MEMORY 5s
START FROM NOW
AS
SELECT
    timestamp,
    symbol,
    price,
    avg(price) OVER (
        PARTITION BY symbol
        ORDER BY timestamp
        ROWS 300 PRECEDING
    ) AS moving_avg
FROM trades;

You then read trades_ma like any other table. Refreshes publish to an in-memory tier first and flush to disk on the interval you set, so a read sees a new row as soon as it is computed rather than waiting for the flush.

On a replicated cluster each node maintains its own copy from the base table it already replicates, so a replica serves the view without any extra replication traffic, and CREATE LIVE VIEW and DROP LIVE VIEW are grantable like any other permission.

Notebooks, agents, and a light mode

The Web Console is no longer a single editor. Notebooks mix SQL, markdown and chart cells in one document, with charts rendered by ECharts and query cells that can refresh themselves.

They pair with the QuestDB MCP server, which relays Claude Code, Codex, Cursor or any other MCP client into the tab you have open. An agent is good at turning a one-line request into a whole dashboard, and much worse at the corrections that follow: move that legend, change that colour, swap those two panels. So let it build the notebook, fix one chart by hand, then tell it to do the same to the other six.

Your Web Console session is the credential. An agent pairs by asking you to approve the token dialog below, in your own console tab, and it can do nothing at all if nobody is signed in there. Your database credentials never reach the agent, and everything it runs goes through the session you already have.

Notebooks are also why the console has a light theme now, where dark used to be the only option: a table pane and a SQL editor were fine dark, but people who spend their day in Jupyter or Marimo expect a light background. There's a settings panel too, and notebooks and tabs can be exported and imported, so the notebook an agent just built for you is something you can hand to a colleague or keep in a repository.

Everything else in this release

All of the above sits on top of QuestDB 10.0.1, so this release also carries the engine work that went into it. The 10.0 post has the detail:

• Schema evolution on Parquet tables: ALTER COLUMN ... TYPE now works on Parquet partitions, converting lazily at the query path instead of failing or leaving the data unconverted.
• Per-query memory limits, queries that stop when the client disconnects, and ALTER TABLE ... REBASE WAL.

QuestDB Enterprise adds one more fix on top. sys.acl_permissions is an append-only log that the ACL loader collapses at read time, and on a deployment re-granting on a schedule it had reached 10.7 million rows. Every replicated GRANT then made the replica reload its access lists with a two-minute full-table scan while holding the monitor PGWire login needs, so replica logins hung for that whole window. The table now compacts automatically, and the heavy part of that work holds no lock at all.

There is a long tail of other bug fixes, and a few breaking changes worth reading before you upgrade, including one that stops an instance from starting if it still carries the OSS http.user and http.password settings. The release notes page has the detail, as do the 10.0.0 and 10.0.1 notes for the engine.

Getting the update

Self-managed enterprise customers will find the binaries at the usual download location. BYOC enterprise customers will be contacted for upgrading.

Upgrade replicas before the primary. Sealing a partition is a new WAL event type, and a node still on an older build will fail on it.

Not on QuestDB Enterprise yet? Learn more about QuestDB Enterprise and BYOC, or contact the QuestDB team for a conversation or a demo.