Operating cold storage

This page covers running cold storage in production: preparing the object store prefix, configuring the cluster, moving the manager role between instances, and diagnosing partitions that are not progressing. Read the concept page first for the partition lifecycle and the read path.

note

Cold storage is available in QuestDB Enterprise only, and is disabled by default.

Plan the deployment

Once DROP LOCAL evicts a partition, the object store holds the only copy of it. The prefix is part of the database rather than a copy of it, which shapes four decisions worth making before the first policy goes on a production table:

• DROP LOCAL is an immutability cutoff, not just a disk-space setting. Once a partition is sealed, writes targeting it are skipped at apply time. Set it beyond the longest late-arrival, replay, and correction window for the table, and test the boundary against your real ingestion pattern.
• DROP REMOTE sets the final retention boundary. It is the one stage that physically deletes data, and there is no undo. Leave it out of the policy until remote retention is a deliberate decision.
• QuestDB owns the prefix. External tools may read data.parquet. Nothing else should write, replace, delete, or expire objects under the root, so disable any provider lifecycle or archive rule that covers it.
• Back up the database and the prefix together. They restore as a pair. See Backup and restore below.

1. Prepare the object store prefix

Use a prefix dedicated to one QuestDB cluster. Do not share it with replication or backup storage, and do not point a second cluster at it.

Access requirements differ by role:

Instances	Required access
Every instance	Read
The manager only	Read, create, conditional create, write, delete

Prefer cloud instance roles, workload identities, or managed identities over long-lived credentials in server.conf. Where credentials are unavoidable, load them from a file with QDB_COLD_STORAGE_OBJECT_STORE_FILE.

Disable any provider lifecycle rule that would transition objects to an archive tier or expire them. QuestDB manages object retention itself through DROP REMOTE and remote garbage collection.

2. Configure every instance

Cold storage settings are not reloadable, so this step requires a restart.

Designate exactly one instance as the manager:

server.conf on the manager

cold.storage.enabled=true
cold.storage.role=manager
cold.storage.object.store=s3::bucket=${BUCKET_NAME};root=cold;region=${AWS_REGION};

Every other instance uses the same store specification and the default refresher role:

server.conf on every other instance

cold.storage.enabled=true
cold.storage.role=refresher
cold.storage.object.store=s3::bucket=${BUCKET_NAME};root=cold;region=${AWS_REGION};

The manager can run on a primary or on a replica. Putting it on a replica moves upload, manifest, and garbage-collection work off the primary.

Object store connection strings

The same syntax as replication and backup storage. For provider setup in more depth, including bucket creation, credentials, and the NFS mount requirements, see Configure object storage.

AWS S3

cold.storage.object.store=s3::bucket=${BUCKET_NAME};root=cold;region=${AWS_REGION};

Azure Blob Storage

cold.storage.object.store=azblob::endpoint=https://${STORE_ACCOUNT}.blob.core.windows.net;container=${BLOB_CONTAINER};root=cold;account_name=${STORE_ACCOUNT};

Google Cloud Storage

cold.storage.object.store=gcs::bucket=${BUCKET_NAME};root=cold;

Filesystem or NFS

cold.storage.object.store=fs::root=/mnt/cold/final;atomic_write_dir=/mnt/cold/scratch;

Omitting the credential parameters lets the provider SDK resolve an instance role, workload identity, or managed identity.

3. Apply a policy

Confirm the table is WAL-backed and partitioned, then attach a policy with a TO REMOTE stage:

Check eligibility

SELECT table_name, walEnabled, partitionBy, designatedTimestamp
FROM tables()
WHERE table_name = 'trades';

Convert at 7 days, upload at 14, evict locally at 30

ALTER TABLE trades SET STORAGE POLICY(
    TO PARQUET 7 DAYS,
    TO REMOTE 14 DAYS,
    DROP LOCAL 30 DAYS
);

The first transition can wait for the periodic policy sweep, which runs every five minutes by default. Upload completions, manifest flushes, and seals also trigger a coalesced recheck, so convergence is not bound to the sweep interval.

4. Verify

Policy is attached and active

SELECT * FROM storage_policies
WHERE table_dir_name LIKE 'trades~%';

Per-partition remote state

SELECT * FROM table_cold_partitions('trades');

Which partitions are served remotely

SHOW PARTITIONS FROM trades;

A partition being served from the object store reports isRemotelyServed as true and readOnly as true, and no longer has a local data.parquet file.

For the first read, use a narrow time interval and project only the columns you need, so request count and latency are easy to observe:

First cold read

SELECT timestamp, symbol, price
FROM trades
WHERE timestamp IN '2026-02-10';

Manager handoff

The manager role moves between instances at runtime, with no restart. Role switching requires database administrator (system admin) privileges. See SWITCH COLD STORAGE ROLE for the full syntax.

The supported handoff is two steps, in this order:

1. On the current manager

SWITCH COLD STORAGE ROLE TO REFRESHER;
SWITCH COLD STORAGE STATUS;
-- wait for state = REFRESHER

Demotion flushes pending manifest state and releases the manager lock.

2. On the replacement

SWITCH COLD STORAGE ROLE TO MANAGER;
SWITCH COLD STORAGE STATUS;
-- wait for state = MANAGER with a positive term

Then update cold.storage.role in server.conf on both instances. A hot-switched role does not survive a restart: each instance boots from its configuration again.

Uploads and garbage collection pause during the short interval when no instance holds the manager role. Reads continue everywhere throughout.

Promoting a replica

Neither ordinary nor forced promotion waits for WAL apply. Before promoting a replica to manager, either verify it has caught up with replicated WAL, or set cold.storage.gc.enabled=false and re-enable garbage collection once catch-up is proven.

cold.storage.gc.partition.grace.period is the supported catch-up window, not a substitute for verifying catch-up. Lag beyond it can cause a live remote object to be reclaimed.

Forced takeover

If the previous manager died before it could demote, changing configuration and restarting the replacement is not enough: it finds a foreign lock and starts as a refresher instead.

Break glass only

SWITCH COLD STORAGE ROLE TO MANAGER FORCE;

danger

FORCE deletes the existing lock and claims ownership. It is a recovery path, not a fencing protocol. Stop the old instance, or prove it is dead, before issuing it. Two active managers write byte-divergent Parquet for the same partition and corrupt cold reads.

Deleting _manager.lock with object store tooling bypasses QuestDB's control path and carries the same fencing requirement. It is not a safer alternative.

Remote garbage collection

Garbage collection is the only path that physically deletes remote objects. It is enabled by default and should stay enabled during normal operation.

The manager reclaims objects for partitions the table no longer references. A partition is first marked deleting, then deleted on a later pass once cold.storage.gc.partition.grace.period has elapsed. The grace window lets in-flight readers drain, and lets WAL apply restore a partition that a lagging replica-turned-manager has not yet caught up on.

Dropping a whole table is gated the same way: the table is marked for removal, further writes are rejected, and local removal waits on remote objects being reclaimed within cold.storage.gc.table.grace.period.

Disable garbage collection with cold.storage.gc.enabled=false only as a temporary failover safeguard.

Monitoring

Lifecycle state

Partitions not yet durable in the object store

SELECT timestamp, state, seq_txn, size, last_modified, partition_path
FROM table_cold_partitions('trades')
WHERE state = 'pending';

A manager reports its own in-memory view. A refresher reports its mirrored copy, which it refreshes when the catalog generation changes. While a role transition is in flight, table_cold_partitions() returns zero rows rather than blocking. SWITCH COLD STORAGE STATUS reports the live role at any time.

Cold read metrics

The Prometheus endpoint exposes fifteen cold-read metrics under the questdb_cold_chunk_ prefix. The ones worth alerting on:

questdb_cold_chunk_download_started_total
questdb_cold_chunk_download_finished_total
questdb_cold_chunk_download_failed_total
questdb_cold_chunk_download_coalesced_total
questdb_cold_chunk_in_flight_downloads
questdb_cold_chunk_pending_batches
questdb_cold_chunk_busy_leases
questdb_cold_chunk_pinned_bytes

Watch rates and ratios rather than raw totals:

• Sustained download_failed_total points at credentials, network, or a missing object.
• pending_batches at its configured cap means queries are being rejected with backpressure errors. Narrow the queries before raising the cap.
• pinned_bytes or busy_leases staying non-zero after query traffic stops indicates leases that were never released, and is worth reporting.
• download_started_total relative to questdb_cold_chunk_acquire_miss_chunks_total is the coalescing factor. A ratio near one means requests are not being merged.

Backup and restore

Backups capture the local transaction state, Parquet metadata, and symbol indexes needed to reopen a cold partition. They do not copy the object store's data.parquet bytes into the backup.

Protect the database backup and the cold storage prefix as a single recovery set. A restored instance must be able to reach the same objects, or a deliberately copied equivalent. If a cold object is lost after local eviction, QuestDB has no local source from which to regenerate it.

A restore onto a different host or backend can change an object's version even when the bytes are identical. The first read of an affected partition fails, QuestDB re-checks the object in the background, and the retry succeeds. An object that is genuinely different is refused rather than silently accepted.

Worth testing before relying on cold storage for retention:

• Restore with access to the existing prefix.
• Restore from a copied prefix, where object versions differ.
• Query retry after a restore changes an object's version.
• The two-step manager handoff, after verifying WAL catch-up.
• Recovery of an accidentally removed object from a provider version or backup.

Troubleshooting

Symptom	Likely cause	What to do
table_cold_partitions() returns no rows	No eligible non-active partition, the first sweep has not run, or the role is still resolving at startup	Check the policy, SHOW PARTITIONS, and the role with SWITCH COLD STORAGE STATUS
A partition stays pending	Upload is queued, or the manager cannot write the object and its sidecar	Verify manager ownership, worker count, credentials, network, and quota. Do not edit the manifest
A partition stays live	It has not reached DROP LOCAL, or the seal has not applied	Check the DROP LOCAL horizon and WAL apply health on every instance
Manifest says sealed but isRemotelyServed is false	The instance is waiting on catalog refresh, object access, or metadata staging	Verify read access to data.parquet and _pm.a, then allow the next recheck
A query reports a stale partition	The object is not the version QuestDB installed	Retry the query; QuestDB re-checks the object in the background. If it persists, restore the expected object, since a genuinely different one is refused deliberately
A cold query is slow or hits backpressure	Broad scan, wide projection, object store cold start, or saturated read limits	Narrow the time range and columns, reduce concurrency, then inspect the metrics above before tuning
Historical writes are missing	Their partition was already sealed	Review apply logs and the DROP LOCAL horizon. There is no in-place unseal
A cold object is missing	External deletion, a lifecycle rule, or incomplete recovery of the prefix	Restore the exact expected object. A cold partition cannot self-heal from local data

See also

• Cold storage for the lifecycle and read path
• Cold storage configuration for every key
• SWITCH COLD STORAGE ROLE
• Storage Policy
• Backup and restore