When to use store-and-forward

The QWP WebSocket transport always uses a store-and-forward (SF) substrate. What changes between deployments is where that substrate keeps unacked data and what durability bar it acknowledges against. This page is the decision guide.

If you are new to SF, start with Concepts.

Memory-only mode vs SF mode

The single switch that decides this is whether you set sf_dir in the connect string.

Memory-only mode: sf_dir unset

Unacked rows live in a buffer in process memory. Default cap is 128 MiB. This is not the same as sf_durability=memory, which still writes to the sender directory and only skips forcing each write to disk.

Choose memory-only mode when:

• The producer process is short-lived or ephemeral (a CLI job, a CI worker, a serverless function).
• A sender restart is acceptable as a fresh start — losing any in-flight data when the sender stops is acceptable.
• You only need to tolerate transient network blips and short server outages (think: rolling upgrades, brief network partitions).
• Your data volume comfortably fits in RAM during the longest outage you care about.

SF mode: sf_dir=/path/to/dir

Unacked rows are written to memory-mapped files in the sender's sender directory, <sf_dir>/<sender_id>/. Default cap is 10 GiB.

Choose SF mode when:

• The producer process is long-running and must ride out outages measured in minutes or hours. A running sender retries indefinitely, so what caps your tolerance is how much unacked data you can hold: sf_max_total_bytes and the disk behind sf_dir, not a timer.
• In-flight data must not be lost when the sender stops or its host reboots — crash, OOM kill, planned redeploy.
• You ingest at rates where minutes of buffering exceeds RAM you can spare.
• You operate unattended at the edge (sensors, ETL jobs) where the server may sometimes be unreachable for extended periods.

Both modes share the same wire behaviour, the same failover loop, and the same connect-string keys for everything other than storage. You can switch between them without changing application code — only the connect string.

Comparison at a glance

Aspect	Memory-only mode	SF mode
Buffered data location	Process RAM	Disk (<sf_dir>/<sender_id>/)
Default capacity	128 MiB	10 GiB
Unacked data after a sender crash (kill -9, OOM)	Lost	Recovered and replayed on restart
Unacked data after the sender's host reboots	Lost	Recovered, if the disk persists
Cross-sender rescue (orphan adoption)	n/a	Yes (opt-in)
Setup cost	Zero	Provisioning a writable directory
Operational cost	Zero	Sizing, monitoring, lock collisions

Durable-ack: when to opt in

The client keeps every batch until the server acknowledges it, then trims it: it frees the space a whole segment at a time, once every batch in the segment is acknowledged. By default the server acknowledges a batch once it is committed to the WAL on the primary. With the server's default cairo.commit.mode of nosync, that commit doesn't wait for the disk, so a crash of the primary's host can still lose it. Whether the batch has been replicated to the object store or to replicas is a separate matter.

When the connect string sets request_durable_ack, the client keeps the batch until a separate durable-ack message confirms more:

• local: the commit is on the server's own disk. The server must run in adaptive durability mode.
• replicated, or its original name on: the WAL has been uploaded to the configured object store (S3, Azure Blob, GCS, or NFS). QuestDB Enterprise only.
• local,replicated: both. The client waits for the object store.

Choose durable-ack when

• A power loss of the server must not lose in-flight data, and the server runs in adaptive durability mode with a group-commit window above 0 (local).
• You require object-store durability before considering a write acknowledged — e.g. compliance requirements, end-to-end exactly-once pipelines with cross-region recovery.
• Loss of an entire primary node (and its local disk) must not lose in-flight data — replicas haven't downloaded the WAL yet, only the object store has.
• You are willing to trade later trim (and so larger steady-state SF disk usage) for the stronger guarantee.

Stay on the default OK trim when

• Losing the last commits in a power loss of the primary is acceptable, or the primary already flushes every commit before it acknowledges it (sync mode, or adaptive durability mode with a group-commit window of 0).
• You want minimum steady-state disk usage.
• The server cannot provide the tier you need: QuestDB open source has no replicated tier, and only a server in adaptive durability mode has the local tier. (The handshake fails loudly if you opt in but the server cannot deliver — see below.)

Caveats

• Server support is required. The client sends the requested tiers in X-QWP-Request-Durable-Ack on the upgrade (true for on). The server must confirm all of them in X-QWP-Durable-Ack (enabled for on). If it does not, the connect fails loudly, by design: for example a server not in adaptive durability mode asked for local, an OSS build asked for replicated, an uninitialised primary, a missing registry, or a replica. Silently waiting for acknowledgements that never arrive would let the SF disk fill up.
• Idle keepalive. The server only sends pending durable acknowledgements during inbound recv events. The client sends a WebSocket PING every durable_ack_keepalive_interval_millis (default 200 ms) when there are pending confirmations and the producer is idle.
• Disk pressure. Steady-state SF disk usage is roughly ingest_rate × time_to_durability: for local, about one group-commit window, plus up to one durable_ack_keepalive_interval_millis when the producer is idle; for replicated, the upload time. Size sf_max_total_bytes accordingly.

Orphan adoption: when to enable

A sender that exits without draining its sender directory leaves unacked data on disk. If another process restarts under the same sender_id and same sf_dir, it picks up the orphan automatically as part of normal recovery. But if no process ever uses that sender_id again, the data sits on disk forever.

Setting drain_orphans=on tells the foreground sender to scan <sf_dir>/* at startup for sibling sender_ids with unacked data and spawn background drainers to clear them.

Enable orphan adoption when

• You have a fleet of senders writing to a shared sf_dir (multi-tenant host, container restart) and want any survivor to rescue dead siblings' data.
• Your deployment can dynamically allocate sender_id (e.g. one per process instance), so dead instances leave permanent orphans that no natural restart will adopt.
• You prefer "automatic eventual delivery" over "operator manually reattaches the sender directory."

Leave it off when

• Each sender_id is statically pinned to a specific process — there are no orphans by construction; a restart of the same process recovers its own sender directory.
• You want explicit operator control over data movement in a shared sf_dir.
• You run a single producer per host.

Drainer concurrency is capped by max_background_drainers (default 4). Each drainer opens its own connection — they share the network path but not the WebSocket.

drain_orphans=on does not interfere with regular recovery: the foreground sender still recovers its own sender_id first, then drainers spawn for other sender directories.

Migrating from HTTP/TCP ILP

If you are currently using HTTP or TCP ILP ingest, the comparison is:

Capability	HTTP ILP	TCP ILP	QWP WebSocket + SF
Non-blocking producer	No (request waits)	No (TCP backpressure)	Yes (buffer absorbs publishes)
No data loss on a sender crash	No	No	Yes (SF mode)
Server outage tolerance	Best-effort retry	None	Reconnect loop that retries indefinitely, bounded by sf_max_total_bytes
Multi-host failover	Yes (HTTP only)	No	Yes
Ack once the data is on the server's disk	Only in sync mode, or adaptive durability mode with a group-commit window of 0	No	Yes (request_durable_ack=local, adaptive durability mode)
Cross-region durability ack	No	No	Yes (request_durable_ack=on)
Cluster-wide ordering	Best-effort	Best-effort	FSN order within each sender stream, not across senders

QWP replay is at-least-once, so a reconnect can write rows twice. Use table-level DEDUP UPSERT KEYS or DEDUP SKIP KEYS with stable event timestamps and row identity to suppress replayed rows. See Delivery semantics for the deduplication requirements.

The transition is application-transparent — Sender.fromConfig accepts a ws:: or wss:: connect string and the public builder API is the same. The most common migration is HTTP ILP → QWP WS+SF, with sf_dir set, retaining HTTP for backward compatibility while the QWP path becomes the primary.

For specifically the multi-host HA path on HTTP ILP, see the existing ILP overview "Multiple URLs for High Availability" section. QWP failover (documented in Client failover concepts) replaces and extends it.

Decision flowchart

Next steps

• Configuration — the connect-string keys.
• Operating and tuning — sender directory layout, sizing, observability.