Operating and tuning store-and-forward

This page is the operator-facing guide for SF in production: how to provision the sender directories, what to watch, and how to tune the limits. For the underlying model see Concepts; for the choice between memory-only mode and SF mode see When to use.

Sender directory layout

In SF mode every sender owns one sender directory:

<sf_dir>/<sender_id>/
├── .lock              # advisory exclusive lock (kernel-released on process exit)
├── .lock.pid          # UTF-8 text: holder PID + '\n' (diagnostic only)
├── .failed            # optional drainer-failure sentinel (UTF-8 reason text)
├── .ack-watermark     # optional durable-ack high-water mark
├── .symbol-dict       # the sender's SYMBOL dictionary
├── sf-0000000000000001.sfa
├── sf-0000000000000002.sfa
└── ...

<sf_dir> is the group root — the directory you point the connect string at. <sender_id> is the sender directory's name; it defaults to default but should be set explicitly when more than one sender shares the host.

Client APIs and messages often call a sender directory a slot, as in UnreplayableSlotException, the Go client's QuarantinedSlotPath(), and lock errors such as sf slot already in use. Node.js messages call it a store-and-forward journal, and its error names call it a replay store, as in QwpReplayStoreLockedError.

.lock and .lock.pid

The .lock file is held under an advisory exclusive lock for the engine's lifetime — POSIX clients use flock / fcntl, Windows uses LockFileEx. The lock is released automatically when the file descriptor closes, including on hard process exit (kernel cleanup).

A second sender pointing at the same sender directory will fail to start with an error that names the holder's PID, read from .lock.pid. The PID file is overwritten on every successful acquire; an absent or empty .lock.pid reports holder=unknown rather than failing the lookup.

Neither .lock nor .lock.pid is deleted on clean shutdown. Stale files are harmless — the next acquirer silently overwrites them.

The Node.js client writes the same .lock and .lock.pid files, but takes the lock with a .lock.owner directory beside them instead of flock. A crash leaves that directory behind, and a new sender takes it over only when it names a process on the same host that no longer exists. In a container, a restart often reuses the process ID or changes the host name, so the sender directory stays locked until you delete .lock.owner: see Recovering a sender directory after a crash. The two locks do not interoperate, so a Node.js and a Java process must never open one sender directory at the same time.

Cross-platform interop: a POSIX client and a Windows client must not share a sender directory on a network filesystem. Their lock primitives are incompatible.

.failed

Present iff a previous drainer attempt gave up on the sender directory — reconnect budget exhausted, terminal auth failure, or irrecoverable corruption. The file contents are a UTF-8 reason for human operators; the presence is the signal that the orphan scanner uses to exclude the directory from auto-drain on subsequent scans.

Operator action: read the reason, fix the underlying cause (rotate credentials, restore the missing peer, etc.), then delete .failed. The next sender that scans <sf_dir> will pick the directory up again.

Segment files

Segments are named sf-<gen>.sfa where <gen> is a 16-character zero-padded hexadecimal generation counter. The number reflects allocation order, not the FSN range — that lives in the file header and is read at recovery time.

Pre-allocation reserves real disk blocks at file creation. On Linux this is posix_fallocate; on macOS, F_PREALLOCATE / F_ALLOCATEALL. The substrate refuses to fall back silently to ftruncate on filesystems where these are unsupported — sparse files would risk a SIGBUS later when the mmap'd region writes into a hole. On filesystems where the native layer must fall back to ftruncate, size sf_max_segment_bytes conservatively against free space.

The Go client never falls back. Where reservation isn't supported, such as on some network and FUSE filesystems and on Unix systems other than Linux and macOS, creating a Go sender fails with ErrSfDurability. Move sf_dir to a supported local filesystem.

Lock collisions in practice

Two sender_ids in the same sf_dir never collide — they are independent sender directories. The same sender_id started twice will collide, and the second start fails loudly.

A common cause is a redeploy where the old process hasn't fully exited when the new one comes up. Solutions:

• Wait for the old process to release the lock (the kernel releases on exit; kill -9 is sufficient). A crashed Node.js sender can leave its .lock.owner directory behind, as described above.
• Use a deployment unit that orders shutdown before startup.
• For containerised deployments, set sender_id from a per-pod stable identity so two pods with the same template name don't collide.

drain_orphans=on does not override the lock: a sender directory that another process holds is skipped, not stolen.

Sizing capacity

Two limits matter:

sf_max_segment_bytes: per-segment file size (default 4 MiB)

This is the rotation threshold and the unit of trim, and each batch must fit in one segment. Segments that are smaller release disk faster but waste more space on the active tail; larger segments waste less on the active tail but hold acked batches in the same file as the still-unacked tail until every batch in the segment is acked.

For most workloads 4 MiB is fine. Raise it if you are appending very large batches and pre-allocation cost matters; lower it if you observe disk usage staying high under slow ack cadence.

sf_max_total_bytes: per-sender capacity (default 128 MiB memory-only / 10 GiB SF)

This is the hard cap on resident SF storage — sealed segments plus the active segment. When this fills, producer appendBlocking calls block (with cooperative yield) for up to sf_append_deadline_millis waiting for ACK-driven trim to free space; on timeout the call throws.

Size this against your worst expected outage times your ingest rate:

sf_max_total_bytes ≥ ingest_rate × max_tolerated_outage

A 5-minute outage at 10 MB/s of compressed data implies at least 3 GB. Add safety margin for trim latency — in particular, request_durable_ack=on or replicated extends time-to-trim by the WAL→object-store upload window, and local by about one group-commit window, plus up to one durable_ack_keepalive_interval_millis when the producer is idle.

In memory-only mode the default 128 MiB is deliberately small: it forces you to think about backpressure rather than letting an outage silently balloon process RSS.

Backpressure observability

appendBlocking distinguishes two reasons it can stall:

• Wire-publishing backpressure. The server is acking but the producer is faster than ack throughput. The exception message names this state. Solutions: scale the server, slow the producer, or raise sf_max_total_bytes.
• Reconnect backpressure. The I/O loop is in the retry loop and the substrate is filling. The exception message includes the attempt count and outage start time. Solutions: address the cluster outage, raise sf_max_total_bytes, or accept that the producer will start throwing once the cap is exhausted.

The getTotalBackpressureStalls() counter (see Observability below) records every producer thread that hit the cap.

Recovery on restart

When an SF-mode sender opens, it runs this sequence:

1. Acquire <sf_dir>/<sender_id>/.lock. Fail loudly on contention.
2. Scan every *.sfa file:
   • Validate magic, version, header.
   • Walk the batches forward verifying each CRC32C-Castagnoli checksum.
   • The first invalid batch becomes end-of-data; any non-zero bytes past that point are logged as a torn-tail count.
3. Sort segments by baseSeq and verify no gaps. A gap means the chain is incomplete: the reference client sets the sender directory aside as <sf_dir>/<sender_id>.unreplayable-<n> and starts on an empty one.
4. Open .ack-watermark (if present) and read the cumulative durable-ack FSN. Reject a watermark that exceeds the on-disk ceiling — it would seed ackedFsn past every un-acked batch and silently drop the un-acked tail.
5. Seed ackedFsn = max(lowestBaseSeq - 1, watermark).
6. Allocate the next segment generation as max(existing-gen) + 1.
7. Bump the connection generation so the I/O loop replays from disk against a fresh wireSeq window.

A clean shutdown that drained everything is indistinguishable from a fresh start: no segments, no replay.

Recovery failures

Symptom	Likely cause	Operator action
A lock error that names the holder's PID: sf slot already in use (Java), slot already in use (Go), store-and-forward slot is already in use (Rust, C, C++, Python), slot … is already locked (.NET), or store-and-forward directory is already in use (Node.js)	Two processes claiming the same sender_id. Most clients call the sender directory a slot.	Stop the duplicate. The lock releases on its exit. A crashed Node.js sender can leave its .lock.owner directory behind; see .lock and .lock.pid.
FSN gap in recovered segments (Java)	A segment was deleted or damaged out of band.	The client sets the sender directory aside as <sender_id>.unreplayable-<n>, reports a DATA_LOSS error, and starts on an empty one. Re-ingest the set-aside rows from their source.
"Watermark exceeds publishedFsn"	.ack-watermark is corrupt; the engine falls back to the no-watermark seed.	Logged as WARN. Replay will re-send the lowest segment's batches; rely on server deduplication.
Torn tail count > 0	The previous process crashed while writing a batch.	Informational; the CRC + zero-fill design discards the partial batch.

Close and shutdown

close() semantics depend on close_flush_timeout_millis:

Value	Behaviour
60000 on Java and .NET, 5000 on the other clients (default)	Block up to that long waiting for ackedFsn ≥ publishedFsn. On timeout the un-acked tail stays on disk (SF) or is lost (memory-only). How the timeout is reported depends on the client; see Close and shutdown.
0 or -1	Skip the drain wait. Pending data persists on disk (SF) for the next sender, or is lost (memory-only).
any other positive value	That timeout in milliseconds.

In every branch close():

• Performs a non-blocking safety-net check that rethrows any latched terminal error not already delivered through an async handler or a synchronous producer call.
• Releases the sender directory's lock and unmaps segment files.

The safety-net check is what makes "close-and-forget" callers safe: if the only API your code uses is close(), terminal errors still surface rather than silently sinking into a no-op handler.

Orphan adoption in operations

With drain_orphans=on, the foreground sender — after acquiring its own lock — scans <sf_dir>/* for siblings that:

• are not its own sender_id,
• contain at least one *.sfa file,
• do not have a .failed sentinel.

Up to max_background_drainers drainers run concurrently. Each drainer opens its own engine and WebSocket connection, runs recovery + replay, and exits when the orphan's ackedFsn ≥ publishedFsn.

Drainer failure modes

• Reconnect budget exhausted. Drainer writes .failed with reason, releases the lock, exits.
• Auth-terminal upgrade error. Same.
• Irrecoverable corruption. Same.

Sender directories with .failed are excluded from auto-drain on subsequent scans — operator action is required to clear the sentinel.

Observing drainers

• getActiveBackgroundDrainers() — count of currently-running drainers (best-effort: a just-finished drainer may still count for a few ms).
• getTotalBackgroundDrainersSucceeded() / …Failed() — cumulative outcomes since process start.
• The BackgroundDrainerListener callback delivers per-drainer events (progress watermark, durable-ack-mismatch escalation, terminal outcome) for richer dashboards.
• On-disk .failed sentinels are the canonical record of giveup events surviving sender restart.

Observability counters

A conformant client exposes at minimum:

Counter	What it tells you
getTotalReconnectAttempts()	How often the wire has broken across the sender's lifetime.
getTotalReconnectsSucceeded()	How many of those recovered.
getTotalFramesReplayed()	Batches re-sent after reconnects (the protocol calls a batch a frame). A spike usually means a fresh outage; sustained growth means a flapping wire.
getTotalServerErrors()	Count of error responses received (any category).
getDroppedErrorNotifications()	Error-inbox overflow count. Non-zero means a busy error stream or a slow handler.
getTotalErrorNotificationsDelivered()	Errors delivered to the user handler.
getTotalBackpressureStalls()	Producer threads that hit sf_max_total_bytes.
getLastTerminalError()	The latched SenderError, or null.
getActiveBackgroundDrainers()	Running orphan drainers right now.
getTotalBackgroundDrainersSucceeded() / …Failed()	Cumulative drainer outcomes.

Suggested dashboards

• Reconnect health: reconnect_attempts - reconnect_succeeded over time. A non-zero difference for more than a few seconds means the wire is currently down. There is no budget-exhaustion event to alert on, since a running sender retries indefinitely, so pick a duration threshold that matches your buffer headroom and alert on that instead.
• Replay volume: frames_replayed rate. Bursts are expected; sustained replay means a chronic instability.
• Backpressure: backpressure_stalls rate. Any non-zero rate is a capacity signal.
• Error rate by category: instrument your error handler to bucket by category. A SCHEMA_MISMATCH is terminal by default and stops the sender, so alert on it, and page on DATA_LOSS.

The default error handler logs every received SenderError: ERROR for terminal and abandoned errors, WARN for retriable ones. Replace it only if you are also routing the errors somewhere else (Sentry, structured logs): silence is forbidden by the contract.

Multi-sender deployments

When several senders share a host and a sf_dir:

• Give each one a unique sender_id. The defaults sender_id=default is fine for a single-sender host but collides for any second sender.
• Consider drain_orphans=on if dynamic sender identities mean dead instances can leave permanent orphans.
• Size sf_max_total_bytes × number_of_senders against available disk.
• Plan for the worst-case lock-collision recovery: a misconfigured fleet that all share sender_id=default will leave only one sender alive on each host. That is the design — fail loudly rather than silently corrupt a shared sender directory.

Next steps

• Configuration — the full connect-string key reference.
• Client failover concepts — what the reconnect loop does between disconnects.