Skip to content

Protected record retention and delivery

The gateway owns bounded volatile record retention. This is not durable Server storage; current Server acceptance remains separate.

Ownership, IPC and protection decisions

  • Owner: existing gateway, new C++ record-manager boundary. It does not fork. Decoder-child and decoder-service restarts preserve earlier-run history; gateway/owner restart or appliance outage destroys all RAM history. A fresh buffer-owner UUID advertises that boundary. No payload-bearing checkpoints.
  • Separate bounded AF_UNIX record handoff from each protected decoder child, beside (not inside) existing JSON diagnostics/IQ handoff. Canonical bytes from Decoder::publish, never reconstructed JSON. Same-UID sockets, bounded framed fragments/reassembly, nonblocking publication and cumulative source drops. Partial handoffs and sequence holes are explicit admission loss; a crashed producer's unobserved tail remains unknown, not invented exact coverage.
  • 128 MiB aggregate payload pool, independent consumers and bounded fixed indexes. Equal hard shares charge owned physical allocation, including 256-byte rounding and dependency-only nodes/entries counted once, not forward wire bytes. Bootstrap wire reservations are also shared. Reclaim from the over-budget/incoming stream or reject; never globally evict a compliant neighbor. New streams rebalance shares; empty obsolete runs may release their bootstrap. a sole stream may use the whole pool. Byte and optional monotonic-age eviction, plus index/epoch bounds. ACKs do not pin records. Original configuration/boundary records remain bounded bootstrap dependencies, not duplicate payload histories.
  • Protected copies everywhere: bounded mapped/locked/non-dump/non-fork pools, required gateway/child mlockall(CURRENT|FUTURE) and dump protection, including OpenSSL/libwebsockets allocations and send buffers. Account indexes, bootstrap, reassembly, in-flight copies and process locks separately. Increase/verify gateway memlock provisioning, not global swap policy; kernel cgroup limits are ineffective here. Allocation/locking failure never falls back to heap/disk. Resize provisions a replacement under an explicit peak limit before publishing it; failure preserves old protected state/configuration.
  • No ring pointers survive asynchronous writes. Each admitted record has one authoritative immutable copy. A bounded protected send copy survives fragments; eviction cannot invalidate it. Cancellation/close zero and release it. Ingest and egress have time/byte/message budgets so controls and acquisition progress.

Delivery protocol decisions

/ws/records, rfsentry-records-v1, existing configured API TLS listener (8443 default). IQ-only 8444 and QoS unchanged. Delivery framing is separately versioned; record bodies retain exact envelope bytes/hash/original identity.

  • First bounded message authenticates by credential ID/token in the encrypted message body, reusing the existing auth-message convention. No URL secrets, custom browser headers or logging. Exact configured Origin for browsers; native absent-Origin permitted only here, with the same required authentication deadline/resource limits. No anonymous record access, even with legacy auth off.
  • Host-only managed definitions contain revocable token hashes and records.read scopes by stable logical stream UUID. Operator-only configuration/credential commands use the existing authenticated control listener. Status readers do not gain payload access; record readers gain no status/control/IQ authority. Rotation/revocation rechecks and closes existing subscriptions.
  • Cursor = original namespace/producer/stream/run/uint64 sequence, with an explicit inclusive-start versus after-position bit. Sequences/revisions/ACK identifiers are decimal strings, including zero/>2^53/MAX; no arithmetic wrap. Buffer-owner UUID prevents pretending pre-restart history is known. Bounded range/follow requests have independent subscription identities, cancellation and credits.
  • Capture a replay-to-live fence when accepting a subscription. Deliver all available requested records through it, then follow successors, including those published during replay. Publish epoch transitions explicitly. Missing/expired positions produce a gap and pause; a client must explicitly choose continuation, never silently jump to latest. Min/max alone never assert hole-free coverage.
  • Deliver original required configuration and boundary records as marked supplemental bootstrap before dependent frames. Supplemental delivery does not advance the forward cursor. If dependencies are unavailable, stop with an explicit reason rather than delivering deceptively self-contained frames.
  • Credits and bounded sent-but-unacknowledged metadata constrain each consumer. Validate ACKs against completed sends and that subscription's original cursor; progress ACK is not Pi durability. Lost ACK/reconnect can repeat identical canonical bytes. Same identity/different content is a conflict; different receptions with equal payloads remain different. Slow consumers cannot retain history indefinitely. Availability/expiry messages are consumer metadata, never impersonated producer records.

Implemented wire protocol

Every command is a text WebSocket message {version:1,type,payload} of at most 4,096 bytes (including fragments). Binary server messages contain this 40-byte, big-endian delivery header, then the complete unchanged canonical RFSR:

Offset Width Meaning
0 4 ASCII RFD1
4 1 Delivery major version 1
5 1 Bit 0 supplemental; all other bits zero
6 2 Reserved zero
8 4 Complete message length = 40 + canonical length
12 4 Canonical length, at most 1 MiB
16 8 Connection delivery ordinal; starts at 1, never wraps
24 16 Subscription UUID (nonzero)
40 variable Original envelope 1.0 and its original integrity trailer

The ordinal is not original sequence. Original sequence zero and MAX are valid. Text cursor identities contain namespace (1/2/3), producerUuid, streamUuid, runEpoch (each 32 hexadecimal digits), and sequence (exact unsigned decimal string). All original bytes remain immutable; wrapper and receipt/availability messages never become producer evidence.

{"version":1,"type":"auth","payload":{"credentialId":"reader-id","token":"<private token from secure storage>"}}

Reply authenticated includes owner epoch, authorized streams and durable:false. Authentication is mandatory within five seconds; no more than eight unauthenticated and 32 total record connections. Native absent-Origin is allowed only with this same authentication. No TLS validation bypass is provided. The loopback HTTP test harness is not an instruction to expose unencrypted record delivery.

{"version":1,"type":"subscribe","payload":{"subscriptionId":"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa","ownerEpoch":"<authenticated owner epoch>","streamUuid":"<authorized logical stream UUID>","start":{"mode":"cursor","cursor":{"namespace":1,"producerUuid":"<original producer UUID>","streamUuid":"<logical stream UUID>","runEpoch":"<decoder run UUID>","sequence":"9007199254740993"},"after":true},"follow":true,"credits":"65536"}}

Alternatively, explicitly select start:{mode:"oldest"} or start:{mode:"latest"}. These are deliberate choices, not a stale-resume fallback. end is an optional inclusive original cursor for a bounded range; it must reference known coverage. The endpoint rejects reversed ranges, including across decoder-run epochs and an exclusive start at its own inclusive end. subscribed returns the snapshot fence. caught-up marks completion through it exactly once; follow:true continues through subsequent publications/epochs, otherwise ends. An empty accepted snapshot is explicit even though its public fence is null: history-only completes it with zero records, including when publication arrives before selection. Follow reports initial catch-up once before later records. Known but entirely expired history (or an explicit unavailable cursor) remains a gap, not successful empty completion. Cancellation prevents new selection. epoch-boundary reports the prior tail extent as unknown. Any unavailable expected sequence/epoch/dependency (including a cursor ahead of admitted coverage) produces gap and pauses; explicitly cancel/resubscribe with a new chosen position. Unknown owner epoch cannot silently resume.

Supplemental configurations/boundaries are delivered before dependent frames, marked in the wrapper. Deduplicate by full original identity plus canonical hash, not payload-only hashes or protocol counters. Supplemental bytes are genuine original records, but do not advance the forward cursor. Retained content can be evicted while sending: the bounded protected send copy remains valid. A disconnect mid-fragment discards that incomplete message; reconnect can replay the same bytes.

{"version":1,"type":"ack","payload":{"deliverySequence":"12"}}
{"version":1,"type":"credit","payload":{"bytes":"16384"}}
{"version":1,"type":"cancel","payload":{}}

ACK is cumulative through a completed ordinal on that connection, including supplemental sends. Optional cursor must equal that delivery's original identity. Duplicate last ACK is idempotent; invented/future/mismatched ACKs are rejected. ACK releases bounded outstanding metadata, not ring retention. ACK and credit do not generate per-frame replies; authoritative status exposes both positions. Credit replenishment is explicit and capped at 4 MiB, including delivery headers. Absent negotiation, at most 32 completed deliveries remain unacknowledged. An authenticated reply may advertise deliveryLimits with version:1, maximumOutstandingDeliveries:4096 and maximumCredits:"4194304". A client can then include the integer maximumOutstandingDeliveries (1..4096) in subscribe; subscribed echoes the effective count and maximum byte credit. The count is independent of byte credit: neither limit permits exceeding the other. Omitted fields preserve the 32-delivery default, including clients using the original v1 exchange. A Server seeing no advertisement uses that legacy window and 16-record groups. The RFD1 envelope, subprotocol and canonical bytes are unchanged. The Server requests 4096 and uses groups capped at half the effective count, 2048 records, or 1 MiB canonical bytes plus one maximum-record overshoot. No consumer can pin history. Cancellation stops new work; already-written/in-flight bytes may precede its cancelled response and remain ACKable. Credential revocation prevents further application sends, but cannot retract bytes already placed in TCP/TLS buffers. Such closing allocations remain reported separately until libwebsockets releases them.

availability takes an authorized streamUuid, optional offset and limit (1..16). It returns original epoch identities, exact oldest/newest/highest-seen, bounded contiguous ranges/truncation, volatile owner epoch and unknown pre-owner history. Min/max alone never means complete coverage. Supplemental-only evidence is not promoted back to ordinary forward coverage. Full status pages are bounded; follow nextOffset for additional epochs.

Policy, protection and operational status

Default protected payload budget is 134217728 bytes aggregate, 131072 fixed record entries, 16 streams/128 epochs, 4 MiB bootstrap wire reservation within the same physical pool and no age expiry. For N known streams, physical shares are floor(payloadCells/N)*256, entry shares floor(maximumRecords/N), epoch shares floor(128/N) and bootstrap wire shares floor(bootstrapBytes/N). Expired forward control slots still own cells/index entries while referenced; shared references do not multiply their charge. No consumer pins allocations. Cells are 256 bytes; wire occupancy and physical ownership are distinct. Required indexes are allocated/protected in addition to payload capacity. Each of at most 16 producers has a bounded 1 MiB protected reassembly; authenticated readers each have one bounded 1 MiB + 40-byte protected copy, plus bounded metadata/control/TLS scratch. The negotiated window retains metadata only, never additional copies of the ring's payload. With 32 consumers the aggregate completed-delivery metadata bound is 131,072 entries; aggregate protected consumer output is at most 32 × (1 MiB + 40 bytes + LWS padding). Existing protected ring/index/producer budgets and memlock checks remain in force. Kernel socket queues do not swap and are independently finite. Output uses at most 16 KiB per writable callback; ingestion has packet/time budgets.

The service provisions LimitMEMLOCK=1G; this is an allowance, not an allocation. The default transition peak budget is 512 MiB, checked against observed process VmLck plus conservative replacement/index/scratch costs before provisioning. Physical resize uses an independently protected replacement, canonical migration and atomic configuration publication, preserving old valid state on failure. Age/bootstrap-only changes avoid payload recopying; shrinking bootstrap below current required dependencies is rejected. A physical migration may incur bounded handoff pressure; losses remain explicit, never IQ acquisition backpressure.

Process MCL_CURRENT|MCL_FUTURE, zero core limit and non-dumpability protect TLS, heap, stack and transient copies. Mapped pools additionally use MADV_DONTDUMP and MADV_DONTFORK. Gateway does not fork. Payload-bearing decoder state is created only after its isolated child starts/protects memory. There is no file/swap/dump/ log fallback. Kernel OOM/process failure can still destroy volatile history; there is no effective memory-cgroup guarantee on the qualified Pi.

/etc/rfsentry/records.d/config.json (or configured directory) is one host-only model with schemaVersion:1, decimal-string revision, policy and scoped token hashes. Updates use both expected revision and exact file digest. Manual edits require increasing revision and explicit reload. Neither policy nor credential reload changes capture settings/identities. Shipped credentials are empty. Policies and identities may persist; live records never do. policy.enabled:false rejects new admission while retained history remains readable until expiry/eviction.

Management commands on authenticated rfsentry-control-v1:

Command Required permission / result
records.capabilities Diagnostic/status-read; endpoint, modes, limits, schemas
records.configuration Status-read; public configuration without token hashes
records.status Status-read; counters, memory, bounded epoch/consumer projections
records.configure Operator job-control; policy plus expected revision/digest
records.credential.put/remove Operator job-control; scoped definition/revocation plus CAS
records.reload Operator job-control; validate/reconcile manually edited model

The existing C++ rfsentry-jobctl implements records-status [OFFSET], records-capabilities, records-configuration, records-policy FILE, records-credential FILE, records-revoke ID, and records-reload. Policy and credential commands first read the authoritative revision/digest, then CAS update. Reader-definition JSON contains id, enabled, tokenSha256, streams. Generate a random private token locally; install its hash with the operator command, never put plaintext in API configuration, URL or command-line arguments. Store the reader's separate {credentialId,token} file privately (0600), outside Git.

sudo rfsentry-jobctl records-status
sudo rfsentry-jobctl records-policy /private/policy.json
sudo rfsentry-jobctl records-credential /private/reader-definition.json
sudo rfsentry-jobctl records-revoke reader-id

The C++ reference rfsentry-record-client derives TLS host/port/certificate from existing appliance configuration, authenticates in the body, verifies C++ canonical framing/hashes, uses independent progress and prints payload-free summary evidence. --seconds, --pause-at/--pause-for, --reconnect-at, --latest, --hashes exercise bounded replay/follow. Use a process-specific memlock allowance when launching it; on a workstation, equivalent configuration points at the appliance and trusted CA. No browser/source workspace or optional server is implemented by this client.

Status counters reset with owner epoch; decoder send/drop diagnostics retain their own run epochs. Status includes exact retained/admitted/evicted/retry/conflict/loss bytes/records, partial handoffs and unknown disconnect tails, rates since owner start, allocated/occupied/bootstrap/index/process locks, applied/requested policy revisions, consumer sent/ACK original/transport positions, credits, pending/in-flight bytes, ACK age, reconnects/cancellations and faults. activeConsumers excludes unauthenticated/closing peers; allocatedConsumerConnections includes still-owned closing/TLS resources. Source send failures are authoritative per-run diagnostics; a crash before its next snapshot has an explicitly unknown tail, not a zero-loss claim. All status is bounded and excludes payloads/credentials.

bytesRetained/epoch bytes are forward wire bytes. occupiedPayloadBytes and epoch allocatedBytes count rounded allocation for all owned nodes, including dependency-only evidence; ownedRecords includes those index slots. Bootstrap bootstrapBytes is wire bytes and bootstrapAllocatedBytes is rounded cells. Per-epoch ownership sums give per-stream physical/index/bootstrap usage. Allocated pool/index capacity remains separate from occupancy; no canonical version changes. Record sizes, stream counts, age limits and losses change retention coverage; no fixed outage duration is promised. Measurements belong in the delivery record.

Qualification commands and limits

components/rfsentry-core/tools/test.sh
make -C components/rfsentry-api -j1 all test test-records test-records-sanitize
RECORDS_RECEIVE=1 python3 components/rfsentry-api/test/records_integration.py
RECORDS_BROWSER=1 python3 components/rfsentry-api/test/records_integration.py

test-records-snapshots (also required by normal/sanitizer record gates) builds an isolated gateway with a compile-time test-only selection barrier. Real loopback/verified-TLS subscriptions are accepted, synthetic records admitted, then selection released explicitly. No sleep-dependent race or production test endpoint: the deployed gateway is built without RFSENTRY_RECORDS_TEST_SCHEDULER.

The two optional test modes require prepared core receive fixtures/module and build/record-fixture / ../rfsentry-decoder/build/service-fixture. All acquisition routes in the harness are disabled; only isolated generated traffic is used. The independent OTA receive regression remains the separate full core gate. Sanitizers interpose process locking and reserve huge shadow address spaces; actual mapped-pool locking is kernel-tested through wrappers. Full process/TLS locking is separately verified in normal service execution. API LSan ptrace is incompatible with mandatory non-dumpability, so ASan/UBSan integration disables LSan only there; native ownership/pool lifetime tests and core leak checks remain. Genuine local memory64 and actual browser WS authentication are required; no remote CDP or wasm32/lowered-memory64 fallback. >4-GiB qualification remains Phase 12.

Replay is promised only while available. Neither input never decoded, lost handoff records nor history lost with the owner can be recovered by this ring. See the delivery record for source-specific qualification and its limitations.