Protected record retention and delivery¶
The gateway owns bounded volatile record retention. This is not durable Server storage; current Server acceptance remains separate.
Ownership, IPC and protection decisions¶
- Owner: existing gateway, new C++ record-manager boundary. It does not fork. Decoder-child and decoder-service restarts preserve earlier-run history; gateway/owner restart or appliance outage destroys all RAM history. A fresh buffer-owner UUID advertises that boundary. No payload-bearing checkpoints.
- Separate bounded AF_UNIX record handoff from each protected decoder child,
beside (not inside) existing JSON diagnostics/IQ handoff. Canonical bytes from
Decoder::publish, never reconstructed JSON. Same-UID sockets, bounded framed fragments/reassembly, nonblocking publication and cumulative source drops. Partial handoffs and sequence holes are explicit admission loss; a crashed producer's unobserved tail remains unknown, not invented exact coverage. - 128 MiB aggregate payload pool, independent consumers and bounded fixed indexes. Equal hard shares charge owned physical allocation, including 256-byte rounding and dependency-only nodes/entries counted once, not forward wire bytes. Bootstrap wire reservations are also shared. Reclaim from the over-budget/incoming stream or reject; never globally evict a compliant neighbor. New streams rebalance shares; empty obsolete runs may release their bootstrap. a sole stream may use the whole pool. Byte and optional monotonic-age eviction, plus index/epoch bounds. ACKs do not pin records. Original configuration/boundary records remain bounded bootstrap dependencies, not duplicate payload histories.
- Protected copies everywhere: bounded mapped/locked/non-dump/non-fork pools,
required gateway/child
mlockall(CURRENT|FUTURE)and dump protection, including OpenSSL/libwebsockets allocations and send buffers. Account indexes, bootstrap, reassembly, in-flight copies and process locks separately. Increase/verify gateway memlock provisioning, not global swap policy; kernel cgroup limits are ineffective here. Allocation/locking failure never falls back to heap/disk. Resize provisions a replacement under an explicit peak limit before publishing it; failure preserves old protected state/configuration. - No ring pointers survive asynchronous writes. Each admitted record has one authoritative immutable copy. A bounded protected send copy survives fragments; eviction cannot invalidate it. Cancellation/close zero and release it. Ingest and egress have time/byte/message budgets so controls and acquisition progress.
Delivery protocol decisions¶
/ws/records, rfsentry-records-v1, existing configured API TLS listener
(8443 default). IQ-only 8444 and QoS unchanged. Delivery framing is separately
versioned; record bodies retain exact envelope bytes/hash/original identity.
- First bounded message authenticates by credential ID/token in the encrypted message body, reusing the existing auth-message convention. No URL secrets, custom browser headers or logging. Exact configured Origin for browsers; native absent-Origin permitted only here, with the same required authentication deadline/resource limits. No anonymous record access, even with legacy auth off.
- Host-only managed definitions contain revocable token hashes and records.read scopes by stable logical stream UUID. Operator-only configuration/credential commands use the existing authenticated control listener. Status readers do not gain payload access; record readers gain no status/control/IQ authority. Rotation/revocation rechecks and closes existing subscriptions.
- Cursor = original namespace/producer/stream/run/uint64 sequence, with an explicit inclusive-start versus after-position bit. Sequences/revisions/ACK identifiers are decimal strings, including zero/>2^53/MAX; no arithmetic wrap. Buffer-owner UUID prevents pretending pre-restart history is known. Bounded range/follow requests have independent subscription identities, cancellation and credits.
- Capture a replay-to-live fence when accepting a subscription. Deliver all available requested records through it, then follow successors, including those published during replay. Publish epoch transitions explicitly. Missing/expired positions produce a gap and pause; a client must explicitly choose continuation, never silently jump to latest. Min/max alone never assert hole-free coverage.
- Deliver original required configuration and boundary records as marked supplemental bootstrap before dependent frames. Supplemental delivery does not advance the forward cursor. If dependencies are unavailable, stop with an explicit reason rather than delivering deceptively self-contained frames.
- Credits and bounded sent-but-unacknowledged metadata constrain each consumer. Validate ACKs against completed sends and that subscription's original cursor; progress ACK is not Pi durability. Lost ACK/reconnect can repeat identical canonical bytes. Same identity/different content is a conflict; different receptions with equal payloads remain different. Slow consumers cannot retain history indefinitely. Availability/expiry messages are consumer metadata, never impersonated producer records.
Implemented wire protocol¶
Every command is a text WebSocket message {version:1,type,payload} of at most
4,096 bytes (including fragments). Binary server messages contain this 40-byte,
big-endian delivery header, then the complete unchanged canonical RFSR:
| Offset | Width | Meaning |
|---|---|---|
| 0 | 4 | ASCII RFD1 |
| 4 | 1 | Delivery major version 1 |
| 5 | 1 | Bit 0 supplemental; all other bits zero |
| 6 | 2 | Reserved zero |
| 8 | 4 | Complete message length = 40 + canonical length |
| 12 | 4 | Canonical length, at most 1 MiB |
| 16 | 8 | Connection delivery ordinal; starts at 1, never wraps |
| 24 | 16 | Subscription UUID (nonzero) |
| 40 | variable | Original envelope 1.0 and its original integrity trailer |
The ordinal is not original sequence. Original sequence zero and MAX are
valid. Text cursor identities contain namespace (1/2/3), producerUuid,
streamUuid, runEpoch (each 32 hexadecimal digits), and sequence (exact
unsigned decimal string). All original bytes remain immutable; wrapper and
receipt/availability messages never become producer evidence.
{"version":1,"type":"auth","payload":{"credentialId":"reader-id","token":"<private token from secure storage>"}}
Reply authenticated includes owner epoch, authorized streams and durable:false.
Authentication is mandatory within five seconds; no more than eight unauthenticated
and 32 total record connections. Native absent-Origin is allowed only with this
same authentication. No TLS validation bypass is provided. The loopback HTTP test
harness is not an instruction to expose unencrypted record delivery.
{"version":1,"type":"subscribe","payload":{"subscriptionId":"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa","ownerEpoch":"<authenticated owner epoch>","streamUuid":"<authorized logical stream UUID>","start":{"mode":"cursor","cursor":{"namespace":1,"producerUuid":"<original producer UUID>","streamUuid":"<logical stream UUID>","runEpoch":"<decoder run UUID>","sequence":"9007199254740993"},"after":true},"follow":true,"credits":"65536"}}
Alternatively, explicitly select start:{mode:"oldest"} or start:{mode:"latest"}.
These are deliberate choices, not a stale-resume fallback. end is an optional
inclusive original cursor for a bounded range; it must reference known coverage.
The endpoint rejects reversed ranges, including across decoder-run epochs and
an exclusive start at its own inclusive end. subscribed
returns the snapshot fence. caught-up marks completion through it exactly once;
follow:true continues through subsequent publications/epochs, otherwise ends.
An empty accepted snapshot is explicit even though its public fence is null:
history-only completes it with zero records, including when publication arrives
before selection. Follow reports initial catch-up once before later records.
Known but entirely expired history (or an explicit unavailable cursor) remains
a gap, not successful empty completion. Cancellation prevents new selection.
epoch-boundary reports the prior tail extent as unknown. Any unavailable expected
sequence/epoch/dependency (including a cursor ahead of admitted coverage) produces
gap and pauses; explicitly cancel/resubscribe
with a new chosen position. Unknown owner epoch cannot silently resume.
Supplemental configurations/boundaries are delivered before dependent frames, marked in the wrapper. Deduplicate by full original identity plus canonical hash, not payload-only hashes or protocol counters. Supplemental bytes are genuine original records, but do not advance the forward cursor. Retained content can be evicted while sending: the bounded protected send copy remains valid. A disconnect mid-fragment discards that incomplete message; reconnect can replay the same bytes.
{"version":1,"type":"ack","payload":{"deliverySequence":"12"}}
{"version":1,"type":"credit","payload":{"bytes":"16384"}}
{"version":1,"type":"cancel","payload":{}}
ACK is cumulative through a completed ordinal on that connection, including
supplemental sends. Optional cursor must equal that delivery's original identity.
Duplicate last ACK is idempotent; invented/future/mismatched ACKs are rejected.
ACK releases bounded outstanding metadata, not ring retention. ACK and credit do
not generate per-frame replies; authoritative status exposes both positions.
Credit replenishment is explicit and capped at 4 MiB, including delivery headers.
Absent negotiation, at most 32 completed deliveries remain unacknowledged.
An authenticated reply may advertise deliveryLimits with version:1,
maximumOutstandingDeliveries:4096 and maximumCredits:"4194304". A client can
then include the integer maximumOutstandingDeliveries (1..4096) in subscribe;
subscribed echoes the effective count and maximum byte credit. The count is
independent of byte credit: neither limit permits exceeding the other. Omitted
fields preserve the 32-delivery default, including clients using the original
v1 exchange. A Server seeing no advertisement uses that legacy window and
16-record groups. The RFD1 envelope, subprotocol and canonical bytes are unchanged.
The Server requests 4096 and uses groups capped at half the effective count,
2048 records, or 1 MiB canonical bytes plus one maximum-record overshoot.
No consumer can pin history. Cancellation stops new work;
already-written/in-flight bytes may precede its cancelled response and remain
ACKable. Credential revocation prevents further application sends, but cannot
retract bytes already placed in TCP/TLS buffers. Such closing allocations remain
reported separately until libwebsockets releases them.
availability takes an authorized streamUuid, optional offset and limit
(1..16). It returns original epoch identities, exact oldest/newest/highest-seen,
bounded contiguous ranges/truncation, volatile owner epoch and unknown pre-owner
history. Min/max alone never means complete coverage. Supplemental-only evidence
is not promoted back to ordinary forward coverage. Full status pages are bounded;
follow nextOffset for additional epochs.
Policy, protection and operational status¶
Default protected payload budget is 134217728 bytes aggregate, 131072 fixed
record entries, 16 streams/128 epochs, 4 MiB bootstrap wire reservation within
the same physical pool and no age expiry. For N known streams, physical shares
are floor(payloadCells/N)*256, entry shares floor(maximumRecords/N), epoch
shares floor(128/N) and bootstrap wire shares floor(bootstrapBytes/N).
Expired forward control slots still own cells/index entries while referenced;
shared references do not multiply their charge. No consumer pins allocations.
Cells are 256 bytes; wire occupancy and physical ownership are distinct. Required
indexes are allocated/protected in addition to payload capacity. Each of at most
16 producers has a bounded 1 MiB protected reassembly; authenticated readers each
have one bounded 1 MiB + 40-byte protected copy, plus bounded metadata/control/TLS
scratch. The negotiated window retains metadata only, never additional copies
of the ring's payload. With 32 consumers the aggregate completed-delivery metadata
bound is 131,072 entries; aggregate protected consumer output is at most
32 × (1 MiB + 40 bytes + LWS padding). Existing protected ring/index/producer
budgets and memlock checks remain in force. Kernel socket queues do not swap and are independently finite. Output
uses at most 16 KiB per writable callback; ingestion has packet/time budgets.
The service provisions LimitMEMLOCK=1G; this is an allowance, not an allocation.
The default transition peak budget is 512 MiB, checked against observed process
VmLck plus conservative replacement/index/scratch costs before provisioning.
Physical resize uses an independently protected replacement, canonical migration
and atomic configuration publication, preserving old valid state on failure.
Age/bootstrap-only changes avoid payload recopying; shrinking bootstrap below
current required dependencies is rejected. A physical migration may incur bounded
handoff pressure; losses remain explicit, never IQ acquisition backpressure.
Process MCL_CURRENT|MCL_FUTURE, zero core limit and non-dumpability protect TLS,
heap, stack and transient copies. Mapped pools additionally use MADV_DONTDUMP
and MADV_DONTFORK. Gateway does not fork. Payload-bearing decoder state is created
only after its isolated child starts/protects memory. There is no file/swap/dump/
log fallback. Kernel OOM/process failure can still destroy volatile history;
there is no effective memory-cgroup guarantee on the qualified Pi.
/etc/rfsentry/records.d/config.json (or configured directory) is one host-only
model with schemaVersion:1, decimal-string revision, policy and scoped token
hashes. Updates use both expected revision and exact file digest. Manual edits
require increasing revision and explicit reload. Neither policy nor credential
reload changes capture settings/identities. Shipped credentials are empty. Policies
and identities may persist; live records never do. policy.enabled:false rejects
new admission while retained history remains readable until expiry/eviction.
Management commands on authenticated rfsentry-control-v1:
| Command | Required permission / result |
|---|---|
records.capabilities |
Diagnostic/status-read; endpoint, modes, limits, schemas |
records.configuration |
Status-read; public configuration without token hashes |
records.status |
Status-read; counters, memory, bounded epoch/consumer projections |
records.configure |
Operator job-control; policy plus expected revision/digest |
records.credential.put/remove |
Operator job-control; scoped definition/revocation plus CAS |
records.reload |
Operator job-control; validate/reconcile manually edited model |
The existing C++ rfsentry-jobctl implements records-status [OFFSET],
records-capabilities, records-configuration, records-policy FILE,
records-credential FILE, records-revoke ID, and records-reload. Policy and
credential commands first read the authoritative revision/digest, then CAS update.
Reader-definition JSON contains id, enabled, tokenSha256, streams. Generate
a random private token locally; install its hash with the operator command, never
put plaintext in API configuration, URL or command-line arguments. Store the reader's
separate {credentialId,token} file privately (0600), outside Git.
sudo rfsentry-jobctl records-status
sudo rfsentry-jobctl records-policy /private/policy.json
sudo rfsentry-jobctl records-credential /private/reader-definition.json
sudo rfsentry-jobctl records-revoke reader-id
The C++ reference rfsentry-record-client derives TLS host/port/certificate from
existing appliance configuration, authenticates in the body, verifies C++ canonical
framing/hashes, uses independent progress and prints payload-free summary evidence.
--seconds, --pause-at/--pause-for, --reconnect-at, --latest, --hashes exercise
bounded replay/follow. Use a process-specific memlock allowance when launching it;
on a workstation, equivalent configuration points at the appliance and trusted CA.
No browser/source workspace or optional server is implemented by this client.
Status counters reset with owner epoch; decoder send/drop diagnostics retain their
own run epochs. Status includes exact retained/admitted/evicted/retry/conflict/loss
bytes/records, partial handoffs and unknown disconnect tails, rates since owner
start, allocated/occupied/bootstrap/index/process locks, applied/requested policy
revisions, consumer sent/ACK original/transport positions, credits, pending/in-flight
bytes, ACK age, reconnects/cancellations and faults. activeConsumers excludes
unauthenticated/closing peers; allocatedConsumerConnections includes still-owned
closing/TLS resources. Source send failures are authoritative per-run diagnostics;
a crash before its next snapshot has an explicitly unknown tail, not a zero-loss
claim. All status is bounded and excludes payloads/credentials.
bytesRetained/epoch bytes are forward wire bytes. occupiedPayloadBytes
and epoch allocatedBytes count rounded allocation for all owned nodes, including
dependency-only evidence; ownedRecords includes those index slots. Bootstrap
bootstrapBytes is wire bytes and bootstrapAllocatedBytes is rounded cells.
Per-epoch ownership sums give per-stream physical/index/bootstrap usage. Allocated
pool/index capacity remains separate from occupancy; no canonical version changes.
Record sizes, stream counts, age limits and losses change retention coverage;
no fixed outage duration is promised. Measurements belong in the
delivery record.
Qualification commands and limits¶
components/rfsentry-core/tools/test.sh
make -C components/rfsentry-api -j1 all test test-records test-records-sanitize
RECORDS_RECEIVE=1 python3 components/rfsentry-api/test/records_integration.py
RECORDS_BROWSER=1 python3 components/rfsentry-api/test/records_integration.py
test-records-snapshots (also required by normal/sanitizer record gates) builds
an isolated gateway with a compile-time test-only selection barrier. Real
loopback/verified-TLS subscriptions are accepted, synthetic records admitted,
then selection released explicitly. No sleep-dependent race or production test
endpoint: the deployed gateway is built without RFSENTRY_RECORDS_TEST_SCHEDULER.
The two optional test modes require prepared core receive fixtures/module and
build/record-fixture / ../rfsentry-decoder/build/service-fixture. All acquisition
routes in the harness are disabled; only isolated generated traffic is used.
The independent OTA receive regression remains the separate full core gate.
Sanitizers interpose process locking and reserve huge shadow address spaces;
actual mapped-pool locking is kernel-tested through wrappers. Full process/TLS
locking is separately verified in normal service execution. API LSan ptrace is
incompatible with mandatory non-dumpability, so ASan/UBSan integration disables
LSan only there; native ownership/pool lifetime tests and core leak checks remain.
Genuine local memory64 and actual browser WS authentication are required; no remote
CDP or wasm32/lowered-memory64 fallback. >4-GiB qualification remains Phase 12.
Replay is promised only while available. Neither input never decoded, lost handoff records nor history lost with the owner can be recovered by this ring. See the delivery record for source-specific qualification and its limitations.