Skip to content

Decoded-record archive contract (segment format 1, index schema 2)

The archive preserves canonical RFSR 1.0 envelopes, including producer identity, timing, configuration, validation evidence and payload. A GOES HRIT data record contains the 892-byte payload after Reed–Solomon decoding. The separately enabled product processor reconstructs files and imagery from these records. The Server roadmap tracks acceptance and the open archive performance gate.

Subscription and commit

Each explicitly enabled stream has its own verified-TLS record connection, subscription UUID, completed delivery ordinal, credit and reconnect state. CA/hostname verification, TLS 1.2 minimum, the registered leaf pin and exact rfsentry-records-v1 subprotocol precede authentication. The appliance decides actual grant scope; narrow credentials retain their stream restrictions. Disabling a stream or changing its registration closes its connection. Saving ingestion controls is an explicit per-stream retry/reconnect decision, including when the saved policy is unchanged after external grant recovery. Status polling/remounting the React view never performs that action.

Up to eight subscriptions per instance share one supervised archive child. A negotiated source permits 4096 unacknowledged deliveries and 4 MiB of wire credit. Groups normally reach 1 MiB canonical bytes (one maximum-record overshoot permitted), with a 2048-record ceiling. There is no default age timer; an optional maximum dirty age can be configured separately. With an older appliance, the effective window remains 32 and groups stop at 16; the Server never waits for an unattainable larger count. Binary RFD1 and original canonical envelopes on the network are unchanged; optional version-1 JSON limit negotiation is described in record delivery.

Each stream receives/validates into one batch while one earlier batch commits. Validated immutable records are also immediately available through an instance-owned RAM tier, with 8 MiB charged allocation per active stream and 64 MiB aggregate, including conservative entry/index accounting. Pending records remain unevictable, and durable records are evicted first. Necessary partial flushes report their reason: ordered boundaries, legacy credit/count limits, cache pressure, explicit drain or shutdown. Low-volume browser reads do not wait for persistence and never trigger a commit. Reading pauses at the next batch's bound; total received but uncommitted wire bytes cannot exceed 4 MiB. Gaps, epoch boundaries and the initial caught-up fence are ordered barriers behind earlier records. Late completions retain their original connection/subscription and configuration binding; they cannot ACK a replacement connection or a revoked configuration. Only completed durable groups return ACK/credit. Accepted RAM records survive connection replacement and can finish persistence under their original storage binding without ACKing a new connection. Source deliveries not yet validated/admitted are replayed.

Staged entries share immutable canonical bytes with the cache. Persistence flattens one bounded worker batch when a flush is due; each batch is at most 40 + 2048*16 + 1 MiB + maximumRecordBytes bytes. At most 16 storage requests can be pending, including retiring generations. Source receive/validation buffers and the 4-MiB unACKed wire credit remain separately bounded. There is no unbounded queue of canonical records. The worker reuses one at-most- 5-MiB input frame and a bounded append buffer; decoding, segment append frames and read responses retain their separate byte/count limits. These are buffer bounds, not a claim about total RSS (TLS, SQLite and allocator overhead are separate).

Internal commit IPC accepts RFB1: four magic/version bytes, a big-endian u32 count, 16-byte stream and owner UUIDs, then records with a big-endian u32 length, one supplemental byte, three reserved zero bytes and exact canonical bytes. New RFB2 batches add an eight-byte big-endian first-Server-receipt UTC second before each canonical envelope. Deferring persistence therefore does not reset retention age. Both formats preserve exact canonical bytes. The outer big-endian u32 length remains bounded. The isolated child independently validates framing and canonical integrity. JSON remains for small commands and bounded read responses; canonical commit bytes are never hex-expanded into JSON.

Worker requests use a FIFO with at most 16 active/waiting requests. At most two bulk archive reads and one metadata reader may occupy it, leaving HTTP capacity for administration during a storage stall. Available-cache reads and known catalog/capability metadata do not take disk-worker turns. Excess requests receive archive-queue-full; ordinary contention queues fairly instead of returning archive-busy or reconnecting ingestion. Queue occupancy is exposed in archiveQueue status. Configuration snapshots are fully validated per generation and invalidated by file identity, permissions, size and nanosecond change/modification times, including same-generation manual edits and external CLI updates. Storage selection and mount/root validation still apply to queued work and durable completion.

The worker has a five-second I/O inactivity deadline, renewed only by successful byte transfer or a completed-work progress frame. Startup recovery, indexed retention, active SQLite VM work and completed SQLite VFS I/O emit rate-limited progress frames; these never indicate durable completion or authorize an ACK. A progressing exchange can exceed five seconds in total. A blocked storage operation emits no progress and retains the five-second limit. A timed-out child is killed and reaped before replacement. An uninterruptible child blocks replacement; a local state lock also prevents a second writer after a service restart. The child receives SIGKILL when its parent dies; a blocked syscall retains the lock until the kernel releases it. A maintenance turn examines at most 256 data/context candidates and unlinks at most 16 segments, then rejoins the FIFO behind queued ingestion/readers. Dependency counts and context candidates use matching indexes; pending RAM dependencies are pinned and retired cache entries are invalidated.

The successful commit response follows this order:

  1. Decode and validate the exact canonical envelopes using the shared C++ core; verify source/stream and required configuration/boundary dependencies.
  2. Append checksummed frames through held, validated, no-follow descriptors. Reuse descriptors within the group, prepared SQLite statements across groups, and immutable decoded records within the worker. Batch writes per segment; context segments precede dependent data even when their IDs are newer.
  3. Synchronize every changed segment. New segment creation synchronizes its directory; appending an existing segment does not repeat unchanged-directory synchronization. SQLite retains ownership of WAL synchronization.
  4. Commit record indexes, dependencies, forward coverage and resume state in one SQLite WAL transaction with synchronous=FULL.
  5. Revalidate the archive binding, then ACK the group's last completed connection delivery ordinal and return consumed byte credit.

Supplemental records are genuine originals, deduplicated and retained before their dependents; their deliveries never advance the forward cursor. Resume uses the original (namespace, producer, stream, run, uint64 sequence) plus volatile source owner epoch. Connection ordinals/UUIDs are fresh on reconnect. Counters/cursors at the HTTP/browser boundary are decimal strings. SQLite stores original sequence/revision values as fixed-width big-endian bytes, preserving the entire uint64 range and unsigned ordering independently of signed internal row IDs. Original producer/stream/run tuples are dictionary entries; private compact references never cross the public record protocol.

Owner loss, unavailable history/dependencies and run boundaries are persisted as explicit gaps. The default continuation policy pauses. Selecting available deliberately chooses available original history and records the choice without claiming missing records arrived. There is no automatic latest fallback. An explicitly selected empty available snapshot follows future records from oldest history, while preserving the prior durable cursor and recorded loss. If the requested run is entirely unavailable, the explicit policy may choose the oldest advertised history with that uncertainty recorded. Availability pagination is limited to the protocol's 128 epochs and strictly increasing offsets. Known expired forward history with no remaining ranges is a distinct condition: the subscriber backs off until records become available, preserving its cursor and recorded gap. It does not turn that condition into a new empty-history snapshot. Both processing intentions use this archiving path; processing cannot hold ACKs.

Files, index and recovery

The explicitly bound archive directory contains a private .rfsentry-archive identity document and segment-<16 hex digits>.rfs files. The identity binds an archive UUID/storage generation to the instance, filesystem and root inode. Another root gets another index/cursor namespace. Copying an identity document to an unrelated root is unsupported and rejected by the physical binding. Current-boot mount identity is also remembered in local state; a remount within that boot requires explicit rebinding, including after helper/service restart.

New segments target 64 MiB, with separate data channels per stream and shared context segments. Format-1 segment bytes are unchanged; index schema compatibility is separate and legacy schema-1 databases are explicitly refused by the schema-2 writer. There is no file per record. All multibyte fields below are big endian; reserved bytes are zero. Each file and frame carries SHA-256 integrity checks.

Segment offset Bytes Meaning
0 8 RFSSEG01, segment format 1
8 16 Archive UUID
24 8 Local positive segment identifier
32 16 Stream UUID; zero denotes the context channel
48 8 Server creation UTC seconds
56 8 Reserved
64 32 SHA-256 of preceding 64 bytes
Frame offset Bytes Meaning
0 4 RFA1
4 2 Frame version 1
6 2 Supplemental flag, 0 or 1
8 4 Complete frame length
12 4 Canonical envelope length
16 8 First Server receipt UTC seconds
24 16 Source buffer-owner epoch
40 8 Reserved
48 variable Exact canonical RFSR bytes, including original trailer
end − 32 32 SHA-256 of preceding complete frame bytes

Schema-2 SQLite indexes/WAL (PRAGMA user_version=2) live under local operational state in archives/<archiveUuid>/index.sqlite, separate from segment storage. The index records identities/content hashes, exact offsets, dependencies, committed ends, forward cursors, retained/deleted state and explicit gaps. A fixed page cache and streamed recovery avoid whole-archive RAM hydration. Runtime index-file safety checks use anchored, no-follow metadata inspection. They must never independently open/close SQLite's database, WAL or shared-memory files in the owning process: those closes release SQLite's POSIX locks and can let another reader reset a still-mapped WAL index. SQLite owns these descriptor lifetimes; ownership, mode, hard-link and directory-binding checks remain required. New databases use incremental auto-vacuum metadata from their first table. There is no automatic migration or VACUUM of a legacy database. Schema/version and column encodings are inspected read-only before writable recovery. Old binaries reject the new user_version; incomplete or incompatible layouts stop. The separate WAL checkpoint trigger defaults to 16 MiB and is configurable; it neither caps WAL size nor changes FULL commit durability. Explicit metadata/WAL/ temporary/physical budgets, per-filesystem reserves and in-flight admission are documented in the operator chapter.

Live records use stable AUTOINCREMENT ordinals, binary canonical SHA-256 values, compact original keys, and integer dependency edges. Exact retired records live in a separate WITHOUT ROWID table. They retain original hash and forward status, epoch association and normalized policy, plus up to three stable dependency ordinals so later quarantine still prevents ACKs. Physical extents survive while a mixed segment exists. After durable unlink, bounded cleanup removes extent bindings and obsolete segment rows; exact completed segment ranges and the AUTOINCREMENT high-water prevent ID reuse or resurrection. Configurations and owners are reclaimed only after their references disappear. Exact retired hash evidence remains charged and grows with lifetime receptions; no constant-space claim or implicit guarantee expiry is made.

Normal startup checks critical index/schema and committed boundaries, recovers context before data (including rotated context segments), synchronizes valid unindexed tails and commits them idempotently. Uncommitted partial tails/header creations are discarded safely. Uniquely indexed committed record damage is handled by the quarantine contract below. Committed truncation, damaged segment headers, index ambiguity and I/O failures remain hard faults. Data is never silently truncated to conceal a checksum failure. Same identity/same bytes is a retry; different content is an integrity fault. Same payload at different identities remains separate receptions. Lost ACKs are harmless. Full historical payload verification is a bounded background traversal of stable live ordinals. The ordinary SQLite integrity attempt yields after its startup work/read allowance or WAL pressure; deferred is not verified. Full offline checks are explicit.

Record quarantine and continued availability

A repeatable record framing/checksum/canonical-validation failure can be isolated when the trusted index identifies exactly one committed physical extent. The position index checks its owner, bounds and neighboring extents. Ambiguous or overlapping index rows, changing reads, missing/truncated committed bytes and invalid unindexed tails do not become fabricated gaps.

Quarantine preserves the original identity and expected canonical hash. A FULL SQLite transaction records the observed hashes, failure reason, segment/offset, detection time, explicit gap and damaged-frame evidence, then marks that record unavailable. records.deleted=2 remains distinct from exact policy-deletion entries in retired_records; normal retained rows remain 0. Schema-2 accounting and compact references preserve required quarantine dependencies. Evidence is capped at 64 MiB per archive, including an honestly labeled partial excerpt when the remaining allowance is insufficient. Existing segment bytes are preserved. Ordinary retention removes wholly retired segments; quarantined evidence and its required segments remain charged until explicit handling. Retired children retain bounded dependency ordinals so a context quarantined later still prevents a duplicate-delivery ACK.

Only after that transaction commits does the worker publish a notice bound to archive and storage identity. A cached copy cannot hide the new disk fault or authorize a delayed ACK. Pending accepted input remains owned. Healthy records, other consumers and new valid commits continue. A damaged configuration or boundary remains unavailable to its dependents; it is not replaced with a different original context. Known record/request rejections keep the helper alive; real storage/index failures retain worker termination and recovery. Commit validation occurs before creating new segment files, so a rejected group cannot strand a header that forces unrelated work through cold recovery.

Startup still verifies retained committed history. Known quarantine entries are skipped only using their persistent, validated physical bindings; they do not create duplicate incidents. Unchanged committed segments need no new write transaction or synchronization. Valid unindexed tails retain synchronization before index/cursor commit, including recovery of header-only segments.

Coverage reports quarantined; diagnostic reads report unavailable with reason record-quarantined. Binary reads preserve the existing explicit gap policy: pause at the gap, or continue healthy originals with ordered gap evidence. Quarantined dependencies produce required-dependency-unavailable. No damaged canonical bytes are delivered for reconstruction. Archive status reports quarantinedRecords, quarantineEvidenceBytes and the evidence limit; stream gaps retain identity and validation details. Historical browser backfill requests continuation explicitly, separately from its live/fallback policy.

Quarantine is loss containment, not a claim that missing data was recovered or that hardware is healthy. Exact repair requires verified original bytes; hashes must not be recomputed merely to bless altered data. Older executables retain their strict startup rejection of damaged segment bytes, so downgrading does not bypass this integrity protection.

Loss of the entire local index/retention journal is reported as archive-index-lost. Segments alone cannot prove intentional prior deletions; automatic rebuilding would risk resurrecting expired history. Preserve both archive and local state in host backups. Recovery from a physically lost index needs an explicit future operator recovery procedure, not a cursor reset or automatic overwrite.

Retention and reads

Retention is independently enabled per stream. Age uses first Server receipt UTC seconds, not source observation/capture/decode time; host wall-clock changes therefore affect age expiration. Byte limits count canonical retained bytes. There is no default deletion limit or upgrade activation.

Deletion first commits identity/hash tombstones and segment deletion intent. It then unlinks wholly retired segments, synchronizes the directory and finishes bookkeeping. Interrupted deletion resumes idempotently. Required dependencies and the latest context for a live run survive. Partial segment reclamation has up to one 64-MiB data-segment slack per stream and context-segment slack, plus pinned dependencies/framing overhead. Physical segment usage and retained canonical bytes are distinct. Tombstones remain in the local index, so old retries do not silently restore intentionally expired history.

Authenticated administrators and read-only principals can make bounded reads; mutating ingestion/retention uses administrator role, CSRF and revision/digest checks. Examples (all UUIDs must be configured/real identities):

GET /api/v1/archive/coverage?streamUuid=<32hex>&limit=16&after=<optional returned key>
GET /api/v1/archive/records?namespace=1&producerUuid=<32hex>&streamUuid=<32hex>&runEpoch=<32hex>&first=0&last=18446744073709551615&limit=16

Coverage returns bounded contiguous intervals with retained-forward, retained-supplemental or retention-deleted states, explicit gaps and a continuation key. Unlisted history is unknown/unavailable; min/max is not proof of continuity. Record ranges return at most 64 entries / 2 MiB canonical bytes, with exact hexadecimal envelopes, identities/hashes/dependency identities, explicit deleted entries and a next cursor. A missing record is never represented by a fabricated envelope. Storage faults/corruption return explicit errors. Archive range reads require the selected binding to be write-ready for recovery; full/read-only volumes return the storage fault until restored, rather than an empty range or a claim that inaccessible records are available. Administration and cached operational status remain responsive during that fault. Use these diagnostic endpoints to verify archived bytes. The browser uses /api/v1/records/history and /api/v1/records/batch to read from a selected Server stream.

Filesystem assumptions and qualification limits

The filesystem, kernel, device/controller and any server-side storage backend must honor successful file/directory fsync, ordered creation and atomic SQLite WAL commits. SQLite documents FULL WAL synchronization and its local-storage requirement: synchronous, WAL. A writable probe or passing SIGKILL test does not establish NAS/controller power-loss behavior. Backups and underlying hardware durability remain host responsibilities.

No fallback path is allowed when storage is missing/replaced/unwritable/full. Stalls stop ACK advancement and bounded delivery work; acquisition and other consumers remain independent. Qualification results and incomplete cases are recorded in the historical Phase 6 report and accepted-source evidence and limits. Phase 6 performance acceptance is separate; Server Phase 8/narrow Phase 9 acceptance does not alter the durability-before-ACK contract.

Phase 7 browser record access

Browser record access derives from the endpoint's permitted local session or normal username/password session. A configured no-auth endpoint requires no operator credential entry. Restricted machine tokens remain separate. Dedicated bearer tokens with role records.read carry an explicit list of up to 128 stream UUIDs. They cannot access administration, credentials or unrelated archive endpoints. Browser requests require an explicitly trusted TLS endpoint and an authorized exact Origin. Session onboarding can authorize that origin without changing the running listener. Preflight is an empty-body 204.

  • GET /api/v1/records/capabilities returns version/format, archive UUID, scoped streams and effective page bounds.
  • GET /api/v1/records/coverage?streamUuid=...&limit=...&after=... retains the existing coverage semantics within the token's scope.
  • GET /api/v1/records/batch?streamUuid=...&start=oldest&limit=512 starts bounded forward reading. Subsequent requests use the original namespace, producerUuid, runEpoch and decimal-string sequence, plus archiveUuid and the returned fixed fence token while catching up. allowGaps=true is an explicit continuation policy; the default pauses at missing history.

The content type is application/vnd.rfsentry.record-batch.v1. RFSB v1 uses big-endian integers: 4-byte RFSB magic, u16 version 1, u16 reserved zero, u32 metadata length, u32 entry count; then bounded UTF-8 JSON metadata and entries. Each entry is u32 canonical length, u8 supplemental flag, three reserved zero bytes, and exact canonical RFSR bytes. No hex expansion occurs. Metadata is at most 64 KiB; entries including their headers total at most 4 MiB. A page admits at most 512 forward records and 1,536 total envelopes. Original dependency closure can include both a new configuration and an older boundary's configuration; when necessary, a page ends earlier to keep the envelope limit. Existing worker framing remains capped at 5 MiB; binary read results use bounded CBOR inside the isolated worker channel.

Metadata binds archive/stream identity, original cursor/resume, the captured fence, caught-up state, ordered gap/epoch events and any paused gap. The requested original cursor's canonical hash permits identity-bound return from another source. Unknown cursors are reported as cursor-not-archived, never silently advanced. Existing epoch admission rank and unsigned sequence order govern reads; run UUIDs are not sorted as timestamps. Context is delivered before dependents without claiming new forward progress. Each read reuses segment/index integrity checks and the existing mount, recovery and fair-admission protections.

Capabilities also advertise an optional available version-2 format and cache generation. Only requests with view=available&generation=<UUID> receive application/vnd.rfsentry.record-batch.v2. RFSB v2 keeps the v1 header layout; entry byte 5 becomes 0 for durable and 1 for volatile, with bytes 6–7 still zero. Metadata adds availableGeneration, availableCursor, durableCursor and archiveState. Page durable is the conjunction of entry durability. Available pages preserve the same original cursor/hash and logical fence while crossing the durable-history/RAM boundary. Coverage and unextended batch reads stay durable-only; cached input cannot be mislabeled as archived durability.

Cache generation replacement reopens from browser-local recoverable progress. awaiting-source-cursor preserves that position while the Server recovers or replays an unACKed volatile tail. Blocked/disabled sources retain explicit unavailable-history reporting. No automatic cursor rewind or jump to live is permitted. POST /api/v1/ingestion/drain is an administrator/CSRF-protected asynchronous request, not a read-side effect or an activation request. Clean shutdown drains within the existing service stop budget; an incomplete drain is logged, never ACKed, and requires original-source recovery. The packaged unit uses KillMode=mixed: SIGTERM goes to the main process so its archive helper remains available for the drain. The unchanged 15-second stop deadline still forces termination of the entire service cgroup.

A captured fence makes the catch-up/follow transition race-safe: later arrivals are read after that fence on the next request, without skipping history. Browser ACKs do not certify Server durability. The durable view serves archived canonical records, and the explicitly negotiated available view may include validated RAM records. Neither performs native Server product reconstruction.

Dated history and binary version 3

Capabilities advertise history.version=1 and binary version 3. The scoped route GET /api/v1/records/history?streamUuid=...&archiveUuid=...&fromUtc=...&beforeUtc=...&limit=64 pages backwards by first Server receipt. UTC bounds are decimal seconds. The next-page marker adds before=<original-identity-hex> with the returned receipt second as beforeUtc; the tuple is exclusive. Initial date selection includes the specified second. Each request examines at most 4,096 rows using the existing record_retention index and emits at most 64 actual consecutive original ranges. Supplemental-only context does not establish forward coverage. Ranges split at epoch/sequence holes and 30-minute receipt buckets. next describes remaining indexed work; unlisted history is not proof of absence outside the requested page. No new index migration or whole-archive sort is required.

The response binds archive/stream identity, a durable snapshot fence and its canonical hash. Each range supplies its first original cursor, last sequence, record/byte counts and first/last receipt seconds. The browser uses these bounds for newest-first planning, then reads each chosen range forwards. Planning and reads retain normal scoped authorization, fair worker admission, mount checks and archive integrity validation.

GET /api/v1/records/batch?...&version=3 explicitly negotiates application/vnd.rfsentry.record-batch.v3. Its 16-byte RFSB header retains the existing layout with version 3. Each entry has the v2 eight-byte prefix followed by a big-endian u64 first-Server-receipt UTC second, then exact canonical bytes. Receipt time is transport metadata and does not modify original canonical hashes. The existing 4-MiB/512-forward/1,536-envelope/64-KiB-metadata bounds still apply, including the larger entry header. Available reads also require the existing view=available&generation=... negotiation. Unextended v1/v2 requests are unchanged.

Version 3 additionally accepts first=<original-identity-hex> for an inclusive start, mutually exclusive with a resume cursor. This avoids synthesizing a predecessor at sequence zero or losing precision at uint64 maximum. A fixed fence bounds the historical snapshot; required supplemental dependencies still precede forward originals. A missing inclusive start reports first-not-archived. Confirmed tombstones/gaps follow the explicit available-history policy. Neither history planning nor reading forces pending records durable or advances ACKs.