Optional rfSentry Server¶
Use rfSentry Server to keep decoded records after the appliance’s RAM buffer rolls over. Install it on a separate Linux host, choose a storage volume, and enable the streams you want to archive. You can also enable GOES product processing and open the resulting imagery and messages in the browser. The Pi and browser work independently of the Server.
Archiving and browser consumption¶
The administration foundation provides an HTTPS/static GUI, independent identity, private configuration/credentials, explicit archive-root readiness, appliance registrations and disabled directed peers. Phase 6 adds verified-TLS resumable ingestion, durable exact decoded-record archiving, explicit retention and bounded coverage/readback APIs. The single administration UI uses React + TypeScript, compiled into static assets served by C++.
Archive only is the default mode for a stream. After pairing, enable ingestion for each stream to start receiving records. Retention and product processing each require a separate choice. See optional Server processing and choosing a browser source. The acceptance record covers native GOES reconstruction and browser consumption. Sustained archive performance remains an open gate.
Initial packages target Debian 13 amd64 and arm64, configurable HTTPS 8445.
They do not install or depend on the appliance, SDR/USB/Chrony/QoS services,
workstation utility, compiler, frontend toolchain or another application backend.
Do not run the appliance install.sh to install this component.
Install and enroll on a trusted LAN¶
On the separate intended server host:
sudo apt-get install ./rfsentry-server_amd64.deb # choose arm64 when applicable
sudo rfsentry-serverctl --instance default validate
sudo rfsentry-serverctl --instance default trust
A genuinely fresh instance generates a provisional private HTTPS authority/listener in C++
and listens on 0.0.0.0:8445, with anonymous full administration and no
default password. This is the selected trusted-LAN convenience policy, not safe
Internet exposure. Archive/appliance configuration remains unset. Existing
instances preserve their listener/authentication through migration/upgrades.
Fresh identity selection uses the system FQDN when available, otherwise the
actual hostname. The default-route IPv4 address is retained as an administration
alias. With an FQDN available, new defaults omit the unqualified short hostname
because it cannot satisfy the appliance's DNS suffix policy. Existing aliases
are preserved through upgrades and require explicit operator review.
Certificates and exact Host/Origin checks cover the
same names; no .local suffix is guessed.
Before pairing, the CLI reports the canonical URL and provisional public
CA/fingerprint. Obtain that fingerprint through trusted host access and import
the public CA only when the instance must remain independent. Pairing with a
private-CA appliance replaces the provisional listener leaf with one signed by
that appliance. Clients which installed the appliance root from its /trust/
page then trust the appliance and every Server it approves. No ACME or manual
certificate generation is needed. Automatic HTTPS cannot establish secure
first-contact trust by itself; do not use certificate-ignore, sandbox or
web-security bypasses.
After upgrading an installation with an old guessed hostname, deliberately refresh its detected identities and restart only this Server instance:
sudo rfsentry-serverctl --instance default identity-refresh
sudo rfsentry-serverctl --instance default trust
sudo systemctl restart rfsentry-server@default.service
This reuses the existing private authority and preserves UUID, bind/port, authentication, pairing, stream enablement, storage binding and archived records. It replaces the canonical address/alias list with current detected identities; review the reported URLs and explicitly add any custom aliases afterward. Operator-provided certificates are refused unchanged. Upgrades do not otherwise change existing addresses, and DHCP/DNS changes require a deliberate refresh. FQDN lookup is bounded to two seconds. Default-route IPv4 selection uses the lowest-metric main-table route; policy routing/ECMP or unusual multiple addresses require explicit alias selection. IPv6 listening remains explicit.
Open the GUI, choose Appliances, and select a discovered appliance or enter its canonical HTTPS address. On that appliance, run:
Enter the six-character code and click Connect securely. One code enrolls this server with full appliance API, diagnostics, controls, raw IQ and all present/future decoded streams—not one code per job. PAKE mutual confirmation binds identities, full grant, canonical endpoint, the Server CSR/enrollment public key and the actual TLS connection. The Server generates and keeps both private keys locally; the appliance receives only the CSR, public enrollment key and proof. Afterward normal CA/hostname/certificate verification protects appliance authentication. No private CA key/global operator token is copied. Pairing does not start reception or activate an inactive ingestion policy. Existing enabled policies survive replacement of their credential. Bind storage and explicitly enable streams next.
Review Server certificate names above the Connect button. Every name must
match the appliance's configured DNS suffixes or IP ranges. A name such as
archive does not match a suffix of example.net; use archive.example.net
or an approved private IP, or review the issuance policy explicitly. Issuance
rejection identifies the cause and leaves the old registration unchanged.
Correct the problem and retry with a valid code. A confirmed rejection is
labeled Enrollment rejected before commit and cannot be resumed as a lost
reply. An uncertain/committed lost reply retains its private pending credential
and offers Resume verified connection, without another code.
Discovery is C++ Avahi/DNS-SD on the Server's local link. Candidates remain
unverified until pairing; there is no network scan/automatic trust. If multicast
is blocked or Avahi unavailable, manual HTTPS entry works. Configure discovery
interfaces/enablement in secondary settings; appliance advertisement is derived
from appliance.conf rather than another identity file.
Enable authentication with the first account in Instance settings, or host-only
sudo rfsentry-serverctl require-auth operator (password on stdin). Host-only
bootstrap instead enables authentication and prints a private 15-minute first-admin
token; enter it in setup. No login is required in the fresh LAN default.
TLS/origin/listener/discovery changes are saved, restart required; status distinguishes
saved versus actual running listener. No web operation launches privileged commands.
Use normal client trust, not certificate-ignore/security flags. Additional exact
HTTPS Origins are an explicit CORS allowlist, separate from Administration
aliases, which authorize this service's hostnames/IPs. Every alias must be
covered by the certificate; choose automatic private HTTPS when changing names.
tls-auto ORIGIN BIND [ALIAS...] is the corresponding host command. Cookie
mutations require CSRF and an allowed Origin. Cookies are host-only; signing in
at one address does not sign in at another. Read-only users cannot save, probe
storage or contact appliances.
Host-only recover-admin USER reads a new password from stdin and revokes sessions
for that user. Restart clears all RAM cookie sessions.
Appliance-issued leaves are valid for 90 days. The per-instance renewal timer checks twice daily and renews with 30 days remaining. Renewal is authenticated by the persistent enrollment signing key and full-appliance registration, not the current Server leaf, so reconnecting after that leaf expired can still recover automatically. SIGHUP installs the renewed certificate for new TLS connections while established sessions remain intact. A lost enrollment key, revoked registration or expanded hostname set requires a new six-character code. If the expired GUI cannot be opened, run:
printf '%s\n' 'SIXCODE' | sudo rfsentry-serverctl --instance default \
certificate-recover https://appliance.example:8443
The code is read from standard input rather than placed in the process list. Activate a host-CLI recovery or renewal with:
This signals only the main Server process, preserving established connections and archive workers. The renewal service uses the same reload action and skips stopped instances. Listener address, port and alias changes still require a planned restart; certificate-only replacement does not. Issuer rotation under the same appliance root requires no client change; root replacement does.
Starting an enabled Server instance also starts its matching renewal timer. Disabling or stopping an instance does not cause package upgrades to re-enable that Server; a later explicit start restores its timer dependency.
Independent instances and private roots¶
Each NAME owns account rfsentry-srv-NAME, UUID, port and private directories:
/etc/rfsentry-server/instances/NAME: managed configuration and previous pointer./var/lib/rfsentry-server/NAME: identity/credential generations and local operational/WAL/index state, separate from external archives.- An explicitly selected archive root; never an automatic fallback directory.
Private directories/files are 0700/0600. Local state requires a supported local
filesystem, not a network archive root. Configuration updates use exact revision
and digest checks and atomic pointer publication; stale changes are rejected.
Use GUI/API or a private validated serverctl configure FILE request, not hand edits
to credential generations. Unsupported schemas fail rather than being guessed.
sudo rfsentry-serverctl --instance research init
sudo rfsentry-serverctl --instance research tls-auto \
https://server.example:18445 127.0.0.1
sudo systemctl enable --now rfsentry-server@research.service
Creating an additional instance does not start it. Set a distinct port before starting. No master, automatic trust or shared admin credentials exists.
Storage selection and faults¶
Prepare an explicit directory and service-account access on the host. Then:
sudo rfsentry-serverctl --instance default authorize-archive /srv/archive/default
sudo systemctl daemon-reload
sudo systemctl restart rfsentry-server@default.service
Check and save the returned filesystem/mount/root binding in Storage. This is sandbox authorization and capacity/permission readiness, not crash durability or universal NAS support. Probes are bounded, disposable, no-follow writes only within the selected root; periodic checks are read-only. Missing/replaced mounts, read-only/full/permission/path-substitution faults leave administration available. No write is redirected to an underlying root directory. Replaced storage requires explicit rebind. Path changes do not migrate/delete data. The component never formats, mounts, recursively chowns or scans arbitrary directory trees.
Explicit ingestion and record retention¶
Record WebSocket subscriptions probe transport liveness after 15 seconds of
silence and time out after 30 seconds without received traffic. A healthy quiet
appliance responds to the ping without creating records or forcing a commit.
record-delivery-liveness-timeout enters the existing bounded reconnect and
durable-resume path; retained pending records follow the normal disconnect
persistence policy. A missing/restarted source still produces explicit gap
handling. This does not change retention, dirty-age settings or ACK authority.
In Appliances, check permitted metadata and add the desired stream policies. For each stream, enable record ingestion and save. Disabling it closes only that subscription and persists across restart. Full grants cover later-added streams without pairing again; narrow grants retain their stream restrictions. No subscription controls acquisition, tuning, jobs or another SDR connection.
Canonical envelopes and original payloads go into checksummed append-only segments under the bound root. Indexes/WAL and durable cursors remain on local storage. The C++ subscriber ACKs completed connection ordinals only after segment/directory synchronization and a durable SQLite index/resume commit. Supplemental configuration/boundaries are retained before dependent data and do not advance the original forward cursor. Reconnect uses original source identities with fresh transport IDs and exact uint64 strings.
The default missing-history policy pauses. Ring expiry, owner restarts and missing dependencies appear as explicit gaps. Deliberately select continuation from available history to proceed without claiming missing records arrived. An empty available snapshot follows future oldest history, preserving the old durable cursor until a record is committed. Integrity conflicts pause ingestion.
Enable retention separately and set age/byte limits explicitly. Age uses first Server receipt UTC; required dependencies survive, and deleted identities retain tombstones so retries cannot repopulate expired history. Deletion bookkeeping survives restart. Switching roots creates a separate archive/index generation; it never inherits another root's progress. Back up archive and local state together. Full/read-only/inaccessible storage returns an explicit archive fault until restored; administration remains available and ACK advancement stops.
Catch-up negotiates a bounded record window with compatible appliances and keeps receiving while an earlier batch becomes durable. The limits are 4096 outstanding deliveries / 4 MiB wire credit and at most 2048 records per group. The default persistence target is 1 MiB of canonical records. A batch can exceed that target by one maximum-size record. Compatible browser readers can access validated records from bounded Server RAM while the batch is pending. The browser shows separate available and durable positions. Durable-only readers wait for persistence, and the Server acknowledges delivery after the durable commit.
Storage's Record persistence controls set the 64-KiB–1-MiB target, optional maximum dirty age (blank by default), and separate WAL checkpoint trigger (16 MiB by default). Necessary partial groups report their reason: ordered boundaries, credit/count limits, cache pressure, explicit drain or shutdown. Older appliances retain their 32-delivery window and 16-record groups. The cache charges at most 8 MiB per active stream / 64 MiB total and backpressures pending input instead of dropping it. Reads do not force commits. Retention yields after bounded indexed work and preserves pending dependencies. New archive segments target 64 MiB, increasing possible physical retention slack to one such data segment per stream. Existing archive formats and identities are preserved. Actual catch-up performance awaits operator evaluation.
Stream/overview panels expose received versus durable counts/bytes, rates,
bounded backlog, catch-up/live state, cursor/ACK, reconnects, gaps, usage,
retention and storage faults. configuredMode, enable flags and retention limits
always describe the committed saved policy. runtimeMatchesConfiguration is
false and state is reconfiguring until the observed ingestion policy matches;
observedRuntime preserves the earlier mode/state separately. A retiring peer
may still report activity during that interval. Missing runtime observations do
not prove active ingestion or effective retention. Both Server processing modes archive records; native reconstruction additionally requires explicit processing activation for the stream. Diagnostic coverage/range APIs
support exact byte verification. Dedicated stream-scoped browser credentials
use the bounded binary record path for catch-up and continued reads. The format,
durability assumptions and recovery limits are in
components/rfsentry-server/ARCHIVE.md.
Index schema 2 and storage capacity¶
The 0.6.11 storage correction uses SQLite index schema 2. Current qualification and acceptance boundaries are in the validation chapter. Canonical RFSR records and format-1 segment/frame bytes are unchanged. This is shared archive infrastructure for all supported record protocols; it does not add Server reconstruction or activate another roadmap phase.
The index stores each original producer/stream/run tuple in a dictionary, full SHA-256 values as 32-byte binary values, and original sequences/revisions as lossless fixed-width big-endian bytes. Internal IDs never replace original identities on the public protocol. Live rows use stable, non-reused integer ordinals for verification and compact dependency references. Original owner bytes remain in segment frames; the current owner is retained in progress state. Configuration records own normalized digest/revision entries, and dependent records reference those original records.
Retention commits an exact retired-record entry and removes the live row and its dependency edges in one FULL-synchronous transaction. Retired evidence keeps the original identity binding, full canonical hash, forward/supplemental status, epoch, receipt and normalized policy reference. Exact membership can be noncontiguous. Unknown sequence holes remain unknown. A same-identity/different- content retry remains a conflict after retirement; same-content retries cannot restore expired payloads.
Each retired entry also retains a bounded set of stable dependency ordinals. This preserves the no-ACK rule if a still-retained dependency is quarantined later, without retaining the old pair-of-full-identities dependency tables.
Retired physical extents remain indexed while a segment contains retained content. Only after unlink and directory synchronization may bounded cleanup discard those extents and completed segment bookkeeping. Exact completed segment-ID ranges and a durable high-water mark prevent reuse or resurrection. Quarantine remains separate and charged; policy retention does not erase its required evidence or dependencies.
Exact conflict evidence has residual lifetime growth. It is much smaller than a fully indexed live record, but retaining exact hashes for arbitrarily old identities is not constant-space storage. The metadata budget includes it. Exhausting that budget pauses admission; it does not silently expire this guarantee or enable deletion. Size metadata budgets from measured bytes per record and arrival rates, not a universal database/archive ratio.
Small canonical records can exhaust metadata capacity before the archive-byte retention limit. Measure the operator's actual record mix, arrival rate and retired-index growth when sizing allowances; no universal storage ratio is qualified. Ingestion evidence keeps its measured workload and durability limitations separate.
Limits and accounting¶
Storage → Record persistence exposes recordStorage.capacity:
| Setting | Default | Meaning |
|---|---|---|
maximumMetadataBytes |
32 GiB | Allocated database/WAL/SHM files, other retained index generations in this instance, and charged temporary work. |
maximumPhysicalBytes |
Unset | Optional combined limit for selected archive segments and accounted metadata. |
localReserveBytes |
256 MiB | Free-space floor for the local index and its private temporary directory. |
maximumWalBytes |
64 MiB | WAL admission allowance, separate from the checkpoint trigger. Must be at least twice that trigger. |
maximumTemporaryBytes |
32 MiB | Bound on temporary logical extents, charged before growth and until the owning file closes. |
resumeHeadroomBytes |
16 MiB | Additional filesystem/metadata headroom before resuming. WAL-specific hysteresis is capped as described below. |
maintenancePages |
64 | Maximum page-reclamation steps per maintenance turn; supported range 1–1,024. |
maintenanceMilliseconds |
20 ms | Work-admission budget for page reclamation; supported range 1–100 ms. A storage syscall/sync can outlast it. |
cacheKiB |
4,096 KiB | SQLite page-cache allowance; configurable to 65,536 KiB on capable hosts. |
Values named Bytes remain exact decimal strings. The existing archive-root
storage.minimumFreeBytes and canonical-data retention policies remain separate.
Older configurations without a capacity object retain their existing settings;
effective defaults include a WAL allowance large enough for the saved checkpoint
trigger. Changing capacity settings never turns on ingestion or retention.
Admission measures the actual index, archive and temporary filesystems. Roles on the same filesystem share one free-space floor (the larger reserve), while their proposed writes are added together. Independent volumes are checked independently. Canonical retained bytes are informational content accounting and are not added again to physical segment bytes. Unrelated filesystem consumers are reflected in actual available space, even though their files are outside rfSentry's allocation budget.
The capacity view distinguishes database logical size, allocated blocks, used
pages, reusable free pages, WAL/SHM allocation, temporary charges and physical
segment allocation. Used pages include retired evidence and indexes. A freelist
is reusable SQLite space, not already-returned filesystem space. Allocated-block
figures are filesystem reports; shared/CoW extents and filesystem metadata can
make filesystem-wide usage differ. Temporary work uses page-rounded logical
charges plus working reserve; filesystem preallocation can differ. Segment
allocation is updated after synchronization/reconciliation, and other index
generations carry their inventory timestamp. Unavailable or stale measurements
are not displayed as zero. Per-table dbstat analysis is diagnostic work, not
status polling.
The selected archive's segment budget does not enumerate abandoned data roots outside the current storage binding. Their consumption still reduces filesystem headroom. Retained SQLite generations under this instance's local state are included without opening their database files; more than 256 generations needs an explicit inventory review. SQLite WAL stays on qualified direct local storage. A network or FUSE archive is not a fallback location for its index.
Limits are admission controls with bounded in-flight reservations and working
margin. External consumption can race a successful check: SQLITE_FULL,
ENOSPC, read-only storage and failed synchronization still stop durable ACKs.
Expected capacity pressure preserves the healthy archive owner for reads where
possible and retries prerequisites. Actual I/O/transaction faults retain
recovery. Admission resumes only after measured headroom satisfies the request
and hysteresis. Retention being disabled never authorizes emergency deletion.
WAL recovery first reserves the next write, then requires additional headroom
of min(resumeHeadroomBytes, (maximumWalBytes - writeReservation) / 2).
This permits legitimate small WAL budgets to recover after checkpoint/truncation,
including a 1-MiB checkpoint trigger and 8-MiB maximum with omitted/default
16-MiB resume headroom. Filesystem reserves and metadata hysteresis retain their
full configured values. A write that exceeds the WAL allowance still blocks.
Normal ingestion and pending recovery batches shrink to fit small WAL budgets;
the original identities, ordering and durability-before-ACK rule are unchanged.
A held reader must release its snapshot before reclamation can recover space;
no restart or policy edit is needed when the actual pressure clears.
Server outputs and browser download policy¶
Processing follows the existing explicit enabled/paused stream intent. Its
compiled goes-hrit processor declares the goes-color child: Natural Color
(C02+C13) and Day/Night (C02+C07+C13). No separate stream activation is needed.
Recent retained product pages are matched first, with a durable bounded cursor
for older retained products. Exact scans and received navigation must agree;
missing constituents/calibration remain waiting or recorded deferred outcomes.
Base imagery, messages and ingestion continue independently. Native dimensions,
received masks, recipe revisions and constituent lineage accompany the output.
The current storage blocker now remains visible ahead of historical continued frame-gap warnings. Once the actual storage condition clears, existing automatic retries resume; no Pause/Resume ritual or tail/checkpoint reset is required. Processing shows the receipt time of its last durably processed input separately from current operation, last progress and last publication. This is source-time age, not a percentage or subtraction across sequence epochs.
For browser products, update the appliance-served client as well as the Server, then reload. All products → Retention and storage contains Server download policies: native imagery When selected (default) or Recent products with an hour window; originals On demand (default) or Retain automatically; and an aggregate download-staging allowance (default 256 MiB). Thumbnails and text arrive first. Opening native imagery or exporting original evidence queues that immutable revision through the existing resumable worker. Staging limits must fit the requested encoded assets; they do not lower native image quality. Local asset roles and native offline readiness are shown in product details.
Safely identified unsupported schemas are counted and deferred while compatible products continue. Upgrade/reload triggers a bounded resnapshot when adapters change. Canonical originals, prior locally retained products and pending byte ranges remain intact. Successful polling still says nothing about upstream production progress. No local recipe is enabled to mask missing Server output.
Only the proven equivalent non-image all-received/unknown coverage label is
normalized. Bounded maintenance clears equivalent conflict markers while leaving
all immutable revisions/assets intact. Genuine imagery or scientific differences
remain conflicts. Detailed deferred recipe contexts are retained for the most
recent 256 tasks; older retry contexts expire explicitly, without requeueing the
same failed input. Changed constituents can trigger a fresh recipe attempt.
The accepted baseline passed the exact-source main/release gates described in validation. The historical sanitizer archive-I/O watchdog incident remains unexplained; acceptance and later green runs do not diagnose it or establish sustained speed.
Optional Server processing¶
Processing is independently enabled in the Processing administration tab. Version 0.6.12 fixes worker CPU/memory observation under the installed service's procfs privacy policy. A failed stream shows its concrete error beside the stream; committed products and recovery state remain retained. Missing peak measurements are not presented as measured zero usage.
Processing forms retain unsaved edits through live status polling, validation
errors and failed saves. Concurrent saved changes are reported instead of being
silently overwritten. Editing any resource/storage allowance selects Manual;
otherwise blank CPU/RAM caps are filled from current measured headroom when
available, with RAM rounded down to whole MiB. Review those caps before saving.
Selecting Auto preserves explicit caps; blank CPU/RAM fields use Auto headroom.
Every setting has help available by hover, keyboard focus or tapping its ?
button; Escape dismisses it. Size fields accept M/MB, G/GB (decimal) and
Mi/MiB, Gi/GiB (binary), including whole-byte decimal fractions such as
1.5 GiB. Bare numbers retain the previous MiB interpretation. Bandwidth also
accepts /s. Units are converted with exact integer arithmetic.
The page displays the selected <archive-root>/products-v1/ destination for
assets, checkpoints and staging, and the local-state products/ directory for
catalogs. The asset allowance controls capacity, not placement; there is no
separate product-root selector. Changing archive selection does not migrate data.
Pairing and upgrades do not enable processing. Choose a retained source stream and compatible processor, then explicitly enable it. Recent history target keeps the existing minutes value (30 by default). Live processing starts near the current committed tail independently. Missing history is processed newest-first, with records inside each bounded range decoded forwards. With older backfill off, the recent target bounds history; with it on, the active product-retention policy and available original records set the older boundary.
A live appliance connection is not required. Pause/resume preserves products and recovery. If a saved foreground owner is substantially stale relative to the retained tail, a durable handoff preserves its unfinished work while a new owner handles current reception. Ordinary decoder errors retain their existing bounded continuation/recovery behavior. Historical work yields first under resource pressure; checkpoint/replay allowance must hold retained state plus a replacement. No processing or storage cap is automatically raised.
The Processing page shows Live reception, the historical primary range and any temporary recovery handoff separately. Historical range reads may extend through bounded assembly context; that does not mean all intervening history is complete. Live input time, historical coverage, publication time and acquisition ACKs are different positions. Independent browser copies retain their own policy.
The first processing-settings save explicitly transitions configuration schema 3 to schema 4. Older binaries cannot interpret that configuration; a rollback requires a reviewed compatible configuration, never an archive reset. The record database remains schema 2. GOES native imagery/text, partial-to-complete revisions, authenticated delivery and durable browser import are implemented. Compact thumbnails and exact native pixel-coverage masks are produced on the Server. The registered color task adapter completes the existing Natural Color and Day/Night recipe path. Functional acceptance is recorded above; hardware working sets remain workload-specific measurements.
In the GOES workspace, choose Configure source → Server products, select the trusted Server and original stream, and Apply. This mode uses existing local archives/viewers and suppresses corresponding local reconstruction/recipe jobs, including after reload. Old preferences keep records mode. Direct IQ, direct appliance records and older archive-only Servers remain supported. Fallback to local reconstruction requires the configured explicit fallback policy and its own valid recovery state. Cached products remain viewable offline.
The importer fetches details, thumbnails and text first. Native/original assets follow the selected policy described above and share the aggregate staging allowance. Quota, cancellation and transfer failure preserve prior published products and durable pending work. Discovery, asset transfer and local publication have separate progress indicators; none acknowledges acquisition records.
Auto measures effective affinity/cgroup CPU and available/cgroup memory before admission. Operator controls cap CPU allowance, workers, aggregate memory/cache and background I/O. Protected memory defaults to 512 MiB, with another 512 MiB when acquisition co-location is explicitly selected; at least one CPU core is protected when effective capacity permits it. Manual mode requires explicit CPU/memory caps. An ordered context has one assembler owner; independent contexts can run concurrently. Observed RSS and future peaks are shown separately. Peak worker RSS remains visible after a failure, alongside the declared bound. Immutable-product tasks use the same worker allowance and can run after an idle record owner yields a durable safe point. The task framework is exercised by a test-only observation processor and production GOES color recipes. A record owner can also yield at a durable point when a pending recipe fits only after its reservation is released. An individually oversized or unsupported item receives a durable failure/deferred outcome so later eligible work continues, without reducing native dimensions. Aggregate admission or recovery-storage pressure instead pauses affected work until its condition clears; no progress advances if its outcome cannot persist.
GOES's default declaration reserves about 3.125 GiB per active owner, including
assembly, original files, codec/output and checkpoint replacement peaks. This is
a conservative admission declaration, not a measured minimum working set or a
real-time throughput claim. Advanced processor settings (assemblyBytes,
fileBytes, outputBytes) must fit both processing and old-plus-new checkpoint
storage allowances. A small native fixture was tested at 16/1/2 MiB respectively;
that fixture does not establish a budget for real full-disk imagery. Do not
multiply the old browser defaults by a chosen worker count without admission.
Checkpoint-capable adapters report an exact frozen save size when available, so
disk admission reserves the actual replacement plus retained state. An older
configuration revision remains incompatible for restore even when its already
prepared product outputs can be recovered safely.
Products occupy separate catalog/staging/checkpoint storage. Default allowances are 8 GiB assets, 128 MiB catalog/index working space, 2 GiB checkpoints/replay, 1 GiB staging and a 64 MiB free-space floor on the selected product/archive volume. A separate local catalog volume uses the configured index reserve plus its SQLite commit allowance, not the archive-sized product reserve. Shared volumes use the higher floor once, plus both write allowances. Preflight and SQLite growth admission use the same reserve scope. Raising the product reserve to protect a large archive does not also require that much free space on an independent system/index volume. Neither existing reserve setting is rewritten. Failed temporary writes remain charged until recovery reclaims their owned files. Missing/replaced directories or mounts fail explicitly. Metadata reservations and recoverable storage pressure use bounded retries; clearing the pressure does not require a Server restart. Registered immutable tasks trigger a lazy product-catalog schema-2 upgrade, separate from record schema 2. Input replacement fences stale tasks, and task output/completion share one publication transaction. Expired task inputs are counted and cannot pin source-product retention. Pairing and upgrades still enable no processing. Product pressure pauses optional work; record ACKs continue to require record fsync/index durability. Per-stream product retention is independently explicit; it does not change record or browser retention.
From 0.6.14, product reads use independent catalog snapshots while publication continues. Four readers and eight queued requests bound interactive admission; ordinary publishing does not cause immediate catalog-busy rejection. Product WAL and shared-memory files count toward the catalog/index allowance, with a 32-MiB WAL ceiling and non-waiting checkpoint reclamation. This changes only the product catalog, not record persistence or its WAL settings.
Products are exposed through scoped /api/v1/products/ GET operations. Assets
are immutable and exact range requests bind the revision and hash. Expired
change cursors require a bounded snapshot. The product catalog is additive,
with its own versioned recovery protocol documented in
PRODUCTS.md.
The validation chapter
distinguishes local fixtures, exact-source CI/installed evidence, hardware
measurements and operator acceptance.
The server runtime/durable-publication checkpoint at signed 510bfdf passed
full CI and
installed amd64/ARM64 qualification,
including durable product delivery and mounted ingestion faults with independent
index-volume exhaustion. This qualifies that historical checkpoint. The accepted 744b840 baseline
includes production recipes, fetch policies and subsequent persistence/retention
and live-history repairs. Its release passed both installed architectures.
Minimum production working sets and sustained throughput are not inferred from
those tests; Phase 6 performance acceptance remains open.
No production installation or automatic activation is implied by this source.
Obsolete partial cleanup (0.6.19)¶
Superseded partial renderings are automatically retired independently of the stream's product-age/byte retention switch. Current/best observations, original files and conflicting complete evidence remain. Pending reads and active recipe inputs are protected. Existing obsolete history is scanned in bounded background passes; cleanup does not require a reset or processing restart ritual.
The Processing page reports retired partial revisions, pending ownership checks, queued packs and reclaimed asset bytes. Disk usage can fall later than metadata retirement because shared packs require bounded compaction. Unused checkpoints are reclaimed ahead of ordinary asset garbage so a large expiry backlog cannot strand processing at its checkpoint allowance. Original-role assets remain until whole-product retention expires their observation. Catalog schema 5 keeps compact replay identities and cannot be opened by an older Server binary.
Refresh appliance/browser assets before running this Server update. Updated clients replace obsolete pending downloads with current revisions, preserve previously retained products and report unavailable selected downloads explicitly. Their archive cleanup proceeds in short background passes and is visible under Startup & activity. The Server's three-day product policy still requires Enforce this stream’s Server product retention; it does not set browser retention.
Product pack storage (0.6.18)¶
New publication batches use product catalog schema 4. Small assets share streamed packs targeting up to 64 MiB, with time and resource bounds; large assets still stream without fitting in RAM. Old loose assets stay readable. The catalog's physical asset usage includes dead space inside retained packs. Maintenance reclaims wholly unused packs, or copies a bounded mixed pack with reserved replacement space before atomically changing asset locations.
Pack data and required directory entries must synchronize before products and processing progress become recoverable. A failed write/sync preserves prior progress. Older binaries cannot open schema 4; do not downgrade against the new catalog. Checkpoint and record archive formats are unchanged.
Writers close before their final allocation charge is recorded. A bounded checkpoint reconciliation pass and maintenance scans correct older persisted open-file allocation charges using current file attributes. This changes neither payloads nor configured limits. Open range readers keep their file allocation charged until the descriptor closes.
Timing counters distinguish packed assets, pack synchronizations and their elapsed time. Batch adaptation uses CPU work while a separate publication deadline limits waiting. Required durability barriers and operator I/O/storage limits remain in effect; sustained catch-up is an acceptance measurement.
Processing throughput diagnostics (0.6.17)¶
Under Processing → Processing timings and batches, inspect the current and maximum page target, completed batches, limiter waiting, snapshot reason/size and elapsed timings. Targets grow for inexpensive catch-up work and shrink for output pressure; a large page may commit only its consumed prefix. Native pixels, original identities and gap ordering are preserved.
GOES catch-up can grow to the existing 512-record archive page ceiling. Near-live pages still shrink for latency. Historical processing coalesces intermediate mosaics over at most five minutes of source time or two minutes of active work; complete candidates, original files/messages and terminal partials remain eligible immediately. Deferred strips survive checkpoints; native geometry, pixels and evidence are unchanged. Age-only product retention uses the receipt index without scanning every asset to calculate an unused byte total.
Full processor snapshots now use a bounded 1–64 MiB replay allowance based on snapshot size and configured checkpoint space, with a 15-minute active-work age limit. Small checkpoints remain small. Idle state changes and failure recovery still snapshot when required. Product/progress commits remain durable in between. Routine cleanup is coalesced and performs no directory sync when nothing was removed; real storage/WAL pressure triggers bounded reclamation promptly. Each new file is synchronized after its final rename/link state; verified durable duplicates reuse their existing bytes. Interrupted staging still needs a file barrier. Asset directory barriers are combined at batch preparation; synchronized file data and both directory barriers precede durable publication. Independent admitted jobs can proceed while a new file is being synchronized; its staging reservation remains charged until safe completion.
The I/O allowance is still an actual ceiling. A 500-MiB snapshot at 16 MiB/s needs at least 31 seconds of allowance, plus verification and durability costs. CPU headroom does not remove that delay. The implementation does not increase this operator setting automatically. Timers overlap: checkpoint/staging totals include worker IPC and limiter waiting. Compare deltas over the same interval, including a checkpoint, and watch the input receipt timestamp against wall time to judge whether backlog is shrinking. No history is skipped to make this look faster.
Short catch-up intervals do not establish sustained realtime throughput. Include a full checkpoint and storage synchronization costs when measuring backlog recovery. Product evidence and the Server roadmap retain the unqualified performance scope and unresolved watchdog observation.
Reclamation and checkpoints¶
Genuinely new databases enable auto_vacuum=INCREMENTAL before creating tables.
Maintenance uses positive, bounded incremental_vacuum(1) steps, bounded
dictionary/extent cleanup, and non-waiting checkpoints. Turns rejoin the existing
ingestion/read queue. Filesystem space is reported as returned only when actual
allocation falls; a reader can prevent WAL reuse or physical truncation after
pages have become free. SQLite's checkpoint trigger and journal_size_limit
are not hard WAL caps. See the SQLite reclamation rules
and WAL checkpoint behavior.
An existing auto_vacuum=NONE database cannot be converted by merely setting a
pragma after tables exist. Schema 1, unknown versions, incomplete schema 2, or
incompatible reclamation mode stop with an actionable error. There is no
automatic migration, full VACUUM, rebuild or erase. Old binaries reject schema
2. A missing main database with sidecars or existing segments remains a recovery
blocker; the Server never guesses the lost retention journal.
Reviewed fresh-generation reset procedure¶
This is a preparation/runbook, not authorization to execute a live reset. Use a reviewed, exactly qualified package and a host-specific inventory before destructive work. A clean generation is preferred to a large legacy migration.
- Stop the named instance and verify
ActiveState=inactive,MainPID=0, and no archive worker remains. Identify independent or paused maintenance holding the database/candidate files. Terminate the identified maintenance without resuming its queued destructive steps. A stopped service alone does not prove those independent processes have released their descriptors. For an independently SIGSTOP-paused maintenance process, the reviewed action is SIGKILL of that reverified process identity, followed by proof that it has exited and released its descriptors. Never send SIGCONT to let cleanup finish. - Record the instance UUID, archive UUID/storage generation, root inode/mount
binding, package/source identity, configuration digest, and each stream's
separate ingestion, retention, processing and continuation intent. Verify
private ownership/permissions. Keep
/etc/rfsentry-server/instances/NAME,/var/lib/rfsentry-server/NAME/identity.json, its completecredentials/directory, grants, certificate/enrollment material and settings intact. - Identify exact database families, including moved files: main, matching WAL
and SHM, plus any compaction candidate. An original main file may require its
WAL to contain all committed pages. Do not detach it or use
immutable=1to pretend the WAL does not matter. A candidate whose verification was paused is not an authoritative backup. Neither database alone restores history without the matching segments and archive identity document. - Preflight each real filesystem. Include preserved originals/candidates, existing segment files, the new metadata allowance and WAL/temporary work, package/staging space, and reserves. Charge WAL/temporary allowances once within the metadata budget, and combine reservations on shared volumes. Recheck actual available space immediately before execution. If preservation does not fit, stop and present exact sizes and alternatives; do not delete a preserved database to make the plan fit.
- Choose explicitly between a new empty, separately bound archive directory
(preserving old history) and discarding the reviewed old generation. The
discard list is only that generation's verified
segment-*.rfsfiles and.rfsentry-archivemarker, plus separately reviewed database files if their disposal is authorized. Never recursively remove the instance state or configuration roots. Keeping the archive directory itself preserves its inode; replacing it requires an explicit fresh storage probe/binding through the existing configuration API/CLI and sandbox authorization. - Install the reviewed package while the instance stays stopped. On the empty
selected root, the Server creates a new archive UUID/storage generation and
a fresh schema-2 database under
archives/<newArchiveUuid>/index.sqlite. Never transplant the old marker, old cursor, progress table or clean-stop receipt into it. Existing Server, appliance, stream and original producer identities are preserved. Old archive-specific fences must be rejected. - Start only the instance and subscriptions that the operator authorizes, preserving the recorded inactive flags. An empty archive has no durable history to ACK. Initial replay uses the configured appliance's currently retained originals, with its real availability boundary and the existing continuation policy. Record history before that boundary as unknown/missing; never claim the former durable cursor is present. Other authorized archives may hold evidence, but this procedure does not add peer replication/import or promise replay beyond an existing supported source. No browser OPFS is reset.
- Verify new generation/schema/reclamation mode, identity and access, effective settings and independent capacity measurements. Confirm original records become durable only after segment/directory sync and FULL SQLite commit, then check source-observed ACK progress, bounded queues, truthful gap reporting and old-fence rejection. Read back exact originals. Check that previously inactive streams and receiver settings remain unchanged.
Rollback to the old history is possible only while its original database/WAL, all required segments, marker and compatible configuration/package remain coherent. If the old segments were deliberately discarded, an old database file is not a history rollback. Keep any optional preserved generations explicitly inactive and account for their space; their existence must not be mistaken for the new archive's progress. The reset remains a separate operator acceptance step after source and isolated qualification.
Registrations, credentials and operational truth¶
Pairing stores stable identities, authenticated private trust and one independently
revocable full-appliance credential per server. Existing/advanced records.read
registrations remain narrow, separate from full grants and server-admin sessions.
Explicit checks validate TLS/pin and then identity/configured stream metadata for
full grants, or permitted availability for readers. No receiver/job/subscription/
history payload is opened. Narrow-reader empty history cannot establish original
source identity. Saving a registration or opening administration does not
activate ingestion. Restart resumes only explicitly enabled subscriptions and
never wakes hardware.
Codes last five minutes, allow at most five initiated attempts appliance-wide,
are single-use and expire on gateway restart. Use pair-status (no code revealed),
pair-cancel, and sudo rfsentry-jobctl records-revoke server-SERVER_UUID on the
appliance. Revocation ends that server's authorized connections/leases without
ending independent continuous jobs. A new code rotates enrollment and preserves
existing stream intent. Removing only a stored server credential is not remote
revocation. Interrupted pending enrollment can resume from its private saved
credential; discarding it does not revoke a grant the appliance already committed.
Admin GUI credentials provide user/token rotation and revocation. Values are write-only/redacted and use explicit unchanged/replace/remove where applicable. Stream byte/age limits have validated units and explicit unset states. Directed peer configuration remains disabled/inactive, not reachable/authenticated/active until separately checked/implemented in later authorized phases.
Upgrade, removal and qualification¶
Startup and background verification¶
Normal startup checks the selected storage, archive identity, index/schema, segment boundaries and unfinished write tails. It restores the original durable cursor and verifies required resume data before subscribing. A successful stop records a generation-bound clean checkpoint after pending records finish; startup invalidates that receipt before further writes. Older installations without a receipt use tail reconciliation rather than a full historical payload scan. Missing storage/indexes, incompatible identities and unsafe write boundaries remain explicit blockers.
Historical record verification runs in bounded turns that yield to ingestion and readers. A separate read-only process checks SQLite integrity at low CPU/I/O priority. Its ordinary startup attempt is bounded by read/work admission rather than an unbounded full-database startup scan. It releases its snapshot and reports Deferred when the 32-MiB/500-ms startup allowance is reached, or when WAL pressure approaches the configured allowance. A single blocking storage operation can outlast the work budget. It is attempted once per service start or selected-archive change; a full offline integrity check is explicit diagnostic work. Deferred does not mean verified; record integrity checks on reads and ingestion remain enforced. Confirmed structural index corruption blocks new writes. Eligible damaged records retain their durable quarantine evidence and explicit gaps.
Use Activity for separate write-readiness, historical-verification and index- verification status, elapsed work, last progress, source connections, HTTP/session counts, and graphs of received/durable flow, pending bytes and verification work. The endpoint uses cached observations and remains available while the archive is busy. Recent history is bounded to 600 samples and 128 coalesced events in Server RAM: reloading the browser preserves it, restarting the service resets it. No additional TSDB or per-record metrics writes are required. Significant transitions and rate-limited progress are also emitted to the system journal.
Graphs report whether Activity data is loading, unavailable, sampled but zero, or plotted; an empty-looking plot is not used for all four conditions. A temporary Activity request failure keeps the last successful samples visible, shows the reason and retries automatically. Returning the tab to the foreground requests a fresh observation. After upgrading an already open administration page, refresh that Server page once to load the newly installed static UI; this does not restart or navigate an appliance reception page.
Ready/Live does not mean historical verification is complete. Conversely, background verification need not prevent normal receiving and durable commits. No percentage or ETA is invented when total work is unknown. Original cursor identities remain exact strings, and byte displays include both readable units and exact byte counts. Gap, cursor and retention summaries keep the full structured details behind expandable controls. Gap downloads contain the bounded recent status snapshot; they do not contain or reconstruct the missing payloads.
Package maintenance¶
Reinstall/upgrade preserves identity, private configuration/credentials and archives; only already-active instances are restarted. Disabled/stopped instances stay so. Removal stops/removes the packaged server; purge deliberately retains private operator state/drop-ins/accounts and never deletes external archives. Retained private state needs appropriate host backup protection. Same-schema rollback is supported only when the binary understands the saved configuration and both catalog schemas. Processing-enabled configuration uses schema 4 and the product catalog advances through schema 6; older binaries may reject them. Downgrades need compatible stopped-state configuration/credentials and product recovery assets. The package never silently downgrades a schema or resets an archive.
Automatic leaf renewal occurs on service start within its last 30 days, preserving the private authority. A continuously running instance still needs a planned restart before the one-year leaf expires. Provided certificates remain explicit; authority rollover requires new client trust. Failed generation/renewal retains the last valid saved configuration. See the component README for full limits.
The build/qualification scripts are in components/rfsentry-server/. Installed
tests run only in explicit disposable Debian 13/systemd guests. Local headless
Chromium uses normal trusted TLS and its real sandbox, not remote CDP. The validation chapter
and retained Phase 6 report distinguish native execution, emulation, CI evidence
and outstanding limits; >4-GiB memory64 qualification remains Phase 12. Normal v* releases now
have one coordinated publisher/checksum manifest; client-v* remains independent.
No production release tag is created by qualification.
Processing failures and recovery (0.6.15)¶
Failure journaling adds product-catalog schema 3; it does not change the canonical record archive's schema 2. Older product-catalog binaries cannot open the upgraded catalog. Any rollback therefore needs a compatible stopped-state product catalog and its referenced assets/checkpoints, in addition to compatible configuration. Do not reset or downgrade the live archive as a processing-error workaround.
An ordinary failed file, message, image or derivative is recorded while later valid work continues. The Processing page shows the current operation, original attempted/durable/checkpoint positions, last actual progress/publication, and failed/deferred and recovery counts. “Awaiting input” means the available range has been consumed; resource waits, storage blockage, recovery and stale supervision are shown separately. A status refresh does not count as progress.
Use View recorded failures for the cause, original range/hash, processing stage, native diagnostic and recovery disposition. Reprocess this item is a deliberate bounded retry after a decoder/policy correction. It uses retained input at lower priority, preserving live processing and existing products. Inactive or paused streams remain inactive. A completed replay describes the attempted range; resulting product revisions/outcomes describe its result. Expired source data or diagnostic detail is reported explicitly.
Expected bad input is isolated immediately. Unsafe worker exceptions/crashes replace the worker and replay committed state. Two repeatable failed executions lead to an adapter isolation decision and durable outcome; unchanged failed work must not wedge restart. Memory admission and storage failures instead wait for their actual condition to clear. Failed outcome/checkpoint commits never advance processing. Record ingestion and its durability-before-ACK rule stay independent.
Failed thumbnails leave valid native imagery available. Unsupported views are not advertised as ready. Current GOES contracts bound details to 16 MiB and 65,536 message records, and text views to 16 MiB; individual limits are reported with original evidence retained. Product/failure retention does not delete independently retained browser observations or alter record retention policy.
Browser product mode shows transfer and upstream processing health separately. Successful polling of an unchanged catalog is not evidence of production progress. Older Servers without health capability report it as unknown. Server-products mode retains its explicit fallback policy and does not silently start local assembly/recipes.
For native diagnostics, retain the exact executable/package identified by the
reported ELF build ID. Use readelf -n to compare it and addr2line -Cfipe with
reported module-relative addresses to resolve matching frames. Signal reports
include the instruction and executable base; catch-site-only stacks are labeled.
No unrestricted core dump is enabled. See qualification and limits
for accepted-source evidence and the unexplained sanitizer watchdog incident.
The separate mount-table incident and recovery issue #7
remains deferred unless it recurs. Its later successful restart did not establish
the trigger or fix automatic recovery; do not weaken missing-mount checks.