Skip to content

Optional rfSentry Server

Use rfSentry Server to keep decoded records after the appliance’s RAM buffer rolls over. Install it on a separate Linux host, choose a storage volume, and enable the streams you want to archive. You can also enable GOES product processing and open the resulting imagery and messages in the browser. The Pi and browser work independently of the Server.

Archiving and browser consumption

The administration foundation provides an HTTPS/static GUI, independent identity, private configuration/credentials, explicit archive-root readiness, appliance registrations and disabled directed peers. Phase 6 adds verified-TLS resumable ingestion, durable exact decoded-record archiving, explicit retention and bounded coverage/readback APIs. The single administration UI uses React + TypeScript, compiled into static assets served by C++.

Archive only is the default mode for a stream. After pairing, enable ingestion for each stream to start receiving records. Retention and product processing each require a separate choice. See optional Server processing and choosing a browser source. The acceptance record covers native GOES reconstruction and browser consumption. Sustained archive performance remains an open gate.

Initial packages target Debian 13 amd64 and arm64, configurable HTTPS 8445. They do not install or depend on the appliance, SDR/USB/Chrony/QoS services, workstation utility, compiler, frontend toolchain or another application backend. Do not run the appliance install.sh to install this component.

Install and enroll on a trusted LAN

On the separate intended server host:

sudo apt-get install ./rfsentry-server_amd64.deb # choose arm64 when applicable
sudo rfsentry-serverctl --instance default validate
sudo rfsentry-serverctl --instance default trust

A genuinely fresh instance generates a provisional private HTTPS authority/listener in C++ and listens on 0.0.0.0:8445, with anonymous full administration and no default password. This is the selected trusted-LAN convenience policy, not safe Internet exposure. Archive/appliance configuration remains unset. Existing instances preserve their listener/authentication through migration/upgrades. Fresh identity selection uses the system FQDN when available, otherwise the actual hostname. The default-route IPv4 address is retained as an administration alias. With an FQDN available, new defaults omit the unqualified short hostname because it cannot satisfy the appliance's DNS suffix policy. Existing aliases are preserved through upgrades and require explicit operator review. Certificates and exact Host/Origin checks cover the same names; no .local suffix is guessed.

Before pairing, the CLI reports the canonical URL and provisional public CA/fingerprint. Obtain that fingerprint through trusted host access and import the public CA only when the instance must remain independent. Pairing with a private-CA appliance replaces the provisional listener leaf with one signed by that appliance. Clients which installed the appliance root from its /trust/ page then trust the appliance and every Server it approves. No ACME or manual certificate generation is needed. Automatic HTTPS cannot establish secure first-contact trust by itself; do not use certificate-ignore, sandbox or web-security bypasses.

After upgrading an installation with an old guessed hostname, deliberately refresh its detected identities and restart only this Server instance:

sudo rfsentry-serverctl --instance default identity-refresh
sudo rfsentry-serverctl --instance default trust
sudo systemctl restart rfsentry-server@default.service

This reuses the existing private authority and preserves UUID, bind/port, authentication, pairing, stream enablement, storage binding and archived records. It replaces the canonical address/alias list with current detected identities; review the reported URLs and explicitly add any custom aliases afterward. Operator-provided certificates are refused unchanged. Upgrades do not otherwise change existing addresses, and DHCP/DNS changes require a deliberate refresh. FQDN lookup is bounded to two seconds. Default-route IPv4 selection uses the lowest-metric main-table route; policy routing/ECMP or unusual multiple addresses require explicit alias selection. IPv6 listening remains explicit.

Open the GUI, choose Appliances, and select a discovered appliance or enter its canonical HTTPS address. On that appliance, run:

sudo rfsentry-jobctl pair-create

Enter the six-character code and click Connect securely. One code enrolls this server with full appliance API, diagnostics, controls, raw IQ and all present/future decoded streams—not one code per job. PAKE mutual confirmation binds identities, full grant, canonical endpoint, the Server CSR/enrollment public key and the actual TLS connection. The Server generates and keeps both private keys locally; the appliance receives only the CSR, public enrollment key and proof. Afterward normal CA/hostname/certificate verification protects appliance authentication. No private CA key/global operator token is copied. Pairing does not start reception or activate an inactive ingestion policy. Existing enabled policies survive replacement of their credential. Bind storage and explicitly enable streams next.

Review Server certificate names above the Connect button. Every name must match the appliance's configured DNS suffixes or IP ranges. A name such as archive does not match a suffix of example.net; use archive.example.net or an approved private IP, or review the issuance policy explicitly. Issuance rejection identifies the cause and leaves the old registration unchanged. Correct the problem and retry with a valid code. A confirmed rejection is labeled Enrollment rejected before commit and cannot be resumed as a lost reply. An uncertain/committed lost reply retains its private pending credential and offers Resume verified connection, without another code.

Discovery is C++ Avahi/DNS-SD on the Server's local link. Candidates remain unverified until pairing; there is no network scan/automatic trust. If multicast is blocked or Avahi unavailable, manual HTTPS entry works. Configure discovery interfaces/enablement in secondary settings; appliance advertisement is derived from appliance.conf rather than another identity file.

Enable authentication with the first account in Instance settings, or host-only sudo rfsentry-serverctl require-auth operator (password on stdin). Host-only bootstrap instead enables authentication and prints a private 15-minute first-admin token; enter it in setup. No login is required in the fresh LAN default.

TLS/origin/listener/discovery changes are saved, restart required; status distinguishes saved versus actual running listener. No web operation launches privileged commands. Use normal client trust, not certificate-ignore/security flags. Additional exact HTTPS Origins are an explicit CORS allowlist, separate from Administration aliases, which authorize this service's hostnames/IPs. Every alias must be covered by the certificate; choose automatic private HTTPS when changing names. tls-auto ORIGIN BIND [ALIAS...] is the corresponding host command. Cookie mutations require CSRF and an allowed Origin. Cookies are host-only; signing in at one address does not sign in at another. Read-only users cannot save, probe storage or contact appliances. Host-only recover-admin USER reads a new password from stdin and revokes sessions for that user. Restart clears all RAM cookie sessions.

Appliance-issued leaves are valid for 90 days. The per-instance renewal timer checks twice daily and renews with 30 days remaining. Renewal is authenticated by the persistent enrollment signing key and full-appliance registration, not the current Server leaf, so reconnecting after that leaf expired can still recover automatically. SIGHUP installs the renewed certificate for new TLS connections while established sessions remain intact. A lost enrollment key, revoked registration or expanded hostname set requires a new six-character code. If the expired GUI cannot be opened, run:

printf '%s\n' 'SIXCODE' | sudo rfsentry-serverctl --instance default \
  certificate-recover https://appliance.example:8443

The code is read from standard input rather than placed in the process list. Activate a host-CLI recovery or renewal with:

sudo systemctl reload rfsentry-server@default.service

This signals only the main Server process, preserving established connections and archive workers. The renewal service uses the same reload action and skips stopped instances. Listener address, port and alias changes still require a planned restart; certificate-only replacement does not. Issuer rotation under the same appliance root requires no client change; root replacement does.

Starting an enabled Server instance also starts its matching renewal timer. Disabling or stopping an instance does not cause package upgrades to re-enable that Server; a later explicit start restores its timer dependency.

Independent instances and private roots

Each NAME owns account rfsentry-srv-NAME, UUID, port and private directories:

  • /etc/rfsentry-server/instances/NAME: managed configuration and previous pointer.
  • /var/lib/rfsentry-server/NAME: identity/credential generations and local operational/WAL/index state, separate from external archives.
  • An explicitly selected archive root; never an automatic fallback directory.

Private directories/files are 0700/0600. Local state requires a supported local filesystem, not a network archive root. Configuration updates use exact revision and digest checks and atomic pointer publication; stale changes are rejected. Use GUI/API or a private validated serverctl configure FILE request, not hand edits to credential generations. Unsupported schemas fail rather than being guessed.

sudo rfsentry-serverctl --instance research init
sudo rfsentry-serverctl --instance research tls-auto \
  https://server.example:18445 127.0.0.1
sudo systemctl enable --now rfsentry-server@research.service

Creating an additional instance does not start it. Set a distinct port before starting. No master, automatic trust or shared admin credentials exists.

Storage selection and faults

Prepare an explicit directory and service-account access on the host. Then:

sudo rfsentry-serverctl --instance default authorize-archive /srv/archive/default
sudo systemctl daemon-reload
sudo systemctl restart rfsentry-server@default.service

Check and save the returned filesystem/mount/root binding in Storage. This is sandbox authorization and capacity/permission readiness, not crash durability or universal NAS support. Probes are bounded, disposable, no-follow writes only within the selected root; periodic checks are read-only. Missing/replaced mounts, read-only/full/permission/path-substitution faults leave administration available. No write is redirected to an underlying root directory. Replaced storage requires explicit rebind. Path changes do not migrate/delete data. The component never formats, mounts, recursively chowns or scans arbitrary directory trees.

Explicit ingestion and record retention

Record WebSocket subscriptions probe transport liveness after 15 seconds of silence and time out after 30 seconds without received traffic. A healthy quiet appliance responds to the ping without creating records or forcing a commit. record-delivery-liveness-timeout enters the existing bounded reconnect and durable-resume path; retained pending records follow the normal disconnect persistence policy. A missing/restarted source still produces explicit gap handling. This does not change retention, dirty-age settings or ACK authority.

In Appliances, check permitted metadata and add the desired stream policies. For each stream, enable record ingestion and save. Disabling it closes only that subscription and persists across restart. Full grants cover later-added streams without pairing again; narrow grants retain their stream restrictions. No subscription controls acquisition, tuning, jobs or another SDR connection.

Canonical envelopes and original payloads go into checksummed append-only segments under the bound root. Indexes/WAL and durable cursors remain on local storage. The C++ subscriber ACKs completed connection ordinals only after segment/directory synchronization and a durable SQLite index/resume commit. Supplemental configuration/boundaries are retained before dependent data and do not advance the original forward cursor. Reconnect uses original source identities with fresh transport IDs and exact uint64 strings.

The default missing-history policy pauses. Ring expiry, owner restarts and missing dependencies appear as explicit gaps. Deliberately select continuation from available history to proceed without claiming missing records arrived. An empty available snapshot follows future oldest history, preserving the old durable cursor until a record is committed. Integrity conflicts pause ingestion.

Enable retention separately and set age/byte limits explicitly. Age uses first Server receipt UTC; required dependencies survive, and deleted identities retain tombstones so retries cannot repopulate expired history. Deletion bookkeeping survives restart. Switching roots creates a separate archive/index generation; it never inherits another root's progress. Back up archive and local state together. Full/read-only/inaccessible storage returns an explicit archive fault until restored; administration remains available and ACK advancement stops.

Catch-up negotiates a bounded record window with compatible appliances and keeps receiving while an earlier batch becomes durable. The limits are 4096 outstanding deliveries / 4 MiB wire credit and at most 2048 records per group. The default persistence target is 1 MiB of canonical records. A batch can exceed that target by one maximum-size record. Compatible browser readers can access validated records from bounded Server RAM while the batch is pending. The browser shows separate available and durable positions. Durable-only readers wait for persistence, and the Server acknowledges delivery after the durable commit.

Storage's Record persistence controls set the 64-KiB–1-MiB target, optional maximum dirty age (blank by default), and separate WAL checkpoint trigger (16 MiB by default). Necessary partial groups report their reason: ordered boundaries, credit/count limits, cache pressure, explicit drain or shutdown. Older appliances retain their 32-delivery window and 16-record groups. The cache charges at most 8 MiB per active stream / 64 MiB total and backpressures pending input instead of dropping it. Reads do not force commits. Retention yields after bounded indexed work and preserves pending dependencies. New archive segments target 64 MiB, increasing possible physical retention slack to one such data segment per stream. Existing archive formats and identities are preserved. Actual catch-up performance awaits operator evaluation.

Stream/overview panels expose received versus durable counts/bytes, rates, bounded backlog, catch-up/live state, cursor/ACK, reconnects, gaps, usage, retention and storage faults. configuredMode, enable flags and retention limits always describe the committed saved policy. runtimeMatchesConfiguration is false and state is reconfiguring until the observed ingestion policy matches; observedRuntime preserves the earlier mode/state separately. A retiring peer may still report activity during that interval. Missing runtime observations do not prove active ingestion or effective retention. Both Server processing modes archive records; native reconstruction additionally requires explicit processing activation for the stream. Diagnostic coverage/range APIs support exact byte verification. Dedicated stream-scoped browser credentials use the bounded binary record path for catch-up and continued reads. The format, durability assumptions and recovery limits are in components/rfsentry-server/ARCHIVE.md.

Index schema 2 and storage capacity

The 0.6.11 storage correction uses SQLite index schema 2. Current qualification and acceptance boundaries are in the validation chapter. Canonical RFSR records and format-1 segment/frame bytes are unchanged. This is shared archive infrastructure for all supported record protocols; it does not add Server reconstruction or activate another roadmap phase.

The index stores each original producer/stream/run tuple in a dictionary, full SHA-256 values as 32-byte binary values, and original sequences/revisions as lossless fixed-width big-endian bytes. Internal IDs never replace original identities on the public protocol. Live rows use stable, non-reused integer ordinals for verification and compact dependency references. Original owner bytes remain in segment frames; the current owner is retained in progress state. Configuration records own normalized digest/revision entries, and dependent records reference those original records.

Retention commits an exact retired-record entry and removes the live row and its dependency edges in one FULL-synchronous transaction. Retired evidence keeps the original identity binding, full canonical hash, forward/supplemental status, epoch, receipt and normalized policy reference. Exact membership can be noncontiguous. Unknown sequence holes remain unknown. A same-identity/different- content retry remains a conflict after retirement; same-content retries cannot restore expired payloads.

Each retired entry also retains a bounded set of stable dependency ordinals. This preserves the no-ACK rule if a still-retained dependency is quarantined later, without retaining the old pair-of-full-identities dependency tables.

Retired physical extents remain indexed while a segment contains retained content. Only after unlink and directory synchronization may bounded cleanup discard those extents and completed segment bookkeeping. Exact completed segment-ID ranges and a durable high-water mark prevent reuse or resurrection. Quarantine remains separate and charged; policy retention does not erase its required evidence or dependencies.

Exact conflict evidence has residual lifetime growth. It is much smaller than a fully indexed live record, but retaining exact hashes for arbitrarily old identities is not constant-space storage. The metadata budget includes it. Exhausting that budget pauses admission; it does not silently expire this guarantee or enable deletion. Size metadata budgets from measured bytes per record and arrival rates, not a universal database/archive ratio.

Small canonical records can exhaust metadata capacity before the archive-byte retention limit. Measure the operator's actual record mix, arrival rate and retired-index growth when sizing allowances; no universal storage ratio is qualified. Ingestion evidence keeps its measured workload and durability limitations separate.

Limits and accounting

Storage → Record persistence exposes recordStorage.capacity:

Setting Default Meaning
maximumMetadataBytes 32 GiB Allocated database/WAL/SHM files, other retained index generations in this instance, and charged temporary work.
maximumPhysicalBytes Unset Optional combined limit for selected archive segments and accounted metadata.
localReserveBytes 256 MiB Free-space floor for the local index and its private temporary directory.
maximumWalBytes 64 MiB WAL admission allowance, separate from the checkpoint trigger. Must be at least twice that trigger.
maximumTemporaryBytes 32 MiB Bound on temporary logical extents, charged before growth and until the owning file closes.
resumeHeadroomBytes 16 MiB Additional filesystem/metadata headroom before resuming. WAL-specific hysteresis is capped as described below.
maintenancePages 64 Maximum page-reclamation steps per maintenance turn; supported range 1–1,024.
maintenanceMilliseconds 20 ms Work-admission budget for page reclamation; supported range 1–100 ms. A storage syscall/sync can outlast it.
cacheKiB 4,096 KiB SQLite page-cache allowance; configurable to 65,536 KiB on capable hosts.

Values named Bytes remain exact decimal strings. The existing archive-root storage.minimumFreeBytes and canonical-data retention policies remain separate. Older configurations without a capacity object retain their existing settings; effective defaults include a WAL allowance large enough for the saved checkpoint trigger. Changing capacity settings never turns on ingestion or retention.

Admission measures the actual index, archive and temporary filesystems. Roles on the same filesystem share one free-space floor (the larger reserve), while their proposed writes are added together. Independent volumes are checked independently. Canonical retained bytes are informational content accounting and are not added again to physical segment bytes. Unrelated filesystem consumers are reflected in actual available space, even though their files are outside rfSentry's allocation budget.

The capacity view distinguishes database logical size, allocated blocks, used pages, reusable free pages, WAL/SHM allocation, temporary charges and physical segment allocation. Used pages include retired evidence and indexes. A freelist is reusable SQLite space, not already-returned filesystem space. Allocated-block figures are filesystem reports; shared/CoW extents and filesystem metadata can make filesystem-wide usage differ. Temporary work uses page-rounded logical charges plus working reserve; filesystem preallocation can differ. Segment allocation is updated after synchronization/reconciliation, and other index generations carry their inventory timestamp. Unavailable or stale measurements are not displayed as zero. Per-table dbstat analysis is diagnostic work, not status polling.

The selected archive's segment budget does not enumerate abandoned data roots outside the current storage binding. Their consumption still reduces filesystem headroom. Retained SQLite generations under this instance's local state are included without opening their database files; more than 256 generations needs an explicit inventory review. SQLite WAL stays on qualified direct local storage. A network or FUSE archive is not a fallback location for its index.

Limits are admission controls with bounded in-flight reservations and working margin. External consumption can race a successful check: SQLITE_FULL, ENOSPC, read-only storage and failed synchronization still stop durable ACKs. Expected capacity pressure preserves the healthy archive owner for reads where possible and retries prerequisites. Actual I/O/transaction faults retain recovery. Admission resumes only after measured headroom satisfies the request and hysteresis. Retention being disabled never authorizes emergency deletion.

WAL recovery first reserves the next write, then requires additional headroom of min(resumeHeadroomBytes, (maximumWalBytes - writeReservation) / 2). This permits legitimate small WAL budgets to recover after checkpoint/truncation, including a 1-MiB checkpoint trigger and 8-MiB maximum with omitted/default 16-MiB resume headroom. Filesystem reserves and metadata hysteresis retain their full configured values. A write that exceeds the WAL allowance still blocks. Normal ingestion and pending recovery batches shrink to fit small WAL budgets; the original identities, ordering and durability-before-ACK rule are unchanged. A held reader must release its snapshot before reclamation can recover space; no restart or policy edit is needed when the actual pressure clears.

Server outputs and browser download policy

Processing follows the existing explicit enabled/paused stream intent. Its compiled goes-hrit processor declares the goes-color child: Natural Color (C02+C13) and Day/Night (C02+C07+C13). No separate stream activation is needed. Recent retained product pages are matched first, with a durable bounded cursor for older retained products. Exact scans and received navigation must agree; missing constituents/calibration remain waiting or recorded deferred outcomes. Base imagery, messages and ingestion continue independently. Native dimensions, received masks, recipe revisions and constituent lineage accompany the output.

The current storage blocker now remains visible ahead of historical continued frame-gap warnings. Once the actual storage condition clears, existing automatic retries resume; no Pause/Resume ritual or tail/checkpoint reset is required. Processing shows the receipt time of its last durably processed input separately from current operation, last progress and last publication. This is source-time age, not a percentage or subtraction across sequence epochs.

For browser products, update the appliance-served client as well as the Server, then reload. All products → Retention and storage contains Server download policies: native imagery When selected (default) or Recent products with an hour window; originals On demand (default) or Retain automatically; and an aggregate download-staging allowance (default 256 MiB). Thumbnails and text arrive first. Opening native imagery or exporting original evidence queues that immutable revision through the existing resumable worker. Staging limits must fit the requested encoded assets; they do not lower native image quality. Local asset roles and native offline readiness are shown in product details.

Safely identified unsupported schemas are counted and deferred while compatible products continue. Upgrade/reload triggers a bounded resnapshot when adapters change. Canonical originals, prior locally retained products and pending byte ranges remain intact. Successful polling still says nothing about upstream production progress. No local recipe is enabled to mask missing Server output.

Only the proven equivalent non-image all-received/unknown coverage label is normalized. Bounded maintenance clears equivalent conflict markers while leaving all immutable revisions/assets intact. Genuine imagery or scientific differences remain conflicts. Detailed deferred recipe contexts are retained for the most recent 256 tasks; older retry contexts expire explicitly, without requeueing the same failed input. Changed constituents can trigger a fresh recipe attempt.

The accepted baseline passed the exact-source main/release gates described in validation. The historical sanitizer archive-I/O watchdog incident remains unexplained; acceptance and later green runs do not diagnose it or establish sustained speed.

Optional Server processing

Processing is independently enabled in the Processing administration tab. Version 0.6.12 fixes worker CPU/memory observation under the installed service's procfs privacy policy. A failed stream shows its concrete error beside the stream; committed products and recovery state remain retained. Missing peak measurements are not presented as measured zero usage.

Processing forms retain unsaved edits through live status polling, validation errors and failed saves. Concurrent saved changes are reported instead of being silently overwritten. Editing any resource/storage allowance selects Manual; otherwise blank CPU/RAM caps are filled from current measured headroom when available, with RAM rounded down to whole MiB. Review those caps before saving. Selecting Auto preserves explicit caps; blank CPU/RAM fields use Auto headroom. Every setting has help available by hover, keyboard focus or tapping its ? button; Escape dismisses it. Size fields accept M/MB, G/GB (decimal) and Mi/MiB, Gi/GiB (binary), including whole-byte decimal fractions such as 1.5 GiB. Bare numbers retain the previous MiB interpretation. Bandwidth also accepts /s. Units are converted with exact integer arithmetic.

The page displays the selected <archive-root>/products-v1/ destination for assets, checkpoints and staging, and the local-state products/ directory for catalogs. The asset allowance controls capacity, not placement; there is no separate product-root selector. Changing archive selection does not migrate data.

Pairing and upgrades do not enable processing. Choose a retained source stream and compatible processor, then explicitly enable it. Recent history target keeps the existing minutes value (30 by default). Live processing starts near the current committed tail independently. Missing history is processed newest-first, with records inside each bounded range decoded forwards. With older backfill off, the recent target bounds history; with it on, the active product-retention policy and available original records set the older boundary.

A live appliance connection is not required. Pause/resume preserves products and recovery. If a saved foreground owner is substantially stale relative to the retained tail, a durable handoff preserves its unfinished work while a new owner handles current reception. Ordinary decoder errors retain their existing bounded continuation/recovery behavior. Historical work yields first under resource pressure; checkpoint/replay allowance must hold retained state plus a replacement. No processing or storage cap is automatically raised.

The Processing page shows Live reception, the historical primary range and any temporary recovery handoff separately. Historical range reads may extend through bounded assembly context; that does not mean all intervening history is complete. Live input time, historical coverage, publication time and acquisition ACKs are different positions. Independent browser copies retain their own policy.

The first processing-settings save explicitly transitions configuration schema 3 to schema 4. Older binaries cannot interpret that configuration; a rollback requires a reviewed compatible configuration, never an archive reset. The record database remains schema 2. GOES native imagery/text, partial-to-complete revisions, authenticated delivery and durable browser import are implemented. Compact thumbnails and exact native pixel-coverage masks are produced on the Server. The registered color task adapter completes the existing Natural Color and Day/Night recipe path. Functional acceptance is recorded above; hardware working sets remain workload-specific measurements.

In the GOES workspace, choose Configure source → Server products, select the trusted Server and original stream, and Apply. This mode uses existing local archives/viewers and suppresses corresponding local reconstruction/recipe jobs, including after reload. Old preferences keep records mode. Direct IQ, direct appliance records and older archive-only Servers remain supported. Fallback to local reconstruction requires the configured explicit fallback policy and its own valid recovery state. Cached products remain viewable offline.

The importer fetches details, thumbnails and text first. Native/original assets follow the selected policy described above and share the aggregate staging allowance. Quota, cancellation and transfer failure preserve prior published products and durable pending work. Discovery, asset transfer and local publication have separate progress indicators; none acknowledges acquisition records.

Auto measures effective affinity/cgroup CPU and available/cgroup memory before admission. Operator controls cap CPU allowance, workers, aggregate memory/cache and background I/O. Protected memory defaults to 512 MiB, with another 512 MiB when acquisition co-location is explicitly selected; at least one CPU core is protected when effective capacity permits it. Manual mode requires explicit CPU/memory caps. An ordered context has one assembler owner; independent contexts can run concurrently. Observed RSS and future peaks are shown separately. Peak worker RSS remains visible after a failure, alongside the declared bound. Immutable-product tasks use the same worker allowance and can run after an idle record owner yields a durable safe point. The task framework is exercised by a test-only observation processor and production GOES color recipes. A record owner can also yield at a durable point when a pending recipe fits only after its reservation is released. An individually oversized or unsupported item receives a durable failure/deferred outcome so later eligible work continues, without reducing native dimensions. Aggregate admission or recovery-storage pressure instead pauses affected work until its condition clears; no progress advances if its outcome cannot persist.

GOES's default declaration reserves about 3.125 GiB per active owner, including assembly, original files, codec/output and checkpoint replacement peaks. This is a conservative admission declaration, not a measured minimum working set or a real-time throughput claim. Advanced processor settings (assemblyBytes, fileBytes, outputBytes) must fit both processing and old-plus-new checkpoint storage allowances. A small native fixture was tested at 16/1/2 MiB respectively; that fixture does not establish a budget for real full-disk imagery. Do not multiply the old browser defaults by a chosen worker count without admission. Checkpoint-capable adapters report an exact frozen save size when available, so disk admission reserves the actual replacement plus retained state. An older configuration revision remains incompatible for restore even when its already prepared product outputs can be recovered safely.

Products occupy separate catalog/staging/checkpoint storage. Default allowances are 8 GiB assets, 128 MiB catalog/index working space, 2 GiB checkpoints/replay, 1 GiB staging and a 64 MiB free-space floor on the selected product/archive volume. A separate local catalog volume uses the configured index reserve plus its SQLite commit allowance, not the archive-sized product reserve. Shared volumes use the higher floor once, plus both write allowances. Preflight and SQLite growth admission use the same reserve scope. Raising the product reserve to protect a large archive does not also require that much free space on an independent system/index volume. Neither existing reserve setting is rewritten. Failed temporary writes remain charged until recovery reclaims their owned files. Missing/replaced directories or mounts fail explicitly. Metadata reservations and recoverable storage pressure use bounded retries; clearing the pressure does not require a Server restart. Registered immutable tasks trigger a lazy product-catalog schema-2 upgrade, separate from record schema 2. Input replacement fences stale tasks, and task output/completion share one publication transaction. Expired task inputs are counted and cannot pin source-product retention. Pairing and upgrades still enable no processing. Product pressure pauses optional work; record ACKs continue to require record fsync/index durability. Per-stream product retention is independently explicit; it does not change record or browser retention.

From 0.6.14, product reads use independent catalog snapshots while publication continues. Four readers and eight queued requests bound interactive admission; ordinary publishing does not cause immediate catalog-busy rejection. Product WAL and shared-memory files count toward the catalog/index allowance, with a 32-MiB WAL ceiling and non-waiting checkpoint reclamation. This changes only the product catalog, not record persistence or its WAL settings.

Products are exposed through scoped /api/v1/products/ GET operations. Assets are immutable and exact range requests bind the revision and hash. Expired change cursors require a bounded snapshot. The product catalog is additive, with its own versioned recovery protocol documented in PRODUCTS.md.

The validation chapter distinguishes local fixtures, exact-source CI/installed evidence, hardware measurements and operator acceptance. The server runtime/durable-publication checkpoint at signed 510bfdf passed full CI and installed amd64/ARM64 qualification, including durable product delivery and mounted ingestion faults with independent index-volume exhaustion. This qualifies that historical checkpoint. The accepted 744b840 baseline includes production recipes, fetch policies and subsequent persistence/retention and live-history repairs. Its release passed both installed architectures. Minimum production working sets and sustained throughput are not inferred from those tests; Phase 6 performance acceptance remains open. No production installation or automatic activation is implied by this source.

Obsolete partial cleanup (0.6.19)

Superseded partial renderings are automatically retired independently of the stream's product-age/byte retention switch. Current/best observations, original files and conflicting complete evidence remain. Pending reads and active recipe inputs are protected. Existing obsolete history is scanned in bounded background passes; cleanup does not require a reset or processing restart ritual.

The Processing page reports retired partial revisions, pending ownership checks, queued packs and reclaimed asset bytes. Disk usage can fall later than metadata retirement because shared packs require bounded compaction. Unused checkpoints are reclaimed ahead of ordinary asset garbage so a large expiry backlog cannot strand processing at its checkpoint allowance. Original-role assets remain until whole-product retention expires their observation. Catalog schema 5 keeps compact replay identities and cannot be opened by an older Server binary.

Refresh appliance/browser assets before running this Server update. Updated clients replace obsolete pending downloads with current revisions, preserve previously retained products and report unavailable selected downloads explicitly. Their archive cleanup proceeds in short background passes and is visible under Startup & activity. The Server's three-day product policy still requires Enforce this stream’s Server product retention; it does not set browser retention.

Product pack storage (0.6.18)

New publication batches use product catalog schema 4. Small assets share streamed packs targeting up to 64 MiB, with time and resource bounds; large assets still stream without fitting in RAM. Old loose assets stay readable. The catalog's physical asset usage includes dead space inside retained packs. Maintenance reclaims wholly unused packs, or copies a bounded mixed pack with reserved replacement space before atomically changing asset locations.

Pack data and required directory entries must synchronize before products and processing progress become recoverable. A failed write/sync preserves prior progress. Older binaries cannot open schema 4; do not downgrade against the new catalog. Checkpoint and record archive formats are unchanged.

Writers close before their final allocation charge is recorded. A bounded checkpoint reconciliation pass and maintenance scans correct older persisted open-file allocation charges using current file attributes. This changes neither payloads nor configured limits. Open range readers keep their file allocation charged until the descriptor closes.

Timing counters distinguish packed assets, pack synchronizations and their elapsed time. Batch adaptation uses CPU work while a separate publication deadline limits waiting. Required durability barriers and operator I/O/storage limits remain in effect; sustained catch-up is an acceptance measurement.

Processing throughput diagnostics (0.6.17)

Under Processing → Processing timings and batches, inspect the current and maximum page target, completed batches, limiter waiting, snapshot reason/size and elapsed timings. Targets grow for inexpensive catch-up work and shrink for output pressure; a large page may commit only its consumed prefix. Native pixels, original identities and gap ordering are preserved.

GOES catch-up can grow to the existing 512-record archive page ceiling. Near-live pages still shrink for latency. Historical processing coalesces intermediate mosaics over at most five minutes of source time or two minutes of active work; complete candidates, original files/messages and terminal partials remain eligible immediately. Deferred strips survive checkpoints; native geometry, pixels and evidence are unchanged. Age-only product retention uses the receipt index without scanning every asset to calculate an unused byte total.

Full processor snapshots now use a bounded 1–64 MiB replay allowance based on snapshot size and configured checkpoint space, with a 15-minute active-work age limit. Small checkpoints remain small. Idle state changes and failure recovery still snapshot when required. Product/progress commits remain durable in between. Routine cleanup is coalesced and performs no directory sync when nothing was removed; real storage/WAL pressure triggers bounded reclamation promptly. Each new file is synchronized after its final rename/link state; verified durable duplicates reuse their existing bytes. Interrupted staging still needs a file barrier. Asset directory barriers are combined at batch preparation; synchronized file data and both directory barriers precede durable publication. Independent admitted jobs can proceed while a new file is being synchronized; its staging reservation remains charged until safe completion.

The I/O allowance is still an actual ceiling. A 500-MiB snapshot at 16 MiB/s needs at least 31 seconds of allowance, plus verification and durability costs. CPU headroom does not remove that delay. The implementation does not increase this operator setting automatically. Timers overlap: checkpoint/staging totals include worker IPC and limiter waiting. Compare deltas over the same interval, including a checkpoint, and watch the input receipt timestamp against wall time to judge whether backlog is shrinking. No history is skipped to make this look faster.

Short catch-up intervals do not establish sustained realtime throughput. Include a full checkpoint and storage synchronization costs when measuring backlog recovery. Product evidence and the Server roadmap retain the unqualified performance scope and unresolved watchdog observation.

Reclamation and checkpoints

Genuinely new databases enable auto_vacuum=INCREMENTAL before creating tables. Maintenance uses positive, bounded incremental_vacuum(1) steps, bounded dictionary/extent cleanup, and non-waiting checkpoints. Turns rejoin the existing ingestion/read queue. Filesystem space is reported as returned only when actual allocation falls; a reader can prevent WAL reuse or physical truncation after pages have become free. SQLite's checkpoint trigger and journal_size_limit are not hard WAL caps. See the SQLite reclamation rules and WAL checkpoint behavior.

An existing auto_vacuum=NONE database cannot be converted by merely setting a pragma after tables exist. Schema 1, unknown versions, incomplete schema 2, or incompatible reclamation mode stop with an actionable error. There is no automatic migration, full VACUUM, rebuild or erase. Old binaries reject schema 2. A missing main database with sidecars or existing segments remains a recovery blocker; the Server never guesses the lost retention journal.

Reviewed fresh-generation reset procedure

This is a preparation/runbook, not authorization to execute a live reset. Use a reviewed, exactly qualified package and a host-specific inventory before destructive work. A clean generation is preferred to a large legacy migration.

  1. Stop the named instance and verify ActiveState=inactive, MainPID=0, and no archive worker remains. Identify independent or paused maintenance holding the database/candidate files. Terminate the identified maintenance without resuming its queued destructive steps. A stopped service alone does not prove those independent processes have released their descriptors. For an independently SIGSTOP-paused maintenance process, the reviewed action is SIGKILL of that reverified process identity, followed by proof that it has exited and released its descriptors. Never send SIGCONT to let cleanup finish.
  2. Record the instance UUID, archive UUID/storage generation, root inode/mount binding, package/source identity, configuration digest, and each stream's separate ingestion, retention, processing and continuation intent. Verify private ownership/permissions. Keep /etc/rfsentry-server/instances/NAME, /var/lib/rfsentry-server/NAME/identity.json, its complete credentials/ directory, grants, certificate/enrollment material and settings intact.
  3. Identify exact database families, including moved files: main, matching WAL and SHM, plus any compaction candidate. An original main file may require its WAL to contain all committed pages. Do not detach it or use immutable=1 to pretend the WAL does not matter. A candidate whose verification was paused is not an authoritative backup. Neither database alone restores history without the matching segments and archive identity document.
  4. Preflight each real filesystem. Include preserved originals/candidates, existing segment files, the new metadata allowance and WAL/temporary work, package/staging space, and reserves. Charge WAL/temporary allowances once within the metadata budget, and combine reservations on shared volumes. Recheck actual available space immediately before execution. If preservation does not fit, stop and present exact sizes and alternatives; do not delete a preserved database to make the plan fit.
  5. Choose explicitly between a new empty, separately bound archive directory (preserving old history) and discarding the reviewed old generation. The discard list is only that generation's verified segment-*.rfs files and .rfsentry-archive marker, plus separately reviewed database files if their disposal is authorized. Never recursively remove the instance state or configuration roots. Keeping the archive directory itself preserves its inode; replacing it requires an explicit fresh storage probe/binding through the existing configuration API/CLI and sandbox authorization.
  6. Install the reviewed package while the instance stays stopped. On the empty selected root, the Server creates a new archive UUID/storage generation and a fresh schema-2 database under archives/<newArchiveUuid>/index.sqlite. Never transplant the old marker, old cursor, progress table or clean-stop receipt into it. Existing Server, appliance, stream and original producer identities are preserved. Old archive-specific fences must be rejected.
  7. Start only the instance and subscriptions that the operator authorizes, preserving the recorded inactive flags. An empty archive has no durable history to ACK. Initial replay uses the configured appliance's currently retained originals, with its real availability boundary and the existing continuation policy. Record history before that boundary as unknown/missing; never claim the former durable cursor is present. Other authorized archives may hold evidence, but this procedure does not add peer replication/import or promise replay beyond an existing supported source. No browser OPFS is reset.
  8. Verify new generation/schema/reclamation mode, identity and access, effective settings and independent capacity measurements. Confirm original records become durable only after segment/directory sync and FULL SQLite commit, then check source-observed ACK progress, bounded queues, truthful gap reporting and old-fence rejection. Read back exact originals. Check that previously inactive streams and receiver settings remain unchanged.

Rollback to the old history is possible only while its original database/WAL, all required segments, marker and compatible configuration/package remain coherent. If the old segments were deliberately discarded, an old database file is not a history rollback. Keep any optional preserved generations explicitly inactive and account for their space; their existence must not be mistaken for the new archive's progress. The reset remains a separate operator acceptance step after source and isolated qualification.

Registrations, credentials and operational truth

Pairing stores stable identities, authenticated private trust and one independently revocable full-appliance credential per server. Existing/advanced records.read registrations remain narrow, separate from full grants and server-admin sessions. Explicit checks validate TLS/pin and then identity/configured stream metadata for full grants, or permitted availability for readers. No receiver/job/subscription/ history payload is opened. Narrow-reader empty history cannot establish original source identity. Saving a registration or opening administration does not activate ingestion. Restart resumes only explicitly enabled subscriptions and never wakes hardware.

Codes last five minutes, allow at most five initiated attempts appliance-wide, are single-use and expire on gateway restart. Use pair-status (no code revealed), pair-cancel, and sudo rfsentry-jobctl records-revoke server-SERVER_UUID on the appliance. Revocation ends that server's authorized connections/leases without ending independent continuous jobs. A new code rotates enrollment and preserves existing stream intent. Removing only a stored server credential is not remote revocation. Interrupted pending enrollment can resume from its private saved credential; discarding it does not revoke a grant the appliance already committed.

Admin GUI credentials provide user/token rotation and revocation. Values are write-only/redacted and use explicit unchanged/replace/remove where applicable. Stream byte/age limits have validated units and explicit unset states. Directed peer configuration remains disabled/inactive, not reachable/authenticated/active until separately checked/implemented in later authorized phases.

Upgrade, removal and qualification

Startup and background verification

Normal startup checks the selected storage, archive identity, index/schema, segment boundaries and unfinished write tails. It restores the original durable cursor and verifies required resume data before subscribing. A successful stop records a generation-bound clean checkpoint after pending records finish; startup invalidates that receipt before further writes. Older installations without a receipt use tail reconciliation rather than a full historical payload scan. Missing storage/indexes, incompatible identities and unsafe write boundaries remain explicit blockers.

Historical record verification runs in bounded turns that yield to ingestion and readers. A separate read-only process checks SQLite integrity at low CPU/I/O priority. Its ordinary startup attempt is bounded by read/work admission rather than an unbounded full-database startup scan. It releases its snapshot and reports Deferred when the 32-MiB/500-ms startup allowance is reached, or when WAL pressure approaches the configured allowance. A single blocking storage operation can outlast the work budget. It is attempted once per service start or selected-archive change; a full offline integrity check is explicit diagnostic work. Deferred does not mean verified; record integrity checks on reads and ingestion remain enforced. Confirmed structural index corruption blocks new writes. Eligible damaged records retain their durable quarantine evidence and explicit gaps.

Use Activity for separate write-readiness, historical-verification and index- verification status, elapsed work, last progress, source connections, HTTP/session counts, and graphs of received/durable flow, pending bytes and verification work. The endpoint uses cached observations and remains available while the archive is busy. Recent history is bounded to 600 samples and 128 coalesced events in Server RAM: reloading the browser preserves it, restarting the service resets it. No additional TSDB or per-record metrics writes are required. Significant transitions and rate-limited progress are also emitted to the system journal.

Graphs report whether Activity data is loading, unavailable, sampled but zero, or plotted; an empty-looking plot is not used for all four conditions. A temporary Activity request failure keeps the last successful samples visible, shows the reason and retries automatically. Returning the tab to the foreground requests a fresh observation. After upgrading an already open administration page, refresh that Server page once to load the newly installed static UI; this does not restart or navigate an appliance reception page.

Ready/Live does not mean historical verification is complete. Conversely, background verification need not prevent normal receiving and durable commits. No percentage or ETA is invented when total work is unknown. Original cursor identities remain exact strings, and byte displays include both readable units and exact byte counts. Gap, cursor and retention summaries keep the full structured details behind expandable controls. Gap downloads contain the bounded recent status snapshot; they do not contain or reconstruct the missing payloads.

Package maintenance

Reinstall/upgrade preserves identity, private configuration/credentials and archives; only already-active instances are restarted. Disabled/stopped instances stay so. Removal stops/removes the packaged server; purge deliberately retains private operator state/drop-ins/accounts and never deletes external archives. Retained private state needs appropriate host backup protection. Same-schema rollback is supported only when the binary understands the saved configuration and both catalog schemas. Processing-enabled configuration uses schema 4 and the product catalog advances through schema 6; older binaries may reject them. Downgrades need compatible stopped-state configuration/credentials and product recovery assets. The package never silently downgrades a schema or resets an archive.

Automatic leaf renewal occurs on service start within its last 30 days, preserving the private authority. A continuously running instance still needs a planned restart before the one-year leaf expires. Provided certificates remain explicit; authority rollover requires new client trust. Failed generation/renewal retains the last valid saved configuration. See the component README for full limits.

The build/qualification scripts are in components/rfsentry-server/. Installed tests run only in explicit disposable Debian 13/systemd guests. Local headless Chromium uses normal trusted TLS and its real sandbox, not remote CDP. The validation chapter and retained Phase 6 report distinguish native execution, emulation, CI evidence and outstanding limits; >4-GiB memory64 qualification remains Phase 12. Normal v* releases now have one coordinated publisher/checksum manifest; client-v* remains independent. No production release tag is created by qualification.

Processing failures and recovery (0.6.15)

Failure journaling adds product-catalog schema 3; it does not change the canonical record archive's schema 2. Older product-catalog binaries cannot open the upgraded catalog. Any rollback therefore needs a compatible stopped-state product catalog and its referenced assets/checkpoints, in addition to compatible configuration. Do not reset or downgrade the live archive as a processing-error workaround.

An ordinary failed file, message, image or derivative is recorded while later valid work continues. The Processing page shows the current operation, original attempted/durable/checkpoint positions, last actual progress/publication, and failed/deferred and recovery counts. “Awaiting input” means the available range has been consumed; resource waits, storage blockage, recovery and stale supervision are shown separately. A status refresh does not count as progress.

Use View recorded failures for the cause, original range/hash, processing stage, native diagnostic and recovery disposition. Reprocess this item is a deliberate bounded retry after a decoder/policy correction. It uses retained input at lower priority, preserving live processing and existing products. Inactive or paused streams remain inactive. A completed replay describes the attempted range; resulting product revisions/outcomes describe its result. Expired source data or diagnostic detail is reported explicitly.

Expected bad input is isolated immediately. Unsafe worker exceptions/crashes replace the worker and replay committed state. Two repeatable failed executions lead to an adapter isolation decision and durable outcome; unchanged failed work must not wedge restart. Memory admission and storage failures instead wait for their actual condition to clear. Failed outcome/checkpoint commits never advance processing. Record ingestion and its durability-before-ACK rule stay independent.

Failed thumbnails leave valid native imagery available. Unsupported views are not advertised as ready. Current GOES contracts bound details to 16 MiB and 65,536 message records, and text views to 16 MiB; individual limits are reported with original evidence retained. Product/failure retention does not delete independently retained browser observations or alter record retention policy.

Browser product mode shows transfer and upstream processing health separately. Successful polling of an unchanged catalog is not evidence of production progress. Older Servers without health capability report it as unknown. Server-products mode retains its explicit fallback policy and does not silently start local assembly/recipes.

For native diagnostics, retain the exact executable/package identified by the reported ELF build ID. Use readelf -n to compare it and addr2line -Cfipe with reported module-relative addresses to resolve matching frames. Signal reports include the instruction and executable base; catch-site-only stacks are labeled. No unrestricted core dump is enabled. See qualification and limits for accepted-source evidence and the unexplained sanitizer watchdog incident. The separate mount-table incident and recovery issue #7 remains deferred unless it recurs. Its later successful restart did not establish the trigger or fix automatic recovery; do not weaken missing-mount checks.