GOES HRIT verification and limits¶
Curated source-specific evidence, not a new run or qualification of current source. Dates, measurements, failure outcomes and original source identifiers retain their recorded meaning. Old tags/assets have been retired and original commits may no longer resolve after the privacy rewrite. Where identifiers were sanitized earlier, the original hardware/run association cannot be independently recovered from the public derivative. Current acceptance lives in the roadmap.
Scope: receive-chain and capture-center foundation, independent product/file comparisons and partial live evidence. The aviation/SATCOM roadmap owns current gates. A ten-minute live interval recorded 10,768 validated CADUs and no transport/input/handoff drops; 208 complete EMWIN products survived reload. The independent 63-CADU OTA excerpt begins/ends mid-file and proves no completed product. Live imagery start-to-completion and concurrent receiver isolation remain open; synthetic fixtures and independent imagery comparisons do not close them.
Authoritative signal geometry¶
The implementation follows the NOAA/NESDIS HRIT/EMWIN reception specification:
- 1694.1 MHz center frequency;
- 927 ksymbols/s BPSK;
- 400 kbit/s user data;
- rate-1/2, constraint-length-7 convolutional coding;
- four-way interleaved CCSDS Reed-Solomon (255,223);
- square-root raised-cosine filtering with alpha 0.3; and
- approximately 1.205 MHz necessary bandwidth.
The manifested worker input is 1.854 Msps complex IQ, exactly two samples per symbol, produced by RF Sentry's existing rational WASM channelizer. The normal 2.4 or 3.2 Msps hardware capture remains authoritative and shared with the spectrum, waterfall, detector, DVR, and other decoder consumers.
Open-source qualification¶
The receive chain was evaluated against established implementations before it was integrated:
pietern/goestoolsrevision80ece1a7ab8a93fb5dfa50d47387ae7c4a8f2a73is the primary algorithmic reference. It is BSD-2-Clause and its bundledlibcorrectrevisiona3c8890b43486822900a2c65a33701fe35bdc0e2is BSD-3-Clause. Its complete native application and device/network stack are not embedded in the browser.- Phil Karn's
libfecfixed CCSDS Reed-Solomon procedure is the reference for error correction. Its LGPL terms are compatible with RF Sentry's GPL distribution, and attribution is retained in source and the decoder notice. - SatDump revision
1aedd602055c154401384a8cc7876713bb371254was inspected as a GPL-3.0 independent pipeline/oracle. It is not a runtime service or browser dependency. libaec1.1.3 is the BSD-2-Clause independent reference for the bounded CCSDS 121.0/Rice image decoder. Its upstream vector is exercised directly; RF Sentry does not embed the native library in the browser.
RF Sentry uses a small freestanding worker-WASM implementation of the required receive subset instead of porting a device-owning desktop stack. It performs complex DC removal, bounded AGC, BPSK carrier recovery, symbol selection, encoded-ASM correlation, soft Viterbi decoding, HRIT NRZ-M decoding, CCSDS derandomization, dual-basis conversion, and four-way RS validation. It emits exact RS-validated CADUs and receiver-quality telemetry. Validated CADUs transfer directly to a dedicated SATCOM product worker; they do not pass through React state or the generic decoder media library.
Independent validation¶
Routine CI generates compact raw CU8 IQ independently of the decoder. The fixture encoder and decoder do not share implementation code, and the test asserts the encoded HRIT ASM against the pinned goestools reference vector. The samples traverse the production rational channelizer and decoder WASM. Coverage includes:
- a valid CADU with known spacecraft/virtual-channel/frame-counter fields;
- carrier and DC offset;
- noise-only input;
- an integrity-destroying corrupted transmission; and
- a source-IQ discontinuity inside a frame.
An optional qualification test accepts an external stereo 16-bit WAV through
RFSENTRY_GOES_OTA_WAV. It was exercised with the independently recorded
Lucas Teske GOES-16 HRIT sample described as 1694.1 MHz, 5 Msps, captured on
2017-01-17. A 32 MiB local prefix with SHA-256
b34eadddd172d8713be686d29b28541fe86fa3158bdf17f9b57a4ba48530a326
produced validated CADUs through the same production channelizer/decoder. The
recording is not redistributed because its page does not grant an explicit
redistribution license.
Run the deterministic gate with:
cd components/rfsentry-web
../../scripts/node-toolchain.sh run npx vitest run \
test/goesHritRawIqPipeline.test.ts
Run the optional independent qualification with:
RFSENTRY_GOES_OTA_WAV=/path/to/recording.wav \
../../scripts/node-toolchain.sh run npx vitest run \
test/goesHritRawIqPipeline.test.ts
A separate optional product qualification reads independently published goestools VCDU packet recordings and compares RF Sentry's reconstructed bytes with output from the pinned, unmodified goestools executable. Three five-minute recordings supplied 16,144 frames and 1,852 file sessions: 242 complete files, including nine packetized Rice image segments, and 1,610 explicitly unsupported legacy GOES-N EMWIN fragments. All 233 non-image comparable files matched the reference output byte-for-byte. All nine packetized Rice images decoded to the declared pixel geometry with no missing scanlines; this also exposed and fixed a low-entropy reference-bit ordering defect that a simpler Rice vector did not exercise. The same run parsed 232 DCS products and one weather bulletin. The recordings and derived reference output remain qualification inputs rather than checked-in assets.
Official NOAA OSPO LRIT reference files provide a second product-format source independent of both RF Sentry and the goestools recordings. The qualification opens two expanded Rice-header imagery products (500 x 500 and 5424 x 339), a readable severe-weather bulletin, and a deflated EMWIN radar GIF. It verifies exact image geometry, rendered previews, ZIP CRC/length, and the resulting product-family presentation. NOAA's expanded reference files retain their Rice header, whereas live source packets contain one compressed scanline per packet; RF Sentry recognizes both forms without treating decoded pixel length as an on-wire progress denominator.
Run it with:
RFSENTRY_GOESTOOLS_PACKET_DIR=/path/to/published-vcdu-recordings \
RFSENTRY_GOESTOOLS_REFERENCE_DIR=/path/to/goeslrit-output \
RFSENTRY_NOAA_LRIT_DIR=/path/to/noaa-ospo-lrit-samples \
../../scripts/node-toolchain.sh run npx vitest run \
test/satcomGoestoolsQualification.test.ts
The normal gate intentionally reports environment-gated skips:
- the million-observation aviation soak runs only with
RFSENTRY_AVIATION_SOAK=1; and - the external HRIT recording runs only when
RFSENTRY_GOES_OTA_WAVnames the locally held recording, which RF Sentry cannot redistribute; and - the published VCDU/reference comparison runs only when the two
RFSENTRY_GOESTOOLS_*paths name locally held evidence; and - the official-format qualification runs only when
RFSENTRY_NOAA_LRIT_DIRnames locally held NOAA OSPO reference files.
The second test was run for this tranche and passed. It exercised WAV PCM conversion, production rational channelization, carrier/timing acquisition, encoded-frame synchronization, soft Viterbi decoding, derandomization, four-way Reed-Solomon correction, and validated CADU output. Generated fixtures remain the routine deterministic tests; they are not represented as independent OTA evidence.
Gain and measurement semantics¶
The installed satcom1 Nooelec NESDR SMArTee XTR reports an Elonics E4000
tuner (RTL-TCP tuner type 1), not an R82xx tuner. Its supported manual gain
steps are retained in 0.1 dB driver units:
The gateway learns the tuner type and reported gain count from the RTL-TCP dongle header, publishes the matching table, rejects unsupported changes after capability discovery, and performs one explicit nearest-step normalization if the receiver's initial saved setting predates discovery. Existing R820T/R828D receivers retain their own 29-step table.
Classic RTL-TCP has command delivery but no gain readback. The UI and GOES reports therefore distinguish requested/supported gain, accepted command, and hardware readback rather than calling an accepted request verified. Tuner AGC, RTL digital AGC, and browser audio AGC are separate states.
A controlled local sample test at fixed 1694.1 MHz and 2.4 Msps, with tuner AGC and RTL digital AGC disabled, measured raw IQ RMS changing from about -43.00 dBFS at -1.0 dB gain through -38.53 dBFS at 14.0 dB, -26.57 dBFS at 29.0 dB, and -20.74 dBFS at 42.0 dB. No clipping was observed. This confirms that the receiver's hardware gain responds; waterfall autoscaling can conceal that amplitude shift visually.
Capture-center artifact and correction boundaries¶
RF Sentry's normal GOES geometry deliberately places the 1694.1 MHz VFO 100 kHz below the 1694.2 MHz logical capture center. With ordinary zero-IF tuning, that is also the physical E4000 LO and the artifact is shown at 1694.2 MHz. After the decoder channelizer translates the wanted VFO to baseband, the capture-center artifact moves to a 100 kHz offset. A blocker applied only after that translation cannot remove it.
The installed /usr/bin/rtl_tcp and librtlsdr0 are Debian
2.0.2-2+b1; the process loads /lib/aarch64-linux-gnu/librtlsdr.so.0 and
the serial-keyed service supplies no hidden correction arguments. The exact
installed path has several distinct layers that must not be conflated:
| Layer | Present behavior and evidence |
|---|---|
| RTL2832 demodulator compensation | librtlsdr initializes zero-IF DC cancellation and IQ estimation/compensation. Classic RTL-TCP exposes no control or readback for it. |
| E4000 calibration | The installed source contains E4000 calibration/LUT routines, but e4k_init() compiles LUT generation out and explicitly disables time-varying E4000 DC correction. Installed librtlsdr exports no supported runtime calibration call. RF Sentry does not issue speculative register writes. |
| Driver offset tuning | Installed librtlsdr supports offset tuning for E4000, and classic RTL-TCP carries it as command 0x0a. At 2.4 Msps the driver uses a 2.040 MHz offset and RTL digital translation while preserving the 1694.2 MHz logical capture center. RF Sentry exposes this lane-locally and reapplies it on acquisition, sample-rate change, retune, and reconnect. |
| RF Sentry software residual correction | A lane-local one-pole complex-mean estimator operates before VFO translation. It defaults to 20 ms (about 8 Hz), is adjustable from 1–500 ms, and resets on acquisition, geometry changes, and discontinuities. It removes constant/very-slow complex bias, not the broader zero-IF pedestal. |
| Display-only suppression | None. RF Sentry does not delete FFT bins, paint over the line, or auto-notch arbitrary narrow signals. |
Software correction is applied independently in realtime DSP, scientific and display FFT, candidate-analysis, and decoder-channelizer paths. It operates on derived samples only; authoritative CU8 retained by DVR/recording is unchanged. Reports record both correction settings and show raw/residual complex means.
Classic RTL-TCP proves only that RF Sentry transmitted a command and streaming
continued. It has no per-command result, driver-state query, or hardware
readback. The UI therefore says command sent, not accepted or read back.
Support on this appliance was verified separately by opening serial
90000004 through the exact installed librtlsdr: both
rtlsdr_set_offset_tuning(1) and rtlsdr_get_offset_tuning() reported
enabled, then the diagnostic restored offset tuning off before release.
Raw IQ frames store the active offset-tuning bit in the capture-options word
at header byte 76. DVR/replay can therefore reproduce the capture mode. The
header's tunerHz remains the logical target; the UI separately shows the
driver-derived physical LO and never describes it as readback.
Gain-dependent live artifact characterization¶
A fixed-geometry sweep used 1694.2 MHz logical center, 1694.1 MHz VFO, 2.4 Msps, both tuner and RTL digital AGC disabled, and -1.0, 19.0, and 42.0 dB supported E4000 gains. Full-capture RMS rose from -42.01 through -34.98 to -21.71 dBFS, while normalized complex mean stayed about 0.0010–0.0013. There was no clipping, source gap, or overflow. Higher gain therefore raises wanted energy and surrounding noise above an approximately fixed conversion term; it does not calibrate the artifact away.
A 1,048,576-bin fixed-scale analysis resolved two components: a broad roughly 77–78 kHz center pedestal, about 24–27 dB above local shoulders at low gain, plus a coherent exact-center component. The sweep restored the original gain before release and did not alter other receiver lanes.
Driver offset-tuning comparison¶
Controlled six-second low-gain captures were taken offset-off, offset-on, then offset-off. With offset tuning off, the broad center feature was 26.60 and 23.99 dB above the local median with 76.8 and 78.5 kHz +3 dB widths. With it on, the broad pedestal disappeared; the remaining feature was the exact-center coherent component, approximately one reporting FFT bin. The repeat-off capture rules out a one-time reconnect coincidence.
At 2.4 Msps the logical 1694.2 MHz target remains unchanged while the driver-derived E4000 LO becomes about 1692.160 MHz. High-gain captures retained the complete HRIT-shaped occupied spectrum. Production decoder replays also remained viable: offset-off produced 94/98 validated CADUs with residual software correction off/on, and offset-on produced 102/106. These are separate live captures, so the small difference is not claimed as a controlled frame- rate improvement. It does prove that offset tuning removes the broad artifact without clipping the required 1.205 MHz channel or preventing HRIT decoding.
Exact-IQ software-correction comparison¶
The same 3.168 seconds of locally captured high-gain CU8 was replayed through the production channelizer/worker with correction off and on. Raw bytes were asserted unchanged. The runs produced 63 versus 62 validated CADUs, 38 versus 41 corrected RS bytes, two versus zero uncorrectable frames, and no frame-sync loss. The coherent center component fell by 36.4 dB; this does not establish a statistically significant frame-rate benefit.
The same raw low-gain offset-off capture explains why the live toggle seemed ineffective. At the deployed 26 ms setting (about 6.1 Hz), it reduced the exact center bin from -64.6 to -85.2 dBFS, but bins at about 100 Hz and beyond—and therefore the visible ~78 kHz pedestal—were unchanged. It was operating in the intended display/analysis/decoder paths, but correcting a much narrower component than the visible feature. Widening that filter by tens of kilohertz would alter wanted HRIT and is not an acceptable substitute for driver offset tuning.
The measured starting configuration for this E4000/GOES path is driver offset tuning enabled, with software mean correction retained as optional residual cleanup. Reception remains judged by validated/FEC evidence. The GOES view now measures center-feature frequency, FFT dBFS peak, local contrast, and +3 dB width independently from complex mean.
With the browser lane stopped, the gateway-level raw comparison is reproducible with distinct output names:
cd components/rfsentry-api
node test/live_client.mjs /etc/rfsentry/auth.token \
/etc/rfsentry/tls/server.crt --goes-dc-sweep-seconds=6 \
--goes-dc-capture=/tmp/goes-offset-off.cu8 \
--goes-offset-tuning=off
node test/live_client.mjs /etc/rfsentry/auth.token \
/etc/rfsentry/tls/server.crt --goes-dc-sweep-seconds=6 \
--goes-dc-capture=/tmp/goes-offset-on.cu8 \
--goes-offset-tuning=on
The sidecar JSON records gain, logical center, driver-derived physical LO,
capture-options bit, continuity, clipping, complex mean, and center-spur FFT
summary. Files are created mode 0600 and never overwrite prior evidence.
Receive-to-product architecture¶
The supported product path is:
shared lane IQ
-> manifested rational channelizer
-> GOES-R HRIT worker/WASM demodulator and FEC
-> exact 892-byte RS-validated CADUs
-> dedicated SATCOM product worker
-> VCDU / M_PDU / CCSDS source packets
-> bounded HRIT/LRIT file assemblies
-> product-specific interpretation and previews
-> IndexedDB metadata + OPFS original/preview objects
-> bounded keyed UI snapshots
The Pi still owns only receiver transport and static web assets. Acquisition and product processing continue while the SATCOM product views are hidden, but only while the owning RF Sentry browser session and decoder job remain open. Closing that browser/session ends the decoder; this work does not install a Pi product daemon or a second receiver connection.
Physical receiver, lane, configured target, service profile, decoded relay spacecraft ID, source-product satellite, product family, observation time, reception time, and acquisition session are separate fields. The configured target is persisted per lane/receiver/route. A target such as GOES-19 is operator intent, not decoded identity. Image/header metadata can establish a different originating spacecraft for a relayed product without changing that target.
Each lane/job/live-or-replay/source-epoch tuple owns an independent assembler. Virtual channel, APID, packet sequence, file, and image-segment state cannot cross an acquisition-session boundary. A DVR seek opens a new source epoch, so packets from separate replay cursor intervals cannot combine. Live and replay CADUs likewise cannot combine into a file, and simultaneous receiver lanes cannot combine segments merely because their LRIT identifiers match.
The file layer implements:
- VCDU counter wrap, duplicate rejection, and gap accounting;
- M_PDU first-header pointers and source packets spanning frame boundaries;
- CCSDS source-packet sequence flags, wrap, duplicates, gaps, and CRC-16;
- HRIT/LRIT primary and applicable secondary headers;
- interleaved virtual channels and APIDs;
- file completion versus incomplete, corrupt, unsupported, abandoned, and still-receiving state;
- bounded active files, packet/file sizes, processing queues, decompression, image dimensions, segment groups, and idle expiry; and
- authoritative packet/frame/CRC/file-length/segment provenance retained with every product.
A validated CADU is never presented as a validated product. Reception that begins mid-file remains visible as activity but cannot fabricate missing headers. A new file, decoder restart, retune/session change, timeout, or safety eviction terminates the affected assembly explicitly.
Supported product handlers¶
The current GOES-R HRIT/EMWIN profile supports:
- uncompressed one-bit and eight-bit LRIT imagery;
- bounded CCSDS 121.0/Rice scanline decoding;
- image-segment retention and complete mosaic assembly;
- explicit magenta missing-line presentation for reconstructable partial Rice images, without synthesizing received pixels;
- received GIF, JPEG, and PNG broadcast graphics;
- bounded stored/deflate EMWIN ZIP members with length and CRC-32 checks;
- readable weather/administrative text with WMO heading, AWIPS identifier, issuing office, issue time, and correction marker where supplied;
- NOAA DCS container and platform transport records, including address, receive time, quality/receiver fields, and retained raw payload; and
- unknown formats as original bytes plus parsed LRIT headers and an explicit unsupported explanation.
RF Sentry does not infer DCS sensor units, station geography, or payload meaning from opaque station-specific bytes. Those records remain searchable by platform address and exportable in their original LRIT container. Likewise, received weather graphics are broadcast graphics, not raw radar volumes.
The single enabled RF service profile is NOAA GOES-R HRIT/EMWIN. The CCSDS and LRIT file machinery is reusable, but other HRIT/LRIT downlinks are not called supported until their modulation, framing, compression, channel conventions, and product handlers are independently qualified. In particular, the published GOES-N qualification archive contains legacy file type 214 EMWIN fragments; RF Sentry preserves them as unsupported rather than pretending that GOES-R and GOES-N service profiles are interchangeable. GRB is outside this profile.
Product archive and operator activity¶
Original reconstructed LRIT files are authoritative. Browser OPFS stores originals and rendered previews when available; IndexedDB stores metadata, headers, parsed text/records, content hashes, and reception occurrences. IndexedDB blob fallback keeps the archive functional where OPFS is unavailable. SHA-256 content deduplication retains one original object while preserving each live/replay reception occurrence. An older replay occurrence cannot move a newer live product backward in the archive or masquerade as a new current product.
The default browser-local policy is 8 GiB and 30 days. Both values are
operator-configurable. Retention removes originals and derivative previews
together; originals and disposable previews are accounted separately.
Archive queries run in the product worker, and large binaries never enter
localStorage or React state.
The product workspace provides overview, all-products, imagery, weather and messages, environmental, Acquisition Activity, and reception-diagnostic views. Product detail supports original export, rendered-image export, deletion, protocol/provenance inspection, reception history, text reading, DCS raw-record inspection, image fit/native zoom, and time-ordered animation of compatible images.
Acquisition Activity publishes at most four keyed snapshots per second while visible. It shows stable per-file rows, channel/APID, stage, unique assembled versus wire-observed bytes, exact progress only when a declared total exists, packet/frame gaps, duplicates, rejection, file start/end and joined-mid-file counts, segment state, waiting/failure reason, and five-second rolling rates. Source-time validated/useful rates are separate from replay-processing speed. Hiding the verbose view stops activity-row serialization while overview and product updates, assembly, accounting, processing, and persistence continue. Hiding the complete GOES workspace stops all product presentation publications. Product publications are keyed upserts rather than repeated copies of the bounded 1,000-item recent index.
Workspace and test contract¶
The Decoder Workspace provides a compact lane selector and protocol-neutral workspace navigation. SATCOM contains a GOES subsection without taking ownership of Aviation, FT8/FT4, NRSC-5, station intelligence, or the active decoder. Lane changes and workspace changes do not open a second receiver connection.
The GOES surface offers continuous bounded alignment trends plus four explicit tests: an 8-second signal check, 30-second decode check, 2-minute stability test, and 10-minute reliability test. Tests observe the current lane and never silently retune, change gain/sample rate, or stop another decoder. They are cancellable and retain honest partial reports.
Reports use uncalibrated dBFS, locally estimated FFT signal/background contrast, clipping, IQ/decoder discontinuities, encoded synchronization, validated frames, corrected bytes, uncorrectable frames, lock losses, and reacquisition. They never convert dBFS to dBm or use frequency/brightness as a GOES identity claim. Configuration changes are flagged because they invalidate controlled before/after comparisons. Reports are bounded and retained in the browser only.
Reports distinguish carrier tracking, timing tracking, frame synchronization, and Reed-Solomon failure. An uncorrectable CADU candidate no longer increments lock-loss/reacquisition by itself; frame lock is dropped only after a sustained two-second absence of validated frames. Processed-IQ duration derives from decoder samples/rate and is shown alongside wall-clock duration. Nominal frame opportunities use the verified 927 ksymbol/s and 16,384 encoded symbols per CADU, while attempts remain explicitly labeled accepted synchronization candidates—not total reception completeness. Source gaps, input discontinuities, capture-frame drops, and decoder-chunk drops are reported as separate counters. Validated binary CADUs are queued before lower-priority JSON presentation events; the bounded handoff has its own explicit drop counter, and any nonzero value makes a diagnostic interval unstable.
An IQ frame marked source-overflow has a precise local meaning: the
gateway's bounded per-browser outbound WebSocket queue had to evict a frame.
It does not by itself identify a USB/tuner loss. A decoder-input-overflow or
decoder-queue-overflow instead identifies a browser-side presentation or
decoder backlog. The deployed raw diagnostic client sustained this receiver
at 2.4 Msps without a gateway overflow; a short browser startup interval did
overflow the gateway queue while other browser-local hydration work was
active. New reports preserve these classes rather than folding them into RF
or FEC lock loss.
Raw-IQ RMS, clipping, DC, and continuity summaries are produced independently of signal-track promotion, at a bounded four-Hz maximum publication cadence. The contrast estimate integrates the 1.205 MHz HRIT band and compares it with guard-separated adjacent FFT bins. If an interactive fine trace is too narrow to contain a defensible background region, the estimator uses the current full-capture spectrum instead. These are relative FFT/dBFS diagnostics, not a calibrated demodulator SNR or received power measurement.
Decoder counters may restart after an input discontinuity. A test report sums the monotonic intervals on both sides of those resets instead of allowing a reset to erase already validated frames from the report.
Current acceptance boundary¶
The receive-to-product implementation remains an operator-acceptance candidate. Live operation now establishes sustained RF-to-product delivery and durable retention on the designated SATCOM1 receiver. Independent evidence separately establishes exact transfer/file reconstruction plus representative DCS, weather-text, ZIP, graphics, and Rice handlers. The short independent OTA excerpt contains 63 validated CADUs but starts and ends mid-file, so it is not claimed as a completed-product reference.
Still open at this checkpoint:
- operator visual acceptance of product browsing, export, deletion, and activity accounting on the current broadcast;
- a live file-start-to-completion imagery observation; the current live run completed weather products while independent evidence qualified imagery;
- an explicit concurrent second-receiver isolation exercise with the product worker active;
- richer interpretation of station-specific DCS payloads only where an authoritative definition is available;
- profile-specific implementations and independent qualification for other HRIT/LRIT services; and
- optional downstream product/map overlays or multispectral enhancements.
These boundaries do not make unknown content disappear: reconstructed originals, headers, integrity state, and acquisition provenance remain inspectable and exportable.