Skip to content

FCC Amateur Corpus Storage Benchmark

Date: 2026-09-07
Status: benchmark complete; recommendation approved for Phase 9B

Decision boundary

This benchmark does not implement or select RF Sentry's permanent FCC data plane. It compares three browser-side candidates against the same current FCC weekly Amateur Radio Service archive and the workloads RF Sentry expects to perform.

The recommendation is:

Use SQLite/WASM with OPFS as the durable relational truth, plus an independent RAM-first hot tier for compact enrichment and common dossier data. Keep RF Sentry observations in IndexedDB as designed in Phase 9A.

The operator approved that choice on 2026-09-07. The recommendation is based on query breadth, predictable relational reconstruction, FTS5 search, integrity checking, and lower long-term maintenance cost. SQLite's slower weekly build and larger durable footprint are real costs, not dismissed because it was favored before the benchmark.

Environment and corpus

The measurements ran in the actual RF Sentry origin on the established remote Windows Chrome session:

  • Chrome 151 on Windows 11
  • 32 GiB reported device memory
  • 16 logical processors
  • cross-origin isolation and SharedArrayBuffer available
  • OPFS and IndexedDB available
  • initial dynamic origin quota approximately 10 GiB
  • persistent-storage grant not present
  • SQLite 3.53.0, official WebAssembly build, FTS5 enabled
  • SQLite VFS: official opfs-sahpool
  • ZIP reader: zip.js 2.11.2

The FCC response used for every candidate was:

  • ETag: "bc95906-65ad03144f0b8"
  • Last-Modified: Sun, 06 Sep 2026 13:08:06 GMT
  • ZIP bytes: 197,744,902
  • decompressed .dat bytes: 727,435,437

The source manifest and parser produced:

File Physical source lines Logical records
AM.dat 1,692,886 1,692,886
CO.dat 235,543 235,529
EN.dat 1,695,148 1,695,148
HD.dat 1,695,148 1,695,148
HS.dat 5,150,613 5,150,613
LA.dat 4,807 4,807
SC.dat 61,789 61,789
SF.dat 86 86
Total 10,536,020 10,536,006

CO.dat contains 14 continuation lines belonging to preceding comment records. Treating every physical line as an independent pipe record is wrong. The benchmark joins those continuations for relational use while retaining the unaltered source ZIP and raw byte offsets. This is a permanent-importer requirement.

Candidate representations

OPFS plus custom RAM indexes

  • Lossless raw .dat files in OPFS.
  • One sorted binary USI -> byte offset/length index per record type.
  • NDJSON license summaries and RAM callsign/previous-callsign maps.
  • Indexed mode slices records from OPFS; Eager mode hydrates all raw files.
  • Search and future relationship traversal require application-owned indexes and query code.

SQLite/WASM plus OPFS

  • Every logical record retained with type, USI, callsign keys, and raw record.
  • Relational summary table for the hot-license projection.
  • Relational indexes for USI, callsign, previous callsign, status, and names.
  • FTS5 corpus for investigative entity/comment/attachment/condition search.
  • The same RAM callsign, summary, and prior-callsign maps as the other candidates; SQLite and RAM are complementary, not alternatives.

Chunked IndexedDB plus RAM indexes

  • Eighteen large durable Blob artifacts per generation: raw files, binary indexes, summaries, and manifest/count data.
  • The same custom query/runtime model as the OPFS candidate.
  • An object-per-FCC-record prototype was rejected as technically non-credible: observed insertion rates projected to tens of minutes and roughly 12 million IndexedDB objects when search indexes were included.

Import, hydration, and replacement

Times are wall-clock seconds. The clean download used 24 verified 8 MiB range requests through the fixed same-origin relay and completed in 47.907 seconds. The importer validates counts before activation.

Workload OPFS/RAM SQLite/OPFS Chunked IDB
Empty-origin download 47.907 47.907 47.907
First build after ZIP is ready 99.684 350.742 81.645
Complete download + first build 147.591 398.649 129.552
Weekly replacement build 106.153 348.669 81.078
Atomic generation activation 0.006 0.004 0.009
Indexed startup hydration 6.827 31.890 6.627
Eager startup hydration 9.447 37.170 7.287

SQLite index and FTS construction accounted for about 44.1 seconds of its first build and 43.1 seconds of replacement. Its remaining cost came from inserting and retaining the full relational/raw corpus.

These are background-import costs, not decode-path costs. A valid old generation remains active while a new generation is built and validated.

Durable storage

Sizes below use decimal GB and include the 197.7 MB source ZIP where noted.

Storage state OPFS/RAM SQLite/OPFS Chunked IDB
One generation artifact 1.466 GB 2.049 GB database about 1.397 GB attributable
Normal origin after first build, including ZIP 1.664 GB 2.353 GB 1.693 GB
Replacement peak with old + new + ZIP 3.130 GB 4.499 GB 3.154 GB
Old-generation deletion reclaimed space yes not in shared SAH pool yes

The official SAH-pool VFS retained its allocated slot capacity after the old SQLite file was unlinked. This is reusable capacity rather than weekly unbounded growth, but it would leave a roughly 4.5 GB high-water allocation if both generations share one pool.

The permanent SQLite design should therefore use separate generation-scoped SAH-pool directories/workers. After atomic activation, close the old worker and delete its whole pool directory. Steady storage should then return near one 2.05 GB database plus the source archive, while replacement peak remains about 4.5 GB.

RF Sentry query workloads

Hot-tier measurements used 1,000 callsigns sampled across the sorted corpus. Dossier and related-record measurements used 100 USIs. Values are mean milliseconds unless noted.

Workload OPFS Indexed OPFS Eager SQLite Indexed SQLite Eager IDB Indexed IDB Eager
Callsign to all USIs 0.0011 0.0013 0.0012 0.0009 0.0015 0.0009
Current license resolution 0.0030 0.0016 0.0020 0.0016 0.0030 0.0021
Prior-callsign/multi-USI traversal 0.0030 0.0018 0.0028 0.0016 0.0028 0.0030
Complete USI dossier 7.815 0.013 0.232 0.117 3.067 0.019
History/comment/condition traversal 6.942 0.011 0.128 0.097 3.127 0.009
Active + TX + name filter, total ms 69.2 64.5 39.2 47.9 100.1 66.8
Investigative text search, total ms 1,594.6 570.0 2.1 1.3 881.3 574.9

All candidates exceed the compact enrichment target: normal callsign lookup, current-record resolution, and prior-callsign traversal are effectively instantaneous and require no persistent-storage access.

SQLite is substantially better for Indexed-mode dossiers and dramatically better for unanticipated forensic search. Eager custom representations make a known-USI dossier very fast, but do not solve arbitrary multi-field search or new relationship queries without building and maintaining additional custom indexes.

The text-search rows are representative rather than bit-for-bit identical. SQLite counted 1,994 matching indexed documents using FTS token semantics; the custom candidates counted 2,012 raw substring occurrences. Permanent acceptance fixtures must normalize search semantics before using these values as exact result-equivalence tests. The timing magnitude and maintenance tradeoff remain clear.

RAM-first policies

Browser-wide memory measurement is approximate. Chrome's measureUserAgentSpecificMemory() is useful for coarse sizing but proved stop-the-world enough to disturb realtime testing, so memory and DSP-impact measurements were run separately.

Policy Expected resident working tier on this corpus
Indexed custom/IDB about 0.8 GB above an idle RF Sentry page
Eager custom/IDB about 1.52 GB above an idle RF Sentry page
Indexed SQLite + hot maps/cache about 0.48 GB incremental in the measured live session
Eager SQLite + hot maps/page cache about 1.62 GB incremental in the measured live session

Peak transient browser memory during import/replacement was approximately 1.7 GB above the relevant baseline for each candidate. SQLite's absolute browser footprint was higher in its run because the live RF Sentry page also held its configured 1 GB IQ timeshift history.

On the measured 32 GiB client, Auto correctly resolves to Eager. Indexed remains appropriate for constrained clients. The user-selected policy should remain authoritative, with storage pressure and unsupported memory telemetry reported rather than silently overriding it.

Realtime RF/DSP impact

The long idle-decoder SQLite benchmark ran while a live 2.4 MS/s WFM lane and the normal spectrum/waterfall pipeline were active for 813.6 seconds:

  • IQ cadence: 73.25 transport frames/s
  • spectrum cadence: 42.77 frames/s
  • audio state remained running
  • DSP state remained ready
  • p99 browser frame interval: 6.10 ms at a 165 Hz display target

One 14.1-second UI interval occurred during whole-origin memory measurement. That was instrumentation interference, not importer work, and is why the decoder runs below disabled that API.

A diagnostic IQ decoder was then kept active during one complete non-replacement run per candidate:

Candidate Observed duration IQ frames/s Spectrum frames/s Added decoder source-overflow resets UI p99 Worst UI interval
OPFS/RAM 140.7 s about 73.5 about 42.9 15 6.12 ms 12.10 ms
Chunked IDB 127.3 s about 73.5 about 42.4 2 6.11 ms 18.19 ms
SQLite/OPFS 548.6 s 73.30 42.70 15 6.13 ms 2,956 ms

Audio and DSP stayed running, decoder processing continued, and no renderer frame gap over 50 ms occurred in the OPFS or IDB runs. SQLite had two such gaps, including one 2.96-second pause. In all three runs, decoder reset growth occurred as a short burst immediately after the benchmark worker/archive path started; it did not continue during the remaining parse, write, indexing, hydration, or query work.

Operator decision (2026-09-07): the brief startup-only reset burst is accepted as a future QoS improvement and is not a Phase 9B blocker. Production still keeps archive/import work in a dedicated worker, uses bounded transactional batches and cooperative task-queue yields, and never invokes heavyweight whole-origin memory measurement during reception. A persistent reset rate, wedged receiver, or degradation throughout the several-minute import remains unacceptable. Receiver-aware scheduling, relay throttling, and adaptive backpressure are intentionally deferred unless later measurements justify their complexity.

Maintainability comparison

Candidate Strength Long-term cost
OPFS/RAM Simple durable artifacts; fastest direct Eager dossiers RF Sentry must own every index, join, search feature, migration, and query optimizer
SQLite/OPFS Relational integrity, SQL, FTS5, mature planner, flexible forensic queries Slowest weekly build; larger storage; WASM/VFS lifecycle and schema migrations
Chunked IDB Fastest measured build; clean transactional generation deletion Still a custom database/query engine; Blob slicing and custom indexes for every relationship/search shape

The custom candidates' import advantage is substantial, not modest. However, the product requirement is a complete investigative/reference corpus whose future query shapes are deliberately broad. Maintaining a private relational and search engine would be the larger long-term cost. SQLite earns its added complexity here, provided the production importer is throttled and generation storage is isolated as described above.

Transport finding discovered by the benchmark

The long run exposed a Phase 9A gateway crash unrelated to storage choice. libwebsockets invokes protocol lifecycle callbacks without per-connection user storage. The new HTTP callback unconditionally derived its embedded FCC peer from that null user pointer, and receiver/relay wakeups could therefore segfault the gateway.

A narrow null-session guard was added, built, integration-tested, deployed, and live-verified. Receiver start, a sustained decoder, and a clean 197.7 MB 24-range FCC transfer then completed without another gateway crash. The relay still emits occasional libwebsockets Unhandled state 282 diagnostics after completed range requests; those did not affect correctness or service availability, but should remain a transport-hardening follow-up.

Browser compatibility gate

Chrome behavior above is measured, not inferred, and Chrome is the reference Phase 9B browser. No Firefox executable or remote Firefox debug endpoint was available, so Firefox import time, storage quota, OPFS cleanup, and memory behavior remain unmeasured and are not a release blocker.

The official SQLite WASM documentation reports OPFS support in Firefox and describes the SAH-pool VFS as broadly supported. If Firefox support is later requested, verify:

  • OPFS worker access and generation deletion;
  • SQLite SAH-pool behavior if approved;
  • quota and persistent-storage behavior;
  • import/hydration/query correctness;
  • continued audio, waterfall, DSP, and decoder operation.

Reference: https://www.sqlite.org/wasm/doc/trunk/persistence.md

Approved permanent Phase 9B architecture

Following approval:

  1. Keep the verified source ZIP and provenance manifest in OPFS.
  2. Build a complete SQLite generation in a dedicated enrichment worker and a generation-scoped SAH-pool directory.
  3. Retain every FCC file/type/field, exact logical raw record, relationships, counts, and schema/source provenance; preserve multiline comments and the original ZIP.
  4. Validate manifest counts, foreign/key consistency appropriate to ULS, required indexes, FTS population, representative dossiers, and integrity_check before activation.
  5. Atomically switch a tiny active-generation pointer only after validation.
  6. Close the prior generation worker, then remove its entire pool directory.
  7. Hydrate callsign-to-USI, current-summary, previous-callsign, and common relationship data into RAM at startup. Never query durable storage on the decode hot path.
  8. Implement Auto, Eager, and Indexed as RAM/cache policies over the same SQLite truth; Auto selects Eager on the measured 32 GiB client.
  9. Coordinate tabs with one enrichment owner per generation using Web Locks and BroadcastChannel, because one SAH pool must not be concurrently owned by multiple workers.
  10. Use bounded batches and cooperative yields. Record the observed startup-only reset burst for later receiver-aware QoS work without making zero added source gaps a Phase 9B acceptance requirement.
  11. Keep RF Sentry observations and learned station state in IndexedDB, apart from FCC regulatory truth.
  12. Request durable browser storage when supported, show quota/persistence state, and preserve the previous valid generation on every failure.

The benchmark gate itself made no permanent storage-format change. The production importer was implemented only after the operator approved this report and remains tracked separately in the Phase 9B acceptance ledger.