FCC Amateur Corpus Storage Benchmark¶
Date: 2026-09-07
Status: benchmark complete; recommendation approved for Phase 9B
Decision boundary¶
This benchmark does not implement or select RF Sentry's permanent FCC data plane. It compares three browser-side candidates against the same current FCC weekly Amateur Radio Service archive and the workloads RF Sentry expects to perform.
The recommendation is:
Use SQLite/WASM with OPFS as the durable relational truth, plus an independent RAM-first hot tier for compact enrichment and common dossier data. Keep RF Sentry observations in IndexedDB as designed in Phase 9A.
The operator approved that choice on 2026-09-07. The recommendation is based on query breadth, predictable relational reconstruction, FTS5 search, integrity checking, and lower long-term maintenance cost. SQLite's slower weekly build and larger durable footprint are real costs, not dismissed because it was favored before the benchmark.
Environment and corpus¶
The measurements ran in the actual RF Sentry origin on the established remote Windows Chrome session:
- Chrome 151 on Windows 11
- 32 GiB reported device memory
- 16 logical processors
- cross-origin isolation and
SharedArrayBufferavailable - OPFS and IndexedDB available
- initial dynamic origin quota approximately 10 GiB
- persistent-storage grant not present
- SQLite 3.53.0, official WebAssembly build, FTS5 enabled
- SQLite VFS: official
opfs-sahpool - ZIP reader: zip.js 2.11.2
The FCC response used for every candidate was:
- ETag:
"bc95906-65ad03144f0b8" - Last-Modified:
Sun, 06 Sep 2026 13:08:06 GMT - ZIP bytes: 197,744,902
- decompressed
.datbytes: 727,435,437
The source manifest and parser produced:
| File | Physical source lines | Logical records |
|---|---|---|
AM.dat |
1,692,886 | 1,692,886 |
CO.dat |
235,543 | 235,529 |
EN.dat |
1,695,148 | 1,695,148 |
HD.dat |
1,695,148 | 1,695,148 |
HS.dat |
5,150,613 | 5,150,613 |
LA.dat |
4,807 | 4,807 |
SC.dat |
61,789 | 61,789 |
SF.dat |
86 | 86 |
| Total | 10,536,020 | 10,536,006 |
CO.dat contains 14 continuation lines belonging to preceding comment
records. Treating every physical line as an independent pipe record is wrong.
The benchmark joins those continuations for relational use while retaining the
unaltered source ZIP and raw byte offsets. This is a permanent-importer
requirement.
Candidate representations¶
OPFS plus custom RAM indexes¶
- Lossless raw
.datfiles in OPFS. - One sorted binary
USI -> byte offset/lengthindex per record type. - NDJSON license summaries and RAM callsign/previous-callsign maps.
- Indexed mode slices records from OPFS; Eager mode hydrates all raw files.
- Search and future relationship traversal require application-owned indexes and query code.
SQLite/WASM plus OPFS¶
- Every logical record retained with type, USI, callsign keys, and raw record.
- Relational summary table for the hot-license projection.
- Relational indexes for USI, callsign, previous callsign, status, and names.
- FTS5 corpus for investigative entity/comment/attachment/condition search.
- The same RAM callsign, summary, and prior-callsign maps as the other candidates; SQLite and RAM are complementary, not alternatives.
Chunked IndexedDB plus RAM indexes¶
- Eighteen large durable Blob artifacts per generation: raw files, binary indexes, summaries, and manifest/count data.
- The same custom query/runtime model as the OPFS candidate.
- An object-per-FCC-record prototype was rejected as technically non-credible: observed insertion rates projected to tens of minutes and roughly 12 million IndexedDB objects when search indexes were included.
Import, hydration, and replacement¶
Times are wall-clock seconds. The clean download used 24 verified 8 MiB range requests through the fixed same-origin relay and completed in 47.907 seconds. The importer validates counts before activation.
| Workload | OPFS/RAM | SQLite/OPFS | Chunked IDB |
|---|---|---|---|
| Empty-origin download | 47.907 | 47.907 | 47.907 |
| First build after ZIP is ready | 99.684 | 350.742 | 81.645 |
| Complete download + first build | 147.591 | 398.649 | 129.552 |
| Weekly replacement build | 106.153 | 348.669 | 81.078 |
| Atomic generation activation | 0.006 | 0.004 | 0.009 |
| Indexed startup hydration | 6.827 | 31.890 | 6.627 |
| Eager startup hydration | 9.447 | 37.170 | 7.287 |
SQLite index and FTS construction accounted for about 44.1 seconds of its first build and 43.1 seconds of replacement. Its remaining cost came from inserting and retaining the full relational/raw corpus.
These are background-import costs, not decode-path costs. A valid old generation remains active while a new generation is built and validated.
Durable storage¶
Sizes below use decimal GB and include the 197.7 MB source ZIP where noted.
| Storage state | OPFS/RAM | SQLite/OPFS | Chunked IDB |
|---|---|---|---|
| One generation artifact | 1.466 GB | 2.049 GB database | about 1.397 GB attributable |
| Normal origin after first build, including ZIP | 1.664 GB | 2.353 GB | 1.693 GB |
| Replacement peak with old + new + ZIP | 3.130 GB | 4.499 GB | 3.154 GB |
| Old-generation deletion reclaimed space | yes | not in shared SAH pool | yes |
The official SAH-pool VFS retained its allocated slot capacity after the old SQLite file was unlinked. This is reusable capacity rather than weekly unbounded growth, but it would leave a roughly 4.5 GB high-water allocation if both generations share one pool.
The permanent SQLite design should therefore use separate generation-scoped SAH-pool directories/workers. After atomic activation, close the old worker and delete its whole pool directory. Steady storage should then return near one 2.05 GB database plus the source archive, while replacement peak remains about 4.5 GB.
RF Sentry query workloads¶
Hot-tier measurements used 1,000 callsigns sampled across the sorted corpus. Dossier and related-record measurements used 100 USIs. Values are mean milliseconds unless noted.
| Workload | OPFS Indexed | OPFS Eager | SQLite Indexed | SQLite Eager | IDB Indexed | IDB Eager |
|---|---|---|---|---|---|---|
| Callsign to all USIs | 0.0011 | 0.0013 | 0.0012 | 0.0009 | 0.0015 | 0.0009 |
| Current license resolution | 0.0030 | 0.0016 | 0.0020 | 0.0016 | 0.0030 | 0.0021 |
| Prior-callsign/multi-USI traversal | 0.0030 | 0.0018 | 0.0028 | 0.0016 | 0.0028 | 0.0030 |
| Complete USI dossier | 7.815 | 0.013 | 0.232 | 0.117 | 3.067 | 0.019 |
| History/comment/condition traversal | 6.942 | 0.011 | 0.128 | 0.097 | 3.127 | 0.009 |
| Active + TX + name filter, total ms | 69.2 | 64.5 | 39.2 | 47.9 | 100.1 | 66.8 |
| Investigative text search, total ms | 1,594.6 | 570.0 | 2.1 | 1.3 | 881.3 | 574.9 |
All candidates exceed the compact enrichment target: normal callsign lookup, current-record resolution, and prior-callsign traversal are effectively instantaneous and require no persistent-storage access.
SQLite is substantially better for Indexed-mode dossiers and dramatically better for unanticipated forensic search. Eager custom representations make a known-USI dossier very fast, but do not solve arbitrary multi-field search or new relationship queries without building and maintaining additional custom indexes.
The text-search rows are representative rather than bit-for-bit identical. SQLite counted 1,994 matching indexed documents using FTS token semantics; the custom candidates counted 2,012 raw substring occurrences. Permanent acceptance fixtures must normalize search semantics before using these values as exact result-equivalence tests. The timing magnitude and maintenance tradeoff remain clear.
RAM-first policies¶
Browser-wide memory measurement is approximate. Chrome's
measureUserAgentSpecificMemory() is useful for coarse sizing but proved
stop-the-world enough to disturb realtime testing, so memory and DSP-impact
measurements were run separately.
| Policy | Expected resident working tier on this corpus |
|---|---|
| Indexed custom/IDB | about 0.8 GB above an idle RF Sentry page |
| Eager custom/IDB | about 1.52 GB above an idle RF Sentry page |
| Indexed SQLite + hot maps/cache | about 0.48 GB incremental in the measured live session |
| Eager SQLite + hot maps/page cache | about 1.62 GB incremental in the measured live session |
Peak transient browser memory during import/replacement was approximately 1.7 GB above the relevant baseline for each candidate. SQLite's absolute browser footprint was higher in its run because the live RF Sentry page also held its configured 1 GB IQ timeshift history.
On the measured 32 GiB client, Auto correctly resolves to Eager. Indexed remains appropriate for constrained clients. The user-selected policy should remain authoritative, with storage pressure and unsupported memory telemetry reported rather than silently overriding it.
Realtime RF/DSP impact¶
The long idle-decoder SQLite benchmark ran while a live 2.4 MS/s WFM lane and the normal spectrum/waterfall pipeline were active for 813.6 seconds:
- IQ cadence: 73.25 transport frames/s
- spectrum cadence: 42.77 frames/s
- audio state remained
running - DSP state remained
ready - p99 browser frame interval: 6.10 ms at a 165 Hz display target
One 14.1-second UI interval occurred during whole-origin memory measurement. That was instrumentation interference, not importer work, and is why the decoder runs below disabled that API.
A diagnostic IQ decoder was then kept active during one complete non-replacement run per candidate:
| Candidate | Observed duration | IQ frames/s | Spectrum frames/s | Added decoder source-overflow resets | UI p99 | Worst UI interval |
|---|---|---|---|---|---|---|
| OPFS/RAM | 140.7 s | about 73.5 | about 42.9 | 15 | 6.12 ms | 12.10 ms |
| Chunked IDB | 127.3 s | about 73.5 | about 42.4 | 2 | 6.11 ms | 18.19 ms |
| SQLite/OPFS | 548.6 s | 73.30 | 42.70 | 15 | 6.13 ms | 2,956 ms |
Audio and DSP stayed running, decoder processing continued, and no renderer frame gap over 50 ms occurred in the OPFS or IDB runs. SQLite had two such gaps, including one 2.96-second pause. In all three runs, decoder reset growth occurred as a short burst immediately after the benchmark worker/archive path started; it did not continue during the remaining parse, write, indexing, hydration, or query work.
Operator decision (2026-09-07): the brief startup-only reset burst is accepted as a future QoS improvement and is not a Phase 9B blocker. Production still keeps archive/import work in a dedicated worker, uses bounded transactional batches and cooperative task-queue yields, and never invokes heavyweight whole-origin memory measurement during reception. A persistent reset rate, wedged receiver, or degradation throughout the several-minute import remains unacceptable. Receiver-aware scheduling, relay throttling, and adaptive backpressure are intentionally deferred unless later measurements justify their complexity.
Maintainability comparison¶
| Candidate | Strength | Long-term cost |
|---|---|---|
| OPFS/RAM | Simple durable artifacts; fastest direct Eager dossiers | RF Sentry must own every index, join, search feature, migration, and query optimizer |
| SQLite/OPFS | Relational integrity, SQL, FTS5, mature planner, flexible forensic queries | Slowest weekly build; larger storage; WASM/VFS lifecycle and schema migrations |
| Chunked IDB | Fastest measured build; clean transactional generation deletion | Still a custom database/query engine; Blob slicing and custom indexes for every relationship/search shape |
The custom candidates' import advantage is substantial, not modest. However, the product requirement is a complete investigative/reference corpus whose future query shapes are deliberately broad. Maintaining a private relational and search engine would be the larger long-term cost. SQLite earns its added complexity here, provided the production importer is throttled and generation storage is isolated as described above.
Transport finding discovered by the benchmark¶
The long run exposed a Phase 9A gateway crash unrelated to storage choice. libwebsockets invokes protocol lifecycle callbacks without per-connection user storage. The new HTTP callback unconditionally derived its embedded FCC peer from that null user pointer, and receiver/relay wakeups could therefore segfault the gateway.
A narrow null-session guard was added, built, integration-tested, deployed,
and live-verified. Receiver start, a sustained decoder, and a clean 197.7 MB
24-range FCC transfer then completed without another gateway crash. The relay
still emits occasional libwebsockets Unhandled state 282 diagnostics after
completed range requests; those did not affect correctness or service
availability, but should remain a transport-hardening follow-up.
Browser compatibility gate¶
Chrome behavior above is measured, not inferred, and Chrome is the reference Phase 9B browser. No Firefox executable or remote Firefox debug endpoint was available, so Firefox import time, storage quota, OPFS cleanup, and memory behavior remain unmeasured and are not a release blocker.
The official SQLite WASM documentation reports OPFS support in Firefox and describes the SAH-pool VFS as broadly supported. If Firefox support is later requested, verify:
- OPFS worker access and generation deletion;
- SQLite SAH-pool behavior if approved;
- quota and persistent-storage behavior;
- import/hydration/query correctness;
- continued audio, waterfall, DSP, and decoder operation.
Reference: https://www.sqlite.org/wasm/doc/trunk/persistence.md
Approved permanent Phase 9B architecture¶
Following approval:
- Keep the verified source ZIP and provenance manifest in OPFS.
- Build a complete SQLite generation in a dedicated enrichment worker and a generation-scoped SAH-pool directory.
- Retain every FCC file/type/field, exact logical raw record, relationships, counts, and schema/source provenance; preserve multiline comments and the original ZIP.
- Validate manifest counts, foreign/key consistency appropriate to ULS,
required indexes, FTS population, representative dossiers, and
integrity_checkbefore activation. - Atomically switch a tiny active-generation pointer only after validation.
- Close the prior generation worker, then remove its entire pool directory.
- Hydrate callsign-to-USI, current-summary, previous-callsign, and common relationship data into RAM at startup. Never query durable storage on the decode hot path.
- Implement Auto, Eager, and Indexed as RAM/cache policies over the same SQLite truth; Auto selects Eager on the measured 32 GiB client.
- Coordinate tabs with one enrichment owner per generation using Web Locks
and
BroadcastChannel, because one SAH pool must not be concurrently owned by multiple workers. - Use bounded batches and cooperative yields. Record the observed startup-only reset burst for later receiver-aware QoS work without making zero added source gaps a Phase 9B acceptance requirement.
- Keep RF Sentry observations and learned station state in IndexedDB, apart from FCC regulatory truth.
- Request durable browser storage when supported, show quota/persistence state, and preserve the previous valid generation on every failure.
The benchmark gate itself made no permanent storage-format change. The production importer was implemented only after the operator approved this report and remains tracked separately in the Phase 9B acceptance ledger.