{"row":{"id":"ADR-038","title":"Admit nika-bm25 as the first L1 memory satellite (CRAFT-from-scratch BM25 scorer)","file":"adr-038-nika-bm25-admission.md","status":"accepted","date":"2026-05-12","cites":10,"supersedes":[],"superseded_by":[],"layers":["L1"],"tags":["memory","L1","bm25","ir"],"room":true},"pin":{"release_tag":"v0.107.0","commit":"5d61b7e13f1f6a10c17414f61146648f76bcf2bd"}}

[{"k":"h","depth":1,"text":"ADR-038: Admit nika-bm25 as the first L1 memory satellite","id":"adr-038-admit-nika-bm25-as-the-first-l1-memory-satellite"},{"k":"h","depth":2,"text":"Status","id":"status"},{"k":"p","inline":[{"k":"strong","text":"Proposed"},{"k":"text","text":" 2026-05-12 · pre-admission gate-readiness pass per Diamond Phase 1 entry plan §2 W2.4. Status flips to "},{"k":"strong","text":"Accepted"},{"k":"text","text":" when the W3 12-gate ceremony lands the crate in "},{"k":"code","text":"crates/nika-bm25/"},{"k":"text","text":" and registers it in the workspace "},{"k":"code","text":"[workspace] members"},{"k":"text","text":" array."}]},{"k":"h","depth":2,"text":"Context","id":"context"},{"k":"p","inline":[{"k":"text","text":"The Diamond memory subsystem architecture (locked separately in private studio doctrine and surfaced publicly via "},{"k":"code","text":"crates/nika-kernel-ai/src/memory.rs:1-413"},{"k":"text","text":") ships as 1 L2 orchestrator ("},{"k":"code","text":"nika-memory"},{"k":"text","text":") + 8 L1 satellites — "},{"k":"code","text":"nika-hnsw"},{"k":"text","text":" · "},{"k":"code","text":"nika-bm25"},{"k":"text","text":" · "},{"k":"code","text":"nika-rrf"},{"k":"text","text":" · "},{"k":"code","text":"nika-fsrs"},{"k":"text","text":" · "},{"k":"code","text":"nika-graph-algos"},{"k":"text","text":" · "},{"k":"code","text":"nika-rdfs-reasoner"},{"k":"text","text":" · "},{"k":"code","text":"nika-temporal"},{"k":"text","text":" · "},{"k":"code","text":"nika-autodesc"},{"k":"text","text":" ★. Each satellite is independently admissible via the 12-gate ceremony (ADR-003) and publishable standalone on crates.io as a SuperNovae contribution to the Rust RDF/ML ecosystem."}]},{"k":"p","inline":[{"k":"text","text":"Per Diamond Phase 1 entry plan §3 (W3 wave) "},{"k":"code","text":"nika-bm25"},{"k":"text","text":" is the "},{"k":"strong","text":"simplest L1 satellite to admit first"},{"k":"text","text":" because :"}]},{"k":"list","ordered":false,"items":[[{"k":"text","text":"BM25 has "},{"k":"strong","text":"zero deps beyond a hand-rolled scorer"},{"k":"text","text":" (no Oxigraph integration yet · no embedding-model dep)"}],[{"k":"text","text":"The algorithm is a battle-tested "},{"k":"strong","text":"IR standard for 15 years"},{"k":"text","text":" (Robertson · Walker · SIGIR 1994 · plus the variant catalog in Manning · Raghavan · Schütze "},{"k":"em","text":"Introduction to Information Retrieval"},{"k":"text","text":" 2008 ch.11)"}],[{"k":"text","text":"It is testable in isolation against toy corpora · no "},{"k":"code","text":"nika-kernel"},{"k":"text","text":" plumbing needed yet"}],[{"k":"text","text":"It proves the satellite-admission flow before tackling the harder satellites ("},{"k":"code","text":"nika-hnsw"},{"k":"text","text":" needs an empirical embedding distribution · "},{"k":"code","text":"nika-autodesc"},{"k":"text","text":" is the moat differentiator · both should land later)"}]]},{"k":"p","inline":[{"k":"text","text":"Per Phase 1 entry plan §2 the kernel-side memory trait surface ("},{"k":"code","text":"MemoryRemember"},{"k":"text","text":" / "},{"k":"code","text":"MemoryRecall"},{"k":"text","text":" / "},{"k":"code","text":"MemoryForget"},{"k":"text","text":" / "},{"k":"code","text":"MemoryStore"},{"k":"text","text":" blanket / "},{"k":"code","text":"MemoryLifecycle"},{"k":"text","text":" standalone) is already shipped at "},{"k":"code","text":"crates/nika-kernel-ai/src/memory.rs"},{"k":"text","text":" (607 LOC · "},{"k":"code","text":"trait_variant::make"},{"k":"text","text":" pattern per FCI-001 Decision A1). The kernel-side "},{"k":"code","text":"MemoryError"},{"k":"text","text":" enum maps to NIKA_601..604 per Diamond W2.1 + W2.2. "},{"k":"code","text":"nika-bm25"},{"k":"text","text":" will :"}]},{"k":"list","ordered":false,"items":[[{"k":"text","text":"Implement "},{"k":"code","text":"MemoryRecall"},{"k":"text","text":" (lexical-scoring branch · alongside semantic HNSW)"}],[{"k":"text","text":"Reserve its own NIKA-620..629 sub-range within the Memory 600-649 category (per Phase 1 entry plan §2 architect Q2 sub-allocation lock)"}]]},{"k":"h","depth":2,"text":"Decision","id":"decision"},{"k":"p","inline":[{"k":"text","text":"Admit "},{"k":"code","text":"nika-bm25"},{"k":"text","text":" to the Diamond workspace at Wave 3 as the first L1 memory satellite. Implementation = "},{"k":"strong","text":"CRAFT from scratch"},{"k":"text","text":" (ADR-001) hand-rolled · NOT a wrapper around "},{"k":"code","text":"tantivy"},{"k":"text","text":" (which would violate ADR-001 + introduce ~500K LOC of search-engine infrastructure we do not need at L1)."}]},{"k":"p","inline":[{"k":"text","text":"Target shape ·"}]},{"k":"code","lang":"text","text":"crates/nika-bm25/\n├── Cargo.toml          name=nika-bm25 · version=0.0.1 · license=AGPL-3.0-or-later\n├── src/\n│   ├── lib.rs          public API surface · MemoryRecall impl\n│   ├── scorer.rs       Bm25 struct · k1/b/avgdl params · score() formula\n│   ├── index.rs        inverted-index builder · tokenize + postings + IDF\n│   └── query.rs        query parsing + multi-term scoring\n└── tests/\n    └── manning_fixtures.rs  1:1 fidelity against published BM25 textbook examples"},{"k":"p","inline":[{"k":"text","text":"Default parameter values · "},{"k":"code","text":"k1=1.2"},{"k":"text","text":" (term saturation per Robertson) · "},{"k":"code","text":"b=0.75"},{"k":"text","text":" (length normalization · industry default). Configurable via "},{"k":"code","text":"Bm25::with_params()"},{"k":"text","text":" builder."}]},{"k":"p","inline":[{"k":"text","text":"Reserve "},{"k":"strong","text":"NIKA-620..629"},{"k":"text","text":" sub-range for "},{"k":"code","text":"nika-bm25"},{"k":"text","text":" error codes (Phase 1 entry plan §2 architect Q2 allocation · within the Memory 600-649 category locked Diamond W2.1)."}]},{"k":"h","depth":2,"text":"Consequences","id":"consequences"},{"k":"h","depth":3,"text":"Positive","id":"positive"},{"k":"list","ordered":false,"items":[[{"k":"text","text":"Proves the L1 satellite admission flow (lower-bound complexity reference for the 7 remaining satellites)"}],[{"k":"text","text":"Adds a real-world "},{"k":"code","text":"MemoryRecall"},{"k":"text","text":" implementer the kernel mock can be tested against (parity with the abstract trait)"}],[{"k":"text","text":"Lexical scoring complements future semantic HNSW (the orchestrator fuses both via RRF · nika-rrf admits at W4)"}],[{"k":"text","text":"Publishable on crates.io as "},{"k":"code","text":"nika-bm25"},{"k":"text","text":" (sovereign IR primitive · AGPL · zero vendor lock · per Rule 3 vendor-neutral default)"}],[{"k":"text","text":"~600 LOC well under the 15k crate cap and 1500-LOC file cap (ADR-023 + nika-invariants §\"Max LOC per file\")"}]]},{"k":"h","depth":3,"text":"Negative","id":"negative"},{"k":"list","ordered":false,"items":[[{"k":"text","text":"Adds 1 admitted crate to the workspace count (was 7 admitted · target 40-42 · acceptable)"}],[{"k":"text","text":"Hand-rolled BM25 means we own the math forever (mitigation · IR textbook fixtures freeze the parity surface · "},{"k":"code","text":"cargo mutants"},{"k":"text","text":" keeps the impl honest)"}],[{"k":"text","text":"Lexical-only · cannot do semantic similarity until "},{"k":"code","text":"nika-hnsw"},{"k":"text","text":" ships (intentional · v0.1 scope keeps the surface small)"}]]},{"k":"h","depth":3,"text":"Neutral","id":"neutral"},{"k":"list","ordered":false,"items":[[{"k":"text","text":"NIKA-620..629 sub-range becomes RESERVED in the Memory category (no concrete codes yet · landed when nika-bm25 errors crystallize during impl)"}],[{"k":"text","text":"The crate ships zero "},{"k":"code","text":"nika-kernel"},{"k":"text","text":" plumbing at v0.1 (just the algorithm) · plumbing waves later when the orchestrator consumes it"}]]},{"k":"h","depth":2,"text":"12-Gate admission readiness (Wave 3 evidence anchors)","id":"12-gate-admission-readiness-wave-3-evidence-anchors"},{"k":"p","inline":[{"k":"text","text":"Per ADR-003 the admission ceremony runs all 12 gates. Pre-flight status before W3 ·"}]},{"k":"list","ordered":false,"items":[[{"k":"text","text":"Gate · Check · Pre-flight verdict · Where it lands"}],[{"k":"text","text":"**1** · SPEC · `docs/crate-specs/nika-bm25.md` exists · ⏸ pending W3 · New file at W3 ; must cite this ADR + the Diamond Phase 1 entry plan §3"}],[{"k":"text","text":"**2** · LOC cap · ≤15k · ✅ trivially (target ~600 LOC) · `wc -l crates/nika-bm25/src/*.rs`"}],[{"k":"text","text":"**3** · Single-file cap · ≤1500 LOC · ✅ trivially (scorer ~200 · index ~250 · query ~150) · Per ADR-023 file modularity discipline"}],[{"k":"text","text":"**4** · Function cap · ≤100 LOC · ✅ design splits per-term scoring + corpus stats · Per nika-invariants §\"Max LOC per fn\""}],[{"k":"text","text":"**5** · Zero `.unwrap()` in `src/` · ✅ enforced via `workspace.lints.clippy unwrap_used = \"deny\"` · Per dx/.claude/rules/security.md §\"Propagate errors\""}],[{"k":"text","text":"**6** · Layer registry (L0/0.5/1/2/3/4/5) · ✅ L1 satellite layer · `[workspace.metadata.diamond.layers]` row added in W3 admission commit"}],[{"k":"text","text":"**7** · Forward-compat invariants (8 patterns) · ✅ FCI-001 `trait_variant::make` (impls `MemoryRecall`) · FCI-002 `#[non_exhaustive]` on public error enum · FCI-003 versioned BM25 index format · Per `docs/architecture/forward-compat-invariants.md`"}],[{"k":"text","text":"**8** · Test coverage (TDD · RED→GREEN) · ⏸ pending W3 · Insta snapshots + proptest fuzzing + 1:1 textbook fidelity tests"}],[{"k":"text","text":"**9** · Clippy `-D warnings` · ✅ enforced via workspace lints · `cargo clippy --workspace --all-targets -- -D warnings`"}],[{"k":"text","text":"**10** · `cargo fmt --check` · ✅ enforced via lefthook pre-commit · Pattern 1 atomic stage+commit"}],[{"k":"text","text":"**11** · Documentation (`#![warn(missing_docs)]`) · ⏸ pending W3 · Crate-level `//!` doc + per-item `///` doc"}],[{"k":"text","text":"**12** · License + AGPL header on every file · ✅ enforced via hygiene vector · `SPDX-License-Identifier: AGPL-3.0-or-later`"}]]},{"k":"p","inline":[{"k":"text","text":"Plus the mandatory pattern gates per "},{"k":"code","text":"nika/engine/.claude/CLAUDE.md"},{"k":"text","text":" ·"}]},{"k":"list","ordered":false,"items":[[{"k":"strong","text":"MUTATION ≥90%"},{"k":"text","text":" via "},{"k":"code","text":"cargo mutants -p nika-bm25"}],[{"k":"strong","text":"PROPERTY"},{"k":"text","text":" testing via "},{"k":"code","text":"proptest"},{"k":"text","text":" (TF-IDF monotonicity · score positivity · empty-corpus invariants)"}],[{"k":"strong","text":"CANARY E2E"},{"k":"text","text":" ↔ a "},{"k":"code","text":"tests/canary-bm25.nika.yaml"},{"k":"text","text":" workflow (or documented exemption if the scorer is too pure for an end-to-end canary)"}],[{"k":"strong","text":"PARITY LEGACY"},{"k":"text","text":" ↔ N/A (brouillon branch v0.79 has no BM25 implementation · CRAFT-from-scratch · no legacy to mirror)"}],[{"k":"strong","text":"REVIEW SWARM"},{"k":"text","text":" (3 agents · spn-nika:code-reviewer + spn-rust:rust-pro + feature-dev:code-reviewer)"}]]},{"k":"h","depth":2,"text":"Consumer signal","id":"consumer-signal"},{"k":"p","inline":[{"k":"text","text":"Per "},{"k":"code","text":"dx/.claude/rules/steal-pattern.md"},{"k":"text","text":" §\"Step 0 — Consumer-signal gate\" (Wave 10 of the DS federation discipline · adopted as a general anti-speculative-port pattern), Tier B/C steal-pattern ports require ≥1 consumer signal before status flips Proposed → Accepted. ADR-038 is an "},{"k":"strong","text":"internal Diamond admission"},{"k":"text","text":", not a Tier B/C steal-pattern port, but the same anti-speculative discipline applies ·"}]},{"k":"list","ordered":false,"items":[[{"k":"strong","text":"Locked decision"},{"k":"text","text":" · the Diamond memory architecture (private studio doctrine + public trait surface at "},{"k":"code","text":"crates/nika-kernel-ai/src/memory.rs"},{"k":"text","text":") lists "},{"k":"code","text":"nika-bm25"},{"k":"text","text":" as one of the 8 mandatory L1 memory satellites · ✅ satisfies « locked decision » in §Step 0"}],[{"k":"strong","text":"Scheduled launch"},{"k":"text","text":" · Phase 1 entry plan §3 schedules "},{"k":"code","text":"nika-bm25"},{"k":"text","text":" as W3 admission with ~7h focused effort · ✅ satisfies « scheduled launch »"}],[{"k":"strong","text":"Consumer ticket"},{"k":"text","text":" · "},{"k":"code","text":"nika-memory"},{"k":"text","text":" L2 orchestrator (deferred to W11+) requires both lexical and semantic recall paths · BM25 IS the lexical half · the orchestrator is the consumer-of-record once it ships"}]]},{"k":"h","depth":2,"text":"Evidence / Affected code","id":"evidence--affected-code"},{"k":"list","ordered":false,"items":[[{"k":"code","text":"crates/nika-kernel-ai/src/memory.rs:218-223"},{"k":"text","text":" — "},{"k":"code","text":"MemoryRecall"},{"k":"text","text":" trait definition that "},{"k":"code","text":"nika-bm25"},{"k":"text","text":" will implement"}],[{"k":"code","text":"crates/nika-error/src/codes/registry.rs:140-186"},{"k":"text","text":" — NIKA_600 placeholder + NIKA_601..604 active + NIKA-620..629 reserved for this crate"}],[{"k":"text","text":"W3 admission plan · private SuperNovae DX surface (not engine-bound)"}],[{"k":"text","text":"Commit anchor pre-W3 · "},{"k":"code","text":"git log --oneline -1"},{"k":"text","text":" on "},{"k":"code","text":"main"},{"k":"text","text":" at admission time will land in the W3 closeout commit body"}]]},{"k":"h","depth":2,"text":"Alternatives considered","id":"alternatives-considered"},{"k":"h","depth":3,"text":"Alt A — Wrap `tantivy` (or `meilisearch-sdk`)","id":"alt-a--wrap-tantivy-or-meilisearch-sdk"},{"k":"p","inline":[{"k":"text","text":"Skip the hand-roll · adopt a published Rust search engine. Rejected because :"}]},{"k":"list","ordered":false,"items":[[{"k":"text","text":"Violates ADR-001 (CRAFT not extraction) · we own the math"}],[{"k":"code","text":"tantivy"},{"k":"text","text":" is ~500K LOC across the workspace · we want a ~600 LOC primitive"}],[{"k":"text","text":"The orchestrator consumer needs "},{"k":"strong","text":"just BM25 scoring"},{"k":"text","text":" (not full-text-search infra · query parsers · faceting · etc.)"}],[{"k":"text","text":"Wrapper-of-wrapper guard (steal-pattern.md §\"Wrapper-of-wrapper guard\") · always port the leaf"}]]},{"k":"h","depth":3,"text":"Alt B — Defer `nika-bm25` until `nika-hnsw` admits","id":"alt-b--defer-nika-bm25-until-nika-hnsw-admits"},{"k":"p","inline":[{"k":"text","text":"Implement HNSW first then BM25. Rejected because :"}]},{"k":"list","ordered":false,"items":[[{"k":"text","text":"HNSW requires an empirical embedding distribution (100k+ real BGE-M3 vectors) we do not have yet"}],[{"k":"text","text":"BM25 has zero dependencies on the embedding model · can land independently"}],[{"k":"text","text":"W5 of the Phase 1 entry plan explicitly DEFERS "},{"k":"code","text":"nika-hnsw"},{"k":"text","text":" to a trigger-gated wave (« when ≥100k real vectors exist »)"}]]},{"k":"h","depth":3,"text":"Alt C — Admit a placeholder `nika-bm25-stub` first then real impl later","id":"alt-c--admit-a-placeholder-nika-bm25-stub-first-then-real-impl-later"},{"k":"p","inline":[{"k":"text","text":"Stub the trait surface · land real algorithm later. Rejected because :"}]},{"k":"list","ordered":false,"items":[[{"k":"text","text":"Stubs are an anti-pattern per ADR-037 §Context (« Stubs that pretend to implement reserved traits accumulate residue »)"}],[{"k":"text","text":"The 12-gate ceremony explicitly forbids stub admission (gate 5 "},{"k":"code","text":"cargo mutants ≥90%"},{"k":"text","text":" cannot pass on a "},{"k":"code","text":"todo!()"},{"k":"text","text":" impl)"}],[{"k":"text","text":"CRAFT-from-scratch in 7h focused beats stub-then-rewrite in calendar time"}]]},{"k":"h","depth":2,"text":"Related","id":"related"},{"k":"list","ordered":false,"items":[[{"k":"strong","text":"ADR-001"},{"k":"text","text":" — Orphan branch · CRAFT not extraction (rejects Alt A wrapper approach)"}],[{"k":"strong","text":"ADR-003"},{"k":"text","text":" — 12-gate admission ceremony (governs the W3 admission close)"}],[{"k":"strong","text":"ADR-016"},{"k":"text","text":" — Cancel-safety discipline (lib functions must be cancel-safe per "},{"k":"code","text":"MemoryRecall::recall()"},{"k":"text","text":" contract)"}],[{"k":"strong","text":"ADR-023"},{"k":"text","text":" — File modularity discipline (≤1500 LOC/file enforced)"}],[{"k":"strong","text":"ADR-037"},{"k":"text","text":" — Bottom-up Diamond progression (no feature-milestone defer · admit when ready)"}],[{"k":"code","text":"docs/architecture/crate-layer-registry.md"},{"k":"text","text":" — L1 row for "},{"k":"code","text":"nika-bm25"},{"k":"text","text":" lands in Diamond W2.5"}],[{"k":"code","text":"docs/architecture/forward-compat-invariants.md"},{"k":"text","text":" — FCI-001/002/003 patterns this crate respects"}],[{"k":"code","text":"dx/.claude/rules/no-legacy-no-back-compat.md"},{"k":"text","text":" — NUKE-LEGACY · BM25 ships as canonical lexical scorer · zero alias / shim / dual-format support"}],[{"k":"code","text":"dx/.claude/rules/naming-memory-subsystem.md"},{"k":"text","text":" — Diamond ⊥ Memory orthogonality · "},{"k":"code","text":"nika-bm25"},{"k":"text","text":" is a L1 crate organized SELON Diamond methodology"}],[{"k":"text","text":"Robertson & Walker 1994 · « Some simple effective approximations to the 2-Poisson model for probabilistic weighted retrieval »"}],[{"k":"text","text":"Manning · Raghavan · Schütze 2008 · "},{"k":"em","text":"Introduction to Information Retrieval"},{"k":"text","text":" ch.11 (fixture source for parity tests)"}]]},{"k":"h","depth":2,"text":"Notes","id":"notes"},{"k":"p","inline":[{"k":"strong","text":"Promotion gate"},{"k":"text","text":" · ADR-038 stays "},{"k":"code","text":"proposed"},{"k":"text","text":" until ·"}]},{"k":"list","ordered":true,"items":[[{"k":"code","text":"crates/nika-bm25/"},{"k":"text","text":" scaffolded with the 4-file shape"}],[{"k":"text","text":"12 gates ✅ per the readiness table above (with W3-pending gates filled in)"}],[{"k":"text","text":"Workspace "},{"k":"code","text":"Cargo.toml"},{"k":"text","text":" "},{"k":"code","text":"[workspace] members"},{"k":"text","text":" array includes "},{"k":"code","text":"crates/nika-bm25"}],[{"k":"code","text":"[workspace.metadata.diamond.layers] nika-bm25 = \"L1\""},{"k":"text","text":" added (Diamond W2.5)"}],[{"k":"code","text":"docs/architecture/crate-layer-registry.md"},{"k":"text","text":" L1 section has the "},{"k":"code","text":"nika-bm25"},{"k":"text","text":" row (Diamond W2.5)"}],[{"k":"text","text":"Single atomic admission commit · message format "},{"k":"code","text":"feat(nika-bm25): admit to workspace — all 12 gates passed"},{"k":"text","text":" (per nika/engine commit-granularity.md)"}]]},{"k":"p","inline":[{"k":"text","text":"On promotion · status flips to "},{"k":"code","text":"accepted"},{"k":"text","text":" · "},{"k":"code","text":"enables: []"},{"k":"text","text":" will list nika-rrf and nika-hnsw admission ADRs once they land."}]},{"k":"p","inline":[{"k":"strong","text":"Out-of-scope for ADR-038"},{"k":"text","text":" (deferred to future ADRs) ·"}]},{"k":"list","ordered":false,"items":[[{"k":"text","text":"BM25F (per-field weighting) — nice-to-have at v0.2 if a consumer asks"}],[{"k":"text","text":"BM25+ (lower-bound saturation) — same as BM25F · deferred"}],[{"k":"text","text":"Custom tokenizers (CJK · IDF caches · etc.) — orchestrator-driven · post v0.95 Cortex"}],[{"k":"text","text":"Persistent index serialization format — FCI-003 versioning bake-in at W3 · concrete schema in a follow-up ADR"}]]}]

ADR-038

Admit nika-bm25 as the first L1 memory satellite (CRAFT-from-scratch BM25 scorer)

accepted · 2026-05-12 · L1 · cites 10

ADR-038: Admit nika-bm25 as the first L1 memory satellite #

Status #

Proposed 2026-05-12 · pre-admission gate-readiness pass per Diamond Phase 1 entry plan §2 W2.4. Status flips to Accepted when the W3 12-gate ceremony lands the crate in crates/nika-bm25/ and registers it in the workspace [workspace] members array.

Context #

The Diamond memory subsystem architecture (locked separately in private studio doctrine and surfaced publicly via crates/nika-kernel-ai/src/memory.rs:1-413) ships as 1 L2 orchestrator (nika-memory) + 8 L1 satellites — nika-hnsw · nika-bm25 · nika-rrf · nika-fsrs · nika-graph-algos · nika-rdfs-reasoner · nika-temporal · nika-autodesc ★. Each satellite is independently admissible via the 12-gate ceremony (ADR-003) and publishable standalone on crates.io as a SuperNovae contribution to the Rust RDF/ML ecosystem.

Per Diamond Phase 1 entry plan §3 (W3 wave) nika-bm25 is the simplest L1 satellite to admit first because :

BM25 has zero deps beyond a hand-rolled scorer (no Oxigraph integration yet · no embedding-model dep)
The algorithm is a battle-tested IR standard for 15 years (Robertson · Walker · SIGIR 1994 · plus the variant catalog in Manning · Raghavan · Schütze Introduction to Information Retrieval 2008 ch.11)
It is testable in isolation against toy corpora · no nika-kernel plumbing needed yet
It proves the satellite-admission flow before tackling the harder satellites (nika-hnsw needs an empirical embedding distribution · nika-autodesc is the moat differentiator · both should land later)

Per Phase 1 entry plan §2 the kernel-side memory trait surface (MemoryRemember / MemoryRecall / MemoryForget / MemoryStore blanket / MemoryLifecycle standalone) is already shipped at crates/nika-kernel-ai/src/memory.rs (607 LOC · trait_variant::make pattern per FCI-001 Decision A1). The kernel-side MemoryError enum maps to NIKA_601..604 per Diamond W2.1 + W2.2. nika-bm25 will :

Implement MemoryRecall (lexical-scoring branch · alongside semantic HNSW)
Reserve its own NIKA-620..629 sub-range within the Memory 600-649 category (per Phase 1 entry plan §2 architect Q2 sub-allocation lock)

Decision #

Admit nika-bm25 to the Diamond workspace at Wave 3 as the first L1 memory satellite. Implementation = CRAFT from scratch (ADR-001) hand-rolled · NOT a wrapper around tantivy (which would violate ADR-001 + introduce ~500K LOC of search-engine infrastructure we do not need at L1).

Target shape ·

text

1crates/nika-bm25/2├── Cargo.toml          name=nika-bm25 · version=0.0.1 · license=AGPL-3.0-or-later3├── src/4│   ├── lib.rs          public API surface · MemoryRecall impl5│   ├── scorer.rs       Bm25 struct · k1/b/avgdl params · score() formula6│   ├── index.rs        inverted-index builder · tokenize + postings + IDF7│   └── query.rs        query parsing + multi-term scoring8└── tests/9    └── manning_fixtures.rs  1:1 fidelity against published BM25 textbook examples

Default parameter values · k1=1.2 (term saturation per Robertson) · b=0.75 (length normalization · industry default). Configurable via Bm25::with_params() builder.

Reserve NIKA-620..629 sub-range for nika-bm25 error codes (Phase 1 entry plan §2 architect Q2 allocation · within the Memory 600-649 category locked Diamond W2.1).

Consequences #

Positive #

Proves the L1 satellite admission flow (lower-bound complexity reference for the 7 remaining satellites)
Adds a real-world MemoryRecall implementer the kernel mock can be tested against (parity with the abstract trait)
Lexical scoring complements future semantic HNSW (the orchestrator fuses both via RRF · nika-rrf admits at W4)
Publishable on crates.io as nika-bm25 (sovereign IR primitive · AGPL · zero vendor lock · per Rule 3 vendor-neutral default)
~600 LOC well under the 15k crate cap and 1500-LOC file cap (ADR-023 + nika-invariants §"Max LOC per file")

Negative #

Adds 1 admitted crate to the workspace count (was 7 admitted · target 40-42 · acceptable)
Hand-rolled BM25 means we own the math forever (mitigation · IR textbook fixtures freeze the parity surface · cargo mutants keeps the impl honest)
Lexical-only · cannot do semantic similarity until nika-hnsw ships (intentional · v0.1 scope keeps the surface small)

Neutral #

NIKA-620..629 sub-range becomes RESERVED in the Memory category (no concrete codes yet · landed when nika-bm25 errors crystallize during impl)
The crate ships zero nika-kernel plumbing at v0.1 (just the algorithm) · plumbing waves later when the orchestrator consumes it

12-Gate admission readiness (Wave 3 evidence anchors) #

Per ADR-003 the admission ceremony runs all 12 gates. Pre-flight status before W3 ·

Gate · Check · Pre-flight verdict · Where it lands
**1** · SPEC · `docs/crate-specs/nika-bm25.md` exists · ⏸ pending W3 · New file at W3 ; must cite this ADR + the Diamond Phase 1 entry plan §3
**2** · LOC cap · ≤15k · ✅ trivially (target ~600 LOC) · `wc -l crates/nika-bm25/src/*.rs`
**3** · Single-file cap · ≤1500 LOC · ✅ trivially (scorer ~200 · index ~250 · query ~150) · Per ADR-023 file modularity discipline
**4** · Function cap · ≤100 LOC · ✅ design splits per-term scoring + corpus stats · Per nika-invariants §"Max LOC per fn"
**5** · Zero `.unwrap()` in `src/` · ✅ enforced via `workspace.lints.clippy unwrap_used = "deny"` · Per dx/.claude/rules/security.md §"Propagate errors"
**6** · Layer registry (L0/0.5/1/2/3/4/5) · ✅ L1 satellite layer · `[workspace.metadata.diamond.layers]` row added in W3 admission commit
**7** · Forward-compat invariants (8 patterns) · ✅ FCI-001 `trait_variant::make` (impls `MemoryRecall`) · FCI-002 `#[non_exhaustive]` on public error enum · FCI-003 versioned BM25 index format · Per `docs/architecture/forward-compat-invariants.md`
**8** · Test coverage (TDD · RED→GREEN) · ⏸ pending W3 · Insta snapshots + proptest fuzzing + 1:1 textbook fidelity tests
**9** · Clippy `-D warnings` · ✅ enforced via workspace lints · `cargo clippy --workspace --all-targets -- -D warnings`
**10** · `cargo fmt --check` · ✅ enforced via lefthook pre-commit · Pattern 1 atomic stage+commit
**11** · Documentation (`#![warn(missing_docs)]`) · ⏸ pending W3 · Crate-level `//!` doc + per-item `///` doc
**12** · License + AGPL header on every file · ✅ enforced via hygiene vector · `SPDX-License-Identifier: AGPL-3.0-or-later`

Plus the mandatory pattern gates per nika/engine/.claude/CLAUDE.md ·

MUTATION ≥90% via cargo mutants -p nika-bm25
PROPERTY testing via proptest (TF-IDF monotonicity · score positivity · empty-corpus invariants)
CANARY E2E ↔ a tests/canary-bm25.nika.yaml workflow (or documented exemption if the scorer is too pure for an end-to-end canary)
PARITY LEGACY ↔ N/A (brouillon branch v0.79 has no BM25 implementation · CRAFT-from-scratch · no legacy to mirror)
REVIEW SWARM (3 agents · spn-nika:code-reviewer + spn-rust:rust-pro + feature-dev:code-reviewer)

Consumer signal #

Per dx/.claude/rules/steal-pattern.md §"Step 0 — Consumer-signal gate" (Wave 10 of the DS federation discipline · adopted as a general anti-speculative-port pattern), Tier B/C steal-pattern ports require ≥1 consumer signal before status flips Proposed → Accepted. ADR-038 is an internal Diamond admission, not a Tier B/C steal-pattern port, but the same anti-speculative discipline applies ·

Locked decision · the Diamond memory architecture (private studio doctrine + public trait surface at crates/nika-kernel-ai/src/memory.rs) lists nika-bm25 as one of the 8 mandatory L1 memory satellites · ✅ satisfies « locked decision » in §Step 0
Scheduled launch · Phase 1 entry plan §3 schedules nika-bm25 as W3 admission with ~7h focused effort · ✅ satisfies « scheduled launch »
Consumer ticket · nika-memory L2 orchestrator (deferred to W11+) requires both lexical and semantic recall paths · BM25 IS the lexical half · the orchestrator is the consumer-of-record once it ships

Evidence / Affected code #

crates/nika-kernel-ai/src/memory.rs:218-223 — MemoryRecall trait definition that nika-bm25 will implement
crates/nika-error/src/codes/registry.rs:140-186 — NIKA_600 placeholder + NIKA_601..604 active + NIKA-620..629 reserved for this crate
W3 admission plan · private SuperNovae DX surface (not engine-bound)
Commit anchor pre-W3 · git log --oneline -1 on main at admission time will land in the W3 closeout commit body

Alternatives considered #

Alt A — Wrap `tantivy` (or `meilisearch-sdk`) #

Skip the hand-roll · adopt a published Rust search engine. Rejected because :

Violates ADR-001 (CRAFT not extraction) · we own the math
tantivy is ~500K LOC across the workspace · we want a ~600 LOC primitive
The orchestrator consumer needs just BM25 scoring (not full-text-search infra · query parsers · faceting · etc.)
Wrapper-of-wrapper guard (steal-pattern.md §"Wrapper-of-wrapper guard") · always port the leaf

Alt B — Defer `nika-bm25` until `nika-hnsw` admits #

Implement HNSW first then BM25. Rejected because :

HNSW requires an empirical embedding distribution (100k+ real BGE-M3 vectors) we do not have yet
BM25 has zero dependencies on the embedding model · can land independently
W5 of the Phase 1 entry plan explicitly DEFERS nika-hnsw to a trigger-gated wave (« when ≥100k real vectors exist »)

Alt C — Admit a placeholder `nika-bm25-stub` first then real impl later #

Stub the trait surface · land real algorithm later. Rejected because :

Stubs are an anti-pattern per ADR-037 §Context (« Stubs that pretend to implement reserved traits accumulate residue »)
The 12-gate ceremony explicitly forbids stub admission (gate 5 cargo mutants ≥90% cannot pass on a todo!() impl)
CRAFT-from-scratch in 7h focused beats stub-then-rewrite in calendar time

ADR-001 — Orphan branch · CRAFT not extraction (rejects Alt A wrapper approach)
ADR-003 — 12-gate admission ceremony (governs the W3 admission close)
ADR-016 — Cancel-safety discipline (lib functions must be cancel-safe per MemoryRecall::recall() contract)
ADR-023 — File modularity discipline (≤1500 LOC/file enforced)
ADR-037 — Bottom-up Diamond progression (no feature-milestone defer · admit when ready)
docs/architecture/crate-layer-registry.md — L1 row for nika-bm25 lands in Diamond W2.5
docs/architecture/forward-compat-invariants.md — FCI-001/002/003 patterns this crate respects
dx/.claude/rules/no-legacy-no-back-compat.md — NUKE-LEGACY · BM25 ships as canonical lexical scorer · zero alias / shim / dual-format support
dx/.claude/rules/naming-memory-subsystem.md — Diamond ⊥ Memory orthogonality · nika-bm25 is a L1 crate organized SELON Diamond methodology
Robertson & Walker 1994 · « Some simple effective approximations to the 2-Poisson model for probabilistic weighted retrieval »
Manning · Raghavan · Schütze 2008 · Introduction to Information Retrieval ch.11 (fixture source for parity tests)

Notes #

Promotion gate · ADR-038 stays proposed until ·

crates/nika-bm25/ scaffolded with the 4-file shape
12 gates ✅ per the readiness table above (with W3-pending gates filled in)
Workspace Cargo.toml [workspace] members array includes crates/nika-bm25
[workspace.metadata.diamond.layers] nika-bm25 = "L1" added (Diamond W2.5)
docs/architecture/crate-layer-registry.md L1 section has the nika-bm25 row (Diamond W2.5)
Single atomic admission commit · message format feat(nika-bm25): admit to workspace — all 12 gates passed (per nika/engine commit-granularity.md)

On promotion · status flips to accepted · enables: [] will list nika-rrf and nika-hnsw admission ADRs once they land.

Out-of-scope for ADR-038 (deferred to future ADRs) ·

BM25F (per-field weighting) — nice-to-have at v0.2 if a consumer asks
BM25+ (lower-bound saturation) — same as BM25F · deferred
Custom tokenizers (CJK · IDF caches · etc.) — orchestrator-driven · post v0.95 Cortex
Persistent index serialization format — FCI-003 versioning bake-in at W3 · concrete schema in a follow-up ADR

read at v0.107.0 · the decision record ships with the engine

Every decision, with its status This one in the repo