Benchmarks
What each benchmark tests, what the number means, and the number itself. No runtime LLM in the decision path.
Primary five
Tests: financial-compliance entity extraction — corporate subsidiary hierarchy reconstruction from SEC Exhibit 21 filings.
Means: F1 on 400-entity corpus with SHA-verified ground truth. Checkpoint arc E → E.2 documented. Corpus SHA-256 (E.2): a6a98dbb…dab99d81.
Tests: security operations center alert triage — classify and prioritize SOC alerts with HEIMDALL confidence gate.
Means: held-out accuracy on the HEIMDALL classifier. No LLM at runtime. Honest refusal counted in the methodology.
Tests: cross-domain factual verification against a sealed knowledge base — verified, refuted, or refused.
Means: 51% overall vs 31% / 25% always-confident baselines. Refusals on 67.5% of the 200-claim corpus counted correctly. Corpus SHA-256: cd5de198…1911ad2.
Tests: deterministic semantic search over codebases — intent-based retrieval without embedding models at runtime.
Means: 25× improvement vs grep on intent queries (P@5 0.250 deterministic on published task).
Tests: document reading-level calibration — does output land within one tier of target?
Means: 15/15 within-one-tier on the sealed calibration set.
Memory · RAVEN v1.1
Tests: memory validation — contradiction detection, importance ranking, honest refusal in memory pipelines.
Means: validation F1 0.847, recall 0.921 on published memory-validation corpus.
Tests: per-class memory decay — facts, preferences, and identity claims age on different clocks.
Means: 100% decay-aware recall across 310 queries.
Tests: structured memory refusal — five typed reasons with recommended actions and audit hashes.
Means: 100% precision on typed refusal taxonomy.
Posture
No runtime LLM in the decision path. Honest refusals are counted, not hidden. Internal receipts for now. Public corpora aren’t published yet.