Synthesis & hypotheses

Beyond storing facts, knomit actively maintains and grows the corpus. All of it runs through one engine, driven by either the MCP tools (knomit_review, knomit_hypothesize) or the HTTP synthesis-run endpoints. The work is work-stealing: it borrows cycles from the calling model, one work item at a time, rather than running a separate headless service.

The pipeline

A session presents one work item at a time; the model responds, the next item is served, until the phase completes. Sessions track three independent axes:

• status — lifecycle: active · completed · abandoned
• phase — workflow: work · reflect · done (advanced by atomic CAS transitions)
• effort — the discovery dial: normal · medium · high

Answering the right work item

Each work item carries an id. Echo it back as the optional item_id parameter on the next knomit_review / knomit_hypothesize call, and the server rejects the call if that item is no longer the current one. This matters because the queue can grow between rendering an item and answering it — applying a distill item enqueues RAPTOR follow-up items — so without the id a response reasoned about against one item’s facts can be applied to a different one. Omitting item_id preserves the previous behaviour of answering whatever item is current.

A response is decoded and validated before the item is claimed, and the claim itself is an atomic compare-and-swap, so a failed validation leaves the item unanswered and retryable, while a lost claim applies nothing twice.

Distill work items are chunked at roughly 256 KiB of marshalled fact JSON per item, so a large cluster arrives as several items rather than one oversized prompt. Prune clusters are not chunked.

A single item can still exceed what one tool result should carry, so an oversized item is additionally paged at roughly 24 KiB of delivered facts per page. A result carrying more_available: true is a partial view of the item: keep calling with session_id, item_id and the next page until it is false, then answer once with the completion_token from the final page. The token is the server’s proof you read every page, and a response without it is rejected — the item stays available, so you can page properly and resubmit. Items that fit in one page carry no token.

Both bounds are constants, not configuration knobs — there is no setting to tune either.

Note

Only epistemic facts seed the pipeline — pragmatic facts (policies, heuristics) never enter synthesis.

knomit_review — prune · distill · reflect

Stage	What it does
Prune (dedup)	Detects near-duplicate facts and merges them. Tiebreak: a non-hypothesis always wins; then higher confidence; then more sources. Domains and entities are unioned.
Distill (synthesis)	Clusters related facts and distills them into a higher-order synthesis fact. Evidence weight uses SumProductNorm = Σ(c·s) / (Σ(c·s)+1); hypothesis sources are excluded from the weight.
Reflect (methodology)	Reflects on hypothesis→outcome transitions to record reasoning lessons as methodology facts. Reinforcement appends the methodology’s path to the transition fact’s refs — git is the only source of truth, no side-channel counter. New-methodology proposals are hard-capped (KNOMIT_REFLECT_PROPOSE_CAP, default 1) and gated by a novelty/cosine floor (KNOMIT_REFLECT_NOVELTY_THRESHOLD).

knomit_review does not generate new hypotheses.

KNOMIT_REFLECT_NOVELTY_THRESHOLD defaults to the active embedding model’s calibrated ReflectNovelty value — 0.69 under the default EmbeddingGemma — not to a fixed number, because a cosine floor is only meaningful against the distribution the model produces. A literal 0.85 exists in the code as a fallback for a model whose calibrated value is missing, which no normal deployment reaches: an embedder is mandatory. See Embeddings.

The final result (done: true) carries a populated summary with the session’s running totals — Pruned, Merged, Updated, Synthesized. The counts are informational: if they cannot be read back, the summary degrades to zeros rather than failing the session. knomit_hypothesize reports no summary.

Tip

Run review at higher effort to surface emergent consequences — see Emergent discovery.

RAPTOR — multi-depth distillation

Distillation is recursive: synthesis facts can themselves be clustered and distilled again at greater depth, RAPTOR-style. This runs through the same work-item queue; the item’s priority orders the depth, so deeper summaries build on shallower ones without a separate scheduler.

Recursion is hard-capped at depth 3: a distill item at that depth enqueues no follow-ups, so a productive session always drains its queue. The cap is a constant in the code, not a setting — there is nothing to tune.

knomit_hypothesize — generate predictions

Walks synthesis facts on the agent branch and, per item, lets the model decide whether to write a falsifiable hypothesis fact (skipping is the expected outcome for most). It is a distinct, user-initiated operation — never an auto-follow-up to review. On a later dedup collision with a confirmed observation, the hypothesis is retracted and the observation links to it via refs.

Tip

Run hypothesize at higher effort to surface emergent keystones — unstated load-bearing premises ranked by blast radius. See Emergent discovery.

Methodology in the loop

When the model reasons, knomit can surface relevant past methodology. RelevantMethodology ranks methodology facts by a composite score (0.6·vector + 0.4·tag_overlap, floored at KNOMIT_METHODOLOGY_MIN_SCORE, default 0.15). Methodology facts are identified by type=methodology, never by path. The server never auto-cites — it injects candidate methodology into the prompt and the model decides.

LLM provider

Synthesis and distillation use an LLM, configured by [llm] (default provider gemini, model gemini-2.5-flash). Embeddings are a separate, local model — see Embeddings and Configuration.