Everything that was measured

01How to read this page

• Pre-registration. Criteria are written down before the run. A rung measured on the queries that motivated it is a hypothesis, not a result, and is labelled exploratory.
• Paired tests. Every A-versus-B claim is paired on the same queries, with a bootstrap confidence interval and a McNemar test on the discordant pairs. Wins and losses are reported separately, because a net zero from 0 changes and a net zero from 40 wins and 40 losses are different facts.
• Nothing is reopened. Once a query set is frozen and a verdict recorded, it stands. A new question needs a new query set.
• pp means percentage points. CI95 is a 95% bootstrap interval.

02W1 · Four-facet fusion lost to one facet

default withdrawn 479 queries · one real 1,700-bookmark library · pre-registered

The premise of the whole project was that fusing four facets beats any one of them. Three criteria were registered before the run. All three failed.

Fusion cost 5.4pp of Recall@5 and made queries 3.5× slower. Config A by query type: content-style 0.959, vague 0.706, episodic 0.279.

Why it lost

Flat-weight reciprocal rank fusion has no way to express confidence. Two weak facets that happen to agree score 0.0279; one strong facet that is certain scores 0.0164. The coincidence wins. That is not a tuning problem, it is what the formula does.

What survived the same run

Both shipped. Note that graph expansion only works as an addition — returned as its own group rather than merged into the ranking.

03W2/W3 · The episodic gate shipped, then lost

default reverted after shipping

The episodic gate detects “the thing I saved around the same time as X” and restricts retrieval to that saving window. On its 616-query holdout it won cleanly and was shipped.

The second row is the same feature, measured on a query set built afterwards to ask a different question: what does the gate do when it fires on a query it should not have? Recall@5 fell from 0.9058 to 0.7175 and Recall@1 from 0.801 to 0.363.

The stratification is the whole answer

When the gate is right it adds nothing. When it is wrong it throws the answer away. Verdict gate_precision_unqualified; the default reverted to no gating.

04The other side of the gate · no verdict

descriptive only · below the pre-registered sample floor

The precision probe asked what happens when the gate fires and should not have. This asks the opposite: how often does it fail to fire when it should? The protocol was pre-registered before the run, mirroring the precision protocol.

All sixteen phrasings are ways of saying “the one I put away” — 之前收起来的那个, the link I set aside, 我塞进清单里的那篇. None of them contains a word in the gate's trigger vocabulary, which currently keys on 保存, 收藏, saved, bookmark and eleven others.

The obvious move — add these sixteen phrasings to the vocabulary — is exactly what the protocol forbids, because selecting a vocabulary on the probes that measure it is circular. Reaching a verdict requires generating at least 25 probes in a new round with new seeds, at frozen parameters, and then passing both the miss-rate bar and the 361-probe precision bar. Until then the vocabulary is unchanged.

05Five candidate fixes, five verdicts

After W1 killed fusion, five obvious repairs were each measured rather than argued about.

The third row is the interesting one. A real bug was found and fixed — the trigram facet went from useless on Chinese to working — and end-to-end recall did not move. A fix that is genuinely a fix and changes no outcome is a normal result, and reporting it is the only thing that keeps the other four honest.

06karakeep round trip · unfaithful

roundtrip_unfaithful 2,376 bookmarks · 616 holdout queries · protocol frozen first

Question: if a library is pushed through the karakeep bridge and read back, is it the same library? Three criteria were registered first.

The cause is fully attributed

• Bodies survive byte-identical: 1,876 of 1,876.
• Summaries survive: 2,375 of 2,375, 100%.
• Topics match 0% and entities 1.18% — because karakeep's tags are the browser's folder labels, not topics.
• The keyword line collapses from 19,016 distinct terms to 13; mean terms per page falls from 10.32 to 0.76; the most common tag is 未分类 on 1,124 pages.
• Vectors move by a median cosine of 0.9846 — small, and enough to reshuffle a top-5.

Grafting the source enrichment back produced 2,376 of 2,376 byte-identical embed texts, residual zero, which closes the attribution. Re-running facetmark index repairs it: 0 karakeep bodies needed re-fetching, all 2,376 rows re-enriched, and the graph came back matching except for 212 semantic edges (26,485 against 26,697).

07The decay layer, measured twice — the second run overturned the first

The decay layer demotes pages that look stale. Round one measured it and found exactly nothing:

Round two ran the same bytes with a local health check first.

ΔRecall@5 went from +0.0000pp in round one to −1.4610pp CI95 [−2.5974, −0.4870] in round two. The mechanism is countable: of 37 rank changes, 12 fell out of the top 20 entirely — 10 of those had been in the top 5 and 5 had been rank 1. Twenty-four rose, 21 of them by a single place, and exactly 1 crossed into the top 5. Net −10 + 1 = −9, and −9/616 = −1.4610pp.

Why the threshold still has not changed

Two bugs are cancelling, and the cancellation is load-bearing.

• Bug one: the cold-layer condition treats “the URL died” as “the saved copy is useless”. But facetmark stores the body. A dead URL is precisely when the local snapshot matters most, and drifted is worse still, because then the snapshot is the only surviving record.
• Bug two: with rrf_k = 60, one unit-weight facet tops out at 1/61 = 0.016393, which is below the rescue threshold of 0.02. In the shipped single-facet profile the rescue valve is therefore always open and the demotion has never once executed.
• Remove either one alone and results get measurably worse. Both are pinned by tests/test_decay_reach.py so neither can be quietly “cleaned up”.

What changed instead was the instrumentation. cold_census() now reports the three conditions separately, and facetmark stats and facetmark health --check name never_opened_selects_everything and health_never_checked out loud.

One more detail worth keeping: 4 of the 8 damaged targets have char_count = 0 and are still retrieved correctly, through title and lexical facets. Body loss is not the same as retrieval loss.

08One real library, end to end

Synthetic corpora hide integration failures. This is one actual browser export, imported and indexed with the shipped code path.

The latency number is honest and unflattering: it is a cold, unfetched index on a laptop, and it is the number that would be quietly omitted from a launch post.

09What none of this measures

• Whether anyone else's queries look like these. Every query set was written by the author of the tool. This is the single largest threat to every number on this page and no amount of bootstrapping touches it.
• Whether the decay layer helps, because in the shipped profile it cannot fire at all.
• Whether the intent facet would help a different library. It was measured on this one, generated by one model, and it lost.
• Whether the karakeep bridge works against a live karakeep. The contract is pinned and replayed; a running instance has never been tested.
• Whether the reranker helps with a real cross-encoder. What ships offline is term overlap. An ablation run under that reranker measures the harness, not the idea, and must not be quoted as evidence that reranking works.
• Long-term behaviour. Every measurement is a snapshot. Nobody has run this for a year and watched what a growing library does to the session clustering.