01 RAG · Introduction and retrieval baseline
Making retrieval observable first
検索拡張生成 / 检索增强生成
The Water Margin knowledge-exploration experiments began with lexical retrieval: turn the question and the source text into terms, then compute which text blocks are more relevant.
RAG (Retrieval-Augmented Generation) retrieves material first, then uses it to help generate an answer. GraphRAG (Graph-based Retrieval-Augmented Generation) uses graph structure to support that process; how exactly the graph is used still varies between implementations.
Water Margin (Shuihu zhuan) is a classic Chinese novel about outlaws. The experiments use its Chinese text, so the examples below stay in Chinese, each with a translation.
The first MVP used character bigram / trigram pieces and BM25, and did not put vector retrieval or a graph database on the main path. The value of that choice is explainability: which terms matched, what score each block received and where it was filtered out can all be recorded and compared.
This is not a “BM25 beats vector retrieval” selection study. The project never ran a full A/B comparison of the two. Whether semantic retrieval is needed depends on what failures the lexical approach leaves behind, and on the real gain after introducing it.
- TermsHow text is matchedN-gram / Analyzer
- RankingWho comes firstBM25 / RRF
- SelectionWhat fits inEvidence budget
- AnswerIs the evidence enoughCheck source text and quotes
Early on the project already had extracted people and relations, followed by an answer-centred local graph. But “having graph data”, “showing a graph” and “using a graph to decide which evidence to retrieve” are three different things. The graph-retrieval topic returns to this line.
BasisInitial BM25 MVP · experiment dataAnswer-centred local graph · experiment dataRetrieval strategy experiment · experiment data
PART 01 / FOUNDATIONS
Foundations
From term matching to rank fusion: what each of four common steps is for.
02 BM25 · Relevance scoring
BM25 | Turning keyword matches into a relevance ranking
関連度スコアリング / 相关性评分
Search for “林冲 宝刀” (Lin Chong, treasured sword) and many texts may contain those words. BM25 uses how often a word appears, how rare it is and how long the text is to decide which results should rank first.
Search term: 宝刀 (“treasured sword”)
林冲买了一口宝刀。
Lin Chong bought a treasured sword.
林冲看见宝刀，又仔细端详这口宝刀。
Lin Chong saw the treasured sword, then studied it closely.
TF | Term frequency
Term Frequency
The more often a word appears, the stronger the match signal usually is.
BM25’s term-frequency contribution saturates: it does not grow linearly without limit as a word repeats.
Bar length is the share of texts containing the word. It is not a relevance score.
IDF | Inverse document frequency
Inverse Document Frequency
The rarer a word, the better it usually separates truly relevant results.
“林冲” (Lin Chong) appears in many texts, so its statistical discriminating power may be weaker than a rare word’s. Rarity is not the same as semantic importance.
Short text“宝刀” appears once
林冲把宝刀拿在手里，翻来覆去地看。
Lin Chong held the treasured sword in his hands and turned it over and over.
Long text“宝刀” appears once
林冲清早出门，经过街边的店铺，和熟人说了几句话。路上人来人往，有人挑着担子，有人停下歇脚。走到街角时，他看见一口宝刀，拿起来仔细查看。随后他又沿街走了一段，听见附近酒家传来谈笑声，便转身向家中走去。
Lin Chong left home early, passed the shops along the street and exchanged a few words with acquaintances. People came and went; some carried loads, some stopped to rest. At the corner he saw a treasured sword and picked it up to examine it. He then walked on a little, heard laughter from a nearby tavern, and turned back toward home.
Length normalization
Length Normalization
Keeps a long text from winning only because it contains more.
BM25 corrects the score using document length and the corpus average length. Here the hit count is fixed, so only how concentrated the term is in a short or long text is compared.
A second-ranked piece of evidence finally gets selected
Before · fixed threshold 5
3.39 < 5The relevant evidence ranks second and is still excluded.
After · absolute threshold removed
Still ranks 2ndNo longer blocked by the score gate; it continues to be filtered by capacity and budget.
Of the 31 answerable questions, those with at least one core evidence block in the context
Removing the threshold gives evidence a chance to be selected, but does not guarantee it reaches the context: the number of candidates, capacity and budget still limit selection.
Learn more: BM25 scoring and the historical threshold experimentCheck the S0 / S1 data, the evaluation conditions, and the difference between scoring and evidence selection.
| Historical strategy | Hit@3 | Context has ≥1 core block |
|---|---|---|
| S0 · threshold 5 | 0.65 | 18/31 ≈ 0.58 |
| S1 · no threshold | 0.65 | 20/31 ≈ 0.65 |
Chapters 7–11, 59 blocks, 31 answerable questions; budget 2,000, at most 4 blocks, skipping a block that does not fit and continuing. Hit@3 checks whether the top 3 contain at least one core block; context coverage checks the blocks finally selected. The two are different.
The 3.39, rank 2 and threshold 5 come from a historical diagnosis. The legacy channel was based on BM25 and also added entity and adjacent-block bonuses, so 3.39 is strictly a historical retrieval total score. The sample is small and labelled by one person; do not generalize it to all queries.
BasisRetrieval strategy experiment · experiment dataRetrieval implementation notes · experiment dataElastic · BM25 similarity ↗
03 N-gram · Consecutive character pieces
N-gram | Cutting text into consecutive pieces
文字 N-gram / 字符 N 元片段
Chinese has no natural spaces to mark word boundaries. A character N-gram does not need to decide what the words are first; it slides a window over the text and produces fixed-length character pieces.
“Lin Chong buys a treasured sword”
This is a simplified character-cutting sketch, not the project’s raw retrieval output. All terms of the three cuts are always readable without any interaction.
N is the number of consecutive characters. A standard character N-gram usually slides one character at a time, so neighbouring terms overlap.
N-gram overlap happens when terms are generated; chunk overlap happens when text is split into chunks.
- Strength: no dictionary needed; it can still match consecutive characters inside unknown words.
- Limitation: it can produce pieces with no real meaning, and it does not naturally understand synonyms.
Project practice · Python bigram / trigram piecesRetrieval implementation notes · experiment dataOpenSearch · N-gram tokenizer ↗
04 Analyzer · Text analysis
Analyzer · Text analysis
トークナイザー / アナライザー / 文本分析器
An Analyzer turns raw text into searchable terms.
Character Filter
Character preprocessing · clean punctuation out of the raw text
林冲→！！买了一把宝刀！！！林冲买了一把宝刀Tokenizer
Term splitting · break continuous text into searchable terms
林冲买了一把宝刀→[林冲] [买] [了] [一把] [宝刀]Token Filter
Term filtering · remove some terms by rule
[林冲] [买]→[了][一把][宝刀][林冲] [买] [宝刀]
The example reads: “Lin Chong bought a treasured sword.” The terms are Lin Chong (a name), buy, a completed-action particle, “one (measure)” and treasured sword. The splitting and filtering rules are invented for this example: it removes the particle and “one (measure)”. Not every Analyzer would do that.
The Character Filter works on raw characters, the Tokenizer produces terms, and the Token Filter screens the terms further.
03 · N-gram: other ways to cut characters.
Next steps: later research on term filtering and query expansion.
The project implements part of the text analysis and filtering in custom Python code.
BasisRetrieval implementation notes · experiment dataRetrieval strategy experiment · experiment dataOpenSearch · Text analysis ↗
05 RRF · Rank fusion
RRF · Rank fusion
逆順位融合 / 倒数排名融合
The same set of content can be ranked differently by different retrieval methods. RRF computes a final order from the ranks each leaderboard gives.
Article A is 1st on one leaderboard and 2nd on the other, and becomes 1st after fusion.
Teaching sketch · invented A / B / C, equal-weight RRF, k = 60Leaderboard 1
- Rank 1Article A
- Rank 2Article B
For example, bigram / trigram character BM25.
Leaderboard 2
- Rank 1Article C
- Rank 2Article A
For example, single-character BM25.
Final leaderboard
- Rank 1Article A0.03252
- Rank 2Article C0.01639
- Rank 3Article B0.01613
Why is A first?
Leaderboard 1: rank 1.
Leaderboard 2: rank 2.
1/61 + 1/62 ≈ 0.03252The two rank contributions are added together, so A gets the highest fused score.
RRF fuses ranks; it does not add the raw scores of different retrievers.
Historical measurement · two-lexical RRF · Hit@3
The share of answerable questions where the top 3 contain at least one core evidence block went up.
This is the overall result of adding a single-character channel and fusing, so not all of the gain can be credited to RRF itself. A fused ranking also does not mean the answer is correct.
Learn more: RRF parameters and the full evaluation dataSee k = 60, how candidates missing from a leaderboard are treated, and the three historical strategies side by side.
Each leaderboard contributes 1 / (k + rank). The figure fuses with equal weights and k = 60. An article absent from one leaderboard’s candidates contributes 0 there; it is not treated as ranked last. Scores are rounded to five decimals.
Chapters 7–11, 59 blocks, 31 answerable questions. The legacy channel combines bigram / trigram BM25 with entity and adjacent-block bonuses; single-character BM25 reaches Hit@3 of 0.71, and the two-lexical equal-weight RRF (k = 60) reaches 0.77. The contributions of dropping the threshold, channel granularity and fusion need to be separated.
BasisRetrieval strategy experiment · experiment dataBudget comparison experiment · experiment dataMulti-chapter knowledge retrieval design · experiment dataCormack et al. · SIGIR 2009 ↗
PART 02 / FIELD NOTES
Field notes from the project
The previous part covered text analysis, relevance scoring and rank fusion. Now we look at how these techniques make up a real retrieval system, and how we followed the retrieval chain to locate problems and adjust the approach.
S The retrieval pipeline
From the foundations to the real retrieval flow
検索パイプライン / 检索链路
The earlier techniques are not separate facts; they are different steps on one chain behind an answer.
- User question“鲁智深和林冲怎么认识？”
How did Lu Zhishen and Lin Chong come to know each other? - Text analysis and term generationSee 03 N-gram04 Analyzer
The question and the source blocks are both turned into two sets of terms: bigram / trigram character pieces, and single characters with high-frequency function characters removed.
- Candidate retrieval and relevance scoringSee 02 BM25
Channel 1Bigram / trigram terms → BM25 ranking
Channel 2Single-character terms → BM25 ranking
- Multi-channel result fusionSee 05 RRF
Merge the two rankings by rank alone into one candidate order.
- Evidence selection and Context assemblyPractice A Retrieval and evidence
In fused order, whole source blocks are loaded into the Context, within a Token budget and a block limit.
Context boundary: the model sees only the evidence that was loaded.
Model generates the answerIt answers only from the Context it actually received; afterwards the system checks the quotes against the source text.
The figure simplifies the current mainline and is not the full call chain. The 2,000 / 4,000 / 8,000 in Practice A are experiment configurations from different times (legacy strategy → RRF); the current default is RRF · 8,000 estimated Tokens · at most 12 blocks.
Learn more: what the diagram leaves outHow the mainline differs from historical configurations, the knowledge-exploration branch, the event skeleton, and how relation queries relate to RRF.
Mainline and historical configurations. The mainline runs two BM25 rankings over source blocks, fuses them with RRF (k = 60, equal weights), then loads whole blocks in rank order; when a block does not fit, selection stops rather than skipping it to fill with lower-ranked blocks. The legacy strategy used threshold 5, at most 4 blocks and 2,000 estimated Tokens, and kept filling past blocks that did not fit. The 0.65 → 0.77 in Part 01 is a historical experiment from the legacy channel to two-lexical RRF; it is not the isolated gain of any single step in this figure.
The knowledge-exploration branch. The project also has a side branch for knowledge exploration: pre-organized character profiles, event / relation / state records and chapter summaries serve as “knowledge”, source blocks are a second source, and each has its own budget. Knowledge candidates are also ranked by two lexical rankings fused with RRF; when the question names two or more distinctive people, a further ranking by character participation is added. That ranks existing knowledge records and is not the same experiment as the “relation-edge ranking channel” in Practice B.
Event skeleton and relation queries. For questions about a person’s journey, the event skeleton is determined by the knowledge base before the model is called, then given to the model as context so it can write a note for each stage. Queries such as “what relations or shared events do these two people have” read records by ID: they do not rank, and they are not an RRF channel.
Quote checking. After the answer, quotes are checked for whether they can be found verbatim in the source they cite. That shows a quote can be located; it does not mean the conclusion is semantically supported.
The figure also omits the session’s evidence memory, follow-up handling and query-word cleaning.
Once these steps are combined, an answer can still miss something even when retrieval has found relevant content. So at which step does the problem occur?See Practice A ↓
Architecture basisCurrent Q&A mainline · experiment dataQ&A orchestration · experiment dataKnowledge-exploration retrieval · experiment dataKnowledge-exploration pipeline · experiment data
A Retrieval and evidence selection
Retrieval Results and Evidence Selection
根拠の選択 / トークン予算 / 证据选择 / Token 预算
Retrieval seems to have found relevant content, yet the answer still lacks key information. Where on the chain does the problem lie?
01An incomplete answer: first find which step went wrong
The actual answer left out important information. Rather than tuning parameters right away, work backwards along the chain above: first check whether the Context the model actually received contains it; then go back to the source text to confirm the necessary facts; finally compare the candidate retrieval (the ranking after BM25 scoring and RRF fusion) and the evidence-packing records, to tell whether it was never retrieved or was retrieved but not loaded.
- Compare lastCandidate retrieval and evidence packing
- Check firstContext
- Problem noticedModel answer
02The key evidence was retrieved, but did not enter the Context
The offline record for “how did Lu Zhishen and Lin Chong come to know each other” shows that the sworn-brotherhood passage was already among the candidates: 8th in the legacy channel, and 4th after switching to RRF. But under a 2,000-Token evidence budget, the packing rules still kept it out, and the Context did not contain that passage. For this question, a better rank was not enough; it also had to have a chance to be loaded.
The model sees only the selected blocksEven if 4th place is useful, it was not sent into this tier’s context.
4th place is an offline RRF ranking, not the legacy strategy’s 4th. The figure explains the mechanism as “stop at the first block over budget”; block widths and the cut points of other blocks are illustrative. This is not the full trace of any real model call below.
Supporting information · early real configuration · chapters 7–11 evaluation set
Legacy strategy · 2,000 Token
- Average source length
- 639 characters / chunk
- Usually only room for
- about 2–3 source blocks
Configuration sourceRetrieval strategy experiment · experiment data
03Change the retrieval strategy and budget, then observe the real answer
The real question: “鲁智深和林冲怎么认识？” (How did Lu Zhishen and Lin Chong come to know each other?) We changed the retrieval strategy from the legacy channel to RRF fusion and widened the evidence budget at the same time, giving more necessary evidence a chance to enter the Context, then checked whether the answer could tell the first meeting, the sworn brotherhood and the Wild Boar Forest rescue apart.
Retrieval strategy and budget→Evidence coverage of the Context→Actual answer
Legacy strategy · 2,000
Token · evidence budget (conservative estimate)
Context: the sworn-brotherhood passage is missing.
Not enough evidence; could not answer.
Returned insufficient.
RRF · 4,000
Token · evidence budget (conservative estimate)
Context: has the sworn-brotherhood evidence, and also pulled in highly ranked Wild Boar Forest passages.
More evidence, but events still confused.
Mixed the sworn brotherhood in chapter 7 with the recognition in Wild Boar Forest in chapter 9; only 1 of 3 quotes could be located.
RRF · 8,000
Token · evidence budget (conservative estimate)
Context: budget widened, so more evidence had a chance to enter.
This answer was more complete, with no renewed confusion, and the quotes could be located.
The report records: meeting on the street, getting to know Lin Chong’s father, becoming sworn brothers; both quotes could be located.
Each group differs in retrieval strategy, budget and selection rules at once, so the whole effect cannot be credited to a bigger budget. Each configuration has a single real model call, n = 1, which is not statistical causal proof. We did not verify, block by block, how this question’s Context differed between 4,000 and 8,000, so we do not infer which block was added; a quote that can be located is not the same as verified semantic support.
More evidence does not guarantee a correct answer.
A bigger budget improves the chance of coverage, but may also bring noise; it does not make the model automatically better at understanding evidence.
The Context still lacks necessary facts: keep checking retrieval, candidate ranking and evidence packing. If the wording simply does not match (for example, asking “revenge” while the source text says “kill” or “stab”), see Next steps.
The evidence is sufficient but the answer is still confused: check evidence understanding, event association, prompt design and answer generation, instead of only adding budget.
Learn more: candidate packing and the budget experiment recordSee the full coverage and noise statistics, the candidate limit, and the different packing rules.
A larger budget may also bring in noise. Over the whole evaluation set, going from the 4,000 to the 8,000 tier, RRF’s count of questions where all core blocks were selected rose from 21/31 to 27/31, and the share of selected blocks not marked core or relevant rose from 61% to 79%.
Context Selection / Context Packing (assembling the context) picks and organizes the context. This article uses evidence coverage and budget to describe that job; it is not treated as a separate basic scoring algorithm.
In the per-question offline record, the core block ch007_chunk_004 for the “how did they come to know each other” question moved from 8th in the legacy channel to 4th under RRF; it was not selected at the 2,000 tier and was selected at the 4,000 tier. A separate real-answer record shows the confusion between the Wild Boar Forest scene and the sworn brotherhood; the two kinds of record cannot be stitched into one full trace.
| Budget estimated Tokens | All core questions / 31 | Average blocks selected | Irrelevant blocks share |
|---|---|---|---|
| 2,000 | 12 / 31 | 2.3 | 45% |
| 4,000 | 21 / 31 | 5.0 | 61% |
| 8,000 | 27 / 31 | 10.6 | 79% |
| 16,000 | 30 / 31 | 21.6 | 89% |
“All core” checks whether every manually labelled core block was selected; irrelevant blocks are counted as those outside core ∪ rel, an upper bound on noise, and an unlabelled block is not necessarily harmful. The table covers the whole evaluation set, not just the “how did they come to know each other” question.
From the 4,000 to the 8,000 tier, 6 more core blocks and 176 more irrelevant blocks were added. The corpus has only 59 blocks, so the result cannot be taken directly as a recommended budget for a larger corpus.
Top-K limits candidates; the block limit and the Token budget limit what is selected. With the limit kept at 4 blocks, raising the budget cannot improve things further. The legacy strategy skips a block that does not fit and keeps “filling gaps”; stopping at the first overflow can suppress some noise, but it is not the best rule for every task.
BasisBudget comparison experiment · experiment dataPer-question rankings and failure attribution · experiment dataRetrieval strategy experiment · experiment data
B The winding road of graph retrieval
The winding road of graph retrieval
グラフ検索 / 图检索
After adding graph relations, the retrieval hit rate actually went down.
The earlier retrieval chain finds evidence mainly through term matching, relevance scoring and rank fusion. So can the people, events and relations we have already extracted add a new retrieval ability to that chain?
01Letting relation edges join the ranking brought no improvement
The project already had knowledge about people, events and relations. The first attempt made relation edges a third retrieval signal, fused by RRF with equal weights alongside the original two lexical rankings.
Hit@3: 0.77 → 0.68
0.77: the original two-lexical RRF.
0.68: with the relation-edge ranking channel added.
Frequently mentioned people are tied to many relation edges, and relation descriptions do not necessarily use the question’s wording; with equal-weight fusion, that noise may be pushed to the front and disturb the existing ranking.
This is the result for particular relation texts, matching rules and an equal-weight fusion setup. It cannot be read as graph retrieval being ineffective in itself; the noise explanation above comes from the report’s analysis and is not a general causal conclusion.
02Did the graph relations fail, or the way we used them?
This failure shows that adding a relation-ranking channel directly did not meet expectations, but it does not prove structured relations are useless. The research did not stop there; it began looking for another way to use them.
Instead of letting every relation edge compete in the ranking, could we use them to find related people and events, and then trace the corresponding source evidence?
03Letting relations assist evidence exploration
The system has since already given structured relations a different job: helping find related people and events and then returning to the matching source evidence, rather than adding one more leaderboard.
Relation edges→ranking channel→RRF fusion→compete with lexical results for rank
People / events / relation records→look up links→source text→evidence display and event paths
- Person-relation lookupRead the relation records between two people.
- Shared-event lookupFind the events both people took part in.
- Source-evidence locationRelation and event records carry their sources, so they lead back to the matching source passage.
- Journey paths and event organizationFor questions about a person’s journey, the knowledge base first divides events into ordered stages, then gives them to the model to write a note for each stage.
Implementation checkImplementation notes · experiment data
Lin Chong and Lu Zhishen
ch007_chunk_004Original
“林沖大喜，就當結義智深為兄。”Translation (for reference)
“Lin Chong was overjoyed, and took Zhishen as his sworn elder brother.”
Source evidence for the two meeting and swearing brotherhood.
ch009_chunk_001Original
“林沖方纔閃開眼看時，認得是魯智深。”Translation (for reference)
“When Lin Chong finally opened his eyes and looked, he recognized it was Lu Zhishen.”
“Recognized” here cannot be taken as the first time they met.
The event titles are teaching summaries and the connectors express organization; they do not pretend to be an actual query path or the original schema. The quotes keep the original traditional characters; the translations are added for reading support only.
One failure does not mean the whole research direction has no value.
The first ranking strategy did not work, but the system has since used structured relations for evidence association and event organization. What job a relation should do depends on the problem and the experimental result; adding a channel does not guarantee improvement.
The lookup, source association and event organization described above are implemented; their stable contribution to answer accuracy and completeness still needs to be verified, and it does not mean all multi-hop questions are solved. See Next steps for how to verify.
Learn more: the relation-retrieval experiment and implementation limitsSee the early equal-weight fusion setup, the lookup and journey-path organization implemented later, and the limits of the evidence.
Retrieval strategy report S4 / S6, 31 answerable questions. A hub person’s name matching introduces noise, and “getting to know” does not necessarily match the relation type “sworn brotherhood” literally; this drop applies only to that experimental setup.
In the current service, kb_pair looks up relations and shared events between two people, and kb_record links events, relations, states and sources; journey paths build an event skeleton through plan_for_question. The person-pair lookup reads by ID, does not take part in ranking, and does not automatically enter the model Context. Being queryable and organizable is not the same as having proven better answer quality.
BasisRetrieval strategy experiment · experiment dataPer-question rankings and failure attribution · experiment dataMulti-chapter knowledge retrieval design · experiment dataPython knowledge-exploration engine · experiment dataAnswer-centred local graph · experiment dataPython / FalkorDB provider · experiment data
PART 03 / OPEN QUESTIONS
Next steps
The practice left two kinds of problem: when the user’s wording differs from the source text, evidence can still be missed; and the contribution of relations and structured knowledge to answer quality is not verified yet.
→ Next steps
Next: helping queries find evidence more accurately
クエリ拡張 / クエリ書き換え / 查询扩展 / 查询改写
Current lexical retrieval still struggles with some differences in wording. For example, a user says “revenge” (报仇), while the source text uses action words such as “kill” (杀) and “stab” (搠).
Analyzer / Token Filter · term handling and filtering
Adjust term handling, domain dictionaries and filter rules, and observe whether retrieval matching improves and noise is reduced.
Query Expansion / Rewrite
Keep the original question and add related search expressions, check whether evidence that was missed before can be found, and guard against query drift.
These directions are still hypotheses to be verified, and need comparison against the original strategy on the same evaluation set. The contribution of structured relations to answer accuracy and completeness likewise needs to be evaluated by comparison before any conclusion.
Basis of the problemRetrieval strategy experiment · experiment dataBudget comparison experiment · experiment dataMulti-chapter knowledge retrieval design · experiment data
Back to the engineering judgment
Locate which layer failed first, then choose the technique.
If the terms did not match, changing the budget will not help; if the evidence ranked high but was squeezed out, adding graph relations may not help either. A more reliable starting point is to make matching, ranking, selection and evidence support all checkable.
Learn more: evidence sources and limits of the articleSee how measurements, teaching sketches and unverified directions differ, and the limits on scale and independence.
The experiment figures come from committed historical reports. Those reports and the code are in a private repository, so this article does not link to them; the methods, numbers and limits are recorded in Experiment data and evidence at the end of this article. We did not rerun the LLM experiments or complete an independent re-evaluation.
The standalone Analyzer example, N-gram, the RRF numeric example, the budget cut and the relation-organization figure are teaching sketches. A negative experiment is not generalized into a ranking of techniques, and future research is not counted as an existing result.
The experiments are small (chapters 7–11, 59 source blocks, 31 answerable questions) and labelled by one person, so one or two questions can change a number; the results should not be generalized to other corpora or larger scales.
Experiment data and evidence
This section does one thing: it puts the experiment numbers, methods and limits that the article cites where anyone can read them. The full internal reports and the source code are not public, and are not copied here.
Evaluation setup and retrieval comparisonsThe evaluation set and metric definitions, then Hit@3 for the retrieval strategies, the removed threshold and the relation-edge channel.
Evaluation setup
| Corpus | Water Margin chapters 7–11: 59 source-text blocks (chunks), about 639 characters each on average. |
|---|---|
| Evaluation set | 33 questions: 31 with answers and 2 without. Core evidence blocks were labelled by one person; Hit@3 and core-coverage figures mainly concern the 31 answerable questions. |
| Metrics | Hit@3: whether the top 3 contain at least one core block. Context coverage: whether the blocks finally loaded into the context contain a core block. The two differ. |
| Budget basis | A conservative estimate of 1 character ≈ 1 Token, not measured model Tokens; capacity varies with block length and selection rules. |
| Real answers | One model call per configuration (n = 1); this is not a statistical experiment. A quote is “locatable” if it can be found verbatim in the block it cites. |
| Scale and independence | A small corpus and one annotator, so one or two questions can change a number. There is no independent re-evaluation and no variance across repeated calls. |
Retrieval strategies compared
Hit@3 of three retrieval configurations on the 31 answerable questions (lexical retrieval only, no vector retrieval):
| Configuration | Hit@3 | Note |
|---|---|---|
| Legacy channel | 0.65 | Bigram / trigram BM25, plus entity and adjacent-block bonuses. |
| Unigram BM25 | 0.71 | A different term granularity. |
| Two-lexical RRF | 0.77 | Ranks of the bigram / trigram and the single-character channels fused, k = 60, equal weights. |
0.65 → 0.77 is the overall result of adding a single-character channel and fusing. The contributions of dropping the threshold, channel granularity and fusion were not controlled separately, so the gain cannot all be credited to RRF.
Removing the score threshold
S0 and S1 use the same scoring and ranking; only whether the absolute threshold applies differs. Budget 2,000, at most 4 blocks, skipping a block that does not fit and continuing.
| Strategy | Hit@3 | Context has ≥1 core block |
|---|---|---|
| S0 · threshold 5 | 0.65 | 18/31 ≈ 0.58 |
| S1 · no threshold | 0.65 | 20/31 ≈ 0.65 |
A diagnostic case: the “buying the sword” evidence block ch007_chunk_012 had a historical retrieval total score of 3.39, ranked 2nd, and was excluded by threshold 5. The score is based on BM25 but also includes entity and adjacent-block bonuses, so strictly it is a “historical retrieval total score”, not a pure BM25 score.
Adding a relation-edge channel
A relation-edge ranking added to the two-lexical RRF with equal-weight fusion (S4 / S6 in the report), 31 answerable questions:
| Configuration | Hit@3 |
|---|---|
| Two-lexical RRF | 0.77 |
| With the relation-edge channel | 0.68 |
The report’s analysis of the drop: frequently mentioned people are tied to many relation edges; relation descriptions do not necessarily use the question’s wording; with equal-weight fusion that noise may be pushed to the front. This is the result for that setup (particular relation texts, matching rules and equal-weight fusion) and cannot be read as graph retrieval being ineffective in itself.
Evidence budget and real answersCore-chunk coverage and noise at each budget, and the answer records of three configurations on one question.
Evidence budgets compared
Two-lexical RRF, no block limit, stop at the first overflow (when a block does not fit, selection stops; it does not skip it to fill with lower-ranked blocks). Results over the whole evaluation set:
| Budget (estimated Tokens) | Questions with all core blocks selected | Average blocks selected | Irrelevant-block share |
|---|---|---|---|
| 2,000 | 12 / 31 questions | 2.3 | 45% |
| 4,000 | 21 / 31 questions | 5.0 | 61% |
| 8,000 | 27 / 31 questions | 10.6 | 79% |
| 16,000 | 30 / 31 questions | 21.6 | 89% |
“Irrelevant blocks” are those outside the manually labelled core ∪ rel; this is an upper bound on noise, and an unlabelled block is not necessarily harmful.
From the 4,000 to the 8,000 tier, 6 more core blocks and 176 more irrelevant blocks were added. The corpus has only 59 blocks, so the result cannot be taken directly as a recommended budget for a larger corpus.
The legacy strategy (threshold 5, at most 4 blocks, 2,000 estimated Tokens) skips a block that does not fit and keeps filling; with the limit at 4 blocks, raising the budget cannot improve things further.
Real answers and ranking records
The same real question, “How did Lu Zhishen and Lin Chong come to know each other?”, with one model call per configuration:
| Configuration | Context | Answer record | Quotes locatable |
|---|---|---|---|
| Legacy strategy · 2,000 | The sworn-brotherhood passage is missing. | Not enough evidence; returned insufficient. | — |
| RRF · 4,000 | Has the sworn-brotherhood evidence, and also pulled in highly ranked Wild Boar Forest passages. | Answered, but mixed the sworn brotherhood in chapter 7 with the recognition in Wild Boar Forest in chapter 9. | 1 / 3 |
| RRF · 8,000 | Budget widened, so more evidence had a chance to enter. | This answer was more complete and no longer showed the earlier confusion of events: meeting on the street, getting to know Lin Chong’s father, becoming sworn brothers. | 2 / 2 |
Rank of the core block ch007_chunk_004: 8th in the legacy channel, 4th under RRF; not selected at the RRF 2,000 tier, selected at the 4,000 tier. This is a per-question offline record and is not the full trace of the real calls above, so the two cannot be stitched into one causal chain.
The three configurations differ in retrieval strategy, budget and selection rules at once, so the effect cannot be credited to a bigger budget alone; we also did not verify, block by block, which block was added between 4,000 and 8,000 for this question.
“Quotes locatable” only means a passage can be found verbatim in the block it cites; it does not show the conclusion is semantically supported. Each configuration was called once, so variance across repeated calls was not measured.
Implementation notes, limits and sourcesNotes from checking the retrieval implementation, the limits of the experiments, and where the numbers come from.
Implementation notes
The system diagram in the article simplifies the implementation below, checked against the code as of October 2026 (a private repository, commit 0d26b5c).
| Part | Description |
|---|---|
| Current Q&A mainline | Retrieves source-text blocks only: bigram / trigram terms and single-character terms each get a BM25 ranking, fused with RRF (k = 60, equal weights); whole blocks are loaded in rank order until the block limit is reached or a block exceeds the budget; the default is 8,000 estimated Tokens, at most 12 blocks. One retrieval and one model call per question, then the quotes are checked. |
| Legacy strategy (historical) | A single BM25 channel (with entity and adjacent-block bonuses), threshold 5, at most 4 blocks, 2,000 estimated Tokens, skipping blocks that do not fit and continuing. |
| Knowledge-exploration branch | Pre-organized character profiles, event / relation / state records and chapter summaries serve as “knowledge”; source blocks are a second source; each has its own budget. Knowledge candidates also get two lexical rankings fused with RRF; when the question names two or more distinctive people, a further ranking by character participation is added. |
| Event skeleton | For questions about a person’s journey, the knowledge base determines the event skeleton before the model is called and gives it to the model as part of the context, so the model writes a note for each stage. |
| Relation queries | Queries for the relations and shared events between two people, and for records and their sources, read by record ID: they do not rank, are not an RRF channel, and do not automatically enter the model Context. |
| Multi-chapter retrieval design | The knowledge-exploration branch’s retrieval design comes from a multi-chapter offline experiment covering chapters 7–11; this article uses none of its numbers. |
| Evolution | Initial BM25 MVP (2061743) → answer-centred local graph (e32245a) → Python knowledge-exploration engine (0b36e0c) → Relation Provider (Python / FalkorDB, d687c29) → the mainline version checked here (0d26b5c). These are commit identifiers in a private repository, given only for identification, with no links. |
Being queryable and organizable is not the same as having shown a stable gain in answer accuracy or completeness; that has not been verified by experiment.
Limits and how far to trust this
- A small corpus (59 blocks), a small evaluation set (31 answerable questions) and one annotator for the core blocks: one or two questions can change a number.
- There is no full A/B comparison of vector retrieval and lexical retrieval, so the article does not claim that BM25 beats vector retrieval.
- Each real-answer configuration was called once (n = 1), with no variance across repeated calls; the answers were judged by the author against the labels and the source text, not scored by a third party.
- Budgets are conservative character-based estimates, not measured model Tokens.
- The experiments were not re-run and there is no independent re-evaluation.
- The failure of the relation-edge channel only concerns that setup; the contribution of the implemented relation queries and event organization to answer quality is still to be verified.
Where the numbers come from
These numbers come from experiment reports and code committed in the graphrag-shuihu project. That repository is currently private, so this site does not link to it; this section is the public record of the numbers and the methods.
This section excerpts only methods, results and limits; it does not copy the full internal reports or any source code. If it ever disagrees with an original report, the report wins and this section should be corrected.
For general explanations of BM25, N-gram, Analyzer and RRF, see these public documents:
BasisElastic · BM25 similarity ↗OpenSearch · Text analysis ↗OpenSearch · N-gram tokenizer ↗Cormack et al. · SIGIR 2009 · RRF ↗