What to think about when RAG gives an incomplete answer

01 · Baseline · Basics

A RAG kept deliberately simple

What follows investigates a failure, not tuning, so every link is as simple as it can be. After this section you only need to remember this line:

1. Official docs
2. Structure-aware chunks
3. embedding
4. cosine retrieval
5. Top-5 as evidence
6. Qwen generates
7. Citation check

Corpus

26 pages of official Cloudflare documentation (Pages, Workers, R2, D1, Wrangler, bindings, variables and secrets, limits and pricing), snapshot v1-2026-10-04

Chunking

336 chunks, of which 334 have body text that can be embedded (the other 2 are headings only); median length 827 characters

Retrieval

bge-m3 (1024 dimensions) + flat cosine; the Top-5 go to generation

Generation

qwen3.8-flash: may only use the evidence it is given, must cite chunks and quote the source word for word; the system checks that each citation really exists

Enough?

Whether the evidence is enough to answer is judged by a hand-written “evidence checklist”; if it is not enough, the system declines and the model is not called (the checklist was written for one case only and is not general)

How the chunks are cut, which models were used and what the environment was are in the “Reproduction notes” at the end; they are not expanded here.

02 · What working looks like · Basics

A successful case: Q5

Question怎么给 Pages Functions 配置 R2 bucket 绑定，并在代码里访问它？Translation: How do I configure an R2 bucket binding for Pages Functions, and access it in code?

First, what it looks like when it works, so that when the failures come you know the whole system is running.

The single chunk at rank 1 covers “how to configure an R2 binding for a Pages project”, “redeploy” and “access it with context.env” together, so the evidence status is sufficient (sufficient). Qwen cited the chunks at ranks 1, 2, 3, 4 in 4 places; the quoted text exists word for word in each, and the check passed.

Even the successful case has a gap: the chunk with the Wrangler configuration syntax for Pages sits at rank 8, outside the Top-5. We recorded the gap and did not patch it.

03 · When there is no evidence · Basics

Q9a: asking about something the corpus does not contain

QuestionWorkers KV 的免费额度是多少？超出后如何计费？Translation: How large is the Workers KV free allowance? How is usage beyond it billed?

Check the corpus first: 34 chunks (13 pages) in this snapshot mention KV, but all of them are about bindings or configuration; none gives the free allowance or the billing. This is not a retrieval failure: no chunk was “missed” by retrieval.

Retrieval still returned 5 chunks, scoring 0.574–0.593, about the limits and free allowances of Workers, Pages and R2. Q5’s rank 5 scored 0.608: the score alone cannot tell “has evidence” from “has none”.

• Normal path: the evidence status is insufficient, and the system declines without calling the model.
• In the experiment we bypassed that gate on purpose and let Qwen face the same 5 distractor chunks. Its reply was: “提供的证据中未包含 Workers KV 的免费额度及超出后的计费信息。Translation: The evidence provided does not include the Workers KV free allowance or how usage beyond it is billed.” No number, no other product’s allowance passed off as KV’s, and no citation.

“The evidence provided does not include it” must not be written as “the Cloudflare docs do not have it”.Not in the corpus ≠ retrieval missed it ≠ the vendor does not have it. The model was called once, which shows it did not overstep this time, not that it is stable; and a thing missing entirely is also the easiest kind to decline.

04 · The question that is really trouble · Going further

Q7: everything looks relevant, yet the answer is incomplete

Question环境变量、secret 和普通配置应该分别怎么设置？本地开发和线上部署有什么区别？Translation: How should environment variables, secrets and plain configuration each be set? What is the difference between local development and a production deployment?

A complete answer needs these four pieces:

Look at the Top-10 first. Reading the headings, you would think retrieval did well:

All of it is secrets, local development and environment variables; nothing is off topic. But it covers only 1/4 of the four pieces needed: just C4. C1, C2 and C3 are in the corpus, within the first 60, but ranked 52, 36 and 36.

semantic similarity ≠ evidence sufficiency.“Related to the question” and “enough to answer it” are two things. Retrieval does the former.

One open labelling point: chunks like “Compare secrets and environment variables” are marked irrelevant in the benchmark, yet to a human reader they help with C1/C2. Whether they count as evidence is a Gold question the author has yet to review; nothing was changed here, and no score was adjusted for it.

05 · Decide which layer, then experiment · Going further

Six hypotheses, and every one really ran

First decide which layer fails. Is the evidence missing from the corpus? No: C1–C3 are in the first 60. Does it fall just outside the Top-5? No either: they are at ranks 36–52. So it is “there, but ranked deep”. Each hypothesis below stands for a common mechanism, and every step is: hypothesis → experiment → result → judgement. Experiments that did not help are written here too, because a technique does not work just by being added: each mechanism answers a different failure.

Chinese-terms version: Cloudflare Workers 里的 environment variables、secrets 和普通配置（vars）应该分别怎么设置？本地开发（Wrangler）和线上部署（production）有什么区别？
(In English: In Cloudflare Workers, how should environment variables, secrets and plain configuration (vars) each be set? What is the difference between local development (Wrangler) and a production deployment (production)?)

Document-aligned English version: How do I configure plain text environment variables in the Wrangler configuration file, add encrypted secrets with wrangler secret put for production, and use .dev.vars or .env files for local development in Cloudflare Workers and Pages?

Side by side

Below is where C1, C2 and C3 first appear after each attempt. Green means in the Top-10.

Six approaches, and not one gets C1–C3 all into the Top-10.Each mechanism answers a different failure; this question’s failure is not at the layer any of them works on.

06 · Recovery Loop · Putting it together

When search cannot find it, help the user carry on

By now Q7 could stop at “not found, so we do not know”. There is another way: the system admits the evidence is not enough and helps the user turn the question into one that is easier to answer.

1. Original questionevidence incomplete
2. Qwen proposes 3 narrower questions
3. The user picks onetreated as a new ordinary query
4. Retrieve again
5. A grounded answer+ citation check

“Recovery Loop” is a name I gave this approach, not a general term. The model sees only the original question and the current Top-5 evidence, and the rules are general: do not answer the original question, do not add Cloudflare facts, give only 2–3 narrower questions. It was not told the categories, which chunks rank low, what experiments came before, or “the right way to split”. Parameters are as before: qwen3.8-flash, temperature 0, thinking off, called once; once generated the suggestions are frozen, with no retry and no polishing, and only then are they searched.

Its reason: “原问题同时询问了环境变量、secret和普通配置的设置方法，以及本地与线上部署的区别，涉及多个独立主题且当前证据主要聚焦于本地开发中的 secret 处理。Translation: The original question asks at once how to set environment variables, secrets and plain configuration, and how local and production deployments differ: several separate topics, while the current evidence is mainly about handling secrets in local development.” The questions it suggested:

• S1在 Cloudflare Workers 中，如何区分设置普通环境变量（vars）和敏感信息（secrets）？Translation: In Cloudflare Workers, how do I set plain environment variables (vars) and sensitive values (secrets) separately?
• S2线上部署时，如何通过 Wrangler CLI 或 Dashboard 为 Worker 添加 secrets？Translation: For a production deployment, how do I add secrets to a Worker through the Wrangler CLI or the Dashboard?
• S3本地开发和线上部署在加载 .dev.vars/.env 文件与生产环境配置时有何具体区别？Translation: When loading .dev.vars/.env files and production configuration, what exactly differs between local development and a production deployment?

Say the user taps S2. That is one new ordinary query, on the same bge-m3, with nothing changed. The result for each of the three suggestions:

Its top three are Via the dashboard, Via Wrangler, Upload secrets alongside code: the dashboard path, wrangler secret put, and --secrets-file for uploading secrets with the code.

S2: C3 goes from rank 36 to rank 2.The piece that ranked 36 for the original question gets into the top 3 once the question asks only “how do I set a secret in production”.

A final answer for S2 only (with S2’s own Top-5 and the same citation rules):

在线上部署时，可以通过以下两种方式添加 Worker secrets：

1. **通过 Dashboard**：
   - 在 Cloudflare dashboard 中进入 **Workers & Pages** 页面 [1]。
   - 选择你的 Worker > **Settings** [1]。
   - 在 **Variables and Secrets** 下选择 **Add** [1]。
   - 选择类型 **Secret**，输入 **Variable name** 和 **Value**（值将在 Wrangler 和 dashboard 中隐藏）[1]。
   - 可选地，选择 **Add variable** 以添加更多 secrets [1]。
   - 最后选择 **Deploy** 以实施更改 [1]。

2. **通过 Wrangler CLI**：
   - 使用 `wrangler secret put <KEY>` 命令。这会创建一个新的 Worker 版本并立即部署 [2]。
   - 如果使用 gradual deployments，则使用 `wrangler versions secret put <KEY>` 命令。这仅创建新版本，随后需使用 `wrangler versions deploy` 进行部署 [2]。
   - 注意：Wrangler 3.73.0 之前的版本需要指定 `--x-versions` 标志 [2]。
   - 此外，可以使用 `--secrets-file` 标志在部署代码的同时上传 secrets，例如 `npx wrangler deploy --secrets-file .env.production` 或 `npx wrangler versions upload --secrets-file secrets.json` [3]。

For a production deployment, Worker secrets can be added in two ways:

1. **Through the Dashboard**:
   - In the Cloudflare dashboard, go to the **Workers & Pages** page [1].
   - Select your Worker > **Settings** [1].
   - Under **Variables and Secrets**, select **Add** [1].
   - Choose the type **Secret**, and enter the **Variable name** and **Value** (the value is hidden in Wrangler and the dashboard) [1].
   - Optionally, select **Add variable** to add more secrets [1].
   - Finally, select **Deploy** to apply the change [1].

2. **Through the Wrangler CLI**:
   - Use the `wrangler secret put <KEY>` command. This creates a new Worker version and deploys it immediately [2].
   - With gradual deployments, use the `wrangler versions secret put <KEY>` command instead. This only creates a new version, which you then deploy with `wrangler versions deploy` [2].
   - Note: Wrangler versions before 3.73.0 need the `--x-versions` flag [2].
   - You can also upload secrets together with the code using the `--secrets-file` flag, for example `npx wrangler deploy --secrets-file .env.production` or `npx wrangler versions upload --secrets-file secrets.json` [3].

The limits stay on record; recovery is not a cure-all:

• S1 (the difference between plain variables and secrets) is not solved: C1 moves up to rank 31 and C2 to rank 16, still outside the Top-10. The most basic piece of Q7 is still not recovered.
• S3 (local development versus production configuration) mostly just covers C4, which could already be found; no new progress.
• One case, one set of suggestions, one call; the final answer did not use the “evidence checklist” gate, because none was written for this question.

A good RAG need not answer every question in one go, but it should know when the evidence is not enough and help the user continue.The result here is partial: one sub-question was answered completely and with grounds, and the other still was not.

07 · Where it stands, and a look back

When you meet a RAG failure, think this way; what worked, what did not, what was not done

1. Relevant ≠ enoughQ7’s Top-10 has nothing off topic, yet covers only 1/4 of the four pieces needed.
2. A high cosine ≠ can answerWith no KV allowance in the corpus, the Top-5 scores are 0.574–0.593; Q5, which has an answer, scores 0.608 at rank 5. A score threshold cannot block it.
3. Not in the corpus ≠ retrieval missed itCheck the corpus before blaming retrieval; and the answer may only say “the evidence provided does not include it”, never “the official docs do not have it”.
4. A reranker, deduplication, rewriting and splitting each have conditions where they applyQ8’s problem was fixed by k=6; on Q7 none of these got the missing evidence into the Top-10. Decide which layer the failure is at, then choose the mechanism.
5. Do not overfit the baseline to one querynomic is better locally on Q7, but bge-m3 is still stronger on the whole benchmark; change the baseline on the whole benchmark, not on one case.
6. Recovery for compound questions can be a product capability; there is no need to keep stacking retrievalLet the system say the evidence is not enough and offer narrower questions, and S2 takes C3 from rank 36 to rank 2 and gets an answer with citations.

Looking back: what this round did

Worked

• With evidence it answers with citations; with none in the corpus it declines instead of making something up (Q5, Q9a).
• Located a typical failure: Q7’s Top-10 has nothing off topic, yet covers only 1/4 of the pieces the answer needs.
• Top-k works for Q8: at k=6 the missing piece is filled in (1/2 → 2/2).
• Having the system admit the evidence is not enough and then offer narrower questions: the missing “how do I set a secret in production” piece goes from rank 36 to rank 2, with an answer that cites its sources.

Did not work

• Top-k, deduplication, a reranker, splitting and rewriting: not one got Q7’s missing evidence into the Top-10.
• Changing the embedding: nomic reaches 3/4 on Q7, but the “plain variables” piece falls to rank 134, and it is not as good as bge-m3 on the whole benchmark either.
• The “plain variables” and “secrets are encrypted” pieces never reach the Top-10 on the bge-m3 baseline, even after recovery.

Not done

• Not a product: no interface, no online deployment; no vector database, no BM25 / hybrid retrieval, no metadata filtering; no fine-tuning.
• The reranker, rewriting and deduplication are only experiments; none entered the baseline.
• The evaluation is small: 9 questions, Q7 only one of them; each experiment that calls a model ran once.
• Generation quality was not evaluated, and different generation models were not compared.

After reading, you should be able to answer this: facing a retrieval failure, what to judge first — is the evidence missing from the corpus, outside the top k, or there but ranked deep — and can the system, for now, admit it is not enough and help the person carry on.

08 · Reproduction notes

If you want to reproduce it, roughly what you need

This is not a complete setup guide. It tells an experienced engineer where the material came from, what was used and where the experiments are. The code and corpus are in the author’s repository and are not public at the moment; the page has no download.

Data

The official Cloudflare documentation, taking each page’s Markdown export (index.md): 26 pages, snapshot batch v1-2026-10-04, fetched within a very short window on 2026-10-04, with a sha256 recorded per file. It is a span in which the data was taken and does not correspond to one commit of the official repository; a later update means a new batch, never overwriting this one. Three layers: raw (the responses as saved) → normalized (only the interface elements the documents record removed; for the long pages that mix several topics, only the relevant sections kept) → chunks.

Chunking

Cut by structure: a heading, a paragraph, a list item, a table, a code block and a <details> are each a block; a heading is bound to the explanation right below it; a chunk never crosses a heading; each chunk tries to stay within about 2400 characters, with zero overlap between neighbours; tables and code blocks are never split. Only one chunk in the whole corpus is over the limit (5759 characters, a compatibility matrix table), and it is kept whole. A chunk’s id carries a hash of its text.

Embedding

Input = document title + heading path + chunk text. The baseline is bge-m3 (1024 dimensions); nomic-embed-text-v1.5 and snowflake-arctic-embed-l-v2.0 were compared. An earlier bake-off also ran bge-small-en-v1.5 (512-token window, which truncates 57 chunks). The current baseline is still bge-m3.

Reranker

BAAI/bge-reranker-v2-m3, used only in the Q7 experiments and not in the baseline; run on CPU through a third-party ONNX export.

Generation

qwen3.8-flash (Alibaba Cloud Model Studio’s OpenAI-compatible interface), temperature 0, enable_thinking=false, JSON output; the key and address are read only from environment variables: DASHSCOPE_API_KEY, DASHSCOPE_BASE_URL.

Environment

Python 3.11, ONNX Runtime 1.30.0, numpy 2.4.6, 4 CPU threads, no GPU; the model files are in a local cache, not in the repository.

experiments/rag/cloudflare-docs-practice-03/
  snapshots/            corpus snapshots (raw, normalized, manifest)
  ingest/               chunking rules and chunks
  embedding/            embedding bake-off and benchmark
  case-01/              Q5's full run, the Q9a boundary test
  q8-decomposition/     hand-written splitting
  topk-sensitivity/     Top-k
  q7-diversity/         deduplication and MMR
  q7-reranker/          reranker
  q7-query-split/       splitting by sentence
  q7-query-rewrite/     rewriting
  q7-embedding-comparison/  embedding comparison
  q7-recovery-loop/     suggestions, re-retrieval and the final answer

Each experiment folder has its own report.md and an offline check script; the numbers on this page come from the results.json there.