Deep-search systems are brittle when user queries omit important constraints such as time, location, scope, or definitions.
Clarify-Then-Search evaluates whether an LLM can ask questions that produce measurable downstream gains rather than merely
sounding plausible. The benchmark contains 518 paired instances derived from real-world Baidu search queries, with a clear intent
query and a corresponding underspecified query. A two-phase closed-book protocol prevents the Rewriter from accessing the hidden
intent, while a fixed WebDancer backend executes the rewritten query. Final utility is measured by
restore_score_100, a weighted nugget-recall score against static, evidence-grounded golden nuggets. Across the evaluated
models, clarification improves over the no-interaction baseline, while interaction diagnostics reveal that information gain—not
question fluency—is the central bottleneck.
Unzip, set your judge endpoint (Qianfan OpenAI-compatible), and run evaluation. The judge computes nugget coverage (full/partial/none) and outputs per-item and summary stats.
# 1) unzip
unzip clarify-then-search-518-release.zip
cd release
# 2) env (Qianfan OpenAI-compatible)
export QIANFAN_API_KEY="YOUR_KEY"
# optional:
export QIANFAN_BASE_URL="https://qianfan.baidubce.com/v2"
export EVAL_MODEL_NAME="ernie-4.5-turbo-128k"
# 3) run one candidate
python ./code/eval_gold.py \
--gold_a data/gold_public_hard518.jsonl \
--gold_b results/candidates/ebk1__cand_hard518.jsonl \
--out_dir outputs/ebk1Core artifacts needed to evaluate any model against the static golden nuggets.
release/
data/
hard_518_queries.csv
gold_public_hard518.jsonl
results/
candidates/
clarify_only/
code/
eval_gold.py
make_candidate.py
prepare_public_eval.pyClarify-Then-Search: A Clarification Benchmark for Deep Search with End-to-End Nugget Restoration evaluates whether LLM clarification improves downstream deep search under a closed-book protocol.
Author-hosted version. The official ACM Digital Library link will be added after publication.
The benchmark contains 518 paired information-seeking instances selected from scenarios where clarification is expected to provide high utility.
Each instance has a hidden intent query fused_query and a corresponding underspecified query blurred_query.
Under a closed-book protocol, a system observes only blurred_query, asks k clarification questions, receives constrained answers,
rewrites to q̂, and is evaluated by running a fixed deep-search backend on q̂ and scoring nugget restoration against a static gold built from fused_query.
Why no supervised “gold” clarification labels? We evaluate whether the model asks for information that is answerable under the hidden intent and useful for downstream deep search, avoiding a single canonical clarification target.
fused_query), blur key constraints to create underspecified inputs (blurred_query), and run WebDancer once
per intent query to archive evidence and extract weighted, traceable golden nuggets that are reused across all evaluations.
We provide a static golden reference per intent query (JSONL). Each gold record contains weighted nuggets and traceability fields. At evaluation time, a candidate answer is scored by weighted nugget recall with partial credit: full=1, partial=0.5, none=0.
restore_score_100 = 100 * ( sum_j w_j * s(cov_j) ) / ( sum_j w_j )
s(full)=1, s(partial)=0.5, s(none)=0
Only gold.nuggets are required by eval_gold.py for scoring; other fields are included for debugging and analysis.
GPT > ERNIE > orig ordering at k=1.
Clarification consistently improves over the no-interaction baseline. GPT-5.2 is strongest with one and two questions, while ERNIE-4.5-Turbo-128K becomes the overall top-performing Clarifier at three questions.
| Clarifier | k=1 mean (Δ) | k=2 mean (Δ) | k=3 mean (Δ) |
|---|---|---|---|
| orig no interaction | 19.434 | 19.434 | 19.434 |
| Qwen3-235B-A22B-Instruct | 22.527 (+3.093) | 26.020 (+6.586) | 26.235 (+6.801) |
| ERNIE-4.5-Turbo-128K | 23.460 (+4.026) | 26.351 (+6.917) | 28.339 (+8.904) |
| DeepSeek-V3.2 | 23.234 (+3.799) | 26.628 (+7.194) | 26.486 (+7.052) |
| Kimi-K2-Instruct | 22.687 (+3.252) | 25.593 (+6.159) | 26.568 (+7.134) |
| GPT-5.2 | 26.474 (+7.040) | 27.073 (+7.639) | 26.297 (+6.863) |
| Claude-Sonnet-4.5 | 25.911 (+6.477) | 26.203 (+6.769) | 27.029 (+7.594) |
| Gemini-2.5-Pro | 25.874 (+6.440) | 26.904 (+7.470) | 27.126 (+7.691) |
The results indicate that useful clarification depends on eliciting answerable, orthogonal constraints—not simply generating fluent or topically plausible questions. Additional turns help primarily by reducing no-signal interactions.
Please cite our paper if you use the benchmark, code, or evaluation artifacts.
@inproceedings{huang2026clarifythensearch,
title = {Clarify-Then-Search: A Clarification Benchmark for Deep Search with End-to-End Nugget Restoration},
author = {Huang, Deqiang and Zhou, Jingbo and Lu, Xinjiang and Xu, Tong and Wu, Hua and Chen, Enhong},
booktitle = {Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2},
year = {2026},
address = {Jeju Island, Republic of Korea},
publisher = {ACM},
numpages = {12},
isbn = {979-8-4007-2259-2},
doi = {10.1145/3770855.3817586}
}Code: MIT License (see LICENSE)
Data: CC BY 4.0 (see DATA_LICENSE)
This release includes model-generated outputs and automatically judged scores. Provided answers may contain errors.