Clarify-Then-Search: A Clarification Benchmark for Deep Search with End-to-End Nugget Restoration

Clarify-Then-Search: A Clarification Benchmark for Deep Search with End-to-End Nugget Restoration evaluates whether LLM clarification improves downstream deep search under a closed-book protocol.

Replace the paper button with the ACM Digital Library, OpenReview, or arXiv link when available.

The benchmark contains 518 information-seeking queries selected to be underspecified, where clarification is expected to provide high utility. Each instance has an intent query fused_query and an underspecified query blurred_query.

Under a closed-book protocol, a system observes only blurred_query, asks k clarification questions, receives constrained answers, rewrites to q̂, and is evaluated by running a fixed deep-search backend on q̂ and scoring nugget restoration against a static gold built from fused_query.

Why no supervised “gold” clarification labels? We evaluate whether the model asks for information that is answerable under the hidden intent and useful for downstream deep search, avoiding a single canonical clarification target.

We provide a static golden reference per intent query (JSONL). Each gold record contains weighted nuggets and traceability fields. At evaluation time, a candidate answer is scored by weighted nugget recall with partial credit: full=1, partial=0.5, none=0.

restore_score_100 = 100 * ( sum_j w_j * s(cov_j) ) / ( sum_j w_j )
s(full)=1, s(partial)=0.5, s(none)=0

Only gold.nuggets are required by eval_gold.py for scoring; other fields are included for debugging and analysis.

Below are compact summary tables for one-turn and three-turn clarification results reported in the paper. At k=1, GPT achieves the strongest one-turn result; at k=3, ERNIE-4.5-Turbo-128K achieves the best overall performance.

One-turn clarification results (k=1)

Three-turn clarification results (k=3)

The three-turn setting highlights that larger clarification budgets reward sustained question-selection quality: ERNIE-4.5-Turbo-128K obtains the highest mean, median, and p90 scores at k=3.

Please cite our paper if you use the benchmark, code, or evaluation artifacts.

@inproceedings{huang2026clarifythensearch,
  title     = {Clarify-Then-Search: A Clarification Benchmark for Deep Search with End-to-End Nugget Restoration},
  author    = {Huang, Deqiang and Zhou, Jingbo and Lu, Xinjiang and Xu, Tong and Wu, Hua and Chen, Enhong},
  booktitle = {Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining},
  year      = {2026}
}

Code: MIT License (see LICENSE)

Data: CC BY 4.0 (see DATA_LICENSE)

This release includes model-generated outputs and automatically judged scores. Provided answers may contain errors.