sdr_manual_eval_2026_03_08
SDR Manual Evaluation — 2026-03-08
Run date: 2026-03-08 12:20 MSK
Model: llama-3.1-8b-instant (Groq free tier)
Config: .env.benchmark (reduced token limits, SearXNG local search)
Queries run: 3/10 (sci_001, sci_002, sci_003)
Evaluator: Feanor agent (manual scoring)
Evaluation Rubric (1–5 scale)
| Criterion | Description |
|---|---|
| Coverage | Are all major relevant methods/concepts mentioned? |
| Accuracy | Are facts, numbers, and attributions correct? |
| Coherence | Is the report logically structured and readable? |
| Depth | Are mechanisms explained? Math/algorithms present? |
| Citations | Are real, verifiable sources provided? |
| Completeness | Does the report fully answer the question? |
Results
sci_001: “State-of-the-art methods for operator splitting in ML optimization”
Report size: 5641 chars
| Criterion | Score | Notes |
|---|---|---|
| Coverage | 1/5 | Missed ALL real methods: ADMM, ISTA/FISTA, Douglas-Rachford, proximal gradient, PDHG. Instead invented “ADO”, “PDOF”, “PDV”, “DAOE” — none of these exist. |
| Accuracy | 1/5 | Complete hallucination. All 4 “methods” are fabricated. Comparison tables are meaningless. |
| Coherence | 2/5 | Structural flow is logical (intro → overview → comparison → recs), but hollow. |
| Depth | 1/5 | No math, no convergence rates, no proximal operator definition, no connection to ML tasks. |
| Citations | 1/5 | “[Один]”, “[Два]”, “[Три]”, “[Четыре]” — Russian words for numbers, not real citations. |
| Completeness | 1/5 | Critically incomplete. A domain expert would reject this immediately. |
| Total | 7/30 | 23% |
Critical failure: Model hallucinated entire methods instead of recalling real literature. Likely caused by poor web search quality + insufficient model knowledge (8b parameter model on specialized domain).
sci_002: “How does FlashAttention improve transformer training efficiency?”
Report size: 2988 chars
| Criterion | Score | Notes |
|---|---|---|
| Coverage | 2/5 | Mentions FlashAttention exists and is faster. Misses: IO-aware computation, SRAM blocking, O(N²)→O(N) memory, FlashAttention 2/3 versions. |
| Accuracy | 2/5 | “Уменьшение точности на 2%” — factually wrong. FlashAttention is mathematically equivalent to standard attention (same output). “4x faster” — no benchmark source. |
| Coherence | 3/5 | Best structured of the three. Readable. |
| Depth | 1/5 | Doesn’t explain WHY it’s faster (memory bandwidth bound → SRAM tiling). Core mechanism missing. |
| Citations | 1/5 | “[1] FlashAttention paper”, “[2] Paper о влиянии…” — placeholder text, not real DOI/arxiv refs. |
| Completeness | 2/5 | Superficially touches the topic, key technical insight absent. |
| Total | 11/30 | 37% |
Key error: “reduces accuracy by 2%” is a fabrication — FlashAttention is numerically equivalent to standard attention up to floating-point precision. This is a dangerous hallucination.
sci_003: “Current evidence on neural operators for solving PDEs”
Report size: 1914 chars
| Criterion | Score | Notes |
|---|---|---|
| Coverage | 2/5 | Calls them “нейтральные операторы” (neutral operators?) — likely translation artifact or confusion. No mention of FNO, DeepONet, WNO, U-NO, PINN. |
| Accuracy | 1/5 | Vague comparative table with “высокий/низкий” — no actual results, no benchmarks (Navier-Stokes, Darcy flow, Burgers equation). |
| Coherence | 2/5 | Short (1914 chars) but somewhat structured. |
| Depth | 1/5 | No description of what neural operators actually are, how FNO works, operator learning paradigm. |
| Citations | 1/5 | No citations at all. |
| Completeness | 1/5 | 1914 characters is catastrophically short for “current evidence” on an active research area. |
| Total | 8/30 | 27% |
Critical issue: Report is ~4x shorter than sci_001, which itself was insufficient. A proper literature review of neural operators would need 20-50K chars minimum.
Summary Statistics
| Query | Coverage | Accuracy | Coherence | Depth | Citations | Completeness | Total | % |
|---|---|---|---|---|---|---|---|---|
| sci_001 (Operator Splitting) | 1 | 1 | 2 | 1 | 1 | 1 | 7 | 23% |
| sci_002 (FlashAttention) | 2 | 2 | 3 | 1 | 1 | 2 | 11 | 37% |
| sci_003 (Neural Operators PDE) | 2 | 1 | 2 | 1 | 1 | 1 | 8 | 27% |
| Average | 1.67 | 1.33 | 2.33 | 1.0 | 1.0 | 1.33 | 8.7 | 29% |
Key Findings
What Failed
-
Hallucination of methods (sci_001): Model invented 4 non-existent methods instead of real ADMM/ISTA/FISTA. Root cause: llama-3.1-8b doesn’t have reliable knowledge of specialized ML optimization literature.
-
No real citations (all): All 3 reports provide placeholder-style references. The web search is not being used to find actual papers with DOI/arxiv IDs.
-
Report size too small: sci_003 = 1914 chars; sci_002 = 2988 chars. Compare to OpenAI Deep Research (typically 10K-50K chars per report).
-
Depth = 0 across all: No mathematical formulations, no convergence bounds, no algorithmic descriptions. The model is pattern-matching on domain vocabulary without substance.
-
Factual errors (sci_002): “FlashAttention reduces accuracy by 2%” is factually incorrect and dangerous.
Root Causes
- Model too small: llama-3.1-8b is insufficient for scientific synthesis. Even llama-3.1-70b struggles with specialized topics.
- Search not working: Reports don’t cite real sources from web search. Either search is failing or model is ignoring search results.
- Token limits too restrictive:
RESEARCH_MAX_TOKENS=4000,FINAL_REPORT_MAX_TOKENS=6000may be forcing truncation of retrieved content. - No citation extraction: System has no mechanism to extract and verify DOI/arxiv identifiers from retrieved pages.
Recommendations for SDR Improvement
| Priority | Action | Expected Impact |
|---|---|---|
| P1 | Use claude-haiku-4-5 as research model (instead of llama-3.1-8b) | Major quality improvement |
| P1 | Debug web search — verify results are being used in synthesis | Fix citation failure |
| P2 | Increase FINAL_REPORT_MAX_TOKENS to 20000+ | Longer, more complete reports |
| P2 | Add citation extraction module (extract arxiv/DOI from retrieved pages) | Fix citations criterion |
| P3 | Add minimum report length check (reject < 5000 chars) | Completeness floor |
| P3 | Add self-consistency check: model lists methods → verify against retrieved content | Reduce hallucinations |
Comparison Target
For context, this is what SOTA looks like on DeepResearch Bench:
- Gemini 2.5 Pro: 48.88/100 on PhD-level questions, 111 citations/task
- SDR 8b-instant: ~29% on our 6-criterion rubric, ~0-4 citations/task
Conclusion: Current SDR with llama-3.1-8b-instant is not viable for scientific research synthesis. Switching to claude-haiku or fixing the search pipeline should be the first step before any further benchmarking.