Coding Week 7

Week 6 ended with the dbo-only pipeline fully deployed to production, a working Validator node with agentic probe, and several prompt and evaluation harness fixes. Week 7 shifts focus entirely to evaluation. Running the pipeline across multiple models and benchmarks, understanding where it succeeds and where it fails, and setting up the infrastructure for clean, reproducible comparisons going forward.

Cross-Model Evaluation on DB26

The primary goal this week was to run the full 50-question DB26 evaluation across all three models the project is working with: Claude Sonnet 4.6, DeepSeek v3.2, and Qwen 3.5 122B.

json{
  "benchmark": "DB26",
  "questions": 50,
  "results": {
    "Claude Sonnet 4.6":  { "result_set_match": "24/50 (48%)", "avg_f1": 0.55 },
    "DeepSeek v3.2":      { "result_set_match": "18/50 (36%)", "avg_f1": 0.41 },
    "Qwen 3.5 122B":      { "result_set_match": "21/50 (42%)", "avg_f1": 0.45 }
  }
}

Claude Sonnet 4.6 remains the strongest model by a clear margin. DeepSeek and Qwen are reasonable open-source baselines, with Qwen slightly ahead of DeepSeek after the fix described below. The baseline at the start of the project was F1=0.32, so all three models are now significantly above that.

Fixing Qwen's NoneType Response Issue

During the first Qwen evaluation run, 12 out of 50 questions crashed the pipeline with an error: expected string or bytes-like object, got 'NoneType'. This meant the Qwen API was returning an empty response for those questions. It was not a refusal, just a Null content field.

The pattern was clear: all 12 affected questions were complex multi-hop, aggregation, or multi-entity questions. Qwen handled simple ASK questions (yes/no questions) and single-hop SELECT questions without any issue. The empty responses only appeared on questions that required more internal reasoning.

The exact root cause is unclear. It could be Qwen's internal reasoning mode consuming output tokens on complex questions and returning empty rather than truncated output, or it could be transient API behaviour on OpenRouter for longer generations. Either way, the fix is the same: add retry logic directly in the LLM call. When the response content is None, retry the same call up to a maximum of 3 times before failing. This is more cost efficient than simply increasing the token limit for all models, which would increase costs on every single call regardless of whether the problem occurs. The retry only fires an extra call when the response is actually empty.

for attempt in range(3):
    response = client.chat.completions.create(
        model=model, messages=messages,
        temperature=0, max_tokens=max_tokens
    )
    content = response.choices[0].message.content
    if content is not None:
        break

Since Claude and DeepSeek never return None in practice, the retry loop exits on the first attempt for those models with zero extra cost. For Qwen, most questions (38/50) also succeed on the first try. For the remaining 12, the retry succeeded on the second or third attempt, dropping pipeline errors from 12 to 1 on the same benchmark. The remaining single error (Q5, a complex full-scan taxonomic query) is a known difficult question that all three models struggle with.

Adding DB25 as a Second Benchmark

Week 7 also added DB25 (the 2025 edition of the Text2SPARQL benchmark, 100 questions) as a second evaluation dataset. The motivation is cross-benchmark generalisation: a pipeline that only performs well on one benchmark has limited scientific credibility. Running on a second dataset helps verify that the results are not specific to DB26.

json{
  "benchmark": "DB25",
  "questions": 100,
  "results": {
    "DeepSeek v3.2": { "result_set_match": "46/100 (46%)", "avg_f1": 0.53 }
  }
}

DeepSeek achieves F1=0.53 on DB25, which is notably better than its DB26 performance (0.41). DB25 appears to contain more straightforward single-hop questions on average, making it a somewhat easier benchmark for the current pipeline.

It is worth noting that DB25 has some dataset-level limitations that affect evaluation reliability. Several gold queries use invalid SPARQL syntax (SELECT DISTINCT COUNT(?var) is not valid SPARQL, the correct form is SELECT (COUNT(DISTINCT ?var) AS ?count)), and several gold queries are so expensive that they time out even at 40 seconds on the evaluation endpoint. These questions automatically score zero regardless of what the pipeline generates, which means the actual pipeline performance on well-formed questions is better than the aggregate F1 suggests. For this reason, DB26 remains the primary evaluation benchmark going forward, with DB25 used as a secondary reference only.

The infrastructure for DB25 evaluation was also built this week: a gold result cache script that pre-executes all 100 gold queries once and saves the results to disk, eliminating gold-side timing variance between runs. The same approach was used for DB26.

README and Documentation

A comprehensive README was published to the repository this week covering the full pipeline architecture, setup instructions from scratch, how to run evaluations, current benchmark results, and the key architectural decisions (dbo-only index, agentic probe, class-safe swap). This is the first proper documentation the repository has had since the project began.

Challenges

The endpoint stability when running the DB25 evaluation and building the gold result cache was a real blocker this week. The evaluation endpoint went down twice during gold cache builds. Once mid-run, producing a half-built cache that had to be discarded and rebuilt. After the mentors increased the server swap to 8GB on the endpoint, the stability improved significantly, but the experience highlighted how dependent the evaluation pipeline is on endpoint availability. The gold cache approach (pre-executing gold queries once and reusing them) helps reduce the number of hits the endpoint sees, hence having reduced burden to execute more queries back to back.

The DB25 dataset limitations as mentioned in the previous section was also a challenge to reason about fairly. Deciding not to patch/correct the gold queries (even though fixing the COUNT syntax would be technically correct) seems to be the right call scientifically, since patching would make the gold results non-comparable to any other system evaluated on the same benchmark. Accepting the zero scores from broken gold queries and noting the limitation explicitly I feel was more honest.

What's Next

The pipeline has a concrete next milestone: breaching the F1=0.614 mark, which is currently 2nd place on the official Text2SPARQL 2026 leaderboard. This is the next target the mentors have set for the pipeline in the coming weeks. Since the full evaluation on Claude Sonnet is being held for a milestone run rather than regular iteration (due to high costs of the model), we will be using DeepSeek and Qwen for regular evaluations instead. The idea is that if a fix improves F1 on open-source models, it will improve on Claude too.

The main tracks for the coming weeks are:

Targeted pipeline improvements from the failure analysis done this week. The first planned task is to extend the Validator node with a fourth routing option that detects when a subject entity returns no data at all from the live agentic probe, which is a strong signal that the entity URI itself is wrong rather than the predicate. In such cases, instead of giving up, the Validator would route back to the Entity Linker with a cleaned version of the entity name (for example stripping disambiguation suffixes like _(series) that produce dead URIs in DBpedia). The second task is adding hop count as an explicit Planner output field so the Query Builder receives it as a structured signal rather than inferring it from natural language alone.

Evaluating on the QALD-9-Plus dataset (DBpedia subset) as a third benchmark for further cross-dataset generalisation.

Running evaluations again after each targeted fix using DeepSeek and Qwen to track whether improvements are measurable before the final Claude milestone run.