Final Report: Agentic Question Answering over DBpedia

Contributor: Malla Siddharth Reddy

Mentors: Tommaso Soru, Ronit Banerjee, Gandharva Naveen, Abdulsobur

This project builds an agentic LangGraph pipeline that translates natural language questions into SPARQL queries against DBpedia, using a live agentic probe to recover from failed queries instead of a static predicate index. Full architecture and setup instructions are in the README (all links provided in the Links section below); this post summarises the work, the results, and what this summer actually taught me.

What I Built

Planner — extracts entities, concepts, answer type, aggregator (COUNT/SUM/GROUP_BY/ORDER_BY), join type (SINGLE/UNION/INTERSECTION), whether a type filter is needed, and the number of relationship hops the question requires
Entity Linker — Redis surface-form index with LLM disambiguation over ambiguous candidates, Unicode normalisation, and a heuristic fallback when Redis has no match
Ontology Explorer — semantic search over a dbo-only Nomic Embed v1.5 index, replacing an earlier AI-labelled dbp: index after a controlled comparison showed no measurable difference in performance but offered a simpler solution
Schema Introspector — enriches predicate candidates with domain, range, and label metadata from the DBpedia OWL ontology
Query Builder — generates SPARQL with num_hops enforced as a hard structural constraint, plus explicit rules for aggregation syntax, join type, type filters, and triple direction
Query Executor — a class-safe deterministic dbo: to dbp: namespace swap as a first fallback
Validator — a two-stage agentic probe that discovers what properties actually exist for an entity, plus three targeted recovery mechanisms: dead URI detection, two-hop probe chaining, and type-aware probe filtering
A full evaluation harness reporting F1, precision, recall, and agent steps per question (along with median and mode), and a live streaming demo UI rebuilt to reflect this exact pipeline

Results

F1 = 0.6119 with Claude Sonnet 4.6 on the Text2SPARQL 2026 DB26 benchmark, a 90% relative improvement over the pre-GSoC baseline (0.32). Qwen 3.5 122B reached F1 = 0.5109, a ~65% improvement over its own baseline. Full raw results are in eval-results/ and a per-question failure breakdown is in docs/final-results-analysis.md.

Challenges and What I Learned

This project taught me far more than I expected going in. The real DBpedia data underneath everything is messy in ways no tutorial prepares you for, inconsistent entity naming, disambiguation suffixes that silently break lookups, properties that exist under one namespace but not another, gold benchmark queries that use invalid syntax. Working through that messiness, rather than around it, was most of the actual engineering this summer.

I also got to properly compare architectures instead of just picking one and hoping. Running a full two-index system against a simpler dbo-only design side by side, and learning firsthand that the simpler, more defensible approach can match a more complex one that looks more thorough on paper. That tradeoff between simplicity and complexity came up again and again, and I don't think I understood it as well before this summer as I do now.

Beyond the technical side, I'm just genuinely grateful for the chance to work on something real, with real data, amazing mentors giving real feedback, and a real benchmark to be honest against. It has been the best learning experience I've had, and I'm walking away from it a noticeably better engineer than I was in April. Keeping all this in mind I have also decided to continue contributing to DBpedia post GSoC.

What's Left

The detailed reasoning behind each of these is in the "What Would Move F1 Beyond the Current Best" section at the end of docs/final-results-analysis.md (link provided below), scroll to the bottom for the full breakdown. In short:

Extending the agentic probe to check the object side of a relationship, not just the subject
A wider hop budget for questions needing more than two chained relationships
Result-set aware scoring that credits legitimate subsets or supersets of the gold answer
A dedicated query strategy for full-scan superlative questions with no entity to anchor a probe on