Reliable PDF Search Agent - SEP 2026 Group 2: CoDex

A modular Retrieval-Augmented Generation (RAG) system for answering questions over collections of PDF documents, with grounded citations, explicit abstention when evidence is weak, and a reproducible evaluation pipeline.

Project documentation and submitted PDFs are maintained in the companion documentation repository.

This repository contains the implementation code. The project is still under active milestone development, so some advanced modules are implemented as stubs or are not yet wired into the end-to-end pipeline.

Recognition

• 1st place at the Tag der jungen Softwareentwickler*innen (TdSE) 2026 at TU Braunschweig: a jury from research, industry and the student body awarded CoDex first prize among the 33 projects of this year's Softwareentwicklungspraktikum (16 July 2026). Coverage: TU Braunschweig Magazin (German).

Quick start

Prerequisite: use Python 3.10 or newer. The project uses modern type-hint syntax such as str | None, so Python 3.9 and older will fail before the requirements can help.

python3 --version              # must be 3.10 or newer
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements-dev.txt
pip install -e .
cp .env.example .env          # add your GEMINI_API_KEY (optional)
pytest                         # unit tests; some future-milestone tests are skipped
make smoke                     # CI-compatible smoke eval using the echo provider

Repository layout

code/
  configs/                # YAML pipeline/evaluation configs (baseline, hybrid, rerank)
  datasets/
    README.md             # dataset structure and usage notes
    sample/               # tiny in-repo corpus + QA (drives tests & smoke suite)
    adversarial/          # unanswerable + contradiction cases (M5 / M7)
    open_rag_bench/       # reserved for downloaded Open RAG Benchmark data (gitignored)
  reports/                # timestamped eval outputs (JSON + Markdown)
  scripts/                # one-off utilities (benchmark download, adversarial build)
  src/
    rag/
      ingest/             # PDF parser, header/footer cleaner, tables, OCR
      chunking/           # recursive, layout-aware, semantic
      retrieval/          # dense, BM25, hybrid (RRF), cross-encoder reranker
      generation/         # LLM clients (Gemini/HF/Ollama/echo), prompt, citations,
                          # abstention, confidence
      pipeline.py         # orchestrator
      config.py, types.py # dataclass schemas shared across layers
      cache.py, cost.py   # prompt cache + token-to-euro helpers
      logging_utils.py    # JSON logging + step timer
    eval/
      dataset.py          # EvalItem + JSONL/suite loaders
      metrics/            # retrieval, answer, grounding, abstention,
                          # robustness, efficiency
      judge.py            # LLM-as-judge for abstractive answers
      benchmarks/         # sample + Open RAG Benchmark adapters
      report.py, run.py   # report rendering + CLI entry point
    api/                  # FastAPI app (/query, /index, /stats, /health)
    ui/                   # Streamlit management dashboard
      assets/             # UI images and static assets
  tests/                  # pytest suite (mirrors src/ layout)
  .uploads/               # local uploaded documents used by the UI/API
  .gitlab-ci.yml          # lint + unit tests + smoke-eval regression gate
  Makefile                # install / test / smoke / eval / api / ui targets
  pyproject.toml          # pytest, ruff, mypy config
  requirements.txt        # runtime deps
  requirements-dev.txt    # runtime + dev deps
  .env.example            # environment variables template

Stack

Component	Current choice/status	Notes
Language	Python 3.10+	pyproject.toml requires Python >= 3.10
PDF parsing	PyMuPDF, cleaner utilities, table extraction, OCR fallback	Core parsing works; table extraction renders tables as Markdown, and OCR is best-effort for scanned PDFs
Chunking	Recursive, layout-aware, and semantic chunking	Recursive chunking is the baseline; layout-aware and semantic strategies are available through config
Retrieval	BM25, dense, and hybrid RRF retrievers	Hybrid retrieval is available through the retriever factory; cross-encoder reranking is still pending
Embeddings	sentence-transformers models	Baseline YAML uses all-MiniLM-L6-v2; hybrid/rerank use BAAI/bge-small-en-v1.5
Vector store	In-memory default, Chroma planned	Persistent store settings exist in config; full persistence is not the default path yet
LLM generator	Gemini, Hugging Face, Ollama, or Echo	Provider selected by YAML/environment; baseline uses Echo for CI-safe smoke runs
LLM judge	Configured judge section, implementation stub	Judge parameters are in YAML; actual judge scoring still needs implementation
Eval	Custom JSONL harness and report writer	Smoke/adversarial suite loading, core metrics, per-item diagnostics, and JSON/Markdown reports exist; benchmark download/adapter still pending
Backend	FastAPI + Uvicorn	/query, /index, /stats, /health style API
UI	Streamlit	Upload/index, chat, traces, and eval report views
Tests / CI	pytest + GitLab CI	Several milestone tests are intentionally skipped until their modules are complete

LLM Provider Strategy

Provider choice is abstracted behind rag.generation.llm_client.LLMClient and selected through YAML plus environment overrides. The current setup uses:

• Gemini as the default hosted provider for normal generation and judge runs. gemini-2.5-flash is used for generation-oriented runs, while gemini-2.5-pro is configured as the judge model.
• Hugging Face Inference as a hosted fallback when Gemini is unavailable or when a separate model family is useful for comparison.
• Ollama as the local/cluster provider path. This is the direction for runs on the institute cluster where models can be hosted through an Ollama endpoint instead of an external API.
• Echo as an offline stub for CI, smoke runs, and development without API credentials. Echo verifies indexing, retrieval, report generation, and the end-to-end control flow, but it does not measure real answer quality.

Switching provider or model is a configuration change: set generation.provider / generation.model in YAML or override the provider with RAG_GENERATION_PROVIDER. For Ollama, set OLLAMA_HOST to the local or cluster endpoint.

Evaluation Modes

The project currently uses two different evaluation modes:

• CI smoke evaluation: make smoke / GitLab smoke-eval runs configs/baseline.yaml, which uses generation.provider: echo. This mode is deterministic and does not need API keys or a GPU. Its retrieval, efficiency, and report-generation results are meaningful; answer and grounding quality are expected to be poor because Echo returns a simulated answer.
• Real generation evaluation: local or cluster runs can set RAG_GENERATION_PROVIDER=gemini, ollama, or huggingface. These reports should be used for answer quality and citation quality. Recent smoke runs with a real/local model reached perfect retrieval on the sample suite and strong answer/grounding scores.

Reports are written to reports/runs/<timestamp>_<suite>/report.json and report.md. The JSON and Markdown reports include aggregate metrics plus per-item diagnostics: retrieved chunks, parsed citations, invalid citations, per-question answer scores, retrieval hits, latency, and issue labels.

Evaluation Pipeline Details

The repository has two related evaluation entry points.

Standard eval

Use eval.run for quick smoke tests, provider comparisons, and deterministic diagnostics:

python -m eval.run --config configs/hf-strict-smoke.yaml --tag my-run

The standard eval pipeline runs in this order:

1. Load the YAML config and the selected suite.
2. Parse and index the suite documents.
3. Answer each question with the configured provider.
4. Save each completed item incrementally.
5. Compute built-in metrics.
6. Write report.json and report.md.

Incremental raw outputs are written while the run is still in progress:

reports/<run-root>/<timestamp>_<suite>/raw/items/<item_id>.json
reports/<run-root>/<timestamp>_<suite>/raw/items.jsonl

The final standard report contains:

retrieval:   recall@5, MRR, nDCG@5, hit@5
answer:      exact match, token F1
grounding:   citation precision, citation recall
abstention:  false refusal rate, false answer rate
efficiency:  latency, tokens, cost summary

RAGAS eval

Use eval.ragas_run when the goal is judge-based RAGAS scoring:

python -m eval.ragas_run --suite full --config configs/ollama-full.yaml

The RAGAS pipeline has three stages:

1. Prepare inputs: answer every question and collect the generated answer, retrieved contexts, and reference answer.
2. Score inputs: call RAGAS metrics with the configured judge model and embedding model.
3. Write report: aggregate scores and write the final RAGAS artifacts.

RAGAS input preparation writes:

raw/inputs/<item_id>.json
raw/ragas_inputs.jsonl

RAGAS scoring writes:

raw/scores/<item_id>.json
raw/ragas_scores.jsonl

The final RAGAS outputs are:

ragas_report.json
ragas_scores.csv

This matters for long runs: if a run is interrupted during answer generation, only ragas_inputs.jsonl may exist. If it reaches scoring, partial scores are still preserved in ragas_scores.jsonl and raw/scores/. RAGAS may write NaN for a metric when the judge model cannot produce a parseable or applicable score; these values should be counted separately when comparing runs.

Prompt templates

Generation prompts are selected with generation.prompt_template.

• default: short baseline prompt for context-grounded answers.
• strict_citation: structured prompt with role, <context>, <user_query>, numbered instructions, strict no-outside-knowledge behavior, concise answer preference, and mandatory [doc_id:pN] citations.

Use strict_citation for prompt-quality experiments and RAGAS comparisons. Keep retrieval, chunking, model, and dataset fixed when comparing prompt variants so score changes can be attributed to the prompt.

Report tracking policy

Generated raw folders and LLM caches should stay local-only. The repository tracks selected comparison summaries under:

reports/comparisons/

Use meaningful comparison folder names that encode the provider/model, prompt, dataset size, and score coverage, for example:

20260618_ollama-llama3-1-8b_default-prompt_openragbench-10docs_87q-77scored
20260618_hf-llama3-8b_strict-citation-prompt_smoke_8q

Datasets

• datasets/sample/: a small hand-written corpus (photosynthesis, mitochondria, transformers) + 8 QA pairs. Drives the smoke eval suite and test fixtures. Stays in git.
• datasets/adversarial/: unanswerable questions (for abstention metrics) and a coffee / type-2-diabetes contradiction pair. Stays in git.
• datasets/open_rag_bench/: the Vectara benchmark (1000 arXiv PDFs + 3045 QA pairs). This directory is gitignored and reserved for the downloaded benchmark. The download script and adapter are still milestone work.

Current Status

Implemented and usable now:

• YAML-driven runtime and evaluation configuration in configs/.
• PDF/text parsing, recursive chunking, BM25 retrieval, an initial dense retriever, and RRF-based hybrid retrieval.
• LLM client abstraction for Gemini, Hugging Face, Ollama, and Echo.
• Prompt construction, citation extraction and validation, abstention helper logic, confidence heuristic, token/cost tracing, and disk-backed prompt caching.
• FastAPI backend and Streamlit management UI.
• Smoke/adversarial JSONL suite loading, core metric functions, per-item diagnostics, and JSON/Markdown evaluation reports.
• GitLab CI for Ruff linting, unit tests, and artifact-producing smoke evals.

Still incomplete or not fully wired:

• Cross-encoder reranking.
• Full pipeline/config use of dense and hybrid retrieval modes.
• Calibrated confidence in the final answer flow.
• LLM-as-a-judge execution.
• Open RAG Benchmark download/adapter and full benchmark run.
• Production-hardening for complex scanned PDFs, irregular tables, and unusual document layouts.

Milestones

Tracking links point to the GitLab milestone pages.

ID	Focus	Status	Tracking
M0	Repo scaffold, stack decided, stubs in place	done	M0 milestone
M1	Evaluation harness for all six metric categories	partially done - smoke/adversarial/robustness suite loading, YAML-driven eval config, core metrics, robustness slicing, per-item diagnostics, Markdown/JSON reports, and CI smoke artifacts exist; LLM judge execution and Open RAG Benchmark integration remain pending	M1 milestone
M2	Baseline RAG v1	partially done - parsing, recursive chunking, BM25-backed pipeline, dense retriever, providers, prompt/citation extraction, citation validation helper, disk prompt cache, API flow, abstention helper, and confidence heuristic exist; full dense/hybrid pipeline integration and calibrated confidence remain pending	M2 milestone
M3	PDF-parsing robustness	implemented for Part-A scope - PyMuPDF/pdfplumber parsing, repeated header/footer stripping, table extraction, OCR fallback, recursive/layout-aware/semantic chunking, and focused tests exist; further work can harden very complex PDFs	M3 milestone
M4	Hybrid retrieval + reranker	partially done - BM25, dense retrieval, RRF hybrid fusion, and factory construction for hybrid mode exist; cross-encoder reranking and full pipeline/config integration remain pending	M4 milestone
M5	Strict citations + calibrated confidence	partially done - strict citation prompt/citation extraction, citation validation helper, abstention rules, and confidence heuristic exist; pipeline-level validation integration and calibration remain pending	M5 milestone
M6	Streamlit management UI	done - corpus upload/path indexing, chat playground, citation display with page previews, traces, and latest evaluation report views are implemented	M6 milestone
M7	Robustness + adversarial suite	implemented for Part-A scope - adversarial JSONL files, robustness suite loading, robustness slice metrics/reporting, and focused tests exist	M7 milestone
M8	Full benchmark run, ablation table, final report	pending	M8 milestone

See the full plan for per-milestone deliverables and the CI regression-gate strategy.

Team split

• Hiba - Ingestion: rag/ingest/*, rag/chunking/*, adversarial corpus.
• Mohammed - Retrieval: rag/retrieval/*, eval/metrics/retrieval.py.
• Finn - Generation: rag/generation/*, eval/metrics/answer.py, eval/metrics/grounding.py, eval/judge.py.
• Ines - Platform & evaluation harness: src/api/, src/ui/, eval/run.py, eval/report.py, eval/dataset.py, eval/metrics/efficiency.py, rag/config.py, rag/cost.py, rag/logging_utils.py, configs/*.yaml, GitLab CI, cost/latency instrumentation.

Everyone contributes to tests/ and to the final written report.

Make targets

Target	What it does
make install	Install runtime deps.
make install-dev	Install runtime + dev deps.
make test	Run pytest.
make lint	Ruff + mypy.
make smoke	Run the CI-safe smoke eval against configs/ci-smoke.yaml.
make eval	Run the full eval path against configs/baseline.yaml; requires the benchmark adapter/download work to be completed.
make api	Launch the FastAPI backend.
make ui	Launch the Streamlit UI.

See docs/linting.md for the linting rules that should be checked before pushing code changes.

Team Management Automation

The CI config includes scheduled jobs that are independent from the RAG app and use GitLab CI only as a cron runner:

Job	Purpose
issue-reminders	Emails assignees their open GitLab issues across code and abgaben.
stale-issue-comments	Comments on open issues with no activity for STALE_ISSUE_DAYS days and mentions assignees.
weekly-team-digest	Emails a team digest with new issues, closed issues, recent commits, and upcoming due dates.

The stale-comment job writes an internal marker in its comment so it does not repeat the same reminder every day. Keep STALE_ISSUE_DRY_RUN=true until the first dry-run output has been reviewed.

The weekly digest supports TEAM_DIGEST_RECIPIENTS and optional TEAM_DIGEST_CC as comma-separated email lists.

Open RAG Benchmark + RAGAS

Download and prepare the Vectara Open RAG Benchmark. For local Ollama runs, start small: verify smoke, then try about 50 benchmark PDFs before attempting the full dataset. --subset all can take a long time, use a lot of disk, and make debugging provider/model issues unnecessarily expensive.

python scripts/download_benchmark.py --subset 10
python scripts/download_benchmark.py --subset 50
python scripts/download_benchmark.py --subset all

--subset selects benchmark documents/PDFs, not questions. For example, --subset 10 can still produce many more than 10 questions because all QA pairs attached to the selected documents are prepared.

The downloader fetches vectara/open_ragbench from Hugging Face and writes a prepared text corpus plus QA file under datasets/open_rag_bench/prepared/. That directory is gitignored because the benchmark is large.

Check the prepared size before running:

find datasets/open_rag_bench/prepared/corpus -type f | wc -l
wc -l datasets/open_rag_bench/prepared/qa.jsonl

Run the full benchmark with Ollama/RAGAS:

ollama pull llama3.1:8b
ollama pull nomic-embed-text
ollama serve

export OLLAMA_HOST=http://localhost:11434
export OLLAMA_MODEL=llama3.1:8b

python -m eval.ragas_run \
  --suite full \
  --config configs/ollama-full.yaml \
  --judge-model llama3.1:8b \
  --embedding-model nomic-embed-text \
  --batch-size 1

Contacts

• Baptiste Bremond - baptiste.bremond@tu-braunschweig.de
• Michel Besserve - michel.besserve@tu-braunschweig.de