DnD Campaign Chatbot

A self-hosted RAG (Retrieval Augmented Generation) chatbot for querying D&D campaign notes. Polish-language content.

Features

• Hybrid search: BM25 (keyword) + vector (semantic) with RRF fusion and optional cross-encoder reranking
• Contextual prefixes: LLM-generated chunk context for better retrieval (Anthropic technique)
• Auto-indexing: Watches content directory, incrementally indexes new/modified markdown files
• Backend failover: Hosts are declared in backends.yml in preference order; generation moves to a cloud fallback when no local host answers, and retrieval drops to keyword-only rather than failing
• Follow-up questions: A follow-up is rewritten into a standalone query before retrieval, so history does not pollute the search
• REST API: Fast context endpoint for external integrations (used by Nanobot Telegram bot)
• Web UI: Chainlit chat interface with streaming responses
• Offline eval: A hand-written test set measures recall@k and MRR, so retrieval changes are compared rather than guessed at
• Polish language: Optimized for Polish content with wiki-link parsing

Architecture

Markdown files (dnd-summaries/content/)
        |
        v
   +---------+    contextual     +--------+
   |  ingest  |---prefixes (LLM)-| Ollama |
   |          |    embeddings     |(remote)|
   +----+-----+                  +--------+
        |
        v
   +----------+
   | ChromaDB |  (vector store, port 8100 external / 8000 internal)
   +----+-----+
        |
        v
   +----------+    hybrid search    +--------+
   |   chat   |----LLM response----| Ollama |
   |(Chainlit|    (streaming)      |(remote)|
   |+FastAPI) |                     +--------+
   +----------+
     port 7860

Which host serves generation and which serves embeddings is resolved at request time from src/backends.yml, not fixed at startup — the GPU desktop is off most of the time, and the chain is rebuilt when that changes.

Three Docker Compose services:

Service	Container	Port	Role
chromadb	dnd-chromadb	8100:8000	Vector store (persistent on NAS)
ingest	dnd-ingest	—	Indexes markdown, watches for changes
chat	dnd-chat	7860:7860	Web UI + REST API

Quick Start

1. Configure

cp .env.example .env

Edit .env:

CONTENT_PATH=/path/to/your/markdown/content
OLLAMA_BASE_URL=http://your-ollama-host:11434
LLM_MODEL=SpeakLeash/bielik-11b-v3.0-instruct:Q4_K_M
EMBEDDING_MODEL=bge-m3

2. Start

docker compose up -d

3. Use

• Web UI: http://localhost:7860
• Context API: http://localhost:7860/api/context?q=your+question
• Query API: http://localhost:7860/api/query?q=your+question
• Health: http://localhost:7860/api/health

Configuration

All settings via .env file:

Variable	Default	Description
CONTENT_PATH	./_content	Path to markdown files
OLLAMA_BASE_URL	http://ollama:11434	Ollama server URL
LLM_MODEL	gemma3:12b	Model for chat and contextual prefixes
EMBEDDING_MODEL	bge-m3	Model for embeddings
OPENROUTER_API_KEY	—	Cloud fallback for generation; empty disables the fallback
OPENROUTER_MODEL	google/gemini-3.6-flash	Model used on the fallback
RETRIEVER_K	6	Number of context chunks to retrieve
CHUNK_SIZE	1000	Text chunk size for indexing
CHUNK_OVERLAP	200	Overlap between chunks
RERANKER_ENABLED	false	Enable cross-encoder reranking — see what the reranker buys for the measured trade-off
CONNECT_TIMEOUT	3	Connect timeout; a powered-off host swallows the SYN, so this is what makes it fail fast
LLM_READ_TIMEOUT	180	Read timeout for generation — generous, since bielik-11b needs 30-40s
EMBED_READ_TIMEOUT	30	Read timeout for embeddings
LOG_LEVEL	INFO	Root log level

.env names the hosts and models; src/backends.yml is what maps them onto roles, in preference order. Adding another Ollama host is an entry in that file and no code change. It reads ${VAR} and ${VAR:-default} from the environment, so .env keeps driving the deployment.

RAG Pipeline

1. Ingestion (src/ingest.py): Markdown → chunks → contextual prefixes via LLM (cached) → embeddings via bge-m3 → ChromaDB
2. Condensing (src/rag.py): A follow-up question is rewritten into a standalone query, used for retrieval only
3. Retrieval (src/retriever.py): BM25 + vector search → RRF fusion → optional cross-encoder reranking (bge-reranker-v2-m3, off by default). Falls back to keyword-only when no embedding host answers
4. Generation (src/rag.py, served by src/chat.py and src/cl_app.py): Retrieved chunks as context → LLM streaming response, from the first live backend

API Endpoints

Endpoint	Speed	Description
GET /api/context?q=...	~1.5s	RAG chunks only, no LLM — for external integrations. Keeps answering in keyword-only mode
GET /api/query?q=...	~30-40s	Full LLM-generated answer with sources; 503 when no LLM backend is live
GET /api/health	instant	Live backends, retrieval mode and document count. Always 200 while the process answers — status says whether it is whole or degraded

Content Format

Pattern	Type	Example
DD.MM.YYYY.md	Session notes	15.11.2025.md
postacie/*.md	Character	Thok Darkhide.md
lokacje/*.md	Location	Luskan.md

Wiki-links ([[Name]] and [[Name|Display]]) are resolved during ingestion.

Commands

# Start
docker compose up -d

# Logs
docker compose logs -f chat
docker compose logs -f ingest

# Rebuild after code changes
docker compose build && docker compose up -d

# Force full reindex (delete ChromaDB collection)
curl -X DELETE http://localhost:8100/api/v2/tenants/default_tenant/databases/default_database/collections/dnd_campaign
docker compose restart ingest

# Stop
docker compose down

Development

# Run locally (needs Ollama + ChromaDB running)
uv run python src/ingest.py
uv run python src/chat.py

# Tests (no network, no ChromaDB)
uv run pytest

# Lint
uv run ruff check .

Retrieval eval

CHROMA_URL=http://localhost:8100 \
OLLAMA_BASE_URL=http://192.168.4.184:11434 \
CACHE_DIR=/mnt/nas/docker/dnd-chatbot/ingest_cache \
RERANKER_ENABLED=false \
uv run python src/eval/run_eval.py

Reports recall@k and MRR over src/eval/testset.jsonl, broken down by category and difficulty, and saves a timestamped JSON to src/eval/results/ so runs stay comparable. Toggle RERANKER_ENABLED for the A/B.

To force keyword-only retrieval while an embedding host is up, point OLLAMA_BASE_URL at a dead address — no live embedding backend is exactly what the degraded path is.

What hybrid search buys (measured)

33 questions against the live collection (310 chunks), reranker off:

	keyword-only	hybrid
substring recall@1	0.576	0.758
substring MRR	0.712	0.853
file recall@3	0.576	0.788
paraphrase recall@6	0.875	1.000
paraphrase MRR	0.505	0.900

paraphrase is the category that answers the question: it is the only one where keyword-only retrieval drops below 1.000 at k=6, because its questions avoid the notes' own vocabulary. Everywhere the question reuses a proper noun from the notes, BM25 alone is already at the ceiling — and on literal lookups hybrid fusion is actually slightly worse (easy_lookup MRR 0.900 → 0.750), because vector hits displace exact keyword matches near the top.

What the reranker buys (measured)

Same 33 questions, hybrid retrieval, both passes inside the same container:

	reranker off	reranker on
substring recall@1	0.758	0.909
substring recall@6	1.000	0.970
substring recall@8	1.000	0.970
substring MRR	0.853	0.935
file recall@1	0.424	0.576
file MRR	0.617	0.704
latency/query	0.06 s	3.27 s

The gain is at the very top of the ranking, which is what matters when only RETRIEVER_K chunks reach the model: six questions gain the first position and one loses it. It also repairs what fusion costs on literal lookups — easy_lookup MRR goes 0.900 keyword-only → 0.750 hybrid → 0.900 with the reranker, so the two mechanisms are complementary rather than competing.

The costs are real and worth stating. Latency goes up 55x, which is what puts /api/context in the 1.5-4s band. And in one question of 33 the cross-encoder demotes the only relevant chunk from position 5 to position 11: raising RETRIEVER_K from 6 to 8 does not recover it (recall@8 equals recall@6), so that one is paid for, not tuned away. Reranking only reorders the fetch_k candidates, so recall@20 is untouched at 1.000.

Tracing

Optional Langfuse tracing, off unless LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY are set. deploy/langfuse/ holds the self-hosted stack for the NUC and its README says what each trace records.

License

MIT