RAG Codebase
A code-aware retrieval-augmented generation (RAG) system for asking natural-language questions about a codebase and getting grounded, citation-backed answers. It chunks source on AST boundaries with tree-sitter (not character windows), retrieves with hybrid dense + BM25 search fused by Reciprocal Rank Fusion (with an optional cross-encoder and the measured-best listwise LLM reranker), builds a code graph (imports / calls / containment) with personalized PageRank for connected context, answers with Claude using [n] citations plus an LLM-as-judge faithfulness check, and — the centerpiece — ships a rigorous evaluation harness (recall@k / MRR / NDCG, bootstrap confidence intervals, paired significance tests). Provider-agnostic (Anthropic or any OpenAI-compatible endpoint, incl. local Ollama); semantic search via sentence-transformers embeddings with an optional HNSW (approximate nearest neighbor) backend for scale. New to RAG / embeddings / the eval methodology? Start with docs/LEARN.md, a from-scratch walkthrough. Design rationale lives in outline.md; the full measured write-up is RESULTS.md. language specs (Python, JS, TS, Go, Rust, Ruby, Java, C/C++, C, PHP, Kotlin, Scala, Swift, Lua, Bash, Perl, Objective-C) + a generic pattern classifier, with line-window fallback. by Reciprocal Rank Fusion; an optional cross-encoder and a validated listwise LLM reranker refine the pool (the cross-encoder is off by default — measured net-negative on code, §3d). resolution; personalized-PageRank traversal for connected, token-cheap context. don't cover this"), and a RAGAS-style faithfulness check that verifies each claim against its cited source. endpoint (OpenRouter, Together, Groq, Azure, local Ollama / LM Studio / vLLM). confidence intervals, paired-bootstrap significance, a holdout split, and ablations. (full retriever), so quality is validated off the self-made golden set. lever — it significantly sharpens ranking (MRR/NDCG, p<0.001) and lifts answer-correctness, with a setup-dependent recall gain. brute-force matmul is the default. hierarchical layouts) plus a live localhost server with persistent edits + reset. Cursor, Claude Desktop). (retry/backoff + timeouts), and secret hygiene (gitignored dotenv + secret scanner, CI). The pipeline is digest once, probe per query. At index time: discover files → AST-chunk → embed (header + code) into a vector store → tokenize into a BM25 index → build the code graph → persist. At query time: embed the question → dense + BM25 search → RRF fuse → (rerank) → grounded answer with citations → faithfulness verification. Design decisions, each empirically motivated: retrieves as noise. Chunks land on definition boundaries: methods are their own chunks, classes get a summary chunk, oversized functions are windowed carrying their signature, and a module chunk captures imports + top-level code. A chunk's id is its file:line location, so the retrieved unit is already its own citation. / docstring — the embedding then carries location and signature, exactly what code queries key off. (The header is embedded, not displayed.) embeddings smear together; BM25 nails them, aided by a code-aware tokenizer that splits getcurrentuser / getCurrentUser. Dense and lexical lists are fused by Reciprocal Rank Fusion (rank-based, so no score calibration). A cross-encoder reranker is wired in but off by default (measured net-negative on code retrieval — §3d); the listwise LLM reranker (--accurate) is the validated accuracy lever. connected chunks (callees, callers, imports, enclosing class) via a dictionary lookup, so the model gets a few precise neighbors plus a compact structural map instead of whole files. into atomic claims and checks each against its cited source (faithfulness = supported / total), backed by a free structural check that every [n] resolves to a real source. contribution, reported with confidence intervals and paired significance rather than vibes.