The most useful answer a document chatbot can give: “I don't know”
Chat with your PDFs, Word files and Markdown. Every claim has a clickable citation.
Ask the handbook
A tiny fictional company handbook. Ask something it covers — and something it does not.
A small browser version with a deterministic fake model — no API key, nothing leaves your device. The real project is in the repo.
What it solves
A general chatbot answers confidently even when it has never seen your files. That is how wrong answers about your contract or policy appear.
How it works
- Hybrid search: vector search (pgvector) plus full-text, merged with Reciprocal Rank Fusion.
- Answers use only retrieved passages, each cited with file and section (plus page for PDFs).
- Retrieval is measured with hit-rate@k and MRR on 30 golden questions. The prompt tells the model to say “I don't know” when the sources don't cover the question, but that is not measured: none of the 30 questions is unanswerable.
The honest verdict
Whether re-ranking helps depends on the embedder. With the offline word-matching embedder, a cross-encoder lifted hit@1 on the 12 paraphrased questions from 4/12 to 7/12. With a real embedding model, plain hybrid search got 6/12 and the cross-encoder 7/12, and a cheap lexical re-ranker scored no worse (the gap is within noise). The set is only 30 questions over 3 short documents: a direction, not a benchmark.
Next demoDistillery →