SABITH S B All projects
Live demoOpen source · MITPersonal project2026

OKFBench Chat

An open-source prototype that asks one question: how does the way knowledge is organised change an AI answer? Ask the same thing through three retrieval paths and inspect exactly what the model was given.

Python · FastAPI · SQLAlchemy · PostgreSQL · BM25 · OpenAI-compatible LLM API · vanilla JS

Role
Solo build
Status
Live demo
Built
Oct 2026
Code
~550 lines of Python

Same question, three retrieval paths

Most RAG demos hide the interesting part. OKFBench Chat puts it first: pick a retrieval mode, ask a question, and every run shows the answer with citations, the sources that were retrieved, how long retrieval and generation each took, and the token usage reported by the model provider.

The knowledge base describes Infant Guard, a fictional newborn-care product, written twice: once as linked concept pages and once as longer prose documents. That lets the same domain be searched in structurally different ways.

OKFBench Chat in hybrid mode: a cited answer about the seven modules on the left; run metrics and retrieved concept and document sources on the right.
Hybrid mode: cited answer, run metrics and retrieved sources.

Try it

Ask “How does an approved nutrition plan reach parents?” in each mode and compare the sources before the timings. The free hosting tier sleeps when idle, so the first request can take a little while.

Live demo
okfbench-chat.onrender.com
Open the app
GitHub · MIT
github.com/s4b11/okfbench
View repository

There's also a 73-second narrated walkthrough in my LinkedIn post.

How the three modes work

All three modes share the same source limit (five by default) and the same answer model, so the only thing that changes between runs is how the context was found.

Mode 1
OKF linked concepts

Keyword-ranks concept pages to pick up to two seeds, then follows their Markdown links breadth-first, up to two hops, pulling in related concepts the question never named.

Mode 2
BM25 keyword search

A from-scratch Okapi BM25 index (k1 = 1.5, b = 0.75) over the prose corpus, ranking whole documents.

Mode 3
Hybrid

Interleaves OKF concepts and BM25 documents under the same source budget, and labels each block so the model knows which is which.

  1. QuestionBrowser posts to /api/chat
  2. RetrieveOKF, BM25 or hybrid
  3. GroundContext block with source IDs
  4. GenerateLLM answers from context only
  5. RecordTimings, tokens, sources stored per session

Implementation

  • OKF loader: parses YAML front-matter and resolves relative Markdown links into a concept graph, with cycle-safe traversal and a configurable hop depth.
  • Grounded prompting: the model is told to answer only from the supplied context, say what is missing, and cite concept or document IDs in brackets.
  • Provider-agnostic LLM layer: any OpenAI-compatible endpoint works. With no API key it falls back to an offline preview, so retrieval can still be explored.
  • Run history: latency split, token counts and sources are stored per anonymous session; SQLite locally, PostgreSQL in production.
  • Tests: eight regression tests cover link resolution, hop-depth and cycle limits, BM25 ranking, provider defaults, the session API and error handling.

What it is not

OKF is a knowledge format; RAG is a retrieval-and-generation pipeline. They aren't competitors, so the project is an explorer, not a leaderboard:

  • No accuracy claims. The two knowledge versions cover the same domain but aren't fact-identical, so runs are examples to inspect, not a controlled benchmark.
  • Scores stay local. Each retriever's score only means something within that retriever; it's never shown as confidence.
  • Timing is scoped. Reported latency covers retrieval and generation only, not database writes or rendering.
  • No dense embeddings yet. The document baseline is BM25; embedding retrieval is the planned addition.