The problem
Organisations now have to answer detailed questions against overlapping AI regulations: the EU AI Act, GDPR, the NIST AI Risk Management Framework, and ISO/IEC 42001. Getting an answer right matters, but so does the cost of getting it.
If every question is sent to a large, expensive model, the bill grows fast, and most questions do not actually need that much power. I wanted to find out whether a system could decide, question by question, how much model to use, without giving up answer quality.
The approach
I built the retrieval layer from scratch rather than reaching for an off-the-shelf framework, so I understood every part of it and could defend every design choice. Retrieval works in three stages. A symbolic exact-match pass catches precise terms like article numbers. A weighted mix of BM25 keyword search and FAISS vector search finds the broader context. A cross-encoder then reranks the shortlist so the most relevant passages rise to the top.
On top of retrieval sits a rule-based router. It reads each question and assigns it to a cost tier: a smaller local model for routine questions, or a larger one when the question needs it. Both models run locally through Ollama, so there are no per-call API fees. The whole system runs behind a FastAPI service in Docker, with a Streamlit dashboard for watching cost and quality side by side.
How I measured it
I did not want to trust a single number. I built a gold evaluation set of 25 carefully annotated questions, then measured retrieval quality with context precision, scored answers with both an LLM-as-judge and RAGAS, and ran a counterfactual analysis comparing the router's choices against an oracle that always knows the cheapest correct tier.
The quality figure above 100 percent is not a typo. The routed system was slightly better than always using the expensive model on this set, because steering simpler questions to the smaller model avoided some overlong, meandering answers.
The router matched the oracle's exact choice about one time in five, which sounds low in isolation. Looking closer, the reason is simple: on this corpus the cheap tier was almost always the right call, so even a router that leans cheap lands close to optimal on both cost and quality. It is a good reminder that a headline accuracy number can hide what is really going on, and I would rather explain that than bury it.
What I would do differently
I would widen the evaluation set well beyond 25 questions so the routing signal is stronger and more general. I would add a learned router alongside the rule-based one and compare them fairly. And I would test the system on live, changing regulation text rather than a fixed snapshot, since real governance work never sits still.
Stack: Python, FastAPI, Docker, FAISS, BM25, cross-encoder reranking, Ollama (Llama 3 8B and phi3:mini), RAGAS, Streamlit. Supervised by Prof. Dr. Iftikhar Ahmed and Prof. Dr. Talha Ali Khan at the University of Europe for Applied Sciences.