PoliMillionaire quiz QA agent: web-search RAG with a local LLM

I built the core of an agent that plays "Who wants to be a PoliMillionaire?", a multiple-choice quiz modelled on "Who Wants to Be a Millionaire?". When a question arrives it searches the web, selects what it needs with two-stage retrieval and reasons with an open-weights LLM running locally, in both text and speech modes. In the conclusions of the submitted notebook, the top prize reached in text mode is $1,000,000 in all six categories.

The project was carried out in a team with other colleagues for the Natural Language Processing course at Politecnico di Milano (academic year 2025/26). The submitted notebook is largely mine: web search, retrieval, reasoning with the model, game loops and experiments. Speech transcription and the maths pipeline were built by my colleagues, and I describe them only for context.

Context

The assignment, released on 15 April 2026 and due on 2 June, asked for a chatbot able to play the course's online quiz, reachable through an API with a Python client provided by the lecturers. The quiz has six categories (Entertainment, Ancient History and Politics, Science and Nature, Maths, Philosophy and Psychology, News). Each has 15 levels, with one question and four options per level, and there are 30 seconds to answer: a game ends at the first wrong or late answer.

There are two ways to play. In text mode, question and options arrive as text. In speech mode they arrive as audio files that must be transcribed before answering; here the 30-second timer starts only when the last option is requested, a detail that shaped the architecture.

The main constraints:

• models must run locally, so only open-weights models and no LLM APIs;
• RAG is encouraged, including through a real-time search API, as long as it is free and returns raw content (HTML pages, PDFs) rather than generated answers;
• agentic techniques, such as tool calling or combining several models, are encouraged.

The system in brief

For each question a router looks at the category. Maths goes to a dedicated solver, a specialised model that reasons without searching the web; every other category goes to web-search RAG, which is my part. In speech mode, before all this, the question and the options are transcribed with Whisper, with a small text-cleaning step. The maths solver and the transcription were built by my colleagues.

My part

Web search

The search starts from the question text alone: I remove stop-words and punctuation and run two queries in parallel, a broad one and one restricted to the category's trusted domains (for example encyclopedias and film databases for Entertainment, news agencies for News). The options stay out of the search: they are not needed, and this way in speech mode the search can start before they arrive (see below).

The results are merged without duplicates, I drop domains that are poor sources of text (video, social media, Q&A sites) and put trusted domains first. The selected pages are downloaded in parallel, each with a time limit, and cleaned with trafilatura; if no download succeeds, the pipeline falls back on the search result snippets. Each document is cut to its opening part, where the answer usually is.

Two-stage retrieval

Documents are split into character windows with overlap. A bi-encoder (all-MiniLM-L6-v2) compares the question with every chunk and keeps 40; a cross-encoder (ms-marco-MiniLM-L-6-v2) scores each question-chunk pair and reranks them. An adaptive threshold on the cross-encoder score drops the chunks that are not relevant enough, and the best of those that remain reach the model. If none passes the threshold the pipeline still answers, without context, relying on the model's own knowledge. Here too the query is the question alone, so the cross-encoder rewards relevance to the topic rather than lexical similarity with the options.

Model and prompt

The model is Qwen2.5-7B-Instruct, loaded once when the notebook starts in a 4-bit version (NF4, with bitsandbytes), on the Colab T4 GPU. Generation is deterministic and short, and the assistant turn starts with a reasoning header, which pushes the model to analyse the options before choosing.

The prompt is an answer contract. Each option keeps the id assigned by the server; the model writes one line per option, saying whether it supports or contradicts it and why, then chooses and closes with an ANSWER line carrying the id and text copied from the list. Each category has a persona and a solved example that show the format. The retrieved context enters the message with the source titles; when it is missing, the prompt tells the model to use its own knowledge.

A small model often picks the right text but the wrong number, for example by counting from 1. The parser therefore trusts the option text over the printed id; if it cannot resolve the text, it looks for a single option quoted in the output and, as a last resort, falls back on the lowest id.

Timing and speech mode

The server allows 30 seconds; I set an internal budget of 25 as a margin, and for every answer the notebook logs the time of each stage (search, retrieval, inference) and compares the total with the budget.

The pipeline is split in two halves precisely for speech mode. search_documents does search and download and does not need the options; answer_from_docs does retrieval, inference and parsing. In speech mode the timer starts only when the last option is requested, so the question is transcribed and the search starts before the options are requested: the slow part happens off the clock, and only the rest runs against it. In text mode the two halves simply run one after the other.

Game loops and experiments

A small class wraps the course client and normalises questions and options into plain Python structures. On top of it are the game loops for the two modes and run_experiment, which plays a chosen number of games in a category and mode and returns a table with one row per game (level reached, mean time per stage). Every configuration parameter can be changed for a single run without editing the notebook: overrides are applied and then restored, and an unknown name raises an error. Two plots summarise a series: the distribution of the maximum level reached and the mean time per game.

Results

The table gives the best result per category and mode, from the conclusions of the submitted notebook.

These are best results, not averages: a game ends at the first mistake or the first late answer, so they should not be read as accuracy. In speech mode the result is lower in four categories out of six; among the improvements listed in the notebook's conclusions is correcting entities that were transcribed badly.

Limitations

• Best results, not averages. The table gives the best result per category and mode, not how often it is reached.
• Simple search. The queries are the question stripped of stop-words, with no LLM rewriting, and the chunks are character windows that can cut words and sentences.
• Silent failures. If search or download fails, the pipeline does not stop: the model answers from its own knowledge alone, and the result does not show it.
• Speech. Answers depend on the transcription, and spoken proper names and technical terms can be transcribed badly before they even reach the model.
• Budget measured, not enforced. The 25-second budget is a reference the logged times are compared against; no stage is cut short.

Possible improvements

The notebook's conclusions list six ideas:

• first ask the LLM whether it needs any information to answer;
• in speech mode, use an LLM to check whether the transcribed entity makes sense in the context of the question, and correct it;
• split documents by sentences or words instead of characters;
• run a grid search for the best RAG parameters;
• use a search API or RSS feeds from known, trusted news sources;
• add an agentic chain in which a first agent optimises the search query.

What I learned

• Time drives the architecture. In the quiz what counts is answering well inside a narrow window, and knowing where the timer starts decided how to split the pipeline in two halves.
• With a small model, format matters as much as content. A clear answer contract and a parser that trusts the option text keep the model's choice readable even when it gets the number wrong.
• One place for configuration. Keeping every parameter in one module and being able to change it per run made experiments quick to repeat.

The code belongs to the team and is not public: the information on this page comes from the notebook submitted to the lecturer.