Decision model that scores options instead of generating text

Problem

An autoregressive LLM can hallucinate because it decodes tokens freely. For choosing among known options that is a defect, not a feature: you do not need new text, you need the right choice and an honest confidence in it.

Solution

The solver never generates. It takes the context and the options, emits one score per option and applies a softmax across that set. With no fixed class count, the same model handles 2, 4 or 7 options.

The dataset builds itself: a generator LLM produces multiple-choice problems and a critic LLM validates them before they are persisted. Solver failures are grouped into weaknesses, and the generator receives requests targeting them, prioritising confusion pairs as the most actionable kind.

The LLMs live only in the data factory. There is none on the decision path.

Architecture

By default only the scoring head trains, with a frozen encoder; LoRA is a config switch. Temperature is fitted after training so the probabilities mean something.

Technical decisions

• Strict separation between data generation and decision.
• Post-training temperature calibration: a confident wrong model is worse than a hesitant one.
• SQLite persistence behind its own repository, so you can audit which problems went in and which ones failed.
• Swappable LLM clients (Ollama HTTP or the Claude CLI) behind a shared protocol, no agent frameworks.

Result

Working MVP with a closed training loop and targeted generation, covered by tests over solver, repository, metrics and calibration.