Grounded Q&A over approved employment-law sources, with abstention
An assistant that answers only from an approved set of sources and cites the passage it used. When the sources do not cover the question, it says so instead of guessing.
The problem
A question-answering assistant is the most requested kind of AI tool and the easiest to misuse. The failure that hurts is a fluent, confident answer with no basis in the material the team has approved. This prototype tests whether an assistant can be built to refuse exactly those answers, and how to measure that it does.
Constraints
- Public-domain material only for the prototype: Department of Labor wage-and-hour fact sheets, EEOC guidance, and one state’s leave statute, roughly 150 to 300 pages with section and date information.
- Every answer carries citations to passages the reader can open.
- A question outside the material gets a clear “not covered by the approved sources” instead of an improvised answer.
- The list of approved sources is managed on a login-protected admin page, with every change logged.
Planned design
- Ingest splits each document by section and keeps the source, the heading, the effective date and the page. It indexes the sections for keyword search (BM25) and for embedding search.
- Retrieval merges the two rankings. The top passages go to the model with the instruction to answer only from them and to cite each claim.
- An abstention check rejects any answer whose claims are not supported by the retrieved passages.
- A chat screen with a sources panel; each citation opens the passage.
- The admin page, where an approver adds and removes sources.
Evaluation plan
About 40 questions in three groups: 25 that the material answers, with the correct passage recorded for each; 10 that it does not answer or that are out of scope; and 5 adversarial ones, including questions built on a false premise and an instruction planted inside a document. Two retrieval configurations run against the same set.
| Measure | How it is scored |
|---|---|
| Answer correctness | Rubric graded, with a model judge checked against a hand-graded sample |
| Citation precision | Share of citations that support the claim they are attached to |
| Abstention accuracy | Correct refusals on unanswerable questions, and false refusals on answerable ones |
| Retrieval recall at k | Whether the correct passage is among the top k retrieved |
| Speed and cost | Median and 95th-percentile time per question; cost per question |
Human review points
Answers are advisory, and every claim is one click from its source. The admin page is the control point: only an approver can change what the assistant is allowed to read.
Known limitations going in
- Public guidance is not a firm’s own guidance. The obvious next corpus is internal, and ingest is built to accept it without changes.
- Effective dates matter in employment law. The first version shows them but does not reason about them.
Changelog
- v0.0, Oct 13, 2026. Plan published. Build starts.