EEOC charge intake: LLM extraction with source spans and human review

The problem

When a company is accused of discrimination, the first document that arrives is usually an EEOC charge: a form and a notice letter. Someone has to read both, type a dozen fields into a case system, work out the deadline for the response, and write a short summary of what is alleged. The work is routine, and a wrong date is expensive.

This prototype tests whether a language model can do the first pass well enough to be worth a pilot, and whether we can tell when it gets something wrong.

Constraints

• No real charges. The test set is made up: generated charge forms and notice letters with known correct answers, including deliberately hard cases (a missing charge number, a rotated scan, particulars that run to several pages, two respondents).
• The model never guesses. A field it cannot support with a passage from the document comes back empty, with a reason.
• Nothing leaves the tool until a person has accepted each field.

Planned design

1. A generator produces about 30 charge-and-notice PDF pairs, each with a file of correct answers.
2. An extraction service (FastAPI) converts each PDF to text with page positions, asks the model for a fixed set of fields in JSON, and attaches to every field a source span (page number and quoted text) and a confidence score.
3. A span check confirms that each quoted span appears in the document text. A field whose span cannot be found is emptied before a reviewer ever sees it.
4. A review screen (React) shows the PDF on the left and the fields on the right, with accept, edit and reject for each field. Export is disabled until every field has a decision.
5. A model adapter switches providers by configuration, so the same evaluation runs against two approved models.

Fields

Charge number, charging party, respondent, EEOC office, bases, issues, earliest and latest dates of harm, the continuing-action flag, the response due date from the notice, and a three-sentence summary of the particulars.

Evaluation plan

The results table at the top of this page fills in when the suite runs. For each of two models it will report:

Measure	How it is scored
Field-level accuracy	Exact match against the correct answers, per field and overall
Unsupported-output rate	Share of filled fields whose cited span is not in the document
Empty-field correctness	Precision and recall of empty fields where the correct answer is absent
Speed	Median and 95th-percentile time per document
Cost	Per document, at list prices on the day of the run

Each change between versions goes in the changelog below with its effect on these numbers.

Human review points

Every field. The reviewer sees the quoted evidence next to the value, and the export button stays disabled until each field is accepted, edited or rejected.

Security and logging

Secrets come from environment variables; the repository ships an example file and nothing else. Every run writes a structured log with a request id, the model, token counts, cost and the outcome for each field. The generated documents contain no real names.

Known limitations going in

• Generated charges are cleaner than real scans. OCR quality on real intake is the first thing a pilot would have to measure.
• The due date is calculated by a rule from the notice date, so it is only as good as the extracted notice date.

Changelog

• v0.0, Oct 13, 2026. Plan published. Build starts.