Accuracy & methodology

Every decision we report is produced by the same four-stage pipeline. Each stage is measurable and each stage can fail, so we track them separately rather than quoting one blended number.

1. Ingest.We pull the official meeting recording published by the jurisdiction (Granicus, CivicPlus, YouTube, or the city’s own stream). The raw audio and any agenda/minutes PDFs are stored immutably so we can re-process later with a better model.
2. Transcribe. Audio is transcribed locally with whisper.cpp (large-v3). Speaker turns are diarized and, where the clerk names members, mapped to names.
3. Extract. A local large language model (Qwen, 27B) reads the transcript and pulls each discrete decision: type, motion, second, vote tally, affected parcels and zones, and a one-to-two sentence summary — each with a confidence score.
4. Gate. Extractions at or above our confidence threshold are delivered automatically. Everything below it is queued for human review before it can reach you.

Word Error Rate (WER) is the standard transcription metric: the share of words inserted, deleted, or substituted versus a human reference. On clean, single-speaker English, whisper large-v3 lands around 5–8% WER. Commission audio is not clean.

Real municipal recordings have gaveling, cross-talk, dais microphones of varying quality, and public-comment speakers at a far mic. On our Central Florida sample we expect WER in the 8–15% range, worse on public comment and better on the dais where the decisions actually happen. This matters less than the raw number suggests: a misspelled street name or a dropped filler word rarely changes the decision. We measure WER specifically on the decision-bearing segments (motion, vote, disposition), not the whole meeting, because that is the part you rely on.

The extraction model emits a calibrated confidence between 0.00 and 1.00 for every field of every decision. In practice, cleanly stated roll-call votes score high — typically 0.85–0.97. Voice votes, amended motions, consent-agenda bundles, and decisions where the chair’s wording is ambiguous score lower, often 0.45–0.70.

A confidence score is not a correctness guarantee — it is the model’s own estimate. Its value comes from being calibrated: across a labeled sample, decisions scored 0.90 should be correct about 90% of the time. Part of the QC pass below is checking that calibration and re-tuning where it drifts.

We auto-deliver a decision only when its confidence is 0.60 or higher. Anything below 0.60 is routed to a human-review queue and does not reach your inbox until a person confirms or corrects it.

Why 0.60 and not higher? Set the bar too high and we hold back correct decisions, making you slower than the newspaper — the exact failure we exist to prevent. Set it too low and we send you noise, which is worse, because one wrong alert in front of a client costs more than ten missed ones. 0.60 is our starting point, it is stored in configuration, and we tune it per decision type as the QC data comes in. The threshold, and the fact that a human clears everything beneath it, is the core of why you can trust an alert without re-watching the meeting yourself.

Precision over recall is a deliberate choice: when we are unsure, we do not fire. You should expect us to occasionally be an hour behind on a genuinely ambiguous vote. You should not expect us to tell you something false.

Before we publish headline accuracy numbers, we run a blind QC pass:

• A stratified sample of 200 decisions drawn across jurisdictions and decision types, so no single city or an easy decision type dominates the score.
• A human reviewer establishes ground truth for each from the official recording — independent of the model’s output.
• We score exact-match on decision type and vote tally, and field-level accuracy on parcels, zones, and mover/seconder.
• We report the results broken down by decision type and by jurisdiction, not just a single blended figure, and we publish the full 200-row dataset — our extraction next to the human ground truth — so you can audit us directly.

• We do not claim 100% coverage of every utterance in a meeting.
• We do not claim to replace the official record. Always confirm a decision against the jurisdiction’s minutes before relying on it in a matter.
• We do not hide low-confidence extractions from ourselves to make a number look better — they go to review, and they count in the QC score.