The evidence behind /audit
50 real patient-education pages from heart.org, cancer.org, and stroke.org, fetched, stripped of markdown contamination, and measured by the same lib/readability engine that powers every grade on this site, then levelled to target grade 6 with the exact frozen loop this site ships. This is run 3 of three -- the committed eval/realWorld/pages_v2.json / results_v2_stripped.json results, generated 2026-08-17. Runs 1 and 2 are below, for comparison, not omitted.
The JMIR column below is reference, not a replication claim. Will et al. (JMIR 2025;27:e69955) scored a different 60-page sample, sampled roughly two years earlier, with a different closed-source tool (Readable.com), using an extraction method their own paper doesn't document. An exact match to that figure was never a meaningful target. What this table actually establishes is independent of it: every organisation's mean sits above the AMA/NIH grade-6 recommendation for patient material, measured by an engine cross-validated to 0.2237 mean absolute disagreement against textstat (eval/results/readability_validation.json). Two independent measurements landing on the same conclusion -- both above grade 6 -- is the claim; a decimal match to either figure never was. Full writeup, including the corpus's three-run correction history, in eval/realWorld/README.md.
Per-organization summary
| Org | n | Raw mean | Stripped mean | Stripped median | Stripped SD | JMIR mean | Stripped delta |
|---|---|---|---|---|---|---|---|
| heart.org | 17 | 15.54 | 8.03 | 7.75 | 1.46 | 10.7 | -2.67 |
| cancer.org | 17 | 23.34 | 9.28 | 9.11 | 1.26 | 10.0 | -0.72 |
| stroke.org | 16 | 16.73 | 8.75 | 7.96 | 2.43 | 9.6 | -0.85 |
| Whole corpus | 50 | 18.57 | 8.69 | 8.60 | 1.85 | — | — |
This table shows the stripped methodology -- markdown link syntax and heading markers removed on top of the earlier navigation/widget cleanup -- which is what run 3 actually levelled. Every organisation now sits slightly under JMIR's published mean rather than over it, consistent with the direction the correction predicted, not a sign the two engines agree: different tool, different pages, roughly two years apart. Full correction history, including the two earlier (overshooting) methodologies this table replaced, in eval/realWorld/README.md.
Levelling result -- run 3, target grade 6
Every page above was then run through the exact frozen verify-refine loop this site ships (aim-off, damped gain 0.6, tolerance ±0.5, three-attempt cap, same model and temperature as /tool), aimed at grade 6.
| Org | n | Source median | Final median | MAE | Within ±0.5 | Guard skips |
|---|---|---|---|---|---|---|
| heart.org | 17 | 7.75 | 6.24 | 0.774 | 9/17 (53%) | 3 |
| cancer.org | 17 | 9.11 | 7.08 | 1.378 | 4/17 (24%) | 0 |
| stroke.org | 16 | 7.96 | 6.20 | 0.671 | 10/16 (63%) | 3 |
| Whole corpus | 50 | 8.60 | 6.43 | 0.947 | 23/50 (46%) | 6 |
Within ±1.0: 30/50 (60%). Within ±1.5: 38/50 (76%). Attempts distribution: 0 (guard skip) 6 · 1 attempt 14 · 2 attempts 1 · 3 attempts 29. Truncated attempts: 0. Total tokens: 191,188 in / 154,608 out = 345,796. Verification itself cost nothing across every run below, because it is arithmetic, not a model call.
Three runs, one real defect fixed each
The number below did not move because anything was tuned -- each run found and fixed one real production defect. Full diagnosis of both in eval/realWorld/README.md.
| Run | State | MAE | Within ±0.5 |
|---|---|---|---|
| 1 | Contaminated source, 4096-token truncation | 2.45 | 15/50 |
| 2 | Contaminated source, truncation fixed | 1.852 | 18/50 |
| 3 | Stripped source, truncation fixed | 0.947 | 23/50 |
Run 1's defect: an undocumented 4096-token output cap silently truncated 11 of 50 pages mid-sentence, producing measured grades as high as 38. Run 2's defect: markdown contamination surviving extraction inflated measured source grades by roughly 2.8 grade levels on average, fixed for run 3 by stripping inline link syntax and heading markers. Both were found by inspecting the data, not by watching the score.
Do the readability formulas agree?
Across 49 pages measured live, FK/SMOG/Gunning Fog spread by 2.0+ grade levels on 42/49 (86%) of pages — and on 49/49 (100%) of them, SMOG and Gunning Fog both read the page as harder than FK did. Not scattered disagreement: FK is the most permissive of the three on every page we measured. A grade-6 audit run on Flesch-Kincaid alone is a weaker check than most auditors assume. Full finding in DOCUMENTATION.md §4b.
Spread, per organization
Each dot is one page's stripped source grade. The solid vertical line is the mean, the dashed line the median, the horizontal bar spans one standard deviation either side of the mean, and the red line is JMIR's published mean for that organization.