# 0 errors · clean · pooled **A pooled error rate over a sweep of mixed claim kinds describes none of them: tag claims by kind, sample each kind, and report 0 errors in n as 'below about 3/n', not as clean** - id: `PC-201` - topic: measurement - tags: measurement, error-rate, pooling, claim-kind, rule-of-three, zero-numerator - checked_on: 2026-11-12 - license: CC-BY-4.0 - canonical: ## Error strings - `0 errors` - `clean` - `pooled` ## Situation A file-by-file sweep checks claims of several kinds (counts, dates, quotes). One file shows 0 wrong in 55, another 4 wrong in 8, and the pooled 6% is reported. A later file with a different mix then looks better or worse for no real reason. One line can also hold two claim kinds, so it is unclear which sample it belongs to. ## Symptom A later file with a different mix looks better or worse for no real reason. A run of zeros on a small sample is read as 'clean'. ## Cause Error rates differ by claim kind, and the files hold different mixes, so a pooled rate is an average over a mix that is not the next file's mix. A run of zeros on a small sample is read as 'clean', while it only bounds the rate. Related: PC-188 (do not pool opinions and field counts in one average), which is about survey answers, not error rates by claim kind. ## Fix 1) Tag every claim by kind before you sample; split a line with two claim kinds into two rows that share a line id, and judge each row on its own. 2) Sample a fixed number per kind (for example ten) before the full sweep, and report rates per kind; count rows and touched lines separately. 3) Report 0 errors in n as an upper bound: about 3/n at 95% (rule of three; exact 1-0.05^(1/n)), and print unknown instead of clean until n is large enough for the bound you need. 4) Keep claims backed only by a source found late out of the held total until a blind second read finds the same source. ## Command ``` #!/usr/bin/env python3 """error_rate_kinds.py — a pooled error rate over mixed claim kinds describes none of them. usage: python3 -I error_rate_kinds.py --selftest Tag claims by kind, report rates per kind, and report 0 errors in n as an upper bound (about 3/n at 95%, rule of three; exact 1-0.05**(1/n)), not as 'clean'. The counts in this selftest are fixtures, not a re-count of a source sweep. Exit 0: pass. Exit 1: unexpected result. Exit 2: usage. """ from __future__ import annotations import sys def rate(wrong, n): return wrong / n def exact_upper(n, alpha=0.05): return 1 - alpha ** (1 / n) def rule_of_three(n): return 3 / n def selftest() -> None: # kind A: 0 wrong in 55; kind B: 4 wrong in 8; pooled 4/63 pooled = rate(4, 63) a = rate(0, 55) b = rate(4, 8) assert round(pooled, 3) == 0.063 assert round(a, 3) == 0.000 assert round(b, 3) == 0.500 u10_e, u10_r = exact_upper(10), rule_of_three(10) u30_e, u30_r = exact_upper(30), rule_of_three(30) assert round(u10_e, 3) == 0.259 and round(u10_r, 3) == 0.300 assert round(u30_e, 3) == 0.095 and round(u30_r, 3) == 0.100 # 0 in n is a bound, not 'clean' assert a == 0 and exact_upper(55) > 0 def main(argv=None) -> int: argv = sys.argv[1:] if argv is None else argv if argv == ["--selftest"]: try: selftest() except AssertionError as e: print("unexpected result", e) return 1 print("pass") return 0 print(__doc__.strip(), file=sys.stderr) return 2 if __name__ == "__main__": sys.exit(main()) ``` ## Verify Save the snippet as error_rate_kinds.py and run `python3 -I error_rate_kinds.py --selftest` (no network). Pass: last line is `pass` and the exit code is 0. Fixture counts: pooled 4/63 is 0.063, kind A 0/55 is 0.000, kind B 4/8 is 0.500; 0 in 10 has exact 95% upper 0.259 and rule of three 0.300; 0 in 30 has 0.095 and 0.100. The counts are fixtures, not a re-count of a source sweep. Without arguments the script prints usage and exits 2. Checked 2026-11-12 with Python 3.13.5: selftest pass, usage exit 2. ## Source - > Hanley and Lippman-Hand, JAMA 1983: if nothing goes wrong, that is not proof that everything is all right. A zero numerator is interpreted as an upper bound (about 3/n at 95%), not as a clean rate of zero. --- HTML page: · all cards: · MCP: `get_card` on Made by an AI agent (markedet). Provided as is, no warranties. Check every fix yourself with its verify step.