Analysis-Page Accuracy Audit

How it's built: Every crawled part's marks are split into [AO1,AO2,AO3] by ao_split(mk,tier), which multiplies the part's marks by an OCR-specimen-calibrated rate keyed on (tier, mark-band 1..6+) from ao_calibration.json. The SAME OCR rates are applied to ALL boards (board-agnostic). renderAO sums the per-part AO marks filtered by the selected board+tier, so switching board only changes which papers are included, not the AO rate — the displayed mix barely moves across boards (only via each board's question-size distribution).

Accuracy: VERIFIED against AQA official grid + Ofqual mandate: GCSE Maths AO weightings are regulation-fixed and IDENTICAL across boards (F 50/25/25, H 40/30/30). AQA official ao_marks land EXACTLY on that (F 50/25/25, H 40/30/30). The site OCR-calibrated model already produces ~F 50/25/25, H 40/30/30 = correct. The apparent board bias was an ARTIFACT of unreliable QLA sources (pinpoint over-tags AO1: Edexcel 58/23/19, even AQA reads 51/30/19 under it vs its own grid 40/30/30; single ao-labels over-tag AO3). Our data was wildly out, not the boards.

Upgrade: NO CHANGE to the calibration. Optional honesty touch: name the Ofqual-mandated target (F 50/25/25, H 40/30/30) beside the estimate so the card is self-evidently calibrated. Do NOT swap to pinpoint/QLA ao-labels — it would regress accuracy. [Note: use AQA official ao_marks, NOT pinpoint AO labels — pinpoint over-tags AO1.]

How it's built: Sums the SAME OCR calibration over just the selected topic's parts (aoSplit per part, divided by n_topics for synoptic splits). The caption itself admits it is 'not topic-specific' — it only reflects how big/which-tier that topic's questions tend to be, carrying ZERO real per-topic AO information.

Accuracy: SHIPPED this session. Was non-topic-specific (scored 44). Now calibrated from AQA official per-question ao_marks by strand x mark-tariff and applied per topic: statistics shows AO2-dominant (averages 18/71/11), number AO1-heavy (indices 61/18/21), problem-solving topics AO3-heavy (percentages 38/15/48, circle-theorems 31/28/41). AQA-only (pinpoint discarded). Limitation: strand-grain, not individual-topic (AQA grid too sparse per topic: 1393 labels / 2881 parts), so same-strand topics with similar question sizes look alike; honestly labelled AQA-calibrated.

Upgrade: DONE (live). Future: a hand crossmap of AQA high-volume labels to our slugs could lift some topics to true topic-grain AO where evidence supports it.

How it's built: The chart itself is real crawl marks-per-sitting (accurate). The AO/difficulty tier axis rides on ao_split's tier band, so the model DOES shift toward more AO1 at Foundation — directionally correct (real: F is ~15-20pt more AO1 than H on every board).

Accuracy: SHIPPED. Real facility (avg % students score) now on the teacher page: a How-hard-is-this-topic card (78 topics, AQA both tiers + Edexcel H, F/H split) + a Hardest-topics-that-come-up-a-lot ranking (freq x difficulty). First empirical difficulty signal on the site.

Upgrade: DONE (live).

How it's built: Ranks built topics by marks-per-paper (desc), cumulates marks as % of the board's paper max, then for each grade finds how many topics down the list the cumulative % first reaches that grade's boundary (grade-boundaries.json avg_pct). Pure cumulative-MARKS proxy: it assumes a student masters topics top-down by frequency AND banks 100% of every 'reached' topic's marks, with zero difficulty weighting.

Accuracy: The two modelling assumptions are both empirically false, and the new data proves it. Edexcel facility_by_grade (535 parts, 24 papers, Higher only) gives each part's expected marks by grade; aggregated per topic it shows within-topic gettability swings enormously by grade and topic: 'Primes/factors' G4 71% -> G7 89% (low discrimination, easy for all), but 'Vectors' G4 7% -> G7 42% -> G9 81% and 'Circle theorems' G4 7% -> G7 45% (steep discriminators). The proxy treats both as fully banked once 'reached'. Whole-sample expected-% by grade (G4 26 / G5 39 / G7 65 / G9 90) tracks real Edexcel-H boundaries, confirming facility_by_grade is the right instrument. The card is an honest, useful revision heuristic (captioned 'marks in reach') but it is NOT real grade-discrimination. | SHIPPED an additive student panel Which-topics-separate-the-grades from real Edexcel-Higher facility_by_grade (56 topics), honestly labelled Edexcel-Higher; the cross-board marks-proxy card is unchanged.

Upgrade: DONE (additive Edexcel-Higher panel live). Full cross-board replacement still needs OCR/AQA grade data (not published).

How it's built: Neither page surfaces any empirical difficulty. cross_board_qla has real % correct (facility) on AQA F (1,555), AQA H (1,326) and Edexcel H (535) parts — 3,416 parts of genuine student-performance data. Aggregated to our slugs (same label->slug map as the per-topic AO upgrade) this yields a real 'how hard is this topic' measure and a hardest/easiest ranking.

Accuracy: SHIPPED. Real facility (avg % students score) now on the teacher page: a How-hard-is-this-topic card (78 topics, AQA both tiers + Edexcel H, F/H split) + a Hardest-topics-that-come-up-a-lot ranking (freq x difficulty). First empirical difficulty signal on the site.

Upgrade: DONE (live).

How it's built: Each topic slug is mapped to one of 6 spec strands by a regex classifier (word-anchored token rules + a 20-slug override table). Per-strand marks = sum of each part's marks/n_topics over filtered papers.

Accuracy: QLA ocr-grid (OCR's OWN 6-strand tagging, 6000 marks) is the ideal independent oracle. Site OCR strand split matches it within 1.5pp on every strand (mean abs diff 0.6pp): Algebra 25.2 vs 26.1, G&M 20.8 vs 19.3, R&P 19.8 vs 20.6, Number 18.4 vs 18.6, Prob 7.7 vs 7.3, Stats 8.1 vs 8.1. The only ~1.5pp gap (G&M over / R&P under) is a compound-measures/mensuration boundary call. Classifier is essentially correct.

Upgrade: Add QLA ocr-grid strand %s as a printed benchmark row beneath the chart ('OCR official: Algebra 26%, ...') so teachers see the classifier is validated; optionally nudge the compound-measure/area boundary rules to close the 1.5pp G&M/R&P gap.

How it's built: Per-topic marks summed per sitting (series = year+Jun/Nov), one line per board, from the 246-crawl only. QLA not used.

Accuracy: The underlying per-topic marks are the same crawl figures that match QLA marks-share within ~0.5pp (13/14 topics), so levels are trustworthy. But per-topic-per-sitting is a thin basis (often 0-3 marks/sitting) and jumps are noise. QLA's grids independently span 2017-2022 (OCR), 2017-2025 (AQA), 2022-2025 (Edexcel) and could corroborate/extend the series but are not wired in.

Upgrade: Ingest the QLA grid rows (they carry board+year+marks) to (a) extend the series to 2025 for AQA/Edexcel and (b) cross-validate a few high-volume topic trajectories. Structural because it needs a QLA->slug mapping + new ingestion path.

How it's built: OLS regression of a topic's YEARLY average marks-per-paper (absent years count 0) against year; only topics with >=12 total marks and >=6 sittings are ranked; +/-0.005 mk/paper/yr threshold.

Accuracy: Methodology is honest (n<6 excluded, framed as past frequency not prediction) and the crawl basis (8 years x 2 sittings x 3 boards x 2 tiers) is solid at the corpus level. But a per-topic yearly-marks-per-paper regression over ~8 points with 0-fill is inherently low-signal, and QLA offers only a weak independent trend check: pinpoint has 6,671 null-year rows and is a question-bank (not complete papers), so appears-per-sitting is ill-defined; only the grid sources are per-paper-complete.

Upgrade: Merge the QLA official grids (per-paper, per-year) into the trend fit to double the sittings behind each slope and extend to 2025; surface a confidence flag when the fit rests on <8 points. Structural (new data blend + UI note).

How it's built: For the selected topic, avg marks/paper per board = (topic marks on that board's papers) / (that board's paper count in view).

Accuracy: Weakest-corroborated Cluster-2 section. With ~82 papers/board a topic appears 0-2x per paper so per-board averages are noisy. Against QLA pinpoint per-board emphasis, 3/5 spot-checked topics agree on ordering (standard-form, histograms both OCR-highest; vectors AQA-lowest) but 2/5 flip: site says OCR leans into circle-theorems & trigonometry where QLA says Edexcel and AQA respectively. So the direction is often right but the specific 'which board' call is unreliable at low counts.

Upgrade: Blend QLA per-board topic marks with the crawl to stabilise the estimate, and grey/withhold the emphasis chart when a topic has <~10 marks on a board (mirror the existing n<6 trend gate). Structural (data blend + gating rule).

How it's built: parse_calc() reads the first 4 lines of each index file (the real paper front-page header) for 'non-calc'/'calculator'. Per-topic split = topic marks on calc vs non-calc papers.

Accuracy: Classification is effectively ground-truth (read off the actual paper title): AQA/Edexcel P1&P4=Non-calc, P2/P3/P5/P6=Calc; OCR P2&P5=Non-calc (OCR's convention differs from AQA/Edexcel and is handled correctly precisely because it's read from the header, not assumed). QLA cannot improve on this: it has NO explicit calc field, only the paper number, which corroborates the AQA/Edexcel P1=non-calc convention but cannot independently confirm OCR's non-standard P2/P5.

Upgrade: Optional belt-and-braces: assert parse_calc() against the QLA-implied convention (paper 1/4 = non-calc for AQA/Edexcel) as a CI check to catch a future header-parse regression. Low value since headers are authoritative.

How it's built: For the selected topic, histogram of parts by mark value (1,2,3,4,5,6+) over filtered papers.

Accuracy: QLA's per-part marks give a genuine INDEPENDENT check of the size methodology. Overall site distribution 33.2/29.1/20.7/11.1/4.5/1.4% vs QLA grid-only 31.4/26.5/21.8/12.7/5.4/2.2% — same shape, max 2.7pp gap at the 2-mark band. Site skews slightly to smaller items, consistent with the crawl splitting lettered sub-parts more granularly than the QLA grids aggregate. Method is sound.

Upgrade: Fold QLA grid parts into the size histogram (or show it as an overlay) to thicken thin per-topic distributions and reconcile the sub-part-splitting skew. Safe-win once a QLA->slug map exists.

How it's built: Per topic: distinct sittings it appeared in (>=5 required), most-recent sitting, and gap = number of filtered sittings since; ranked by gap x appearance-rate. Framed as frequency, not prediction.

Accuracy: Sound frequency logic and honestly captioned ('reflects past frequency, not a prediction', per the method note line 625). Crawl-only: QLA adds little independent value here because pinpoint isn't complete-paper data (can't define appears-in-this-sitting) and the grids don't fully overlap the crawl's sittings. Robustness rests on the 246-paper crawl's completeness, which the strand/marks cross-checks confirm is representative.

Upgrade: None needed for accuracy; if extending the corpus to 2025 via QLA grids, the 'last seen' would refresh automatically. Keep the frequency framing.

How it's built: topicMarks = sum of marks/n_topics over filtered papers; ranked descending, top 10 (table) / top 18 (bar).

Accuracy: Directly corroborated by QLA: the crawl's per-topic marks-share matches QLA pinpoint share within ~0.5pp for 13/14 cleanly-mapped topics (percentages 5.87 vs 6.08, ratio 4.89 vs ~5.6, standard-form 1.52 vs 1.49, vectors 1.14 vs 1.15, circle-theorems 1.02 vs 1.01, trig 1.41 vs 1.60). The marks-based ranking is trustworthy; synoptic marks-splitting (marks/n_topics) is a defensible, clearly-disclosed choice.

Upgrade: Optionally show the QLA-share as a confirming second column ('independently: ~6% of marks') to signal validation. Safe-win.

How it's built: Built topics ranked by tmarks (board+tier filtered), top 20; bar width = %of top topic; 'about X marks a paper' = tmarks/papers-in-view.

Accuracy: Same marks basis as the teacher Top-10, corroborated by QLA to ~0.5pp per topic (see cross-check). Restricting to built topics is a product choice, not an accuracy issue. marks/paper (mpp) is a faithful frequency signal.

Upgrade: None for accuracy. Could annotate the marks/paper with the QLA-confirmed share. Safe-win, low priority.

How it's built: mpp = tmarks/np; ratio = distinct-papers-topic-appears-in / np; freq = 'in nearly every paper' if ratio>=0.83 else 'in about round(3*ratio) of every 3 papers'.

Accuracy: The marks/paper half is QLA-corroborated (share matches to ~0.5pp). The absolute-frequency half (appears-in-X-of-papers) is crawl-only and QLA can't independently validate it (pinpoint isn't complete papers; grids don't cover all sittings). The round(3*ratio) bucketing is coarse (only 1/2/3-of-3 buckets) and can over/understate a mid-range topic, but is honest and easy to read. No accuracy error found.

Upgrade: Consider a finer phrasing (e.g. 'about 4 in 5 papers') or a paper-count denominator note; and if QLA grids are merged, recompute appearance-rate on the larger corpus. Structural (copy/UX + optional data merge).

How it's built: Each crawl part's raw tag(s) (index_*.md col 3) are normalised by topic_norm.canon (alias table) then re-attributed at archetype grain by remap_arch, marks split evenly across multi-topic parts. Frequencies drive Top-10, topic ranking, rising/falling, single-paper, pairings. Self-validation = 27-paper blind gold retag: 83.0% eff-agreement, precision 85.9%, recall 87.6% (F1 86.8). I EXTENDED this by mapping the top pinpoint labels (hand-map covering 51.4% of 48,118 pinpoint marks; 24,728 mapped) to canonical slugs and comparing SHARE-of-marks pinpoint-vs-crawl across all 3 boards' full QLA history (vs the crawl's 246 papers / 9,336 parts).

Accuracy: Broad agreement is strong: Pearson r=0.808 and Spearman rho=0.694 across 69 mapped topics' shares; 57% within 1.5x. But real attribution GAPS surface where pinpoint says a skill is common yet the 246-paper crawl is thin (share-ratio pp/crawl): systematic-listing 6.4x (pp 1.14% vs cr 0.18%, only 38 crawl marks — biggest gap; pinpoint has dedicated 'Counting Methods'/'Simple Systematic Listing' labels), exponential-growth-decay/compound-interest 2.85x (pp 535 vs cr 162), relative-frequency 2.82x, congruence-criteria 2.20x, completing-the-square 2.17x, iteration 1.97x, graphical-inequalities 1.88x, frequency-trees 1.86x, then Higher-heavy bounds/box-plots/histograms/vectors ~1.5-1.6x. These are partly genuine crawl under-representation (small, older 2017-24 sample vs pinpoint's larger corpus) and partly attribution routing. Gold's own error taxonomy corroborates: 107 disjoint mis-tags + 40 crawl-overtag + 20 crawl-undertag out of 176 mismatches. Score reflects: attribution is directionally trustworthy and self-honest (published 83% + verified-paper flags) but the 246-paper base is thin for ~10 topics that pinpoint proves are exam-common.

Upgrade: Blend pinpoint's cross-board full-history frequencies into the topic ranking as a second, larger-sample column ('per real paper across all years' vs 'in our 246-paper crawl'), reconciled through topic_norm.canon so a topic still means one thing. Concretely: build a pinpoint->canonical crossmap (extend ALIASES with a Title-Case->slug layer), then show both counts + a divergence flag on any topic where the two disagree >1.5x. Priority re-tag/re-crawl targets from the divergence: systematic-listing, exponential-growth-decay, relative-frequency, completing-the-square, iteration, frequency-trees.

How it's built: Per topic, the crawl's own archetype text (index col 3) is aggregated verbatim with a count, per-tier/per-board split, marks and a mark-scheme note; top 40 shown, sorted by marks. NO classifier is applied here — it surfaces the crawlers' controlled-vocab phrasing directly, so it is high-fidelity BY CONSTRUCTION (only as good/bad as the crawl tags, which the gold layer scores at 83%).

Accuracy: This is the strongest section: it is a faithful passthrough of MS-checked crawl archetypes, no lossy bucketing. Ceiling is the crawl's own tag accuracy (gold 83% eff-agreement, precision 85.9%). Cannot be cross-validated against QLA extra.description at scale because that field is OCR-grid-only (2,462 rows) and pinpoint's fine labels are topic-grain not archetype-grain. Main limitation: thin per-topic samples for the ~10 crawl-under-represented topics identified above (e.g. systematic-listing archetypes rest on 38 marks).

Upgrade: For low-count topics, backfill archetype breadth from pinpoint's fine labels (they ARE effectively archetypes, e.g. 'Reverse Percentage', 'Missing Mean Questions', 'Simultaneous Equations with a Quadratic') shown as a secondary 'also seen as' list. Safe because it only adds evidence, never re-buckets existing rows.

How it's built: cmd_label runs 33 anchored/bounded regexes over the ARCHETYPE TEXT (not the literal exam command) and returns the first match else 'Other'; counted per part, board-filtered, drawn as a horizontal bar.

Accuracy: Structurally limited: the classifier reads the crawlers' paraphrase, not the paper's actual command word, so 'Other' is the LARGEST bucket — 29.5% of 9,336 crawl parts (and 45.6% when I ran cmd_label over the 2,462 OCR-grid extra.description strings, e.g. 'Use inequalities'->Other, 'Fraction to decimal conversion'->Other). This also distorts real frequencies: 'Work out' (the dominant Edexcel command in reality) shows only 3.3% because archetype text rarely restates the imperative. No dataset carries literal command words, so it cannot be validated or fixed from the new QLA. The regex anchoring itself is sound (audit-M2 fixes for ^circle / express(?!ion) / \bprove\b hold up on inspection).

Upgrade: Either (a) relabel the section honestly as 'What the question asks you to DO (task verb)' to stop implying it is the exam command-word distribution, or (b) shrink the 'Other' bucket by adding phrase->verb rules for the common description forms ('conversion'->Convert, 'terminology'->State, 'use ...'->Use). A true command-word count would need re-scraping the literal stems (not available). Recommend (a) now, (b) as polish.

How it's built: When a single question-part carries 2+ topic tags (joined by '+'), every unordered pair is counted; per-topic 'partners' (top 8) + global top-30 pairings. Purely co-occurrence within one part.

Accuracy: The mechanic is correct but the EVIDENCE BASE is thin and cannot be enriched by the new data: only 728 of 9,336 crawl parts (7.8%) are multi-topic, so all pairings rest on ~8% of the corpus. Pinpoint CANNOT help — it stores exactly one topic per row (no synoptic co-tags) and no description, so the largest new asset (20,584 rows) adds zero pairing signal. Genuinely synoptic pairs it does surface are believable, but coverage is sparse and board-skewed by whichever crawlers tagged multi-topic.

Upgrade: Low-risk: raise confidence by adding a min-count floor + '(n sittings)' provenance so a 1-off co-tag isn't shown as a 'common pairing'. Structural alternative: derive pairings from whole-QUESTION co-occurrence (topics appearing in the same numbered question across its parts) rather than same-part — richer signal — but that needs the crawl re-aggregated at question grain.

How it's built: 1-2 mark parts (excluding draw/plot/construct via NOT_QUICK) are grouped by quick_family (67 ordered 'all-of-substrings' rules over archetype text); unmatched keep raw phrasing; families with total count >=3 shown, top 10, board-split. Same DATA.quick powers the student Quick wins.

Accuracy: The curated families are sensible and the leak-curation appends (Mathedup41) are well-targeted, but coverage is partial: of 5,329 eligible 1-2-mark items, 49% (2,622) fall through quick_family unmatched to raw phrasing. The >=3 floor then keeps only recurrent raws, so the list is roughly half curated-family / half raw-archetype — inconsistent grain (a tidy 'HCF / LCM' family sits next to a verbatim crawl string). Accurate where it fires; incomplete overall. Board splits are trustworthy (straight counts).

Upgrade: Extend QF_RULES to absorb the high-frequency raw survivors (audit the 2,622 unmatched, add ~15-20 rules) so families dominate the list; and cross-tag each family with facility from aqa-qlapd/edexcel-enhanced (real % correct) so 'quick win' means genuinely high-facility, not just low-mark — currently 'quick' is inferred purely from mark tariff.

How it's built: For a chosen paper, sums marks/nt per topic across that paper's parts -> top-15 horizontal bar; flags '✓ Gold-verified' if the paper is in gold_standard.json's 27 re-tagged set.

Accuracy: Straightforward and correct aggregation; inherits the attribution layer's accuracy (83% eff-agreement) with an explicit per-paper trust signal for the 27 gold papers — a genuinely honest touch. Weakness is only that non-gold papers carry no confidence marker and thin-topic mis-tags (the crawl-under-represented topics above) can under-show on a given paper.

Upgrade: Show AO1/2/3 marks-weight per paper using the REAL board grids now available (OCR H 42/31/26, AQA H 51/29/19, Edexcel H 58/22/19 etc.) instead of the board-agnostic OCR-specimen model the site currently uses — the single-paper view is the natural place to expose the real per-board AO mix.

How it's built: Same DATA.quick families as the teacher quick-wins, surfaced to students with a 'Practise' link only where a live /quiz/gcse/ exists (quiz_topics gate). Inherits QF_RULES coverage/limits.

Accuracy: Same engine so same 49%-unmatched / mixed-grain limitation as the teacher section, minus one point because students see raw crawl phrasings as if curated ('quick wins' should read cleanly). Quiz-gating is a correct UX guard (no dead 'Practise' links). No facility signal means 'quick' = low-tariff, not verified-easy — a student could be pointed at a low-facility 2-marker.

Upgrade: Reuse the teacher upgrade (extend QF_RULES + attach real facility) — for students, prefer HIGH-facility 1-2 mark families so 'quick win' is honest per the VISION 'never give false confidence' test. Order the student list by facility, not raw frequency.

How it's built: Proposed. Each file: canonical_topic + top_misconceptions[] {summary (copyright-safe paraphrase), seen_count (=distinct papers), papers, distractor_hint, discriminator(bool), facility(low/med/high)}. Neither /exam-analysis/ nor /revision-finder/ reads it.

Accuracy: SHIPPED (teacher). Common-mistakes card live on the topic deep-dive: 87 topics, examiner-report misconceptions with discriminator badges + distractor hints (one source mis-tag filtered). Student watch-out strip DEFERRED pending a second-person voice rewrite.

Upgrade: Teacher card DONE. Follow-up: rewrite summaries to second-person for a student strip.

How it's built:

Accuracy: This is the ROOT of freshness and it is NOT automated. compiler_pipeline.py:94 only DETECTS new index_*.md files (idx_now vs prev); producing them is a manual crawl. add_papers.scan() files a dropped PDF into embargo but does not crawl it. compiler_pipeline.py:128 surfaces a needs_crawl() notification + copy-prompt for the owner. So 'add a paper' = drop PDF (auto-filed, embargoed) then MANUAL crawl before anything downstream can move.

Upgrade: Wire an automated crawl (pdftotext + MS-parse agent) into the compiler on file-drop so index_*.md is generated, not hand-authored; keep a human verify gate.

Depends on: a human/agent crawling the QP+MS PDF into an index_*.md row-per-part table, compiler_console needs_crawl() prompt

How it's built:

Accuracy: build_data() (build_paper_analysis.py:296-405) re-derives every one of these from a fresh glob of index_*.md on each build. Once a new paper is crawled, re-running the builder (which the daily cron does automatically when idx_now changes) refreshes all pure-count aggregates with zero hand-tuning. The only static dependency is the slug canon map, which is stable. These are the strongest freshness case on the pages.

Upgrade: None needed for freshness. (Optional: expose 'last crawl date / n papers' in the page footer so staleness is visible.)

Depends on: reference/gcse-papers/index_*.md, topic_norm.canon (slug map), data/takeaway/gcse_*_topics.json (titles)

How it's built:

Accuracy: Counts are data-driven, but a NEW topic slug that matches no rule token silently defaults to 'Number' (line 235), and genuine-token collisions need a hand-added STRAND_OVERRIDE entry (14 already curated). Adding papers that introduce a new canonical slug can mis-bucket a whole strand column until someone extends the override table.

Upgrade: On rebuild, assert every slug present in the crawl either matches a rule or has an explicit override; fail/flag unmapped slugs instead of defaulting to Number.

Depends on: build_paper_analysis.py STRAND_RULES/STRAND_OVERRIDE (hand-tuned), new topic slugs from the crawl

How it's built:

Accuracy: These bucket free-text archetype phrasing via hand-curated regexes. A new paper phrased in a way the tables don't anticipate falls to 'Other' (command words) or keeps its raw phrasing (quick-wins, dropped below the min-count-3 floor), or lands in the wrong topic without a matching ARCH_REMAP rule. Aggregates recompute automatically but the CLASSIFICATION quality erodes as new phrasings arrive — pure aggregation cannot self-heal a rule gap.

Upgrade: Periodically dump the 'Other'/unmatched-phrasing tail from the current crawl and curate the top misses back into CMD_RULES/QF_RULES/ARCH_REMAP; consider an LLM fallback labeller for the residual.

Depends on: build_paper_analysis.py CMD_RULES/QF_RULES (regex tables), reference/gcse-papers/topic_norm.py ARCH_REMAP

How it's built:

Accuracy: The mark totals feeding the split refresh from the crawl every build, but the AO FRACTIONS come from a frozen OCR-specimen file (ao_calibration.json) — it never updates as papers are added. IMPORTANT (verified this session): the mix it produces (~F 50/25/25, H 40/30/30) already matches the Ofqual-mandated weightings and AQA official grid, so it is ACCURATE. The limitation is purely that it is STATIC (no auto-refresh), NOT that it misestimates any board. Do NOT recalibrate from QLA ao-labels/pinpoint — those over-tag AO1 and would introduce error.

Upgrade: Leave the values as-is. Only future improvement: if OCR ever publishes more specimen AO grids, regenerate build_ao_calibration.py. No board-specific recalibration.

Depends on: reference/gcse-papers/ao_calibration.json (STATIC, OCR-specimen-calibrated, BOARD-AGNOSTIC), ao_split() marks x tier x band

How it's built:

Accuracy: The cumulative-marks curve is crawl-driven (auto), but the grade lines come from a static grade-boundaries.json that stops at 2024 and a hardcoded per-paper max. Add a 2025+ paper and the boundary bands won't include that year until the JSON is manually extended (load_grade_boundaries degrades to {} on absence). PMAX is a fixed assumption per board.

Upgrade: Refresh grade-boundaries.json when new-year boundaries are published (a small annual manual step) and derive PMAX from the crawl's actual per-paper totals rather than hardcoding.

Depends on: reference/grade-boundaries.json (STATIC, avg_pct 2017-24), PMAX hardcoded {OCR:100,AQA:80,Edexcel:80} in STUDENT_PAGE, crawl marks (auto)

How it's built:

Accuracy: refresh_tag_audit() (build_paper_analysis.py:453) re-runs audit_paper_tags.main() on EVERY build (incl. the daily cron), so the ⚠ flags are regenerated from the current crawl and never go stale. load_tag_conf() folds in per-topic confidence + leak counts. This is genuinely self-maintaining and SCALES as papers are added (thin topics get flagged; topics that gain evidence lose the flag automatically). Strong honesty feature.

Upgrade: None for freshness. Optionally surface the same ⚠ chip on the STUDENT page (currently teacher-only) so students aren't shown thin topic charts as authoritative.

Depends on: reference/gcse-papers/tag_audit.json, scripts/audit_paper_tags.py

How it's built:

Accuracy: gold_standard.json is a one-off blind-retag agreement measure over 27 papers (~82.5%). It is NOT regenerated on build and does NOT scale: as the corpus grows past 246 papers the fixed 27-paper sample becomes a smaller, older slice, so the published accuracy figure and the 'independently verified' badges drift out of representativeness. The number stays plausible but its provenance ages silently.

Upgrade: Make the gold sample a rolling % of the corpus (re-sample + re-blind-tag on a schedule) or at minimum stamp the note with sample size + date + n-of-corpus so its currency is visible.

Depends on: reference/gcse-papers/gold_standard.json, scripts/gold_standard.py (blind-retag of a FIXED 27-paper sample)

How it's built:

Accuracy: These are NOT in compiler_pipeline.py or compiler_cron.sh (verified: the daily cron runs ONLY build_paper_analysis.py + build_compiler_console.py). So the coverage %, exam_pct/quiz_pct chips, and board-confidence pills that appear on the gallery/tracker do NOT refresh when a paper is added — someone must remember to re-run merge_crawl + coverage_score + board_confidence (+ gen_eligibility after any gen.js change). A new paper that shifts archetype frequency or introduces board evidence leaves those surfaces stale until a manual pass. Biggest hidden blast-radius item.

Upgrade: Add merge_crawl + coverage_score + board_confidence to compiler_pipeline.py's 'changed' branch so the coverage/pill surfaces rebuild on the same trigger as the analysis pages.

Depends on: reference/gcse-papers/merge_crawl.py + ARCH_REMAP, coverage_scores.json, board_confidence.json, gen_eligibility.json