Evidence signal
Day 2, Week 1
Evidence e(k,c) ranks hospitals' true per-class counts with Spearman >= 0.7 on S1
Came in under the bar on its one allowed retry, so the project took Fallback F1 as planned. A sign-aware amendment (M1) scored 0.11, also under the bar.
If the bar is not met: One retry (bias row, or evidence averaged over 3 rounds); if still below 0.7, Fallback F1
Baseline quality
End of Week 1
Tier A FedAvg reaches balanced accuracy >= 0.45
Passed on the design doc's retry: the last dense block fine-tuned by FedAvg, features re-extracted.
If the bar is not met: Unfreeze the last dense block and re-extract features. One retry.
Problem is real
End of Week 2 (30 Sep)
On S1 or S2, under Camp A, attack A1 gives the attacker >= 2x its FedAvg weight on a rare row AND drops that class's F1 by >= 0.15, with balanced accuracy falling < 0.03
The attack succeeds on neither split: Fallback F1 - an empirical study of how rare-class-aware aggregation behaves on a real hospital split.
If the bar is not met: Stop building EARN. Fallback F1.
EARN works
End of Week 4 (14 Oct)
Under A1 and A2, EARN's rare-class F1 stays within 0.05 of no-attack; with no attack within 0.02 of Camp A and above FedAvg; honest centre 2 reaches trust >= 0.8
Exploratory re-run on an oracle evidence signal: conditions not met on S1, not met on S2. Reported as exploratory, never as the gate.
If the bar is not met: Fallback F2.
Indian baseline quality
Week 7, second dataset
FedAvg on the age-band split reaches balanced accuracy >= 0.40 over 7 classes
Cleared by 0.008 - narrow enough to say so. 40 rounds, seed 42, on the constructed age-band split.
If the bar is not met: DermaCon-IN is used for external validation only, and the shortfall is reported
Does EARN help on Indian data?
Week 7, second dataset
Rare-class F1 within 0.05 of FedAvg, the specialist weighted above its size share, and no effect on the near-IID control
EARN's holder threshold is an absolute count of 20 images, calibrated on Fed-ISIC2019; at this dataset's scale it admits almost nobody, so trust and evidence never engage. A scale-relative threshold was then pre-registered on Fed-ISIC2019 and rejected there: a bar set by the round's own evidence is one a claiming attacker clears by claiming.
If the bar is not met: Reported as measured; the cause is diagnosed rather than tuned away
| What | State | Owner · branch |
|---|---|---|
| A holder threshold that is both scale-free and unspoofableWaiting on the round-mean rule was rejected (R8): a bar set by this round's evidence is one a claiming attacker clears by claiming | Team- | |
| A working evidence signal for EARNWaiting on a signal that clears G0a - the one allowed retry came in under the bar | Team- | |
| The threshold amendment re-run on the grid's own features | Teammain | |
| CARE-FL full-text check | Team- | |
| Tier B repeated on a metric that matches the attack - rare-class recall or the to-nevus share - with three seeds and a longer post-turn windowWaiting on the one-seed run (R11) was scored on rare macro-F1, which its addendum shows is confounded here by calibration drift in the honest baseline | Teammain |