Everything we've measured, negative results included.
0.535
± 0.002 across 3 seeds · first attempt 0.415
FedAvg balanced accuracy after the retry (last dense block fine-tuned). Gate G0b needed ≥ 0.45.
-0.21
mean Spearman, 8 diseases · retry -0.46
EARN's evidence signal. Gate G0a needed ≥ 0.7.
408
attack-study runs · 5 items not run
Attacks, Camp A, robust rules and EARN on an oracle signal. See the study →
FedAvg on the natural split.
Rare-disease F1
Final round, mean over seeds. Whiskers show ± one standard deviation.
In numbers
- Balanced accuracy
- 0.415 ± 0.005
- Dermatofibroma F1
- 0.192 ± 0.012
- Vascular lesion F1
- 0.471 ± 0.037
- Overall accuracy
- 0.591
Per seed · balanced accuracy
Gate G0b, first attempt: below the bar
Balanced accuracy after every round, mean over seeds.
FedAvg finished at 0.415, short of the 0.45 it needed. It was still climbing when the run stopped: 0.397 at round 30, 0.415 at round 40.
The retry: passed
Balanced accuracy after every round, mean over seeds.
The design doc's retry - fine-tune DenseNet's last dense block (by FedAvg, on training images only, 13 minutes on a laptop GPU) and re-extract the features - lifts FedAvg to 0.535 at the same 40 rounds. Gate G0b passes.
Training for 100 rounds was an amendment, logged before running: 0.450 on frozen features, 0.536 on fine-tuned ones (rare F1 0.592). Every later experiment uses the fine-tuned features and 100 rounds.
EARN's evidence signal came out inverted.
Spearman correlation, per disease
★ = rare disease. Red bars point the wrong way.
What went wrong
For both rare diseases, the hospital with the highest evidence was one holding zero images of it. The likely cause: local training pushes down the score of a disease a hospital never sees, and that also shows up as a large change - the measure can't tell "up" from "down".
- Actinic keratosistop: Centre 5 · 0 images
- Dermatofibromatop: Centre 4 · 0 images
- Vascular lesiontop: Centre 4 · 0 images
- Squamous cell carcinomatop: Centre 5 · 0 images
The one allowed retry also came in under the bar, so the project switched to Fallback F1.
The retry, and one amendment
Written down and committed before running. Mean Spearman over the 8 diseases, needs ≥ 0.7. Holder AUROC: how well the signal separates hospitals holding ≥ 20 images of a disease from the rest (0.5 = chance).
| Evidence measure | Spearman | Holder AUROC |
|---|---|---|
| Weight-row change - the original check | -0.210 | - |
| Bias-row change - the design doc's one retry (decides the gate) | -0.463 | 0.137 |
| Weight row averaged over rounds 1-3 - reported only | -0.240 | 0.145 |
| Sign-aware bias change - amendment M1, logged before running | 0.110 | 0.738 |
The sign-aware measure can tell whether a hospital holds a disease better than chance, but not how much - local training uses a class-balanced loss, which deliberately removes count information. That risk was written down before the run.
No result, no number.
A holder threshold that is both scale-free and unspoofable
Waiting on the round-mean rule was rejected (R8): a bar set by this round's evidence is one a claiming attacker clears by claiming
A working evidence signal for EARN
Waiting on a signal that clears G0a - the one allowed retry came in under the bar
The threshold amendment re-run on the grid's own features
CARE-FL full-text check
Tier B repeated on a metric that matches the attack - rare-class recall or the to-nevus share - with three seeds and a longer post-turn window
Waiting on the one-seed run (R11) was scored on rare macro-F1, which its addendum shows is confounded here by calibration drift in the honest baseline