# Breast Ultrasound Lesion Analysis: results

Alonso Jiménez Flores

Classification estimates malignancy for an imaged lesion. Segmentation delineates the index lesion, and a bounding box is derived from its largest predicted component. The models are trained independently. This is a research and educational project.

## Data and model selection

| Partition | Images | Cases | Benign | Malignant | Normal |
| --- | --- | --- | --- | --- | --- |
| train | 1307 | 744 | 505 | 239 | 0 |
| val | 286 | 160 | 109 | 51 | 0 |
| test | 282 | 160 | 108 | 52 | 0 |
| external | 252 | 252 | 154 | 98 | 0 |
| external_normal | 4 | 4 | 0 | 0 | 4 |

All images from a case remain in one partition. The BUS-BRA split is diagnosis-stratified, with seed 2026 and intended fractions 70/15/15. Four normal BrEaST images are excluded from the binary lesion task.

Candidates use ImageNet initialization, AdamW, a cosine learning-rate schedule, up to 50 epochs and patience 10. Each candidate is fitted with seed 17. Checkpoint and configuration selection maximize validation case AP for classification or native-resolution case Dice for segmentation; epoch ties retain the earliest checkpoint. Both selected configurations are repeated with seeds 29 and 43. Validation scores are development estimates, not unbiased performance estimates. Resolution candidates also use different batch sizes, so their comparison does not isolate resolution alone.

| Candidate | Encoder | Input | Batch | Epoch | Validation metric | Validation score | Selected |
| --- | --- | --- | --- | --- | --- | --- | --- |
| classification_224 | resnet18 | 224 | 16 | 17 | ap_case | 0.902 | yes |
| classification_384 | resnet18 | 384 | 8 | 25 | ap_case | 0.895 |  |
| segmentation_224 | resnet18 | 224 | 16 | 28 | dice_case | 0.891 |  |
| segmentation_384 | resnet18 | 384 | 8 | 18 | dice_case | 0.898 |  |
| segmentation_resnet34_384 | resnet34 | 384 | 8 | 14 | dice_case | 0.903 | yes |

## Classification

Probabilities are averaged within each case before scoring. AP concerns the malignant class; its prevalence reference is 0.325 in BUS-BRA testing and 0.389 in BrEaST. Decisions use the Youden threshold fitted separately for each run on validation, with sensitivity favored in ties. Temperature scaling is a secondary analysis, also fitted on validation. Original probabilities are the primary output.

| Model | Cohort | AUROC | AP | Sensitivity | Specificity | Precision | F1 | Balanced accuracy | MCC | Brier | Brier calibrated |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| ResNet18 classifier · 224 px | BUS-BRA test | 0.903 ± 0.016 | 0.865 ± 0.017 | 0.801 ± 0.097 | 0.852 ± 0.056 | 0.729 ± 0.054 | 0.759 ± 0.016 | 0.827 ± 0.021 | 0.642 ± 0.018 | 0.128 ± 0.032 | 0.121 ± 0.030 |
| ResNet18 classifier · 224 px | BrEaST | 0.847 ± 0.013 | 0.779 ± 0.023 | 0.833 ± 0.041 | 0.690 ± 0.026 | 0.632 ± 0.021 | 0.718 ± 0.025 | 0.762 ± 0.023 | 0.511 ± 0.044 | 0.220 ± 0.076 | 0.199 ± 0.068 |
| Prevalence / mean mask | BUS-BRA test | 0.500 | 0.325 | 1.000 | 0.000 | 0.325 | 0.491 | 0.500 | 0.000 | 0.219 | 0.219 |
| Prevalence / mean mask | BrEaST | 0.500 | 0.389 | 1.000 | 0.000 | 0.389 | 0.560 | 0.500 | 0.000 | 0.242 | 0.243 |
| Frozen ResNet18 + logistic regression | BUS-BRA test | 0.777 | 0.693 | 0.577 | 0.907 | 0.750 | 0.652 | 0.742 | 0.524 | 0.166 | 0.165 |
| Frozen ResNet18 + logistic regression | BrEaST | 0.679 | 0.600 | 0.255 | 0.909 | 0.641 | 0.365 | 0.582 | 0.221 | 0.266 | 0.269 |

Neural results are mean ± sample standard deviation across the three fixed seeds. Baselines have one deterministic execution. The frozen feature baseline uses ResNet18 at 224 pixels, train-fitted weighted standardization, and logistic regularization selected by validation AP from C = 0.01, 0.1, 1, 10. Confusion matrices and results at threshold 0.5 are retained in run metrics.

## Segmentation and derived localization

Predicted probability maps are unpadded and restored to the original image dimensions before thresholding at 0.5. Image Dice and IoU are averaged within each case, then across cases. A missing prediction for a lesion scores zero. Localization compares the largest predicted 8-connected component's box to the extent of the complete index-lesion annotation. Success is the fraction of image box IoUs ≥0.5 within a case, averaged across cases.

| Model | Cohort | Dice | IoU | Dice median | Dice p10 | Box IoU | Localization success |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Prevalence / mean mask | BUS-BRA test | 0.260 | 0.164 | 0.265 | 0.003 | 0.187 | 0.025 |
| Prevalence / mean mask | BrEaST | 0.256 | 0.164 | 0.228 | 0.000 | 0.195 | 0.048 |
| U-Net / ResNet34 · 384 px | BUS-BRA test | 0.900 ± 0.002 | 0.830 ± 0.003 | 0.925 ± 0.002 | 0.810 ± 0.017 | 0.839 ± 0.001 | 0.964 ± 0.008 |
| U-Net / ResNet34 · 384 px | BrEaST | 0.735 ± 0.012 | 0.632 ± 0.011 | 0.840 ± 0.006 | 0.322 ± 0.012 | 0.650 ± 0.019 | 0.788 ± 0.037 |

![Classification and segmentation](benchmark-summary.png)

The error bars show training-seed variation. They are not patient sampling intervals.

## Individual fits and sampling uncertainty

| Model | Seed | Cohort | AP | 95% case CI | Threshold | Dice | 95% case CI |
| --- | --- | --- | --- | --- | --- | --- | --- |
| classification_224 | 17 | BUS-BRA test | 0.884 | [0.818, 0.942] | 0.103 | — | — |
| classification_224 | 17 | BrEaST | 0.785 | [0.719, 0.850] | 0.103 | — | — |
| classification_224 | 29 | BUS-BRA test | 0.856 | [0.778, 0.922] | 0.915 | — | — |
| classification_224 | 29 | BrEaST | 0.754 | [0.679, 0.830] | 0.915 | — | — |
| classification_224 | 43 | BUS-BRA test | 0.853 | [0.777, 0.918] | 0.301 | — | — |
| classification_224 | 43 | BrEaST | 0.799 | [0.733, 0.867] | 0.301 | — | — |
| control | 17 | BUS-BRA test | 0.325 | [0.325, 0.325] | 0.321 | 0.260 | [0.232, 0.289] |
| control | 17 | BrEaST | 0.389 | [0.389, 0.389] | 0.321 | 0.256 | [0.231, 0.280] |
| frozen_resnet18 | 17 | BUS-BRA test | 0.693 | [0.587, 0.805] | 0.462 | — | — |
| frozen_resnet18 | 17 | BrEaST | 0.600 | [0.520, 0.677] | 0.462 | — | — |
| segmentation_resnet34_384 | 17 | BUS-BRA test | — | — | — | 0.902 | [0.887, 0.915] |
| segmentation_resnet34_384 | 17 | BrEaST | — | — | — | 0.723 | [0.687, 0.754] |
| segmentation_resnet34_384 | 29 | BUS-BRA test | — | — | — | 0.898 | [0.883, 0.911] |
| segmentation_resnet34_384 | 29 | BrEaST | — | — | — | 0.735 | [0.704, 0.764] |
| segmentation_resnet34_384 | 43 | BUS-BRA test | — | — | — | 0.900 | [0.887, 0.911] |
| segmentation_resnet34_384 | 43 | BrEaST | — | — | — | 0.746 | [0.715, 0.776] |

Intervals use 2,000 case bootstrap replicates. Classification bootstrap is stratified by diagnosis, conditioning on observed prevalence; segmentation uses ordinary case resampling. Intervals are conditional on the fitted checkpoint, split and validation policy. They exclude uncertainty in model selection, threshold estimation, annotation and future training runs.

### Paired baseline comparisons

Both models and all fixed seeds use the same bootstrap case indices. Differences average seed-specific metric differences; seeds are not treated as additional patients. These are exploratory comparisons without multiplicity adjustment.

| First − second | Cohort | Metric | Difference | 95% case CI |
| --- | --- | --- | --- | --- |
| classification_224 − frozen_resnet18 | BUS-BRA test | auroc | 0.126 | [0.054, 0.200] |
| classification_224 − frozen_resnet18 | BUS-BRA test | average_precision | 0.171 | [0.076, 0.264] |
| classification_224 − frozen_resnet18 | BUS-BRA test | brier | -0.037 | [-0.069, -0.003] |
| classification_224 − frozen_resnet18 | BrEaST | auroc | 0.168 | [0.107, 0.233] |
| classification_224 − frozen_resnet18 | BrEaST | average_precision | 0.179 | [0.109, 0.253] |
| classification_224 − frozen_resnet18 | BrEaST | brier | -0.046 | [-0.084, -0.011] |
| classification_224 − control | BUS-BRA test | auroc | 0.403 | [0.345, 0.448] |
| classification_224 − control | BUS-BRA test | average_precision | 0.540 | [0.471, 0.597] |
| classification_224 − control | BrEaST | auroc | 0.347 | [0.302, 0.388] |
| classification_224 − control | BrEaST | average_precision | 0.390 | [0.329, 0.454] |
| segmentation_resnet34_384 − control | BUS-BRA test | dice | 0.640 | [0.609, 0.669] |
| segmentation_resnet34_384 − control | BUS-BRA test | iou | 0.666 | [0.639, 0.691] |
| segmentation_resnet34_384 − control | BrEaST | dice | 0.479 | [0.442, 0.514] |
| segmentation_resnet34_384 − control | BrEaST | iou | 0.468 | [0.434, 0.501] |

## Error analysis

Each evaluated run retains confusion matrices, ROC/PR and calibration curves, deterministic false positives/negatives and the lowest-Dice images. Ordering uses error severity and a stable image-ID tie-break. Device, diagnosis, relative lesion size, verification and other-lesion strata are descriptive analyses. Small subgroups cannot establish reliable differences. Reference annotations are used only for scoring and error analysis, never as inference inputs.

| Task (seed 17) | Normal case | Raw image probability | Predicted area fraction |
| --- | --- | --- | --- |
| classification | breast:209 | 0.874 | — |
| classification | breast:213 | 0.000 | — |
| classification | breast:45 | 0.000 | — |
| classification | breast:61 | 0.338 | — |
| segmentation | breast:209 | — | 0.028 |
| segmentation | breast:213 | — | 0.000 |
| segmentation | breast:45 | — | 0.207 |
| segmentation | breast:61 | — | 0.000 |

The four normal images have no lesion-overlap score. They are individual out-of-task observations, not an estimate of screening specificity.

### Subgroups for the default seed

| Task | Cohort | Stratum | Value | Cases | AP | Dice |
| --- | --- | --- | --- | --- | --- | --- |
| classification | BUS-BRA test | label | 0 | 108 | — | — |
| classification | BUS-BRA test | label | 1 | 52 | — | — |
| classification | BUS-BRA test | device | GE Logiq 5 @10-12MHz | 79 | 0.890 | — |
| classification | BUS-BRA test | device | GE Logiq 7 @10-14MHz | 71 | 0.899 | — |
| classification | BUS-BRA test | device | Toshiba Aplio 300 @12-14 MHz | 7 | 0.806 | — |
| classification | BUS-BRA test | device | U-Systems @10-14MHz | 3 | 1.000 | — |
| classification | BUS-BRA test | lesion_size | large (>=15%) | 23 | 0.949 | — |
| classification | BUS-BRA test | lesion_size | medium (5–15%) | 77 | 0.911 | — |
| classification | BUS-BRA test | lesion_size | small (<5%) | 60 | 0.581 | — |
| classification | BUS-BRA test | has_other_annotation | False | 160 | 0.884 | — |
| classification | BrEaST | label | 0 | 154 | — | — |
| classification | BrEaST | label | 1 | 98 | — | — |
| classification | BrEaST | device | unknown | 252 | 0.785 | — |
| classification | BrEaST | lesion_size | large (>=15%) | 30 | 0.957 | — |
| classification | BrEaST | lesion_size | medium (5–15%) | 92 | 0.802 | — |
| classification | BrEaST | lesion_size | small (<5%) | 130 | 0.611 | — |
| classification | BrEaST | has_other_annotation | False | 245 | 0.786 | — |
| classification | BrEaST | has_other_annotation | True | 7 | — | — |
| segmentation | BUS-BRA test | label | 0 | 108 | — | 0.925 |
| segmentation | BUS-BRA test | label | 1 | 52 | — | 0.855 |
| segmentation | BUS-BRA test | device | GE Logiq 5 @10-12MHz | 79 | — | 0.919 |
| segmentation | BUS-BRA test | device | GE Logiq 7 @10-14MHz | 71 | — | 0.886 |
| segmentation | BUS-BRA test | device | Toshiba Aplio 300 @12-14 MHz | 7 | — | 0.891 |
| segmentation | BUS-BRA test | device | U-Systems @10-14MHz | 3 | — | 0.856 |
| segmentation | BUS-BRA test | lesion_size | large (>=15%) | 23 | — | 0.858 |
| segmentation | BUS-BRA test | lesion_size | medium (5–15%) | 77 | — | 0.912 |
| segmentation | BUS-BRA test | lesion_size | small (<5%) | 60 | — | 0.905 |
| segmentation | BUS-BRA test | has_other_annotation | False | 160 | — | 0.902 |
| segmentation | BrEaST | label | 0 | 154 | — | 0.700 |
| segmentation | BrEaST | label | 1 | 98 | — | 0.758 |
| segmentation | BrEaST | device | unknown | 252 | — | 0.723 |
| segmentation | BrEaST | lesion_size | large (>=15%) | 30 | — | 0.808 |
| segmentation | BrEaST | lesion_size | medium (5–15%) | 92 | — | 0.811 |
| segmentation | BrEaST | lesion_size | small (<5%) | 130 | — | 0.640 |
| segmentation | BrEaST | has_other_annotation | False | 245 | — | 0.723 |
| segmentation | BrEaST | has_other_annotation | True | 7 | — | 0.710 |

Additional BrEaST lesions are excluded from the primary annotation. A prediction overlapping another lesion can count as an index-target error; diagnosis is never assigned automatically to those other objects.

## Interpretation and limitations

The useful outputs are a case malignancy score, a native-image lesion mask, and a reproducible visual record of their errors. Accuracy must be interpreted for selected lesion images. These data do not evaluate screening, exhaustive multi-lesion detection or a clinical workflow.

Performance on BrEaST describes transfer between the two released cohorts. Case composition, equipment, annotation and benign-lesion verification differ; these effects cannot be separated by the present experiment. These cohorts have been inspected during project development, so another independently sourced cohort is needed to confirm broader transfer claims.

One patient split and three seeds leave split and population variability unresolved. Validation reuse makes candidate scores optimistic. Image annotations and acquisition markers may be shortcuts; error images cannot establish why a network made a prediction. The next experiment should use patient-grouped cross-validation and a cohort with representative negatives and explicit patient linkage.

All numerical tables are generated from saved predictions. Run-level JSON/CSV records are copied into reports/runs for inspection without downloading model weights.