Table 2 Random forests performance measures in the development and validation data across all outcomes

Development

 AUC (95% CI)

0.62 (0.60, 0.64)

0.66 (0.64, 0.68)

0.67 (0.64, 0.69)

 Sensitivity / Specificity (optimal cut-off)

0.51 / 0.68 (0.58)

0.67 / 0.56 (0.53)

0.71 / 0.53 (0.67)

 OOB error / RMSE

39.84%

38.72%

29.77%

2.89

2.94

3.33

 pseudo-R²

0.112

0.207

0.186

 Calibration intercept (95% CI)

0.11 (0.04, 0.18)

0.01 (-0.08, 0.10)

0.01 (-0.12, 0.14)

0.04 (-0.46, 0.54)

-0.06 (-0.33, 0.22)

-0.18 (-0.64, 0.28)

 Calibration slope (95% CI)

0.83 (0.71, 0.95)

1.00 (0.83, 1.16)

0.95 (0.76, 1.13)

0.98 (0.84, 1.13)

1.02 (0.92, 1.13)

1.05 (0.93, 1.17)

 Calibration-in-the-large

0.01

 < 0.01

-0.03

-0.02

-0.01

 < 0.01

 E/O

0.98

0.99

1.04

 Brier score

0.23

0.22

0.20

 ECI

0.07

0.09

0.16

 Pearson's r (observed vs. predicted)

0.33

0.45

0.43

Validation

 AUC (95% CI)

0.62 (0.60, 0.65)

0.68 (0.66, 0.71)

0.66 (0.63, 0.69)

 Sensitivity / Specificity (optimal cut-off)

0.56 / 0.64 (0.56)

0.86 / 0.39 (0.47)

0.75 / 0.50 (0.66)

 RMSE

2.94

3.04

3.33

 pseudo-R²

0.152

0.252

0.220

 Calibration intercept (95% CI)

0.06 (-0.07, 0.18)

-0.06 (-0.17, 0.06)

0.01 (-0.16, 0.19)

-0.26 (-0.96, 0.45)

-0.04 (-0.42, 0.35)

-0.25 (-0.72, 0.22)

 Calibration slope (95% CI)

0.94 (0.71, 1.16)

1.20 (0.99, 1.41)

0.96 (0.72, 1.21)

1.12 (0.91, 1.33)

1.14 (1.00, 1.29)

1.14 (1.02, 1.25)

 Calibration-in-the-large

0.02

0.05

-0.01

0.14

0.28

0.25

 E/O

0.96

0.92

1.02

 Brier score

0.23

0.22

0.20

 ECI

0.07

0.38

0.17

 Pearson's r (observed vs. predicted)

0.39

0.5

0.47