Skip to contents

02. Model Performance & Benchmarking

Source: vignettes/model-performance.Rmd
model-performance.Rmd

Introduction

Once a SuperSurv ensemble is trained, its predictive performance should be evaluated on data not used to fit the model. An ensemble is not guaranteed to outperform every component learner in every sample. Because survival outcomes may be right-censored, ordinary classification accuracy is not an appropriate summary.

Instead, we evaluate the model across three critical dimensions: 1. Calibration: Does the predicted survival probability match the actual observed survival rate? 2. Discrimination (AUC & C-index): Can the model correctly rank which patient will survive longer? 3. Overall Accuracy (Brier Score): A combined measure of both calibration and discrimination.

This tutorial demonstrates how to extract these metrics and visualize the benchmark comparisons using SuperSurv’s built-in evaluation suite.

1. Data Preparation & The Extrapolation Rule

We begin by loading the metabric dataset and defining our evaluation time grid (new.times).

Crucial Methodological Note: Your new.times grid should never exceed the maximum observed follow-up time in your training cohort. For example, if your training data only spans 1 to 100 days, predicting survival at day 150 is extrapolation. Non-parametric models (like Survival Trees) mathematically cannot extrapolate, and parametric models (like Weibulls) will generate highly unreliable tail estimates. Always bind your evaluation grid within the limits of your observed data!

library(SuperSurv)
library(survival)

data("metabric", package = "SuperSurv")
set.seed(123)
train_idx <- sample(1:nrow(metabric), 0.7 * nrow(metabric))
train <- metabric[train_idx, ]
test  <- metabric[-train_idx, ]

X_tr <- train[, grep("^x", names(metabric))]
X_te <- test[, grep("^x", names(metabric))]

# Our max follow-up is well beyond 200, so this grid is safe.
new.times <- seq(50, 200, by = 25)

2. Train the Benchmark Ensemble

We will fit an ensemble consisting of a Cox model, a Weibull model, and a Survival Tree using the default Least Squares meta-learner.

my_library <- c("surv.coxph", "surv.weibull")
if (has_rpart) {
  my_library <- c(my_library, "surv.rpart")
}

fit_supersurv <- SuperSurv(
  time = train$duration,
  event = train$event,
  X = X_tr,
  newdata = X_te,
  new.times = new.times,
  event.library = my_library,
  cens.library = my_library,
  control = list(saveFitLibrary = TRUE),
  verbose = FALSE,
  nFolds = 3
)

3. Extracting Integrated Metrics

The eval_benchmark() function generates predictions on the test set and returns both integrated and time-specific numerical results. The built-in evaluator uses a marginal reverse Kaplan–Meier censoring estimate. For conditional censoring models, resampling, inference, or formal model comparisons, use specialist software such as riskRegression::Score().

# Evaluate performance directly using the fitted model and test data
performance_results <- eval_benchmark(
  object = fit_supersurv,
  newdata = X_te,
  time = test$duration,
  event = test$event,
  eval_times = new.times
)

performance_results$summary
#>                     Model       IBS IPCW_LogLoss     Uno_C      iAUC
#> 1      SuperSurv_Ensemble 0.2056052    0.5986356 0.6317163 0.6657561
#> 2   surv.coxph_screen.all 0.2070730    0.6044041 0.6290759 0.6620635
#> 3 surv.weibull_screen.all 0.2068304    0.6039515 0.6289231 0.6616319
#> 4   surv.rpart_screen.all 0.2156195    0.6198497 0.5931278 0.6268625
head(performance_results$by_time)
#>   Time              Model     Brier   LogLoss    CD_AUC   C_Index
#> 1   50 SuperSurv_Ensemble 0.1427394 0.4590400 0.6387032 0.6322948
#> 2   75 SuperSurv_Ensemble 0.1864146 0.5616281 0.6117365 0.6319181
#> 3  100 SuperSurv_Ensemble 0.2158827 0.6233480 0.6323065 0.6319049
#> 4  125 SuperSurv_Ensemble 0.2218279 0.6339276 0.6675448 0.6317163
#> 5  150 SuperSurv_Ensemble 0.2187880 0.6262853 0.6990790 0.6316829
#> 6  175 SuperSurv_Ensemble 0.2179843 0.6231089 0.6992792 0.6316231

The compatibility function eval_summary() returns the earlier compact table. Lower IBS and IPCW log-loss indicate better probabilistic prediction under their respective definitions; higher iAUC and Uno’s C-index indicate better discrimination.

4. Visualizing Longitudinal Benchmarks

A single integrated number rarely tells the whole story. Different models excel at different follow-up periods (e.g., a Cox model might dominate short-term survival, while a Random Forest dominates long-term).

The plot_benchmark() function visualizes the time-specific results. If the optional patchwork package is unavailable, it returns a named list of individual plots rather than failing.

The plotting examples below run only when the optional ggplot2 package is installed; numerical evaluation above remains available without it.

plot_benchmark(
  object = fit_supersurv,
  newdata = X_te,
  time = test$duration,
  event = test$event,
  eval_times = new.times
)

Interpreting the Curves:

  • IPCW Brier Score Plot: Look for the curve that stays the lowest. The SuperSurv ensemble should ideally hug the bottom edge.
  • Time-Dependent AUC Plot: Look for the curve that stays the highest (closest to 1.0).
  • Uno’s C-index Plot: Evaluates the global C-index using the specific predictions generated at each time point. Look for the curve that stays the highest.

5. Assessing Clinical Calibration

Before deploying a model to the clinic, doctors need to know if the probabilities are reliable. If the model predicts a patient has a 40% chance of surviving past Time = 100, do exactly 40% of similar patients actually survive?

We use plot_calibration() to evaluate this at a specific clinical milestone. The function groups patients into risk quantiles and plots their predicted probability against the actual observed Kaplan-Meier survival rate.

# Assess calibration specifically at Time = 150
plot_calibration(
  object = fit_supersurv,
  newdata = X_te,
  time = test$duration,
  event = test$event,
  eval_time = 150,
  bins = 5 # Group patients into 5 risk quintiles
)

Interpretation: A perfectly calibrated model will follow the 45-degree dashed black line. Points above the line indicate the model is under-predicting survival (being too pessimistic), while points below the line indicate it is over-predicting survival (being too optimistic).