Skip to contents

Refits the complete RF workflow inside folds that hold out whole studies or outermost multilevel groups. Each training fold learns transformations, sampling-covariance pooling, heterogeneity weights and any EM completion.

Usage

mars_rf_validate(object, folds = 5L, seed = 1L, permutation_repeats = 0L)

Arguments

object

A fitted mars_rf object.

folds

Number of folds, at least two and no more than independent groups.

seed

Reproducible fold assignment and refitting seed. RNG state is restored.

permutation_repeats

Number of held-out permutations per moderator; zero skips importance. Study-invariant predictors are permuted between groups as whole blocks. Varying predictors require exchangeable groups of equal size; otherwise their permutation importance is unavailable.

Value

A list with row-level held-out predictions, pooled sampling-variance weighted RMSE and R-squared, outcome-specific metrics, fold diagnostics and permutation importance. Importance is the increase in weighted squared error on held-out data, averaged over repetitions and folds using evaluation weights. Negative values are possible. Results condition on the supplied tuning choices.

Details

This evaluates population prediction for new groups. Tuning or selecting moderators with these same folds requires an additional outer validation loop. Errors in a fold are recorded, not replaced with in-sample predictions; pooled metrics use successful folds and report coverage. With EM, held-out continuous predictors are completed using training-fold conditional-normal moments. Missing categorical prediction moderators are unsupported. Coverage counts independent groups represented by successful evaluated rows. Errors predict observed effects rather than latent true effects; studies with greater precision or more observations can contribute more to pooled scores. Multivariate pooled R-squared can reflect different outcome means, so inspect by_effect. Permutations require exchangeable groups and compatible within-group ordering; with EM they act on completed held-out moderators. At least two held-out groups are required. Structural effect identity is not included in permutation importance.

Examples

fit <- mars_rf_univariate(teacher_expectancy, yi ~ year + weeks,
  "study", "vi", num_trees = 5, seed = 1)
validation <- mars_rf_validate(fit, folds = 3, seed = 2)
validation$metrics
#>       rmse         r2 
#> 0.21085223 0.06697152