
Harmonized Standardized Mean Differences
mars authors
HSMD-Sensitivity-Analysis.RmdPurpose and measurement scale
HSMD asks how conclusions change when studies use harmonized standardizers. It is a sensitivity analysis alongside conventional SMD, rather than a correction for publication bias or an automatic improvement in effect-size estimation.
For each arm, the coefficient of variation is CV=s/|m|. At a selected benchmark quantile, the harmonized SD is s_q=|m|CV_q. The outcome needs a meaningful ratio-scale origin and nonzero group means. Zero means are rejected; negative means trigger a warning to review the measurement scale. Changing measurement units by a positive multiplier preserves results. Adding a constant to scores can change CVs and HSMD, so arbitrary recentering is not a remedy for zero means. Very small nonzero means can give unstable CVs even without numerical errors; inspect their distribution and substantive plausibility.
The implementation estimates separate treatment and control benchmarks, as in the worked examples of Jung and Aloe (2025). Each available row contributes to its arm’s empirical CV distribution. With multiple effects per study, this gives studies with more rows greater benchmark representation. Use outcome-specific groups or externally justified benchmarks when that weighting is inappropriate.
Start with conventional and harmonized effects
out <- hsmd(hsmd_example_1, mT, sdT, nT, mC, sdC, nC, study=Study)
out$summary
#> method component est se ci.lb ci.ub tau2
#> 1 SMD (Intercept) -0.3482182 0.10640132 -0.5567610 -0.13967543 0.11743967
#> 2 HSMD_25 (Intercept) -0.5182606 0.22512743 -0.9595023 -0.07701898 0.72841243
#> 3 HSMD_50 (Intercept) -0.2504580 0.10868797 -0.4634825 -0.03743353 0.12560799
#> 4 HSMD_75 (Intercept) -0.1918735 0.08000593 -0.3486822 -0.03506475 0.04819645
#> avwW I2 n_effects n_studies
#> 1 0.04191511 60.01581 16 16
#> 2 0.04546883 89.56394 16 16
#> 3 0.04186181 61.64808 16 16
#> 4 0.04132897 38.45138 16 16
out$cv_benchmarks
#> $treatment
#> [1] 0.2854079 0.5067338 0.8359677
#>
#> $control
#> [1] 0.2166862 0.5257705 0.6481069
out$heterogeneity
#> method component tau2
#> SMD SMD between-study 0.11743967
#> HSMD_25 HSMD_25 between-study 0.72841243
#> HSMD_50 HSMD_50 between-study 0.12560799
#> HSMD_75 HSMD_75 between-study 0.04819645Compare effect direction, magnitude, uncertainty, heterogeneity, and analysis counts across SMD and the three benchmark scenarios. A change across scenarios shows sensitivity to the standardizer assumptions. It does not identify which scenario is true. Differences between scenario estimates are not formal tests: the scenarios reuse observations and benchmarks.
The pooled SD is
s_p=\sqrt{\frac{(n_T-1)s_T^2+(n_C-1)s_C^2}{n_T+n_C-2}},\qquad d=(m_T-m_C)/s_p.
The selected variance approximation is v_d=1/n_T+1/n_C+d^2/[2(n_T+n_C)]. For
measure="g", J=1-3/[4(n_T+n_C-2)-1], g=Jd, and v_g=J^2v_d. Other SMD software may use exact
corrections or different variance approximations. These formulas are
applied to the reconstructed SDs for each HSMD scenario. The resulting
standard errors and confidence intervals do not propagate CV
benchmark uncertainty. Interpret them conditionally within the
sensitivity analysis, not as fully calibrated inference for a new
estimator.
Custom quantiles and grouped benchmarks
custom <- hsmd_effect_sizes(hsmd_example_1, mT, sdT, nT, mC, sdC, nC,
probs=c(.251,.254,.5))
custom$prob_names
#> [1] "25p1" "25p4" "50"Distinct probabilities retain distinct labels: 25.1% becomes
25p1. Duplicate probabilities are rejected. Grouping is
useful when outcomes or domains have different defensible variability
distributions. The following artificial groups illustrate the API;
choose groups substantively in an actual analysis.
dat <- hsmd_example_1
dat$domain <- rep(c("A","B"), length.out=nrow(dat))
grouped <- hsmd_effect_sizes(dat, mT, sdT, nT, mC, sdC, nC,
cv_by=domain, probs=.5)
grouped$cv_benchmarks
#> $treatment
#> [,1]
#> A 0.5067338
#> B 0.5606226
#>
#> $control
#> [,1]
#> A 0.5052311
#> B 0.5257705
fixed <- hsmd_effect_sizes(dat, mT, sdT, nT, mC, sdC, nC,
cv_by=domain, probs=.5,
cv_benchmarks=grouped$cv_benchmarks)
stopifnot(isTRUE(all.equal(fixed$effect_sizes, grouped$effect_sizes)))Supplied grouped benchmarks need group names on rows and
probabilities in the requested column order. Ungrouped benchmarks are
numeric vectors named treatment and control
within the supplied list. Benchmarks must be positive and finite. Each
automatically estimated group needs observed CVs; supplying benchmarks
allows effect-size calculation when SDs are entirely unavailable. Use
hsmd_effect_sizes() if insufficient conventional effects
prevent pooling.
Missing SDs and analysis counts
incomplete <- hsmd_example_1
incomplete$sdT[1] <- NA_real_
missing_fit <- hsmd(incomplete, mT, sdT, nT, mC, sdC, nC,
study=Study, probs=.5)
missing_fit$summary[, c("method","est","n_effects","n_studies")]
#> method est n_effects n_studies
#> 1 SMD -0.3198513 15 15
#> 2 HSMD_50 -0.2491451 16 16The conventional fit has 15 effects; the harmonized fit has 16.
Original missing SDs and unrelated missing columns no longer remove
computable harmonized effects. Missing model predictors still follow the
selected missing policy. Benchmark estimation precedes
model-specific moderator filtering, so fitted models remain conditional
on that chosen benchmark population. To separate changes in the
standardizer from changes in study composition, repeat the analysis on
an explicit common subset, ideally using the same supplied
benchmarks.
Dependent outcomes
For correlated outcomes from the same participants, select the
multivariate structure and provide the assumed within-study outcome
correlation r2do. This example duplicates outcomes solely
to demonstrate model routing.
multi <- hsmd_example_1[rep(seq_len(nrow(hsmd_example_1)), each=2), ]
multi$Study <- rep(seq_len(nrow(hsmd_example_1)), each=2)
multi$effect <- rep(1:2, nrow(hsmd_example_1))
multi$mT <- multi$mT + ifelse(multi$effect==2, .5, 0)
multi$N <- multi$nT + multi$nC
multi$r2do <- .5
mv <- hsmd(multi, mT, sdT, nT, mC, sdC, nC, study=Study,
effectID=effect, sample_size=N, structure="DIAG1",
varcov_type="outcome", measure="g", probs=.5,
estimation_method="MLE")
mv$summary
#> method component est se ci.lb ci.ub tau2
#> 1 SMD 1 -0.3269361 0.07725749 -0.4783580 -0.17551421 NA
#> 2 SMD 2 -0.2556310 0.07809070 -0.4086860 -0.10257607 NA
#> 3 HSMD_50 1 -0.2550980 0.08780626 -0.4271951 -0.08300088 NA
#> 4 HSMD_50 2 -0.1907958 0.08833032 -0.3639200 -0.01767151 NA
#> avwW I2 n_effects n_studies
#> 1 0.04020362 NA 32 16
#> 2 0.04020362 NA 32 16
#> 3 0.04020291 NA 32 16
#> 4 0.04020291 NA 32 16
mv$heterogeneity
#> method component tau2
#> SMD.1 SMD 1 0.04305889
#> SMD.2 SMD 2 0.04507179
#> HSMD_50.1 HSMD_50 1 0.06806320
#> HSMD_50.2 HSMD_50 2 0.06952497outcome assumes equal arm sample sizes across a study’s
outcomes and a common correlation for outcome pairs. Unequal participant
overlap needs a richer covariance specification and is rejected by this
path. Singleton studies do not need a correlation. Sample sizes come
from the explicitly selected nT/nC columns
even if stale nt/nc columns exist in the
input.
For outcomes i,j on the same
participants, the working covariance is \operatorname{Cov}(d_i,d_j)=(1/n_T+1/n_C)r+d_i
d_jr^2/[2(n_T+n_C)]. For g, both axes are multiplied by their
correction factors, J_iJ_j. The
diagonal uses the selected effect’s sampling variance.
Non-positive-definite outcome covariance is rejected.
varcov_type="multilevel" supplies diagonal sampling
covariance; it does not represent correlated sampling errors merely
because a random-effects model is fitted.
Multilevel summaries and reporting
Nested models use structure="multilevel" and a formula
such as ~ 1 + (1 | district/StudyID), with
study=district. Inspect out$heterogeneity for
every named random-effect variance. Multivariate and multilevel
summary$tau2 and summary$I2 are deliberately
unavailable because variance components cannot be assigned to regression
coefficients by position. Full model objects retain their covariance
matrices and model-specific heterogeneity diagnostics. Inspect optimizer
and Hessian warnings before using uncertainty estimates; the HSMD
wrapper does not repair failed underlying fits.
A report should state the measurement scale, CV source/grouping, chosen quantiles, d/g convention, covariance assumptions, analysis counts, and changes across scenarios. State that uncertainty is conditional on benchmarks. Refit objects created before these repairs to obtain consistent covariance and summaries.