
Higgins design-by-treatment incoherence tests
network_design_incoherence.RdIntended question: is there disagreement across treatment designs anywhere in the network? Assess whether treatment means can be described by a coherent network after accounting for the specified sampling covariance and treatment heterogeneity. Available for all three joint network interfaces.
Usage
network_design_incoherence(
object,
method = c("wald", "conditional", "ML"),
adjust = c("none", "holm", "bonferroni"),
local_covariance = c("gls", "observed")
)
# S3 method for class 'nma_design_incoherence'
print(
x,
...,
digits = getOption("digits"),
estimable_only = TRUE,
detailed = FALSE
)
# S3 method for class 'nma_design_incoherence'
summary(
object,
...,
digits = getOption("digits"),
estimable_only = TRUE,
detailed = FALSE
)Arguments
- object
A fitted joint network model from
network_meta()ornetwork_meta_coherent(); forsummary(), a result returned bynetwork_design_incoherence().- method
wald(default) for the Higgins/White design-by-treatment Wald test with heterogeneity refitted under the alternative by REML;conditionalfor a Wald test with fitted covariance fixed; orMLfor a likelihood-ratio test with both models refitted by ML.- adjust
Multiplicity correction for the local design-deviation tests:
none,holm, orbonferroni. All requested pairs form the family, including nonestimable hypotheses. The global test is separate and remains unadjusted.- local_covariance
Covariance for local symmetric side-splitting Wald tests:
gls(default) conditions on fitted heterogeneity;observeduses the mean-coefficient block of the inverse joint observed information, including heterogeneity uncertainty, to reproduce Stata'snetwork sidesplitdefault calculation (White, 2015).observedrequiresmethod="wald". It affects local SEs, confidence limits, Wald statistics and p-values; gamma estimates, heterogeneity fits and the global test are unchanged.- x
A result from
network_design_incoherence().- ...
Additional print arguments forwarded to
print.data.frame(); reserved additional arguments for summary.- digits
Decimal places to display, an integer from 1 to 22. Defaults to
getOption("digits"); stored values retain full precision. Variance-component labels are preformatted diagnostic strings. P-values below 0.0001 print as<0.0001; other p-values use at least four decimal places.- estimable_only
Logical; show only estimable tests (default
TRUE).- detailed
Logical; show all diagnostic columns (default
FALSE). Combine withestimable_only=FALSEto inspect unavailable tests and reasons.
Value
A list of class nma_design_incoherence containing:
- global
One-row table with
statistic,df,p_value, andestimable.- local
The same fields for each treatment pair, plus
comparison. A local row can jointly test several mean-stratum parameters.p_adjustedcontains multiplicity-adjusted p-values. For one estimable coefficient,gamma,se_gamma,ci_lower, andci_uppergive the signed departure, its model-based SE and pointwise 95% interval;contraststates direction.- local_coefficients
One row per pair and mean stratum, including
gamma,se_gamma,ci_lower,ci_upper, andestimable. Unavailable coefficients have missing estimates.- local_covariance
Requested local coefficient covariance:
glsorobserved. Fixed heterogeneity makes the two calculations identical.- adjustment
Local multiplicity method, family description and size.
- method, assumption
Descriptions of the calculation and interpretation.
With method="wald" or method="ML", test tables also contain null_heterogeneity,
alternative_heterogeneity, convergence codes (zero indicates convergence), and
reason. Scalar variance components are labelled; unstructured matrices are
flattened in column-major basic-effect coordinate order. These strings are
summaries, not confidence intervals. Nonestimable tests have estimable=FALSE
and missing statistics/p-values; this is different from failing to reject a null.
Printing shows the global test and estimable local tests, with shared settings
reported once. Constant local df is printed above the results. With no local
adjustment, the duplicate adjusted-p column is omitted. Alternative-model
heterogeneity, convergence codes and nonestimability reasons remain in the
returned tables. Use print(result, detailed=TRUE, estimable_only=FALSE) to
inspect all diagnostic fields and rows. Display filtering does not change the
multiplicity family. summary(result, digits=3) remembers display preferences;
explicit print arguments override them. No stored values are rounded.
Details
Printed output states the research question and global null hypothesis that all identifiable design-by-treatment interaction parameters are zero. The local section separately states that each pair's identifiable targeted deviation parameters are zero under the symmetric side-splitting model (White, 2015).
The null mean model is \(E(y_i)=X_i d\). The alternative adds columns
\(W_i\gamma\); the null hypothesis is that all identifiable added
coefficients are zero. Covariance remains sampling covariance plus the selected
signed random-treatment covariance. Only columns increasing the mean-design rank
are retained. Thus df is the number of independently testable added parameters,
not the number of graphical triangles.
Global test. A study design is the set of treatments observed in that
study, across its rows. The function interacts design indicators with the basic
treatment-effect design Z. Mean strata receive separate interactions when
specified. Existing moderator slopes remain in the mean model; this extension
does not add design-specific moderator slopes. The joint test assesses
between-design disagreement. Within-study multi-arm contrast identities are
preserved, so a triangle supplied solely by one common multi-arm design does not
create an independent incoherence parameter.
Local tests. For each unordered treatment pair A/B, eligible designs contain both treatments. Treatment B is perturbed by +1/2 and A by -1/2 in those studies; other treatments are unchanged. The perturbation of a contrast is the second treatment's shift minus the first's. One added column is constructed per mean stratum. These are White's (2015, section 3.6, Table 1) symmetric side-splitting tests, preserving multi-arm identities. They are not independent direct/indirect splits, and a significant result does not uniquely identify a problematic comparison or study. Qin et al. (2024, equation 3) describe the symmetric side-splitting model and its restrictions relative to the full design-by-treatment interaction model.
The local mean model is \(E(y)=Xd+w_{AB}\gamma_{AB}\). The sign of gamma
follows treatment B minus A, shown in contrast. For a single estimable
coefficient, local reports gamma, se_gamma, ci_lower, and ci_upper.
The intervals are pointwise 95% normal Wald intervals, regardless of adjust
or the primary fit's confidence level. With augmented matrix A and fitted total
covariance V, GLS coefficient covariance is \((A^T V^{-1} A)^{-1}\). Its added
coefficient block supplies gamma's covariance with local_covariance="gls";
heterogeneity estimation uncertainty is then not additionally propagated.
For multiple mean strata,
local_coefficients reports each retained gamma separately while local
retains the joint test. Scalar gamma fields are missing when more than one
coefficient is tested. A likelihood-ratio test still reports Wald coefficient
intervals, which need not agree with its p-value.
Refitted Wald calculation. Fit the augmented design-by-treatment mean
model by REML and jointly test its identifiable added coefficients using their
model covariance and a chi-squared reference. Heterogeneity is estimated under
this alternative, unless fixed through tau2 or heterogeneity="none".
This follows Higgins et al. and White et al. (2012). Failure to identify or
estimate the alternative is reported as nonestimable, rather than as agreement.
Observed-information local covariance. With method="wald" and
local_covariance="observed", let theta contain the free heterogeneity
covariance parameters, V=V(theta), and A the augmented local mean matrix.
The joint observed information is the Hessian, with respect to both beta and
theta, of
$$\tfrac12[\log|V|+(y-A\beta)^T V^{-1}(y-A\beta)+\log|A^T V^{-1}A|].$$
It is evaluated at the REML heterogeneity estimate and its GLS beta estimate.
The beta block of its inverse supplies the local coefficient covariance,
retaining the cross-information between mean and heterogeneity parameters.
This is not the Hessian of the profiled REML criterion with respect to beta.
gls instead inverts only the mean-information block. The observed option is
model-based and asymptotic, not a small-sample correction or a sandwich variance.
Pointwise 95% intervals still use normal quantiles and Holm still adjusts only
local p-values. White's Stata mvmeta default includes heterogeneity uncertainty;
its nouncertainv option describes the alternative conditional calculation.
The observed calculation supports the fitted linear variance-component kernels
and interior unstructured heterogeneity covariance. Fixed tau2 and
heterogeneity="none" have no estimated variance parameters, so both covariance
options reduce to GLS. An estimated variance on a boundary, singular information,
or non-positive-definite joint information makes the local test unavailable:
estimable=FALSE and reason explains why. No automatic GLS fallback or
positive-definiteness repair is performed. Use detailed=TRUE and
estimable_only=FALSE to inspect such rows. The default global GLS Wald test
is unaffected; conditional and ML tests reject the observed option explicitly.
Conditional calculation. Refit the augmented mean by generalized least
squares using the original fitted total covariance. Compute
\(Q=\hat\gamma^T\operatorname{Var}(\hat\gamma)^{-1}\hat\gamma\) and use a
chi-squared reference distribution with df degrees of freedom. This conditions
on fitted heterogeneity and does not propagate its estimation uncertainty.
ML calculation. Refit both null and alternative using ML, even when
the primary fit used REML. Calculate \(D=2(\ell_{alternative}-\ell_{null})\)
and compare with chi-squared(df). Heterogeneity is re-estimated unless the
original model fixed it through tau2 or heterogeneity="none". REML likelihoods
are not compared across different mean designs. Alternative optimization or
identifiability failures are marked in the output.
Interpretation. A small p-value indicates evidence against the specified
coherent mean model, conditional on its covariance and effect-modifier
assumptions. It does not measure the size or clinical importance of disagreement.
Use network_nodesplit() to examine direct-minus-indirect effects and intervals,
and network_influence() to investigate sensitivity. Large p-values can reflect
low power; they do not establish coherence or transitivity. Global and local
reference distributions are asymptotic. Local p-values are not multiplicity
adjusted unless adjust is supplied, and tests use model-based covariance even
for a primary CR1 fit. Local tests overlap: their statistics and df cannot be
summed to obtain within-design heterogeneity. Use network_q_decomposition().
References
Higgins, J. P. T., Jackson, D., Barrett, J. K., Lu, G., Ades, A. E., and White, I. R. (2012). Consistency and inconsistency in network meta-analysis: concepts and models for multi-arm studies. Research Synthesis Methods, 3, 98–110. doi:10.1002/jrsm.1044 .
White, I. R., Barrett, J. K., Jackson, D., and Higgins, J. P. T. (2012). Consistency and inconsistency in network meta-analysis: model estimation using multivariate meta-regression. Research Synthesis Methods, 3, 111–125. doi:10.1002/jrsm.1045 .
White, I. R. (2015). Network meta-analysis. The Stata Journal, 15(4), 951–985. Section 3.6, Table 1. doi:10.1177/1536867X1501500403 .
Qin, L., Zhao, S., Guo, W., Tong, T., and Yang, K. (2024). A comparison of two models for detecting inconsistency in network meta-analysis. Research Synthesis Methods, 15(6), 851–871. doi:10.1002/jrsm.1734 .
Examples
d <- data.frame(study=1:9, a=rep(c("A","A","B"),3),
b=rep(c("B","C","C"),3), effect=rep(c(.2,.6,.2),3), variance=.04)
fit <- network_meta(d,"study","a","b","effect","variance")
network_design_incoherence(fit)$global
#> statistic df p_value estimable null_heterogeneity alternative_heterogeneity
#> 1 1 1 0.3173105 TRUE study 0 study 0
#> null_convergence alternative_convergence reason
#> 1 0 0
network_design_incoherence(fit,method="ML")$local
#> statistic df p_value estimable null_heterogeneity alternative_heterogeneity
#> 1 1 1 0.3173105 TRUE study 0 study 0
#> 2 1 1 0.3173105 TRUE study 0 study 0
#> 3 1 1 0.3173105 TRUE study 0 study 0
#> null_convergence alternative_convergence reason comparison contrast gamma
#> 1 0 0 A vs B B - A -0.2
#> 2 0 0 A vs C C - A 0.2
#> 3 0 0 B vs C C - B -0.2
#> se_gamma ci_lower ci_upper p_adjusted
#> 1 0.2 -0.5919928 0.1919928 0.3173105
#> 2 0.2 -0.1919928 0.5919928 0.3173105
#> 3 0.2 -0.5919928 0.1919928 0.3173105
network_design_incoherence(fit,local_covariance="observed")$local
#> statistic df p_value estimable null_heterogeneity alternative_heterogeneity
#> 1 1 1 0.3173105 TRUE study 0 study 0
#> 2 1 1 0.3173105 TRUE study 0 study 0
#> 3 1 1 0.3173105 TRUE study 0 study 0
#> null_convergence alternative_convergence reason comparison contrast gamma
#> 1 0 0 A vs B B - A -0.2
#> 2 0 0 A vs C C - A 0.2
#> 3 0 0 B vs C C - B -0.2
#> se_gamma ci_lower ci_upper p_adjusted
#> 1 0.2 -0.5919928 0.1919928 0.3173105
#> 2 0.2 -0.1919928 0.5919928 0.3173105
#> 3 0.2 -0.5919928 0.1919928 0.3173105