Metan vs. heterogi confidence intervals for I^2

[Peter Kurotschka]

Hello everybody,

I am performing several meta-analyses using the metan command in STATA v 18.0 for a systematic review ob observational studies focusing on prognostic factors. As most meta-analyses include 3-6 studies, the I^2 point estimate is unreliable, so that I wanted to include the confidence intervals of I^2 into the forestplots rather than (or in addition to) the p-values.

Here my command (example):

metan log_aOR log_aOR_lci log_aOR_uci if Outcome_regrouped_3==1 & Test_Final=="Diabetes", npts(N) random eform effect(adjusted OR) sortby (Year) forestplot(lcols (Study Year) favours (Risk reduced # Risk increased) fp(2) xlabel (0.7 1 2 4) astext (80) boxscale (100) diamopts (fcolor(dknavy)) leftjustify maxlines(1) null(1) nlineopts(lcolor(navy) lwidth(medthin) lpattern(dash_dot)) hlineopts(lpattern(solid)))

Question 1) Any suggestion on how to add the 95%CI of I^2 to the forest plots?

Question 2) cf. the command above:
For the outcome == 1 and the exposure "Diabetes" I have 4 studies in my dataset, D-L pooled OR=1.34 (95CI 1.06-1.69), I^2= 74.6%, p<.0001. Here is the metan output for heterogeneity:


---------------------------------------------------------
Measure | Value df p-value
---------------------+-----------------------------------
Cochran's Q | 23.61 6 0.001
| -[95% Conf. Interval]-
H | 1.984 1.000 3.581
I² (%) | 74.6% 0.0% 92.2%
---------------------------------------------------------


When I use the heterogi command to recalculate these confidence intervals, I get different results for the 95%CI.

heterogi 23.61 6
------------------------------------------------------
Statistic | Estimate [95% Conf. Interval]
----------+-------------------------------------------
H | 2.0 1.4 2.9
I^2 | 75 46 88
------------------------------------------------------

Why? seems to me that I am misinterpreting the I^2 95%CI of the metan command or are the CIs calculated with a different method?

Thanks in advance
Peter

[David Fisher]

Hi Peter,

As stated in the documentation for heterogi, the 95% CI is estimated using the "test-based" method of Higgins and Thompson (2002). There is an option, ncchi2, which instead uses the noncentral chi-squared distribution.

As for metan: re-reading the documentation now, I admit that it is far from clear (I am the maintainer and co-author). Briefly: if a fixed-effect model is specified, the noncentral chi-squared distribution is used (as in heterogi with the ncchi2 option). However, if a random-effects model is specified, the noncentral chi-squared distribution is generalized to incorporate the between-trial heterogeneity, which implies a Gamma distribution. This option is not available for heterogi. A suitable reference for this choice is: Hedges LV, Pigott TD. 2001. The power of statistical tests in meta-analysis. Psychological Methods 6: 203-217. doi: 10.1037/1082-989X.6.3.203

The heterogi command is fairly old now (my installation shows a date of 2006). More recently, personal communication with Julian Higgins, co-author of both metan and heterogi, showed a preference for the noncentral chi-squared (and its generalization, the Gamma) distribution over the older "test-based" method.

So to answer your question, yes the 95% CIs are calculated using different methods. But more importantly: I-squared estimation is indeed unreliable, and this is reflected in the wide confidence limits. Although authors such as Higgins and Thompson have suggested methods for generating 95% CIs (and although maintainers of code packages such as myself have implemented them), it is questionable how useful these figures actually are. In your own example, if you have 3 studies and an I-squared of 75%, then clearly those three studies are heterogeneous -- it should be obvious from a glance at the forest plot. The precise value of I-squared is of negligible importance, compared to how you choose to interpret such results. For example, does the heterogeneity appear to be qualitative or quantitative? Might the heterogeneity be explained by specific study characteristics such as case-mix or differences in measurement e.g. of outcome and/or intervention? Evidence synthesis in such situations can be tricky, and may need to be subjective or "hypothesis-generating".

Best wishes,
David.

[Peter Kurotschka]

Hi David,

Thank you very much for your very detailed answer and for pointing out the importance of interpreting heterogeneity rather than focusing solely on quantifying I².
Regarding my example (which included 7 studies, not 3/4 as I previously stated, apologies for the typo), one very large study appears to be an outlier, with the effect size going in the opposite direction.

We will likely report that we explored heterogeneity by visually inspecting the forest plots and conducting sensitivity analyses, such as leaving one study out, to better understand the results.

For my understanding, have I interpreted the `metan` output correctly below? The I² has a 95% confidence interval ranging from 0% to 92.2%?
Last question (even if for this one it won´t be very meaningful): you haven't implemented a metan option to show confidence intervals of I² in the forest plot, correct?

Thank you again for maintaining `metan`; I find this command incredibly useful.

Best,
Peter

Measure | Value df p-value
---------------------+-----------------------------------
Cochran's Q | 23.61 6 0.001
| -[95% Conf. Interval]-
H | 1.984 1.000 3.581
I² (%) | 74.6% 0.0% 92.2%
---------------------------------------------------------


Best
Peter

[David Fisher]

Hi Peter,

For my understanding, have I interpreted the `metan` output correctly below? The I² has a 95% confidence interval ranging from 0% to 92.2%?

Yes, that is correct. This is one of the reasons why CIs for I-squared are often not particularly useful: they are so imprecise that the bounds often hit 0% or 100%.

Last question (even if for this one it won´t be very meaningful): you haven't implemented a metan option to show confidence intervals of I² in the forest plot, correct?

Yes, that is correct, I have not.

However, metan and forestplot are designed to be flexible; and it is fairly straightforward to include a CI within the forest plot if you wish. Below is some example code, using the metan example dataset:

use http://fmwww.bc.edu/repec/bocode/m/metan_example_data, clear
metan tdeath tnodeath cdeath cnodeath, random label(namevar=id, yearvar=year) extraline(yes) nogr clear
local p_het = strofreal(r(p_het), "%5.3f")
local IsqLCI = strofreal(100*max(0, (r(Q_lci)-(r(k)-1))/r(Q_lci)), "%4.1f")
local IsqUCI = strofreal(100*min(1, (r(Q_uci)-(r(k)-1))/r(Q_uci)), "%4.1f")
local fmt : format _LABELS
replace _LABELS = subinstr(_LABELS, ", p = `p_het'", " [`IsqLCI'% - `IsqUCI'%], p = `p_het'", 1) in l
format `fmt' _LABELS
forestplot, useopts

To clarify: using the method-of-moments DerSimonian-Laird heterogeneity variance estimator, we have the relationship: I² = (Q - df)/Q, where df = number of studies - 1. Therefore, given a (non-central chi-square or Gamma) confidence interval for Q, we can derive a corresponding confidence interval for I².

Best wishes,
David.