I2 Residual in Meta-Regression

[noraadonnelly]

I am working on a meta-regression of a random effects meta-analysis with 53 studies. The overall I-Square in the meta-analysis is 79% with P=0.000.
To do the meta-regression I have taken all the aspects of the included studies that could have contributed to the heterogeneity (follow-up duration, study size, study population and period of the study etc).
I looked at each aspect individually (with the command metareg eff followup, wsse (SEeff)) then put them in a model (metareg eff smallvlarge-area, wsse (SEeff)). This model originally had an I-Square residual of 36.8% and an adjusted R Square of 58%.

I then removed covariates which had the highest p value one by one, this brought down the I-Square residual and increased the adjusted R Square. I’ve now a model with just 3 variables, all of which are significant (and remain significant with the permutation tests). The adjusted R Square is 100%, the I-Square residual is 34%, and the tau 2 is 0.

If I’ve understood the model correctly, the final model shows all the between-study variance is explained by the covariates (with an adjusted R Square of 100%) and there is no remaining between-study variance (the tau 2 is 0).

Is it odd that the I-Square residual, this is 34% if the adjusted R Square is 100%? Should this be 0%
Is there something wrong if, though I’ve explained all the between-study variance explained by the covariates, I am still left with an I-Square residual of 34%? I am guessing it’s not possible to account for all the between-study heterogeneity? Any advice would be greatly appreciated.

[Sven Trelle]

Hi Nora(?)
I cannot tell exactly from what you have written but this paper might be a starting point: Rücker G, Schwarzer G, Carpenter JR, Schumacher M. Undue reliance on I(2) in assessing heterogeneity may mislead. BMC Med Res Methodol. 2008 Nov 27;8:79. http://www.biomedcentral.com/1471-2288/8/79
The Stata Journal article accompanying the metareg command is also probably worth reading: The Stata Journal (2008) 8, Number 4, pp. 493–519.

Could it be an issue with the output format i.e. number of digits displayed? You might have a look at e(tau2) to see whether tau-square is really==0
The R² reflects reduction in tau-square. So if tau-square==0 it is always 100%

Best
Sven

[Paul T Seed]

I also find this odd. I would have expected i-sq, tau-sq and R-sq to be consistent. There may be differences in the way that I-sq allows for the extra variables in the meta-regression model, as compared to tau-sq and R-sq; which implies that I-sq may be incorrectly calculated.. Rounding or units of measurement would could give a false "perfect" result for tau-sq, but not R-sq.

The most likely explanation for your "perfect" results (100% R-sq, zero tau-sq) is that you have a "perfect" model, due to overfitting - you have fitted a large number of predictors and chosen the best. It is likely that to some extent you are fitting to the error, and not the signal. Similar problems come up with stepwise regression and for similar reasons. The result is distorted and exaggerated estimates of model performance. However, only in a situation like meta-regression or multi-level modelling, where there are two or more error terms, could you come up with a "perfect" model.

Best
Paul

[Sarah Roos]

Hi Paul,

Could you please explain more how is it possible that in meta-regressions 100% R-sq is possible? or possibly give a reference where I can study more about it! I'm doing a meta-regression and I have 100% R-sq but I don't know how to explain it!

Thank you, Sarah

[Adige Shapsugh]

Hi All this picture is what I get from the (Hedges et. al.) book named "An Introduction to Meta Analysis." This might give you an idea.