Admetan P for heterogeneity between groups

[Suhail Doi]

Hi David,

It seems that admetan in some cases returns a between groups heterogeneity P that is too small and a Q that is much larger than expected. Could you possibly please see if this is indeed the case?

Regards
Suhail

[David Fisher]

Hi Suhail,
Sorry, I'm just trying to keep on top of all my communications after the Christmas holidays, and I've only just found this message!
Could you give me an example, either with a sample dataset, or (sorry again if this is the case) by pointing me to previous correspondence on this matter?
Many thanks, and best wishes,
David.

[Suhail Doi]

Hi David,

No worries and happy new year!

This is an example data-set below:

* Example generated by -dataex-. To install: ssc install dataex
clear
input int(n1 cases1 noncases1 n2 cases2 noncases2) float qi byte prevg
1080 148 932 1073 120 953   5 0
  70  15  55   70  17  53 3.5 1
 241  37 204  257  36 221 3.5 0
 216  44 172  179  29 150   2 0
 168   9 159   82   4  78   1 0
 629 160 469  651 172 479   5 1
  63   4  59   61  17  44   4 1
  80  20  60   86  23  63 4.5 1
 106  35  71   97  27  70 3.5 1
 150   9 141  154   8 146 3.5 0
  92  18  74  100  19  81   4 0
  27   3  24   27   1  26   3 0
  76   9  67   63   7  56 3.5 0
  23   0  23   27   3  24 1.5 0
  27   2  25   22   1  21  .5 0
 350   7 343  371   9 362 4.5 0
  69  13  56   62  23  39   4 1
 144  20 124  125  27  98   4 1
 103   6  97  134   7 127 3.5 0
 116  11 105  116  19  97   4 0
  27   1  26   29   1  28   3 0
  56   6  50   58   7  51 2.5 0
 129  23 106  128  24 104 2.5 0
  33   7  26   28  16  12   3 1
  15   3  12   16   3  13 3.5 0
end

If I run the following code:

admetan cases1 noncases1 cases2 noncases2 , by( prevg ) ivhet   rr counts forestplot(astext(85) boxscale(35) spacing(0.95) leftjustify range(0.3 3)) coch

then admetan reports a between groups Q of 5.9 and p=0.015. The correct Q (MetaXL) seems to be 1.33 and p=0.249

Regards
Suhail

[David Fisher]

Hi Suhail,

I'm just trying to recreate this using MetaXL. I can't see a function to directly calculate the between-subgroup heterogeneity, is that right? If I calculate the Q statistic for each group separately, I obtain the same values as from admetan; that is, 7.9 for prev=0 and 19.5 for prev=1. Hence, the total subgroup heterogeneity is 27.4 and the overall heterogeneity is 33.3, giving a difference of 5.9 as reported by admetan.

This also makes intuitive sense to me, since both the IVHet weights for individual studies, and the overall pooled effect, are the same as for a fixed-effect analysis. Therefore, the Q statistics should also be the same. Do you propose a different way of calculating Q under the IVHet model? I can't see one in the main IVHet paper, but maybe I'm not looking in the right place.

Could you take me through your calculations, either numerically or in terms of MetaXL functions, so I can recreate your analysis?

Many thanks,

David.

[Suhail Doi]

Hi David,

Yes, we do not have a specific function but we do have the MACochranQ function so users can use =CHIDIST(Q,df) to get the P value in excel.

Regarding the difference we got, you used a Q-test to partition the variance, and test the between-subgroups portion of the variance and you are right, using this approach admetan and MetaXL match exactly. What I did was to use the Q-test to assess the dispersion of the summary effects about the combined effect which means do a meta-analysis of the two summary effects. This procedure is mathematically equivalent to the variance partitioning method under iv standard errors only. So if I run the iv model for the subgroups I get:

0.862	(0.749	0.992)
1.108	(0.957	1.283)

A meta-analysis of these two results gives a Cochran's Q of 5.899 and P=0.015 which exactly replicates the variance partitioning method.

However, if I use the QE or IVHet model subgroup results (QE below) then we get:

0.850	(0.565	1.279)
1.095	(0.934	1.285)

A meta-analysis of these two results gives a Cochran's Q of 1.282 and P=0.258 which is completely different from that under variance partitioning as additional variance over random error is ignored by the variance partitioning method.

I did not realize that variance partitioning was applied to all models and it seems to me that assessing the dispersion of summary effects may allow for the additional variance of newer models and even for the RE model (even though it continues to suffer from overdispersion). It of course does not matter when Q is computed for single groups as additional variance is only applied to summary effects.

What do you think about which is correct and apologies if I am missing something obvious here

Regards
Suhail.

[Suhail Doi]

Hi again David,

I just checked that with the random effects model admetan also returns a between Q of 5.9 and P=0.015 while it should be Q of 6.4 and P=0.011 so I think I understand whats happening here - the model weights are not being used for the between and that's why the results may differ. But then this wont work even if that was updated with robust models as there are no weights to update and even with the random effects model, after weights are updated, we still need to use the statistics reported using the iv weights to address the statistical significance of the total variance or the variance within groups.

Perhaps the easiest fix in the Cochran Q statistics table is keep the subgroup and overall as is but replace between by the computation from dispersion of summary effects and suppress the last row (F) so that not much needs to be changed in terms of programming.

Again apologies if I have misunderstood something about admetan!

Regards
Suhail

[David Fisher]

Hi Suhail,

I now understand, and I agree there is work to be done here. To be clear: at present, admetan follows metan in using fixed-effect variance partitioning for subgroups, regardless of the model used to estimate pooled effects. I have done some quick reading, and it is not clear to me that other software (e.g. metafor in R, RevMan, etc.) handles this in a consistent manner, so I am reluctant to make large-scale changes to admetan without further consideration. However, I can see that methodology has advanced in this area, and I will likely add some form of "dispersion" option in a future update. Of course, you should be able to perform the same calculations using the output from admetan as you have done using MetaXL, even if the default output isn't what you expect.

Thanks for pointing me in this direction!

Best wishes,

David.

[Suhail Doi]

Hi David,

Agree, this can be done but then the only option for the sub-grouped forest plot seems to be to use the nohet option as it seems that its not possible to edit the P value directly on the forest plot in Stata unless for example it is exported to a graphics program (e.g. Paint) and edited there.

Before we leave this topic, since we know that the between groups Q gives the same information as the Z(diff) (Q(bet) = [Z(diff)]^2) and if we perform the Z test for difference in subgroup means using the variances computed under the model, then why treat the Q(bet) differently?.

Regards
Suhail

[David Fisher]

Hi Suhail,

To answer your first question: editing can certainly be done without needing Paint! (or even the, rather clunky, Stata Graph Editor... ) See the penultimate example in the admetan help file. You could either edit the saved dataset manually before running forestplot, useopts ; or automate it all using macros or scalars.

And your second question: As I say, I don't necessarily disagree

Best wishes,

David.

[Suhail Doi]

Hi David,

That is a brilliant solution:

tempfile myfile
admetan cases1 noncases1 cases2 noncases2 , by( prevg ) ivhet   rr counts nogr saving(`myfile')
preserve
use `myfile', clear
list _LABELS
replace _LABELS = "Heterogeneity between groups: p = 0.258" in 34
forestplot, eform(RR) astext(85) boxscale(35) spacing(0.95) leftjustify range(0.3 3)
restore

Now problem is solved and no updates needed!!

Regards
Suhail

[Suhail Doi]

Addendum:

Got this query - thought best to clarify here for readers of the thread

Does it make a difference under what MA model the Q is generated from with MetaXL when considering dispersion of summary effects?

The answer is no because iv weights are required but then these are the only weights used in MetaXL (and admetan) for generating Q.

[Suhail Doi]

Hi David,

I was pleasantly surprised to note that the between subgroup het (aka test for interaction) was updated in metan 4.02 to allow for differences across models (as discussed in this thread) so all that programming we had to do for each forest plot is no longer necessary - a good reason to migrate to metan from admetan !

I also agree with your position in metan - I2 must be derived from Q regardless of model and I am not sure on what basis the Stata group have deviated from this fundamental principle

Regards
Suhail