"reversed" p-values and negative CIs in Meta-analysis

[Sarah Marchina]

Hello everyone,
I am pretty new to STATA and I am definitely not a "coder" hence I use STATA 16 which has a GUI for meta-analysis (thanks for that!).
Originally, I ran a meta-analysis with 20 studies plus subgroup analyses with my STATA just fine, but my p-values were strangely reversed (CI crosses 1 p-value <0.05, CI doesn't cross 1 p-value > 0.05) and I had negative CI in the forest plot as well. When I used the same version of STATA (16) on our hospital's computer, I encountered in the set-up tab the "CI symmetry tolerance" field which I hadn't seen in either my version nor a friend of mine's version (which are both STATA 16). Anyhow, needless to say I wasn't able to run the analysis at all since I got an error saying "confidence intervals not symmetric".
My question, what do I do? I cannot change the data as that is what I extracted from the studies.

I found a post in this forum (06/11/2020) which suggests to convert to log odds-ratios, run the analysis and if necessary lower the threshold in this CI symmetry tolerance field. Thanks to the author providing the code I was able to do this and get a forest plot with Odds Ratios using the provided eform(Odds ratios) code, BUT, I was only able to run the overall analysis. The forest plot code doesn't seem to work when I try to run my subgroup analyses (2 and 4 groups respectively). Since I cannot use the GUI anymore and I am completely helpless when it comes to codes, I hope someone can provide me with the code for running a subgroup analysis with log odds ratios, but then creating a forest plot with Odds Ratios (showing the 2 or 4 groups). Also, I wasn't able to use the GUI for adding reference lines anymore and when I tried to edit the forest plot, the "adding reference line" option was greyed out and I couldn't use it either.

I would really appreciate your help with this! Please let me know if you need more information or clarification on my description. Thank you so much!!

[Jared Greathouse]

First show me your example data using dataex and the precise code you used

[Sarah Marchina]

Hi Jared,
This is my data example. I hope I did this correctly. I never used commands before and feel awfully clumsy. Sorry, I wasn't able to get all data points to show the same number of decimals, still working on figuring it all out.
The last row is the encoding for the 2 groups for the subgroup analysis I am trying to run (1 Volume; 0 Growth)


copy starting from the next line ------ ----------------

* Example generated by -dataex-. To install: ssc install dataex
clear
input double(OR CI95low CI95upper) str8 Volumeversusgrowth
1.123              1.021             1.273 "1"
1.05                .73              1.51 "0"
1.12               1.01              1.24 "0"
.75                 .6               .93 "1"
.035               .002               .51 "1"
1.13               1.03              1.24 "1"
1.96                  1              3.82 "1"
1.14                .93              1.41 "0"
.41                .19               .91 "1"
1.78               1.12              2.64 "0"
.142                .02              .996 "1"
3.062              1.196             7.839 "1"
1.03                  1              1.07 "1"
1.04               1.02              1.05 "1"
1.037              1.008             1.068 "0"
1.54               1.04              2.22 "0"
1.0235414534288638  1.002004008016032 1.044932079414838 "1"
1.016260162601626 1.0060362173038229 1.027749229188078 "1"
1.6               1.04              2.46 "1"
1.17               1.02              1.33 "0"
end

copy up to and including the previous line - ----------------


I did the initial analysis using the GUI in STATA16. When that didn't work I tried this code which I found in this forum (I attached the link in case it is easier to read there)

Hi Jae, If you can share your data or a portion of it, I would be happy to help solve your problem. Basically, whenever you are working with per-computed odds ratios (or risk ratios, hazard ratios, etc.), you have to provide your effect size and CI variables in the log metric. That is, in the control panel, under the -Setup- tab, you have to provide variables that contain log odds-ratios (NOT odds ratios) and CIs for the log odds-ratios. You can generate these variables for example by typing:

Code: 
 . generate double logor = log(<or_var>) . generate double cil     = log(<or_cilvar>) . generate double ciu    = log(<or_ciuvar>)

where you need to replace <or_var>, <or_cilvar>, and <or_ciuvar> by the names of the variables containing odds ratios, lower CI limit, and upper CI limit in your data set. You can then use variables, logor, cil, and ciu in the -Setup- tab. Alternatively, you can type:

Code: 
 . meta set logor cil ciu

The CI defined by variables cil and ciu should be symmetric. That is logor - cil should be almost the same as ciu - logor. If you have checked that the CIs are now symmetric but the above command produces an error, then you may relax the CI symmetry tolerance under the Setup tab (try values like .01 or .1 before you try 1). Alternatively, you can type:

Code: 
 . meta set logor cil ciu, civartolerance(.1)

If you type help meta set, check the last example in the help file that walks you through the same steps I did here (the ex works with risk ratios instead of odds ratios but the concept is the same). Finally, to construct a forest plot for the odds ratios, you can use option eform() in meta forestplot:

Code: 
 . meta forestplot, eform(Odds ratios)

I hope this solves your problem. If you still have an issue, please check the description of option civartolereance() in the documentation of the meta set command. Also, feel free to e-mail our tech support and they would be happy to help you further.

But that led to the mentioned problems in my original post.
Thank you for your help! I appreciate it a lot! Please let me know if I can provide more information!

best,
Sarah

[Andrew Musau]

I do not use this command, except if I am answering the odd question here on Statalist.

Originally, I ran a meta-analysis with 20 studies plus subgroup analyses with my STATA just fine, but my p-values were strangely reversed (CI crosses 1 p-value <0.05, CI doesn't cross 1 p-value > 0.05) and I had negative CI in the forest plot as well.

Wouldn't this be expected if you are reporting odds ratios? In the log-odds scale, a coefficient is significant if the confidence interval does not include 0. This value becomes 1 if you are looking at odds ratios as OR= exp(log odds) and exp(0)= 1

di exp(0)

Res.:

. di exp(0)
1

In any case, are data transformations necessary? If I want meta forestplot to display odds ratios (or risk-ratios), I would just specify the -rr- option. Note that if you run a command using the GUI interface, the resulting command line is displayed on the results window. You can copy this command line and add the option -rr- after the comma. Consider the following subgroup analysis, first reporting log-risk ratios and then reporting risk ratios.

webuse bcgset, clear
meta forestplot, subgroup(alloc)

webuse bcgset, clear
meta forestplot, subgroup(alloc) rr

[Sarah Marchina]

Hi Andrew, thanks for your insight! I agree with your first part, however, I was recommended to use log odds ratios for the analysis because our CIs are extremely asymmetric (I couldn't actually run it with ORs, it gave me an error). So, I again used the GUI to enter my logOR and CIs, and entered my subgroup encoding variable as well. After running the analysis, I used your code to try to get a forestplot for the ORs (not logORs), but I seem to be doing something wrong. It gives me the following error messages:

Code used:

meta forestplot, subgroup(Volumeversusgrowth) rr

option rr may only be specified when the effect size is log risk-ratio
r(198)

Code used:

meta forestplot, subgroup(Volumeversusgrowth) or

option or may only be specified when the effect size is log odds-ratio
r(198)


For my overall analysis (i.e. not subgroups) I used the
Code

meta forestplot, eform(Odds ratios)

which worked like a charm. What I don't know is how to use this for the subgroups, so that I get a nice graph like yours but displaying ORs.
Maybe I am thinking about this the wrong way? If you could help me out, I would be very grateful.
Thank you very much!

[Andrew Musau]

meta forestplot, eform(Odds ratios) subgroup(Volumeversusgrowth)

[Sarah Marchina]

Hi Andrew,
You are awesome! It worked! Only one more question if you allow.
I am attempting to get a reference line at 1 (x-axis), but for some weird reason I cannot use the option in the graph editor as it is greyed out. Is there an additional code to add to the above? Or is there a trick to get it to work in the graph editor?

Sorry for the basic questions. I am very grateful for your help!

best,
Sarah

[Andrew Musau]

What do you mean by greyed out? To add an xline at 1:

meta forestplot, subgroup(Volumeversusgrowth) eform(Odds ratios) xline(1)

[Sarah Marchina]

Fantastic! It worked like a charm! Thank you Andrew.
With greyed out I mean, the option is there but I cannot click it because it is greyed out (not sure what the proper term would be if there is one).
To give you some more info, once I create the forest plot, I can click on "graph" and there is an option called "reference lines", but it is not clickable. I also started the graph editor, but it still won't let me click the option reference lines. I am not sure what I am doing wrong or how to make these options appear black so that they can be clicked, but your piece of code just saved me. :-)

You know, if I look at the code you gave me, it seems so beautifully simple and clear. But honestly, I would have not been able to come up with that myself. I am very grateful for you help.
Thank you.