[Hark Huang]

You are misinterpreting your -margins- output: these are not marginal effects, they are adjusted margins.

The output that you show simply gives the predicted probability of y in each of the four combinations of values of A and B. The p-values test the hypotheses that the probability of y is zero in each of those conditions. That is clearly a useless hypothesis, the test of which is of no interest to even the most ardent advocates of null hypothesis significance testing. The confidence intervals in that output are probably useful, as in most contexts we are interested in knowing the uncertainty associated with the probabilities of y in each combination. But testing the hypothesis that the probability that y = 0 is basically never of any use. (If I were rewriting the margins command, I would omit the p-value column from that output.)

If you want marginal effects, the command is:

margins A, dydx(B)
margins B, dydx(A)

Now, it is possible that when you run those commands you will encounter a difference in statistical significance between the logistic regression coefficients and the marginal effects, and in that case there is something to discuss about how to interpret them. If that happens and you are unsure what to make of it, post back.

Hi Clyde,

Thanks for the explanation. Just wondering is there any way in Stata that we can show the significant level of coefficient (of an independent variable) on it's adjusted margins?

For example

logistic DvEdu i.Gender i.Year
margins Gender, post
est store a1
esttab a1

The coefficient (odds ratio) of Gender is not significant in regression but the adjusted margins is significant. As you explained, they are test two different hypothesis. I'm wondering if there is any way when using esttab to present the adjusted margins, the significant level can be consistent with the original coefficient?

Best wishes

Hark

[Clyde Schechter]

I don't use -esttab- myself, so I can't say whether it can be made to do what you ask or not. If you are running version 17, the new -table- command almost surely can, though I haven't tried using it for that and am not really sure how it can be done.

All of that said, remember that if you are looking at the statistical significance of an adjusted margin, you are testing the null hypothesis that the adjusted probability of DvEdu in one (or both) genders is zero. If ever a null hypothesis were a straw man, this would be the occasion, would it not? To me it makes no sense to do this. The fact that people are tempted to do tests of this sort is one of the reasons I am a strong supporter of the American Statistical Association's position that statistical significance as a concept should be abandoned. See https://www.tandfonline.com/doi/full...5.2019.1583913 for the "executive summary" and https://www.tandfonline.com/toc/utas20/73/sup1 for all 43 supporting articles. Or https://www.nature.com/articles/d41586-019-00857-9 for the tl;dr.

[Scott Forrester]

You are misinterpreting your -margins- output: these are not marginal effects, they are adjusted margins.

The output that you show simply gives the predicted probability of y in each of the four combinations of values of A and B. The p-values test the hypotheses that the probability of y is zero in each of those conditions. That is clearly a useless hypothesis, the test of which is of no interest to even the most ardent advocates of null hypothesis significance testing. The confidence intervals in that output are probably useful, as in most contexts we are interested in knowing the uncertainty associated with the probabilities of y in each combination. But testing the hypothesis that the probability that y = 0 is basically never of any use. (If I were rewriting the margins command, I would omit the p-value column from that output.)

If you want marginal effects, the command is:

margins A, dydx(B)
margins B, dydx(A)

Now, it is possible that when you run those commands you will encounter a difference in statistical significance between the logistic regression coefficients and the marginal effects, and in that case there is something to discuss about how to interpret them. If that happens and you are unsure what to make of it, post back.

Dear
Clyde Schechter ,

Thank you very much for your clarification regarding predictive margins and average marginal effects. Judging from your statement that the hypotheses in predictive margins test the probability of y being zero, I infer the highly significant p-values should not be trusted. May I then ask what the point of predictive margins is? Can I rely on the interpretations/ visuals of predictive margins, or is it better I rely (only) on margins at representative values (can I trust the p-values there)? I ask because hypothesis testing and significance are important in my field.

Iteration 0:   log pseudolikelihood = -571.57912  
Iteration 1:   log pseudolikelihood = -532.23893  
Iteration 2:   log pseudolikelihood = -532.18749  
Iteration 3:   log pseudolikelihood = -532.18748  

Logistic regression                                     Number of obs =    825
                                                        Wald chi2(29) =      .
                                                        Prob > chi2   =      .
Log pseudolikelihood = -532.18748                       Pseudo R2     = 0.0689

                                  (Std. err. adjusted for 251 clusters in CASE)
-------------------------------------------------------------------------------
              |               Robust
   action | Odds ratio   std. err.      z    P>|z|     [95% conf. interval]
--------------+----------------------------------------------------------------
  return_numb |
           2  |   .6471243   .2042051    -1.38   0.168     .3486458    1.201133
           3  |   .6153148   .1985942    -1.50   0.132     .3268663    1.158309
           4  |   .6620303    .211529    -1.29   0.197     .3539205    1.238369
           5  |   .3662528   .1222703    -3.01   0.003     .1903771    .7046074
           6  |   .6113714   .1889009    -1.59   0.111     .3336582    1.120233
           7  |   .3931677   .1341579    -2.74   0.006     .2014319      .76741
           8  |   .7439665   .2535814    -0.87   0.386     .3814353    1.451062
           9  |   .7439864   .2591318    -0.85   0.396     .3759152    1.472448
          10  |   .3691542   .1152612    -3.19   0.001     .2001874    .6807364
              |
          T_C |
          TG  |   .6547502   .1168591    -2.37   0.018     .4614813    .9289602
              |
         risk |
           2  |   .8149563   .1437907    -1.16   0.246     .5766981    1.151649
           3  |   .7330664   .1372685    -1.66   0.097     .5078713    1.058115
           4  |   .7390198   .1588192    -1.41   0.159     .4849851    1.126118
              |
        round |
           2  |   1.535104   .3076627     2.14   0.032     1.036433    2.273704
           3  |   1.585242   .3189178     2.29   0.022      1.06869    2.351472
           4  |   1.536097   .3030808     2.18   0.030      1.04345    2.261339
              |
     1.reby_n |   1.011697   .1822325     0.06   0.949     .7107669    1.440037
              |
          REF |
           2  |   .5471091   .1748062    -1.89   0.059     .2924876    1.023388
           3  |   127.2733    5700.61     0.11   0.914     9.53e-37    1.70e+40
           4  |   .1204785   .0832186    -3.06   0.002      .031114    .4665129
              |
       device |
      Tablet  |   .3749142   .3347597    -1.10   0.272     .0651473    2.157581
  Smartphone  |   .7929138   .2907949    -0.63   0.527     .3864154    1.627037
              |
     system |
     Android  |   1.315018   .2658361     1.35   0.176     .8848258    1.954364
       Apple  |     3.6318   4.634496     1.01   0.312     .2977927    44.29247
              |
1.instruclick |   .9226272   .2002472    -0.37   0.711       .60295    1.411794
     correct1 |   1.362939    .365479     1.15   0.248     .8057907    2.305315
     correct2 |   .6414579    .132948    -2.14   0.032     .4273158    .9629138
     correct3 |   1.371108   .3108411     1.39   0.164     .8792195    2.138187
     correct4 |   .7983574    .145048    -1.24   0.215     .5591759    1.139846
 all4_correct |   1.545989   .6444779     1.05   0.296     .6829126    3.499835
        _cons |   2.621298   1.217175     2.08   0.038      1.05504     6.51274
-------------------------------------------------------------------------------

. margins,dydx(T_C ) at(return_numb =(1 2 3 4 5 6 7 8 9 10))

Average marginal effects                                   Number of obs = 825
Model VCE: Robust

Expression: Pr(action), predict()
dy/dx wrt:  1.T_C
1._at:  return_numb =  1
2._at:  return_numb =  2
3._at:  return_numb =  3
4._at:  return_numb =  4
5._at:  return_numb =  5
6._at:  return_numb =  6
7._at:  return_numb =  7
8._at:  return_numb =  8
9._at:  return_numb =  9
10._at: return_numb = 10

------------------------------------------------------------------------------
             |            Delta-method
             |      dy/dx   std. err.      z    P>|z|     [95% conf. interval]
-------------+----------------------------------------------------------------
0.T_C        |  (base outcome)
-------------+----------------------------------------------------------------
1.T_C        |
         _at |
          1  |   -.093954    .040058    -2.35   0.019    -.1724663   -.0154416
          2  |   -.098521   .0414953    -2.37   0.018    -.1798503   -.0171917
          3  |  -.0985711   .0414781    -2.38   0.017    -.1798666   -.0172756
          4  |  -.0984648   .0413876    -2.38   0.017     -.179583   -.0173465
          5  |  -.0932946   .0391296    -2.38   0.017    -.1699872    -.016602
          6  |  -.0985701    .041474    -2.38   0.017    -.1798575   -.0172826
          7  |   -.094603   .0400965    -2.36   0.018    -.1731908   -.0160152
          8  |  -.0978502   .0411933    -2.38   0.018    -.1785876   -.0171129
          9  |    -.09785   .0413172    -2.37   0.018    -.1788303   -.0168698
         10  |  -.0934485   .0395823    -2.36   0.018    -.1710284   -.0158685
------------------------------------------------------------------------------
Note: dy/dx for factor levels is the discrete change from the base level.

Or using predictive margins:

. margins, over (T_C) at(return_numb =(1 2 3 4 5 6 7 8 9 10))

Predictive margins                                         Number of obs = 825
Model VCE: Robust

Expression: Pr(action), predict()
Over:       T_C
1._at: 0.T_C
            return_numb =  1
       1.T_C
            return_numb =  1
2._at: 0.T_C
            return_numb =  2
       1.T_C
            return_numb =  2
3._at: 0.T_C
            return_numb =  3
       1.T_C
            return_numb =  3
4._at: 0.T_C
            return_numb =  4
       1.T_C
            return_numb =  4
5._at: 0.T_C
            return_numb =  5
       1.T_C
            return_numb =  5
6._at: 0.T_C
            return_numb =  6
       1.T_C
            return_numb =  6
7._at: 0.T_C
            return_numb =  7
       1.T_C
            return_numb =  7
8._at: 0.T_C
            return_numb =  8
       1.T_C
            return_numb =  8
9._at: 0.T_C
            return_numb =  9
       1.T_C
            return_numb =  9
10._at: 0.T_C
            return_numb = 10
        1.T_C
            return_numb = 10

------------------------------------------------------------------------------
             |            Delta-method
             |     Margin   std. err.      z    P>|z|     [95% conf. interval]
-------------+----------------------------------------------------------------
     _at#T_C |
       1#CG  |   .6583481   .0521328    12.63   0.000     .5561697    .7605265
       1#TG  |   .5612762     .05893     9.52   0.000     .4457755    .6767769
       2#CG  |    .561895   .0519277    10.82   0.000     .4601185    .6636715
       2#TG  |   .4600941   .0560303     8.21   0.000     .3502767    .5699115
       3#CG  |   .5502696   .0558427     9.85   0.000       .44082    .6597193
       3#TG  |    .448428   .0541197     8.29   0.000     .3423554    .5545007
       4#CG  |   .5671264   .0551948    10.27   0.000     .4589466    .6753061
       4#TG  |   .4653793   .0529944     8.78   0.000     .3615122    .5692465
       5#CG  |   .4296472   .0589794     7.28   0.000     .3140498    .5452447
       5#TG  |   .3334668   .0571017     5.84   0.000     .2215496    .4453839
       6#CG  |   .5487826   .0525505    10.44   0.000     .4457856    .6517796
       6#TG  |   .4469436    .054987     8.13   0.000     .3391711    .5547161
       7#CG  |   .4459835   .0638217     6.99   0.000     .3208953    .5710717
       7#TG  |   .3484162   .0589294     5.91   0.000     .2329168    .4639157
       8#CG  |   .5936784   .0570543    10.41   0.000     .4818541    .7055028
       8#TG  |   .4925525   .0586106     8.40   0.000     .3776778    .6074272
       9#CG  |   .5936844   .0605536     9.80   0.000     .4750016    .7123673
       9#TG  |   .4925587    .064427     7.65   0.000     .3662841    .6188334
      10#CG  |   .4314587   .0569875     7.57   0.000     .3197653     .543152
      10#TG  |   .3351153   .0522692     6.41   0.000     .2326694    .4375611
------------------------------------------------------------------------------

I would be most grateful for your input.

Thank you very much in advance!

[Clyde Schechter]

Judging from your statement that the hypotheses in predictive margins test the probability of y being zero, I infer the highly significant p-values should not be trusted.

There is a certain irony to my responding to this because I believe there is only very limited use for p-values and I dislike statistical significance as a concept. But putting myself in the shoes of somebody who does support those things, I would say this. The p-values associated with testing the probability of y being 0 are no more or less trustworthy than the p-values associated with marginal effects. The problem is that the null hypothesis that y = 0 in this situation is almost never of any interest or importance.

May I then ask what the point of predictive margins is?

Not all research is done to test hypotheses about the effect of x on y. In many research programs, we are interested in finding estimates of the frequency with which some outcome occurs (or estimates of the mean value of some continuous outcome) in our target population as a whole, or in selected subpopulations, or conditional on certain values of some explanatory variables. This is what predictive margins provide. We would generally ignore the p-values as, if we could seriously entertain the null hypothesis that the frequency is zero, we certainly wouldn't study it with a sample size of only 825. But we would be very interested in the estimate itself and the confidence interval around the estimate.

Moreover, in studies where the goal is to study the effect of x on y, in addition to testing some hypothesis about the marginal effect of x, we would generally also be interested in the predictive margins of y conditional on key values of x, and, depending on circumstances, we might be interested in the unconditional predictive margin of y.

Can I rely on the interpretations/ visuals of predictive margins, or is it better I rely (only) on margins at representative values (can I trust the p-values there)?

You can rely on them to the same extent you can rely on any statistical results. The question is not their reliability, it is their relevance to answering your research questions.

[Scott Forrester]

Dear Clyde Schechter

Thank you very much for your informative reply!

The problem is that the null hypothesis that y = 0 in this situation is almost never of any interest or importance.

May I please clarify: Does this mean that predictive margins test the null hypothesis that no action (at all) was taken as opposed to action =1 (an action was taken), where action is the binary dependent variable, which measures whether an action in the portfolio was taken.

We would generally ignore the p-values as, if we could seriously entertain the null hypothesis that the frequency is zero, we certainly wouldn't study it with a sample size of only 825.

I'm sorry to ask again, does frequency = 0 mean that action is zero - i.e., no one in my sample takes an action in their portfolio.

Regarding the confidence intervals, we seek tighter intervals around the coefficient and no overlap in order to draw meaningful conclusions about the data?

Thank you very much for your time and knowledge!

[Clyde Schechter]

When I refer to the null hypothesis y = 0, I mean, in the context of your study, a hypothesis that says the probability that anybody takes an action is zero; nobody takes an action.

As to your last question, you are evaluating an intervention here: you have treatment and control groups. And you work in the null hypothesis significance testing framework. So you would test your hypothesis by looking at the p-value for the single T_C variable. Equivalently, you could use the confidence interval to do hypothesis testing by seeing whether it excludes 0.

What you describe in "Regarding the confidence intervals, we seek tighter intervals around the coefficient and no overlap in order to draw meaningful conclusions about the data?" is true in certain contexts, but is not the whole story. For example, if I am evaluating an intervention effect, and not using null hypothesis significance testing, I would also look at the T_C variable's coefficient, and I would also look at its confidence interval. But I would not be particularly interested in whether or not it excludes 0. Rather, I would look at the upper and lower limits of the confidence interval and ask whether, in practical, substantive, real-world terms, I would consider the effect to be large enough to call meaningful at both ends of the interval. If so, I would consider the intervention to have shown a meaningful effect in my study. If both ends of the confidence interval lay in territory where an effect of that size would not be meaningful in real-world terms, I would call the intervention ineffective. And if the two ends of the interval gave opposite impressions: one would constitute a meaningful effect and the other would not, then I would call the study inconclusive.

And, to anticipate a question about what I mean by real-world meaningfulness, my answer is that this is a judgment call. In studies where I am working solely as a data analyst and do not have deep content knowledge, I leave that judgment to others. In studies where I do have content expertise, I use my own judgment. Either way, in publication I would show the estimate and the confidence interval, and while I would also indicate my conclusion, it is up to my reader(s) to apply their own judgment and draw their own conclusions as well. The kind of judgment applied boils down to something like this: given the cost and burden of implementing the intervention, is the result big enough to justify it? If not, is it close enough that one might want to try developing some alternative intervention that could produce similar results but with lesser cost and burden (assuming that such developing appears feasible)? It's a different philosophy from null hypothesis significance testing.