[Elise Sobrie]

Dear all

After reading a couple of times your posts, I understand what you are saying. However, it is not clear to me whether to follow the p-values of the margins or those from the coefficients in the Original model. I know different hypotheses etc. are beign tested, but just one clear answer which to follow?

Thank you in advance!

[Richard Williams]

As I already stated, my preference is to use the p-values from the original coefficients. I think that is also what is more common. Many analyses do not even present adjusted predictions or marginal effects, but you just about always have the coefficients.

But I am not the ultimate authority on such things, so you should decide what is best for you given what you want to test.

[Evelyn Mare]

I have encountered a similar problem. My logit models suggests a p value of <.05 and my logit model a p value >.15

As I already stated, my preference is to use the p-values from the original coefficients. I think that is also what is more common.

I would lean towards presenting the AMEs, because they are easier to interpret, yet I am wrangling with the insignificance.

 mi est, post: svy: logit ro i.d_e i.pel  d_pov i.p_bsc i.p_bnvq_s4  i.involv_s4 i.p_sy_s4  c.p_age_sample_s1##c.p_age_sample_s1 i.p_sex_s4
mimrgns , dydx(i.pek ) cmdmargins predict(pr)

  
  Logit coefficients     
Multiple-imputation estimates
Imputations
=
30

Survey: Logistic regression
Number of obs
=
4,460

Number of strata = 3
Population size
=
4,549.841

Number of PSUs = 200

Average RVI
=
0.3858

Largest FMI
=
0.9258

Complete DF
=
197

DF adjustment: Small sample
DF: min
=
2.31

avg
=
175.31

max
=
195.02

Model F test: Equal FMI
F( 34, 157.1)
=
90.89

Within VCE type: Linearized
Prob > F
=
0.0000





ro
Coef. Std. Err.
t
P>t
[95% Conf.
Interval]














d_pel

stage1
1.556983 1.072342
1.45
0.148
-.558007
3.671973

stage2
-.5627057 .9420055
0.60
0.551
-2.420539
1.295127

stage3
2.444653 1.197903
2.04
0.043
.0821298
4.807177

stage4
.3289372 1.40129
0.23
0.815
-2.434705
3.092579

stage5
-.6252651 1.675261
0.37
0.709
-3.929289
2.678759


Code:
  
  Margins     


dy/dx Std. Err. t
P>t [95% Conf. Interval]



d_pel

stage1
.1470687 .1426974 1.03
0.304 -.1343723 .4285098

stage2
-.0241331 .0347124 -0.70
0.488 -.0925933 .0443271

stage3
.2947183 .218825 1.35
0.180 -.1368519 .7262885

stage4
.0199796 .0946079 0.21
0.833 -.1666073 .2065665

stage15
-.0261499 .0563228 -0.46
0.643 -.1372328 .084933



Note: dy/dx for
factor levels is the discrete
change from the base level.

[Edmondo Ricci]

Clyde Schechter

Dear Clyde,

Thank you for your post. Since tests on marginal effect and coefficients are different tests, I can see statistical significance can be different.

But would it be even possible that coefficients are positive and marginal effect is negative?

I managed to make a dataset where I get following results.

1. reg => positive
2. ivreg => positive
3. probit coefficient => positive
4. probit marginal effect => positive
5. ivprobit coefficient => positive
6. ivprobit marginal effect => negative

This is strange in so many levels. #5 and #6 having different sign is strange. #2 and #6 having different sign is strange.

And if this is possible, in which situation would it occur?

And if this is possible, shouldn't there be somewhere in the range of x such that ivprobit marginal effect is positive?

And how should I interpret this? Is X causing increase in Y or decrease in Y? All others are positive & significant. Only marginal effects from ivprobit is negative (and sometimes significant)

[Clyde Schechter]

I'm unable to respond to your specific question because I do not use instrumental variables in my work and I have only a very limited understanding of how they work.

I can say that with linear models, the marginal effect is equal to the coefficient. With non-linear models, the marginal effect must be conditioned on particular values of the predictor variables (or averaged over the distribution of the predictor variables) and can vary considerably, whereas regression coefficients are unconditional. So there is no necessary type of agreement between a regression coefficient and all of the infinitely many marginal effects associated with that variable.

[Edmondo Ricci]

Clyde Schechter

Dear Clyde,

Thank you again

[Teresa Schützeichel]

Following this thread, I am still confused as to why the significance levels of the ivprobit original coefficients and those of its average marginal effects should not be identical. I ran an ivprobit which returned positive and significant coefficients (varying p-values for the different specifications), however the corresponding average marginal effects after running margins, dydx(*) predict (pr) were all positive and insignificant (p>0.1 for all specifications).
The other thing is the command 'margins, dydx (*) predict (pr)' returns the average marginal effects of the instrumental variable as well, ideally this should not happen..any idea why this is the case?
Thanks in advance.

[Richard Williams]

Teresa, welcome to Statalist.

i suggest reading the Statalist FAQ, esp. point 12 about asking Qs effectively. Showing exactly what you typed and how Stata responded can make it easier to see what you are talking about.

Also, while I might add a link to old threads, I like to start a new thread rather than add on to a long older one. If something has 20+ posts I usually don’t want to go to the trouble of getting up to date on what has already been talked about.

i personally do not get too surprised about difference in significance levels between coefficients and marginal effects. Marginal effects can be computed in many ways, e.g. atmeans, asobserved, or at values chosen by the user. These different ways might produce different significance levels. I usually just focus on the significance of the coefficients.

[Scott Forrester]

Clyde Schechter

3. Failure to take into account that the coefficients and odds ratios are different metrics from the marginal effect on probability. The non-linearity of these models then produces "paradoxical" results. In fact, in many situations, if you run -margins, dydx(x) at(x = (numlist spanning a wide range of values)) you will get an interesting mix of large and small, "significant" and "non-significant" marginal effects. Drawing a graph of the logistic curve and pointing out that it has a steep section in the middle and flat sections at the ends probably makes the point better than any number of words and sentences. So a unit increment in x may correspond to a rather large increase in predicted outcome probability if we are starting out in the steep area, and a barely visible increase if we are out at the far ends. Once again, due to the noise in our data and sampling variation, we are estimating these marginal effects with a certain degree of precision, and some, but not all, marginal effects will be large enough that we can bound them away from zero at that degree of precision. Just where we are on the logistic curve isn't always obvious from looking at the regression output or the marginal effects, as it depends on the sample distributions of the predictor variables too. The predicted probability for the sample as a whole, or with all variables set at their means, can be helpful for figuring that out. Once you know where you are on the curve, it is easier to see graphically why a marginal effect might be surprisingly large or small in the face of a particular logistic regression coefficient.

Dear Clyde, I'm sorry I seem to comment lots on your post. May I please kindly ask for your advice, especially regarding your phrase "The non-linearity of these models then produces "paradoxical" results."
In my code from this post (https://www.statalist.org/forums/for...on%C2%A0/page2) I have insignificant logit coefficients but highly significant average marginal effects. For example, not every return_numb is significant in my logit regression, but every average marginal effect is.
How do I best answer questions that deal with this conundrum? Can I really say it's due to the non-linearity of my model/ data/ research question (and hence the use of the logit model because of my binary dependent variable)?

How do I best report these results if my supervisor is an avid fan of significance and hypothesis testing? Do I show the whole logit regression (even if insignificant) and then those margins on themes/topics that I find interesting in hopes of the margins making up for the insignificance of the logit regression?

Thank you very much in advance!

I greatly appreciate your advice and support!

[Clyde Schechter]

I'm sorry I seem to comment lots on your post.

No apologies called for. It's obviously a topic that's of importance to you, highly relevant to a project you are working on.

"The non-linearity of these models then produces "paradoxical" results."

While this is true, I think in the case of the model you are referring to from your other thread at https://www.statalist.org/forums/for...on%C2%A0/page2, this is not much in play. So I'm not going to elaborate on this part of it now. While it would probably help your professional development to work through this, it's nearing the end of my work day and I'd rather focus my effort on what is more relevant to your immediate problem now.

I have insignificant logit coefficients but highly significant average marginal effects. For example, not every return_numb is significant in my logit regression, but every average marginal effect is.

The key thing to understand is that logit coefficients and average marginal effects are not just different expressions of the same thing. They are different things. The logit coefficients are numbers that you can use to write down a formula that predicts the probability of outcome action in terms of the values of the variables in your model. To say that one of them is statistically significant is simply to say that if you were to rerun your study with a bunch of other samples and look at the coefficients of the same variable in each sample, you might well find that they center around zero, and some could well be of the opposite sign. The logit coefficient is one way of expressing how changes in a predictor variable affect the outcome probability. But they do it very indirectly. They are the logarithms of odds ratios. So if you exponentiate the coefficient, you get the odds ratio, which tells you by what factor, according to your model, the odds (not the probability) of the outcome will be multiplied (not incremented/decremented) when the predictor/explanatory variable is incremented by 1.0. Now, this has been around for a long time and many people are comfortable with it just because of repeated exposure, and they think they understand it. Many people, however, don't really understand it. If you think about it, it's a pretty unnatural way to think about it.

The average marginal effects are different beasts. To calculate these, the model's predicted probability for action is calculated for every observation in the data set twice: once using x = 0 and again using x= 1 (x here refers to one of your return_num variables, or any other 0/1 indicator variable.) Then the average predicted values at x = 0 and x = 1 are calculated from those results, and the difference is subtracted. So this number is actually the model's prediction of the actual difference in probability of action, on average, in your data set between x = 1 and x = 0. It is, to my mind at least, a much more natural expression of the effect of a variable. It's not abstract; it's the change in outcome probability you would actually observe if everybody changed from x = 0 to x = 1. It's an increment/decrement, not a ratio. It takes into account the joint distribution of all of the variables in the model.

Given that they are two rather different things, you can't expect them to resemble each other all that closely. It is fair to insist that they should at least have the same sign. And they always do. But the similarity ends there. If you look at your outputs in #18 of the other thread, you can see that the odds ratios vary appreciably: the largest is more than twice the smallest. But the average marginal effects are nearly all the same. The odds ratios are not statistically significant, but the average marginal effects are. There is nothing really surprising about any of this. And it's not that one is right and the other is wrong: they are both correct, but they are describing different things that are only indirectly and somewhat distantly related to each other.

So which should you choose? Well, basing it on which gives you statistically significant results is not science. In fact, if you do it knowing that it is not science, some would regard it as scientific misconduct. To answer that question, you need to do a deep dive into the meaning of your research question and why it is being asked in the first place. What do you hope to be able to do with the results? Now, you have never given a statement of your research question here. (And if you had, we're so far from my discipline that I probably wouldn't be able to help you figure this out in detail anyway.) If we were discussing the treatment vs control variable, and if your research question was to determine whether or not the intervention had a significant effect on the outcome probability, then I would say comfortably that the average marginal effect is the best statistic to use. But you are interested, it appears, in the effects of the return_num variables. I have no clue what those even are, let alone why one would want to know about some aspect of their effect on outcome and what one might do with the information. So I can't advise you. Finally, you have mentioned your boss's preference for null hypothesis significance testing more than once, so I suspect that presenting results that will satisfy your his or her needs is a high priority. On the highly likely assumption that your boss is somebody I have never met and know nothing about, my best advice would be to discuss this with him or her and work it out together.

Finally, some unsolicited career advice: if your boss is not a person that you can have that kind of discussion with, do your best to read his or her mind for the moment and set your sights on finding a better job.

Added: Even within the hypothesis testing mode, the overall model p-value is a rather uninteresting statistic that answers a rather bizarre question that is rarely of concern to anyone. It is well worth ignoring. It purports to be a test of the joint hypothesis that all of the model coefficients are zero. And, in a sense, it is, except that what it actually tests is whether the vector of all the coordinate-estimates lies within a high-dimensional hyper-ellipse centered around the origin or not. Most people are unaware of that true meaning and most people would, if they understood that, agree that it is not an interesting or sensible way of defining "all coordinates are indistinguishable from zero."

[Scott Forrester]

Clyde Schechter
Dear Clyde,
Thank you so very much for your advice! You have been most helpful!