How to test whether the two variables are 2 standard deviation different from each other?

[NIki Young]

Dear Stata experts,

I have below comments from one referee for my paper, can someone guide me how to test whether the two variables are 2 standard deviation different from each other in stata?
"The author points to some effects that are positive or negative — but the issue is not the difference between the effect and zero but between one effect and the other. So, say for one income level the effect is 0.005 and for another it is something else like -0.001. The real question is not whether one of these is significantly different from zero at 5% but whether the two coefficients are 2 standard deviations different from each other. If the SE of one variable is large, this may not be the case even if one coefficient is very significant. Here we are less interested in whether either coefficient is different from zero and much more interested in whether one coefficient is different from the other at any real level of significance."
My income level is a category variable with 7 levels, my dependent variable is loan default. This questions is about regression result of a logit model.

Thank you very much in advance.

[Clyde Schechter]

The reviewer made a small mistake in describing the situation: the issue is not whether they are 2 standard deviations apart. That doesn't even make sense: standard deviation of what? But what I'm sure was intended is for you to test the significance of the difference between the two levels. You don't provide the details of your regression output, but since you have said income is a categorical variable with 7 levels, let's assume for the sake of illustration that the two levels of interest are are levels 2 and 5. Then what the reviewer wants is for you to run the test:

lincom 2.income - 5.income

You run that immediately after the regression itself. Stata will give you the difference between the coefficients of income level 2 and income level 5, along with the standard error (not standard deviation) of that difference and will also give you a z-test.

Note: I'm assuming you used factor variable notation to represent your income variable in the regression. If you didn't, now's as good a time as any to start using factor variable notation: read -help fvvarlist- for an explanation.

Added: The review is correct in principle. The fact that one thing is statistically significant and another is not is simply a meaningless observation. It tells you nothing about whether they are significantly different from each other. Even for those who take statistical significance seriously, it is important to remember that the difference between statistically significant and not statistically significant is, itself, not statistically significant, nor even statistically meaningful.

[NIki Young]

Thank you very much for the response. I run the code. I have the same results as my regression. I choose the income group 4 (mean group) as the reference group. So when I run
lincom 2.income - 4.income It gave me:

( 1) [default]2.income2 - [default]4b.income2 = 0

------------------------------------------------------------------------------
default | Coef. Std. Err. z P>|z| [95% Conf. Interval]
-------------+----------------------------------------------------------------
(1) | -.6078613 .3443761 -1.77 0.078 -1.282826 .0671034
------------------------------------------------------------------------------

Which is actually the same as what is show in my original results. So how should I intepretate this result to respond the reviewer?

Below is my orginal regression result:
--------------------------------------------------------------------------------------------
| Robust
default | Coef. Std. Err. z P>|z| [95% Conf. Interval]
---------------------------+----------------------------------------------------------------
1.incomeverified | .599432 .2321633 2.58 0.010 .1444004 1.054464
|
income2 |
1 | -.8782643 1.085802 -0.81 0.419 -3.006398 1.249869
2 | -.6078613 .3443761 -1.77 0.078 -1.282826 .0671034
3 | -.1656721 .1335863 -1.24 0.215 -.4274963 .0961522
5 | -.3601979 .1678176 -2.15 0.032 -.6891143 -.0312815
6 | .2328037 .1482003 1.57 0.116 -.0576636 .5232709
7 | .2594992 .1557526 1.67 0.096 -.0457703 .5647688
|
incomeverified#income2 |
1 1 | 0 (empty)
1 2 | 2.787478 .8822793 3.16 0.002 1.058243 4.516714
1 3 | .4686156 .328828 1.43 0.154 -.1758755 1.113107
1 5 | -1.16149 .5804709 -2.00 0.045 -2.299192 -.0237879
1 6 | -1.749303 .5839082 -3.00 0.003 -2.893742 -.6048639
1 7 | -2.237697 .5767056 -3.88 0.000 -3.368019 -1.107375

[Clyde Schechter]

Given that income group 4 is the reference category for your variable, it is correct that the difference between group 2 and group 4 is the same as the coefficient for group 2. So you should just report that the expected difference between the groups is -.61, with 95% CI -1.28 to +.07. If they want a p-value it's 0.078.

(I had tacitly assumed that you were asked to compare two categories, neither of which was the reference grouop.)

[NIki Young]

Hi Clyde, Thank you very much for your response. I want to verify, probably the table is not clear. The coefficient is -0.61, the standard error is 0.34. So the expected difference between the groups should be 0.34?
And the -0.61 is the 0.61 log odds decrease of probability of default if income group 2 compares to income group 4. But I thought the standard error here is the distance of the observed value and the regression line. Why it is relevant to the difference between the groups....

Moreover, take income group 3 as an example, it is insignificant compares to group 4. However with quite low standard error. Is that what the referee mean by saying P value is not important but the difference between the groups matters? I mean this sentence "If the SE of one variable is large, this may not be the case even if one coefficient is very significant." So I should simply said group 3 is different from group 4 at 0.21 significant level with 0.13 standard error...Is this correct?

About that standard deviation, would it be possible that he thought this chart is for marginal effect. Then it is relevant to 2 standard deviation somehow? Becasue I truly didn't mention this is the original regression results, not the marginal effect.

[Clyde Schechter]

The coefficient is -0.61, the standard error is 0.34. So the expected difference between the groups should be 0.34?

No. The expected difference (in log odds) between groups should be -0.61.

But I thought the standard error here is the distance of the observed value and the regression line.

No, that is not at all what the standard error means. The standard error of a regression coefficient is an estimate of the standard deviation of the sampling distribution of the coefficient on repeated random resampling of the data, with the same sample size, from the same population. It is relevant to the difference between groups because it is a "yardstick" that is free of the scale of the independent variable to enable comparisons of different coefficients.

About that standard deviation, would it be possible that he thought this chart is for marginal effect. Then it is relevant to 2 standard deviation somehow?

No. Marginal effects have standard errors, not standard deviations, just like regression coefficients do.

However with quite low standard error. Is that what the referee mean by saying P value is not important but the difference between the groups matters?

I cannot read the reviewer's mind. There are two reasonable interpretations to this:

1. The reviewer may belong to the school of statistical thought which says that p-values are, in general, unimportant and should seldom, if ever, be used for anything. While I belong to this school of thought myself, it is a minority view at this time. If the reviewer did not object more generally to the use of p-values in your paper, this is probably not what was meant.
2. The reviewer may be pointing out that comparing the p-value of one coefficient to the p-value of another coefficient is meaningless. Even people who take p-values very seriously, if they understand them properly, will agree that comparing one p-value with another is meaningless at best, misleading at worst. Rather it is the difference between the coefficients that matters, not the difference between their p-values. And if you want to know if the coefficients are statistically significantly different, you have to look at the difference between the coefficients themselves and the standard error of that difference. (Which is what the output of -lincom- shows you.)

I would present the finding simply as the difference in coefficients between groups 3 and 4 is -0.17 +- S.E. 0.13. If you want to mention that the p-value is 0.215 you can do that, too, providing the reviewer is not objecting to the use of p-values overall. An alternative to presenting the standard error is to prevent the 95% confidence interval--many people find that more convenient, though the two approaches are entirely equivalent: either one can be calculated from knowing the other.

Finally, I just noticed for the first time that the variable income2 participates in an interaction with incomeverified. Consequently, the coefficients for income2 are not about the variable income2 itself. They are about income2 conditional on incomeverified = 0. When incomeverified = 1, there will be different results. So if you are speaking about the income2 coefficients as being effects of income group without further qualification, you are misrepresenting the results entirely. With the interaction model, you must have separate effects for each incomegroup conditional on the value of incomeverified. It is most simple to get those from the -margins- command, rather than going through the tedious and error-prone process of trying to calculate them from the regression output table.

[NIki Young]

Thank you very much Clyde.

Still refering to the original results I posted.

So this is the results of income2 for incomeverified==0,

income2 |
1 | -.8782643 1.085802 -0.81 0.419 -3.006398 1.249869
2 | -.6078613 .3443761 -1.77 0.078 -1.282826 .0671034
3 | -.1656721 .1335863 -1.24 0.215 -.4274963 .0961522
5 | -.3601979 .1678176 -2.15 0.032 -.6891143 -.0312815
6 | .2328037 .1482003 1.57 0.116 -.0576636 .5232709
7 | .2594992 .1557526 1.67 0.096 -.0457703 .5647688


AND this is the results of income2 for incomeverified==1

incomeverified#income2 |
1 1 | 0 (empty)
1 2 | 2.787478 .8822793 3.16 0.002 1.058243 4.516714
1 3 | .4686156 .328828 1.43 0.154 -.1758755 1.113107
1 5 | -1.16149 .5804709 -2.00 0.045 -2.299192 -.0237879
1 6 | -1.749303 .5839082 -3.00 0.003 -2.893742 -.6048639
1 7 | -2.237697 .5767056 -3.88 0.000 -3.368019 -1.107375

Am I right?

[NIki Young]

The code I run is

logit default i.incomeverified##ib4.income2

[NIki Young]

And when I run margins for this regression as below ,

margins,dydx(i.incomeverified ib4.income2)

incomeverified shows "not estimable".

What is the reason for this?

Thank you.

[NIki Young]

And how to intepretate the marginal effect result:

----------------------------------------------------------------------------------
| Delta-method
| dy/dx Std. Err. z P>|z| [95% Conf. Interval]
-----------------+----------------------------------------------------------------
1.incomeverified | . (not estimable)
|
income2 |
1 | . (not estimable)
2 | -.1102621 .0064779 -17.02 0.000 -.1229586 -.0975656
3 | -.028074 .0014621 -19.20 0.000 -.0309395 -.0252084
5 | .0108446 .0018573 5.84 0.000 .0072044 .0144848
6 | .028063 .0023184 12.10 0.000 .0235191 .0326069
7 | .0326274 .0026951 12.11 0.000 .0273452 .0379096

Does this mean verified income level 2 is 11 percentage point less likely to default than income level 4?

[NIki Young]

No. The expected difference (in log odds) between groups should be -0.61.


No, that is not at all what the standard error means. The standard error of a regression coefficient is an estimate of the standard deviation of the sampling distribution of the coefficient on repeated random resampling of the data, with the same sample size, from the same population. It is relevant to the difference between groups because it is a "yardstick" that is free of the scale of the independent variable to enable comparisons of different coefficients.


No. Marginal effects have standard errors, not standard deviations, just like regression coefficients do.


I cannot read the reviewer's mind. There are two reasonable interpretations to this:

1. The reviewer may belong to the school of statistical thought which says that p-values are, in general, unimportant and should seldom, if ever, be used for anything. While I belong to this school of thought myself, it is a minority view at this time. If the reviewer did not object more generally to the use of p-values in your paper, this is probably not what was meant.
2. The reviewer may be pointing out that comparing the p-value of one coefficient to the p-value of another coefficient is meaningless. Even people who take p-values very seriously, if they understand them properly, will agree that comparing one p-value with another is meaningless at best, misleading at worst. Rather it is the difference between the coefficients that matters, not the difference between their p-values. And if you want to know if the coefficients are statistically significantly different, you have to look at the difference between the coefficients themselves and the standard error of that difference. (Which is what the output of -lincom- shows you.)

I would present the finding simply as the difference in coefficients between groups 3 and 4 is -0.17 +- S.E. 0.13. If you want to mention that the p-value is 0.215 you can do that, too, providing the reviewer is not objecting to the use of p-values overall. An alternative to presenting the standard error is to prevent the 95% confidence interval--many people find that more convenient, though the two approaches are entirely equivalent: either one can be calculated from knowing the other.

Finally, I just noticed for the first time that the variable income2 participates in an interaction with incomeverified. Consequently, the coefficients for income2 are not about the variable income2 itself. They are about income2 conditional on incomeverified = 0. When incomeverified = 1, there will be different results. So if you are speaking about the income2 coefficients as being effects of income group without further qualification, you are misrepresenting the results entirely. With the interaction model, you must have separate effects for each incomegroup conditional on the value of incomeverified. It is most simple to get those from the -margins- command, rather than going through the tedious and error-prone process of trying to calculate them from the regression output table.

Still refering to the original results I posted.

So this is the results of income2 for incomeverified==0,

income2 |
1 | -.8782643 1.085802 -0.81 0.419 -3.006398 1.249869
2 | -.6078613 .3443761 -1.77 0.078 -1.282826 .0671034
3 | -.1656721 .1335863 -1.24 0.215 -.4274963 .0961522
5 | -.3601979 .1678176 -2.15 0.032 -.6891143 -.0312815
6 | .2328037 .1482003 1.57 0.116 -.0576636 .5232709
7 | .2594992 .1557526 1.67 0.096 -.0457703 .5647688


AND this is the results of income2 for incomeverified==1

incomeverified#income2 |
1 1 | 0 (empty)
1 2 | 2.787478 .8822793 3.16 0.002 1.058243 4.516714
1 3 | .4686156 .328828 1.43 0.154 -.1758755 1.113107
1 5 | -1.16149 .5804709 -2.00 0.045 -2.299192 -.0237879
1 6 | -1.749303 .5839082 -3.00 0.003 -2.893742 -.6048639
1 7 | -2.237697 .5767056 -3.88 0.000 -3.368019 -1.107375

Am I right?

What is the reason that incomeverified shows "not estimable" when I run marginal effect.

margins,dydx(i.incomeverified ib4.income2)

And how to intepretate the marginal effect result:

----------------------------------------------------------------------------------
| Delta-method
| dy/dx Std. Err. z P>|z| [95% Conf. Interval]
-----------------+----------------------------------------------------------------
1.incomeverified | . (not estimable)
|
income2 |
1 | . (not estimable)
2 | -.1102621 .0064779 -17.02 0.000 -.1229586 -.0975656
3 | -.028074 .0014621 -19.20 0.000 -.0309395 -.0252084
5 | .0108446 .0018573 5.84 0.000 .0072044 .0144848
6 | .028063 .0023184 12.10 0.000 .0235191 .0326069
7 | .0326274 .0026951 12.11 0.000 .0273452 .0379096

Does this mean verified income level 2 is 11 percentage point less likely to default than income level 4?

[Clyde Schechter]

So this is the results of income2 for incomeverified==0,

income2 |
1 | -.8782643 1.085802 -0.81 0.419 -3.006398 1.249869
2 | -.6078613 .3443761 -1.77 0.078 -1.282826 .0671034
3 | -.1656721 .1335863 -1.24 0.215 -.4274963 .0961522
5 | -.3601979 .1678176 -2.15 0.032 -.6891143 -.0312815
6 | .2328037 .1482003 1.57 0.116 -.0576636 .5232709
7 | .2594992 .1557526 1.67 0.096 -.0457703 .5647688

Assuming that these are from the regression table of your logit command, that is correct.

AND this is the results of income2 for incomeverified==1

incomeverified#income2 |
1 1 | 0 (empty)
1 2 | 2.787478 .8822793 3.16 0.002 1.058243 4.516714
1 3 | .4686156 .328828 1.43 0.154 -.1758755 1.113107
1 5 | -1.16149 .5804709 -2.00 0.045 -2.299192 -.0237879
1 6 | -1.749303 .5839082 -3.00 0.003 -2.893742 -.6048639
1 7 | -2.237697 .5767056 -3.88 0.000 -3.368019 -1.107375

This is incorrect. These are the differences between the results when incomeverified == 1 and the results when incomeverified == 0. Use the -margins- command to get the predicted probability of outcome in all combinations of incomeverified and income2:

margins incomeverified#income2

What is the reason that incomeverified shows "not estimable" when I run marginal effect.

margins,dydx(i.incomeverified ib4.income2)

And how to intepretate the marginal effect result:

----------------------------------------------------------------------------------
| Delta-method
| dy/dx Std. Err. z P>|z| [95% Conf. Interval]
-----------------+----------------------------------------------------------------
1.incomeverified | . (not estimable)
|
income2 |
1 | . (not estimable)
2 | -.1102621 .0064779 -17.02 0.000 -.1229586 -.0975656
3 | -.028074 .0014621 -19.20 0.000 -.0309395 -.0252084
5 | .0108446 .0018573 5.84 0.000 .0072044 .0144848
6 | .028063 .0023184 12.10 0.000 .0235191 .0326069
7 | .0326274 .0026951 12.11 0.000 .0273452 .0379096

These arise because the estimation sample does not contain any observations with income2 == 1 and incomeverified == 1. Due to the absence of relevant data, the marginal effects that Stata points out is not estimable are simply undefined.

[NIki Young]

Hi Clyde, thank you again.

I probbaly didn't mention the code I run, I run "logit default i.incomeverified##ib4.income2". There are two # between them.
So it will return the coefficient for "i.incomeverified", "ib2.income2" and the interaction terms. Thus the first part result show below is the coefficient regardless of incomeverification. Should be the results for income2 independently.
income2 |
1 | -.8782643 1.085802 -0.81 0.419 -3.006398 1.249869
2 | -.6078613 .3443761 -1.77 0.078 -1.282826 .0671034
3 | -.1656721 .1335863 -1.24 0.215 -.4274963 .0961522
5 | -.3601979 .1678176 -2.15 0.032 -.6891143 -.0312815
6 | .2328037 .1482003 1.57 0.116 -.0576636 .5232709
7 | .2594992 .1557526 1.67 0.096 -.0457703 .5647688

My question now is how to intepretate the results of the second part.

incomeverified#income2 |
1 1 | 0 (empty)
1 2 | 2.787478 .8822793 3.16 0.002 1.058243 4.516714
1 3 | .4686156 .328828 1.43 0.154 -.1758755 1.113107
1 5 | -1.16149 .5804709 -2.00 0.045 -2.299192 -.0237879
1 6 | -1.749303 .5839082 -3.00 0.003 -2.893742 -.6048639
1 7 | -2.237697 .5767056 -3.88 0.000 -3.368019 -1.107375

Can I say that when the income is verified, income group 23 is more likely to default compare to income group 4, while income group 567 is less likely to default than income group 4. ?

[Clyde Schechter]

Thus the first part result show below is the coefficient regardless of incomeverification. Should be the results for income2 independently.

No, no, no, no, no!!!!! In an interaction model there is no such thing as an estimate regardless of incomeverification. Those numbers are the coefficients of income2 conditional on incomeverified = 0.

My question now is how to intepretate the results of the second part.

incomeverified#income2 |
1 1 | 0 (empty)
1 2 | 2.787478 .8822793 3.16 0.002 1.058243 4.516714
1 3 | .4686156 .328828 1.43 0.154 -.1758755 1.113107
1 5 | -1.16149 .5804709 -2.00 0.045 -2.299192 -.0237879
1 6 | -1.749303 .5839082 -3.00 0.003 -2.893742 -.6048639
1 7 | -2.237697 .5767056 -3.88 0.000 -3.368019 -1.107375

I explained what these are in #12. Please re-read that.

Can I say that when the income is verified, income group 23 is more likely to default compare to income group 4, while income group 567 is less likely to default than income group 4. ?

No. I explained in #12 how you can get the results for incomeverified = 1.

[NIki Young]

If I have other variables in the model such as age and gender. "logit default i.incomeverified##ib4.income2 age gender". Does it make a difference for the intepretation?