Comparing Coefficients of two Independent Samples

[Lara Wagner]

I ran two logistic regressions for two independent samples (identical variables in both regressions).
Now I would like to find out if the difference between two specific coefficients I used for both estimates as an independent variable is signficantly different. The values are different, but I need evidence for significance.

Coef 1 = .119, N1= 8000
Coef 2 = .147, N2=17000

Could you tell me how to test this in Stata?

[Bruce Weaver]

Hello Lara. You could estimate a model using all of the data, and include an indicator for group/sample and its interaction with the variable of interest. The z-test for that product term will provide a test of the difference you want. Here is an example using the low birth weight data.

clear
webuse lbw
tabulate race smoke

logit low age if !smoke
logit low age if smoke
logit low c.age##smoke

display "b_age for non-smokers = " -.0828798
display "b_age for smokers =     " -.0097967
display "Smokers - non-smokers = "  -.0097967 - (-.0828798)
// Notice that the difference = b for the c.age#smoke product term,
// so the z-test for that term is a test of the difference you want.

HTH.

[Roman Mostazir]

It is difficult to suggests specifically as you are not telling us much about the models you fitted. Please read the FAQ section on how to make meaningful post. On your question, my first hunch is why you needed to run two separate model when all your variables are identical? Couldn't you have done it in one model and compare the coefficients? For example, you can have an independent sample of USA people and another from UK asking them whether they are happy or not. Fitting one logit model having country as dummy and happiness as outcome, will give you the difference of average odds of happiness between two countries. It is a simple solution I am suggesting based on what information you have provided. If it does not fit, you need to tell us more.

[Rich Goldstein]

if I understand your question correctly, you want to start with -suest- and then use either -test- or -testnl-; see the help for -suest- and particularly the examples

[Lara Wagner]

Thank you so far. Maybe I need to clarify why I need two seperate regressions.
I simply want to test if my treatment is working better for males or females.

DV = response (0/1)
IDV1=Gender (0/1)
IDV2=Treatment (0/1)

1. I seperate the sample by gender: sample 1 = male, sample 2 = female
N_male= 8000
N_female=17000

2. I run two logistic regressions on my DV "response"
I receive the following Coefficients:
Coef. "treatment" for the sample of males = .119 (Odds Ratio = 0.13)
Coef. "treatment" for the sample of females = .147 (Odds Ratio = 0.16)

3. Interpretation: The treatment has stronger Impact on females.

Question: Is the difference between .119 and .147 significant? How would I test this in stata?

[Rich Goldstein]

maybe your response crossed with mine, but I repeat that -suest- is what you want (since you apparently don't want to go the interaction route); see #4 above and there is a similar example in the -suest- help file

[Bruce Weaver]

If I wanted to test whether the treatment effect differs by sex, I would include both males and females in the model, and look at the Treatment x Sex interaction.

logit Treatment##Sex // change variable names to match yours

The z-test for the Treatment#Sex term gives the test you want.

HTH.

[Brad Anderson]

Richard Williams has a very useful discussion of the issues here: http://www3.nd.edu/~rwilliam/stats3/RW_ESRA2013.pdf

An often cited article is Paul Alison's 1999 article "Comparing Logit and Probit Coefficients between Groups in Sociological Methods and Research. There are also several older threads that discuss this issue. A test of the conditional effect using an interaction term assumes equal unobserved heterogeneity between the groups being compared. I'd also add that you might want to ask yourself if the differences are substantively impotant? With an n of 8k in one group and an n of 17k in the second, almost any difference will be sufficiently large to reject the null hypothesis of no difference. But is the observed difference actually clinically or substantively meaningful? Indeed, are the effects expressed by these coefficients substantively meaningful? They translate to effects on the odds of about 1.13 and 1.16, respectively. Of course, what is clinically or substantively meaningful varies greatly across subject matter.

[Roman Mostazir]

There is a small typo in Bruce's command (#7) and as he said, that is what you want (fit them in the same model):

 logit DV Treatment##Sex // change variable names to match yours

The interaction term is the difference between the average odds between two group and its relevant tests.

Brad's points #8 are worth to consider. You are likely to get a significant value with such sample size, but is it a meaningful difference in your discipline, that is something needs to be considered by yourself.

[Bruce Weaver]

Re my typo in #7, good catch Roman! Thanks for pointing that out. And excellent point about the distinction between statistical significance (which is almost guaranteed with large samples) and practical importance.

Cheers,
Bruce

[Lara Wagner]

Thank you so much for your valuable time and answers! As I understand from #8, Brad, the comparison of logit models actually doesn't meet statistical requirements.
If I want proper results I should calculate interaction effects.
Also the -suest- command doesn't work here, because it's for linear regression only. Right?

[Rich Goldstein]

-suest- is not limited to linear regression and may certainly be used for logistic regression

whether to use -suest- or an interaction depends on whether you want/should constrain the variances; this has been discussed several times on this list