Reg with interaction terms vs. subgroup regression

[Vincent Li]

Hi there,

I intend to study how education and occupation are related to fathers' and mothers' time investment in parenting. Because I focus on mothers' and fathers' time investment respectively, I choose two ways to deal with the impact of gender: (a) adding interaction terms (gender##edu, gender##ocp); (b) regression on each gender group with suest.

About adding interaction terms in the model, the code is:

reg daily_time2 gender_p peduy speduy ocpcate_dum2 ocpcate_dum3 ocpcate_dum4 ocpcatesp_dum2 ocpcatesp_dum3 ocpcatesp_dum4  ///
live_c splive_c sblive_c gplive_c othlive_c marital_s livearea_dum2 livearea_dum3 fam_culecores fam_livenvi  ///
page peth ///
gender age mig13 sibrank_dum2 sibrank_dum3 sibrank_dum4 selftime_edu selftime_re cog_13 depression_13 schmisb_13 ///
classz pincoml private   ///
i.gender_p##c.peduy i.gender_p##c.speduy ///
i.gender_p##i.ocpcate_dum2 i.gender_p##i.ocpcate_dum3 i.gender_p##i.ocpcate_dum4  ///
i.gender_p##i.ocpcatesp_dum2 i.gender_p##i.ocpcatesp_dum3 i.gender_p##i.ocpcatesp_dum4, vce(robust)

peduy means the parent's years of schooling; speduy means the spouse's years of schooling.
3 dummy variables for the parent's occupation (ocpcate_dum) and 3 dummy variables for the spouse's occupation (ocpcatesp_dum).

The results are (results of covariates are not shown):

Linear regression Number of obs = 9,508
F(43, 9464) = 58.70
Prob > F = 0.0000
R-squared = 0.1909
Root MSE = 2.2099

---------------------------------------------------------------------------------------
| Robust
daily_time2 | Coef. Std. Err. t P>|t| [95% Conf. Interval]
----------------------+----------------------------------------------------------------
gender_p | -2.025758 .3423148 -5.92 0.000 -2.696768 -1.354747
peduy | .0128846 .0160059 0.80 0.421 -.0184903 .0442596
speduy | .0066183 .0176536 0.37 0.708 -.0279866 .0412231
ocpcate_dum2 | -.6571733 .1419606 -4.63 0.000 -.9354466 -.3789
ocpcate_dum3 | -.7833145 .1963067 -3.99 0.000 -1.168118 -.3985113
ocpcate_dum4 | -.8941722 .1701738 -5.25 0.000 -1.227749 -.5605951
ocpcatesp_dum2 | .1721731 .2258591 0.76 0.446 -.2705593 .6149054
ocpcatesp_dum3 | .5095036 .2955635 1.72 0.085 -.0698643 1.088872
ocpcatesp_dum4 | .480492 .2422597 1.98 0.047 .005611 .955373

1.gender_p | 0 (omitted)
peduy | 0 (omitted)
|
gender_p#c.peduy |
1 | -.0035093 .0190895 -0.18 0.854 -.0409288 .0339103
|
speduy | 0 (omitted)
|
gender_p#c.speduy |
1 | -.0191733 .0198346 -0.97 0.334 -.0580534 .0197068
|
1.ocpcate_dum2 | 0 (omitted)
|
gender_p#ocpcate_dum2 |
1 1 | .5448046 .196568 2.77 0.006 .1594891 .9301202
|
1.ocpcate_dum3 | 0 (omitted)
|
gender_p#ocpcate_dum3 |
1 1 | .9546625 .265345 3.60 0.000 .4345293 1.474796
|
1.ocpcate_dum4 | 0 (omitted)
|
gender_p#ocpcate_dum4 |
1 1 | .9423153 .2270512 4.15 0.000 .4972463 1.387384
|
1.ocpcatesp_dum2 | 0 (omitted)
|
gender_p#ocpcatesp_dum2 |
1 1 | .1734514 .2376521 0.73 0.465 -.2923977 .6393005
|
1.ocpcatesp_dum3 | 0 (omitted)
|
gender_p#ocpcatesp_dum3 |
1 1 | -.3412283 .3165165 -1.08 0.281 -.9616686 .279212
|
1.ocpcatesp_dum4 | 0 (omitted)
|
gender_p#ocpcatesp_dum4 |
1 1 | -.1851254 .2621893 -0.71 0.480 -.6990728 .328822
|
_cons | 1.751216 .6408113 2.73 0.006 .4950884 3.007344
---------------------------------------------------------------------------------------

.
end of do-file

As for the subgroup regression, the code is as below:

reg daily_time2 gender_p peduy speduy ocpcate_dum2 ocpcate_dum3 ocpcate_dum4 ocpcatesp_dum2 ocpcatesp_dum3 ocpcatesp_dum4  ///
live_c splive_c sblive_c gplive_c othlive_c marital_s livearea_dum2 livearea_dum3 fam_culecores fam_livenvi  ///
page peth ///
gender age mig13 sibrank_dum2 sibrank_dum3 sibrank_dum4 selftime_edu selftime_re cog_13 depression_13 schmisb_13 ///
classz pincoml private if gender_p==0
est store daily_m

reg daily_time2 gender_p peduy speduy ocpcate_dum2 ocpcate_dum3 ocpcate_dum4 ocpcatesp_dum2 ocpcatesp_dum3 ocpcatesp_dum4  ///
live_c splive_c sblive_c gplive_c othlive_c marital_s livearea_dum2 livearea_dum3 fam_culecores fam_livenvi  ///
page peth ///
gender age mig13 sibrank_dum2 sibrank_dum3 sibrank_dum4 selftime_edu selftime_re cog_13 depression_13 schmisb_13 ///
classz pincoml private if gender_p==1
est store daily_f

suest daily_m daily_f

test [daily_m_mean]peduy = [daily_f_mean]peduy
test [daily_m_mean]speduy = [daily_f_mean]speduy
test [daily_m_mean]ocpcate_dum2 = [daily_f_mean]ocpcate_dum2
test [daily_m_mean]ocpcate_dum3 = [daily_f_mean]ocpcate_dum3
test [daily_m_mean]ocpcate_dum4 = [daily_f_mean]ocpcate_dum4
test [daily_m_mean]ocpcatesp_dum2 = [daily_f_mean]ocpcatesp_dum2
test [daily_m_mean]ocpcatesp_dum3 = [daily_f_mean]ocpcatesp_dum3
test [daily_m_mean]ocpcatesp_dum4 = [daily_f_mean]ocpcatesp_dum4

The results are (just show simultaneous results and tests):

Simultaneous results for daily_m, daily_f

Number of obs = 9,508

--------------------------------------------------------------------------------
| Robust
| Coef. Std. Err. z P>|z| [95% Conf. Interval]
---------------+----------------------------------------------------------------
daily_m_mean |
gender_p | 0 (omitted)
peduy | .0042709 .0164781 0.26 0.795 -.0280256 .0365674
speduy | .0038929 .0179366 0.22 0.828 -.0312622 .0390481
ocpcate_dum2 | -.6382543 .1423337 -4.48 0.000 -.9172232 -.3592853
ocpcate_dum3 | -.7577436 .1963139 -3.86 0.000 -1.142512 -.3729753
ocpcate_dum4 | -.8749675 .1694062 -5.16 0.000 -1.206998 -.5429374
ocpcatesp_dum2 | .1403245 .226276 0.62 0.535 -.3031682 .5838173
ocpcatesp_dum3 | .5180557 .2937077 1.76 0.078 -.0576008 1.093712
ocpcatesp_dum4 | .4593349 .2422361 1.90 0.058 -.0154392 .9341089

_cons | 1.7019 1.04548 1.63 0.104 -.3472028 3.751003
---------------+----------------------------------------------------------------
daily_m_lnvar |
_cons | 1.968507 .0307041 64.11 0.000 1.908328 2.028686
---------------+----------------------------------------------------------------
daily_f_mean |
gender_p | 0 (omitted)
peduy | .0117149 .0106283 1.10 0.270 -.0091161 .032546
speduy | -.0058764 .0095221 -0.62 0.537 -.0245395 .0127866
ocpcate_dum2 | -.0844407 .1358409 -0.62 0.534 -.350684 .1818027
ocpcate_dum3 | .1370791 .1776663 0.77 0.440 -.2111405 .4852988
ocpcate_dum4 | .086198 .149824 0.58 0.565 -.2074517 .3798476
ocpcatesp_dum2 | .3252663 .0723523 4.50 0.000 .1834585 .4670742
ocpcatesp_dum3 | .1314536 .1096267 1.20 0.230 -.0834108 .3463179
ocpcatesp_dum4 | .2622342 .0977692 2.68 0.007 .0706101 .4538582

_cons | -.543665 .5717879 -0.95 0.342 -1.664349 .5770187
---------------+----------------------------------------------------------------
daily_f_lnvar |
_cons | .9294638 .0360856 25.76 0.000 .8587372 1.00019
--------------------------------------------------------------------------------

. test [daily_m_mean]peduy = [daily_f_mean]peduy

( 1) [daily_m_mean]peduy - [daily_f_mean]peduy = 0

chi2( 1) = 0.14
Prob > chi2 = 0.7042

. test [daily_m_mean]speduy = [daily_f_mean]speduy

( 1) [daily_m_mean]speduy - [daily_f_mean]speduy = 0

chi2( 1) = 0.23
Prob > chi2 = 0.6305

. test [daily_m_mean]ocpcate_dum2 = [daily_f_mean]ocpcate_dum2

( 1) [daily_m_mean]ocpcate_dum2 - [daily_f_mean]ocpcate_dum2 = 0

chi2( 1) = 7.92
Prob > chi2 = 0.0049

. test [daily_m_mean]ocpcate_dum3 = [daily_f_mean]ocpcate_dum3

( 1) [daily_m_mean]ocpcate_dum3 - [daily_f_mean]ocpcate_dum3 = 0

chi2( 1) = 11.42
Prob > chi2 = 0.0007

. test [daily_m_mean]ocpcate_dum4 = [daily_f_mean]ocpcate_dum4

( 1) [daily_m_mean]ocpcate_dum4 - [daily_f_mean]ocpcate_dum4 = 0

chi2( 1) = 18.06
Prob > chi2 = 0.0000

test [daily_m_mean]ocpcatesp_dum2 = [daily_f_mean]ocpcatesp_dum2

( 1) [daily_m_mean]ocpcatesp_dum2 - [daily_f_mean]ocpcatesp_dum2 = 0

chi2( 1) = 0.61
Prob > chi2 = 0.4363

. test [daily_m_mean]ocpcatesp_dum3 = [daily_f_mean]ocpcatesp_dum3

( 1) [daily_m_mean]ocpcatesp_dum3 - [daily_f_mean]ocpcatesp_dum3 = 0

chi2( 1) = 1.52
Prob > chi2 = 0.2175

. test [daily_m_mean]ocpcatesp_dum4 = [daily_f_mean]ocpcatesp_dum4

( 1) [daily_m_mean]ocpcatesp_dum4 - [daily_f_mean]ocpcatesp_dum4 = 0

chi2( 1) = 0.57
Prob > chi2 = 0.4505

My questions are:
Do the results of interaction terms and subgroup regression tell us the same thing?
I think I can get that, for example, the impact of spouse's occupation (ocpcatesp_dum) has no gender heterogeneity. However, in the Simultaneous results, the impact of spouse's occupation for fathers and mothers "look" quite different:

for mother:
ocpcatesp_dum2 | .1403245 .226276 0.62 0.535 -.3031682 .5838173
ocpcatesp_dum3 | .5180557 .2937077 1.76 0.078 -.0576008 1.093712
ocpcatesp_dum4 | .4593349 .2422361 1.90 0.058 -.0154392 .9341089

for father:
ocpcatesp_dum2 | .3252663 .0723523 4.50 0.000 .1834585 .4670742
ocpcatesp_dum3 | .1314536 .1096267 1.20 0.230 -.0834108 .3463179
ocpcatesp_dum4 | .2622342 .0977692 2.68 0.007 .0706101 .4538582

test results:
test [daily_m_mean]ocpcatesp_dum2 = [daily_f_mean]ocpcatesp_dum2

( 1) [daily_m_mean]ocpcatesp_dum2 - [daily_f_mean]ocpcatesp_dum2 = 0

chi2( 1) = 0.61
Prob > chi2 = 0.4363

. test [daily_m_mean]ocpcatesp_dum3 = [daily_f_mean]ocpcatesp_dum3

( 1) [daily_m_mean]ocpcatesp_dum3 - [daily_f_mean]ocpcatesp_dum3 = 0

chi2( 1) = 1.52
Prob > chi2 = 0.2175

. test [daily_m_mean]ocpcatesp_dum4 = [daily_f_mean]ocpcatesp_dum4

( 1) [daily_m_mean]ocpcatesp_dum4 - [daily_f_mean]ocpcatesp_dum4 = 0

chi2( 1) = 0.57
Prob > chi2 = 0.4505

How to interpret the results of subgroup regression?

Thank you so much.

[Clyde Schechter]

First, let's be clear that the two approaches are not, as you have implemented them, equivalent. In the separate regressions, all of the variables in the model receive regression coefficients separately for each sex. In the interaction approach, those variables which are not interacted with gender are constrained to receive a single common regression coefficient that is applied regardless of gender. So you cannot expect the results of these two approaches to necessarily agree on anything. To make them fully equivalent you would need to extend the gender interaction to include all model variables.

Now, concerning the interpretation of the separate regressions, a simple inspection of the outputs for the males and females can be misleading. If you want to know how much the effects of occupation and education differ across the sexes, what is relevant is the difference between the male and female coefficients. You will not find that in the regression outputs: you will find the coefficients themselves, and while you can mentally calculate their difference, what you can't do by inspection is get the confidence interval for the difference. That's a more complicated calculation, which the -lincom- command can do for you. (The -test- command does it, behind the scenes, as well, but the output shows you only the chi square statistic and p-value, which are far less informative.) What you can perhaps glean from inspecting the regression outputs is that the confidence intervals around the male and female coefficients are pretty wide, and in almost all instances the male and female confidence intervals overlap extensively and contain the opposite coefficient. That is a hint that the male-female differences are either not very large or, if they are, the data are too imprecise to say much about just how large they are. In any case, the best way to see the actual differences is to use -lincom- instead of -test-.

[Vincent Li]

Thank you Clyde.

Now I understand why there are so many differences between results of the interaction approach and that of the subgroup regression. Thank you so much for the clarification.

Then I tried -lincom- command and have two further questions.

code:

reg daily_time2 gender_p peduy speduy ocpcate_dum2 ocpcate_dum3 ocpcate_dum4 ocpcatesp_dum2 ocpcatesp_dum3 ocpcatesp_dum4  ///
live_c splive_c sblive_c gplive_c othlive_c marital_s livearea_dum2 livearea_dum3 fam_culecores fam_livenvi  ///
page peth ///
gender age mig13 sibrank_dum2 sibrank_dum3 sibrank_dum4 selftime_edu selftime_re cog_13 depression_13 schmisb_13 ///
classz pincoml private if gender_p==0
est store daily_m

reg daily_time2 gender_p peduy speduy ocpcate_dum2 ocpcate_dum3 ocpcate_dum4 ocpcatesp_dum2 ocpcatesp_dum3 ocpcatesp_dum4  ///
live_c splive_c sblive_c gplive_c othlive_c marital_s livearea_dum2 livearea_dum3 fam_culecores fam_livenvi  ///
page peth ///
gender age mig13 sibrank_dum2 sibrank_dum3 sibrank_dum4 selftime_edu selftime_re cog_13 depression_13 schmisb_13 ///
classz pincoml private if gender_p==1
est store daily_f

suest daily_m daily_f

lincom [daily_m_mean]peduy-[daily_f_mean]peduy
lincom [daily_m_mean]speduy-[daily_f_mean]speduy
lincom [daily_m_mean]ocpcate_dum2-[daily_f_mean]ocpcate_dum2
lincom [daily_m_mean]ocpcate_dum3-[daily_f_mean]ocpcate_dum3
lincom [daily_m_mean]ocpcate_dum4-[daily_f_mean]ocpcate_dum4
lincom [daily_m_mean]ocpcatesp_dum2-[daily_f_mean]ocpcatesp_dum2
lincom [daily_m_mean]ocpcatesp_dum3-[daily_f_mean]ocpcatesp_dum3
lincom [daily_m_mean]ocpcatesp_dum4-[daily_f_mean]ocpcatesp_dum4

results:

Simultaneous results for daily_m, daily_f

Number of obs = 9,508

--------------------------------------------------------------------------------
| Robust
| Coef. Std. Err. z P>|z| [95% Conf. Interval]
---------------+----------------------------------------------------------------
daily_m_mean |
gender_p | 0 (omitted)
peduy | .0042709 .0164781 0.26 0.795 -.0280256 .0365674
speduy | .0038929 .0179366 0.22 0.828 -.0312622 .0390481
ocpcate_dum2 | -.6382543 .1423337 -4.48 0.000 -.9172232 -.3592853
ocpcate_dum3 | -.7577436 .1963139 -3.86 0.000 -1.142512 -.3729753
ocpcate_dum4 | -.8749675 .1694062 -5.16 0.000 -1.206998 -.5429374
ocpcatesp_dum2 | .1403245 .226276 0.62 0.535 -.3031682 .5838173
ocpcatesp_dum3 | .5180557 .2937077 1.76 0.078 -.0576008 1.093712
ocpcatesp_dum4 | .4593349 .2422361 1.90 0.058 -.0154392 .9341089
⋯⋯
_cons | 1.7019 1.04548 1.63 0.104 -.3472028 3.751003
---------------+----------------------------------------------------------------
daily_m_lnvar |
_cons | 1.968507 .0307041 64.11 0.000 1.908328 2.028686
---------------+----------------------------------------------------------------
daily_f_mean |
gender_p | 0 (omitted)
peduy | .0117149 .0106283 1.10 0.270 -.0091161 .032546
speduy | -.0058764 .0095221 -0.62 0.537 -.0245395 .0127866
ocpcate_dum2 | -.0844407 .1358409 -0.62 0.534 -.350684 .1818027
ocpcate_dum3 | .1370791 .1776663 0.77 0.440 -.2111405 .4852988
ocpcate_dum4 | .086198 .149824 0.58 0.565 -.2074517 .3798476
ocpcatesp_dum2 | .3252663 .0723523 4.50 0.000 .1834585 .4670742
ocpcatesp_dum3 | .1314536 .1096267 1.20 0.230 -.0834108 .3463179
ocpcatesp_dum4 | .2622342 .0977692 2.68 0.007 .0706101 .4538582
⋯⋯
_cons | -.543665 .5717879 -0.95 0.342 -1.664349 .5770187
---------------+----------------------------------------------------------------
daily_f_lnvar |
_cons | .9294638 .0360856 25.76 0.000 .8587372 1.00019
-------------------------------------------------------------------------

( 1) [daily_m_mean]ocpcate_dum2 - [daily_f_mean]ocpcate_dum2 = 0

------------------------------------------------------------------------------
| Coef. Std. Err. z P>|z| [95% Conf. Interval]
-------------+----------------------------------------------------------------
(1) | -.5538136 .1967528 -2.81 0.005 -.9394419 -.1681852
------------------------------------------------------------------------------

. lincom [daily_m_mean]ocpcate_dum3-[daily_f_mean]ocpcate_dum3

( 1) [daily_m_mean]ocpcate_dum3 - [daily_f_mean]ocpcate_dum3 = 0

------------------------------------------------------------------------------
| Coef. Std. Err. z P>|z| [95% Conf. Interval]
-------------+----------------------------------------------------------------
(1) | -.8948227 .2647725 -3.38 0.001 -1.413767 -.3758781
------------------------------------------------------------------------------

. lincom [daily_m_mean]ocpcate_dum4-[daily_f_mean]ocpcate_dum4

( 1) [daily_m_mean]ocpcate_dum4 - [daily_f_mean]ocpcate_dum4 = 0

------------------------------------------------------------------------------
| Coef. Std. Err. z P>|z| [95% Conf. Interval]
-------------+----------------------------------------------------------------
(1) | -.9611655 .2261542 -4.25 0.000 -1.40442 -.5179115
------------------------------------------------------------------------------

. lincom [daily_m_mean]ocpcatesp_dum2-[daily_f_mean]ocpcatesp_dum2

( 1) [daily_m_mean]ocpcatesp_dum2 - [daily_f_mean]ocpcatesp_dum2 = 0

------------------------------------------------------------------------------
| Coef. Std. Err. z P>|z| [95% Conf. Interval]
-------------+----------------------------------------------------------------
(1) | -.1849418 .2375619 -0.78 0.436 -.6505546 .280671
------------------------------------------------------------------------------

. lincom [daily_m_mean]ocpcatesp_dum3-[daily_f_mean]ocpcatesp_dum3

( 1) [daily_m_mean]ocpcatesp_dum3 - [daily_f_mean]ocpcatesp_dum3 = 0

------------------------------------------------------------------------------
| Coef. Std. Err. z P>|z| [95% Conf. Interval]
-------------+----------------------------------------------------------------
(1) | .3866021 .3134999 1.23 0.218 -.2278465 1.001051
------------------------------------------------------------------------------

. lincom [daily_m_mean]ocpcatesp_dum4-[daily_f_mean]ocpcatesp_dum4

( 1) [daily_m_mean]ocpcatesp_dum4 - [daily_f_mean]ocpcatesp_dum4 = 0

------------------------------------------------------------------------------
| Coef. Std. Err. z P>|z| [95% Conf. Interval]
-------------+----------------------------------------------------------------
(1) | .1971007 .2612224 0.75 0.451 -.3148858 .7090872
------------------------------------------------------------------------------

.
end of do-file

The two questions are:
First, in the results of subgroup regression, let's say the ocpcate_dum2 variable, for female, the coefficient is negative and statistically significant; for male, the coefficient is negative but not statistically significant. The lincom result says these two coefficients are statistically different from each other. Could I draw the conclusion that there is a heterogeneity between males and females in terms of the impact of ocpcate_dum2 variable, and the ocpcate_dum2 is not statistically correlated to the dependent variable for males?
For the variable ocpcatesp_dum2, for female, the coefficient is positive but not statistically significant; for male, the coefficient is positive and statistically significant. The lincom result shows that there is no statistically difference between female's and male's coefficients. How to report the result when the two coefficients have no statistically difference but have different statistical significance? Should I draw the conclusion depending on the results of the original models (the results for the whole sample including two gender groups) instead of the separate result of two gender groups?

Second, for the interaction approach and the subgroup regression approach, which one do you prefer to do the analysis?

Sorry that the questions are too detailed.

Thanks for you patience and help.

[Clyde Schechter]

While I am among those who would like to see the use of the concept of statistical significance largely eliminated, I will put that aside (and spare you my rant about it) and pretend that I still believe in it. Even for those who take statistical significance seriously, it is important to remember that the difference between a statistically significant finding and a non-statistically significant finding is, itself, not statistically significant. The fact that a male coefficient is significant and a female one is not (or vice versa) tells you absolutely nothing about whether the difference between the male and female coefficients is statistically significant. You should never draw any conclusion from the fact that one thing is significant and the other is not.

That basically dispatches your first question. Conclusions about the difference between male and female coefficients should be drawn from the -lincom- results. The statistical significance, or lack thereof, of the separate male and female coefficients has nothing at all to do with it.

As between the interaction approach and the subgroup regression approach, I incline towards the interaction approach. It's mostly a matter of taste--the conclusions, when both methods are done correctly, will always be the same. One reason for preferring the interaction approach is that it is applicable in any kind of regression model, whereas the after separate subgroup regressions, for certain kinds of regression there is no good way to then compare and contrast the results of the two separate regressions.

[Alice Yang]

Hi Clyde,

One reason for preferring the interaction approach is that it is applicable in any kind of regression model, whereas the after separate subgroup regressions, for certain kinds of regression there is no good way to then compare and contrast the results of the two separate regressions.

I believe one such case is when using xtreg, I can't suest model_1 model_2 and then use lincom. I wonder would a chow test be appropriate to compare the coefficients of the independent variable between two subgroup regressions with two-way fixed effects using xtreg?

Thanks!

[Clyde Schechter]

Yes, if you can do a Chow test, it is equivalent to using an interaction model (or to using -suest- when that is possible.) Here is an old FAQ from StataCorp on how to do a Chow test. https://www.stata.com/support/faqs/s...cs/chow-tests/ It is really just a long-winded way of constructing an interaction model.

[Alice Yang]

Hi Clyde,

Thanks for your reply. I believe there is a user-written command -chowtest- that can be used. I have searched on statalist but it seems there's very little information about this command. This is related to my another question https://www.statalist.org/forums/for...-clarification, I also wonder whether I should include control variables and also i.year when using -chowtest-.

[Clyde Schechter]

Yes you would need to include all of the covariates ("control" variables) and the panel and year fixed effects as well in order for it to give you a correct answer. Without those, you would be testing equality of coefficients in models that are not the ones you are interested in comparing.

[Alice Yang]

I see, thanks for your help!

[Vincent Li]

Clyde,

Thank you so much! Now I totally understand it.
As for "the use of the concept of statistical significance largely eliminated", do you think using the concept of effect size is more appropriate or more helpful when interpreting the results?
Thanks again.

And it's great that Alice got the answer of her question here.

Thank you!

[Clyde Schechter]

Yes, I think effect sizes are much more important. And I think that whenever we estimate an effect size we should also give some indication of the uncertainty in that estimate. Standard errors do that, but I think confidence intervals convey that more clearly.

[Vincent Li]

Thanks a lot, Clyde. I'd like to try to use effect sizes and confidence intervals to report the results in the following analysis.

[Aparajita Agarwal]

Picking up on this thread again - I am encountering a similar discrepancy between sub-group analysis and interaction effects. What is perplexing is that I find this discrepancy even without controls in my model. I am using ppmlhdfe, I have a Treatment variable which is a continuous measure of "bite of treatment" multiplied by a post variable. My moderator is a binary variable: 0 if an agent and customer are from different caste, and 1 if an agent and customer are from different caste. My DV is number of transactions between the agent and customer. The model includes agent-customer dyad fixed effects and week fixed effects

Models 1 and 2 give totally different results and they are opposite in signs too.

Model 1: ppmlhdfe n_txn c.Treatment c.Treatment#i.samecaste , absorb (agent_customerdyad i.yr_week) vce(cluster agentID)

Model 2a: ppmlhdfe n_txn c.Treatment if samecaste == 0 , absorb (agent_customerdyad i.yr_week) vce(cluster agentID)
Model 2b: ppmlhdfe n_txn c.Treatment if samecaste == 1 , absorb (agent_customerdyad i.yr_week) vce(cluster agentID)

Note that the variable samecaste gets dropped out with the fixed effects as it is time invariant. Hence I do not add a ##

Why would the signs flip in this case? Any advice? And how should I decide which of these to trust?

Thank you