Obtaining residuals with by command

[Chinmay Sharma]

Hi All,

I have a dataset that looks like:

* Example generated by -dataex-. To install: ssc install dataex
clear
input float(ID y x)
1  23  2
1   2  3
1   2  1
1   3  2
1   2 22
2   2  1
2   2  2
2   2  3
2 123  1
2   3  .
end

Here, I have information on individuals, identified by ID, on a regressand y and regressor x. I wish to run separate regressions of y on x for each individual, and compute residuals for each individual separately. (Note that this is different from running a regression of y on x, and computing residuals, as I am allowing for different slopes for each individual, which will affect residuals.

I tried:

by id: regress y x

This will simply run all the regressions, but of course does not store estimates in order to use the predict command afterwards. Any tips on this will be most useful.


CS

[Maarten Buis]

regress y i.id##x
predict resid, resid

[Nick Cox]

Maarten Buis's answer is pointing to a different model, which indeed may be better in general.

On what you asked, note e.g. rangestat (SSC) as introduced in https://www.statalist.org/forums/for...s-within-range

Posts #14 to #17 are just one of several examples showing how residuals can be calculated after several regressions.

[Attaullah Shah]

asreg (can be downloaded from SSC) can estimate both by-group and rolling regressions with residuals.

ssc install asreg
bys ID: asreg y x, fit
list ID y x _Nobs _b_x _b_cons _residuals

     +-------------------------------------------------------------+
     | ID     y    x   _Nobs         _b_x     _b_cons   _residuals |
     |-------------------------------------------------------------|
  1. |  1    23    2       5   -.27329193   8.0397516    15.506832 |
  2. |  1     2    3       5   -.27329193   8.0397516   -5.2198758 |
  3. |  1     2    1       5   -.27329193   8.0397516   -5.7664596 |
  4. |  1     3    2       5   -.27329193   8.0397516   -4.4931677 |
  5. |  1     2   22       5   -.27329193   8.0397516   -.02732919 |
     |-------------------------------------------------------------|
  6. |  2     2    1       4          -33          90          -55 |
  7. |  2     2    2       4          -33          90          -22 |
  8. |  2     2    3       4          -33          90           11 |
  9. |  2   123    1       4          -33          90           66 |
 10. |  2     3    .       .            .           .            . |
     +-------------------------------------------------------------+

[Chinmay Sharma]

Thank you!

[Chinmay Sharma]

Maarten Buis I obtain the error message:

x: Factor Variables may not contain non-integer values.

x does contain some non-integer values, but I don't see why that should be restrictive.

[Chinmay Sharma]

Interestingly, I used Attaullah Shah 's suggestion, as well as Maarten Buis , and the residuals computed are different (correlation 0f 0.95). Could be because one of them exlcudes a constant.

[Attaullah Shah]

Please note Nick's comment

Maarten Buis's answer is pointing to a different model

[Maarten Buis]

I forgot the c. prefix for the variable x. After that the models should result in the same residuals (the only difference is that the residual variance is assumed to the same for both IDs in the interaction model (homoscedasticity) while it is allowed to differ in the separately estimated models, but that difference does not affact the parameter estimates, and thus does not influence the residuals):

. clear

. input float(ID y x)

            ID          y          x
  1. 1  23  2
  2. 1   2  3
  3. 1   2  1
  4. 1   3  2
  5. 1   2 22
  6. 2   2  1
  7. 2   2  2
  8. 2   2  3
  9. 2 123  1
 10. 2   3  .
 11. end

.
. // do this manually
. qui reg y x if ID == 1

. predict resid if ID == 1, resid
(5 missing values generated)

.
. qui reg y x if ID == 2

. predict resid2 if ID == 2, resid
(6 missing values generated)

.
. replace resid = resid2 if ID== 2
(4 real changes made)

. drop resid2

.
. // do this with an interaction
. qui reg y i.ID##c.x

. predict resid2, resid
(1 missing value generated)

.
. // admire the result
. l

     +---------------------------------------+
     | ID     y    x       resid      resid2 |
     |---------------------------------------|
  1. |  1    23    2    15.50683    15.50683 |
  2. |  1     2    3   -5.219876   -5.219876 |
  3. |  1     2    1   -5.766459   -5.766459 |
  4. |  1     3    2   -4.493168   -4.493168 |
  5. |  1     2   22   -.0273292   -.0273292 |
     |---------------------------------------|
  6. |  2     2    1         -55         -55 |
  7. |  2     2    2         -22         -22 |
  8. |  2     2    3          11          11 |
  9. |  2   123    1          66          66 |
 10. |  2     3    .           .           . |
     +---------------------------------------+

[Chinmay Sharma]

thank you Attaullah Shah Maarten Buis . Why should it be a different model? Both models are running within group regressions of y on x...

[Nick Cox]

It's really the same model in different guises. My intuition was that run separately, each model can't know about the other's y and x values. But that's the point reversed: Maarten's formulation gets you separate models fitted all at once. The scope for regress to impress is unlimited, even when you think you know something about it!

Sorry therefore for misleading comment in #3.

Meanwhile, Attaullah Shah's results from asreg match those of Maarten Buis, Hence to me there is at least one puzzle left: What did you do that led to your comment in #7 that the residuals are different? I can't see any code or results to comment. As Attaullah's results show, neither model excludes a constant.

[Chinmay Sharma]

Nick Cox Thanks for your response. They are now equivalent- I had used Maarten Buis ' previous recommendation of running

regress y i.id##x predict resid, resid

as opposed to:

 
 qui reg y i.ID##c.x predict resid2, resid

which resulted in different residuals. I guess not puttting a c in front of x did result in a different model.

[Maarten Buis]

Yes, to see why see help fvvarlist