Predicted values and residuals with by

[Bülent Köksal]

Dear Stata Users,

I would like to estimate a regression like

by firm: reg y x1 x2

then I would like to obtain fitted values based on the parameter estimates for each firm. How can I achieve this?

Please correct me if I am wrong but I think if I run

predict r, residuals

after the regression above, parameter estimates from the last regression is used.

Thank you.

[Konrad Zdeb]

Dear Colleague,

Please not that this forum requires its users to use their full names when posting. With respect to your question, in order to obtain prediction or residuals after estimation you can use the predict command. If you check help predict, you will find sample syntax for obtaining the desired values. If this is not clear, I would suggest that you post an extract or dummy data via the input command as it will be easier for someone to provide you with the required code.

[Nick Cox]

That is correct. You need a loop over firms here. It is sufficient to save fitted values, as residuals can then be calculated all at once. One recipe is

egen group = group(firm)
gen fitted = .
su group, meanonly

forval g = 1/`r(max)' {
    regress y x1 x2 if group == `g'
    predict work
    replace fitted = work if group == `g'  
    drop work
}

gen residual = y - fitted

See http://www.stata.com/support/faqs/da...ach/index.html

Please note our strong preference for full real names here. That means, usually, given name and family name.

[daniel klein]

You may need to loop over the levels of firm as in

quietly levelsof firm ,local(firms)
foreach firm of local firms {
    regress y x1 x2 if (firm == `firm')
    predict resid_`firm' ,residuals
}

assuming firm identifies firms by an integer value.

Best
Daniel

[Bülent Köksal]

Thank you all for quick replies. I entered a username but I did not see a space for full name. I also do not see how to change my username under the settings.

[daniel klein]

I also do not see how to change my username under the settings.

You cannot do it yourself. Do not hesitate to click the CONTACT US button in the lower right corner. You are not the first one to miss the preference for full real names here on Statalist.

Best
Daniel

[Priyesh VP]

Hi,

My data looks like the following. I want to run regression by each nic2digit and year . I would like to get residuals of this regression for each 'code'.
I have tried the command

statsby _b e(N), by(nic2digit year)clear verbose nodots: regress ACC_w CFO_w PPE_w Rev_w LaCFO LeCFO

but this generates coefficients in a new data file for each nic2digit year combinations. But I want the predicted value/residuals for each 'code' after running regression by each nic2digit and year . The 'code' is unique for every company. 'nic2digit' is the industry identifier.. There can be different 'code' under each 'nic2digit' Can someone help me with it??

* Example generated by -dataex-. To install: ssc install dataex
clear
input long code int(year nic2digit) float(ACC_w CFO_w PPE_w Rev_w LaCFO LeCFO)
 11 1999  8            .            .         .             .            .            .
 11 2000  8            .            .         .             .            .            .
 11 2001  8            .            .         .             .            .            .
 11 2002  8            .            .         .             .            .    .05129731
 11 2003  8 -.0035705785    .05129731 .57081646      .5389193            .    .02149475
 11 2004  8  -.064360715    .02149475  .7319333      .6442248    .05129731    .09297886
 11 2005  8   -.14587593    .09297886  .9083844      .7705522    .02149475    .07464965
 11 2006  8   .001665048    .07464965  .9456084      .9281254    .09297886     .0779597
 11 2007  8   .018513855     .0779597   .897733     1.0945843    .07464965    .07567864
 11 2008  8    .02610365    .07567864  .8343296     1.1711544     .0779597  .0012792398
 11 2009  8    .10809576  .0012792398  .7411367      1.256305    .07567864    .02340646
 11 2010  8   .074156255    .02340646  .6722737      1.265667  .0012792398     .0412007
 11 2011  8    .03789344     .0412007   .603212     1.3171893    .02340646    .05202771
 11 2012  8   .032831736    .05202771  .5085389     1.1589074     .0412007     .1516278
 11 2013  8   -.11012533     .1516278  .7010964     1.0286334    .05202771    .07685912
 11 2014  8  -.028768126    .07685912  .6780298      .9607556     .1516278     .0752905
 11 2015  8   -.01868411     .0752905  .6458484      .9593536    .07685912            .
289 1999 52   -.01046043    .07836749  .6197376      1.916488            .    .06201674
289 2000 52   .002710243    .06201674  .6055002       1.72499    .07836749   .034581266
289 2001 52   .016388513   .034581266  .5678845     1.6537362    .06201674    .05632088
289 2002 52  -.016591826    .05632088 .56229544     1.4730192   .034581266     .0962406
289 2003 52   -.04691729     .0962406  .5957895     1.4854136    .05632088    .04815172
289 2004 52  -.013778798    .04815172  .5920439      1.471961     .0962406   .018306635
289 2005 52   .015353953   .018306635  .6179966     1.4623164    .04815172    .06809493
289 2006 52   -.01211859    .06809493  .6357931      1.543966   .018306635            .
289 2007 52            .            .         .             .    .06809493            .
289 2008 52            .            .         .             .            .            .
289 2009 52            .            .         .             .            .            .
289 2010 52            .            .         .             .            .            .
289 2011 52            .            .         .             .            .     .2212753
289 2012 52    -.1381082     .2212753  .6271381     1.5425204            .    .11337147
289 2013 52  -.014867387    .11337147   .627367     1.0816482     .2212753   .032804012
289 2014 52   -.06929935   .032804012  .6669453     1.1240424    .11337147     .0384489
289 2015 52   -.08314164     .0384489  .6258627     1.2276536   .032804012            .
365 1999 30            .            .         .             .            .            .
365 2000 30            .            .         .             .            .            .
365 2001 30            .            .         .             .            . -.0043263286
365 2002 30    .05809642 -.0043263286 .43692285      .5581691            .    .11870936
365 2003 30    -.0775951    .11870936  .3355012      .4123985 -.0043263286   .022299957
365 2004 30    .08607177   .022299957  .4145841      .9940466    .11870936   .008239828
365 2005 30      .244347   .008239828  .4922633     1.4301988   .022299957    .21812125
365 2006 30   -.07961965    .21812125 .22461157      .8472432   .008239828   -.30929035
365 2007 30     .4094186   -.30929035 .13937888     .55072874    .21812125    .08773988
365 2008 30   .015734997    .08773988 .13802543     .51798517   -.30929035   -.07442024
365 2009 30     .1510003   -.07442024  .1898739      .4478916    .08773988   -.06114458
365 2010 30    .13817917   -.06114458 .15522642      .3803483   -.07442024    .23097613
365 2011 30    -.1772575    .23097613 .17284165      .3218299   -.06114458   -.05273814
365 2012 30    .11156725   -.05273814 .15883617     .29010478    .23097613  -.007510649
365 2013 30    .05078186  -.007510649 .14450423     .21438725   -.05273814  -.020489113
365 2014 30      .066911  -.020489113 .11699347      .1403223  -.007510649   -.07074534
365 2015 30    .06130627   -.07074534 .10675485     .03154876  -.020489113            .
400 1999 21  -.009715475     .0388619  .4677307      .3247745            .     .1261745
400 2000 21  -.030872483     .1261745 .41879195      .2214765     .0388619   .015564202
400 2001 21   -.05447471   .015564202  .4856031     .04357977     .1261745 -.0040650405
400 2002 21    -.2703252 -.0040650405  .6321138   .0020325202   .015564202  .0023121387
400 2003 21   -.03699422  .0023121387   .716763     .03468208 -.0040650405    .12642486
400 2004 21    -.1015544    .12642486  .7212435      .5284974  .0023121387    .10670732
400 2005 21  -.033536587    .10670732  .5609756      .8841463    .12642486    -.2972247
400 2006 21     .4726947    -.2972247  .3267681      .7162041    .10670732     .0938647
400 2007 21      .051914     .0938647  .4488726     .56318825    -.2972247   -.05990445
400 2008 21      .168688   -.05990445  .3289232      .6196252     .0938647   .016607355
400 2009 21  -.012940796   .016607355 .19389626     .17577915   -.05990445  -.014462013
400 2010 21    -.3735056  -.014462013  .6935982     .04300039   .016607355  -.011289936
400 2011 21  -.003122748  -.011289936  .8575547             0  -.014462013  -.002899952
400 2012 21  -.001449976  -.002899952  .8629773 -.00024166264  -.011289936            .
400 2013 21            .            .         .             .  -.002899952            .
400 2014 21            .            .         .             .            .            .
400 2015 21            .            .         .             .            .            .
414 1999 46            .            .         .             .            .            .
414 2000 46            .            .         .             .            .    .10636216
414 2001 46   -.09983687    .10636216 .22903752      1.395106            .    .09618574
414 2002 46   -.14726368    .09618574 .23349917       2.66136    .10636216    .20976925
414 2003 46   -.19778244    .20976925  .2091699      1.859155    .09618574   -.08047337
414 2004 46    .09467456   -.08047337  .2005917      1.895858    .20976925    -.1187779
414 2005 46    .12564367    -.1187779  .2279437     2.7092345   -.08047337    .01638807
414 2006 46  -.007866274    .01638807  .2182891     2.2477877    -.1187779    .08676671
414 2007 46   -.08894952    .08676671  .2095498     1.9050478    .01638807    .10735586
414 2008 46   -.09045725    .10735586 .17097415      2.392147    .08676671     .1649399
414 2009 46   -.15375052     .1649399 .14338997      2.220058    .10735586   -.09326247
414 2010 46    .11383507   -.09326247 .12309275      2.726213     .1649399    .10933204
414 2011 46   -.09639334    .10933204 .11612486      3.250526   -.09326247     .1197113
414 2012 46   -.12742658     .1197113 .09059233       2.53783    .10933204     .1598692
414 2013 46   -.17570126     .1598692 .06281191      3.027706     .1197113   -.11782376
414 2014 46    .10063418   -.11782376 .06058078      2.789052     .1598692   -.14030588
414 2015 46     .1349778   -.14030588 .07696103      3.078046   -.11782376            .
415 1999 62  -.033755273   .014064698 .27566808             .            .   .023255814
415 2000 62    .12684989   .023255814  .5285412     1.2198732   .014064698   .073897496
415 2001 62   -.05403258   .073897496 .35359555     .25744936   .023255814   -.12821469
415 2002 62    .15430488   -.12821469  .3317182      .7752516   .073897496    .05435592
415 2003 62   -.05212212    .05435592  .3380491       .641102   -.12821469   -.08753943
415 2004 62    .10015773   -.08753943  .1884858      .4416404    .05435592  .0008396305
415 2005 62  -.004198153  .0008396305  .2031906      .4819479   -.08753943   -.07029877
415 2006 62    .08435852   -.07029877  .2513181       .473638  .0008396305    .09372846
415 2007 62  -.014472777    .09372846  .2046864     1.2549965   -.07029877    .14536454
415 2008 62    .14131413    .14536454 .12826283      1.260126    .09372846    .50344926
415 2009 62    -.3258477    .50344926 .08454426     1.1947747    .14536454   .033081625
415 2010 62      .134484   .033081625 .08222462      .9063886    .50344926   -.14627369
415 2011 62     .3423638   -.14627369  .0724422     1.0449538   .033081625    .08888542
415 2012 62    .13934234    .08888542 .05998594      1.447317   -.14627369    .05757972
415 2013 62    .10372615    .05757972  .3581514      .9354638    .08888542    .08161873
end

[William Lisowski]

Previous discussion of the post #7 problem appears at other topics started by the author; see post #6 and following at https://www.statalist.org/forums/for...r-and-industry
and also see
https://www.statalist.org/forums/for...ustry-and-year

[Priyesh VP]

Previous discussion of the post #7 problem appears at other topics started by the author; see post #6 and following at https://www.statalist.org/forums/for...r-and-industry
and also see
https://www.statalist.org/forums/for...ustry-and-year

 egen group = group(nic2digit year)
 gen residual = .
 foreach i of var group {
  regress  ACC_w CFO_w PPE_w Rev_w LaCFO LeCFO  if group == `i'
  predict temp, residuals
  replace residual=temp if group == `i'
  drop temp
 }

I am getting residuals when I run the above code. But when I manually check it with the coefficients generated from the below code :

statsby _b, by(nic2digit year)clear verbose nodots: regress ACC_w CFO_w PPE_w Rev_w LaCFO LeCFO

I see some difference in the value of residuals ...please advice

[William Lisowski]

I would run

statsby _b, by(group) clear verbose nodots: regress ACC_w CFO_w PPE_w Rev_w LaCFO LeCFO

so that you can confirm that the coefficients you are using from your statsby output are identical to those reported from the regress command for that group in the loop that generates predictions. That seems to me to be the most likely problem.

[Priyesh VP]

 egen group = group(nic2digit year)
gen residual = .
foreach i of var group {
regress ACC_w CFO_w PPE_w Rev_w LaCFO LeCFO if group == `i'
predict temp, residuals
replace residual=temp if group == `i'
drop temp
}

I am getting residuals when I run the above code. But when I manually check it with the coefficients generated from the below code :

statsby _b, by(nic2digit year)clear verbose nodots: regress ACC_w CFO_w PPE_w Rev_w LaCFO LeCFO

I see some difference in the value of residuals ...please advice

Could you please tell me how are these two codes different from each other ?? I wanted to run regression by nic2digit and year, and use this coefficient for each comapny 'code's and predict residuals.

statsby _b, by(group) clear verbose nodots: regress ACC_w CFO_w PPE_w Rev_w LaCFO LeCFO
statsby _b, by(nic2digit year)clear verbose nodots: regress ACC_w CFO_w PPE_w Rev_w LaCFO LeCFO

[Priyesh VP]

I would run

statsby _b, by(group) clear verbose nodots: regress ACC_w CFO_w PPE_w Rev_w LaCFO LeCFO

so that you can confirm that the coefficients you are using from your statsby output are identical to those reported from the regress command for that group in the loop that generates predictions. That seems to me to be the most likely problem.

I am exteremely thankful for your replies.But,
when I run the quoted code, it is not giving me any nic2digit and year identifier. it simply gives unique groups and coefficients.
And it gives same coefficients as given by the code

  statsby _b, by(nic2digit year)clear verbose nodots: regress ACC_w CFO_w PPE_w Rev_w LaCFO LeCFO

[William Lisowski]

You claim you have computed residuals by hand using the statsby results and they do not agree with the residuals computed by Stata.

The reason is that the coefficients produced by your regress loop are different than those produced by statsby, because you did something different when you ran statsby or you made a mistake.

You have created a group variable that identifies each combination of nic2digit and year.

You must run the the statsby using the group variable instead of the nic2digit and year so you can correctly compare the coefficients saved by statsby with the coefficients FOR THE SAME GROUP as reported in your Results window by Stata.

Then find the residual you computed by hand that differed from what Stata computed. What group was it in? Are the coefficients reported by statsby the same as those reported by regress?

This is what you have to do. This is how you find your mistake.

[Priyesh VP]

The residuals I get by running the below code is not what I wanted. It generated residuals after running a pooled regression with all the observations. But what I was looking for is to run regression for each nic2digit and year and use this coefficient to find residuals for each firms belong to respective nic2digit and year. 'Code' is the firm identifier in my data set.

There is no problem with coefficients generated by statsby command. But how to get company specific residuals from these coefficients is my main concern.
please advice

egen group = group(nic2digit year)
gen residual = .
foreach i of var group {
regress ACC_w CFO_w PPE_w Rev_w LaCFO LeCFO if group == `i'
predict temp, residuals
replace residual=temp if group == `i'
drop temp }

[William Lisowski]

The foreach command you have used loops over a list of variable names, not the values of a single variable. In that it is a step backward from the code you presented in post #1 of https://www.statalist.org/forums/for...ustry-and-year .

egen group = group(nic2digit year)
gen residual = .
levelsof group, local(grouplist)
foreach i of local grouplist {
regress ACC_w CFO_w PPE_w Rev_w LaCFO LeCFO if group == `i'
predict temp, residuals if group == `i'
replace residual=temp if group == `i'
drop temp 
}

Subsequent to my post in #10 I realized that

statsby _b, by(group nic2digit year) clear verbose nodots: regress ACC_w CFO_w PPE_w Rev_w LaCFO LeCFO

would give you stored coefficients with the group identifier and the nic2digit and year identifiers.