Announcement

Collapse
No announcement yet.
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Saving Residuals for different Regressions

    Hello,

    I'm currently working on regressing panel-data for every year and industry. What i basically want to do is I want to run regression (1) and use the estimators regression (1) to estimate regression (2) for every year and industry.

    I have the following regressions:
    HTML Code:
    scaled_TACC = a1*inverse_lagTA+a2*scaled_delta_Rev+a3*scaled_PPE
    where i want to use the coefficients to estimate the second regression
    HTML Code:
    NACC= a^1*inverse_lagTA+a^2scaled_delta_Rev_Rec+a^3*scaled_PPE
    I'm currently running the following code:
    Code:
    gen NACC = .
    gen nobs = .
    
    foreach i of local ids {
         foreach y of local years {
              count if !missing(scaled_TACC, inverse_lagTA, term2, scaled_PPE) &  year == `y' & SIC == `i'
              replace nobs = r(N) if year == `y' & SIC == `i'
        
              if r(N) >= 15 & least_number_firms_any_year >= 15 {
                  regress scaled_TACC inverse_lagTA scaled_delta_Rev scaled_PPE if year == `y' & SIC == `i', noconstant
                  predict xb, xb
                  replace DACC = xb if year == `y' & SIC == `i'
                  drop xb
             }
         }
    }
    which only allows me to calculate regression (2) if the would be exactly similar. So my question is how can i fit regression 2 into my code?
    I hope it is clear what I want to do.
    Thank you in advance for your comments and suggestion!
    Last edited by Hannes Meyer; 11 Jul 2018, 15:43.

  • #2
    I don't understand what you want here. What is the correspondence between the variables in your Stata code and the terms in your regression equations? And how do the results of the first regression fit into the second one? It looks to me as if the second equation is simply an independent (non-linear) regression equation that happens to use some of the same predictor variables, but in a very different way. What am I missing here?

    Comment


    • #3
      I edited the regressions above to make it more clear. I hope it's clear now. (Title is wrong as well, since I need the coefficients not the residuals) The first equation estimates the total accruals, while the second one estimates the normal levels of accruals based on total accruals. But regression 2 is adapted to capture any potential accounting discrestion arising from credit sales. In the end i want to estimate discretionary accruals (DACC) by subtracting regression 1 from regression 2.
      HTML Code:
      DACC = TACC/A - NACC

      Comment


      • #4
        I'm still very confused here.

        If NACC is the outcome variable in the second equation, and TACC/A (which, I guess, refers to the variable scaled_TACC), is the outcome variable in the second equation, then both of these variables must already exist in your data set. So you don't need any regressions to just subtract them. Just -gen DACC = scaled_TACC-NACC-. I guess I'm missing something very big here, but I really can't figure out what you are trying to do.

        A few things might help. First post a data example, so we can see what variables you actually have at your disposal and also have a platform for testing out solutions to your problem. Please use the -dataex- command to do this. If you are running version 15.1 or a fully updated version 14.2, it is already part of your official Stata installation. If not, run -ssc install dataex- to get it. Either way, run -help dataex- to read the simple instructions for using it. -dataex- will save you time; it is easier and quicker than typing out tables. It includes complete information about aspects of the data that are often critical to answering your question but cannot be seen from tabular displays or screenshots. It also makes it possible for those who want to help you to create a faithful representation of your example to try out their code, which in turn makes it more likely that their answer will actually work in your data.

        Second, while it is clear that you are trying to loop over combinations of SIC and year, it is completely unclear to me what is supposed to happen inside the loop. Please provide, in words or in Stata code, but not in equations, a step-by-step description of what is supposed to go on in the loop.

        Also, even though I don't see a role for either regression equation, I am intrigued by the second one and where it comes from. I have never in my career seen a model like that where the coefficient of the second variable is the square of the coefficient of the first variable, and the coefficient of the third variable is its cube. It is hard for me to imagine a process in any field of science that leads to such a model. Where does it come from?

        Comment


        • #5
          This is what my data acutally looks like:
          Code:
          * Example generated by -dataex-. To install: ssc install dataex
          clear
          input float(year SIC industry FirmID scaled_TACC inverse_lagTA scaled_delta_Rev scaled_delta_Rev_Rec scaled_PPE)
          2007 73 62  1           .             .           .            .             .
          2008 73 62  1 -.064041965 .000011043622   .05474323     .0437217     .08837107
          2009 73 62  1    .0424292   9.74488e-06    .1889922    .08669044     .09105615
          2010 73 62  1  -.05635441  6.462662e-06    .0953695    .13293695      .0886031
          2011 73 62  1  -.05877345  5.911633e-06    .0542215    .03515648     .08550586
          2012 73 62  1   .02002721  5.692783e-06   .08318295    .06664541     .08107662
          2008 37 30  2           .             .           .            .             .
          2009 37 30  2           .   .0003973977   -.6110033    -.6405323      .3016928
          2004 34 27  3           .             .           .            .             .
          2006 34 27  3           .             .           .            .             .
          2007 34 27  3 -.005745843  .00002158842  -.01185802  .0003084769     .27716365
          2008 34 27  3 .0032013026 .000022198585    .0925523    .09375273     .28016526
          2009 34 27  3  .015786149 .000022660246  -.05269361   -.05668016      .2667238
          2010 34 27  3           . .000022313105   .02909406    .02614882      .2558447
          2004 26 19  4           .             .           .            .             .
          2005 26 19  4   .10586265  6.913292e-06  -.13408984   -.16030768     .17349124
          2006 26 19  4   -.1597143  6.655715e-06 -.033877507     .2150405     .18316044
          2007 26 19  4  -.16363212  7.725956e-06    .0273716    .05493992     .23311955
          2008 26 19  4 -.028950347  7.767656e-06  -.10793702   -.12260135      .1743541
          2009 26 19  4  .011278813  8.731598e-06   -.3397753    -.3023656     .17712587
          2010 26 19  4  -.07318856  .00001035321    -.305591    -.3190612      .1950248
          2011 26 19  4   -.3314434  .00001105249  -.08390167   -.06395346      .1874068
          2012 26 19  4  -.23247333 .000012657474   -1.558052   -1.5283642     .10567626
          2004 49 42  5           .             .           .            .             .
          2005 49 42  5  -.05223614 1.7889087e-07    .0842576     .0608229      .2076923
          2006 49 42  5   .24512535 1.6385384e-07    .3596592    .53334427      .1153531
          2007 49 42  5    .1557842 1.3190872e-07  -.45455745    -.4571956     .08415776
          2004 36 29  6           .             .           .            .             .
          2005 36 29  6  -.05002641  1.257577e-08 .0033954575  -.017379712     .15106013
          2006 36 29  6  -.15567543 1.1600255e-08   -.0995534   -.06220057     .13854185
          2007 36 29  6  -.07791793 1.1424915e-08   .06868659    .07618134     .11820217
          2008 36 29  6   .05697122 1.0922396e-08   .05729889    .04324177     .12131505
          2009 36 29  6  .017382467 1.0586155e-08  -.00696569  .0082466155     .11810973
          2010 36 29  6 -.011082317 1.0534522e-08   -.0832122   -.10855825     .12375956
          2011 36 29  6  -.05715425  9.725072e-09   .04136073    .06483705     .10188958
          2012 36 29  6    .0229272   9.59297e-09   .03691375    .03478411     .10324914
          2005 17 12  7           .             .           .            .             .
          2009 17 12  7           .             .           .            .             .
          2010 17 12  7  .017283676  .00011043106   -.5992419    -.7328324     .06559948
          2012 17 12  7           .             .           .            .             .
          2004 56 49  8           .             .           .            .             .
          2005 56 49  8           .   .0001835101   .12763603    .13548623     .06779468
          2004 59 52  9           .             .           .            .             .
          2005 59 52  9           .  .00028730818   .09389692     .0882809 2.0111572e-06
          2004 39 32 10           .             .           .            .             .
          2005 39 32 10   .05832781  .00004244612    .2757242    .28684217     .12767793
          2006 39 32 10     .179272  .00004167882    .2514947    .25221267     .12241229
          2007 39 32 10  -.08350068  .00004188807    .3481759     .3321634      .1674311
          2008 39 32 10   -.2565574  .00003603986    .1743413     .2134869     .20598297
          2004 36 29 11           .             .           .            .             .
          2004 28 21 12           .             .           .            .             .
          2005 28 21 12  -.05657782  .00005476912   .14782524    .14710678      .5028072
          2004 34 27 13           .             .           .            .             .
          2006 34 27 13           .             .           .            .             .
          2008 34 27 13           .             .           .            .             .
          2004 16 11 14           .             .           .            .             .
          2005 16 11 14 -.025869593  .00023994873   -.4895321    -.5307421     .18082464
          2005 35 28 15           .             .           .            .             .
          2006 35 28 15  -.48599875  .00002007347  -.20515086   -.16990185     .13613826
          2007 35 28 15    .4586299 .000022241993 -.006605872  .0017126334     .13727757
          2008 35 28 15   -.6166614   .0000227056  -.23325463    -.1768085     .08936924
          2009 35 28 15           .  .00003327012  -.22317597   -.09328942     .12629338
          2010 35 28 15           .  .00004149894   .13947794 -.0019919493     .16997966
          2011 35 28 15  -.01843829 .000034658442    .4297993     .4407167     .11371434
          2012 35 28 15  -.09829796 .000034454246   -.3006133   -.23649393     .12624036
          2004 27 20 16           .             .           .            .             .
          2005 27 20 16 -.007226659  4.179917e-07  -.02777764    -.0172292      .3266697
          2006 27 20 16   .07796557  3.828485e-07  -.02110682   -.03032352      .2887838
          2007 27 20 16  -.12072296  3.201046e-07   .06983498    .04880827     .24225163
          2008 27 20 16           . 2.6130544e-07   .03439041   .035526045      .1887276
          2009 27 20 16           . 3.5599035e-07  -.04732642    -.0605237      .2508849
          2010 27 20 16 -.032181077  3.407966e-07   .12324805    .09463716      .2340168
          2011 27 20 16  -.05775018  2.775277e-07    .0602224    .04454209     .19368657
          2012 27 20 16  -.05578507 2.3880597e-07  -.09019701   -.10457313     .16496716
          2005 34 27 17           .             .           .            .             .
          2006 34 27 17   .12396564 3.9417255e-06   .17406686     .1382114     .12176591
          2007 34 27 17   .19435707  3.406216e-06   .18023315     .1901551      .1043859
          2008 34 27 17   .08675928 3.0479064e-06  .034589373    .02248566     .08616287
          2009 34 27 17  -.03993307  2.802736e-06   -.1535431   -.10132599     .06595671
          2010 34 27 17  -.02116891  2.866856e-06  -.04312395   -.04442452     .06187355
          2011 34 27 17    .0765991  2.655893e-06  .002962155  .0019357105     .06152748
          2012 34 27 17   .04550003  2.540388e-06    .0607046    .05360237     .06205109
          2006 47 40 18           .             .           .            .             .
          2007 47 40 18   -.1790309  .00004863209  -.07100036  -.008597959      .3942361
          2008 47 40 18  -.04211431  .00004032517   -.1380649   -.12663227      .3067211
          2004 27 20 19           .             .           .            .             .
          2005 27 20 19   .05074746  .00004016114   .13332526    .11277413   .0044886097
          2006 27 20 19  -.01479626  .00003739479  -.01894035   -.04204906     .00347622
          2007 27 20 19  -.05075476  .00003866375  .007910605    .03464017    .003326204
          2008 27 20 19   -.1830412  .00004013503   .15533605    .13090642   .0041317004
          2009 27 20 19   .21591674  .00003730301   .07617088    .09461588    .005405505
          2010 27 20 19   -.3190426 .000035629677   .04302636   .028077574    .007479809
          2011 27 20 19    -.305586  .00005512308   .08569715    .07061802     .00956016
          2012 27 20 19  .002814049  .00005770482     .222179    .19024684    .007306584
          2004 36 29 20           .             .           .            .             .
          2005 36 29 20           .   .0000941112   .10851935    .06079575      .6305165
          2006 36 29 20           .  .00008462737   .16897386    .08950724     .54505545
          2007 36 29 20    .2003906  .00007831763  .033038605    .09227767      .4794851
          2008 36 29 20 -.034147244  .00008571526   .20678908     .1119508      .6075081
          2009 36 29 20  -.05691832  .00006822186    -.240616   -.20843142      .5406997
          end
          format %ty year
          So my goal is to estimate the levels of discretionary accruals, by substracting total accruals from normal accruals. The problem is only the level of total accruals is known. Since the normal and abnormal (discretionary) levels are unknown. I have to guess the normal levels based on the total amount of accruals. So I estimate the total level of accruals and use the coefficients to calculate the normal levels.
          Oh my god, I'm so sorry! It isn't the square or cube. It is a_circumflex_2 and a_circumflex_3 if that makes sense, I didn't knew how to make that work. I acutally tried HTML_Codes for a_circumflex but they didn't showed in the preview. So a_circumflex is the estimated coefficient from regression 1.
          Thank you for your patience!

          Comment


          • #6
            I think I get it this time. See if this does what you want:
            Code:
            * Example generated by -dataex-. To install: ssc install dataex
            clear
            input float(year SIC industry FirmID scaled_TACC inverse_lagTA scaled_delta_Rev scaled_delta_Rev_Rec scaled_PPE)
            2007 73 62  1           .             .           .            .             .
            2008 73 62  1 -.064041965 .000011043622   .05474323     .0437217     .08837107
            2009 73 62  1    .0424292   9.74488e-06    .1889922    .08669044     .09105615
            2010 73 62  1  -.05635441  6.462662e-06    .0953695    .13293695      .0886031
            2011 73 62  1  -.05877345  5.911633e-06    .0542215    .03515648     .08550586
            2012 73 62  1   .02002721  5.692783e-06   .08318295    .06664541     .08107662
            2008 37 30  2           .             .           .            .             .
            2009 37 30  2           .   .0003973977   -.6110033    -.6405323      .3016928
            2004 34 27  3           .             .           .            .             .
            2006 34 27  3           .             .           .            .             .
            2007 34 27  3 -.005745843  .00002158842  -.01185802  .0003084769     .27716365
            2008 34 27  3 .0032013026 .000022198585    .0925523    .09375273     .28016526
            2009 34 27  3  .015786149 .000022660246  -.05269361   -.05668016      .2667238
            2010 34 27  3           . .000022313105   .02909406    .02614882      .2558447
            2004 26 19  4           .             .           .            .             .
            2005 26 19  4   .10586265  6.913292e-06  -.13408984   -.16030768     .17349124
            2006 26 19  4   -.1597143  6.655715e-06 -.033877507     .2150405     .18316044
            2007 26 19  4  -.16363212  7.725956e-06    .0273716    .05493992     .23311955
            2008 26 19  4 -.028950347  7.767656e-06  -.10793702   -.12260135      .1743541
            2009 26 19  4  .011278813  8.731598e-06   -.3397753    -.3023656     .17712587
            2010 26 19  4  -.07318856  .00001035321    -.305591    -.3190612      .1950248
            2011 26 19  4   -.3314434  .00001105249  -.08390167   -.06395346      .1874068
            2012 26 19  4  -.23247333 .000012657474   -1.558052   -1.5283642     .10567626
            2004 49 42  5           .             .           .            .             .
            2005 49 42  5  -.05223614 1.7889087e-07    .0842576     .0608229      .2076923
            2006 49 42  5   .24512535 1.6385384e-07    .3596592    .53334427      .1153531
            2007 49 42  5    .1557842 1.3190872e-07  -.45455745    -.4571956     .08415776
            2004 36 29  6           .             .           .            .             .
            2005 36 29  6  -.05002641  1.257577e-08 .0033954575  -.017379712     .15106013
            2006 36 29  6  -.15567543 1.1600255e-08   -.0995534   -.06220057     .13854185
            2007 36 29  6  -.07791793 1.1424915e-08   .06868659    .07618134     .11820217
            2008 36 29  6   .05697122 1.0922396e-08   .05729889    .04324177     .12131505
            2009 36 29  6  .017382467 1.0586155e-08  -.00696569  .0082466155     .11810973
            2010 36 29  6 -.011082317 1.0534522e-08   -.0832122   -.10855825     .12375956
            2011 36 29  6  -.05715425  9.725072e-09   .04136073    .06483705     .10188958
            2012 36 29  6    .0229272   9.59297e-09   .03691375    .03478411     .10324914
            2005 17 12  7           .             .           .            .             .
            2009 17 12  7           .             .           .            .             .
            2010 17 12  7  .017283676  .00011043106   -.5992419    -.7328324     .06559948
            2012 17 12  7           .             .           .            .             .
            2004 56 49  8           .             .           .            .             .
            2005 56 49  8           .   .0001835101   .12763603    .13548623     .06779468
            2004 59 52  9           .             .           .            .             .
            2005 59 52  9           .  .00028730818   .09389692     .0882809 2.0111572e-06
            2004 39 32 10           .             .           .            .             .
            2005 39 32 10   .05832781  .00004244612    .2757242    .28684217     .12767793
            2006 39 32 10     .179272  .00004167882    .2514947    .25221267     .12241229
            2007 39 32 10  -.08350068  .00004188807    .3481759     .3321634      .1674311
            2008 39 32 10   -.2565574  .00003603986    .1743413     .2134869     .20598297
            2004 36 29 11           .             .           .            .             .
            2004 28 21 12           .             .           .            .             .
            2005 28 21 12  -.05657782  .00005476912   .14782524    .14710678      .5028072
            2004 34 27 13           .             .           .            .             .
            2006 34 27 13           .             .           .            .             .
            2008 34 27 13           .             .           .            .             .
            2004 16 11 14           .             .           .            .             .
            2005 16 11 14 -.025869593  .00023994873   -.4895321    -.5307421     .18082464
            2005 35 28 15           .             .           .            .             .
            2006 35 28 15  -.48599875  .00002007347  -.20515086   -.16990185     .13613826
            2007 35 28 15    .4586299 .000022241993 -.006605872  .0017126334     .13727757
            2008 35 28 15   -.6166614   .0000227056  -.23325463    -.1768085     .08936924
            2009 35 28 15           .  .00003327012  -.22317597   -.09328942     .12629338
            2010 35 28 15           .  .00004149894   .13947794 -.0019919493     .16997966
            2011 35 28 15  -.01843829 .000034658442    .4297993     .4407167     .11371434
            2012 35 28 15  -.09829796 .000034454246   -.3006133   -.23649393     .12624036
            2004 27 20 16           .             .           .            .             .
            2005 27 20 16 -.007226659  4.179917e-07  -.02777764    -.0172292      .3266697
            2006 27 20 16   .07796557  3.828485e-07  -.02110682   -.03032352      .2887838
            2007 27 20 16  -.12072296  3.201046e-07   .06983498    .04880827     .24225163
            2008 27 20 16           . 2.6130544e-07   .03439041   .035526045      .1887276
            2009 27 20 16           . 3.5599035e-07  -.04732642    -.0605237      .2508849
            2010 27 20 16 -.032181077  3.407966e-07   .12324805    .09463716      .2340168
            2011 27 20 16  -.05775018  2.775277e-07    .0602224    .04454209     .19368657
            2012 27 20 16  -.05578507 2.3880597e-07  -.09019701   -.10457313     .16496716
            2005 34 27 17           .             .           .            .             .
            2006 34 27 17   .12396564 3.9417255e-06   .17406686     .1382114     .12176591
            2007 34 27 17   .19435707  3.406216e-06   .18023315     .1901551      .1043859
            2008 34 27 17   .08675928 3.0479064e-06  .034589373    .02248566     .08616287
            2009 34 27 17  -.03993307  2.802736e-06   -.1535431   -.10132599     .06595671
            2010 34 27 17  -.02116891  2.866856e-06  -.04312395   -.04442452     .06187355
            2011 34 27 17    .0765991  2.655893e-06  .002962155  .0019357105     .06152748
            2012 34 27 17   .04550003  2.540388e-06    .0607046    .05360237     .06205109
            2006 47 40 18           .             .           .            .             .
            2007 47 40 18   -.1790309  .00004863209  -.07100036  -.008597959      .3942361
            2008 47 40 18  -.04211431  .00004032517   -.1380649   -.12663227      .3067211
            2004 27 20 19           .             .           .            .             .
            2005 27 20 19   .05074746  .00004016114   .13332526    .11277413   .0044886097
            2006 27 20 19  -.01479626  .00003739479  -.01894035   -.04204906     .00347622
            2007 27 20 19  -.05075476  .00003866375  .007910605    .03464017    .003326204
            2008 27 20 19   -.1830412  .00004013503   .15533605    .13090642   .0041317004
            2009 27 20 19   .21591674  .00003730301   .07617088    .09461588    .005405505
            2010 27 20 19   -.3190426 .000035629677   .04302636   .028077574    .007479809
            2011 27 20 19    -.305586  .00005512308   .08569715    .07061802     .00956016
            2012 27 20 19  .002814049  .00005770482     .222179    .19024684    .007306584
            2004 36 29 20           .             .           .            .             .
            2005 36 29 20           .   .0000941112   .10851935    .06079575      .6305165
            2006 36 29 20           .  .00008462737   .16897386    .08950724     .54505545
            2007 36 29 20    .2003906  .00007831763  .033038605    .09227767      .4794851
            2008 36 29 20 -.034147244  .00008571526   .20678908     .1119508      .6075081
            2009 36 29 20  -.05691832  .00006822186    -.240616   -.20843142      .5406997
            end
            format %ty year
            
            capture program drop once
            program define once
                capture regress scaled_TACC inverse_lagTA scaled_delta_Rev scaled_PPE
                if c(rc) == 0 & e(N) >= 15 & e(df_m) == 3 {
                    gen nobs = e(N)
                    predict NACC, xb
                    replace NACC = NACC + _b[scaled_delta_Rev]*(scaled_delta_Rev_Rec-scaled_delta_Rev)
                    gen DACC = scaled_TACC - NACC
                    exit 0
                }
                else if !inlist(c(rc), 2000, 2001) {  //    UNEXPECTED REGRESSION ERROR
                    exit c(rc)
                }
                else    {    //    NO OR INSUFFICIENT OBSERVATIONS TO REGRESS; IGNORE
                    exit 0
                }
            end
            
            runby once, by(SIC year)
            Notes: To use this code you will need to install the -runby- command, written by Robert Picard and me, available from SSC. -runby- can be used in place of looping, and if your real data set is large, it is appreciably faster. But you can think of it as just a fancy version of the -by:- prefix applied to program once.

            Program once is the meat of the problem. I begin by regressing the observed scaled_TACC on the predictors you specified. I notice that you only want regressions with at least 15 observations. I noted during testing, however, that there are, at least in this data example, some situations where the missing data is so afflicted with missing values of the variables that you end up with less than even three observations, in which case Stata then finds colinearity among the three predictors and omits one or more of them from the regression model. So, just as a safety precaution against some inadvertent colinearity arising even when there are 15 observations, I have added the condition that the model must contain all three predictors (e(df_m) == 3). When those conditions are met, we record the number of observations in variable nobs, and make a preliminary calculation of NACC using -predict, xb-. We then "correct" NACC, which was calculated using variable scaled_delta_Rev, by adding the coefficient of scaled_delta_Rev times the difference between scaled_delta_Rev_Red and scaled_delta_Rev. The result will be the estimate you would get from your second equation in the original post (now that I understand the coefficients properly.) Finally we calculate DACC as the difference between scaled_TACC and NACC.

            In the event the regression did not run successfully due to a problem other than too few observations, this is an unexpected error and we note as much by exiting with the corresponding error code. If the regression failed, but it was just due to too few observations, we treat that as an expected error and ignore it.

            Unless you have some other use for them later on, I did not perceive any reason to save the coefficients or residuals from the regression, so I did not do that. If you really need them, it is a simple modification to include those steps in -program once-.

            If your data set is large, even with -runby- this could take a long time to run. So that you can see that Stata is making progress through the data, you may choose to add the -status- option to the -runby- command. If you do that, Stata will give you a periodic update of its progress.
            Last edited by Clyde Schechter; 11 Jul 2018, 17:21. Reason: Additional typo in code: scaled_delta_rev changed to scaled_delta_Rev

            Comment


            • #7
              So i basically need at least 15 firms per industry and year with fully available data. Everything is working perfectly, thank you so much!

              Comment


              • #8
                Hello,

                I have encountered another problem with the code or rather a similiar code, where i tried to adapt the code you gave me. So I adapted your code and I have been running three regressions in a row realising that everytime i ran the regression it dropped observations. At the start i had over 136,000 while in the there where only about 700 left.

                Code:
                . capture program drop once
                
                . 
                . program define once
                  1. 
                .     capture regress scaled_CFO inverse_lagTA scaled_Sales scaled_deltaSales
                  2. 
                .     if c(rc) == 0 & e(N) >= 10 & e(df_m) == 3 {
                  3. 
                .         gen nobs = e(N)
                  4. 
                .         predict NCFO, xb
                  5. 
                .         gen CF_Manipulation2 = scaled_CFO - NCFO
                  6. 
                .         exit 0
                  7. 
                .     }
                  8. 
                .     else if !inlist(c(rc), 2000, 2001) {  //    UNEXPECTED REGRESSION ERROR
                  9. 
                .         exit c(rc)
                 10. 
                .     }
                 11. 
                .     else    {    //    NO OR INSUFFICIENT OBSERVATIONS TO REGRESS; IGNORE
                 12. 
                .         exit 0
                 13. 
                .     }
                 14. 
                . end
                
                . 
                . 
                . 
                . runby once, by(SIC year) status
                
                  elapsed ----------- by-groups ----------    ------- observations ------       time
                     time      count     errors    no-data        processed         saved  remaining
                ------------------------------------------------------------------------------------
                 00:00:01        107         11          0           17,152        17,137   00:00:07
                 00:00:02        214         12          0           39,635        39,617   00:00:05
                 00:00:03        320         13          0           62,072        62,042   00:00:04
                 00:00:04        414         13          0           99,887        99,857   00:00:01
                (now reporting every 5 seconds)
                 00:00:05        573         23          0          136,495       136,440   00:00:00
                
                --------------------------------------
                Number of by-groups    =           573
                by-groups with errors  =            23
                by-groups with no data =             0
                Observations processed =       136,495
                Observations saved     =       136,440
                --------------------------------------
                
                . 
                . 
                . 
                . capture program drop once
                
                . 
                . program define once
                  1. 
                .     capture regress scaled_PROD inverse_lagTA scaled_Sales scaled_deltaSales lag_scaled_delta_Sales
                  2. 
                .     if c(rc) == 0 & e(N) >= 10 & e(df_m) == 4 {
                  3. 
                .         gen nobs = e(N)
                  4. 
                .         predict NPROD, xb
                  5. 
                .         gen PROD_Manipulation2 = scaled_PROD - NPROD
                  6. 
                .         exit 0
                  7. 
                .     }
                  8. 
                .     else if !inlist(c(rc), 2000, 2001) {  //    UNEXPECTED REGRESSION ERROR
                  9. 
                .         exit c(rc)
                 10. 
                .     }
                 11. 
                .     else    {    //    NO OR INSUFFICIENT OBSERVATIONS TO REGRESS; IGNORE
                 12. 
                .         exit 0
                 13. 
                .     }
                 14. 
                . end
                
                . 
                . 
                . 
                . runby once, by(SIC year) status
                
                  elapsed ----------- by-groups ----------    ------- observations ------       time
                     time      count     errors    no-data        processed         saved  remaining
                ------------------------------------------------------------------------------------
                 00:00:01        124         91          0           19,091           324   00:00:06
                 00:00:02        256        222          0           55,718           327   00:00:03
                 00:00:03        387        339          0           98,179           544   00:00:01
                 00:00:04        520        460          0          135,883           797   00:00:00
                 00:00:04        550        470          0          136,440         1,134   00:00:00
                
                --------------------------------------
                Number of by-groups    =           550
                by-groups with errors  =           470
                by-groups with no data =             0
                Observations processed =       136,440
                Observations saved     =         1,134
                --------------------------------------
                
                . 
                . 
                . 
                . capture program drop once
                
                . 
                . program define once
                  1. 
                .     capture regress scaled_DISEXP inverse_lagTA lag_scaled_Sales
                  2. 
                .     if c(rc) == 0 & e(N) >= 10 & e(df_m) == 2 {
                  3. 
                .         gen nobs = e(N)
                  4. 
                .         predict NDISEXP, xb
                  5. 
                .         gen DISEXP_Manipulation2 = scaled_DESEXP - NDISEXP
                  6. 
                .         exit 0
                  7. 
                .     }
                  8. 
                .     else if !inlist(c(rc), 2000, 2001) {  //    UNEXPECTED REGRESSION ERROR
                  9. 
                .         exit c(rc)
                 10. 
                .     }
                 11. 
                .     else    {    //    NO OR INSUFFICIENT OBSERVATIONS TO REGRESS; IGNORE
                 12. 
                .         exit 0
                 13. 
                .     }
                 14. 
                . end
                
                . 
                . 
                . 
                . runby once, by(SIC year) status
                
                  elapsed ----------- by-groups ----------    ------- observations ------       time
                     time      count     errors    no-data        processed         saved  remaining
                ------------------------------------------------------------------------------------
                 00:00:01         80         17          0            1,134           703   00:00:00
                
                --------------------------------------
                Number of by-groups    =            80
                by-groups with errors  =            17
                by-groups with no data =             0
                Observations processed =         1,134
                Observations saved     =           703
                --------------------------------------
                Is there a way to save all observations after each regression and not dropping them?

                Comment


                • #9
                  After -runby-, memory contains only the results for those by-groups that provoked no errors in -program once- and resulted in actual regression results. If you run -runby- successively like this, in each cycle you start with just the data that "survived" the preceding -runby-. That is not what you want, because, for example a by-group might have adequate data for the third round even though it fails for the first or second.

                  So what you have to do is, after each -runby-, integrate the results back with the starting data. To do that you need to create a variable that uniquely identifies all of the observations in the starting data. The simplest way to do this is just:

                  Code:
                  gen long starting_obs_no = _n
                  tempfile building
                  save `building'
                  Then for each round of -runby- you do it like this:

                  Code:
                  runby once, by(SIC year) status
                  merge 1:1 starting_obs_no using `building', nogenerate
                  save "`building'", replace
                  At the end, you will have all of your original observations in memory; those for which the various regressions could be carried out will also have the results you created.

                  Note: Had I known from the start that you would be doing several manipulations like this, I would have built them all into -program once- from the very beginning, and it could all have be done in a single -runby-, avoiding this problem. But as you have already undertaken these code modifications, I'm building on what you have done rather than taking a different approach.

                  Comment


                  • #10
                    Ah I see. But it's not working for me, depending on the regression I choose to run first i only get a new variable for the first regression. For example if I choose to regress R_CF first and then R_PROD I only get R_CF and the other way around. I'm using this code right now:
                    Code:
                    gen long starting_obs_no = _n
                    tempfile building
                    save `building'
                     
                     capture program drop once
                    program define once 
                        capture regress scaled_CFO inverse_lagTA scaled_Sales scaled_deltaSales
                        if c(rc) == 0 & e(N) >= 10 & e(df_m) == 3 {
                             gen nobs = e(N) 
                             predict NCFO, xb
                            gen R_CF = scaled_CFO - NCFO
                             exit 0
                         }
                         else if !inlist(c(rc), 2000, 2001) {  //    UNEXPECTED REGRESSION ERROR
                             exit c(rc)
                         }
                        else    {    //    NO OR INSUFFICIENT OBSERVATIONS TO REGRESS; IGNORE
                             exit 0
                         } 
                     end
                    
                    runby once, by(SIC year) status
                    merge 1:1 starting_obs_no using `building', nogenerate
                    save "`building'", replace
                    
                    
                     capture program drop once
                    program define once 
                        capture regress scaled_PROD inverse_lagTA scaled_Sales scaled_deltaSales lag_scaled_delta_Sales
                        if c(rc) == 0 & e(N) >= 10 & e(df_m) == 4 {
                             gen nobs = e(N) 
                             predict NPROD, xb
                            gen R_PROD = scaled_PROD - NPROD
                             exit 0
                         }
                         else if !inlist(c(rc), 2000, 2001) {  //    UNEXPECTED REGRESSION ERROR
                             exit c(rc)
                         }
                        else    {    //    NO OR INSUFFICIENT OBSERVATIONS TO REGRESS; IGNORE
                             exit 0
                         } 
                     end
                    
                    runby once, by(SIC year) status
                    merge 1:1 starting_obs_no using `building', nogenerate
                    save "`building'", replace
                    
                    
                     capture program drop once
                    program define once 
                        capture regress scaled_DISEXP inverse_lagTA lag_scaled_Sales 
                        if c(rc) == 0 & e(N) >= 10 & e(df_m) == 2 {
                             gen nobs = e(N) 
                             predict NDISEXP, xb
                            gen R_DISEXP = scaled_DISEXP - NDISEXP
                             exit 0
                         }
                         else if !inlist(c(rc), 2000, 2001) {  //    UNEXPECTED REGRESSION ERROR
                             exit c(rc)
                         }
                        else    {    //    NO OR INSUFFICIENT OBSERVATIONS TO REGRESS; IGNORE
                             exit 0
                         } 
                     end
                    
                    runby once, by(SIC year) status
                    merge 1:1 starting_obs_no using `building', nogenerate
                    save "`building'", replace
                    My results:
                    Code:
                    . gen long starting_obs_no = _n
                    
                    . 
                    . tempfile building
                    
                    . 
                    . save `building'
                    file C:\Users\Hannes\AppData\Local\Temp\ST_0900000g.tmp saved
                    
                    . 
                    .  
                    . 
                    .  capture program drop once
                    
                    . 
                    . program define once 
                      1. 
                    .     capture regress scaled_CFO inverse_lagTA scaled_Sales scaled_deltaSales
                      2. 
                    .     if c(rc) == 0 & e(N) >= 10 & e(df_m) == 3 {
                      3. 
                    .          gen nobs = e(N) 
                      4. 
                    .          predict NCFO, xb
                      5. 
                    .         gen R_CF = scaled_CFO - NCFO
                      6. 
                    .          exit 0
                      7. 
                    .      }
                      8. 
                    .      else if !inlist(c(rc), 2000, 2001) {  //    UNEXPECTED REGRESSION ERROR
                      9. 
                    .          exit c(rc)
                     10. 
                    .      }
                     11. 
                    .     else    {    //    NO OR INSUFFICIENT OBSERVATIONS TO REGRESS; IGNORE
                     12. 
                    .          exit 0
                     13. 
                    .      } 
                     14. 
                    .  end
                    
                    . 
                    . 
                    . 
                    . runby once, by(SIC year) status
                    
                      elapsed ----------- by-groups ----------    ------- observations ------       time
                         time      count     errors    no-data        processed         saved  remaining
                    ------------------------------------------------------------------------------------
                     00:00:01        103         12          0           20,310        20,293   00:00:06
                     00:00:02        197         13          0           33,577        33,557   00:00:07
                     00:00:03        298         13          0           64,666        64,646   00:00:04
                     00:00:04        391         13          0          105,529       105,509   00:00:02
                    (now reporting every 5 seconds)
                     00:00:06        574         23          0          149,114       149,068   00:00:00
                    
                    --------------------------------------
                    Number of by-groups    =           574
                    by-groups with errors  =            23
                    by-groups with no data =             0
                    Observations processed =       149,114
                    Observations saved     =       149,068
                    --------------------------------------
                    
                    . 
                    . merge 1:1 starting_obs_no using `building', nogenerate
                    
                        Result                           # of obs.
                        -----------------------------------------
                        not matched                            46
                            from master                         0  
                            from using                         46  
                    
                        matched                           149,068  
                        -----------------------------------------
                    
                    . 
                    . save "`building'", replace
                    file C:\Users\Hannes\AppData\Local\Temp\ST_0900000g.tmp saved
                    
                    . 
                    . 
                    . 
                    . 
                    . 
                    .  capture program drop once
                    
                    . 
                    . program define once 
                      1. 
                    .     capture regress scaled_PROD inverse_lagTA scaled_Sales scaled_deltaSales lag_scaled_delta_Sales
                      2. 
                    .     if c(rc) == 0 & e(N) >= 10 & e(df_m) == 4 {
                      3. 
                    .          gen nobs = e(N) 
                      4. 
                    .          predict NPROD, xb
                      5. 
                    .         gen R_PROD = scaled_PROD - NPROD
                      6. 
                    .          exit 0
                      7. 
                    .      }
                      8. 
                    .      else if !inlist(c(rc), 2000, 2001) {  //    UNEXPECTED REGRESSION ERROR
                      9. 
                    .          exit c(rc)
                     10. 
                    .      }
                     11. 
                    .     else    {    //    NO OR INSUFFICIENT OBSERVATIONS TO REGRESS; IGNORE
                     12. 
                    .          exit 0
                     13. 
                    .      } 
                     14. 
                    .  end
                    
                    . 
                    . 
                    . 
                    . runby once, by(SIC year) status
                    
                      elapsed ----------- by-groups ----------    ------- observations ------       time
                         time      count     errors    no-data        processed         saved  remaining
                    ------------------------------------------------------------------------------------
                     00:00:01        134        101          0           22,328           333   00:00:06
                     00:00:02        273        239          0           62,345           336   00:00:03
                     00:00:03        405        358          0          106,873           498   00:00:01
                     00:00:04        541        478          0          148,569           790   00:00:00
                     00:00:04        574        496          0          149,114         1,088   00:00:00
                    
                    --------------------------------------
                    Number of by-groups    =           574
                    by-groups with errors  =           496
                    by-groups with no data =             0
                    Observations processed =       149,114
                    Observations saved     =         1,088
                    --------------------------------------
                    
                    . 
                    . merge 1:1 starting_obs_no using `building', nogenerate
                    
                        Result                           # of obs.
                        -----------------------------------------
                        not matched                       148,026
                            from master                         0  
                            from using                    148,026  
                    
                        matched                             1,088  
                        -----------------------------------------
                    
                    . 
                    . save "`building'", replace
                    file C:\Users\Hannes\AppData\Local\Temp\ST_0900000g.tmp saved
                    
                    . 
                    . 
                    . 
                    . 
                    . 
                    .  capture program drop once
                    
                    . 
                    . program define once 
                      1. 
                    .     capture regress scaled_DISEXP inverse_lagTA lag_scaled_Sales 
                      2. 
                    .     if c(rc) == 0 & e(N) >= 10 & e(df_m) == 2 {
                      3. 
                    .          gen nobs = e(N) 
                      4. 
                    .          predict NDISEXP, xb
                      5. 
                    .         gen R_DISEXP = scaled_DISEXP - NDISEXP
                      6. 
                    .          exit 0
                      7. 
                    .      }
                      8. 
                    .      else if !inlist(c(rc), 2000, 2001) {  //    UNEXPECTED REGRESSION ERROR
                      9. 
                    .          exit c(rc)
                     10. 
                    .      }
                     11. 
                    .     else    {    //    NO OR INSUFFICIENT OBSERVATIONS TO REGRESS; IGNORE
                     12. 
                    .          exit 0
                     13. 
                    .      } 
                     14. 
                    .  end
                    
                    . 
                    . 
                    . 
                    . runby once, by(SIC year) status
                    
                      elapsed ----------- by-groups ----------    ------- observations ------       time
                         time      count     errors    no-data        processed         saved  remaining
                    ------------------------------------------------------------------------------------
                     00:00:02        120         91          0           21,083           263   00:00:10
                     00:00:03        251        221          0           59,080           266   00:00:04
                     00:00:04        382        337          0          104,874           451   00:00:02
                     00:00:05        513        460          0          140,450           551   00:00:00
                    (now reporting every 5 seconds)
                     00:00:05        574        503          0          149,114           740   00:00:00
                    
                    --------------------------------------
                    Number of by-groups    =           574
                    by-groups with errors  =           503
                    by-groups with no data =             0
                    Observations processed =       149,114
                    Observations saved     =           740
                    --------------------------------------
                    
                    . 
                    . merge 1:1 starting_obs_no using `building', nogenerate
                    
                        Result                           # of obs.
                        -----------------------------------------
                        not matched                       148,374
                            from master                         0  
                            from using                    148,374  
                    
                        matched                               740  
                        -----------------------------------------
                    
                    . 
                    . save "`building'", replace
                    file C:\Users\Hannes\AppData\Local\Temp\ST_0900000g.tmp saved
                    
                    .
                    What am I doing wrong?

                    Comment


                    • #11
                      The most recent example data in this thread does not include the variables necessary to run this code. Please post a suitable example data set that has all the needed variables and I will try to troubleshoot.

                      I do notice that, except for the first round, the -runby- output shows the vast majority of by-groups returning errors. The code within the program is pretty simple and straightforward, so I suspect there is something wrong with your data. But send a good example, and let's see. If the example has to be too long to include the -dataex- within a post, attach a .dta file.

                      Comment


                      • #12
                        Since I have no clue how to generate a good example and how to only attach an example of my data, I'll try this:

                        Code:
                        * Example generated by -dataex-. To install: ssc install dataex
                        clear
                        input float(year SIC lag_TA inverse_lagTA delta_Sales scaled_delta_Sales scaled_CFO scaled_COGS scaled_delta_Inv scaled_PROD scaled_DISEXP scaled_Sales lag_scaled_Sales scaled_deltaSales lag_delta_Sales lag_scaled_delta_Sales)
                        2007 73         .             .          .            .            .         .             .         .         .         .         .            .          .           .
                        2008 73     90550 .000011043622       4957    .05474323    .05614578  1.076455    .007631143 1.0840862 .27510768 1.4494975         .    .05474323          .           .
                        2009 73    102618   9.74488e-06      19394     .1889922     .0757372  1.092547     .01126508 1.1038122 .26394978  1.468027 1.4494975     .1889922       4957   .04830537
                        2010 73    154735  6.462662e-06      14757     .0953695    .12048987   .778531    .003651404  .7821825  .2003296 1.0689436  1.468027     .0953695      19394   .12533687
                        2011 73    169158  5.911633e-06       9172     .0542215    .04879462  .7501744    .003818915  .7539933 .19526714 1.0320233 1.0689436     .0542215      14757   .08723797
                        2012 73    175661  5.692783e-06      14612    .08318295    .07772926   .779769 -.00015939793  .7796096  .2009154 1.0770006 1.0320233    .08318295       9172   .05221421
                        2008 37         .             .          .            .            .         .             .         .         .         .         .            .          .           .
                        2009 37  2516.371   .0003973977  -1537.511    -.6110033   -.10840014  .8165517     -.0425601  .7739916 .28876743  1.145097         .    -.6110033          .           .
                        2006 34         .             .          .            .            .         .             .         .         .         .         .            .          .           .
                        2007 34  46321.13  .00002158842   -549.277   -.01185802    .06075676  .7129575   -.011480548  .7014769 .22440113  .9829571         .   -.01185802          .           .
                        2008 34  45047.92 .000022198585   4169.288     .0925523    .09419383  .7569247   -.013041202  .7438835 .25030145 1.1032913  .9829571     .0925523   -549.277  -.01219317
                        2009 34  44130.15 .000022660246  -2325.377   -.05269361    .12867936  .7250485    -.02997794  .6950706 .23792766 1.0735427 1.1032913   -.05269361   4169.288    .0944771
                        2010 34  44816.71 .000022313105     1303.9    .02909406   .013111983  .7341704    .003480599  .7376509 .25094822 1.0861907 1.0735427    .02909406  -2325.377  -.05188638
                        2005 26 144648.89  6.913292e-06 -19395.947   -.13408984    -.0692905  1.728967    -.06106215 1.6679047 .50664717 2.2659085         .   -.13408984          .           .
                        2006 26 150246.83  6.655715e-06  -5089.988  -.033877507     .2660844 1.6180238    .005278867 1.6233027  .4693682 2.1476073 2.2659085  -.033877507 -19395.947  -.12909389
                        2007 26 129433.82  7.725956e-06   3542.811     .0273716  -.029764405 1.9049562     .07255238 1.9775087 .56005466  2.520315 2.1476073     .0273716  -5089.988  -.03932502
                        2008 26 128738.97  7.767656e-06   -13895.7   -.10793702  -.014753057 1.8063644      -.051805 1.7545594 .55602294  2.425981  2.520315   -.10793702   3542.811  .027519336
                        2009 26 114526.57  8.731598e-06 -38913.297    -.3397753    .15665005 1.8228785    -.09677205 1.7261065  .5064454 2.3872626  2.425981    -.3397753   -13895.7  -.12133167
                        2010 26  96588.41  .00001035321 -29516.545     -.305591   -.04264147  1.933141    .011429073   1.94457   .576119  2.525028 2.3872626     -.305591 -38913.297   -.4028775
                        2011 26  90477.36  .00001105249  -7591.201   -.08390167     -.081294   2.07344     .01186793  2.085308 .57923615 2.6116726  2.525028   -.08390167 -29516.545   -.3262313
                        2012 26   79004.7 .000012657474 -123093.43    -1.558052    .11218737 1.0018961     -.1504976  .8513986 .49622685 1.4328743 2.6116726    -1.558052  -7591.201  -.09608544
                        2005 49   5590000 1.7889087e-07     471000     .0842576    .16547406 .22468694   -.005903399 .21878354   .240966 1.0268337         .     .0842576          .           .
                        2006 49   6103000 1.6385384e-07    2195000     .3596592     .4936916  .2120269   -.062756024 .14927085 .28412256 1.3001802 1.0268337     .3596592     471000   .07717516
                        2007 49   7581000 1.3190872e-07   -3446000   -.45455745    .18493603 .14127424   .0007914523 .14206569  .2483841 .59213823 1.3001802   -.45455745    2195000   .28953964
                        2005 17         .             .          .            .            .         .             .         .         .         .         .            .          .           .
                        2009 17         .             .          .            .            .         .             .         .         .         .         .            .          .           .
                        2010 17  9055.423  .00011043106  -5426.388    -.5992419   -.13912028 2.1338818    -.10407686  2.029805 .26633853 2.3823624         .    -.5992419          .           .
                        2012 17         .             .          .            .            .         .             .         .         .         .         .            .          .           .
                        2005 56  5449.292   .0001835101    695.526    .12763603    .09634133 2.0803726     .05293201 2.1333046  .4335514  2.627594         .    .12763603          .           .
                        2005 59  3480.583  .00028730818    326.816    .09389692            .         .             .         . .26493752  5.428855         .    .09389692          .           .
                        2005 39  23559.28  .00004244612   6495.864     .2757242    .14875387  2.421509    .010970072  2.432479    .57315  3.133033         .     .2757242          .           .
                        2006 39     23993  .00004167882   6034.113     .2514947    .18211696 2.5387044    .002214771  2.540919  .6233606  3.327892  3.133033     .2514947   6495.864   .27073994
                        2007 39 23873.146  .00004188807   8312.055     .3481759   -.11569816  2.827488     .12907319  2.956561  .7282655 3.6927755  3.327892     .3481759   6034.113   .25275734
                        2008 39 27747.057  .00003603986   4837.458     .1743413     .1740237  2.643331    -.09395148 2.5493796  .7036378  3.351549 3.6927755     .1743413   8312.055    .2995653
                        2005 28 18258.465  .00005476912   2699.062    .14782524    .12240525 2.1619291     .03790023 2.1998293  .2399218  2.771998         .    .14782524          .           .
                        2006 34         .             .          .            .            .         .             .         .         .         .         .            .          .           .
                        2005 16  4167.557  .00023994873  -2040.153    -.4895321   -.09673293 1.0698402     .17034416 1.2401843  .1557944 1.3340057         .    -.4895321          .           .
                        2005 35         .             .          .            .            .         .             .         .         .         .         .            .          .           .
                        2006 35     49817  .00002007347     -10220   -.20515086   .029588293  .8347151    .020916555  .8556316 .18449526 1.0225023         .   -.20515086          .           .
                        2007 35     44960 .000022241993       -297  -.006605872   .014924377  .9382117     .04326067  .9814724  .1554493 1.1263567 1.0225023  -.006605872     -10220  -.22731316
                        2008 35     44042   .0000227056     -10273   -.23325463     .2298488  .7991236     -.1900913  .6090323 .14890333  .9165796 1.1263567   -.23325463       -297 -.006743563
                        2009 35     30057  .00003327012      -6708   -.22317597     .1156802  .9464684    -.10307083  .8433976 .16541904 1.1198722  .9165796   -.22317597     -10273   -.3417839
                        2010 35     24097  .00004149894       3361    .13947794    -.1956675 1.2258787     .06731129   1.29319 .22861767 1.5363324 1.1198722    .13947794      -6708   -.2783749
                        2011 35     28853 .000034658442      12401     .4297993    .13981216  1.407514   -.007486223 1.4000278  .2161647 1.7128894 1.5363324     .4297993       3361   .11648702
                        2012 35     29024 .000034454246      -8725    -.3006133   -.07400772 1.1602123     .03507442 1.1952866 .21509784 1.4021844 1.7128894    -.3006133      12401    .4272671
                        2005 27   2392392  4.179917e-07     -66455   -.02777764    .08848634  .5918449    .000150059 .59199494  .2853972  1.055286         .   -.02777764          .           .
                        2006 27   2611999  3.828485e-07     -55131   -.02110682    .12118726 .55488497   -.003376341  .5515086   .248373  .9454548  1.055286   -.02110682     -66455   -.0254422
                        2007 27   3123979  3.201046e-07     218163    .06983498   -.04574583  .5072499  -.0007112724  .5065386 .21712662  .8603419  .9454548    .06983498     -55131 -.017647685
                        2008 27   3826939 2.6130544e-07     131610    .03439041    .13706987  .4358133   .0016292394  .4374426 .17993833  .7366984  .8603419    .03439041     218163   .05700718
                        2009 27   2809065 3.5599035e-07    -132943   -.04732642    .15615197  .5974899   -.004387581 .59310234 .24736238  .9563171  .7366984   -.04732642     131610   .04685189
                        2010 27   2934302  3.407966e-07     361647    .12324805    .11937694 .59415084  -.0016552488  .5924956 .26375848 1.0387492  .9563171    .12324805    -132943  -.04530651
                        2011 27   3603244  2.775277e-07     216996     .0602224   .016931131  .5293286  .00040435785 .52973294 .21755396  .9061279 1.0387492     .0602224     361647   .10036706
                        2012 27   4187500 2.3880597e-07    -377700   -.09019701    .09077015  .4130627  -.0003343284  .4127284  .1606209  .6895045  .9061279   -.09019701     216996   .05181994
                        2005 27  24899.69  .00004016114   3319.758    .13332526    .05967242    1.3625   -.001570421 1.3609294   .310077 1.7533756         .    .13332526          .           .
                        2006 27  26741.69  .00003739479   -506.497   -.01894035    .03983992  1.251938   -.009204467 1.2427336  .2859917 1.6136607 1.7533756   -.01894035   3319.758   .12414166
                        2007 27  25864.02  .00003866375      204.6   .007910605    .05463989 1.2824323   .0041905325 1.2866228  .3239576 1.6763295 1.6136607   .007910605   -506.497 -.019583076
                        2008 27  24915.89  .00004013503   3870.336    .15533605    .09943362 1.4681816    .007054654 1.4752364   .317116  1.895455 1.6763295    .15533605      204.6  .008211628
                        2009 27  26807.49  .00003730301    2041.95    .07617088    .11698336 1.4250764    .020373896 1.4454503   .300103 1.8378783  1.895455    .07617088   3870.336   .14437518
                        2010 27  28066.49 .000035629677   1207.599    .04302636    -.3644054 1.3348736   -.009606188 1.3252673   .311618 1.7984613 1.8378783    .04302636    2041.95   .07275401
                        2011 27  18141.22  .00005512308   1554.651    .08569715  -.007516968 2.1274223  -.0009848841 2.1264374 .51285374 2.8681176 1.7984613    .08569715   1207.599   .06656658
                        2012 27 17329.576  .00005770482   3850.268      .222179      .113582  2.454281     .02450527  2.478786  .5413355  3.224627 2.8681176      .222179   1554.651   .08971085
                        2005 36 10625.728   .0000941112   1153.097    .10851935    .17582203  .7820503      .0381268   .820177 .10503666 1.0013665         .    .10851935          .           .
                        2006 36 11816.508  .00008462737   1996.681    .16897386   -.08049147  .8456426     .03964945   .885292 .10057937 1.0694301 1.0013665    .16897386   1153.097   .09758357
                        2007 36 12768.518  .00007831763    421.854   .033038605   .063572615  .7989164    .009737622   .808654  .0913922 1.0227329 1.0694301   .033038605   1996.681    .1563753
                        2008 36 11666.535  .00008571526   2412.512    .20678908    .07919138 1.0304368     .03970785 1.0701445 .12294199  1.326126 1.0227329    .20678908    421.854   .03615932
                        2009 36 14658.058  .00006822186  -3526.963     -.240616    .11598971  .6532929   -.022839246  .6304536 .07525574  .8148646  1.326126     -.240616   2412.512   .16458607
                        2010 36  15180.76  .00006587285    -13.765 -.0009067399   -.06415726  .6408803    .018460538  .6593409 .06515234  .7859005  .8148646 -.0009067399  -3526.963   -.2323311
                        2011 36 14244.012 .000070204944   2672.059    .18759175    .09088232  .7988997    .018098764  .8169984 .09972584 1.0251765  .7859005    .18759175    -13.765 -.000966371
                        2005 51  9897.685  .00010103373  -1115.883   -.11274183    .04724276  .9389247  -.0004239375  .9385008 .04105182  1.102383         .   -.11274183          .           .
                        2007 51         .             .          .            .            .         .             .         .         .         .         .            .          .           .
                        2008 51 10142.917  .00009859096    701.838    .06919489     .2102885 1.1784943    .007710603 1.1862049 .10860021   1.51191         .    .06919489          .           .
                        2009 51 10326.962   .0000968339   1924.686    .18637486     .2653732 1.2473785    .010970506  1.258349 .09878326 1.6713398   1.51191    .18637486    701.838  .067961715
                        2011 51         .             .          .            .            .         .             .         .         .         .         .            .          .           .
                        2012 51 17127.486  .00005838569    762.853    .04453969     .2105515 1.2088088  -.0014135174 1.2073952  .0560673  1.476739         .    .04453969          .           .
                        2005 50  5794.274  .00017258417  -1348.064    -.2326545     .1864924  2.548504    -.03111658 2.5173874 .24594246 3.0115464         .    -.2326545          .           .
                        2006 50  5872.596  .00017028244   3347.222     .5699731     .1451794  2.978324    -.02221607 2.9561074 .28444013  3.541355 3.0115464     .5699731  -1348.064   -.2295516
                        2007 50  6802.668  .00014700115   4747.756     .6979256    .08271432 3.1685615     .07748195 3.2460434  .2601334 3.7551005  3.541355     .6979256   3347.222    .4920455
                        2008 50  10064.32   .0000993609     421.06     .0418369    .12089023  2.188607    -.02207541 2.1665316 .21314128 2.5799816 3.7551005     .0418369   4747.756    .4717413
                        2009 50  15541.87  .00006434232  -1382.913   -.08897983    .03426518 1.3857815   -.003045901 1.3827356 .09586678 1.5817176 2.5799816   -.08897983     421.06   .02709198
                        2010 50 15006.026  .00006663989   3228.182     .2151257    .06066996 1.6119145    .021660764 1.6335752   .123786  1.853324 1.5817176     .2151257  -1382.913  -.09215717
                        2011 50 15551.771 .000064301355   3785.402     .2434065    .08716898  1.739948    .007571935 1.7475202 .13522884 2.0316935  1.853324     .2434065   3228.182   .20757647
                        2012 50 15739.535  .00006353428   1738.283    .11044055    .14221379 1.7839378    .010902228   1.79484  .1576825  2.117897 2.0316935    .11044055   3785.402    .2405028
                        2006 38         .             .          .            .            .         .             .         .         .         .         .            .          .           .
                        2008 38         .             .          .            .            .         .             .         .         .         .         .            .          .           .
                        2009 38  32759.69 .000030525323  -5025.988    -.1534199    .04398173  .6112211   -.004758256  .6064628  .1565949 1.0190045         .    -.1534199          .           .
                        2010 38 31374.127   .0000318734   6673.626    .21271113    .10960853  .7194507    .010027498  .7294782 .19510433 1.2767173 1.0190045    .21271113  -5025.988   -.1601953
                        2005 50         .             .          .            .            .         .             .         .         .         .         .            .          .           .
                        2005 57 12050.197   .0000829862    783.619    .06502956    .04554938  2.579206    .006520225 2.5857265  .6698572  3.223767         .    .06502956          .           .
                        2006 57 11837.733  .00008447563   2627.099    .22192585 -.0007927189  2.738998  -.0017288782  2.737269  .7239562  3.503553  3.223767    .22192585    783.619   .06619671
                        2008 57         .             .          .            .            .         .             .         .         .         .         .            .          .           .
                        2009 57     10251  .00009755146   -4025.97    -.3927392   -.05546474  2.427254    .023188176  2.450442  .7432397  3.133258         .    -.3927392          .           .
                        2010 57 10323.521  .00009686617     868.47    .08412536   .003106789  2.474816   -.006043868  2.468772  .7053314  3.195373  3.133258    .08412536   -4025.97   -.3899803
                        2011 57 10887.182  .00009185114  -1198.237   -.11005943   .027678974 2.2836692  -.0015362103  2.282133  .6579264   2.91988  3.195373   -.11005943     868.47   .07976995
                        2012 57 11271.666  .00008871803   -365.778  -.032451104  -.019048294 2.1899583     .00960967  2.199568  .6227053 2.7878296   2.91988  -.032451104  -1198.237  -.10630523
                        2007 50         .             .          .            .            .         .             .         .         .         .         .            .          .           .
                        2005 50         .             .          .            .            .         .             .         .         .         .         .            .          .           .
                        2005 27         .             .          .            .            .         .             .         .         .         .         .            .          .           .
                        2005 22         .             .          .            .            .         .             .         .         .         .         .            .          .           .
                        2006 22     13816  .00007237985   4330.032     .3134071  .0022084536 2.2285807    .002683917 2.2312646 .24496387  2.642808         .     .3134071          .           .
                        2007 22 12941.748  .00007726932    521.133    .04026759  -.014155814  2.309697    .032501716 2.3421984  .2769337 2.8616045  2.642808    .04026759   4330.032    .3345786
                        end
                        format %ty year



                        What bothers me is If i am running I am getting a lot of errors for the other regressions but not the first one, even though it's a different one now:
                        Code:
                        . tempfile building
                        
                        . 
                        . save `building'
                        file C:\Users\Hannes\AppData\Local\Temp\ST_0900000h.tmp saved
                        
                        .  capture program drop once
                        
                        . 
                        . program define once 
                          1. 
                        .     capture regress scaled_PROD inverse_lagTA scaled_Sales scaled_deltaSales lag_scaled_delta_Sales
                          2. 
                        .     if c(rc) == 0 & e(N) >= 10 & e(df_m) == 4 {
                          3. 
                        .          gen nobs = e(N) 
                          4. 
                        .          predict NPROD, xb
                          5. 
                        .         gen R_PROD = scaled_PROD - NPROD
                          6. 
                        .          exit 0
                          7. 
                        .      }
                          8. 
                        .      else if !inlist(c(rc), 2000, 2001) {  //    UNEXPECTED REGRESSION ERROR
                          9. 
                        .          exit c(rc)
                         10. 
                        .      }
                         11. 
                        .     else    {    //    NO OR INSUFFICIENT OBSERVATIONS TO REGRESS; IGNORE
                         12. 
                        .          exit 0
                         13. 
                        .      } 
                         14. 
                        .  end
                        
                        . 
                        . 
                        . 
                        . runby once, by(SIC year) status
                        
                          elapsed ----------- by-groups ----------    ------- observations ------       time
                             time      count     errors    no-data        processed         saved  remaining
                        ------------------------------------------------------------------------------------
                         00:00:01        116         28          0           20,807        18,375   00:00:06
                         00:00:02        223         48          0           51,817        45,045   00:00:04
                         00:00:03        333         63          0           77,238        67,654   00:00:03
                         00:00:04        433         76          0          110,734        97,230   00:00:01
                        (now reporting every 5 seconds)
                         00:00:06        574        108          0          149,114       131,538   00:00:00
                        
                        --------------------------------------
                        Number of by-groups    =           574
                        by-groups with errors  =           108
                        by-groups with no data =             0
                        Observations processed =       149,114
                        Observations saved     =       131,538
                        --------------------------------------
                        
                        . 
                        . merge 1:1 starting_obs_no using `building', nogenerate
                        
                            Result                           # of obs.
                            -----------------------------------------
                            not matched                        17,576
                                from master                         0  
                                from using                     17,576  
                        
                            matched                           131,538  
                            -----------------------------------------
                        
                        . 
                        . save "`building'", replace
                        file C:\Users\Hannes\AppData\Local\Temp\ST_0900000h.tmp saved
                        
                        . 
                        . 
                        . 
                        . 
                        . 
                        .  capture program drop once
                        
                        . 
                        . program define once 
                          1. 
                        .     capture regress scaled_DISEXP inverse_lagTA lag_scaled_Sales 
                          2. 
                        .     if c(rc) == 0 & e(N) >= 10 & e(df_m) == 2 {
                          3. 
                        .          gen nobs = e(N) 
                          4. 
                        .          predict NDISEXP, xb
                          5. 
                        .         gen R_DISEXP = scaled_DISEXP - NDISEXP
                          6. 
                        .          exit 0
                          7. 
                        .      }
                          8. 
                        .      else if !inlist(c(rc), 2000, 2001) {  //    UNEXPECTED REGRESSION ERROR
                          9. 
                        .          exit c(rc)
                         10. 
                        .      }
                         11. 
                        .     else    {    //    NO OR INSUFFICIENT OBSERVATIONS TO REGRESS; IGNORE
                         12. 
                        .          exit 0
                         13. 
                        .      } 
                         14. 
                        .  end
                        
                        . 
                        . 
                        . 
                        . runby once, by(SIC year) status
                        
                          elapsed ----------- by-groups ----------    ------- observations ------       time
                             time      count     errors    no-data        processed         saved  remaining
                        ------------------------------------------------------------------------------------
                         00:00:01        148        119          0           23,895           263   00:00:05
                         00:00:02        286        249          0           63,956           379   00:00:03
                         00:00:03        417        372          0          108,784           451   00:00:01
                         00:00:04        548        482          0          148,594           696   00:00:00
                         00:00:04        574        503          0          149,114           740   00:00:00
                        
                        --------------------------------------
                        Number of by-groups    =           574
                        by-groups with errors  =           503
                        by-groups with no data =             0
                        Observations processed =       149,114
                        Observations saved     =           740
                        --------------------------------------
                        
                        . 
                        . merge 1:1 starting_obs_no using `building', nogenerate
                        
                            Result                           # of obs.
                            -----------------------------------------
                            not matched                       148,374
                                from master                         0  
                                from using                    148,374  
                        
                            matched                               740  
                            -----------------------------------------
                        
                        . 
                        . save "`building'", replace
                        file C:\Users\Hannes\AppData\Local\Temp\ST_0900000h.tmp saved
                        
                        .  capture program drop once
                        
                        . 
                        . program define once 
                          1. 
                        .     capture regress scaled_CFO inverse_lagTA scaled_Sales scaled_deltaSales
                          2. 
                        .     if c(rc) == 0 & e(N) >= 10 & e(df_m) == 3 {
                          3. 
                        .          gen nobs = e(N) 
                          4. 
                        .          predict NCFO, xb
                          5. 
                        .         gen R_CF = scaled_CFO - NCFO
                          6. 
                        .          exit 0
                          7. 
                        .      }
                          8. 
                        .      else if !inlist(c(rc), 2000, 2001) {  //    UNEXPECTED REGRESSION ERROR
                          9. 
                        .          exit c(rc)
                         10. 
                        .      }
                         11. 
                        .     else    {    //    NO OR INSUFFICIENT OBSERVATIONS TO REGRESS; IGNORE
                         12. 
                        .          exit 0
                         13. 
                        .      } 
                         14. 
                        .  end
                        
                        . 
                        . 
                        . 
                        . runby once, by(SIC year) status
                        
                          elapsed ----------- by-groups ----------    ------- observations ------       time
                             time      count     errors    no-data        processed         saved  remaining
                        ------------------------------------------------------------------------------------
                         00:00:01        131         96          0           21,969           293   00:00:06
                         00:00:02        250        208          0           59,023           320   00:00:03
                         00:00:03        366        308          0          101,760           490   00:00:01
                         00:00:04        486        428          0          135,993           490   00:00:00
                         00:00:05        574        487          0          149,114           817   00:00:00
                        
                        --------------------------------------
                        Number of by-groups    =           574
                        by-groups with errors  =           487
                        by-groups with no data =             0
                        Observations processed =       149,114
                        Observations saved     =           817
                        --------------------------------------
                        
                        . 
                        . merge 1:1 starting_obs_no using `building', nogenerate
                        
                            Result                           # of obs.
                            -----------------------------------------
                            not matched                       148,297
                                from master                         0  
                                from using                    148,297  
                        
                            matched                               817  
                            -----------------------------------------
                        
                        . 
                        . save "`building'", replace
                        file C:\Users\Hannes\AppData\Local\Temp\ST_0900000h.tmp saved

                        Comment


                        • #13
                          Thanks. But, in your example data, you have only 0, 1, or 2 observations for each combination of SIC and year, which means that there are never enough observations to actually do the regressions you are asking for. I'm going to speculate that although the n's may not always be 0, 1, or 2, when you take account missing values on the regression variables, that lack of a "quorum" for your regressions is, in fact, quite typical of your data set. (You need at least 3 observations for NDISEXP, 5 for NPROD, and 4 for NCFO, counting only observations must have non-missing values for all of the variables mentioned in the corresponding regression commands. In addition, by your earlier request, we are not returning any results when the final estimation sample is smaller than 10.)

                          The output shows that you are getting results for some SIC year combinations, but only a handful. This suggests that the code is less likely to be the problem than the data is.

                          Try running this diagnostic code on your data:
                          Code:
                          by SIC year, sort: egen OK = total(!missing(scaled_DISEXP, inverse_lagTA, lag_scaled_Sales)) 
                          replace OK = (OK > 10)
                          egen flag = tag(SIC year)
                              
                          count if OK & flag
                          This will tell you how many SIC year combinations in your data are rich enough in data to do the regression needed to calculate NDISEXP and R_DISEXP. Based on the output you show, I'm betting the result will be 574-487 = 87. If that's what you get, then the code is running just fine and doing what you've asked it to do and the problem is entirely in your data.

                          If it's not, then I will need to see an example of your data that includes SIC year combinations that have enough non-missing data to support these calculations but nevertheless do not produce results for you. (Those would be the SIC-year combinations that have OK = 1 after the above diagnostic code.) A -dataex- example with a few of those would probably help me troubleshoot.


                          Comment


                          • #14
                            This is my result:
                            Code:
                            . by SIC year, sort: egen OK = total(!missing(scaled_DISEXP, inverse_lagTA, lag_scaled_Sales))
                            
                            .
                            . replace OK = (OK > 10)
                            (131,662 real changes made)
                            
                            .
                            . egen flag = tag(SIC year)
                            
                            .
                            .    
                            .
                            . count if OK & flag
                              395
                            I hope this helps as it should have enough non-missing data:
                            Code:
                            * Example generated by -dataex-. To install: ssc install dataex
                            clear
                            input float(year SIC lag_TA inverse_lagTA delta_Sales scaled_delta_Sales scaled_CFO scaled_COGS scaled_delta_Inv scaled_PROD scaled_DISEXP scaled_Sales lag_scaled_Sales scaled_deltaSales lag_delta_Sales lag_scaled_delta_Sales OK) byte flag
                            2012 1  45540.83 .000021958316   350.701  .007700803   .10811333 1.1662098  -.013963074  1.1522467 .15363275 1.4119183 1.4198906  .007700803  5703.447  .12523809 1 1
                            2012 1         .             .         .           .           .         .            .          .         .         .         .           .         .          . 1 0
                            2012 1 26269.045  .00003806762  -402.047 -.015304972 .0001515108 .12079046   .007740784  .12853125 .13456921   .302379  .4402527 -.015304972  1829.132   .0696307 1 0
                            2012 1 11519.912  .00008680622  1755.691    .1524049    .1608679  .5561112    .04202984    .598141 .18213376  .9719664         .    .1524049         .          . 1 0
                            2012 1  2062.263   .0004849042   113.307   .05494304   .05690157 1.1042346  .0022596535  1.1064942 .23254067 1.4778076  1.402961   .05494304      78.4  .03801649 1 0
                            2012 1  2407.273   .0004154078   714.268   .29671252  -.04343297  .8230625   .014034968   .8370975  .3595367  1.648404 1.6843607   .29671252   362.944  .15076977 1 0
                            2012 1         .             .         .           .           .         .            .          .         .         .         .           .         .          . 1 0
                            2012 1         .             .         .           .           .         .            .          .         .         .         .           .         .          . 1 0
                            2012 1         .             .         .           .           .         .            .          .         .         .         .           .         .          . 1 0
                            2012 1   994.904   .0010051221    68.243   .06859254    .1967034 1.3577833  -.008859146   1.348924  .3011155 1.8795743 1.6980897   .06859254  -221.878  -.2230145 1 0
                            2012 1   660.471    .001514071  -213.525    -.323292     .200139  2.613677 .00027404685  2.6139514 .25582048  3.045734  3.232816    -.323292   232.479   .3519897 1 0
                            2012 1    187180  5.342451e-06 16851.566   .09002867   .15965644 .29720196   -.02758425   .2696177 .08322175  .5164611  .8840067   .09002867  42643.62  .22782145 1 0
                            2012 1 12785.716  .00007821228 -2743.666  -.21458837    .1563136  2.227256   -.07743298  2.1498232 .12184746  2.438419 2.6392286  -.21458837    233.36 .018251618 1 0
                            2012 1  7090.963  .00014102457    25.183  .003551422   .02770583 .25621343   .029259074   .2854725 .09764513  .4570877  .4696194  .003551422   187.181  .02639712 1 0
                            2012 1         .             .         .           .           .         .            .          .         .         .         .           .         .          . 1 0
                            2012 1      6061   .0001649893  1723.707   .28439316    .0938266  2.550213   -.08183336  2.4683795 .23756658 2.8635056         .   .28439316         .          . 1 0
                            2012 1  4360.646   .0002293238   391.175   .08970574    .0863333 .28358045    .02373914   .3073196 .13098633 .54237103         .   .08970574         .          . 1 0
                            2012 1  3736.074  .00026766065  -113.516 -.030383766   .33064845  .0107621   -.05967387 -.04891177 .04148767 .24657676 .29596487 -.030383766 -3852.264 -1.0310994 1 0
                            2012 1 10756.604  .00009296615   2003.37    .1862456   .02835672  .4868705    .02439311  .51126367 .10940963  .7572162         .    .1862456         .          . 1 0
                            2012 1 17619.836  .00005675422  -167.295 -.009494697           . .03330905            .          . .01155516 .05799713 .06540017 -.009494697   149.995 .008512849 1 0
                            2012 1  1136.342   .0008800167   317.297   .27922666   .10953568  3.375036    .04722082  3.4222565   .597351 4.1605177  3.382522   .27922666    70.196  .06177365 1 0
                            2012 1         .             .         .           .           .         .            .          .         .         .         .           .         .          . 1 0
                            2012 1  39611.42 .000025245246  1142.509   .02884292   .12747724 1.0925163    .02418078  1.1166971    .25285 1.5216364 1.6990057   .02884292  2080.079  .05251211 1 0
                            end
                            format %ty year
                            Last edited by Hannes Meyer; 16 Jul 2018, 17:14.

                            Comment


                            • #15
                              Thank you. Yes, there was a bug in the code. Each of the three versions of -program once- attempted to generate a variable called nobs. But only the first of those can succeed. I have revised the code, giving each of the three nobs variables a distinctive name reflecting which regression it's associated with. This now runs correctly for me. My apologies for the error and the delay in getting it fixed.


                              Code:
                              * Example generated by -dataex-. To install: ssc install dataex
                              clear
                              input float(year SIC lag_TA inverse_lagTA delta_Sales scaled_delta_Sales scaled_CFO scaled_COGS scaled_delta_Inv scaled_PROD scaled_DISEXP scaled_Sales lag_scaled_Sales scaled_deltaSales lag_delta_Sales lag_scaled_delta_Sales OK) byte flag
                              2012 1  45540.83 .000021958316   350.701  .007700803   .10811333 1.1662098  -.013963074  1.1522467 .15363275 1.4119183 1.4198906  .007700803  5703.447  .12523809 1 1
                              2012 1         .             .         .           .           .         .            .          .         .         .         .           .         .          . 1 0
                              2012 1 26269.045  .00003806762  -402.047 -.015304972 .0001515108 .12079046   .007740784  .12853125 .13456921   .302379  .4402527 -.015304972  1829.132   .0696307 1 0
                              2012 1 11519.912  .00008680622  1755.691    .1524049    .1608679  .5561112    .04202984    .598141 .18213376  .9719664         .    .1524049         .          . 1 0
                              2012 1  2062.263   .0004849042   113.307   .05494304   .05690157 1.1042346  .0022596535  1.1064942 .23254067 1.4778076  1.402961   .05494304      78.4  .03801649 1 0
                              2012 1  2407.273   .0004154078   714.268   .29671252  -.04343297  .8230625   .014034968   .8370975  .3595367  1.648404 1.6843607   .29671252   362.944  .15076977 1 0
                              2012 1         .             .         .           .           .         .            .          .         .         .         .           .         .          . 1 0
                              2012 1         .             .         .           .           .         .            .          .         .         .         .           .         .          . 1 0
                              2012 1         .             .         .           .           .         .            .          .         .         .         .           .         .          . 1 0
                              2012 1   994.904   .0010051221    68.243   .06859254    .1967034 1.3577833  -.008859146   1.348924  .3011155 1.8795743 1.6980897   .06859254  -221.878  -.2230145 1 0
                              2012 1   660.471    .001514071  -213.525    -.323292     .200139  2.613677 .00027404685  2.6139514 .25582048  3.045734  3.232816    -.323292   232.479   .3519897 1 0
                              2012 1    187180  5.342451e-06 16851.566   .09002867   .15965644 .29720196   -.02758425   .2696177 .08322175  .5164611  .8840067   .09002867  42643.62  .22782145 1 0
                              2012 1 12785.716  .00007821228 -2743.666  -.21458837    .1563136  2.227256   -.07743298  2.1498232 .12184746  2.438419 2.6392286  -.21458837    233.36 .018251618 1 0
                              2012 1  7090.963  .00014102457    25.183  .003551422   .02770583 .25621343   .029259074   .2854725 .09764513  .4570877  .4696194  .003551422   187.181  .02639712 1 0
                              2012 1         .             .         .           .           .         .            .          .         .         .         .           .         .          . 1 0
                              2012 1      6061   .0001649893  1723.707   .28439316    .0938266  2.550213   -.08183336  2.4683795 .23756658 2.8635056         .   .28439316         .          . 1 0
                              2012 1  4360.646   .0002293238   391.175   .08970574    .0863333 .28358045    .02373914   .3073196 .13098633 .54237103         .   .08970574         .          . 1 0
                              2012 1  3736.074  .00026766065  -113.516 -.030383766   .33064845  .0107621   -.05967387 -.04891177 .04148767 .24657676 .29596487 -.030383766 -3852.264 -1.0310994 1 0
                              2012 1 10756.604  .00009296615   2003.37    .1862456   .02835672  .4868705    .02439311  .51126367 .10940963  .7572162         .    .1862456         .          . 1 0
                              2012 1 17619.836  .00005675422  -167.295 -.009494697           . .03330905            .          . .01155516 .05799713 .06540017 -.009494697   149.995 .008512849 1 0
                              2012 1  1136.342   .0008800167   317.297   .27922666   .10953568  3.375036    .04722082  3.4222565   .597351 4.1605177  3.382522   .27922666    70.196  .06177365 1 0
                              2012 1         .             .         .           .           .         .            .          .         .         .         .           .         .          . 1 0
                              2012 1  39611.42 .000025245246  1142.509   .02884292   .12747724 1.0925163    .02418078  1.1166971    .25285 1.5216364 1.6990057   .02884292  2080.079  .05251211 1 0
                              end
                              format %ty year
                              
                              tab SIC year
                              
                              gen long starting_obs_no = _n
                              tempfile building
                              save `building'
                               
                               capture program drop once
                              program define once 
                                  capture regress scaled_CFO inverse_lagTA scaled_Sales scaled_deltaSales
                                  if c(rc) == 0 & e(N) >= 10 & e(df_m) == 3 {
                                       gen nobs_NCFO = e(N) 
                                       predict NCFO, xb
                                      gen R_CF = scaled_CFO - NCFO
                                       exit 0
                                   }
                                   else if !inlist(c(rc), 2000, 2001) {  //    UNEXPECTED REGRESSION ERROR
                                       exit c(rc)
                                   }
                                  else    {    //    NO OR INSUFFICIENT OBSERVATIONS TO REGRESS; IGNORE
                                       exit 0
                                   } 
                               end
                              
                              runby once, by(SIC year) status
                              merge 1:1 starting_obs_no using `building', nogenerate
                              save "`building'", replace
                              
                              
                               capture program drop once
                              program define once 
                                  capture regress scaled_PROD inverse_lagTA scaled_Sales scaled_deltaSales lag_scaled_delta_Sales
                                  if c(rc) == 0 & e(N) >= 10 & e(df_m) == 4 {
                                       gen nobs_NPROD = e(N) 
                                       predict NPROD, xb
                                      gen R_PROD = scaled_PROD - NPROD
                                       exit 0
                                   }
                                   else if !inlist(c(rc), 2000, 2001) {  //    UNEXPECTED REGRESSION ERROR
                                       exit c(rc)
                                   }
                                  else    {    //    NO OR INSUFFICIENT OBSERVATIONS TO REGRESS; IGNORE
                                       exit 0
                                   } 
                               end
                              
                              runby once, by(SIC year) status
                              merge 1:1 starting_obs_no using `building', nogenerate
                              save "`building'", replace
                              
                              
                               capture program drop once
                              program define once 
                                  capture regress scaled_DISEXP inverse_lagTA lag_scaled_Sales 
                                  if c(rc) == 0 & e(N) >= 10 & e(df_m) == 2 {
                                       gen nobs_NDISEXP = e(N) 
                                       predict NDISEXP, xb
                                      gen R_DISEXP = scaled_DISEXP - NDISEXP
                                       exit 0
                                   }
                                   else if !inlist(c(rc), 2000, 2001) {  //    UNEXPECTED REGRESSION ERROR
                                       exit c(rc)
                                   }
                                  else    {    //    NO OR INSUFFICIENT OBSERVATIONS TO REGRESS; IGNORE
                                       exit 0
                                   } 
                               end
                              
                              runby once, by(SIC year) status
                              merge 1:1 starting_obs_no using `building', nogenerate
                              save "`building'", replace

                              Comment

                              Working...
                              X