Anyone knows what is an "adjusted within-R2"?

[Sergio Correia]

Dear all,
I'm trying to understand what is the adjusted R2 in the context of a fixed effects regression (xtreg, xtivreg2) and would like some feedback/comments. My problem is that the formula I thought makes more sense is not used anywhere in Stata, so I'm not sure of what's going on...

Background: explanation of the adjusted R2 from Quora

Common Formulas:
rss: Sum-of-squares of the residuals
tss: Sum-of-squares of the dependent variable (after demeaning)
wss: Within-sum-of-squares; similar to TSS but instead of demeaning wrt to the entire sample, demean the dependent variable within each group.

N: Number of obs.
L: Number of fixed effects (including the constant)
K: Number of other regressors besides the fixed effects (and the constant)

R2 = 1 - rss / tss
R2 Adjusted = 1 - [ rss / (N-L-K) ] / [ tss / (N-1) ]
R2 Within = 1 - rss / wss

Controversial Formula:
R2 Within Adjusted = 1 - [ rss / (N-L-K) ] / [ wss / (N-L) ]

The above is how I would write it based on the Quora link. Within each term in brackets, the numerator is is divided by the degrees of freedom available after computing that number.

This formula has two nice properties:

1. With no regressors besides the fixed effects (K=0), it collapses to the within-R2 (which is also zero).
2. If there is only one fixed effect (L=1), it collapses to the adjusted-R2.

Problem: It's not what other packages do, not sure if I'm missing something...

1) xtreg computes it as
1 - [ rss / (N-L-K) ] / [ wss / (N-1) ]
(The only difference is -1 instead of -L)

2) (areg doesn't compute within estimates)

3) xtivreg2 (ssc) computes it as
1 - [ rss / (N-L-K) ] / [ wss / (N-0) ]

Note how the only difference is whether to substract the number of fixed effects L, substract 1, or substract 0.


Any opinions? Is that a bug in the other programs? Should I follow what they do?

Best,
Sergio





Previous discussions
[1] http://www.stata.com/statalist/archi.../msg00615.html
[2] http://www.stata.com/statalist/archi.../msg00201.html

Alternative Formulas (less relevant)
[1] http://stats.stackexchange.com/quest...-r/63766#63766
[2] http://stats.stackexchange.com/quest...re/55932#55932
[3] http://mlrv.ua.edu/2007/Vol33_1/Leach-Henson-p1-11.pdf

[Andrew Musau]

Sergio: Stata computes

R2 = SSreg/ SStotal

where

SStotal = total sum of squares
SSreg = regression sum of squares (or explained sum of squares)
____________________________________


Adjusted R2 = R2 - [(1- R2)*(p/n-p-1)]

where p is the number of regressors (excluding constant), and n is the number of observations. After running the regressions, you can always compute the R2 and adjusted R2 manually using the following commands

scalar R2 = e(mss)/ (e(mss)+ e(rss))
di R2
scalar aR2 = R2 - ((1- R2)* (e(df_m) / (e(N)-e(df_m)-1)))
di aR2

Stata will simply implement the above formula using the output from a fixed effects regression (as it would with OLS). The following example illustrates

clear 
input y x1 x2 x3 x4 id year
78 19  45 44  15   1 1
23 19  17 47  72   1 2
10 19  32 62  65   1 3
34 19  11 21  20   1 4
77 19  42 23  100  1 5
91 55  12 13  14   2 1
62 55  27 37  47   2 2
33 55  13 14  15   2 3
16 55  58 68  78   2 4
99 55  80 90  70   2 5
20 51  18 62  82   3 1
38 39  39 11  63   3 2
40 87  46 93  90   3 3
56 03  64 80  28   3 4
73 200 88 103  36  3 5
115 70  85 18  85  4 1
49 51  67 22 76    4 2
57 28  49 26  96   4 3
74 32  31 41  77   4 4
110 16  12 60  80  4 5
24 112  26 20  26  5 1
111 123 81 82  37  5 2
64 45  59 39  49   5 3
39 72  31 29  92   5 4
79 80  16 77  107  5 5
37 47  19 89  12   6 1
23 61  38 45  22   6 2
32 83  82 83  66   6 3
120 115 91 116 108 6 4
7 150  54 93  72   6 5
92 28  30 41  90   7 1
100 28 40 96 102   7 2
108 28  50 29  59  7 3
116 28  60 42  76  7 4
128 28  70 80  94  7 5
39 7  55 103  106  8 1
51 50  27 98  62   8 2
73 61  19 81  74   8 3
94 86  112 99  53  8 4
103 99  67 102  10 8 5
89 80  105 54  69  9 1
62 90  97 108  62  9 2
13 100  102 92  39 9 3
100 110 81 66  85  9 4
115 120 92 50  67  9 5
40 37  75 19  14  10 1
92 65  87 5  34   10 2
56 72  92 15  40  10 3
119 128  23 21 63 10 4
84 80  82 67  29  10 5
end

Running the fixed effects model

. xtset id year
       panel variable:  id (strongly balanced)
        time variable:  year, 1 to 5
                delta:  1 unit

. xtreg y x*, fe

Fixed-effects (within) regression               Number of obs      =        50
Group variable: id                              Number of groups   =        10

R-sq:  within  = 0.1448                         Obs per group: min =         5
       between = 0.0000                                        avg =       5.0
       overall = 0.0538                                        max =         5

                                                F(4,36)            =      1.52
corr(u_i, Xb)  = -0.3074                        Prob > F           =    0.2161

------------------------------------------------------------------------------
           y |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
          x1 |   .2398038   .1525894     1.57   0.125    -.0696618    .5492694
          x2 |   .2376799   .2046493     1.16   0.253    -.1773682     .652728
          x3 |   .0511757   .2110712     0.24   0.810    -.3768965     .479248
          x4 |   .0676377   .1813813     0.37   0.711    -.3002206     .435496
       _cons |   32.27467   17.37543     1.86   0.071    -2.964345    67.51368
-------------+----------------------------------------------------------------
     sigma_u |  22.933562
     sigma_e |   31.13536
         rho |  .35172042   (fraction of variance due to u_i)
------------------------------------------------------------------------------
F test that all u_i=0:     F(9, 36) =     1.94               Prob > F = 0.0774

. scalar R2 = e(mss)/ (e(mss)+ e(rss))
 
. scalar aR2 = R2 - ((1- R2)* (e(df_m) / (e(N)-e(df_m)-1)))

. di R2
.14475503

. di aR2
-.16408343

The adjusted R2 can be seen on the ereturn list below. Note that, in this case p= 13 (df_m) because you have 4 regressors + (10-1) individual dummies.

. ereturn list

scalars:
               e(rank) =  5
                e(rss) =  34898.78424281887
               e(df_m) =  13
               e(df_r) =  36
                 e(r2) =  .1447550276722099
               e(rmse) =  31.13536049914366
                e(mss) =  5906.815757181132
               e(r2_a) =  -.1640834345572697
                 e(ll) =  -234.6515333278909
               e(ll_0) =  -238.5607166714206
                e(tss) =  59530.49999999999
                  e(N) =  50
               e(df_b) =  4
               e(r2_w) =  .1447550276722099
               e(df_a) =  9
                  e(F) =  1.523300681328726
                e(F_f) =  1.938003049221467
               e(Tbar) =  5
               e(Tcon) =  1
              e(g_min) =  5
                e(rho) =  .3517204164928344
              e(sigma) =  38.66987126953578
            e(sigma_e) =  31.13536049914366
               e(r2_b) =  .0000103737332557
               e(r2_o) =  .0537633657871388
               e(corr) =  -.3074031244891103
            e(sigma_u) =  22.93356209991883
                 e(ui) =  22.93356209991883
                e(N_g) =  10
              e(g_max) =  5
              e(g_avg) =  5

macros:
            e(cmdline) : "xtreg y x*, fe"
                e(cmd) : "xtreg"
       e(marginsnotok) : "E U UE SCore STDP XBU"
            e(predict) : "xtrefe_p"
              e(model) : "fe"
             e(depvar) : "y"
               e(ivar) : "id"
                e(vce) : "conventional"
         e(properties) : "b V"

matrices:
                  e(b) :  1 x 5
                  e(V) :  5 x 5

functions:
             e(sample)   

I am not familiar with the algorithm used by other software, but Stata's algorithm is consistent (FE and OLS).

[Sergio Correia]

Hi Andrew,

Thanks for the detailed reply!
I agree that in general both the R2 and adjusted R2 are consistent, but why xtreg uses that formula makes no sense for me:

1. It is negative if you run it without any regressors.
2. The entire point of the adjusted R2 is to fix the bias of the standard R2, which xtreg's formula doesn't do at all. You cannot use the same formula if to obtain the within total-sum-of-squares you used e(df_a) instead of just 1 degrees of freedom. Note that fixed effects e(df_a) can be HUGE so the effect is not trivial at all.

[Andrew Musau]

I have taken a look at EViews. It reports the R2 from OLS with N-1 individual dummies after running a FE regression. Experience indicates that researchers use this R2 estimate (after all, it makes more sense).

. reg y x* i.id

      Source |       SS       df       MS              Number of obs =      50
-------------+------------------------------           F( 13,    36) =    1.95
       Model |  24631.7158    13  1894.74737           Prob > F      =  0.0562
    Residual |  34898.7842    36  969.410673           R-squared     =  0.4138
-------------+------------------------------           Adj R-squared =  0.2021
       Total |     59530.5    49  1214.90816           Root MSE      =  31.135

------------------------------------------------------------------------------
           y |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
          x1 |   .2398038   .1525894     1.57   0.125    -.0696618    .5492694
          x2 |   .2376799   .2046493     1.16   0.253    -.1773682     .652728
          x3 |   .0511757   .2110712     0.24   0.810    -.3768965     .479248
          x4 |   .0676377   .1813813     0.37   0.711    -.3002206     .435496
             |
          id |
          2  |   5.516458     20.443     0.27   0.789    -35.94386    46.97678
          3  |  -19.72369   21.75645    -0.91   0.371    -63.84782    24.40044
          4  |   25.48315   21.16516     1.20   0.236    -17.44178    68.40809
          5  |  -1.339484   22.04283    -0.06   0.952    -46.04442    43.36545
          6  |  -26.87833   23.29873    -1.15   0.256    -74.13035    20.37368
          7  |   54.39856   20.76321     2.62   0.013     12.28882     96.5083
          8  |   7.928215   22.87142     0.35   0.731    -38.45717    54.31359
          9  |  -6.158041    24.9925    -0.25   0.807    -56.84518     44.5291
         10  |   11.91863   23.21828     0.51   0.611    -35.17024    59.00749
             |
       _cons |   27.16012   18.15717     1.50   0.143    -9.664322    63.98457
------------------------------------------------------------------------------

[Sergio Correia]

That would be the same R2 reported by areg, and makes sense if you don't have many fixed effects. But in a context of many fixed effects, most of the R2 will be explained by the FEs, and that's where the within R2 makes sense.

I guess the choice between standard-R2 and within-R2 depends on your objectives and how you think of your fixed effects. But in any case, the adjusted R2 reported by xtreg,fe is simply misleading in my opinion, as it has a *huge* downwards bias.

[Sergio Correia]

As an update, I did a few Montecarlos that showed that the xtreg_fe formula is quite unbiased, in particular under two conditions:
1) if T is small (e.g. each individual has on avg. only 3-5 observations)
2) if N is not so large (in which case it doesn't matter whether you compute R2 or adjusted R2)

I know changing the reported estimates would break backwards compatibility, but if the adjusted R2 is clearly wrong, why not fix it?

Sergio

[Andrew Musau]

I agree with your take on the adjusted within R2. Usually, you do not see people reporting it (in most cases, it has no meaning). To see this, you can look at the formula

Adjusted R2 = R2 - [(1- R2)*(p/n-p-1)]

The problem arises because n (the number of observations) is a function of p (the model degrees of freedom). In simple terms, if you have more individuals in the panel, you have more data. The usual case is that you have a few regressors (say 7), and a large cross-sectional dimension (100+). Since fixed effects regression regards the individual effects as parameters to be estimated, you will have a problem with the within adjusted R2. (p/n-p-1) in the formula above is usually greater than 1 and [(1- R2)*(p/n-p-1)] exceeds the within R2 so that you get a negative adjusted R2.

Stata usually does not display the value of the within adjusted R2 in the main result window - it is just a redundant statistic in the ereturn list. There are other ways of assessing what value a single regressor adds to the model prior to running it.