Random Effects vs Fixed Effects model issues with Hausman Test (V_b Vb not positive definite)
Dear Statalister,
I have an unbalanced panel dataset for 5817 (N) firms per 8 years (T). I'm running a random effect model where my Y is time variant across id and across years, while the Xs are time variant except for one ( a dummy if the firm is foreign owned 1, 0 otherwise).
My main indepent predictor is Market potential measure at the district level, weighted both for the population and for the population with a HE degree.
However before choosing the RE model, I did the classical Hausman test testing a FE vs RE effect model, however I got the error posted below: as I know, in principle you choose a RE when you are interested in the effect of a time invariant variable which you don't want to be absorbed by a fixed effect, still for completion I wanted to test the validity of this. The theory to which I refer to mostly use OLS or FE model using industry and time dummies, therefore I don't get the huge difference between a RE model with all these fixed effects and a FE model. To be completerly clear I tested with Hausman, - xtoverid and testparm i.year to check whether variation across years made sense and yes they do (coefficients = 0 and p value = 0.000). I don't post it to avoid unnecessary long post.
I also read all the threads about the this but still I cant' get my head around it.
All the models tested had robust errors clustered at the firm and district level but in order to test HAUSMAN I tooke them away and used xtoverid instead, which is not the real issue at hand right now.
Of course the dummy forown and other time and industry dummies were removed from the fixed effect, but it is something I have to control for standing by the literature.
I add that before I tried these:
And the test worked just fine, but it is not the kind of model tested in the literature and it might result incomplete and biased.
I also tested my XTREG , RE vs POOLED OLS using
and according to the test POOLED OLS is not a choice as all year dummies were = 0 and p values was 0.000
Moreover I also used xtoverid after the xtreg , re (it is not possible after xtreg, fe but I don't know why).
Can someone help me?
Thanks
Dalila
I have an unbalanced panel dataset for 5817 (N) firms per 8 years (T). I'm running a random effect model where my Y is time variant across id and across years, while the Xs are time variant except for one ( a dummy if the firm is foreign owned 1, 0 otherwise).
My main indepent predictor is Market potential measure at the district level, weighted both for the population and for the population with a HE degree.
However before choosing the RE model, I did the classical Hausman test testing a FE vs RE effect model, however I got the error posted below: as I know, in principle you choose a RE when you are interested in the effect of a time invariant variable which you don't want to be absorbed by a fixed effect, still for completion I wanted to test the validity of this. The theory to which I refer to mostly use OLS or FE model using industry and time dummies, therefore I don't get the huge difference between a RE model with all these fixed effects and a FE model. To be completerly clear I tested with Hausman, - xtoverid and testparm i.year to check whether variation across years made sense and yes they do (coefficients = 0 and p value = 0.000). I don't post it to avoid unnecessary long post.
I also read all the threads about the this but still I cant' get my head around it.
All the models tested had robust errors clustered at the firm and district level but in order to test HAUSMAN I tooke them away and used xtoverid instead, which is not the real issue at hand right now.
Code:
* Example generated by -dataex-. To install: ssc install dataex clear input long id int year str81 oa11cd double tuemp float(lmpgenOA lmpheOA capempl forown) double cost_sales 1225 2012 "E00168192" 2273233.39658444 9.132033 8.421536 .55452245 0 -3303.433 403 2018 "E00000007" 1990078.66108787 9.065825 9.269802 5.070008 0 -1957.633 1225 2013 "E00168192" 1434669.76744186 9.132033 8.421536 .6686352 0 -1907.644 403 2017 "E00000007" 1762047.00162075 9.065825 9.271615 4.5478215 0 -1796.352 403 2016 "E00000007" 1618102.15482841 9.065825 9.274349 3.947793 0 -1621.86 466 2017 "E00174692" 445506.745817593 7.933118 8.937097 .17657636 0 -1594.043 466 2019 "E00174692" 462221.467006514 7.933118 8.926232 .2028176 0 -1579.259 466 2018 "E00174692" 458740.962140621 7.933118 8.933418 .20365784 0 -1547.68 331 2012 "E00004701" 559875.518672199 8.006558 8.54924 .1146307 0 -1534 146 2018 "E00166755" 1961920 11.81126 11.8221 .3007664 1 -1486.374 end label values id firmid label def firmid 146 "GB00594001", modify label def firmid 331 "GB00975677", modify label def firmid 403 "GB01090741", modify label def firmid 466 "GB01207120", modify label def firmid 1225 "GB02433585", modify
Code:
xtreg ltuemp lmpgenOA lmpheOA capempl forown cost_sales size1 i.industry1 i.year i.lad, fe
note: forown omitted because of collinearity
note: 2.industry1 omitted because of collinearity
note: 3.industry1 omitted because of collinearity
note: 4.industry1 omitted because of collinearity
note: 5.industry1 omitted because of collinearity
note: 6.industry1 omitted because of collinearity
note: 7.industry1 omitted because of collinearity
note: 2.lad omitted because of collinearity
note: 3.lad omitted because of collinearity
note: 4.lad omitted because of collinearity
note: 8.lad omitted because of collinearity
note: 10.lad omitted because of collinearity
note: 11.lad omitted because of collinearity
note: 13.lad omitted because of collinearity
note: 14.lad omitted because of collinearity
note: 16.lad omitted because of collinearity
note: 17.lad omitted because of collinearity
note: 18.lad omitted because of collinearity
note: 20.lad omitted because of collinearity
note: 21.lad omitted because of collinearity
note: 22.lad omitted because of collinearity
note: 23.lad omitted because of collinearity
note: 24.lad omitted because of collinearity
note: 25.lad omitted because of collinearity
note: 26.lad omitted because of collinearity
note: 27.lad omitted because of collinearity
note: 29.lad omitted because of collinearity
note: 30.lad omitted because of collinearity
note: 31.lad omitted because of collinearity
note: 32.lad omitted because of collinearity
note: 33.lad omitted because of collinearity
Fixed-effects (within) regression Number of obs = 25,149
Group variable: id Number of groups = 4,091
R-sq: Obs per group:
within = 0.0340 min = 1
between = 0.0559 avg = 6.1
overall = 0.0545 max = 8
F(20,21038) = 37.05
corr(u_i, Xb) = -0.0296 Prob > F = 0.0000
-----------------------------------------------------------------------------------------
ltuemp | Coef. Std. Err. t P>|t| [95% Conf. Interval]
------------------------+----------------------------------------------------------------
lmpgenOA | -.0938133 .15202 -0.62 0.537 -.3917841 .2041576
lmpheOA | -.0652593 .0681192 -0.96 0.338 -.1987781 .0682596
capempl | .0089819 .0006072 14.79 0.000 .0077918 .0101721
forown | 0 (omitted)
cost_sales | -.0021398 .0001144 -18.71 0.000 -.002364 -.0019157
size1 | .0699393 .0087211 8.02 0.000 .0528453 .0870333
|
industry1 |
Construction | 0 (omitted)
Logistics | 0 (omitted)
Manufacturing | 0 (omitted)
Others | 0 (omitted)
R&D | 0 (omitted)
wholesale and retail | 0 (omitted)
|
year |
2013 | .032932 .0132008 2.49 0.013 .0070574 .0588067
2014 | .0369939 .0130772 2.83 0.005 .0113615 .0626263
2015 | .060668 .013114 4.63 0.000 .0349634 .0863725
2016 | .0691917 .0131206 5.27 0.000 .0434743 .0949092
2017 | .0886862 .0132313 6.70 0.000 .0627519 .1146206
2018 | .0995565 .0133961 7.43 0.000 .073299 .1258139
2019 | .1163816 .0182189 6.39 0.000 .0806711 .1520921
|
lad |
Barnet | 0 (omitted)
Bexley | 0 (omitted)
Brent | 0 (omitted)
Bromley | -.2900522 .5408589 -0.54 0.592 -1.350177 .7700727
Camden | -.1497671 .704691 -0.21 0.832 -1.531016 1.231481
City of London | .7345571 .8806155 0.83 0.404 -.9915169 2.460631
Croydon | 0 (omitted)
Ealing | -.392667 .7541058 -0.52 0.603 -1.870772 1.085438
Enfield | 0 (omitted)
Greenwich | 0 (omitted)
Hackney | -.8681928 .9271746 -0.94 0.349 -2.685526 .9491405
Hammersmith and Fulham | 0 (omitted)
Haringey | 0 (omitted)
Harrow | -1.099072 .7116874 -1.54 0.123 -2.494034 .2958902
Havering | 0 (omitted)
Hillingdon | 0 (omitted)
Hounslow | 0 (omitted)
Islington | -.0945124 .7519595 -0.13 0.900 -1.568411 1.379386
Kensington and Chelsea | 0 (omitted)
Kingston upon Thames | 0 (omitted)
Lambeth | 0 (omitted)
Lewisham | 0 (omitted)
Merton | 0 (omitted)
Newham | 0 (omitted)
Redbridge | 0 (omitted)
Richmond upon Thames | 0 (omitted)
Southwark | -.4545496 .8367965 -0.54 0.587 -2.094735 1.185636
Sutton | 0 (omitted)
Tower Hamlets | 0 (omitted)
Waltham Forest | 0 (omitted)
Wandsworth | 0 (omitted)
Westminster | 0 (omitted)
|
_cons | 13.19484 1.01411 13.01 0.000 11.2071 15.18257
------------------------+----------------------------------------------------------------
sigma_u | 1.3506759
sigma_e | .4966646
rho | .88089059 (fraction of variance due to u_i)
-----------------------------------------------------------------------------------------
F test that all u_i=0: F(4090, 21038) = 35.64 Prob > F = 0.0000
est store fe
xtreg ltuemp lmpgenOA lmpheOA capempl forown cost_sales size1 i.industry1 i.year i.lad, re
Random-effects GLS regression Number of obs = 25,149
Group variable: id Number of groups = 4,091
R-sq: Obs per group:
within = 0.0325 min = 1
between = 0.2410 avg = 6.1
overall = 0.2355 max = 8
Wald chi2(51) = 1995.62
corr(u_i, X) = 0 (assumed) Prob > chi2 = 0.0000
-----------------------------------------------------------------------------------------
ltuemp | Coef. Std. Err. z P>|z| [95% Conf. Interval]
------------------------+----------------------------------------------------------------
lmpgenOA | -.0859057 .0251848 -3.41 0.001 -.135267 -.0365444
lmpheOA | .0815867 .0225095 3.62 0.000 .037469 .1257044
capempl | .0106728 .0005328 20.03 0.000 .0096285 .0117171
forown | .231343 .0385795 6.00 0.000 .1557287 .3069573
cost_sales | -.0023917 .000106 -22.57 0.000 -.0025995 -.002184
size1 | .1316934 .0081154 16.23 0.000 .1157876 .1475993
|
industry1 |
Construction | .5043068 .0769176 6.56 0.000 .3535512 .6550625
Logistics | .1817653 .1153128 1.58 0.115 -.0442437 .4077743
Manufacturing | .2062247 .0911777 2.26 0.024 .0275197 .3849297
Others | -.4832861 .0474155 -10.19 0.000 -.5762187 -.3903534
R&D | -.4489571 .2482047 -1.81 0.070 -.9354295 .0375153
wholesale and retail | .6985405 .0564901 12.37 0.000 .5878218 .8092591
|
year |
2013 | .0315211 .0132515 2.38 0.017 .0055487 .0574936
2014 | .0334569 .0129714 2.58 0.010 .0080333 .0588804
2015 | .0557066 .0128702 4.33 0.000 .0304814 .0809318
2016 | .0656223 .0128763 5.10 0.000 .0403852 .0908593
2017 | .0876851 .013092 6.70 0.000 .0620253 .113345
2018 | .0985067 .0133509 7.38 0.000 .0723394 .1246741
2019 | .1274615 .0180757 7.05 0.000 .0920338 .1628892
|
lad |
Barnet | -.4041992 .2943551 -1.37 0.170 -.9811245 .1727261
Bexley | -.4295678 .3678848 -1.17 0.243 -1.150609 .2914731
Brent | -.0348001 .3083596 -0.11 0.910 -.6391739 .5695737
Bromley | -.1309259 .3161117 -0.41 0.679 -.7504934 .4886416
Camden | -.1044641 .2709851 -0.39 0.700 -.6355851 .4266569
City of London | .096831 .2767392 0.35 0.726 -.4455679 .6392299
Croydon | -.0945046 .3074663 -0.31 0.759 -.6971275 .5081183
Ealing | .042697 .2994096 0.14 0.887 -.544135 .629529
Enfield | -.2932872 .3209769 -0.91 0.361 -.9223903 .3358159
Greenwich | -.5613284 .3651217 -1.54 0.124 -1.276954 .1542971
Hackney | -.171142 .3144232 -0.54 0.586 -.7874002 .4451162
Hammersmith and Fulham | -.291473 .2908778 -1.00 0.316 -.861583 .278637
Haringey | -.1314029 .3726172 -0.35 0.724 -.8617193 .5989135
Harrow | .1846775 .3012802 0.61 0.540 -.4058207 .7751758
Havering | .0261472 .3316642 0.08 0.937 -.6239026 .6761971
Hillingdon | -.1525145 .2830821 -0.54 0.590 -.7073453 .4023162
Hounslow | .1601133 .2899418 0.55 0.581 -.4081621 .7283887
Islington | -.1997624 .274643 -0.73 0.467 -.7380527 .3385279
Kensington and Chelsea | -.1181314 .2888664 -0.41 0.683 -.6842991 .4480363
Kingston upon Thames | -.1546803 .3191289 -0.48 0.628 -.7801614 .4708009
Lambeth | -.1992022 .3160978 -0.63 0.529 -.8187426 .4203382
Lewisham | -1.018139 .3848164 -2.65 0.008 -1.772365 -.2639127
Merton | -.0995425 .3314845 -0.30 0.764 -.7492402 .5501551
Newham | -.0542702 .3401734 -0.16 0.873 -.7209979 .6124575
Redbridge | .39405 .3720598 1.06 0.290 -.3351737 1.123274
Richmond upon Thames | -.0310623 .3072969 -0.10 0.919 -.6333532 .5712287
Southwark | -.1837637 .2786828 -0.66 0.510 -.729972 .3624446
Sutton | -.0537442 .3736738 -0.14 0.886 -.7861314 .6786431
Tower Hamlets | -.0557447 .2797957 -0.20 0.842 -.6041342 .4926447
Waltham Forest | .5709957 .3773261 1.51 0.130 -.1685499 1.310541
Wandsworth | -.6133194 .3097433 -1.98 0.048 -1.220405 -.0062336
Westminster | .006925 .2687566 0.03 0.979 -.5198283 .5336783
|
_cons | 11.68975 .2939594 39.77 0.000 11.1136 12.2659
------------------------+----------------------------------------------------------------
sigma_u | 1.1483457
sigma_e | .4966646
rho | .84241734 (fraction of variance due to u_i)
-----------------------------------------------------------------------------------------
. est store re
. hausman fe re
Note: the rank of the differenced variance matrix (19) does not equal the number of coefficients being tested (20); be sure this is what you expect, or
there may be problems computing the test. Examine the output of your estimators for anything unexpected and possibly consider scaling your
variables so that the coefficients are on a similar scale.
---- Coefficients ----
| (b) (B) (b-B) sqrt(diag(V_b-V_B))
| fe re Difference S.E.
-------------+----------------------------------------------------------------
lmpgenOA | -.0938133 -.0859057 -.0079076 .1499193
lmpheOA | -.0652593 .0815867 -.146846 .0642927
capempl | .0089819 .0106728 -.0016909 .0002912
cost_sales | -.0021398 -.0023917 .0002519 .000043
size1 | .0699393 .1316934 -.0617541 .0031934
year |
2013 | .032932 .0315211 .0014109 .
2014 | .0369939 .0334569 .003537 .00166
2015 | .060668 .0557066 .0049614 .0025171
2016 | .0691917 .0656223 .0035695 .0025204
2017 | .0886862 .0876851 .0010011 .001915
2018 | .0995565 .0985067 .0010498 .0010994
2019 | .1163816 .1274615 -.01108 .0022801
lad |
5 | -.2900522 -.1309259 -.1591264 .4388642
6 | -.1497671 -.1044641 -.045303 .6505048
7 | .7345571 .096831 .6377261 .8360018
9 | -.392667 .042697 -.435364 .6921196
12 | -.8681928 -.171142 -.6970508 .8722332
15 | -1.099072 .1846775 -1.283749 .6447707
19 | -.0945124 -.1997624 .1052499 .7000103
28 | -.4545496 -.1837637 -.2707859 .7890274
------------------------------------------------------------------------------
b = consistent under Ho and Ha; obtained from xtreg
B = inconsistent under Ha, efficient under Ho; obtained from xtreg
Test: Ho: difference in coefficients not systematic
chi2(19) = (b-B)'[(V_b-V_B)^(-1)](b-B)
= 410.47
Prob>chi2 = 0.0000
(V_b-V_B is not positive definite)
Of course the dummy forown and other time and industry dummies were removed from the fixed effect, but it is something I have to control for standing by the literature.
I add that before I tried these:
Code:
xtreg ltuemp lmpgenOA lmpheOA capempl forown cost_sales, fe est store fe xtreg ltuemp lmpgenOA lmpheOA capempl forown cost_sales, re est store re hausman fe re
I also tested my XTREG , RE vs POOLED OLS using
Code:
- testparm i.year
Moreover I also used xtoverid after the xtreg , re (it is not possible after xtreg, fe but I don't know why).
Can someone help me?
Thanks
Dalila
Comment