Variables become insignificant when moving from OLS regression to fixed effects estimation

[Solomon Lin]

Hi,

I know that fixed effects estimation (demeaning across time) removes time-invariant observables and so variables like race and gender drop out. I also have read that when you run fixed effects estimation, variables that don't vary much over the period will have high standard errors. However, when I transition from OLS regression to fixed effects estimation, almost all of my regressors become insignificant.

1) Do you know how I can explain this?
2) Do you think this is a big issue for my regression? The joint significance test has a p-value of zero but many of the individual coefficients have p-values above 40% with some hitting the 80% - 90% mark.

Just to provide some more information, I am running a regression of Log Total Annual Hours Worked against typical personal and demographic variables (e.g. age, age squared, education, marital status, number of children etc.).

Thanks!

[Clyde Schechter]

At the risk of being a nag, I will start by reiterating (from my response to your earlier post) that log-transforming an outcome variable which takes on zero variables is virtually never appropriate, so to a certain extent, I don't think the rest of your questions might be deferred until you do a better analysis. These same issues might or might not arise there.

But there are some general principles that apply to the differences between OLS and fixed effects regression that are useful to think about in any case. The first thing to remember is that with panel data, the results of OLS regression are, in general, simply wrong. The standard errors and p-values computed by OLS are based on the assumption that the observations are independent, which is almost never the case with panel data. Not to mention omitted variable bias from not accounting for the fixed effects. So unless your fixed effects regression output ended with Stata telling you that the null hypothesis of all u_i = 0 could not be rejected, not even close to being rejected, you should just delete your OLS results from your files (or at least not look at them any more).

So comparison of the fixed effects results to OLS is, at best, the comparison of valid with invalid, or, sometimes, the comparison of two invalid models. I'll just note that in addition to the standard errors and p-values of OLS being just plain wrong, you have another phenomenon going on. Some of your variables drop out of the model because they are invariant within person, as you noted. Those variables necessarily exert their effects on outcomes only between persons, not within. But a fixed effects regression is a purely within-person analysis. In OLS you are getting a (badly proportioned) mixture of between and within person effects operating. In any case, given that between and within person effects can be very different (and the same variable can even work in opposite directions in those ways), it should never be surprising when -xtreg, fe- results are drastically different from (invalid ) OLS results.

If you feel there are reasons why you need to explore both the between-person and within-person effects (which, in general, sounds like a good idea to me, though my observation is that it is not widely done) look at -xtreg, be- rather than OLS as a supplement to -xtreg, fe-. Or, consider -xtreg, re-, where the results are a weighted average of -xtreg, fe- and -xtreg, be-. Yes, I know, you will be hounded for a Hausman test if you use random effects: so, do it.

The co-occurrence of a joint significance test with a very low p-value when all of the predictors separately have high p-values is not unusual in any way. It seems paradoxical if you misconceive "statistical significance" as meaning "is non-zero." But that isn't what statistical significance means. To say that a predictor is statistically significant is to say that given your data, and its level of (im)precision, and conditional on your model being correct, the occurrence of a coefficient this large or larger is an uncommon event--it occurs in less than 5% (assuming you used the conventional .05 level) of samples if in fact the population value of that coefficient were zero. What does the joint significance test tell us? In this case it says that given your data and its level of (im)precision, and conditional on your model being correct, the occurrence of a set of coefficients whose inverse variance weighted sum of squares and cross-products of coefficients is as large as what was observed in this analysis would be an uncommon event if all of the corresponding population coefficients were zero. As you can see there is no logical contradiction, nor even any logical tension between a significant joint test and no significant individual ones (nor the other way around).

A much less mathematical way to think of it is this. Suppose four people enter a room, the doors are locked, and there are no windows. Nobody utters a word. A gunshot rings out. The police arrive and find one person shot to death. The three survivors say nothing. We have pretty clear evidence that one of the three committed a murder, but, at least pending further investigation for finger prints, gun ownership records, etc., at this point we cannot point to any one of the three as the culprit.

[Solomon Lin]

At the risk of being a nag, I will start by reiterating (from my response to your earlier post) that log-transforming an outcome variable which takes on zero variables is virtually never appropriate, so to a certain extent, I don't think the rest of your questions might be deferred until you do a better analysis. These same issues might or might not arise there.

Hi Clyde,

Thank you for the reply - it is very much appreciated and I completely understand your points. However, I am running a project and the specifications of the project means that I have to compare the OLS result with the fixed effect result to show how the bias has been corrected.

I have a further question about what you said. When you run xtreg with fixed effects, rho is reported. I know that it is the share of the estimated variance of the overall error accounted for by the individual effect. Can I interpret a rho of 0.61 as a high rho? And say that it implies that individual effects (which are removed by fixed effects) accounted for a lot of the variation?

[Ariel Karlinsky]

In additon to Clyde's great comment, I will add the usual go-to guide regarding panel data estimation with stata:
https://www.princeton.edu/~otorres/Panel101.pdf

Also, I would add (more like reiterate what Clyde wrote) that even if you need to compare OLS to panel estimation, if you KNOW that your data is panel data than you KNOW that OLS is wrong.
Think about it this way: you draw random values from some known distribution (say standard normal). you then run some distribution-matching tests and you have some random chance that one of these will "stick". say a student t distribution for example. but you KNOW that it's wrong. you know that it's actually standard normal. so there's nothing "surprising" that your results differ. if this is some home-work project then the objective of this (if the professor knows what he's doing) is to showcase how important assumptions are and how different estimation methods yield different results, signaling the importance of "knowing" where your data comes from. In this case since the data is panel data, the observations are clearly NOT independent so OLS is indeed, plainly wrong. just like the example with you simulating random data from some known distribution.

[Solomon Lin]

In additon to Clyde's great comment, I will add the usual go-to guide regarding panel data estimation with stata:
https://www.princeton.edu/~otorres/Panel101.pdf

Also, I would add (more like reiterate what Clyde wrote) that even if you need to compare OLS to panel estimation, if you KNOW that your data is panel data than you KNOW that OLS is wrong.
Think about it this way: you draw random values from some known distribution (say standard normal). you then run some distribution-matching tests and you have some random chance that one of these will "stick". say a student t distribution for example. but you KNOW that it's wrong. you know that it's actually standard normal. so there's nothing "surprising" that your results differ. if this is some home-work project then the objective of this (if the professor knows what he's doing) is to showcase how important assumptions are and how different estimation methods yield different results, signaling the importance of "knowing" where your data comes from. In this case since the data is panel data, the observations are clearly NOT independent so OLS is indeed, plainly wrong. just like the example with you simulating random data from some known distribution.

Hi Ariel,

Thank you for the response. Would you be able to help me interpret rho (which is given when you run the regression on Stata)?

I know that it is the share of the estimated variance of the overall error accounted for by the individual effect. Can I interpret a rho of 0.61 as a high rho? And say that it implies that individual effects (which are removed by fixed effects) accounted for a lot of the variation in the error and hence the dependent variables were not able to explain as much as the variation (so their predictive power is low)?

[Clyde Schechter]

A rho of 0.61 seems pretty high. It says that 61% of the variance not explained by your predictors is attributable to time-invariant differences between people. The remaining 39% is not accounted for by anything observed in your data. It does not say anything, one way or the other, about how much of the outcome variance is attributable to your predictors. That could be a lot or a little: you have to look at R² to see that.

I just want to emphasize that there are no absolute cutoffs on rho that distinguish "high" from "low." There aren't even any generally accepted rules of thumb for this, so far as I know. It is really very context dependent. If the outcome is measured with high levels of precision, as is typically the case in physical science experiments, a rho of .61 would not be particularly high, as values in the range of 0.8 or even 0.9+ are common. By contrast, when doing opinion surveys on topics that people don't invest a lot of time thinking about or frequently change their minds about, a rho of even 0.3 might be celebrated as a great achievement compared to the expected values down in the 0.1 range. Rho also depends on the inclusion criteria for the study. If you do a random sample of a population in which there is a great deal of person-to-person variation in the outcome, and the outcome is measurable with a fair amount of precision, moderate or high values of rho will result. Using the exact same measures, if you restrict person-to-person variation by doing another study that focuses on, say, the top decile of the outcome group, (or impose some other selection criterion that entails approximately that kind of effect) rho will plummet.

So, if you have been instructed to make a judgment about rho, those are the factors you should consider. If you haven't been asked to do that, given the number of factors that can affect it and the complexity of evaluating them, I would probably just report rho without commenting on it.