xtreg vs xtpoisson difference in observations

[Neha Bhardwaj Up]

Hello all,

I have a dataset of 4789 observations (unbalanced 194 groups and 25 years) . The way this dataset is constructed, there are 133 groups (3325 obs) with all zero outcomes out of the 194 groups.

My predicament is that when using

xtreg y x, fe

all the 4789 observations are retained. However, on using

xtpoisson y x, fe

the 133 groups (3325 obs) with all zero outcomes are dropped and only 1464 observations (61 groups) are used to generate the estimates.

Please can you guide me on why this is happening. I am using STATA 15.

Thank you in advance!

[Joro Kolev]

Instead of having "predicaments" is better to just try:

 
 xtreg y x, fe

and see whether it gives the same result.

I think it will drop all your groups for which the dependent variable is constant, because the within transformation wipes out all variables that are constant within panel.

[Neha Bhardwaj Up]

Dear Joro, Thats exactly what I wrote in my post that on trying with

xtreg y x, fe

the observations are 4789 (none dropped) while on using

xtpoisson

the observations that are constant on the panel are wiped. Results will obviously not be the same because the two estimation techniques are different. My 'predicament' is as to why

xtreg y x, fe

is not wiping out the variables that are constant within panel.

[Joro Kolev]

Something is wrong in how you understand your data.

Here is an example where I regress a time constant variable on a time varying variable. The -xtreg, fe- drops everything as it should:

. clear

.  webuse nlswork
(National Longitudinal Survey.  Young Women 14-26 years of age in 1968)

.   xtset idcode
       panel variable:  idcode (unbalanced)

. xtsum race hours

Variable         |      Mean   Std. Dev.       Min        Max |    Observations
-----------------+--------------------------------------------+----------------
race     overall |  1.303392   .4822773          1          3 |     N =   28534
         between |             .4862111          1          3 |     n =    4711
         within  |                    0   1.303392   1.303392 | T-bar = 6.05689
                 |                                            |
hours    overall |  36.55956   9.869623          1        168 |     N =   28467
         between |             7.846585          1       83.5 |     n =    4710
         within  |             7.520712  -2.154726   130.0596 | T-bar = 6.04395

. xtreg race hours, fe

Fixed-effects (within) regression               Number of obs     =     28,467
Group variable: idcode                          Number of groups  =      4,710

R-sq:                                           Obs per group:
     within  =      .                                         min =          1
     between = 0.0001                                         avg =        6.0
     overall =      .                                         max =         15

                                                F(1,23756)        =          .
corr(u_i, Xb)  =      .                         Prob > F          =          .

------------------------------------------------------------------------------
        race |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
       hours |          0  (omitted)
       _cons |   1.303299          .        .       .            .           .
-------------+----------------------------------------------------------------
     sigma_u |  .48563517
     sigma_e |          0
         rho |          1   (fraction of variance due to u_i)
------------------------------------------------------------------------------
F test that all u_i=0: F(4709, 23756) = .                    Prob > F =      .

.

[Neha Bhardwaj Up]

Joro, Here is an sample dataset by which I'll explain my problem again.

* Example generated by -dataex-. To install: ssc install dataex
clear
input str1 country int year byte(y x1) float Country_Group
"A" 1995 0 0 1
"A" 1996 0 0 1
"A" 1997 0 0 1
"B" 1995 0 0 2
"B" 1996 1 0 2
"B" 1997 0 0 2
"C" 1995 0 0 3
"C" 1996 1 0 3
"C" 1997 1 0 3
"D" 1995 0 0 4
"D" 1996 1 1 4
"D" 1997 1 1 4
"E" 1995 1 0 5
"E" 1996 1 0 5
"E" 1997 1 0 5
end

As we can see in country A, there is no variation in y and x1. However, in rest of the countries B,C,D,E, there is some variation in y and x1.

1. XTREG with FE

. xtreg y x1, fe

Fixed-effects (within) regression               Number of obs     =         15
Group variable: Country_Gr~p                    Number of groups  =          5

R-sq:                                           Obs per group:
     within  = 0.3333                                         min =          3
     between = 0.0385                                         avg =        3.0
     overall = 0.1346                                         max =          3

                                                F(1,9)            =       4.50
corr(u_i, Xb)  = -0.4036                        Prob > F          =     0.0629

------------------------------------------------------------------------------
           y |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
          x1 |          1   .4714045     2.12   0.063    -.0663911    2.066391
       _cons |         .4   .1175889     3.40   0.008     .1339953    .6660047
-------------+----------------------------------------------------------------
     sigma_u |  .43461349
     sigma_e |  .38490018
         rho |  .56043956   (fraction of variance due to u_i)
------------------------------------------------------------------------------
F test that all u_i=0: F(4, 9) = 3.20                        Prob > F = 0.0679

2. XTPOISSON with FE

. xtpoisson y x1, fe
note: 1 group (3 obs) dropped because of all zero outcomes

Iteration 0:   log likelihood = -5.6108445  
Iteration 1:   log likelihood = -4.7999143  
Iteration 2:   log likelihood = -4.7999143  

Conditional fixed-effects Poisson regression    Number of obs     =         12
Group variable: Country_Group                   Number of groups  =          4

                                                Obs per group:
                                                              min =          3
                                                              avg =        3.0
                                                              max =          3

                                                Wald chi2(0)      =          .
Log likelihood  = -4.7999143                    Prob > chi2       =          .

------------------------------------------------------------------------------
           y |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
          x1 |    49.3125          .        .       .            .           .
------------------------------------------------------------------------------

Thus, as can be seen, xtpoisson drops 3 observations (pertaining to country A because they have all zero outcomes) while this does not happen in case of XTREG. Any clarification will be helpful.

[Joro Kolev]

I do not think that there is an intuitive explanation here. The likelihood of the non-linear model requires something which the linear model does not. To know what the nonlinear model requires we need to look at the likelihood function. Or if we do not want to dig into the mechanics of the estimation, to just trust that Stata does what makes sense.

As you can see from the output below, if we try -xtlogit-, which is appropriate for this y 0/1 variable, the -xtlogit- drops not only the group for which all ys are 0, but also the group for which all ys are 1. So we learn that the likelihood of the logit requires even more than the likelihood of the poisson from the data.

. xtpoisson y x1, fe nolog
note: 1 group (3 obs) dropped because of all zero outcomes

Conditional fixed-effects Poisson regression    Number of obs     =         12
Group variable: Country_Group                   Number of groups  =          4

                                                Obs per group:
                                                              min =          3
                                                              avg =        3.0
                                                              max =          3

                                                Wald chi2(0)      =          .
Log likelihood  = -4.7999143                    Prob > chi2       =          .

------------------------------------------------------------------------------
           y |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
          x1 |    49.3125          .        .       .            .           .
------------------------------------------------------------------------------

. xtlogit y x1, fe nolog
note: multiple positive outcomes within groups encountered.
note: 2 groups (6 obs) dropped because of all positive or
      all negative outcomes.
convergence not achieved

Conditional fixed-effects logistic regression   Number of obs     =          9
Group variable: Country_Group                   Number of groups  =          3

                                                Obs per group:
                                                              min =          3
                                                              avg =        3.0
                                                              max =          3

                                                LR chi2(0)        =       2.20
Log likelihood  = -2.1972246                    Prob > chi2       =          .

------------------------------------------------------------------------------
           y |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
          x1 |     50.625          .        .       .            .           .
------------------------------------------------------------------------------

.

[Clyde Schechter]

What -xtpoisson- is complaining about is not that y is constant within some group, but that it is always zero in that group. Even in your example data you can see that: when Country_Group = 5, y is also constant--but it is always 1, and that group is not dropped.

The problem arises because of the form of the log-likelihood for the Poisson distribution if all values of y are 0: all of the terms in the log likelihood that depend on the values of y vanish, and as a result the maximum likelihood estimate for any coefficient will be negative infinity. Such an estimation can never converge, and Stata simply checks for this condition ahead of time and avoids the problem rather than getting mired in it.

The problem is similar with the logistic model, except that it has a similar difficulty with either all 0 or all 1: the likelihood is maximized with the coefficient being negative infinity or positive infinity, respectively. So notice that in your logistic regression, both groups 1 and 5 are omitted.

[Joao Santos Silva]

Dear All,

Clyde's explanation is not entirely correct. It is true that if y is always zero the corresponding fixed effect would go to minus infinity, but I do not think that Stata is aware of that because it does not include that check in standard Poisson regression models. That is why users introduced that check in commands such as ppml and ppmlhdfe.

What is going on here is the following. The Poisson FE estimator is based on the probability that Y=y conditional on the sum of y. If the sum of y is zero, we have that Pr(Y=0|x and sum of y) = 1 and therefore does not contribute to the likelihood function. Stata then drops these observations because they are not informative about the parameters and to make computations easier. In the case that y is always 1, the sum is 3 and that value can be obtained with different sequences of y, for example 3, 0, 0 or 0, 0, 3 or 2, 0, 1. So, these observations contain information about the parameters and are not dropped.

As a side note, I find it misleading that Stata reports the number of observations after deleting groups with all zeros; these observations did not have to be dropped, they were dropped only for convenience. Reporting the reduced number of observations often leads to confusion and unfounded fears of sample selection issues. Maybe someone in Stata is reading this?

Best wishes,

Joao

[Neha Bhardwaj Up]

Thank you all for your insights.