tobit regression with collinearity

[Anya hewertson]

Hi i am running a tobit regresison for data across 2 years 2007-2008:
My variables include 10 log price categories for alcohol types on trade and off trade : l_p_wine_on l_p_wine_off etc
I also have a log income variable : log_income
My dependent variables are the expenditure shares of the alcohol type expenditure divided by total expenditure : e.g exp_share_wine_on expshare_wine_off
I am looking at the price elasticities of demand and the cross price elasticities of demand vary across each alcohol type and vary across socio-economic groups, government regions and gender

My prices for alcohols are constant throughout the year (i am using the average year price) however they vary between years

here is a data-ex for some of my variables

* Example generated by -dataex-. To install: ssc install dataex
clear
input float(l_p_wine_on l_p_beer_on l_p_spirits_on l_p_wine_off l_p_spirits_off
>  l_p_beer_off expshare_wine_on expshare_beer_off logincome) byte(socio_group 
> gor) int year byte sexhrp
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0           0 
>  4.433789 6 2 2007 1
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0   .01142119 
>  5.898746 3 2 2007 1
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0           0 
>  4.898213 6 2 2007 2
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584    .0550356  .015000853 
>  6.399842 1 2 2007 2
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0 .0016348386 
>  5.584012 3 2 2007 2
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0  .015073973 
>  7.020905 2 2 2007 1
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0           0 
>  5.225338 6 2 2007 1
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0           0 
>  4.911331 3 2 2007 2
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0           0 
>  5.219934 6 2 2007 1
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0           0 
>  5.533279 6 2 2007 1
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0           0 
> 4.2492094 6 2 2007 2
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0   .00609936 
>  6.168564 6 2 2007 2
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0           0 
>  4.835587 6 2 2007 2
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0           0 
>  5.940566 6 2 2007 2
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0  .006249688 
>  5.331317 6 2 2007 1
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0           0 
>  5.786775 6 2 2007 1
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0  .003858888 
>  7.201894 2 2 2007 2
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0           0 
>  6.476967 2 2 2007 2
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0           0 
>  5.009435 6 2 2007 2
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0  .010382757 
>  6.377679 3 2 2007 1
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0           0 
>  5.982862 6 2 2007 1
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0           0 
>   6.11283 2 2 2007 1
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0 .0023888294 
>  6.279646 6 2 2007 1
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0  .001813489 
>  6.294915 1 2 2007 2
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0  .005435922 
>  6.704463 1 2 2007 2
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0           0 
>  6.747566 3 2 2007 1
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0  .005957043 
>   6.11456 1 2 2007 2
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584  .014408222  .016718158 
>  6.605068 1 2 2007 1
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584  .018981254           0 
>  6.019785 1 2 2007 2
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0           0 
>  6.088818 1 2 2007 1
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0           0 
>  4.779476 6 2 2007 1
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0  .008590408 
>  6.514719 6 2 2007 2
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584  .005628793  .018012136 
>  6.960443 1 2 2007 2
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0  .005657709 
>  6.424075 3 2 2007 1
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0           0 
>  5.920457 6 2 2007 1
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584  .008473212           0 
>  6.898255 1 2 2007 2
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0   .02177079 
>  5.623837 6 2 2007 1
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0           0 
>  4.812526 6 2 2007 2
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0           0 
>  6.182973 1 2 2007 2
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0           0 
>  4.514611 6 2 2007 2
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0           0 
>  6.109314 1 2 2007 1
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0           0 
>  5.362559 3 2 2007 1
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0           0 
>   5.30903 6 2 2007 1
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0           0 
>   5.26414 6 2 2007 2
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0           0 
> 4.3593974 3 2 2007 2
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584  .017695729           0 
>   4.77104 6 2 2007 2
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0           0 
>  5.069847 6 2 2007 2
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0   .01336186 
>  6.690271 2 2 2007 1
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0           0 
>   5.80408 1 2 2007 1
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0           0 
>  6.628306 5 2 2007 2
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0  .016276948 
>  6.522627 5 2 2007 1
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0           0 
>  5.519619 6 2 2007 2
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0  .035966147 
>  6.422951 3 2 2007 1
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0           0 
>  6.557673 1 2 2007 1
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0           0 
>  6.602438 2 2 2007 1
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0           0 
>  6.402017 3 2 2007 2
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584 .0029820926           0 
>  7.401286 1 2 2007 1
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0  .015186014 
>  7.176426 2 2 2007 1
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0           0 
>  5.746554 6 2 2007 1
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0           0 
>  5.474176 6 2 2007 1
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0   .01607261 
>  6.874416 3 2 2007 1
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0  .010095213 
>  6.662046 1 2 2007 2
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584  .024986824    .0423164 
>  6.069906 3 2 2007 1
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584  .001250104           0 
>  7.438652 1 2 2007 1
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584   .03693495           0 
>  7.021414 1 2 2007 2
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0   .02268917 
>    6.5658 1 2 2007 1
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0           0 
>  6.958667 2 2 2007 1
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0           0 
>  6.192117 1 2 2007 2
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584 .0016070686           0 
>  5.815264 6 2 2007 2
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0           0 
>   5.34921 6 2 2007 1
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0           0 
>     6.279 1 2 2007 1
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0           0 
>  5.516609 1 2 2007 1
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0           0 
>  5.554516 6 2 2007 2
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0           0 
>  6.347932 1 2 2007 1
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584   .01880577           0 
>   5.93925 1 2 2007 1
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0   .07133046 
>  6.985651 2 2 2007 1
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584  .005728897  .005415315 
>   7.12227 1 2 2007 1
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584   .01053234  .021376746 
>    6.8088 2 2 2007 1
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584 .0019776237           0 
>  6.519822 3 2 2007 1
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0           0 
>  6.490757 6 2 2007 1
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0           0 
>  6.787439 1 2 2007 1
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0           0 
>  6.457868 1 2 2007 2
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0           0 
>  6.921752 2 2 2007 1
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0           0 
>  7.098411 2 2 2007 2
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0           0 
>  4.400603 6 2 2007 2
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584  .013181653           0 
>  6.857086 1 2 2007 2
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584  .003744323           0 
>  6.710182 1 2 2007 1
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0           0 
>  6.136498 6 2 2007 1
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584   .02200635           0 
>  7.438652 1 2 2007 1
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584 .0019496685           0 
>  6.911319 1 2 2007 1
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0 .0006786454 
>  6.854755 1 2 2007 1
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0           0 
>  7.438652 2 2 2007 1
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0           0 
>  6.609726 1 2 2007 1
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584  .004789272           0 
>  6.868133 3 2 2007 1
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0           0 
>  5.182907 4 2 2007 2
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0           0 
>  5.823194 1 2 2007 2
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0           0 
>  4.812526 6 2 2007 2
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0   .04808098 
>  5.530222 4 2 2007 2
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0           0 
>  5.793585 3 2 2007 1
.60158 -.010050327 .662688 -.9416085 -.967584 -.967584           0   .11235794 
>  6.436151 1 2 2007 2
end
label values gor gor
label def gor 2 "north west", modify
label values sexhrp sexhrp
label def sexhrp 1 "male", modify
label def sexhrp 2 "female", modify

I am then running a tobit regression as follows:

 tobit expshare_wine_on l_p_wine_on l_p_beer_on l_p_cider_on l_p_spirits_on l_p_alcopops_on l_p_wine_off l_p_beer_off l_p_spirits_off l_p_cider_off l_p_alcopops_off logincome i.socio_group i.gor i.year i.sexhrp , ll(0)

I have censored the data at zero since some households report no consumption of alcohol

However my results are as follows:

. tobit expshare_wine_on l_p_wine_on l_p_beer_on l_p_cider_on l_p_spirits_on l_p_alcopops_on l_p_wine_off
>  l_p_beer_off l_p_spirits_off l_p_cider_off l_p_alcopops_off logincome i.socio_group i.gor i.year i.sex
> hrp , ll(0)
note: l_p_beer_on omitted because of collinearity
note: l_p_cider_on omitted because of collinearity
note: l_p_spirits_on omitted because of collinearity
note: l_p_alcopops_on omitted because of collinearity
note: l_p_wine_off omitted because of collinearity
note: l_p_beer_off omitted because of collinearity
note: l_p_spirits_off omitted because of collinearity
note: l_p_cider_off omitted because of collinearity
note: l_p_alcopops_off omitted because of collinearity
note: 2008.year omitted because of collinearity

Refining starting values:

Grid node 0:   log likelihood = -5976.9775

Fitting full model:

Iteration 0:   log likelihood = -5976.9775  
Iteration 1:   log likelihood = -640.92644  
Iteration 2:   log likelihood =   1103.185  
Iteration 3:   log likelihood =  1808.8673  
Iteration 4:   log likelihood =  1909.2432  
Iteration 5:   log likelihood =   1910.562  
Iteration 6:   log likelihood =  1910.5625  
Iteration 7:   log likelihood =  1910.5625  

Tobit regression                                Number of obs     =     11,962
                                                   Uncensored     =      2,312
Limits: lower = 0                                  Left-censored  =      9,650
        upper = +inf                               Right-censored =          0

                                                LR chi2(14)       =     927.97
                                                Prob > chi2       =     0.0000
Log likelihood =  1910.5625                     Pseudo R2         =    -0.3207

-------------------------------------------------------------------------------------------
         expshare_wine_on |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
--------------------------+----------------------------------------------------------------
              l_p_wine_on |   -.028118   .0162136    -1.73   0.083    -.0598992    .0036632
              l_p_beer_on |          0  (omitted)
             l_p_cider_on |          0  (omitted)
           l_p_spirits_on |          0  (omitted)
          l_p_alcopops_on |          0  (omitted)
             l_p_wine_off |          0  (omitted)
             l_p_beer_off |          0  (omitted)
          l_p_spirits_off |          0  (omitted)
            l_p_cider_off |          0  (omitted)
         l_p_alcopops_off |          0  (omitted)
                logincome |   .0125922   .0006706    18.78   0.000     .0112778    .0139066
                          |
              socio_group |
                       2  |   .0014811   .0010997     1.35   0.178    -.0006745    .0036368
                       3  |  -.0078991   .0012672    -6.23   0.000    -.0103829   -.0054152
                       4  |  -.0098159    .003836    -2.56   0.011    -.0173351   -.0022968
                       5  |   .0065436   .0035439     1.85   0.065    -.0004031    .0134903
                       6  |  -.0027114   .0010429    -2.60   0.009    -.0047556   -.0006672
                          |
                      gor |
              north west  |  -.0004291   .0014892    -0.29   0.773    -.0033481      .00249
              merseyside  |  -.0009579   .0014654    -0.65   0.513    -.0038303    .0019145
yorkshire and the humber  |   .0017352   .0014609     1.19   0.235    -.0011284    .0045987
           east midlands  |  -.0012567   .0021903    -0.57   0.566      -.00555    .0030366
           west midlands  |  -.0024181   .0018331    -1.32   0.187    -.0060112    .0011751
                 eastern  |  -.0016493   .0017609    -0.94   0.349     -.005101    .0018023
                          |
                     year |
                    2008  |          0  (omitted)
                          |
                   sexhrp |
                  female  |   .0011694    .000803     1.46   0.145    -.0004046    .0027435
                    _cons |  -.0853166   .0110156    -7.75   0.000     -.106909   -.0637242
--------------------------+----------------------------------------------------------------
   var(e.expshare_wine_on)|   .0007865   .0000268                      .0007357    .0008408
------------------------------------------------------------------------------------------

Q1. Why are my price variables other than the price of the same dependent variable omitted (as i am trying to work out cross price elasticity of demand) i understand its due to collinearity but what is causing this and how do i overcome it?
Q2. Why is the year dummy variable omitted?

I am following a model which has done close to the same thing and they didn't have this problem

Thanks so much in advance

[FernandoRios]

Hi Anya
Based on your data extract, and your output, it doesnt seem you have any information for the 2008 year. That may be who it is being omitted. Because of this, for the regression, prices do not change (as you are using only 2007 data i believe), thus they are also dropped due to co linearity (or more precisely because they are constants).
Before you try to venture more into why Tobit is not working, why dont you create some summary statistics, to see if anything is odd
Something like:

sum expshare_wine_on l_p_wine_on l_p_beer_on l_p_cider_on l_p_spirits_on l_p_alcopops_on l_p_wine_off l_p_beer_off l_p_spirits_off l_p_cider_off l_p_alcopops_off logincome i.socio_group i.gor i.year i.sex hrp if year==2007
sum expshare_wine_on l_p_wine_on l_p_beer_on l_p_cider_on l_p_spirits_on l_p_alcopops_on l_p_wine_off l_p_beer_off l_p_spirits_off l_p_cider_off l_p_alcopops_off logincome i.socio_group i.gor i.year i.sex hrp if year==2008

That will let you see if all data is indeed available, and if your prices indeed vary across time.

HTH
Fernando

[Anya hewertson]

Thanks so much for your reply,
I originally appended my 2007 and 2008 datasets together to make this overall dataset- could that be a problem?
I have used the codes you provided and this is the outcome:

 sum expshare_wine_on l_p_wine_on l_p_beer_on l_p_cider_on l_p_spirits_on l_p_alcopops_on l_p_wine_off l_p_beer_off l_p_spirits_off l_p_cider_off l_p_alcopops_off logincome i.socio_group i.gor i.year i.sexhrp if year==2007

    Variable |        Obs        Mean    Std. Dev.       Min        Max
-------------+---------------------------------------------------------
expshare_w~n |      6,136    .0029185    .0090731          0   .1457427
 l_p_wine_on |      6,136      .60158           0     .60158     .60158
 l_p_beer_on |      6,136   -.0100503           0  -.0100503  -.0100503
l_p_cider_on |      6,136   -.1743534           0  -.1743534  -.1743534
l_p_spirit~n |      6,136     .662688           0    .662688    .662688
-------------+---------------------------------------------------------
l_p_alcopo~n |      6,136    .5877866           0   .5877866   .5877866
l_p_wine_off |      6,136   -.9416085           0  -.9416085  -.9416085
l_p_beer_off |      6,136    -.967584           0   -.967584   -.967584
l_p_spirit~f |      6,136    -.967584           0   -.967584   -.967584
l_p_cider_~f |      6,136   -1.366492           0  -1.366492  -1.366492
-------------+---------------------------------------------------------
l_p_alcopo~f |      6,136    -.198451           0   -.198451   -.198451
   logincome |      6,125    6.131905    .8331424  -2.120264   7.438652
             |
 socio_group |
          2  |      6,136    .1134289    .3171423          0          1
          3  |      6,136    .1357562    .3425574          0          1
-------------+---------------------------------------------------------
          4  |      6,136    .0202086    .1407247          0          1
          5  |      6,136    .0110821    .1046953          0          1
          6  |      6,136    .3536506    .4781413          0          1
             |
         gor |
 north west  |      6,136    .2257171    .4180878          0          1
-------------+---------------------------------------------------------
 merseyside  |      6,136    .2459257    .4306698          0          1
yorkshire..  |      6,136    .2180574    .4129602          0          1
east midl..  |      6,136    .0454694    .2083482          0          1
west midl..  |      6,136    .0816493     .273852          0          1
    eastern  |      6,136    .0971317    .2961611          0          1
-------------+---------------------------------------------------------
             |
        year |
       2007  |      6,136           1           0          1          1
             |
      sexhrp |
     female  |      6,136    .3855932    .4867748          0          1



and for 2008: 

sum expshare_wine_on l_p_wine_on l_p_beer_on l_p_cider_on l_p_spirits_on l_p_alcopops_on l_p_wine_off l _p_beer_off l_p_spirits_off l_p_cider_off l_p_alcopops_off logincome i.socio_group i.gor i.year i.sexhrp if year==2008

    Variable |        Obs        Mean    Std. Dev.       Min        Max
-------------+---------------------------------------------------------
expshare_w~n |      5,843    .0027839    .0092569          0   .2517651
 l_p_wine_on |      5,843    .6471033           0   .6471033   .6471033
 l_p_beer_on |      5,843    .0295588           0   .0295588   .0295588
l_p_cider_on |      5,843    .5988365           0   .5988365   .5988365
l_p_spirit~n |      5,843    .7080358           0   .7080358   .7080358
-------------+---------------------------------------------------------
l_p_alcopo~n |      5,843    .7178398           0   .7178398   .7178398
l_p_wine_off |      5,843   -.9038682           0  -.9038682  -.9038682
l_p_beer_off |      5,843   -.9416085           0  -.9416085  -.9416085
l_p_spirit~f |      5,843   -.9162907           0  -.9162907  -.9162907
l_p_cider_~f |      5,843   -1.290984           0  -1.290984  -1.290984
-------------+---------------------------------------------------------
l_p_alcopo~f |      5,843   -.1743534           0  -.1743534  -.1743534
   logincome |      5,837    6.168688    .8281505  -.5447271   7.527783
             |
 socio_group |
          2  |      5,843     .119288    .3241549          0          1
          3  |      5,843    .1364025    .3432449          0          1
-------------+---------------------------------------------------------
          4  |      5,843    .0210508    .1435661          0          1
          5  |      5,843    .0099264    .0991441          0          1
          6  |      5,843     .356495    .4790048          0          1
             |
         gor |
 north west  |      5,843    .2255691    .4179923          0          1
-------------+---------------------------------------------------------
 merseyside  |      5,843    .2406298    .4275025          0          1
yorkshire..  |      5,843    .2238576    .4168634          0          1
east midl..  |      5,843    .0453534    .2080959          0          1
west midl..  |      5,843    .0855725    .2797557          0          1
    eastern  |      5,843    .0982372    .2976606          0          1
-------------+---------------------------------------------------------
             |
        year |
       2008  |      5,843           1           0          1          1
             |
      sexhrp |
     female  |      5,843    .3717269    .4833073          0          1

The prices therefore do vary between the two years

What shall i try now