Omitted dummy variables due to collinearity, but missing values should prevent them entering the model

[Francesco Defendi]

Dear Statalisters,
I am using Stata SE 17 on Windows 10. I'm conducting a panel data analysis with the goal of investigating the impact of innovation in attracting foreign direct investments (FDI). My dataset consists of 37 EU countries, measured on 14 variables across 14 years. Here is a sample of my dataset:

* Example generated by -dataex-. For more info, type help dataex
clear
input str18 COUNTRY int YEAR double(newdocs citedpubbl edupop publicrd venture businessrd nordbusexp pctapps trademarks designs SMEsmax innosales infdi1) long country_n
"Austria" 2004               2.2 10.879386829323456    34               .72               .0485 1.53                  .     4.804806595897  5.750369762148659  5.96433700911233  53.26164874551972 10.599653308926985  25.66539714709834 2
"Austria" 2005                 2  10.90807814642101  20.5               .74 .052000000000000005 1.71                  .  5.082864356344017   5.44394195271982 6.113029790484267 54.073638071027744  12.12044275080061 3.1251494779180087 2
"Austria" 2006                 2  11.14425359820278  21.2               .72 .044000000000000004 1.73                  .  5.365545654591564    7.5000552377227 7.070428505610924  54.88562739653577 13.641232192674234 17.717340682610388 2
"Austria" 2007               1.9 11.766702785799323  21.1               .74 .028000000000000004 1.78                  . 5.0537632261303145 7.7373498636957745  9.22138609767386  48.83508332512358 12.442621838711055  1.464380045626025 2
"Austria" 2008                 2  11.25539528832986  22.2               .78               .0215 1.88 .46944562318694916   4.99638723207233  7.753349369779991 7.113023003917563  42.78453925371138 11.244011484747876 3.5734768412160807 2
"Austria" 2009              2.15 10.744087790860396  23.5               .81               .0285 1.94 .46944562318694916  4.939011238014344  9.559393223111512 9.190448731846583  42.77785207971112 11.244011484747876 -5.615650034432947 2
"Austria" 2010               2.3    11.092568305056 30.95               .84  .03956166277617255 1.87  .3533407195307009  5.252049058377892 11.693662770783515 8.594888189423665 42.326375213338586 11.916767341477245  5.331213212198524 2
"Austria" 2011               2.2   11.0989727110624  38.4 .8200000000000001  .05062332555234509 1.84  .3533407195307009  5.129995316751745 13.827932318455517 7.999327647000747 42.326375213338586 11.916767341477245 1.2746210241656093 2
"Austria" 2012               2.2   11.0627916896433  38.4               .85 .044624883377768744 2.05  .4575665576828351  4.757594759441145   13.3241487743434 8.298052792230653 44.708133971291865  9.846163545405481 .10489199786555246 2
"Austria" 2013 2.008113571981588 11.082658226067501  38.4               .85  .04394477403736861 2.09  .4575665576828351  4.964945807646287 13.406225799062174 8.000981365169235 44.708133971291865  9.846163545405481 .38737336166369846 2
end
label values country_n country_n
label def country_n 2 "Austria", modify

As an initial exploratory model for fixed effects, I'm running a least square dummy variables (LSDV) model. What happens is that 6 countries dummy variables are dropped from the model, apparently due to multicollinearity. To check that, I have investigated the correlation matrix of my -indepvars- with such dummies. A first possibility is that the countries dropped have at least one variable that does not vary across years, i.e. the value for the variable remains the same across all the time series for the given country. However, this is not the case in any of the dropped countries above. None of them have a variable that takes on the same value for every given year. We can see from the correlation table below that there is no one single variable correlating 100% with any of the country dummy. What we instead see is that for all the dropped countries, at least one variable has complete missing values. Here is the code:

*
. xi: regress infdi1 i.country_n newdocs citedpubbl edupop publicrd venture businessrd nordbusexp  pctapps trademarks designs
> SMEsmax  innosales
i.country_n       _Icountry_n_2-38    (naturally coded; _Icountry_n_2 omitted)
note: _Icountry_n_15 omitted because of collinearity.
note: _Icountry_n_17 omitted because of collinearity.
note: _Icountry_n_23 omitted because of collinearity.
note: _Icountry_n_25 omitted because of collinearity.
note: _Icountry_n_36 omitted because of collinearity.
note: _Icountry_n_37 omitted because of collinearity.

      Source |       SS           df       MS      Number of obs   =       309
-------------+----------------------------------   F(42, 266)      =     13.80
       Model |  146034.029        42  3477.00069   Prob > F        =    0.0000
    Residual |  67043.2023       266  252.042114   R-squared       =    0.6854
-------------+----------------------------------   Adj R-squared   =    0.6357
       Total |  213077.231       308  691.809193   Root MSE        =    15.876

--------------------------------------------------------------------------------
        infdi1 | Coefficient  Std. err.      t    P>|t|     [95% conf. interval]
---------------+----------------------------------------------------------------
 _Icountry_n_3 |  -39.68065   10.59999    -3.74   0.000     -60.5512    -18.8101
 _Icountry_n_4 |   28.69714   24.12934     1.19   0.235    -18.81165    76.20593
 _Icountry_n_5 |  -9.110369   22.86466    -0.40   0.691     -54.1291    35.90836
 _Icountry_n_6 |   216.9712   22.30024     9.73   0.000     173.0638    260.8787
 _Icountry_n_7 |  -4.527713   18.12123    -0.25   0.803    -40.20701    31.15158
 _Icountry_n_8 |  -50.57355   11.24234    -4.50   0.000    -72.70884   -28.43825
 _Icountry_n_9 |   6.945694   16.86783     0.41   0.681    -26.26575    40.15714
_Icountry_n_10 |  -32.77956    14.4839    -2.26   0.024    -61.29724    -4.26188
_Icountry_n_11 |  -35.29605   11.20799    -3.15   0.002    -57.36371   -13.22839
_Icountry_n_12 |  -30.76903   10.30141    -2.99   0.003     -51.0517   -10.48636
_Icountry_n_13 |  -52.51921    19.5321    -2.69   0.008    -90.97638   -14.06203
_Icountry_n_14 |   3.337269     18.891     0.18   0.860    -33.85764    40.53218
_Icountry_n_15 |          0  (omitted)
_Icountry_n_16 |  -31.11075    14.9042    -2.09   0.038    -60.45597   -1.765537
_Icountry_n_17 |          0  (omitted)
_Icountry_n_18 |  -25.14016   13.35924    -1.88   0.061    -51.44347    1.163155
_Icountry_n_19 |   4.665286    23.9956     0.19   0.846    -42.58018    51.91076
_Icountry_n_20 |  -36.70157   25.56419    -1.44   0.152    -87.03547    13.63233
_Icountry_n_21 |    134.045   20.08224     6.67   0.000     94.50467    173.5854
_Icountry_n_22 |   201.7102   26.09115     7.73   0.000     150.3388    253.0817
_Icountry_n_23 |          0  (omitted)
_Icountry_n_24 |  -35.33513   13.78443    -2.56   0.011    -62.47559   -8.194669
_Icountry_n_25 |          0  (omitted)
_Icountry_n_26 |  -77.66236   15.07972    -5.15   0.000    -107.3531   -47.97157
_Icountry_n_27 |  -17.43932   23.78558    -0.73   0.464    -64.27127    29.39263
_Icountry_n_28 |  -32.29503   16.40757    -1.97   0.050    -64.60025     .010194
_Icountry_n_29 |  -1.508972   23.43871    -0.06   0.949    -47.65798    44.64004
_Icountry_n_30 |  -13.44188   24.38399    -0.55   0.582    -61.45206     34.5683
_Icountry_n_31 |  -14.65724   23.39618    -0.63   0.532    -60.72251    31.40802
_Icountry_n_32 |   11.82224   13.69928     0.86   0.389    -15.15057    38.79505
_Icountry_n_33 |  -23.05757   16.79391    -1.37   0.171    -56.12347    10.00832
_Icountry_n_34 |  -43.37165   16.82267    -2.58   0.010    -76.49418   -10.24912
_Icountry_n_35 |  -16.72277   15.32675    -1.09   0.276    -46.89995    13.45442
_Icountry_n_36 |          0  (omitted)
_Icountry_n_37 |          0  (omitted)
_Icountry_n_38 |  -58.98577   14.03119    -4.20   0.000    -86.61209   -31.35944
       newdocs |   6.546929   2.884637     2.27   0.024     .8673014    12.22656
    citedpubbl |   5.621024   1.619945     3.47   0.001     2.431478    8.810571
        edupop |    1.00983   .2822136     3.58   0.000     .4541734    1.565487
      publicrd |   23.46202   13.69844     1.71   0.088    -3.509142    50.43318
       venture |  -39.16915   28.68726    -1.37   0.173    -95.65213    17.31383
    businessrd |  -20.99256   7.292466    -2.88   0.004    -35.35086   -6.634264
    nordbusexp |   12.61612   4.261034     2.96   0.003     4.226477    21.00577
       pctapps |   1.376031   2.827341     0.49   0.627    -4.190784    6.942846
    trademarks |  -5.781484   .5329737   -10.85   0.000    -6.830868   -4.732101
       designs |   1.289679   .8997131     1.43   0.153    -.4817863    3.061144
       SMEsmax |   .1067005   .2173757     0.49   0.624    -.3212953    .5346963
     innosales |   .0906274   .3701102     0.24   0.807    -.6380908    .8193456
         _cons |  -45.79126   28.25667    -1.62   0.106    -101.4264    9.843927
--------------------------------------------------------------------------------

. pwcorr _Icountry_n_15 _Icountry_n_17 _Icountry_n_23 _Icountry_n_25 _Icountry_n_36 _Icountry_n_37 newdocs citedpubbl edupop p
> ublicrd venture businessrd nordbusexp innosmes pctapps trademarks designs newprodsmes newmarksmes innosales

             | _Icou~15 _Icou~17 _Icou~23 _Icou~25 _Icou~36 _Icou~37  newdocs
-------------+---------------------------------------------------------------
_Icountry~15 |   1.0000
_Icountry~17 |  -0.0278   1.0000
_Icountry~23 |  -0.0278  -0.0278   1.0000
_Icountry~25 |  -0.0278  -0.0278  -0.0278   1.0000
_Icountry~36 |  -0.0278  -0.0278  -0.0278  -0.0278   1.0000
_Icountry~37 |  -0.0278  -0.0278  -0.0278  -0.0278  -0.0278   1.0000
     newdocs |  -0.1531   0.0326  -0.1740  -0.1914  -0.2170   0.0270   1.0000
  citedpubbl |   0.1402   0.0617  -0.1498  -0.1727  -0.1314  -0.2255   0.5676
      edupop |   0.0889   0.0666  -0.0239  -0.1722  -0.2059        .   0.2860
    publicrd |   0.2166   0.0080  -0.1834  -0.2058  -0.1086  -0.1613   0.5916
     venture |        .        .        .        .        .  -0.1129   0.1209
  businessrd |   0.0676   0.4475  -0.1358  -0.1671  -0.1285  -0.0980   0.6700
  nordbusexp |  -0.0550        .  -0.1211   0.0226   0.4278  -0.0601  -0.1834
    innosmes |        .  -0.0660  -0.3449        .   0.0095  -0.1118   0.4023
     pctapps |   0.0391   0.3285  -0.1016  -0.1372  -0.1154  -0.0988   0.6496
  trademarks |   0.0690  -0.0712  -0.0963  -0.1240  -0.1404  -0.0809   0.0653
     designs |  -0.1337  -0.0806  -0.1031  -0.1173  -0.1631  -0.1251   0.3172
 newprodsmes |   0.2048  -0.0849   0.2063   0.0203  -0.0024  -0.2914   0.3060
 newmarksmes |   0.0868   0.1800  -0.0164  -0.0766   0.0997  -0.2878   0.3435
   innosales |  -0.1254  -0.0002  -0.1159  -0.0865   0.3267  -0.2128   0.1543

             | citedp~l   edupop publicrd  venture busine~d nordbu~p innosmes
-------------+---------------------------------------------------------------
  citedpubbl |   1.0000
      edupop |   0.5480   1.0000
    publicrd |   0.6768   0.4121   1.0000
     venture |   0.4714   0.4025   0.2397   1.0000
  businessrd |   0.6688   0.3670   0.7089   0.2528   1.0000
  nordbusexp |  -0.2355  -0.2011  -0.1506  -0.3138  -0.1642   1.0000
    innosmes |   0.6663   0.2552   0.5391   0.2747   0.4693   0.1264   1.0000
     pctapps |   0.7072   0.3495   0.7418   0.2943   0.9086  -0.1670   0.5003
  trademarks |   0.3248   0.4248   0.0907   0.3081   0.1209  -0.1308   0.2329
     designs |   0.4506   0.1728   0.3118   0.2472   0.3621  -0.2186   0.3142
 newprodsmes |   0.6240   0.2506   0.4711   0.2896   0.4172   0.0676   0.7858
 newmarksmes |   0.6178   0.1713   0.3606   0.2701   0.4947   0.1592   0.7493
   innosales |   0.0855  -0.2519   0.0241  -0.1382   0.1074   0.3120   0.2043

             |  pctapps tradem~s  designs newpro~s newmar~s innosa~s
-------------+------------------------------------------------------
     pctapps |   1.0000
  trademarks |   0.0913   1.0000
     designs |   0.3681   0.6228   1.0000
 newprodsmes |   0.4313   0.2034   0.2837   1.0000
 newmarksmes |   0.4474   0.2385   0.2970   0.7794   1.0000
   innosales |   0.0507  -0.1319  -0.0694   0.1733   0.3042   1.0000

And then if we drop the variables that have all missing values for the given countries, Stata does not drop the dummies anymore:

*
. xi: regress infdi1 i.country_n newdocs citedpubbl  publicrd  businessrd   pctapps trademarks designs SMEsmax  innosales
i.country_n       _Icountry_n_2-38    (naturally coded; _Icountry_n_2 omitted)

      Source |       SS           df       MS      Number of obs   =       422
-------------+----------------------------------   F(45, 376)      =     10.85
       Model |  376799.153        45  8373.31452   Prob > F        =    0.0000
    Residual |  290217.467       376  771.854965   R-squared       =    0.5649
-------------+----------------------------------   Adj R-squared   =    0.5128
       Total |   667016.62       421  1584.36252   Root MSE        =    27.782

--------------------------------------------------------------------------------
        infdi1 | Coefficient  Std. err.      t    P>|t|     [95% conf. interval]
---------------+----------------------------------------------------------------
 _Icountry_n_3 |   -28.2702   13.38353    -2.11   0.035    -54.58615   -1.954256
 _Icountry_n_4 |   24.24601   31.04683     0.78   0.435    -36.80117    85.29319
 _Icountry_n_5 |   16.97781   28.33762     0.60   0.549    -38.74227    72.69789
 _Icountry_n_6 |   137.1536   27.71099     4.95   0.000     82.66567    191.6415
 _Icountry_n_7 |   5.228704   23.78324     0.22   0.826    -41.53612    51.99353
 _Icountry_n_8 |  -42.37346   15.37538    -2.76   0.006    -72.60597   -12.14095
 _Icountry_n_9 |   12.61542    21.0514     0.60   0.549    -28.77781    54.00865
_Icountry_n_10 |  -26.85992    20.6996    -1.30   0.195     -67.5614    13.84156
_Icountry_n_11 |  -21.32491   13.17534    -1.62   0.106    -47.23148    4.581669
_Icountry_n_12 |  -12.45769   13.80738    -0.90   0.368    -39.60706    14.69167
_Icountry_n_13 |  -25.95055   24.42883    -1.06   0.289    -73.98479    22.08369
_Icountry_n_14 |  -3.152964   24.76806    -0.13   0.899    -51.85423     45.5483
_Icountry_n_15 |   13.24128   17.85137     0.74   0.459    -21.85976    48.34231
_Icountry_n_16 |  -10.78169   17.10134    -0.63   0.529    -44.40793    22.84456
_Icountry_n_17 |  -7.212702    28.1325    -0.26   0.798    -62.52945    48.10405
_Icountry_n_18 |  -22.79081   18.63951    -1.22   0.222    -59.44155    13.85993
_Icountry_n_19 |   26.55662   30.40193     0.87   0.383    -33.22248    86.33573
_Icountry_n_20 |   13.13162   28.92817     0.45   0.650    -43.74964    70.01288
_Icountry_n_21 |   99.24663   21.73214     4.57   0.000     56.51487    141.9784
_Icountry_n_22 |   157.6805   28.54731     5.52   0.000     101.5481    213.8129
_Icountry_n_23 |   50.34346   32.37068     1.56   0.121    -13.30679    113.9937
_Icountry_n_24 |  -30.22928   18.08496    -1.67   0.095    -65.78961    5.331054
_Icountry_n_25 |   23.79676   32.78206     0.73   0.468    -40.66239    88.25591
_Icountry_n_26 |   -42.6865   17.13695    -2.49   0.013    -76.38277    -8.99024
_Icountry_n_27 |   8.445098    29.7548     0.28   0.777    -50.06157    66.95176
_Icountry_n_28 |  -11.27378   21.68761    -0.52   0.603    -53.91799    31.37042
_Icountry_n_29 |  -5.421011   30.97143    -0.18   0.861    -66.31992     55.4779
_Icountry_n_30 |   12.71005   30.43586     0.42   0.676    -47.13578    72.55587
_Icountry_n_31 |  -20.50129   30.08854    -0.68   0.496    -79.66417    38.66159
_Icountry_n_32 |   6.574371   17.41182     0.38   0.706    -27.66237    40.81111
_Icountry_n_33 |  -39.01039   21.29306    -1.83   0.068    -80.87879    2.858012
_Icountry_n_34 |  -25.11505   21.70173    -1.16   0.248    -67.78702    17.55692
_Icountry_n_35 |  -14.98214   19.05556    -0.79   0.432    -52.45097    22.48669
_Icountry_n_36 |  -14.36639   27.73303    -0.52   0.605    -68.89767    40.16488
_Icountry_n_37 |   26.49698   32.62507     0.81   0.417    -37.65348    90.64744
_Icountry_n_38 |  -58.04496   16.42962    -3.53   0.000    -90.35042    -25.7395
       newdocs |    3.57746   4.202534     0.85   0.395    -4.685955    11.84087
    citedpubbl |   8.428897   2.052776     4.11   0.000     4.392537    12.46526
      publicrd |    .152932   19.57373     0.01   0.994    -38.33475    38.64062
    businessrd |  -7.091957    9.91889    -0.71   0.475     -26.5954    12.41149
       pctapps |   .1559082   3.595107     0.04   0.965    -6.913126    7.224942
    trademarks |  -2.883817   .4868718    -5.92   0.000     -3.84115   -1.926484
       designs |  -1.008264   1.007529    -1.00   0.318     -2.98936    .9728332
       SMEsmax |  -1.165689    .280415    -4.16   0.000    -1.717067   -.6143111
     innosales |   2.516477   .4543427     5.54   0.000     1.623106    3.409848
         _cons |   -21.5053   38.12557    -0.56   0.573    -96.47135    53.46075
--------------------------------------------------------------------------------

I'm thus quite puzzled and have two questions:

1. Shouldn't Stata treat missing values with listwise deletion by default? This would prevent the dummies of the countries presenting missing values along one whole variable to enter the model in the first place. Also, I am not understanding why Stata treats them as collinear. I have found this and this previous posts about similar issues, and I understand this might be due to the dummies being collinear to the coefficients of the fixed effects, i.e. the other dummies in the model, but I'm not really sure if that is the case.
2. I'm not planning to use LSDV as my final model. Instead, since my variable are heteroskedastic, autocorrelated and cross-sectionally dependent, I'm planning to use Driscoll-Kraay standard errors with the community command -xtscc-. Is by any chance the issue presented in this post affecting the consistency of other models?

Thank you for your time,
Francesco Defendi

[Carlo Lazzaro]

Francesco:
welcome to this forum.
It seems that the culprit is -nordbusexp- (here missing values seem to have a bearing on ensuing OLS estimates) along with -country- (omitted for collinearity).
In addition:
1) -xi._ prefix is redundant if you go -fvvarlist-. Some community-contributed modules still need -xi:- because they do not support -fvvarlist- notation-;
2) if you already detected nuisances such as heteroskedasticity, auto and cross-sectinal correlation of -epsilon- an LSDV with default standard errors is not reliable;
3) please note that the conmmunity-contributed module -xtscc- was conceived for T>N panel datasets.

[Francesco Defendi]

Hi Carlo! Thank you for replying to this thread and a lot of other threads in the past regarding panel data: you have been already helpful to me indirectly in the past weeks

I have still two doubts:

• Shouldn't the dummy variables not even enter the model, in case there are variable completely missing for the relative countries?
• In Daniel Hoechle's 2007 paper, introducing the command -xtscc-, it states that: "Because this nonparametric technique of estimating standard errors places no restrictions on the limiting behavior of the number of panels, the size of the cross-sectional dimension in finite samples does not constitute a constraint on feasibility—even if the number of panels is much larger than T. Nevertheless, because the estimator is based on an asymptotic theory, one should be somewhat cautious with applying this estimator to panels that contain a large cross-section but only a short time dimension.". Given that the gap between cross-sectional units and time series is not large, I guess I can then apply -xtscc- to my dataset anyways, right?

Thanks again for your time!

[Carlo Lazzaro]

Francesco:
1) Not quite. Stata applies, as you wrote, listwise deletion, that ends up in complete case analysis (and not available case analysis). Put differently, all the observations with missing values in any variable will be ruled out from the ensuing statistic;
2) yes, you can. Following the very same Daniel Hoechle's statement reported in your reply, my previous warning in 3) meant to be cautious in applying this estimator when the T>N requirement is not perfectly met, as in your case, as far as I can get it.

[Francesco Defendi]

Thanks Carlo for you replies.

1) So indeed if I have, say, a country that is behaving like this:

. list COUNTRY venture if COUNTRY == "Iceland"

     +-------------------+
     | COUNTRY   venture |
     |-------------------|
196. | Iceland         . |
197. | Iceland         . |
198. | Iceland         . |
199. | Iceland         . |
200. | Iceland         . |
     |-------------------|
201. | Iceland         . |
202. | Iceland         . |
203. | Iceland         . |
204. | Iceland         . |
205. | Iceland         . |
     |-------------------|
206. | Iceland         . |
207. | Iceland         . |
208. | Iceland         . |
209. | Iceland         . |
210. | Iceland         . |
     +-------------------+

I'm expecting that the relative country-dummy is ruled out from the ensuing statistics, since it refers to observations that have missing values. However, this is not happening, and Stata reports that it is dropping the dummy "due to collinearity", as presented in the initial query.

[Carlo Lazzaro]

Francesco:
it might be that Stata ranks omission due to collinearity higher than for missing values and, as such, the reason justifying omission is collinearity.
Elaborating on your previos -regress-, let's assume that there's a variabe with all missing values:

gen nordbusexp2=.
. regress infdi1 i.country_n newdocs citedpubbl edupop publicrd venture businessrd nordbusexp2  pctapps trademarks designs SMEsmax  innosales
no observations
r(2000);

.

[Francesco Defendi]

Carlo, I am really thankful for your help and I promise this is the last question for this thread:

We identified that -nordbusexp- is the culprit of the above issue. But what do you mean with:

Francesco:
[...] (here missing values seem to have a bearing on ensuing OLS estimates) [...]

Does this implicitly mean that dropping -nordbusexp- is the way to go?

Thanks again!

[Carlo Lazzaro]

Francesco:
most depends on -nordbusex- being relevant for giving a true and fair view of the data generating process you're investigating.
If its role is relevant in this respect, you may want to consider diagnosing its missing mechanism first and then dealing with the missing values accordingly.
Conversely, if it is just a control variable, I would get rid of it.