Unbalanced panel dataset

Dominik Bammer

Join Date: Apr 2022

Posts: 6
#1

Unbalanced panel dataset

14 Apr 2022, 01:12

Dear all,

I am running a DiD regression with an unbalanced panel dataset. The data consists of 41 countries with monthly observations from 2009 to 2020. When I xtreg stata drops almost half of the countries since the results only contain 24 countries. It is important to get all 41 countries included. What can be the issue here?

The code used is the following:

Code:

xtreg DVGTcirculareconomy i.treated##i.DiD1Time i.id i.period inflation gdp householdexp industrialization hdi educationexp recycling2 randd internet electricity age15plus work co2emission

DV = DVGTcirculareocnomy
treated = 1 for treated group and 0 for control group
DiD1Time = 1 post treatment and 0 pre treatment
Inflation to co2emission are control variables where I don't have complete data of all, thus the unbalanced dataset.

Super grateful for any help or advice.

Thank you and best regards,
Dominik
Tags: difference-in-difference, fixed effects, panel data, regression
Øyvind Snilsberg

Join Date: Oct 2021

Posts: 591
#2

14 Apr 2022, 02:08

Stata drops observations that have a missing value for any one of the variables used in the model, so to include all countries you must omit incomplete control variables from the regression.

Code:

xtset id xtreg DVGTcirculareconomy i.treated##i.DiD1Time i.period, fe
1 like
Comment
Dominik Bammer

Join Date: Apr 2022

Posts: 6
#3

14 Apr 2022, 11:26

Dear Øyvind,
thanks for your reply. Does that mean there is no other way than dropping all incomplete variables? Because in some other posts I read that stata has no problem dealing with unbalanced panel data.

Best regards,
Dominik
Comment
Carlo Lazzaro

Join Date: Apr 2014

Posts: 17613
#4

14 Apr 2022, 11:44

Dominik:
you're correct, but your statement does not conflict with Øyvind Snilsberg 's advice.
A panel dataset is unbalanced when when one or more panels do not have all the observations available for the whole period the panel dataset stretches over.
No matter if the panel dataset is balanced or not, Stata appplies listwise deletion and rules out all the observations with at least one missing value in any variable.

Last edited by Carlo Lazzaro; 14 Apr 2022, 11:46.

Kind regards,
Carlo
(StataNow 18.5)
1 like
Comment
Dominik Bammer

Join Date: Apr 2022

Posts: 6
#5

14 Apr 2022, 15:30

Dear Carlo,

thanks a lot for the clarification, I do understand it now. What is still weird though is that I do have some missing values in CV for a panel where it does return results. I then copied the values to another panel to check if there is an issue with the data but stata still delisted this country then. Also countries get deleted in a very suspicious way. I start to think there is a problem with the setting of the panel data.

Code:

xtset id period, monthly

Is there any possibility that the underlying panel does not get used in the correct way?

Many thanks and best regards,
Dominik
Comment
Carlo Lazzaro

Join Date: Apr 2014

Posts: 17613
#6

14 Apr 2022, 23:34

Dominik:
your -xtset- is fine, whereas the description of what's going on is not.
As per FAQ, please post an excerpt of your data via -dataex-. Thanks.

Kind regards,
Carlo
(StataNow 18.5)
Comment

Dominik Bammer

Join Date: Apr 2022
Posts: 6

15 Apr 2022, 01:16

Dear Carlo,

I used the two periods just randomly to have an example of all countries included.

Code:

* Example generated by -dataex-. For more info, type help dataex
* dataex id country period DVGTcirculareconomy inflation gdp educationexp treated DiD1Time if period==626 | period==702

clear
input byte id str15 country int period double(DVGTcirculareconomy inflation gdp educationexp) byte(treated DiD1Time)
 1 "Austria"         626                  1  2.429543  3771.833450004125 5.48040008544922 1 0
 1 "Austria"         702                  3  2.142162 3929.7369689093016 5.22535991668701 1 1
 2 "Belgium"         626                  1  3.360553 3384.5478392228742                . 1 0
 2 "Belgium"         702                 24  2.168331  3598.038848169059 6.37663984298706 1 1
 3 "Bulgaria"        626                  0         .  552.3946097541863 3.49359011650085 1 0
 3 "Bulgaria"        702                  2         .  678.3273438182007                . 1 1
 4 "Croatia"         626                  0         .  967.2272698315703                . 1 0
 4 "Croatia"         702                  0         .  1132.527281579487                . 1 1
 5 "Cyprus"          626                  0         . 2445.5799442559974                . 1 0
 5 "Cyprus"          702                  7         .  2715.923559168918                . 1 1
 6 "Czechia"         626                  0  3.711559 1406.6031932067253 4.22070980072021 1 0
 6 "Czechia"         702                  2  2.321083 1671.9187094723704 4.26692008972168 1 1
 7 "Denmark"         626                  0  2.609603  4367.700122015874 7.23782014846802 1 0
 7 "Denmark"         702                 13  1.078431  4804.022308662944                . 1 1
 8 "Estonia"         626                  0  4.398021 1376.4453492436662 4.68269014358521 1 0
 8 "Estonia"         702                 13  3.499261  1670.582768413737 5.23887014389038 1 1
 9 "Finland"         626                  0   2.94289 3694.5650118604212 7.14803981781006 1 0
 9 "Finland"         702                 19  1.368369  3873.775225140493 6.26929998397827 1 1
10 "France"          626                  1  2.301711 3063.4893993487062                . 1 0
10 "France"          702                  8  2.287808  3250.878224701549 5.41316986083984 1 1
11 "Germany"         626                  3  2.158979 3392.4877869558336 4.95658016204834 1 0
11 "Germany"         702                  5  1.853659 3648.3187673636007 4.99274015426636 1 1
12 "Greece"          626                  0  1.675075 1535.6825448594896                . 1 0
12 "Greece"          702                  3  .8519518 1582.0042616208607 3.59395003318787 1 1
13 "Hungary"         626                  0       5.5  968.8292566285149 4.15342998504639 1 0
13 "Hungary"         702                  1       3.4  1220.350163082231 4.62730979919434 1 1
14 "Ireland"         626                  0  2.244898 3894.7160960325646 5.76611995697021 1 0
14 "Ireland"         702                 13  .7920792  6164.196090448814 3.38309001922607 1 1
15 "Italy"           626 .07710861679850667  3.336605 2640.2530401762515 4.05601978302002 1 0
15 "Italy"           702                 13  1.482213 2682.6262494323314  4.2557201385498 1 1
16 "Latvia"          626                  0  3.290951  1054.494650115947 6.54672002792358 1 0
16 "Latvia"          702                  0  2.644246 1319.9653549676918 4.24517011642456 1 1
17 "Lithuania"       626                  0  3.638564 1073.4465581584425 4.75802993774414 1 0
17 "Lithuania"       702                  6  2.466976 1395.2589570866026 3.89495992660522 1 1
18 "Luxembourg"      626                  0  2.645503  8169.038215414039 4.09041023254395 1 0
18 "Luxembourg"      702                 42  1.458703  8855.123392296293 3.66626000404358 1 1
19 "Malta"           626                  0         .  1797.450439491186 6.34545993804932 1 0
19 "Malta"           702                  0         . 2300.8737177363137                . 1 1
20 "Netherlands"     626                  8  2.482345 3745.8489475659376  5.4102201461792 1 0
20 "Netherlands"     702                 34  2.125159  4060.370568375321 5.35764980316162 1 1
21 "Poland"          626  .8738435791632605       3.9  975.7110222722009 4.82954978942871 1 0
21 "Poland"          702                  1         2 1217.3162079109266 4.61579990386963 1 1
22 "Portugal"        626                  0   3.14852 1579.4293975879154 4.94940996170044 1 0
22 "Portugal"        702                 14  1.575729 1787.7360917081257 4.67515993118286 1 1
23 "Romania"         626                  0         .  678.0114354887513 2.96379995346069 1 0
23 "Romania"         702                  4         .   909.957468859748 3.34474992752075 1 1
24 "Slovakia"        626                  0  3.737113 1279.4704152347153 3.86286997795105 1 0
24 "Slovakia"        702                  4  2.610807   1503.97815685127   3.975830078125 1 1
25 "Slovenia"        626                  0  2.311957 1706.6008656477281 5.62023019790649 1 0
25 "Slovenia"        702                  0  1.867036 1993.0817708038094 4.93603992462158 1 1
26 "Spain"           626                  0  1.898629 2085.7469370127933  4.4693398475647 1 0
26 "Spain"           702                 22  2.246648  2353.847627839412 4.17816019058228 1 1
27 "Sweden"          626                  0  1.512367  4140.579237587834  7.5369701385498 1 0
27 "Sweden"          702                  6  2.051345  4499.926610175876 7.64084005355835 1 1
28 "United Kingdom"  626 .03553762627322321       3.1  3624.321185379476                . 1 0
28 "United Kingdom"  702                  6       2.3  3927.373180982715 5.23890018463135 1 1
29 "Australia"       626  .3526067893286666         .  4685.178310130045 4.86899995803833 0 0
29 "Australia"       702                  3         .  4973.690110832822 5.10789012908936 0 1
30 "Bosnia"          626                  0         . 356.87847517886024                . 0 0
30 "Bosnia"          702                  0         . 457.31541750434474                . 0 1
31 "Canada"          626                  3  1.926298  3584.125762469478                . 0 0
31 "Canada"          702                  3  2.990798  3802.001583117395                . 0 1
32 "Japan"           626                  6        .5  2838.993970064627                . 0 0
32 "Japan"           702                  1        .9 3073.5540298118717                . 0 1
33 "Montenegro"      626                  0         .  507.8367384480909                . 0 0
33 "Montenegro"      702                  0         .  626.9453314012117                . 0 1
34 "New Zealand"     626                  0         0 3094.4891704968295 7.15993022918701 0 0
34 "New Zealand"     702                  4         0  3423.110148755386 6.04850006103516 0 1
35 "North Macedonia" 626                  0         . 371.24665131842215                . 0 0
35 "North Macedonia" 702                  0         .  439.2124595205335                . 0 1
36 "Norway"          626                  0  .7494647  6198.203096469914 7.37470006942749 0 0
36 "Norway"          702                  1  3.016023 6450.8501830370105 7.64411020278931 0 1
37 "Serbia"          626                  0         . 452.59831049901624 4.16420984268188 0 0
37 "Serbia"          702                  2         .  531.8009351002689 3.58231997489929 0 1
38 "Singapore"       626                  0         .  4375.869575085115 3.07085990905762 0 0
38 "Singapore"       702                  4         .  5185.627684932321 2.64509010314941 0 1
39 "South Africa"    626                  0  6.035666 482.08511178250086 6.07742023468018 0 0
39 "South Africa"    702                  4  5.048543  479.0053750651366 6.15898990631104 0 1
40 "South Korea"     626                  7  2.661791 2259.3952057282318                . 0 0
40 "South Korea"     702                  2  1.118895  2636.329640607084 4.45807981491089 0 1
41 "Switzerland"     626                  0 -.9542716  7013.121028293715 4.85834980010986 0 0
41 "Switzerland"     702                  6  1.230301  7472.330934927584 4.86093997955322 0 1
end
format %tm period

What is weird is that in the results 2 Belgium is included even though it has some missing values for some CV. For example among many others Austria on the other hand is not included and I cannot figure out why.

Many thanks!

Best regards,
Dominik

Last edited by Dominik Bammer; 15 Apr 2022, 01:18.

Comment

Carlo Lazzaro

Join Date: Apr 2014

Posts: 17613
#8

15 Apr 2022, 12:22

Dominik_
if you include -educationexp- in the right-hand side of your regression equation, the first observation of "Belgium" is ruled out from the regression.
Cannot say about "Austria" (perfect collinearity with -panelid-?).
To be more precise, Stata omits all the observations with missing values in any variable included in the statistical procedure; put differently, if you do not include -educationexp- among the predictors, other things being equal bith the observation odìf the "Belgium" panel will be considered as valid.

Kind regards,
Carlo
(StataNow 18.5)
Comment
Dominik Bammer

Join Date: Apr 2022

Posts: 6
#9

15 Apr 2022, 15:13

Dear Carlo,

I am really sorry for all the questions. I do think I understand what you mean but what is weird is that 2 Belgium is in fact included in the results while 1 Austria is not.
I then tried several very basic "tests" to see where the problem is. I copied the values of Belgium to Austria to see if it works. However, Austria was still not included in the results (not omitted due to collinearity but simply deleted), Belgium was still in the results even though they had the same values. Furthermore, I once gave Austria the id 42 and it then appeared in the results (weirdly with the negative values of Belgium from the regular regression) but suddenly Belgium didn't anymore.
I really don't know how to continue.

Very grateful for further thoughts or corrections!

Best regards,
Dominik
Comment

Carlo Lazzaro

Join Date: Apr 2014
Posts: 17613

#10

16 Apr 2022, 02:36

Dominik:
now it's clear: Austria is omitted as it's the reference category for -country-.
Belgium is included because in the regression specification that you can find below, 1 out 2 observations is included (due to the lack of missing values)
That said, it's better to -encode- (caveat emptor: in other instances, well doumented in the related entry of Stata -pdf manual, -encode- may show naty behaviours) -country-, as in the following toy-example:

Code:

. encode country, gen(country_num)

. xtset country_num period

Panel variable: country_num (strongly balanced)
 Time variable: period, 2012m3 to 2018m7, but with gaps
         Delta: 1 month

. reg DVGTcirculareconomy i.treated##i.DiD1Time i.country_num i.period inflation gdp educationexp, vce(cluster country_num)
note: 40.country_num omitted because of collinearity.
note: 702.period omitted because of collinearity.

Linear regression                               Number of obs     =         50
                                                F(4, 27)          =          .
                                                Prob > F          =          .
                                                R-squared         =     0.7416
                                                Root MSE          =     7.7802

                               (Std. err. adjusted for 28 clusters in country_num)
----------------------------------------------------------------------------------
                 |               Robust
DVGTcircularec~y | Coefficient  std. err.      t    P>|t|     [95% conf. interval]
-----------------+----------------------------------------------------------------
       1.treated |    18.7093   30.65833     0.61   0.547     -44.1964      81.615
      1.DiD1Time |   2.312968    3.23626     0.71   0.481    -4.327288    8.953224
                 |
treated#DiD1Time |
            1 1  |   6.265147   6.904769     0.91   0.372    -7.902269    20.43256
                 |
     country_num |
        Belgium  |   17.00452    4.32514     3.93   0.001     8.130062    25.87897
        Czechia  |   18.16289   23.55347     0.77   0.447    -30.16484    66.49061
        Denmark  |  -6.048976    11.3333    -0.53   0.598    -29.30299    17.20504
        Estonia  |   21.95581   22.73314     0.97   0.343    -24.68875    68.60037
        Finland  |   4.651183   5.282942     0.88   0.386     -6.18852    15.49089
         France  |   5.840308   6.126262     0.95   0.349    -6.729742    18.41036
        Germany  |   5.302484   3.792752     1.40   0.173    -2.479599    13.08457
         Greece  |    17.2994   23.86366     0.72   0.475    -31.66479     66.2636
        Hungary  |   20.38516   28.75552     0.71   0.484     -38.6163    79.38661
        Ireland  |  -1.944665    8.87131    -0.22   0.828    -20.14709    16.25776
          Italy  |   15.95323   13.94222     1.14   0.263    -12.65383     44.5603
         Latvia  |    16.8447   22.89349     0.74   0.468    -30.12885    63.81826
      Lithuania  |   22.13161   25.84992     0.86   0.399    -30.90805    75.17127
     Luxembourg  |  -10.59938   35.27037    -0.30   0.766    -82.96819    61.76944
    Netherlands  |   18.54948   .5184205    35.78   0.000     17.48577    19.61319
    New Zealand  |    23.1287   28.68408     0.81   0.427    -35.72618    81.98358
         Norway  |  -2.571989     8.7774    -0.29   0.772    -20.58173    15.43775
         Poland  |   20.08412   25.62857     0.78   0.440    -32.50136     72.6696
       Portugal  |   21.77709   19.81147     1.10   0.281     -18.8727    62.42687
       Slovakia  |   21.00256    26.0265     0.81   0.427     -32.3994    74.40452
       Slovenia  |   12.46355   16.77355     0.74   0.464    -21.95293    46.88002
   South Africa  |   43.79208   60.11876     0.73   0.473    -79.56142    167.1456
    South Korea  |   31.61593   41.07231     0.77   0.448    -52.65748    115.8893
          Spain  |     23.172   16.64424     1.39   0.175    -10.97917    57.32317
         Sweden  |   -7.81151    12.7781    -0.61   0.546       -34.03    18.40698
    Switzerland  |          0  (omitted)
 United Kingdom  |  -.5559368   2.558583    -0.22   0.830    -5.805716    4.693842
                 |
      702.period |          0  (omitted)
       inflation |   .0615973   2.978891     0.02   0.984    -6.050583    6.173777
             gdp |   .0071305   .0083656     0.85   0.402    -.0100343    .0242953
    educationexp |   2.457489   4.076871     0.60   0.552    -5.907559    10.82254
           _cons |   -61.7519   75.36169    -0.82   0.420    -216.3813    92.87751
----------------------------------------------------------------------------------

.

Last edited by Carlo Lazzaro; 16 Apr 2022, 02:38.

Kind regards,
Carlo
(StataNow 18.5)

Comment

Dominik Bammer

Join Date: Apr 2022

Posts: 6
#11

19 Apr 2022, 03:05

Dear Carlo,

I know understand what went wrong and what I need to do to resolve the issue. Thank you so much for all the provided help!

Best regards,
Dominik
Comment

Announcement