IV2/ivreghdfe partialing out for panel

[Giorgio Di Stefano]

Dear all,

I am running a dynamic panel with clustering with ivreghdfem which is similar to the iv2 command, with an AR(1) Driscoll-Kraay standard errors. I am getting a warning in big red letters

Warning: estimated covariance matrix of moment conditions not of full rank.
         overidentification statistic not reported, and standard errors and
         model tests should be interpreted with caution.
Possible causes:
         number of clusters insufficient to calculate robust covariance matrix
         covariance matrix of moment conditions not positive definite
         covariance matrix uses too many lags
         singleton dummy variable (dummy with one 1 and N-1 0s or vice versa)
partial option may address problem.

Indeed, I have too many categorical variables and dummies in the model, so there me a problem of overidentification and degrees of freedoms.

I just want to know how to solve with partialing out



The code I am running is

qui growthgdp l.gdp cpi u Output dummy1 dummy2 c.indicator1##c.indicator1, absorb(time panelid) vce(cluster panelid, dkraay(1))

This is just a small sample of my variables. I have too many dummies and categoricals, as said above, which I consider them all as endogenous in the model

Note that when I run the command without the

quite

option Stata solves by its own the partial problem by stating at the end

Partialled-out:       _cons
                      nb: total SS, model F and R2s are after partialling-out;
                          any small-sample adjustments include partialled-out
                          variables in regressor count K

and

Absorbed degrees of freedom:
-----------------------------------------------------+
Absorbed FE | Categories - Redundant = Num. Coefs |
-------------+---------------------------------------|
ts | 40 40 0 *|
id | 30 1 29 |
-----------------------------------------------------+
* = FE nested within cluster; treated as redundant for DoF computation

This is the display output. However, I cannot see the regression output
So

1 )How am I solving it, and do I run the partialling option?

and
2) Will I be able to run margins after that ?

Thank you for any help

Carlo Lazzaro maybe you can be of my help

[Andrew Musau]

What is the total number of observations? You need to compare the number of estimated parameters to the number of observations to see if you have enough degrees of freedom.

[Giorgio Di Stefano]

What is the total number of observations? You need to compare the number of estimated parameters to the number of observations to see if you have enough degrees of freedom.

These are the outputs I get

Estimates efficient for homoskedasticity only
Statistics robust to heteroskedasticity and clustering on ts
and kernel-robust to common correlated disturbances (Driscoll-Kraay)
  kernel=Bartlett; bandwidth=1
  time variable (t):  ts
  group variable (i): id

Number of clusters (ts) =           40                Number of obs =      578
                                                      F( 88,    39) =  1.5e+08
                                                      Prob > F      =   0.0000
Total (centered) SS     =  2033.988801                Centered R2   =   0.8995
Total (uncentered) SS   =  2033.988801                Uncentered R2 =   0.8995
Residual SS             =  204.4318054                Root MSE      =    .6659

and after stata partials out aimatically I get

c.ind_vetopoints#c.ind_vetopoints
Partialled-out: _cons
nb: total SS, model F and R2s are after partialling-out;
any small-sample adjustments include partialled-out
variables in regressor count K

and

Absorbed degrees of freedom:
-----------------------------------------------------+
 Absorbed FE | Categories  - Redundant  = Num. Coefs |
-------------+---------------------------------------|
          ts |        40          40           0    *|
          id |        30           1          29     |
-----------------------------------------------------+
* = FE nested within cluster; treated as redundant for DoF computation

Many of may key variables and their combinations are dropped out due to collinearity. .I understand that I have a problem of over identification here and weak instruments, if I am not wrong. Nonetheless, the command does not have a test option for weak instrument. It is similar to iv2 , according to the author. ,Sergio Correa. Any ideas on how to move on? Could that be first testeed with iv2 and then move on?

[Andrew Musau]

When you cluster, the effective number of observations is the number of clusters (as observations are not independent within clusters). So the estimated covariance matrix of moment conditions is not of full rank as the number of estimated parameters is greater than the number of clusters. So work with a model which has less than 40 estimated parameters. If you have no use for coefficients on the dummies, absorb them.

... growthgdp l.gdp cpi u Output , absorb(time panelid dummy1 dummy2 c.indicator1##c.indicator1 ) vce(cluster panelid, dkraay(1))

[Giorgio Di Stefano]

When you cluster, the effective number of observations is the number of clusters (as observations are not independent within clusters). So the estimated covariance matrix of moment conditions is not of full rank as the number of estimated parameters is greater than the number of clusters. So work with a model which has less than 40 estimated parameters. If you have no use for coefficients on the dummies, absorb them.

... growthgdp l.gdp cpi u Output , absorb(time panelid dummy1 dummy2 c.indicator1##c.indicator1 ) vce(cluster panelid, dkraay(1))

Now I see. Thanks, Andrew. Actually, I run a cross validation lasso approach to reduce the number of variables. The quadratic relation

 
 c.indicator1##c.indicator1

is the cause for the too many variables. From the quadratic relation I get

c. |
ind_weighted_im~1#|
c. |
ind_weighted_~n_1 |
|
c. |
ind_weighted_~r_0#|
c. |
ind_weighted_~r_0


How can I generate /include them as separated variables in the model?

[Andrew Musau]

The quadratic relation

c.indicator1##c.indicator1

is the cause for the too many variables. From the quadratic relation I get

c. |
ind_weighted_im~1#|
c. |
ind_weighted_~n_1 |
|
c. |
ind_weighted_~r_0#|
c. |
ind_weighted_~r_0


How can I generate /include them as separated variables in the model?

The interaction of a continuous variable with itself defines two variables, the variable and its quadratic term. Note that multiplying a 0/1 indicator by itself results in the same variable. It should be the other categorical variables that result in many indicators.

[Giorgio Di Stefano]

The interaction of a continuous variable with itself defines two variables, the variable and its quadratic term. Note that multiplying a 0/1 indicator by itself results in the same variable. It should be the other categorical variables that result in many indicators.

Ι have broken down all categorical variables in single dummies. Most of the variables that are dropped out as collinears are from the intercaction the between a continuous variables and its quadratic term. How am I solving this?

[Andrew Musau]

I would go back and look at the specification. Why do you need that many interactions if some of them are redundant? Start with a parsimonious model. Below, using ivreghdfe from SSC, I will illustrate how either you can absorb indicators that are controls in your model or partial them out. Without doing this, I replicate your problem.

webuse nlswork, clear
set seed 06172022
replace year= runiformint(1, 50)
replace id = runiformint(1,2000)
bys id year: keep if _n==1
xtset id year

*REPLICATES PROBLEM
ivreghdfe ln_w age c.age#c.age not_smsa i.year (tenure = union south), absorb(idcode) dkraay(1)

*SOLUTION 1 - ABSORB THE YEAR DUMMIES
ivreghdfe ln_w age c.age#c.age not_smsa (tenure = union south), absorb(idcode year) dkraay(1)

*SOLUTION 2 - PARTIAL OUT THE YEAR DUMMIES
ivreghdfe ln_w age c.age#c.age not_smsa i.year (tenure = union south), absorb(idcode) dkraay(1) partial(i.year)

Res.:

. *REPLICATES PROBLEM

.
. ivreghdfe ln_w age c.age#c.age not_smsa i.year (tenure = union south), absorb(idcode) dkraay(1)
Note: kernel=Bartlett and bw=1 implies zero lags used.  Standard errors and
      test statistics are not autocorrelation-consistent.
(dropped 2 singleton observations)
Warning: time variable year has 11803 gap(s) in relevant range
(MWFE estimator converged in 1 iterations)

IV (2SLS) estimation
--------------------

Estimates efficient for homoskedasticity only
Statistics robust to heteroskedasticity and clustering on year
and kernel-robust to common correlated disturbances (Driscoll-Kraay)
  kernel=Bartlett; bandwidth=1
  time variable (t):  year
  group variable (i): idcode

Number of clusters (year) =         50                Number of obs =    16456
                                                      F( 53,    49) =   207.48
                                                      Prob > F      =   0.0000
Total (centered) SS     =  3159.431288                Centered R2   =  -0.6498
Total (uncentered) SS   =  3159.431288                Uncentered R2 =  -0.6498
Residual SS             =  5212.504764                Root MSE      =    .6015

------------------------------------------------------------------------------
             |               Robust
     ln_wage |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
      tenure |   .1477497   .0092637    15.95   0.000     .1291335    .1663659
         age |    .019787   .0062524     3.16   0.003     .0072223    .0323517
             |
 c.age#c.age |  -.0006372   .0000956    -6.67   0.000    -.0008292   -.0004451
             |
    not_smsa |  -.2471012   .0132605   -18.63   0.000    -.2737491   -.2204533
             |
        year |
          1  |          0  (empty)
          2  |   .0765209   .0189027     4.05   0.000     .0385346    .1145073
          3  |  -.0267462   .0147152    -1.82   0.075    -.0563175    .0028251
          4  |   .0499182   .0168675     2.96   0.005     .0160217    .0838147
          5  |  -.0228926   .0158287    -1.45   0.154    -.0547017    .0089164
          6  |   .0919789   .0160498     5.73   0.000     .0597257    .1242321
          7  |   .0626689   .0166113     3.77   0.000     .0292872    .0960505
          8  |   .0479868   .0146815     3.27   0.002     .0184832    .0774904
          9  |   .0440458   .0125645     3.51   0.001     .0187965    .0692952
         10  |   .0561821    .014261     3.94   0.000     .0275237    .0848406
         11  |   .1083785   .0141256     7.67   0.000     .0799921    .1367649
         12  |  -.0391415   .0152323    -2.57   0.013    -.0697519    -.008531
         13  |   .1309383   .0141032     9.28   0.000     .1025968    .1592797
         14  |    .050156   .0164273     3.05   0.004     .0171442    .0831679
         15  |   .0878776   .0162405     5.41   0.000      .055241    .1205141
         16  |   .0372372   .0137872     2.70   0.009     .0095307    .0649436
         17  |   .0265132   .0155492     1.71   0.095    -.0047341    .0577606
         18  |   .0416571   .0173223     2.40   0.020     .0068466    .0764676
         19  |   .0596874   .0156622     3.81   0.000      .028213    .0911619
         20  |  -.0136783   .0168761    -0.81   0.422    -.0475921    .0202355
         21  |   .0153517   .0161908     0.95   0.348     -.017185    .0478884
         22  |   .0634344   .0186086     3.41   0.001      .026039    .1008298
         23  |   .0600096   .0149255     4.02   0.000     .0300157    .0900035
         24  |   .0109941   .0142447     0.77   0.444    -.0176318    .0396199
         25  |   .0532859   .0158377     3.36   0.001     .0214589    .0851128
         26  |  -.0149129   .0161928    -0.92   0.362    -.0474536    .0176278
         27  |   .0502523   .0153842     3.27   0.002     .0193365     .081168
         28  |   .1001255   .0157426     6.36   0.000     .0684895    .1317614
         29  |   .0075959   .0129407     0.59   0.560    -.0184093    .0336012
         30  |   .0629403   .0165918     3.79   0.000     .0295979    .0962828
         31  |   .0362259    .013759     2.63   0.011      .008576    .0638757
         32  |   .0339586   .0152742     2.22   0.031     .0032641    .0646532
         33  |   .0003327    .014548     0.02   0.982    -.0289025    .0295679
         34  |   .0705133   .0132382     5.33   0.000     .0439102    .0971165
         35  |   .0236562   .0171444     1.38   0.174    -.0107968    .0581091
         36  |    .011931    .016234     0.73   0.466    -.0206925    .0445545
         37  |   .0242023    .015233     1.59   0.119    -.0064096    .0548143
         38  |   .0220125   .0129075     1.71   0.094    -.0039261    .0479512
         39  |   .0512212   .0140947     3.63   0.001     .0228968    .0795457
         40  |   .0240988   .0156945     1.54   0.131    -.0074405    .0556381
         41  |   .0239378   .0157256     1.52   0.134    -.0076639    .0555396
         42  |   .0431318   .0154143     2.80   0.007     .0121557     .074108
         43  |   .0618076   .0177229     3.49   0.001     .0261921     .097423
         44  |   .0897484   .0139248     6.45   0.000     .0617655    .1177314
         45  |   .0465936   .0194055     2.40   0.020     .0075969    .0855904
         46  |   .0443533   .0131111     3.38   0.001     .0180056    .0707009
         47  |   .0253242   .0132789     1.91   0.062    -.0013608    .0520091
         48  |   .0642028   .0173008     3.71   0.001     .0294354    .0989701
         49  |   .0257316   .0180245     1.43   0.160      -.01049    .0619533
         50  |   .0402937   .0127254     3.17   0.003     .0147209    .0658664
------------------------------------------------------------------------------
Underidentification test (Kleibergen-Paap rk LM statistic):             44.928
                                                   Chi-sq(2) P-val =    0.0000
------------------------------------------------------------------------------
Weak identification test (Cragg-Donald Wald F statistic):              200.555
                         (Kleibergen-Paap rk Wald F statistic):        207.416
Stock-Yogo weak ID test critical values: 10% maximal IV size             19.93
                                         15% maximal IV size             11.59
                                         20% maximal IV size              8.75
                                         25% maximal IV size              7.25
Source: Stock-Yogo (2005).  Reproduced by permission.
NB: Critical values are for Cragg-Donald F statistic and i.i.d. errors.
------------------------------------------------------------------------------
Warning: estimated covariance matrix of moment conditions not of full rank.
         overidentification statistic not reported, and standard errors and
         model tests should be interpreted with caution.
Possible causes:
         number of clusters insufficient to calculate robust covariance matrix
         covariance matrix of moment conditions not positive definite
         covariance matrix uses too many lags
         singleton dummy variable (dummy with one 1 and N-1 0s or vice versa)
partial option may address problem.
------------------------------------------------------------------------------
Instrumented:         tenure
Included instruments: age c.age#c.age not_smsa 2.year 3.year 4.year 5.year
                      6.year 7.year 8.year 9.year 10.year 11.year 12.year
                      13.year 14.year 15.year 16.year 17.year 18.year 19.year
                      20.year 21.year 22.year 23.year 24.year 25.year 26.year
                      27.year 28.year 29.year 30.year 31.year 32.year 33.year
                      34.year 35.year 36.year 37.year 38.year 39.year 40.year
                      41.year 42.year 43.year 44.year 45.year 46.year 47.year
                      48.year 49.year 50.year
Excluded instruments: union south
Partialled-out:       _cons
                      nb: total SS, model F and R2s are after partialling-out;
                          any small-sample adjustments include partialled-out
                          variables in regressor count K
------------------------------------------------------------------------------

Absorbed degrees of freedom:
-----------------------------------------------------+
 Absorbed FE | Categories  - Redundant  = Num. Coefs |
-------------+---------------------------------------|
      idcode |      1998           0        1998     |
-----------------------------------------------------+

.
.
.
. *SOLUTION 1 - ABSORB THE YEAR DUMMIES

.
. ivreghdfe ln_w age c.age#c.age not_smsa (tenure = union south), absorb(idcode year) dkraay(1)
Note: kernel=Bartlett and bw=1 implies zero lags used.  Standard errors and
      test statistics are not autocorrelation-consistent.
(dropped 2 singleton observations)
Warning: time variable year has 11803 gap(s) in relevant range
(MWFE estimator converged in 6 iterations)

IV (2SLS) estimation
--------------------

Estimates efficient for homoskedasticity only
Statistics robust to heteroskedasticity and clustering on year
and kernel-robust to common correlated disturbances (Driscoll-Kraay)
  kernel=Bartlett; bandwidth=1
  time variable (t):  year
  group variable (i): idcode

Number of clusters (year) =         50                Number of obs =    16456
                                                      F(  4,    49) =   256.19
                                                      Prob > F      =   0.0000
Total (centered) SS     =  3144.295236                Centered R2   =  -0.6578
Total (uncentered) SS   =  3144.295236                Uncentered R2 =  -0.6578
Residual SS             =  5212.504764                Root MSE      =    .6005

------------------------------------------------------------------------------
             |               Robust
     ln_wage |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
      tenure |   .1477497    .009248    15.98   0.000     .1291651    .1663343
         age |    .019787   .0062418     3.17   0.003     .0072436    .0323304
             |
 c.age#c.age |  -.0006372   .0000954    -6.68   0.000    -.0008289   -.0004455
             |
    not_smsa |  -.2471012    .013238   -18.67   0.000    -.2737039   -.2204985
------------------------------------------------------------------------------
Underidentification test (Kleibergen-Paap rk LM statistic):             44.928
                                                   Chi-sq(2) P-val =    0.0000
------------------------------------------------------------------------------
Weak identification test (Cragg-Donald Wald F statistic):              201.237
                         (Kleibergen-Paap rk Wald F statistic):        208.122
Stock-Yogo weak ID test critical values: 10% maximal IV size             19.93
                                         15% maximal IV size             11.59
                                         20% maximal IV size              8.75
                                         25% maximal IV size              7.25
Source: Stock-Yogo (2005).  Reproduced by permission.
NB: Critical values are for Cragg-Donald F statistic and i.i.d. errors.
------------------------------------------------------------------------------
Hansen J statistic (overidentification test of all instruments):        39.074
                                                   Chi-sq(1) P-val =    0.0000
------------------------------------------------------------------------------
Instrumented:         tenure
Included instruments: age c.age#c.age not_smsa
Excluded instruments: union south
Partialled-out:       _cons
                      nb: total SS, model F and R2s are after partialling-out;
                          any small-sample adjustments include partialled-out
                          variables in regressor count K
------------------------------------------------------------------------------

Absorbed degrees of freedom:
-----------------------------------------------------+
 Absorbed FE | Categories  - Redundant  = Num. Coefs |
-------------+---------------------------------------|
      idcode |      1998           0        1998     |
        year |        50          50           0    *|
-----------------------------------------------------+
* = FE nested within cluster; treated as redundant for DoF computation

.
.
.
. *SOLUTION 2 - PARTIAL OUT THE YEAR DUMMIES

.
. ivreghdfe ln_w age c.age#c.age not_smsa i.year (tenure = union south), absorb(idcode) dkraay(1) partial(i.year)
Note: kernel=Bartlett and bw=1 implies zero lags used.  Standard errors and
      test statistics are not autocorrelation-consistent.
(dropped 2 singleton observations)
Warning: time variable year has 11803 gap(s) in relevant range
(MWFE estimator converged in 1 iterations)

IV (2SLS) estimation
--------------------

Estimates efficient for homoskedasticity only
Statistics robust to heteroskedasticity and clustering on year
and kernel-robust to common correlated disturbances (Driscoll-Kraay)
  kernel=Bartlett; bandwidth=1
  time variable (t):  year
  group variable (i): idcode

Number of clusters (year) =         50                Number of obs =    16456
                                                      F(  4,    49) =   255.32
                                                      Prob > F      =   0.0000
Total (centered) SS     =  3144.295236                Centered R2   =  -0.6578
Total (uncentered) SS   =  3144.295236                Uncentered R2 =  -0.6578
Residual SS             =  5212.504764                Root MSE      =    .6015

------------------------------------------------------------------------------
             |               Robust
     ln_wage |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
      tenure |   .1477497   .0092637    15.95   0.000     .1291335    .1663659
         age |    .019787   .0062524     3.16   0.003     .0072223    .0323517
             |
 c.age#c.age |  -.0006372   .0000956    -6.67   0.000    -.0008292   -.0004451
             |
    not_smsa |  -.2471012   .0132605   -18.63   0.000    -.2737491   -.2204533
------------------------------------------------------------------------------
Underidentification test (Kleibergen-Paap rk LM statistic):             44.928
                                                   Chi-sq(2) P-val =    0.0000
------------------------------------------------------------------------------
Weak identification test (Cragg-Donald Wald F statistic):              200.555
                         (Kleibergen-Paap rk Wald F statistic):        207.416
Stock-Yogo weak ID test critical values: 10% maximal IV size             19.93
                                         15% maximal IV size             11.59
                                         20% maximal IV size              8.75
                                         25% maximal IV size              7.25
Source: Stock-Yogo (2005).  Reproduced by permission.
NB: Critical values are for Cragg-Donald F statistic and i.i.d. errors.
------------------------------------------------------------------------------
Hansen J statistic (overidentification test of all instruments):        39.074
                                                   Chi-sq(1) P-val =    0.0000
------------------------------------------------------------------------------
Instrumented:         tenure
Included instruments: age c.age#c.age not_smsa
Excluded instruments: union south
Partialled-out:       2.year 3.year 4.year 5.year 6.year 7.year 8.year 9.year
                      10.year 11.year 12.year 13.year 14.year 15.year 16.year
                      17.year 18.year 19.year 20.year 21.year 22.year 23.year
                      24.year 25.year 26.year 27.year 28.year 29.year 30.year
                      31.year 32.year 33.year 34.year 35.year 36.year 37.year
                      38.year 39.year 40.year 41.year 42.year 43.year 44.year
                      45.year 46.year 47.year 48.year 49.year 50.year _cons
                      nb: total SS, model F and R2s are after partialling-out;
                          any small-sample adjustments include partialled-out
                          variables in regressor count K
------------------------------------------------------------------------------

Absorbed degrees of freedom:
-----------------------------------------------------+
 Absorbed FE | Categories  - Redundant  = Num. Coefs |
-------------+---------------------------------------|
      idcode |      1998           0        1998     |
-----------------------------------------------------+

[Giorgio Di Stefano]

Deleted. Please see #10

[Giorgio Di Stefano]

Now I what you mean I was using absorbing by panel ID and time and clustering by time at the same time

absorb(ts id) cluster (ts)  dkraay(1)

The code I am now using is


[CODE]ivreghdfe growthgdp l.gdp cpi u Output , absorb(time panelid dummy1 dummy2 c.indicator1##c.indicator1, iyear i.ts , absorb(id) dkraay(1) partial(i.ts). [CODE] One observation is that I do not see the output for weak instrument test and over identification . Still when I add the qui ivreghdfe does not run

Secondly, how do I get rid of the redundant relation from the quadratic.

And last, I do not get the syntax in your code
(tenure = union south). Are those the instruments?

[Andrew Musau]

You have a number of syntax errors and other fundamental issues that I am not willing to devote any more time to address here. Before I bail out of this thread, here are some recommendations:

1. Review the documentation of ivreghdfe and ivreg2 to see what the correct syntax is.

help ivreghdfe
help ivreg2

2. -quietly- suppresses the output of a command. See

help qui

3. If var1 is a 0/1 indicator, then var1*var1 = var1 because 0*0=0 and 1*1=1.

4. Specifying Driscoll-Kraay standard errors automaically clusters on the time dimension given that you have properly xtset your data.

5. Once you are clear with the above points, reread my replies above.

[Giorgio Di Stefano]

Thank you so much Andrew. I have now figured out how doing that. Thank you so much for your time and inputs!