ivreg2 with partial option

[Teresa Fuchs]

Dear all,

I’m doing a IV regression with the endog(varlist_en) option and cluster at country level.

However, the overidentification statistic and the Durbin-Wu-Hausman statistic can’t be reported because

Warning: estimated covariance matrix of moment conditions not of full rank.
overidentification statistic not reported, and standard errors and
model tests should be interpreted with caution.
Possible causes:
number of clusters insufficient to calculate robust covariance matrix
singleton dummy variable (dummy with one 1 and N-1 0s or vice versa)
partial option may address problem

Therefore, I tried the partial option to solve this problem.

xi: ivreg2 lnGermanFDIs lnIPRP lnInfrastructureIndex lnCapitalLaborRatios lnQualityofPublicSchools lnOrganizedCrimeIndex lnLawmakinginstitutionquality lnDistancekm Commonlanguage i.Country i.Year i.Industry (lnGDP lnEnvPolicyIndex lnTariffRate = Tractorsagriculturalworker Landagriculturalworker Regionalcapitallaborratios RegionalOrganizedCrime Regionalpublicschoolquality Regionalinfrastructurequality Regionaltractorsagriculturalwo Regionallandagriculturalworker), partial(lnIPRP lnInfrastructureIndex lnCapitalLaborRatios lnQualityofPublicSchools lnOrganizedCrimeIndex lnLawmakinginstitutionquality lnDistancekm Commonlanguage) robust cluster(Countryj) endog(lnGDP lnEnvPolicyIndex lnTariffRate)

However, this does not change my problem, the error message is still the same.
I also tried partial(_cons) as suggested by help ivreg2, but with the same result.

[Mark Schaffer]

Try partialling out the factor variables as well: i.Country, i.Year, i.Industry. To do this you'll also need to switch to the modern Stata syntax, i.e., drop the xi: prefix.

[Teresa Fuchs]

Thank you, Mark.
If I add i.Year i.Country i.Industry, the error message disappears and I get the Hansen J statistic and the endogeneity test.

However, I don’t really understand what happens if I use the partial option. If I look at my regression tables, the exogenous variables are partialled out and disappear from the regression tables. However, the estimates of my main variable of interest (Environmental Policy) remain the same as if I don’t use the partial option. So what happens to my exogenous variables?

[Mark Schaffer]

The exogenous regressors ha've been partialled out. There's a brief discussion in the ivreg2 help file. It's discussed in detail in various econometrics and statistics textbooks. If you google around for "Frisch-Waugh-LovellTheorem" you'll find sources that discuss it. Angrist & Pischke's Mostly Harmless Econometrics is another source (they call it "regression anatomy"). Filoso's reganat program for Stata and the corresponding SJ article (13:1) is another useful source.

[Teresa Fuchs]

Thank you for the references.
As far as I understand the Frisch-Waugh-Lovell Theorem, the estimators as well as the residuals remain the same, if we transform the regression by the partial option.
In my thesis I want to display the regression with the control variables included (public school quality etc.). However, if I do it like this, the overidentification and DWH statistics are not displayed. If I use the partial option, I get these statistics, but not the estimators of the control variables.
Is it possible to “merge” the tables, meaning that I use the results from the tables without the partial option and use the statistics from the regressions with partial option, with a comment on it?

Or is this unusual or even wrong?

[Mark Schaffer]

So long as you are using IV or 2-step FEGMM, the coefficient estimates and J stats will be the same whether or not you partial out. (It's not true with CUE GMM but you're not using that.)

I think the way you are presenting it is fine. Maybe a better description for footnote (a) would be to add that you are using FWL to partial out the exogenous regressors.

The reference for partialling out exogenous regressors in the IV setting is David E.A. Giles (1984), "Instrumental variables regressions involving seasonal data", Economics Letters, Volume 14, Issue 4, 1984, Pages 339-343.

Partialling-out exogenous regressors is fairly common in theory papers on IV and GMM because it simplifies the presentation - you don't have to carry around the exogenous regressors everywhere. An alternative would be to refer to one of these theory papers. For example, in Hahn-Hausman (2002), "Notes on bias in estimators for simultaneous equation models", in Economics Letters, vol. 75, p. 237 the quote is

"This model specification accounts for other RHS predetermined (or exogenous) variables, which have been ‘partialled out’ of the specification."

I'm sure you can find other examples if you look around.

[Teresa Fuchs]

I used the partial option for all my regressions to get the DWH and Hansen J statistic. It worked perfectly, but now I am doing a number of robustness tests and for 2 specifications, the estimates for environmental policy are not the same if I use the robust option and if not.
These are the codes:

ivreg2 lnGermanFDIs lnInfrastructureIndex lnQualityofPublicSchools lnCapitalLaborRatios lnOrganizedCrimeIndex lnLawmakinginstitutionquality lnDistancekm Commonlanguage lnHiddenTradeBarriers lnClusteringIndex lnHiringandFiringPractices lnBusinesscostsofcorruptionNew Commoncurrency lnOpennesstotrade lnWagesinmanufacturingNew  i.Country i.Year i.Industry (lnGDP lnEnvPolicyIndex lnIPRP = Tractorsagriculturalworker Landagriculturalworker Regionalcapitallaborratios RegionalOrganizedCrime Regionalpublicschoolquality Regionalinfrastructurequality Regionaltractorsagriculturalwo Regionallandagriculturalworker), robust cluster(Countryj) endog(lnGDP lnEnvPolicyIndex lnIPRP)

ivreg2 lnGermanFDIs lnInfrastructureIndex lnQualityofPublicSchools lnCapitalLaborRatios lnOrganizedCrimeIndex lnLawmakinginstitutionquality lnDistancekm Commonlanguage lnHiddenTradeBarriers lnClusteringIndex lnHiringandFiringPractices lnBusinesscostsofcorruptionNew Commoncurrency lnOpennesstotrade lnWagesinmanufacturingNew  i.Country i.Year i.Industry (lnGDP lnEnvPolicyIndex lnIPRP = Tractorsagriculturalworker Landagriculturalworker Regionalcapitallaborratios RegionalOrganizedCrime Regionalpublicschoolquality Regionalinfrastructurequality Regionaltractorsagriculturalwo Regionallandagriculturalworker), robust partial(lnInfrastructureIndex lnQualityofPublicSchools lnCapitalLaborRatios lnOrganizedCrimeIndex lnLawmakinginstitutionquality lnDistancekm Commonlanguage lnHiddenTradeBarriers lnClusteringIndex lnHiringandFiringPractices lnBusinesscostsofcorruptionNew Commoncurrency lnOpennesstotrade lnWagesinmanufacturingNew  i.Country i.Year i.Industry) cluster(Countryj) endog(lnGDP lnEnvPolicyIndex lnIPRP)

and

ivreg2 lnGermanFDIs lnInfrastructureIndex lnQualityofPublicSchools lnCapitalLaborRatios lnLawmakinginstitutionquality lnOrganizedCrimeIndex lnDistancekm Commonlanguage lnHiddenTradeBarriers lnClusteringIndex lnHiringandFiringPractices lnBusinesscostsofcorruptionNew Commoncurrency lnOpennesstotrade lnWagesinmanufacturingNew  i.Country i.Year i.Industry (lnGDP lnStringency lnClarity lnIPRP = Tractorsagriculturalworker Landagriculturalworker Regionalcapitallaborratios RegionalOrganizedCrime Regionalpublicschoolquality Regionalinfrastructurequality Regionaltractorsagriculturalwo Regionallandagriculturalworker), robust cluster(Countryj) endog(lnGDP lnStringency lnClarity lnIPRP)

ivreg2 lnGermanFDIs lnInfrastructureIndex lnQualityofPublicSchools lnCapitalLaborRatios lnLawmakinginstitutionquality lnOrganizedCrimeIndex lnDistancekm Commonlanguage lnHiddenTradeBarriers lnClusteringIndex lnHiringandFiringPractices lnBusinesscostsofcorruptionNew Commoncurrency lnOpennesstotrade lnWagesinmanufacturingNew  i.Country i.Year i.Industry (lnGDP lnStringency lnClarity lnIPRP = Tractorsagriculturalworker Landagriculturalworker Regionalcapitallaborratios RegionalOrganizedCrime Regionalpublicschoolquality Regionalinfrastructurequality Regionaltractorsagriculturalwo Regionallandagriculturalworker), robust partial(lnInfrastructureIndex lnQualityofPublicSchools lnCapitalLaborRatios lnLawmakinginstitutionquality lnOrganizedCrimeIndex lnDistancekm Commonlanguage lnHiddenTradeBarriers lnClusteringIndex lnHiringandFiringPractices lnBusinesscostsofcorruptionNew Commoncurrency lnOpennesstotrade lnWagesinmanufacturingNew  i.Country i.Year i.Industry) cluster(Countryj) endog(lnGDP lnStringency lnClarity lnIPRP)

Is there any explanations why this happens?

[Mark Schaffer]

Hard to say without seeing the output. Do you mean that in most specifications the coefficient estimates for all 3 endogenous regressors are the same whether or not you partial out, but for these two estimations only the coefficients on the endogenous regressors change? The only thing I can think of is that collinearities mean that some endogenous regressors are being reclassified as exogenous, and which ones this happens to depends on whether there has been partialling-out, but that's just a guess. Can you show us the output?

[Teresa Fuchs]

Exactly that’s the point.

The output tables:

ivreg2 lnGermanFDIs lnInfrastructureIndex lnQualityofPublicSchools lnCapitalLaborRatios lnOrganizedCrimeIndex lnLawmakinginstitutionquality lnDistancekm Commonlanguage lnHiddenTradeBarriers lnClusteringIndex lnHiringandFiringPractices lnBusinesscostsofcorruptionNew Commoncurrency lnOpennesstotrade lnWagesinmanufacturingNew  i.Country i.Year i.Industry (lnGDP lnEnvPolicyIndex lnIPRP = Tractorsagriculturalworker Landagriculturalworker Regionalcapitallaborratios RegionalOrganizedCrime Regionalpublicschoolquality Regionalinfrastructurequality Regionaltractorsagriculturalwo Regionallandagriculturalworker), robust cluster(Countryj) endog(lnGDP lnEnvPolicyIndex lnIPRP)

ivreg2 lnGermanFDIs lnInfrastructureIndex lnQualityofPublicSchools lnCapitalLaborRatios lnOrganizedCrimeIndex lnLawmakinginstitutionquality lnDistancekm Commonlanguage lnHiddenTradeBarriers lnClusteringIndex lnHiringandFiringPractices lnBusinesscostsofcorruptionNew Commoncurrency lnOpennesstotrade lnWagesinmanufacturingNew  i.Country i.Year i.Industry (lnGDP lnEnvPolicyIndex lnIPRP = Tractorsagriculturalworker Landagriculturalworker Regionalcapitallaborratios RegionalOrganizedCrime Regionalpublicschoolquality Regionalinfrastructurequality Regionaltractorsagriculturalwo Regionallandagriculturalworker), robust partial(lnInfrastructureIndex lnQualityofPublicSchools lnCapitalLaborRatios lnOrganizedCrimeIndex lnLawmakinginstitutionquality lnDistancekm Commonlanguage lnHiddenTradeBarriers lnClusteringIndex lnHiringandFiringPractices lnBusinesscostsofcorruptionNew Commoncurrency lnOpennesstotrade lnWagesinmanufacturingNew  i.Country i.Year i.Industry) cluster(Countryj) endog(lnGDP lnEnvPolicyIndex lnIPRP)

[Mark Schaffer]

That's odd. Can you contact me off-list with a copy of the dataset and I will try to reproduce this behaviour?

[Teresa Fuchs]

Thank you, I’ve just sent you my data set as a private message

[Mark Schaffer]

I've looked into it, and the source seems to be some badly conditioned X'X and Z'Z matrices. In the estimation without partialling out, the condition number for X'X is 2x10^11 and for Z'Z is 2x10^19, which is very high. The condition numbers are saved in the e(condxx) and e(condzz) macros.

As soon as you partial out even just the constant, ivreg2 can cope with it and the estimation results become stable. I cross-checked against ivregress and old ivreg and depending on the estimation equation and whether the constant is automatically included or included by hand, I can get them to report rather different results for the same estimation equation, and I think for the same reason (badly conditioned X'X and Z'Z).

[Teresa Fuchs]

So you told me that by partiallling out, the the X'X and Z'Z matrices become smaller and easier to deal with and my results become easier to deal with and the regression results are more likely to be right. Is there any literature on this? I can’t find anything related to my problem in the standard econometric text books.

I do this regression to check robustness of my basic results by adding additional variables. I am now unsure how to proceed. I have 2 options:
(1) Use the partial option to get more reliable results of my main variables of interest. The environmental policy variable, the one I am interested in most, is negative and significant. This confirms my results. But then I can make no statement about the additional variables.
(2) Do not use partial or only partial out the constant, then the policy variable is no longer significant.
I think the first option is the better one as my results of the main explanatory variable are more reliable?

[Mark Schaffer]

There's literature around on condition numbers, ill-conditioned matrices, etc. that some googling should reveal. For Stata. the programs coldiag and coldiag2 by Harkness and Hendrickx respectively might be directly useful and/or have some literature references.

I agree, you should use option #1. Partialling out just the country dummies should help a lot. You can supplement this by checking with Stata's official ivregress and the old Stata ivreg estimation routines. The latter doesn't support factor variables so you would need to use the xi prefix. And if you xtset the data so that country is the panel variable, you can also cross-check using Stata's official xtivreg as well as xtivreg2.

[Teresa Fuchs]

All right, so just to get it right: my regressions with and without the partial option lead to different estimates in this special case because of badly conditioned X'X and Z'Z matrices. As the X'X and Z'Z matrices become smaller and easier to deal with, the regression results are more likely to be "right" as a result. Therefore, I use the partial option in this special case as main results of my robustness test.
To keep the estimates of my explanatory variables, that I added to test robustness of my results, I only partial out the country fixed effects.

ivreg2 lnGermanFDIs lnInfrastructureIndex lnQualityofPublicSchools lnCapitalLaborRatios lnOrganizedCrimeIndex lnLawmakinginstitutionquality lnDistancekm Commonlanguage lnHiddenTradeBarriers lnClusteringIndex lnHiringandFiringPractices lnBusinesscostsofcorruptionNew Commoncurrency lnOpennesstotrade lnWagesinmanufacturingNew  i.Country i.Year i.Industry (lnGDP lnEnvPolicyIndex lnIPRP = Tractorsagriculturalworker Landagriculturalworker Regionalcapitallaborratios RegionalOrganizedCrime Regionalpublicschoolquality Regionalinfrastructurequality Regionaltractorsagriculturalwo Regionallandagriculturalworker), robust partial(i.Country) cluster(Countryj) endog(lnGDP lnEnvPolicyIndex lnIPRP)

But what is the argument for partialling out i.Country? I also tried i.Year and i.Industry, whereat the results remain the same.