P-values for Hansen and difference-in-Hansen tests!!!

[Hamza Almustafa]

Dear Stata experts,

I have got lost and confused regarding the p-values for Hansen and difference-in-Hansen tests threshold proposed by Roodman (2009, p129) "Also, because of the risks, do not take comfort in a Hansen test p-value below 0.1. View higher values, such as 0.25, as potential signs of trouble".

Researchers reports the p-value of these tests under two different arguments based on that quoted sentence, first group reports p-value under 0.25 threshold as the best estimations, which i believe its true according to Roodman statement, while others report higher values are best results!!

Thus, according to Roodman's statement which group would you support?

Thanks

[Sebastian Kripfganz]

If we take the p-value at its face value, then we would reject the null hypothesis that all overidentifying restrictions are jointly valid if the p-value is smaller than the significance level (e.g. 0.1 or 0.05).

Given that p-values tend to become inflated with an increasing number of instruments, we should not be very confident about our non-rejection if the p-value is "slightly" larger than the significance level.

The question is: How do we quantify "slightly" in this context? With a very large number of instruments, "slightly" becomes large to the point that even a p-value of 1.000 should not make us confident. As Roodman (2009, p. 98) says,

it can weaken the Hansen test to the point where it generates implausibly good p-values of 1.000

It really depends on the degree to which you have "too many" instruments. Coming back to your initial quote from Roodman (2009, p.129):

do not take comfort in a Hansen test p-value below 0.1. View higher values, such as 0.25, as potential signs of trouble.

I understand this sentence as a suggested rule of thumb which tells you that any p-value between 0.1 and 0.25 should be fine as it is reasonably larger than a significance level of 0.05 and it is reasonably small enough to not suspect severe p-value inflation.

Note that this is just my interpretation of Roodman's conclusion which does not mean that I necessarily agree with this rule of thumb. While I do agree with the underlying argumentation, these specific thresholds are very arbitrary and you should not use them as your sole justification for a particular specification.

It is more important that you can defend the choice of your instruments based on theoretical reasoning and that you take every precaution to avoid having a large number of overidentifying restrictions. If you have done so, then you should not worry about having a p-value of 0.09 on the one side or 0.3 on the other. In other words, do not take the thresholds of 0.1 and 0.25 as too serious (in particular the upper threshold) as there is no theoretical justification for exactly these two numbers.

[Hamza Almustafa]

Hello Mr Sebastian, thanks for your informative answer and helpful comments

[Rania Shahine]

Dear Sebastian,
If I get p-values for a set of System GMM regressions as follows; 0.0858, 0.0915, 0.0656. Is it safe to use the 0.05 significance level and conclude that I fail to reject the null hypothesis of the Hansen test and that my set of instruments are overidentified? I understand that exceeding the 0.1 significance level would be better, but what if I couldn't get these any higher? Would that be okay?
Thanks!

[Sebastian Kripfganz]

Dear Sebastian,
If I get p-values for a set of System GMM regressions as follows; 0.0858, 0.0915, 0.0656. Is it safe to use the 0.05 significance level and conclude that I fail to reject the null hypothesis of the Hansen test and that my set of instruments are overidentified? I understand that exceeding the 0.1 significance level would be better, but what if I couldn't get these any higher? Would that be okay?
Thanks!

Double posting, answered here:
https://www.statalist.org/forums/for...63-hansen-test