Significance at 5% or 10% for a regression output

[lal mohan kumar]

Dear Stata Members

I would like to ask a question on interpreting the significance of results based on a regression result. I did a panel regression with controls and my variable of interest is significant at 10% only. Now the first question is how do I interpret that significance? Of course, the safe way could be to interpret that the coefficient of variable of interest is significant at 10%(without using the adjectives like moderately/weakly significant as some Statisticians dont recommend such usage). However, research articles that I have come across usually interpret and extend their findings for those results were the coefficient of variable of interest is significant at 1% level. Even in the case of robustness test, I have seen that the coefficient of variable of interest remains statistically significant at 1%. Now my doubt is that if I am getting a significance at say 5% or 10% level for my baseline results, should I resist myself from extending the work and close it there? I have seen a few studies which highlighted the statistical INsignificance(also Clyde's advice:https://www.statalist.org/forums/for...-what-can-i-do), but when it comes to significance at 10% or 5%, what should one do with regard to 1)reporting the results 2)extending the working given the condition that baseline is not significant at 1%.

Any comments or thoughts related to this(or related posts) will be highly helpful

[Carlo Lazzaro]

The issue is that your last statement about taking the matter forward is conditional on having the chance to increase your sample size.
If this chance is not feasible, there's nothing you can do but live with the results you got (provided that the model is not misspecified).
As an aside, please note that 10%, 5%, 1% are arbitrary cut-offs: 5% has been selected as a sort of magic wand that split the world in significant and non significant results; by the way, it is a customary rule that has been recently questioned (see https://www.amstat.org/asa/files/pdf...eStatement.pdf).
I would rather take a look at confidence interval.
Eventually, if you end up with non-significant results they'are as informative as their sigificant counterparts and, as such, can be valuable inputs for further discussion on the data generating process that you investigated.

[Maarten Buis]

Statisticians are moving away from the concept of significance. The 10%, 5%, or 1% boundaries are completely arbitrary, so why would we label a result slightly above an arbitrary boundary "insignificant" and a result slightly below "significant"? This has led to so many problems that the advise now is to avoid the "s-word" at all cost. You can report the p-value, but don't dichotomize it in significant and insignificant. See e.g. this special issue of the American Statistician: https://www.tandfonline.com/doi/full...5.2019.1583913

[Nick Cox]

Excellent advice so far with which I agree completely. The scenarios range from bad news -- your model fit may be too weak to be publishable if that is what you hoping -- to relatively good news: your P-values just mean that you are fitting too many predictors, and a more parsimonious model might work better. But there are still too many scenarios to discuss adequately. For example, views on simplifying existing models range from it being obviously the best tactic to it being obnoxious cherry-picking, P-hacking and the like.

I should mention e.g. lasso as one way to proceed.

There is no context here to guide advice. On this question you could be an undergraduate or research assistant being told exactly what to do, or you could be a researcher working independently towards a journal paper. The advice might vary depending on the situation.

[lal mohan kumar]

Thank you all for the excellent advice. Thanks to Maarten for sharing that paper!