[Joro Kolev]

The answers to such general question are very tribal, and different tribes see it differently.

The view of economists and econometricians is that you should interpret the magnitude of your estimate, whether or not your estimate is "significant" (more on "significant" below). Statistically "insignificant" elephant might be very interesting, and highly statistically significant mouse might not be that interesting.

Good economists and econometricians have a notion of what the orders of magnitude of what they are estimating are. Just signs of effects are not that interesting whether they are significant or not, without the magnitudes being large enough to merit interest.

The question you are raising is more general than you suggest. There is not one "significance", coefficients are significant at certain level. This is best seen by considering the confidence interval -- the more you raise the significance level, the wider the confidence interval becomes. Whether your confidence interval falls on both sides of the 0 or not, depends on the significance level you have set.

[lal mohan kumar]

Thanks Nick Cox and Joro Kolev. I will revisit this thread once again once I read some recommended readings

[Christopher Bratt]

I strongly recommend reading at least the comment in Nature. It's already listed above among several relevant references, but if one wants only a single and easy-to-read article, then this would be my recommendation:

https://www.nature.com/articles/d41586-019-00857-9

But then, the criticism of "statistical significance" is far from new. I remember enjoying this one, ages ago, published in 1994 by Jacob Cohen. It's an enjoyable article, well worth the time it takes to read it:

http://www.iro.umontreal.ca/~dift391...h_is_round.pdf

Cohen's article starts with the following sentence (in 1994!):

After 4 decades of severe criticism, the ritual of null hypothesis significance testing-mechanical dichotomous decisions around a sacred .05 criterion still persists.

One of the most common, and most damaging, misunderstandings is that a p > .05 suggests support to the null hypothesis (or no effect). Even a p = .55 does not indicate support to the null (or nil) hypothesis. In most sciences, the null hypothesis meaning no association is probably always wrong. More than anything else, statistical significance reflects sample size. Whatever one chooses to do, I would always consider the point estimate in addition to estimates of uncertainty. The p-value does not identify an effect or no effect, it tells us something about the uncertainty associated with an estimated association. But if the effect size is close to zero, why bother at all with significance?

Exploratory dropping parameters because p > .05 rather than effect sizes comes at a cost. For instance, there is little reason to believe that p = .15 in a small sample suggests that p will be > .05 in a new sample. And exploratory dropping parameters from a model because of the p-value may also bias other parts of the model. (A better solution is a priori to formulate alternative models based on theory, and select the one that explains the data as well as other models, but with fewer parameters.)

I would not use "statistical significance", but I would use p-values. Or, much better, I would use confidence intervals, and I would read them as compatibility intervals: Which estimates are compatible with your data, given your model?

But... with a given confidence interval (or compatibility interval): Which 'effect' is most compatible with your data? The point estimate (and values around it)! I believe many advocates of statistical significance decide to disregard this. Again, the p-value tells us something about uncertainty, not much more. It does not tell us anything about what effect size is most likely.

A point made by the comment in Nature: We should acknowledge and embrace uncertainty and not pretend to have yes/no answers.

Personally, I tend to check if confidence intervals are non-overlapping. That of course, is a much more conservative test than p < .05. It is also a much more intuitive and meaningful test, but I cannot dismiss differences because confidence intervals overlap somewhat. Anyway, using statistical significance as a yes/no answer not only gives a false sense of having found the true answer. Critics strongly argue that it also has been more damaging to science than helpful (except for in sciences where you need to conclude with a yes or no answer, e.g. take a pill or not). The comment in Nature touches this point too.

[lal mohan kumar]

Thanks Christopher Bratt for recommending that reading. Personally, I think lack of significance is not an issue for covariates as we rarely interpret them. The issue comes when your theory says a particular variable, which is the variable of interest (X) affects Y and you run the regression and results shows expected sign but not significance

Let me reproduce something I saw online written by Prof David Lane https://onlinestatbook.com/
"Consider the following hypothetical example. A researcher develops a treatment for anxiety that he or she believes is better than the traditional treatment. A study is conducted to test the relative effectiveness of the two treatments: 20 subjects are randomly divided into two groups of 10. One group receives the new treatment and the other receives the traditional treatment. The mean anxiety level is lower for those receiving the new treatment than for those receiving the traditional treatment. However, the difference is not significant. The statistical analysis shows that a difference as large or larger than the one obtained in the experiment would occur 11% of the time even if there were no true difference between the treatments. In other words, the probability value is 0.11. A naive researcher would interpret this finding as evidence that the new treatment is no more effective than the traditional treatment. However, the sophisticated researcher, although disappointed that the effect was not significant, would be encouraged that the new treatment led to less anxiety than the traditional treatment. The data support the thesis that the new treatment is better than the traditional one even though the effect is not statistically significant. This researcher should have more confidence that the new treatment is better than he or she had before the experiment was conducted. However, the support is weak and the data are inconclusive. What should the researcher do? A reasonable course of action would be to do the experiment again. Let's say the researcher repeated the experiment and again found the new treatment was better than the traditional treatment. However, once again the effect was not significant and this time the probability value was 0.07. The naive researcher would think that two out of two experiments failed to find significance and therefore the new treatment is unlikely to be better than the traditional treatment. The sophisticated researcher would note that two out of two times the new treatment was better than the traditional treatment. Moreover, two experiments each providing weak support that the new treatment is better, when taken together, can provide strong support. Using a method for combining probabilities, it can be determined that combining the probability values of 0.11 and 0.07 results in a probability value of 0.045. Therefore, these two non-significant findings taken together result in a significant finding".
Source: https://onlinestatbook.com/2/logic_o...gnificant.html

Though I do agree that the general theme in the above example that one sample study results is not the final word and many times if we repeat the experiment we may get some significance. But it can also happen that many times when we do this experiment or do regression or some tests with different samples and if we cannot find any significance even after say many repetitions can we put blatantly that X doesn't impact Y. In other words is not significance and non-significance symmetric in treatment?
Also some researchers often say to collect more data so that relationship can be established but I feel if something is not there, should we try to make sure that we have searched for it Gazillion times to establish the absence.
These are my thoughts but let me read more from expert sources that were cited

[Richard Williams]

Clyde B. Schechter has an excellent discussion of statistical significance problems in

https://methods.sagepub.com/foundati...icance-problem

It is part of Sage Research Methods Foundations, which includes entries from many contributors to this list:

https://methods.sagepub.com/foundations

If you are an academic, you should harass your library to buy free online access!

[lal mohan kumar]

Thanks prof Richard Williams (I am a heavy user of your resources Stats 1, Stats2). I cannot access that paper as it has restricted access, but I shall get it somehow. Thanks for the direction

[Dirk Enzmann]

https://methods.sagepub.com/foundations
If you are an academic, you should harass your library to buy free online access!

This is the problem: That kind of publication is not accessible to many and will not be read as it should because access is only possible to those lucky few who work in institutions that decided or could afford to decide to buy free access. To wait until one's institution makes this step is a bad solution. Therefore, if you want to be read you should refuse to publish in this kind of handbook.

[Richard Williams]

This is the problem: That kind of publication is not accessible to many and will not be read as it should because access is only possible to those lucky few who work in institutions that decided or could afford to decide to buy free access. To wait until one's institution makes this step is a bad solution. Therefore, if you want to be read you should refuse to publish in this kind of handbook.

Open access is great but as a practical matter most books and journal articles still need to paid for by somebody. For example, none of the Stata Press books are free, and most archived journal articles aren't free.

Also, open source outlets tend to not be as prestigious, at least not yet, and hence can be less attractive to anyone trying to get tenure or be promoted.

Personally, if someone wants one of my publications and can't get it for free, I usually just email it if asked. For that matter I often send articles that haven't been explicitly asked for but which might address some question I have been asked.

[Dirk Enzmann]

... most books and journal articles still need to paid for by somebody.

I completely agree "in principle". My comment was addressed specifically to the https://methods.sagepub.com/foundations series that is special because it is especially expensive and I can't afford privately to buy this (in contrast to most books). At least what I found when looking at some entries by you and by Clyde: You can't simply buy it but have to subscribe to the complete series. If this is correct the publishing policy is prohibitive for individuals. As far as I can see the series is not available at my university (nor at any other university in my city). On the web site I couldn't even find an ISBN! Previously SAGE published the small green books (Series: Quantitative Applications in the Social Sciences) that could be ordered. Again: I completely understand that high quality publications can't be produced without costs and thus can't be free (much of what appears to be free of charge has already be paid by somebody).However, the SAGE series you pointed to is special and the publishing poiicy excludes people like me.

This is why I wrote that one should seriously consider not to publish in this series.

[Dirk Enzmann]

Add on: There is an ISBN (I didn't find initially). But when I searched for the title "Statistical Significance Problem" (Edited by Paul Atkinson, Sara Delamont, Alexandru Cernat, Joseph W. Sakshaug & Richard A. Williams) with the ISBN 9781529750720 there is no way to buy it at a book seller available to me. Normally I can order books at Amazon that have been published outside my country (Germany) by going to www.amazon.com (instead of www.amazon.de). But if you enter the titel and / or ISBN there it can't be found and thus can't be ordered. SAGE does not tell you the price and order policies, either.

[Richard Williams]

For the moment, you can purchase online from Sage:

https://us.sagepub.com/hi/nam/sage-r...ons/book245688

I can't imagine many individuals buying a personal copy though. It is clearly aimed at libraries. It is 10 volumes, 5,600 pages, and costs either $2,120 or $2,650. If anybody wants to stop by my house though I'll be happy to show them my copy. ;-) The print version looks really cool and impressive but I never use it; I just use the online materials which have these nice search functions and easy hyperlinks.

Sage Online is a fantastic resource for University libraries though. The user can download (for free) any of the Sage little green books, as well as zillions of other Sage materials. I'm sure my library pays a good price for them, but in exchange, my students hardly ever buy textbooks anymore. Instead, I get stuff from Sage or other sources I get for free through my library or that I just get for free, period.

I do wish access was easier for people whose library resources aren't as good as mine. On the other hand, those people are no worse off than if the SRMF had never been written. I doubt that most or any of this stuff would have been done for free if Sage hadn't commissioned it.

Again, if you are at a university, I would encourage you to push your library to get it as well as the whole Sage online offerings. But, if you can't get anybody to buy it, you can try contacting authors of entries you are interested in. I don't know what other authors do, but I usually honor requests for copies of my work.

[lal mohan kumar]

Dear all
A related paper on significance, p-values etc got published recently in Journal of Economic Perspectives and I am referring it here so that interested users can refer it

Statistical Significance, p-Values, and the Reporting of Uncertainty by Guido W. Imbens

[Eric de Souza]

#27: I was just going to refer to the paper by Imbens which is part of a symposium of statistical significance. If you follow the link below and scroll down you will come to the three papers which compose the symposium. And they are complimentary (ungated).
https://www.aeaweb.org/issues/646

[lal mohan kumar]

Thanks Eric de Souza. These will be helpful for me and all others in this community.