Again on revisiting the role of p-value

[Carlo Lazzaro]

Dear All,
those interested in this debate may find the following article interesting:

https://link.springer.com/article/10.1007/s13246-021-01068-1.

[Jared Greathouse]

I've long been a proponent that p-values are useless to most applied settings (in theoretical settings this may change). Research decisions, as well as the usefulness of said decisions, should be based on how well you design your analysis, as Don Rubin and others suggest.

When I become a professor one day, and I'm teaching statistics to would be public policy analysts, a student will need to do more than show me a nice low p-value when they try to convince me a tax, for example, meaningfully affected... sugar consumption. They can't simply present a graph, nor can they rely on omnibus testss.

No for me, this hypothetical student would need to convince me that they've adjusted (or at minimum considered) heterogeneous treatment effects. That they've selected an appropriate set of comparison units beyond "well, these places seem similar", whether they use synthetic controls, differences-in-differences, matching, or some other technique of interest.

All of this of course has little to nothing to do with p-values, and has everything to do with how you've structured your study in the first place. That's how statisticians, in my opinion, should focus on their work, not "this value is lower than this value so my finding is meaningful." You can have a p value as low as the Underworld and it'll mean nothing, to me, if someone's design is poor.

[Doug Hess]

I find it very frustrating when journals replace reporting p-values with just stars. I wish there was some joint social science committee of journals that would come out with a statement on this.

[Clyde Schechter]

Yes, significance stars are absolutely the most egregious misuse of the concept of statistical significance. Even if one still retains some faith in p-values (I do, but very minimally), significance stars are beyond the pale. It should be a felony to use them. ;-)

[Jared Greathouse]

The discussion ended in my mind when I read papers from this issue of JASA, and this paper in Euro J Epidemiol. I know I'm technically in my second semester of my Ph.D program, and everyone hasn't taken the requisite courses in stats just yet, but when people older than myself do Difference-in-Differences and don't even mention parallel trends and why they matter or other design relevant assumptions, and simply do the equivalent of "This number lower than that number, good, this number higher than that number, bad". We're awash in fancy new estimators (not that I'm any stranger to this), yet quite dry with scientific thinking and planning. It sort of makes me wish we taught GLMs and other estimators last. What good is it, after all, that you can tell me why logit and probit are (pretty much!) the same but when it comes to really designing a paper (causal or otherwise), you revert your work to this black-box-y sort of mysticism? Clyde Schechter

Doug Hess I think the whole "p-values replacement with stars" thing is laughable. It's just a way of avoiding the issue. It means methods teachers no longer need to critically interrogate the foundations of their pedagogy, and for that matter, that their own work doesn't need to change a lick. It means you now don't need to go into class next week and say "Hey everyone, just so you know, I'm banning the phrase statistical significance from papers, so I hope you've thought long and well about uncertainty and the limits of your estimator and the quality of your data and the complexities of modeling a really messy world. Don't try and overly flatter me with talk of theory either, I wanna hear about DESIGN and the empirical setup of your paper and why it is a sensible one. So everybody, summon forth your inner statistician and econometrician, because the old rules no longer apply this go-round." I'm quite certain many students would panic internally of this was said by somebody who was grading their papers.