Simple mean comparison: Upaired t-test and normality

[Konstantin Fischer]

Dear Statalists!

I got a bit confused about the assumptions/requierments of the t-test and think here are some very good experts.

As far as I know, the "unpaired" two-sample t-test requires an approx. normal distribution to investigate the difference in the sample means. In my concrete situation, this assumption does not hold and the data in both samples are - even with a relatively large sample (n=100) - more than a little bit skewed.

Now I read a lot about the assumption and some state that it does not hold for the "unpaired" t-test. The authors refer to various simulations that there is no significant difference in the power (compared to a non-parametric test such as Wilcoxon-rank sum test) and the t-statistic gets with a large number even normally distributed (central-limit-t.?).

If I apply the non-parametric test I get significantly different results that would change the complete story.

My questions: Is it statistically acceptable to apply the unpaired t-test with skewed data and a relatively large dataset?

• If yes, are there good papers that provide more information regarding the rationale/arguments behind this effect?
• If not, is there a way to (test) estimate the power of the current t-test compared to a non-parametric (or any other)?

If you have any other good ideas on how to deal with this situation please let me know. Thank you for your help!

[Marcos Almeida]

A t test can be (somewhat) robust to departs from the normality assumption. But the higher the sample, the bigger the false positive rate under these terms. A nonparametric test seems to be a reasonable approach under skewed data.

That said, under ‘ordinary’ situations and ‘average-sized’ data sets, they won’t differ much.

Also, it is not clear in the message how large and how skewed is the variable, neither do we know whether the variable is really continuous.

To end, depending on the study objectives, these tests can be taken as exploratory analysis. A ‘full’ model adjusted for a bunch of covariates tend to be a natural approach. Being the Yvar very skewed, a non-parametrical regression is something to consider.

[Nick Cox]

If the difference between means is of interest, I would get a bootstrap confidence interval for that difference. That gives you a test indirectly, and more information yet on (as it were) the shape of uncertainty.

Another possibility is a generalised linear model with some identity link.

[Bruce Weaver]

Good morning Konstantin Fischer. I teach my students that all t-tests have a common format, as follows (and see also the attached slides):

    statistic - parameter|H0
t = ------------------------
        SE_statistic

Further, I tell them that the normality assumption applies to the sampling distribution of the statistic that is in the numerator of that ratio, and that in the real world, approximate normality is usually the best we can hope for.

Using the unpaired t-test as a specific example, the statistic in the numerator is Xbar1 - Xbar2, the difference between the two sample means. Textbooks typically say that the two samples must be drawn from normally distributed populations (with equal variances). I tell my students that if it were possible to sample from two perfectly normally distributed populations with exactly equal variances, then the sampling distribution of Xbar1-Xbar2 would be perfectly normal, and the unpaired t-test would be an exact test. But in the real world, perfectly normal distributions and exact homogeneity of variance do not exist. Therefore, in the real world, the unpaired t-test is really an approximate test. And as we do with other tests that are recognized as approximate tests (e.g., Chi-square tests), we must ask whether the approximation is good enough to be useful (as George Box might say). Sample size is certainly one relevant issue here: As n increases, the sampling distribution of the statistic in the numerator will converge on the normal distribution. But just how large n needs to be depends on the shape(s) of the underlying raw score distribution(s).

Another thing I always ask myself is whether it is fair and honest to use means and SDs for description. If it is (and the sample size is "large enough"), then I feel fairly comfortable using a t-test. But if I believe that median and IQR (for example) are clearly better for description, then I would likely use quantile regression rather than a t-test. In your post, you said the data were "more than a little bit skewed", which makes me think you might prefer median & IQR for description.

Here is one other thing to bear in mind: If you use the Wilcoxon-Mann-Whitney test as a test of location (rather than a test of stochastic superiority), then "small differences in variances and moderate degrees of skewness can produce large deviations from the nominal type I error rate." That is a quote from the abstract of an interesting simulation study by Fagerland & Sandvik (2009).

Fagerland, M. W., & Sandvik, L. (2009). The Wilcoxon-Mann-Whitney test under scrutiny. Statistics in medicine, 28(10), 1487-1497.

HTH.

[Konstantin Fischer]

Wow, thank you Marcos Almeida, Nick Cox and Bruce Weaver for the help!

Now, I got a much better understanding and will think about the future strategy.

In particularly read more about bootstrap confidence interval and will think about to transform my variable (ln) in a more normally distributed distribution.

Thank you!!!