Faster fisher exact test

[olivier Perruchet]

Hello guys,
I want to use the fisher exact test to two independant samples. My problem is with that function :

tabulate v`t' vvect ,exact(1000)

it takes me like 4 hours to execute my test on a dataset of 128 variables and all of them own 1024 observations.
That's pretty long and i wonder if someone know an other way to do that test in a faster way ?

[Marcos Almeida]

I recommend to try with - exact(2) - option. I’ve never needed to go beyond 5.

[olivier Perruchet]

I cannot you a low setting as this one i will get the following error :

exceeded memory limits using exact(2); try again with larger #; see help tabulate for details

I don't think the slowness of the calculate is because of the setting, but more because of the function itself.
But maybe it doesn't exist such a function on Stata ?

[Andy Hiegel]

Dr. Almeda.

I had the same problem with STATA, but I was able to get results with exact (3). Should I be concerned with the validity of my results because I had to use this modified command?

Thanks,
Andy

[Rich Goldstein]

no

[Maarten Buis]

That is to be expected. The exact test, even in moderate sample sizes, requires an unreasonable amount of computations. The test is a permutation test, that computes a statistic for every possible table given the margins. Here is combinatorics at work: even with moderate sample sizes you get a humongous number of possible tables. Stata does use a smart algorithm to cut this down a bit, but there is only so much it can do.

Calling a test an exact test is good marketing, but that does not mean it is any good. The exact test tends to be overly conservative. Many approximate tests have much better coverage than the exact test, and are substantially faster. See:
Alan Agresti & Brent A. Coull (1998) Approximate is Better than “Exact” for Interval Estimation of Binomial Proportions, The American Statistician, 52:2, 119-126. https://doi.org/10.1080/00031305.1998.10480550

[Joro Kolev]

The Agresti & Coull (1998) reference Maarten gave in #6 is seminal. It shows that even for small samples asymptotic tests might be preferred.

But otherwise the situation with the exact Fisher test is fortunate, because when the sample is large and the exact test is hard to compute, asymptotic tests are expected to perform well.