Post-hoc fisher's exact test to determine which level is significant

[noora kazanji]

Dear STATA community,

I have a dataset that looks at result 1 versus result 2 and which surgical diagnoses may be associated. A snippet of the data is included below:




I calculated the p value (in bold as 0.017) using chi-squared, then repeated it using Fishers exact (p = 0.045) given the cell counts were small. I'd like to know if I can further determine which surgical diagnosis may be responsible for this significance, if possible. Are there post-hoc tests for Fisher's exact?

Appreciate any help!

[Carlo Lazzaro]

Noora:
do you mean something along the following lines?:

. set obs 10
Number of observations (_N) was 0, now 10.

. g id=_n

. g Diagnosis=1 in 1/3

. replace Diagnosis=2 in 4/7

. replace Diagnosis=3 in 8/10

. g Outcome=1 in 1/2

. replace Outcome=0 in 3/5

. replace Outcome=1 in 6/10


. label define Outcome 0 "Wrong diagnosis" 1 "Right diagnosis"


. label val Outcome Outcome

. logit Outcome i.Diagnosis

note: 3.Diagnosis != 0 predicts success perfectly;
      3.Diagnosis omitted and 3 obs not used.

Iteration 0:   log likelihood = -4.7803567  
Iteration 1:   log likelihood = -4.6822309  
Iteration 2:   log likelihood = -4.6821312  
Iteration 3:   log likelihood = -4.6821312  

Logistic regression                                     Number of obs =      7
                                                        LR chi2(1)    =   0.20
                                                        Prob > chi2   = 0.6576
Log likelihood = -4.6821312                             Pseudo R2     = 0.0205

------------------------------------------------------------------------------
     Outcome | Coefficient  Std. err.      z    P>|z|     [95% conf. interval]
-------------+----------------------------------------------------------------
   Diagnosis |
          2  |  -.6931472   1.581139    -0.44   0.661    -3.792122    2.405828
          3  |          0  (empty)
             |
       _cons |   .6931472   1.224745     0.57   0.571    -1.707309    3.093603
------------------------------------------------------------------------------

.

[Joseph Coveney]

A snippet of the data is included below: . . . I'd like to know if I can further determine which surgical diagnosis may be responsible for this significance, if possible.

You can both handle your complete separation problem and do your exploratory pairwise evaluations using various approaches. Run the following code to see a couple of them in action.

version 18.0

clear *

input str18 PSI byte(count1 count2)
Crayniopharyngioma 0 3
GBM 6 1
Epilepsy 8 1
"Acoustic Neuroma" 3 0
Meningioma 2 2
end

sort PSI
list, noobs

*
* Begin here
*
encode PSI, generate(psi) label(PSIs)
quietly reshape long count, i(psi) j(out)
quietly replace out = out - 1
quietly drop if !count
quietly expand count

bayes, prior({out:}, t(7)) nomleinitial nchains(2) rseed(640467876) ///
    nomodelsummary nodots ///
    thinning(5): logit out ibn.psi, noconstant asis
tempfile posterior
quietly bayes, saving(`posterior')

set more on
foreach graph in trace ac histogram {
    foreach parm in 1.psi 2.psi 3.psi 4.psi 5.psi {
        bayesgraph `graph' {`parm'}
        more
    }
}

frame create Posterior
frame Posterior {
    quietly use `posterior'
    quietly expand _frequency

    // Second-largest difference eye-balling posterior means in regression table
    generate double del21 = eq1_p2 - eq1_p1
    histogram del21, xline(0)
    more

    // Largest eye-ball difference between posterior means
    generate double del32 = eq1_p3 - eq1_p2
    histogram del32, xline(0)
    more
}

/* For entertainment */
firthlogit out i.psi, nolog // from SSC
estimates store Full

constraint define 1 _b[2.psi] = 0
quietly firthlogit out i.psi, constraints(1)
lrtest Full

constraint define 2 _b[3.psi] = _b[2.psi]
quietly firthlogit out i.psi, constraints(2)
lrtest Full

exit

Notes:
1. You can edit the version number back to what you have. (I don't remember when -bayes- came about.)
2. If you want to run the entertainment section of the code, then you'll need to get the user-written -firthlogit- from SSC first.
3. If your snippet is representative, then you've got awfully sparse data if you plan on making bold claims about responsible surgical diagnoses.

[noora kazanji]

Dear Carlo,

Thank you for your reply. Apologize about the delay in mine. Yes, logistic regression is a great idea. However, interestingly, I no longer saw significant results as i did with the Fisher's exact test:








As my numbers are small, would you say the Fishers exact is more appropriate to use statistically?

Thank you for your time and thoughts!
-Noora