P-value in sensitivity and specificity

[Amir Jabbari]

Hi everyone,
I want to use a new method (Cold_forcepsbiopsy) to diagnose a disease and I want to compare its result with a gold standard (MRI). To calculate sensitivity and specificity I run "tabulate" and specify the -col- option. I want to know is it correct to specify the -chi- option to obtain p-value? and if yes, what is the interpretation of P-value in sensitivity and specificity?

[Rich Goldstein]

what is your null hypothesis? if you want to test sensitivity and specificity separately, the chi-squared statistic is clearly irrelevant (and while I can imagine an hypothesis where it is relevant I don't think that null is interesting); why do you want a test here anyway? if your goal is to determine whether the new method is better than the old, then you need to decide what you mean by better (e.g., does the new method better discriminate between those who have the event and those who don't (use C statistic); another and quite different criterion might be whether calibration is better with the new method); so, what is your substantive research question here?

[Amir Jabbari]

Dear Rich Goldstein,
Thanks for your valuable questions.
In fact, I have 3 methods to diagnose disease (Cold_forceps biopsy and brush cytology) and my gold standard (MRI). I want to assess that which one (Cold_forceps biopsy or brush cytology) have better diagnostic value. My null hypothesis is that there is no difference among Cold_forceps biopsy and brush cytology. So I calculated sensitivity and specificity separately and specify its P-value to compare both p_value. Because my sample size is the same in both methods so I want to compare both p_value to decide which one is better. Whichever of the two p_values are more significant is better. Is it true?

[Rich Goldstein]

"no difference among Cold_forceps and brush cytology" with respect to what: sensitivity, specificity, both, discrimination, calibration ...?

also, if the p-value you refer to in #3 is based on the table in #1, then, no, I don't think that this is giving you anything you want (but since I am still unclear on what you want, take this with a grain of salt)

[Amir Jabbari]

"no difference among Cold_forceps and brush cytology" with respect to 4 parameters of diagnostic value including (sensitivity, specificity, positive predictive value, and negative predictive value). (sensitivity and specificity can be specify with the -col- option and positive predictive value and negative predictive value with the -row- option).
and yes my means of p-value in #3 is based on the table in #1.

[daniel klein]

Not exactly my area of expertise but comparing p-values strikes me as flawed. If you want to test whether the sensitivity differs between two methods, then you should test the difference between the two sensitivity estimates. Because both estimates are based on the same sample, this would be something along the lines of a paired t-test.

[Rich Goldstein]

is your N really only 33? and you really want to compare 4 different criteria? too much for too little data, in my opinion; and I agree with daniel klein that comparing p-values is the wrong way to go; but also, as above, those tabulations and chi-squared values are not testing any of your criteria; the null for each of those tables is that the row variable is independent of the column variable and I don't see how that is related to what you want at all

[Amir Jabbari]

Dear Daniel Klein,
Thank you for your sincere contribution.

[Amir Jabbari]

Yes dear Rich Goldstein, my sample size is 33 and I want to compare 4 different criteria.
Thanks four your valuable comments.

[daniel klein]

I agree with Rich Goldstein: 33 observations is a little thin.

If you really want to force your way through, here is one approach:

capture program drop bootit
program bootit , rclass
    tempname F sens1 sens2
    tabulate MRI_result cold_BX , matcell(`F')
    scalar `sens1' = `F'[2,2]/(`F'[2,1]+`F'[2,2])
    tabulate MRI_result brush_cytology , matcell(`F')
    scalar `sens2' = `F'[2,2]/(`F'[2,1]+`F'[2,2])
    return scalar diff = `sens1'-`sens2'
end

bootstrap diff = r(diff) , reps(500) : bootit

The code estimates the difference in sensitivity between the two methods and uses nonparametric bootstrap to get standard errors and CIs. This is the best I can come up with. However, I would not base any real-life decisions on these results.

[Amir Jabbari]

Dear Daniel Klein,
I appreciate you.