Command for computing effect size for paired/dependent samples t-test

[Eitan Hemed]

Hi everyone,
I was trying to find how can i find calculate by using syntax the effect size (specifically, Cohen's d) following a paired/dependent samples t-test, but unfortunately found only the esize unpaired subcommand. Is that the one that should be used?

Any help would be appreciated,
Thanks!

Eitan

[Clyde Schechter]

No, -esize- doesn't deal with paired samples.

But this is not hard to do "by hand." So suppose your two paired measures are variables m1 and m2

gen delta = m1 - m2
summ delta
display =`r(mean)'/`r(sd)'

Note: Not tested. Beware of typos.

[Jerome Anderson]

But if you look at https://blog.stata.com/2013/09/05/me...e-in-stata-13/ the author says you can use esize for paired samples. The example given was a pre-post with 15 observations in each group. I used the formula you (Clyde) gave on pre-post data I have and got an effect size of ~ 0,72. When I used esize using both unpaired and twosample commands on that same file the results I got were in both cases ~0.42. That's a rather large difference.

Any thoughts on this?

[Clyde Schechter]

I cannot find any mention of using -esize- for paired data in the blog post you linked to. In fact, the word "paired" does not even appear anywhere in it. The only example I can think of with "15 observations in each group" is the one about math scores in a treated and untreated group, but these are unpaired data, where, coincidentally, the group sizes are the same.

[Jerome Anderson]

But a pre- post- design is by definition requires a paired test, does it not? The design was described as follows: "Consider a study where 30 school children are randomly assigned to classrooms that incorporated web-based instruction (treatment) or standard classroom environments (control). At the end of the school year, the children were given tests to measure reading and mathematics skills. The reading test is scored on a 0-15 point scale and, the mathematics test, on a 0-100 point scale." A random assignment to treatment and control groups, with pre-intervention and post-intervention scores, would require a paired sample t-test, would it not? Granted, the word "paired" is not mentioned, but the design would need to be paired.

But the larger question concerns the value of Cohen's d. Using your method with my data, I got effect size of 0.72. Using Stata's esize command, the effect size was 0.42. That's too large a difference to ignore. Which of the two options is the better (or correct) one to use with paired groups?

[Bruce Weaver]

Jerome, for the two-group pre-post design you describe in #5 (with random allocation to groups), many authors recommend ANCOVA. E.g.,

• https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1121605/

[Clyde Schechter]

Jerome,

Reading your excerpt from that example, I see no mention of pre-intervention testing, only testing at the end of the study. The data set itself only contains one test score per child in each subject. There is no paired design here. Yes, a pre-post design with two groups would require an analysis that is at least partially based on paired outcomes (and would actually be more complicated than just a paired t-test), but the example you refer to is not such a design. It is a simple two-groups with one outcome assessment study. No pairing of any kind.

[Bruce Weaver]

After reading #7, I had to re-read #5 to see if I had misunderstood. I agree with Clyde that the quoted material in #5 ("Consider a study...") says nothing at all suggesting paired scores. But Jerome did ask about the two-group pre-post design after that:

A random assignment to treatment and control groups, with pre-intervention and post-intervention scores, would require a paired sample t-test, would it not?

That's what I was responding to in #6.

[Clyde Schechter]

Bruce,

Yes, you're quite right as a response to that quote from #6. I guess it's unclear just what Jerome is asking about.

[Jerome Anderson]

Thanks to both of you for your replies. Yes, I now see the data set does not have pre and post, just post for the treatment and control groups. My error. Although why there would not be a pre I have no idea -- I would think a pre-treatment baseline would be essential. But that is another issue.

There is one remaining question, and that is this (taken from my 1st and 2nd posts): But the larger question concerns the value of Cohen's d. Using your method with my data, I got effect size of 0.72. Using Stata's esize command, the effect size was 0.42. That's too large a difference to ignore. Which of the two options is the better (or correct) one to use with paired groups?

[Clyde Schechter]

But, if I understand correctly, you applied that esize command following an unpaired analysis of unpaired data. And as far as I know -estat esize- is not available following paired analyses. The results are a correct analysis for unpaired data.

The formula I provided in #2 is not applicable to the unpaired data in that example. And I do not know what you did to come up with the results you did. I don't know what you would have used for m1 and m2. In any case, applying this calculation to unpaired data does not produce correct, or even meaningful, results.

[Jerome Anderson]

Clyde, thanks for continuing the conversation. I think things are coming clear. Let me walk through my thought process on this, starting with the initial t-test.

If we have truly paired data, and run a t-test using both the paired and unpaired t-test commands, we obtain different results. Hence if the data are truly paired, the proper test is the paired sample t-test. Use of the two sample or unpaired test is inappropriate. Stata’s options for t-tests are one sample, two sample (with 2 options) and paired.

When it comes to calculating the effect size, Stata provides two esize options. One is twosample, the other is unpaired. Note this is different from the t-test commands. With t-tests, we have paired and two sample. With esize, we have unpaired and twosample. This was one source of my confusion.

In the example at https://blog.stata.com/2013/09/05/me...e-in-stata-13/ , the two sample t-test was used as well as the twosample esize command. I made the mistake of thinking these data were paired when they were not. That was another source of my confusion. Based on this mistake, I had thought I could use the two sample option for paired data, and I was confused as to why your formula was needed for the paired option. I now see I was mistaken to think the twosample esize option was appropriate.

Now my question is why Stata has separate esize commands for unpaired and two sample, when it seems the same command could cover both situations. Unpaired data are by definition two sample data, and vice versa, aren’t they? And error messages from Stata are somewhat conflicting. I used the sample data set from the example in the blog, and got the following results:

. use "F:\Stata_Materials\171128_t-Test_Sample_File.dta"
(Fictitious web-based learning experiment data)

. esize twosample math, by(treated)

Effect size based on mean comparison

Obs per group:
Control = 15
Treated = 15
---------------------------------------------------------
Effect Size | Estimate [95% Conf. Interval]
--------------------+------------------------------------
Cohen's d | -.9419085 -1.691029 -.1777553
Hedges's g | -.916413 -1.645256 -.1729438
---------------------------------------------------------

. esize unpaired math, by(treated)
may not combine esize unpaired with option by()
r(198);

. esize unpaired math
by() option required
r(100);

One error message states we cannot use esize unpaired with the by option; the other error message states we must use the by option. While the documentation shows how to use each command, the error messages are not helpful.

The answers to my original questions are now (1) if the data truly are paired (as are the data with which I am now working), to calculate the effect size following a paired sample t-test I use your formula, and (2) the reason the effect sizes from your calculation and the twosample esize command are different is because they are supposed to be different because the two sample option is not appropriate for paired data. It was a long journey to get to this point, but I think I understand all of this now. Thanks again for helping me think this through.

[Clyde Schechter]

Yes, it sounds like you understand it all now.

Concerning the two esize commands, unpaired vs twosample. You are correct that unpaired data is two samples and vice versa. The difference between these commands is how they expect the data to be laid out. So, the twosample esize command is probably used in most circumstances. It is used when the two samples (two groups) are separate sets of observations in the data. The distinction between the two groups in the data is marked by a second, dichotomous variable. So a typical example would be:

esize twosample outcome, by(subgroup)

where outcome is the outcome variable, and subgroup is a dichotomous variable distinguishing the two subgroups being compared. This is the usual way two-sample data are organized in Stata data sets.

But there are occasionally data sets where the outcomes for one group are in one variable and the outcomes for the second group are in a different variable, and there is no variable corresponding to subgroup. Each observation in the data set contains an outcome for somebody in one group and, in a different variable, an outcome for somebody in the other group, but there is no actual matching or pairing of these people: it is coincidental which person from the first group happens to be in the same observation as the person from the second group in that observation. This is where the unpaired version of -esize- comes in:

esize unpaired outcome_var_for_group_1 = outcome_var_for_group_2

The error messages you are getting from Stata are a bit messed up here, and if somebody from Tech Support is following this thread, perhaps they will arrange to get this fixed in a future update. The message "may not combine esize unpaired with option by()" is accurate: you cannot use the -by()- option with -estat unpaired-. (By contrast, with -estat twosample- it is obligatory.) The second message you are getting,

. esize unpaired math
by() option required
r(100);

is incorrect. The error message should be different: it should tell you that you need to specify two variables for the group outcomes, and they must be connected by an = sign. It is not correct to say "by() option required," as it is neither required nor even allowed! The message also fails to point up the actual error you made.

Added: It is pretty unusual to find a data set that is organized for use with -esize unpaired-. And I would be hard pressed to think of a situation where that would be the best way to organize the data. I would even go so far as to say that I, personally, would never create such a data set in my own work. Most two-group datasets use the "vertically stacked" layout that goes with -esize twosample-, so I think you should focus your efforts on learning and understanding the application of that one, and, more or less, ignore -estat unpaired-.

[Jerome Anderson]

Thanks, Clyde. I appreciate especially your explanation of the lack of clarity in the error message for the unpaired command. You confirmed what I thought the message should be.

I really do appreciate your help w/ this.

[Weiwen Ng]

...
There is one remaining question, and that is this (taken from my 1st and 2nd posts): But the larger question concerns the value of Cohen's d. Using your method with my data, I got effect size of 0.72. Using Stata's esize command, the effect size was 0.42. That's too large a difference to ignore. Which of the two options is the better (or correct) one to use with paired groups?

Folks, a point of information, if I may. Most variants of Cohen's d are the change score divided by some estimator of the standard deviation of either the baseline score, or the change score. Clyde's code actually divides by the SD of the change score. That may explain why the estat esize postestimation command gave a smaller effect size than Clyde's code, because estat esize appears to use the pooled SD of the baseline score for Cohen's d.

I have no opinion on which measure of effect size is more valid.