Cohen’s f squared in a linear mixed effects regression model

[Martine Nurek]

Hi there,

I have conducted a vignette study in which 83 surgeons responded to 32 clinical vignettes (fictitious patient cases). Patient characteristics were manipulated systematically across the vignettes. Per vignette, surgeons indicated the likelihood that they would perform surgery A vs. surgery B (on an 11-point visual analogue scale). I am using a linear mixed effects regression model to measure the influence of the manipulated patient characteristics on surgeons’ judgments (code below).

A reviewer has asked me to provide Cohen’s f squared for each predictor. I’m not sure how to do this, or whether it can even be done (as it relies on R squared, which is not straightforward in a mixed effects linear regression model).

I am seeking guidance on either of the following:

1. How to compute f squared per predictor in a linear mixed effects regression model (if possible); OR
2. An alternative measure of effect size per predictor.

Sincere thanks in advance!
Martine

My STATA code

mixed Judgment i.PATIENT_AP_CRUCIATE_recoded i.PATIENT_AGE_recoded i.PATIENT_ASA_recoded i.PATIENT_BMI_recoded i.PATIENT_SITE_PAIN_recoded || ID:, reml

…where:

Judgment = surgeon’s response (11-point scale)
i.PATIENT_AP_CRUCIATE_recoded = manipulated patient variable
i.PATIENT_AGE_recoded = manipulated patient variable
i.PATIENT_ASA_recoded = manipulated patient variable
i.PATIENT_BMI_recoded = manipulated patient variable
i.PATIENT_SITE_PAIN_recoded = manipulated patient variable
ID = surgeon identifier

Stata version: Stata/MP 17.0

[Joseph Coveney]

There is a step-by-step illustration of the procedure for a linear mixed model fitted with Stata here.

[Martine Nurek]

Thank you so much Joseph!

[Martine Nurek]

I was able to use this method to compute 'Cohen's f' for predictors in my model that vary within-subjects. However, my model also contains predictors that vary strictly between-subjects, and I'd like to compute 'f squared' for those too. I'm not sure that this method can/should be used for those variables. Is anyone able to advise?

[Joseph Coveney]

I was able to use this method to compute 'Cohen's f' for predictors in my model that vary within-subjects. However, my model also contains predictors that vary strictly between-subjects, and I'd like to compute 'f squared' for those too. I'm not sure that this method can/should be used for those variables.

The note in bold font at the very top of that UCLA FAQ raises concerns about the suitability of the method for such between-subjects predictors.

The effect size measurement is essentially a measure of the reduction of residual variance. Because between-subjects predictors would be expected to affect between-subjects variance component more, perhaps you can turn the tables and apply the same approach to that variance component, that is, hold the residual variance constant and look at changes to the random-effects variance analogously to what is illustrated there in the FAQ. I suspect, though, that that's easier said than done and even then there's no guarantee of its validity.

As an alternative, consider summing or averaging the outcomes for each subject and reducing the multilevel (mixed) model to a conventional ANOVA containing only the averaged outcomes for each subject and the between-subject predictors. From there, you can use conventional methods for computing effect sizes. You can easily derive Cohen's f² from the returned results of estat esize invoked after anova or regress as shown in this post. (Worked example's do-file and log file is attached to this post in that thread.)

[Martine Nurek]

Thanks so much Joseph, I was thinking exactly along these lines, and agree that the second option is probably more defensible.
To confirm: within-subjects variables would be excluded entirely from this anova / regression, right?

That is:
-effect sizes for the within-subjects variables are computed using the UCLA procedure
-effect sizes for the between-subjects variables are computed by averaging the DV per participant and using conventional regression to calculate R² and therefore f².

Have I understood correctly? Huge thanks again!

[Joseph Coveney]

. . . within-subjects variables would be excluded entirely from this anova / regression, right?

You mentioned above in #1 that the within-subjects variables (fictitious patient characteristics) "were manipulated systematically" and so their averages (or other measure of central tendency) would be identical between the surgeons, anyway, wouldn't they? You could compute the effect measure both ways in order to see whether it makes a substantive difference in the conclusion, but for that reason alone I suspect that it won't in your case.

[Martine Nurek]

Yes, exactly: the within-subjects variables (fictitious patient characteristics) were manipulated systematically, so all surgeons were in all conditions and ultimately saw exactly the same set of vignettes.
By "their averages would be identical between the surgeons", I assume you are referring to the averages of these manipulated variables? If so, then yes, these would be constant across surgeons.

[Joseph Coveney]

By "their averages would be identical between the surgeons", I assume you are referring to the averages of these manipulated variables?

Yes.

[Martine Nurek]

Joseph, I tried it both ways and they produced reasonably similar results. I decided to go with the first option, namely:
-constraining between-subjects variance to measure f² for within-subjects variables (UCLA procedure) and
-constraining within-subjects variance to measure f² for between-subjects variables (inverted UCLA procedure).

However, it returned some perplexing results, tabulated below (note: this is a partial tabulation; some results omitted for simplicity):

Factor	b [95% CI]	p	Cohen's f squared
Patient Factor 1 (within-subjects) 0 = normal 1 = abnormal	-1.93 [-2.17,-1.68]	<0.001	0.092
Patient Factor 2 (within-subjects) 0 = normal 1 = abnormal	-0.57 [-0.82, -0.33]	<0.001	0.008
Surgeon Factor 1 (between-subjects) 0 = below median 1 = equal to/above median	1.26 [0.54, 1.99]	0.001	0.173
Surgeon Factor 2 (between-subjects) 0 = below median 1 = equal to/above median	0.78 [-0.10, 1.66]	0.082	0.045

Notably, Patient Factor 1 has a larger coefficient, but a much smaller effect size, than Surgeon Factor 1.
Both are binary variables (coded 0 vs. 1), so it's not a scaling issue.
One could argue that the coefficients are not that different (overlapping confidence intervals). Still, the effect sizes are markedly so, which does confuse me.

Importantly, Patient Factor 1 is a within-subjects factor while Surgeon Factor 1 is a between-subjects factor. If I've understood correctly, this means that their f² statistics concern different types of variance. A larger f² for Surgeon Factor 1 could thus be telling us that: the between-subjects variance explained by Surgeon Factor 1 > the within-subjects variance explained by Patient Factor 1.
Would you agree with this interpretation? Do you have another interpretation?

I will add that the results are very similar when I use the "aggregation" method for calculating f² for the between-subjects variables (i.e., averaging the DV per participant and then using conventional regression to calculate R² and therefore f²). When I do this, the f² for Surgeon Factor 1 is 0.15, and therefore still considerably larger than that for Patient Factor 1.

I'm exceedingly grateful for any insights that you're able to offer! Thank you so much in advance.

[Joseph Coveney]

Would you agree with this interpretation? Do you have another interpretation?

Your apples-and-oranges take on the difference sounds reasonable to me and I don't offhand have any alternative to offer.

I do however have to confess that I've never used any of these post hoc effects-size estimates (omega squared, eta squared etc.) and the variance-explained stuff (R², adjusted R²) is among the output from regress, anova and xtreg that I tend not to pay a whole lot of attention to.

I'm guessing that the referee asked for Cohen's f² in order to get some kind of handle on the regression coefficients resulting from linear models fitted to your eleven-point visual-analog scale (VAS) scores.

If it were me, I would've established a minimum clinically significant / medically important difference in the VAS scores (the hard part, I know, but it's been done), powered the study on the basis of that as the minimum detected difference (i.e., minimum effect size in expressed in mean difference of the VAS score) and then provided interpretation of the regression results in terms of that.

[Martine Nurek]

Thanks so much Joseph -- I really appreciate your prompt response, and willingness to muddle through this with me!
I like your idea regarding the predefined VAS difference/criterion and will bear this in mind for next time.