Effect Size with Robust variance estimates

[Ryan Luo]

Hello,

Why does "estat esize" not work with "Robust variance estimates"? For example, if I run a regression with regress X Y, robust, I receive the error: "estat esize only works with vce(ols)." Is there any way to compute effect sizes for regression results with robust variance estimates?

Thanks in advance!

[George Ford]

The values of eta2 don't change with robust estimation.

clear all

sysuse auto, clear

reg price mpg weight length
estat esize

program define eta_squared
    version 14
    syntax varlist(min=2)

    // Run the regression
    quietly regress `varlist' , robust   // set error type here

    // Store the dependent variable name
    local dv: word 1 of `varlist'

    // Calculate total sum of squares (SST)
    quietly sum `dv'
    local SST = r(Var) * (r(N) - 1)

    // Calculate model sum of squares (SSM)
    local SSM = e(mss)

    // Calculate eta-squared
    local eta2 = `SSM' / `SST'

    // Display results
    display "Eta-squared: " %6.4f `eta2'

    // Calculate partial eta-squared for each predictor
    local predictors: list varlist - dv
    foreach var of local predictors {
        // Run regression without this predictor
        local other_vars: list predictors - var
        quietly regress `dv' `other_vars' , robust   // set error type here
        local SSM_reduced = e(mss)
        
        // Calculate partial eta-squared
        local partial_eta2 = (`SSM' - `SSM_reduced') / (`SST' - `SSM_reduced')
        
        display "Partial eta-squared for `var': " %6.4f `partial_eta2'
    }
end

eta_squared price mpg weight length

[Ryan Luo]

I see. Thank you, George.

[Leonardo Guizzetti]

Robust variance estimates assume there is no constant variance in the sample, so in that sense, a standardized effect size isn’t uniquely determined.

[Ryan Luo]

Hi Leonardo, are you suggesting that when running a regression with robust variance estimates, we shouldn't use 'estat esize'? If that's the case, do you know the proper way to calculate effect sizes for this type of analysis? Thank you.

[Leonardo Guizzetti]

I'm saying that you should estimate effect size with a non-robust variance estimate, as that seems to be more aligned with the theoretical approach of a (common) effect size. I have not seen this done in literature, and I'm not sure of the validity of using a robust variance estimation as a plug-in estimate, but that doesn't mean it's not a valid approach.

[George Ford]

I'm sure LG is correct, but it makes no difference in the calculation, at least in the example I provided.

[Ryan Luo]

Thank you both!