Calculate omega reliability coefficient for two variables?

[WJ Wang]

Dear Statalists,

I am currently working on an inventory and want to examine the reliability of each dimension. Since all my items are categorical (Likert scales ranging from 1 to 5), it has been suggested that the McDonald's omega is a better measure of reliability than Cronbach's alpha. However, each dimension of the inventory only has two items, and the user-written -omegacoef- command only allows three or more items per dimension.

Given the situation, I was wondering if I could calculate the omega coefficients by hand using the results of -polychoric- command (or any other commands), which provide the polychoric correlations? If yes, could anyone kindly provide the formula/code for doing so? I know that the -psych- package in R has this function, but I had difficulty finding any Stata equivalents/translating the function to Stata language on my own.

Any assistance from all of you would be greatly appreciated. Thanks a lot in advance!

[Frode Andre]

Dear WJ Wang,

You may find the following thread on Alpha and Omega useful: Reliability coefficient omega and alpha - Statalist

[Bruce Weaver]

. ssc describe omegacoef

----------------------------------------------------------------------------------------------------------------------------------------
package omegacoef from http://fmwww.bc.edu/repec/bocode/o
----------------------------------------------------------------------------------------------------------------------------------------

TITLE
      'OMEGACOEF': module to calculate the omega reliability coefficient

DESCRIPTION/AUTHOR(S)
      
       -omegacoef- calculates McDonald’s omega to estimate scale
      reliability. Omega has many desirable statistical properties that
      make it preferable to the widely used Cronbach’s alpha.
      
      KW: omega
      KW: alpha
      KW: reliability
      KW: psychometrics
      
      Requires: Stata version 12
      
      Distribution-Date: 20210909
      
      Author: Brian Shaw, Indiana University
      Support: email [email protected]
      

INSTALLATION FILES                            (type net install omegacoef)
      omegacoef.ado
      omegacoef.sthlp
----------------------------------------------------------------------------------------------------------------------------------------
(type ssc install omegacoef to install)

[WJ Wang]

Dear WJ Wang,

You may find the following thread on Alpha and Omega useful: Reliability coefficient omega and alpha - Statalist

Dear Frode - Thank you for your reply! I tried this method before, but for my categorical variables the program returns a "Type mismatch" error message. Also, for only two items per dimension, the significance test for the omega coefficient's z-score was super not significant, which is sort of concerning for me. Would there be any other alternatives that you may suggest?

[WJ Wang]

. ssc describe omegacoef

----------------------------------------------------------------------------------------------------------------------------------------
package omegacoef from http://fmwww.bc.edu/repec/bocode/o
----------------------------------------------------------------------------------------------------------------------------------------

TITLE
'OMEGACOEF': module to calculate the omega reliability coefficient

DESCRIPTION/AUTHOR(S)

-omegacoef- calculates McDonald’s omega to estimate scale
reliability. Omega has many desirable statistical properties that
make it preferable to the widely used Cronbach’s alpha.

KW: omega
KW: alpha
KW: reliability
KW: psychometrics

Requires: Stata version 12

Distribution-Date: 20210909

Author: Brian Shaw, Indiana University
Support: email [email protected]


INSTALLATION FILES (type net install omegacoef)
omegacoef.ado
omegacoef.sthlp
----------------------------------------------------------------------------------------------------------------------------------------
(type ssc install omegacoef to install)

Dear Bruce - Thank you for your response! Unfortunately -omegacoef- does not support only two items per dimension, so I'll still be needing a different program...

[Frode Andre]

Dear Frode - Thank you for your reply! I tried this method before, but for my categorical variables the program returns a "Type mismatch" error message. Also, for only two items per dimension, the significance test for the omega coefficient's z-score was super not significant, which is sort of concerning for me. Would there be any other alternatives that you may suggest?

One is advised against using only 2 items, preferably at least 3 items is desired per factor, see Hair, J. et al (2005). Multivariate data analysis (6th ed.). Upper saddle River, New Jersey. Pearson Education International, pages 783-784.
Note that the code in post #12 will produce a number for you. It uses the SEM framework, you may want to look into the variable name convention for these techniques
However, McDonald’s omega intends to provide a measure of reliability. And a factor consisting of only 2 items is oftentimes regarded as unreliable (by definition).

[WJ Wang]

One is advised against using only 2 items, preferably at least 3 items is desired per factor, see Hair, J. et al (2005). Multivariate data analysis (6th ed.). Upper saddle River, New Jersey. Pearson Education International, pages 783-784.
Note that the code in post #12 will produce a number for you. It uses the SEM framework, you may want to look into the variable name convention for these techniques
However, McDonald’s omega intends to provide a measure of reliability. And a factor consisting of only 2 items is oftentimes regarded as unreliable (by definition).

Thanks a lot for the clarification, Frode! Dr. Enzmann, the contributor of the code, also replied in post #19 saying that at least 3 items are required. Since there have been studies reporting omega coefficients based on 2 items (like mentioned in this article), I was just wondering if it's something replicable in Stata. I'll definitely take a better look. Thanks again for your help!

[Joseph Coveney]

From the parallel thread:

. . . I also saw journal articles examining 2-item dimensions with omega coefficients (using, for example, the -psych- package in R, as mentioned in this article), I was wondering if it is possible to find a Stata equivalent.

From #7 above:

. . . there have been studies reporting omega coefficients based on 2 items (like mentioned in this article), I was just wondering if it's something replicable in Stata.

I don't know about that article or about R's psych package, but, yeah, for what it's worth because of the additional constraints (on residual variance) you can apply McDonald's algorithm in Stata to compute an omega with only two indicator variables when they're ordered categorical and you use gsem with the ordered-probit family-link option.

See the illustration below.

For your case begin at the "Begin here" comment; the code that comes before it is just to create the toy dataset used in illustration and to confirm the method for computing McDonald's omega and its confidence bounds.

I've also attached the do file and corresponding log file to this post if you want to explore this further.

version 18.0

log close _all
log using "McDonald Omega with two ordinal indicators.smcl", nomsg name(lo)

clear *

// seedem
set seed 184536679

*
* McDonald's omega with continuous indicator variables
*
quietly drawnorm y1 y2 y3, double corr(1 0.5 0.5 \ 0.5 1 0.5 \ 0.5 0.5 1) n(3000)

sem (y? <- F), nocnsreport nodescribe nofootnote nolog

nlcom (omega: ([y1]F + [y2]F + [y3]F)^2 * [/]var(F) / ///
    (([y1]F + [y2]F + [y3]F)^2 * [/]var(F) + ///
        [/]var(e.y1) + [/]var(e.y2) + [/]var(e.y3))), noheader

// From omegacoef (SSC)
omegacoef y?

*
* Confirmation with corresponding discretized indicator variables
*
forvalues i = 1/3 {
    generate byte x`i' = 1
    forvalues cut = 1/4 {
        quietly replace x`i' = x`i' + 1 if invnormal(`cut' / 5) > y`i'
    }
}
gsem (x? <- F, oprobit), nocnsreport nodvheader nolog
nlcom (omega: ([x1]F + [x2]F + [x3]F)^2 * [/]var(F) / ///
    (([x1]F + [x2]F + [x3]F)^2 * [/]var(F) + 3)), noheader

*
* Begin here
*
// With only two ordered-categorical indicator variables:
gsem (x1 x2 <- F, oprobit), nocnsreport nodvheader nolog
assert e(converged)
nlcom (omega: ([x1]F + [x2]F)^2 * [/]var(F) / ///
    (([x1]F + [x2]F)^2 * [/]var(F) + 2)), noheader

log close lo

exit

I do agree though with the contributors above and in the other thread that a scale with only two items isn't much to hang your hat on.

[WJ Wang]

From the parallel thread:
From #7 above:I don't know about that article or about R's psych package, but, yeah, for what it's worth because of the additional constraints (on residual variance) you can apply McDonald's algorithm in Stata to compute an omega with only two indicator variables when they're ordered categorical and you use gsem with the ordered-probit family-link option.

See the illustration below.

For your case begin at the "Begin here" comment; the code that comes before it is just to create the toy dataset used in illustration and to confirm the method for computing McDonald's omega and its confidence bounds.

I've also attached the do file and corresponding log file to this post if you want to explore this further.

version 18.0

log close _all
log using "McDonald Omega with two ordinal indicators.smcl", nomsg name(lo)

clear *

// seedem
set seed 184536679

*
* McDonald's omega with continuous indicator variables
*
quietly drawnorm y1 y2 y3, double corr(1 0.5 0.5 \ 0.5 1 0.5 \ 0.5 0.5 1) n(3000)

sem (y? <- F), nocnsreport nodescribe nofootnote nolog

nlcom (omega: ([y1]F + [y2]F + [y3]F)^2 * [/]var(F) / ///
(([y1]F + [y2]F + [y3]F)^2 * [/]var(F) + ///
[/]var(e.y1) + [/]var(e.y2) + [/]var(e.y3))), noheader

// From omegacoef (SSC)
omegacoef y?

*
* Confirmation with corresponding discretized indicator variables
*
forvalues i = 1/3 {
generate byte x`i' = 1
forvalues cut = 1/4 {
quietly replace x`i' = x`i' + 1 if invnormal(`cut' / 5) > y`i'
}
}
gsem (x? <- F, oprobit), nocnsreport nodvheader nolog
nlcom (omega: ([x1]F + [x2]F + [x3]F)^2 * [/]var(F) / ///
(([x1]F + [x2]F + [x3]F)^2 * [/]var(F) + 3)), noheader

*
* Begin here
*
// With only two ordered-categorical indicator variables:
gsem (x1 x2 <- F, oprobit), nocnsreport nodvheader nolog
assert e(converged)
nlcom (omega: ([x1]F + [x2]F)^2 * [/]var(F) / ///
(([x1]F + [x2]F)^2 * [/]var(F) + 2)), noheader

log close lo

exit

I do agree though with the contributors above and in the other thread that a scale with only two items isn't much to hang your hat on.

Dear Joseph - Thank you so much for the solution! I'll definitely test it on my data. One quick question: since this solution also essentially follows a SEM framework, it is highly possible that the significance test for the coefficient would not be significant. Should I even bother to look at it, or do I interpret it the same way as other significance tests for coefficients?

Thank you again for your explanation and the code!

[Joseph Coveney]

. . . since this solution also essentially follows a SEM framework, it is highly possible that the significance test for the coefficient would not be significant.

I don't follow you here, sorry. Maybe others on the list know what you're driving at and can respond.

[WJ Wang]

I don't follow you here, sorry. Maybe others on the list know what you're driving at and can respond.

My bad, I tried your code and it worked perfectly (thank you!!!), just that the significance test for the omega coefficient was not significant, and I was wondering how should I interpret it. Thanks again for your help and I'll keep on digging!

[Joseph Coveney]

. . . the significance test for the omega coefficient was not significant, and I was wondering how should I interpret it.

Oh, the z thing; I see now what you're referring to.

I look upon the standard error and confidence bounds as indicators of precision of the estimate. With the idealized 3000-observation artificial dataset in #9 above, the confidence interval went from [0.73, 0.76] with three ordered-categorical indicator variables to [0.36, 0.94] with two. (If you look at the SEM table in the two-indicator case, you can see that the fit is really shaky, not unexpected with a scarcely identified model.) In your case, the reported standard error is tenfold the coefficient, the lower confidence bound is shown in negative territory, and the upper bound exceeds one. If your sample size permits, you might want to consider bootstrapping omega as a crosscheck on what nlcom reports.

I wouldn't bother performing some kind of NHST on omega inasmuch as the null hypothesis implies that there is absolutely zero population correlation between the items, something that I wouldn't have entertained as ontologically tenable.

On a technical note: if the sense of your two items point in the opposite direction, then be sure to use the absolute value of the factor loadings before summing them in computing omega. I didn't show that above (my artificial dataset doesn't require it), but you'll need to take reverse-sense items (reversed-scoring for detecting acquiescence or method bias) into account.

[WJ Wang]

Oh, the z thing; I see now what you're referring to.

I look upon the standard error and confidence bounds as indicators of precision of the estimate. With the idealized 3000-observation artificial dataset in #9 above, the confidence interval went from [0.73, 0.76] with three ordered-categorical indicator variables to [0.36, 0.94] with two. (If you look at the SEM table in the two-indicator case, you can see that the fit is really shaky, not unexpected with a scarcely identified model.) In your case, the reported standard error is tenfold the coefficient, the lower confidence bound is shown in negative territory, and the upper bound exceeds one. If your sample size permits, you might want to consider bootstrapping omega as a crosscheck on what nlcom reports.

I wouldn't bother performing some kind of NHST on omega inasmuch as the null hypothesis implies that there is absolutely zero population correlation between the items, something that I wouldn't have entertained as ontologically tenable.

On a technical note: if the sense of your two items point in the opposite direction, then be sure to use the absolute value of the factor loadings before summing them in computing omega. I didn't show that above (my artificial dataset doesn't require it), but you'll need to take reverse-sense items (reversed-scoring for detecting acquiescence or method bias) into account.

Thank you Joseph for the clarification! Now that makes more sense. I'll definitely double check the bootstrapped results, though I doubt if they're going to be significantly different given the already large CIs. And yes, my two items were measuring the opposite poles of the dimension, so I reverse-coded one of them before applying your code. The two items are theoretically clustered under the same dimension (though with a Spearman's rho = .3 ish in my case, not a great one), so I was just struggling with blending the omega results into the existing evidence.