Reliability coefficient omega and alpha

[Amelie Nickel]

Dear all,

using

ssc install omega

, I calculated the reliability coefficient omega and alpha, to test the correlation of 6 items. I was wondering why the two coefficients differ so much (alpha = 0.5651 and omega = 0.9352). Can this be correct at all?

Thanks in advance

[Dirk Enzmann]

Can you use -dataex- (see FAQ #12.2) to let us understand the data better?

[Amelie Nickel]

Thank you for the advice, hopefully this is clarifying it a bit more. It is cross-sectional data including 27 countries. The example dataset includes 5 countries ("AT" Austria, "BE" Belgium, "BG" Bulgaria, "CH" Switzerland, "CY" Cyprus) and the (aggregated) variables tested for correlation.

* Example generated by -dataex-. To install: ssc install dataex
clear
input str2 cntry float(c_votu_2018 c_demoindex_2018 c_cpi_2018 polintr_r_ag trstplt_ag trstprt_ag trstprl_ag)
"AT" 75.59 8.29 76  2.545818 4.2409444 4.1343956 5.419935
"BE" 88.38 7.78 75 2.3397508   4.25698 4.2588367 4.794974
"BG" 53.85 7.03 42 2.0289521 2.1160924 2.1438031 2.485391
"CH" 45.12 9.03 85  2.572727  5.369314   5.24723 6.399862
"CY" 66.74 7.59 59 1.9961587  2.829205  2.733333 3.707412
end
label var cntry "Country" 
label var c_votu_2018 "Voter Turnout last parliamentary election" 
label var c_demoindex_2018 "Democracy Index 2018" 
label var c_cpi_2018 "Corruption Perceptions Index 2018" 
label var polintr_r_ag "How interested in politics_recoded_aggregated" 
label var trstplt_ag "Trust in politicians_aggregated" 
label var trstprt_ag "Trust in political parties_aggregated" 
label var trstprl_ag "Trust in country's parliament_aggregated"

To measure alpha and omega coefficients I used the following commands:

 alpha c_votu_2018 c_demoindex_2018 c_cpi_2018 polintr_r_ag trstplt_ag trstprt_ag trstprl_ag

omega c_votu_2018 c_demoindex_2018 c_cpi_2018 polintr_r_ag trstplt_ag trstprt_ag trstprl_ag

[Dirk Enzmann]

Unfortunately, the data set is too small to achieve convergence using -omega-, thus I can't check the issue as I would like to (can you provide us with an example data set with enough cases to test -omega-?).

However, here is an alternative to -omega- defining the .ado-program -omega2-, perhaps you can try whether it shows the same problem you encountered with -omega- (note that the program forces the loadings to be positive, it does not check for negative correlated items in the same way as -omega- and -alpha do):

/* Calculate omega (see Baldwin, 2019, p. 341) ignoring negative loadings

   Reference:

   Baldwin, S. A. (2019). Psychological Statistics and Psychometrics
      using Stata. College Station, TX: Stata Press.
*/      

program omega2, rclass
   version 13
   syntax varlist [if] [in]
   marksample touse
   qui count if `touse'
   local N = r(N)
   local nvars : word count `varlist'
   local i = 0
   local bcom = "("
   local buni = "("
   foreach v of varlist `varlist' {
      local ++i
      if `i' < `nvars' {
         local bcom = "`bcom'" + "abs(_b[`v':SCORE]) + "
         local buni = "`buni'" + "_b[/var(e.`v')] + "
      }
      else {
         local bcom = "`bcom'" + "abs(_b[`v':SCORE]))^2*_b[/var(SCORE)]"
         local buni = "`buni'" + "_b[/var(e.`v')])"
      }
   }
   qui sem (SCORE -> `varlist') if `touse', var(SCORE@1)
   nlcom(omega:`bcom'/(`bcom' + `buni')), post noheader
   return scalar omega = e(b)[1,1]
   return scalar k = `nvars'
   return scalar N = `N'
end

[Dirk Enzmann]

I had a closer look at -omega- from SSC and found that -omega- calculates omega using standardized items, whereas you compared this measure of reliability to alpha using unstandardized items. If the variances of the items differ, reliability measures based on unstandardized and standardized items may be vastly different, especially if you compare alpha (unstandardized) to omega (standardized).

Here is a different version of -omega2- that allows to calculate omega based on unstandardized or based on standardized items:

/* Calculate omega (see Baldwin, 2019, p. 341) ignoring negative loadings

   Reference:
   
   Baldwin, S. A. (2019). Psychological Statistics and Psychometrics
      using Stata. College Station, TX: Stata Press.
*/      

program omega2, rclass
   version 13
   syntax varlist [if] [in] [, Std]
   marksample touse
   qui count if `touse'
   local i = 0
   local vlist = ""
   foreach v of varlist `varlist' {
      local ++i
      tempvar `v'`i'
      if "`std'" == "" qui gen ``v'`i'' = `v'
      else egen ``v'`i'' = std(`v') if `touse'
      local vlist = "`vlist'" + " ``v'`i''"
   }
   local N = r(N)
   local nvars : word count `varlist'
   local i = 0
   local bcom = "("
   local buni = "("
   foreach v of varlist `varlist' {
      local ++i
      if `i' < `nvars' {
         local bcom = "`bcom'" + "abs(_b[``v'`i'':SCORE]) + "
         local buni = "`buni'" + "_b[/var(e.``v'`i'')] + "
      }
      else {
         local bcom = "`bcom'" + "abs(_b[``v'`i'':SCORE]))^2*_b[/var(SCORE)]"
         local buni = "`buni'" + "_b[/var(e.``v'`i'')])"
      }
   }
   qui sem (SCORE -> `vlist') if `touse', var(SCORE@1) iterate(1000) dif
   if "`std'" == "" di _n as txt "unstandardized items"
   else di _n as txt "standardized items"
   di as txt "number of items in the scale: " as res %3.0f `nvars'
   nlcom(omega:`bcom'/(`bcom' + `buni')), post noheader
   return scalar omega = e(b)[1,1]
   return scalar k = `nvars'
   return scalar N = `N'
end

When comparing the reliability of alpha and omega both using unstandardized items (or both using unstandardized items), the measures become more similar:

. sysuse auto
(1978 Automobile Data)

. alpha price trunk weight length

Test scale = mean(unstandardized items)

Average interitem covariance:     214279.3
Number of items in the scale:            4
Scale reliability coefficient:      0.2887

. omega2 price trunk weight length

unstandardized items
number of items in the scale:   4

------------------------------------------------------------------------------
             |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
       omega |   .4217923   .0995086     4.24   0.000     .2267591    .6168255
------------------------------------------------------------------------------

. alpha price trunk weight length, std

Test scale = mean(standardized items)

Average interitem correlation:      0.6049
Number of items in the scale:            4
Scale reliability coefficient:      0.8596

. omega2 price trunk weight length, std

standardized items
number of items in the scale:   4

------------------------------------------------------------------------------
             |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
       omega |   .8806162   .0231515    38.04   0.000     .8352401    .9259923
------------------------------------------------------------------------------

[ericmelse]

Dear Professor Enzmann,

Note that Philip B. Ender, formerly affiliated with UCLA Academic Technology Services, Statistical Computing and Consulting, coded and distributed a package with the same name omega2, see here.
'It is a program to compute omega squared after an ANOVA.'
I mention this just in case people who start using your code in #5 bump into conflicts using the ado file with the same name.
Possibly, when you consider extending your code into a distributable package, you better give it another name.

[Dirk Enzmann]

Thanks a lot Eric for alerting me!

But before I will contribute this to SSC I have to change and add some things, e.g. add an option not to reverse variables negatively correlated with the total score, report reversed variables, report average correlation/covariance, calculate item-total-correlations, item-rest-correlations and omega if single items are deleted one by one, allow to generate an item mean score better than -alpha- does (which simply multiplies items by -1 and then calculates the item mean scores which is somewhat silly with Likert-like variables reversed in that manner), write a help-file etc.

[ericmelse]

Sounds very exciting to me that you continue on this project. An important and most welcome contribution.

[Amelie Nickel]

Thank you a lot Dirk Enzmann for your detailed and very helpful information and for providing the -omega2-program and also thanks ericmelse for your hint.
So first, here is the example dataset including more cases.

* Example generated by -dataex-. To install: ssc install dataex
clear
input float(c_votu_2018 c_demoindex_2018 c_cpi_2018 polintr_r_ag trstplt_ag trstprt_ag trstprl_ag)
75.59 8.29 76  2.545818 4.2409444 4.1343956  5.419935
88.38 7.78 75 2.3397508   4.25698 4.2588367  4.794974
53.85 7.03 42 2.0289521 2.1160924 2.1438031  2.485391
45.12 9.03 85  2.572727  5.369314   5.24723  6.399862
66.74 7.59 59 1.9961587  2.829205  2.733333  3.707412
60.84 7.69 59 1.8854516  3.561463  3.552486 4.1784177
76.15 8.68 80  2.863386  3.964087 3.9927225   5.09884
 84.6 9.22 88 2.8712556  5.201153  5.304516  6.166667
63.67 7.97 73 2.2904413  3.885411  3.748538   4.85873
71.76 8.08 58  2.280312  2.551956 2.4960294  4.082754
86.73 9.14 85  2.618586  4.901611  5.031774  5.913941
 48.7  7.8 72 2.3945136  3.539634 3.0397756 4.1369233
67.55 8.53 80  2.575771  3.434325  3.497243 4.2066913
 46.9 6.57 48 2.0254283  1.681111 1.8780624 2.2525084
69.67 6.63 46 1.9843185  3.925355  3.779421  4.542258
62.77 9.15 73  2.501356  3.929792  3.819322  4.616361
 91.2 9.58 76  2.714452  4.488784 4.3079667  5.172738
72.93 7.71 52 2.0398247  3.043136  2.927362  4.250753
 47.8  7.5 59  2.142701  3.106845 2.8270845 3.2979546
54.58 7.38 58 2.2835333  2.689616  2.554039  3.359639
81.93 8.89 82 2.6632776  5.385455  5.384569  5.925473
78.22 9.87 84       2.6  5.280401  5.401004     6.755
61.74 6.67 60  2.184193 3.1107266  3.163548  3.830357
48.57 7.84 64  2.364929 2.7301435 2.8095694 4.1494594
48.93 6.41 39 1.9345795  2.821682  2.627907 3.9335024
87.18 9.39 85 2.9089134  4.909804  4.988197  6.167759
52.64  7.5 60 2.2013679  2.670254  2.702012  3.584891
65.81  7.1 50 2.2218115  3.348924  3.507003  3.711754
end

All Items are positively correlated and when using standardized items, the alpha and omega coefficients are almost the same (although still surprisingly high). Based on unstandardized items the coefficients do still differ a bit, similar to your example.

. alpha c_votu_2018 c_demoindex_2018 c_cpi_2018 polintr_r_ag trstplt_ag trstprt_ag trstprl_ag, std

Test scale = mean(standardized items)

Average interitem correlation:      0.7695
Number of items in the scale:            7
Scale reliability coefficient:      0.9590

. omega2_2 c_votu_2018 c_demoindex_2018 c_cpi_2018 polintr_r_ag trstplt_ag trstprt_ag trstprl_ag, std

standardized items
number of items in the scale:   7

------------------------------------------------------------------------------
             |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
       omega |   .9518519   .0003432  2773.13   0.000     .9511791    .9525246
------------------------------------------------------------------------------

. alpha c_votu_2018 c_demoindex_2018 c_cpi_2018 polintr_r_ag trstplt_ag trstprt_ag trstprl_ag

Test scale = mean(unstandardized items)

Average interitem covariance:     9.793409
Number of items in the scale:            7
Scale reliability coefficient:      0.5880

. omega2_2 c_votu_2018 c_demoindex_2018 c_cpi_2018 polintr_r_ag trstplt_ag trstprt_ag trstprl_ag

unstandardized items
number of items in the scale:   7

------------------------------------------------------------------------------
             |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
       omega |    .784228   .0015795   496.51   0.000     .7811322    .7873237
------------------------------------------------------------------------------

As far as I know, there is some research (e.g. Trizano-Hermosilla & Alvarado 2016) that advocates omega to test reliability instead of alpha. Referring on that, I would use -omega2- based on standardized items.
Would you agree with me on that (if I may add this short question)?



Trizano-Hermosilla, I., & Alvarado, J. M. (2016). Best Alternatives to Cronbach's Alpha Reliability in Realistic Conditions: Congeneric and Asymmetrical Measurements. Frontiers in psychology, 7, 769. https://doi.org/10.3389/fpsyg.2016.00769

[Dirk Enzmann]

I can't say more than what we can find already in Trizano-Hermozilla & Alvarado's paper. If I understand their conclusion correctly, they are arguing that with normally distributed items omega shows least bias and that the alternative measures of reliability (GLB -- greatest lower bound -- and GLBa -- greatest lower bound algebraically) show (too) large positive bias. However, applied to shorter scales with skewed items (and applied to longer scales with about 50% skewed items) the bias of GLB and GLBa becomes negative and is far less than the negative bias of omega. In comparsion to alpha, omega always performs (slightly) better than alpha.

Note that if you want to construct additive (mean) scores of your items you should consider whether to standardize the items because their variances differ substantially. Don't forget to recode (reverse) items that correlate negatively with the total score -- and you should never use the -generate- option of -alpha- with unstandardized items (see #7).

[Amelie Nickel]

I will keep that in mind and use standardized items.
Thank you again Dirk Enzmann for your helpful support and for not getting tired of pointing out the FAQ and correct forms of posting in Statalist and even more thanks for replying even though those were not followed correctly.

[Yapeng Wang]

Unfortunately, the data set is too small to achieve convergence using -omega-, thus I can't check the issue as I would like to (can you provide us with an example data set with enough cases to test -omega-?).

However, here is an alternative to -omega- defining the .ado-program -omega2-, perhaps you can try whether it shows the same problem you encountered with -omega- (note that the program forces the loadings to be positive, it does not check for negative correlated items in the same way as -omega- and -alpha do):

/* Calculate omega (see Baldwin, 2019, p. 341) ignoring negative loadings

Reference:

Baldwin, S. A. (2019). Psychological Statistics and Psychometrics
using Stata. College Station, TX: Stata Press.
*/

program omega2, rclass
version 13
syntax varlist [if] [in]
marksample touse
qui count if `touse'
local N = r(N)
local nvars : word count `varlist'
local i = 0
local bcom = "("
local buni = "("
foreach v of varlist `varlist' {
local ++i
if `i' < `nvars' {
local bcom = "`bcom'" + "abs(_b[`v':SCORE]) + "
local buni = "`buni'" + "_b[/var(e.`v')] + "
}
else {
local bcom = "`bcom'" + "abs(_b[`v':SCORE]))^2*_b[/var(SCORE)]"
local buni = "`buni'" + "_b[/var(e.`v')])"
}
}
qui sem (SCORE -> `varlist') if `touse', var(SCORE@1)
nlcom(omega:`bcom'/(`bcom' + `buni')), post noheader
return scalar omega = e(b)[1,1]
return scalar k = `nvars'
return scalar N = `N'
end

Dear Dirk, thank you for providing the command. I am using your command to calculate omega coefficient. But Stata 16.1 (MP) returned the following error message. I was wondering if you could help me out.

* Example generated by -dataex-. To install: ssc install dataex
clear
input byte(Q158_rvs Q159_rvs Q160 Q161 Q162 Q163_rvs)
 9 7  .  9  3  7
 8 6  .  .  1  5
 1 .  .  1  5  3
 1 3  9  7  7  1
 4 3  7  .  2  3
 2 1 10  1  1  4
 3 1 10  9  2  1
 1 5  8  7  4  4
 1 1 10 10  1  1
 1 1 10  3  1  1
 2 2  7  7  6  4
 3 4  8  5  7  3
 3 5  7  8  6  5
 2 4  5  4  7  5
 3 2  5  4  9  3
 2 1  2  1  5  3
 6 1  8  1  7  6
 8 3  1  1  5  4
 6 5  6  3  7  7
 1 3  5  5  3  4
 4 1  8  8 10  1
 4 3  6  9  9  5
 3 2  7  6  7  7
 4 3  5  5 10  5
 3 4  8  5  6 10
 1 1  9  9  9  5
 1 5  8 10  8  3
 1 1 10  1 10  1
 1 1 10  5  1  4
 1 4  7 10  2  4
 4 4  2  1  1  6
10 7 10  8  5  6
 4 4 10  5  5  4
 7 8  9  9  2  3
 8 2  6  9  5  3
 4 7  8  .  1  4
 5 5  6  6  6  4
 7 5  6  5  6  7
 4 4  4  2  5  7
 8 7  .  . 10  3
 3 3  9  2  1  1
 1 4 10  4  1  1
 1 1 10 10  1  1
 2 1 10  6  4  2
 1 1  1  7  1  1
 1 1 10  3  2  1
 1 1  8  2  2  1
 3 4  9  7  8  3
 3 2  5  4  3  3
 3 2  7  5  3  3
 7 7  5  5  4  3
 3 3  5  6  8  4
 2 2  9  6  3  5
 3 3  7  9  9  3
 2 1  1  1  8  3
 3 1  7  6  7  3
 1 1  7  5  4  .
 . .  .  .  .  .
 6 2  5  1  4  7
 3 1  6  3  6  6
 1 1  8  3  5  4
 1 1  3  1  1  2
 8 2  9  4  5  7
 1 1 10  7  8  4
 3 1  4  5  6  .
 6 6  5  5  5  4
 3 3  3  5  1  9
 3 1  5  7  8  3
 1 6  5  3  5  3
 6 3 10  5  3  3
 5 6  6  5  5  3
 6 6  5  5  5  2
 3 3  5  5  5  3
 5 5  1  1  1  2
 1 1  5  4  4  3
 1 1 10  6 10  1
 2 1  9  8  9  2
 1 1 10  1  1  1
 1 1  3  1  1  4
 1 1 10  1  5  2
 4 4  6  1  5  5
 1 1  5  1  1  3
 2 2  5  1  1  3
 1 1  9  2  3  4
 1 7 10  1  9  3
 1 1  7  9  3  4
 4 1  9 10  9  5
 3 1 10  7  5  5
 1 2  9 10 10  3
 6 3 10 10  5  5
 1 2  9 10  9  3
 2 1 10  2  8  4
 2 1 10  7 10  2
 1 2  9 10  9  3
 1 2 10  9 10  4
 6 6  3  5  5  8
 4 3  7  7  7  6
 4 4  8  7  7  4
 1 1  5  1  1  5
 1 1  5  4  4  6
end
label values Q158_rvs Q158_rvs_L
label def Q158_rvs_L 1 "completely agree", modify
label def Q158_rvs_L 10 "completely disagree", modify
label values Q159_rvs Q159_rvs_L
label def Q159_rvs_L 1 "completely agree", modify
label values Q160 Q160
label def Q160 1 "Completely disagree", modify
label def Q160 2 "2", modify
label def Q160 3 "3", modify
label def Q160 4 "4", modify
label def Q160 5 "5", modify
label def Q160 6 "6", modify
label def Q160 7 "7", modify
label def Q160 8 "8", modify
label def Q160 9 "9", modify
label def Q160 10 "Completely agree", modify
label values Q161 Q161
label def Q161 1 "Completely disagree", modify
label def Q161 2 "2", modify
label def Q161 3 "3", modify
label def Q161 4 "4", modify
label def Q161 5 "5", modify
label def Q161 6 "6", modify
label def Q161 7 "7", modify
label def Q161 8 "8", modify
label def Q161 9 "9", modify
label def Q161 10 "Completely agree", modify
label values Q162 Q162
label def Q162 1 "Completely disagree", modify
label def Q162 2 "2", modify
label def Q162 3 "3", modify
label def Q162 4 "4", modify
label def Q162 5 "5", modify
label def Q162 6 "6", modify
label def Q162 7 "7", modify
label def Q162 8 "8", modify
label def Q162 9 "9", modify
label def Q162 10 "Completely agree", modify
label values Q163_rvs Q163_rvs_L
label def Q163_rvs_L 1 "a lot better off", modify
label def Q163_rvs_L 10 "a lot worse off", modify

[Joseph Coveney]

I am using your command to calculate omega coefficient. But Stata 16.1 (MP) returned the following error message.

Answered here.

[Dirk Enzmann]

I can't replicate the error message with the data you provided.

Some comments:

• Note that -omega2- as shown in #4 and #5 is an ad-hoc program that didn't make it into a quasi official .ado- (such as .ado programs at SSC).
• -omega2- as shown in #5 allows to calculate omega based on unstandardized items (option "std"), whereas omega in #4 only produces omega based on standardized items, you should have used the version in #5. If you want omega based on unstandardized items, you better use -omega- from SSC. Will this produce your error message, as well?
• When using -omega- (from SSC) you should take care that all items correlate positively with the total score of the remaining items (see the -omega- options "reverse" or "noreverse"). Using the data you provided this is not the case with Q160, Q161, and Q162 as

  alpha _all, item

  shows.
• If you want to compare omega with alpha, additionally you should use standardized items and the option of -alpha- to ignore cases with missing items (option "casewise"), e.g.

  alpha _all, item case std

  , or you should use -omega- with the option "usemissing".

[Yapeng Wang]

I can't replicate the error message with the data you provided.

Some comments:

• Note that -omega2- as shown in #4 and #5 is an ad-hoc program that didn't make it into a quasi official .ado- (such as .ado programs at SSC).
• -omega2- as shown in #5 allows to calculate omega based on unstandardized items (option "std"), whereas omega in #4 only produces omega based on standardized items, you should have used the version in #5. If you want omega based on unstandardized items, you better use -omega- from SSC. Will this produce your error message, as well?
• When using -omega- (from SSC) you should take care that all items correlate positively with the total score of the remaining items (see the -omega- options "reverse" or "noreverse"). Using the data you provided this is not the case with Q160, Q161, and Q162 as

  alpha _all, item

  shows.
• If you want to compare omega with alpha, additionally you should use standardized items and the option of -alpha- to ignore cases with missing items (option "casewise"), e.g.

  alpha _all, item case std

  , or you should use -omega- with the option "usemissing".

Thank you for your additional comments. Joseph found it out. Both commands only work for lower case variable names.