Effects coding vs. dummy coding - differences in model fit

[Cordula Kiel]

Dear all,

I am trying to run conditional logit models and identified a problem which I do not understand so far.
If I use a main-effects only conditional logit model, pseudo R2 values are the same if I use dummy coding compared to using effects coding.
But if I include interaction effects, pseudo R2 values are better for the dummy-coded model as compared to the effects-coded model.
To my understanding, this should not be the case.
For both models, I created the interaction terms by multiplying the main effects variables with each other.

Does anybody have any idea what might be wrong here? Or does anybody know good literature on that? I googled a lot but did not find any examples on that.

Thanks a lot in advance!

[Richard Williams]

Cordula, it would help to see your code and output. Use code tags; see pt 12 of the FAQ. There may be some substantive reason for what you describe, but it is also possible you made a mistake somewhere and we can't tell without seeing exactly what you did.

[Cordula Kiel]

I am not sure how I should include all that in here because it would be a huge amount of data.
I created a long list of interaction variables out of effects-coded categorial variables (-1; 0; 1) and/or continuous variables by multiplying them (e.g. female*productattribute1).
Then I run a -clogit- model including all the interaction variables and the attribute-specific variables.

Then I did the same thing again - creating variables, but with dummy-coded variables (1, 0) instead and running -clogit- again with the same command.

I am wondering whether it might be a problem that for example if I have two effects coded variables and multiply them to create an interaction term, -1*-1 would be 1 and 1*1 would also be 1, so I am confused how this then gets accounted for in my results (same result but totally different meaning).

[William Lisowski]

Multiplying an effects-coded categorical variable to create an interaction is incorrect technique. Without seeing more of what you've done, I cannot comment further.

[Cordula Kiel]

I took this method from Fairchild & McQuillin (2010): "Evaluating mediation and moderation effects in school psychology: A presentation of methods and review of current practice" who did it the same way. I found no other reference with clear examples on interaction terms of effects-coded variables.

[William Lisowski]

I found the phrase "effects coding" in only one place in the referenced article, and within the numerical example worked, as far as I was able to tell, the moderator variable was not effects-coded.

However, the informative FAQ from UCLA regarding effects coding (http://www.ats.ucla.edu/stat/mult_pk...ral/effect.htm) only briefly discusses using effects-coded indicators for interaction. Assuming what they say is correct, your problem may well be the use of moderator variables in conditional logistic regression, and in particular your apparent understanding that pseudo-R squared for logistic regression behaves like R squared in a linear regression model. This is generally not so, or at best, so-so.

It is not clear to me that using effects coding in the conditional logistic regression context will gain you anything. I would explore using dummy variables created with Stata's factor variable notation (see the output of help factor variable) followed by using the margins command to aid in interpreting your results. Effects coding was, in general, an attempt to get regressions to produce more meaningful output by complicating the inputs, in the absence of tools like margins.

[Richard Williams]

I am inclined to agree with William. Just use dummy variable coding combined with margins and avoid the problem.

But, if you really really want to use effects coding, a pared down example that illustrates the problem might help us better to advise you.

[Cordula Kiel]

Thanks for your replies!
The reason why I would like to use effects coding is because Bech & Gyrd-Hansen (2005) showed that the alternative-specific constant of an opt-out option in cannot be interpreted in a meaningful way when using dummy coding - while it would be possible with effects coding. I did a choice experiment and have an alternative-specific constant for the opt-out option (i.e. "choosing none of the provided alternatives"), and would like to estimate the utility of that option independent from the base levels of the alternative-specific variables.

[William Lisowski]

Back to your original question

Does anybody have any idea what might be wrong here? Or does anybody know good literature on that? I googled a lot but did not find any examples on that.

the answer appears to be "No". It doesn't help that the article you cited is behind a paywall and unavailable for review by those who might be interested in effects coding but want to better understand the advice you are trying to follow.

Alternatively, some Statalist members might be willing to explore further, but in the absence of details from you about your data and its structure, the commands you issued, and the results reported by Stata for both your dummy coded and effects coding, for both the main effects only and the interaction models, we would need psychic powers to render advice beyond suggesting that one possibility is that you have made some sort of error you have overlooked.

If you are interested in increasing the likelihood that Statalist readers will be able to assist you, you should review the Statalist FAQ linked to from the top of the page, as well as from the Advice on Posting link on the page you used to create your post. Note especially sections 9-12 on how to best pose your question. The more you help others understand your problem, the more likely others are to be able to help you solve your problem.

[Cordula Kiel]

Thanks!
I did indeed understand that. I just don't find a useful way to put all these data in here, because I created around 40 interaction terms and to check the coding, probably data for more than one respondent would be needed, and I have several lines per case. If I find a way to put it in here so that it is still manageable to read, I will do so.

It would just have been helpful if anybody knows anything about how to do effects coding with interaction variables, to provide some advice on that. Because the general procedure should to my understanding be independent of the data. I mean if someone says that just multiplying effects coded variables with each other is wrong, then he or she should have an idea on the correct procedure then.

[Cordula Kiel]

For those who are interested in the general question: I also found a report which is available for free, also describing interaction effects for effects coded variables.
Kugler et al. (2012): "Effect coding versus dummy coding in analysis of data from factorial experiments"

[William Lisowski]

So let's return to your original post. My understanding is that you have two conditional logit models - (a) main effects only and (b) main and interaction effects - that you are fitting using two codings of the categorical variables - (1) dummy coding and (2) effects coding. Of the four fits of the model, you find that (a1) and (a2) match your expectations, but (b1) and (b2) do not match your expectations. Your problem is that you do not know if your expectations are incorrect or your programs contain an error, either due to misunderstanding effects coding or to other causes.

I assert that while the choice between (1) and (2) will affect the parameterization and thus the coefficient estimates, both should produce the same predicted values for any combination of the data. So, rather than reading the tea leaves of regression summary statistics like psudo R-squared, I suggest you calculate and compare predicted probabilities. Here's an outline of the steps I envision you needing to take.

1. Fit (a1) and (a2) and use the predict command to produce predicted probabilities for each.
2. Compare the two predictions. The assert command will efficiently do this, but you may see minuscule differences due to rounding. I would do something like assert abs(phat_a1-phat_a2)/max(abs((phat_a1),abs(phat_a2)) < .0001 but this should be the subject of some investigation and experimentation, there may be better comparisons for near-equality.
3. If those two sets of predictions are not equal, go back and find the programming errors in step 1 (including any possible errors in coding the categorical variables) and step 2 and correct them.
4. With that, you've now debugged the code for comparing the results on the case where the results match your expectations.
5. Repeat steps 1 and 2 with (b1) and (b2).
6. If these two sets of predictions are not equal, you've demonstrated that you have a programming error in the code for (b1) and/or (b2). Find it and correct it.

This process might be improved by simplifying your models (a) and (b) to include only two of your categorical variables, one continuous variable (if your models indeed include any continuous variables) and, for (b), just the interaction of the two categorical variables and the interaction of one indicator variable and one categorical variable. If steps 1-6 can be made to work with the simplified models, then you can build back up to the full models, comparing predictions as you go.

[Richard Williams]

I still wish Cordula would give us a simple example that illustrates the problem. You don't need a model with 40 interactions, you just need a very simple model with one interaction. My suspicion is that Cordula has a coding error, but without seeing the coding I could be wrong. Maybe there is some reason that effect coding produces different results when using clogit, but if so I don't know what it is. I'd like to first rule out the possibility of a programming error.

Here is a simple example where I use the xi3 command, which, alas, has been dropped by UCLA; but you can still find it hiding on the web by typing

net describe xi3, from(http://www.ats.ucla.edu/stat/stata/ado/analysis)

. webuse nhanes2f, clear

. xi3: logit diabetes weight [email protected]
e.female          _Ifemale_0-1        (naturally coded; _Ifemale_0 omitted)
e.black           _Iblack_0-1         (naturally coded; _Iblack_0 omitted)

Iteration 0:   log likelihood = -1999.0668  
Iteration 1:   log likelihood = -1962.1763  
Iteration 2:   log likelihood =  -1957.709  
Iteration 3:   log likelihood =  -1957.702  
Iteration 4:   log likelihood =  -1957.702  

Logistic regression                             Number of obs     =     10,335
                                                LR chi2(4)        =      82.73
                                                Prob > chi2       =     0.0000
Log likelihood =  -1957.702                     Pseudo R2         =     0.0207

------------------------------------------------------------------------------
    diabetes |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
      weight |   .0221153   .0027981     7.90   0.000     .0166311    .0275996
   _Iblack_1 |   .2609779   .0627594     4.16   0.000     .1379717    .3839841
   _Ife1Wbl0 |   .2264517   .0535388     4.23   0.000     .1215176    .3313859
   _Ife1Wbl1 |   .0801847   .1140125     0.70   0.482    -.1432757     .303645
       _cons |  -4.443909   .2282746   -19.47   0.000    -4.891319   -3.996499
------------------------------------------------------------------------------

. tab1 _I*

-> tabulation of _Ifemale_1  

   female(1 |
  vs. grand |
      mean) |      Freq.     Percent        Cum.
------------+-----------------------------------
         -1 |      4,909       47.49       47.49
          1 |      5,428       52.51      100.00
------------+-----------------------------------
      Total |     10,337      100.00

-> tabulation of _Iblack_1  

black(1 vs. |
grand mean) |      Freq.     Percent        Cum.
------------+-----------------------------------
         -1 |      9,251       89.49       89.49
          1 |      1,086       10.51      100.00
------------+-----------------------------------
      Total |     10,337      100.00

-> tabulation of _Ife1Wbl0  

   female(1 |
  vs. grand |
    mean) @ |
   black==0 |      Freq.     Percent        Cum.
------------+-----------------------------------
         -1 |      4,409       42.65       42.65
          0 |      1,086       10.51       53.16
          1 |      4,842       46.84      100.00
------------+-----------------------------------
      Total |     10,337      100.00

-> tabulation of _Ife1Wbl1  

   female(1 |
  vs. grand |
    mean) @ |
   black==1 |      Freq.     Percent        Cum.
------------+-----------------------------------
         -1 |        500        4.84        4.84
          0 |      9,251       89.49       94.33
          1 |        586        5.67      100.00
------------+-----------------------------------
      Total |     10,337      100.00

. xi3: logit diabetes weight [email protected]
i.female          _Ifemale_0-1        (naturally coded; _Ifemale_0 omitted)
i.black           _Iblack_0-1         (naturally coded; _Iblack_0 omitted)

Iteration 0:   log likelihood = -1999.0668  
Iteration 1:   log likelihood = -1962.1763  
Iteration 2:   log likelihood =  -1957.709  
Iteration 3:   log likelihood =  -1957.702  
Iteration 4:   log likelihood =  -1957.702  

Logistic regression                             Number of obs     =     10,335
                                                LR chi2(4)        =      82.73
                                                Prob > chi2       =     0.0000
Log likelihood =  -1957.702                     Pseudo R2         =     0.0207

------------------------------------------------------------------------------
    diabetes |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
      weight |   .0221153   .0027981     7.90   0.000     .0166311    .0275996
   _Iblack_1 |   .6682228   .1863494     3.59   0.000     .3029848    1.033461
   _Ife1Wbl0 |   .4529035   .1070776     4.23   0.000     .2430352    .6627718
   _Ife1Wbl1 |   .1603694    .228025     0.70   0.482    -.2865513    .6072901
       _cons |  -4.931339   .2417952   -20.39   0.000    -5.405249   -4.457429
------------------------------------------------------------------------------

. tab1 _I*

-> tabulation of _Ifemale_1  

   female=1 |      Freq.     Percent        Cum.
------------+-----------------------------------
          0 |      4,909       47.49       47.49
          1 |      5,428       52.51      100.00
------------+-----------------------------------
      Total |     10,337      100.00

-> tabulation of _Iblack_1  

    black=1 |      Freq.     Percent        Cum.
------------+-----------------------------------
          0 |      9,251       89.49       89.49
          1 |      1,086       10.51      100.00
------------+-----------------------------------
      Total |     10,337      100.00

-> tabulation of _Ife1Wbl0  

 female=1 @ |
   black==0 |      Freq.     Percent        Cum.
------------+-----------------------------------
          0 |      5,495       53.16       53.16
          1 |      4,842       46.84      100.00
------------+-----------------------------------
      Total |     10,337      100.00

-> tabulation of _Ife1Wbl1  

 female=1 @ |
   black==1 |      Freq.     Percent        Cum.
------------+-----------------------------------
          0 |      9,751       94.33       94.33
          1 |        586        5.67      100.00
------------+-----------------------------------
      Total |     10,337      100.00

.

[William Lisowski]

A little too late to add in edit, but I realize now that in post #12 above, the absolute values in the denominator are unneeded, since I recommended predicting probability rather than the linear combination xb, which also avoids worrying about division by zero.

assert abs(phat_a1-phat_a2)/max(phat_a1,phat_a2) < .0001

[Richard Williams]

Incidentally, looking at xi3 more closely, I like this coding better.

. webuse nhanes2f, clear

. xi3: logit diabetes weight e.female*e.black, nolog
e.female          _Ifemale_0-1        (naturally coded; _Ifemale_0 omitted)
e.black           _Iblack_0-1         (naturally coded; _Iblack_0 omitted)

Logistic regression                             Number of obs     =     10,335
                                                LR chi2(4)        =      82.73
                                                Prob > chi2       =     0.0000
Log likelihood =  -1957.702                     Pseudo R2         =     0.0207

------------------------------------------------------------------------------
    diabetes |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
      weight |   .0221153   .0027981     7.90   0.000     .0166311    .0275996
  _Ifemale_1 |   .1533182   .0632612     2.42   0.015     .0293286    .2773078
   _Iblack_1 |   .2609779   .0627594     4.16   0.000     .1379717    .3839841
   _Ife1Xbl1 |  -.0731335   .0626949    -1.17   0.243    -.1960133    .0497462
       _cons |  -4.443909   .2282746   -19.47   0.000    -4.891319   -3.996499
------------------------------------------------------------------------------

. xi3: logit diabetes weight i.female*i.black, nolog
i.female          _Ifemale_0-1        (naturally coded; _Ifemale_0 omitted)
i.black           _Iblack_0-1         (naturally coded; _Iblack_0 omitted)

Logistic regression                             Number of obs     =     10,335
                                                LR chi2(4)        =      82.73
                                                Prob > chi2       =     0.0000
Log likelihood =  -1957.702                     Pseudo R2         =     0.0207

------------------------------------------------------------------------------
    diabetes |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
      weight |   .0221153   .0027981     7.90   0.000     .0166311    .0275996
  _Ifemale_1 |   .4529035   .1070776     4.23   0.000     .2430352    .6627718
   _Iblack_1 |   .6682228   .1863494     3.59   0.000     .3029848    1.033461
   _Ife1Xbl1 |  -.2925341   .2507796    -1.17   0.243     -.784053    .1989849
       _cons |  -4.931339   .2417952   -20.39   0.000    -5.405249   -4.457429
------------------------------------------------------------------------------

. logit diabetes weight i.female##i.black, nolog

Logistic regression                             Number of obs     =     10,335
                                                LR chi2(4)        =      82.73
                                                Prob > chi2       =     0.0000
Log likelihood =  -1957.702                     Pseudo R2         =     0.0207

------------------------------------------------------------------------------
    diabetes |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
      weight |   .0221153   .0027981     7.90   0.000     .0166311    .0275996
    1.female |   .4529035   .1070776     4.23   0.000     .2430352    .6627718
     1.black |   .6682228   .1863494     3.59   0.000     .3029848    1.033461
             |
female#black |
        1 1  |  -.2925341   .2507796    -1.17   0.243     -.784053    .1989849
             |
       _cons |  -4.931339   .2417952   -20.39   0.000    -5.405249   -4.457429
------------------------------------------------------------------------------

.