Simple Multilevel Model: Random Vs Fixed part

[Frank Sulley]

Hi everyone, first time poster here and I hope that you can help me with a very simple multilevel model as follow:

Level 2 IV: lawyer (lawyer gender, lawyer specialty)
Level 1 IV: multiple customer reviews for each lawyer (lawyer empathy assessed at the review level)
Cross level interaction: gender X empathy
Level 1 DV: lawyer helpfulness assessed at the review level

This is the code I thought I would be using:

mixed helpfulness i.l_gender##c.r_empathy i.specialty || lawyer_id: r_empathy, vce(cluster lawyer_id)

Additionally, is

|| lawyer_id: r_empathy

correct in the model if I assume that gender does not vary, but empathy does (in each single review of the lawyer) ?

Thanks in advance for any help !

[Daniel Schaefer]

I don't think that's quite right. It seems to me that this model assumes that the first-order effect of empathy on helpfulness depends on some unmeasured feature of the lawyer - so you allow the size of the effect to vary depending on the lawyer. That's the random effect added by the second-level r_empathy term. That said, you are already explicitly using a measured feature of the lawyer to explain why the effect of r_empathy on helpfulness differs by the Lawyer - it differs based on the gender of the lawyer. I think there are a few possibilities here:

1. Lawyer's gender completely (or almost completely) explains why the effect of empathy on helpfulness differs from lawyer to lawyer. In that case, the variance term for the second-level r_empathy should be very small, or the model wont converge at all. You should omit the second-level random effect in this case.

2. Lawyer's gender partially explains why the effect of empathy on helpfulness differs from lawyer to lawyer. There are still unmeasured differences between lawyers that modify the size of this effect. You have the right model.

3. Gender doesn't explain differences in the effect size of empathy on helpfulness. The size of the gender interaction term is very small. If the variance term on the second-level random effect is also relatively small, there aren't differences between lawyers in terms of the size of the effect of empathy on helpfulness, although it is still possible that the group means are different, and therefore a fixed-effects only model is more appropriate.

Finally, you don't need vce(cluster lawyer_id) here. You're already explicitly modeling this with the mixed effects model, so it is at best redundant and at worst incorrect.

[Daniel Schaefer]

It seems that there is a bit of debate on whether its appropriate to exclude level-1 random effects in cross-level interactions. See:

https://www.researchgate.net/post/Cross-level-interaction-without-specifying-a-random-slope-for-the-Level-1-variable

and the links and resources posted in that thread.

See also:

https://doi.org/10.1093/esr/jcy053

[Erik Ruzek]

This is what Raudenbush and Bryk call a "slopes as outcome" model, in which the variation in the slopes of empathy on helpfulness at the lawyer level are predicted by a lawyer-level characteristic (gender). Practically, this is implemented via the model proposed in #1 (note that it is customary to allow the random slope and intercept to covary and you can do so by adding covariance(unstructured) to the mixed statement after the comma).

The simulation study linked in #3 shows that if you do not include the random slope in the model, then the test of significance of the cross-level interaction is "anti-conservative," meaning you are more likely to reject the null. Table 2 in that paper shows that no matter the amount of variation in the random slope, you only get acceptable coverage rates for the cross-level interaction when you include the random slope. There is no case, no matter the conditions, when not including the random slope leads to remotely acceptable coverage rates.

Accordingly, as their title indicates, you should always include the random slope when testing a cross-level interaction in a multilevel model.

[Frank Sulley]

Thanks everybody for your insights, and I am sorry, but I believe that by using empathy I might have created a bit of confusion.
Let's assume that "lawyer tidy appearance" at the review level is the main variable (instead of empathy, and assuming again that in the review of the lawyer, that day for that specific customer he/she looked tidy or not).
In this sense I would expect "tidy appearance" to vary among reviews and within each single lawyer.
Would the following model apply ?

 mixed helpfulness i.l_gender##c.r_appearance i.specialty || lawyer_id: r_appearance

Or should I use instead

mixed helpfulness i.l_gender##c.r_appearance i.specialty || lawyer_id:

Once again here I am assuming that the effect of lawyer tidy appearance for that customer influences perceptions of lawyer helpfulness for that customer depending on the lawyer gender.

[Clyde Schechter]

If you include the random slope on r_appearance, you are estimating a model in which the effect of appearance on helpfulness varies across lawyers. You did not mention any such expectation in your description, so I assume you want the model without the random slope.

In the model without the random slope, the appearance varies across reviews of the same lawyer, and it can vary across lawyers as well. What is constrained to be the same for all lawyers, however, is the effect of appearance on helpfulness. This structure of relationships would appear to correspond best to what you said in #5. Of course, when we try to put our models into words we often don't quiet convey our meaning exactly.

Added: And I am not conveying my meaning above exactly! Because the bottom level of each model contains an interaction between the lawyer's sex and the reviewer's appraisal of appearance, this, alone, causes the effect of appearance on helpfulness to vary among lawyers to the extent that it differs between male and female lawyers. However, in the model without random slopes, the only way in which the effect of appearance on helpfulness differs between two lawyers is by virtue of them being different sexes. For all male lawyers, the effect of appearance on helpfulness is the same. And all female lawyers exhibit the same effect of appearance on helpfulness, though that may differ from the effect exhibited by all males.

If you add the random slope to the model, then in addition to sex differences among lawyers inducing a difference in the effect of appearance on helpfulness, each lawyer may, due to unobserved (or at least unmodeled) factors, have an idiosyncratic effect of appearance on helpfulness. Given two lawyers of the same sex, their effects of appearance on sex may differ.

[Frank Sulley]

Wow, thanks a lot Clyde, now my head is really spinning. I was indeed simply interested in seeing if the effect of appearance on helpfulness was conditional to the gender of the lawyer.
Since I would assume that appearance on helpfulness should vary between and within lawyers, my bet would be a random slope model.

Coming from a different perspective, I originally started testing the model with fixed effect, which would do what I needed.
But unfortunately because of the fe, I could not quantify the coefficient of gender (omitted in the regression) and I couldn't figure out how to run a simple slope test and plotting.

reghdfe helpfulness i.l_gender##c.r_appearance, absorb(i.specialty lawyer_id)

[Frank Sulley]

Given two lawyers of the same sex, their effects of appearance on sex may differ.

I am inclined toward the random model because each different customer might assign to lawyer appearance a different weight (e.g., based on different taste) even "within the same lawyer".
Is this in line with what you were suggesting when using

 
 mixed helpfulness i.l_gender##c.r_appearance i.specialty || lawyer_id: r_appearance

[Erik Ruzek]

I would encourage you to add the covariance between the r_appearnce slopes and lawyer_id intercepts to your model unless you have very strong reason to believe they do not covary. The small addition to your code is the following:

mixed helpfulness i.l_gender##c.r_appearance i.specialty || lawyer_id: r_appearance, cov(un)

[Frank Sulley]

I would encourage you to add the covariance between the r_appearnce slopes and lawyer_id intercepts to your model unless you have very strong reason to believe they do not covary. The small addition to your code is the following:

mixed helpfulness i.l_gender##c.r_appearance i.specialty || lawyer_id: r_appearance, cov(un)

Thanks Erik, I tried but the model doesnt converge
Iteration 7: log likelihood = -30611.903 (backed up)
could not calculate numerical derivatives -- discontinuous region with missing values encountered
could not calculate numerical derivatives -- discontinuous region with missing values encountered
r(430);

EDIT:
It does converge only if eliminate the r_appereance as following

mixed helpfulness i.l_gender##c.r_appearance i.specialty || lawyer_id: , cov(un)

[Daniel Schaefer]

I want to be clear that I don't think the paper linked in #3 is the final word on the matter. The paper essentially gives an argument about omitted variable bias. I think its reasonable to advise people that, if you expect some level-2 variable to explain differences in the strength of a relationship for level-1 units, there may be reason to think other unaccounted for features of the level-2 clusters help to explain those level-1 differences. That said, all of the models reported in table 2 assume that there is some (relatively large) unexplained variation that should be accounted for by a random effect. Essentially, table 2 only considers models that are a priori incorrectly specified when the random effect is excluded. Just take a look at the empirical analysis, especially figure 2: there are definitely many cases where omitting the random slope has a serious effect on the t-ratio statistic, but there are other cases where there appears to be little or no change.

I want to take a moment to note the link in #3 doesn't actually go where it says it goes, and it is too late to edit. I wanted to highlight this debate, but didn't set up the link correctly.

[Frank Sulley]

Thanks Erik, I tried but the model doesnt converge
Iteration 7: log likelihood = -30611.903 (backed up)
could not calculate numerical derivatives -- discontinuous region with missing values encountered
could not calculate numerical derivatives -- discontinuous region with missing values encountered
r(430);

EDIT:
It does converge only if eliminate the r_appereance as following

mixed helpfulness i.l_gender##c.r_appearance i.specialty || lawyer_id: , cov(un)

Any idea on what to do?

[Erik Ruzek]

That is very strange behavior. Can you post model results from the following two models?

mixed helpfulness i.l_gender##c.r_appearance i.specialty || lawyer_id:
mixed helpfulness i.l_gender##c.r_appearance i.specialty || lawyer_id: r_appearance

Also, can you post your data or a slice of your data using dataex?

[Frank Sulley]

Sure, here they are, thanks !

 mixed helpfulness i.l_gender##c.r_appearance i.l_specialty || lawyer_id: 

Performing EM optimization: 

Performing gradient-based optimization: 

Iteration 0:   log likelihood = -32231.184  
Iteration 1:   log likelihood = -32230.821  
Iteration 2:   log likelihood = -32230.821  

Computing standard errors:

Mixed-effects ML regression                     Number of obs     =     37,646
Group variable: lawyer_id                       Number of groups  =      4,553

                                                Obs per group:
                                                              min =          1
                                                              avg =        8.3
                                                              max =         10

                                                Wald chi2(16)     =   61990.36
Log likelihood = -32230.821                     Prob > chi2       =     0.0000

-----------------------------------------------------------------------------------------
            helpfulness |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
------------------------+----------------------------------------------------------------
             2.l_gender |  -.0414463   .0117187    -3.54   0.000    -.0644145    -.018478
           r_appearance |  -.8116196   .0044598  -181.99   0.000    -.8203606   -.8028786
                        |
l_gender#c.r_appearance |
                     2  |   .0520492   .0063937     8.14   0.000     .0395179    .0645806
                        |
            l_specialty |
                     3  |   .0019181   .0151456     0.13   0.899    -.0277668    .0316029
                     4  |  -.0600767   .0166578    -3.61   0.000    -.0927254    -.027428
                     5  |  -.1347807   .0229009    -5.89   0.000    -.1796657   -.0898957
                     6  |   .0306961   .0204527     1.50   0.133    -.0093905    .0707827
                     7  |   .1193425   .0196422     6.08   0.000     .0808445    .1578406
                     8  |    .042894   .0176195     2.43   0.015     .0083605    .0774276
                     9  |   .0097216   .0222773     0.44   0.663    -.0339412    .0533843
                    10  |   .0308514   .0138002     2.24   0.025     .0038034    .0578993
                    11  |   .0094409   .0206395     0.46   0.647    -.0310117    .0498936
                    12  |   .1090057   .0207063     5.26   0.000      .068422    .1495893
                    13  |  -.0874618   .0205804    -4.25   0.000    -.1277987    -.047125
                    14  |  -.0126572    .013427    -0.94   0.346    -.0389735    .0136592
                    15  |  -.0050992   .0222885    -0.23   0.819    -.0487838    .0385854
                        |
                  _cons |   5.802319   .0095033   610.56   0.000     5.783693    5.820945
-----------------------------------------------------------------------------------------

------------------------------------------------------------------------------
  Random-effects Parameters  |   Estimate   Std. Err.     [95% Conf. Interval]
-----------------------------+------------------------------------------------
lawyer_id: Identity          |
                  var(_cons) |   .0224744   .0013006      .0200645    .0251738
-----------------------------+------------------------------------------------
               var(Residual) |   .3067974   .0023778      .3021722    .3114935
------------------------------------------------------------------------------
LR test vs. linear model: chibar2(01) = 527.46        Prob >= chibar2 = 0.0000




. mixed helpfulness i.l_gender##c.r_appearance i.l_specialty || lawyer_id: r_appearance

Performing EM optimization: 

Performing gradient-based optimization: 

Iteration 0:   log likelihood = -29507.274  
Iteration 1:   log likelihood = -29399.739  
Iteration 2:   log likelihood = -29371.151  
Iteration 3:   log likelihood = -29371.119  
Iteration 4:   log likelihood = -29371.119  

Computing standard errors:

Mixed-effects ML regression                     Number of obs     =     37,646
Group variable: lawyer_id                       Number of groups  =      4,553

                                                Obs per group:
                                                              min =          1
                                                              avg =        8.3
                                                              max =         10

                                                Wald chi2(16)     =   22770.38
Log likelihood = -29371.119                     Prob > chi2       =     0.0000

-----------------------------------------------------------------------------------------
            helpfulness |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
------------------------+----------------------------------------------------------------
             2.l_gender |  -.0367458   .0118669    -3.10   0.002    -.0600044   -.0134871
           r_appearance |  -.7681997   .0070704  -108.65   0.000    -.7820574   -.7543421
                        |
l_gender#c.r_appearance |
                     2  |   .0474926   .0098724     4.81   0.000     .0281431    .0668421
                        |
            l_specialty |
                     3  |  -.0015247   .0152823    -0.10   0.921    -.0314775    .0284281
                     4  |  -.0328039   .0172058    -1.91   0.057    -.0665266    .0009188
                     5  |  -.0510631   .0250656    -2.04   0.042    -.1001907   -.0019354
                     6  |  -.0155674   .0207004    -0.75   0.452    -.0561394    .0250046
                     7  |   .0154958   .0200517     0.77   0.440    -.0238048    .0547964
                     8  |  -.0211747   .0177895    -1.19   0.234    -.0560416    .0136922
                     9  |  -.0157277   .0229506    -0.69   0.493      -.06071    .0292546
                    10  |   .0005851   .0140053     0.04   0.967    -.0268647     .028035
                    11  |  -.0038071   .0211521    -0.18   0.857    -.0452644    .0376502
                    12  |    .037177   .0214596     1.73   0.083     -.004883    .0792371
                    13  |   .0054956   .0210911     0.26   0.794    -.0358422    .0468334
                    14  |  -.0093506   .0134541    -0.69   0.487      -.03572    .0170189
                    15  |  -.0078694   .0225537    -0.35   0.727    -.0520738     .036335
                        |
                  _cons |   5.759324   .0098156   586.75   0.000     5.740086    5.778562
-----------------------------------------------------------------------------------------

------------------------------------------------------------------------------
  Random-effects Parameters  |   Estimate   Std. Err.     [95% Conf. Interval]
-----------------------------+------------------------------------------------
lawyer_id: Independent       |
               var(r_appe~e) |   .0271236   .0008358       .025534    .0288122
                  var(_cons) |   5.00e-21   2.86e-21      1.63e-21    1.54e-20
-----------------------------+------------------------------------------------
               var(Residual) |   .2451107   .0018757      .2414618    .2488148
------------------------------------------------------------------------------
LR test vs. linear model: chi2(2) = 6246.86               Prob > chi2 = 0.0000

Note: LR test is conservative and provided only for reference.

[Erik Ruzek]

That's helpful. A few thoughts:

• You have a tiny amount of variation in helpfulness between lawyers - they aren't that different from each other. This is reflected in the small variance estimate for lawyer _id of .02.
• When you add in the random slope for appearance, the variance of the lawyer random intercept is essentially 0. So the cov(un) statement is probably pushing the data beyond where it is able to go, hence the non convergence.
• Things might be helped by centering the appearance variable around the overall mean of appearance such that 0 is equal to a lawyer with a sample average value of r_appearance. Alternatively you could center appearance uniquely around each lawyer's mean.

I'd try centering appearance and re-run the models to see if that improves things.