Crossed effects multilevel models

[Ahuti Friebel]

Hi,

I collected data from an experience sampling study over 14 days, where participants belonged to five different batches (done for convenience, and was not a test variable), and responded to the same questionnaire five times each day of the 14 days. I'm building models to predict for eg sleep from phone use, and have the following syntax:

mixed sleep phone_use || _all: R. Questionnaire_number || _all: R. ID || Batch_No: || Study_day_number:

(questionnaire number and participant ID are crossed because each participant responded to all five questionnaires every day; study day number i.e. 1 to 14 is nested within batch number because the study was conducted over different days for participants belonging to separate batches).

a) Can anyone share shed some light regarding the way I've built up my syntax? I'm a bit lost regarding the use of four levels in a crossed effects model, any advice would be helpful! and b)What sort of post estimation analyses can I conduct to verify that the model is doing what I've asked it to do, eg robustness checks etc?

I must mention that the models are converging, but I just want to make sure that they are the correct models and verify the output.

best wishes,
Ahuti

[Clyde Schechter]

I think your syntax is OK for what you are trying to do.

But I wouldn't do it if I were you. While it is legal to do random effects for a level (Batch) with only five categories, five is an awfully small sample of Batch space and your variance components at that level are going to be very poorly estimated. It would probably make more sense to eliminate the Batch level from the model and instead include i.Batch among the fixed effects.

I would probably do the same with study day number, although 14 is a bit more reasonable--this is a gray zone. My guess is that you have no real interest in the day-variation and are just including it in the model to deal with the dependency among observations. If that is the case, then even if I viewed 14 as an adequate number of categories for a level, I would still put study day number in the fixed effects instead: why complicate the model to get estimates of a nuisance parameter that you don't care about?

If you do both of those things you are left with a simpler model that has two crossed random effects.

I just want to make sure that they are the correct models

You can be nearly certain that they are not the correct models and that, in fact, no model you can realistically work with will be correct. The world is just not that simple. What you can hope for is that the model is a reasonable approximation to the real world data generating process and that the results are wrong but nevertheless useful.

[Ahuti Friebel]

Hi Clyde, thank you so much for your response and suggestions -- I'm new to multilevel analyses so am not entirely familiar with its nuances, in STATA or otherwise. Based on what you have suggested, my question to you is this: Won't including batch number and study day number as fixed effects, take away from accounting for the nested nature of the data, and so, the added variance in the data from the final analyses?

[Clyde Schechter]

Well, you just have to do it the right way. Since you want to model study day number as nested within batch you would do this by specifying i.batch and i.batch#i.study_day_number in the fixed effects to reflect the desired nesting.

[Ahuti Friebel]

Oh! I didn't realise I could separate the clusters in my data into both random and fixed effects and represent the fixed effects in the way you've written it out. Based on your suggestion, the model will now account for all four levels in the data, but treat those levels differently?

[Clyde Schechter]

Yes. It will treat batch and study day number as fixed effects: this will account for the nesting of observations and reduce residual variance accordingly. You will also get actual estimates of those effects at the level of the individual batch and day, although you will probably want to disregard those and will be irked that they clutter up the output. You then have that structure nested within crossed random effects for study ID and questionnaire number. For these two levels of the model you will not get estimates of the category-specific random effects, but you will get estimates of the variance components at each of those levels.

[Ahuti Friebel]

Hi Clyde, I've just performed analyses again based on your suggested syntax, and models have converged. Results are similar to the ones STATA had calculated based on the first syntax I had posted about (direction of relationship, significance), but estimates are different. Thank you so much for all your help, I'm incredibly appreciative. I can now be a bit more confident in the analyses!