Multilevel ordinal logistic model using longitudinal data

[Rose Matthews]

Hi I've got have 4 (q1-q4) questions in a questionnaire (Knee Injury and Osteoarthritis Outcome Score - KOOS),
that together (the 4 questions) all relate to a specific function
of the knee (lets call this F1). Each of the 4 questions are ordinal scores from 1(worst outcome) to 5 (best outcome)

I want to test how each question improves/doesn't improve following treatment as compared to control

I've tried latent analysis, factor analysis - however on discussion with seniors I was told I should treat every question separately
and can not gather them together. So forget K1.....

I'm trying one last attempt with a multilevel ordinal logistic model

I've reshaped by data:

///for every q* there is q1_preop and q1_postop and so forth. 

reshape long q1_ q2_ q3_ Q4_, i(ptid) j(pre_post) string
label define q1 1 "preop" 2 "postop"
encode pre_post, gen(pre_postno) label(q1)

//Focus on q1 

///procedure is binary 0= control 1=treatment

///q1 as the value of pre and post procedure - which is indicated by 1 or 2 by pre_postno

melogit q1_ i.procedure pre_postno procedure##pre_postno ||ptid:, vce(robust) or level(95)

///Calculate the predicted probabilities of improvement from pre_postno =1 and pre_postno=2 as for treatment=1 and treament = 0 

margins procedure, over(pre_postno)

Q1. Do you think this model makes sense?

Q2, Personally, I don't think the model makes sense as the margins command is calculating the probability of being 1 or 2 i.e pre or post procedure
As you can see in the output attached where red= procedure= 0 and blue= procedure= 1

This shows that mean probability for control (red) there is a 99% chance of what??
his doesn't make sense as the q1 is a ordinal score of 1-5.

[Clyde Schechter]

The -margins- command makes no sense. The -melogit- command is inappropriate because your outcome variable, the question response, has 4 levels, not 2. Also, you are using factor-variable notation in a sloppy way that, in this case, you might get away with, but eventually it will catch up with you. I think you want something like:

meologit q1 i.procedure##i.pre_postno || ptid:, vce(robust) /* ??? why not vce(cluster ptid) */ or
margins procedure#pre_postno
margins procedure, dydx(pre_postno)

The first -margins- commands will give you the probability of each response to q1 for both treatment groups pre and post procedure. The second one will give you the difference in probability of each response level from pre to post in both treatment groups. The actual estimate of treatment effectiveness, however, is found not in the -margins- output but in the -meologit- output itself as the coefficient of the interaction term.

Also, you might want to consider simplifying this by treating the item responses as if they were a continuous variable. This is not strictly legitimate unless you think it is reasonable to believe that the 4 responses are spaced more or less equally along whatever dimension it is that the item measures. But if that is reasonable, using -mixed- will give you a smaller, more digestible, more comprehensible set of results.

[Rose Matthews]

Thanks for the insight, I’ll try this out

re this (1)

Also, you might want to consider simplifying this by treating the item responses as if they were a continuous variable.

i’ve tried a simple regression, just trying different things to confirm the regression outpu

re this (2):
This is not strictly legitimate unless you think it is reasonable to believe that the 4 responses are spaced more or less equally along whatever dimension it is that the item measures

I’ve seen this written a couple of times, what does -spaces more or less equally- mean ?

does it mean, every individual is asked the same 4 questions and answers the same no of questions (from 1-4)

clarification on point 2 would be really appreicated

[Clyde Schechter]

I don't know what the subject matter of the questions is. Let me suppose that q1 is about pain and asks patients to report how much pain they have in the knee. They could be asked to just pick a number from 1 to 4, or perhaps they are offered 4 options such as "none," "mild," "moderate," or "severe." To say that the options seem equally spaced means that you believe the difference in the patient's psychological perception of the difference of severity of mild pain vs none is the same as the patient's perception of the difference of severity of moderate pain vs mild, and that, in turn, is the same as the perception of the difference of severity of severe vs moderate pain.

It is difficult to make such judgments, as it involves both your own subjectivity and your ability to mind-read your respondents' subjectivity. Nevertheless, some people are comfortable saying that. A good example is the classic 5-point Likert scale, whose options are "Strongly Disagree," "Somewhat Disagree," "Neither Agree nor Disagree," "Somewhat Agree," and "Strongly Agree." Many people who use 5-point Likert scales in their research will assert that these five points are equally spaced, and an even larger number will treat these scales (or rather the attached numbers 1 through 5) as a continuous variable in regression analyses without even realizing this could be an issue.

And, by comparison, here's a scale that I think most people would agree is not equally spaced: "Extremely Unhappy," "Very Unhappy," "Somewhat Unhappy," "Extremely Happy." (And as obviously badly designed as that response set is, believe it or not, I have not infrequently seen things just like that.)

[Rose Matthews]

Thanks for this.

Just to confirm, if my interaction value from :

meologit q1 i.procedure##i.pre_postno || ptid:, vce(robust)  or level(95)

i.procedure##i.pre_postno = OR = 0.83 (p= 0.08)

The interpretation would be that for procedure = 1 , the odds of achieving a score in q1 lower than 5 is more likely with the treatment (x0.87) than control. However one can not infer much as it is not statistically significant.

Remembering that:
procedure = 1= treatment; 0 = control
q1 = ordinal from 0-5 (5 = best, 0 = worst)
pre_postno = 1= post treatment 0 = pre-procedure

Many thanks

[Clyde Schechter]

The interpretation would be that for procedure = 1 , the odds of achieving a score in q1 lower than 5 is more likely with the treatment (x0.87) than control. However one can not infer much as it is not statistically significant.

That is the conclusion most people would draw. But I disagree. This is based on a widespread but fallacious misunderstanding of statistical significance. Statistical significance does not mean that there is an effect and non-significance means there isn't. You can't dichotomize that way, even though so many people do so. The correct interpretation is that the data are consistent with a lack of effect, but they are also consistent with the presence of such an effect. The absence of evidence is not evidence of absence. I would prefer to ignore the p-value and look at the confidence interval. The latter provides a range of values that are largely consistent with the data. Most of that range will be in OR < 1 territory; some of it will be in OR > 1 territory. I would interpret that as an inconclusive finding.