help interpreting of model with change score as dependent variable - treatment effect

[Jennifer Carson]

Hello, I'm working on a study where the goal is to determine the effect of a treatment. Everyone received the treatment, and I have a pre treatment measurement and a post treatment measurement for each person. I have calculated the change between pre and post (specifically, Post minus pre), and am using this as a dependent variable in a regression model. I also have a couple of demographic variables in the model. See code and results below.

reg change female age

      Source |       SS           df       MS      Number of obs   =        35
-------------+----------------------------------   F(2, 32)        =      5.19
       Model |  2232.18795         2  1116.09397   Prob > F        =    0.0112
    Residual |  6881.38348        32  215.043234   R-squared       =    0.2449
-------------+----------------------------------   Adj R-squared   =    0.1977
       Total |  9113.57143        34  268.046218   Root MSE        =    14.664

------------------------------------------------------------------------------
      change |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
      female |  -16.27532   5.262576    -3.09   0.004    -26.99483   -5.555801
         age |  -.4479529   .2852654    -1.57   0.126    -1.029019    .1331137
       _cons |   48.06454   20.07094     2.39   0.023     7.181372    88.94772
------------------------------------------------------------------------------

.

Questions:
1. How to interpret the value of _cons (48.06) ? In a model without the covariates I thought it was the average of the change scores, but with the covariates present the value changed. Can one conclude there is a statistically significant treatment effect here since p=0.023?
2. How to interpret the coefficient for female (-16.28)? is it saying that the treatment was not as effective for females? what does the number actually represent?

thank you in advance for any comments!

[Clyde Schechter]

Your regression output tells you that you have the following equation as the best linear equation that can be fit to your data:

expected value of change = 48.06454 - 16.27532*female - 0.4479529*age

This linear relationship explains just under 25% of the variance in change, and the root mean squared error is about 14.7.

So given a person's sex and age you can calculate the expected value of change for people meeting that description. In particular, the constant term, 48.06454 is the expected change for a male of age zero. My guess is there aren't any people of age 0 in your study, so this constant term is not really meaningful in your context.

If would like the average values of change by sex, you can get that, after the -regress- command is run, with

margins female

If you want average values of change at certain ages, say, 10, 35, and 55, you can do

margins, at(age = (10 35 55))

If you want average values of change, separately for each sex at those ages you can do:

margins female, at(age = (10 35 55))

If you are interested in the expected difference in change between the sexes, that is given by the coefficient of the female variable

In short, interpreting this model is probably done using the -margins- command. The best introduction to that command I know of is from the excellent Richard Williams: https://www3.nd.edu/~rwilliam/stats/Margins01.pdf.

[Jennifer Carson]

Thank you - this is very helpful!!! A follow up question: is there a way to get some kind of average treatment effect/change and determine whether or not it is statistically significant? Or does that not make sense given that the change depends on whether the person is male or female?

[Clyde Schechter]

Well, in your situation, if you just want an average treatment effect, just -summ change- and look at the mean. If the differences between the effects in males and females are appreciable, such an average effect is really not very useful, as the results would be very sensitive to the gender mix of the sample, and certainly would not generalize to a sample with an appreciably different gender mix. But if the male and female treatment effects are nearly equal, and if the age effect is small, then an average treatment effect might be a useful summary statistic.

Don't ask whether or not it is statistically significant. The American Statistical Association now recommends abandonment of the concept of statistical significance. See https://www.tandfonline.com/doi/full...5.2019.1583913 and the articles it references for the full story, or https://www.nature.com/articles/d41586-019-00857-9 for a shorter pep talk on the topic. Focus instead on whether the magnitude is large enough to matter for practical purposes. And include in your reports and thinking the uncertainty associated with your estimates, as quantified by standard errors or confidence intervals, and, to a lesser degree, by p-values. But do not categorize into "significant" and "not significant."