measuring consistency over time

[Monica Muller]

Hi,

I have a panel data that measures the daily level of engagement in an intervention in different units of an organization. I also have units' monthly performance. My hypothesis is that we'll see the effect of the intervention on performance if people consistently were engaged in the intervention in each month. In other words I need something more than average monthly engagement to predict monthly performance. Is there a way to create a new variable that measures the consistency of engagement in each unit-month? Should I think of ICC measures? Below is a very small sample of my data that only shows 2 units over 6 months with missing days.

I really appreciate your help.

* Example generated by -dataex-. To install: ssc install dataex
clear
input double unit float(year month day intervention x1 perf)
3 2014 3  3 3.1110635          0  3.681864
3 2014 3  4  3.214602          0  3.681864
3 2014 3  8  3.463133          0  3.681864
3 2014 3 11  3.458333          0  3.681864
3 2014 3 16 3.8407454          0  3.681864
3 2014 3 18  3.938024          0  3.681864
3 2014 3 19 3.6066666          0  3.681864
3 2014 3 23  3.730527          0  3.681864
3 2014 3 25  3.285587          0  3.681864
3 2014 3 30  3.724418          0  3.681864
3 2014 4  1  3.652296 .006289308  3.588401
3 2014 4  2  3.772857          0  3.588401
3 2014 4  5  3.277775          0  3.588401
3 2014 4  6  3.647735          0  3.588401
3 2014 4 12 3.7626054          0  3.588401
3 2014 4 13 3.8420615          0  3.588401
3 2014 4 18  3.929562          0  3.588401
3 2014 4 23  3.928355          0  3.588401
3 2014 5  6  3.739881          0  3.625499
3 2014 5 10 3.7718964          0  3.625499
3 2014 5 11  3.868339          0  3.625499
3 2014 5 12  4.080385          0  3.625499
3 2014 5 13 4.2436934          0  3.625499
3 2014 5 14 4.1212697          0  3.625499
3 2014 5 15  3.620952          0  3.625499
3 2014 5 21  3.667033          0  3.625499
3 2014 5 25  3.864689          0  3.625499
3 2014 6  1  3.950181          0  3.647307
3 2014 6  2 4.0233035          0  3.647307
3 2014 6  6  3.979147          0  3.647307
3 2014 6  7 4.0089684          0  3.647307
3 2014 6  8  3.601786          0  3.647307
3 2014 6 14  4.000992          0  3.647307
3 2014 6 20   4.02127          0  3.647307
3 2014 6 21 3.6988945          0  3.647307
3 2014 6 24 3.6065714          0  3.647307
3 2014 6 27 3.4779956          0  3.647307
3 2014 7  1  3.682246          0  3.660105
3 2014 7  3  3.839359          0  3.660105
3 2014 7  6  3.786766          0  3.660105
3 2014 7  8 3.5608134          0  3.660105
3 2014 7  9  3.834944          0  3.660105
3 2014 7 12  3.633594          0  3.660105
3 2014 7 14  3.598512          0  3.660105
3 2014 7 25 3.6499574          0  3.660105
3 2014 8  1  3.895816          0  3.640486
3 2014 8  9 3.7872024          0  3.640486
3 2014 8 11  3.989059          0  3.640486
3 2014 8 12  3.719251          0  3.640486
3 2014 8 16  3.901501          0  3.640486
3 2014 8 26  4.154808          0  3.640486
4 2014 3  1  4.487683          0  3.877455
4 2014 3  2  4.466119          0  3.877455
4 2014 3  8  4.530067          0  3.877455
4 2014 3 13  4.389437          0  3.877455
4 2014 3 14 4.3569727          0  3.877455
4 2014 3 16  4.428704          0  3.877455
4 2014 3 17 4.3523016 .005050505  3.877455
4 2014 3 18 4.4167166 .005050505  3.877455
4 2014 3 20 4.2942405 .005050505  3.877455
4 2014 3 23  3.559831 .005050505  3.877455
4 2014 3 27   3.38417          0  3.877455
4 2014 3 29 3.3134615 .005050505  3.877455
4 2014 4 12  3.174099          0 3.7853694
4 2014 4 17 3.4076686          0 3.7853694
4 2014 4 24  3.346627          0 3.7853694
4 2014 4 25  3.646145          0 3.7853694
4 2014 4 26  3.246712 .005102041 3.7853694
4 2014 5  2 3.9026785          0 3.9066565
4 2014 5  3  3.709762 .005154639 3.9066565
4 2014 5  4  3.611012          0 3.9066565
4 2014 5  7  3.899127          0 3.9066565
4 2014 5 14   3.74919 .010309278 3.9066565
4 2014 5 16  4.003702          0 3.9066565
4 2014 5 17 4.3048353          0 3.9066565
4 2014 5 20 4.3275514          0 3.9066565
4 2014 5 21  4.305546          0 3.9066565
4 2014 5 22  4.118285          0 3.9066565
4 2014 5 27  4.046372          0 3.9066565
4 2014 5 29  4.280924          0 3.9066565
4 2014 5 30 4.2469954          0 3.9066565
4 2014 6  1 4.4469843          0 3.8373895
4 2014 6  3 4.1550426          0 3.8373895
4 2014 6  6  4.354739          0 3.8373895
4 2014 6  9  3.083121          0 3.8373895
4 2014 6 11 3.6067686          0 3.8373895
4 2014 6 24  3.718174          0 3.8373895
4 2014 6 26  3.464996          0 3.8373895
4 2014 6 29 3.3165674          0 3.8373895
4 2014 7  1 3.8384674          0 3.7793496
4 2014 7  2 3.3438776          0 3.7793496
4 2014 7  4  3.890944          0 3.7793496
4 2014 7  5 4.0652776          0 3.7793496
4 2014 7  6 3.4663436 .005050505 3.7793496
4 2014 7 13 3.5659156          0 3.7793496
4 2014 7 16 3.7170634          0 3.7793496
4 2014 7 21  3.601885 .005050505 3.7793496
4 2014 7 25  3.659416          0 3.7793496
4 2014 7 30  3.862675          0 3.7793496
4 2014 8  1  3.818193          0      3.88
end

[Clyde Schechter]

I don't understand. Which variable measures "engagement" here?

[Monica Muller]

Hi Clyde,

The intervention variable measures how much engaged people have been in the intervention. The organization started this intervention at the same time for all units. Unfortunately, I don't have any measures before the intervention, but people reported how much engaged they've been in the intervention every day.

[Clyde Schechter]

Yes, but which variable in your -dataex- example contains the reports of how engaged they've been?

[Monica Muller]

The fourth variable, called intervention.

[Monica Muller]

I mean the fifth variable, called intervention.

[Clyde Schechter]

So the task is to come up with some measure of the "consistency" of engagement (as measured by the variable intervention), to us as a predictor of outcome (perf). It seems implied, though I don't see it explicitly stated, that the level of engagement itself is also important. That is, consistent low level of enagement might help performance, but presumably consistent high level would be better. This suggests to me that ultimately perf is to be modeled in terms of both the level of engagement and its "consistency", perhaps through an interaction.

I don't think the ICC would be a particularly good approach here. First, if assessed at the level of the variable unit, it is a constant for the entire population and it assesses the extent to which there is greater consistency of engagement within units than there is across units or not. But I think you want something that can be measured in each unit. Perhaps there is a lower level of aggregation of the data within unit that would support this, but I don't see one.

I think a better approach might even be a simpler one: use the unit's standard deviation of intervention as the measure of consistency. So something like

egen consistency = sd(intervention), by(unit)

xtset unit
xtreg perf c.intervention##c.consistency, re

margins, at(intervention = (3(.25)4.5) consistency = (.25(.05).40))
marginsplot

(I picked those -at()- values because they are round numbers that roughly span the ranges of intervention and, as defined here, consistency, in the data. You should replace those with suitable values that are relevant to your own data and that perhaps emphasize "interesting" values, without necessarily spanning the range.

Another measure, similar to the standard deviation, would be the median absolute deviation from the median (-egen- function -mad()-), although it looks like the distribution of intervention within units is fairly symmetrical and there aren't many outliers to worry about, at least in the example data.

[Monica Muller]

Thank you very much, Clyde. It's super helpful.

[Monica Muller]

Sorry, I've one more question. After I run the margins code I get the attached table. All the rows are empty and under Delta-method standard error is written "not estimable". Do you know what I'm doing wrong? Thanks.

[Clyde Schechter]

Please show the actual regression command, the complete regression output, and the -margins- command. There are several possibilities that cannot be distinguished just from this -margins- output.

[Monica Muller]

I'm so sorry. I just noticed that I made a dumb mistake that eliminated the interaction term altogether. It works perfectly now. I apologize.