Estimating variation in fixed effects

[Alyssa Clare]

Hope this isn't too basic. The conceptual question is that I'd like to estimate 1) how much variation there is in the odds of an outcome across counties, and 2) whether that variation declines after a policy is implemented (i.e. whether the effect of where you live is minimized).

I'm trying to answer this using the below model with fixed effects (dummies) for counties:

logistic outcome i.prepost controlvars i.county, vce(cluster county)

Is there a way to estimate in Stata the variation in county fixed effects, and the change in variation in county fixed effects across values of prepost?

Many thanks.

[Clyde Schechter]

Are you sure that what you want is to estimate variation in the county effects? If the goal is to determine if the intervention makes the outcome less dependent on where you live, this sounds to me like a question that would be probed by a model in which there are county#prepost interaction terms and you test whether the county marginal effects are closer to zero in the post period.

[Alyssa Clare]

Thanks so much for your response. Sorry if I'm misunderstanding - I'm imagining you're suggesting something like the below?

margins r.county, over(r.prepost)

Wouldn't an approach like this tell us whether there were county reductions in the likelihood of the outcome, but we wouldn't know whether counties that started out higher reduced more (such that there was less variability depending on where you live)?

[Clyde Schechter]

No, I'm thinking of

logistic outcome i.county##i.prepost controlvars, vce(cluster county)
margins prepost, dydx(county)
margins prepost, dydx(county) pwcompare

[Alyssa Clare]

I see. Thanks so much, Clyde! Really appreciate your help.

[Alyssa Clare]

Clyde,

This has been a very helpful approach to assessing whether each county's marginal effects are closer to zero in the post-period, which gives valuable information, but if it's possible to do so, I think I'd still also like to generate a summary measure of the variation (SD) in county effects pre and post, and test the change in that variation.

I could generate pre and post probabilities for each county using the model in #4 and use sdtest to test whether the standard deviations in county effects changed. However, this wouldn't account for the error in the estimates. Is there a better way to do this?

Thank you!

[Steve Samuels]

Here is code which will estimate a standard deviation for county fixed effects, with and without analytic weights. I don't have a logit dataset handy, so I use regress. For simplicity, I generate the county indicators "by hand". Note that the SDs with and without analytic weights are similar.

However I don't see a way of comparing SDs except by bootstrapping the entire process. Better, in my opinion, would be a random effects model in which the the county random effect is allowed to interact with covariates.

set more off
sysuse auto, clear
rename trunk county
levelsof county, local(county)
local k : word count `county'
di `k'  // 18

/* Create 0-1 indicators cty1 -cty18 */
qui tab county, gen(cty)
des cty* /* link county indicators to  values */

/* IMPORTANT! PUT county INDICATORS FIRST */
regress mpg  cty* foreign length gear , nocons

/* Extract intercepts & Std errors */
matrix r = r(table)
mat b   = r[1,1..`k']'
mat se  = r[2,1..`k']'

/* aweight matrix   aweight = 1/se^2 */
mata: st_matrix("awt",(1:/st_matrix("se")):^2)
matrix d =  b, awt
matrix  colnames d = intercept awgt
clear   /* drop main data */

/* convert matrix d to dataset */
svmat d, names(col)  /* convert d to dataset*/
save countries, replace
// drop if intercept ==0  /* For logit, when there is perfect prediction */
list
/* Get mean & SD & density plot */
sum intercept
sum intercept [aw = awgt]
kdensity intercept [aw = awgt]
graph export gcty.png, replace

with results

. /* Get mean & SD & density plot */
. sum intercept
    Variable |        Obs        Mean    Std. Dev.       Min        Max
-------------+---------------------------------------------------------
   intercept |         18    49.65785    1.702936   46.09109     52.409

. sum intercept [aw = awgt]

    Variable |     Obs      Weight        Mean   Std. Dev.       Min        Max
-------------+-----------------------------------------------------------------
   intercept |      18   .12698833    49.63772   1.652512   46.09109     52.409

[Alyssa Clare]

Steve,

Thank you so much for this code. This is quite helpful. I will also try the random effects model route you recommended.

Thanks again,
Alyssa