Honestdid interpretation

[Zachary Bartsch]

I have read the support documentation for honestdid, and I think that I must be missing something.

First, I run csdid2, and yields

Then, I run:

honestdid, pre(1/5) post(1) mvec(0(0.2)1) delta(rm) coefplot parallel(8)

and

honestdid, pre(1/5) post(1/6) mvec(0(0.2)1) delta(rm) coefplot parallel(8)

I have 2 questions:

1) Why are the CI bounds almost identical if they are supposed to be referring to different post-treatment effects?
2) Why are the top listed CI positive if all of the post-treatment coefficients are negative?
--I undderstand deviations from trend, but the original is also positive (which isn't sensible to me).

What am I missing?

Here are support pages for csdid & honestdid:

https://github.com/friosavila/csdid2...main/README.md

https://github.com/mcaceresb/stata-h...file#honestdid

[Zachary Bartsch]

I solved my own problems.
I'm sharing here in case others find them useful.

Consider this example from the honestdid documentation: https://github.com/mcaceresb/stata-h...file#honestdid

local mixtape https://raw.githubusercontent.com/Mixtape-Sessions
use `mixtape'/Advanced-DID/main/Exercises/Data/ehec_data.dta, clear
qui sum year, meanonly
replace yexp2 = cond(mi(yexp2), r(max) + 1, yexp2)

csdid dins, time(year) ivar(stfips) gvar(yexp2) long2 notyet
csdid_estat event, window(-4 5) estore(csdid)
estimates restore csdid

local plotopts xtitle(Mbar) ytitle(95% Robust CI)
honestdid, pre(3/6) post(7/12) mvec(0.5(0.5)2) coefplot  name(original ,replace)

If you run it how it is written, then you will get the documentation results perfectly.

My confusion above is due to misinterpreting the index values. The pre() and post() do not refer to periods relative to the initial treatment time - nor the coefficients to use in testing. Rather, they refer to matrix row of the stored results. I don't use the 'did' package, but in csdid, the first 2 listed rows are for the average pre and average post treatment effects. That's why the documentation code says pre(3/6) - it's beginning at the 3rd row of the csdid estimation results.

If you want to test the deviation from trend for multiple coefficients, then you need the l_vec() option, which specifies a matrix that is the same length as your post treatment period (example at the bottom). It includes the weights on the post-treatment coefficients.

I also learned that the above code does not work for csdid2 in exactly the same way. That's because the 1) csdid2 uses estat rather than csdid_estat, 2) csdid2 uses revent() rather than window() options (see other thread), & 3) revent() indexes differently than does window(). revent() indexes to the jth treatment period while window() indexes to the jth coefficient.

To get the documentation results using csdid2, you'd use the below code with or without revent().

With revent():

csdid2 dins, time(year) ivar(stfips) gvar(yexp2) long2 notyet
estat event, revent(-5/5) estore(csdid)
estimates restore csdid

local plotopts xtitle(Mbar) ytitle(95% Robust CI)
honestdid, pre(3/6) post(7/12) mvec(0.5(0.5)2) coefplot  name(csdid2revent,replace)

Without revent():

csdid2 dins, time(year) ivar(stfips) gvar(yexp2) long2 notyet
estat event,  estore(csdid)
estimates restore csdid

local plotopts xtitle(Mbar) ytitle(95% Robust CI)
honestdid, pre(9/12) post(13/18) mvec(0.5(0.5)2) coefplot  name(csdid2revent,replace)

You can use the l_vec() option to specify post-treatment coefficients to test. Again replicating the above:

csdid2 dins, time(year) ivar(stfips) gvar(yexp2) long2 notyet
estat event,  estore(csdid)
estimates restore csdid

matrix l_vec=1\0\0\0\0\0
local plotopts xtitle(Mbar) ytitle(95% Robust CI)
honestdid, pre(9/12) post(13/18) mvec(0.5(0.5)2) coefplot name(csdid2lvec,replace) l_vec(l_vec)

[FernandoRios]

Thank you zachary
I ll prepare an example on my page as a good opportunity to document a bit more what can csdid2 do with honestdid
best wishes

[Ashani Abayasekara]

Hi Zachary,

I'm trying to understand and interpret the honestdid package at the moment and found your blog article about it at https://economistwritingeveryday.com/tag/honestdid/ very useful. I now better understand the intuition behind the approach, but am still not fully clear about interpreting the results (CIs). I'm copying an excerpt from your article to highlight the section I would like more clarification on:

__________________________________________________ __________________________________________________ __________________________________________________ __________________________________________________ __________________________________________________ _
What does the above table tell us? It gives us several values of M and the confidence interval for the difference between the coefficient and the trend at the 95% level of confidence. The first CI is the original time-0 coefficient. When M is zero, then the null assumes the same linear trend as during the pretreatment. Again, M is the ratio by which maximum deviations from the trend during the pretreatment are used as the null hypothesis during the post-treatment period. So, above, we can see that the initial treatment effect deviates from the linear pretreatment trend. However, if our standard is the maximum deviation from trend that existed prior to the treatment, then we find that the alpha is just barely greater than 0.05 (because the CI just barely includes zero).
__________________________________________________ __________________________________________________ __________________________________________________ __________________________________________________ _________________________________________________

In particular, I don't quite follow the last bit which I have bolded, and would appreciate a clearer explanation on this. How do we know that the initial treatment effect deviates from the pre-treatment trend, and that alpha is barely greater than 0.05 if we consider the maximum deviation from the trend? Also, how do we determine the cut-off point here, or up to how much as big as the maximum violation in the pre-treatment period is the significant result robust to ?

Another question is that when I test this command on my data (which I unfortunately cannot share due to privacy standards), I get an error message saying that "CI is open at upper end-point; CI length may not be accurate." I was wondering whether I can still proceed with analysing and interpreting results produced by this method under these circumstances.

Thank you very much.

Ashani