Test of balance (means, std. dev.; Differences, p-values for joint-F-test) for 3 treatments

[Janina Gleed]

Hi,

I have three treatments from a survey (control; Empirical; Normative) and wish to create a table like the one below. I have managed to recreate it to some degree, however, I have come across some difficulties (listed below with my attempted code) that I haven’t yet managed to successfully overcome from what I have found online so far. I feel I have got a bit lost and may be missing something obvious so any guidance would be massively appreciated! If I have missed any useful information, please let me know. Am I on the right track here?

Intended table

Columns (1) – (3):

• means & std. dev. of respondent covariates in the different treatments

Columns (4) – (6):

• differences in means between the groups indicated in the column header together with p-values in parentheses.
• The p-values of the joint F-test are determined by regressing the treatment indicator on the vector of demographic controls.
• The F-test tests the joint hypothesis that none of the covariates predicts treatment assignment.

(i) Version 1

1. How would I best add columns (4) – (6)?

sort Groupid
by Groupid: eststo: quietly estpost summarize NBpriorBehOthr NBpriorNorOthr Altr Gender 
esttab, main(mean) aux(sd) label nodepvar nonumber
esttab using TableA.3a.doc, replace title("Table A.3: Test of balance")
eststo clear

(ii) Version 2

1. What reasons could there be for the ‘Treatment: mean & sd’ not showing?
2. How to incorporate the difference in means and associated p-values from an F-test?
3. How to add the third group (Normative treatment)? I have written the code for the test of differences but unfortunately haven’t managed to incorporate the treatment’s summary statistics successfully so I have just added in after for reference.

Code for the above table below (additional code for third group mentioned last)

global DESCVARS NBpriorBehOthr NBpriorNorOthr Altr Gender 
mata: mata clear
 
* First test of differences (Empirical treatment)
local i = 1
 
foreach var in $DESCVARS {
    reg `var' dGroupid1,
    outreg,  keep(dGroupid1) rtitle("`: var label `var''") stats(b) ///
        noautosumm store(row`i')  starlevels(10 5 1) starloc(1)
    outreg, replay(T1_C) append(row`i') ctitles("",T1_C ) ///
        store(T1_C) note("")
    local ++i
}
outreg, replay(T1_C)
 
* Then Summary statistics
local count: word count $DESCVARS
mat sumstat = J(`count',4,.)
 
local i = 1
foreach var in $DESCVARS {
    quietly: summarize `var' if Groupid==dGroupid1
    mat sumstat[`i',1] = r(mean)
    mat sumstat[`i',2] = r(sd)
    quietly: summarize `var' if Groupid==dGroupid2
    mat sumstat[`i',3] = r(mean)
    mat sumstat[`i',4] = r(sd)
    local i = `i' + 1
}
frmttable, statmat(sumstat) store(sumstat) sfmt(f,f,f,f)

*And export
outreg using C:\Users\janin\OneDrive\Documents\Survey\TableA3g.doc, ///
    replay(sumstat) merge(T1_C) nocenter note("") plain replace ///
    ctitles("", Control, "", Treatment, "", T1_C \ "", mean, sd, mean, sd) ///
    multicol(1,2,2;1,3,2)
 
* Second test of differences (Normative treatment)
local i = 1
 
foreach var in $DESCVARS {
    reg `var' dGroupid2,
    outreg,  keep(dGroupid2) rtitle("`: var label `var''") stats(b) ///
        noautosumm store(row`i')  starlevels(10 5 1) starloc(1)
    outreg, replay(T2_C) append(row`i') ctitles("",T2_C ) ///
        store(T2_C) note("")
    local ++i
}
outreg, replay(T2_C)
             
 
** When searching for a way of adding F-test p-values, I found code here which I attempted to use. However I am unsure how to add it to my tables.
reg dGroupid2 $DESCVARS
est sto test
qui testparm*
estadd scalar p_value = r(p)
esttab test, p label scalar(F p_value)
esttab using TableA.3b.doc, replace title("Table A.3: Test of balance")
eststo clear

dataex NBpriorBehOthr NBpriorNorOthr Altr Gender dGroupid1 dGroupid2 dGroupid3

* Example generated by -dataex-. For more info, type help dataex
clear
input byte(NBpriorBehOthr NBpriorNorOthr) float Altr byte(Gender dGroupid1 dGroupid2 dGroupid3)
 80  90   -.56070167 6 1 0 0
 83  50     1.393947 6 1 0 0
 80 100    1.1791126 5 1 0 0
 40  73    -.8701696 6 1 0 0
 60  80   -1.0538094 6 1 0 0
 89  85   -1.2374493 5 1 0 0
 65  50       1.3122 6 1 0 0
 70  61    .28059033 5 1 0 0
 78  93     -.833873 5 1 0 0
 40  85     .1184658 5 1 0 0
 50  82  -.011676595 5 1 0 0
 90  95     .4717521 6 1 0 0
 46  66   -.16654183 6 1 0 0
 90 100   -.53892374 5 1 0 0
 61  62            . 5 1 0 0
 70 100            . 5 1 0 0
 90  98    -1.370274 5 1 0 0
 40  95    -1.788369 5 1 0 0
 97  97     .9489719 5 1 0 0
 60  90      -.31336 6 1 0 0
 88  78    -.6865296 5 1 0 0
 71  70    -.8556509 5 1 0 0
 80  75   -.50504696 5 1 0 0
 47  80     .9417126 5 1 0 0
 40  70    -.2074153 5 1 0 0
 92  95    -.9954724 5 1 0 0
 87 100     .9489719 6 1 0 0
 90  92     .0832196 6 1 0 0
 90 100    .22977516 5 1 0 0
 83  83     .6365591 6 1 0 0
 98  99     -.200156 6 1 0 0
100 100    -1.113516 6 1 0 0
 80 100   -.44455275 5 1 0 0
 60  80 -.0044172797 6 1 0 0
 60  75    -.3819013 6 1 0 0
 75  90    -1.604729 5 1 0 0
 80 100     .9973673 5 1 0 0
 91  95     .9489719 5 1 0 0
 80  70     .0442407 6 1 0 0
 90  90   -1.1140411 5 1 0 0
 85  95    -.3886355 5 1 0 0
 80 100    -.8701696 5 1 0 0
 71  86    .29373953 6 1 0 0
 70  75     .7725913 6 1 0 0
 61 100    .39805225 5 1 0 0
 97  97   -1.0581238 5 1 0 0
 57  79   -.26228225 5 1 0 0
 61   0            . 6 1 0 0
 95  98    -.7997336 5 1 0 0
 90  97    -.1759583 5 1 0 0
 80  95    -.3837959 5 1 0 0
 85  82   -.02135567 6 1 0 0
 81 100     .7774308 5 1 0 0
 80  68   -.57227534 5 1 0 0
 90 100     .4023666 6 1 0 0
  1   0    -.6865296 6 1 0 0
 95  91   -.40315405 6 1 0 0
 75  70     .8395571 6 1 0 0
 66  85   -1.0175129 5 1 0 0
 87  88     .3464493 6 1 0 0
 80 100    2.1029394 6 1 0 0
 60  50     -.105785 5 1 0 0
 70  90    1.6189854 6 1 0 0
 91  98    1.6144085 5 1 0 0
 44  77     .6981037 5 1 0 0
 69  78     .6104667 5 0 1 0
 90  80            . 5 0 1 0
 76 100   -1.0538094 5 0 1 0
 20   5   -.25849307 5 0 1 0
  .   .     .8250384 6 0 1 0
 70  70            . 6 0 1 0
 10  90   -.26091287 5 0 1 0
 45  71    1.0922635 6 0 1 0
 70  70    -.8094127 6 0 1 0
 70  85  -.016516132 6 0 1 0
 47  47    -.3789564 5 0 1 0
 85  85    .29053202 6 0 1 0
 81  73   -.22703607 6 0 1 0
 61  82            . 5 0 1 0
 59  39   -.21414946 6 0 1 0
 60  84            . 5 0 1 0
 60  70   -.44455275 5 0 1 0
100 100     .3483439 6 0 1 0
 80  70      .706995 5 0 1 0
 60  80    .05391978 6 0 1 0
 73 100    -.7007857 5 0 1 0
 95   0   -.21251744 6 0 1 0
 80  70    -.6281926 6 0 1 0
 76  88     .5306144 6 0 1 0
 83  97   -.38647825 6 0 1 0
 16  20     .4867959 5 0 1 0
 78  59     .6615445 6 0 1 0
 75  90   -.01893589 5 0 1 0
 68  73   -1.0027317 5 0 1 0
 79  67    -.0745906 6 0 1 0
 94  98     .8371373 5 0 1 0
 90  84    .10878672 6 0 1 0
 17  95    -1.715776 5 0 1 0
 20  90    2.5844736 5 0 1 0
 60  40    1.1035746 6 0 1 0
end
label values NBpriorBehOthr NBpriorBehOthr
label values Gender Gender_lbl
label def Gender_lbl 5 "male", modify
label def Gender_lbl 6 "female", modify

[Andrew Musau]

Here is a start assuming 2 groups. You can extend it to more than 2 groups using a similar logic. You should probably report standard errors on the means instead of standard deviations as it will be confusing to report SDs for the means and SEs on the differences. All results are obtained from regress so that it is easier to append and export using appendmodels and estout, respectively. The former is enclosed below whereas the latter is from SSC. Assuming a balanced sample, you can report sample sizes as I do. If not, you should not report as each combination of means will likely be based on a different sample size. Otherwise, you can add an extra column to report the sample size as in https://www.statalist.org/forums/for...rs-with-esttab with varying sample sizes.

* Example generated by -dataex-. For more info, type help dataex
clear
input byte(NBpriorBehOthr NBpriorNorOthr) float Altr byte(Gender dGroupid1 dGroupid2 dGroupid3)
 80  90   -.56070167 6 1 0 0
 83  50     1.393947 6 1 0 0
 80 100    1.1791126 5 1 0 0
 40  73    -.8701696 6 1 0 0
 60  80   -1.0538094 6 1 0 0
 89  85   -1.2374493 5 1 0 0
 65  50       1.3122 6 1 0 0
 70  61    .28059033 5 1 0 0
 78  93     -.833873 5 1 0 0
 40  85     .1184658 5 1 0 0
 50  82  -.011676595 5 1 0 0
 90  95     .4717521 6 1 0 0
 46  66   -.16654183 6 1 0 0
 90 100   -.53892374 5 1 0 0
 61  62            . 5 1 0 0
 70 100            . 5 1 0 0
 90  98    -1.370274 5 1 0 0
 40  95    -1.788369 5 1 0 0
 97  97     .9489719 5 1 0 0
 60  90      -.31336 6 1 0 0
 88  78    -.6865296 5 1 0 0
 71  70    -.8556509 5 1 0 0
 80  75   -.50504696 5 1 0 0
 47  80     .9417126 5 1 0 0
 40  70    -.2074153 5 1 0 0
 92  95    -.9954724 5 1 0 0
 87 100     .9489719 6 1 0 0
 90  92     .0832196 6 1 0 0
 90 100    .22977516 5 1 0 0
 83  83     .6365591 6 1 0 0
 98  99     -.200156 6 1 0 0
100 100    -1.113516 6 1 0 0
 80 100   -.44455275 5 1 0 0
 60  80 -.0044172797 6 1 0 0
 60  75    -.3819013 6 1 0 0
 75  90    -1.604729 5 1 0 0
 80 100     .9973673 5 1 0 0
 91  95     .9489719 5 1 0 0
 80  70     .0442407 6 1 0 0
 90  90   -1.1140411 5 1 0 0
 85  95    -.3886355 5 1 0 0
 80 100    -.8701696 5 1 0 0
 71  86    .29373953 6 1 0 0
 70  75     .7725913 6 1 0 0
 61 100    .39805225 5 1 0 0
 97  97   -1.0581238 5 1 0 0
 57  79   -.26228225 5 1 0 0
 61   0            . 6 1 0 0
 95  98    -.7997336 5 1 0 0
 90  97    -.1759583 5 1 0 0
 80  95    -.3837959 5 1 0 0
 85  82   -.02135567 6 1 0 0
 81 100     .7774308 5 1 0 0
 80  68   -.57227534 5 1 0 0
 90 100     .4023666 6 1 0 0
  1   0    -.6865296 6 1 0 0
 95  91   -.40315405 6 1 0 0
 75  70     .8395571 6 1 0 0
 66  85   -1.0175129 5 1 0 0
 87  88     .3464493 6 1 0 0
 80 100    2.1029394 6 1 0 0
 60  50     -.105785 5 1 0 0
 70  90    1.6189854 6 1 0 0
 91  98    1.6144085 5 1 0 0
 44  77     .6981037 5 1 0 0
 69  78     .6104667 5 0 1 0
 90  80            . 5 0 1 0
 76 100   -1.0538094 5 0 1 0
 20   5   -.25849307 5 0 1 0
  .   .     .8250384 6 0 1 0
 70  70            . 6 0 1 0
 10  90   -.26091287 5 0 1 0
 45  71    1.0922635 6 0 1 0
 70  70    -.8094127 6 0 1 0
 70  85  -.016516132 6 0 1 0
 47  47    -.3789564 5 0 1 0
 85  85    .29053202 6 0 1 0
 81  73   -.22703607 6 0 1 0
 61  82            . 5 0 1 0
 59  39   -.21414946 6 0 1 0
 60  84            . 5 0 1 0
 60  70   -.44455275 5 0 1 0
100 100     .3483439 6 0 1 0
 80  70      .706995 5 0 1 0
 60  80    .05391978 6 0 1 0
 73 100    -.7007857 5 0 1 0
 95   0   -.21251744 6 0 1 0
 80  70    -.6281926 6 0 1 0
 76  88     .5306144 6 0 1 0
 83  97   -.38647825 6 0 1 0
 16  20     .4867959 5 0 1 0
 78  59     .6615445 6 0 1 0
 75  90   -.01893589 5 0 1 0
 68  73   -1.0027317 5 0 1 0
 79  67    -.0745906 6 0 1 0
 94  98     .8371373 5 0 1 0
 90  84    .10878672 6 0 1 0
 17  95    -1.715776 5 0 1 0
 20  90    2.5844736 5 0 1 0
 60  40    1.1035746 6 0 1 0
end
label values NBpriorBehOthr NBpriorBehOthr
label values Gender Gender_lbl
label def Gender_lbl 5 "male", modify
label def Gender_lbl 6 "female", modify

estimates clear
local i 1
foreach var in NBpriorBehOthr NBpriorNorOthr Altr Gender{
    rename dGroupid1 dGroupid1`i'
    eststo m`i': regress `var' 0.dGroupid1`i', nocons robust
    rename dGroupid1`i' holding
    recode holding (1=0) (0=1), gen(dGroupid1`i')
    eststo k`i': regress `var' 0.dGroupid1`i', nocons robust
    eststo d`i': regress `var' 0.dGroupid1`i', robust  
    drop dGroupid1`i'
    rename holding dGroupid1
    local ++i
}



*PROGRAM TO APPEND MODELS
capt prog drop appendmodels
*! version 1.0.0  14aug2007  Ben Jann
program appendmodels, eclass
    // using first equation of model
    version 8
    syntax namelist
    tempname b V tmp
    foreach name of local namelist {
        qui est restore `name'
        mat `tmp' = e(b)
        local eq1: coleq `tmp'
        gettoken eq1 : eq1
        mat `tmp' = `tmp'[1,"`eq1':"]
        local cons = colnumb(`tmp',"_cons")
        if `cons'<. & `cons'>1 {
            mat `tmp' = `tmp'[1,1..`cons'-1]
        }
        mat `b' = nullmat(`b') , `tmp'
        mat `tmp' = e(V)
        mat `tmp' = `tmp'["`eq1':","`eq1':"]
        if `cons'<. & `cons'>1 {
            mat `tmp' = `tmp'[1..`cons'-1,1..`cons'-1]
        }
        capt confirm matrix `V'
        if _rc {
            mat `V' = `tmp'
        }
        else {
            mat `V' = ///
            ( `V' , J(rowsof(`V'),colsof(`tmp'),0) ) \ ///
            ( J(rowsof(`tmp'),colsof(`V'),0) , `tmp' )
        }
    }
    local names: colfullnames `b'
    mat coln `V' = `names'
    mat rown `V' = `names'
    eret post `b' `V'
    eret local cmd "regress"
end
local m
local k
local d
foreach l in m k d{
    forval i=1/4{
        local `l' "``l''  `l'`i'"
    }
}
eststo M: appendmodels `m'
qui count if  !dGroupid1
estadd scalar N= r(N)
eststo K: appendmodels `k'
qui count if  dGroupid1
estadd scalar N= r(N)
eststo D: appendmodels `d'
qui count
estadd scalar N= r(N)
local coef 0.dGroupid1
esttab M K D, se scalars(N) coeflab(`coef'1 "NBpriorBehOthr" `coef'2 "NBpriorNorOthr" ///
`coef'3 "Altr" `coef'4 "Gender") varwidth(20) mlab("Group A" "Group B" "Difference")

Res.:

. esttab M K D, se scalars(N) coeflab(`coef'1 "NBpriorBehOthr" `coef'2 "NBpriorNorOthr" ///
> `coef'3 "Altr" `coef'4 "Gender") varwidth(20) mlab("Group A" "Group B" "Difference")

--------------------------------------------------------------------
                              (1)             (2)             (3)  
                          Group A         Group B      Difference  
--------------------------------------------------------------------
NBpriorBehOthr              65.21***        74.05***        8.840  
                          (4.093)         (2.329)         (4.733)  

NBpriorNorOthr              72.06***        83.31***        11.25*  
                          (4.348)         (2.504)         (5.043)  

Altr                       0.0592         -0.0517          -0.111  
                          (0.144)         (0.109)         (0.182)  

Gender                      5.514***        5.415***      -0.0989  
                         (0.0849)        (0.0614)         (0.105)  
--------------------------------------------------------------------
N                              35              65             100  
--------------------------------------------------------------------
Standard errors in parentheses
* p<0.05, ** p<0.01, *** p<0.001

[Janina Gleed]

Hello Andrew,

Thank you very much for your guidance as this really helped me move forwards with my code and I have taken lots from it since.

I am sorry for my delayed response as I only noticed now that my initial reply had not posted.

Thank you,
Janina