Loop built around program does not 'loop'

[Mohamed El Alili]

Hello fellow stata-ticians,

I have been working on a cost effectiveness analysis in which I wrote program with xtreg. The next step is to let this program loop over 1000 simulations, and to do that I built a forvalues loop. However, the loop stops right away after the first analysis. What can I do to make it work? (I suspect the bootsrap program (in bold) that ends my loop, i also tried it with removing 'end' after 'restore' but it still does not work. Thanks in advance!

Find below the stata code:

local n = 10

forvalues j=1(1) `n' {
local y = `j'

use Dataset`y', clear
gen ID_dataset = `y'

* step 1: Produce initial estimates and store in Matrix observe!
quietly xtreg LROT trtGrp, i(idHospital) mle
mat betas = e(b)
scalar betaEFFECT = betas[1,1]
quietly xtreg Costs trtGrp, i(idHospital) mle
mat betas = e(b)
scalar betaCOST = betas[1,1]
matrix observe = (betaEFFECT, betaCOST)

*Step 2: Write bootstrap program myboot --> Bootstrap (bsample)
capture program drop myboot
program define myboot, rclass
preserve
bsample, str(idHospital)
xtreg LROT trtGrp, i(idHospital) mle
mat betasE = e(b)
return scalar betaEFFECT = betasE[1,1]
xtreg Costs trtGrp, i(idHospital) mle
mat betasC = e(b)
return scalar betaCOST = betasC[1,1]
restore
end

* Step 3: simulate with monte carlo simulations!
simulate BETAEFFECT = r(betaEFFECT) BETACOST = r(betaCOST), reps(50) saving(bootsamples_imp`y', replace) seed(30): myboot

*Step 4: give bootstrap results!
bstat, stat(observe) n(2664)

estat bootstrap, all

*** Extract data from bivariate regression analysis
mat betaCE= e(b) /* extract the matrix of regression coefficients */
mat se = e(se) /* extract standard errors */
mat vari = e(V) /* extract the variance-covariance matrix within-imputation variability (uncertainty per imp) and between-imputation variability (uncertainty due to missing information) */
mat limits = e(ci_bc) /* extract lower en upper limit bc interval */

gen CD = betaCE[1,colnumb(betaCE,"BETACOST")] /* create differential costs */
gen ED = betaCE[1,colnumb(betaCE,"BETAEFFECT")] /* create differential EFFECTS */
gen varCD = vari[colnumb(vari,"BETACOST"),rownumb(vari,"BETACOST")] /* extract the variance of the mean differential costs from the VC matrix */
gen varED = vari[colnumb(vari,"BETAEFFECT"),rownumb(vari,"BETAEFFEC T")] /* extract the variance of the mean differential EFFECTS from the VC matrix */
gen cov = vari[colnumb(vari,"BETACOST"),rownumb(vari,"BETAEFFECT" )] /* extract the covariance between mean differential costs and EFFECTS */
gen N = e(N) /* extract sample size GECHECKT */
gen cor = cov /(varED*varCD) /* calculate the correlation coefficient, UITLEG: "[COVARIANCE MEAN DIFFERENTAL COSTS AND EFFECTS] / ([variance of the mean differential COSTS])* ([variance of the mean differential EFFECTS]) */

gen LLcost = limits[1, colnumb(limits,"BETACOST")]
gen UPcost = limits[2, colnumb(limits,"BETACOST")]
gen LLeffect = limits[1, colnumb(limits,"BETAEFFECT")]
gen UPeffect = limits[2, colnumb(limits,"BETAEFFECT")]
gen SE_CD = se[1,1]
gen SE_ED = se[1,2]

save postBoots_samples_imp`y', replace

}

[William Lisowski]

Welcome to Statalist.

You are likely correct that the program define with its associated end command is causing your problem. Here's a simplified example.

forvalues j=1(1)3 {
di `j'
capture program drop foo
program define foo
di "running foo"
end
foo
}

and running that from the do-file editor results in the following.

. do "/var/folders/xr/lm5ccr996k7dspxs35yqzyt80000gp/T//SD12471.000000"

. forvalues j=1(1)3 {
  2. di `j'
  3. capture program drop foo
  4. program define foo
  5. di "running foo"
  6. end
--Break--
r(1);

end of do-file

--Break--
r(1);

.

But you don't need to define the program within the loop. Define it once, before you start the loop.

capture program drop foo
program define foo
di "running foo"
end
forvalues j=1(1)3 {
di `j'
foo
}

. do "/var/folders/xr/lm5ccr996k7dspxs35yqzyt80000gp/T//SD12471.000000"

. capture program drop foo

. program define foo
  1. di "running foo"
  2. end

. forvalues j=1(1)3 {
  2. di `j'
  3. foo
  4. }
1
running foo
2
running foo
3
running foo

. 
end of do-file

.

[Mohamed El Alili]

Thank you William, I will try to figure this out by defining the program first. I will post it here whether it works for me or not!

[Mohamed El Alili]

It worked, I placed the program (in bold in first post) before the forvalues loop and it went through all the datasets! Many thanks again!

[Giulio Zanella]

Hi William, I have a question related to this old post because I'm facing a similar problem but your proposed solution does not work for me because I would like to loop over programs that contain the looping variable and so differ from each other. Here's what I mean, using your own example (in boldface my modification):

capture program drop foo
program define foo
di "running foo for the `j'th time"
end

forvalues j=4(1)6 {
di `j'
foo
}

This does not work because it produces

4
running foo for the th time
5
running foo for the th time
6
running foo for the th time

instead of

4
running foo for the 4th time
5
running foo for the 5th time
6
running foo for the 6th time

which is what I need.

Of course the following (which was my first natural attempt) does not work either:

forvalues j=4(1)6 {
capture program drop foo`j'
program define foo`j'
di "running foo for the `j'th time"
end
}

forvalues j=4(1)6 {
foo `j'
}

Any hint would be greatly appreciated, thanks.

[William Lisowski]

The way you make a program aware of a local macro defiined outside the program is to pass the value of the local macro in as an argument. In the code below, the value of the local macro j is the first (and only!) argument on the line that calls the foo program, and within the foo program, `1' refers to the first argument passed into foo on the line that called it.

cls
capture program drop foo
program define foo
di "running foo for the `1'th time"
end

forvalues j=4(1)6 {
di `j'
foo `j'
}

foo million

4
running foo for the 4th time
5
running foo for the 5th time
6
running foo for the 6th time

. 
. foo million
running foo for the millionth time

For a much more comprehensive explanation of passing arguments into programs, read section 18.4 of the Stata User's Guide PDF included in your Stata installation and accessible through Stata's Help menu.

[Giulio Zanella]

I see. Thank you!

[Jonathan Horowitz]

I'm running into a similar problem, where it doesn't seem to quite work. I can display text fine, but when I try to get the program to loop over a variable in a regression it doesn't work. It's probably just that I need to put something in parentheses or quotes or brackets, but I can't quite find an example in the program manual that seems to solve it

I've customized this program here: https://stats.idre.ucla.edu/stata/fa...-imputed-data/
So that loops over both write and math as the first variable. But I'm getting an error, suggesting it's not parsing the `j' correctly in the bootstrap command.

capture program drop  myboot
program define myboot, rclass

   mi estimate: regress read `1' female

   return scalar b_a = el(e(b_mi),1,1)
   return scalar b_f = el(e(b_mi),1,2)
   return scalar b_c = el(e(b_mi),1,3)
end


foreach j of varlist write math {
di "running `j'"
bootstrap `j'=r(b_a) b_female=r(b_f) intercept=r(b_c), reps(5) : myboot
}

I get:

. foreach j of varlist write math {
  2. di "running `j'"
  3. bootstrap `j'=r(b_a) b_female=r(b_f) intercept=r(b_c), reps(5) : myboot
  4. }
running write
(running myboot on estimation sample)
'r(b_c)' evaluated to missing in full sample

[Bjarte Aagnes]

In your program myboot you refere to `1' , but when used myboot is called without any argument. Using the syntax command help returning informative error messages:

program define myboot, rclass

   syntax varlist(max=1 numeric)    

   mi estimate: regress read `varlist' female

   ...

end

[Jonathan Horowitz]

Thank you! It took me a few minutes for me to figure out where the varlist went and its relationship to `j' earlier, but I think I've got it running. Here's the full code, written with two arguments:

capture program drop  myboot
program define myboot, rclass
   syntax varlist(max=2 numeric)
   mi estimate: regress read `varlist' female

   return scalar b_a = el(e(b_mi),1,1)
   return scalar b_f = el(e(b_mi),1,2)
   return scalar b_c = el(e(b_mi),1,3)
end


foreach j in write math {
di "running `j'"
di `j'
bootstrap `j'=r(b_a) b_female=r(b_f) intercept=r(b_c), reps(5) : myboot `j'
}

[Bjarte Aagnes]

syntax varlist(max=1 numeric)

restrict the number of variables given in any single call to the program to one, which seems correct.

[Jonathan Horowitz]

Ah yes, that was part of my attempt to generalize it to multiple variables in the varlist (two arguments), which I mostly undid when I posted it here. But you are correct.

[Ting Meng]

Dear all,
I have a similar problem when I need to loop two parameters (dd and theta) in program myest. It reports error as "varlist required".
Would you please help me to figure it out? Many thanks.



capture program drop myest
program define myest
version 14
args lnf Xb
tempvar dd theta
quietly replace `lnf' = ln(normal((ln(300/((1-exp(-1*`dd'-0.1))/(`dd'+0.1)*exp(-1*`d'*2)))-`Xb')/`theta')) if $ML_y1==2
quietly replace `lnf' = ln(1-normal((ln(300/((1-exp(-1*`dd'-0.1))/(`dd'+0.1)*exp(-1*`d'*19)))-`Xb')/`theta')) if $ML_y1==20
quietly replace `lnf' = ln(normal((ln(300/exp(-1*$ML_y1*`dd')*((1-exp(-1*`dd'-0.1))/(`d'+0.1)))-`Xb')/`theta')-normal((ln(300/exp(-1*($ML_y1-1)*`d')*((1-exp(-1*`d'-0.1))/(`d'+0.1)))-`Xb')/`theta')) if ($ML_y1<20)&($ML_y1>2)
end

forvalues dd = 0.1(0.1)0.2 {
forvalues theta = 1(1)2 {
* display `i'
* display `j'
myest
ml model lf myest (y= x1 x2)
ml maximize
display `dd'
display `theta'
display e(ll)
}
}