Very slow loop

[Fredrick orwa]

Dear team,
Can someone advice me. My loop is so slow:
levelsof serotype, local(levels)
foreach i of local levels{
cap betabin count_carr pcv7 pcv10 pcv13 lmic umic hic age5_14 age15more ageallages if serotype=="`i'", n(popnsampled_carr) link(cloglog)
di "`i'"
if _rc==0{
predict mu if serotype=="`i'"
replace imputed_preva=1 if serotype=="`i'"
replace est_eb =(mu/sig + count_carr)/(mu/sig + (1-mu)/sig + popnsampled_carr) if serotype=="`i'"
drop mu
}
***block
else if _rc!=0{
di "model1 did not converge"
cap betabin count_carr postpcv hic age5_14 age15more ageallages if serotype=="`i'", n(popnsampled_carr) link(cloglog)
if _rc==0{
predict mu if serotype=="`i'"
replace imputed_preva=1 if serotype=="`i'"
replace est_eb =(mu/sig + count_carr)/(mu/sig + (1-mu)/sig + popnsampled_carr) if serotype=="`i'"
drop mu
}
else if _rc!=0{
cap betabin count_carr if serotype=="`i'", n(popnsampled_carr) link(cloglog)
if _rc==0{
predict mu if serotype=="`i'"
replace imputed_preva=1 if serotype=="`i'"
replace est_eb =(mu/sig + count_carr)/(mu/sig + (1-mu)/sig + popnsampled_carr) if serotype=="`i'"
drop mu
}

else if _rc!=0{
replace no_coverge_preva=1 if serotype=="`i'"
}
}
}
**end block
}

[Clyde Schechter]

I take it you have a large data set (large number of observations) and perhaps a large number of different serotypes as well. The problem is likely due to all those -if serotype == "`i'"- clauses. Each of those requires Stata to go through every observation in the data set to check the value of serotype, and it does this every time through the loop. This kind of problem can be greatly speeded up by using Robert Picard's and my -runby- command, available from SSC. In fact, this is precisely what -runby- was written for.

capture program drop one_serotype
program define one_serotype
    cap betabin count_carr pcv7 pcv10 pcv13 lmic umic hic age5_14 age15more ageallages , n(popnsampled_carr) link(cloglog)
    di serotype[1]
    if _rc==0{
        predict mu
        gen imputed_preva=1
        gen est_eb =(mu/sig + count_carr)/(mu/sig + (1-mu)/sig + popnsampled_carr)
        drop mu
    }
    ***block
    else if _rc!=0{
        di "model1 did not converge"
        cap betabin count_carr postpcv hic age5_14 age15more ageallages , n(popnsampled_carr) link(cloglog)
        if _rc==0{
            predict mu
            gen imputed_preva=1
            gen est_eb =(mu/sig + count_carr)/(mu/sig + (1-mu)/sig + popnsampled_carr)
            drop mu
        }
        else if _rc!=0{
            cap betabin count_carr , n(popnsampled_carr) link(cloglog)
            if _rc==0{
                predict mu
                gen imputed_preva=1
                gen est_eb =(mu/sig + count_carr)/(mu/sig + (1-mu)/sig + popnsampled_carr)
                drop mu
            }

            else if _rc!=0{
                gen no_coverge_preva=1
            }
        }
    }
    exit
end

runby one_serotype, by(serotype) status

Notes:

1. You didn't provide any example data, so I have not tested this.
2. I didn't scrutinize every detail of your original code, so it is possible that some of my modifications are incorrect. Basically, I encapsulated your outer loop in a program, removed all the -if serotype == "`i'"- clauses, and changed all the -replace- commands to -gen-. I also added indentation to make the code readable.
3. To debug this, add the -verbose- option to the -runby- command so that you will see the output as you go along, including any error messages generated along the way. Once you've got it running well, take away the -verbose- option and all of the output, including your -display- messages will disappear.
4. The -status- option gives you a progress report as you go. That tells you how many observations have been processed so far, in how much time, and an estimate of time to completion.

[Fredrick orwa]

Thanks so much Clyde.Am still lost a little but am working on my code as per your modification

[Fredrick orwa]

Thanks Clyde I have managed to edit my code as per your request.It is still proving slow but let me hope it doesn't run for 24 hours like mine.

[Fredrick orwa]

Thanks so much Clyde this is very much better.Within 40 minutes its done