Why the massive difference in run times in these two loops?

[David Roodman]

I run this on my computer in Stata 13.1:

mata t = 1
forvalues i=1/1000 {
    mata r=t
}

mata for (i=1000;i;i--) r=t

The first loop takes 30.38 seconds and the second takes 0.04 seconds, a ratio of 760. If I change

mata r=t

to

mata r=1

, the difference largely disappears.

The practical situation is that I am running -simulate- on what is essentially a Mata function. My -simulate- command line calls a little Stata program, which in turn runs the Mata function, then saves its results in r() macros to pass back to -simulate-. I was surprised to discover that assignments rather like this were dominating my run time and causing 100,000 simulations to take far longer than they should.


UPDATE: I just tried this another computer, and there was no problem. Then I opened a fresh instance of Stata on the original computer and again there was no problem. So there is something about the current environment in my current Stata instance that is happening as I run the simulations that is causing this massive slowdown.... Here is the log:

. mata t = 1
r; t=0.03 8:53:19

. forvalues i=1/1000 {
  2.         mata r=t
  3. }
r; t=30.19 8:53:49

. 
. mata for (i=1000;i;i--) r=t
r; t=0.04 8:53:49

. 
end of do-file

r; t=30.38 8:53:49
. di 30.38/.04
759.5
r; t=0.00 8:54:29

. do "C:\Users\David\AppData\Local\Temp\STD03000000.tmp"

. mata t = 1
r; t=0.05 8:55:04

. forvalues i=1/1000 {
  2.         mata r=1
  3. }
r; t=0.33 8:55:04

. 
. mata for (i=1000;i;i--) r=t
r; t=0.04 8:55:04

[Sergio Correia]

Yep, both loops are also fast on my side.

Have you checked if your memory usage is exploding (memory leaks), or maybe you end up with too many global objects in the mata namespace? (mata desc might be useful, not sure).

[David Roodman]

Thank you Sergio. Those are good suggestions and my intuition too is that it is something like that.

mata desc didn't reveal an unwanted accumulation of objects.

I decided to bypass the problem by not using -simulate-. But I am using the -parallel- package. So instead of doing "parallel sim", which calls -simulate- in each parallel instance, I'm just using "parallel:" to run a custom ado command that itself runs the Mata Monte Carlo simulation function, passing it the number of simulations to perform, then stores the matrix of results as Stata variables using getmata. It is much faster.

[Belinda Foster]

Memory leaks are often the culprit for this sort of problems. In fact, if you let your Stata copy open for a few days in a row, it can even crash. Results may also get distorted; it has happened to me more than once. So before a simulation I think it is best to restart Stata if it has been open for awhile.

[David Roodman]

Interesting. In struggling with this problem, it did seem to me that simulate got slower as it went along.

[Sebastian Kripfganz]

I can confirm both the problems that simulate considerably slows down as it goes along and that Stata can eventually crash if I do not clear the memory or restart the Stata instance in between. It is a bit annoying.

[Hua Peng (StataCorp)]

We would like to find out what the issue is. It would be great help if David Roodman can send us a do-file which exhibits the problem using -simulate-. David, either post code here or through tech support will be greatly appreciated.