Open do-file in separate Stata console from master dofile?

[Johan Karlsson]

Dear Statalist,

So the project that I am currently working on requires the creation of a lot of synthetic data for machine learning. To speed up the process, I have divided it across 12 different dofiles to run the program i parallel.
However, this also then requires me to click on each dofile to start a separate console, and then run it.

My question is now: Is there a way to execute this process from a master dofile that would allow the dofiles to run in parallel on separate consoles?

I have tried the "parallel" command previously and although it runs, it only seemed to create some auxilliary files in my dofile folder, whereas the dofiles themselves weren't executed.

Sincerely
Johan

[Leonardo Guizzetti]

Though I'm not familiar with these programs, the user contributed -parallel- (st0572 from Stata Journal 19(3):667--684) would appear to do what you want. There is also -batcher- (in the SSC) which is a Windows-only solution to parallelize tasks.

[Bjarte Aagnes]

OS?

[Johan Karlsson]

Though I'm not familiar with these programs, the user contributed -parallel- (st0572 from Stata Journal 19(3):667--684) would appear to do what you want. There is also -batcher- (in the SSC) which is a Windows-only solution to parallelize tasks.

Sounds interesting! I will look into this one. Thank you Leonardo!

[Johan Karlsson]

OS?

Windows!

[Bjarte Aagnes]

if on MS Windows you can also try multishell

ssc install multishell

[Oscar Ozfidan]

I am using batcher and handling approx 600 instances on the host (max 34) at a time, 300 instances on one VM (about max 12-14 at a time) and another 300 instances on a second VM (about max 12-14 at a time) with no problems, so about 60 instances at one time in all. I have to manually kick off the batcher on VMs just once and go on with my day to come back too see it al done. I am currently working on a solution to get the batcher started on VMs through python which would be kicked off by the master do file on the host. I am not sure if Jesse fixed some issues I was having with batcher but if you run into some quirks with it I have fixed it all myself and tested extensively and could share the revised ado file ....

[Jesse Wursten]

I am using batcher and handling approx 600 instances on the host (max 34) at a time, 300 instances on one VM (about max 12-14 at a time) and another 300 instances on a second VM (about max 12-14 at a time) with no problems, so about 60 instances at one time in all. I have to manually kick off the batcher on VMs just once and go on with my day to come back too see it al done. I am currently working on a solution to get the batcher started on VMs through python which would be kicked off by the master do file on the host. I am not sure if Jesse fixed some issues I was having with batcher but if you run into some quirks with it I have fixed it all myself and tested extensively and could share the revised ado file ....

The quirks mentioned here have been fixed in the latest version on SSC, for anyone stumbling upon this thread.