Stata Crashing Because of Tmp Files When Running Parallel Instances

[Ricardo Dahis]

Dear StataList Users,

I have the following problem.

I'm running multiple parallel Stata sessions in a server, using commands such as nohup stata -b dofile.do &. Each dofile.do is a standalone piece of code, so I run many at the same time calling multiple Stata sessions in multiple cores.

However, the programs eventually crash throwing errors such as "file /tmp/St14980.000003 already exists" or "file /tmp/St22536.000002 could not be opened". It does not happen every time in the same place, and sometimes it doesn't happen at all. But it does happen with probability >0. For instance, later, if I run the same dofile.do that crashed, it runs smoothly.

Is there a good solution for this? Is there a correct way to run parallel Stata instances?

It's hard to fix it since errors are unpredictable; they depend on some way that Stata has in creating temporary files, naming and deleting them. I've read this but still couldn't figure it out.

Thanks a lot!

[William Lisowski]

Cross-posted on Stack Exchange at https://stackoverflow.com/questions/...llel-instances

Welcome to Statalist, Ricardo. Please review the Statalist FAQ linked to from the top of the page, as well as from the Advice on Posting link on the page you used to create your post. Section 8 on cross-posting requests

But if you do post elsewhere, we ask that you provide cross-references in URL form to searchable archives. That way, people interested in your question can quickly check what has been said elsewhere and avoid posting similar comments. Being open about cross-posting saves everyone time.

[William Lisowski]

It is possible that you can use the article linked below to create a shell script that, before running the stata command creates a distinctly-named directory to be used for temporary files and runs stata with the environment variable STATATMP set appropriately. This way, each instance of Stata will place its temporary files in a different directory, and then the file naming convention, whatever it is, won't matter.

https://www.stata.com/support/faqs/d...ment-variable/

[Ricardo Dahis]

Solved. Thanks!

[wbuchanan]

The parallel package described at the 2017 North American Stata Users Group meeting has a way to handle this automatically. It might be worth checking that program out.

[Andrew Wade]

Hello,
I had the exact same problem to that listed above.
I resolved by placing the do files (to run in parallel) in separate folders.
That way the temp data files are placed in separate files.
Maybe Stata could address this issue by using some form of randomisation to generate temp files names that don't conflict, when jobs are running in parallel.

Regards,
Andrew