Forking a shell on a Mac

[Clyde Schechter]

I work on Windows. I have a workflow for one of my projects that looks like this:

foreach x of .... {
     // DO THINGS BASED ON `x' TO CREATE AND EXPORT SOME TEXT FILES

    shell executable_compiled_from_c++_program_using_those_t ext_files_as_inputs_creating_new_output_files

    // USE OUTPUT FILES CREATED BY THE ABOVE EXECUTABLE AS INPUTS
    // TO CREATE FINAL OUTPUT FILES (.DTA AND .XLSX).
}

Now, that C++ program, in addition to creating output files, because it runs for a long time, also outputs progress reports to the window created by -shell-. Those progress reports are of no value once the program has completed, and they disappear with the -shell- window.

I now need to run a script similar to this, but where the number of iterations of the loop is going to be very large and the total execution time will be measured in months. Since each iteration of the loop could, in principle, be done independently of all the others, I have persuaded some colleagues to let me run this on their computers, dividing up the work. The catch is that some of them use Macs. According to the Stata documentation, -shell-, when run on Mac, will redirect all those progress reports from the C++ program to the Results window. That's not necessary, and not desirable as the information in the Results window about which iteration of the loop we are on will scroll off the screen.

From the documentation, it looks to me as if the -xshell- command is what I need. Am I correct in understanding that -xshell- on Mac does both of the following?

1. Opens a separate window for the C++ program which will receive the progress report outputs and then close once the C++ program terminates, sending nothing to the Results window.
2. Suspends Stata until the program run under -xshell- terminates.

I don't have a Mac available to try this out on. I know there are a lot of Mac users on Statalist, so I'm hoping one of them will advise me.

[William Lisowski]

I have no experience with xterm, but my understanding is that it is no more than an xwindows alternative to the macOS Terminal window, and as such offers no advantage over the latter for what you are doing, since you are not creating xwindows graphics in what you are doing. In particular it is not a priori clear to me that it will close itself once the C++ program terminates and return to Stata. And in any event it installs user installation.

Here are some examples of alternative approaches using shell and winexec that may start you in a useful direction. Steps 1 and 2 are the equivalent of creating a macOS executable compiled from your C++ code. Let me know if you have any questions.

1) I used a text editor to create ~/Downloads/executable_creating_new_output_file.sh containing the following script.

#!/bin/tcsh
echo "bedtime"
sleep 10
ls -l > ~/Downloads/new_output_file.txt
echo "Hello, world!"
exit

2) I opened the macOS Terminal window and made that script file executable.

chmod +x ~/Downloads/executable_creating_new_output_file.sh

3) From within Stata I ran the following command.

shell /System/Applications/Utilities/Terminal.app/Contents/MacOS/Terminal ~/Downloads/executable_creating_new_output_file.sh

• A macOS terminal window opened
• Control did not return to Stata
• The script ran in the terminal window, completed its tasks, creating ~/Downloads/new_output_file.txt
• The terminal window did not close, and technically the Terminal app continued running
• When I manually quit the Terminal app, control returned to Stata

4) From within Stata I ran the following command.

winexec /System/Applications/Utilities/Terminal.app/Contents/MacOS/Terminal ~/Downloads/executable_creating_new_output_file.sh

• A macOS terminal window opened
• Control returned to Stata
• The script ran in the terminal window, completed its tasks, creating ~/Downloads/new_output_file.txt
• The terminal window did not close, and technically the Terminal app continued running

I will note that once control returned to Stata, I was able to run a second winexec command opening a new Terminal window running simultaneously with the initial one.

[Clyde Schechter]

Thanks, William. I really appreciate this. Very helpful.

[Tiago Pereira]

Not sure if this is going to be useful, Clyde. However, on a Mac and linux you can use sed and sleep, which can be used to run independent instances of your simulations/analyses. Assume that you need 1000 simulations/iterations and has available 10 cores. Simulations can be run independently. Suppose your do-file has the following structure:

local min = 123456
local max = 123456

forvalues i = `min'/`max' {

[run this code]

}

You can used sed to edit your do files. Let's say you want to run simulations 1-100 using core 1, 101-200 using core 2, and so on.

    cd "the path that contains your do-file for analysis"
j = 1
for i in {1..1000..100}; do
 echo "Number: $i"
 j=$(($i+99))
 echo "Number: $j"
     sed -e "s/local min = 123456789/local min = ${i}/g" mydofile.do > script.txt
      sed -i "s/local max = 123456789/local max = ${j}/g" script.txt
       /usr/local/stata16/xstata-se do "script.txt"  & sleep 10s
   done

So, basically, every 10 seconds you will trigger a new Stata and run a separate analysis, each time with increments of 100 (e.g., 1-100, 101-200, 201-300, etc).

[Clyde Schechter]

Thank you. I'm not sure how I might apply this to my situation, but I will look into it further.