Keeping shell open and active until shell command is complete

[Drew Berrett]

I am currently using Stata 14.2 to execute a shell command but am running into an issue with the shell closing before the command is completed. A similar topic was previously discussed here:

https://www.stata.com/statalist/arch.../msg00295.html

However, the provided solution does not appear to work in my case.

Specifically, I am currently attempting to submit the following code:

shell CsvGenerator.exe orgs  ///
    -s 08/01/2018 -e 06/10/2019 -c -u ///
    /*userloginaddress*/ ///
    /*organizationid*/

The CsvGenerator.exe program is a multi-step program intended to search our company database for user information for an organization (which consists of several sub-organizations) and then ultimately exports the data as a .csv. The program is iterative in nature in that individual exports are created for each sub-organization which are then all appended together for the final full organization .csv export).

Is seems that due to this iterative nature, Stata will close the Windows shell as soon as the first operation within the CsvGenerator.exe is complete. As mentioned, a previously suggested solution is to include a check for the final .csv export or file creation before concluding. However, in my case, including this check does not appear to have convinced Stata to keep the shell open any longer. This is the version with the check in place:

capture noisily confirm file /*file path & name*/
while _rc {
shell CsvGenerator.exe orgs  ///
    -s 08/01/2018 -e 06/10/2019 -c -u ///
    /*userloginaddress*/ ///
     /*organizationid*/ 
}

I would appreciate any solutions for keeping the shell open so that the CsvGenerator.exe can complete all tasks instead of just the first one in its sequence.

[William Lisowski]

I do not believe you have fully understood the advice in the referenced post.

Thee code below, with the exception of your shell command, is taken from the referenced post. You will replace both occurrences of

c:\wherever\output.txt

with the path to and name of a file created by CavGenerator.exe just before it is fully finished. Presumably that could be the "final full organization .csv export", provided it is not created initially from the first export and then repeatedly updated with the results of each export as they become available.

shell CsvGenerator.exe orgs  ///
      -s 08/01/2018 -e 06/10/2019 -c -u ///
      /*userloginaddress*/ ///
      /*organizationid*/ 

capture confirm file c:\wherever\output.txt
while _rc {      /* file does not exist! */
    sleep 1000   /* wait 1000 ms = 1 second before trying again */
    capture confirm file c:\wherever\output.txt
}

[Drew Berrett]

Thank you William for your feedback. You are right, it looks like I misinterpreted where I should be placing the shell command in relation to the capture confirm.

Unfortunately, in using the formatting that you suggested, it still does not appear to be working. What seems to be happening (and please pardon the non-technical description due to my intermediate level of Stata skill) is that the shell will still close before the capture confirm has had a chance to actually do its check. That's my guess anyway. Once the shell program (CsvGenerator) completes it's first operation, the shell closes and then Stata moves on to do the capture confirm check. What I need is for the capture confirm to occur before the shell closes.

Of course, if there were some way to just tell Stata to keep the shell open indefinitely, that would be a far easier solution.

[William Lisowski]

Once the shell program (CsvGenerator) completes it's first operation, the shell closes and then Stata moves on to do the capture confirm check. What I need is for the capture confirm to occur before the shell closes.

Of course, if there were some way to just tell Stata to keep the shell open indefinitely, that would be a far easier solution.

You misunderstand what is happening. Stata turns control over to a Windows command shell in which the CsvGenerator program is run, and then Stata waits for that Windows command shell to terminate. Stata does not as you asserted in post #1 "close the Windows shell as soon as the first operation within the CsvGenerator.exe is complete" because Stata has no control over the Windows command shell or knowledge of what it is doing.

I expect that CsvGenerator does nothing more than submit a program to your corporate database. After doing that, CsvGenerator says "my job is done" and does not wait for the database to complete its work before terminating and returning control to the Windows command shell, which in turn terminates and returns control to Stata.

So because of that, we have to add a loop in your do-file so that - once CsvGenerator terminates and the Windows shell terminates and control is returned to Stata - Stata can continually check if the CsvGenerator output file it needs actually exists, at which time the looping ceases and Stata continues to execute the remainder of the do-file.

Of course, if there were some way to just tell CsvGenerator to not return to the Windows shell until its output file is ready, that would be a far easier solution. Perhaps there is such an option to CsvGenerator - you might ask your IT support if that is the case.

Since the Stata loop checking the output file ceases immediately, it suggests that you are not deleting the output file created by an earlier run of CsvGenerator before running your do-file, so that the old version is immediately found and the loop allows Stata execution to continue. The code below suggests a way to delete the output file automatically. But another possibility is that CsvGenerator creates an empty output file at the start, which is filled as the work is done. In the latter case, you need to talk to your IT support to find out to know positively when the task started by CsvGenerator is done.

erase c:\wherever\output.txt

shell CsvGenerator.exe orgs  ///
      -s 08/01/2018 -e 06/10/2019 -c -u ///
      /*userloginaddress*/ ///
      /*organizationid*/

capture confirm file c:\wherever\output.txt
while _rc {      /* file does not exist! */
    sleep 1000   /* wait 1000 ms = 1 second before trying again */
    capture confirm file c:\wherever\output.txt
}

[Drew Berrett]

William,

You were correct. I was assuming Stata was somehow in charge of deciding when the shell should end its session so that Stata could move on.

The problem appeared to be with the default behavior of the Windows command shell itself. Adding cmd /k in front of everything kept the windows command shell open so that the CsvGenerator.exe could complete all functions. The below code ultimately was all I needed (highlighting added to denote added code):

 
 shell (cmd /k CsvGenerator.exe orgs  ///       -s 08/01/2018 -e 06/10/2019 -c -u ///       /*userloginaddress*/ ///       /*organizationid*/)

Your feedback was much appreciated.

[William Lisowski]

I'm not a Window user, so this could be a dumb question.

The documentation for cmd /k that turns up in a web search leads me to believe that the Windows command shell will stay open as an interactive session until the user closes it. What causes the Windows command shell to eventually terminate and return control to Stata once CsvGenerator is done?

[Drew Berrett]

To be honest, I'm not sure why cmd /k was required in this case but not in others. Historically, I have manually opened Windows Command Prompt and ran the CsvGenerator.exe program there without any issues of things closing pre-maturely. The cmd /k was not required until I attempted to run the CsvGenerator.exe program through the Stata shell command. This has caused me to wonder if the shell that Stata calls up does not have the same properties as the standard Windows Command Prompt.

Again, I'm no guru on software or back-end coding of any sort. Just my observations.