run all do files in folder sequentially

[tim bradshaw]

Hi there - I would like to run all do files in one particular folder sequentially. The order in which they're run doesn't matter. If it helps, all of the do files have the same prefix. I'm finding this simple task surprisingly difficult!

Any advice very much appreciated.

[Nick Cox]

Copy the filenames into a master .do file with commands

do whatever.do
do somethingelse.do

(or run as the case may be)

and then do or run that.

[Leonardo Guizzetti]

I like Nick's solution because it is straightforward and efficient for a small number of files. I have made a somewhat more general example which will automatically grab do files (based on pattern matching) and in turn, execute each one. This works because of the macro extended function which can extract file names and put them as a list into a macro variable.

* write two test files to some directory.
cd "C:\temp\example"
* file 1
file open file1 using "file1.do", write text replace
file write file1 `"display "This is file 1.""'
file close file1
* file 2
file open file2 using "file2.do", write text replace
file write file2 `"display "This is file 2.""'
file close file2

* verify the files exist, and their contents
dir "*.do"
do "file1.do"
do "file2.do"

* now loop over the files
local flist : dir . files "*.do", respectcase
di `"`flist'"' // show you the filelist
foreach fname of local flist {
  di as red "Executing file: `fname'"
  do "`fname'"
}

[tim bradshaw]

I was looking for Leonardo's solution (to avoid copying and pasting a large number of file names), and it did the trick. Thanks

[Nick Cox]

The two suggestions aren't really different. I didn't spell it out but if you have a long list of filenames then copying and pasting from a directory listing is a natural act, and with editing as well equivalent to Leonardo's suggestion. The amount of work is about the same....

[Antonio Schiavone]

I like Nick's solution because it is straightforward and efficient for a small number of files. I have made a somewhat more general example which will automatically grab do files (based on pattern matching) and in turn, execute each one. This works because of the macro extended function which can extract file names and put them as a list into a macro variable.

* write two test files to some directory.
cd "C:\temp\example"
* file 1
file open file1 using "file1.do", write text replace
file write file1 `"display "This is file 1.""'
file close file1
* file 2
file open file2 using "file2.do", write text replace
file write file2 `"display "This is file 2.""'
file close file2

* verify the files exist, and their contents
dir "*.do"
do "file1.do"
do "file2.do"

* now loop over the files
local flist : dir . files "*.do", respectcase
di `"`flist'"' // show you the filelist
foreach fname of local flist {
di as red "Executing file: `fname'"
do "`fname'"
}

Hi there, I was wondering whether there is an easy way of doing the same including all the subfolders in the main directory. I've been trying to include the program I found here (https://www.stata.com/statalist/arch.../msg01016.html) but I don't seem to be able to get it this way.

[Clyde Schechter]

cd my_main_directory // IF YOU'RE NOT ALREADY THERE
clear
filelist, pattern(*.do)

capture program drop one_do_file
program define one_do_file
    local todo = dirname[1] + "/" + filename[1]
    do `"`todo'"'
    clear
    exit
end

runby one_do_file, by(dirname filename)

-filelist- is written by Robert Picard. -runby- is wrirtten by Robert Picard and me. Both are available from SSC.

[Antonio Schiavone]

Thanks a lot, Clyde Schechter. It seems to work, but it stops after the first do file and gives me this message:


--------------------------------------
Number of by-groups = 42
by-groups with errors = 41
by-groups with no data = 1
Observations processed = 42
Observations saved = 0
--------------------------------------


Would you know why? I tried substituting [ _n] where you had [1], but that didn't work either.
I tried changing the directory to one of the subfolders and it did work on the first do file as well. So I don't think there is an issue with the do files, but rather with the loop stopping after the first file.
Forgive me if this is trivial, I usually work on R and this language is still a bit confusing to me.

[Clyde Schechter]

Add the -verbose- option to the -runby- command and you will get running output. That way you will be able to see what's going wrong.

I just tried the code from #2 in one of my own directories and got similar results to what you did. The major problem is that most of my do-files are written assuming that all of the files that it needs to read or write to will be in the same directory as the do-file itself, or perhaps in a subdirectory. But the code above does not change the working directory accordingly. So the do-files that were being executed were throwing errors about files not being found. That can be partly ameliorated done by modifying the code to do that:

capture program drop one_do_file
program define one_do_file
    local go_to - dirname[1]
    local return_to `c(pwd)'
    local todo = dirname[1] + "/" + filename[1]
    cd `"`go_to'"'
    do `"`todo'"'
    cd `"`return_to'"'
    clear
    exit
end

However, this may not entirely solve the problem if the do-files are throwing other kinds of errors, or if they were not written assuming that files being accessed would be located in the current directory or a subdirectory thereof. (For example, one of the do-files attempted to access a file in an unrelated directory, which was specified with a full path name, but that directory and file no longer exist.)

In short, it may or may not be feasible to fully automate the running of all these files, depending on the source of the errors. You will have to see what the specific errors the do-files are throwing are.

[Antonio Schiavone]

Ok so as you guessed the issue was with directories, i.e. once the first do file is over -runby- goes back to the list of do-files but cannot find the next one, since the directory has changed. I found a solution using your last suggestion:

capture program drop one_do_file
program define one_do_file
      local go_to = "main directory containing all do-files and subfolders"
      local todo = dirname[1] + "/" + filename[1]
      cd `"`go_to'"'
      do `"`todo'"'
      cd `"`go_to'"'
      clear
      exit
end

I believe the main difference with your solution to be the main directory. You used dirname[1] which is often a subfolder, hence making it impossible to reach do-files in different subfolders. This should work in any case where the directories are the problem.
Again, thank you very much

[Tim Plante]

I copied/pasted the last part of Antonio Schiavone's code into a do file and put it in the same folder as all of my other do files of interest. That "run all do files" do file was intentionally the first to appear alphabetically in that folder. The "run all do files" do file would run itself in an endless loop, not moving along to the other do files.

Below, I put in a count to get it to ignore the first do file (itself) and run the remaining do files in that same folder. This script assumes that you have the pwd as the folder with all of the do files of interest and that the "run all do files" do file is the first one listed alphabetically.

* now loop over the files
local flist : dir . files "*.do", respectcase
di `"`flist'"' // show you the filelist

local count=0
foreach fname of local flist {
    if `count'==0     {
        local count = `count'+1
    }
    else {
  di as red "Executing file: `fname'"
  do "`fname'"
  local count = `count'+1
    }
}