Foreach loop not capturing all files in folder

[Luke Robertson]

Hi,

I am using a panel data set with separate waves sorted into separate files each with the same ending "_indresp.dta". The file structure within the directory "~/datafiles" looks something like "ba_indresp.dta", "bb_indresp.dta", "bc_indresp.dta" etc until "br_indresp.dta" so 18 files to loop over. I am using a foreach loop to aggregate each file into a single one, "clean.dta", as shown:

local files: dir "~/datafiles" files "*_indresp.dta"
local dir1 "~/datafiles"

foreach f of local files {
     local w = substr(`"b`f'_"', 1, 3)
     use `"b`f'"', clear
     keep `w'a `w'b `w'c
     rename `w'* *
     append using `"`dir1'/clean.dta"'
     save `"`dir1'/clean.dta"', replace
}

However, the issue is that it only reaches the "bj_indresp.dta" file before ending the loop meaning the remainder of the files are not appended to the aggregate "clean.dta" file.

Any help would be appreciated, thanks.

[Mike Lacy]

I'm not offhand sure what the problem is, but I would offer something perhaps more generally useful to you, namely some simple things to help you to diagnose this (or other) problems yourself. (Perhaps you have already tried some diagnostic tricks, in which case the following ideas might be moot..)

1. Verifying the contents of the -files- local. One way to do this is to insert a -macro dir- right after the creation of that local.
2. Investigating what is happening with the expansion of your various macros by using -set trace on- at the beginning of your code.

[Justin Niakamal]

I tend to find Robert Picard's filelist (SSC) useful for these kinds of issues.

Something along the lines of

filelist, dir("~/datafiles"") pattern("*_indresp.dta")
gen fname = dirname + "/" + filename 

list

[Luke Robertson]

Thank you, Mike and Justin! You have both been great help to resolve this issue.

I have opted to use the "filelist" module to more easily manipulate files in my directory so thank you Justin for the suggestion.

For completeness of this question, I created a loop using the "filename" variable to traverse the folder alphabetically and completely as shown:

filelist, dir("~/datafiles") pattern("*_indresp.dta")

local files = _N
forvalues i = 1/`files' {
    display filename[`i']
}

Resulting in the output of:

ba_indresp.dta
bb_indresp.dta
bc_indresp.dta
bd_indresp.dta
be_indresp.dta
bf_indresp.dta
bg_indresp.dta
bh_indresp.dta
bi_indresp.dta
bj_indresp.dta
bk_indresp.dta
bl_indresp.dta
bm_indresp.dta
bn_indresp.dta
bo_indresp.dta
bp_indresp.dta
bq_indresp.dta
br_indresp.dta