Running Same Commands in multiple datasets

[Giorgio Depolli]

Hi everyone,

I am quite new to state and I have 150 different quarterly datasets which contains the same variables. I was to run the same commands for each dataset.

I have put these datasets into one folder, but I am not sure how to use loop and macro commands to run the same commands for these datasets simultaneously.

If anyone could help me and provide me with a general framework regarding this question would be much appreciated.

Thank you.

[Ali Atia]

cd "yourfolder"
local files: dir . files "*dta"

foreach file of local files {
    use "`file'", clear
    //do something
}

[Giorgio Depolli]

Thank you very much the code worked. However, I have troubles with saving the data. Currently, I am only trying to apply the code on 4 quarters and when I applied your command, I can see that stata applies those command to all four quarters (by looking at deleted observations, etc) but when I saved the results, the saved results only correspond to one quarter. So, it only saved the last quarter.

Could you please tell me what command do I need to use to save changes in all four quarters? In other words, to have 4 different datasets, not just 1 after saving it?

[Ali Atia]

I'm not sure I entirely understand your data structure or desired output. Maybe this code will work - it produces 600 datasets, one for each quarter in each file. Please share a data example and an excerpt of your code if this isn't what you're looking for.

cd "yourfolder"
local files: dir . files "*dta"
forval quarter = 1/4{
    foreach file of local files {
        use "`file'", clear
        //do something
        keep if quarter==`quarter'
        local filenodta = subinstr("`file'",".dta","",.)
        save "`filenodta'_`quarter'.dta",replace
    }
}

[Giorgio Depolli]

I am sorry, I should have explained it better. So, I have q1.dta q2.dta q3.dta q4.dta i.e. dataset for each quarter. When I applied your previously suggested framework, I can see that my commands are applied to each of these .dta. However, when I press the save button in the end, It only generates "x.dta" which is the same as "q4.dta" but with the changes that I applied with my commands.

What should I add to my command so that it saves the results for each .dta (loop) and not just the last one?

I hope now my question is clear. Thank you for your help.

[Ali Atia]

Just add

save "`file'",replace

at the end of the loop, before the closing curly bracket.

[Giorgio Depolli]

Thank you for your suggestion.

I added save, replace and it gave me an error - "invalid file specification" in the end. All the commands before save, replace are running perfectly fine, but when it came to saving, I got that error. Do you know what might be causing that?

[Ali Atia]

See edit - though I see the problem was solved in the other thread: https://www.statalist.org/forums/for...d-saving-files

[Giorgio Depolli]

Yes, the problem was solved! Thank you for your help!

[Oscar Ozfidan]

In the description you mention "to run the same commands for these datasets simultaneously." So if you want to run your commands concurrently in parallel on multiple datasets I recommend batcher from SSC. how many instances your computer can handle will depend on some factors but you can adjust that using max option.

[Sonnen Blume]

cd "yourfolder"
local files: dir . files "*dta"

foreach file of local files {
use "`file'", clear
//do something
}

Hi Atia, thanks for this nice solution. This code works for all .dta files in the same folder. Is there any way to apply this for sub-folders in a parent folder as well?