Getting file list in directory

[Konrad Zdeb]

Colleagues,

I would like to get the list of CSV files in the directory but, as always, I messed up something with quotes.

* Path and files
global importfolder ///
    `"`C:\Users\me\files\data to play with'"'
global filelist : dir ${importfolder}  *.csv

[Nick Cox]

  
global importfolder "C:\Users\me\files\data to play with"
global filelist : dir "$importfolder"  *.csv

You just need plain " " here to bind over the spaces. The single quotes suggest some sort of local macro but you don't have one, so they are mistaken here.

EDIT: Copied a point from Maarten Buis.

[Maarten Buis]

I suspect you want:

global importfolder ///
`"C:\Users\me\files\data to play with"'
global filelist : dir `"${importfolder}"' files *.csv

i.e. no inner single quotes, as that would assume your path is actually a local macro and added quotes around the global macro again, as quotes get stripped off.

[Konrad Zdeb]

Thanks for all the comments. Following the "help quotes" section, I was trying to be fancy and encapsulate "" in the macro so I could call the path without typing "" around macro.

[Konrad Zdeb]

Strangely,

I got an error:

. global importfolder ///
`"C:\Users\me\files\data to play with"'

. global filelist : dir `"${importfolder}"' files *.csv
varlist not allowed

[Nick Cox]

What's the problem? Spaces within filenames. What's the solution? Plain double quotation marks " "

[Konrad Zdeb]

Thanks, I'm fighting with this. Presently, I have:

* Define path and get CSVs.
local importfolder ///
    `"C:\Users\me\stuff for stata"'
di `""`importfolder'*.csv""'
global filelist : dir `""`importfolder'*.csv""'
invalid syntax

The naming of the files does not adhere to any naming convention, so have spaces capital and small letters and different lengths.

To add to my previous post, it appears that the problem relates to the macro definition:

dir "C:\Users\me\stuff for stata\*.csv"
1853.6k   8/29/14  9:32  Age_24_and_under.csv

global files : dir "C:\Users\me\stuff for stata\*.csv"
invalid syntax
r(198);

[Nick Cox]

Indeed: your syntax is invalid; it's just fantasy syntax that you would like to work. Stata requires you (1) to separate the directory name and the wildcard pattern and (2) to use the keyword files.



This is documented. Start with help macro.

[Konrad Zdeb]

I'm trying to follow the advice found here, but it doesn't seem to work. To my mind what is in the filelist doesn't make sense.

* Alternative
cd "C:\Users\me\stuff"
local filelist: dir . files "*.csv"
di `filelist'
aged 18-24, claiming for over 6 months.csvage_24_and_under.csv

* Import files.        
foreach datafile of local filelist {
   preserve
   insheet using `datafile', clear
   save temp, replace
   restore
   append using temp
   }

invalid '18'
r(198);

end of do-file

I find it hard to believe that this can't be more simplified, this would take two lines of code in R:

temp <- list.files(path="whataver", pattern="*.csv", full.names = T)
myfiles <- lapply(temp, read.delim) 

[Nick Cox]

I think this is exactly the same problem as explained earlier, indeed more than once in this thread.

  
 insheet using "`datafile'", clear

There are spaces in the filename, and you must bind with " ".

I wrote fs (SSC) to make handling lists of files easier.

[Konrad Zdeb]

Thanks very much for showing interest. I did most of the stuff I wanted, the only outstanding task is to reshape the data with multiple stubs and certain identifiers for time series missing but I'll get there at some point, not that I wouldn't appreciate help. The attached file contains data extract (tab delimited due to forum limitations). For those who may be interested in this, full data sets are available via NOMIS.

/// Import
clear
cd "C:\path_with_csvs"
local filelist: dir . files "*.csv"
di `filelist'

* Get the base data
use "C:\proper_path\sns-igzs.dta", clear
keep igz INTGEONAME laname

* Import files.        
foreach datafile of local filelist {  
    preserve
    import delimited using "`datafile'", clear varnames(1)
    save temp, replace
    restore
    merge 1:1 igz using ///
        "C:\proper_path\temp.dta", ///
         nogenerate
    drop v*
    destring, replace ignore(-)
    dropmiss *, force
    }

* remove the temporary .dta file and save
compress
save jsadta, replace

// Renvars
renvars, subst(_total)
renvars, subst(_tot)
renvars, subst(_to)
describe

// Make panel data

*cut out the "_may09 and other" at the end to obtain the actual stubs
foreach v of varlist aged1824_oct04 - claimingforover6months_jul14 {
    local stubs `"`stubs' `=substr("`v'",1,length("`v'")-6)'"'
}

* Reshape the data to make it long for charts and analysis (doesn't work)
reshape long `stubs', i(igz) j(date)
    
* Make proper time series variable
(...)

I wrote fs (SSC) to make handling lists of files easier.

This is excellent. It would save me half an hour faffing around with the dir command.