Loading the most recent data set

[Konrad Zdeb]

Colleagues,

I have a number of data sets that are saved with use of the following code:

local c_date = c(current_date)
local c_time = c(current_time)
local c_time_date = "`c_date'"+"_" +"`c_time'"
local time_string = subinstr("`c_time_date'", ":", "_", .)
local time_string = subinstr("`time_string'", " ", "_", .)
display "`time_string'"
save .\Data\nl_full_`time_string'.dta, replace

As you can presume the ls will produce the following output:

. ls
  <dir>   9/12/14 15:33  .                 
  <dir>   9/12/14 15:33  ..                
  14.5M   9/12/14 14:43  nl_full_12_Sep_2014_14_43_53.dta
  14.5M   9/12/14 15:33  nl_full_12_Sep_2014_15_33_02.dta

I would like to draft some code that will always load the most recent data set, either using the system information on file modification or, preferably, date information stored in the file name.

[Phil Schumm]

I don't believe you can get the date last modified of a file easily within Stata, however if you're satisfied with the information in the filename, then simply use a proper ISO date (or date and time) stamp, so that the filenames will sort in order of date (or time).

[Konrad Zdeb]

Phil,

thanks for your comments. It's a useful suggestion. It also crossed my mind that on the lines of this discussion, I could attempt to write the generated file name to the text file and the read the content of the file and pass the name to the use command. It looks like quite a lot of hassle to do relatively simple thing but it may be worth exploring. Alternatively, which appears to be simpler solution, I could attempt to executre shell command dir with /O:D switch and get the list of file names sorted by date, export the results to file and read appropriate line in the file and pass it to local macro.

[Phil Schumm]

To get the most recent dataset based on a timestamp encoded in the filename, you could use

loc flist : dir . files "*.dta"
loc lastfile : word `:list sizeof flist' of `flist'

assuming that you are using an ISO-format datetime string so that the filenames sort in temporal order. Note that I'm pretty sure that the macro extended function dir returns files in alphabetic order of filename (you'll need to check this); if not, you could sort the list with list sort.

Depending on what you're trying to do, one option would simply be to carry a copy of the most recently saved dataset in a file named latest.dta. IOW, you would do the following

save .\Data\nl_full_`time_string'.dta
copy .\Data\nl_full_`time_string'.dta .\Data\latest.dta, replace

This makes it easy to load the most recent dataset.

P.S. I just noticed that you were originally using the replace option with your save command. However, if you are encoding a timestamp in the name, you shouldn't need this option (i.e., you should never repeat the same filename), and using it can instead hide problems—either in your code or in the way your code is being used.

[Joe Canner]

Konrad,

Building on Phil's suggestion, you should be able to do what you want just with macros. The following produces a macro sorted in alphabetical order:

 local myfilelist    : dir . files "*.dta"

After which you can use the last filename in the list. If necessary, you can ensure that the list is sorted by:

 local sortedlist: list sort myfilelist

Regards, Joe

[Konrad Zdeb]

Phil/Joe,

Thank you very much for showing the interest in my little problem. With respect to the replace option, it's an artefact from earlier version of the code

[Donovan Pollack]

Hello,

I have a similar issue but my file names do not have the time information. How can you retrieve the the "last modified" information via dofile without information in the filename?

thanks

Donovan

[William Lisowski]

Here is some technique. For details on how this works, see the output of help creturn.

. sysuse auto, clear
(1978 Automobile Data)

. display c(filename)
/Applications/Stata/ado/base/a/auto.dta

. display c(filedate)
13 Apr 2016 17:45

. tempfile foo

. save `foo'
file /var/folders/xr/lm5ccr996k7dspxs35yqzyt80000gp/T//S_83239.000001 saved

. clear

. use `foo'
(1978 Automobile Data)

. display c(filename)
/var/folders/xr/lm5ccr996k7dspxs35yqzyt80000gp/T//S_83239.000001

. display c(filedate)
23 Dec 2018 17:21

[Donovan Pollack]

this is very helpful thank you!