Is there a way to get the data type of a variable without loading the dataset into memory?

[Michael Anbar]

I have an ado program that merges (in a complicated way, not just the "merge" command) two large data sets, and I'd like the program to perform type checking on each dataset before beginning the merge. Specifically, given a variable name and a "using" dataset, I'd like to get the data type of that variable in that dataset. For the dataset in memory, this is easily done with macro extended functions, but for a dataset stored on disk, I can't find a way to do this without manually parsing the dta format.

Obviously, this information is available to Stata without loading the dataset because it's built into the dta format and commands like describe can access it. I don't want to load a 300 GB dataset into memory just to check the data types, and I'd prefer not to have to parse the text of "describe using ..." manually.

[Clyde Schechter]

I am not aware of any way to do this in Stata. If somebody else knows of a way, I hope they'll chime in here, because I have confronted this situation in my own work (although I have never handled a 300 GB dataset, so for me it's more a matter of convenience). If nobody has a solution, I'd suggest adding it to the Wish List for Stata 15 thread.

[Nick Cox]

describe using

reports on types. If you want more, I guess you need to open and close log and parse its contents.

Or you could read the first part of the .dta file and extract the information, but I would stick with the above.

EDIT: That didn't really add to what was said previously. My mind wandered.

[Clyde Schechter]

If you want more, I guess you need to open and close log and parse its contents.

That's what I've done when I've needed the information. What I think Michael and I are both hoping for is something like a change to -describe using- so that this kind of information would be accessible in some more convenient way. What would work very well for me is if -describe using- had a -replace- option, just like that of -describe-. Then I could save the new data set and pull the information easily from there without having to write parsing code.

Now, I don't know if that solution would work for Michael, because he might object to having to -preserve- his original data set before bringing the -describe using- results into memory if it, too, is of the order of 300 GB, .

[Steve Samuels]

Withdrawn. Bad answer.

[Michael Anbar]

That's what I've done when I've needed the information. What I think Michael and I are both hoping for is something like a change to -describe using- so that this kind of information would be accessible in some more convenient way. What would work very well for me is if -describe using- had a -replace- option, just like that of -describe-. Then I could save the new data set and pull the information easily from there without having to write parsing code.

Now, I don't know if that solution would work for Michael, because he might object to having to -preserve- his original data set before bringing the -describe using- results into memory if it, too, is of the order of 300 GB, .

I don't think this will work for me, especially since on Windows, -preserve- seems to write to disk, and that's a major performance bottleneck even on an SSD. I was hoping there was a way to use a macro extended function, or even a Mata function, that would work like "local vartype: type <varname>" except on a dataset on disk.

If nobody has a solution, I'd suggest adding it to the Wish List for Stata 15 thread.

Is Statacorp usually responsive to feature requests of this sort? I've only been using Stata since version 11, but from what I've seen of the wishlist threads, the features that are implemented (if any) are those that attract market share, not those that help programmers who are diving deeper into the internals.

[wbuchanan]

Michael Anbar while helping user programmers is great, if the feature is only valuable to a small portion of the user community it wouldn't make sense for Stata to invest time, effort, and financial resources to an additional feature that doesn't benefit the greatest number of current and future users. One possible alternative would be to parse the .dta header and grab the data from there. As an extended macro function it would be extremely poor for performance given the high penalty on I/O operations. If the results were cached, then it runs the risk of tying up needed system resources that might negatively affect performance in other ways. I'm assuming the files were created by someone (at some point), so perhaps they could send you a file specification/layout so you could read that smaller file into memory/parse it instead?

[Clyde Schechter]

One problem with doing this by parsing the .dta file header is that this changes from version to version in Stata. So if you write the code in version 14, it may well fail once we hit version 15 or 16.

[wbuchanan]

It also depends on the underlying processor architecture of the machine that wrote the file and the machine that is reading the file. For example, if the file was written by a big endian machine and was being read by a little endian machine, the order of the bytes would need to be swapped.

[daniel klein]

For example, if the file was written by a big endian machine and was being read by a little endian machine, the order of the bytes would need to be swapped.

Why exactly would this be a problem? The byte order of the .dta file is recorded somewhere in the header. Otherwise Stata would also choke on such things.

Best
Daniel

Edit

And for "somewhere in the header" read: in the second byte. For more

help dta

Edit 2

That is true for Stata versions < 13 ..., but see Clyde's comment above.

[wbuchanan]

If the machine is big endian and the file was written on a little endian system, the program would need to swap the byte order. If the file was written on a big endian machine and was being read on a big endian machine the byte order would not be swapped. I never said it was a problem, but it is another factor that needs to be considered in the programming and I'm not sure if there is a Stata command that returns the endianness of the system.

[daniel klein]

Indeed, it is something that needs consideration. I was just thinking that, compared to Clyde's point, it is a minor inconvenience. Anyone interested in writing such a thing, see Mata function byteorder().

Best
Daniel

[wbuchanan]

Daniel Klein, thanks for pointing out that Mata function. I definitely was not aware of it previously and assume others may not have known about it as well.