How to resolve "op. sys. refuses to provide memory"?

[Titir Bhattacharya]

I'm trying to append a couple of fairly large .dta files and am getting this error:

op. sys. refuses to provide memory:
Stata's data-storage memory manager has already allocated 1528823820 bytes and it just attempted to
allocate another 16m bytes. The operating system said no. Perhaps you are running another
memory-consuming task and the command will work later when the task completes. Perhaps you are on a
multiuser system that is especially busy and the command will work later when activity quiets down.
Perhaps a system administrator has put a limit on what you can allocate; see help memory. Or perhaps
that's all the memory your computer can allocate to Stata.

Apart from Stata, only a bunch of files on adobe were open while the append was being run and am not using a multi-user system. Can anyone suggest how to resolve this issue? It would have made my work infinitely simpler were I able to append all the files. Is there any way to allocate more memory to stata?

Looking forward to your suggestions.

Thanks!

[Richard Williams]

Do you need all the variables? If not you could use only the subset of variables you need.

[Marc Kaulisch]

In addition, are the data files compressed before saving?

PS: A quick addition: Sometimes string variables are better stored as numeric ones. That can save space as well.

[Titir Bhattacharya]

Do you need all the variables? If not you could use only the subset of variables you need.

Thanks Richard. In fact I do not need all the variables and I'm going try with only a subset of variables.

[Titir Bhattacharya]

In addition, are the data files compressed before saving?

PS: A quick addition: Sometimes string variables are better stored as numeric ones. That can save space as well.

Thanks Marc, I'm going to consider your suggestion. And no, data files aren't compressed before saving. Is there a way to do that in the code?

[Leonardo Guizzetti]

Thanks Marc, I'm going to consider your suggestion. And no, data files aren't compressed before saving. Is there a way to do that in the code?

Especially for large datasets, using -compress- prior to saving your dataset to disk will ask Stata to make sure that all variables use the smallest possible storage type for each variable. For example, an integer variable can be stored as a double, but wastes space unnecessarily because the additional bytes of storage are not used. NB: this is different from compression in the sense of some abstract mathematical representation like a zip file.

[Marc Kaulisch]

Thanks Marc, I'm going to consider your suggestion. And no, data files aren't compressed before saving. Is there a way to do that in the code?

As Leonardo already pointed to -compress- my routine is usually to do

compress
datasignature set, reset
save .., replace

If you have some string variables trimming, destringing or encoding them before compressing can save space as well.

-datasignature- allows me to control for unwanted changes when using the file later.