RAM usage on Xeon scalable CPU issue?

[Ciaran OFlynn]

Hi all,

I just updated my workstation to 2 Xeon 6244, from a singe Xeon W-3223. The aim was to up my RAM from 384GB to 512GB, in order to open some large csv files (300GB+) without having to use the CHUNKY command first.

On relatively smaller 200GB+ files, my old Xeon 3223 would chug along and eventually load all the data into memory, maxing at about 350GB. When I tried to load a large file into memory with the new CPUs, it just hangs at 20GB utilisation (4%). Its been like that for 30 minutes now with zero movement.

If I tried to open this same file on my old system, it would just creep the RAM utilisation up to the max of 384gb, and then crash. Here, it just doesn't seem to want to access it for CSVs. I have tested it, and it will open a 188GB dta happily.

Any thoughts?

[Daniel Feenberg]

First, do:

update all,force

You might be lucky. Here at NBER we have a server with 2TB of physical memory, I am aware that a few users have datasets of 850GB or more and they do work. It does take several minutes to load the full file from disk. We didn't have to change any configuration paprameters to get this ability. I expect you have an OS or Java problem. We are using Linux for the servers and have never tried running Stata for Windows on a large machine.

Before you appeal to Stata tech support, try seeing if the problem is with I/O or storage. Here is an I/O-free way of consuming a lot of memory:

set obs 10000000
gen double recno=_n
forvalues i = 1(1)1000 {
  quietly gen  x`i' = mod(_n,1000)+`i'
}

When asking Stata for tech support, it helps to provide information about your installation: Include the output of these commands:
.

about
query compilenumber

It is possible the difficulty has nothing to do with Stata. Do you have another package or language that could be tested with a large memory demand?

[Alecia Cassidy]

How did you fix it? I think I may have a related problem (https://www.statalist.org/forums/for...mory-available). Are you using Windows 11? In my case, available RAM is not used. Instead, the RAM used by Stata seems to not surpass 16 GB.