Downsides to increasing set matsize

[Richard Williams]

I am frequently increasing matsize, e.g. I give commands like

set matsize 7500

Would anything horrible happen if I made the change permanent? The manual says "increasing matsize increases the amount of memory consumed by Stata, increasing the probability of page faults and thus of making Stata run more slowly." But I've got 64 GB of memory so do I really have to worry?

Incidentally, when I run equivalent models in Mplus or lavaan, they don't whine at me about such things. I wish Stata could just set the matsize dynamically. Or at least give better messages about what the matsize needed to be set to. I don't want to experiment to have to figure out what the optimal matsize is for every problem.

[Clyde Schechter]

This is a good question. I did once encounter a situation where I had set matsize to a large number and then subsequently could not run a certain do-file because Stata couldn't obtain enough memory to hold everything needed, even though the same analysis had run previously. Now, that machine had much less memory than yours, and perhaps this would only bite under more extreme circumstances on your instillation.

But I think the request that this all be handled dynamically is a very good one. In fact, I would have posted this question on the Wishlist for Stata 16 thread.

[Richard Williams]

Clyde, you talked me into it!

https://www.statalist.org/forums/for...32#post1412332

Meanwhile, if anybody has any advice on what I should do in the meantime, please me know.

[Mike Lacy]

Yes, Rich, there was and presumably still is a real downside to setting -matsize- to a large value. See this discussion I initiated back in 2014.
The short version is that, at least under v.13, I found that setting matsize to a larger size grossly degraded performance to the point that my MP2 version of Stata ran slower than an IC version. Tech Support chimed in and indicated that moving to dynamic allocation of matsize was on their to-do list, but was a difficult task.

[Richard Williams]

Thanks Mike. I will play around with it and see if my 64 GB of memory makes me immune to problems or not. If they can't give dynamic memory I wish they could at least give you ballpark figures of what to set matsize to.

[Clyde Schechter]

Reading the details of Tech Support's response in Mike's thread, it sounds like your 64 GB of memory may not help you much, as cache seems to play a role here. But it'll be interesting to see what you find.

[Richard Williams]

I'm testing it now. The annoying this, if I set matsize too low, it takes several minutes to tell me I need to increase it. So these tests may take a while.

[Richard Williams]

I just ran the same big SEM model with matsizes ranging between 5,000 and 11,000. The times ranged between 10:57 and 11:15, and even that little variation may reflect that I was doing other things with my computer at the same time. Maybe my 64 GB does give me immunity. But I should probably try a few other examples too.