Matrix Size Adjustment

[Long Hong]

Dear all,

I have two questions on the Stata Matrix Size.

1. In general, if you set the matrix size = 11000, the Stata will perform using the size regardless of your actual needed matrix size (e.g. you actually only need matrix size 100 to run the regression). Therefore, it will take a much longer time for you to run a simple OLS regression if you set matsize 11000. Am I right on this?

2. If this is the case, is there anyway that Stata will automatically adjust its matrix size according to the need? Going back to the OLS example, if my OLS regression only needs Matrix size 100, but I do not know it and I set the matsize = 11000, is there any way that I will make Stata to adjust automatically according to the real need of the regression (i.e. matsize = 100)?

I am currently working on some regression that require hours for one iteration of the MLE and I figured out that the size of the matrix is one of the reasons. Therefore, I would like to reduce the matrix size to the min level to save MLE iteration, but I do not know the real matrix size needed.

Hope my question is not confusing. I look forward to hearing from you! Thank you very much in advance.

Best regards
Long

[Clyde Schechter]

My understanding is that when creating a matrix Stata will only use the actual size matrix needed for the information and calculations at hand. When you set the matrix size you are actually setting the maximum size allowed for any matrix. But a given m x n matrix will still only require storage for m x n data elements (plus some fixed overhead). I don't believe you need to do anything to restrict the size of the matrix to only what is actually needed: this is automatic.

I do not know why Stata even requires you to set a matrix size upper limit. In earlier versions of Stata you also had to set the amount of memory you wanted allocated to Stata by the operating system (though there was a default amount if you didn't specify it explicitly). Modern versions of Stata have eliminated that and memory is allocated dynamically, increasing or decreasing as needed. Why Stata requires you to explicitly put an upper limit on the size of matrices, I do not know. But given that they didn't change this when they went to dynamic memory allocation, there must be some reason why it is difficult or impossible to do that.

[Long Hong]

Dear Prof. Schechter,

Thank you very much for your reply! I saw this statement in the 'remark' part of the -matsize- help file.

You may change matsize with data in memory, but increasing matsize increases the amount of memory consumed by Stata, increasing the probability of page faults and thus of making Stata run more slowly

My understanding is that It basically says that if you set a memory that is way higher than your need, your Stata is going to very slow in processing. This is pretty true for my case. Changing matsize from 1000 to 10000 increases the processing time more than 10 times... Therefore, I am wondering the allocation of matsize is not automatic in Stata. But I am not sure if this is really the case.

For now, I can only try using 'trial & error' strategy to find the real matsize I need, but it is very inefficient in my current project, which takes more than 50 hours to try once. So, I am trying to see if there is any command that can detect the real matrix size (at least approx.) needed.

Thank you very much and have a lovely weekend!

Best,
Long

[Carole J. Wilson]

If you are running regressions, you should only need a matsize of 2+ the number of variables in the model.