cfbinout

[Kaibalyapati Mishra]

Hello,

I am using Stata BE and encountered the following error while trying to run a control function logit model using cfbinout:
.
cfbinout logit hee i.hh_size_grp i.soc_grp i.relgn i.sctr i.marital i.Quartile i.rel_code i.state (ses_index = jan_suraksha)
J(): 3900 unable to allocate real <tmp>[896518,896518]
r(3900);

I suspect this may be due to memory or matrix size limitations in Stata BE, possibly due to high-cardinality variables like i.state.

• Does cfbinout create large matrices internally that exceed BE's limits?
• Would simplifying the model (e.g., dropping some fixed effects or collapsing categories) help?
• Or is upgrading to Stata SE or MP necessary for this type of model?

Any suggestions or workarounds would be greatly appreciated.

Thank you in advance!

[Chen Samulsion]

The return code 3900 demonstrates out of memory. Try to set trace on to check what's going wrong?

[M-2]   error . . . . . . . . . . . . . . . . . . . . . . . . Return code 3900
        out of memory
        Mata is out of memory; the operating system refused to supply
        what Mata requested.  There is no Mata or Stata setting that
        affects this, and so nothing in Mata or Stata to reset in order
        to get more memory.  You must take up the problem with your
        operating system.

[Clyde Schechter]

You don't need trace to see what's going wrong here. -help limits- will tell you that Stata BE allows at most 800 x 800 for a matrix. You are orders of magnitude beyond that.

But also, even if you had STATA MP or SE, which have larger limits, I suspect this would still fail. An 896,518 x 896,518 matrix of floats would require a little over 3.2 terabytes of memory--which I suspect you are nowhere near close to having. In fact, there is no software at all that would do this unless you have that much RAM available (after subtracting out the RAM taken up by your operating system and other open apps that may be running in the background.)

[Kaibalyapati Mishra]

Thanks for the response.
But how does someone go about estimating these with a larger sample size? In Stata 19, they have cfprobit, I suspect similar problems might occur as well. So, manual estimation is the way to go?

[Harald Tauchmann]

Dear Kaibalyapati:
Thank you for bringing this issue to our attention. As already mentioned above, The error is due to mata running out of memory. Without knowing your data and your model (e.g. how many variables are generated by expanding i.hh_size_grp i.soc_grp i. ...?), it is not straightforward to pinpoint precisely what is causing the problem and what a possible remedy might be. Because it is a square matrix that gets too large, my guess is that the size of the model (not the size of the data) is causing the problem. I guess you cannot provide me with any (artificial) data that would allow me to replicate the error ... which would help a lot.
Best wishes,
Harald

PS: A simple logit model (which treats ses_index as exogenous) runs? Does the error also occur with the terza(2017) option?

[Harald Tauchmann]

Dear Kaibalyapati:
This seems to be a problem with how the analytic derivation of the variance-covariance matrix is programmed. Please try again with the noanalytic option (or alternatively, but less recommended, terza(2017)) specified. I hope this solves the problem.
Best wishes, Harald

[Harald Tauchmann]

Dear Kaibalyapati:
Thanks again for pointing us to this issue. The error you encountered was due to sloppy coding (which only came into play with analytic standard errors and large samples). We have fixed this bug. Thanks to Kit Baum, the corrected version of cfbinout.ado is now available from SSC. Please install the new version and let me know if the corrected ado solves your (specific) problem.
Best wishes,
Harald