moptimize + svy + moptimize_util_matsum and running out of memory

[Richard Foltyn]

Dear Stata list,

I have implemented an MLE using -moptimize- and use survey weights via -moptimize_init_svy-.
I compute the Hessian using -moptimize_util_matsum- to take care of the survey weights.

The problem is this:
1. My model does not have the single-index property, so I cannot just specify some equation p = x'b and compute the score / Hessian wrt. p
2. To work around this, I specify k equations for each element of b which contain only a constant.
3. The effect is that the per-observation contributions to the Hessian become quite large (as I have many parameters)
4. Since I need to pass the contribution of each observation to -moptimize_util_matsum-, I need to build a matrix of (vectorized) lower triangular Hessians for each observation, before calling -moptimize_util_matsum-
5. This matrix would require ~ 10 GB of RAM for the whole sample

My question is this: can I do something more intelligent? E.g. progressively build the Hessian as I compute each observation's contribution, without having to store the entire matrix beforehand?
This would be simple without the svy stuff, but I need the weights.

Thanks,
Richard

[Christophe Kolodziejczyk]

I think optimize() would more suitable in this case, since it does not assume the single-index property.