Optimization problems in mata: missing values returned by evaluator and initial values not feasible

[Alice Zhang]

Post contained sensitive data.

[German Rodriguez]

Hi Alice. May I recommend that you use code tags for program listings? And more importantly, consider using dataex from SSC or built into Stata 15.1, to provide some sample data. That would make it easier for forum members to help troubleshoot your problem and test potential solutions.

In the meantime a preliminary suggestion. Consider copying the data from Stata to Mata just once rather that on every iteration. You can then pass the data as extra arguments to the evaluator or declare them as external, which is probably easier. The next task would be to make sure that GMM_DLW() works, calling it with your starting values. I find that I rarely need to specify a technique, as the defaults usually work.

[Trang Tran]

Hello,

I'm currently encountering the same problem as Alice did. My code is very similar to Alice's. Please kindly find below:

mata:
void GMM_DLW(todo,betas,crit,g,H)
{
PHI=st_data(.,("phi"))
PHI_LAG=st_data(.,("phi_lag"))
Z=st_data(.,("const","l_lag","k"))
X=st_data(.,("const","l","k"))
X_lag=st_data(.,("const","l_lag","k_lag"))
C=st_data(.,("const"))
Y=st_data(.,("y"))
OMEGA=PHI-X*betas'
OMEGA_lag=PHI_LAG-X_lag*betas'
OMEGA_lag_pol=(C,OMEGA_lag)
g_b = invsym(OMEGA_lag_pol'OMEGA_lag_pol)*OMEGA_lag_pol' OMEGA
XI=OMEGA-OMEGA_lag_pol*g_b
crit=(Z'XI)'(Z'XI)
crit
}


void DLW()
{
initialvalue=st_data(1,("const","beta_lols","beta_ kols"))
S=optimize_init()
optimize_init_evaluator(S, &GMM_DLW())
optimize_init_evaluatortype(S,"d0")
optimize_init_technique(S, "nm")
optimize_init_nmsimplexdeltas(S, 0.01)
optimize_init_which(S,"min")
optimize_init_params(S, initialvalue)
p=optimize(S)
p
st_matrix("beta_dlw",p)
}

end
*-----------------------------------------END MATA PROGRAM---------------------------------------*
cap program drop dlw
program dlw, rclass
preserve
sort firmid yearaccount
mata DLW()
end

*------------------------------------OLS estimates-----------------------------------------------*
reg y l k i.yearaccount
gen beta_lols=_b[l]
gen beta_kols=_b[k]
*------------------------------------ACF estimates-----------------------------------------------*
dlw
The code runs just fine until the previous line "dlw", and shows the return error stating "missing value returned by evaluator". i also tried different approach just like Alice did (different techniques, different initial values), but it either shows initial values not feasible" or "missing values returned by evaluator"
Below is a sample of my data. Would you mind checking what's wrong? I try to figure out what's wrong with the code or my data, but since I'm new to Mata, it's still quite hard for me to see through. I would be much appreciated to receive your comments. Thank you!

-------------Data sample------------------------

clear
input float(firmid e y l m k phi phi_lag const l_lag k_lag yearaccount)
4 0 7.991247 3.3322046 7.296923 7.687755 6.139946 6.139946 1 2.3025851 9.789052 2005
4 0 8.071547 3.3322046 7.46044 8.06427 6.139946 6.139946 1 3.3322046 7.687755 2006
4 0 8.626635 3.0910425 8.0748415 7.974518 6.139946 6.139946 1 3.3322046 8.06427 2007
4 0 8.73257 3.218876 8.213391 8.180602 6.139946 6.139946 1 3.0910425 7.974518 2008
4 0 9.017573 3.6888795 8.219066 9.815081 6.139946 6.139946 1 3.4011974 . 2011
4 0 9.096193 3.8066626 8.263784 9.75112 6.139946 6.139946 1 3.6888795 9.815081 2012
5 0 10.411207 . 10.377305 7.597796 6.139946 6.139946 1 3.6888795 . 2003
5 0 10.134768 2.397895 10.07436 8.226087 6.139946 6.139946 1 2.484907 8.413038 2005
5 0 10.08645 2.397895 10.020196 8.2744875 6.139946 6.139946 1 2.397895 8.226087 2006
5 0 6.370093 3.3322046 5.88396 9.031751 6.139946 6.139946 1 2.397895 8.2744875 2007
5 0 9.873952 3.4011974 9.819534 8.840881 6.139946 6.139946 1 3.3322046 9.031751 2008
5 0 7.072663 2.564949 6.336652 8.751653 6.139946 6.139946 1 2.564949 . 2011
5 0 7.088763 2.3025851 6.296508 8.484472 6.139946 6.139946 1 2.564949 8.751653 2012
6 0 7.652978 . 7.332772 9.218064 6.139946 6.139946 1 2.3025851 . 2003
14 1 -1.517234 . -1.517234 -1.517234 17.728449 6.139946 1 2.833213 9.178294 2003
14 0 9.399361 4.3438053 9.012427 9.272181 6.139946 17.728449 1 4.65396 . 2005
14 0 9.390632 4.4426513 9.00764 9.148253 6.139946 6.139946 1 4.3438053 9.272181 2006
14 0 10.50526 4.787492 10.176756 10.468596 6.139946 6.139946 1 4.4426513 9.148253 2007
14 0 10.300865 4.787492 9.911077 10.234726 6.139946 6.139946 1 4.787492 10.468596 2008
14 0 9.354046 3.4011974 9.098346 9.610126 6.139946 6.139946 1 3.4011974 . 2011
14 0 9.309422 3.4011974 9.015363 9.454276 6.139946 6.139946 1 3.4011974 9.610126 2012
19 0 6.142509 . 4.802734 7.367684 6.139946 6.139946 1 2.833213 . 2003
19 0 5.93965 1.609438 4.801359 7.683763 6.139946 6.139946 1 1.609438 7.28209 2005
19 0 5.869711 1.609438 4.852671 7.749963 6.139946 6.139946 1 1.609438 7.683763 2006
22 1 -1.517234 . -1.517234 4.7074246 17.728449 6.139946 1 1.609438 7.749963 2003
24 0 7.775477 . 7.425755 7.994472 6.139946 17.728449 1 3.4011974 8.560674 2003
24 0 7.852661 2.3025851 7.029836 8.302186 6.139946 6.139946 1 2.890372 7.708349 2005
24 0 7.985444 2.3025851 7.155255 8.341574 6.139946 6.139946 1 2.3025851 8.302186 2006
24 0 9.036418 3.8501475 8.282225 8.497074 6.139946 6.139946 1 2.3025851 8.341574 2007
24 0 8.876464 3.912023 8.111609 8.35964 6.139946 6.139946 1 3.8501475 8.497074 2008
24 0 8.973935 3.8066626 8.492818 8.902836 6.139946 6.139946 1 3.8066626 . 2011
24 0 8.915784 3.6888795 8.427103 8.843771 6.139946 6.139946 1 3.8066626 8.902836 2012
27 0 5.085203 . 4.848594 7.070619 6.139946 6.139946 1 3.6888795 . 2003
37 0 6.755027 . 6.509042 9.166101 6.139946 6.139946 1 2.890372 7.305394 2003
39 0 5.408792 0 5.246429 5.757968 6.139946 6.139946 1 2.397895 9.132043 2005
39 0 5.306926 0 5.144407 5.674651 6.139946 6.139946 1 0 5.757968 2006
39 0 7.01772 0 6.967852 5.612407 6.139946 6.139946 1 0 5.674651 2007
39 0 6.833528 0 6.783866 5.826724 6.139946 6.139946 1 0 5.612407 2008
39 0 7.893643 1.94591 7.677697 5.665166 6.139946 6.139946 1 1.3862944 . 2011
39 0 7.744301 1.609438 7.518174 6.474321 6.139946 6.139946 1 1.94591 5.665166 2012
47 1 13.16552 . 12.626302 13.731174 17.728449 6.139946 1 1.0986123 . 2003
52 0 5.112889 . 4.3392553 5.400571 6.139946 17.728449 1 7.564757 13.764865 2003
52 0 6.033503 1.3862944 4.0185995 5.253344 6.139946 6.139946 1 1.3862944 5.270702 2005
52 0 5.992105 1.3862944 4.025992 5.171124 6.139946 6.139946 1 1.3862944 5.253344 2006
52 0 5.859267 1.609438 4.3859615 8.761147 6.139946 6.139946 1 1.3862944 5.171124 2007
52 0 5.878017 1.609438 4.4227295 8.654763 6.139946 6.139946 1 1.609438 8.761147 2008
64 0 7.965521 . 6.737676 7.182449 6.139946 6.139946 1 1.609438 8.654763 2003
76 0 8.104374 . 7.42704 7.81574 6.139946 6.139946 1 2.70805 7.676908 2003
76 0 8.295699 3.8286414 7.722983 8.46675 6.139946 6.139946 1 3.912023 7.749483 2005
76 0 8.409802 4.060443 7.783864 8.678591 6.139946 6.139946 1 3.8286414 8.46675 2006
76 0 8.55051 3.218876 7.610639 7.203002 6.139946 6.139946 1 4.060443 8.678591 2007
76 0 8.366251 3.218876 7.467252 7.032799 6.139946 6.139946 1 3.218876 7.203002 2008
76 0 6.930906 2.1972246 6.014615 6.204829 6.139946 6.139946 1 2.1972246 . 2011
76 0 6.759707 2.1972246 5.843416 6.111237 6.139946 6.139946 1 2.1972246 6.204829 2012
127 0 6.773249 . 5.999051 6.839532 6.139946 6.139946 1 1.3862944 . 2003
127 0 6.796026 1.7917595 6.535209 10.228866 6.139946 6.139946 1 2.70805 6.71127 2005
127 0 6.81532 1.7917595 6.456455 10.293436 6.139946 6.139946 1 1.7917595 10.228866 2006
127 0 6.761408 2.772589 6.134595 6.941509 6.139946 6.139946 1 1.7917595 10.293436 2007
127 0 8.039039 2.772589 7.678749 6.732007 6.139946 6.139946 1 2.772589 6.941509 2008
127 0 7.266953 1.7917595 6.335191 7.181549 6.139946 6.139946 1 2.772589 . 2011
127 0 7.409464 1.0986123 6.45965 7.11539 6.139946 6.139946 1 1.7917595 7.181549 2012
131 1 4.5235014 . 3.604804 3.3370035 17.728449 6.139946 1 2.70805 . 2003
137 0 7.151162 . 6.708904 7.873322 6.139946 17.728449 1 1.0986123 3.3283646 2003
137 0 7.172937 1.3862944 6.556751 7.734456 6.139946 6.139946 1 1.94591 7.62458 2005
137 0 7.123507 1.3862944 6.522733 7.644602 6.139946 6.139946 1 1.3862944 7.734456 2006
139 1 9.282405 . 8.701028 9.136814 17.728449 6.139946 1 1.3862944 7.644602 2003
139 1 9.252378 4.1743875 8.964696 8.338585 17.728449 17.728449 1 4.4998097 9.081575 2005
139 1 9.575624 4.1743875 9.366274 8.493541 17.728449 17.728449 1 4.1743875 8.338585 2006
139 0 10.618588 4.6051702 10.3999 10.34509 6.139946 17.728449 1 4.1743875 8.493541 2007
139 0 10.483187 4.6051702 10.253186 10.122982 6.139946 6.139946 1 4.6051702 10.34509 2008
147 0 10.57672 . 10.216813 8.923101 6.139946 6.139946 1 3.912023 . 2003
150 0 7.751947 . 7.242707 8.732776 6.139946 6.139946 1 5.123964 8.887859 2003
150 0 8.102473 2.397895 7.763387 8.518409 6.139946 6.139946 1 3.6888795 8.580092 2005
150 0 8.120337 2.397895 7.782797 8.625431 6.139946 6.139946 1 2.397895 8.518409 2006
150 0 8.972783 5.075174 8.279635 8.384996 6.139946 6.139946 1 2.397895 8.625431 2007
150 0 8.953792 4.941642 7.775137 8.24514 6.139946 6.139946 1 5.075174 8.384996 2008
150 0 8.607829 4.6051702 7.388729 9.302 6.139946 6.139946 1 4.6051702 . 2011
150 0 8.487694 4.4998097 7.314911 9.183899 6.139946 6.139946 1 4.6051702 9.302 2012
154 0 4.291909 . 2.619951 5.505832 6.139946 6.139946 1 2.995732 . 2003
154 0 4.079224 0 3.03777 6.251801 6.139946 6.139946 1 .6931472 5.4574 2005
154 0 4.054163 .6931472 3.004341 6.22512 6.139946 6.139946 1 0 6.251801 2006
154 0 4.907889 .6931472 4.2011642 4.7571487 6.139946 6.139946 1 .6931472 6.22512 2007
154 0 4.961726 .6931472 4.2762165 4.602966 6.139946 6.139946 1 .6931472 4.7571487 2008
155 0 6.012449 1.3862944 5.480117 7.48287 6.139946 6.139946 1 .6931472 . 2005
155 0 5.864272 1.3862944 5.353446 7.531978 6.139946 6.139946 1 1.3862944 7.48287 2006
157 0 3.3128736 0 .8152279 6.791188 6.139946 6.139946 1 1.0986123 8.44392 2005
157 0 3.123886 0 .4706438 6.705055 6.139946 6.139946 1 0 6.791188 2006
157 0 4.426929 0 3.497463 6.91532 6.139946 6.139946 1 0 6.705055 2007
157 0 4.401068 0 3.522031 6.791637 6.139946 6.139946 1 0 6.91532 2008
158 0 7.375624 . 6.975004 8.797326 6.139946 6.139946 1 0 6.791637 2003
158 0 6.235763 1.7917595 4.870352 5.563499 6.139946 6.139946 1 2.6390574 8.708045 2005
158 0 6.070176 1.7917595 4.74731 5.691 6.139946 6.139946 1 1.7917595 5.563499 2006
158 0 7.383511 2.0794415 6.832253 8.974078 6.139946 6.139946 1 1.7917595 5.691 2007
158 0 7.266808 2.0794415 6.710926 8.963437 6.139946 6.139946 1 2.0794415 8.974078 2008
158 0 7.504173 1.94591 6.708467 7.745322 6.139946 6.139946 1 1.94591 . 2011
158 0 7.241334 2.1972246 6.143169 7.910933 6.139946 6.139946 1 1.94591 7.745322 2012
161 0 8.282575 . 8.03246 7.375763 6.139946 6.139946 1 1.7917595 . 2003
161 0 6.92925 2.484907 6.487758 8.418052 6.139946 6.139946 1 2.484907 7.627576 2005
161 0 6.579892 2.0794415 6.168408 8.337404 6.139946 6.139946 1 2.484907 8.418052 2006
182 0 7.403433 3.135494 7.079796 8.026395 6.139946 6.139946 1 2.0794415 8.337404 2005
end
label values e r4
label def r4 0 "No", modify
label def r4 1 "Yes", modify
[/CODE]

[Shelly Xu]

I've been having the same problems as you recenly. Because there was a data leak in my memory before entering the mata environment, so the mata environment can't identify it. I checked the variables that will be used in the mata’s function, and found that there were missing values in some variables. So, I deleted the missing values before entering the mata environment. Then my problem was solved. In addition, there are references also show that the inappropriate initial value setting may also cause such a mistake. Hopefully this will help you or anyone else with the same problem.