Problem with using a mata likelihood evaluator for ml in ado file "evaluator example_lf() not found"

[Sarwar Islam]

Hi,

I'm building a new likelihood estimation ado command in Stata and I've written the likelihood in Mata. However, I wanted to avoid using a library and include the Mata code at the bottom of the ado file (this is the second approach mentioned here: http://www.stata.com/statalist/archi.../msg00712.html).

Using this approach with ml seems to return an error where the mata evaluator is not found, but mata functions defined normally work as they should with the code included below the ado program.

I tried exploring this issue by testing whether stata is compiling the evaluator by running "mata: example_lf()" which spits out an "expected argument" error (as we would expect) which shows that it does indeed compile when the program is called.

I whipped up a quick simple example to highlight the issue (adapted from Maximum Likelihood Estimation with Stata, 4th Edition):

------------ foomata.ado ------------------------

program foomata
version 13.1
if replay() {
if (`"`e(cmd)'"' != "foomata") error 301
Replay `0'
}
else Estimate `0'
end

program Estimate, eclass sortpreserve
syntax varlist(fv) [if] [in] [, ///
vce(passthru) ///
noLOG ///
noCONStant ///
Level(cilevel) /// -Replay- option
* /// -mlopts-
]
// check syntax
mlopts mlopts, `options'
gettoken lhs rhs : varlist

// mark the estimation sample
marksample touse

// fit the full normal model
ml model lf probit_lf() ///
(`lhs' = `rhs'), `constant' ///
if `touse' ///
`vce' ///
`log' ///
`mlopts' ///
maximize

ereturn local cmd foomata

Replay, level(`level')
end

program Replay
syntax [, Level(cilevel)]
ml display, level(`level')
end

mata:
void probit_lf(transmorphic scalar ML, real rowvector b, real colvector lnfj)
{
real vector depvar, xb
depvar = moptimize_util_depvar(ML, 1)
xb = moptimize_util_xb(ML, b, 1)

lnfj = depvar:*ln(normal(xb)) + (1:-depvar):*ln(normal(-xb))
}
end

------------ foomata.ado ------------------------

------------ runcommand.do -------------------

set more off

capture program drop _all

sysuse cancer, clear

foomata died age


------------ runcommand.do -------------------

Running the do file will return the error "evaluator probit_lf() not found". I know that using a library or pre-compiling the mata code will make the command work, but I wanted to know if there was a fix to make this approach of including the mata code at the bottom of the ado file work with ml model.


Thanks in advance for the help, I look forward to hearing about any potential solutions.

Sarwar

[Matthew J. Baker]

Sarwar --

I think the issue is you have to pass the variables to the evaluator and use the evaluator functions from within Mata. I might set your problem up as follows, as an example:

program foomata
    version 12.1
    if replay() {
        if (`"`e(cmd)'"' != "foomata") error 301
        Replay `0'
    }
    else Estimate `0'
end

program Estimate, eclass sortpreserve
    syntax varlist(fv) 
    gettoken lhs rhs : varlist

    marksample touse

    mata: M=moptimize_init()
    mata: moptimize_init_evaluator(M,&probit_lf())
    mata: moptimize_init_depvar(M,1,"`lhs'")
    mata: moptimize_init_eq_indepvars(M,1,"`rhs'")
    mata: moptimize(M)
    mata: moptimize_result_post(M)
    ereturn display
end
mata:
void probit_lf(transmorphic scalar ML, real rowvector b, real colvector lnfj)
{
    real vector depvar, xb
    depvar = moptimize_util_depvar(ML, 1)
    xb = moptimize_util_xb(ML, b, 1)
lnfj = depvar:*ln(normal(xb)) + (1:-depvar):*ln(normal(-xb))
}
end

So, the optimizer is started from within Mata, and the variables are passed to the Mata function as arguments. You can also add additional arguments and pass them to the moptimize routine (as described in the help files for Mata). Note I pared away a few things to clarify the example a bit. You can test it by running the program, and then doing something like:

sysuse auto
foomata foreign mpg

Best,

Matt

p.s. I probably should have gotten rid of the "Replay" as it isn't working!

[Jeff Pitblado (StataCorp)]

Matt gives a work around for Sarwar, the only shortcoming I see is that parsing the standard
ml options will require more work from Sarwar.

The problem with Sarwar's original code in foomata.ado is that the Mata function
probit_lf() is local to the foomata command, and thus not accessible to the
ml command. This is similar to putting the code for an ado-language likelihood evaluator
in the same ado-file as the command calling ml.

Sarwar's options are to put the Mata code in a separate file and either compile it into an
mo file (Mata object file) or create a Mata library file.

To learn more about creating Mata object files see [M-3] mata mosave.

To learn more about create Mata libraray files see [M-3] mata mlib.

[Matthew J. Baker]

Jeff and Sarwan --

Of course, Jeff is %100 correct and is one of the three wise men when it comes to these sorts of problems in Stata (Gould and Sribney rounding out the triumvirate). The only thing I might add in defense of my solution is that other users can very easily see the whole command with my solution by opening up the ado file, or using the type or viewsource command. If the code is compiled and packaged as Jeff describes, it is harder to see how the Mata function(s) is constructed.

Best,

Matt

[Sarwar Islam]

Thank you for your quick replies Jeff and Matthew,

The advantage of using ml which Jeff mentions (Parsing the standard ml options for moptimize() would require more work where ml do all these nicely and neatly for you) and the advantage of using mata code in the ado file that Matthew mentions are both things that I have thought about which is why I was hoping that there would be a workaround for using mata code in the ado file without using moptimize() and find a way for it to work with ml. However, it seems like this isn't possible, so will probably have to stick to using a mata library.

Another reason for my reluctance to use a library is because from what I understand, if the library is compiled in Stata 14 at the time of the user written command's release, if a user tries to run this command in Stata 13, there is a backward compatibility problem as it seems that Stata 13 is unable to detect the library that was compiled in Stata 14. I've discussed potential workarounds of this by instead actually compiling the library at the time of running the command by the user in their version of Stata instead of pre-compiling it and have a code within it that checks for the correct version etc (This way might also be a bit messy - but could work.)

I'd appreciate any other suggestions for a fix on this library compiling issue in Stata 14 or any other potential workaround solutions to this problem.

Thanks again for both of your inputs, it has definitely helped clarify my understanding on this matter.