Mata library not found

[Stephen Jenkins]

I am trying to run jobs on my institution's server (Windows; Stata 13). I not have write-access to many directories. I need to be able to access Ben Jann's -moremata- package (on SSC). I have used the -ssc copy- facility to copy the following files to my current working directory on the server

. dir *mlib
   0.3k  12/24/15 19:09  lmoremata.mlib   
   0.0k  12/24/15 19:09  lmoremata10.mlib

Following Bill Goul's advice in http://www.stata.com/statalist/archi.../msg00836.html, I have also done the following but, as you'll see, it appears that Stata still can't find one of the components of the package

. mata
------------------------------------------------- mata (type end to exit) -------
: mata mlib index
.mlib libraries to be searched are now
    lmatabase;lmatasem;lmatapss;lmatapath;lmataopt;lmatagsem;lmatafc;lmataado;lmo
> remata;lmoremata10

: end
---------------------------------------------------------------------------------

. mata: mata mlib query
.mlib libraries to be searched are
    lmatabase;lmatasem;lmatapss;lmatapath;lmataopt;lmatagsem;lmatafc;lmataado;lmo
> remata;lmoremata10

. mata: mata which mm_colvar()
function mm_colvar() not found
r(111);

Advice please about how to make the functions in -moremata- accessible to my program. thanks.

[Christophe Kolodziejczyk]

on my computer I have several versions of moremata

lmoremata;lmoremata10;lmoremata14;lmoremata_old

lmoremata has no member, lmoremata10 has 1 member, whereas lmoremata_old has plenty of functions/members. Maybe you need this file. Anyway mata describe should help you to inspect the content of the libraries you copy.

Am I correct in saying that the moremata package is not on the server, but you can download files from the internet?

mata:
mata describe using lmoremata
mata describe using lmoremata10
mata describe using lmoremata_old
end

mata:
------------------------------------------------- mata (type end to exit) -------------
: mata describe using lmoremata
(library contains 0 members)


: mata describe using lmoremata10
(library contains 1 member)


      # bytes   type                        name and extent
-------------------------------------------------------------------------------
          452   auto numeric matrix         _mm_colrunsum10()
-------------------------------------------------------------------------------

: mata describe using lmoremata_old
(library contains 185 members)


      # bytes   type                        name and extent
-------------------------------------------------------------------------------
        1,736   auto real colvector         ___mm_ranks()
          288   auto real colvector         __mm_ranks()
        1,608   auto transmorphic matrix    _mm_callf0()
        1,712   auto transmorphic matrix    _mm_callf1()
        1,816   auto transmorphic matrix    _mm_callf2()
        1,920   auto transmorphic matrix    _mm_callf3()
        2,024   auto transmorphic matrix    _mm_callf4()
        2,128   auto transmorphic matrix    _mm_callf5()
          688   auto real scalar            _mm_cebinomial()
        1,832   auto real matrix            _mm_collapse()
          264   auto void                   _mm_colrunsum()
          104   auto numeric matrix         _mm_colrunsum_goto10()
          844   auto real colvector         _mm_composition()
          352   auto real colvector         _mm_composition2()
          280   auto struct scalar          _mm_composition2setup()
          872   auto struct scalar          _mm_compositionsetup()
        2,056   auto void                   _mm_expand()
        1,032   auto real colvector         _mm_expandclusters()
          400   auto pointer scalar         _mm_findkdel0()
          400   auto pointer scalar         _mm_findkern()
          400   auto pointer scalar         _mm_findkint()
        1,192   auto real vector            _mm_finvert_brent()
          140   auto real scalar            _mm_finvert_brent_f()
          888   auto real vector            _mm_finvert_nr()
          192   auto real rowvector         _mm_finvert_nr_fdf()
          968   auto real colvector         _mm_freq()
        1,008   auto real colvector         _mm_freq2()
          368   auto real scalar            _mm_gini()
        4,084   auto void                   _mm_matlist()
          748   auto real matrix            _mm_mgof_approx()
          840   auto real scalar            _mm_mgof_cr()
        4,560   auto real matrix            _mm_mgof_ee()
          536   auto real scalar            _mm_mgof_ksmirnov()
          320   auto real scalar            _mm_mgof_lr()
        2,620   auto real matrix            _mm_mgof_mc()
          864   auto real colvector         _mm_mgof_mc_smpl()
          520   auto real colvector         _mm_mgof_mc_usmpl()
          544   auto real scalar            _mm_mgof_mlnp()
        2,592   auto real matrix            _mm_mgof_rc()
          304   auto real scalar            _mm_mgof_x2()
          728   auto real scalar            _mm_npartitions()
          936   auto real scalar            _mm_npieces()
          584   auto real colvector         _mm_panels()
        1,608   auto real colvector         _mm_partition()
        1,320   auto real colvector         _mm_partition2()
          468   auto real colvector         _mm_partition2_expand()
          944   auto struct scalar          _mm_partition2setup()
          632   auto struct scalar          _mm_partitionsetup()
        1,196   auto real scalar            _mm_plot()
        1,456   auto real scalar            _mm_polint()
          992   auto real colvector         _mm_quantile()
        1,544   auto real colvector         _mm_quantilew()
        1,584   auto real colvector         _mm_quantilewalt()
          928   auto real colvector         _mm_ranks()
          592   auto real colvector         _mm_ranks_mid()
          600   auto real scalar            _mm_rbinomial()
        1,288   auto real colvector         _mm_relrank()
        1,012   auto void                   _mm_repeat()
          848   auto real scalar            _mm_strexpand()
          320   auto string scalar          _mm_unabkern()
          768   auto transmorphic matrix    _mm_uniqrows()
          132   auto transmorphic matrix    mm__jumble2()
        1,372   auto real vector            mm_benford()
        3,800   auto struct scalar          mm_bs()
        3,744   auto struct scalar          mm_bs2()
        4,660   auto real matrix            mm_bs_report()
          324   auto structdef scalar       mm_bsstats()
          856   auto transmorphic matrix    mm_callf()
          860   auto structdef scalar       mm_callf_o10()
          848   auto struct scalar          mm_callf_setup()
          320   auto real matrix            mm_cauchy()
          304   auto real matrix            mm_cauchyden()
          172   auto real matrix            mm_cauchytail()
        1,616   auto real matrix            mm_cebinomial()
        2,840   auto real matrix            mm_collapse()
          368   auto real rowvector         mm_colmse()
          336   auto numeric matrix         mm_colrunsum()
          368   auto real rowvector         mm_colsse()
          388   auto real rowvector         mm_colvar()
          344   auto real colvector         mm_composition()
          808   auto real matrix            mm_compositions()
          528   auto struct scalar          mm_compositionsetup()
        1,680   auto transmorphic matrix    mm_cond()
          944   auto real colvector         mm_cut()
          336   auto real matrix            mm_ecdf()
        1,716   auto real colvector         mm_exactbin()
          472   auto transmorphic matrix    mm_expand()
        1,304   auto real colvector         mm_fastlinbin()
          736   auto real vector            mm_finvert()
          640   auto real colvector         mm_freq()
          552   auto real colvector         mm_freq2()
          508   auto real rowvector         mm_gini()
          984   auto real matrix            mm_histogram()
        1,192   auto void                   mm_hunt()
        1,472   auto string matrix          mm_infile()
        1,848   auto string matrix          mm_insheet()
          448   auto real matrix            mm_invcauchy()
          784   auto string scalar          mm_invtokens()
        2,296   auto real colvector         mm_ipolate()
          416   auto real matrix            mm_iqrange()
          160   auto real scalar            mm_isconstant()
        4,152   auto struct scalar          mm_jk()
        3,628   auto real matrix            mm_jk_report()
          276   auto structdef scalar       mm_jkstats()
          116   auto transmorphic matrix    mm_jumble2()
          168   auto real matrix            mm_kdel0()
           80   auto real scalar            mm_kdel0_biweight()
          208   auto real scalar            mm_kdel0_cosine()
           80   auto real scalar            mm_kdel0_epan2()
          120   auto real scalar            mm_kdel0_epanechnikov()
          128   auto real scalar            mm_kdel0_gaussian()
           96   auto real scalar            mm_kdel0_parzen()
           96   auto real scalar            mm_kdel0_rectangle()
           80   auto real scalar            mm_kdel0_triangle()
           96   auto real scalar            mm_kdel0_triweight()
          196   auto real matrix            mm_kern()
          160   auto real matrix            mm_kern_biweight()
          192   auto real matrix            mm_kern_cosine()
          152   auto real matrix            mm_kern_epan2()
          208   auto real matrix            mm_kern_epanechnikov()
           68   auto real matrix            mm_kern_gaussian()
          384   auto real matrix            mm_kern_parzen()
          136   auto real matrix            mm_kern_rectangle()
          108   auto real matrix            mm_kern_triangle()
          168   auto real matrix            mm_kern_triweight()
          368   auto real matrix            mm_kint()
        1,640   auto real matrix            mm_kint_biweight()
        2,024   auto real matrix            mm_kint_cosine()
        1,240   auto real matrix            mm_kint_epan2()
        1,552   auto real matrix            mm_kint_epanechnikov()
        1,000   auto real matrix            mm_kint_gaussian()
        4,648   auto real matrix            mm_kint_parzen()
        1,160   auto real matrix            mm_kint_rectangle()
        1,720   auto real matrix            mm_kint_triangle()
        1,936   auto real matrix            mm_kint_triweight()
        1,648   auto real colvector         mm_linbin()
          632   auto void                   mm_locate()
          648   auto real colvector         mm_makegrid()
        3,380   auto void                   mm_matlist()
          416   auto real matrix            mm_meancolvar()
          496   auto real matrix            mm_meanvariance0()
          224   auto real rowvector         mm_median()
        4,460   auto real matrix            mm_mgof()
          372   auto real matrix            mm_mse()
          248   auto real scalar            mm_ncompositions()
          264   auto real scalar            mm_nobs()
          304   auto real scalar            mm_npanels()
          752   auto real scalar            mm_npartitions()
          576   auto real matrix            mm_npieces()
        1,192   auto real scalar            mm_nrroot()
          472   auto real scalar            mm_nuniqrows()
          352   auto real scalar            mm_nunique()
          984   auto void                   mm_outsheet()
        1,616   auto void                   mm_panels()
          488   auto real colvector         mm_partition()
          808   auto real matrix            mm_partitions()
          640   auto struct scalar          mm_partitionsetup()
        1,240   auto string rowvector       mm_pieces()
          268   auto void                   mm_plot()
        1,356   auto real vector            mm_polint()
          212   auto real scalar            mm_posof()
        1,944   auto real matrix            mm_quantile()
        1,024   auto real matrix            mm_ranks()
        1,208   auto real matrix            mm_rbinomial()
          688   auto real colvector         mm_rcomposition()
          604   auto real matrix            mm_realofstr()
        1,616   auto real matrix            mm_relrank()
          360   auto transmorphic matrix    mm_repeat()
        2,992   auto real scalar            mm_root()
          432   auto real colvector         mm_rsubset()
        4,876   auto real colvector         mm_sample()
          740   auto string colvector       mm_smatlist()
        1,032   auto real colvector         mm_srswor()
          776   auto real colvector         mm_srswr()
          336   auto real matrix            mm_sse()
          560   auto string scalar          mm_strexpand()
          632   auto real colvector         mm_subset()
          416   auto structdef scalar       mm_subsetinfo()
          648   auto real matrix            mm_subsets()
          784   auto struct scalar          mm_subsetsetup()
          128   auto real colvector         mm_unorder2()
        1,896   auto real colvector         mm_upswor()
        1,224   auto real colvector         mm_upswr()
          512   auto real matrix            mm_variance0()
          192   auto real matrix            mm_which()
-------------------------------------------------------------------------------

: end
---------------------------------------------------------------------------------------

. 
end of do-file

.

[Stephen Jenkins]

Dear Christophe: many thanks for your helpful remarks. Yes, I can copy files from SSC to the directories on the server that I have write-access to, using -ssc copy-.

I don't have moremata_old.mlib file, nor does it appear to be on SSC (according to -ssc describe l-). But on my desktop the contents of the file you show appear to be in moremata.lib. Here's what I get there:

. mata: mata describe using lmoremata
(library contains 189 members)


      # bytes   type                        name and extent
-------------------------------------------------------------------------------
        1,736   auto real colvector         ___mm_ranks()
          288   auto real colvector         __mm_ranks()
        1,608   auto transmorphic matrix    _mm_callf0()

<snip>

The mm_colvar() function that my program is looking for is in that collection. The moremata*.mlib files are dated as follows

. dir D:\home\stephenj\ado\STBPLUS\l\lmoremata*.mlib
 405.4k   6/03/15 21:13  lmoremata.mlib   
  21.0k   2/11/09  8:56  lmoremata10.mlib 
 271.4k   6/03/15 21:13  lmoremata14.mlib

Meanwhile, back on the server, where the moremata libraries are in the current working directory, I tried your suggestion to try -mata describe using- and got the following:

. mata: mata describe using lmoremata
.\lmoremata.mlib from a more recent version of Stata
r(610);

. mata: mata describe using lmoremata10
could not read .\lmoremata10.mlib
r(612);

Recall the server is using Stata version 13.1, so perhaps the file I copied from SSC yesterday is version 14 (though that doesn't explain why the moremata10.lib file is not readable).

Any further ideas or suggestions, please?

[Christophe Kolodziejczyk]

Maybe you're right, the first error message you get suggests that lmoremata.lib is a Stata14 version. At least it says that the version of Stata on the server is older than the mata lib. So maybe you could check the Stata version and ensure that it is 13.1 and not an older version, although I find it unlikely. I couldn't find a way to determine which version was used to compile a specific mata library.

If the server is not completely closed, i.e. you it can communicate with the internet, then you can download packages in the usual way in a directory which Stata identifies as the PLUS directory. You can do that with

sysdir set PLUS "D:\home\stephenj\ado\STBPLUS"

Have you tried that?


In last resort you could compile the source code yourself.

[Stephen Jenkins]

Thanks again, Christophe. I'm currently trying the "last resort" option. That is, I'm taking account of the fact that moremata_source.hlp contains the source code; I've then

. mata: mata mlib create lpersonal
(file lpersonal.mlib created)

Continuing to follow http://www.stata.com/manuals13/m-3matamlib.pdf, I then (a) compiled the functions I required and then (b) added them to the lpersonal library. Now I get

. mata: mata mlib index
.mlib libraries to be searched are now
    lmatabase;lmatasem;lmatapss;lmatapath;lmataopt;lmatagsem;lmatafc;lmataado;lmoremata - copy;lmoremata;lmoremata10;lpersonal

. mata: mata describe using lpersonal
(library contains 6 members)


      # bytes   type                        name and extent
-------------------------------------------------------------------------------
          388   auto real rowvector         mm_colvar()
        1,680   auto transmorphic matrix    mm_cond()
        2,296   auto real colvector         mm_ipolate()
        1,192   auto real scalar            mm_nrroot()
        1,944   auto real matrix            mm_quantile()
          192   auto real matrix            mm_which()
-------------------------------------------------------------------------------

Assuming I've extracted all the moremata functions that are called, I hope that I'm ready to roll. I'll set my Monte-Carlo analyses running and report back (it'll likely be tomorrow, given the runtime)
Thanks again.

[Stephen Jenkins]

The first of the server jobs has not fallen over, whereas it should have by now if there had been continuing problems with calling moremata. Something is now going right!

[Stephen Jenkins]

PS for some reason, the 5 Monte-Carlo jobs run in 5 separate Stata instances after the first job (the one described as working ok in #6) keep on failing. These jobs are identical to the first, apart from starting with a different seed. The error messages state that one of the moremata functions is not found -- which is strange, as -mata: describe using lpersonal- suggests that it is there, in each of the 5 instances. I've run out of inspiration now, and so I think I am going to have to limit myself to one server job. [I have 2 other instances running successfully on an old laptop and desktop.]

[Stephen Jenkins]

Closure: I think I've discovered the problem (and mea culpa). In #5, I stated

Assuming I've extracted all the moremata functions that are called, ...

But looking closer at the errors, I think the issue is that mmquantile() in the moremata package calls 3 other _mmquantile*() functions. Note the underscore prefix! I hadn't, it seems. Thus I hadn't compiled all the relevant functions as previously assumed.
So, I've compiled those additional functions and added them to my lpersonal library. And I currently have 5 jobs running on the server without error (or at least without the same error as before). Yippee.
[One remaining mystery is how I got one job to run without error. My conjecture is that I may have 'accidentally' compiled the additional _mmquantile() functions in one server instance ... but not saved and added them to a library.]
Thanks again for your help and suggestions along the way.