Compatibility of matsave/matuse and fputmatrix/fgetmatrix

[JanDitzen]

Hi Statalist users,
I have a question about matuse/matsave and fputmatrix/fgetmatrix using Stata 14. Are these two not compatible to each other? Implying that a file saved with matsave can not be processed using fgetmatrix? The background of my problem is, that I would like to add a matrix to an existing .mmat file with an undefined number of matrices already saved in it.

For example:

clear all
mata
A = rnormal(2,2,0,1)
B = rnormal(3,3,0,1)
mata matsave "test.mmat" A B , replace
  end
 clear mata
mata
C = rnormal(4,4,0,1)
fh = fopen("test.mmat", "rw")
fputmatrix(fh, C)
fclose(fh)
 end

clear mata

mata
fh = fopen("test.mmat", "r")
C = fgetmatrix(fh)
B = fgetmatrix(fh)
A = fgetmatrix(fh)
fclose(fh)
 end
clear mata
mata mata matuse "test.mmat"

Mata is not able to perform the line "B = fgetmatrix(fh)" with the error: "fgetmatrix(): 610 file format error" and the line "mata mata matuse "test.mmat"" aborts with the error "file test.mmat not .mmat-format file" (error code 610 as well).
My understanding from the helpfile for mata matsave is, that both are compatible to each other. Also I had a look into the sourcecode for matsave and my understanding is that matsave uses fputmatrix to save matrices.

So my question is, are these two completely different or is there somewhere a mistake in my code?
Thanks!
Jan

[Charles Cadestin]

Dear statalist members,
I encounter the same problem as this one. I try to write a program that will store matrices in ".mmat" format and that will be usable through

mata: mata matuse  foo.mmat

For that, as, explained in the Help file (see Remarks and examples), I use the fopen commands. The command works efficiently for saving. The problem arises when I try to use the data through the "mata: mata matuse foo.mmat" command.

Why? As suggested by

viewsource mmat_.mata

, the command "mata: mata matuse foo.mmat" checks the signature and returns an error if not the same. For this reason, I integrated the signature in my function. But it still does not work. Is there any workaround or I won't be able to use "mata: mata matuse foo.mmat" conjointly with my storage?

Here is an example:

local   MMAT_SUFFIX             `"".mmat""'
local   MMAT_SIGNATURE          `"(char((14, 87, 03, 01)) + "mmat")"'
local   MMAT_VER                `"(char(0) + char(1))"'
capture mata: mata drop store_mmat()
mata:
    void store_mmat(string scalar foo)
    {
        real matrix a,b,c
        real scalar fh
        string scalar filename
        
        a = 1
        b = 2
        c = 3

        signature = `MMAT_SIGNATURE'
        ver = `MMAT_VER'
        datetime = c("current_date") + " " + c("current_time")

        filename = foo + ".mmat"
        
        fh = fopen(filename, "rw")
        _fputmatrix(fh, a)
        _fputmatrix(fh, b)
        _fputmatrix(fh, c)
        _fwrite(fh,signature)
        _fwrite(fh,ver)
        _fwrite(fh,datetime)
        fclose(fh)
        }
end
/* Runs the function */
mata: store_mmat("attempt_1")
/* Does work */
mata:
    signature = `MMAT_SIGNATURE'
    fh = fopen("attemp_1.mmat", "r")
    a = fgetmatrix(fh)
    b = fgetmatrix(fh)
    c = fgetmatrix(fh)
    if (_fread(fh, strlen(signature)) == signature) "works"
    fclose(fh)
    a,b,c
end
/* Does not work */
mata:
    store_mmat("attempt_1")
    mata matuse attempt_1.mmat, replace
end

After having saved using fopen, "mata: mata matuse" fails to recognize my file as a ".mmat":

file attempt_1.mmat not .mmat-format file

Thank you very much!

Charles.

[Jean-Claude Arbaut]

Hi,

I am using Stata 15.1, but I guess that does not change the answer much.

If you inspect carefully mmat_.mata, you will see that there is a signature before data (well, you already saw that), but also that variables are not written directly. Instead, the first _fputmatrix will print a string colvector of variable names, then a second _fputmatrix will print a colvector of _pointers_ to the actual variables (along with the contents pointed to).
If you want to append, you will need to carefully read the name and pointer vectors, extend them, and write back everything in the same format.

Regarding the first question (nov. 2015): there seems also to be a mistake in the documentation of fopen. When mode is "rw", the file pointer is not positioned to the end, but to the begining. When running the code from the first question in this thread, fputmatrix(C) overwrites a part of test.mmat, starting from the begining. Then, the subsequent C=fgetmatrix(fh) successfully reads C, but of course the remaining bytes are not aligned on a correct mata structure, hence Mata signals a file format error when it tries to run B=fgetmatrix(fh). To append, one needs to use mode "a". However, as already explained above, it won't work here as the format is not so simple.

If there is not much data, it could be ok to read everything and write back the original data along with new contents, but it's not a very good solution. The best is probably to devise you own file format, designed to be easily append to. Or you may use several files.
However, if you only want to emulate the effect of mata matsave (that's the second question, april 2018), then just use its source code (if I understand the Stata license correctly, it's allowed, with a few reasonable constraints).

Note: as far as I know, the file format is undocumented. While you can guess using a hex dump, or use mmat_.mata source code, be aware that all of this could change in future versions.

Hope this helps

Jean-Claude Arbaut

[Charles Cadestin]

Hello Jean-Claude, better later than never... I wanted to thank you for your reply which was helpful understanding the function. I finally chose not to go on with the fputmatrix which (1) too complex for me, (2) subject to bugs if the code is changed in future versions (3) i don't know what's behind "it's allowed with a few reasonable constraints".).
Thank you again!
Best,
Charles.