Mata block within a Stata block (-if-, -forvalue- etc)

[David Mant]

Hi All,

the issue is that I don't think Stata parser/compiler is able to do its job when a mata {block} is contained within a Stata {block}, such as -if {...}-, but also -forvalue {...}- etc.
Please try to run the following code and let me know why you think it behaves in such an unexpected way.

The problem is self-explained in the code:

local x = 1
if `x'!=1 {
    di "1) THIS message should not be displayed, and it is not"
    }
if `x'!=1 {
    di "2) THIS message should not be displayed, and it is not"
    mata {
        1000
    }
    di "3) THIS message should not be displayed, but it is!"
}

Please note:
- I'm aware I could have terminated the mata block by using -end-, but this would not work when the block is nested in a Stata -forvalue- for example
- I'm aware I could have created a mata function and call it. However, long story short, please assume we want to insert a mata block {}, not a mata function

Many thanks.

PS: I have raised a similar problem in another thread, but I thought starting a new one would be a good idea, as I've now realised the issue is much wider/simpler and not as specific/complex as I initially thought it was in the other topic. So please let's consolidate the discussion here.

[William Lisowski]

Interestingly enough, this works exactly as expected.

local x = 1
if `x'!=1 {
    di "1) THIS message should not be displayed, and it is not"
    }
if `x'!=1 {
    di "2) THIS message should not be displayed, and it is not"
    mata {  1000 }
    di "3) THIS message should not be displayed, but it is!"
}

as does

local x = 1
if `x'!=1 {
    di "1) THIS message should not be displayed, and it is not"
    }
if `x'!=1 {
    di "2) THIS message should not be displayed, and it is not"
    mata {  
        1000 }
    di "3) THIS message should not be displayed, but it is!"
}

as does

local x = 1
if `x'!=1 {
    di "1) THIS message should not be displayed, and it is not"
    }
if `x'!=1 {
    di "2) THIS message should not be displayed, and it is not"
    mata {  
        1000 
        2000 }
    di "3) THIS message should not be displayed, but it is!"
}

but not

local x = 1
if `x'!=1 {
    di "1) THIS message should not be displayed, and it is not"
    }
if `x'!=1 {
    di "2) THIS message should not be displayed, and it is not"
    mata {  
        1000 
        2000 
    }
    di "3) THIS message should not be displayed, but it is!"
}

which fails in exactly the same way as the code in post #1.

So the problem seems to depend in some sense on line endings. Now, the output of help m2_semicolons would seem to be definitive on this situation, but in fact the more I read it the less I understand and it doesn't seem to address this situation either explicitly or implicitly.

If we're lucky, someone from Statacorp may weigh in on this. But you might want to send a note to Stata Technical Services with a link to this thread asking if they have any thoughts on the issues raised herein.

[daniel klein]

Out of curiosity, where exactly is the syntax

mata {
[...]
}

documented in the first place? If it is not, why would you believe it behaves in a certain way?


Edit:

By the way, William's code does not really work; this is just hidden. Try

local x = 0 // <- change here so expression is true
if `x'!=1 {
    di "1) THIS message should not be displayed, and it is not"
    }
if `x'!=1 {
    di "2) THIS message should not be displayed, and it is not"
    mata {  1000 }
    di "3) THIS message should not be displayed, but it is!"
}

which results in

[...]
.     di "2) THIS message should not be displayed, and it is not"
2) THIS message should not be displayed, and it is not
.     mata {  1000 }
invalid expression
r(3000);
[...]

Best
Daniel

[David Mant]

William Lisowski : Yes, you got the point - the {...} are not dealt with properly. I have other examples when you use the same structure in a -forvalue-, and fails in a similar way. Interesting the other examples you made. I've send a request for support to Stata. Let's see, I'll post here if they don't post directly. Agree that help m2_semicolons talks about the matter, but as you say it keeps clear from killing the issue.

daniel klein : Good question about where the syntax mata {...} is documented in the first place. Short answer is, I don't know. I have been using this syntax for a while, without issues, and I recall I originally found it in a post on this forum. It was suggested by a member because he wanted to nest a mata block within a -forvalue-, and in that case you cannot use -end- to close the mata block, as it causes an error when nested in a loop.

Thanks for your input so far.

[daniel klein]

While I think it would be interesting to read an answer from StataCorp, I do not think this is a serious problem. If your Mata code is more than one line, define a respective function outside the loop; this way your code will also be more readable and easier to debug (in my opinion).

Best
Daniel

[William Lisowski]

where exactly is the syntax ... documented in the first place?

Combining the sources shown below seems to suggest, without providing an explicit example or counterexample, that the syntax is allowed.

help m3 mata

        Syntax 3                        Comment
        ----------------------------------------------------------------------
        mata  istmt                     rarely used
        ----------------------------------------------------------------------
        Syntax 3 is the single-line variant of syntax 1, but it is not
        useful.

help m2 syntax

        istmt :=        stmt
                       [other stuff]

        stmt :=         nothing
                        ;                             (meaning nothing)
                        version number
                        { stmt ... }
                        exp
                        pragma pstmt
                        if (exp) stmt
                        if (exp) stmt else stmt
                        for (exp;exp;exp) stmt
                        while (exp) stmt
                        do stmt while (exp)
                        break
                        continue
                        label:
                        goto label
                        return
                        return(exp)

help m2 semicolons

    Mata allows, but does not require, semicolons.

    Most languages that use semicolons follow the rule that a statement continues up to the
    semicolon.

    Mata follows a different rule:  a statement continues across lines until it looks to be complete,
    and semicolons force the end of statements.

[Jean-Claude Arbaut]

Apparently, this is not about line endings, but about Ado being a line interpreter. The following works:

local x = 1
if `x'!=1 {
    di "1) THIS message should not be displayed, and it is not"
}
if `x'==1 {
    di "2) THIS message should not be displayed, and it is not"
    mata {
        1000
    }
    di "3) THIS message should not be displayed, but it is!"
}

What is the difference? The code inside the if command is executed. That is, when the test evaluates to "true", Ado runs the subsequent lines, when the test evaluates to "false", it looks for an ending curly brace lying alone on a line, and does not recognize that mata could have one. You can put a forvalue or foreach command inside an if, and it will be correctly interpreted, but it seems mata is not supposed to have curly braces inside an Ado program. That's probably why, usually, you only put a function call inside Ado, and define the Mata function outside.

Another example: in the folowing, the for loop will be executed entirely, before Ado complains that there is a missing ending curly brace. The interpreter does not analyse the syntax beforehand.

if 1 {
    forv i=1/10 {
        di `i'
    
}

Another instructive example

loc a "{"
loc b "}"
`a'
di 1
`b'

Here the opening brace works, but not the closing brace. But this works:

loc a "{"
`a'
di 1
}

It seems Ado is first looking for an ending curly brace to close the inner block, when not found it expands macros and only then executes the line.

HTH

Jean-Claude Arbaut

[David Mant]

Yes, all your comments make sense, thank you.
In the meantime, Stata support responded:

Dear David,

This issue is occuring due to the blocks of both mata and how Stata interpr=
ets blocks from loops.

In most cases, Stata is trying to interpret blocks of code as the same type=
. This can causes issues when introducing a different "language" into the l=
oop.

This can be handled two ways.

1) If you only need a couple of mata lines in your loop, you can perform ma=
ta operations in a single line.

forvalues i =3D 1/5 {
mata: 1000
mata: "Hello"
}

2) If you need large amounts of mata code however, you should define your m=
ata block outside of your loop and then call the mata block. For example:

mata:
void hello1000() {
"hello"
1000
}
end

forvalues i =3D 1/5 {
mata: hello1000()
}



Sincerely,

[William Lisowski]

Thank you for passing along the Stata Technical Services response, which supports the very nice explanation in post #7, for which I thank Jean-Claude.

"A block within an enclosing block really should be of the same type" seems to be the operative rule, and an easy enough rule to express that I hope to be able to remember it in the future.

[David Mant]

Yes, I guess all is clear now (whether we like it or not).

My very personal view is that, from Stata, I feel we should expect it to interpret correctly the syntax with nested {...}.
It does not feel rocket science, even recognising the different nature/objectives of Stata and Mata compilers.

I think I'll still be able to sleep tight in the meantime.

Thank you all.