Changing from Mata to Stata within a while loop - error

[Frank Stoer]

Dear Statalist members,

I am running a do file for a dataset including a series of commands. It also includes multiple times of switching between Stata and Mata. Since I have to repeat the process for many datasets I created a while loop, looking something like this:

local i=2016
while `i'>=2003 {
local j=4
while `j'>=1 {

local j=`j'-1
}
local i=`i'-1
}

Normally exiting Mata of course is done with the command end. However, since I am switching within the while loop it errors. It gives error code r(1) - meaning I have hit the Break button. I however have not hit the Break button. The reason I believe the problem is with the command end within the while loop is because the error is given right after this end command. Moreover, the whole Do-file runs perfectly fine for one dataset without the while loop.

Any ideas how to solve this problem?

Thank you in advance,

Frank

[William Lisowski]

I think you want to take your Mata code and make a Mata function out of it, something like the following.

mata:
function gnxl(i,j) {
    a = i
    b = j
}
end

forvalues i = 2016(-1)2003 {
    forvalues j = 4(-1)1 {
        mata: gnxl(`i',`j')
    }
}

With luck, someone more knowledgeable than I am will weigh in with further, and likely better, advice.

[Charlie Joyez]

Does your loop is written within Mata environment or within Stata one? (Within the mata --- end mata delimiters or in a traditional do file.)
Your code looks like traditional Stata, so in a Mata environment, it could lead to errors (but perhaps also what you didn't show us in the loop could cause the break).
If you're under Mata code, William provided you good advice. Otherwise, do not forget to type end mata before your loop.

I've written several programs jumping from Stata to Mata, with loops, and never faced your issue. I therefore tend to think the error comes from your code within the loop rather than from the loop.
Also, please consider using the \code delimeters (from advanced editor) to write your code, it would make it easier to read.

Best,
Charlie

[Frank Stoer]

Thank you for your comments. However, I think I did not explain myself clear enough. The while loop is written in the Stata environment, most of the code in the remainder of the do file is as well. However in three instances I switch to the Mata environment to create and store matrices, which I later use in the Stata environment again. This morning I once again tried to execute only the code within the loop and it did not cause any problems.

I am not sure if I understand what you mean with typ end mata before your loop, since mata is used within the loop. I did however use the end mata command within the loop to go back to the stata environment.

Hereby I ll add a few parts of my do file here for your understanding.


clear

local i=2016
while `i'>=2003 {
local j=4
while `j'>=1 {

clear

import delim using "/Volumes/Harde schijf Frank/Thesis/All raw datasets/Origin_and_Destination_Survey_DB1BMarket_`i'_`j'.c sv"

save "/Volumes/Harde schijf Frank/Thesis/All finished datasets/Origin_and_Destination_Survey_DB1BMarket_`i'_`j'.d ta"

************ Some other Stata commands ************

mata

Y = st_data(., ("airlineD1", "airlineD2", "airlineD3", "airlineD4", "airlineD5", "airlineD6", "airlineD7", "airlineD8", "airlineD9", "airlineD10", "airlineD11", "airlineD12", "airlineD13", "airlineD14", "airlineD15", "airlineD16", "airlineD17", "airlineD18", "airlineD19", "airlineD20", "airlineD21", "airlineD22", "airlineD23", "airlineD24", "airlineD25", "airlineD26", "airlineD27", "airlineD28", "airlineD29", "airlineD30"))
Z=Y'Y

st_matrix("MatrixZ", Z)

end mata

matrix list MatrixZ

clear

********** Some other Stata commands **********

save "/Volumes/Harde schijf Frank/Thesis/All finished datasets/Origin_and_Destination_Survey_DB1BMarket_`i'_`j'.d ta", replace

local j=`j'-1
}
local i=`i'-1
}



Thank you in advance for your help.

Frank

[William Lisowski]

Here is sample code that reproduces the problem you experience.

forvalues f=0/1 {
    sysuse auto, clear
    keep if foreign==`f'
    keep mpg weight
    mata:
        Y=st_data(.,("mpg","weight"))
        Z=Y'Y
        st_matrix("MatrixZ",Z)
    end
    matrix list MatrixZ
    }

And here is a solution along the lines suggested in post #2 abov

capture mata: mata drop fun()
mata:
function fun() {
    Y=st_data(.,("mpg","weight"))
    Z=Y'Y
    st_matrix("MatrixZ",Z)
    }
end
forvalues f=0/1 {
    sysuse auto, clear
    keep if foreign==`f'
    keep mpg weight
    mata fun()
    matrix list MatrixZ
    }

I'm not a Mata expert, so my code may not be as good, or as elegant, as it should be.

[Niels Henrik Bruun]

Hi
I like William suggestion. It is the best for development purposes.
However a solution for post #5 could also be (using curly brackets around Mata code):

forvalues f=0/1 {
    sysuse auto, clear
    keep if foreign==`f'
    keep mpg weight
    mata {
        Y=st_data(.,("mpg","weight"))
        Z=Y'Y
        st_matrix("MatrixZ",Z)
    }
    matrix list MatrixZ
    }

[William Lisowski]

My thanks to Niels. That was the sort of answer I'd hoped to find, but the (portions) of the Mata documentation I waded through did not suggest it (or I didn't see the suggestion). In particular, help m3 mata which covers mata invocation does not suggest anything other than the mata and end pair. I'm sure it's suggested somewhere, or implicit in Stata syntax, but I'm glad to have Niels explain how to mimic "interactive Mata" in a do-file.

[Frank Stoer]

Thanks a lot for your help. It worked.

Frank

[jerome falken]

Thank you Niels, I didn't know this construct!

[Jeff Timberlake]

I stumbled across this exchange while searching for a solution to a similar problem. The issue I'm having right now is that it doesn't seem like the while loop is ending correctly and I'm wondering if Frank had the same problem.

The basic syntax looks like this (the only relevant Stata command is the first one, for reasons I describe below):

local i = 1
while `i' <= 19 {
    
quietly estpost svy: tab gad2_one *_`i' if gender == 1, nomarginal

    mata {

*****Mata commands 1****

        }

*****Stata commands 2****    

     mata {

*****Mata commands 2****

        }

*****Stata commands 3****

    mata {

*****Mata commands 3****

        }

*****Stata commands 4****

local i = `i' + 1
}

Because I'm just learning Mata, there might be a more efficient way to do what I'm trying to do, but it does the job. The problem is that the loop doesn't end when i = 20, so in the first Stata command I get an error message saying "variable *_20 not found," meaning that the while loop didn't terminate at i = 20.

I'm wondering if using the close braces around the mata commands (which solved the problem Frank Stoer was having in this thread, and which was the reason I found it--I was having the same problem) somehow trips up the while loop? Any suggestions you all can offer would be greatly appreciated.

Jeff

[Niels Henrik Bruun]

Dear all
The best answer to how to use both Stata and Mata is to separate Stata and Mata parts as much as possible.
Gather your Mata code into functions or classes, then call these functions from Stata.
If you need to run Stata code between Mata code lines, it is often better to run this from within Mata using the Mata function stata() (-help mf_stata-).

In many cases, the best layout is to use Stata for input validation and possibly for return management, and Mata for most or all the calculations.

[John Mullahy]

What baffles me about #10 (and has baffled me in similar settings in the past) is that if one uses

forval i=`i'(1)19 {

instead of

while `i' <= 19 {

the problem doesn't arise. In fact I've generally tried to avoid while loops in favor of forval loops whenever possible, even though in many cases while-looping might be more intuitive.

[Joseph Coveney]

I'm wondering if using the close braces around the mata commands . . . somehow trips up the while loop?

What baffles me about #10 (and has baffled me in similar settings in the past) is that if one uses [-forvalues-] instead of [-while-] the problem doesn't arise.

I can replicate this (code below), and it seems as if you've uncovered a bug in Stata's -while- looping command.

-while- looks for and finds the next closing brace (which is for enclosing the Mata code section) and mistakes that for its own loop-closing brace. As John's discovered, -forvalues- appears to be keeping track of pairs of open- and close-braces that are enclosed in the loop, something that -while- appears not to be programmed to do (except perhaps in the case of nested -while- loops).

In the code below, the -forvalues- loop executes normally. The corresponding -while- loop executes through the countdown and is aware that the condition has become false as it enters the third time through. And it does skip execution of the code (marked in gray) on that third time through until it (i) sees the closing brace for the first section of embedded Mata code, (ii) mistakes that for its own closing brace, and (iii) resumes program execution immediately afterward.

version 17.0

clear *

*
* -for- loop (executes normally)
*
forvalues i = 1(1)2 {
    
    display in smcl as text _newline(2) "Stata 1 = `i'"
    mata {
        printf("Mata 1 = %2.0f\n", strtoreal(st_local("i")))
    }
    
    display in smcl as text "Stata 2 = `i'"
    mata {
        printf("Mata 2 = %2.0f\n", strtoreal(st_local("i")))
    }
}

*
* -while- loop (error during the third pass)
*
local i 1
while `i' <= 2 {

    display in smcl as text _newline(2) "Stata 1 = `i'"
    mata {
        printf("Mata 1 = %2.0f\n", strtoreal(st_local("i")))
    } // <= this is what the -while- loop mistakes for termination, and control resumes on the next line

    display in smcl as text "Stata 2 = `i'"
    mata {
        printf("Mata 2 = %2.0f\n", strtoreal(st_local("i")))
    }

    local ++i
}

exit

The code that is grayed is actually skipped in the third pass when `i' is 3 and the condition has become false. Program control resumes, however, right after that closing Mata brace that -while- mistakes for its own.

StataCorp should implement a brace counting (brace pairs that are enclosed in the loop, other than nested -while- loops) bookkeeping feature in -while- such as -foreach- seems to have.

For the Jeff, the offending line isn't the one that he shows

quietly estpost svy: tab gad2_one *_`i' if gender == 1, nomarginal

That line is actually skipped; rather, the error is from another one down later in his reacted code display, there seems to be another Stata command referencing *_`i' that he's elided.

[John Mullahy]

Joseph Coveney: Your results in #13 are clear and compelling. To get broader attention I wonder if this discussion should move out of the Mata forum and into the general Stata forum since the central concern seems to be a "Stata issue" and not a "Mata issue."

[Jeff Timberlake]

You guys rule!

@Niels--thanks for the suggestion. As I said, I've just started using Mata commands so I have a lot to learn.

@John--thanks for the general suggestion. I have never used forval before and it worked!

@Joseph--thanks for your contribution. As I said, the problem was fixed with using forval instead of while. I'll be sure to keep that in mind in the future.

Thanks again, guys.

Jeff