System Limit Exceeded -- see manual with loop

[Ben Raymond]

Hi, I get the error "system limit exceeded -- see manual" when I try to run the attached .do file. I am looping over monthly Current Population Data, so I am looping over month and year. These data are freely available here: https://data.nber.org/cps-basic2/dta/. The strange thing is I don't get this error if I run the code without a loop (i.e. for a single month and year combination). Any insight would be greatly appreciated!

[Jared Greathouse]

People won't look at the file. Please provide your full code here, as well as the code to import the data from online, and the exact loop you tried.

[William Lisowski]

The output of help limit tells you what the system limits are, depending on the version of Stata you are running. But then it can be tough to figure out just what limit you might have exceeded.

Please let us know what version you are running. The output of the about command will tell you precisely. On my system

. about

Stata/SE 17.0 for Mac (Intel 64-bit)
Revision 15 Feb 2022

[Ben Raymond]

Here is my version information:

. about

Stata/SE 16.1 for Windows (64-bit x86-64)
Revision 30 Jun 2021

Statalist wont let me post the code due to character limit constraints. The .do file itself is about 7300 lines long with about 45,000 characters. I could trim down a section of the code and post that, but I dont think it would generate the same error given that size limits seems to be the issue. I suspect there must be some limit to the number of characters or lines within a loop? I am using two loops, one nested within another. I am happy to provide more information, but Im not sure what you would find helpful since the code itself is too big to post?

[William Lisowski]

I suspect there must be some limit to the number of characters or lines within a loop?

You are correct. In a discussion on Statalist some time ago, which I cannot quickly locate, a StataCorp employee explained that a block of code enclosed within braces was in many respects the same as a Stata program. And we see from help limits that programs are limited to 3500 lines. So let's explore that. First, a small test to demonstrate the concept.

local nl 10

file open code using code.txt, write text replace 
file write code "clear" _newline
file write code "set obs 1" _newline
file write code "generate x = 1" _newline
file write code "forvalues i=1/2 {" _newline

forvalues i=1/`nl' {
    file write code "quietly replace x = x+`i'" _newline
}

file write code "}" _newline
file write code "describe" _newline

file close code

include code.txt

. include code.txt

. clear

. set obs 1
Number of observations (_N) was 0, now 1.

. generate x = 1

. forvalues i=1/2 {
  2. quietly replace x = x+1
  3. quietly replace x = x+2
  4. quietly replace x = x+3
  5. quietly replace x = x+4
  6. quietly replace x = x+5
  7. quietly replace x = x+6
  8. quietly replace x = x+7
  9. quietly replace x = x+8
 10. quietly replace x = x+9
 11. quietly replace x = x+10
 12. }

. describe

Contains data
 Observations:             1                  
    Variables:             1                  
------------------------------------------------------------------------------------------------
Variable      Storage   Display    Value
    name         type    format    label      Variable label
------------------------------------------------------------------------------------------------
x               float   %9.0g                 
------------------------------------------------------------------------------------------------
Sorted by: 
     Note: Dataset has changed since last saved.

.

Now let's change the local nl from 10 to 10000. Here's the head and tail ends of the log that results.

. include code.txt

. clear

. set obs 1
Number of observations (_N) was 0, now 1.

. generate x = 1

. forvalues i=1/2 {
  2. quietly replace x = x+1
  3. quietly replace x = x+2
  4. quietly replace x = x+3
  5. quietly replace x = x+4
  6. quietly replace x = x+5
 [...]
3497. quietly replace x = x+3496
3498. quietly replace x = x+3497
3499. quietly replace x = x+3498
3500. quietly replace x = x+3499
3501. quietly replace x = x+3500
system limit exceeded - see manual
r(1000);

And we see that it failed on the command that follows the 3500th command within between the braces.

I'm guessing you're going to have to rethink your do-file. What I would suggest - I'm looking at your program at the moment - is that you need to get your crosswalk blocks of code out from within the loops. This will be simple because the replace commands that constitute those blocks don't include local macro references - they're just plain vanilla Stata commands. So you can do something like the following above your loops, using cut-and-paste to move the code lines out of the loop and into the programs.

capture program drop indxw0308
program define indxw0308
replace naics="111" if ((peio1icd=="170")&(year<=2008))
replace naics="112" if ((peio1icd=="180")&(year<=2008))
replace naics="1131" if ((peio1icd=="190")&(year<=2008))
replace naics="1133" if ((peio1icd=="270")&(year<=2008))
replace naics="114" if ((peio1icd=="280")&(year<=2008))
...
replace naics_j2="928110" if ((peio2icd=="9770")&(year<=2008))
replace naics_j2="928110" if ((peio2icd=="9780")&(year<=2008))
replace naics_j2="928110" if ((peio2icd=="9790")&(year<=2008))
replace naics_j2="928110" if ((peio2icd=="9870")&(year<=2008))
end

and then inside the loops

indxw0308
indxw0913
...

I think that will do what you need, but I'm not up for writing a demo of it at the moment.

[Ben Raymond]

Thanks William! I appreciate the help. Good to know that code in braces is basically the same as a Stata program. I will do as you suggest and create programs outside the loop. Thanks again!