about tuples:why observation numbers are out of range ? It should't be

[Fred Lee]

My university has bought Stata 15.1MP with 12 cores.
I only include 12 variables in for tuples command, why the observation numbers are out of range? Should't More than 2 billion observations be allowed?
Here is the sample code and data, could anyone explain this? Thanks a lot!

* Example generated by -dataex-. To install: ssc install dataex
clear
input float(c1 c2 c3) byte(c4 c5) float c6 byte(c7 c8 c9 c10) float c11 double y float x int id
.4201681  3.296207  14.13587 0 0 3.8728664 2 2 44 2  3.178054   54 0  1
.4201681  3.296207  14.13587 0 0 3.8728664 2 1 59 2  3.178054   76 0  1
.4201681  3.296207  14.13587 0 0 3.8728664 2 2 43 2  3.178054 68.5 0  1
.4201681  3.296207  14.13587 0 0 3.8728664 2 1 43 2  3.178054   49 0  1
.4201681  3.296207  14.13587 0 0 3.8728664 2 1 48 2  3.178054   76 0  1
      .6  5.588671         . 0 0 3.8728664 2 2 33 2 3.0445225   56 0  2
      .6  5.588671         . 0 0 3.8728664 2 1 58 2 3.0445225   72 0  2
      .6  5.588671         . 0 0 3.8728664 2 1 57 2 3.0445225   54 0  2
      .6  5.588671         . 0 0 3.8728664 2 1 54 2 3.0445225   75 1  2
      .6  5.588671         . 0 0 3.8728664 2 2 35 2 3.0445225   54 0  2
       1  6.269854  39.35821 1 0  7.760126 2 2 38 1  3.713572   81 0  4
       1  6.269854  39.35821 1 0  7.760126 2 1 43 1  3.713572   73 0  4
       1  6.269854  39.35821 1 0  7.760126 2 1 43 1  3.713572   80 0  4
       1  6.269854  39.35821 1 0  7.760126 2 2 43 1  3.713572   82 0  4
       1  6.269854  39.35821 1 0  7.760126 2 1 33 1  3.713572   78 0  4
     .95 4.5217886 .29449838 0 0  5.648974 2 1 60 5 3.0445225   47 0 11
     .95 4.5217886 .29449838 0 0  5.648974 2 1 47 5 3.0445225   55 0 11
     .95 4.5217886 .29449838 0 0  5.648974 2 2 34 5 3.0445225   47 0 11
     .95 4.5217886 .29449838 0 0  5.648974 2 1 53 5 3.0445225   52 0 11
     .95 4.5217886 .29449838 0 0  5.648974 2 2 39 5 3.0445225   47 0 11
.8333333   5.56452        52 0 0  5.080659 2 2 38 3  2.944439   65 0 13
.8333333   5.56452        52 0 0  5.080659 2 2 41 3  2.944439   80 0 13
.8333333   5.56452        52 0 0  5.080659 2 1 48 3  2.944439   64 0 13
.8333333   5.56452        52 0 0  5.080659 2 1 54 3  2.944439   76 0 13
.8333333   5.56452        52 0 0  5.080659 2 1 40 3  2.944439   76 0 13
       1  4.574711 2.1333334 0 0 3.8728664 2 1 54 1 2.1972246   85 0 24
       1  4.574711 2.1333334 0 0 3.8728664 2 2 42 1 2.1972246   68 0 24
       1  4.574711 2.1333334 0 0 3.8728664 2 1 55 1 2.1972246   83 0 24
       1  4.574711 2.1333334 0 0 3.8728664 2 2 38 1 2.1972246   72 0 24
       1  4.574711 2.1333334 0 0 3.8728664 2 1 54 1 2.1972246   72 0 24
.7735849  6.294971  .4524722 0 0  8.326396 2 1 60 1  3.988984   81 0 25
.7735849  6.294971  .4524722 0 0  8.326396 2 1 37 1  3.988984   65 0 25
.7735849  6.294971  .4524722 0 0  8.326396 2 1 48 1  3.988984   57 0 25
.7735849  6.294971  .4524722 0 0  8.326396 2 2 46 1  3.988984   75 0 25
.7735849  6.294971  .4524722 0 0  8.326396 2 2 36 1  3.988984   73 0 25
       1  4.348082 .18771826 0 0  5.559103 2 2 34 3  2.833213   62 0 29
       1  4.348082 .18771826 0 0  5.559103 2 1 41 3  2.833213   62 0 29
       1  4.348082 .18771826 0 0  5.559103 2 1 37 3  2.833213   59 0 29
       1  4.348082 .18771826 0 0  5.559103 2 2 54 3  2.833213   65 0 29
       1  4.348082 .18771826 0 0  5.559103 2 1 43 3  2.833213   47 0 29
.7619048 4.1311584  10.36379 0 0   7.27337 2 2 33 3 3.0910425   70 0 30
.7619048 4.1311584  10.36379 0 0   7.27337 2 1 57 3 3.0910425   87 0 30
.7619048 4.1311584  10.36379 0 0   7.27337 2 1 55 3 3.0910425   87 0 30
.7619048 4.1311584  10.36379 0 0   7.27337 2 2 55 3 3.0910425   62 0 30
.7619048 4.1311584  10.36379 0 0   7.27337 2 1 60 3 3.0910425   80 0 30
       1  3.851211 1.0779494 0 0  7.251749 1 2 57 1  2.772589   69 0 31
       1  3.851211 1.0779494 0 0  7.251749 1 2 39 1  2.772589   72 1 31
       1  3.851211 1.0779494 0 0  7.251749 1 1 53 1  2.772589   66 0 31
       1  3.851211 1.0779494 0 0  7.251749 1 1 60 1  2.772589   63 0 31
       1  3.851211 1.0779494 0 0  7.251749 1 1 49 1  2.772589   74 0 31
.7027027   6.02369 .40595105 0 1  7.302857 2 2 41 2  3.637586   70 0 37
.7027027   6.02369 .40595105 0 1  7.302857 2 1 47 2  3.637586   65 0 37
.7027027   6.02369 .40595105 0 1  7.302857 2 1 41 2  3.637586   81 0 37
.7027027   6.02369 .40595105 0 1  7.302857 2 1 60 2  3.637586   74 0 37
.7027027   6.02369 .40595105 0 1  7.302857 2 2 41 2  3.637586   74 0 37
.6969697  4.927254 .32009345 0 1  7.199678 2 1 57 3 3.5263605   82 0 38
.6969697  4.927254 .32009345 0 1  7.199678 2 2 33 3 3.5263605   77 0 38
.6969697  4.927254 .32009345 0 1  7.199678 2 1 55 3 3.5263605   84 0 38
.6969697  4.927254 .32009345 0 1  7.199678 2 1 60 3 3.5263605   85 0 38
.6969697  4.927254 .32009345 0 1  7.199678 2 2 55 3 3.5263605   75 0 38
       1   5.25942  .7263445 0 1  8.163983 1 1 43 4  3.367296   47 0 39
       1   5.25942  .7263445 0 1  8.163983 1 1 45 4  3.367296   55 1 39
       1   5.25942  .7263445 0 1  8.163983 1 2 38 4  3.367296   53 0 39
       1   5.25942  .7263445 0 1  8.163983 1 1 38 4  3.367296   62 0 39
       1   5.25942  .7263445 0 1  8.163983 1 2 43 4  3.367296   67 0 39
       1  5.204007  2.397351 0 0  8.326396 1 1 44 4 3.0445225   47 1 41
       1  5.204007  2.397351 0 0  8.326396 1 2 45 4 3.0445225   60 1 41
       1  5.204007  2.397351 0 0  8.326396 1 2 43 4 3.0445225   76 0 41
       1  5.204007  2.397351 0 0  8.326396 1 1 57 4 3.0445225   87 0 41
       1  5.204007  2.397351 0 0  8.326396 1 1 42 4 3.0445225   47 0 41
     .72  6.396729 1.7342253 1 0  6.976133 4 2 44 3 3.2580965   81 0 42
     .72  6.396729 1.7342253 1 0  6.976133 4 2 39 3 3.2580965   87 0 42
     .72  6.396729 1.7342253 1 0  6.976133 4 1 33 3 3.2580965   54 0 42
     .72  6.396729 1.7342253 1 0  6.976133 4 1 44 3 3.2580965   80 0 42
     .72  6.396729 1.7342253 1 0  6.976133 4 1 54 3 3.2580965   78 0 42
       1  5.187944 18.360825 0 0  6.264597 4 2 33 4 2.6390574   76 0 44
       1  5.187944 18.360825 0 0  6.264597 4 1 56 4 2.6390574   75 0 44
       1  5.187944 18.360825 0 0  6.264597 4 1 51 4 2.6390574   67 0 44
       1  5.187944 18.360825 0 0  6.264597 4 2 33 4 2.6390574   70 0 44
       1  5.187944 18.360825 0 0  6.264597 4 1 48 4 2.6390574   77 0 44
.4347826  5.617498 .53189015 0 0   6.13027 4 2 47 2  3.178054   72 0 45
.4347826  5.617498 .53189015 0 0   6.13027 4 2 46 2  3.178054   68 0 45
.4347826  5.617498 .53189015 0 0   6.13027 4 1 60 2  3.178054   73 0 45
.4347826  5.617498 .53189015 0 0   6.13027 4 1 60 2  3.178054   87 0 45
.4347826  5.617498 .53189015 0 0   6.13027 4 1 46 2  3.178054   84 0 45
     .76  5.056246     .1625 0 0  7.036149 2 2 47 5 3.2580965   55 1 46
     .76  5.056246     .1625 0 0  7.036149 2 1 34 5 3.2580965   49 1 46
     .76  5.056246     .1625 0 0  7.036149 2 1 59 5 3.2580965   47 1 46
     .76  5.056246     .1625 0 0  7.036149 2 2 44 5 3.2580965   68 0 46
     .76  5.056246     .1625 0 0  7.036149 2 1 58 5 3.2580965   47 0 46
       1  4.684628  .8477162 0 0  6.322296 2 1 35 2  2.772589   58 0 50
       1  4.684628  .8477162 0 0  6.322296 2 2 33 2  2.772589   72 0 50
       1  4.684628  .8477162 0 0  6.322296 2 1 51 2  2.772589   78 0 50
       1  4.684628  .8477162 0 0  6.322296 2 1 49 2  2.772589   77 0 50
       1  4.684628  .8477162 0 0  6.322296 2 2 43 2  2.772589   64 0 50
       1  5.966865  .4755262 1 0  6.678329 4 2 43 5  3.178054   80 0 51
       1  5.966865  .4755262 1 0  6.678329 4 1 43 5  3.178054   75 0 51
       1  5.966865  .4755262 1 0  6.678329 4 2 38 5  3.178054   72 0 51
       1  5.966865  .4755262 1 0  6.678329 4 1 43 5  3.178054   77 0 51
       1  5.966865  .4755262 1 0  6.678329 4 1 33 5  3.178054   62 0 51
end

cap drop pvalue IV
cap drop pvalueMin numOfIV
generate pvalue = .
generate IV = ""
tuples  c1 c2 c3 c4 c5 c6 c7 c8 c9 c10 c11
quietly forvalues i = 1/`ntuples' {
     replace IV = "`tuple`i''" in `i'
     reg y   i.x   `tuple`i'' ,vce(cluster id )
     replace pvalue = el(r(table),4,1) in `i'
     }
dis `ntuples'

[Mike Lacy]

After your -tuples- command (-ssc describe tuples-), -di `ntuples'- indicates that its value is 2047. Your data set has _N = 100. When i reaches 101, your command replace ... in `i' refers to an observation number greater than _N. Stata sees that you are trying to store data in an observation that does not exist. (By the way: When you are experiencing problems, using -quietly- is not a good idea, as it hides the problems from you.)

I don't know what you are trying to accomplish here. It appears that you want to create a data set containing a large number of regression results. If right, I'd suggest looking at the -post- command, which is one of the facilities in Stata by which you can do that sort of thing. You might also be able to do it by using -set obs- to increase the size of your data set, but that's kind of a hack, so I wouldn't recommend it.

[Fred Lee]

After your -tuples- command (-ssc describe tuples-), -di `ntuples'- indicates that its value is 2047. Your data set has _N = 100. When i reaches 101, your command replace ... in `i' refers to an observation number greater than _N. Stata sees that you are trying to store data in an observation that does not exist. (By the way: When you are experiencing problems, using -quietly- is not a good idea, as it hides the problems from you.)

I don't know what you are trying to accomplish here. It appears that you want to create a data set containing a large number of regression results. If right, I'd suggest looking at the -post- command, which is one of the facilities in Stata by which you can do that sort of thing. You might also be able to do it by using -set obs- to increase the size of your data set, but that's kind of a hack, so I wouldn't recommend it.

Thanks Mike a lot! That will be correct after I move the
replace IV = "`tuple`i''" in `i' after
replace pvalue = el(r(table),4,1) in `i'



Here comes another question, the error code r(3900),indicateing that Mata is out of memory; the operating system refused to supply what Mata requested. There is no Mata or Stata setting that affects this, and so nothing in Mata or Stata to reset in order to get more memory. You must take up the problem with your operating system. Considering that I run the regression on our university's server, the memory is quite large. How can increase the memory setting for Mata? Thanks again!

[daniel klein]

[...] the error code r(3900) [...]
Considering that I run the regression on our university's server, the memory is quite large. How can increase the memory setting for Mata?

This error is most likely produced by tuples (SSC) and has nothing to do with the regressions. You should not be getting this error for only 12 variables (items) even if you were running this on a Netbook. Could you show the tuples command that you are actually using?

My machine (Win7, 8 GB RAM) starts throwing the r(3900) error when there are more than 20-something variables (items) in the list supplied to tuples. Note that this implies running something between 1,000,000 and 10,000,000 (or even 100,000,000) regression models. Is this really what you want?

If you do want that many combinations, read the help for tuples and try the ncr option.

Best
Daniel

[Nick Cox]

tuples  c1 c2 c3 c4 c5 c6 c7 c8 c9 c10 c11

produces 2047 local macros, which is evidently one fewer than 2^11. (The empty tuple in which none of the 11 elements appears is easy enough to fake.)

That shouldn't break anything by itself, but a dataset with at least that many observations is needed for the code in #1 to work.

This is in essence what Mike said in #2. I don't know where the 2 billion observations enter the story. For once I disagree with Daniel Klein: if there is a memory problem here it's not obviously anything to do with a loop over 2047 tuples.

[daniel klein]

For once I disagree with Daniel Klein: if there is a memory problem here it's not obviously anything to do with a loop over 2047 tuples.

I think, this is more of a misunderstanding than disagreement. Nick's statement is the essence of the first paragraph in #4. The loop will not cause the problem. Neither will the tuples command, by itself, with only 2,047 local macros requested.

I was guessing that Fred Lee was actually trying to get (many) more local macros and forgot to mention that the code has changed. I then pointed out how many local macros would usually be needed for tuples to complain.

However, let's assume that code in question is still that shown in #1. The error is still most likely produced by the tuples command (official Stata commands tend not to produce Mata run-time errors). One reason might be that the dataset in memory is (very) large. A (very) large dataset requires a lot of memory. A (very) large dataset, thus, reduces the memory available for tuples.

Best
Daniel

[Nick Cox]

OK. To move forward we need Fred Lee to tell us his real dataset size -- and his real code if it differs from #1.

[Fred Lee]

Thanks all for the discussion! Nick Cox daniel klein My real code right now is and this will not cause any problem.

cap drop pvalue IV
cap drop pvalueMin numOfIV
generate pvalue = .
generate IV = ""
set obs 10000
tuples c1 c2 c3 c4 c5 c6 c7 c8 c9 c10 c11
quietly forvalues i = 1/`ntuples' {
reg y i.x `tuple`i'' ,vce(cluster id )
replace pvalue = el(r(table),4,1) in `i'
replace IV = "`tuple`i''" in `i'
}
dis `ntuples'

I just want to add more variables in command tuple and my machine is Win10, 16 GB RAM. I dont know what is the largest number of variables I can include in the tuple command.
Thanks again.

[Nick Cox]

This is just a calculation in powers of 2: if you have 10000 observations, then 2^12 - 1 = 4095 tuples and 2^13 - 1 = 8191 tuples will map to observations in your dataset, but nothing more.

[Fred Lee]

This is just a calculation in powers of 2: if you have 10000 observations, then 2^12 - 1 = 4095 tuples and 2^13 - 1 = 8191 tuples will map to observations in your dataset, but nothing more.

I mean what is the largest number of variables for RAM of 16G, you know, I can set obs 100000000000000,but the machine can not set that much?

[Nick Cox]

help limits