"variable e not found" message from -describe-

[Brian Holtemeyer]

I don't understand why I'm getting the message

variable e not found
r(111);

when I run the command

des    `k' , clear replace

I have some code wherein I'm doing various data-munging, and I hit the error when I call -describe- with the "clear replace" option. It's baffling to me because the error only occurs when certain variables are described, and the error can be avoided by saving the data set. I'm not sure how I share the data in a way that makes this replicable, but perhaps someone will have an idea based on my code & output below. In my example I loop over 2 variables to show that the variable plays some (??) role, and I loop over `z' to control whether the data are saved, because that also seems to play some role. I also kept just 5 observations to make this even simpler. I'd appreciate any thoughts.

If helpful I can post some -trace- output.

...
* data munging here
...

tempfile TF
des *
list * , sep(0)
assert x==0
qui forv z = 1/3 {
    noi di _newline(1)
    foreach k in ID x {
        preserve
            if `z'==1     sa "my_data", replace                    /* when `z'==1 I save a copy to disk     */
            if `z'==2     sa         `TF', replace                    /* when `z'==2 I save a tempfile         */
            if `z'==3     noi di "on iteration z=3, no saving: "     /* when `z'==3 no saving                 */
            noi di      "`z'; `k'; `c(filename)'"
            keep      `k'
            noi des    `k' , clear replace
        restore
    }
}

Here's the output:

. tempfile TF

. des *

Variable      Storage   Display    Value
    name         type    format    label      Variable label
-----------------------------------------------------------------------------------------------------------------------------------
ID              str6    %9s                   Household ID
x               float   %9.0g                 daily nexus yarn per AE

. list * , sep(0)

     +------------+
     |     ID   x |
     |------------|
  1. | 100001   0 |
  2. | 100002   0 |
  3. | 100003   0 |
  4. | 100004   0 |
  5. | 100005   0 |
     +------------+

. assert x==0

. qui forv z = 1/3 {


1; ID; my_data.dta
1; x; my_data.dta


2; ID; C:\Users\BHOLTE~1\AppData\Local\Temp\ST_6a70_000001.tmp
2; x; C:\Users\BHOLTE~1\AppData\Local\Temp\ST_6a70_000001.tmp


on iteration z=3, no saving:
3; ID; work\my_original_data.dta
on iteration z=3, no saving:
3; x; work\my_original_data.dta
variable e not found
r(111);

end of do-file

r(111);

[William Lisowski]

I cannot reproduce your error. (Note that I changed the location of the saved dataset to write it somewhere I don't care about.)

* Example generated by -dataex-. For more info, type help dataex
clear
input long ID byte x
100001 0
100002 0
100003 0
100004 0
100005 0
end
save "~/Downloads/my_original_data"

use "~/Downloads/my_original_data", clear
tempfile TF
des *
list * , sep(0)
assert x==0
qui forv z = 1/3 {
    noi di _newline(1)
    foreach k in ID x {
        preserve
            if `z'==1     sa "~/Downloads/my_data", replace           /* when `z'==1 I save a copy to disk     */
            if `z'==2     sa         `TF', replace                    /* when `z'==2 I save a tempfile         */
            if `z'==3     noi di "on iteration z=3, no saving: "     /* when `z'==3 no saving                 */
            noi di      "`z'; `k'; `c(filename)'"
            keep      `k'
            noi des    `k' , clear replace
        restore
    }
}

. use "~/Downloads/my_original_data", clear

. tempfile TF

. des *

Variable      Storage   Display    Value
    name         type    format    label      Variable label
------------------------------------------------------------------------------------------------
ID              long    %12.0g                
x               byte    %8.0g                

. list * , sep(0)

     +------------+
     |     ID   x |
     |------------|
  1. | 100001   0 |
  2. | 100002   0 |
  3. | 100003   0 |
  4. | 100004   0 |
  5. | 100005   0 |
     +------------+

. assert x==0

. qui forv z = 1/3 {


1; ID; ~/Downloads/my_data.dta
1; x; ~/Downloads/my_data.dta


2; ID; /var/folders/xr/lm5ccr996k7dspxs35yqzyt80000gp/T//S_71034.000001
2; x; /var/folders/xr/lm5ccr996k7dspxs35yqzyt80000gp/T//S_71034.000001


on iteration z=3, no saving:
3; ID; ~/Downloads/my_original_data.dta
on iteration z=3, no saving:
3; x; ~/Downloads/my_original_data.dta

.

. about

Stata/SE 17.0 for Mac (Apple Silicon)
Revision 13 Oct 2022

[Brian Holtemeyer]

Yeah, I had also noticed that when I constructed the data set using -dataex- I wasn't hitting the same error. This makes me think the error has something to do with some metadata from one of the data sets I used earlier. Here's an excerpt from the trace output:

        -------------------------------------------------------------------------------------------------------- begin describe ---
        - version 9
        - local version : di "version " string(_caller()) ":"
        - syntax [anything] [using] [, SImple REPLACE *]
        - if ("`replace'"!="") {
        = if (""!="") {
          describe_mk `0'
          return add
          exit
          }
        - local varlist `"`anything'"'
        = local varlist `"(e)"'
        - if ("`simple'" == "") {
        = if ("" == "") {
        - `version' _describe `0'
        = version 11: _describe (e), varlist
variable e not found
          return add
          }
        ---------------------------------------------------------------------------------------------------------- end describe ---

my version info:

Stata/MP 17.0 for Windows (64-bit x86-64)
Revision 14 Jun 2021

[Brian Holtemeyer]

I've been working on creating a minimal working example. I'm able to replicate the issue my merging 2 data sets (ie error on line 10). But, as before, not replicable when using -dataex- (ie no error on line 16).

Interestingly, no error on line 10 if I "comment out" lines 2 or 7. Or if I just run the -describe- command on "data2" set without the merge (line 19).

u  "work\data1", clear                                 /* 1 */
sepov     x , p(pl)                                     /* 2 */
keep hid                                              /* 3 */
list *                                              /* 4 */
mer 1:1 hid using "work\data2", assert(3)              /* 5 */
list                                                  /* 6 */
keep x?                                             /* 7 */
list *                                              /* 8 */
dataex                                                /* 9 */
des * , clear replace                                  /* 10*/

clear                                                /* 11*/
input double x1 float x2 double x3                    /* 12*/
.5 .75 .95                                            /* 13*/
.5 .75 .95                                            /* 14*/
end                                                    /* 15*/
des * , clear replace                                  /* 16*/                    

u  "work\data2", clear                              /* 17*/                    
keep x?                                              /* 18*/                    
des * , clear replace                               /* 19*/

output from lines 1-10:

.
.
. u  "work\data1", clear

. keep x

.
. des * , clear replace

.
.
end of do-file

. do "C:\Users\BHOLTE~1\AppData\Local\Temp\statacmd.tmp"

.
.
. u  "work\data1", clear                                                          /* 1 */

. sepov   x , p(pl)                                                                       /* 2 */


Poverty measures for the variable x: some label

Survey mean estimation

pweight:  <none>                                  Number of obs    =         2
Strata:   <one>                                   Number of strata =         1
PSU:      <observations>                          Number of PSUs   =         2
                                                  Population size  =         2

------------------------------------------------------------------------------
    Mean |   Estimate    Std. err.   [95% conf. interval]        Deff
---------+--------------------------------------------------------------------
      p0 |          0           0           0           0           .
      p1 |          0           0           0           0           .
      p2 |          0           0           0           0           .
------------------------------------------------------------------------------

. keep hid                                                                                        /* 3 */

. list *                                                                                          /* 4 */

     +--------+
     |    hid |
     |--------|
  1. | 100001 |
  2. | 100002 |
     +--------+

. mer 1:1 hid using "work\data2", assert(3)                       /* 5 */

    Result                      Number of obs
    -----------------------------------------
    Not matched                             0
    Matched                                 2  (_merge==3)
    -----------------------------------------

. list                                                                                            /* 6 */

     +---------------------------------------+
     |    hid   x1    x2    x3        _merge |
     |---------------------------------------|
  1. | 100001   .5   .75   .95   Matched (3) |
  2. | 100002   .5   .75   .95   Matched (3) |
     +---------------------------------------+

. keep x?                                                                                         /* 7 */

. list *                                                                                          /* 8 */

     +----------------+
     | x1    x2    x3 |
     |----------------|
  1. | .5   .75   .95 |
  2. | .5   .75   .95 |
     +----------------+

. dataex                                                                                          /* 9 */

----------------------- copy starting from the next line -----------------------


	
	
* Example generated by -dataex-. For more info, type help dataex
clear
input double x1 float x2 double x3
.5 .75 .95
.5 .75 .95
end
------------------ copy up to and including the previous line ------------------

Listed 2 out of 2 observations

. des * , clear replace                                                           /* 10*/
variable e not found
r(111);

end of do-file

output from lines 11-16:

. clear                                                                                           /* 11*/

. input double x1 float x2 double x3                                      /* 12*/

             x1         x2          x3
  1. .5 .75 .95                                                                                      /* 13*/
  2. .5 .75 .95                                                                                      /* 14*/
  3. end                                                                                                     /* 15*/

. des * , clear replace                                                           /* 16*/                                 

.
.
.
end of do-file

output from lines 17-19:

. u  "work\data2", clear                                                          /* 17*/                                  

. keep x?                                                                                         /* 18*/                         

. des * , clear replace                                                           /* 19*/                                 

.
.
end of do-file

[William Lisowski]

Interestingly, no error on line 10 if I "comment out" lines 2 or 7

I'm inclined to believe the problem lies in the use of the seppov command in line 2. It is a community contributed command described by

net describe sg117, from(http://www.stata.com/stb/stb51)

I believe that if you replace the use command on line 1 with the output of

u  "work\data1", clear
dataex

your code lines 2 through 10 (including lines 2 and 7) will again fail at line 10.

In any event, if you cannot resolve this problem on your own, you will have to post the dataex results of

u  "work\data1", clear 
dataex
u  "work\data2", clear 
dataex

so that the readers here have some chance of replicating your results.

[Brian Holtemeyer]

Thanks William. I had been thinking this issue was related to metadata, but I think I'm wrong about that. It is - as you pointed out - more likely to do with -sepov-.

Below I show that there are several things that - if done - cause the error to not occur.

When I examine the -trace- output, it seems like the issue is related to a varlist being set to (e). In the screen shot I've compared the -trace- output on iterations 3 (left side) and 4 (right side) of the loop below.

clear
input double hid float(x1 )
    101  5
    102  5
end
sa "work/data2", replace

clear
input double hid float(x1 pl)
    101  2 3
    102  4 3
end
sa "work/data1", replace


qui foreach k in 1 2 3 4 {
    noi di "`k'"
    use "work/data1", clear
    if `k'!=1 sepov     x1 , p(pl)         
    if `k'!=2 keep hid
    mer 1:1 hid using "work/data2", assert(3)  
    if `k'!=3 keep x1
    des * , clear replace                              
}


*output:

. qui foreach k in 1 2 3 4 {
1
2
3
4
variable e not found
r(111);

end of do-file

r(111);

[William Lisowski]

Below is a minimal reproducible example based on your code, and a workaround to the problem - save the working dataset before describe, replace and remove the clear option, which is not required only when the working dataset has been changed since it was last saved.

While I continue to suspect that the use of sepov is at the root of the problem, I think this example - where the describe, replace clear command is given a single variable name known to exist and yet it complains about a different variable name - points out a problem in the innards of the describe command, The problem may perhaps be triggered by something done in sepov - because sepov is written using version control to Stata Release 5 syntax, and much has changed over the intervening twelve releases, perhaps breaking some aspect of backward compatibility - but the error message from describe is clearly unhelpful.

I'd suggest you follow this up with Stata Technical Services at https://www.stata.com/support/tech-support/ so that the describe, replace clear command can perhaps be improved.

I am tempted to see if taking the code from seppov.ado and updating it entirely to Release 17 standards would resolve the problem. This would include changes to the argument list parsing, and updating the survey-related command listed at the bottom of help seppov to their contemporary versions. But with age and experience I've improved my ability to resist (some but not all) temptation.

// net install sg117, from(http://www.stata.com/stb/stb51)

clear all

tempfile data1 data2
input double hid float(x1 )
    101  5
    102  5
end
save "`data2'", replace
clear
input double hid float(x1 pl)
    101  2 3
    102  4 3
end
save "`data1'", replace

use "`data1'", clear
// all of the following four commands are necessary
// to cause the the error
quietly sepov x1 , p(pl)        
keep hid
merge 1:1 hid using "`data2'", assert(3)  
keep x1
//
describe x1
capture noisily describe x1, replace clear
tempfile foo
save `foo'
describe x1, replace
list, clean abbreviate(12)

. use "`data1'", clear

. // all of the following four commands are necessary
. // to cause the the error
. quietly sepov x1 , p(pl)        

. keep hid

. merge 1:1 hid using "`data2'", assert(3)  

    Result                      Number of obs
    -----------------------------------------
    Not matched                             0
    Matched                                 2  (_merge==3)
    -----------------------------------------

. keep x1

. //
. describe x1

Variable      Storage   Display    Value
    name         type    format    label      Variable label
------------------------------------------------------------------------------------------------
x1              float   %9.0g                

. capture noisily describe x1, replace clear
variable e not found

. tempfile foo

. save `foo'
file /var/folders/xr/lm5ccr996k7dspxs35yqzyt80000gp/T//S_80855.000003 saved as .dta format

. describe x1, replace

. list, clean abbreviate(12)

       position   name    type   isnumeric   format   vallab   varlab  
  1.          1     x1   float           1    %9.0g                    

.

[Brian Holtemeyer]

Yes, the fact that saving the data to disk or tempfile avoids the error is one of many interesting wrinkles to this bug. I've submitted this to technical support. Thanks for your help.