Using Levelsof and tokenize together

[Giles German]

I have some code where I am trying to use two macros at the same time. Previously I have stored one of these macros in a foreach and the other in a tokenize option with a macro shift as locals as follows: -

local r 3
keep if site == `r'
keep if TotalDebttoCapital < 0.1
local stock "EUR GOLD HAM"
local stock1 "EUR.PA GOLD.L HAM.F"
tokenize `stock1'
local s
foreach s of local stock {

I am now using the same piece of code using levelsof to store the contents of a variable in two different macro but the tokenize option does not work.

Can you help me.

Data set as follows: -

* Example generated by -dataex-. To install: ssc install dataex
clear
input str9 StockCode str10 exch double(WeekPriceChangePercent NetIncomeYOY TotalDebttoCapital)
"ALMIL"  "ALMIL.PA"       0   .0934 .2846
"1JB"    "1JB.F"      -.176   .0717     .
"2GBX"   "2GBX.DE"    .0322  -.3527 .1104
"36L"    "36L.BE"    -.1926  -.2221 .3222
"III"    "III.L"      .3249    .928 .0834
"DDDD"   "DDDD.L"    -.5089  -.3069     0
"FOUR"   "FOUR.L"     .0659   .0786     0
"EMB"    "EMB.ST"     .4341   .9394 .3278
"HRP"    ""          -.0049   -.144 .6935
"888"    "888.L"      .2855   .7458     0
"1GX"    "1GX.MU"     -.029 20.7962 .4505
"MKLW"   "MKLW.L"     .0967   .1746 .2213
"OTT"    "OTT.F"      .0732   .1746 .2213
"BAG"    "BAG.L"      .2911   .0379  .044
"A2A"    "A2A.MI"     .1956  2.1507 .5634
"AA."    "AA..L"      -.422      75  3.06
"A2X"    "A2X.F"      .7186   .2957     .
"AALB"   ""           .4597    .102 .3734
"CFEB"   "CFEB.BR"    .1269  -.0374 .3009
"AZDA"   "AZDA.BE"    .1162   .0263 .1915
"ABDP"   "ABDP.L"      .609   .0073     0
"AB"     "AB.PA"       -.32  -.0367 .4686
"A8D"    "A8D.F"     -.3438  -.0367 .4686
"AXS"    "AXS.BE"     .0984   .0549 .0008
"ABBY"   "ABBY.L"     .1193   .0478     0
"ABCA"   "ABCA.PA"   -.0238   .2599     0
"ABC"    "ABC.L"      .3958   .1317     0
"ABEO"   "ABEO.PA"    .9073 17.8574  .391
"AUC"    "AUC.MU"     .3949  1.9454 .7847
"DAB"    "DAB.BE"    -.0241  -.0851     0
"ABLX"   "ABLX.BR"    1.978   .9801 .5183
"AB4"    "AB4.BE"    -.1401   .2536 .4613
"ABN"    ""           .2851  -.0765     .
"AB2"    "AB2.MU"     .2728  -.0765     .
"33A"    "33A.BE"    -.5283  1.4816     .
"ACA"    "ACA.L"     -.5378  1.4816     .
"ACAN"   "ACAN.PA"    .3111  -.0925 .1071
"ADX"    "ADX.F"      .2941  -.0925 .1071
"1A8"    "1A8.BE"     .0644  -.4588     0
"ACCEL"  ""           .1563   .0002 .3727
"AO1A"   "AO1A.ST"    .1561   .0002 .3727
"E7SX"   "E7SX.DE"   -.0617 24.7257 .4968
"ACSO"   "ACSO.L"     .4085   .4023 .3112
"LQG"    "LQG.BE"     .4215   .4023 .3112
"AJ3"    "AJ3.BE"    -.0088   .6977 .6397
"AC"     "AC.PA"      .1999   .0619 .3624
"ACE"    "ACE.MI"     .3468   .4992 .6367
"ACE.N"  "ACE.N.BE"  -.0121   .8727 .3944
"AZJ"    "AZJ.BE"    -.5226  -.6728 .4476
"ACKB"   "ACKB.BR"    .1629  -.2138 .6282
"B3K"    "B3K.BE"      .162  -.2138 .6282
"ACS"    "ACS.MI"     .4419   -.139 .3496
"ATI"    "ATI.PA"    -.0535   .3678 .5083
"AIY"    "AIY.BE"     .5936   .0919 .3057
"ACT"    "ACT.L"      .1178 -1.5483     0
"CY3B"   "CY3B.F"     .1963  -.5334 .1012
"AR7B"   "AR7B.F"     .1957  -.1377     0
"A41"    "A41.F"     -.0478   .0506 .2026
"ADES"   "ADES.L"         .   .4833 .4409
"ADN1"   "ADN1.F"     .0594  1.6381 .4394
"ADSX.N" "ADSX.N.DE"  .1545   .4941 .2418
"ADLX"   "ADLX.DE"    .0416   .6771 .6986
"ADM"    "ADM.L"       .074  -.2593 .2623
"ADJX"   "ADJX.DE"     .352  1.6665 .3685
"ADOC"   "ADOC.PA"   -.6599 -1.6287 .1205
"ADVX"   "ADVX.DE"   -.1074   -.198 .3122
"6VAA"   "6VAA.BE"   1.5098  -.4878 .0483
"OXQ"    "OXQ.ST"    -.1509  4.2335     0
"AMS"    "AMS.L"       .455   .1114     0
"S7Q"    "S7Q.BE"     .1899  -.0497     0
"ADVI"   "ADVI.PA"   -.0169  1.0678 .6332
"2AE"    "2AE.F"      .2476  -.3192 .5147
"AE"     "AE.MI"      .2776  6.6073     .
"AED"    "AED.BR"     .1135   .5735 .4088
"AEF"    "AEF.MI"      .967  1.3922 .3239
"AGN"    "AGN.MI"     .1084  1.6833 .3948
"AENF"   "AENF.DE"    .0957  1.6833 .3948
"ADB"    "ADB.MI"     .4686   .6258 .1521
"ADP"    "ADP.PA"     .6251   .0116 .4982
"W7L"    "W7L.BE"      .623   .0116 .4982
"G5L"    "G5L.BE"    -.1683  -.3875 .7185
"AEWU"   "AEWU.L"     .0619   .3156 .1816
"AFAGR"  "AFAGR.HE"   .1056 -1.3068 .0319
"V5W"    "V5W.F"      .0964 -1.3068 .0319
"AFRK"   "AFRK.L"    -.0333 -1.3068 .0319
"AJAX"   ""           .1429 -1.4388 .0704
"IML"    "IML.PA"     .1388    11.4 .4999
"AFHP"   "AFHP.L"     .8892   .4297 .0985
"SSA"    "SSA.L"      -.529   .8099 .7577
"7AG"    "7AG.F"     -.0192  2.9499 .6621
"FORB"   "FORB.BR"    .1257  -.9648 .3174
"AGS"    "AGS.MI"     .1145  -.9648 .3174
"FO4NX"  "FO4NX.DE"   .1224  -.9648 .3174
"AGFB"   "AGFB.BR"    .1148    .129 .2697
"AGK"    "AGK.L"     -.1658  -.2284 .3588
"AGB1"   "AGB1.F"    -.1277   .3456 .2048
"AGR"    "AGR.VI"    -.1305   .3456 .2048
"G5A"    "G5A.F"     -.4611   .3121 .4797
"AGTA"   "AGTA.PA"    .0494  -.1405  .001
"AM1"    "AM1.HE"     .2791   .9241 .3799
end

local r 3
keep if site == `r'
keep if TotalDebttoCapital < 0.1
levelsof StockCode, local(stock)
levelsof exch, local(stock1)
tokenize `stock1'
local s
foreach s of local stock {

The error message I get is as follows

`"44H"' `"5AM"' `"6AZ"' `"6G4"' `"7T0"' `"7TC"' `"A8U"' `"AFAGR"' `"AFRK"' `"ATYM"' `"AVTX"' `"AYS"' `"BCN"' `"BKY"' `"DALR"' `"DLR"' `"DTI"' `
> "EUR"' `"EUZX"' `"F1D"' `"GOLD"' `"HAM"' `"HGR"' `"JAY"' `"MLZAM"' `"N0T"' `"ND3"' `"NOF"' `"NP7"' `"NR4"' `"NWF"' `"R8G"' `"RRS"' `"RXC"' `"
> S32"' `"S8F"' `"S9Y"' `"SOLV"' `"SSH"' `"SZW"' `"TEN"' `"V5W"' `"VCT"' `"VI9"'
`"44H.BE"' `"5AM.BE"' `"6AZ.F"' `"6G4.F"' `"7T0.F"' `"7TC.BE"' `"A8U.F"' `"AFAGR.HE"' `"AFRK.L"' `"ATYM.L"' `"AYS.BE"' `"BCN.L"' `"BKY.L"' `"DA
> LR.L"' `"DLR.F"' `"DTI.BE"' `"EUR.PA"' `"EUZX.DE"' `"F1D.BE"' `"GOLD.L"' `"HAM.F"' `"HGR.BE"' `"JAY.L"' `"MLZAM.PA"' `"N0T.F"' `"ND3.F"' `"NO
> F.MU"' `"NP7.BE"' `"NR4.BE"' `"NWF.F"' `"R8G.BE"' `"RRS.L"' `"RXC.F"' `"S32.L"' `"S8F.BE"' `"S9Y.BE"' `"SOLV.BR"' `"SSH.F"' `"SZW.ST"' `"TEN.
> MI"' `"V5W.F"' `"VCT.L"' `"VI9.F"'
varlist not allowed
r(101);

Can you help me to get this to work? Thanks in advance.

[Nick Cox]

Where does the error occur? Is it within the foreach loop you are not showing us?

It's possible that your values are being misread as attempts to specify a varlist, as the periods may be misread as syntax associated with time series operators. But that's just a guess as you don't show a reproducible example of your problem.

I stopped careful reading at

keep if site == `r'

as there is no variable in your example that is site. Hence your code is misleading as the data input can't be followed by the code you give without failing. You'll correct me if I am wrong.

So I can't make precise suggestions given that I can't see where your code fails and I can't see what's in the loop.

But Method 1 in https://www.stata.com/support/faqs/d...-with-foreach/ is often much cleaner than using levelsof (and I say that as its nominal author).

[Clyde Schechter]

Well, the first error message I get from the above is

. keep if site == `r'
site not found
r(111);

You didn't include the site variable in your -dataex- example. But I don't think that's relevant to the issue you're raising. The problem is the way -tokenize- is intepreting your quotes. When -tokenize- encounters an opening quote (whether " or `") as the first character, it assumes that the entire thing it is being asked to tokenize is whatever is between that opening quote and the closing quote that balances it. In this case, when you run -tokenize `stock1'-, -tokenize- sees `"44H"', and then wonders what everything else after that is supposed to be, because -tokenize- wants only a single string to parse. Not knowing what to make of all that other stuff, it assumes you have mistakenly given it a varlist. I would agree that the error message is not really informative about the nature of the problem--but writing code for really informative error messages is truly hard.

Anyway, if you change that to -tokenize `"`stock1'"'- it will run just fine.

Added: Crossed with #2.

[Giles German]

Same error unfortunately sIte variable added more of the code below. I've tested it this should give you the error above.

* Example generated by -dataex-. To install: ssc install dataex
clear
input str9 StockCode str6 exchange float site double(WeekPriceChangePercent NetIncomeYOY)
"VNP"      "TOR" 0    .648    .9394
"EMB"      "STU" 3   .4341    .9394
"33A"      "BER" 3  -.5283   1.4816
"ACA"      "LSE" 3  -.5378   1.4816
"ADN"      "TOR" 0   .0946    .1782
"ACE.N"    "BER" 3  -.0121    .8727
"AZJ"      "BER" 3  -.5226   -.6728
"5142"     "TYO" 2   .4438   -.2238
"PADEF"    "PNK" 1       0   1.1951
"4401"     "TYO" 2   .2191    .1558
"ASIX"     "NYQ" 1  1.0472   -.4646
"AFAGR"    "HEX" 3   .1056  -1.3068
"V5W"      "FRA" 3   .0964  -1.3068
"AFRK"     "LSE" 3  -.0333  -1.3068
"AEM"      "NYQ" 1   .0348   5.4607
"AEM"      "TOR" 0  -.0174   5.4607
"4955"     "TYO" 2   .8725   -.1157
"AM1"      "HEX" 3   .2791    .9241
"MUNK1EUR" "STO" 0       0    .9241
"5482"     "TYO" 2   -.101    253.2
"AI"       "PAR" 3    .138    .0523
"APD"      "NYQ" 1   .1667    .0317
"4088"     "TYO" 2   .1658    .1091
"BOS"      "TOR" 0  -.1798    .0407
"AKS"      "NYQ" 1  -.3728    .9847
"AKZA"     "AEX" 3   .2736    -.001
"ASR"      "TOR" 0  -.0924   -.8669
"AGI"      "TOR" 0  -.3051    .9648
"ALB"      "NYQ" 1   .4242      .46
"ALC"      "WSE" 0  -.1394    .9933
"AA"       "NYQ" 1   .7179    .5365
"3036"     "TYO" 2  1.9668   -.3806
"AXR"      "TOR" 0  -.1067    .2087
"AXU"      "ASQ" 1  -.0585    .2087
"ALO"      "TOR" 0   .0535   1.1668
"ATI"      "NYQ" 1   .7532    -.696
"ALS"      "TOR" 0    .223   -.6864
"ALTR"     "LIS" 3   .3082   -.3457
"ACJ"      "MUN" 3   .3442   -.3457
"AJM"      "ASX" 0  1.7879     .812
"AML"      "WSE" 0  -.1396    .2035
"AWC"      "ASX" 0    .452   -1.342
"2600"     "HKG" 0   .6971   1.2615
"ACH"      "NYQ" 1   .6854   1.2615
"AMAG"     "VIE" 3   .6568    .1424
"AM8"      "BER" 3   .6807    .1424
"AVD"      "NYQ" 1   .2238    .9402
"ARG"      "TOR" 0  2.0909    .5552
"AP"       "NYQ" 1       0 -59.1355
"ATM"      "ASX" 0  -.3333    1.045
"347"      "HKG" 0   .4349   1.3518
"AAL"      "LSE" 3   .3493   1.2834
"AAM"      "SWX" 0   .3939   1.2834
"APY"      "TOR" 0   .3604   2.1677
"HGR"      "BER" 3   .1916   2.1677
"AU"       "NYQ" 1   -.069   1.0323
"FG1"      "FRA" 3   .4003  24.4706
"ANTO"     "LSE" 3   .4304  24.4706
"1370"     "HKG" 0  -.2591   1.3644
"1104"     "HKG" 0   .3148  33.5672
"APAM"     "AEX" 3   .0775     .237
"APMSF"    "PNK" 1   .2006     .237
"MT"       "AEX" 3   .3699   1.2239
"ARCH"     "NYQ" 1   .2732   1.4379
"AR"       "TOR" 0  -.0257   1.0214
"A8U"      "FRA" 3  -.1759   1.0214
"5208"     "TYO" 2   .8847   -.2369
"AZ"       "TOR" 0   .4894  -1.0383
"AKE"      "PAR" 3   .1543    .4982
"V1S"      "BER" 3   .1651    .4982
"3313"     "HKG" 0   .2143  -1.0072
"5857"     "TYO" 2   .0309  -1.4231
"A19"      "BER" 3  -.0723  -1.4231
"3407"     "TYO" 2   .4392    .2534
"4216"     "TYO" 2   .9301    .1782
"AKG"      "TOR" 0  -.7871   -.5911
"AKG"      "ASQ" 1   -.773   -.5911
"AOT"      "CVE" 0    -.33  -5.7038
"ASH"      "NYQ" 1  -.3322    .5352
"ATYM"     "LSE" 3   .1742   1.8019
"AGO"      "ASX" 0   -.186   1.3017
"NDAX"     "GER" 3   .4552   1.8796
"AIAGF"    "PNK" 1       0   1.8796
"AUZ"      "ASX" 0 13.3464   -.6379
"AVTX"     "AEX" 3       .  -1.0481
"AVY"      "NYQ" 1   .6547    .1687
"ASO"      "LSE" 3   .2121  -1.6615
"AVZ"      "ASX" 0 18.7059  -2.5228
"3A2"      "BER" 3 19.2727  -2.5228
"AZZVF"    "PNK" 1       .  -2.5228
"AXTA"     "NYQ" 1   .1569   -.5539
"BGLPF"    "ASQ" 1   .1204    1.261
"BTO"      "TOR" 0   .0665    1.261
"BCN"      "CVE" 0   .7561   -.5332
"BCRMF"    "PNK" 1    .881   -.5332
"BCN"      "LSE" 3   .6821   -.5332
"5195"     "TYO" 2   .3104    .1288
"BAPUF"    "PNK" 1       0     2.11
"BGM"      "CVE" 0    .551        .
"ABX"      "NYQ" 1  -.1048   1.2308
end

local r 3
keep if site == `r'
keep if TotalDebttoCapital < 0.1
levelsof StockCode, local(stock4)
levelsof exch, local(stock3)
tokenize "`stock4'"
local s
foreach s of local stock3 {
capture noisily {

save "/Users/gilesgerman/Documents/StoKred/StoKredModels/Model2/Equities/Sector`p'/region`r'/`s'.dta", replace

export excel using "/Users/gilesgerman/Documents/StoKred/StoKredModels/Model2/Equities/Sector`p'/region`r'/`s'.xlsx", sheet("DATA") sheetreplace firstrow(variables)

}
macro shift
}

[Clyde Schechter]

Giles, you did not do what I suggested, so you are still getting an error message. It has to be

tokenize `"`stock4'"'

with compound double quotes. Read -help quotes- for more information.

By the way, while you did get site into your example, this time you omitted TotalDebttoCapital, so your -keep if TotalDebttoCapital < 0.1- command threw an error message. When the code you post breaks on an error earlier than the one you are trying to figure out, most responders will simply shrug and move on, and you end up with no response, or an off-point response.


When you are posting code for troubleshooting it is important that you post exactly what you ran and show exactly what you got. Do not "simplify" it or change "unimportant details." There are no unimportant details in code. The way to assure that your example will illustrate the problem you want help with is to put the example into your do-file editor and then run it. Look at the output in the Results window and make sure that it is showing the problem you want help with, not some extraneous problem. When you have that right, then copy/paste from the do-file into the Forum editor.

[Giles German]

Yes Clyde this works perfectly. Apologies for the TotalDebttoCapital issue I only filtered by that to reduce the data set I didn't think it would affect the error. You are right I need to be more careful in future.