Generating new variables using macros and loops

[gi peters]

Hello, I am trying to generate new variables with the use of macros and having issues with the syntax. Would anyone be able to help me with this please? Thank you!

Issue 1: I have 500 variables named bsw1-bsw500. Now I want to generate new variables based from these 500 variables by multiplying each of the variables with a variable called "product1" .
My desired output would be 500 new variables named p1bsw1 p1bsw2 p1bsw3...p1bsw500. This is the code that I am using but it returns an "invalid syntax" error.

gen product1=123456*54321
local bswtemp “bsw1-bsw500”
foreach x of local bswtemp{
gen p1`x'=`x'*product1
}

issue 2 : Again using the 500 bsw1-bsw500 variables, I want to create new variables where each of the bsw variables would have a suffix attached to it. The suffix will come from the categories of the variable wave. Wave is composed of years 2003 2004 2005....So my desired output is that if bsw1 is found in wave==2003, the generated output should be bsw1_2003. If bsw2 is found in both 2003 and 2004, there should be two variables generated named bsw2_2003 and bsw_2004

local bswtemp “bsw1-bsw500”
levelsof wave, local(levels)
foreach x of local bswtemp{
gen`x’`levels’=`x’ if `levels’==`levels’
}

This code returns an error that says "bsw1 is already defined"

[alejoforero]

For your first issue you need to use first the unab command to unabbreviate the variable list before plugging it into the foreach.

unab varlistname: bsw1-bsw500

For your second problem you need to use a nested loop.

local bswtemp “bsw1-bsw500”
levelsof wave, local(levels)
foreach level of local levels{
foreach x of local bswtemp{
gen`x'_`level'=`x' if `level'==wave
}
}

[gi peters]

Thank you alejo! Do I need to use unab again for the second issue?

[William Lisowski]

Instead of unabbreviating the variable list it can instead be plugged into the version of foreach that accepts a variable list (including abbreviated ones). For example

local bswtemp “bsw1-bsw500”
foreach x of varlist `bswtemp' {

or equivalently

foreach x of varlist bsw1-bsw500 {

[Nick Cox]

William Lisowski has solved the main problem here but there is still scope to explain what the problem was. It is tacit in #2 as well as in #1. I will focus on the code for what was labelled "issue 1" in #1 because it is enough to unravel the misunderstanding here.

If I go

local bswtemp bsw1-bsw500

all that I am doing is putting text into a local macro. Informally I think of a local macro as being like a box or bag -- small or large as the case may be. So I put text in that local macro. The local macro has a name. That is like putting some stuff in a box for safe or tidy keeping and labelling it with some suitable description, "family photos" or "stamp collection" (that dates me) or whatever.

I can look inside that box

di "`bswtemp'" 
bsw1-bsw500

and I will see exactly what I put in.

Now when you pass that to foreach the loop machinery sees

foreach x of local bswtemp {

which is exactly equivalent to

foreach x in bsw1-bsw500 {

which you could have typed directly. Use of a local macro did no harm, but was not needed or helpful. That is, the code passes to foreach a list with precisely one item, the text bsw1-bsw500 so that Stata interprets the one statement in the loop in #1 as follows:

gen p1`x' = `x' * product1 

becomes

gen p1bsw1-bsw500 = bsw1-bsw500 * product1

which is not only not what you want, it's illegal as you can't use a hyphen in a new variable name. Hence the "invalid syntax" error.

In a nutshell, there isn't a side-effect of using a local macro with foreach that a varlist inside a local macro is expanded as you wish, which is precisely why foreach offers a syntax that will do that, as used by William.


In passing, let's note another solution:

forval j = 1/500 { 
          gen p1bsw`j' = bsw`j' * p1 
} 

All that said, the big picture here seems to be that every new step implies the creation of many (hundreds!) new variables. That to me smacks of holding data in a wide layout whereas usually you are better off with a long layout.

[gi peters]

Hi Nick Cox , you're right, Im creating 5000 new variables. The reason is that Im rescaling bootstrap weights per cycle for a pooled dataset that includes 10 cycles of survey data and there's 500 bootstrap weights for each cycle. I will try the proposed syntax and see how that goes. Thank you!