Announcement

Collapse
No announcement yet.
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • foreach / forvalues running simultaneously

    Dear all,

    I am trying to write some code to shorten an operation that I am doing to merge data.
    I am repeating this bit of code four times.
    Code:
    use     "C:\OneDrive\OneDrive - University\Year 3\Final stuff\WDI.csv"
    drop if    series!=2
    reshape    long yr, i( countryname countrycode seriesname seriescode country ) j(year)
    rename    yr gdp_growth
    drop    seriescode series seriesname ctry countrycode countryname
    drop if year<1979
    merge m:m country year using "C:\OneDrive\OneDrive - University\Year 3\Final stuff\WDI II.dta", nogenerate
    save "C:\OneDrive\OneDrive - University\Year 3\Final stuff\WDI II.dta", replace
    The bits that vary are
    "rename yr gdp_growth", where "gdp_growth" is either itself, "gdp_exp","inflation", or "population"
    and
    "drop if series!=2" has 2 be one of 2, 3, 4, 5.

    The bit of code I wrote to attempt to do this is the following

    Code:
    foreach x in "gdp_growth" "gdp_exp" "inflation" & forvalues y = 2(1)5 {
        use     "C:\OneDrive\OneDrive - University\Year 3\Final stuff\WDI.csv"
        drop if series!=`y'
        reshape long yr, i( countryname countrycode seriesname seriescode country ) j(year)
        rename yr `x'
        drop seriescode series seriesname ctry countrycode countryname
        drop if year<1979
        merge m:m country year using "C:\OneDrive\OneDrive - University\Year 3\Final stuff\WDI II.dta", nogenerate
        save "C:\OneDrive\OneDrive - University\Year 3\Final stuff\WDI II.dta", replace
    }
    Of course, this does not work, but I was wondering if there was a way to have "y" be one of 2, 3, 4, 5, and "x" be one of "gdp_growth" "gdp_exp" "inflation" "population" within the same loop.

    The only other way I could think of doing that was writing
    Code:
    foreach x in "gdp_growth" "gdp_exp" "inflation" {
        use     "C:\OneDrive\OneDrive - University\Year 3\Final stuff\WDI.csv"
        forvalues y = 2(1)5 {
            drop if series!=`y'
        }
        reshape long yr, i( countryname countrycode seriesname seriescode country ) j(year)
        rename yr `x'
        drop seriescode series seriesname ctry countrycode countryname
        drop if year<1979
        merge m:m country year using "C:\OneDrive\OneDrive - University\Year 3\Final stuff\WDI II.dta", nogenerate
        save "C:\OneDrive\OneDrive - University\Year 3\Final stuff\WDI II.dta", replace
    }
    but of course this doesn't work either, since the forvalues is running inside the foreach, and so is being run nine times rather than the three I want it to be.

    N.B.: I am aware there is a lot of excess wording in my code, though I copied the original reshaping code from the website attached and when trying to remove some of it, the code broke, so I am just sticking with it. What comes to mind is the variables countryname, countrycode, seriesname, seriescode which all feature in the reshape command and then are immediately dropped.

    Thanks for your help!
    How to get data from World Development Indicators and other World Bank sources into Stata

  • #2
    Well, there's no point getting the loops right when what's inside the loops is so very, very wrong.

    First of all, you can't -use- a .csv file. You can bring it in with -import delimited-, but not with -use-.

    Next, you appear to be importing a .csv file in which each row corresponds to a different "series," i.e. a different desired variable, and the yearly values are spread out in columns. But then you drop the variables that identify which series is which, so you end up with a data set that has all these different things mixed together in a way that makes them impossible to separate or distinguish. On top of that you fell into the -merge m:m- trip which creates data salad instead of a meaningful usable data set. Now, it seems you tried to avoid that with a loop over a command -drop if series != `y'-, but since you do that over all values of y from 2 through 5, there cannot be any data left anyhow.

    What it appears to me you really need is a series of two -reshape-s. You take the original data set, and -reshape- it long, along the lines you wrote, and then -reshape- it wide to make the series into variables. If you import this data set into Stata and then run -dataex- to generate an example I can work with, I would be happy to show you code to do this. It is almost surely going to be much shorter, simpler, and clearer than what you are attempting here, and I do not believe any loops are needed at all.

    If you are running version 17, 16 or a fully updated version 15.1 or 14.2, -dataex- is already part of your official Stata installation. If not, run -ssc install dataex- to get it. Either way, run -help dataex- to read the simple instructions for using it. -dataex- will save you time; it is easier and quicker than typing out tables. It includes complete information about aspects of the data that are often critical to answering your question but cannot be seen from tabular displays or screenshots. It also makes it possible for those who want to help you to create a faithful representation of your example to try out their code, which in turn makes it more likely that their answer will actually work in your data.

    When asking for help with code, always show example data. When showing example data, always use -dataex-.

    Comment


    • #3
      Dear Clyde,
      Thank you for your response. Me trying to -use- the .csv files was an oversight that I had not realised was in there - I assumed that it was set to .dta as I was getting no errors.

      With regards to my dataset, the original dataset I have is this
      Code:
      * Example generated by -dataex-. For more info, type help dataex
      clear
      input str3 countrycode str68 seriesname double(yr1960 yr1961 yr1962 yr1963 yr1964) long(series ctry) float country
      "AFG" "GDP growth (annual %)"                                                               .                 .                 .                 .                . 2  3  4
      "AFG" "Inflation, consumer prices (annual %)"                                               .                 .                 .                 .                . 4  3  4
      "AFG" "GDP (constant 2015 US$)"                                                             .                 .                 .                 .                . 1  3  4
      "AFG" "General government final consumption expenditure (constant 2015 US$)"                .                 .                 .                 .                . 3  3  4
      "AFG" "Population, total"                                                             8996967           9169406           9351442           9543200          9744772 5  3  4
      "ALB" "GDP growth (annual %)"                                                               .                 .                 .                 .                . 2  6  5
      "ALB" "Inflation, consumer prices (annual %)"                                               .                 .                 .                 .                . 4  6  5
      "ALB" "GDP (constant 2015 US$)"                                                             .                 .                 .                 .                . 1  6  5
      "ALB" "General government final consumption expenditure (constant 2015 US$)"                .                 .                 .                 .                . 3  6  5
      "ALB" "Population, total"                                                             1608800           1659800           1711319           1762621          1814135 5  6  5
      "DZA" "GDP growth (annual %)"                                                               . -13.6054413259301  -19.685041825216  34.3137287827466 5.83941300668019 2 61  6
      "DZA" "Inflation, consumer prices (annual %)"                                               .                 .                 .                 .                . 4 61  6
      "DZA" "GDP (constant 2015 US$)"                                               23999022855.823  20733849882.3774  16652382861.0539  22366436351.8606  23672504945.322 1 61  6
      "DZA" "General government final consumption expenditure (constant 2015 US$)" 5172653287.50415  3823265603.34789  2586326621.26214  3373469694.78917 3261020719.37949 3 61  6
      "DZA" "Population, total"                                                            11057864          11336336          11619828          11912800         12221675 5 61  6
      "AND" "GDP growth (annual %)"                                                               .                 .                 .                 .                . 2  7  7
      "AND" "Inflation, consumer prices (annual %)"                                               .                 .                 .                 .                . 4  7  7
      "AND" "GDP (constant 2015 US$)"                                                             .                 .                 .                 .                . 1  7  7
      "AND" "General government final consumption expenditure (constant 2015 US$)"                .                 .                 .                 .                . 3  7  7
      "AND" "Population, total"                                                               13410             14378             15379             16407            17466 5  7  7
      "AGO" "GDP growth (annual %)"                                                               .                 .                 .                 .                . 2  5  8
      "AGO" "Inflation, consumer prices (annual %)"                                               .                 .                 .                 .                . 4  5  8
      "AGO" "GDP (constant 2015 US$)"                                                             .                 .                 .                 .                . 1  5  8
      "AGO" "General government final consumption expenditure (constant 2015 US$)"                .                 .                 .                 .                . 3  5  8
      "AGO" "Population, total"                                                             5454938           5531451           5608499           5679409          5734995 5  5  8
      "ATG" "GDP growth (annual %)"                                                               .                 .                 .                 .                . 2 13 10
      "ATG" "Inflation, consumer prices (annual %)"                                               .                 .                 .                 .                . 4 13 10
      "ATG" "GDP (constant 2015 US$)"                                                             .                 .                 .                 .                . 1 13 10
      "ATG" "General government final consumption expenditure (constant 2015 US$)"                .                 .                 .                 .                . 3 13 10
      "ATG" "Population, total"                                                               54132             55005             55849             56701            57641 5 13 10
      "ARG" "GDP growth (annual %)"                                                               .  5.42784287951248 -.852021523482009 -5.30819682651725 10.1302976614338 2 10 11
      "ARG" "Inflation, consumer prices (annual %)"                                               .                 .                 .                 .                . 4 10 11
      "ARG" "GDP (constant 2015 US$)"                                               150797810292.95  158982878501.396  157628310157.913  149261089200.418 164381681829.118 1 10 11
      "ARG" "General government final consumption expenditure (constant 2015 US$)"                .                 .                 .                 .                . 3 10 11
      "ARG" "Population, total"                                                            20481781          20817270          21153042          21488916         21824427 5 10 11
      "ARM" "GDP growth (annual %)"                                                               .                 .                 .                 .                . 2 11 12
      "ARM" "Inflation, consumer prices (annual %)"                                               .                 .                 .                 .                . 4 11 12
      "ARM" "GDP (constant 2015 US$)"                                                             .                 .                 .                 .                . 1 11 12
      "ARM" "General government final consumption expenditure (constant 2015 US$)"                .                 .                 .                 .                . 3 11 12
      "ARM" "Population, total"                                                             1874119           1941498           2009524           2077584          2145004 5 11 12
      "AUS" "GDP growth (annual %)"                                                               .  2.48327050212353  1.29446824362078  6.21494897490116 6.97854037246404 2 14 14
      "AUS" "Inflation, consumer prices (annual %)"                                3.72881355932205   2.2875816993464 -.319488817891384  .641025641025639 2.86624203821656 4 14 14
      "AUS" "GDP (constant 2015 US$)"                                              205048666020.255  210140579058.534  212860782121.407  226089971117.828 241867751030.378 1 14 14
      "AUS" "General government final consumption expenditure (constant 2015 US$)" 31579392269.7312  32738360894.5348  34004370475.3834  35393592667.5068 36494035301.9548 3 14 14
      "AUS" "Population, total"                                                            10276477          10483000          10742000          10950000         11167000 5 14 14
      "AUT" "GDP growth (annual %)"                                                               .  5.53797931684528  2.64867511295262  4.13826758157036 6.12435373762843 2 15 15
      "AUT" "Inflation, consumer prices (annual %)"                                1.94574935198639  3.54223986727562  4.38179863851184   2.7087669949983 3.86856354936625 4 15 15
      "AUT" "GDP (constant 2015 US$)"                                              84896808897.8958   89598376615.323  91971546518.3426  95777575212.1801 101643332719.497 1 15 15
      "AUT" "General government final consumption expenditure (constant 2015 US$)"                .                 .                 .                 .                . 3 15 15
      "AUT" "Population, total"                                                             7047539           7086299           7129864           7175811          7223801 5 15 15
      "AZE" "GDP growth (annual %)"                                                               .                 .                 .                 .                . 2 16 16
      "AZE" "Inflation, consumer prices (annual %)"                                               .                 .                 .                 .                . 4 16 16
      "AZE" "GDP (constant 2015 US$)"                                                             .                 .                 .                 .                . 1 16 16
      "AZE" "General government final consumption expenditure (constant 2015 US$)"                .                 .                 .                 .                . 3 16 16
      "AZE" "Population, total"                                                             3895398           4030325           4171428           4315127          4456691 5 16 16
      "BHS" "GDP growth (annual %)"                                                               .   10.667097429183  10.4519718906977  10.5240552039871 10.5015672018341 2 24 17
      "BHS" "Inflation, consumer prices (annual %)"                                               .                 .                 .                 .                . 4 24 17
      "BHS" "GDP (constant 2015 US$)"                                              2201671025.03956  2436525418.35062  2691190370.18633  2974412730.38912 3286772682.13085 1 24 17
      "BHS" "General government final consumption expenditure (constant 2015 US$)"                .                 .                 .                 .                . 3 24 17
      "BHS" "Population, total"                                                              109532            115119            121092            127340           133705 5 24 17
      "BHR" "GDP growth (annual %)"                                                               .                 .                 .                 .                . 2 23 18
      "BHR" "Inflation, consumer prices (annual %)"                                               .                 .                 .                 .                . 4 23 18
      "BHR" "GDP (constant 2015 US$)"                                                             .                 .                 .                 .                . 1 23 18
      "BHR" "General government final consumption expenditure (constant 2015 US$)"                .                 .                 .                 .                . 3 23 18
      "BHR" "Population, total"                                                              162429            167899            173140            178142           182888 5 23 18
      "BGD" "GDP growth (annual %)"                                                               .  6.05816082548085  5.45303096987189 -.455894289938712 10.9527885460417 2 21 19
      "BGD" "Inflation, consumer prices (annual %)"                                               .                 .                 .                 .                . 4 21 19
      "BGD" "GDP (constant 2015 US$)"                                              22247412282.0175  23595196297.5699  24881849659.0784  24768414727.2515  27481246818.534 1 21 19
      "BGD" "General government final consumption expenditure (constant 2015 US$)"                .                 .                 .                 .                . 3 21 19
      "BGD" "Population, total"                                                            48013505          49362834          50752150          52202008         53741721 5 21 19
      "BRB" "GDP growth (annual %)"                                                               .                 .                 .                 .                . 2 31 20
      "BRB" "Inflation, consumer prices (annual %)"                                               .                 .                 .                 .                . 4 31 20
      "BRB" "GDP (constant 2015 US$)"                                                             .                 .                 .                 .                . 1 31 20
      "BRB" "General government final consumption expenditure (constant 2015 US$)"                .                 .                 .                 .                . 3 31 20
      "BRB" "Population, total"                                                              230985            231718            232623            233632           234588 5 31 20
      "BLR" "GDP growth (annual %)"                                                               .                 .                 .                 .                . 2 26 35
      "BLR" "Inflation, consumer prices (annual %)"                                               .                 .                 .                 .                . 4 26 35
      "BLR" "GDP (constant 2015 US$)"                                                             .                 .                 .                 .                . 1 26 35
      "BLR" "General government final consumption expenditure (constant 2015 US$)"                .                 .                 .                 .                . 3 26 35
      "BLR" "Population, total"                                                             8198000           8271216           8351928           8437232          8524224 5 26 35
      "BEL" "GDP growth (annual %)"                                                               .  4.97842306319197  5.21200350925528  4.35158428167051 6.95668473344102 2 18 21
      "BEL" "Inflation, consumer prices (annual %)"                                .299467254769634  .992676260767344  1.40460703543509  2.14800341332203 4.16876058564369 4 18 21
      "BEL" "GDP (constant 2015 US$)"                                              107366655198.619  112711821523.205   118586365616.34  123746751262.705 132355422615.927 1 18 21
      "BEL" "General government final consumption expenditure (constant 2015 US$)"                .                 .                 .                 .                . 3 18 21
      "BEL" "Population, total"                                                             9153489           9183948           9220578           9289770          9378113 5 18 21
      "BLZ" "GDP growth (annual %)"                                                               .  4.89227721902688  4.89265218028396   4.9564341949162 4.99987991041317 2 27 22
      "BLZ" "Inflation, consumer prices (annual %)"                                               .                 .                 .                 .                . 4 27 22
      "BLZ" "GDP (constant 2015 US$)"                                              110027507.628303  115410358.318665  121056985.731217  127057095.567334 133409797.763359 1 27 22
      "BLZ" "General government final consumption expenditure (constant 2015 US$)"                .                 .                 .                 .                . 3 27 22
      "BLZ" "Population, total"                                                               92068             94700             97392            100165           103069 5 27 22
      "BEN" "GDP growth (annual %)"                                                               .  3.14128045871796 -3.42640982287578  4.73002797546205 6.65075911863318 2 19 23
      "BEN" "Inflation, consumer prices (annual %)"                                               .                 .                 .                 .                . 4 19 23
      "BEN" "GDP (constant 2015 US$)"                                              1651120625.57515  1702986955.13621  1644635642.82313  1722427368.82308 1836981864.11691 1 19 23
      "BEN" "General government final consumption expenditure (constant 2015 US$)" 188031918.649734  194147117.537855  200261893.243473  206376613.886721 189560939.107649 3 19 23
      "BEN" "Population, total"                                                             2431617           2465865           2502897           2542864          2585961 5 19 23
      "BMU" "GDP growth (annual %)"                                                               .  4.67999999999729  4.47000000000328  1.06999999999849 11.1099999999974 2 28 24
      "BMU" "Inflation, consumer prices (annual %)"                                               .                 .                 .                 .                . 4 28 24
      "BMU" "GDP (constant 2015 US$)"                                              1684353374.02034  1763181111.92444  1841995307.62752  1861704657.41911 2068540044.85832 1 28 24
      "BMU" "General government final consumption expenditure (constant 2015 US$)"                .                 .                 .                 .                . 3 28 24
      "BMU" "Population, total"                                                               44400             45500             46600             47700            48900 5 28 24
      end
      label values series series
      label def series 1 "GDP (constant 2015 US$)", modify
      label def series 2 "GDP growth (annual %)", modify
      label def series 3 "General government final consumption expenditure (constant 2015 US$)", modify
      label def series 4 "Inflation, consumer prices (annual %)", modify
      label def series 5 "Population, total", modify
      label values ctry ctry
      label def ctry 3 "AFG", modify
      label def ctry 5 "AGO", modify
      label def ctry 6 "ALB", modify
      label def ctry 7 "AND", modify
      label def ctry 10 "ARG", modify
      label def ctry 11 "ARM", modify
      label def ctry 13 "ATG", modify
      label def ctry 14 "AUS", modify
      label def ctry 15 "AUT", modify
      label def ctry 16 "AZE", modify
      label def ctry 18 "BEL", modify
      label def ctry 19 "BEN", modify
      label def ctry 21 "BGD", modify
      label def ctry 23 "BHR", modify
      label def ctry 24 "BHS", modify
      label def ctry 26 "BLR", modify
      label def ctry 27 "BLZ", modify
      label def ctry 28 "BMU", modify
      label def ctry 31 "BRB", modify
      label def ctry 61 "DZA", modify
      The variable seriesname captures one of five variables - GDP, GDP growth, government final consumption expenditure, inflation, population - it is a download from the UN World Development Indicators. The variable "series" is "seriesname" encoded, "ctry" is "countrycode" encoded, and the variable "country" is the country codes from another dataset which I am ultimately going to merge this with.

      My first bit of code is
      Code:
      drop if    series!=1
      reshape    long yr, i( countryname countrycode seriesname seriescode country ) j(year)
      rename    yr gdp
      drop    seriescode series seriesname ctry countrycode countryname
      drop if year<1979
      save "C:\OneDrive\OneDrive - UniversityYear 3\Final stuff\WDI II.dta", replace
      where I drop all series other than GDP, reshape it to be useful, then save it. It outputs this dataex
      Code:
      * Example generated by -dataex-. For more info, type help dataex
      clear
      input float country int year double gdp
         4 1979                .
         4 1980                .
         4 1981                .
         4 1982                .
         4 1983                .
         4 1984                .
         4 1985                .
         4 1986                .
         4 1987                .
         4 1988                .
         4 1989                .
         4 1990                .
         4 1991                .
         4 1992                .
         4 1993                .
         4 1994                .
         4 1995                .
         4 1996                .
         4 1997                .
         4 1998                .
         4 1999                .
         4 2000                .
         4 2001                .
         4 2002       7228792320
         4 2003       7867259392
         4 2004       7978511360
         4 2005       8874475520
         4 2006       9349916672
         4 2007      10642666496
         4 2008      11060389888
         4 2009      13426265088
         4 2010      15354604544
         4 2011      15420069888
         4 2012      17386481664
         4 2013      18360254464
         4 2014      18860486656
         4 2015      19134212096
         4 2016      19566704640
         4 2017      20084635648
         4 2018      20323487744
         4 2019      21118461952
         4 2020      20621946880
         2 1979                .
         2 1980 332413861028.144
         2 1981  345753887929.57
         2 1982 346888193290.563
         2 1983 347513366631.415
         2 1984 359113027150.361
         2 1985 358471197930.651
         2 1986 366402741716.878
         2 1987 380826813451.636
         2 1988  396781049308.67
         2 1989 407290006698.036
         2 1990 407497014554.954
         2 1991  407108174992.09
         2 1992 397568848057.505
         2 1993 393237664710.067
         2 1994   401306562686.7
         2 1995 419004080516.529
         2 1996 442342732156.009
         2 1997 457494792861.253
         2 1998 465681693043.983
         2 1999 477807465237.001
         2 2000 493083651737.665
         2 2001 510472146755.021
         2 2002     530853298176
         2 2003     546290532352
         2 2004     577191870464
         2 2005     613911691264
         2 2006     654974779392
         2 2007     699888369664
         2 2008     731891040256
         2 2009     738820685824
         2 2010     776887205888
         2 2011     808072904704
         2 2012     824013357056
         2 2013     859514863616
         2 2014     893781606400
         2 2015     919930011648
         2 2016     938506977280
         2 2017     962366668800
         2 2018     986187825152
         2 2019    1006679818240
         2 2020     977091624960
      2004 1979 240584167283.363
      2004 1980  245384642358.41
      2004 1981 228372728441.183
      2004 1982 220721147106.467
      2004 1983 206298786490.683
      2004 1984 207452740169.605
      2004 1985 218969867228.645
      2004 1986 222067762145.339
      2004 1987 225284162241.869
      2004 1988 236197562440.721
      2004 1989 241564236844.108
      2004 1990 257417907712.142
      2004 1991 260303740766.007
      2004 1992 267316217601.015
      2004 1993  264214097653.17
      2004 1994 263422403289.938
      end
      I have confirmed manually that these are identical to the original dataset posted above. It is worth noting that "2004" at the bottom of the first column is not a year, rather is a country code.
      My next bits of code look like this, where "series!=2" is replaced for 3, 4, 5 also.
      Code:
      use     "C:\OneDrive\OneDrive - University\Year 3\Final stuff\WDI.dta"
      drop if    series!=2
      reshape    long yr, i( countryname countrycode seriesname seriescode country ) j(year)
      rename    yr gdp_growth
      drop    seriescode series seriesname ctry countrycode countryname
      drop if year<1979
      merge m:m country year using "C:\OneDrive\OneDrive - University\Year 3\Final stuff\WDI II.dta", nogenerate
      save "C:\OneDrive\OneDrive - University\Year 3\Final stuff\WDI II.dta", replace
      I have exceeded the character limit to posts, so my message continues in the next one.

      Comment


      • #4
        This outputs as below

        Code:
        * Example generated by -dataex-. For more info, type help dataex
        clear
        input float country int year double(gdp_growth gdp)
        2 1979                   .                .
        2 1980                   . 332413861028.144
        2 1981    4.01307781214851  345753887929.57
        2 1982    .328067275768035 346888193290.563
        2 1983    .180223297576504 347513366631.415
        2 1984    3.33790341113666 359113027150.361
        2 1985   -.178726242487656 358471197930.651
        2 1986    2.21260280658906 366402741716.878
        2 1987    3.93667134344298 380826813451.636
        2 1988    4.18936778963459  396781049308.67
        2 1989    2.64855325315459 407290006698.036
        2 1990   .0508256656223978 407497014554.954
        2 1991  -.0954214507040092  407108174992.09
        2 1992   -2.34319218344612 397568848057.505
        2 1993   -1.08941718361486 393237664710.067
        2 1994    2.05191381720317   401306562686.7
        2 1995    4.40997468652051 419004080516.529
        2 1996     5.5700296786392 442342732156.009
        2 1997    3.42541192694441 457494792861.253
        2 1998    1.78950674641077 465681693043.983
        2 1999    2.60387564599263 477807465237.001
        2 2000     3.1971427012105 493083651737.665
        2 2001    3.52647972733993 510472146755.021
        2 2002   3.992607355117798     530853298176
        2 2003   2.908003807067871     546290532352
        2 2004   5.656581401824951     577191870464
        2 2005   6.361804008483887     613911691264
        2 2006   6.688755035400391     654974779392
        2 2007   6.857303619384766     699888369664
        2 2008   4.572539329528809     731891040256
        2 2009   .9468112587928772     738820685824
        2 2010   5.152336597442627     776887205888
        2 2011   4.014183521270752     808072904704
        2 2012   1.972651720046997     824013357056
        2 2013  4.3083696365356445     859514863616
        2 2014  3.9867544174194336     893781606400
        2 2015   2.925590753555298     919930011648
        2 2016    2.01939058303833     938506977280
        2 2017   2.542297840118408     962366668800
        2 2018   2.475271701812744     986187825152
        2 2019   2.077897787094116    1006679818240
        2 2020  -2.939185619354248     977091624960
        4 1979                   .                .
        4 1980                   .                .
        4 1981                   .                .
        4 1982                   .                .
        4 1983                   .                .
        4 1984                   .                .
        4 1985                   .                .
        4 1986                   .                .
        4 1987                   .                .
        4 1988                   .                .
        4 1989                   .                .
        4 1990                   .                .
        4 1991                   .                .
        4 1992                   .                .
        4 1993                   .                .
        4 1994                   .                .
        4 1995                   .                .
        4 1996                   .                .
        4 1997                   .                .
        4 1998                   .                .
        4 1999                   .                .
        4 2000                   .                .
        4 2001                   .                .
        4 2002                   .       7228792320
        4 2003    8.83227825164795       7867259392
        4 2004  1.4141179323196411       7978511360
        4 2005  11.229714393615723       8874475520
        4 2006    5.35740327835083       9349916672
        4 2007  13.826319694519043      10642666496
        4 2008  3.9249837398529053      11060389888
        4 2009  21.390527725219727      13426265088
        4 2010  14.362441062927246      15354604544
        4 2011   .4263547956943512      15420069888
        4 2012  12.752286911010742      17386481664
        4 2013   5.600744724273682      18360254464
        4 2014   2.724543333053589      18860486656
        4 2015  1.4513146877288818      19134212096
        4 2016  2.2603142261505127      19566704640
        4 2017   2.647003173828125      20084635648
        4 2018  1.1892281770706177      20323487744
        4 2019  3.9116034507751465      21118461952
        4 2020 -2.3511006832122803      20621946880
        5 1979                   .                .
        5 1980                   . 4650625160.56601
        5 1981    5.74563529200671 4917833121.09044
        5 1982    2.94859680156867 5062840191.20539
        5 1983    1.10493826182712 5118781449.61318
        5 1984   -1.25159664469854 5054714952.74037
        5 1985    1.78064395977269 5144721429.23007
        5 1986    5.63724317825177  5434741887.0394
        5 1987   -.787842655012852 5391924672.26345
        5 1988   -1.42003965482257 5315357203.75915
        5 1989    9.83654897047623 5838204918.06266
        5 1990   -9.57564016948646 5279159422.75171
        5 1991   -28.0021416560459 3800881722.94428
        5 1992   -7.18711091636945 3527708137.71626
        5 1993    9.55941169434738 3864936281.97555
        5 1994     8.3028665911288 4185836785.30012
        end
        Again, I have confirmed that this is accurate to the original dataset.
        This continues until I arrive at my final dataset of

        Code:
        * Example generated by -dataex-. For more info, type help dataex
        clear
        input str52 countryname float country int year double(population inflation gdp_exp gdp_growth gdp)
        "Africa Eastern and Southern" 2 1979 220740384   15.0698498241015                .                   .                .
        "Africa Eastern and Southern" 2 1980 227305945   15.0665115372509                .                   . 332413861028.144
        "Africa Eastern and Southern" 2 1981 234058404   14.4615910742106                .    4.01307781214851  345753887929.57
        "Africa Eastern and Southern" 2 1982 240999134   12.1399176954734 59507257630.7547    .328067275768035 346888193290.563
        "Africa Eastern and Southern" 2 1983 248146290   11.5675243834069  60879465322.709    .180223297576504 347513366631.415
        "Africa Eastern and Southern" 2 1984 255530063   10.9838625715774 63217587570.6968    3.33790341113666 359113027150.361
        "Africa Eastern and Southern" 2 1985 263161451   13.0065664218827 65751297280.0908   -.178726242487656 358471197930.651
        "Africa Eastern and Southern" 2 1986 271050065   13.8919716455566 67777116748.6101    2.21260280658906 366402741716.878
        "Africa Eastern and Southern" 2 1987 279184536   12.5634432211069 70350015069.2675    3.93667134344298 380826813451.636
        "Africa Eastern and Southern" 2 1988 287524258    12.522253626804 71707148626.8763    4.18936778963459  396781049308.67
        "Africa Eastern and Southern" 2 1989 296024639   12.5582020707012 74297052588.7373    2.64855325315459 407290006698.036
        "Africa Eastern and Southern" 2 1990 304648010   12.4579139789566  75619911507.637   .0508256656223978 407497014554.954
        "Africa Eastern and Southern" 2 1991 313394693   17.6781002638522 78624870650.2006  -.0954214507040092  407108174992.09
        "Africa Eastern and Southern" 2 1992 322270073   16.1676123963339 80694548717.0928   -2.34319218344612 397568848057.505
        "Africa Eastern and Southern" 2 1993 331265579   13.1356607683724 81898738563.8031   -1.08941718361486 393237664710.067
        "Africa Eastern and Southern" 2 1994 340379934    14.852814983397 78263012295.4096    2.05191381720317   401306562686.7
        "Africa Eastern and Southern" 2 1995 349605660   12.2885912601426 75469424042.9245    4.40997468652051 419004080516.529
        "Africa Eastern and Southern" 2 1996 358953595   9.70658632526506 76175354909.2859     5.5700296786392 442342732156.009
        "Africa Eastern and Southern" 2 1997 368440591   10.2495987403067 78713236642.2033    3.42541192694441 457494792861.253
        "Africa Eastern and Southern" 2 1998 378098393   7.49525908567409 77561080831.7551    1.78950674641077 465681693043.983
        "Africa Eastern and Southern" 2 1999 387977990   7.81986477630863 78115148829.6226    2.60387564599263 477807465237.001
        "Africa Eastern and Southern" 2 2000 398113044    8.6014851485149 80433136118.2725     3.1971427012105 493083651737.665
        "Africa Eastern and Southern" 2 2001 408522129   5.84035398311558 83941632353.0068    3.52647972733993 510472146755.021
        "Africa Eastern and Southern" 2 2002 419223712  8.763751983642578      88632262656   3.992607355117798     530853298176
        "Africa Eastern and Southern" 2 2003 430246624  7.449700355529785      94431543296   2.908003807067871     546290532352
        "Africa Eastern and Southern" 2 2004 441630144  5.023420810699463      99302326272   5.656581401824951     577191870464
        "Africa Eastern and Southern" 2 2005 453404064  8.558037757873535     102498344960   6.361804008483887     613911691264
        "Africa Eastern and Southern" 2 2006 465581376  8.898163795471191     107574280192   6.688755035400391     654974779392
        "Africa Eastern and Southern" 2 2007 478166912  8.450775146484375     115165241344   6.857303619384766     699888369664
        "Africa Eastern and Southern" 2 2008 491173152 12.566644668579102     124477849600   4.572539329528809     731891040256
        "Africa Eastern and Southern" 2 2009 504604672  8.954217910766602     123952439296   .9468112587928772     738820685824
        "Africa Eastern and Southern" 2 2010 518468224  5.537538051605225     129971150848   5.152336597442627     776887205888
        "Africa Eastern and Southern" 2 2011 532760416  8.971206665039063     142296317952   4.014183521270752     808072904704
        "Africa Eastern and Southern" 2 2012 547482880  9.158707618713379     146584043520   1.972651720046997     824013357056
        "Africa Eastern and Southern" 2 2013 562601600  5.746948719024658     155988459520  4.3083696365356445     859514863616
        "Africa Eastern and Southern" 2 2014 578075392 5.3702898025512695     158255005696  3.9867544174194336     893781606400
        "Africa Eastern and Southern" 2 2015 593871872  5.250170707702637     159200919552   2.925590753555298     919930011648
        "Africa Eastern and Southern" 2 2016 609978944    6.5946044921875     160711688192    2.01939058303833     938506977280
        "Africa Eastern and Southern" 2 2017 626392896  6.399343490600586     163519578112   2.542297840118408     962366668800
        "Africa Eastern and Southern" 2 2018 643090112  4.720811367034912     165689982976   2.475271701812744     986187825152
        "Africa Eastern and Southern" 2 2019 660046272  4.124350547790527     168937521152   2.077897787094116    1006679818240
        "Africa Eastern and Southern" 2 2020 677243328  5.191455841064453     171381129216  -2.939185619354248     977091624960
        "Afghanistan"                 4 1979  13411060                  .                .                   .                .
        "Afghanistan"                 4 1980  13356500                  .                .                   .                .
        "Afghanistan"                 4 1981  13171679                  .                .                   .                .
        "Afghanistan"                 4 1982  12882518                  .                .                   .                .
        "Afghanistan"                 4 1983  12537732                  .                .                   .                .
        "Afghanistan"                 4 1984  12204306                  .                .                   .                .
        "Afghanistan"                 4 1985  11938204                  .                .                   .                .
        "Afghanistan"                 4 1986  11736177                  .                .                   .                .
        "Afghanistan"                 4 1987  11604538                  .                .                   .                .
        "Afghanistan"                 4 1988  11618008                  .                .                   .                .
        "Afghanistan"                 4 1989  11868873                  .                .                   .                .
        "Afghanistan"                 4 1990  12412311                  .                .                   .                .
        "Afghanistan"                 4 1991  13299016                  .                .                   .                .
        "Afghanistan"                 4 1992  14485543                  .                .                   .                .
        "Afghanistan"                 4 1993  15816601                  .                .                   .                .
        "Afghanistan"                 4 1994  17075728                  .                .                   .                .
        "Afghanistan"                 4 1995  18110662                  .                .                   .                .
        "Afghanistan"                 4 1996  18853444                  .                .                   .                .
        "Afghanistan"                 4 1997  19357126                  .                .                   .                .
        "Afghanistan"                 4 1998  19737770                  .                .                   .                .
        "Afghanistan"                 4 1999  20170847                  .                .                   .                .
        "Afghanistan"                 4 2000  20779957                  .                .                   .                .
        "Afghanistan"                 4 2001  21606992                  .                .                   .                .
        "Afghanistan"                 4 2002  22600774                  .                .                   .       7228792320
        "Afghanistan"                 4 2003  23680872                  .                .    8.83227825164795       7867259392
        "Afghanistan"                 4 2004  24726688                  .                .  1.4141179323196411       7978511360
        "Afghanistan"                 4 2005  25654274  12.68626880645752                .  11.229714393615723       8874475520
        "Afghanistan"                 4 2006  26433058 6.7845964431762695                .    5.35740327835083       9349916672
        "Afghanistan"                 4 2007  27100542  8.680570602416992                .  13.826319694519043      10642666496
        "Afghanistan"                 4 2008  27722280 26.418664932250977                .  3.9249837398529053      11060389888
        "Afghanistan"                 4 2009  28394806 -6.811161041259766                .  21.390527725219727      13426265088
        "Afghanistan"                 4 2010  29185512  2.178537607192993                .  14.362441062927246      15354604544
        "Afghanistan"                 4 2011  30117412  11.80418586730957                .   .4263547956943512      15420069888
        "Afghanistan"                 4 2012  31161378 6.4412126541137695                .  12.752286911010742      17386481664
        "Afghanistan"                 4 2013  32269592  7.385771751403809                .   5.600744724273682      18360254464
        "Afghanistan"                 4 2014  33370804 4.6739959716796875                .   2.724543333053589      18860486656
        "Afghanistan"                 4 2015  34413604  -.661709189414978                .  1.4513146877288818      19134212096
        "Afghanistan"                 4 2016  35383028  4.383892059326172                .  2.2603142261505127      19566704640
        "Afghanistan"                 4 2017  36296112  4.975951671600342                .   2.647003173828125      20084635648
        "Afghanistan"                 4 2018  37171920  .6261491775512695                .  1.1892281770706177      20323487744
        "Afghanistan"                 4 2019  38041756  2.302372455596924                .  3.9116034507751465      21118461952
        "Afghanistan"                 4 2020  38928340                  .                . -2.3511006832122803      20621946880
        "Albania"                     5 1979   2617832                  .                .                   .                .
        "Albania"                     5 1980   2671997                  .                .                   . 4650625160.56601
        "Albania"                     5 1981   2726056                  .                .    5.74563529200671 4917833121.09044
        "Albania"                     5 1982   2784278                  .                .    2.94859680156867 5062840191.20539
        "Albania"                     5 1983   2843960                  .                .    1.10493826182712 5118781449.61318
        "Albania"                     5 1984   2904429                  .                .   -1.25159664469854 5054714952.74037
        "Albania"                     5 1985   2964762                  .                .    1.78064395977269 5144721429.23007
        "Albania"                     5 1986   3022635                  .                .    5.63724317825177  5434741887.0394
        "Albania"                     5 1987   3083605                  .                .   -.787842655012852 5391924672.26345
        "Albania"                     5 1988   3142336                  .                .   -1.42003965482257 5315357203.75915
        "Albania"                     5 1989   3227943                  .                .    9.83654897047623 5838204918.06266
        "Albania"                     5 1990   3286542                  .                .   -9.57564016948646 5279159422.75171
        "Albania"                     5 1991   3266790                  .                .   -28.0021416560459 3800881722.94428
        "Albania"                     5 1992   3247039   226.005421253526                .   -7.18711091636945 3527708137.71626
        "Albania"                     5 1993   3227287   85.0047512387157                .    9.55941169434738 3864936281.97555
        "Albania"                     5 1994   3207536   22.5650526933696                .     8.3028665911288 4185836785.30012
        end
        This time, I have included the countryname variable in the -reshape- command. Additionally, I include regions such as Africa Eastern and Southern - these will have dummies attached to them to exclude/include them from regressions I do on this dataset in future.
        Again, the results here are identical to the original dataset.

        You can see that while my code is long and has excess, it does work. That notwithstanding, if you could show me how to code a beter way to do this, you help would be gladly accepted. Also, with respect to my initial question: is there a way to include an `x' and a `y' variable in the same -foreach-/-forvalues- command, or a way to combine the two into the same loop?
        Apologies for the length of the post, I tried to include as much detail as I could.
        Thanks!

        Comment


        • #5
          Here is a more compact and clearer way to do it:

          Code:
          label define series     1   "gdp" ///
                                  2   "gdp_growth"    ///
                                  3   "gov_expenditure"   ///
                                  4   "inflation" ///
                                  5   "population", modify
          drop seriesname
          decode series, gen(seriesname)
          drop series
          
          reshape long yr, i(countrycode seriesname) j(year)
          rename yr _
          reshape wide _, i(countrycode year) j(seriesname) string
          rename _* *
          The only really hard part of this is that the seriesname variable is not usable as a variable name because it is too long and has blanks and special characters. So the first block of commands in the code I show fixes that. The rest is just -reshape-ing the data twice in the way I suggested in #2. I did not do the part about dropping observations with year < 1979 because in the example data, all the years are < 1979. But in your real data set that is just one line you can add to the code at any point after the first -reshape- command.

          I can't emphasize enough that if you think you should be using -merge m:m- then either you don't understand your data or your don't understand -merge m:m-. It is a trap that in nearly all cases produces incorrect results. In the case of your data, country and year actually jointly uniquely identify observations at the point where you are ready to -merge-. Since there is actually only one observation for each combination of country and year at that point, your m:m actually becomes 1:1, which is legitimate, so you got away with it.

          But you should not rely on that. When you have 1:1 matching in the data sets to be -merged-, use -merge 1:1-. If there are no variables that jointly identify unique observations in either data set, so you can't run 1:1, 1:m, or m:1 then let me assure you that in nearly all cases, -merge m:m- will produce a useless mashup of unrelated observations. I have been using Stata, pretty much every day, since 1994, and in all that time I have only once seen a situation where the data is actually not -merge-able with 1:1, 1:m, or m:1 and -merge m:m- produced a correct result. Usually when you find yourself in that situation it is because you have overlooked one or more variables that you could use to get to 1:1, 1:m, or m:1 (perhaps with some renaming in one of the data sets), or where what is really wanted is done by -joinby-. -merge m:m- is particularly treacherous because it does not give you any warning or error message to tell you that what you are doing is probably wrong, and if you don't look closely at the result, they can appear correct at first glance.

          Comment


          • #6
            You can see that while my code is long and has excess, it does work. That notwithstanding, if you could show me how to code a beter way to do this, you help would be gladly accepted. Also, with respect to my initial question: is there a way to include an `x' and a `y' variable in the same -foreach-/-forvalues- command, or a way to combine the two into the same loop?
            Actually, the code you show in #3 and #4 is materially different from what you had in #1. In particular,
            Code:
            forvalues y = 2(1)5 {
                drop if series != `y'
            }
            drops all observations in the data set. Think about it. On the first iteration, everything with series != 2 is eliminated. So all that is left is series == 2. Then on the next iteration, `y' == 3, and with series == 2, we have 2 != 3, so series 2 is now eliminated as well. So nothing is left.

            You eventually figured that out, or changed that loop for other reasons and finally ended up with code that worked.

            As for how to get `x' and `y' to synchronize in looping, the key is to do it as a single loop and calculate `x' as a function of `y' or the other way around. Here's how I would do it here (if I were going to use loops for this problem):

            Code:
            local new_vble_names gdp gdp_growth gdp_exp inflation population
            forvalues y = 2/5 {
                local x: word `y' of `new_vble_names'
                // DO STUFF WITH `x' AND `y' HERE
            }

            Comment


            • #7
              Clyde Schechter's tip at the end of #6 may be as much as you need.

              For a longer-winded account of loops in parallel see https://journals.sagepub.com/doi/pdf...6867X211063415

              Its main theme is simple. Loops in parallel are just one loop, with different actions at each step of the loop.

              Comment


              • #8
                Thank you both for your help.

                The second and third bits of code that I showed in my original post I knew didn't work - these were the best I could come up with since I could not work out how to do loops in parallel. The first one had -foreach x in "gdp_growth" "gdp_exp" "inflation" & forvalues y = 2(1)5- which of course gave a syntax error, while the second with -forvalues y=2(1)5- repeat itself inside the loop, deleting all of the data as you suggested it would. I definitely could have clarified that I knew that they didn't work yet were my best guesses of what may be close to correct in my original post - sorry for the confusion!
                This is why, in my second/third posts, I repeated the second bit of code in my second post several times with -series!=2- varying between 1, 2, 3, 4, 5. The idea was iterative collapse and merge procedure of roughly 40 lines of code - but potentially one that I thought I may be able to collapse into a loop, of maybe 10, hence the original post.

                The reason I used -merge m:m- was that I was getting the error "Variables do not uniquely identify observations in the master data" in my attempted 1:1 merges, but this fixed itself in the m:m merges. Now that I am looking at it again this morning, I cannot appear to replicate the errors while using a 1:1 merge, so I must have been doing something very wrong.

                Again, thank you very much for both of your help - it is definitely invaluable.

                Comment


                • #9
                  "Variables do not uniquely identify observations in the master data" in my attempted 1:1 merges, but this fixed itself in the m:m merges.
                  I know I'm somewhat obsessed with this issue, but it can't be said too many times: using m:m doesn't fix anything, it sweeps the error messages under the rug and allows invalid data to be propagated and created.

                  When you are working with a mature, well-developed programming system, and I consider Stata to be one such, when you get an error message, your goal should not be to get rid of the error message. It should be to understand what the error message means, and to fix the underlying problem in the data or in earlier code. Quick "fixes" like m:m, or indiscriminate use of -capture-, or -force- options just enable you to proceed from a state of the program and data that you have been put on notice is flat-out wrong. If you are lucky, the problems will resurface soon thereafter in your work. If you are unlucky, they will resurface when somebody else relies on your results to their detriment.

                  Error messages in computing are like pain in life: they are warnings that something is amiss. You ignore such warnings at your peril. The goal should be to fix what is amiss, not suppress the warning.

                  Comment

                  Working...
                  X