Carryforward

[Lindsey Buck]

* Example generated by -dataex-. To    install:    ssc    install    dataex
clear
input float(idc distp_cutoff4 age)
301          . 35
301   .8122066 21
301   .8122066 29
301   .8122066 33
301   .8122066 13
301   .8122066 31
301   .8122066 17
302          . 33
302   .6412213 17
302   .6412213 27
302   .6412213 31
302   .6412213 13
302   .6412213 29
401          . 36
401          . 30
401 -.09761905 16
401 -.09761905 32
401 -.09761905 34
4301 -.59052795 20
4301 -.59052795 12
4301 -.59052795 34
4301 -.59052795 16
4301 -.59052795 32
4301 -.59052795 28
4301 -.59052795 30
4302          . 27
4302          . 23
4302 .029911134 17
4302 .029911134 21
4302 .029911134 25
4302 .029911134 13
5701  .30498785 17
5701  .30498785 35
5701  .30498785 31
5701  .30498785 13
5701  .30498785 37
5701  .30498785 21
5701  .30498785 33
6201          . 16
6201          . 20
6201          . 14
6201    .751409 12
6201    .751409 18
8601          . 33
8601  -.3388582 21
8601  -.3388582 17
8601  -.3388582 35
8601  -.3388582 37
8601  -.3388582 39
8601  -.3388582 13
9601 -.21130952 20
9601 -.21130952 32
9601 -.21130952 36
9601 -.21130952 16
9601 -.21130952 12
9601 -.21130952 34
9601 -.21130952 30
9602   1.682412 16
9602   1.682412 12
9602   1.682412 28
9602   1.682412 26
9602   1.682412 24
9602   1.682412 20
9602   1.682412 30
9801 -.08328358 21
9801 -.08328358 27
9801 -.08328358 29
9801 -.08328358 13
9801 -.08328358 25
9801 -.08328358 23
9801 -.08328358 17
10001          . 27
10001   .5319467 21
10001   .5319467 25
10001   .5319467 31
10001   .5319467 13
10001   .5319467 29
15702          . 27
15702          . 23
15702          . 29
15702          . 25
15702  -.8403576 21
22101  -.8251565 16
22101  -.8251565 24
22101  -.8251565 30
22101  -.8251565 12
22101  -.8251565 26
22101  -.8251565 28
22101  -.8251565 20
22102  -.7766201 21
22102  -.7766201 23
22102  -.7766201 25
22102  -.7766201 29
22102  -.7766201 13
22102  -.7766201 27
22102  -.7766201 17
22103 -.55437726 13
22103 -.55437726 23
22103 -.55437726 27
22103 -.55437726 17
end

bys idc: carryforward distp_cutoff4, replace

Hi all,

Using the above code, I am trying to get my distp_cutoff4 variable to carryover for all observations with the same id. However, it keeps leaving one or two out per id. Can you help?
Thanks so much!
-Lindsey

[Nick Cox]

Thanks for the data example!

I don't know much about carryforward (from SSC, as you are asked to explain: FAQ Advice #12) but the nub here surely lies in the name. The command will carry forward, which doesn't mean backward.

What I guess you want here is to spread a non-missing value to all observations for an identifier if and only if there is only one such non-missing value. That can be done directly, say like this

* Example generated by -dataex-. To    install:    ssc    install    dataex
clear
input float(idc distp_cutoff4 age)
301          . 35
301   .8122066 21
301   .8122066 29
301   .8122066 33
301   .8122066 13
301   .8122066 31
301   .8122066 17
302          . 33
302   .6412213 17
302   .6412213 27
302   .6412213 31
302   .6412213 13
302   .6412213 29
401          . 36
401          . 30
401 -.09761905 16
401 -.09761905 32
401 -.09761905 34
4301 -.59052795 20
4301 -.59052795 12
4301 -.59052795 34
4301 -.59052795 16
4301 -.59052795 32
4301 -.59052795 28
4301 -.59052795 30
4302          . 27
4302          . 23
4302 .029911134 17
4302 .029911134 21
4302 .029911134 25
4302 .029911134 13
5701  .30498785 17
5701  .30498785 35
5701  .30498785 31
5701  .30498785 13
5701  .30498785 37
5701  .30498785 21
5701  .30498785 33
6201          . 16
6201          . 20
6201          . 14
6201    .751409 12
6201    .751409 18
8601          . 33
8601  -.3388582 21
8601  -.3388582 17
8601  -.3388582 35
8601  -.3388582 37
8601  -.3388582 39
8601  -.3388582 13
9601 -.21130952 20
9601 -.21130952 32
9601 -.21130952 36
9601 -.21130952 16
9601 -.21130952 12
9601 -.21130952 34
9601 -.21130952 30
9602   1.682412 16
9602   1.682412 12
9602   1.682412 28
9602   1.682412 26
9602   1.682412 24
9602   1.682412 20
9602   1.682412 30
9801 -.08328358 21
9801 -.08328358 27
9801 -.08328358 29
9801 -.08328358 13
9801 -.08328358 25
9801 -.08328358 23
9801 -.08328358 17
10001          . 27
10001   .5319467 21
10001   .5319467 25
10001   .5319467 31
10001   .5319467 13
10001   .5319467 29
15702          . 27
15702          . 23
15702          . 29
15702          . 25
15702  -.8403576 21
22101  -.8251565 16
22101  -.8251565 24
22101  -.8251565 30
22101  -.8251565 12
22101  -.8251565 26
22101  -.8251565 28
22101  -.8251565 20
22102  -.7766201 21
22102  -.7766201 23
22102  -.7766201 25
22102  -.7766201 29
22102  -.7766201 13
22102  -.7766201 27
22102  -.7766201 17
22103 -.55437726 13
22103 -.55437726 23
22103 -.55437726 27
22103 -.55437726 17
end

bysort id (dist) : gen OK = dist == dist[1] | missing(dist)
bysort id (OK) : replace OK = OK[_N]
bysort id (dist): replace dist = dist[1] if OK 

list, sepby(id)

     +------------------------------+
     |   idc   distp_c~4   age   OK |
     |------------------------------|
  1. |   301    .8122066    29    1 |
  2. |   301    .8122066    17    1 |
  3. |   301    .8122066    13    1 |
  4. |   301    .8122066    31    1 |
  5. |   301    .8122066    33    1 |
  6. |   301    .8122066    21    1 |
  7. |   301    .8122066    35    1 |
     |------------------------------|
  8. |   302    .6412213    31    1 |
  9. |   302    .6412213    27    1 |
 10. |   302    .6412213    29    1 |
 11. |   302    .6412213    13    1 |
 12. |   302    .6412213    17    1 |
 13. |   302    .6412213    33    1 |
     |------------------------------|
 14. |   401    -.097619    34    1 |
 15. |   401    -.097619    32    1 |
 16. |   401    -.097619    16    1 |
 17. |   401    -.097619    30    1 |
 18. |   401    -.097619    36    1 |
     |------------------------------|
 19. |  4301    -.590528    16    1 |
 20. |  4301    -.590528    20    1 |
 21. |  4301    -.590528    30    1 |
 22. |  4301    -.590528    32    1 |
 23. |  4301    -.590528    28    1 |
 24. |  4301    -.590528    12    1 |
 25. |  4301    -.590528    34    1 |
     |------------------------------|
 26. |  4302    .0299111    21    1 |
 27. |  4302    .0299111    13    1 |
 28. |  4302    .0299111    17    1 |
 29. |  4302    .0299111    25    1 |
 30. |  4302    .0299111    23    1 |
 31. |  4302    .0299111    27    1 |
     |------------------------------|
 32. |  5701    .3049878    37    1 |
 33. |  5701    .3049878    35    1 |
 34. |  5701    .3049878    33    1 |
 35. |  5701    .3049878    13    1 |
 36. |  5701    .3049878    21    1 |
 37. |  5701    .3049878    17    1 |
 38. |  5701    .3049878    31    1 |
     |------------------------------|
 39. |  6201     .751409    12    1 |
 40. |  6201     .751409    18    1 |
 41. |  6201     .751409    16    1 |
 42. |  6201     .751409    20    1 |
 43. |  6201     .751409    14    1 |
     |------------------------------|
 44. |  8601   -.3388582    39    1 |
 45. |  8601   -.3388582    13    1 |
 46. |  8601   -.3388582    35    1 |
 47. |  8601   -.3388582    17    1 |
 48. |  8601   -.3388582    37    1 |
 49. |  8601   -.3388582    21    1 |
 50. |  8601   -.3388582    33    1 |
     |------------------------------|
 51. |  9601   -.2113095    12    1 |
 52. |  9601   -.2113095    30    1 |
 53. |  9601   -.2113095    34    1 |
 54. |  9601   -.2113095    32    1 |
 55. |  9601   -.2113095    20    1 |
 56. |  9601   -.2113095    36    1 |
 57. |  9601   -.2113095    16    1 |
     |------------------------------|
 58. |  9602    1.682412    24    1 |
 59. |  9602    1.682412    12    1 |
 60. |  9602    1.682412    30    1 |
 61. |  9602    1.682412    20    1 |
 62. |  9602    1.682412    28    1 |
 63. |  9602    1.682412    16    1 |
 64. |  9602    1.682412    26    1 |
     |------------------------------|
 65. |  9801   -.0832836    25    1 |
 66. |  9801   -.0832836    17    1 |
 67. |  9801   -.0832836    23    1 |
 68. |  9801   -.0832836    13    1 |
 69. |  9801   -.0832836    29    1 |
 70. |  9801   -.0832836    21    1 |
 71. |  9801   -.0832836    27    1 |
     |------------------------------|
 72. | 10001    .5319467    21    1 |
 73. | 10001    .5319467    31    1 |
 74. | 10001    .5319467    29    1 |
 75. | 10001    .5319467    25    1 |
 76. | 10001    .5319467    13    1 |
 77. | 10001    .5319467    27    1 |
     |------------------------------|
 78. | 15702   -.8403576    21    1 |
 79. | 15702   -.8403576    27    1 |
 80. | 15702   -.8403576    29    1 |
 81. | 15702   -.8403576    25    1 |
 82. | 15702   -.8403576    23    1 |
     |------------------------------|
 83. | 22101   -.8251565    30    1 |
 84. | 22101   -.8251565    24    1 |
 85. | 22101   -.8251565    16    1 |
 86. | 22101   -.8251565    26    1 |
 87. | 22101   -.8251565    28    1 |
 88. | 22101   -.8251565    12    1 |
 89. | 22101   -.8251565    20    1 |
     |------------------------------|
 90. | 22102   -.7766201    29    1 |
 91. | 22102   -.7766201    17    1 |
 92. | 22102   -.7766201    25    1 |
 93. | 22102   -.7766201    23    1 |
 94. | 22102   -.7766201    27    1 |
 95. | 22102   -.7766201    13    1 |
 96. | 22102   -.7766201    21    1 |
     |------------------------------|
 97. | 22103   -.5543773    23    1 |
 98. | 22103   -.5543773    13    1 |
 99. | 22103   -.5543773    27    1 |
100. | 22103   -.5543773    17    1 |
     +------------------------------+

Alternatively, or additionally, see https://www.stata.com/support/faqs/d...issing-values/ for the difference between copying forward and backward or
https://www.statalist.org/forums/for...-interpolation for a fairly broad interpolation command.

[Lindsey Buck]

Thank you! Hmm, I tried with this code:

foreach var in  birthyear distp_cutoffb4 {

bysort idc (`var') : gen OK = `var' == `var'[1] | missing(`var')
bysort idc (OK) : replace OK = OK[_N]
bysort idc (`var'): replace `var'= `var'[1] if OK
drop OK
}

and it keeps saying that "Variable OK already defined"? Do you know wwhy?

[Lindsey Buck]

I even tried it so it renamed them as `var'_2 but it doesn't seem to be working at carrying the values forward.

[Lindsey Buck]

So here is my code that doesn't seem to work:

foreach var in birthyear distp_cutoffb4 lincomeb02_2 lincomeb02_3 fsize02_1 fsize02_2 fsize02_3 fsize_incomeb02 bweight_1 bweight_2 bweight_3 lincomeb4_1 lincomeb4_2 lincomeb4_3 fsize4_1 fsize4_2 fsize4_3 fsize_incomeb4 father_figure4 male black dyear4_4 dagech_4 cluster4 lincomeb02_1 eligb4 hspart sex dstate4_4 female male all all_noblack all_black noblack black black_all black_female black_male black_all all_female all_male noblack_male noblack_female noblack_all all_all all bweight loweight hspart everps hother afqtmomst dropout02 college hs02 father_figure incomebd02 fsize4 poorb4 eligb4 breastfed momselfesteem maxindex maxage momselfesteemindex motherdepression incomed4 {

bysort idc (`var') : gen `var'_carry = `var' == `var'[1] | missing(`var')
bysort idc (`var'_carry) : replace `var'_carry = `var'_carry[_N]
bysort idc (`var'): replace `var'= `var'[1] if `var'_carry 
}

[Nick Cox]

I can't see what the problem is there.

[Lindsey Buck]

Weird! I somehow got it to work with this code:

foreach var in birthyear distp_cutoff4 lincomeb02_2 lincomeb02_3 fsize02_1 fsize02_2 fsize02_3 fsize_incomeb02 bweight_1 bweight_2 bweight_3 lincomeb4_1 lincomeb4_2 lincomeb4_3 fsize4_1 fsize4_2 fsize4_3 fsize_incomeb4 father_figure4 male black dyear4_4 dagech_4 cluster4 lincomeb02_1 eligb4 hspart sex dstate4_4 female male all all_noblack all_black noblack black black_all black_female black_male black_all all_female all_male noblack_male noblack_female noblack_all all_all all bweight loweight hspart everps hother afqtmomst dropout02 college hs02 father_figure incomebd02 fsize4 poorb4 eligb4 breastfed momselfesteem maxindex maxage momselfesteemindex motherdepression incomed4 {

bysort idc (year) : carryforward `var', replace
}

I just didn't have (year) in the carryforward code before. Thanks!