create analytical/inverse variance weights for a variable

[farah roslan]

Dear all,

I create a new topic here since my last topic is a bit unclear.

I know how to tabulate the analytical weights but I am not sure how to create the analytical weight for the variable in the fixed-effect model. My dep var.

pv_small

are varied throughout countries therefore I want to create an analytical weight for it.

tab pv_small [aw=pv_small], m

        net |
   addition |
   solar PV |
residential |      Freq.     Percent        Cum.
------------+-----------------------------------
       .001 |.0000838047        0.00        0.00
       .002 |.0000558698        0.00        0.00
      .0025 | .000034919        0.00        0.00
       .005 | .000069837        0.00        0.00
       .007 | .000097772        0.00        0.00
        .01 | .000698372        0.00        0.00
       .015 | .000209512        0.00        0.00
        .02 | .000279349        0.00        0.00
       .021 | .000293316        0.00        0.00
       .025 | .001745931        0.00        0.00
       .026 | .000363154        0.00        0.00
        .03 | .000838047        0.00        0.00
       .033 |.0004609258        0.00        0.00
       .034 | .000474893        0.00        0.00
      .0397 | .000554508        0.00        0.00
        .05 | .004190235        0.00        0.00
       .053 | .000740275        0.00        0.00
       .055 |  .00076821        0.00        0.00
       .061 | .000852014        0.00        0.00
       .062 |.0008659818        0.00        0.00
       .075 | .001047559        0.00        0.00
       .081 | .001131363        0.00        0.00
       .095 | .001326908        0.00        0.01
         .1 |  .05447305        0.02        0.02
        .11 | .001536419        0.00        0.02
       .113 | .001578322        0.00        0.02
       .116 | .001620224        0.00        0.02
       .125 | .001745931        0.00        0.02
       .126 | .001759899        0.00        0.02
       .143 | .001997345        0.00        0.02
       .178 | .002486206        0.00        0.02
        .19 | .002653815        0.00        0.03
         .2 | .013967449        0.00        0.03
       .236 | .003296318        0.00        0.03
       .238 | .006648506        0.00        0.03
        .24 | .003352188        0.00        0.03
       .242 | .003380123        0.00        0.03
        .25 | .045394208        0.01        0.05
       .255 | .003561699        0.00        0.05
        .28 | .007821771        0.00        0.05
        .29 |  .00405056        0.00        0.05
       .296 | .004134365        0.00        0.05
         .3 | .016760939        0.00        0.06
        .33 |.0092185164        0.00        0.06
       .345 |  .00481877        0.00        0.06
        .35 | .009777214        0.00        0.07
       .362 | .005056216        0.00        0.07
       .378 | .005279695        0.00        0.07
       .387 | .005405403        0.00        0.07
       .389 | .005433337        0.00        0.07
         .4 |.0055869795        0.00        0.07
       .424 | .005922198        0.00        0.08
       .439 |  .00613171        0.00        0.08
        .48 | .006704375        0.00        0.08
       .488 | .006816115        0.00        0.08
        .49 |   .0136881        0.00        0.09
       .495 | .006913887        0.00        0.09
         .5 | .083804691        0.02        0.11
        .54 | .007542423        0.00        0.11
        .55 | .015364194        0.00        0.12
        .56 | .015643542        0.00        0.12
         .6 | .025141408        0.01        0.13
        .63 | .008799493        0.00        0.13
        .65 | .009078841        0.00        0.14
        .75 |.0104755864        0.00        0.14
        .79 | .011034285        0.00        0.14
         .8 | .033521877        0.01        0.15
       .827 |  .01155108        0.00        0.16
        .85 | .011872332        0.00        0.16
       .876 | .012235485        0.00        0.16
       .907 |.0126684759        0.00        0.17
        .95 | .013269076        0.00        0.17
        .96 |.0268175006        0.01        0.18
          1 | .111739588        0.03        0.21
       1.09 | .015224519        0.00        0.22
        1.1 | .030728387        0.01        0.22
      1.133 | .015825119        0.00        0.23
       1.15 | .016062565        0.00        0.23
        1.2 | .016760939        0.00        0.24
       1.22 | .017040288        0.01        0.24
      1.269 | .017724693        0.01        0.25
        1.5 | .104755864        0.03        0.28
        1.7 | .023744663        0.01        0.29
          2 | .083804691        0.02        0.31
        2.1 |.0293316406        0.01        0.32
        2.4 | .033521878        0.01        0.33
      2.542 | .035505255        0.01        0.34
       2.84 | .039667553        0.01        0.35
       2.89 | .040365928        0.01        0.36
          3 | .083804691        0.02        0.39
        3.5 |  .04888607        0.01        0.40
          4 | .167609382        0.05        0.45
       4.08 | .056987189        0.02        0.47
       4.12 | .057545886        0.02        0.49
        4.5 | .062853518        0.02        0.50
        4.8 | .067043756        0.02        0.52
          5 | .209511728        0.06        0.59
       5.02 |.0701165914        0.02        0.61
       5.15 | .071932361        0.02        0.63
       5.25 | .073329105        0.02        0.65
        5.4 | .075424223        0.02        0.67
          6 | .167609382        0.05        0.72
       6.09 | .085061764        0.03        0.75
          7 |  .09777214        0.03        0.77
      8.517 | .118960762        0.03        0.81
          9 | .251414074        0.07        0.88
        9.8 | .136880998        0.04        0.92
         10 | .279348971        0.08        1.01
      10.89 | .152105519        0.04        1.05
      11.75 |  .16411752        0.05        1.10
         12 |.6704375299        0.20        1.30
      13.33 | .186186088        0.05        1.35
      13.46 | .188001858        0.06        1.41
      13.62 | .190236648        0.06        1.46
         14 | .391088559        0.12        1.58
         15 | .628535184        0.18        1.76
       15.5 | .216495452        0.06        1.83
         16 | .223479177        0.07        1.89
         17 |  .47489325        0.14        2.03
      17.06 | .238284665        0.07        2.10
      17.27 | .241217843        0.07        2.17
         18 | .502828147        0.15        2.32
      18.27 | .255185291        0.08        2.39
   18.79458 | .262512389        0.08        2.47
         19 | .265381522        0.08        2.55
         20 | .279348971        0.08        2.63
         21 | .293316419        0.09        2.72
       21.6 | .301696894        0.09        2.81
       21.7 | .303093644        0.09        2.90
         22 | .307283868        0.09        2.99
         26 | .363153662        0.11        3.09
         29 | .405056008        0.12        3.21
         30 | .419023456        0.12        3.34
       30.5 |  .42600718        0.13        3.46
         34 |  .47489325        0.14        3.60
         35 | .488860699        0.14        3.74
         38 |.5307630445        0.16        3.90
      41.01 | .572805041        0.17        4.07
         44 | .614567736        0.18        4.25
       50.2 | .701165927        0.21        4.46
     53.008 | .740386504        0.22        4.67
   54.48401 | .761002642        0.22        4.90
      55.77 | .778964611        0.23        5.13
      58.98 | .823800108        0.24        5.37
         67 | .935819052        0.28        5.64
      71.07 | .992666563        0.29        5.94
      72.24 | 1.00900845        0.30        6.23
         78 | 1.08946099        0.32        6.55
      81.31 | 1.13569321        0.33        6.89
         82 | 1.14533078        0.34        7.22
   86.83963 |1.212928065        0.36        7.58
      90.24 | 1.26042253        0.37        7.95
         92 | 1.28500527        0.38        8.33
      94.98 | 1.32662831        0.39        8.72
      95.94 |1.340037047        0.39        9.11
       97.5 | 1.36182623        0.40        9.51
        103 |  1.4386472        0.42        9.94
     104.22 |  1.4556875        0.43       10.37
     104.76 | 1.46322994        0.43       10.80
     108.69 | 1.51812202        0.45       11.24
     116.46 | 1.62664904        0.48       11.72
   128.8272 | 1.79938742        0.53       12.25
     135.33 | 1.89021484        0.56       12.81
      138.1 | 1.92890473        0.57       13.37
     142.09 | 1.98463471        0.58       13.96
     149.25 | 2.08464169        0.61       14.57
     151.15 | 2.11117976        0.62       15.19
        153 | 2.13701963        0.63       15.82
    165.625 | 2.31335909        0.68       16.50
     172.21 | 2.40533441        0.71       17.21
     172.91 | 2.41511158        0.71       17.92
        185 | 2.58397798        0.76       18.68
   189.0797 | 2.64096157        0.78       19.45
   192.4785 | 2.68843355        0.79       20.25
     192.58 | 2.68985127        0.79       21.04
   193.5384 | 2.70323816        0.80       21.83
     197.62 | 2.76024711        0.81       22.64
        200 | 2.79348971        0.82       23.47
     200.32 | 2.79795939        0.82       24.29
     206.64 | 2.88623356        0.85       25.14
    216.262 |3.020628264        0.89       26.03
     219.77 | 3.06962623        0.90       26.93
   233.4798 | 3.26111642        0.96       27.89
     235.86 | 3.29436242        0.97       28.86
   246.3454 | 3.44081624        1.01       29.87
     248.85 | 3.47579965        1.02       30.89
     276.45 | 3.86130132        1.14       32.03
     309.85 | 4.32781402        1.27       33.30
      322.9 | 4.51008905        1.33       34.63
    339.613 | 4.74352719        1.40       36.02
   342.8159 | 4.78826327        1.41       37.43
     357.34 | 4.99112801        1.47       38.90
      378.7 | 5.28947293        1.56       40.45
    418.945 | 5.85159283        1.72       42.17
     442.25 | 6.17710412        1.82       43.99
        479 | 6.69040785        1.97       45.96
        545 |7.612259454        2.24       48.20
   660.7269 | 9.22866938        2.71       50.91
        675 | 9.42802776        2.77       53.68
     678.58 | 9.47803147        2.79       56.47
        824 | 11.5091776        3.39       59.86
      851.5 | 11.8932824        3.50       63.36
     929.05 | 12.9764579        3.82       67.17
   952.1545 | 13.2991696        3.91       71.08
    955.618 |  13.347545        3.93       75.01
    1024.15 | 14.3047628        4.21       79.22
   1092.016 | 15.2526772        4.49       83.70
   1205.594 | 16.8390789        4.95       88.66
    1264.35 |17.65974322        5.19       93.85
   1497.172 | 20.9116779        6.15      100.00
------------+-----------------------------------
      Total |        340      100.00

. 
.

Thank you so much for your helpful answer!

[Clyde Schechter]

The usual situation for using what Stata calls aweights is when your variable is actually the average of several measurements, and the number of such measurements varies from observation to observation. So to calculate proper aweights you have to go back to the way your data set was generated and find out how many measurements were averaged to create the value of pv_small for each observation in your data set. The aweight for that observation is just the number of measurements that were averaged.

If that doesn't make sense in your context, then you probably don't really want aweights, but some other kind of weights. If you explain in greater detail, more concrete advice might be available.

[farah roslan]

Hi Clyde,

Thank you for your reply.

I have 19 panels data from 1998-2015 and my dep var is pv_small. My supervisor asked me to weight the dep var to increase the importance of some observations in the regression, some pv capacity is greater in advanced countries e.g. Japan, while for other countries, e.g. Malaysia, its development still at infancy period. For e.g., PV capacity in 2015 for Japan stood almost 10,000 MW while in Malaysia is only 60 MW.

I have read somewhere in previous https://www.statalist.org/forums/for...la-for-aweight and try to implement the steps but I when I try to run fixed effect model, it gives me an error message.

egen mean_pv_small=mean(pv_small)
gen w_pv_small=pv_small/mean_pv_small

 xtreg lpv_small rois tax grant lgdp import lsharee lcarbon3 i.year, fe robust [aw=w_pv_small]
option [ not allowed

xtreg lpv_small [aw=w_pv_small] rois tax grant lgdp import lsharee lcarbon3 i.year, fe robust
invalid 'rois'

I just can't seem run the command. I am not sure whether I'm doing the right thing.

[Clyde Schechter]

Concerning the error message you are getting, they are purely syntax errors because you have put [aw=w_pv_small] in the wrong place. It has to go immediately before the comma.

That said, these do not sound like aweights. And they are clearly not fweights or pweights. And unfortunately, those are the only kinds of weight that can be used with -xtreg, fe-. If you were using, say, the population averaged -xtreg- I would advise you to use iweights, but -xtreg, fe- won't allow those.

I suppose what you are looking for is closest in spirit to aweights.

All of that said, I am concerned about what you are using as your weighting factor, however. Your proposed weighting variable is just the value of the variable pv_small scaled by the sample average value. Whatever the merits of that in other respects, your outcome variable is called lpv_small. Now, I don't know what that is, but when I see a variable named l-something and that something is the name of another variable, my suspicion is that l-something is either the log or the lag of that other variable. Then again, you actually say that your dependent variable is pv_small itself. If any of those is the case you are weighting the regression by something closely related to the outcome variable itself. I won't go so far as to say that it's completely invalid to do that, but it's really bizarre, and I think it will be very difficult, if not impossible, to find a reasonable interpretation of the results. And, although this content is outside my domain of expertise, so I may well be missing something, I have difficulty grasping how it could be sensible to do this in any context.

If I were you, I would go back to the supervisor and ask for some clarification of just how he or she expects you to do this weighting.

[farah roslan]

Dear Clyde,

Sorry, for running the wrong regression. Previously, I have transformed the pv_small into logarithm form, but when I did that the 'zero' observation turned to be missing values '.' That's why I make an approach to weight the variable first since some of the values are large and some of them are small.

Correct me if I'm wrong, does xtreg, fe only recognise fweights or pweights? Or aweight?

I try to see if aweight works but the error message give me

xtreg pv_small [aw=w_pv_small] rois, fe
invalid 'rois'

[Clyde Schechter]

Please re-read what I wrote at the very beginning of #4. You have put [aw = w_pv_small] in the wrong place, again!

-xtreg, fe- does recognize fweights, pweights, and aweights. None of these are exactly right for what you are trying to do. But aweights are closer to it than fweights and pweights which would be very, very wrong.

That said, if you are using weighting because you wanted to use a log transform but couldn't due to zeroes, consider using -xtpoisson- with no weighting and no transformation of the outcome variable instead of -xtreg-. The Poisson regression has a log link, so you get the benefits of a log transform, but there is no problem with zeroes in the outcome.

[farah roslan]

Dear Clyde,

Thank you for your helpful answer!

I have another question, does apply log transformation will increase/reduce the importance of some observations of the dependent variable in the regression since some of the values are large and small? All previous literature have applied log to the capacity installation. Actually the internal examiner for my PhD thesis has asked me to appropriately scale the observation, pv capacity, to check if the results are driven by one or two small countries.

Thank you again Clyde.

[Clyde Schechter]

Log transformation does not increase or reduce the importance of an observation, at least not directly. Rather, it changes the scale of the observations. So if the original range of the outcome variables goes from an order of magnitude of, say, 1000 to an order of magnitude of, say, 100,000,000, a log10 transformation shrinks this range to from 3 to 8, so that the very large values are no longer as dramatically large compared to the smaller ones.

Now, the term "importance" has no statistical meaning. But what you may be thinking of here is leverage. An observation has high leverage if its exclusion from the data set results in a material change in the regression coefficients. In order to be a high leverage observation, both the outcome and one of the predictor variables needs to have an outlying value. A log transformation, by shrinking the range of the data, may indeed turn a high leverage observation into an ordinary observation. But remember that just because the value of the outcome in an observation is extreme, that does not, by itself, make it a high-leverage observation.

While there are advantages to working with variables whose distributions do not have long tails, bear in mind that the routinely transforming variables to "normalize" them is not appropriate. The most important concern is whether or not the relationship between the outcome and the predictors is linear in the untransformed metric. If so, transforming it will de-linearize the relationship and, in that case, the use of a linear model becomes a mis-specification, from which valid conclusions cannot be drawn. If, however, transforming the outcome variable results in a more linear relationship, then doing so improves the model. It is on this basis that a decision to transform or not transform should be made. (With linear relationships, R² may be the best indicator of how well your linear model fits.)

So, in brief, log-transforming may or may not be a good idea, and while it may remove some high leverage problems from the data, it is not guaranteed to do so. It is not a substitute for or equivalent to weighting. I do think you need to speak to your supervisor at length for more penetrating insight about what is needed here. It isn't possible for me to give you specific advice at a distance, and with no understanding of the project, which is, in any case, about subject matter I have little knowledge of. All I can do is point out the general principles, but not how they apply to your case.