Normalize RMSE after running sureg

[Kensuke Suzuki]

Hi everyone,

I would like to store the RMSEs which are normalized by the variance of the regressions after running seemingly unrelated regressions (sureg).

I have three dependent variables y₁ y₂ and y₃ and two independent variables x₁ and x₂ generated from x using the mkspline command. After running the SUR;
sureg ( y₁ x₁ x₂ ) ( y₂ x₁ x₂ ) ( y₃ x₁ x₂ )
I would like to store the RMSEs of three regression which are normalized by the variance of each regression, something like RMSE = e's^-1e and store them where e is the residual vector and the s is the variance of regression. I need to run the SUR and store the results repeatedly with different knot location and determine the knot which minimize the sum of normalized RMSE.

Thank you very much for your help in advance!

Ken

----
Kensuke SUZUKI (Mr.) M.Econ.
PhD Student, Graduate School of Economics, Nagoya University
Research Fellow, Japan Society for the Promotion of Science (JSPS)
E-mail: [email protected]

[Emad Shehata]

clear all
 input t y1 y2 y3 x1 x2
 1  89.1  99.6  96.7  96.7   101
 2  99.2 102.6  98.1  98.1 100.1
 3    99 125.6   100   100   100
 4   100 130.1 104.9 104.9  90.6
 5 111.6 135.6 124.9 104.9  86.5
 6 122.2 142.2 109.5 109.5  89.7
 7 117.6 157.6 120.8 110.8  90.6
 8 121.1 125.2 112.3 112.3  82.8
 9   136   136 119.3 109.3  70.1
10 154.2 154.2 115.3 105.3  65.4
11 153.6 153.6 121.7 101.7  61.3
12 158.5 155.5 125.4  95.4  62.5
13 140.6 140.7 146.4  96.4  63.6
14 136.2 176.2 127.6  97.6  52.6
15   168 185.8 132.4 102.4  59.7
16 154.3 186.3 132.6 101.6  59.5
17   149   189 155.8 103.8  61.3
 end

sureg (y1 x1 x2) (y2 x1 x2) (y3 x1 x2)
scalar RMSE1= e(rmse_1)
scalar RMSE2= e(rmse_2)
scalar RMSE3= e(rmse_3)

scalar list RMSE1  RMSE2 RMSE3

.

 sureg (y1 x1 x2) (y2 x1 x2) (y3 x1 x2)

Seemingly unrelated regression
----------------------------------------------------------------------
Equation          Obs  Parms        RMSE    "R-sq"       chi2        P
----------------------------------------------------------------------
y1                 17      2    9.055903    0.8556     100.69   0.0000
y2                 17      2    14.52961    0.6932      38.41   0.0000
y3                 17      2    9.839389    0.6125      26.87   0.0000
----------------------------------------------------------------------

------------------------------------------------------------------------------
             |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
y1           |
          x1 |   .5118888   .4340857     1.18   0.238    -.3389034    1.362681
          x2 |  -1.366398   .1364309   -10.02   0.000    -1.633798   -1.098999
       _cons |   181.5685   44.10346     4.12   0.000     95.12732    268.0097
-------------+----------------------------------------------------------------
y2           |
          x1 |   .9680958   .6964624     1.39   0.165    -.3969454    2.333137
          x2 |  -1.355074   .2188946    -6.19   0.000      -1.7841   -.9260485
       _cons |   150.5231   70.76115     2.13   0.033     11.83377    289.2124
-------------+----------------------------------------------------------------
y3           |
          x1 |   .1925461   .4716413     0.41   0.683    -.7318539    1.116946
          x2 |  -.7645066   .1482345    -5.16   0.000    -1.055041   -.4739724
       _cons |   158.7296   47.91914     3.31   0.001     64.80986    252.6494
------------------------------------------------------------------------------

. scalar RMSE1= e(rmse_1)
. scalar RMSE2= e(rmse_2)
. scalar RMSE3= e(rmse_3)

. scalar list RMSE1  RMSE2 RMSE3
     RMSE1 =  9.0559033
     RMSE2 =  14.529611
     RMSE3 =  9.8393891

[Kensuke Suzuki]

Hi Emad,

Thanks!

Ken

[Steve Samuels]

Welcome to Statalist, Kensuke. Emad has done for you what we ask all posters to do in FAQ 12, which is essential reading: illustrate the problem with a self-contained data set or one built into Stata or referenced in the Manual. Put all code and results between CODE delimiters.

The estimated variances of the disturbances are the diagonal entries of the ereturned matrix e(Sigma), which is defined on page 1201 in the Manual Entry for reg3.

To get the normalized values, use this code after the sureg command:

forvalues i = 1/3{
    scalar v`i' =el(e(Sigma),`i',`i')
    scalar norm`i' = e(rmse_`i')/v`i'
    scalar list norm`i'
}
     norm1 =  .11042521
     norm2 =  .06882497
     norm3 =  .10163233

[Kensuke Suzuki]

Hi Steve,

that's exactly what i want to do and it works. Thanks a lot for your help!

Kensuke

[Steve Samuels]

You're welcome, Kensuke. I'm not familiar with this area, but I should point out that the units are not congruent. In your initial post \( \text{RMSE} = e'e/s\) , the \(e'e\) is a sum of squares, not a mean square and not a root mean square, If you want to normalize the root mean square, then perhaps you should divide by the standard deviation of the disturbances:

scalar norm`i' = e(rmse_`i')/sqrt(v`i')

When I do this on the data above:

. scalar list norm`i'
     norm1 =          1
     norm2 =          1
     norm3 =          1

So it looks like the the RMSE is the standard deviation of the disturbances.

[Emad Shehata]

Check NRMSE here
https://en.m.wikipedia.org/wiki/Root...uare_deviation

[Emad Shehata]

Nrmsei = rmsei/ (maxei - minei)
Or
Nrmsei = rmsei/mean (ei)
For each equation i

[Emad Shehata]

NRMSE is looks like CV

[Steve Samuels]

Thank you, Emad. Either version makes sense.

[Emad Shehata]

But I am not sure any mean will be sued
is the mean of error term of each equation which will approach to zero?
or mean of each dependent variable?

[Kensuke Suzuki]

Hi Emad an Steve,

thank you very much for your replies and suggestions. i wanted to normalize RMSE for model selection sake. I have three parameters which defines the dependent variables and the system wide constraints, and these should be given in the estimation of the system. So i'm employing the grid search approach to find the three parameters. log likelihood is usually used to evaluate the goodness-of-prediction of the each model generated by grid-search iteration, but it does not give us reasonable results. Thus i tried to employ RMSE to evaluate the goodness-of-fit and discount the RMSE to take into consideration of the different scales of dependent variables in the three equations.Although i have tried to normalize the RMSE by the arithmetic mean of dependent variable (just as the posted Wikipedia page suggests), i got a criticism saying that this way of normalization is not usual way in the literature. Now i'm trying to construct the other measure, R^2, to evaluate the fitness of the system of equation.

Thanks anyway for your great help and i apologize for my late reply.

Kensuke