How to calculate the standard deviation of residuals in the Fama-French three factors' model?

[Jae Li]

Hello everyone!

I am replicating a paper about calculating the idiosyncratic volatility with respect to the FF three factors' models. The excess returns and FF three factors are daily recorded but the idiosyncratic volatility needs to be computed based on monthly basis. However, I still couldn't get the same result as the paper did after trying many many times. May I ask that, is there anyone who is able to have a look for me? Many thanks for your help and time, I will be very grateful if you can give some tips!

Here is my code:

keep date permno mkt_rf smb hml rf ret vwretd
gen y=year(date)
gen m=month(date)
gen ym=y*100+m
gen exret1=(ret-rf)*sqrt(250) *annualised excess returns

drop if missing(ret)
sort ym permno

reg exret1 mkt_rf smb hml


May I also ask that, does the Root MSE out of the regression analysis mean the idiosyncratic volatility?

[Clyde Schechter]

Before answering your main question, your ym variable, although it will serve the purpose at hand, is not a good way to create a monthly variable: and it certainly will not work for most purposes that you would use a monthly date variable. Do instead

gen ym = mofd(date)
format ym %tm

Now you want to calculate the standard deviation of the residuals of the exret1 regression in each month. So after your regression command, do this:

predict resid, resid
egen monthly_sd_resid = sd(resid), by(ym)

The number stored in e(rmse) after your regression is related to the standard deviation of the residuals in the entire estimation sample, but there is a sqrt((N-1)/N) factor difference. If your N is large, the difference can be ignored for most purposes.

[William Lisowski]

Crossposted on Stack Overflow at https://stackoverflow.com/questions/...a-french-three

[Jae Li]

Clyde Schechter Many thanks for your reply!

After checking many times with the paper, I just realise the idiosyncratic volatility needs to computed based on each firm at each month, so there will be millions of regressions created for calculating each idiosyncratic volatility. I am currently cracking on this issue now and hopefully will resolve it by using the loops in STATA.

Many thanks for your codes, your tips help me clarify a lot of confusions between the steps so I really appreciate that help!

Wish you all the best!

[Robert Picard]

If you need to perform millions of regressions, I'm pretty sure that rangestat (from SSC) will be by far the most efficient tool to do this.

[Jae Li]

Robert Picard woow, sounds exciting! Thank you very much for the tip, I will try it now!

[Jae Li]

@Clyde Schechter Hello, can I ask you one more question?

I am using the codes at below to convert the daily returns to monthly returns. However, I found that I need to tell STATA to run it based on each permno (firm) and ym (month and year), do you know how to achieve that by a loop? (ret=returns)

The dataset looks like this:

permno date ret

10006 02JAN1980 0.02
10057 03JAN1980 0.85
10058 04JAN1980 0.35
10065 06JAN1980 -0.25
10103 07JAN1980 0.24
10137 09JAN1980 0.39
10145 10JAN1980 0.87
10146 11JAN1980 0.88

This is my loop codes:

foreach var in ret {
gen `var'_m= (sum(`var'/100 +1)-1)*100
}

Many thanks for your time, hope to hear from you soon!

[Clyde Schechter]

You don't need, and shouldn't code, another loop for this. Doing this based on each permno and ym can be accomplished with the -by:- prefix.

foreach var in ret {
    by permno ym (date), sort: gen `var'_m= (sum(`var'/100 +1)-1)*100
}

Also, if the only variable you are doing this for is ret, then there is no point in having a loop over just one variable:

by permno ym (date), sort: gen ret_m = (sum(ret/100 +1)-1)*100

will suffice.

[Robert Picard]

Please follow the request in the FAQ to present data examples using dataex (from SSC).

I'm not in finance but my understanding is that you can't simply add returns, you need to multiply them. This is typically done by adding returns in logs.I also infer from what you were trying to do that your returns are in percents so you first need to convert them to a fraction before proceeding. Your data example is not really useful since permno is different for each observation. I added two additional observations for permno 10006 so that you can see what's happening when you convert daily to monthly returns this way. At the end, I show how to manually calculate the return for the month for permno 10006.

* Example generated by -dataex-. To install: ssc install dataex
clear
input float permno str9 date float ret
10006 "02JAN1980"  .02
10006 "03JAN1980"  .03
10006 "04JAN1980" -.06
10057 "03JAN1980"  .85
10058 "04JAN1980"  .35
10065 "06JAN1980" -.25
10103 "07JAN1980"  .24
10137 "09JAN1980"  .39
10145 "10JAN1980"  .87
10146 "11JAN1980"  .88
end
gen ndate = daily(date, "DMY")
format %td ndate

* generate a monthly date - please study: help datetime
gen monthdate = mofd(ndate)
format %tm monthdate

* convert ret to a fraction
gen double ret_frac = ret/100

* no product function in Stata so use logs
gen double lret1 = ln(1+ret_frac)
collapse (sum) lret1, by(permno monthdate)

gen ret_monthly = exp(lret1)-1

* show cumulative returns for permno 10006
dis (1+.02/100) * (1 +.03/100) * (1 + -.06/100) - 1
list if permno == 10006

[Jae Li]

@Clyde Schechter Hello Clyde! Thank you so much for your reply! It was my mistake early about the function, my apologies for that! As in order to convert from daily returns to monthly returns, the correct way is to use product symbol rather than sum symbol in that function. Anyway, your codes help me debug the errors. Many thanks for your help!

[Jae Li]

Robert Picard Hi Robert! Many thanks for your comments and codes!

Yes, you are right about using the production function to convert. I was confused about the sum and product signs early!

By the way, do you know how can I calculate the standard deviation of residuals by running multiple regressions in a loop with respect to 5441 firms based on 280 months and store each standard deviation of residual into STATA? I guess that there would be (5441*280) standard deviations of residuals as a result. Any help will be appreciated!!

Here are my codes but the result of monthly_sd_resid is blank for not known reason:

 foreach var in exret mkt_rf smb hml {
 regress exret mkt_rf smb hml
 predict resid, res
 egen monthly_sd_resid= sd(resid), by(permno ym) *permno is the firm index and ym is the months
 }

[Robert Picard]

You seem to be confused about many things and it looks very much like you are looking for quick shortcuts to avoid having to learn the ropes. I already recommended that you take a look at rangestat (from SSC) yet you seem to want to create loops but what you come up with does not make any sense.

As I have already stated, I am not in finance and I have no idea what a Fama-French model eats in the winter. You have not well described your data and I'm still not clear if you need to regress per permno or permno and ym. Here's a quick example that may do what you want and spot checks the residuals for a specific permno.

* create fake data
clear all
set seed 3123
set obs 5441
gen long permno = _n
expand 280
bysort permno: gen ym = _n
gen mkt_rf = runiform()
gen smb = runiform()
gen hml = runiform()
gen exret = runiform()

rangestat (reg) exret mkt_rf smb hml, interval(permno 0 0)
gen resid = exret - (b_mkt_rf*mkt_rf + b_smb*smb + b_hml*hml + b_cons)

* spot check for permno 2
regress exret mkt_rf smb hml if permno== 2
predict rcheck if e(sample), residual
assert rcheck == resid if permno== 2

[Jae Li]

Robert Picard Many thanks for your reply! You are right I was too anxious about getting the right answer for my question. It's been really tough days about debugging the errors for me as a beginner. I should be more patient in learning STATA brick by brick rather than just running without walking beforehand. My apologies for that!

Many thanks for writing the codes for me! It's a very helpful demonstration to my issue!

[Shaquille Wijngaarde]

Robert Picard Many thanks for your reply! You are right I was too anxious about getting the right answer for my question. It's been really tough days about debugging the errors for me as a beginner. I should be more patient in learning STATA brick by brick rather than just running without walking beforehand. My apologies for that!

Many thanks for writing the codes for me! It's a very helpful demonstration to my issue!

Hello Jae Li,

Did you ever manage to figure out how to code this in a way that works? Am also trying to calculate idiosyncratic volatility, (well actually idiosyncratic risk).
Would you mind sharing your code with me, if you succeeded?

[Kate Lussy]

Hi Jae Li, I am also trying to calculate idiosyncratic volatility, but am struggling. Would it be possible to share your code or your methodology? Thank you very much!