Changes in significance from one model to the next and do you interpret the sign/size of an insignificant variable?

[Prathvajeeth Rajmohan]

hi there: I am trying to see the effect of derivative usage (dummy deriv) on firm value (with controls variables), I estimate 3 models:
1)OLS
2)OLS with industry dummies (industry fixed effects)
3) firm fixed effects (xtreg)

1)

 regress firm value logassets  deriv  blev roa currentratio RND cash dyield yeardummy if inlist(year,2014,2015)

2)

 regress firm value logassets  deriv  blev roa currentratio RND cash dyield yeardummy ind1* if inlist(year,2014,2015)

3)

 xtreg firm value  logassets  deriv  blev roa currentratio RND cash dyield yeardummy  if inlist(year,2014,2015), fe

sorry might seem elementary but my question was basically regarding variables changing in significance from model to model, is this normal?

for example in these regressions:

-natlogassets (is positive in all regressions) is significant in the ols, insignificant in the ols with industry dummies and significant in the firm fixed effects.

-while the RND (is postive in all regressions) is significant in ols and ols with industry dummies and insignificant in firm fixed effects.

Is this normal/ to be expected or out of the ordinary?

and could a possible feasible explanation of the natlogassets significance change be that that industry plays a big role in determine firm size (which is what logassets proxies) so when industry is controlled for that effect of firm size on firm value vanishes?

or if a variable is insignificant does that mean you should'nt bother interpreting its sign/size at all?



I have no variables that change sign, I was taught that its only variables that are one sign and significant and then change sign and stay significant in a different model which are causes of concern
but just was curious about these changes in significance aswell.

Any insight would be appreciated. Thanks

[Clyde Schechter]

Well, it is no surprise that the OLS results differ from the -xtreg, fe- results.. There is no reason to expect that they will be the same, or even close, or even have the same signs. The differences can be arbitrarily large, or they can be quite similar; there is no necessary connection between them. The models are radically different.

What is problematic here is that the results of -xtreg, fe- should be identical to the results of -regress ... ind1*-, assuming that ind1* are indicator variables for the levels of the variable that you used to -xtset- your data before running -xtreg, fe-. The fact that they are not suggests to me that you have made a mistake somewhere along that path. So please repost showing all commands and complete output from:

1. Whatever you did to create the ind1* variables.
2. Your -xtset- command
3. The -regress ... ind1*- command
4. The -xtreg, fe- command.

There is no need to repost anything about the -regress- command that did not include firm indicators as it is not problematic.

[Prathvajeeth Rajmohan]

Hi Clyde thanks so much for the reply, sorry in the rush last night I pasted the commands with slighted altered variable names/years, but the actual query is still exactly the same regarding the changes in signficance.

1)OLS

 .  regress lntobinsq lnassets FXDerivatives10 IRDerivatives10  bookleverage_w1 roa_w1 cratio_w1 RND_ASSETS_PERCENT cash_to_totalassets_w1 div_yield_w1 year2016 if inlist(year,2015,2016)

2)OLS with industry dummies

 .  . regress lntobinsq lnassets FXDerivatives10 IRDerivatives10  bookleverage_w1 roa_w1 cratio_w1 RND_ASSETS_PERCENT cash_to_totalassets_w1 div_yield_w1 year2016 ind2* if inlist(year,2015,2016)

3) firm fixed effects (xtreg)

 .  xtreg lntobinsq lnassets FXDerivatives10 IRDerivatives10 bookleverage_w1 roa_w1 cratio_w1 RND_ASSETS_PERCENT cash_to_totalassets_w1 div_yield_w1 year2016 if inlist(year,2015,2016), fe

- lnassets (is positive in all regressions) is significant in the ols, insignificant in the ols with industry dummies, and significant in the firm fixed effects.

-while the RND (is postive in all regressions) is significant in ols and ols with industry dummies and insignificant in firm fixed effects.

I am focusing on firms from 2015-2016. In model 3 xtreg was for firm specific effects and while in model 2 it was the same as the OLS but industry dummies were introduced according to 2 digit SIC codes:



heres the output as requested:

. xtset firmid year
       panel variable:  firmid (weakly balanced)
        time variable:  year, 2013 to 2016
                delta:  1 unit

. gen sic2 = substr(sic_code,1,2)

. tab sic2, gen(ind2)

       sic2 |      Freq.     Percent        Cum.
------------+-----------------------------------
         10 |         34        4.22        4.22
         13 |         40        4.96        9.18
         14 |          4        0.50        9.68
         15 |         30        3.72       13.40
         17 |          2        0.25       13.65
         20 |         24        2.98       16.63
         21 |          4        0.50       17.12
         22 |          2        0.25       17.37
         23 |          6        0.74       18.11
         26 |          6        0.74       18.86
         27 |         26        3.23       22.08
         28 |         48        5.96       28.04
         29 |          8        0.99       29.03
         30 |         10        1.24       30.27
         32 |          6        0.74       31.02
         33 |         10        1.24       32.26
         34 |         14        1.74       34.00
         35 |         18        2.23       36.23
         36 |         26        3.23       39.45
         37 |         14        1.74       41.19
         38 |         36        4.47       45.66
         39 |          6        0.74       46.40
         41 |          8        0.99       47.39
         42 |          4        0.50       47.89
         44 |         14        1.74       49.63
         45 |         10        1.24       50.87
         47 |          6        0.74       51.61
         48 |         32        3.97       55.58
         49 |         24        2.98       58.56
         50 |         20        2.48       61.04
         51 |          8        0.99       62.03
         52 |          6        0.74       62.78
         53 |          8        0.99       63.77
         54 |          6        0.74       64.52
         55 |          8        0.99       65.51
         56 |          6        0.74       66.25
         57 |         12        1.49       67.74
         58 |         24        2.98       70.72
         59 |         24        2.98       73.70
         60 |          2        0.25       73.95
         65 |         20        2.48       76.43
         67 |         20        2.48       78.91
         70 |          4        0.50       79.40
         72 |          6        0.74       80.15
         73 |         98       12.16       92.31
         75 |          2        0.25       92.56
         78 |          6        0.74       93.30
         79 |         16        1.99       95.29
         80 |          2        0.25       95.53
         87 |         34        4.22       99.75
         89 |          2        0.25      100.00
------------+-----------------------------------
      Total |        806      100.00

model 2: old with industry dummies at sic code 2

. regress lntobinsq lnassets FXDerivatives10 IRDerivatives10  bookleverage_w1 roa_w1 cratio_w1 RND_ASSETS_PERCENT cash_to_totalassets_w1 div_yield_w1 year2016 ind2* if inlist(year,2015,2016)



      Source |       SS           df       MS      Number of obs   =       586
-------------+----------------------------------   F(58, 527)      =     28.36
       Model |  137.191495        58  2.36537061   Prob > F        =    0.0000
    Residual |  43.9489022       527  .083394501   R-squared       =    0.7574
-------------+----------------------------------   Adj R-squared   =    0.7307
       Total |  181.140398       585  .309641705   Root MSE        =    .28878

----------------------------------------------------------------------------------------
             lntobinsq |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-----------------------+----------------------------------------------------------------
              lnassets |  -.0164726   .0102322    -1.47   0.142    -.0351735    .0050283
             FXDer10 |   .0028847   .0332299     0.09   0.918    -.0622945     .068264
              IRDer10 |  -.0243612   .0336888    -0.69   0.558     -.089542    .0428196
       bookleverage_w1 |   .0093596   .0706366     0.13   0.851    -.1295042    .1480235
                roa_w1 |   .0749969   .0031074    23.46   0.000     .0667924    .0790013
             cratio_w1 |  -.0472351   .0137777    -3.61   0.000    -.0712365   -.0210336
    RND_ASSETS_PERCENT |   .0078757   .0038096     2.10   0.035     .0005118    .0154796
cash_to_totalassets_w1 |   .3654757   .1767328     2.12   0.033     .0282884    .7226631
          div_yield_w1 |  -.0523367    .006331    -8.19   0.000    -.0642738   -.0393996
              year2016 |  -.01487409   .0240481    -0.62   0.535    -.0621829     .032301
                 ind21 |  -.5005649   .2991703    -1.67   0.095    -1.088278    .0871479
                 ind22 |  -.4624794   .3026261    -1.53   0.127    -1.056981    .1320221
                 ind23 |  -1.071256   .3587639    -2.99   0.003    -1.776039   -.3664728
                 ind24 |  -.3003776   .2976871    -1.01   0.313    -.8851766    .2844214
                 ind25 |  -.2156552   .3577936    -0.60   0.547     -.918532    .4872216
                 ind26 |  -.0192405   .2994194    -0.06   0.949    -.6074427    .5689617
                 ind27 |   .1150536   .3284582     0.35   0.726    -.5301946    .7603018
                 ind28 |  -.4655083   .3586442    -1.30   0.195    -1.170056    .2390395
                 ind29 |   .4001954   .3177517     1.26   0.208      -.22402    1.024411
                ind210 |  -.1931448   .3261847    -0.59   0.554    -.8339267    .4476372
                ind211 |  -.2545326   .2989492    -0.85   0.395    -.8418111    .3327458
                ind212 |   .0361889    .296574     0.12   0.903    -.5464234    .6188013
                ind213 |  -.0063025    .319818    -0.02   0.984    -.6345772    .6219723
                ind214 |  -.0202293    .306205    -0.07   0.947    -.6217616    .5813029
                ind215 |   -.304089   .3266701    -0.93   0.352    -.9458243    .3376464
                ind216 |  -.2097631   .3166432    -0.66   0.508    -.8318009    .4122747
                ind217 |  -.3755175   .3063212    -1.23   0.221    -.9772781    .2262432
                ind218 |    .086898   .3052206     0.28   0.776    -.5127004    .6864964
                ind219 |  -.0439694   .3035126    -0.14   0.885    -.6402125    .5522736
                ind220 |  -.0035779   .3049097    -0.01   0.991    -.6025656    .5954097
                ind221 |  -.1587454   .2979246    -0.53   0.594    -.7440111    .4265203
                ind222 |  -.1457999   .3273205    -0.45   0.656    -.7888131    .4972133
                ind223 |  -.2651167   .3154939    -0.84   0.401    -.8848969    .3546634
                ind224 |  -.5117701   .3372122    -1.52   0.130    -1.174215     .150675
                ind225 |  -.3961064    .306157    -1.29   0.196    -.9975443    .2053315
                ind226 |  -.3334756   .3109793    -1.07   0.284    -.9443868    .2774356
                ind227 |  -.1227103   .3355284    -0.37   0.715    -.7818477    .5364271
                ind228 |  -.0188824   .2992331    -0.06   0.950    -.6067186    .5689538
                ind229 |  -.1643805   .3010787    -0.55   0.585    -.7558424    .4270813
                ind230 |  -.1484585   .2994224    -0.50   0.620    -.7366666    .4397496
                ind231 |  -.0387996   .3091865    -0.13   0.900     -.646189    .5685897
                ind232 |  -.2579017     .31541    -0.82   0.414    -.8775169    .3617135
                ind233 |  -.4382912   .3127356    -1.40   0.162    -1.052653    .1760702
                ind234 |  -.2403848   .3204628    -0.75   0.454    -.8699261    .3891565
                ind235 |   -.415201   .3108316    -1.34   0.182    -1.025822    .1954202
                ind236 |  -.0164647   .3291576    -0.05   0.960    -.6630868    .6301574
                ind237 |  -.2848765   .3148761    -0.90   0.366     -.903443    .3336899
                ind238 |  -.2659692   .2978472    -0.89   0.372    -.8510827    .3191443
                ind239 |  -.1397646   .3047713    -0.46   0.647    -.7384805    .4589512
                ind240 |          0  (omitted)
                ind241 |  -.3941941   .3025561    -1.30   0.193    -.9885583    .2001701
                ind242 |  -.4472723   .2987312    -1.50   0.135    -1.034122    .1395779
                ind243 |    -.57244   .3366824    -1.70   0.090    -1.233844    .0889644
                ind244 |  -.2404855   .3247416    -0.74   0.459    -.8784325    .3974615
                ind245 |  -.0540642   .2935718    -0.18   0.854    -.6307788    .5226503
                ind246 |    -.60266   .3550028    -1.70   0.090    -1.300054    .0947344
                ind247 |          0  (omitted)
                ind248 |  -.1494382   .3071157    -0.49   0.627    -.7527595    .4538831
                ind249 |          0  (omitted)
                ind250 |  -.2090572   .2973058    -0.70   0.482    -.7931073    .3749928
                ind251 |  -.3388991   .3586909    -0.94   0.345    -1.043539    .3657404
                 _cons |   .5946361   .2949526     2.02   0.044      .015209    1.174063
----------------------------------------------------------------------------------------

firm fixed effects model:

 xtreg lntobinsq lnassets FXDerivatives10 IRDerivatives10 bookleverage_w1 roa_w1 cratio_w1 RND_ASSETS_PERCENT cash_to_totalassets_w1 div_yield_w1 year2016 if inlist(year,2015,2016), fe

Fixed-effects (within) regression Number of obs = 586
Group variable: firmid Number of groups = 306

R-sq: Obs per group:
within = 0.3784 min = 1
between = 0.1894 avg = 1.9
overall = 0.1995 max = 2

F(10,270) = 16.44
corr(u_i, Xb) = -0.7252 Prob > F = 0.0000

----------------------------------------------------------------------------------------
             lntobinsq |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-----------------------+----------------------------------------------------------------
              lnassets |  -.3270572   .0650003    -6.49   0.000    -.5500291   -.2940854
          FXDeriv10 |   .0853074     .06897     1.09   0.264      -.06048    .2110949
           IRDeriv10 |  -.0871809   .0699328    -1.30   0.183    -.2288737    .0464919
       bookleverage_w1 |    .172826    .109698     1.48   0.150    -.0536462    .3782981
                roa_w1 |   .0172843   .0040619     4.00   0.000     .0082573    .0242513
             cratio_w1 |  -.0870082   .0152594    -4.06   0.000    -.0920507   -.0319656
    RND_ASSETS_PERCENT |   .0062209   .0103128     0.50   0.620    -.0151829    .0254246
cash_to_totalassets_w1 |   .2367015   .1961116     1.15   0.261    -.1604009    .6118039
          div_yield_w1 |  -.0441328   .0047986    -9.20   0.000    -.0535802   -.0346855
              year2016 |   .0163724   .0130394     1.26   0.259    -.0092995    .0420442
                 _cons |   3.610123   .4529874     7.97   0.001     2.718286    4.501959
-----------------------+----------------------------------------------------------------
               sigma_u |  .72261949
               sigma_e |  .11846442
                   rho |  .97382789   (fraction of variance due to u_i)
----------------------------------------------------------------------------------------

My main concern was the lnassets shifting in significance, but as ind1* are dummy variables based on 2 digit sic codes in model 2, while the panel was xstet on the level variable firm id, I presume this is not be of concern, but please let me know.

Thanks so much, eagerly waiting for your reply.

[Carlo Lazzaro]

Prash:
after a fast scan of your OLS code, you omitted in both of them to cluster the standard errors on -panelid- (since your observations are not independent): that makes your results biased.
Besides, when compared to your sample size, you seem also to have a number of predictors which appraoches (or even exceeds) the limit of what you can get out of your data.
Setting aside for a while my first remark, you have a pretty high F-test but only a negligible handful of coefficient reaches statistical significance: I would intepret that feature as a sign of quasi-perfect multicollinearity. Did you check it via -estat vif-?
That said, I would rarely consider a pooled OLS for dealing with a panel dataset.
In your case, the F-test appearing at the feet of the -xtreg- outcome table tells you that -xtreg- outperforms pooled OLS.

[Prathvajeeth Rajmohan]

HI Carlo thanks I completely understand, but as I've explained before I need to show with each of these methods as its the evolution of the process, all I wanted to know was if the significance changes in lnasset from significance to ols to insignificance in ols with industry dummies and the back to signifiance in fe is out of the ordinary generally? or not as they are 3 different models.

[Prathvajeeth Rajmohan]

Prash:
after a fast scan of your OLS code, you omitted in both of them to cluster the standard errors on -panelid- (since your observations are not independent): that makes your results biased.
Besides, when compared to your sample size, you seem also to have a number of predictors which appraoches (or even exceeds) the limit of what you can get out of your data.
Setting aside for a while my first remark, you have a pretty high F-test but only a negligible handful of coefficient reaches statistical significance: I would intepret that feature as a sign of quasi-perfect multicollinearity. Did you check it via -estat vif-?
That said, I would rarely consider a pooled OLS for dealing with a panel dataset.
In your case, the F-test appearing at the feet of the -xtreg- outcome table tells you that -xtreg- outperforms pooled OLS.

I ran the inital OLS and tried estat vif and the doesnt seem to be any multicollinriy: the mean vif is 1.8 which is means very little chance of multicollinarity right? Thanks

[Carlo Lazzaro]

Prash:
if the average VIF is 1.8, there's no problem of quasi-perfect multicollinearity.
However, the clustered standard errors issue still remains.

[Clyde Schechter]

Looking back at #3, and the preceding discussion, I note that the ind2* variables were created as indicators for sic2 which, I understand, is an industry level code. By contrast, -xtset- uses firms for the panel variables. Consequently, there is no reason to expect that -regress ... ind2*-'s results will bear any necessary relationship to -xtreg, fe- as a completely different set of fixed effects is being used.

This kind of confusion is the inevitable result of trying to take 3 (or more) -level data and apply the procrustean bed of panel data analysis. There is no entirely valid panel-data analysis of 3-level data: something always has to be wrong. None of the three models in the original post is a correct specification of the underlying process and all estimates derived from them will be wrong in some way. In my opinion, using a 3-level mixed effects model is a better approach, though it too has the limitation that if there are unmeasured variables that influence the outcome and correlate with the predictors, estimation may not be consistent. This is, unfortunately, a situation for which no ideal solution exists.

[Prathvajeeth Rajmohan]

Looking back at #3, and the preceding discussion, I note that the ind2* variables were created as indicators for sic2 which, I understand, is an industry level code. By contrast, -xtset- uses firms for the panel variables. Consequently, there is no reason to expect that -regress ... ind2*-'s results will bear any necessary relationship to -xtreg, fe- as a completely different set of fixed effects is being used.

Hi thanks so much, just to clarify does this mean that for a reader its nothing out of the ordinary in terms of the ln assets becoming insignificant going from the ols to ols with industry dummies right?, as you could argue that its the result of controlling for industry fixed effects. Thanks

[Clyde Schechter]

That's right, it is nothing out of the ordinary at all.

One shift in emphasis: the statistical significance changing is even less meaningful than everything else here. Don't focus on that, it's a distraction. The real meat is in the coefficients and their confidence intervals. And when you change the model, everything is subject to change, often in quite radical ways. Statistical significance is even less stable than anything else, and can change even when the most minimal changes in the data or the modeling occur even while other things stay more or less the same.

The problem with focusing on statistical significance is that it is a (false) dichotomy: effect vs no effect detected. But in the real world, there is hardly ever a situation where there is no effect of one thing on another, and those situations are usually not the subject of research investigations anyway. By applying an arbitrary statistical significance criterion, you can even have two sets of results that are nearly the same in all other respects, but where the p-value flips from just one side of the .05 line to just the other side. That means nothing at all. Significant vs not-significant is a misleading way to think about effects. A better way to think about them is to consider how large they appear to be and how much uncertainty we attach to our estimate of how large they are. These are continuous and more reflective of reality. So when comparing the results of two analyses, the last thing you should look at is whether significance changes--indeed you probably shouldn't look at that at all. The thing to look at is whether the models give similar estimates of the coefficients and the confidence intervals.

That said, once you introduce new variables into a model, everything can change in any way you can imagine. Never expect the results from two different models of the same outcome to produce similar results. (It is sometimes reasonable to expect models of two highly correlated outcomes with the same predictor variables in both models to produce similar results. That's a different question. But even there, it is not guaranteed.)

[Prathvajeeth Rajmohan]

That's right, it is nothing out of the ordinary at all.

One shift in emphasis: the statistical significance changing is even less meaningful than everything else here. Don't focus on that, it's a distraction. The real meat is in the coefficients and their confidence intervals. And when you change the model, everything is subject to change, often in quite radical ways.

Thanks so much Clyde, sorry what I was meant to clarify was: If I presented these regressions in the same table for example for a examiner there's nothing out of the ordinary in terms of the ln assets becoming insignificant going from the ols to ols with industry dummies, and then back to significant in the firm fixed effects regression right?,

as its simply the result of controlling for different fixed effects (industry/firm)?



Sorry to ask again, I've literally been breaking head today over this issue of issue for the entire day, hence why I am so worried.

Thanks for being so helpful.

[Clyde Schechter]

If I presented these regressions in the same table for example for a examiner there's nothing out of the ordinary in terms of the ln assets becoming insignificant going from the ols to ols with industry dummies, and then back to significant in the firm fixed effects regression right?,

as its simply the result of controlling for different fixed effects (industry/firm)?

Absolutely right.

[Prathvajeeth Rajmohan]

Absolutely right.

Thanks so so much, really.

And I presume that its the same for the RND which is significant in both OLS and OLS dummies model and then insignificant in the firm fixed effects model (xtreg regression), this is completely fine and would'nt seem odd to an examiner?

Sorry to ask so many (elementary) questions just very new to this and quite nervous. King regards, again really appreciate all the help ive been getting from the forum.

[Prathvajeeth Rajmohan]

That's right, it is nothing out of the ordinary at all.

One shift in emphasis: the statistical significance changing is even less meaningful than everything else here. Don't focus on that, it's a distraction. The real meat is in the coefficients and their confidence intervals. And when you change the model, everything is subject to change, often in quite radical ways. Statistical significance is even less stable than anything else, and can change even when the most minimal changes in the data or the modeling occur even while other things stay more or less the same.

The problem with focusing on statistical significance is that it is a (false) dichotomy: effect vs no effect detected. But in the real world, there is hardly ever a situation where there is no effect of one thing on another, and those situations are usually not the subject of research investigations anyway. By applying an arbitrary statistical significance criterion, you can even have two sets of results that are nearly the same in all other respects, but where the p-value flips from just one side of the .05 line to just the other side. That means nothing at all. Significant vs not-significant is a misleading way to think about effects. A better way to think about them is to consider how large they appear to be and how much uncertainty we attach to our estimate of how large they are. These are continuous and more reflective of reality. So when comparing the results of two analyses, the last thing you should look at is whether significance changes--indeed you probably shouldn't look at that at all. The thing to look at is whether the models give similar estimates of the coefficients and the confidence intervals.

That said, once you introduce new variables into a model, everything can change in any way you can imagine. Never expect the results from two different models of the same outcome to produce similar results. (It is sometimes reasonable to expect models of two highly correlated outcomes with the same predictor variables in both models to produce similar results. That's a different question. But even there, it is not guaranteed.)

Hi Clyde thanks so much just seen your recent edit.

Could I ask though about another general case and if this applies i.e if RND +ve and significant in OLS and OLS with industry dummies, and then becomes -ve but Insignificant in the firm fixed effects model is this a cause for concern or does the -ve sign switch not matter as its ultimately insignificant. Thanks

[Clyde Schechter]

No change in statistical significance from OLS to OLS with industry FE and FE regression with firm FE is meaningful. It is no cause for concern. Don't even give it another thought. When you change the predictors in a model, anything can happen!