Reporting on models with interaction terms

[Emeka Dim]

Hi,

Please, I would like to ask a question about the reporting of interaction models with main effects and square terms. I am running a multilevel analysis for protest behaviour (demons). I am interested in the cross-level interactions between gender (female), which is a binary individual-level variable, and the number of years a country has been a democracy (dem_yrs), a continuous country-level variable. I include control variables like "urban" "resp_age" "log_gni" "fem_hdi." For instance, I have the following models:

Model A - melogit demons_bi i.female c.dem_yrs urban resp_age log_gni fem_hdi || countrylvl:female, cov(unstr)

Model B - melogit demons_bi i.female c.dem_yrs##c.dem_yrs urban resp_age log_gni fem_hdi || countrylvl:female, cov(unstr)

Model C - melogit demons_bi i.female##c.dem_yrs##c.dem_yrs urban resp_age log_gni fem_hdi || countrylvl:female, cov(unstr)

Model D - melogit demons_bi i.female##c.dem_yrs urban resp_age log_gni fem_hdi || countrylvl:female, cov(unstr)


As I said earlier, I am mainly interested in the interaction effects between "female" and "dem_yrs." If models A, B, and D are significant, i.e. only the interaction of gender and the square-function of years of being a democracy is not significant, can I still report Model D? Is Model D considered as misspecified?

[Clyde Schechter]

If you are, as you say, mainly interested in the interaction effects between female and dem_yrs, then models A and B are irrelevant because they omit the interaction of interest. Relying on the statistical significance of a term is pretty much the worst way to decide whether a term should be included in the model. If you show the full output from models C and D, it would be easier to give you concrete advice. But bear in mind that, regardless of what the output shows, if there is good reason, based on the substantive science, to believe that the relationship with years is quadratic, and that the quadratics differ by gender, then the female#c.dem_yrs#c.dem_yrs term should be retained in the model, even if its coefficient is dead zero. If there is no reason to believe that, then the model definitely should not contain that, unless your point is to try to overthrow existing theory. If theory is silent on the issue, then a decision based on the outputs may be reasonable.

[Emeka Dim]

Thank you for your reply.

My research is informed by theory. However, that theory is silent on this issue, especially on the nature of the relationship between the variables I am interested in. Please, in this case, I replaced dem_yrs with state_leg_fsi, which means the extent to which a state is legitimate. Nevertheless, the issue I need answers to is basically the same, i.e., is it okay to report Model D or is it misspecified?

Here are the models:
Model C:

Model D:

Thus, I simply want to know if Model D is misspecified, given that Model B had the non-linear value of state_leg_fsi as significant.

So, please, what do you think?

I can provide the remaining values for these models. I had to cut them so as to fit the table into one page. I can also provide models for A and B if you would like to see it.

Thank you and looking forward to your reply.

Eugene.

[Clyde Schechter]

So, in going from model C to model D you have dropped not only the female#c.state_leg_fsi#c.state_leg_fsi interaction, but the entire quadratic portion of the model. My approach to whether this is OK is based on the turning points in the quadratic equations produced. When a quadratic regression model gives you an equation y = a*x^2 + b*x + c, the vertex of the parabola (turning point) lies at x = -b/2a, as you may recall from high school algebra. If that turning point lies far outside the range of useful values of x, then, for practical purposes there is no turning point in the useful range of the data and the model is, for all intents and purposes linear. If however, the turning point lies within the range of useful values of x, or just beyond that range, then the model is clearly curved in the useful range of data and a model lacking a quadratic term would be mis-specified.

In Model C, for males, we have b= -0.0393596 and a = 0.005059, which puts the male vertex at state_leg_fsi = 3.89 (to two decimal places). For females, we have b = -.0393596-0.066613, and a = 0.005059 + 0.0035836, which puts the female vertext at state_leg_fsi = 6.13. Now, I have no idea what the range of meaningful and useful values of your state_leg_fsi variable is. If either of 3.89 or 6.13 falls in or near that range, then you should definitely retain the quadratic term and female#quadratic interaction. If both of them are remote from that range, e.g. if state_leg_fsi ranges from 0 to 1, then you can use Model D instead.

[Emeka Dim]

Clyde Schechter Thank for your response. Please, for the part where you said: "Now, I have no idea what the range of meaningful and useful values of your state_leg_fsi variable is." Please, what do you mean by the range of meaningful and useful values of your state_leg_fsi? I can provide you with the descriptive statistics of that variable, please see below

If the picture above is not what you are referring to, please let me know so that I can confirm the model that I can report.

You also said: "If either of 3.89 or 6.13 falls in or near that range, then you should definitely retain the quadratic term and female#quadratic interaction. If both of them are remote from that range, e.g. if state_leg_fsi ranges from 0 to 1, then you can use Model D instead." Please, do you mean that if the values of state_leg_fsi falls within 3.89 and 6.13, then I should not report Model D?

Thank you.

[Clyde Schechter]

The word meaningful is related to, but not the same as, statistical considerations. Sometimes values near the extremes of the observed range of a variable are, whether or not they are outliers in the formal statistical sense, nonetheless represent situations that are unusual, perhaps pathological, and we don't really need our model to handle them well. Of course, the best solution in that case is to exclude them from the model-development analysis in the first place. Suppose under normal circumstances a certain variable ranges between 0 and 5, but there happens to be a small number of observations where it is recorded as 11, or -3. Assuming these are not simply data errors, we might, nevertheless, not want our model to be strongly influenced by these observations because they represent such strange or unusual conditions. In fact, the best solution in this case is to remove them from the modeling analysis and simply state outright that our model only applies to x between 0 and 5. But, if for other reasons removing them from the analysis did not make sense, you could consider that they are not meaningful values for your purposes, and you would not want to be pushed into fitting a quadratic model that had its vertex at, say, 10.5 just to accommodate them. Meaningful is not a statistical term: it is a term about the substance of the data and is, in the end, a value judgment that can only be made by people with expertise in the underlying real-world processes. It is all too often ignored: many people are comfortable with the technical wizardry of algorithms and computers and forget that the purpose of doing statistics in the first place is to understand real things happening in the real world. Things that happen in statistics but don't relate to the real world are not meaningful, in the sense I'm using here.

In any case, looking at the statistics you show, state_leg_fsi ranges from 3.2 to 9.9, and none of those values appears to be an outlier or stand out in anyway. So I think this is the (fairly typical) case where the meaningful range coincides with the observed range. Consequently, turning points at 3.89 and 6.13 indicate that there is real "quadraticness" the needs to be kept in the model. I would not report Model D because it omits key terms and is a mis-specification of the relationships.

By the way, as a general point, the fact that so many people erroneously believe that only statistically significant results should be reported is powerful testimony to how badly statistics is taught in so many places.

[Emeka Dim]

Thank you for your response and clarification.

I had run into this kind of problem earlier, and I did not want to hide the non-linear effect in the models for sake of transparency and curiosity. However, this kind of issue is littered in my results and I have a fear that many of my results would have this issue. Thus, are you suggesting that even if the theory is silent on this issue and I were to see these turning point values within the meaningful range of the variable, I have to report the model with the non-linear effect (i.e. Model C) and reject Model D?

Also, I have another case I would want you to look at. It is similar to the one I presented before, however, I calculated the turning points and they were between 17.7 and 18.7 and this implies that I must reject the linear model with the non-quadratic variable. Now, please I may be wrong in my calculation and I was hoping if you could take a look at the model to confirm if I am right or wrong.

Similar to the first case, I was wondering if my calculation is right.

Furthermore, I notice that if I add weights to the non-linear models, they tend to turn up insignificant. Do you think the weights should be taken seriously?

Thank you.

[Clyde Schechter]

I can confirm your calculations of the turning points, and I concur that this data clearly suggests a quadratic model, not a linear one.

I notice that if I add weights to the non-linear models, they tend to turn up insignificant. Do you think the weights should be taken seriously?

Weights must reflect the actual sampling design. They are not something you can choose to use or not use. If your sample was selected in such a way that it is a simple random sample of the population to which you wish to generalize your results, then you must not use weights. If it was a complex sample, with different people (firms, countries, whatever) had different ex ante probabiities of being selected into the sample, then you must use sampling weights or post-stratification weights, or some other kind of weighting scheme in order to get unbiased estimates.

There are other types of weights besides sampling weights. You can have analytic weights: this arises when the nature of the data collection is such that different observatoins' outcomes have different precision (typically because the outcome variable is an average of a number of measurements, and the number of measurements differs across observations.) Those weights primarily affect variance estimates, rather than bias. So, in a sense, they are somewhat optional if you don't care much about precision.

What Stata calls fweights are not really weights, they are more a way of unpacking a collapsed dataset. But these are never optional when they exist.

[Emeka Dim]

Thank you for your response.

Please, is there a citation or article you can suggest for what you said about the choice of the non-linear models. I am trying to re-run many of the models that I have because a majority of them fall within this non-linear situation. Also, it seems to me that it is almost in every case that the turning points of the quadratic values will fall within the meaningful ranges of the country-level variable. My question is that if this quadratic model has not been suggested in theory, which is so in my case, but only seen in the data output, can one still report the linear models?

Also, would recoding the continuous variables into categorical variables be an alternative route to reporting the linear model?

For the weights, the institution that gathered the data I am using suggested that weights must be applied at the descriptive level. The institution says: "“Withinwt” (the within country weights) should be turned on for all national–level descriptive statistics in countries that contain this weighting variable." The sample design is a clustered, stratified, multi-stage, area probability sample. Thus, I do not know if it will be necessary to apply the weights to the multivariate analysis.

[Clyde Schechter]

I do not have a reference for my approach to the choice of non-linear models. You seem uncomfortable with it. In your case, even though I dislike using statistical significance of terms for modeling selections, let me point out that the c.mil_yrs#c.mil_yrs and c.state_leg_fsi#c.state_leg_fsi terms are strongly statistically significant in these models. So even by that awful criterion, you need the quadratic terms.

What you might, in other circumstances, not need are the i.female#c.x#c.x terms and i.female#c.x terms. The data are clearly screaming at you that the relationships to outcome of mil_yrs and state_leg_fsi are quadratic, not linear. But they are not providing very much support for those relationships differing by gender. That, of course, is your point of interest according to what you said in #1. So you have to leave those terms in to test that. Let me just point out that in your hypothesis test for that, you must include both the linear and quadratic interaction terms with female in a joint test. For example

test 1.female#c.x 1.female#c.x#c.x // REPLACE x BY state_leg_fsi OR mil.yrs

You cannot draw inferences properly based only on the separate significance of those coefficients: only the joint test is legitimate.

My question is that if this quadratic model has not been suggested in theory, which is so in my case, but only seen in the data output, can one still report the linear models?

You said earlier that previous theory is silent on the issue. If there were good prior evidence against a non-linear relationship, then we might be inclined to regard your results as just a fluke and rely on strong prior evidence. But you say there is no prior evidence. So your data are, at this point, the strongest evidence that can be brought to bear on the matter. On top of that, I assume you didn't throw in the quadratic terms just so you could complicate your life. Did you not have some reason for including them? So why are you so doubtful?

Also, would recoding the continuous variables into categorical variables be an alternative route to reporting the linear model?

That may get an award for worst idea of the month. :-) First of all, categorizing inherently continuous variables is a bad idea in almost any circumstance. It throws away information and inherently mis-specifies almost any model it is a part of. It is justifiable only when something truly discontinuous happens to the outcome when the predictor crosses a cutpoint used to define the categories. On top of that, in this case, monkeying around with different models in search of particular results that you want just isn't science. Some people even consider it scientific misconduct.

The sample design is a clustered, stratified, multi-stage, area probability sample. Thus, I do not know if it will be necessary to apply the weights to the multivariate analysis.

It is absolutely necessary. I have no idea why the authors referred only to descriptive statistics in their statement, but there is no difference between descriptive statistics and multivariate analyses in this regard. In fact, you not only need to use the weights. You need to deal with the stratification and clustering as well: otherwise all your standard errors (and the test statistics, p-values, and confidence intervals that rely on them) are wrong. The -svyset- command is Stata's tool for dealing with all of these things in one line of code. You will need to read the PDF chapter on the -svyset- command to learn about it, and you will have to carefully review the information that came with your survey data set to find which variables in the data represent what parts of the -svyset- command. One you have -svyset- your data, your analyses will have to be run using the -svy:- prefix. See -help svy estimation- for more information about this.

[Emeka Dim]

Thank you. I ran a test as you suggested and below is what I found:

test 1.female#c.state_leg_fsi 1.female#c.state_leg_fsi#c.state_leg_fsi

( 1) [ep_tot2]1.female#c.state_leg_fsi = 0
( 2) [ep_tot2]1.female#c.state_leg_fsi#c.state_leg_fsi = 0

chi2( 2) = 10.44
Prob > chi2 = 0.0054

I also ran a lrtest for the following models:
Model 1: xtmixed ep_tot2 i.female##c.state_leg_fsi##c.state_leg_fsi urban resp_age employed i.educ i.partisan i.volun dep_index i.wv log_gni fem_hdi || countrylvl:female, cov(unstr)

Model 2: xtmixed ep_tot2 i.female##c.state_leg_fsi urban resp_age employed i.educ i.partisan i.volun dep_index i.wv log_gni fem_hdi || countrylvl:female, cov(unstr)

Model 3: xtmixed ep_tot2 i.female##c.state_leg_fsi urban resp_age employed i.educ i.partisan i.volun dep_index c.state_leg_fsi##c.state_leg_fsi i.wv log_gni fem_hdi || countrylvl:female, cov(unstr)

. lrtest (model 1) (model 2)
Likelihood-ratio test LR chi2(2) = 27.90
(Assumption: model_e nested in model_f) Prob > chi2 = 0.0000

. lrtest (model 3) (model 1)
Likelihood-ratio test LR chi2(1) = 2.41
(Assumption: model_h nested in model_f) Prob > chi2 = 0.1207

Please, let me know what you think? You also said that "only the joint test is legitimate." Please, what would you mean by that? Is there a test one can use to compare the models other than the lrtest?

Regarding the weights, I have checked the manual of the dataset and it says nothing about the use of the -syvset-. I know what it is and I have used it before with the Canadian General Social Survey datasets.

You said: "That may get an award for worst idea of the month. :-) ...On top of that, in this case, monkeying around with different models in search of particular results that you want just isn't science. Some people even consider it scientific misconduct." As a African, I am very offended by that statement. I will assume that you never knew I was an African. I have always respected your opinion through this thread. You have advised me in the past and I really appreciate it. I humbly asked your opinion on this issue because I concede that you know better than I do, and I am willing to learn from you. I had informed a friend about this problem and he asked me why I bothered about it when I could simply not talk about it or ignore the quadratic term in my models. I have seen many other papers in my area of research interest ignore the quadratic term of their main independent variables. However, I told him that I inquired about the effect of the quadratic term by reason of personal transparency and curiosity. When I see issues similar to what I am dealing with now, I like to ask questions. From where I come from, there is no such thing as a bad question; a question simply needs an answer and the person asking should learn from the answer. This analysis is for my doctoral dissertation, and I am tempted to ignore the quadratic term to complete my thesis. However, seeing the effect of the quadratic term in model B and trying to be silent about it did not sit well with me, hence, why I am here on this platform. Thus, implying scientific misconduct due to me trying to find another possible way (i.e. recategorizing the variable) to tell the story I have seen in my dataset is deeply offensive, Professor. I simply asked a question, and a simple answer would have sufficed. I do not mind not releasing any result from this analysis I am currently doing, as long as I have substantial reason, evidence, and scholarly material to support it.

Thank you and looking forward to your reply. Once again, I really appreciate your advice and support thus far, and I still hope to learn more from you.

Eugene.

[Clyde Schechter]

You said: "That may get an award for worst idea of the month. :-) ...On top of that, in this case, monkeying around with different models in search of particular results that you want just isn't science. Some people even consider it scientific misconduct." As a African, I am very offended by that statement. I will assume that you never knew I was an African.

You are correct that I never knew you were African. For what it's worth, to the extent I considered it at all, I had imagined you to be German or Dutch based on your username, which sounds Germanic or perhaps Scandinavian to my ear. In any case, it had never occurred to me that the expression "monkeying around with different models" would be offensive to anybody. But such expressions, innocent in earlier times and originally having nothing to do with race, ethnicity, or nationality, have acquired new, negative connotations. I meant no offense, and I apologize for giving it.

From where I come from, there is no such thing as a bad question;

I agree there is no such thing as a bad question. But there are bad data practices that are in common use, and turning continuous variables into categories is one of the most egregious. It also comes up frequently here on this Forum, and I suppose I sometimes get weary of saying over and over again what a bad idea it is. Nevertheless, my way of saying it here was flippant and unwarranted; it was meant to be light-hearted, but it did not come across that way. So, another apology due, and offered.

This analysis is for my doctoral dissertation, and I am tempted to ignore the quadratic term to complete my thesis. However, seeing the effect of the quadratic term in model B and trying to be silent about it did not sit well with me, hence, why I am here on this platform. Thus, implying scientific misconduct due to me trying to find another possible way (i.e. recategorizing the variable) to tell the story I have seen in my dataset is deeply offensive, Professor. I simply asked a question, and a simple answer would have sufficed. I do not mind not releasing any result from this analysis I am currently doing, as long as I have substantial reason, evidence, and scholarly material to support it.

My perception of the thread as a whole was the opposite of what you state here. It had seemed to me that despite a few explanations of why the quadratic terms belong in the model, you were looking for some way to avoid using them. I took your persistence in that vein as being motivated by a desire to reach some pre-specified conclusion rather than acknowledging what the data support. I was wrong to attribute any motive--as I am fond of saying here in other contexts, I am not telepathic. I did not mean to imply that you were knowingly or intentionally committing scientific misconduct. In fact, I thought you might not be aware that it can be considered scientific misconduct, by some people, in some circumstances. Statistics is often badly taught, and ethical issues are seldom discussed in statistics classes. Consequently, there are many people out there, and not just in early stages of their careers, who sincerely believe that the way you do research is gather up some data, with little attention to its quality, and keep changing models and transforming variables until you get a p < 0.05 in some purported test of the hypothesis you have already made up your mind is true. So, if there is offense to be taken from this remark, it should be that I attributed ignorance, not malevolence, to you. In any case, I meant no offense of either kind--I just wanted to make a very strong point about a really bad, and really widespread, practice in research, one that is, in my view, a major contributor to a general decline in the quality of scientific publications and public trust in science as a whole. So I feel really strongly about this issue, and I react strongly when I see it, or think I see it. But, again, I intended to be helpful, not to give offense.

test 1.female#c.state_leg_fsi 1.female#c.state_leg_fsi#c.state_leg_fsi

( 1) [ep_tot2]1.female#c.state_leg_fsi = 0
( 2) [ep_tot2]1.female#c.state_leg_fsi#c.state_leg_fsi = 0

chi2( 2) = 10.44
Prob > chi2 = 0.0054

[quote]I also ran a lrtest for the following models:
Model 1: xtmixed ep_tot2 i.female##c.state_leg_fsi##c.state_leg_fsi urban resp_age employed i.educ i.partisan i.volun dep_index i.wv log_gni fem_hdi || countrylvl:female, cov(unstr)

Model 2: xtmixed ep_tot2 i.female##c.state_leg_fsi urban resp_age employed i.educ i.partisan i.volun dep_index i.wv log_gni fem_hdi || countrylvl:female, cov(unstr)

Model 3: xtmixed ep_tot2 i.female##c.state_leg_fsi urban resp_age employed i.educ i.partisan i.volun dep_index c.state_leg_fsi##c.state_leg_fsi i.wv log_gni fem_hdi || countrylvl:female, cov(unstr)

. lrtest (model 1) (model 2)
Likelihood-ratio test LR chi2(2) = 27.90
(Assumption: model_e nested in model_f) Prob > chi2 = 0.0000

. lrtest (model 3) (model 1)
Likelihood-ratio test LR chi2(1) = 2.41
(Assumption: model_h nested in model_f) Prob > chi2 = 0.1207

Please, let me know what you think? You also said that "only the joint test is legitimate." Please, what would you mean by that? Is there a test one can use to compare the models other than the lrtest?[/quote]

Good. So, if we go by hypothesis testing, the first is the joint test of the female interaction terms with state_leg_fsi, and the joint test does reject a hypothesis of no interaction. This argues in favor of keeping the gender interaction terms in the model. This particular test is a Wald test, not a likelihood-ratio test. It is possible to do a likelihood ratio test for this as well: you simply run the model with and without those terms included, and then compare those models with a likelihood-ratio test. From the likeilhood tests you show in this excerpt, it appears you already know that. There are differences between Wald and likelihood-ratio tests, but asymptotically they are equivalent, and given your large sample size, I think the difference between them can be ignored here. The Wald test is simpler to do (just the -test- or -testparm- command) and does not require running two separate models.

Concerning the subsequent likelihood ratio tests comparing model 1 with model 2 and then with model 3, the model 1 vs model 2 test clearly rejects the adequacy of model 2, and again supports the retention of the quadratic terms. The one comparing model 3 with model 1 is problematic. In model 3, you have an interaction of i.female with the linear term in state_leg_fsi, and then you add to the model a quadratic term in stage_leg_fsi, but do not allow i.female to interact with the quadratic term. It is an understandable impulse to do this in light of the non-significance of only the i.female#c.stage_leg_fsi#c.state_leg_fsi term in the outputs shown in #3. But it is a mistake: any model where you have a quadratic term should always contain the linear term as well (which model 3 does), but when any other variable interacts with either the linear or quadratic term, that variable must also interact with the other (which model 3 fails to do). This is actually close to what I had in mind when I said that "only the joint test is legitimate." My meaning is that in quadratic models, both the linear and quadratic term should be there, and anything that involves either must also involve the other: interactions, tests, etc. And just as it is a mistake to interact i.female with the linear term but not the quadratic, it is also a mistake to do a separate significance test (whether Wald, likelihood-ratio, or any other kind) that involves either without the other.

Regarding the weights, I have checked the manual of the dataset and it says nothing about the use of the -syvset-. I know what it is and I have used it before with the Canadian General Social Survey datasets.

Good. Don't get me wrong, I do think that when working with survey data, it often makes sense to first explore the data ignoring the survey design, just to get a feel for it, and to confront any convergence issues that might arise with models that are fit iteratively in a simpler context. But ultimately for "production," the survey design has to be taken into account.

One totally minor point. Since version 13, -xtmixed- was renamed -mixed-. The older -xtmixed- still works as a synonym for -mixed-, but you can save yourself a few keystrokes. Also, in future editions of Stata, they might stop recognizing -xtmixed-, so best to get into the habit of using the newer name. Evidently, this doesn't apply if you are using version 12 or earlier.

[Emeka Dim]

Apology accepted.

You had said: "But such expressions, innocent in earlier times and originally having nothing to do with race, ethnicity, or nationality, have acquired new, negative connotations." Please, the use of such personifications and objectification like "monkeying" has been a racial insult used against Black people, especially Africans, for a very long time. It has never been an innocent expression against Black people in earlier times. As I said, I assumed that you never knew that I was an African. Nevertheless, I accepted your apology wholeheartedly.

I always engage in any conversation with a fair amount of good faith. I am also very inquisitive when it comes to research issues. My people do say that "Onye ajuju anaghi efu uzo" meaning "he who asks questions never get lost." As you can tell from our thread, I am not very grounded in statistical theory. I did most of my foundational education in Nigeria (up to my first degree), and I did not have the privilege to be taught high-school mathematics or statistics at a very high level. Nevertheless, through sheer perseverance, long-suffering, being inquisitive, and humility, I have been able to learn and delve into some interesting statistical analyses that I use today. Nevertheless, I am still learning, and this is why I like to ask questions.

I understand your concern about the poor teaching of statistics. I am the one in this thread and platform asking the questions because I know that what I may have learned in the past could be problematic. In fact, when I see results that align with my beliefs, it gives me every reason to question them because I know that social reality is complex, which the human mind can hardly grapple with, and empirical research is meant to reach into such complexities. The analysis I am doing is not in any way a data-mining exercise. I had taken months reading the literature before I started my analysis. This is why I could confidently tell you that theory is silent on the relationship between the variables I am analyzing.

Regarding your suggestion about the use of the quadratic term, please, is there any scholarly paper that has addressed this issue? I would like to have them so that I can cite them. I believe that majority of these statistical issues are subject to debate, including the need to apply weights to the multivariate analyses. I have several articles that have argued against weighting. For the models, I asked someone else (a statistician), and he suggested that I simply control for the quadratic term in my model, hence the Model 3 you see in the last response. For me, I would prefer to stick with the line of argument that has stronger evidence and reference. If I were to follow your suggestion, it means that I will not have any coherent results to report for my thesis. I can live with that; I will only need to work on something else. However, to reject the results that will inform my thesis, I do need some scholarly evidence and citation to do so. This has nothing to do with a vested interest to publish anything; I do not even know the direction of the relationship thus far. I have not seen or produced most of the margin plots to see the variables I am interacting.

Also, please, do you think the AIC/BIC tests be used to compare the linear and quadratic models accurately?

Thank you and looking forward to your reply.

Eugene Emeka Dim.

[Clyde Schechter]

Regarding your suggestion about the use of the quadratic term, please, is there any scholarly paper that has addressed this issue?

I'll do some searching and see if I can find some and get back to you. I have worked remotely from my home for a number of years, and my collection of references is fairly sparse, consisting mostly of either introductory material (which this issue goes beyond) or arcane and advanced references in my particular areas of research (which would be irrelevant and don't deal with general statistical issues). So I have nothing quickly at hand.

If I were to follow your suggestion, it means that I will not have any coherent results to report for my thesis.

Why? I don't understand this at all.

Also, please, do you think the AIC/BIC tests be used to compare the linear and quadratic models accurately?

AIC and BIC are popular for model selection. Basically, they are log likelihood statistics with a penalty attached for increasing degrees of freedom. I think they can be useful for preventing the kind of overfitting that can result from just relying on statistical significance tests. But that is mostly an issue in smaller samples, or large samples with an enormous number of explanatory variables in play. You have a large sample and a modest number of explanatory variables. So AIC/BIC won't be much different than just a likelihood ratio test of models with and without the quadratic terms in your case. In fact, I think if you calculate them you will find that the results largely replay what you have already seen with likelihood ratio tests comparing models.

Although you have not asked about it, I want to take the opportunity here to explain why I don't like testing quadratic terms or comparing quadratic and non-quadratic models with statistical tests for the purpose of deciding on a quadratic model. There are relatively few true quadratic relationships in the real world. When we use quadratic modeling we are usually doing so because we are trying to model a U-shaped (possibly inverted) relationship between the outcome and explanatory variable. The problem is that quadratic regression is too non-specific, and significance testing of the quadratic term can confirm the model even when the true model is a curve that is not even remotely parabolic. For example, run:

clear*
set obs 50
gen x = _n
gen y = log(x)

regress y c.x##c.x

You will see that the quadratic term is highly significant, even in this modest sample. But if you plot y vs x, you will see the familiar logarithmic curve, which is certainly not quadratic at all. If however, you calculate the location of the turning point, you will see that it comes out a little over 42--which is near the upper end of the range of x. By noticing that you have a strong quadratic coefficient accompanied by a turning point that is far from the center of the data, you can conclude, correctly, that the relationship is, in fact, not U-shaped, but rather that it is curvilinear. And that might even prompt one to consider other non-linear representations of x, and perhaps even find (correct, in this case) logarithmic model as better. [There are other approaches to this same difficulty with quadratic terms in regressions picking up non-specific curvature. Some people recommend fitting separate linear regressions at the left and right edges of the distribution of the explanatory variable to see if they have opposite signs. This is a similar concept, and probably even more specific.]

[Emeka Dim]

Thank you for your reply.

What I meant by "If I were to follow your suggestion, it means that I will not have any coherent results to report for my thesis" is that about 60% of my results have this quadratic effect. All but one of my quadratic models (where gender*country-level variable²) are non-significant. However, to be honest, I have not checked (calculated) all of them to see if their values fall within or outside the meaningful ranges of the turning point.

You said "There are relatively few true quadratic relationships in the real world. When we use quadratic modeling we are usually doing so because we are trying to model a U-shaped (possibly inverted) relationship between the outcome and explanatory variable. The problem is that quadratic regression is too non-specific, and significance testing of the quadratic term can confirm the model even when the true model is a curve that is not even remotely parabolic." So does this mean that they may be several models in my work that would seem to be the quadratic model but are not? I will try to calculate all the quadratic models that I have to see their turning points and make sure of where their values fall.

You said: "There are other approaches to this same difficulty with quadratic terms in regressions picking up non-specific curvature. Some people recommend fitting separate linear regressions at the left and right edges of the distribution of the explanatory variable to see if they have opposite signs. This is a similar concept, and probably even more specific." Please, who has run such analysis and how do the commands look like so that I can try them? Please, if there is an alternative means of analyzing my data, I will be open to attempting them.