This content is restricted to site members. If you are an existing user, please log in. New users may register below.

7 Comments

Newest

Oldest

Inline Feedbacks

View all comments

David Fox

4 months ago

RESOLUTION

Changed sentence to read:

“thus use of AICc should generally be used in the SSD context. An exception is when the data are arbitrarily censored (i.e. where a cut-off is recorded rather than an actual numerical value) so as to give more weight to the left tail as described in the section Left Tail weighting of the SSD below. In such cases n is no longer defined”.

Rick van Dam

5 months ago

This is all useful commentary. It seems we can readily rebut this comment from the reviewer. Is the Wheeler & Bailer (2009) paper relevant and useful to city in support of model averaging? It looks to apply model averaging to dose-response data, which is not the same as SSDs, but not that from them.

Carl Schwarz

5 months ago

Cade (2015) indicates that the best case for model averaging is in predicting the predicted response value at a specific value of the x-axis variable. In the SSD context, this means predicting the hazard at a particular exposure concentration of the test chemical under consideration. Unfortunately, this is not the application here. Rather, the averaging is done on the estimated quantiles from these models, an altogether more challenging proposition and subject to the criticisms of Cade.

Cade (2015) does point out that for non-linear predictions, you should not first model average the coefficient, use the model averaged coefficient to make a prediction, and then transform this final prediction, but you need to first make a prediction for each model, and then model average the predictions. [If it is a linear prediction, then both methods are equivalent.] We never model average coefficients.

Is a predicted quantile ok for model averaging? Yes, it is a non-linear function (albeit) complicated function of each model. As indicated in the comment, finding quantiles on the left tail is highly sensitive to the distribution and fraught with problems when data are sparse and you are “extrapolating/predicting” far outside the range of the data.

Model averaging quantiles is quite common.. see 3.2.1 of the paper by Nysen et al (below) who discuss 2 ways to model average quantiles….The second being to model average the CDF’s, and find the quantile of the model averaged CDF.

The paper by Wheeler, M.W., Bailer, A.J. Comparing model averaging with other model selection strategies for benchmark dose estimation. Environ Ecol Stat16, 37–51 (2009). https://doi.org/10.1007/s10651-007-0071-7 also discusses this issue and find model averaging benchmark quantities (related to quantiles) performs well.

THis paper also discusses removing models from the set that are “nested” to reduce redundancy across the model set.

RE: “We never model average coefficients” – what coefficients are you talking about here? We don’t do this because we’re fitting different distributional forms where the “coefficients” are the cdf parameter values and they’re not comparable. My understanding of ssdtools is that it model averages the quantiles from each distribution. The fitted (and displayed) model-avaeraged cdf is, I believe, obtained by computing the model averaged cdf values for a range of abscissa values and essentially doing an x-y plot of the resulting pairs.Please correct me if this understanding is incorrect.

Carl Schwarz

5 months ago

Here some wordsmithing in the paper is needed. We state “An exception is when the data are censored (i.e. where a cut-off is recorded rather than an actual numerical value) as n is no longer defined.” Here we are refering to an “artifical” censoring that we can add to the observed data to give more weight to the left tail, rather than having censored observation at the endpoint for a species. This needs to be clarified in the sentence, e.g ….An exception is when we apply a censoring to the data to give more weight to the left tail and then n is no longer defined.

I don’t think we would every accept an endpoint for a species that is censored, i.e. for species X, the endpoint is <50. This could be handled in a Bayesian framework quite easily … Angeline and my paper talked about incorporating uncertainty in the endpoint (e.g. each endpoint has a confidence interval), but same approach could be use if the ci is one sided.

David Fox

5 months ago

The title of Cade’s (2015) paper, “Model averaging and muddled multimodel inferences”, would suggest it contains some broad recommendations and cautions for the use of model averaging. This is not the case. Cade’s paper is focussed entirely on a single, specific application of model averaging and that is model averaged regression coefficients and in particular the issue of multicollinearity. There is nothing I can see in Cade’s paper that is relevant to what we are doing.

RESOLUTIONChanged sentence to read:

“thus use of AICc should generally be used in the SSD context. An exception is when the data are arbitrarily censored (i.e. where a cut-off is recorded rather than an actual numerical value) so as to give more weight to the left tail as described in the section Left Tail weighting of the SSD below. In such cases n is no longer defined”.This is all useful commentary. It seems we can readily rebut this comment from the reviewer. Is the Wheeler & Bailer (2009) paper relevant and useful to city in support of model averaging? It looks to apply model averaging to dose-response data, which is not the same as SSDs, but not that from them.

Cade (2015) indicates that the best case for model averaging is in predicting the predicted

response value at a specific value of the x-axis variable. In the SSD context, this means

predicting the hazard at a particular exposure concentration of the test chemical under

consideration. Unfortunately, this is not the application here. Rather, the averaging is done on the estimated quantiles from these models, an altogether more challenging proposition and subject to the criticisms of Cade.

Cade (2015) does point out that for non-linear predictions, you should not first model average the coefficient, use the model averaged coefficient to make a prediction, and then transform this final prediction, but you need to first make a prediction for each model, and then model average the predictions. [If it is a linear prediction, then both methods are equivalent.] We never model average coefficients.

Is a predicted quantile ok for model averaging? Yes, it is a non-linear function (albeit) complicated function of each model. As indicated in the comment, finding quantiles on the left tail is highly sensitive to the distribution and fraught with problems when data are sparse and you are “extrapolating/predicting” far outside the range of the data.

Model averaging quantiles is quite common.. see 3.2.1 of the paper by Nysen et al (below) who discuss 2 ways to model average quantiles….The second being to model average the CDF’s, and find the quantile of the model averaged CDF.

The paper by

Wheeler, M.W., Bailer, A.J. Comparing model averaging with other model selection strategies for benchmark dose estimation.

Environ Ecol Stat16,37–51 (2009). https://doi.org/10.1007/s10651-007-0071-7also discusses this issue and find model averaging benchmark quantities (related to quantiles) performs well.

THis paper also discusses removing models from the set that are “nested” to reduce redundancy across the model set.

RE: “We never model average coefficients” – what coefficients are you talking about here? We don’t do this because we’re fitting different distributional forms where the “coefficients” are the cdf parameter values and they’re not comparable. My understanding of ssdtools is that it model averages the quantiles from each distribution. The fitted (and displayed) model-avaeraged cdf is, I believe, obtained by computing the model averaged cdf values for a range of abscissa values and essentially doing an x-y plot of the resulting pairs.Please correct me if this understanding is incorrect.

Here some wordsmithing in the paper is needed. We state

“An exception is when the data are censored (i.e. where a cut-off is recorded rather than

an actual numerical value) as n is no longer defined.” Here we are refering to an “artifical” censoring that we can add to the observed data to give more weight to the left tail, rather than having censored observation at the endpoint for a species. This needs to be clarified in the sentence, e.g ….An exception is when we apply a censoring to the data to give more weight to the left tail and then n is no longer defined.

I don’t think we would every accept an endpoint for a species that is censored, i.e. for species X, the endpoint is <50. This could be handled in a Bayesian framework quite easily … Angeline and my paper talked about incorporating uncertainty in the endpoint (e.g. each endpoint has a confidence interval), but same approach could be use if the ci is one sided.

The title of Cade’s (2015) paper, “Model averaging and muddled multimodel inferences”, would suggest it contains some broad recommendations and cautions for the use of model averaging. This is not the case. Cade’s paper is focussed entirely on a single, specific application of model averaging and that is model averaged regression coefficients and in particular the issue of multicollinearity. There is nothing I can see in Cade’s paper that is relevant to what we are doing.

Agree…nothing relevant here as we are not using a covariate but rather just model averating predictions.