Reproducing the Dunn et al. (2024) Analysis • SiGN

Dunn et al. (2024) compiled an exhaustive collection of studies examining suboptimal choice. This dataset is included in the SiGN package.

The complete, unfiltered data set is available via subopt_full. For model evaluation, however, Dunn et al. (2024) focused on a curated subset of studies involving pigeons and starlings—species that, particularly in the case of pigeons, were the most extensively studied and yielded the most robust and consistently replicated findings. This filtered data set is provided in the SiGN package as subopt_avian.

Each row in subopt_avian represents a distinct study condition. Many of its columns, specifically 9 through 24, directly correspond to parameters used by the choice_params() function (for specifics, consult the subopt_avian R documentation).

library(SiGN)

# Column Names
names(subopt_avian)
#>  [1] "row"             "study"           "year"            "species"        
#>  [5] "exp"             "condition"       "n"               "cp"             
#>  [9] "il_dur_a"        "il_dur_b"        "tl_dur_a1"       "tl_dur_a2"      
#> [13] "tl_dur_b1"       "tl_dur_b2"       "tl_p_a1"         "tl_p_a2"        
#> [17] "tl_p_b1"         "tl_p_b2"         "tr_p_a1"         "tr_p_a2"        
#> [21] "tr_p_b1"         "tr_p_b2"         "il_sched_a"      "il_sched_b"     
#> [25] "tl_sched_a1"     "tl_sched_a2"     "tl_sched_b1"     "tl_sched_b2"    
#> [29] "forced_exposure" "DOI"             "ref"             "data_version"

One key column is $cp, which reflects the observed choice proportion reported by each study. These values can be directly compared to predictions from the SiGN model.

# Model Predictions
#-------------------------------------------------------------------------------
# Extract relevant parameter columns (cols 9–24 align with choice_params())
data_cols <- subopt_avian[9:24]

# Construct model input list
params <- do.call(choice_params, as.list(data_cols))

# Generate model predictions
preds <- SiGN(params)$details

Before plotting the comparisons, it will be helpful to first compute some additional statistics to aid in model assessment:

A linear regression model fitted to the observed vs predicted values, to provide a visual summary of their linear relationship alongside the 1:1 reference line.
An $R^2$ value from the above regression, quantifying how well a straight line fits the relationship between the SiGN model’s predictions and the observed data. This is the statistic reported by Dunn et al. (2024, see note at the bottom of this document).
An additional $R^2$ value, computed directly from the residuals of the SiGN model, which reflects the true proportion of variance in the observed data explained by the model.
The RMSE (root mean squared error) and MAE (mean absolute error), which both capture the average prediction error — with RMSE giving greater weight to larger deviations.

# Model Assessment
#-------------------------------------------------------------------------------
# Fit a linear model comparing observed vs predicted
reg <- lm(subopt_avian$cp ~ preds$cp)
Rsq_lin_fit <- summary(reg)$r.squared

# Residual-based R2
Rsq_mod_fit <- 1 - sum((subopt_avian$cp - preds$cp)^2) / 
  sum((subopt_avian$cp - mean(subopt_avian$cp))^2)

# Root Mean Squared Error (RMSE)
rmse <- sqrt(mean((subopt_avian$cp - preds$cp)^2))

#Mean Absolute Error (MAE)
mae <- mean(abs(subopt_avian$cp - preds$cp))

The following plot illustrates the correspondence between observed and predicted choice proportions. The dashed line represents perfect prediction (y ~ x), while the solid line depicts the best-fitting linear regression.

# Plot of Results
#-------------------------------------------------------------------------------
# Format annotations for plot
reg_txt <- sprintf(
  "Linear fit: y = %.2f + %.2f x\n",
  reg$coefficients[1], reg$coefficients[2]
)
Rsq_lin_fit_txt <- sprintf("R² (linear fit) = %.2f", Rsq_lin_fit)

Rsq_mod_fit_txt <- sprintf("R² (model fit) = %.2f", Rsq_mod_fit)
rmse_txt <- sprintf("RMSE = %.2f", rmse)
mae_txt <- sprintf("MAE = %.2f", mae)

stats_lin <- paste(reg_txt, Rsq_lin_fit_txt)
stats_mod <- paste(Rsq_mod_fit_txt, rmse_txt, mae_txt, sep = "\n")

# Draw Plot
plot(preds$cp, subopt_avian$cp,
  type = "n", las = 1,
  xlim = c(0, 1), ylim = c(0, 1),
  xlab = "Predicted",
  ylab = "Obtained"
)

# Reference line
abline(a = 0, b = 1, lwd = 1, lty = 3, col = "grey")

# Points
points(preds$cp, subopt_avian$cp, las = 1, cex = 1)

# Regression line
abline(a = reg$coefficients[1], b = reg$coefficients[2], lwd = 2)

# Annotation
text(x = 0.25, y = 0.95, labels = stats_lin, cex = 0.75)
text(x = 0.25, y = 0.75, labels = stats_mod, cex = 0.75)

The RMSE (0.14) and MAE (0.11) indicate that the model’s predictions deviate from the observed values by approximately 11 to 14 percentage points on average. This reflects a reasonably good fit to the subopt_avian data set and aligns with expectations, given the inherent variability across studies, species, and procedural differences.

Note: In Dunn et al. (2024), we reported an $R^2$ value derived from a linear regression of observed outcomes on the model’s predicted values. While this is a common practice in applied modelling, it does not directly quantify how much variance the model itself explains in the data. Had we more fully appreciated this distinction at the time, we would have opted for a residual-based $R^2$ instead. In this vignette, we include both the regression-based and residual-based $R^2$ values: the former to reproduce the original analysis, and the latter to provide a more appropriate measure of model fit. This reflects our commitment to a transparent, evolving methodology and hopefully encourages best practices for evaluating predictive models.

References

Dunn, R. M., Pisklak, J. M., McDevitt, M. A., & Spetch, M. L. (2024). Suboptimal choice: A review and quantification of the signal for good news (SiGN) model. Psychological Review. 131(1), 58-78. https://doi.org/10.1037/rev0000416