We present and show applications of two new test statistics for deciding if one ARIMA model provides significantly better h-step-ahead forecasts than another, as measured by the difference of approximations to their asymptotic mean square forecast errors. The two statistics differ in the variance estimate whose square root is the statistic’s denominator. Both variance estimates are consistent even when the ARMA components of the models considered are incorrect. Our principal statistic’s variance estimate accounts for parameter estimation. Our simpler statistic’s variance estimate treats parameters as fixed. The broad consistency properties of these estimates yield improvements to what are known as tests of Diebold and Mariano (1995) type. These are tests whose variance estimates treat parameters as fixed and are generally not consistent in our context.

We describe how the new test statistics can be calculated algebraically for any pair of ARIMA models with the same differencing operator. Our size and power studies demonstrate their superiority over the Diebold-Mariano statistic. The power study and the empirical study also reveal that, in comparison to treating estimated parameters as fixed, accounting for parameter estimation can increase power and can yield more plausible model selections for some time series in standard textbooks.