More Rigorous Validation for Predictive Models

---

More Rigorous Out-of-Sample Validation

A model that fits historical data well does not necessarily perform well on data it has never seen. With this update, FactDecode has refined how predictive models are trained and evaluated over time.

For each historical validation period, training uses observations whose target outcomes were already known by that point. We also enforce stricter boundaries between training and validation periods to prevent overlapping future outcomes from affecting the evaluation. This reduces the risk of evaluating a model using information it should not have seen, making out-of-sample (OOS) validation more rigorous.

open_in_full

Reselecting Factors for Each Validation Period

Factor selection is an important part of model validation. If factors are chosen after examining the results of an entire historical period, the resulting backtest may appear stronger than the evidence actually supports.

With this update, FactDecode reselects factors within each validation period using only the history permitted for that period. We have also revised the history requirements for factor candidates to avoid excluding factors with shorter histories unnecessarily, while still checking that sufficient data is available for meaningful comparisons.

However, remaining eligible as a candidate does not guarantee that a factor will be selected for the final model. By evaluating factor selection within each historical training window, FactDecode provides a more consistent time-ordered process from factor selection through model validation.

Making Better Use of Eligible Training History

Evaluating historical predictive performance and training a model for current predictions serve different purposes. Previously, part of the data eligible for final-model training was excluded from the actual fitting process.

With the updated approach, FactDecode refits the final model using all eligible training observations whose target outcomes are already known, after factor selection and model settings have been determined. This allows the final model to make use of the later portion of the available training history that was previously excluded from fitting.

At the same time, OOS validation continues to respect each historical evaluation boundary, ensuring that outcomes from the validation period are not used to train the corresponding model.

A Consistent Approach Across Prediction Horizons

The updated training and validation methodology is not limited to specific assets or prediction horizons. It applies to the standard 5-, 20-, 60-, 120-, and 240-observation presets, as well as custom positive-integer horizons such as 30 or 90 observations.

In FactDecode, a prediction horizon represents the number of observations in the target data, not the number of calendar days. When sufficient historical data is unavailable, the analysis is blocked with an explanation rather than forcing a model to run with inadequate data.

Your Saved Analysis Results Remain Unchanged

These improvements apply to newly executed analyses. Predictions and OOS evaluations from previously saved Runs are not recalculated, and results generated using earlier methods remain available in Results.

The purpose of this update is to improve the rigor and consistency of model validation and final training. It does not guarantee better future predictions or investment returns. It also does not imply that the publication timing or historical revisions of every external data source can be fully reconstructed.

FactDecode will continue to evolve as an analysis environment where you can examine not only the predictions themselves, but also how they were evaluated and which factors were selected.