Improved Duplicate Handling for More Reliable Market Data Updates

---

Improved Handling of Duplicate Market Data

FactDecode has improved how it handles multiple daily market data records for the same date retrieved from Twelve Data.

Previously, conflicting closing prices for a single date could prevent an entire dataset from being updated, including otherwise valid observations from subsequent dates.

With this update, duplicate records containing identical values are consolidated into a single record. When closing prices conflict, FactDecode excludes the affected date rather than arbitrarily selecting one of the conflicting values.

This allows newer, valid observations to be incorporated even when inconsistencies remain on a limited number of dates, provided that all data quality requirements are satisfied.

Maintaining Data Quality During Updates

Dates with conflicting observations cannot be excluded without limits.

Exclusions are permitted only when both conditions are met: no more than five dates are excluded, and the excluded dates account for no more than 0.5% of the observed dates in the retrieved dataset.

Existing quality checks, including data coverage and minimum observation requirements, continue to apply.

If the quality requirements are not met, the update is rejected and the previously available dataset version is preserved.

If the data provider subsequently corrects an inconsistency, FactDecode can validate the corrected observations and restore the previously excluded date in a new dataset version.

Clearer Data Quality Information

Data quality information is now available in the data details screen, refresh results, and pre-analysis confirmation screen.

These views provide information about dates excluded for quality reasons, the number of affected dates, and related notices.

This makes it easier to understand the quality status of an updated dataset and identify the latest observation date included in the accepted data.

When an update proceeds with excluded dates, that information remains visible as part of the dataset's quality record.

Existing Analysis Methods and Saved Data Remain Unchanged

This update improves data ingestion quality checks and the presentation of quality information. It does not change the calculation formulas used for Factors or Targets.

For dates excluded due to data quality issues, FactDecode does not insert fabricated market prices. Instead, the remaining observations are processed using the existing calculation methods.

As a result, when dates are excluded, the calendar time between adjacent observations may be longer than usual.

Previously saved dataset versions and historical analysis results are not modified.

Open a dataset in FactDecode to review its latest accepted observations and data quality information in the data details and refresh results.