A forecast comparison is only meaningful when each provider is evaluated on the same operational problem. A lower error on a different portfolio, deadline or measurement boundary does not establish a better result. Before requesting a benchmark, agree what is being predicted, which forecasts would have been available and how observations will be treated.
Define the target quantity
Power in MW is different from energy in MWh. Available generation is different from exported feed-in after restrictions. Metering at a turbine, inverter or grid connection can also represent different quantities. Fix the valid interval and timestamp convention. Document whether timestamps label an interval's beginning or end, and handle daylight-saving transitions consistently.
Fix the information deadline
A day-ahead forecast issued before scheduling is not comparable to an intraday update informed by newer observations. Compare equal lead times and archive each issued forecast. Replacing an old forecast with a later version introduces hindsight. Delivery failures and stale forecasts should remain visible rather than disappearing from the score.
Interpret error measures explicitly
For each interval, error can be defined as forecast minus observation. Mean absolute error averages the absolute errors. Root mean square error squares them before averaging and taking the square root, giving larger deviations more influence. Bias preserves the sign and reveals systematic over- or underprediction. Normalised scores divide by a stated reference such as installed capacity; changing that denominator changes the interpretation. Report units and the exact normalisation.
Agree exclusions before seeing results
Missing measurements, outages and curtailment need an agreed treatment. Exclusions may be appropriate for a particular target, but they can also remove the hardest intervals. Publish how much data was excluded and why. Night-time solar zeros and low-output periods can make aggregate scores appear favourable, so examine meaningful subsets as well as the complete period.
Look beyond one average
Inspect performance by season, asset, lead time and output range. Portfolio aggregation can hide individual-site errors or benefit from their partial cancellation. Compare against a simple baseline on exactly the same data. If uncertainty products are supplied, assess their probabilities and interval coverage separately from point forecasts. Business value additionally depends on the actual decision and cost structure.
Make the evaluation reproducible
Keep the asset list, input data version, issued forecasts, measurements and scoring rules together. Use a period long enough to include relevant weather and operational conditions. Ask the provider to explain anomalous intervals without silently changing the rules. A backtest should produce evidence for a deployment decision, including its limitations, rather than an isolated marketing percentage.
A simple example of a misleading comparison
Suppose one solar forecast is scored over every hour, including night, while another is scored only during daylight. Their average errors answer different questions. The same problem occurs when one wind portfolio contains sites whose errors partly cancel and another contains a single plant. Similar metric names do not make the tests comparable. Start with the same dataset and issue-time rules.
Keep statistical and commercial results separate
A smaller average error can be useful without guaranteeing a proportional financial benefit. The intervals with the largest imbalance cost may differ from those with the largest generation error. A commercial replay needs the actual scheduling policy, market opportunities and relevant prices available at each decision time. State those assumptions alongside the result.
Review the difficult intervals
Ask for examples of large misses, missed deliveries and changed operating states. Those cases reveal how the service behaves when it matters. A transparent evaluation retains the anomalies and explains them instead of quietly removing them from the score. This makes the result actionable: the buyer can see whether additional data, a different process or a revised model addresses the observed limitation.