Reports

Report to Securing Water for Food (SWFF) and USAID on Evaluation of Ignitia Daily Rainfall Forecasts for Subscribers in West Africa

Goddard, Lisa M.; Kruczkiewicz, Andrew J.; Mason, Simon J.; Dinku, Tufa; Lenssen, Nathan

This report provides an independent verification of daily rainfall forecasts over West Africa from a private sector weather service, Ignitia, during the wet season for 2016 and 2017. The question asked by USAID was, "How accurate are Ignitia's weather forecasts for rainfall?" The forecast product under evaluation is daily rainfall likelihood, targeted to the locations of specific subscribers. The forecasts provide qualitative statements, which are underpinned by quantitative probability ranges, for daily rainfall for the day of forecast issuance (24-hour forecast) and the subsequent day (48-hour forecast). Given the probabilistic format of the forecasts, and the associated event specific (rain versus no-rain) outcome, an assessment of accuracy was not possible. Therefore, we assessed the forecast performance based on the probabilistic reliability and their ability to discriminate rainy days from non-rainy days over the period in question (2016-near present) using reference standard forecast and a simple alternative forecast computed from the widely used and adapted Global Ensemble Forecasting System (GFS), which also happens to be an input to the Ignitia forecasts. We note that we were unable to obtain competitor's forecasts (the National Meteorological Services, such as the Ghana Meteorological Agency, International news and media outlets, such as weather.com and BBC, and other private sector companies that produce forecasts on comparable spatiotemporal scales) available over the study period, which would allow us to fully examine the veracity of Ignitia's claim of that other forecasts can only achieve 39% accuracy. We were, however, able to determine the reason for Ignitia's claim that their own forecasts were 84% accurate. The claim is misleading. It is based on only a small subset of their forecasts: those for a high probability (>80%) chance of rain, which make up only about 5% of forecasts issued. Overall it cannot be said that their forecasts are 84% accurate. The results of the analysis show that while Ignitia's forecasts for rainfall perform well for certain categories of rainfall, and better than the raw output from a weather prediction model, in consideration of the full forecast spectrum, their claims are not substantiated. Ignitia's daily rainfall forecasts were found to be reliable, though slightly over-confident. In particular, their forecasts are considerably more reliable than raw model-based predictions. They were also found to have reasonable discrimination between the probabilities issued on rainy days versus those issued on dry days, though this aspect of forecast quality was similar to that found in the raw model output. There is also discernible impact to the reliability using the ENACTS data for Ghana. Particularly for the 24-hr forecasts in 2016, the reliability is nearly perfect for the Ignitia forecasts, but remains very poor and over-confident for GFS. The effect on discrimination is to lower it slightly for both Ignitia and the GFS baseline.

Geographic Areas

Files

  • thumnail for ReporttoSecuringWaterforFoodSWFFandUSAIDonevaluationofIgnitiadailyrainfallforecastsforsubscribersinWestAfrica (1).pdf ReporttoSecuringWaterforFoodSWFFandUSAIDonevaluationofIgnitiadailyrainfallforecastsforsubscribersinWestAfrica (1).pdf application/pdf 1.27 MB Download File

More About This Work

Academic Units
International Research Institute for Climate and Society
Published Here
September 25, 2024