2009 Reports

Position Paper: Verification of African RCOF Forecasts

Mason, Simon J.; Chidzambwa, Simbarashe

Now that the Regional Climate Outlook Forums in Africa have been operating for over ten years, an evaluation of the skill of these forecasts is possible. For most other regions in which RCOFs have been held there are fewer forecasts available for any detailed diagnostic verification, but many of the lessons learnt from a verification of the Africa RCOF forecasts are relevant globally. In addition, the identification of appropriate verification procedures has relevance globally, since forecasts are presented in similar formats at all the RCOFs. Forecasts are verified from three RCOFs in Africa: for Southern Africa forecasts are verified for the October – December (early-season) and January – March (late-season) summer rainfall season; for the Greater Horn, the target seasons are March – May and September – December; and for West Africa forecasts are for July – September. All three regions indicate some evidence of positive skill, meaning that they contain useful information that could potentially have been used to achieve some form of benefit. In addition to the numerous other benefits, such as the development of capacity within the climate services of the National Meteorological Services in most of Africa, the positive skill provides a powerful endorsement to the RCOF process. However, the forecasts do show clear evidence of systematic errors, and in some cases the positive skill may not be immediately apparent to users. There is thus considerable scope for improvement. The most ubiquitous error is for the forecasters to hedge the forecasts towards high probabilities on the normal category. The probabilities for the normal category are therefore consistently higher than they should be, and the normal rainfall occurred notably much less frequently and extensively than implied by the forecasts. This hedging is an effect of an ongoing deterministic interpretation of the forecasts and the wish to avoid the risk of the forecasts being interpreted as in error by two categories (which is possible only if below- or above-normal rainfall is forecast). In addition to this over-forecasting of the normal category, there is little or no evidence of any skill in forecasting increased probabilities for this category. More generally, the probabilities for all categories typically show poor reliability, and there is a need to implement improved procedures for defining the probabilities. In most cases the poor reliability reflects over-confidence (increases and decreases in probabilities are too large), which points to a need to review the scientific bases for some of the predictions. Over the approximately 10-year verification period, below-normal rainfall was predominant in the Greater Horn in both seasons, in West Africa for the July – September period, and in Southern Africa for January – March. The RCOFs did not provide any clear indications of these trends, which has to be acknowledged as a notable failure. Again, the need for a serious review of the scientific bases for how the forecasts are currently made needs to be undertaken, and an assessment of the potential benefit of making greater use of Global Producing Centre products should be conducted. Apart from these considerations of the skill of the forecasts, ambiguities in the precise meaning of the forecasts occur because of the way in which the forecasts are constructed. Specifically it is not clear whether the forecasts are meant to be interpreted only as regional averages, and, if so, what precisely the regions are over which the averages should be calculated. It is recommended that this ambiguity be addressed by careful consideration of how the forecasts are constructed; specifically, greater consistency is required in the ways in which the forecasts are made for each country before the consensus building step.