A community member of Metaculus asks,

Has anyone figured out a rough heuristic for predicting the next week’s sun-hours based on weather forecasts? This week I was like “oh, it’s been pretty cloudy, so it’ll be ~58hrs”. Then it’s 74! The same happened the last couple of weeks.

One’s first instict in situations like this should be a back-of-the-napkin calculation of what accuracy is even theoretically possible. First, some basic knowledge we will need:

• Cloud cover is measured in okta, which is 1/8th of the sky. So one okta is 12.5 % of the sky covered.
• The highest amount of weekly sun in this location in recent history has been 118 hours, so to be conservative, let’s say the maximum possible sun is 120 hours.

The ecmwf publishes probabilistic weather forecasts including cloud coverage, which lets us deduce the standard error of daily cloud coverage forecasting. At the time of writing, their 80 % interval (the narrower bars in the bar-and-whisker plot) seems to cover on average about 5.2 okta.

Since we’re just shooting the breeze here, let’s assume a normal distribution1¹ Eyeballing the distributions, it’s clearly not normal, but maybe close enough to get some numbers going., because that lets us translate the 5.2 okta into a standard deviation, by dividing by two (to get a one-sided z-score) and then looking up what z score corresponds to 10 % (it’s 1.128), and dividing by that also.

\[\frac{5.2}{2 \times 1.128} \approx 2.3\]

So the standard error of the daily cloud coverage forecast is 2.3 okta.2² For a reality check, since the 25–75 % range of the normal distribution has a z-score of ±0.67, we will expect to see cloud coverage differ from the forecast by more than 2.3×0.67 = 1.6 okta more than half the time. This is a very uncertain forecast.

If “percent of sky covered with clouds” translates directly to sun hours, that gives us an upper limit on the accuracy of an estimation of sun hours for next week.

If we are a little sloppy with terminology, a week contains 7×8=56 daily okta. If we take all the daily cloud coverage forecasts in a week together, the standard error for the entire week should be \(2.3 \sqrt{7} \approx 6.1\). Thus, the weekly cloud coverage forecast error is in the ballpark of 11 %.

The maximum weekly number of sun hours was 120. Taking 11 % of this gives us a standard error of the sun-hour estimation of ±13 hours, which in turn means that 90 % of the time, we will see the actual sun-hours be within 1.645×13 = ±21.5 hours of our forecast, even when we are forecasting optimally – simply due to the uncertainty in the cloud cover forecast.

Thus, the initial problem of the actual sun-hours deviating by 16 from the forecast is completely normal for this type of data. We shouldn’t expect to do better even if we had a perfect forecasting method!

In this case we did the translation from input error to forecast error very sloppily. This is probably fine for a majority of forecasts. In some cases we might want a more exact solution, an expression that tells us exactly how uncertain the forecast is based on uncertainty in the inputs.

In the simplest case, the forecast is a one-dimensional function of just one input; it will look like a regular curve on the X and Y plane. The question then becomes: How much does this function change due to small fluctuations in the input? This is what derivatives solve for us!

To use a somewhat simple example, what is the effect of error in radius measurement on the estimation of sphere volume? The volume of a sphere is given by \(V = 4/3 \pi r^3\), with first order derivative

\[\frac{dV}{dr} = 3r^2\]

Which means a 5 % error in radius measurement turns into roughly a 10 % error in volume estimation, due to the square: \(1.05^2 \approx 1.1\). If we want an even more exact result, we can add in higher order derivatives (think Taylor series expansion).