If we want to encode the vector \(\left[6, 2, 8, 4\right]\) in a single expression and we cannot use vectors, we can create a function

\[f(x) = 6 + 2x + 8x^2 + 4x^3\]

and this function now contains all those numbers. One might imagine that this function would be reducible into some simpler expression by combining terms and multiplying out \(x\)​s, but thanks to the increasing powers of \(x\), it is not. Regardless of how this function is rearranged, we will still be able to arrange it back into this shape. We can extract any number in the sequence from this function with some calculus if we want to1¹ If we want to extract, say, the third number (in this case 8), we will need to take the second derivative (to isolate that number as a constant), evaluate the function at zero (to get the constant out) and then divide by two (to remove excess coefficient that came from the derivative action.) More generally, number \(n\) is found through \(f^{(n-1)}(0)/(n-1)!\)., emphasising how the polynomial and the vector really are, in some sense, carrying the same information.

We might be curious about what the \(x\) represents. Like, what are typical values we might plug in for \(x\)? The answer is nothing. We generally don’t evaluate this function with any \(x\) at all. The function, and \(x\), exists only to create a particular structure into which we can store coefficients for later manipulation. it is not intended for anything meaningful to come out if we replace \(x\) with some specific value we might think of.2² That said, we have already seen in an earlier sidenote that the value x=0 is special in that it allows us to extract the constant, which combined with differentiation lets us extract any coefficient. But although there are some \(x\) that result in something meaningful, we shouldn’t think all \(x\) do.

What we just did was turn a vector into a polynomial. Today, the inverse operation is probably more common in that we take the coefficients of a long polynomial and plug them into a vector to e.g. find roots with linear algebra. But back in the time of de Moivre, polynomials were all they had, and so when they needed to work with an entire sequence as a single object, they chucked the sequence in as coefficients of a polynomial function. The polynomial \(f(x)\) is then known as a generating function of the sequence, because we can (through calculus) generate each value of the sequence from the function \(f(x)\).

The probability distribution of a draw from your hand, you know, that vector

\[\left[1/5,\;0,\;0,\;0,\;0,\;0,\;1/5,\;0,\;1/5,\;0,\;0,\;2/5,\;0\right]\]

might then be represented by the polynomial (generating) function3³ Taking the convention that aces can be represented numerically as 1, and queens as 12.

\[G(t) = \frac{1}{5}t^1 + \frac{1}{5}t^7 + \frac{1}{5}t^9 + \frac{2}{5}t^{12}\]

In this case, the numbers in the sequence – and the coefficients of the generating function – are probabilities. When the coefficients of a generating function are probabiilties, we call that function a probability-generating function.

We have some other examples of probability-generating functions. For example, a fair coin flip has a probability-generating function

\[G(t) = 0.5t^0 + 0.5t^1\]

because outcome 0 (tails) has probability 50 % and outcome 1 (heads) has probability 50 %.

If the coin is not fair, but has bias \(p\), then the probability-generating function is

\[G(t) = (1-p)t^0 + pt^1\]

This is often written more compactly as4⁴ Using the fact that for any \(t\), we can say that \(t^0 = 1\), and we usually write \(t^1\) as just \(t\).

\[G(t) = (1-p) + pt\]

The geometric distribution represents how many tails we have to see until the first heads. In the case of a fair coin, we expect to get heads very soon, but when the probability \(p\) of heads is low, we may have to toss for a very long time. We can think systematically to guess the sequence of probabilities for the geometric distribution5⁵ With probability \(p\) we get heads on the first toss. In that case, we will have zero tails, and this will be the first value of the probability sequence. Failing to get heads on the first toss has probability \((1-p)\), which means if we get heads on the second toss, we will have seen an outcome with probability \((1-p)p\). Getting heads on the third toss has probability \((1-p)^2p\) because it requires two tails followed by heads. This goes on forever, but at this point we have caught on to the pattern: the first heads on the nth trial requires \(n-1\) tails followed by a heads, which has probability \((1-p)^{n-1}p\). and if we do, we end up with a sequence that in vector form looks like

\[\left[p,\;\;(1-p)p,\;\;(1-p)^2p,\;\;(1-p)^3p,\;\;\ldots\right]\]

As a generating function, that becomes

\[G(t) = pt^0 + (1-p)pt^1 + (1-p)^2pt^2 + (1-p)^3pt^3 + \ldots\]

It might seem troublesome that this function has an infinite number of terms, but that’s actually not a big deal. Not even to de Moivre! By the mid-1700s, although their proofs were not as rigorous as today’s, they had a decent grasp of polynomials of infinite degree.

In particular, we can define \(v = (1-p)t\) and rewrite this last probability-generating function of the geometric distribution as

\[G(t) = p \left( v^0 + v^1 + v^2 + \ldots \right)\]

and for sensible \(v\), this is equivalent to

\[G(t) = \frac{p}{1-v}\]

Substituting back, we get a very compact way to express the geometric distribution through its probability-generating function:

\[G(t) = \frac{p}{1 - (1 - p)t}\]

But remember that even though this function does not look anything like a series, it still encodes a series as a polynomial – it’s just been rearranged for compactness. We can still extract each individual probability from this through calculus.

That’s it. A probability-generating function is a way to encode a sequence of probabilities into a single object (a function) when one does not have access to the technology of vectors.

Except …

In Theory of Probability, de Finetti provides a nice visual of the transformation implied by the characteristic function. We will have a bunch of

\[\phi(u) = \ldots + p(-2) e^{-2iu} + p(-1) e^{-iu} + p(0) + p(1) e^{iu} + p(2) e^{2iu} + \ldots\]

If we focus on the case where \(u=1\), then each of the \(e^{iuX}\) represent a distance \(X\) traveled counter-clockwise (for \(X\) positive) or clockwise (for \(X\) negative) around the unit circle in the complex plane. Then when we multiply by \(p(X)\) we scale this point on the circle down closer to the origin based on its probability.

Visually, the sequence of \(p(X)e^{iuX}\) has the effect of wrapping the probability distribution of \(X\) around the unit circle in the complex plane into a kind of spiral. When we sum all these points together, we get the barycentre of that spiral, and that is indeed how we can intuit the value of \(\phi(u)\).

This also makes it clear that \(\phi(u)\) is real for symmetrical distributions, but complex in the general case.

I really wish I could animate this because it’s a great way to look at it, but instead, I give you the image we get from de Finetti: