Mathematics of compound symbols

## Probability

Since a compound symbol cij is defined

cij ≜ ⟨ xi, yj
Probability of the compound symbol, p(cij) = p(⟨xi, yj⟩) = p(xi, yj) is a joint probability.

Hence, Similarly for a word wi given by wi ≜ ⟨xi,0, xi,1, xi,2, …, xi,n−1⟩, probability (joint) of the word is
p(wi) = p(⟨xi,0, xi,1, xi,2, …, xi,n−1⟩) = p(xi,0, xi,1, xi,2, …, xi,n−1)

## Joint Entropy

Since the set of all compound symbols, And probability of a compound symbol is the joint probability, p(cij) = p(xi, yj). Thus, entropy of the set of all compound symbols H(C) is given by Thus, H(C) ≡ H(X,Y).

In other words, H(C) and H(X,Y) are one and the same (≡) and not just in quantity (=), i.e., it is true for all values of C, X and Y.

What does it mean (practical) to say, H (C) ≡ H(X,Y)? ## Equivocation

Since,

p(x, y) = p(x|y) p(y)
⇒ log p(x, y) = log p(x|y) + log p(y)
alternatively
p(x, y) = p(y|x) p(x)
⇒ log p(x, y) = log p(y|x) + log p(x)
Thus, What is the practical meaning of H(Y|X)?

Equivocation or conditional entropy, H(Y|X) implies that, given the knowledge of X, H(Y|X) measures the uncertainty remaining in Y. For instance, if Y = f(x) then it implies that knowing x tell us about Y.

What does uncertainty remaining in Y mean, i.e., what does H(Y|X) mean?

To answer let us consider three cases:

1. Given all the knowledge of X if it tells us nothing about Y, then uncertainty remaining in Y, H(Y|X) = H(Y).
2. Given all the knowledge of X if it tells us everything about Y, then uncertainty remaining in Y, H(Y|X) = 0.
3. Given some the knowledge of X so that there is uncertainty in Y, then uncertainty remaining in Y, H(Y|X) ≤ H(Y).
Thus, possible range of uncertainty remaining in Y is
0 ≤ H(Y|X) ≤ H(Y)