Lecture 04 Summary (Information Theory)

Lecture04 Summary

Applications for the rules with more than two symbols (e.g. three symbols; x₁, x₂, x₃)

Chain Rule H(x₁, x₂, x₃) = H(x₁) + H(x₂|x₁) + H(x₃|x₁, x₂)
Conditional Mutual Information I(X, Y, Z) = H(X|Z) − H(X|Y, Z)
Z is considered Side Information.

Using Probability Distribution to derive mutual information.

$I(X; Y) = for all x sub i, for all y sub j, sum of products of p(x sub i, y sub j) and log base 2 of (p(x sub i, y sub j) over (product of p(x sub i) and p(y sub j)))$
Partial Mutual Information

$I(X; Y) = for all y sub j, sum of I(X; y sub j) OR I(X; Y) = for all x sub i, sum of I(x sub i; Y)$

❶

Directed graphs is a convenient and useful way to represent information channel where the symbols (input & outputs) are shown as nodes and the connecting directed arrows indicate transitional probabilities.
For example,
Steps to determining information lost or gained.
1. Compute entropy H(X).
  
  $H(X) = for all x sub i, sum of products p(x sub i) times log of base 2 of (1 over p(x sub i))$
2. Compute Mutual information.
  
  $I(x sub i; Y) = for all y sub j in Y, sum of products p(Y given x sub i) times p(x sub i) times log of base 2 of (p(Y given x sub i) over p(Y))$
  
  $I(X; Y) = for all x sub i in X, sum of I(x sub i, Y)$
3. Compute Information Loss. H(X|Y) = H(X) − I(X; Y) And H(X|Y) ≠ 0 implies that there is information loss through the information channel.

❷

Hard Decision

If there is no ambiguity (transitional probability ≈ 1 )then the occurence of the output symbol is pretty sure. The decision resulting in such output symbols is called hard decision (the output symbols are therefore called hard decisions).
Soft Decision

If there is ambiguity (transitional probability ≈ 0 )then the occurence of the output symbol is not definite. The decision resulting in such output symbols is called soft decision (the output symbols are therefore called soft decisions).

❸

Both Hard & Soft Decision Models results in information loss, I(X; Y)_Soft > I(X; Y)_Hard
H(X|Y)_Soft < H(X|Y)_Hard and Hard decision losses more information than soft decisions. Thus, Quantization is information lossy.
Mechanics of Information Loss
There is less information in Y than in the cartesian product or set X × η (i.e, H(Y) < H(C) = H(X, N)), but more information in Y than information in either X alone or η alone (H(Y) > H(X) & H(Y) > H(N)).
Thus noise information has been added to information.

✪

Lectures on Information Theory