Entropy.md (384B)
1 # Entropy 2 3 Ch 6 4 5 **Definition:** Entropy is the average number of bits communicated by one message if message hoarding is allowed. 6 7 Entropy of a finite set of messages is denoted as H(X). 8 9 Formula: 10 11 $H(X) = -\sum_{x\in S} P(x)log_2(P(x))$ 12 13 X is an ordered pair (S,P) where S is a finite set of messages and P is a function from S -> [0,1] S.T. P(x) = Probability of x for all x in S.