Information Theory Basics (Simplified)
George Legrady


   

Origins
-

-

-


-

Information Theory has influenced the development of the digital world - the computer

Claude Shannon, mathematician from Bell Labs developed the theory to get rid of noise from telephone communication (which at the time was very noisy)

The goal of the theory:
1) how to send a maximum amount of information in a given channel
2) how to measure the capacity of a given channel to carry information

This theory came out of math and engineering but became influencial as it could apply to all kinds of human communication


Measuring Information

-

-
-
-

The smallest unit of information in the digital worlds is a "bit" or byte.
(short for binary digit)

A bit is like a switch - it is either on or its off: [1] or [0]
10011101: 8 bit
1011010110101001:16 bit


Communication Structure

-


-

-

-


-


-

-

-

In all communication, there is a sender, a message/communication and a receiver.

The sender encodes a message, the receiver decodes it.

Meaning of a message is greatly dependent on the culture in which it is transmitted.

The sender and receiver both know the code used to encode and decode the message as it is learned through cultural training)

Between the sender, the message and receiver, noise gets in the way and complicates the process.

A noiseless communication does not exist. There always is some kind of noise entering the communication.

The goal of Information Theory is to reduce the noise to make the communication more effective

Engineers try to remove the noise to get pure signal

Many artists, cultural theorists usually argue that it is the noise that makes the signal interesting


Noise Enters the System
-


-

-

-

Noise is unintended, unplanned, extra information that enters the communication process

Example of noise include distortion, feedback, anything that breaks down the signal

Noise can enter the process from the sender, the communication or the receiver

Noise can be considered as limiting but also as increasing information:
1) Noise in the system limits the amount of desired communication byt the sender,
2) but also adds new kinds of information (which may make the communication interesting)


Information: Basic definition

-

-

-


-

-

Shannon and Weaver (the authors of Information Theory) redefined the meaning of "information": Information must not be confused with meaning!

Information in Information Theory does not necessarily refer to meaningful content

Information is a measure of the predictability of the signal: If I meet you on the street, I can expect that we will say "Hi" to each other.

I can predict that there will be communication, so "the predictability of the signal is high!"

In the design of a telephone system, the critical factor is the number of signals it can carry. What people actually say is irrelevant


Redundancy
-

-

-

-


-


-

-





-
Redundancy is that which is predictable or conventional in a message

Something is redundant if I already know it.

Redundancy does not give new information but is essential to communication

Redundancy provides a way to identity errors in the communication through repetition.
(We repeat ourselves on a bad telephone line)

The English language is about 50% redundant - that means we can delete about 50% of the words and still have usable communication (you me go have drink now)

The opposite of redundancy is entropy or randomness (noise)

Noise usually means lots of new information (as opposed to planned information) but unable to make sense of it:
a) how ya doin, see you later
b) jpe us fpom, drr upi ;syrt (the same text as in (a) but with 1 keyboard character to right)

(b) has lots more new information but very difficult to understand unless one has the code to read it

Entropy: Conditional probability - a balance between random values of 0 and 1

Conventions & Codes
-

-


-

-
Politeness: conventional behavior that reaffirms social relationships

Subculture behavior - specialized communication recognized by members through repetition of conventions

CODE: a system of meaning common to members of a culture

it consists of signs and conventions that determine in what contexts these signs are used: Dress code, behavior code, alphabet, music taste, eating behavior, etc.