Information theory

This course provides the fundamental principles for understanding the concepts of information quantification and the reliability of information to be communicated. Based on the definition of entropy, the redundancy of a source and its relevance through the source coding theorem are defined. The course concludes with a study of the coding theorem in the case of transmission through a noisy channel.

MA105: Probability TS108: Random Processes

* QUANTITATIVE ESTIMATE OF INFORMATION
- Study of sources: entropy, mutual and conditioned entropy,
- Mutual information
- Kullback-Leibler divergence
- Case of ergodic and Markovian sources
* ENTROPIC CODING
- Redundancy of symbols and messages
- Average length of a code, optimal length
- Huffman code
- Extended sources

Signal and image processing

Course and TD handouts