Random processes and information theory

Numerous random phenomena occur in nature, such as fluctuations in temperature or atmospheric pressure. In electronics and telecommunications, the study of random processes is particularly useful in the context of digital communications, where certain signals are impossible to characterize a priori. The exploitation of random processes is also at the root of many approaches to signal processing, whether for characterizing the frequency content of a signal (spectral analysis), or for denoising, coding and watermarking a speech signal. More generally, information sources such as sound and images are random and vary over time.
The Random Process/Information Theory course therefore aims to introduce the properties and tools for processing phenomena that vary randomly over time. It is particularly useful for courses in signal processing and automatic control.

probabilities

Part Random processes:


1/Characterization of random processes:

Notion of mean,

illustrations of a random process in the discrete case,

Higher-order probability density,

properties of autocorrelation and autocovariance functions,

stationarity and ergodicity


2/Markov chains Reminders on oriented graph theory, discrete-time Markov chains


Part Information theory

1/ Quantitative estimation of information

Study of sources: Entropy, mutual and conditioned entropy,
Mutual information

Case of ergodic and Markovian sources


2/ Entropic coding

Redundancy of symbols and messages

Average length of a code, optimal length

Shannon-Fano code

Huffman code

course material