Spectrogram Spectrogram of frequency-modulated wave corrupted by exponentially decaying white noise

Theory-Code (TC) Notebooks

Below is a collection of notebooks with explanations of topics, including both theory and code, from several fields, including signal processing, machine learning and information theory. These are topics that I have found interesting, important or particularly difficult to master. These notebooks are also hosted on github here.

Origin

The first notebooks here grew from my exploration of specific topics that I found particularly interesting, and befuddling, because of the myriad ways in which these topics could be approached. In an effort to unscramble them for myself, I began to document my understanding from near first principles as well as implement them in code. I soon realised that this was a particularly effective approach to mastering a topic and began doing so for other complex/interesting topics that I encountered.

Purpose

The purpose of these notebooks is to break down chosen topics to near first principles in a manner that I would have liked these topics explained to me when I first encountered them. My hope is that they will serve as a friendly and thorough introduction to these topics for anyone with a knowledge of high school mathematics seeking both theory and implementation and as a ready reference material for me when I revisit any of this topics.

Notebooks

Machine Learning I

The Classification and Probability Density Estimation (PDE) tasks are introduced for a Gaussian Mixture Model (GMM). For classification we motivate and implement k-Nearest Neighbours and Radial Basis Functions and for PDE, Parzen Windows. Expectation Maximization is introduced for fitting the GMM with k Means Clustering used to find the means. Precision and Recall are also introduced.

Maximum Likelihood - Code

First a simple probability mass function is introduced to provide an implementation of the foundational Information Theory material given in the related Theory book. A Binomial Distribution is then introduced and the method of Maximum Likelihood is used to determine this distribution. For this distribution the Fisher Information is computed as well.

Maximum Likelihood - Theory

An Information Theory approach to Maximum Likelihood (ML) is introduced. Elementary concepts such as Entropy, Cross-Entropy and Relative Entropy are introduced as foundational knowledge. The ML method is built up from its elements followed by a return to motivate the relationship to Entropy. Finally, the Cramer-Rao Bound and Fisher Information (with supporting material including the Score Function) motivates the use of Maximum Likelihood.

Power Spectral Density - Code

A Stationary Random Process is introduced and each concept from the Theory book (including Energy, Power and the Periodogram is explored for this process. The Einstein-Wiener-Khintchine Theorem allows for an analytic evaluation of the random process. Stationarity is also proved while determining Autocorrelation.

Power Spectral Density - Theory

A thorough introduction to Power Spectral Density (PSD) for a signal, building up from elementary components - Energy, Power, Autocorrelation. The motivation behind the Einstein-Wiener-Khintchine Theorem is explored. The Periodogram as an estimate for the PSD is introduced. This notebook takes care to make clear distinctions between signals and systems at every step, a lack of which, in most popular references, introduces ambiguity.