Information-Theoretic Analysis of Convolutional Autoencoders
Information Theory; neural networks; convolutional autoencoder.
The use of Information Theory concepts to understand deep neural networks has been extensively explored in recent years. The Information Theoretic Learning framework that resulted from such use has been acknowledged as a potentially important tool to comprehend the learning mechanisms employed during the deep neural networks’ training process, for the study of which theoretical and systematic methods of analysis are still lacking. The use of statistical measurements derived from Information Theory such as entropy and mutual information has allowed for a better understanding of how the information flows through the aforementioned networks during their training. It also enabled the creation of systematic methods to design and analyze these networks in a more rigorous manner, which in turn allows the creation of more efficient and robust architectures. This work aims to investigate the possibility of application of a method based on the aforementioned framework for the automatic detection of the bottleneck dimension of a convolutional autoencoder, whose objective is to find the optimal compression for the
images presented to it.