Laplace-Based Estimators for Bayesian Design and Information Theory

Project description

This project focuses on enhancing the efficiency of Information Theory in the context of machine learning. We will devise a framework by addressing computational challenges associated with estimating the expected information gain and the information bottleneck functionals. These functionals involve an outer and inner integral separated by a nonlinear function applied to the inner integral.

Given uncertainties inherent in the mathematical model concerning both parameters of interest and nuisance parameters, we will propose novel estimators leveraging the Laplace approximation to alleviate computational burdens: Laplace's method followed by a Laplace approximation and a Dual Laplace-Based Importance Sampling Estimator.

As for the applications, we will focus on the Bayesian Design of Physical Experiments: Empowering strategic parameter selection for enhanced inference and resource optimization and Information Bottleneck in Neural Network Training: Leveraging the information bottleneck as a loss function for neural networks showcases principled feature compression, promoting model robustness and improved generalization in machine learning tasks.