Sparse representations and capsules

Continuous online learning of sparse representations (2017-2018)

This project was the foundation of our approach to learning representations of data, with ambitious criteria – continuous, online, unsupervised learning of sparse distributed representations, resulting in state-of-the-art performance even given nonstationary input. We reviewed a broad range of historical techniques and experimented some novel mashups of older competitive learning and modern convolutional networks. We obtained some fundamental insights into effective sparse representations, and how to train them.

• Convolutional competitive learning vs sparse autoencoders (part 1)
• Convolutional competitive learning vs sparse autoencoders (part 2)

Predictive Capsules (2018)

We believe that Capsules networks promise inherently better generalization, a key weakness of conventional artificial neural networks.

We published an initial paper on unsupervised sparse Capsules earlier this year, extending the work of Sabour et al to only allow local, unsupervised training, and arguably obtained much better generalization. We are now developing a much better understanding of Capsules and how they might be implemented by Pyramidal neurons.

• Sparse unsupervised Capsules
• Understanding Equivariance

Since we ended this project, Kosiorek et al have developed a better version of the same ideas called “Stacked Capsule Autoencoders”. Their results are better too!