← All research

Earlier research · 2018–2020

Learning without backpropagation through time (RSM)

Brief: Using Reinforcement Learning to discover attentional strategies ↗

How we got here

  1. 2019
  2. 2020
    An Nguyen, Masters · Monash RL for attentional strategiesAbout An’s project →RL to learn attentional strategies for RSM

Project notes

Reinforcement learning of attentional strategies (2019-2020)

Architecture diagram: three RSM memory layers receive input; an attention-filtering module (e.g. a DQN) sends Clear and Skip signals to them, and a predictor and reward feed back to it.

Recently Transformers and other attentional neural architecture have shown great promise in a number of domains. How can we align the deep backpropagation used in Transformers with our desire to use biologically plausible local credit assignment? We aim to use the Bellman equation – better known as discounted Reinforcement Learning – to learn attentional strategies for controlling a set of RSM memory “heads” such that their predictive abilities are maximized.

Learning distant cause & effect with only local credit assignment (2018-2019)

Diagram of the Recurrent Sparse Memory (RSM) processing a digit sequence over times t-1, t and t+1, predicting the next digit class from a sparse code with a loss on the prediction.

In May we introduced our Recurrent Sparse Memory (RSM). We showed that it can learn to associate distant casuses and effects, higher-order and partially observable sequences, and also tested it on natural language modelling. RSM is demonstrated to possess capabilities that previously required deep backpropagation and gated memory networks such as LSTM, while using more computationally and memory efficient local learning rules.

Alternatives to BackPropagation Through Time (BPTT)

Small graph of two coloured node chains (R1, G1, B1 and G2, B2, R2) both leading to a shared black node BK, illustrating a sequence with a shared ambiguous element.

We are intensely interested in biologically plausible alternatives to backpropagation through time (BPTT). BPTT is used to associate causes and effects that are widely separated in time. The problem is that it requires storage of partial derivatives for all synaptic weights for all time-steps up to a fixed horizon (e.g. 1000 steps). Not only is this memory intensive, the finite time window is very restrictive. There is no neurological equivalent to BPTT – nature does it another way, which we hope to copy.

This project led to the development of our RSM algorithm.