Tag
10 posts
Adaptive optimization methods, such as Adam and Adagrad, maintain some statistics over time about the variables and gradients (e.g. moments) which affect the learning rate. These statistics won’t be very accurate when working with sparse tensors, where most of its elements are zero or near zero. We investigated the effects of using Adam with varying…
Reproduced left to right from [8,10,1] Within a 5 day span in October, 4 papers came out that take a significantly different approach to AI hierarchical networks. They are all inspired by biological principles to varying degrees. It’s exciting to see different ways of thinking. Particularly at a time when there is growing skepticism…
Figure 1: The Region-Layer component. The upper surface in the figure is the Region-Layer, which consists of Cells (small rectangles) grouped into Columns. Within each Column, only a few cells are active at any time. The output of the Region-Layer is the activity of the Cells. Columns in the Region-Layer have similar – overlapping –…
Digit classification error over time in our experiments. The image isn’t very helpful but it’s a hint as to why we’re excited :) Project AGI A few weeks ago we paused the “How to build a General Intelligence” series (part 1, part 2, part 3, part 4). We paused it because the next article in…
Erik Laukien is back with a demo of Sparse, Distributed Representation with Reinforcement Learning. This topic is of intense interest to us, although the problem is quite a simple one. SDRs are a natural fit with Reinforcement Learning because bits jointly represent a state. If you associate each bit-pattern with a reward value, it is…
An interesting article by Gerard Rinkus comparing the qualities of sparse distributed representation and quantum computing. In effect, he argues that because distributed representations can simultaneously represent multiple states, you get the same effect as a quantum superposition. The article was originally titled “sparse distributed coding via quantum computing” but I think that gets the…
By Gideon Kowadlo, David Rawlinson and Alan Zhang Can you hear silence or see pitch black? Should we classify no input as a valid state or ignore it? To my knowledge, the machine learning and statistics literature mainly regards an absence of input as missing data. There are several ways that it’s handled. It can…
TL;DR: An SDR is a Sparse Distributed Representation, described below SDRs are biologically plausible data structures SDRs have powerful properties SDRs have received a lot of attention recently There are a few really great new resources on the topic: Presentation by Subutai Ahmad of Numenta Older introductory presentation by Jeff Hawkins Excellent summary of their characteristics in…
This post is about a fantastic new paper by Michael R. Ferrier, titled: Toward a Universal Cortical Algorithm: Examining Hierarchical Temporal Memory in Light of Frontal Cortical Function The paper was posted to the NUPIC mailing list and can be found via: http://numenta.org/community-content.html The paper itself is currently hosted at:…
By David Rawlinson and Gideon Kowadlo Jeff’s new Temporal Pooler This is article 2 in a 3 part series about Temporal Pooling (TP) in MPF/CLA-like algorithms. You can read part 1 here. For the rest of this article we will assume you’ve read part 1. This article is about the new TP proposed by Jeff Hawkins. The…