Posts

CCSLM as Hierarchical Associative Memory

Image
  When we talk about CCSLM, we tend to talk about linear mappings. But there is another way to look at those same mappings that may have been hiding in plain sight: a linear mapping is also an associative memory. A matrix does not merely transform numbers. It stores associations between patterns. And those memories have engineering properties: capacity limits, noise sensitivity, interference, redundancy, and error-correction behavior. These considerations were once central to the study of associative memory, but they can become strangely distant from view when we look at modern neural networks primarily as layers of learned nonlinear functions. CCSLM brings them back into the foreground. A CCSLM network is a sequence of context-selected linear mappings: Each mapping can therefore be considered an associative memory operating on the representation produced by the preceding mappings. This immediately raises an important question: how observable is the original input as we move throug...
Post a Comment
Read more

Concatenated Context Selected Linear Mapping for Neural Networks

Image
 Learn the basics of CCSLM and neural networks from this video: A key point is ReLU(W₁x) = DW₁x where D is a diagonal decision matrix with 0 or 1 entries according to the corresponding ReLU decisions.  From the viewpoint of the weights in the next layer you have a binary context vector in the diagonal of D operating doing column suppression (parameter selection, linear mapping selection) W₂D but W₂D can be computed to a single matrix L=W₂D.  All the data in the main data channel effectively sees is L and hence ReLU neural networks are concatenated context selected linear mappings.
Post a Comment
Read more

Aerial View of ReLU Neural Networks

Image
  Taking a Systems Level view of ReLU (and other) neural networks. Giving the insight necessary to design new types of neural networks.
Post a Comment
Read more

Shifting from a local viewpoint of ReLU to a systems viewpoint

Post a Comment
Read more

Concatenated Context Selected Linear Mappings

  Concatenated Context Selected Linear Mappings Many computational systems can be described as repeatedly applying mathematical transformations to data. One particularly simple and surprisingly expressive architecture can be built from just two ingredients: A method for selecting a context. A collection of linear mappings associated with those contexts. The computation then becomes a sequence of Concatenated Context Selected Linear Mappings (CCSLMs). This description intentionally says nothing about neural networks, activation functions, or even learning. It simply describes a family of computational systems. The Basic Idea Imagine you have many linear transformations available. Instead of always using the same transformation, a context function examines the current state and chooses which one should be applied. Mathematically, y = A c(x) x where x is the current state, c(x) selects a context, A c(x) is the linear mapping associated with that context. The interesting part comes ...
Post a Comment
Read more

Dora the Explorer's Metamorphic River

Image
  A Neural Network as a Metamorphic River.
Post a Comment
Read more

A practical way to view neural networks (whiteboard view)

Image
 A whiteboard view: You can take a decision matrix view of ReLU neural networks. Instead of ReLU(Wx) you create a diagonal matrix with binary 1 or 0 entries according to the ReLU decisions (x>=0?) Then conceptually a layer is DWx. Where D is doing row selection on W. And in fact D does column selection on the weight matrix in the next layer. That is very coarse parameter selection which is linear mapping selection.
Post a Comment
Read more