Posts

Context as Parameter Assignment for Expertise

Image
CCSLM: Context as Parameter Assignment for Expertise Central idea The main role of context is not to determine the output . The main role of context is to assign regions of parameter space in which expertise can grow . Context determines which parameters participate in a particular portion of the learning problem. Repeated exposure to similar contexts causes those parameters to specialize. From context to expertise Context selects or modulates parameters. Training repeatedly adjusts the selected parameters. Over time, particular parameter regions become specialized. These regions can be regarded as future experts . Each expert is fundamentally a linear mapping and linear associative memory . Context is control, not computation Context provides a control pathway. It does not itself transform the data. The selected parameters perform the data transformation. Consequently, the final output is entirely the result of the selected linear mappings and their concatenation. There is no separate...
Post a Comment
Read more

CCSLM as Hierarchical Associative Memory

Image
  When we talk about CCSLM, we tend to talk about linear mappings. But there is another way to look at those same mappings that may have been hiding in plain sight: a linear mapping is also an associative memory. A matrix does not merely transform numbers. It stores associations between patterns. And those memories have engineering properties: capacity limits, noise sensitivity, interference, redundancy, and error-correction behavior. These considerations were once central to the study of associative memory, but they can become strangely distant from view when we look at modern neural networks primarily as layers of learned nonlinear functions. CCSLM brings them back into the foreground. A CCSLM network is a sequence of context-selected linear mappings: Each mapping can therefore be considered an associative memory operating on the representation produced by the preceding mappings. This immediately raises an important question: how observable is the original input as we move throug...
Post a Comment
Read more

Concatenated Context Selected Linear Mapping for Neural Networks

Image
 Learn the basics of CCSLM and neural networks from this video: A key point is ReLU(W₁x) = DW₁x where D is a diagonal decision matrix with 0 or 1 entries according to the corresponding ReLU decisions.  From the viewpoint of the weights in the next layer you have a binary context vector in the diagonal of D operating doing column suppression (parameter selection, linear mapping selection) W₂D but W₂D can be computed to a single matrix L=W₂D.  All the data in the main data channel effectively sees is L and hence ReLU neural networks are concatenated context selected linear mappings.
Post a Comment
Read more

Aerial View of ReLU Neural Networks

Image
  Taking a Systems Level view of ReLU (and other) neural networks. Giving the insight necessary to design new types of neural networks.
Post a Comment
Read more

Shifting from a local viewpoint of ReLU to a systems viewpoint

Post a Comment
Read more

Concatenated Context Selected Linear Mappings

  Concatenated Context Selected Linear Mappings Many computational systems can be described as repeatedly applying mathematical transformations to data. One particularly simple and surprisingly expressive architecture can be built from just two ingredients: A method for selecting a context. A collection of linear mappings associated with those contexts. The computation then becomes a sequence of Concatenated Context Selected Linear Mappings (CCSLMs). This description intentionally says nothing about neural networks, activation functions, or even learning. It simply describes a family of computational systems. The Basic Idea Imagine you have many linear transformations available. Instead of always using the same transformation, a context function examines the current state and chooses which one should be applied. Mathematically, y = A c(x) x where x is the current state, c(x) selects a context, A c(x) is the linear mapping associated with that context. The interesting part comes ...
Post a Comment
Read more

Dora the Explorer's Metamorphic River

Image
  A Neural Network as a Metamorphic River.
Post a Comment
Read more