Science Limelight

Context-Only CCSLM sets the data input to a constant vector, such as (x=all 1's), so that all computational variation is controlled by context . y=Lₙ(Cₙ)... L₂(C₂)L₁(C₁)1 The finite set of context states visited during training becomes the primary organizing structure. Each context selects specialized linear mappings, making context effectively a computational address . This is particularly interesting for robotics and other stateful systems where useful extrinsic context—task, mode, gait phase, contact state, environment, time, etc.—may exist without any sensible analog data vector to supply. It also provides a clean experimental setting for studying context topology, state transitions, specialization, capacity, and computational structure largely independently of conventional input-driven learning.

CCSLM: Context as Parameter Assignment for Expertise Central idea The main role of context is not to determine the output . The main role of context is to assign regions of parameter space in which expertise can grow . Context determines which parameters participate in a particular portion of the learning problem. Repeated exposure to similar contexts causes those parameters to specialize. From context to expertise Context selects or modulates parameters. Training repeatedly adjusts the selected parameters. Over time, particular parameter regions become specialized. These regions can be regarded as future experts . Each expert is fundamentally a linear mapping and linear associative memory . Context is control, not computation Context provides a control pathway. It does not itself transform the data. The selected parameters perform the data transformation. Consequently, the final output is entirely the result of the selected linear mappings and their concatenation. There is no separate...

  When we talk about CCSLM, we tend to talk about linear mappings. But there is another way to look at those same mappings that may have been hiding in plain sight: a linear mapping is also an associative memory. A matrix does not merely transform numbers. It stores associations between patterns. And those memories have engineering properties: capacity limits, noise sensitivity, interference, redundancy, and error-correction behavior. These considerations were once central to the study of associative memory, but they can become strangely distant from view when we look at modern neural networks primarily as layers of learned nonlinear functions. CCSLM brings them back into the foreground. A CCSLM network is a sequence of context-selected linear mappings: Each mapping can therefore be considered an associative memory operating on the representation produced by the preceding mappings. This immediately raises an important question: how observable is the original input as we move throug...

 Learn the basics of CCSLM and neural networks from this video: A key point is ReLU(W₁x) = DW₁x where D is a diagonal decision matrix with 0 or 1 entries according to the corresponding ReLU decisions.  From the viewpoint of the weights in the next layer you have a binary context vector in the diagonal of D operating doing column suppression (parameter selection, linear mapping selection) W₂D but W₂D can be computed to a single matrix L=W₂D.  All the data in the main data channel effectively sees is L and hence ReLU neural networks are concatenated context selected linear mappings.

  Taking a Systems Level view of ReLU (and other) neural networks. Giving the insight necessary to design new types of neural networks.