Science Limelight

Bidirectional initial training of (Atlas LSH or similar neural networks) for nice preconditioning. https://archive.org/details/bidirectional-conditioning-of-context-selected-linear-networks

 

  https://archive.org/details/ccslm-matrix-products-alignment-rank-and-energy

 

  CCSLM via the ReLU entry point Break a ReLU neural network into a main data channel and a context channel . ReLU(Wx) = DWx , where D is a diagonal decision matrix containing 0/1 entries produced by a context function applied to the layer input—in this case, the ReLU decisions. The main data channel therefore only ever sees linear mappings : the W matrices and the D selection matrices. This is a remarkable restriction on apparent expressiveness, but it has potentially very positive consequences. The context channel is doing something quite different. It does not transform the data; it selects which linear mapping acts on the data . A sequence of binary decisions across layers can represent an exponentially large number of distinct symbolic contexts or computational paths, even though each individual decision is simple. The selected mapping can be made explicit by fusing D with an adjacent weight matrix. L = WD is the more useful representation for human comprehension: the co...

The core mathematics of concatenated context selected linear mappings https://archive.org/details/core-ccslm

One for the skeptics but it is not argued as well as possible. It still retains some old fashioned views of neural networks. https://docs.google.com/document/d/e/2PACX-1vRLqFOuoGC3jS9HqJebDW5PHscznODG5XaCrsVN7MmoMRVkDl2r5Iu8sEjLpoN6vuH-PoAer0M1BT1w/pub

Context-Only CCSLM sets the data input to a constant vector, such as (x=all 1's), so that all computational variation is controlled by context . y=Lₙ(Cₙ)... L₂(C₂)L₁(C₁)1 The finite set of context states visited during training becomes the primary organizing structure. Each context selects specialized linear mappings, making context effectively a computational address . This is particularly interesting for robotics and other stateful systems where useful extrinsic context—task, mode, gait phase, contact state, environment, time, etc.—may exist without any sensible analog data vector to supply. It also provides a clean experimental setting for studying context topology, state transitions, specialization, capacity, and computational structure largely independently of conventional input-driven learning.

  When we talk about CCSLM, we tend to talk about linear mappings. But there is another way to look at those same mappings that may have been hiding in plain sight: a linear mapping is also an associative memory. A matrix does not merely transform numbers. It stores associations between patterns. And those memories have engineering properties: capacity limits, noise sensitivity, interference, redundancy, and error-correction behavior. These considerations were once central to the study of associative memory, but they can become strangely distant from view when we look at modern neural networks primarily as layers of learned nonlinear functions. CCSLM brings them back into the foreground. A CCSLM network is a sequence of context-selected linear mappings: Each mapping can therefore be considered an associative memory operating on the representation produced by the preceding mappings. This immediately raises an important question: how observable is the original input as we move throug...

 Learn the basics of CCSLM and neural networks from this video: A key point is ReLU(W₁x) = DW₁x where D is a diagonal decision matrix with 0 or 1 entries according to the corresponding ReLU decisions.  From the viewpoint of the weights in the next layer you have a binary context vector in the diagonal of D operating doing column suppression (parameter selection, linear mapping selection) W₂D but W₂D can be computed to a single matrix L=W₂D.  All the data in the main data channel effectively sees is L and hence ReLU neural networks are concatenated context selected linear mappings.

  Taking a Systems Level view of ReLU (and other) neural networks. Giving the insight necessary to design new types of neural networks.

  Concatenated Context Selected Linear Mappings Many computational systems can be described as repeatedly applying mathematical transformations to data. One particularly simple and surprisingly expressive architecture can be built from just two ingredients: A method for selecting a context. A collection of linear mappings associated with those contexts. The computation then becomes a sequence of Concatenated Context Selected Linear Mappings (CCSLMs). This description intentionally says nothing about neural networks, activation functions, or even learning. It simply describes a family of computational systems. The Basic Idea Imagine you have many linear transformations available. Instead of always using the same transformation, a context function examines the current state and chooses which one should be applied. Mathematically, y = A c(x) x where x is the current state, c(x) selects a context, A c(x) is the linear mapping associated with that context. The interesting part comes ...

  A Neural Network as a Metamorphic River.

 A whiteboard view: You can take a decision matrix view of ReLU neural networks. Instead of ReLU(Wx) you create a diagonal matrix with binary 1 or 0 entries according to the ReLU decisions (x>=0?) Then conceptually a layer is DWx. Where D is doing row selection on W. And in fact D does column selection on the weight matrix in the next layer. That is very coarse parameter selection which is linear mapping selection.

  Atlas LSH Neural Networks: Geometry as Context Most neural networks perform essentially the same computation for every input. Every layer applies the same collection of parameters, regardless of what the input actually is. Atlas LSH neural networks explore a different idea: use the geometry of the input to determine which computation should be performed. The first step is to take a compact geometric sample of the input using locality-sensitive hashing (LSH). Each LSH bit can be viewed as asking a simple geometric question about the input vector—for example, which side of a randomly oriented hyperplane it lies on. A few hundred such bits form a sparse fingerprint describing the input's approximate location in a high-dimensional space. The important point is that these bits are not trying to represent the input in detail. Instead, they capture context . Similar inputs tend to produce similar bit patterns, so the LSH serves as a geometry sampler that identifies the neighborhood in ...