Posts

Concatenated Context Selected Linear Mappings

  Concatenated Context Selected Linear Mappings Many computational systems can be described as repeatedly applying mathematical transformations to data. One particularly simple and surprisingly expressive architecture can be built from just two ingredients: A method for selecting a context. A collection of linear mappings associated with those contexts. The computation then becomes a sequence of Concatenated Context Selected Linear Mappings (CCSLMs). This description intentionally says nothing about neural networks, activation functions, or even learning. It simply describes a family of computational systems. The Basic Idea Imagine you have many linear transformations available. Instead of always using the same transformation, a context function examines the current state and chooses which one should be applied. Mathematically, y = A c(x) x where x is the current state, c(x) selects a context, A c(x) is the linear mapping associated with that context. The interesting part comes ...
Post a Comment
Read more

Dora the Explorer's Metamorphic River

Image
  A Neural Network as a Metamorphic River.
Post a Comment
Read more

A practical way to view neural networks (whiteboard view)

Image
 A whiteboard view: You can take a decision matrix view of ReLU neural networks. Instead of ReLU(Wx) you create a diagonal matrix with binary 1 or 0 entries according to the ReLU decisions (x>=0?) Then conceptually a layer is DWx. Where D is doing row selection on W. And in fact D does column selection on the weight matrix in the next layer. That is very coarse parameter selection which is linear mapping selection.
Post a Comment
Read more

A Practical Way to Think About Neural Networks

  Binary Context: A Practical Way to Think About Neural Networks Much of deep learning can be understood in surprisingly simple terms. Rather than thinking about neurons "computing features," it is often more useful to think of them as selecting which computation should happen next . The selected computation is almost always linear. The interesting part is not the linear mapping itself. The interesting part is how the network chooses which linear mapping to use. Binary context Consider a layer with a binary context vector. The context might come from: ReLU activations (positive or zero) threshold units locality sensitive hash (LSH) bits routing networks mixture-of-experts gates any other binary routing mechanism The binary vector does not directly solve the problem. Instead it answers a much simpler question: Which parameters should participate in the next computation? Once those parameters are selected, the layer performs an ordinary linear mapping. So every layer is essenti...
Post a Comment
Read more

Music to code neural neural networks by

Image
 
Post a Comment
Read more

Atlas LSH Neural Networks: Geometry as Context

Image
  Atlas LSH Neural Networks: Geometry as Context Most neural networks perform essentially the same computation for every input. Every layer applies the same collection of parameters, regardless of what the input actually is. Atlas LSH neural networks explore a different idea: use the geometry of the input to determine which computation should be performed. The first step is to take a compact geometric sample of the input using locality-sensitive hashing (LSH). Each LSH bit can be viewed as asking a simple geometric question about the input vector—for example, which side of a randomly oriented hyperplane it lies on. A few hundred such bits form a sparse fingerprint describing the input's approximate location in a high-dimensional space. The important point is that these bits are not trying to represent the input in detail. Instead, they capture context . Similar inputs tend to produce similar bit patterns, so the LSH serves as a geometry sampler that identifies the neighborhood in ...
Post a Comment
Read more

Walsh Hadamard Agreement & Disagreement for the iterated system HD

Image
  This interactive sketch explores how a binary diagonal matrix D interacts with the Walsh–Hadamard mixing matrix H.  Each entry of D can be flipped between +1 and -1, dividing the nodes into two groups.  The code visualizes two derived operators:  ●​ X = HD + DH  ●​ Y = HD - DH  These have a surprisingly clean interpretation:  ●​ X keeps interactions between nodes with the same sign (“agreement”).  ●​ Y keeps interactions between nodes with opposite signs (“disagreement”). Because the Walsh–Hadamard matrix contains only ±1 entries, the resulting matrix values become simple:  ●​ reinforcement: +2  ●​ cancellation: 0  ●​ opposite reinforcement: -2  The graph views make this intuitive: ●​ The graph of X splits into same-sign clusters.  ●​ The graph of Y becomes a bipartite interaction graph between opposite-sign groups.  As entries of D are flipped, the system transitions between:  ●​ uniform structure (all signs e...
Post a Comment
Read more