Posts
Concatenated Context Selected Linear Mappings
Concatenated Context Selected Linear Mappings Many computational systems can be described as repeatedly applying mathematical transformations to data. One particularly simple and surprisingly expressive architecture can be built from just two ingredients: A method for selecting a context. A collection of linear mappings associated with those contexts. The computation then becomes a sequence of Concatenated Context Selected Linear Mappings (CCSLMs). This description intentionally says nothing about neural networks, activation functions, or even learning. It simply describes a family of computational systems. The Basic Idea Imagine you have many linear transformations available. Instead of always using the same transformation, a context function examines the current state and chooses which one should be applied. Mathematically, y = A c(x) x where x is the current state, c(x) selects a context, A c(x) is the linear mapping associated with that context. The interesting part comes ...
A practical way to view neural networks (whiteboard view)
A whiteboard view: You can take a decision matrix view of ReLU neural networks. Instead of ReLU(Wx) you create a diagonal matrix with binary 1 or 0 entries according to the ReLU decisions (x>=0?) Then conceptually a layer is DWx. Where D is doing row selection on W. And in fact D does column selection on the weight matrix in the next layer. That is very coarse parameter selection which is linear mapping selection.
A Practical Way to Think About Neural Networks
Binary Context: A Practical Way to Think About Neural Networks Much of deep learning can be understood in surprisingly simple terms. Rather than thinking about neurons "computing features," it is often more useful to think of them as selecting which computation should happen next . The selected computation is almost always linear. The interesting part is not the linear mapping itself. The interesting part is how the network chooses which linear mapping to use. Binary context Consider a layer with a binary context vector. The context might come from: ReLU activations (positive or zero) threshold units locality sensitive hash (LSH) bits routing networks mixture-of-experts gates any other binary routing mechanism The binary vector does not directly solve the problem. Instead it answers a much simpler question: Which parameters should participate in the next computation? Once those parameters are selected, the layer performs an ordinary linear mapping. So every layer is essenti...
Atlas LSH Neural Networks: Geometry as Context
Atlas LSH Neural Networks: Geometry as Context Most neural networks perform essentially the same computation for every input. Every layer applies the same collection of parameters, regardless of what the input actually is. Atlas LSH neural networks explore a different idea: use the geometry of the input to determine which computation should be performed. The first step is to take a compact geometric sample of the input using locality-sensitive hashing (LSH). Each LSH bit can be viewed as asking a simple geometric question about the input vector—for example, which side of a randomly oriented hyperplane it lies on. A few hundred such bits form a sparse fingerprint describing the input's approximate location in a high-dimensional space. The important point is that these bits are not trying to represent the input in detail. Instead, they capture context . Similar inputs tend to produce similar bit patterns, so the LSH serves as a geometry sampler that identifies the neighborhood in ...