Science Limelight

  Taking a Systems Level view of ReLU (and other) neural networks. Giving the insight necessary to design new types of neural networks.

  Concatenated Context Selected Linear Mappings Many computational systems can be described as repeatedly applying mathematical transformations to data. One particularly simple and surprisingly expressive architecture can be built from just two ingredients: A method for selecting a context. A collection of linear mappings associated with those contexts. The computation then becomes a sequence of Concatenated Context Selected Linear Mappings (CCSLMs). This description intentionally says nothing about neural networks, activation functions, or even learning. It simply describes a family of computational systems. The Basic Idea Imagine you have many linear transformations available. Instead of always using the same transformation, a context function examines the current state and chooses which one should be applied. Mathematically, y = A c(x) x where x is the current state, c(x) selects a context, A c(x) is the linear mapping associated with that context. The interesting part comes ...

  A Neural Network as a Metamorphic River.

 A whiteboard view: You can take a decision matrix view of ReLU neural networks. Instead of ReLU(Wx) you create a diagonal matrix with binary 1 or 0 entries according to the ReLU decisions (x>=0?) Then conceptually a layer is DWx. Where D is doing row selection on W. And in fact D does column selection on the weight matrix in the next layer. That is very coarse parameter selection which is linear mapping selection.

  Binary Context: A Practical Way to Think About Neural Networks Much of deep learning can be understood in surprisingly simple terms. Rather than thinking about neurons "computing features," it is often more useful to think of them as selecting which computation should happen next . The selected computation is almost always linear. The interesting part is not the linear mapping itself. The interesting part is how the network chooses which linear mapping to use. Binary context Consider a layer with a binary context vector. The context might come from: ReLU activations (positive or zero) threshold units locality sensitive hash (LSH) bits routing networks mixture-of-experts gates any other binary routing mechanism The binary vector does not directly solve the problem. Instead it answers a much simpler question: Which parameters should participate in the next computation? Once those parameters are selected, the layer performs an ordinary linear mapping. So every layer is essenti...