Switched Linear Coding with Rectified Linear Autoencoders - Johnson & Corcoran 2013
ReLU Networks Really Do Switch Linear Mappings There is a useful way to think about ReLU neural networks that is surprisingly easy to overlook: a ReLU network is a system that switches between linear mappings according to the input. This is not just an analogy. It follows directly from the mathematics. Consider a layer: h = ReLU(Wx + b) The ReLU operation sets negative activations to zero and leaves positive ones unchanged. Define Das a diagonal matrix whose entries are 1 where the corresponding activation is positive and 0 otherwise. We can then write: h = D(Wx + b) Now consider the next layer, with weight matrix V: y = VD(Wx + b) Notice what has happened. Multiplying V by D switches columns of V on or off. The input determines which columns participate in the next linear mapping. In other words, the network is not simply applying one fixed sequence of weight matrices. It is selecting an effective sequence of mappings according to the input. For any fixed activation pattern, the...