Posts

Sign Flipping before FWHT: Matrix Synthesis

Image
  A simple way to introduce context based mutability in concatenated context selected linear mappings.  Typically multiplied with a block sparse weight matrix to hold the learnable parameters.  Avoids having to directly switch matrix elements within a mutable matrix reducing memory address calculation and parameter fetching costs.  Disadvantage: lots of geometric disturbance. Better for more shallow neural networks. A difference between element swapping and sign flipping is that 2 swapped elements could be similar in value giving little change, a sign flip is always large. Concept could be used with ordinary weight matrices.

Learned Key Synthesis for OOD Tasks

Image
  Learned key synthesis for OOD tasks

The Hidden Geometry of Intelligence

Image
The Hidden Geometry of Intelligence: Why Big Changes Can Happen Gradually We often look at a neural network and see representations changing dramatically from one layer to the next. A dog enters the network; somewhere deep inside, its representation may look nothing like the original. By the end, the network can recognise a dog—or perhaps distinguish it from a cat. But here is an intriguing possibility: A representation can change dramatically in appearance while undergoing only modest geometric deformation at each step. The secret may lie in the difference between reorientation and deformation . Three simple geometric operations Consider a linear transformation represented by a matrix A. Its singular value decomposition is: A = UΣVᵀ This expresses the transformation as three operations: Vᵀ: Reorient the input representation. Σ: Expand or contract selected directions. U: Reorient the resulting representation again. The two orthogonal matrices, U and V, can rotate or reflect a repre...

From Dog to Cat

Image
  From Dog to Cat: A More Geometric Way to Think About Artificial Intelligence Imagine asking an AI system to turn a picture of a dog into a picture of a cat. One approach is to build a complicated collection of nonlinear functions and train them to produce the desired result. With enough layers and training, this can work remarkably well. But there is a catch: complicated nonlinear transformations can behave unpredictably. A tiny change in the input can sometimes produce a surprisingly large change in the output. A transformation that works well on familiar examples may behave strangely on examples that lie between them. As layers are composed, small distortions can accumulate, creating complicated geometric structures. There may be another way to approach the problem. Think in terms of a journey, not a jump Instead of asking for one complicated function that turns a dog into a cat, imagine breaking the transformation into a sequence of smaller steps. For example: Start with the o...

Switched Linear Coding with Rectified Linear Autoencoders - Johnson & Corcoran 2013

  ReLU Networks Really Do Switch Linear Mappings There is a useful way to think about ReLU neural networks that is surprisingly easy to overlook: a ReLU network is a system that switches between linear mappings according to the input. This is not just an analogy. It follows directly from the mathematics. Consider a layer: h = ReLU(Wx + b) The ReLU operation sets negative activations to zero and leaves positive ones unchanged. Define Das a diagonal matrix whose entries are 1 where the corresponding activation is positive and 0 otherwise. We can then write: h = D(Wx + b) Now consider the next layer, with weight matrix V: y = VD(Wx + b) Notice what has happened. Multiplying V by D switches columns of V on or off. The input determines which columns participate in the next linear mapping. In other words, the network is not simply applying one fixed sequence of weight matrices. It is selecting an effective sequence of mappings according to the input. For any fixed activation pattern, the...

CCSLM and Collision Decision Trees

Image
  CCSLM and Collision Decision Trees What happens when a neural network reuses a finite set of linear mappings across many different decision paths? This note explores Concatenated Context Selected Linear Mappings (CCSLM), where context selects the computation and linear mappings transform the data. Reusing mappings creates collision decision trees : distinct paths can share the same computational operator, while linear associative memory allows that operator to respond differently to different inputs. The report concludes with three fundamental computational elements: decision-tree collisions, geometric reorientations, and singular-value modulation . Together, they offer a compact framework for thinking about context-dependent computation, parameter reuse, and the geometry of neural networks. Note: Collision Decision Trees Audio Summary