Posts

The Hidden Geometry of Intelligence

Image
The Hidden Geometry of Intelligence: Why Big Changes Can Happen Gradually We often look at a neural network and see representations changing dramatically from one layer to the next. A dog enters the network; somewhere deep inside, its representation may look nothing like the original. By the end, the network can recognise a dog—or perhaps distinguish it from a cat. But here is an intriguing possibility: A representation can change dramatically in appearance while undergoing only modest geometric deformation at each step. The secret may lie in the difference between reorientation and deformation . Three simple geometric operations Consider a linear transformation represented by a matrix A. Its singular value decomposition is: A = UΣVᵀ This expresses the transformation as three operations: Vᵀ: Reorient the input representation. Σ: Expand or contract selected directions. U: Reorient the resulting representation again. The two orthogonal matrices, U and V, can rotate or reflect a repre...
Post a Comment
Read more

From Dog to Cat

Image
  From Dog to Cat: A More Geometric Way to Think About Artificial Intelligence Imagine asking an AI system to turn a picture of a dog into a picture of a cat. One approach is to build a complicated collection of nonlinear functions and train them to produce the desired result. With enough layers and training, this can work remarkably well. But there is a catch: complicated nonlinear transformations can behave unpredictably. A tiny change in the input can sometimes produce a surprisingly large change in the output. A transformation that works well on familiar examples may behave strangely on examples that lie between them. As layers are composed, small distortions can accumulate, creating complicated geometric structures. There may be another way to approach the problem. Think in terms of a journey, not a jump Instead of asking for one complicated function that turns a dog into a cat, imagine breaking the transformation into a sequence of smaller steps. For example: Start with the o...
Post a Comment
Read more

Switched Linear Coding with Rectified Linear Autoencoders - Johnson & Corcoran 2013

  ReLU Networks Really Do Switch Linear Mappings There is a useful way to think about ReLU neural networks that is surprisingly easy to overlook: a ReLU network is a system that switches between linear mappings according to the input. This is not just an analogy. It follows directly from the mathematics. Consider a layer: h = ReLU(Wx + b) The ReLU operation sets negative activations to zero and leaves positive ones unchanged. Define Das a diagonal matrix whose entries are 1 where the corresponding activation is positive and 0 otherwise. We can then write: h = D(Wx + b) Now consider the next layer, with weight matrix V: y = VD(Wx + b) Notice what has happened. Multiplying V by D switches columns of V on or off. The input determines which columns participate in the next linear mapping. In other words, the network is not simply applying one fixed sequence of weight matrices. It is selecting an effective sequence of mappings according to the input. For any fixed activation pattern, the...
Post a Comment
Read more

CCSLM and Collision Decision Trees

Image
  CCSLM and Collision Decision Trees What happens when a neural network reuses a finite set of linear mappings across many different decision paths? This note explores Concatenated Context Selected Linear Mappings (CCSLM), where context selects the computation and linear mappings transform the data. Reusing mappings creates collision decision trees : distinct paths can share the same computational operator, while linear associative memory allows that operator to respond differently to different inputs. The report concludes with three fundamental computational elements: decision-tree collisions, geometric reorientations, and singular-value modulation . Together, they offer a compact framework for thinking about context-dependent computation, parameter reuse, and the geometry of neural networks. Note: Collision Decision Trees Audio Summary
Post a Comment
Read more

Soda can & Straw = Variable Capacitor

Image
Easy, money saving variable capacitor for hobby electronics. Why pay $15++ for something you can easily make today from junk. Mostly okay AI image, however just use a normal office stapler. Text version: Soda Can & Straw Capacitor
Post a Comment
Read more

Context-Controlled Linear Mappings Using Binary Sign Flips

Image
Context-Controlled Linear Mappings Using Binary Sign Flips and Fast Walsh–Hadamard Transforms 1. Generating a constrained binary context The first component constructs a binary context vector from analog control signals and fixed Gaussian random vectors. Begin with analog signals, for example random projections of the input. Associate each signal with a fixed random vector whose components are sampled from a Gaussian distribution. Multiply each random vector by its corresponding analog signal and sum the weighted vectors. Binarize the resulting vector into +1 and -1 values. The result is a binary context that controls the signs of selected input dimensions. This construction provides a way to restrict the binary context space. Although an m-dimensional binary vector has 2ⁿ possible patterns in principle, a low-dimensional analog control process will reach only a subset of them. The context mechanism can therefore introduce useful variation without necessarily exposing the trainable map...
1 comment
Read more

Miscellaneous

Image
  How the deep learning community arrived at context-selected linear mappings through empirical convergence driven by gradient stability and gradualism, rather than starting from a deliberate, first-principles understanding of switched linear systems: The late afternoon sun was streaming into Terence Tao’s UCLA office, casting long shadows across floating stacks of journals and half-filled chalkboards. Inside, the usual quiet atmosphere had entirely collapsed. Tao was practically pacing a circuit around his coffee table, hands gesturing frantically toward a printout of the whiteboard diagram. "How can this be?!" Tao exclaims, turning on his heel. "How can this be?! We spend decades treating non-linearity like a foundational wall—a continuous, static warping of the metric space—and the entire deep learning empirical engine was just... bypassing it? Routing around it with soft switches?!" His graduate student, Maya, stood quietly by the doorway holding a steaming ce...
Post a Comment
Read more