From Dog to Cat

 

From Dog to Cat: A More Geometric Way to Think About Artificial Intelligence



Imagine asking an AI system to turn a picture of a dog into a picture of a cat.

One approach is to build a complicated collection of nonlinear functions and train them to produce the desired result. With enough layers and training, this can work remarkably well.

But there is a catch: complicated nonlinear transformations can behave unpredictably.

A tiny change in the input can sometimes produce a surprisingly large change in the output. A transformation that works well on familiar examples may behave strangely on examples that lie between them. As layers are composed, small distortions can accumulate, creating complicated geometric structures.

There may be another way to approach the problem.

Think in terms of a journey, not a jump

Instead of asking for one complicated function that turns a dog into a cat, imagine breaking the transformation into a sequence of smaller steps.

For example:

  1. Start with the original dog.

  2. Gradually reduce features that make it distinctly dog-like.

  3. Reorganise the remaining features.

  4. Introduce features associated with cats.

  5. Arrive at the final cat representation.

The intermediate stages might look unusual. They need not be recognisable as either dogs or cats. What matters is that each step makes a controlled contribution to the overall transformation.

This is the idea of factorized metamorphosis: a complicated transformation is constructed as a sequence of simpler transformations.

The mathematics is surprisingly elegant

A matrix can be decomposed using singular value decomposition (SVD):

A = UΣVᵀ

This gives us three basic operations:

  • Rotation or reflection: Changes the orientation of a representation without changing distances.

  • Singular-value modulation: Expands or contracts selected directions.

  • Another rotation or reflection: Reorients the resulting representation.

Think of the rotation operations as rearranging the coordinate system, while singular-value modulation changes the relative importance or magnitude of different directions.

A sequence of such matrices can progressively reshape a representation.

Importantly, rotations and reflections preserve information. Singular-value modulation also preserves information when it is reversible, although it changes geometry. Information is genuinely lost when dimensions collapse.

Why might this be better?

The attraction is geometric discipline.

A fixed matrix transformation has some very useful properties:

  • Straight lines remain straight.

  • Parallel lines remain parallel.

  • Convex regions remain convex.

  • It cannot fold one part of the input space over another.

  • Its sensitivity to input changes can be bounded using its singular values.

Compare this with arbitrary nonlinear functions, which can bend, stretch and fold the input space in complicated ways.

When many such functions are composed, their local sensitivities multiply. Even relatively modest distortions can accumulate.

The result can be a transformation that works beautifully on the examples it has seen but behaves unpredictably elsewhere.

The interesting part: make the matrices mutable

There is no need for every input to undergo exactly the same sequence of transformations.

The matrices could depend on the current representation.

A dog with a long snout might require a different transformation path from a short-snouted dog. The system could select different rotations and singular-value modulations according to the context.

This makes the overall mapping nonlinear, even though each individual step is a structured matrix operation.

Of course, the matrix-selection mechanism must itself be controlled. If it changes wildly in response to tiny input differences, we simply recreate the original instability in a different form.

The goal is to make nearby inputs undergo nearby transformations.

A different philosophy of AI

The usual approach is to learn a sufficiently powerful function and rely on training to make it behave sensibly.

The alternative is to restrict the kinds of transformations the system is allowed to perform.

Rather than asking an AI to invent an arbitrary route from dog to cat, we give it a geometric vocabulary:

  • Rotate.

  • Reflect.

  • Expand.

  • Contract.

  • Reorient.

  • Repeat.

The final transformation can be substantial, even though every individual step is relatively modest.

This is not a claim that matrix chains can replace all nonlinear functions. Fixed matrix products are ultimately just linear transformations, and genuine nonlinear behaviour requires context-dependent selection or other nonlinear mechanisms.

But it suggests an intriguing research direction:

Perhaps intelligent transformation is less about inventing increasingly complicated functions and more about discovering the right sequence of controlled changes.

Instead of learning just where to go, perhaps an intelligent system should also learn how to get there.

Comments

Post a Comment

Popular posts from this blog

Neon Bulb Oscillators

Image
Neon Tube Oscillators  Fig. 1 High voltage to headphones, don't do this in modern times. Very high impedance headphones are implied. A neon bulb is useful in many ways in an amateur station, but there are undoubtedly many amateurs who are not aware of the fact that the device can be made to oscillate at either audio or radio frequencies. Fig. 1 shows how a single bulb may be hooked up as an audio frequency oscillator. The L-C circuit is formed by the headphones and a condenser, and by varying the capacity the circuit can be made to oscillate at frequencies from a few cycles per second up to the highest audio frequencies.  The variable resistor in series with the bulb and battery controls the strength of oscillation, and also controls the frequency to some extent. A 1 nf variable condenser will be sufficient to cover most of the audio range.  Fig. 2 The circuit in Fig. 2 may be used when an RF oscillator is to be modulated at audio frequencies. The transformer may be an or...
Read more

A practical way to view neural networks (whiteboard view)

Image
 A whiteboard view: You can take a decision matrix view of ReLU neural networks. Instead of ReLU(Wx) you create a diagonal matrix with binary 1 or 0 entries according to the ReLU decisions (x>=0?) Then conceptually a layer is DWx. Where D is doing row selection on W. And in fact D does column selection on the weight matrix in the next layer. That is very coarse parameter selection which is linear mapping selection.
Read more

CCSLM via the ReLU entry point - Flyer

Image
  CCSLM via the ReLU entry point Break a ReLU neural network into a main data channel and a context channel . ReLU(Wx) = DWx , where D is a diagonal decision matrix containing 0/1 entries produced by a context function applied to the layer input—in this case, the ReLU decisions. The main data channel therefore only ever sees linear mappings : the W matrices and the D selection matrices. This is a remarkable restriction on apparent expressiveness, but it has potentially very positive consequences. The context channel is doing something quite different. It does not transform the data; it selects which linear mapping acts on the data . A sequence of binary decisions across layers can represent an exponentially large number of distinct symbolic contexts or computational paths, even though each individual decision is simple. The selected mapping can be made explicit by fusing D with an adjacent weight matrix. L = WD is the more useful representation for human comprehension: the co...
Read more