Science Limelight

The Hidden Geometry of Intelligence: Why Big Changes Can Happen Gradually We often look at a neural network and see representations changing dramatically from one layer to the next. A dog enters the network; somewhere deep inside, its representation may look nothing like the original. By the end, the network can recognise a dog—or perhaps distinguish it from a cat. But here is an intriguing possibility: A representation can change dramatically in appearance while undergoing only modest geometric deformation at each step. The secret may lie in the difference between reorientation and deformation . Three simple geometric operations Consider a linear transformation represented by a matrix A. Its singular value decomposition is: A = UΣVᵀ This expresses the transformation as three operations: Vᵀ: Reorient the input representation. Σ: Expand or contract selected directions. U: Reorient the resulting representation again. The two orthogonal matrices, U and V, can rotate or reflect a repre...

  From Dog to Cat: A More Geometric Way to Think About Artificial Intelligence Imagine asking an AI system to turn a picture of a dog into a picture of a cat. One approach is to build a complicated collection of nonlinear functions and train them to produce the desired result. With enough layers and training, this can work remarkably well. But there is a catch: complicated nonlinear transformations can behave unpredictably. A tiny change in the input can sometimes produce a surprisingly large change in the output. A transformation that works well on familiar examples may behave strangely on examples that lie between them. As layers are composed, small distortions can accumulate, creating complicated geometric structures. There may be another way to approach the problem. Think in terms of a journey, not a jump Instead of asking for one complicated function that turns a dog into a cat, imagine breaking the transformation into a sequence of smaller steps. For example: Start with the o...