The Convex Hull and the Manifold - neural network training and test data

 

The orange polygon is the convex hull of the training data, the blue curve is the natural image manifold, the red dots are training points, and the green dot is a test point that’s outside the convex hull but still on the manifold. 

1. Convex Hull of Training Data

  • Definition (geometry view):
    The convex hull of a set of points is the smallest convex set containing them. Imagine stretching a rubber band around your scattered data points in feature space — the shape you get is the convex hull.

  • In the context of image data:
    If each image is a point in a very high-dimensional space (e.g., a RGB image = 3,072-D vector), the convex hull is a gigantic, convex polytope containing all your training examples.
    Importantly:

    • Many points inside the convex hull do not correspond to valid, natural images.

    • Linear combinations of training images (as in convex combinations) often produce weird blends, not real-world photos.

  • Neural network connection:
    The convex hull is relevant for linear models or models that behave nearly linearly in some regions: if the test point lies outside the convex hull of the training data, you’re extrapolating, which is risky.
    But most deep nets learn nonlinear decision boundaries that can handle points outside the convex hull if they lie on the same low-dimensional manifold.

2. Low-Dimensional Manifold for Natural Images

  • Definition (manifold hypothesis):
    Natural images, despite being high-dimensional vectors, occupy a thin, curved subset of the huge ambient space.
    This manifold has much lower intrinsic dimensionality — perhaps dozens or hundreds of degrees of freedom, instead of thousands.

  • Why this happens:
    Physical constraints (lighting, shapes, textures, viewpoint, etc.) and semantics greatly limit the possible configurations of pixels.
    Random points in pixel space are almost always noise — far off the natural image manifold.

  • Training and test data location:
    Both lie on or near the same manifold.
    Good generalization happens because the test set samples the same manifold as the training set, even if it lies outside the convex hull of the training samples.

3. Relationship Between Convex Hull & Manifold

  • Key idea:
    The convex hull is a crude, convex wrapper around your training points. The low-dimensional manifold is a much thinner, curved structure winding through space.
    The manifold typically pokes outside the convex hull in some places, and is mostly empty space inside the convex hull except along its surface.

  • Reasoning intuition in high dimensions:

    • In very high dimensions, the convex hull of many points still occupies an infinitesimally small fraction of the space.

    • The manifold is even smaller — it’s like a filament or sheet inside the hull.

    • The hull contains many “interpolations” between real images that aren’t realistic at all — they’re just valid convex combinations, not valid points on the manifold.

    • Neural networks don’t simply interpolate in the convex hull; they learn to follow the manifold structure.

Comments

Post a Comment

Popular posts from this blog

Neon Bulb Oscillators

Image
Neon Tube Oscillators  Fig. 1 High voltage to headphones, don't do this in modern times. Very high impedance headphones are implied. A neon bulb is useful in many ways in an amateur station, but there are undoubtedly many amateurs who are not aware of the fact that the device can be made to oscillate at either audio or radio frequencies. Fig. 1 shows how a single bulb may be hooked up as an audio frequency oscillator. The L-C circuit is formed by the headphones and a condenser, and by varying the capacity the circuit can be made to oscillate at frequencies from a few cycles per second up to the highest audio frequencies.  The variable resistor in series with the bulb and battery controls the strength of oscillation, and also controls the frequency to some extent. A 1 nf variable condenser will be sufficient to cover most of the audio range.  Fig. 2 The circuit in Fig. 2 may be used when an RF oscillator is to be modulated at audio frequencies. The transformer may be an or...
Read more

23 Circuits you can Build in an Hour - Free Book

Image
Unleash your inner electronics wizard with 23 captivating circuits you can build in a jiffy! Tired of staring at daunting circuit diagrams that seem to go on forever? Craving hands-on experience that will ignite your passion for electronics? Look no further than this electrifying collection of 23 circuits you can build in an hour or less! From the delightfully simple to the mind-bendingly clever, these circuits are guaranteed to spark your curiosity and inspire further experimentation. Whether you're a seasoned pro or a budding hobbyist, you'll find something to tantalize your taste buds in this smorgasbord of electronic delights. Get the book now, it's free. The Lost SuperFoods (sponsored) Crash GPT course (sponsored) From Zero To Off-Grid Hero (sponsored)
Read more

Q Multiplier Circuits

Image
Q Multipliers Q multipliers were commonly used in amateur radio circuits in the past, especially in shortwave receivers. They were used to improve the selectivity and sensitivity of the receivers, making it possible to receive weaker signals and to distinguish between signals that were close together in frequency. Here are some specific examples of amateur radio circuits from the past that used Q multipliers: The Heathkit AR-3 receiver, which was a popular shortwave receiver in the 1950s, had a built-in Q multiplier. The Drake 2B receiver, which was another popular shortwave receiver in the 1950s and 1960s, had a separate Q multiplier unit that could be added to the receiver. The Collins KWS-1 transmitter, which was a popular amateur radio transmitter in the 1960s and 1970s, used a Q multiplier to generate the SSB signal. From Zero To Off-Grid Hero (sponsored) Operation By cancelling out the resistance and other losses in an LC (aka LCR) circuit its Q (voltage magnification at resonant...
Read more