Information flow in neural networks

Input-manifold information must propagate forward through the network and output-manifold information must propagate backward through the network during training and then forward again during recall.

Here’s how it’s typically framed in the literature:

1. Forward Information Flow (Input Manifold)

  • In the forward pass, the geometry and structure of the input manifold (the set of possible inputs, often low-dimensional in a high-dimensional space) needs to be preserved and transformed in useful ways through each layer.

  • If the network “breaks” the manifold — for example, by collapsing it onto a lower-dimensional set too early (rank collapse, saturation, dead ReLUs) — then information useful for later decision boundaries is lost irreversibly.

  • In deep learning theory, this is connected to:

    • Expressivity and manifold embedding theory — ensuring that layers preserve enough variation to distinguish different classes or outputs.

    • Signal propagation theory (e.g., Poole et al. 2016, Schoenholz et al. 2017) — analyzing whether signals blow up, vanish, or preserve variance as they move forward.

2. Backward Information Flow (Output Manifold)

  • During training, the output manifold is being constructed in the parameter space layer-by-layer through backpropagation.

  • Gradients carry information about how the current layer’s transformation should change so that the output manifold aligns with the target space.

  • If gradients vanish, explode, or become decorrelated from the forward signal, the network can’t “connect” the target geometry to earlier layers.

  • This is connected to:

    • Gradient flow analysis — ensuring the Jacobians don’t degenerate across layers.

    • Neural tangent kernel (NTK) perspectives — where both forward and backward flows are jointly considered for stable learning.

3. Bidirectional Preservation

  • Some recent works explicitly talk about mutual information flow in both directions:

    • In information bottleneck theory (Tishby et al.), too much compression too early kills forward signal; too little kills generalization.

    • In invertible networks (e.g., RevNets), forward and backward flows are explicitly symmetric, guaranteeing no information loss.

    • In critical initialization research, both forward activations and backward gradients are kept at “criticality” to ensure simultaneous signal propagation.

4. Why It Matters

  • If forward manifold information doesn’t make it to the output, you can’t make fine-grained predictions.

  • If backward manifold information (gradients) doesn’t reach the early layers, those layers can’t learn to shape the forward manifold properly.

  • The healthiest networks maintain two-way information highways until the very end of training — only near the output do you intentionally collapse the manifold to decision boundaries.


Comments

Post a Comment

Popular posts from this blog

Neon Bulb Oscillators

Image
Neon Tube Oscillators  Fig. 1 High voltage to headphones, don't do this in modern times. Very high impedance headphones are implied. A neon bulb is useful in many ways in an amateur station, but there are undoubtedly many amateurs who are not aware of the fact that the device can be made to oscillate at either audio or radio frequencies. Fig. 1 shows how a single bulb may be hooked up as an audio frequency oscillator. The L-C circuit is formed by the headphones and a condenser, and by varying the capacity the circuit can be made to oscillate at frequencies from a few cycles per second up to the highest audio frequencies.  The variable resistor in series with the bulb and battery controls the strength of oscillation, and also controls the frequency to some extent. A 1 nf variable condenser will be sufficient to cover most of the audio range.  Fig. 2 The circuit in Fig. 2 may be used when an RF oscillator is to be modulated at audio frequencies. The transformer may be an or...
Read more

23 Circuits you can Build in an Hour - Free Book

Image
Unleash your inner electronics wizard with 23 captivating circuits you can build in a jiffy! Tired of staring at daunting circuit diagrams that seem to go on forever? Craving hands-on experience that will ignite your passion for electronics? Look no further than this electrifying collection of 23 circuits you can build in an hour or less! From the delightfully simple to the mind-bendingly clever, these circuits are guaranteed to spark your curiosity and inspire further experimentation. Whether you're a seasoned pro or a budding hobbyist, you'll find something to tantalize your taste buds in this smorgasbord of electronic delights. Get the book now, it's free. The Lost SuperFoods (sponsored) Crash GPT course (sponsored) From Zero To Off-Grid Hero (sponsored)
Read more

Q Multiplier Circuits

Image
Q Multipliers Q multipliers were commonly used in amateur radio circuits in the past, especially in shortwave receivers. They were used to improve the selectivity and sensitivity of the receivers, making it possible to receive weaker signals and to distinguish between signals that were close together in frequency. Here are some specific examples of amateur radio circuits from the past that used Q multipliers: The Heathkit AR-3 receiver, which was a popular shortwave receiver in the 1950s, had a built-in Q multiplier. The Drake 2B receiver, which was another popular shortwave receiver in the 1950s and 1960s, had a separate Q multiplier unit that could be added to the receiver. The Collins KWS-1 transmitter, which was a popular amateur radio transmitter in the 1960s and 1970s, used a Q multiplier to generate the SSB signal. From Zero To Off-Grid Hero (sponsored) Operation By cancelling out the resistance and other losses in an LC (aka LCR) circuit its Q (voltage magnification at resonant...
Read more