Repetition Coding and Over-Parameterized Weighted Sums: A Signal Processing Perspective on Generalization and Noise Robustness

 

1. Introduction

Good afternoon. Today, I want to take a step back from neural networks and machine learning models to revisit a classic concept from digital signal processing: repetition coding, and explore how its principles resurface—surprisingly—in the behavior of over-parameterized weighted sums, particularly in neural networks and linear models. We'll also touch on how these ideas relate to the variance of weighted combinations of random variables, noise robustness, and generalization.

2. Repetition Coding: The Basics

Let’s start with repetition coding, one of the simplest forms of error control coding in digital communications.

  • The idea is simple: repeat each bit multiple times before transmission.

    • For example: 0 → 000, 1 → 111 (a 3-times repetition code).

  • On the receiving end, apply a majority vote to determine the original bit.

This redundancy combats transmission noise. If one or even two of the three bits get flipped due to noise, the majority vote still gives you the correct value. Of course, this comes at the cost of bandwidth or rate, but it increases robustness.

Key point:

Repetition coding spreads the information across multiple independent channels, so that noise in individual elements averages out, reducing the variance of the estimate of the original bit.

3. Weighted Sums: From Digital Codes to Learning Systems

Now, let’s draw the analogy to weighted sums, especially in models like:

  • Linear regression,

  • Perceptrons,

  • Neural network layers,

  • Extreme Learning Machines (ELMs),

  • Kernel methods with feature expansion.

In all these, we compute:

y=∑i=1Nwixiy = \sum_{i=1}^{N} w_i x_i

We can think of the input vector x\mathbf{x} as a signal, the weights wiw_i as learnable coefficients, and the output yy as a response or prediction.

Now let’s compare critically, under-, and over-parameterized regimes.

  • Under-parameterized: fewer features than data points.

  • Critically parameterized: the number of features equals the number of training points.

  • Over-parameterized: more features than training points; the model can interpolate training data.

4. Redundancy and Error Correction in Over-Parameterized Weighted Sums

Now here's the central idea:

Over-parameterization acts like repetition coding.

Let me explain.

In over-parameterized models—especially when input features are correlated, duplicated, or random projections of similar signals—multiple parts of the input vector support the same target output. Each feature (or neuron) might provide a noisy "vote", but together they combine to reinforce a clean signal.

This becomes clearer when you consider noise.



5. Comparing Regimes via Variance



Hence, variance at the output decreases with over-parameterization, just like how repetition coding reduces the effect of bit-level noise.

6. Repetition Coding Analogy in Over-Parameterization

Just as repetition coding duplicates bits to improve noise resilience:

  • Over-parameterized systems duplicate or spread influence across many correlated or redundant features.

  • Each feature contributes a small, potentially noisy vote to the final prediction.

  • But as in averaging or majority voting, the net effect cancels noise and reinforces the signal.

In fact, this is similar to random feature models or kernel ridge regression, where excess capacity enables distributed representations.

7. Takeaway: Distributed, Redundant Representations Improve Generalization

So the main insight is:

In over-parameterized systems, generalization is not harmed by the number of parameters per se, but is determined by the geometry of the weight vector, the distribution of signal vs noise, and how redundantly the signal is encoded across the input.

This understanding bridges signal processing, information theory, and modern machine learning.

It also underpins the surprising generalization abilities of deep networks and ELMs that operate in high-dimensional feature spaces yet manage to remain robust to noise.

8. Summary

  • Repetition coding spreads bits across multiple noisy channels to improve reliability.

  • Over-parameterized models spread signal across many features or neurons.

  • This can reduce the effective variance in the output via averaging—similar to ensemble methods.

  • The variance of the weighted output depends on the L2 norm of the weights, and in over-parameterized regimes, the norm can be minimized, resulting in lower output variance despite high capacity.

9. Closing Thought

Modern over-parameterized models might look complex, but at heart, they often exploit the same error correction and redundancy principles we’ve known for decades. By understanding these connections, we can better design and analyze systems that generalize well, even in high-dimensional settings.

Thank you.

Comments

Post a Comment

Popular posts from this blog

Neon Bulb Oscillators

Image
Neon Tube Oscillators  Fig. 1 High voltage to headphones, don't do this in modern times. Very high impedance headphones are implied. A neon bulb is useful in many ways in an amateur station, but there are undoubtedly many amateurs who are not aware of the fact that the device can be made to oscillate at either audio or radio frequencies. Fig. 1 shows how a single bulb may be hooked up as an audio frequency oscillator. The L-C circuit is formed by the headphones and a condenser, and by varying the capacity the circuit can be made to oscillate at frequencies from a few cycles per second up to the highest audio frequencies.  The variable resistor in series with the bulb and battery controls the strength of oscillation, and also controls the frequency to some extent. A 1 nf variable condenser will be sufficient to cover most of the audio range.  Fig. 2 The circuit in Fig. 2 may be used when an RF oscillator is to be modulated at audio frequencies. The transformer may be an or...
Read more

23 Circuits you can Build in an Hour - Free Book

Image
Unleash your inner electronics wizard with 23 captivating circuits you can build in a jiffy! Tired of staring at daunting circuit diagrams that seem to go on forever? Craving hands-on experience that will ignite your passion for electronics? Look no further than this electrifying collection of 23 circuits you can build in an hour or less! From the delightfully simple to the mind-bendingly clever, these circuits are guaranteed to spark your curiosity and inspire further experimentation. Whether you're a seasoned pro or a budding hobbyist, you'll find something to tantalize your taste buds in this smorgasbord of electronic delights. Get the book now, it's free. The Lost SuperFoods (sponsored) Crash GPT course (sponsored) From Zero To Off-Grid Hero (sponsored)
Read more

Q Multiplier Circuits

Image
Q Multipliers Q multipliers were commonly used in amateur radio circuits in the past, especially in shortwave receivers. They were used to improve the selectivity and sensitivity of the receivers, making it possible to receive weaker signals and to distinguish between signals that were close together in frequency. Here are some specific examples of amateur radio circuits from the past that used Q multipliers: The Heathkit AR-3 receiver, which was a popular shortwave receiver in the 1950s, had a built-in Q multiplier. The Drake 2B receiver, which was another popular shortwave receiver in the 1950s and 1960s, had a separate Q multiplier unit that could be added to the receiver. The Collins KWS-1 transmitter, which was a popular amateur radio transmitter in the 1960s and 1970s, used a Q multiplier to generate the SSB signal. From Zero To Off-Grid Hero (sponsored) Operation By cancelling out the resistance and other losses in an LC (aka LCR) circuit its Q (voltage magnification at resonant...
Read more