Double Descent and Extreme Learning Machines

Double Descent and Extreme Learning Machines


An Extreme Learning Machine with a fixed random layer followed by a linear output layer.

What is "Double Descent" – Explained Simply

Double descent curve for the linear layer

Recent research shows that something unexpected happens when you keep increasing model size past the point where it exactly fits the training data—this is called the interpolation threshold. Beyond this point:

  • Test error starts to drop again.

  • The model generalizes better, even though it's massively overparameterized.

This gives us a second "dip" in the test error curve—hence the name "double descent."

📊 How This Relates to Extreme Learning Machines (ELMs)

Let’s now bring in Extreme Learning Machines, which are a type of neural network with:

  • One hidden layer,

  • Randomly chosen input-to-hidden weights (they are not trained),

  • Only the output layer (linear layer) is learned using something like the Moore-Penrose inverse or ridge regression.

Think of the ELM as a linear model applied to random features.

🧱 Number of Parameters and Double Descent in ELMs

Here’s how double descent happens in ELMs:

  1. Few hidden units (few parameters):

    • The model is too simple → can’t capture patterns well → high test error.

  2. Hidden units = number of training samples:

    • The model becomes just powerful enough to fit the training data exactly → this is the interpolation point.

    • But it also fits noise → test error increases (the first rise).

  3. Many more hidden units (lots of parameters):

    • Surprisingly, test error drops again.

    • Why? Because the solution chosen (usually the minimum norm or regularized one) tends to be smoother and generalizes better, even though the model is massively overparameterized.

So the error curve goes:

High → Low → High → Low
as you increase the number of hidden units in the ELM.

🔑 Key Idea

Even though ELMs are simple and don’t train their hidden layers, they still show double descent because:

  • The output layer is a linear model, applied to a growing number of features (random nonlinear transforms).

  • When the number of hidden units increases, you effectively have more input features for that final linear layer.

  • This behaves just like a linear regression model with increasing dimensionality—which is known to show double descent.

📘 In One Sentence

Double descent happens in ELMs because increasing the number of hidden units makes the model go from underfitting (moderate input noise sensitivity)→ perfect fitting (high input noise sensitivity) → then better generalization again (low input noise sensitivity) , just like in modern deep learning, but through a linear model on random features.

📚 Want to Read More?

You can check out:

  • "Reconciling modern machine learning and the bias-variance trade-off" – Belkin et al. (2019)

  • "Double Descent in Random Feature Models and ELMs" – Various 2020–2023 ML workshop papers

  • ELM theory from G.-B. Huang et al., often discusses generalization in overparameterized settings.


Further information:

Comments

Post a Comment

Popular posts from this blog

Neon Bulb Oscillators

Image
Neon Tube Oscillators  Fig. 1 High voltage to headphones, don't do this in modern times. Very high impedance headphones are implied. A neon bulb is useful in many ways in an amateur station, but there are undoubtedly many amateurs who are not aware of the fact that the device can be made to oscillate at either audio or radio frequencies. Fig. 1 shows how a single bulb may be hooked up as an audio frequency oscillator. The L-C circuit is formed by the headphones and a condenser, and by varying the capacity the circuit can be made to oscillate at frequencies from a few cycles per second up to the highest audio frequencies.  The variable resistor in series with the bulb and battery controls the strength of oscillation, and also controls the frequency to some extent. A 1 nf variable condenser will be sufficient to cover most of the audio range.  Fig. 2 The circuit in Fig. 2 may be used when an RF oscillator is to be modulated at audio frequencies. The transformer may be an or...
Read more

23 Circuits you can Build in an Hour - Free Book

Image
Unleash your inner electronics wizard with 23 captivating circuits you can build in a jiffy! Tired of staring at daunting circuit diagrams that seem to go on forever? Craving hands-on experience that will ignite your passion for electronics? Look no further than this electrifying collection of 23 circuits you can build in an hour or less! From the delightfully simple to the mind-bendingly clever, these circuits are guaranteed to spark your curiosity and inspire further experimentation. Whether you're a seasoned pro or a budding hobbyist, you'll find something to tantalize your taste buds in this smorgasbord of electronic delights. Get the book now, it's free. The Lost SuperFoods (sponsored) Crash GPT course (sponsored) From Zero To Off-Grid Hero (sponsored)
Read more

Q Multiplier Circuits

Image
Q Multipliers Q multipliers were commonly used in amateur radio circuits in the past, especially in shortwave receivers. They were used to improve the selectivity and sensitivity of the receivers, making it possible to receive weaker signals and to distinguish between signals that were close together in frequency. Here are some specific examples of amateur radio circuits from the past that used Q multipliers: The Heathkit AR-3 receiver, which was a popular shortwave receiver in the 1950s, had a built-in Q multiplier. The Drake 2B receiver, which was another popular shortwave receiver in the 1950s and 1960s, had a separate Q multiplier unit that could be added to the receiver. The Collins KWS-1 transmitter, which was a popular amateur radio transmitter in the 1960s and 1970s, used a Q multiplier to generate the SSB signal. From Zero To Off-Grid Hero (sponsored) Operation By cancelling out the resistance and other losses in an LC (aka LCR) circuit its Q (voltage magnification at resonant...
Read more