Using Extreme Learning Machines as a Data Sponge

 Extreme Learning Machines (ELMs) as the “sponge”

ELMs (and similar fixed-feature random projection networks) already have these sponge properties:

  • Fast training — weights in the hidden layer are fixed random projections; only the output weights are learned (often by linear regression).

  • Good interpolation in the training region, if the random features span the relevant subspace.

  • Low complexity — less tuning, smaller training cost.

If your “sponge” role really is just to store associative patterns and map them into a useful feature space, then an ELM (or random Fourier features, or kernel approximations) can do that far more cheaply than a full backprop-trained deep net.

The trade-off:

  • The feature mapping is fixed, so you lose the adaptive representation learning that deep nets excel at.

  • You may need a lot of random features to match the coverage of a trained net, which can inflate memory.

  • Without adaptation, robustness to domain shift can be worse unless the random mapping is very high-dimensional.

2. Engineering extrapolation into the system

If you swap a dense ReLU net for an ELM, the extrapolation still has to come from somewhere. Options include:

  • External retrieval/memory (RAG, vector databases) — doesn’t require backprop through the memory.

  • Symbolic or algorithmic modules — e.g., planners, calculators, simulators.

  • Learned controllers or routers that select modules — these can be trained without full backprop into the sponge, using reinforcement learning or policy gradients.

  • Feature-space composition — e.g., combining ELM features with structured representations (graphs, trees) before feeding into downstream algorithms.

3. When backprop through the whole system matters

You’re right — in many state-of-the-art systems, the engineered modules are differentiable and trained end-to-end:

  • Vision–language models: image encoder + text decoder + cross-attention are all trained together.

  • Tool-augmented LLMs with differentiable retrievers: the retriever is fine-tuned via gradients from the language loss.

  • Reinforcement learning with differentiable environments (rare, but exists).

End-to-end backprop allows the neural core to co-adapt its representation to the needs of the surrounding modules, which often improves efficiency and sample complexity.
If you replace the core with a fixed-feature ELM, you lose this co-adaptation — the surrounding modules have to work with whatever feature space the ELM gives them.

4. Hybrid compromise

One possible hybrid is:

  • Cheap, partially trainable sponge:

    • Randomized features for most of the projection

    • A small trainable bottleneck layer for adaptation

  • Non-differentiable extrapolation modules:

    • Retrieval, simulation, symbolic reasoning

  • Occasional fine-tuning of the bottleneck or output weights using a meta-learning loop, rather than continuous backprop through everything.

This keeps the sponge inexpensive, but lets the engineered shell do the heavy extrapolation lifting.

Comments

Post a Comment

Popular posts from this blog

Neon Bulb Oscillators

Image
Neon Tube Oscillators  Fig. 1 High voltage to headphones, don't do this in modern times. Very high impedance headphones are implied. A neon bulb is useful in many ways in an amateur station, but there are undoubtedly many amateurs who are not aware of the fact that the device can be made to oscillate at either audio or radio frequencies. Fig. 1 shows how a single bulb may be hooked up as an audio frequency oscillator. The L-C circuit is formed by the headphones and a condenser, and by varying the capacity the circuit can be made to oscillate at frequencies from a few cycles per second up to the highest audio frequencies.  The variable resistor in series with the bulb and battery controls the strength of oscillation, and also controls the frequency to some extent. A 1 nf variable condenser will be sufficient to cover most of the audio range.  Fig. 2 The circuit in Fig. 2 may be used when an RF oscillator is to be modulated at audio frequencies. The transformer may be an or...
Read more

23 Circuits you can Build in an Hour - Free Book

Image
Unleash your inner electronics wizard with 23 captivating circuits you can build in a jiffy! Tired of staring at daunting circuit diagrams that seem to go on forever? Craving hands-on experience that will ignite your passion for electronics? Look no further than this electrifying collection of 23 circuits you can build in an hour or less! From the delightfully simple to the mind-bendingly clever, these circuits are guaranteed to spark your curiosity and inspire further experimentation. Whether you're a seasoned pro or a budding hobbyist, you'll find something to tantalize your taste buds in this smorgasbord of electronic delights. Get the book now, it's free. The Lost SuperFoods (sponsored) Crash GPT course (sponsored) From Zero To Off-Grid Hero (sponsored)
Read more

Q Multiplier Circuits

Image
Q Multipliers Q multipliers were commonly used in amateur radio circuits in the past, especially in shortwave receivers. They were used to improve the selectivity and sensitivity of the receivers, making it possible to receive weaker signals and to distinguish between signals that were close together in frequency. Here are some specific examples of amateur radio circuits from the past that used Q multipliers: The Heathkit AR-3 receiver, which was a popular shortwave receiver in the 1950s, had a built-in Q multiplier. The Drake 2B receiver, which was another popular shortwave receiver in the 1950s and 1960s, had a separate Q multiplier unit that could be added to the receiver. The Collins KWS-1 transmitter, which was a popular amateur radio transmitter in the 1960s and 1970s, used a Q multiplier to generate the SSB signal. From Zero To Off-Grid Hero (sponsored) Operation By cancelling out the resistance and other losses in an LC (aka LCR) circuit its Q (voltage magnification at resonant...
Read more