Hash & Select Extreme Learning Machine

Unit of Hash & Select Extreme Learning Machine

What’s happening step by step:

  1. Input vector:
    Length n, where n= 2^k (e.g. 8, 16, 32).

  2. Random projection RR:

    • A linear random projection spreads out the input information across all dimensions.

    • This ensures that even small input changes can flip some signs after binarization.

  3. Binarization → Hashing:

    • Each coordinate of RxR \mathbf{x} is binarized (sign or threshold).

    • Then grouped into 8-bit chunks → 256 possible keys.

    • This is essentially a locality sensitive hash (LSH). Inputs close in RR-space are likely to share the same or nearby keys.

  4. Second random projection UU:

    • URx produces a different transformed version of the input.

    • These outputs are grouped into blocks of 8 coordinates.

  5. Keyed weight block selection:

    • Each 8-dimensional block chooses a block of 8 weights from a pool of 256 blocks.

    • The chosen block index is determined by one of the 8-bit hash keys from step 3.

    • This ties the hash identity of the input (discrete) with the continuous projection URUR.

  6. Weighted multiplication:

    • The 8 coordinates of the projection block are multiplied by the 8 chosen weights.

    • This creates a final nonlinear, input-dependent weighted feature.

Properties / What this buys you:

  • Information spreading:
    Random projection distributes input energy uniformly, so binarization provides a “fair” hash.

  • Locality-sensitive weight selection:
    Keys cause structurally different weight paths for different inputs. This introduces discrete branching capacity into the ELM.

  • Hybrid continuous-discrete feature map:
    The continuous projection URUR gets modulated by discrete keys from RR. This creates richer feature diversity compared to plain ELMs.

  • Capacity expansion:
    With 256 weight blocks per group, you’ve essentially introduced a combinatorial number of possible weight configurations, vastly increasing representational power with modest parameter counts.

  • Analogy:
    It’s a bit like combining:

    • Random Fourier features (spreading info),

    • Locality sensitive hashing (discrete routing),

    • Mixture-of-experts gating (keyed weight selection).

Ensemble Several Units 


You can use Sum-then-Project or Project-then-Sum again using random projections to ensemble several units of Hash & Select Extreme Learning Machines all acting on the same input, into a more powerful system. 

Hints:
1/ Fast Inward and Outward facing Random Projections:
Use inward facing random projections for the extreme learning machine units and outward facing random projections to ensemble. 

2/ Sum-then-Project Vs Project-then-Sum:

3/ You could increase the block size to 16 for example, giving the extreme learning machine vast memory capacity. That would result in 65536 blocks of 16 weights for each 16 coordinate channel. You could tame that down a bit by using (for example) only 12 bits of each 16 bit key.

4/ Example code using Java:



Comments

Post a Comment

Popular posts from this blog

Neon Bulb Oscillators

Image
Neon Tube Oscillators  Fig. 1 High voltage to headphones, don't do this in modern times. Very high impedance headphones are implied. A neon bulb is useful in many ways in an amateur station, but there are undoubtedly many amateurs who are not aware of the fact that the device can be made to oscillate at either audio or radio frequencies. Fig. 1 shows how a single bulb may be hooked up as an audio frequency oscillator. The L-C circuit is formed by the headphones and a condenser, and by varying the capacity the circuit can be made to oscillate at frequencies from a few cycles per second up to the highest audio frequencies.  The variable resistor in series with the bulb and battery controls the strength of oscillation, and also controls the frequency to some extent. A 1 nf variable condenser will be sufficient to cover most of the audio range.  Fig. 2 The circuit in Fig. 2 may be used when an RF oscillator is to be modulated at audio frequencies. The transformer may be an or...
Read more

23 Circuits you can Build in an Hour - Free Book

Image
Unleash your inner electronics wizard with 23 captivating circuits you can build in a jiffy! Tired of staring at daunting circuit diagrams that seem to go on forever? Craving hands-on experience that will ignite your passion for electronics? Look no further than this electrifying collection of 23 circuits you can build in an hour or less! From the delightfully simple to the mind-bendingly clever, these circuits are guaranteed to spark your curiosity and inspire further experimentation. Whether you're a seasoned pro or a budding hobbyist, you'll find something to tantalize your taste buds in this smorgasbord of electronic delights. Get the book now, it's free. The Lost SuperFoods (sponsored) Crash GPT course (sponsored) From Zero To Off-Grid Hero (sponsored)
Read more

Q Multiplier Circuits

Image
Q Multipliers Q multipliers were commonly used in amateur radio circuits in the past, especially in shortwave receivers. They were used to improve the selectivity and sensitivity of the receivers, making it possible to receive weaker signals and to distinguish between signals that were close together in frequency. Here are some specific examples of amateur radio circuits from the past that used Q multipliers: The Heathkit AR-3 receiver, which was a popular shortwave receiver in the 1950s, had a built-in Q multiplier. The Drake 2B receiver, which was another popular shortwave receiver in the 1950s and 1960s, had a separate Q multiplier unit that could be added to the receiver. The Collins KWS-1 transmitter, which was a popular amateur radio transmitter in the 1960s and 1970s, used a Q multiplier to generate the SSB signal. From Zero To Off-Grid Hero (sponsored) Operation By cancelling out the resistance and other losses in an LC (aka LCR) circuit its Q (voltage magnification at resonant...
Read more