Change of Basis Transforms - Machine Learning & Neural Networks




Change-of-basis transforms are interesting in machine learning and neural networks because they re-express data in a new coordinate system, and sometimes the new basis reveals patterns, structure, or computational shortcuts that were hidden in the original one.

Here’s why they matter and where the “interesting” comes from:

1. Geometry: different coordinates, same space

  • A change of basis is just a rotation/reflection (orthogonal case) or a more general invertible linear transformation.

  • It preserves all the information in the vector — nothing is lost — but the representation changes.

  • In ML, this means you can choose a basis that makes the structure of your problem easier to work with.

2. Sparsity and interpretability

  • Some bases make signals sparse — most coefficients are zero or small.

    • Example: images are not sparse in the pixel basis, but often sparse in a wavelet or DCT basis.

  • Sparsity is gold in ML: it helps with compression, denoising, and learning from fewer examples.

3. Separating important from irrelevant components

  • In the right basis, important signal directions can be isolated from noise directions.

  • PCA (Principal Component Analysis) is essentially a data-driven change of basis that aligns axes with the directions of largest variance.

  • This is useful for dimensionality reduction and avoiding overfitting.

4. Computational efficiency

  • Some fixed transforms (FFT, FWHT, DCT) can be applied in O(Nlog⁡N)O(N \log N) instead of O(N2)O(N^2) — a huge speed gain.

  • In neural networks, these can be structured weight matrices:

    • Example: Replace a dense layer with HDP (Hadamard × diagonal random sign × permutation) for fast random projections.

  • You get something like the same “expressiveness” but with far fewer multiplications.

5. Random features and kernels

  • Change-of-basis transforms with randomly chosen basis vectors can approximate kernels or preserve geometry (JL lemma).

  • Random Fourier features, Fastfood transform, and related techniques are structured changes of basis that make kernel methods scalable.

6. Information flow in networks

  • In a deep net, each layer can be seen as a change of basis + nonlinearity.

  • In the NTK/infinite-width perspective, the initial random weights define a fixed basis in function space, and learning adjusts coefficients.

  • Orthogonal basis changes can improve gradient flow and help avoid signal collapse.

7. Connections to physics and signal processing

  • In physics, choosing the right basis (Fourier, eigenmodes) can turn hard problems into separable ones.

  • In ML, the same logic applies: if your task has a symmetry or repeating structure, picking the right basis makes it easier for the network to learn.

Core takeaway:
Change-of-basis transforms in ML are powerful because they can make patterns more visible, computations faster, features sparser, and training easier — all without losing information. In many cases, the “magic” is choosing a basis that matches the problem’s structure, whether that’s learned (PCA, learned embeddings) or fixed (Fourier, Hadamard, wavelets).

Comments

Post a Comment

Popular posts from this blog

Neon Bulb Oscillators

Image
Neon Tube Oscillators  Fig. 1 High voltage to headphones, don't do this in modern times. Very high impedance headphones are implied. A neon bulb is useful in many ways in an amateur station, but there are undoubtedly many amateurs who are not aware of the fact that the device can be made to oscillate at either audio or radio frequencies. Fig. 1 shows how a single bulb may be hooked up as an audio frequency oscillator. The L-C circuit is formed by the headphones and a condenser, and by varying the capacity the circuit can be made to oscillate at frequencies from a few cycles per second up to the highest audio frequencies.  The variable resistor in series with the bulb and battery controls the strength of oscillation, and also controls the frequency to some extent. A 1 nf variable condenser will be sufficient to cover most of the audio range.  Fig. 2 The circuit in Fig. 2 may be used when an RF oscillator is to be modulated at audio frequencies. The transformer may be an or...
Read more

23 Circuits you can Build in an Hour - Free Book

Image
Unleash your inner electronics wizard with 23 captivating circuits you can build in a jiffy! Tired of staring at daunting circuit diagrams that seem to go on forever? Craving hands-on experience that will ignite your passion for electronics? Look no further than this electrifying collection of 23 circuits you can build in an hour or less! From the delightfully simple to the mind-bendingly clever, these circuits are guaranteed to spark your curiosity and inspire further experimentation. Whether you're a seasoned pro or a budding hobbyist, you'll find something to tantalize your taste buds in this smorgasbord of electronic delights. Get the book now, it's free. The Lost SuperFoods (sponsored) Crash GPT course (sponsored) From Zero To Off-Grid Hero (sponsored)
Read more

Q Multiplier Circuits

Image
Q Multipliers Q multipliers were commonly used in amateur radio circuits in the past, especially in shortwave receivers. They were used to improve the selectivity and sensitivity of the receivers, making it possible to receive weaker signals and to distinguish between signals that were close together in frequency. Here are some specific examples of amateur radio circuits from the past that used Q multipliers: The Heathkit AR-3 receiver, which was a popular shortwave receiver in the 1950s, had a built-in Q multiplier. The Drake 2B receiver, which was another popular shortwave receiver in the 1950s and 1960s, had a separate Q multiplier unit that could be added to the receiver. The Collins KWS-1 transmitter, which was a popular amateur radio transmitter in the 1960s and 1970s, used a Q multiplier to generate the SSB signal. From Zero To Off-Grid Hero (sponsored) Operation By cancelling out the resistance and other losses in an LC (aka LCR) circuit its Q (voltage magnification at resonant...
Read more