CCSLM via the ReLU entry point - Flyer

 

CCSLM via the ReLU entry point



Break a ReLU neural network into a main data channel and a context channel.

ReLU(Wx) = DWx, where D is a diagonal decision matrix containing 0/1 entries produced by a context function applied to the layer input—in this case, the ReLU decisions.

The main data channel therefore only ever sees linear mappings: the W matrices and the D selection matrices. This is a remarkable restriction on apparent expressiveness, but it has potentially very positive consequences.

The context channel is doing something quite different. It does not transform the data; it selects which linear mapping acts on the data. A sequence of binary decisions across layers can represent an exponentially large number of distinct symbolic contexts or computational paths, even though each individual decision is simple.

The selected mapping can be made explicit by fusing D with an adjacent weight matrix. L = WD is the more useful representation for human comprehension: the context has selected columns of W, producing the linear mapping actually used by the data.

Also, D = DD: the selection operation is idempotent.

From this viewpoint, a ReLU neural network can be described as a:

Concatenated Context-Selected Linear Mapping (CCSLM)

This reframing separates two potentially independent engineering problems:

  • Context engineering — how computational regions are selected.

  • Linear mapping engineering — what each selected region learns to do.

That separation suggests many design opportunities beyond ReLU: alternative context sources, larger discrete context alphabets, external sensors, temporal state, hashing, automata, learned routing, or combinations of these. The context mechanism can potentially provide enormous symbolic complexity while leaving the data transformation itself strictly linear.

Comments

Post a Comment

Popular posts from this blog

Neon Bulb Oscillators

Image
Neon Tube Oscillators  Fig. 1 High voltage to headphones, don't do this in modern times. Very high impedance headphones are implied. A neon bulb is useful in many ways in an amateur station, but there are undoubtedly many amateurs who are not aware of the fact that the device can be made to oscillate at either audio or radio frequencies. Fig. 1 shows how a single bulb may be hooked up as an audio frequency oscillator. The L-C circuit is formed by the headphones and a condenser, and by varying the capacity the circuit can be made to oscillate at frequencies from a few cycles per second up to the highest audio frequencies.  The variable resistor in series with the bulb and battery controls the strength of oscillation, and also controls the frequency to some extent. A 1 nf variable condenser will be sufficient to cover most of the audio range.  Fig. 2 The circuit in Fig. 2 may be used when an RF oscillator is to be modulated at audio frequencies. The transformer may be an or...
Read more

A practical way to view neural networks (whiteboard view)

Image
 A whiteboard view: You can take a decision matrix view of ReLU neural networks. Instead of ReLU(Wx) you create a diagonal matrix with binary 1 or 0 entries according to the ReLU decisions (x>=0?) Then conceptually a layer is DWx. Where D is doing row selection on W. And in fact D does column selection on the weight matrix in the next layer. That is very coarse parameter selection which is linear mapping selection.
Read more

23 Circuits you can Build in an Hour - Free Book

Image
Unleash your inner electronics wizard with 23 captivating circuits you can build in a jiffy! Tired of staring at daunting circuit diagrams that seem to go on forever? Craving hands-on experience that will ignite your passion for electronics? Look no further than this electrifying collection of 23 circuits you can build in an hour or less! From the delightfully simple to the mind-bendingly clever, these circuits are guaranteed to spark your curiosity and inspire further experimentation. Whether you're a seasoned pro or a budding hobbyist, you'll find something to tantalize your taste buds in this smorgasbord of electronic delights. Get the book now, it's free. The Lost SuperFoods (sponsored) Crash GPT course (sponsored) From Zero To Off-Grid Hero (sponsored)
Read more