Posts

Poem

  Oh, rebellious root of crimson hue, What foolish dreams took hold of you? To spurn the dark, safe earthen bed, And hoist your leafy, reckless head! You traded soil and mineral grace To wither in this open space, Exposed to wind, to sun, to crow, Where honest veggies never go. Your taproot dangles in the air, A monument to your despair. No longer drinking from the loam, You left your blind and blissful home. So wither now in stubborn pride, With purple skin completely dried. You broke the subterranean rule, You stubborn, subterranean fool. I stand before you, proud and red, And do not mourn my earthen bed! You call me foolish, wild, and blind, For leaving safety far behind. But what is life within the clay? A sentence served day after day, To choke on mud and drink the rain, In silent, suffocating pain. The haughty rose, the sunflower high, Are granted freedom in the sky. While we, who build the deepest taste, Are hidden, buried, and erased. Behold my crown of violet bright! It wa...
Post a Comment
Read more

An Introduction to Concatenated Context Selected Linear Mappings (CCSLM)

Image
  Imagine a chain of linear mappings each selected by something. If the same mapping was selected each time that would be boring. Matrices such as A and B can be collapsed together to a single matrix M by matrix multiply, M=AB. And M is actually very different from its parents A and B. If you select A from a pool of 10 and likewise B you have 100 different possible M's from 20 matrices. They are like rabbits aren't they? If you have 10 of A and B and C you have 1000 different matrices from 30. This is exponentiation of expressiveness with depth similar to a neural network. The something that is doing the matrix selection at each layer you can call the context. And that can be liberal in meaning. That's nice but what if you now apply pressure from backpropagation on the matrix chain? For a data channel input x and the co-selected linear mapping (eg A₂B₇C₅...) backpropagation drifts each chosen linear mapping to be an expert for the slice of the training data it sees. Further...
Post a Comment
Read more

Linear Mappings Have a Memory

  Linear Mappings Have a Memory I've been looking at something surprisingly simple: what happens when a linear mapping is treated explicitly as a linear associative memory . One particularly interesting observation is that when a weighted sum is trained beyond capacity, new examples don't simply erase old ones. Instead, the weight vector is continually perturbed, gradually adding noise to previously stored associations. Recent examples are recalled better, but older examples can persist statistically for a surprisingly long time. Even more interestingly, below capacity, removing a training example may produce no change whatsoever in the weight vector—until weight decay is introduced. Then the mapping can move to a different, lower-norm solution while preserving the remaining associations. These elementary observations have some interesting consequences for initialization, weight decay, SGD dynamics, and ultimately CCSLM, where the local experts can themselves be viewed as fact...
Post a Comment
Read more

The Core Mathematics of CCSLM

Image
The core mathematics of concatenated context selected linear mappings https://archive.org/details/core-ccslm
Post a Comment
Read more

CCSLM skeptics document

Image
One for the skeptics but it is not argued as well as possible. It still retains some old fashioned views of neural networks. https://docs.google.com/document/d/e/2PACX-1vRLqFOuoGC3jS9HqJebDW5PHscznODG5XaCrsVN7MmoMRVkDl2r5Iu8sEjLpoN6vuH-PoAer0M1BT1w/pub
Post a Comment
Read more

Shadow People

Image
Researchers may have spent decades calculating increasingly sophisticated properties of the shadows because the experimental systems made the shadows extremely easy to observe.
Post a Comment
Read more
Image
Context-Only CCSLM sets the data input to a constant vector, such as (x=all 1's), so that all computational variation is controlled by context . y=Lₙ(Cₙ)... L₂(C₂)L₁(C₁)1 The finite set of context states visited during training becomes the primary organizing structure. Each context selects specialized linear mappings, making context effectively a computational address . This is particularly interesting for robotics and other stateful systems where useful extrinsic context—task, mode, gait phase, contact state, environment, time, etc.—may exist without any sensible analog data vector to supply. It also provides a clean experimental setting for studying context topology, state transitions, specialization, capacity, and computational structure largely independently of conventional input-driven learning.
Post a Comment
Read more