Science Limelight

The ReLU (Rectified Linear Unit) activation function is one of the most commonly used activation functions in neural networks, especially in deep learning. Definition The ReLU function is defined as: ReLU ( x ) = max ( 0 , x ) This means: If the input x is  x> 0 x > 0 , the output is x x . If the input x is  x≤ 0 x \leq 0 , the output is 0 0 . Graphically It looks like a straight line with a slope of 1 for positive inputs and flat (zero) for negative inputs. Switching Viewpoint ReLU can also be understood from an alternative perspective. Consider that an electrical switch behaves linearly when "on" (e.g., 1 V in gives 1 V out, 2 V in gives 2 V out) and outputs zero when "off."  From this viewpoint, ReLU acts like a switch that is "on" when x≥0 x \geq 0  and "off" otherwise. The switching decision is (x≥0)? More generally (outside of ReLU) other switching decisions are possible. This switching interpretation can help demystify...

Why Deep Neural Networks Likely Operate as Hierarchical Associative Memory Training Saturation Indicates Memorization, Not Computation Once a deep neural network is trained using backpropagation, attempts to improve performance with alternative training methods generally fail. This plateauing suggests that the network has reached a limit in memorization capacity rather than computational sophistication. In essence, DNNs seem to function like high-dimensional memory systems that map inputs to outputs without performing symbolic or algorithmic computation internally. The lack of further improvement implies that the network is storing associations, not learning new rules or procedures. Absence of Explicit Algorithms Within the Network Despite extensive analysis of trained neural networks, researchers have never discovered internal representations of explicit algorithms (e.g., sorting, arithmetic procedures, logical inference chains). Instead, what emerges are hierarchies of featur...

The CReLU Activation Function CReLU , or   Concatenated Rectified Linear Units , is a type of activation function which allows both negative and positive inputs through by splitting the response into 2 parts. h CReLU(x)= [ ReLU ( x ) , ReLU ( − x ) ] ReLU is the standard Rectified Linear Unit and x is the scalar input value. ReLU activation function Using the CReLU doubles the size of the input to the next layer, increasing the number of parameters.  However  CReLU can improve accuracy on image recognition tasks when used for the lower convolutional layers by avoiding filter pair formation that happens as the network trains around ReLU one-sided behavior. In terms of information flow CReLU has the potential to allow complete information flow through. ReLU blocks information completely about 50% of the time, and is free flowing the other 50%. The Threshold activation function allows only 1 bit of information through, roughly speaking. The impact of ReLU information blockin...

Extreme Learning Machines Are a form of neural network that offer speed, efficiency and low computational complexity through simplicity and directness. Composition of an Extreme Learning Machine (ELM):      Input layer : Takes in the features of the data. Random hidden layer : Processes the input using fixed, randomly assigned weights and activation functions.  Output layer : A linear layer that computes the final output by solving a linear system using the activations from the hidden layer.   Key Features of ELMs: Random Initialization of Hidden Layer Weights: The weights connecting the input to the hidden layer are randomly assigned and remain fixed. Simple Training Process: Only the output layer weights are learned - by solving a simple linear system, usually by means of least squares . Universality: ELMs have been shown to approximate any continuous function, which makes them a universal approximator. High Speed and Efficiency: ELMs are capa...

Brief Introduction to Extreme Learning Machines (ELM) Extreme Learning Machines (ELMs) are a type of feedforward neural network known for fast training speed and good generalization performance. The contain 2 active layers. A fixed random layer with activation functions and an entirely linear read-out layer.  Here's a point-wise introduction: Type: Single-hidden layer feedforward neural networks (SLFNs). Key Idea: Input weights and biases in the hidden layer are randomly assigned and not updated during training. Training: The read-out layer weights are analytically determined using least squares (no iterative backpropagation).  Single shot least square. Advantage: Extremely fast training compared to traditional neural networks. Application Areas: Regression, classification, clustering, and feature learning tasks. Limitation: Performance depends on hidden layer size and random initialization. Current State of the Art (as of 2025) ELM Variants: ...