Matrix Transform Explorer with Nonlinearity

Neural networks need nonlinearities when stacking linear layers one after another.

A linear layer represents a linear transformation, and repeated linear transformations can be represented by just a single linear transformation, so adding additional layers is essentially useless. By introducing a nonlinearity in between the two layers, we add additional expressiveness, because the network can learn to do things that aren't possible with just linear transformations.

Common nonlinearities include sigmoid (

σ (x) = \frac{1}{1 + e^{- x}}

), ReLU (

ReLU (x) = \max (0, x)

), and tanh (

\tanh (x) = \frac{e^{x} - e^{- x}}{e^{x} + e^{- x}}

). So, if we have a two layer network with per-layer weights A and B, we might have something like this:

network_result = A \times ReLU (B \times input)

where × is matrix multiplication, and ReLU is applied element-wise

If you think about it, we're still just transforming one 2D input vector into another 2D vector. So let's try visualizing that in the same way we might try visualizing a matrix multiplication, by showing how it affects a grid of points in the plane.

About this Explorer

This visualization shows the pipeline used in neural networks: input data passes through a linear transformation (Matrix 1), then an activation function, then another linear transformation (Matrix 2). You can adjust each component to understand how they work together to transform data.

Matrix 1

M1[0,0] 1.0

M1[0,1] 0.0

M1[1,0] 0.0

M1[1,1] 1.0

Activation Function

Alpha (α) 0.20

Alpha (α) 1.0

Beta (β) 1.0

Scale 1.0

Leaky ReLU: f(x) = max(αx, x)
Allows small gradient for negative values.

Matrix 2

M2[0,0] 1.0

M2[0,1] 0.0

M2[1,0] 0.0

M2[1,1] 1.0

Available Activation Functions:

Leaky ReLU: $f (x) = {\begin{matrix} x & if x \geq 0 \\ α x & if x < 0 \end{matrix}$
ReLU: $f (x) = \max (0, x)$
Tanh: $f (x) = s \times \tanh (x)$
Sigmoid: $f (x) = s \times \frac{1}{1 + e^{- x}}$
ELU: $f (x) = {\begin{matrix} x & if x \geq 0 \\ α (e^{x} - 1) & if x < 0 \end{matrix}$
Swish: $f (x) = x \times σ (β x)$
GELU: $f (x) = x \times Φ (x)$
Softplus: $f (x) = \frac{1}{β} \ln (1 + e^{β x})$
Mish: $f (x) = x \times \tanh (\ln (1 + e^{x}))$
Linear: $f (x) = s x$

Pipeline: Points → Matrix1 → Activation Function → Matrix2 → Result

Reset Options: Use individual reset buttons for each section, or "Reset All" for complete identity transformation.

Matrix Transform Explorer with Nonlinearity

About this Explorer

Original

Transformed

Matrix 1

Activation Function

Matrix 2

Visualization Options

Available Activation Functions: