This note provides an interactive, visual simulation of a Multi-Layer Perceptron (MLP) for pattern recognition using a 9-3-1 architecture (9 inputs for 3×3 pixel images, 3 hidden neurons with ReLU activation, 1 output with sigmoid for binary classification). This represents the perfect "sweet spot" between XOR (9 weights) and full image recognition (thousands of weights) - small enough to visualize every connection, yet meaningful enough to demonstrate real pattern recognition concepts.

The task is to detect whether a 3×3 pixel pattern contains a vertical line or a horizontal line. The simulation includes a clickable 3×3 grid where you can toggle pixels on/off to create patterns. The network processes the pattern and outputs a probability that it's a horizontal line. The network diagram shows every single connection with line thickness and color (blue=positive, red=negative) representing weight strength, and hidden neurons "light up" based on their activation levels.

The training uses backpropagation with momentum on a dataset of vertical and horizontal line patterns (with noise). As the network trains, you can observe how hidden neurons learn to become feature detectors - for example, one neuron might learn to detect the top row (becoming a "horizontal line detector"), while another might learn to detect the middle column (becoming a "vertical line detector"). This demonstrates the core concept of how Convolutional Neural Networks (CNNs) work: neurons learn to detect specific features rather than processing the entire image at once.

NOTE : This simulation demonstrates feature learning in a "glass box" architecture where every weight is visible. The 9-3-1 architecture with ~28 total parameters is small enough to understand completely, yet complex enough to show how networks decompose problems into features. Refer to this note for theoretical details.

 

Usage Example

Follow these steps to explore how the network learns to detect vertical vs horizontal lines:

1. Initial State: When you first load the simulation, the network has random weights. The input grid is empty (all pixels off). Click on pixels in the 3x3 grid to create a pattern - try drawing a vertical line (all pixels in one column) or a horizontal line (all pixels in one row).
2. Test Before Training: Click pixels to create a pattern and observe the prediction. With random weights, the network will likely make incorrect predictions. The network diagram shows all connections - notice how the line thickness and colors represent weight strength and sign.
3. Start Training: Click the "Train" button. Watch as the input grid cycles through training patterns (vertical and horizontal lines). You'll see:
   • The input grid updating to show each training example
   • Hidden neurons lighting up based on their activation
   • The network diagram updating in real-time
   • Accuracy and loss improving over epochs
4. Adjust Training Delay: Use the "Training Delay" slider to control how fast training progresses. Lower values (0.1x-0.5x) make training faster but harder to observe. Higher values (2x-10x) slow down training so you can carefully watch each pattern being processed. You can adjust this even while training is running.
5. Observe Feature Learning: After a few epochs, stop training and examine the network diagram. You should notice:
   • One or more hidden neurons with thick blue connections to pixels in a specific row (horizontal line detector)
   • Other neurons with connections to pixels in a specific column (vertical line detector)
   • The output neuron combining these features to make the final decision
6. Test After Training: Clear the input grid and draw your own patterns. Try drawing:
   • A vertical line (all pixels in column 0, 1, or 2) - should predict "Vertical"
   • A horizontal line (all pixels in row 0, 1, or 2) - should predict "Horizontal"
   • Mixed patterns - see how the network handles ambiguous cases
7. Experiment with Parameters: Try adjusting the learning rate and momentum during training. Higher learning rates learn faster but may overshoot. Momentum helps the network escape flat regions. Observe how these parameters affect convergence speed and stability.
8. Reset and Retry: If training doesn't converge well, click "Reset" to reinitialize weights and try again. Different random initializations can lead to different learned features, demonstrating the importance of weight initialization in neural networks.

Tip: The key insight to look for is how hidden neurons become specialized feature detectors. After training, you should be able to identify which hidden neuron is looking for horizontal patterns and which is looking for vertical patterns by examining the connection weights in the network diagram.

Parameters

Followings are short descriptions on each parameters

• Input Grid (3×3): A clickable 3×3 grid where you can toggle pixels on/off by clicking. Green pixels are "on" (value 1), gray pixels are "off" (value 0). Click any pixel to toggle it. Create vertical lines (all pixels in one column) or horizontal lines (all pixels in one row) to test the network.
• Network Architecture: The network has 9 inputs (one for each pixel in the 3×3 grid), 3 hidden neurons with ReLU activation, and 1 output neuron with sigmoid activation. Total parameters: (9×3 + 3) + (3×1 + 1) = 28 weights and biases. This is small enough to visualize every connection clearly.
• Network Visualization: The center panel shows the complete network structure with all connections. Line thickness represents weight magnitude, color represents sign (blue=positive, red=negative). Input nodes turn green when activated. Hidden neurons "light up" based on their activation level (darker blue = higher activation). The output node shows the probability of a horizontal line.
• Learning Rate: Controls the step size for weight updates during backpropagation. Higher values learn faster but may overshoot or oscillate. Lower values are more stable but slower. Default is 0.1. Can be adjusted during training.
• Momentum: Controls the momentum factor (0 to 0.99) used in momentum-based gradient descent. Higher values (closer to 0.99) maintain more velocity, helping the network escape local minima and flat regions. Default is 0.9.
• Training Data: The network is trained on a dataset of vertical and horizontal line patterns. Each pattern type has 3 clean examples (one for each column/row) plus 10 noisy variations (with 1 random pixel flipped). Total: 6 clean + 60 noisy = 66 training samples.

Buttons and Controls

Followings are short descriptions on each control

• Clear: Clears the input grid, turning all pixels off.
• Predict: Processes the current pattern through the network and displays the prediction (Vertical or Horizontal line) along with confidence. The probability gauge shows the probability of a horizontal line (0% = vertical, 100% = horizontal). The network diagram updates in real-time to show which neurons are activated.
• Train: Starts training the network on the dataset. Training uses batch gradient descent with momentum. The network processes batches of 8 samples, accumulates gradients, and updates weights. Training continues until you click "Stop". The epoch counter, accuracy, and loss are displayed in real-time. Watch the network diagram to see how weights change and how neurons learn to detect features.
• Stop: Stops the training process. The network retains the weights learned so far.
• Reset Weights: Reinitializes all network weights and biases with random values (He initialization for ReLU). Also resets momentum velocities and the epoch counter. Useful for starting fresh training.

Key Concepts and Implementation

This simulation demonstrates feature learning in a "glass box" architecture where every connection is visible. Here are the key concepts:

• Architecture: 9-3-1: The network has 9 inputs (3×3 pixels), 3 hidden neurons with ReLU activation, and 1 output with sigmoid. Total: 28 parameters - small enough to visualize every single weight, yet complex enough to learn meaningful features. This is the perfect "sweet spot" for understanding how neural networks work.
• Feature Detection: As the network trains, hidden neurons learn to become feature detectors. For example, one neuron might develop strong positive weights to the top row pixels, making it a "top row detector" that fires when it sees horizontal lines. Another neuron might learn to detect the middle column, becoming a "vertical line detector." This is exactly how Convolutional Neural Networks (CNNs) work - they learn hierarchical features automatically.
• Weight Visualization: Every connection is drawn in the network diagram. Line thickness represents weight magnitude, color represents sign (blue=positive, red=negative). As training progresses, you can literally see which pixels each hidden neuron is "looking at" by observing which connections become thick and blue. This makes the learning process transparent and understandable.
• Neuron Activation: Hidden neurons "light up" based on their activation level. When you draw a horizontal line, you'll see neurons connected to horizontal pixels activate. When you draw a vertical line, different neurons activate. This visual feedback shows how the network decomposes the problem into features.
• ReLU Activation: The hidden layer uses ReLU (Rectified Linear Unit) activation, which is standard for image recognition. ReLU helps with gradient flow and allows neurons to learn sparse, specialized features. Weights are initialized using He initialization (sqrt(2/fan_in)) which is optimal for ReLU.
• Sigmoid Output: The output layer uses sigmoid activation to produce a probability (0 to 1) that the pattern is a horizontal line. Values close to 0 indicate vertical lines, values close to 1 indicate horizontal lines.
• Binary Cross-Entropy Loss: The network is trained using binary cross-entropy loss, which measures how well the predicted probability matches the true label (0 for vertical, 1 for horizontal).
• What to Look For: After training, observe the network diagram. You should see one or more hidden neurons with thick blue lines connecting to pixels in a specific row (horizontal detector) or column (vertical detector). This is the moment "math" becomes "machine vision" - the network has learned to decompose the image into features rather than processing it as a whole.