Deep Learning Defined

Deep learning is a way to learn functions by stacking layers of simple units (neurons) so that the whole network can approximate very complex input→output mappings. A neural network is just a composable function: output = layer_L(...layer_2(layer_1(input))).

Why “deep”? Because there are many layers (depth). Why “learning”? Because the network’s numeric knobs (its weights and biases) are tuned to minimize a loss—a number that measures how wrong the network is on your data.

Where It Fits

• Machine learning (ML): learn patterns from data.
• Representation learning: learn useful features automatically (instead of hand-crafting them).
• Deep learning (DL): representation learning with many layers of differentiable transformations.

DL shines when you have large datasets, high-dimensional inputs (images, audio, text), and the need for end-to-end learning.

From Perceptron to Neuron

A perceptron is a mathematical model of a biological neuron that takes numerical inputs, applies weights, adds a bias, and uses an activation function to produce a binary output, classifying data into two categories.

The original perceptron computed: y = step(w·x + b). Modern neurons do: z = w·x + b, then a = φ(z) where φ is an activation function (e.g., ReLU, sigmoid, tanh). Stacking many neurons gives you a layer; stacking layers gives you a network.

The Math You Really Need

Here's the mathematical terms at play:

• Weights (W) and biases (b): the parameters we learn.
• Activation function φ: adds non-linearity (e.g., ReLU(x) = max(0,x)).
• Loss: scalar measuring error, e.g., MSE for regression, cross-entropy for classification.
• Gradient: vector of partial derivatives that tells us how to tweak parameters to reduce loss.
• Gradient descent: update rule θ ← θ − η ∇θ L with learning rate η.

Forward, Loss, Backprop: The Loop

The Forward, Loss, Backprop loop is the core training process for a neural network, where a forward pass makes a prediction, a loss function calculates how wrong it is, and backpropagation computes gradients to update the model's weights, reducing error over many iterations to improve future predictions.

1. Forward: compute predictions from inputs via layers and activations.
2. Loss: compare predictions to targets.
3. Backward: compute gradients of loss w.r.t. each parameter (backpropagation).
4. Update: adjust parameters with gradient descent (or a fancier optimizer).

Multiclass Heads & Cross-Entropy

For K classes, we compute a vector of logits z ∈ ℝ^K, then apply softmax(z)_k = e^{z_k} / Σ_j e^{z_j}. Use cross-entropy loss: L = − Σ_k y_k log(softmax(z)_k) where y is a one-hot label.

Regularization & Generalization

• Overfitting: model learns noise; low training loss, high validation loss.
• Underfitting: model too simple; high training and validation loss.
• Regularization: techniques to improve generalization:

• L2 (weight decay): penalize large weights.
• Early stopping: stop when validation loss worsens.
• Dropout: randomly drop units during training (simulated in code by masking).
• Data augmentation: alter inputs (flips/crops/noise) to create variety.

When NOT to Use Deep Learning

• Tiny dataset with easily engineered features? Try simpler ML (like linear or tree-based models).
• Need perfect interpretability or strict guarantees? DL may be harder to justify.
• Low compute budget or latency constraints? A smaller model may be better.

Rule of thumb: start simple, scale up when the problem/data demands it.

Hardware and Complexity

• Training cost grows with data size, model size, and sequence/image resolution.
• Batch size: how many samples per gradient step. Larger batches use more memory.
• Epoch: one full pass over training data.
• Typical accelerators: GPUs/TPUs; but conceptually all you need is the math we wrote.

Ethics, Safety, and Bias

Neural nets learn what they see. If training data is biased, the model may be biased. Key ideas:

• Dataset curation and evaluation on diverse slices.
• Explainability tools (feature attributions, probes) to audit behavior.
• Safety: avoid harmful outputs; consider rate limits, human review, domain constraints.

Quick Debugging Playbook

• Sanity check: can the model overfit a tiny subset (e.g., 10 samples)?
• Loss not decreasing? Lower lr, check gradient signs and shapes.
• Exploding loss? Clip gradients, reduce lr, check for NaNs.
• Validation worse than training? Add regularization or more data.

You’ve seen what deep learning is, why it works, and you’ve implemented tiny nets from scratch. When you’re ready, port these to a proper framework—but now you’ll know exactly what the framework is doing under the hood.

One Pager Cheat Sheet

• Deep learning is the process of approximating complex input-output mappings by stacking layers of simple units (neurons) in a neural network, tuning the network's numeric knobs (weights and biases) to minimize a loss function through many layers (depth).
• Deep learning (DL) is a subset of machine learning (ML) that excels when working with large datasets, high-dimensional inputs, and the need for end-to-end learning due to its focus on representation learning with many layers of differentiable transformations.
• From Perceptron to Neuron: The mathematical model of a biological perceptron evolved into modern neurons that compute z = w·x + b, apply an activation function φ(z), and can be stacked to create layers and networks in deep learning.
• Deep learning stacks linear layers; without non-linear activations this equals one big linear map.
• In neural networks, the key terms include weights (W), biases (b), activation function φ, loss functions like MSE and cross-entropy, gradients, and gradient descent with a learning rate η.
• Tiny neuron implemented with ReLU activation trained using plain gradient descent to learn y ≈ 2*x + 1 on synthetic data using only standard library.
• The statement that the function is false is because the ReLU function is not differentiable at 0 due to the disagreement between the left derivative (0) and the right derivative (1) at that point.
• The Forward, Loss, Backprop loop is the core training process for a neural network, where a forward pass predicts, a loss function calculates errors, and backpropagation adjusts weights with gradient descent to improve future predictions.
• In the training loop, the forward pass computes predictions, the loss is calculated on those predictions, backpropagation is used to compute gradients, and finally parameters are updated, each step depending on the results of the previous one.
• Two-layer MLP implemented for binary classification on a toy dataset using standard library only.
• The softmax function converts raw logits to a probability distribution by exponentiating and normalizing values, making it a "soft" argmax that is differentiable and pairs well with cross-entropy loss for multi-class classification training.
• Multiclass heads involve computing a vector of logits for K classes, applying the softmax function to obtain probabilities, and calculating the cross-entropy loss L using the formula L = − Σ_k y_k log(softmax(z)_k) with y as a one-hot label.
• In multiclass classification, the typical approach involves a Linear layer for class logits, followed by a softmax function to generate a probability distribution, and cross-entropy loss to maximize likelihood and differentiate predicted probabilities from true values.
• Regularization techniques like L2 weight decay, early stopping, dropout, and data augmentation are used to prevent overfitting and improve the generalization of machine learning models.
• L2 Weight Decay is added as an L2 penalty to the loss function within the training loop.
• Self-attention in transformers allows for direct connections between positions, enabling parallel computation and modeling of long-range dependencies without the need for recurrent steps.
• When NOT to Use Deep Learning: For a tiny dataset with easily engineered features, perfect interpretability or strict guarantees, or low compute budget or latency constraints, consider simpler ML models (like linear or tree-based) and start simple, scaling up only when necessary.
• Build Your Own MLP for XOR using a 2-layer MLP with standard libraries only.
• Training cost increases with data size, model size, and sequence/image resolution, where batch size and epoch are important factors to consider when using accelerators such as GPUs/TPUs.
• Neural nets learn from data and may exhibit bias if the training data is biased, requiring dataset curation, evaluation on diverse slices, and explainability tools such as feature attributions and probes to audit behavior while also ensuring safety through measures like rate limits, human review, and domain constraints.
• The Quick Debugging Playbook includes steps like checking for overfitting on a tiny subset, adjusting lr and gradients if loss isn't decreasing, clipping gradients for exploding loss, and adding regularization for cases where validation performs worse than training.
• Overfitting occurs when a model fits training data too closely but fails to generalize, and combating this involves using an L2 penalty to reduce complexity by shrinking weights and early stopping to prevent memorization of noise, both directly reducing the training/validation gap.
• The correct term is epoch, referring to a single pass through the entire training dataset that allows the model to learn patterns, with multiple epochs used in training to refine parameters and combat overfitting.
• The composition of linear maps is itself a linear map, allowing multiple linear layers without activation functions to be collapsed into a single linear transformation.
• You have gained an understanding of deep learning, implemented small networks from scratch, and are prepared to transfer them to a framework where you will have a clear understanding of the underlying operations.