02 — Learning

Deep Learning Topics

A curated path exploring neural networks, CNNs, RNNs, and Transformers. Each topic includes core concepts, sample implementation, and a link to the full notebook.

Backpropagation

Understanding the core algorithm behind training neural networks by calculating gradients.

GradientsChain RuleOptimization

    
python
 def backward_pass(loss):
    loss.backward()
    optimizer.step()
 

View Notebook

Keras Functional and Subclassing API

Building complex neural network architectures using Keras Functional and Subclassing APIs.

Keras Functional APIModel SubclassingCustom Layers

    
python
 class CustomModel(tf.keras.Model):
    def __init__(self):
        super().__init__()
        self.dense = tf.keras.layers.Dense(32, activation="relu")
 

View Notebook

PyTorch Core Concepts

Fundamentals of PyTorch, including tensors, computational graphs, and basic operations.

TensorsAutogradComputational Graphs

    
python
 import torch
x = torch.tensor([1., 2.], requires_grad=True)
y = x.sum()
y.backward()
 

View Notebook

Dataset and DataLoader

Managing and batching data efficiently in PyTorch using Dataset and DataLoader utilities.

Custom DatasetDataLoaderBatchingShuffling

    
python
 from torch.utils.data import DataLoader
dataloader = DataLoader(dataset, batch_size=32, shuffle=True)
 

View Notebook

Dropout and Regularization

Preventing overfitting in deep neural networks using Dropout and other regularization techniques.

DropoutOverfittingRegularization

    
python
 import torch.nn as nn
model = nn.Sequential(
    nn.Linear(128, 64),
    nn.Dropout(p=0.5),
    nn.ReLU()
)
 

View Notebook

N-Gram Language Models (Part 1)

Introduction to statistical language modeling using N-grams.

Language ModelingN-GramsMarkov Assumption

    
python
 from nltk.util import ngrams
bigrams = list(ngrams(tokens, 2))
 

View Notebook

N-Gram Language Models (Part 2)

Advanced concepts in N-gram language models, including smoothing and perplexity.

SmoothingPerplexityProbability Distribution

    
python
 # Laplace Smoothing Example
prob = (count + 1) / (total_count + vocab_size)
 

View Notebook

Bengio MLP Language Model

Implementation of the classic Neural Probabilistic Language Model by Bengio et al.

Neural Language ModelsWord EmbeddingsMLP

    
python
 class BengioLM(nn.Module):
    def __init__(self, vocab_size, embed_dim):
        super().__init__()
        self.embeddings = nn.Embedding(vocab_size, embed_dim)
 

View Notebook

Multi-Layer Perceptron (MLP)

Building and training fully connected feed-forward neural networks.

MLPFeed-forwardActivation Functions

    
python
 mlp = nn.Sequential(
    nn.Linear(input_dim, hidden_dim),
    nn.ReLU(),
    nn.Linear(hidden_dim, output_dim)
)
 

View Notebook

CNN for Sentence Classification

Applying Convolutional Neural Networks to natural language processing and text classification.

Text CNN1D ConvolutionMax Pooling

    
python
 self.conv1d = nn.Conv1d(in_channels=embed_dim, out_channels=100, kernel_size=3)

View Notebook

Recurrent Neural Networks (RNN)

Working with sequential data using basic Recurrent Neural Networks.

RNNSequential DataHidden State

    
python
 self.rnn = nn.RNN(input_size, hidden_size, batch_first=True)
output, hidden = self.rnn(x)
 

View Notebook

Name Generation with RNN

Building a character-level RNN to generate names by learning from real name datasets.

Character-level RNNName GenerationSequence Modeling

    
python
 output, hidden = rnn(input_tensor, hidden)
topi = output.topk(1)[1][0][0]
 

View Notebook

Name Generation with RNN (Part 2)

Extending the character-level RNN name generator with improved sampling and temperature-based generation.

Character-level RNNTemperature SamplingText Generation

    
python
 def sample(category, start_letter='A'):
    hidden = rnn.initHidden()
    output, hidden = rnn(category_tensor, input[0], hidden)
 

View Notebook

Name Generation with RNN (Part 3)

Advanced name generation using RNNs with attention mechanisms and multi-language support.

RNNMulti-languageAttentionName Generation

    
python
 def generate(category, start_letters='ABC'):
    for start_letter in start_letters:
        print(sample(category, start_letter))
 

View Notebook

RNN Cell from Scratch

Implementing a custom RNN cell from scratch using PyTorch to understand the inner workings of recurrent units.

RNN CellCustom LayersBackpropagation Through Time

    
python
 class RNNCell(nn.Module):
    def forward(self, x, hidden):
        combined = torch.cat((x, hidden), 1)
        return torch.tanh(self.i2h(combined))
 

View Notebook

Self-Attention & Transformers

Understanding and implementing the self-attention mechanism that powers modern Transformer architectures.

Self-AttentionTransformersQuery-Key-ValueScaled Dot-Product

    
python
 scores = torch.matmul(Q, K.transpose(-2, -1)) / math.sqrt(d_k)
attn = F.softmax(scores, dim=-1)
output = torch.matmul(attn, V)
 

View Notebook

GPT-2 Language Model

Exploring and fine-tuning the GPT-2 autoregressive language model for text generation tasks.

GPT-2Autoregressive LMText GenerationCausal Language Model

    
python
 from transformers import GPT2LMHeadModel, GPT2Tokenizer
model = GPT2LMHeadModel.from_pretrained('gpt2')
outputs = model.generate(input_ids, max_length=100)
 

View Notebook