BAB 05

05 - Transformers & LLMs: Architecture, Pretraining, Inference

Estimasi Waktu: 55 menit Level: Intermediate → Advanced

---

The Transformer Revolution

Paper “Attention Is All You Need” (Vaswani et al., 2017) adalah momen “iPhone” untuk AI. Semua yang kita sebut “AI” sekarang — GPT, Claude, Gemini, Llama — adalah turunan dari arsitektur ini.

---

Transformer Architecture: High-Level

INPUT TEXT
    ↓
TOKENIZATION (text → numbers)
    ↓
EMBEDDING (numbers → vectors)
    ↓
TRANSFORMER BLOCK (× N layers)
    ├── Multi-Head Self-Attention
    ├── Feed-Forward Network
    ├── Layer Normalization
    └── Residual Connections
    ↓
OUTPUT HEAD (next token prediction)

---

Step 1: Tokenization

Text dipecah jadi tokens (subwords).

# Contoh tokenization
text = "Hello, world!"
tokens = ["Hello", ",", " world", "!"]
token_ids = [15496, 11, 995, 0]

# GPT-4 tokenizer: ~100K vocabulary
# "makanan" → ["makan", "an"] (2 subwords)
# "transformer" → ["transform", "er"] (2 subwords)

Kenapa subwords? Keseimbangan antara vocabulary size (efisiensi) dan coverage (bisa handle kata baru).

---

Step 2: Embeddings

Setiap token di-map ke vector (dense representation).

# Token ID → Embedding Vector
token_id = 15496
embedding = embedding_matrix[token_id]  # shape: [768] (GPT-2 small)
# atau [4096] untuk GPT-3, [8192] untuk Llama 3 70B

# Positional Encoding: model perlu tahu urutan
# "Anjing gigit orang" ≠ "Orang gigit anjing"
position_vector = positional_encoding(position=0)
final_embedding = token_embedding + position_vector

---

Step 3: Self-Attention (The Core Innovation)

Self-attention memungkinkan setiap token “melihat” dan “berkomunikasi” dengan semua token lain.

Query, Key, Value (QKV):

Untuk setiap token, hitung:
- Query (Q): "Apa yang aku cari?"
- Key (K):   "Apa yang aku punya?"
- Value (V): "Informasi yang aku bawa"

Attention Score = softmax(Q @ K^T / √d_k)
Output = Attention Score @ V

Intuisi:

• “The cat sat on the mat because it was tired”
• “It” harus attend ke “cat” (bukan “mat”)
• Self-attention menghitung ini secara otomatis

Multi-Head Attention:

• Jalankan multiple attention “heads” secara paralel
• Setiap head belajar pola berbeda
• Head 1: syntax, Head 2: semantics, Head 3: long-range dependencies

---

Step 4: Feed-Forward Network (FFN)

Setelah attention, setiap token di-process oleh FFN (2-layer MLP):

FFN(x) = ReLU(x @ W1 + b1) @ W2 + b2

Ini adalah tempat “knowledge” disimpan. FFN layers mengandung factual knowledge, reasoning patterns, dan world knowledge.

---

Step 5: Layer Norm & Residual Connections

# Residual connection: membantu gradient flow di network dalam
output = LayerNorm(x + Attention(x))
output = LayerNorm(output + FFN(output))

---

LLM Training Pipeline

Phase 1: Pretraining ($$$$$)

Data: Trillions of tokens (web, books, code)
Objective: Next Token Prediction
Duration: Weeks-Months
Cost: $10M-$100M+ (GPU clusters)
Output: Base model (bisa complete text, tapi gak bisa chat)

# Simplified pretraining loop
for batch in dataloader:  # millions of batches
    # Shift input: predict next token
    inputs = batch[:, :-1]   # "The cat sat on"
    targets = batch[:, 1:]   # "cat sat on the"

    logits = model(inputs)
    loss = cross_entropy(logits, targets)
    loss.backward()
    optimizer.step()

Phase 2: Fine-Tuning / Instruction Tuning ($)

Data: 100K-1M instruction-response pairs
Objective: Follow instructions, helpful responses
Duration: Hours-Days
Cost: $1K-$100K
Output: Chat model (GPT-4, Claude, etc.)

Phase 3: Alignment (RLHF/DPO) ($$)

Data: Human preference comparisons
Objective: Helpful, harmless, honest (HHH)
Duration: Days
Cost: $10K-$500K
Output: Aligned chat model

---

Inference: How Models Generate Text

Autoregressive Generation:

def generate(model, prompt, max_tokens=100):
    tokens = tokenize(prompt)
    for _ in range(max_tokens):
        logits = model(tokens)          # [seq_len, vocab_size]
        next_token_logits = logits[-1]  # last token's prediction
        next_token = sample(next_token_logits, temperature=0.7)
        tokens.append(next_token)
        if next_token == EOS_TOKEN:
            break
    return detokenize(tokens)

Key Parameters:

Parameter	Effect	Range
Temperature	Randomness (0 = deterministic)	0.0 - 2.0
Top-p	Nucleus sampling (cumulative probability cutoff)	0.0 - 1.0
Top-k	Only sample from top-k tokens	1 - 100
Max tokens	Output length limit	1 - 128K+

---

Key LLM Architectures (2025-2026)

Model	Parameters	Context	Key Innovation
GPT-4o	~1.8T (rumored)	128K	Multimodal native
Claude 3.5 Sonnet	Unknown	200K	Computer use, deep reasoning
Llama 3.1 405B	405B	128K	Open weights, MoE
DeepSeek-V3	671B (37B active)	128K	MoE, cost-efficient training
Gemini 2.0	Unknown	1M+	Ultra-long context, multimodal
Grok-2	Unknown	128K	Real-time X integration

---

The Scaling Laws

Chinchilla Scaling Law (DeepMind, 2022):

• Optimal: 20 tokens of training data per parameter
• 1B parameter model → 20B tokens optimal
• 100B parameter model → 2T tokens optimal
• Most models are under-trained (gak cukup data)

---

Latihan

1. Baca “The Illustrated Transformer” (Jay Alammar)
2. Implement simplified self-attention in NumPy/PyTorch
3. Download Llama 3.2 1B, run inference locally via Ollama
4. Experiment dengan temperature, top-p, top-k — lihat perbedaannya
5. Baca Chinchilla paper abstract (DeepMind, 2022)

Target: Paham full transformer pipeline, bisa explain ke orang non-teknis.