<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[Gradients & Ground Truth]]></title><description><![CDATA[Gradients & Ground Truth]]></description><link>https://gradientsandgroundtruth.hashnode.dev</link><generator>RSS for Node</generator><lastBuildDate>Sat, 21 Nov 2026 16:50:14 GMT</lastBuildDate><atom:link href="https://gradientsandgroundtruth.hashnode.dev/rss.xml" rel="self" type="application/rss+xml"/><language><![CDATA[en]]></language><ttl>60</ttl><item><title><![CDATA[Building a Decoder-Only Transformer from Scratch: A Tiny Shakespeare Experiment]]></title><description><![CDATA[I’m currently learning how large language models work — not at the level of calling an API, but understanding what’s actually happening inside. This was my…]]></description><link>https://gradientsandgroundtruth.hashnode.dev/building-a-decoder-only-transformer-from-scratch-a-tiny-shakespeare-experiment</link><guid isPermaLink="true">https://gradientsandgroundtruth.hashnode.dev/building-a-decoder-only-transformer-from-scratch-a-tiny-shakespeare-experiment</guid><category><![CDATA[pytorch]]></category><category><![CDATA[DeepLearning]]></category><category><![CDATA[nlp]]></category><category><![CDATA[transformers]]></category><category><![CDATA[Python]]></category><dc:creator><![CDATA[Siya]]></dc:creator><pubDate>Sat, 21 Nov 2026 01:57:29 GMT</pubDate><content:encoded><![CDATA[<h2>1. Why I Built This</h2>
<p>I’m currently learning how large language models work — not at the level of calling an API, but understanding what’s actually happening inside. This was my first attempt at building a Transformer from scratch in PyTorch.</p>
<p>I didn’t start with <code>RefinedShakespeareTransformer</code>. I started with a basic baseline model, ran it, saw it wasn’t working well, and kept refining — adding learned positional embeddings, Pre-LN layer normalization, residual connections, a 4× expanded FFN with GELU and dropout — until I arrived at the version documented here. The name reflects that iteration history.</p>
<p>The goal was to understand what a Transformer actually computes. Using <code>nn.Transformer</code> or a pretrained model would have answered “does it work” but not “what is it doing.” I wanted to trace how <code>[B, T, C]</code> flows through Q, K, V projections, what <code>-inf</code> masking before <code>softmax</code> is actually enforcing, and how parallel training across T time steps differs from step-by-step generation at inference.</p>
<p>Tiny Shakespeare made sense for this: ~1.11M characters, 65 unique symbols, and immediately readable structure — speaker names in ALL CAPS, line breaks, punctuation — so you can tell at a glance whether the model is picking up real patterns.</p>
<p>Repository: <a href="https://github.com/Sia3136/Tiny-Shakespeare-Decoder-Only-Transformer-Language-Model.git">Tiny Shakespeare Decoder-Only Transformer Language Model</a></p>
<h2>2. Architecture Choice: Decoder-Only</h2>
<p>Next-token text generation is autoregressive — you predict one token at a time given what came before. A decoder-only Transformer is the standard architecture for this.</p>
<p>Honestly, I didn’t do a deep comparison against encoder-decoder at this stage. For a learning experiment focused on understanding attention mechanics, decoder-only was the natural starting point — it’s simpler, it maps directly onto the next-character prediction objective, and it’s what most introductory resources use for this kind of task. Encoder-decoder adds cross-attention and a separate encoder, which would have added complexity without helping me understand the core mechanics I was trying to learn.</p>
<h2>3. Dataset and Preprocessing</h2>
<p><strong>Character-level tokenization</strong></p>
<pre><code class="language-python">chars = sorted(list(set(text)))
vocab_size = len(chars) # 65

stoi = {ch: i for i, ch in enumerate(chars)}
itos = {i: ch for i, ch in enumerate(chars)}
</code></pre>
<p>65 unique symbols: <code>\n</code>, space, punctuation, <code>A–Z</code>, <code>a–z</code>. With <code>embedding_dim = 32</code>, the token embedding table is <code>65 × 32 = 2,080</code> parameters and the output head is another <code>2,080</code> — small enough to train quickly. A 50k-token BPE vocabulary would require millions of parameters in those two layers alone, which didn’t make sense for a first scratch experiment.</p>
<p>The tradeoff: <code>block_size = 32</code> characters spans only ~5–6 English words. The model spends most of its capacity learning character-by-character spelling rather than sentence-level structure.</p>
<p><strong>Train-validation split and sequential batching</strong></p>
<p>Contiguous 90:10 split: ~1.0M characters for training, ~111k for validation.</p>
<pre><code class="language-python">def prepare_data_matrix(dataset, batch_size, block_size):
    chunk_size = block_size * batch_size
    num_chunks = (len(dataset) - 1) // chunk_size
    truncated = dataset[:num_chunks * chunk_size + 1]
    x_matrix = truncated[:-1].view(batch_size, -1)
    y_matrix = truncated[1:].view(batch_size, -1)
    return x_matrix, y_matrix

def get_sequential_batch(split, step_idx):
    x_mat = train_x_mat if split == "train" else val_x_mat
    y_mat = train_y_mat if split == "train" else val_y_mat
    num_steps = x_mat.shape[1] // block_size
    col_idx = step_idx % num_steps
    start = col_idx * block_size
    end = start + block_size
    return x_mat[:, start:end], y_mat[:, start:end]
</code></pre>
<p><code>prepare_data_matrix</code> reshapes input <code>truncated[:-1]</code> and 1-step-shifted target truncated<code>[1:]</code> into <code>[64, N]</code> matrices. <code>get_sequential_batch</code> slices deterministic, non-overlapping 32-character chunks.</p>
<p><strong>A limitation I noticed after:</strong> Within each 32-character slice, the first position can attend only to itself, the second to the first two positions, and so on. The model cannot access preceding characters outside that slice. The same chunk boundaries also recur across epochs. Random sliding-window offsets could vary the alignment of training windows and reduce dependence on fixed chunk boundaries.</p>
<table>
<thead>
<tr>
<th>Parameter</th>
<th>Value</th>
</tr>
</thead>
<tbody><tr>
<td>Vocabulary size</td>
<td>65</td>
</tr>
<tr>
<td>Batch size</td>
<td>64</td>
</tr>
<tr>
<td>Context length</td>
<td>32</td>
</tr>
<tr>
<td>Embedding dimension</td>
<td>32</td>
</tr>
<tr>
<td>Train/val split</td>
<td>90:10</td>
</tr>
<tr>
<td>Dropout</td>
<td>0.1</td>
</tr>
</tbody></table>
<h2>4. Model Architecture — 16,736 Parameters</h2>
<pre><code class="language-plaintext">class RefinedShakespeareTransformer(nn.Module):
    def __init__(self, vocab_size, embedding_dim, hidden_dim, block_size, dropout_p):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, embedding_dim)
        self.position_embedding = nn.Embedding(block_size, embedding_dim)

        self.ln1 = nn.LayerNorm(embedding_dim)
        self.ln2 = nn.LayerNorm(embedding_dim)

        self.Wq = nn.Linear(embedding_dim, embedding_dim, bias=False)
        self.Wk = nn.Linear(embedding_dim, embedding_dim, bias=False)
        self.Wv = nn.Linear(embedding_dim, embedding_dim, bias=False)

        self.ffn = nn.Sequential(
            nn.Linear(embedding_dim, hidden_dim),
            nn.GELU(),
            nn.Linear(hidden_dim, embedding_dim),
            nn.Dropout(dropout_p)
        )
        self.lm_head = nn.Linear(embedding_dim, vocab_size, bias=False)
        self.dropout_p = dropout_p
        self.block_size = block_size
</code></pre>
<h3>RefinedShakespeareTransformer — forward pass</h3>
<img src="https://cdn.hashnode.com/uploads/covers/6a16cb0eff27a644af276d39/0602c0e9-a157-496e-900a-705cdb7ddb33.png" alt="" style="display:block;margin:0 auto" />

<h3>4.1 Token + Positional Embeddings</h3>
<pre><code class="language-plaintext">tok_emb = self.embedding(idx)                 # [B, T, C]
pos_emb = self.position_embedding(positions)  # [T, C]
emb = tok_emb + pos_emb                       # [B, T, C]
</code></pre>
<p>Self-attention without positional information does not inherently encode the order of tokens. Permuting the input sequence also permutes the resulting representations, rather than giving the model an independent representation of absolute position. Learned positional embeddings provide that positional information.</p>
<p>One thing this forced me to handle: <code>position_embedding</code> has exactly 32 slots. During generation, the context must be cropped to <code>idx[:, -self.block_size:]</code> at every step — otherwise it crashes with an index error at character 33. I only understood why this was necessary after running into it.</p>
<h3>4.2 Single-Head Causal Self-Attention</h3>
<pre><code class="language-python">norm_emb = self.ln1(emb)
Q = self.Wq(norm_emb)
K = self.Wk(norm_emb)
V = self.Wv(norm_emb)

scaled_scores = (Q @ K.transpose(-2, -1)) / math.sqrt(emb.shape[-1])
causal_mask = torch.triu(torch.full((T, T), float('-inf'), device=idx.device), diagonal=1)
masked_scores = scaled_scores + causal_mask
attn_weights = torch.softmax(masked_scores, dim=-1)
attn_weights = F.dropout(attn_weights, p=self.dropout_p, training=self.training)

attn_out = attn_weights @ V
x = emb + attn_out
</code></pre>
<p><code>Wq</code>, <code>Wk</code>, <code>Wv</code> each project <code>32 → 32</code> directly — single-head, no multi-head splitting, no post-attention output projection \(W_O\). I started with one head to verify the matrix math cleanly before adding more complexity.</p>
<p><strong>On the</strong> <code>-inf</code> <strong>mask</strong>: During training, all 32 targets are predicted in parallel. The upper-triangular mask filled with <code>-inf</code> before <code>softmax</code> ensures future positions get \(\exp(-\infty) = 0\) weight — they effectively don’t exist. If you mask after <code>softmax</code>, or fill with 0 before <code>softmax</code>, future token information leaks into the prediction. Getting this ordering right was one of the specific things I wanted to understand by implementing it myself.</p>
<p><strong>On scaling by</strong> \(\frac{1}{\sqrt{d_k}}\)<strong>:</strong> Without it, dot products grow large as <code>d_k</code> increases, potentially pushing <code>softmax</code> toward saturation, where gradients can become very small.</p>
<h3>4.3 Pre-LN Residual Structure</h3>
<p>$$X_1 = X + \mathrm{Attention}(\mathrm{LN}_1(X))$$</p>
<p>$$X_2 = X_1 + \mathrm{FFN}(\mathrm{LN}_2(X_1))$$</p>
<p>Residual connections let gradients flow directly back to the embeddings without passing through every transformation. Pre-LN places <code>LayerNorm</code> on the branch before Attention and FFN, keeping that path clean.</p>
<blockquote>
<p><strong>Note on the architecture diagram:</strong> In Pre-LN, the attention residual bypasses <code>ln1</code>, originating from <code>emb</code> before normalisation. The FFN residual bypasses <code>ln2</code>, originating from the output of the attention addition. The diagram’s + <code>residual</code> labels communicate that residuals exist — if you’re reading the forward pass, trace where each skip connection starts, not just where it rejoins.</p>
</blockquote>
<p><strong>Something I missed:</strong> I didn’t place a final <code>LayerNorm</code> before <code>lm_head</code>, which a standard GPT-style implementation would include. I noticed this in retrospect.</p>
<h3>4.4 Feed-Forward Network</h3>
<pre><code class="language-python">self.ffn = nn.Sequential(
    nn.Linear(embedding_dim, hidden_dim),  # 32 → 128
    nn.GELU(),
    nn.Linear(hidden_dim, embedding_dim),  # 128 → 32
    nn.Dropout(dropout_p)
)
</code></pre>
<p>Self-attention mixes information across positions but doesn’t transform it nonlinearly. The FFN is where the nonlinear computation happens at each position after context has been gathered.</p>
<p>The 4× expansion (<code>32 → 128 → 32</code>) accounts for 8,352 parameters — nearly <strong>50% of the entire 16,736-parameter model.</strong> That surprised me when I calculated it.</p>
<h3>4.5 Language Modelling Head</h3>
<pre><code class="language-python">self.lm_head = nn.Linear(embedding_dim, vocab_size, bias=False)  # 32 → 65 
</code></pre>
<p>Outputs raw logits for all 65 characters. <code>CrossEntropyLoss</code> applies log-softmax internally during training. At inference, <code>torch.softmax(logits[:, -1, :], dim=-1)</code> converts the final time step’s logits to probabilities for <code>torch.multinomial</code>.</p>
<p><strong>What this implementation is missing</strong></p>
<p>Compared to a full GPT-style architecture: multi-head attention, stacked Transformer blocks, attention output projection \(W_O\), <code>final LayerNorm</code>. These were intentional simplifications — the point was to understand one block cleanly, not to replicate GPT.</p>
<h2>5. Training</h2>
<pre><code class="language-python">optimizer = optim.AdamW(model.parameters(),lr=3e-4)
loss_fn = nn.CrossEntropyLoss()
</code></pre>
<p>Each step processes <code>64 × 32 = 2,048</code> character positions. ~490 steps per epoch, trained on Google Colab GPU. I don’t have the exact training time recorded — it was fast enough on Colab’s free GPU that it wasn’t a bottleneck.</p>
<p>Validation was evaluated on the first validation chunk only — <code>get_sequential_batch("val", 0)</code> — not the full 111k-character split. I wrote it that way initially and only recognised it as a limitation afterward.</p>
<table>
<thead>
<tr>
<th>Config</th>
<th>Value</th>
</tr>
</thead>
<tbody><tr>
<td>Optimizer</td>
<td>AdamW</td>
</tr>
<tr>
<td>Learning rate</td>
<td>3e-4</td>
</tr>
<tr>
<td>Epochs</td>
<td>5</td>
</tr>
<tr>
<td>Steps/epoch</td>
<td>~490</td>
</tr>
<tr>
<td>Tokens/step</td>
<td>2,048</td>
</tr>
<tr>
<td>Environment</td>
<td>Google Colab GPU</td>
</tr>
</tbody></table>
<h2>6. Results</h2>
<table>
<thead>
<tr>
<th>Epoch</th>
<th>Train Loss</th>
<th>Val Loss</th>
<th>Val Perplexity</th>
</tr>
</thead>
<tbody><tr>
<td>1</td>
<td>3.2190</td>
<td>2.7295</td>
<td>15.3250</td>
</tr>
<tr>
<td>2</td>
<td>2.6787</td>
<td>2.5459</td>
<td>12.7550</td>
</tr>
<tr>
<td>3</td>
<td>2.5568</td>
<td>2.4437</td>
<td>11.5159</td>
</tr>
<tr>
<td>4</td>
<td>2.4736</td>
<td>2.3576</td>
<td>10.5659</td>
</tr>
<tr>
<td>5</td>
<td>2.4157</td>
<td>2.3076</td>
<td>10.0501</td>
</tr>
</tbody></table>
<p>A uniform predictor over 65 characters has perplexity 65. A perplexity of 10.05 represents an effective branching factor of approximately 10 under the evaluated distribution. It does not mean that exactly 10 characters are equally likely at every position.</p>
<p><strong>Two things to be clear about:</strong></p>
<ol>
<li>Val loss and perplexity are from the first validation chunk only — 2,048 tokens, not the full validation split. These numbers are real but not a complete picture.</li>
<li>Val loss (<code>2.3076</code>) is lower than train loss (<code>2.4157</code>) — this isn’t a bug. Train loss is a running average across the epoch with dropout active, while validation runs after the epoch finishes with <code>model.eval()</code> and dropout off.</li>
</ol>
<h2>7. Generated Output</h2>
<p>Prompt: <code>'\n'</code>, 200 characters:</p>
<p><code>KINIUSERLABWhake. I thek INF hoo Vonor me whimin CAUchemy th? I thithin s to indo is spoklo; Yof me'cetis, feered be utu bato ON: Fowand ho twsoour Fhand harte omthy, Af IONGHizlsiy milalemss I swe. F</code></p>
<p><strong>What it got:</strong> Speaker-like formatting, uppercase after newlines, plausible punctuation, real short function words (<code>I</code>, <code>to</code>, <code>is</code>, <code>be</code>, <code>me</code>), English-sounding fragments (<code>whimin</code>, <code>feered</code>, <code>harte</code>).</p>
<p><strong>What it didn’t get:</strong> Longer sequences collapse into pseudo-words (<code>KINIUSERLABWhake</code>, <code>CAUchemy</code>, <code>spoklo</code>). No consistent word boundaries or sentence grammar. I sampled at temperature 1.0 with no top-k or top-p filtering, which lets low-probability characters appear mid-word.</p>
<p>This experiment highlighted an interesting gap between the evaluation metric and generated output. The decreasing perplexity showed that the model was improving its next-character predictions on the evaluated validation chunk. However, this did not translate into coherent word formation or sentence-level generation. The generated text made that distinction visible.</p>
<h2>8. Limitations</h2>
<table>
<thead>
<tr>
<th>Area</th>
<th>Issue</th>
</tr>
</thead>
<tbody><tr>
<td>Model capacity</td>
<td>1 layer, 1 head, <code>C=32</code> — too small to do much</td>
</tr>
<tr>
<td>Context window</td>
<td>32 chars ≈ 5–6 words</td>
</tr>
<tr>
<td>Batching</td>
<td>Fixed chunk boundaries, same every epoch</td>
</tr>
<tr>
<td>Validation</td>
<td>Only chunk 0 evaluated</td>
</tr>
<tr>
<td>Tokenization</td>
<td>Character-level — model learns spelling from scratch</td>
</tr>
<tr>
<td>Architecture</td>
<td>No multi-head, no stacked blocks, no \(W_O\) , no final LN</td>
</tr>
</tbody></table>
<h2>9. What I Would Do Differently</h2>
<p>These are things I’d change next time — not things I implemented here.</p>
<table>
<thead>
<tr>
<th>Area</th>
<th>Change</th>
<th>Why</th>
</tr>
</thead>
<tbody><tr>
<td>Batching</td>
<td>Random sliding-window offsets</td>
<td>Reduce dependence on fixed chunk boundaries</td>
</tr>
<tr>
<td>Validation</td>
<td>Loop over all val chunks</td>
<td>Reliable generalisation estimate</td>
</tr>
<tr>
<td>Evaluation</td>
<td>Report BPC (\(\mathcal{L} / \ln 2\))</td>
<td>Standard char-level metric</td>
</tr>
<tr>
<td>Attention</td>
<td>Multi-head with \(W_O\)</td>
<td>Different heads learn different relationships</td>
</tr>
<tr>
<td>Architecture</td>
<td>Stacked blocks, final LN, <code>C=128+</code>, <code>T=128–256</code></td>
<td>More capacity</td>
</tr>
<tr>
<td>Tokenization</td>
<td>Try subword tokenization</td>
<td>Compare how it changes text represented per context window</td>
</tr>
<tr>
<td>Generation</td>
<td>Temperature + top-k/top-p</td>
<td>Controllable output</td>
</tr>
<tr>
<td>Optimization</td>
<td>LR warmup + cosine schedule</td>
<td>Better training dynamics</td>
</tr>
</tbody></table>
<h2>10. What This Built</h2>
<p>The clearest thing this experiment produced was understanding the gap between a <em>working implementation</em> and <em>a capable model</em>. Loss going down doesn’t mean the output makes sense — the pseudo-words in the generated text make that concrete.</p>
<p>I’m still learning this area. The specific questions I want to investigate next come directly from the gaps here: what does multi-head attention actually contribute, how does context length change what the model can represent, and whether fixing the sliding-window batching changes the loss curve in practice. Those feel like answerable questions, which is what I wanted from a first experiment.</p>
<p>References:<br /><a href="https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt">Tiny Shakespeare dataset</a><a href="https://arxiv.org/abs/1706.03762"><br />Attention Is All You Need — Vaswani et al. (2017)</a><br /><a href="https://docs.pytorch.org/docs/2.15/index.html">PyTorch Documentation</a><br /><a href="https://github.com/Sia3136/Tiny-Shakespeare-Decoder-Only-Transformer-Language-Model.git">Project repository</a></p>
]]></content:encoded></item></channel></rss>