Training Tiny Manas: How I Built a Small Language Model Based on the Manas Epic
Abstract
After several years of building systems around LLMs, I decided to study the model itself: from the way text becomes numbers to the way backpropagation changes millions of parameters. The first major achievement in that process was Tiny Manas, a small decoder-only Language Model that I implemented and trained to continue the Kyrgyz epic Manas.
The experiment follows the complete path from plain text to next-token prediction. My Kyrgyz byte-level BPE tokenizer turns the epic into tokens. Token and position embeddings create their internal representations. Eight Transformer blocks collect and process context through Causal Multi-Head Self-Attention and Feed-Forward Networks. The LM Head scores the next token, Cross-Entropy measures the error, and backpropagation with AdamW gradually changes the weights.
The final Tiny Manas contains 26,877,696 parameters and has a context window of 256 tokens. I trained it from scratch on 465,069 tokens from one edition of Manas, using an M5 MacBook Pro with 16 GB of unified memory. A complete run took 24.4 minutes. Increasing the model from 13.2M to 26.9M parameters reduced validation perplexity by 25.4% and test perplexity by 23.1%. A more obvious hypothesis failed: doubling the context window made training slower and the held-out results worse.
Tiny Manas learned names, rhythm, forms of address, and local chains of action from the epic. It still repeats itself, invents broken words, and eventually loses the plot. This essay therefore does not present the model as a Kyrgyz counterpart to ChatGPT. It explains how a Language Model works, how I verified each part of my implementation, where I caught a measurement bug, how I encountered overfitting, and why the final result looks the way it does.
Part I. How a Language Model Works
We will begin by assembling the theory into one clear chain: how ordinary text becomes tokens, where the tokens get their internal numerical representations, how a Transformer works with context, and how the model learns to predict a continuation. Without this foundation, the numbers from my experiment later on would be little more than a nice-looking table.
1. Why look inside an LLM at all?
I have worked closely with AI for several years. I started with simple chatbots and eventually moved into complex agent systems in which multiple models use tools, retain state, and carry out long workflows. At some point, though, I caught myself facing an uncomfortable thought. I could build increasingly complex things around Large Language Models, or LLMs, but I could not honestly say that I understood the model itself from the inside.
An awkward realization, to be honest. You can already assemble fairly complex systems on the outside, while the model at the center is still a fog. That did not sit well with me.
Progress in any field cannot happen only by moving sideways. Sometimes you have to stop and dig down: study the mathematics, work through the mechanics by hand, and understand why the system works at all. So I began devoting a significant part of my time outside work to studying how LLMs work. I studied tokenization, embeddings, backpropagation, Attention, and the other components usually hidden behind a single API call.
Eventually those separate ideas began to form one stable picture. I reached my first real achievement: I could write and train my own Language Model. Not a large LLM on the scale of ChatGPT, of course, but a Small Language Model. I called it Tiny Manas, and trained it to continue the Kyrgyz epic Manas.
This essay is more than a report on the final numbers. It is my attempt to assemble everything I learned into one continuous story and make the core ideas understandable even to a reader without a technical background. There will be technical details. But each one will begin with a reason: what problem did we run into, why did it need solving, and only then how does the solution work?
2. What is a Language Model?
Let us begin with the simplest definition. Put plainly, a Language Model predicts the next piece of text. It receives the beginning of a text and tries to continue it. We write:
Manas raised his ...
and the model must decide which piece of text is most likely to come next. Perhaps sword. Perhaps voice. Perhaps something completely different.
Once the magic is removed, the work has three broad parts:
- prepare the text and turn it into numbers a computer can process;
- train a model to find patterns in those numbers;
- use the trained model to generate new text.
The last part is called inference, or generation. During inference the model is no longer learning. It simply uses what it learned earlier and predicts new pieces of text one at a time.
But what does it mean to train a model? Imagine that it has millions of tiny muscles. In reality those muscles are called weights, or parameters, and each one is an ordinary number. The architecture fixes the number of parameters before training begins. If we build a model with 26.9 million parameters, training will not create more or remove any. It will only change the values of those same 26.9 million numbers. And those numbers look roughly like this:
Right after initialization, the parameters are almost random and poorly coordinated. The model can already calculate an answer, but the result resembles a newborn trying to control every muscle at once.
During training, the model makes a prediction, measures its mistake, and adjusts the parameters slightly so that a similar mistake becomes less likely. One update changes almost nothing. Thousands of updates create structure.
As with training a body, there are two extremes. Too little training produces underfitting: the model has not learned even the main patterns. Too much repetition of a limited dataset produces overfitting: the model knows the training text almost by heart and breaks on new material. It is like memorizing the answers to one version of an exam without understanding the subject. Or training nothing but the bench press and then failing to do five squats. Ideally, that is not how we train.
This is why the text is usually divided into three parts. The training split changes the parameters. The validation split acts as a recurring exam: it helps us choose good decisions and decide when to stop. The test split remains untouched until the final honest evaluation. If training loss keeps falling while validation loss rises, the model is not becoming smarter. It is beginning to memorize.
Inference begins after training ends. If training was an athlete's long preparation, inference is one performance at the competition. The muscles no longer change. The model simply uses the ability it has built.
3. How text becomes numbers
A computer does not read the word Манас the way we do. To an algorithm, it is not an epic hero or even a word. At the lowest level, a computer stores bits: zeros and ones. You have probably seen a movie hacker breaking into the Pentagon while streams of zeros and ones fly across the screen. This is one of those rare moments when the movies contain at least a little truth. The Latin word Manas, for example, can be shown in UTF-8 like this:
01001101 01100001 01101110 01100001 01110011
Each group of eight bits is called a byte. We can write the same information as ordinary numbers from 0 to 255:
77 97 110 97 115
Long chains of zeros and ones are hard to read, so from now on we will show each group of eight bits as one ordinary number. 01001101 and 77, for example, are two ways of writing the same byte. The information has not changed; we have simply written it in a shorter, clearer form. Instead of forty zeros and ones, Manas now looks like five understandable numbers: 77 97 110 97 115. Great, we turned a word into numbers! Can we feed them straight into the model now? Technically, yes. But another problem appears immediately.
Did you notice that even the short word Manas takes up five whole positions and consists of five separate numbers? A Kyrgyz word may take even more. The longer the sequence, the more work the model will have to do later and the sooner it fills its limited context window. The context window is the fixed number of tokens the model can work with at once. If one short word consumes many tokens, less useful text fits inside that window: fewer words, fewer sentences, and less context.
So we need some way to compress this long sequence: make it shorter without losing the original text. This is where a tokenizer enters the scene. Familiar word, right? We hear about “tokens” constantly now, but what is a token, really? It is the smallest piece a Language Model operates on. It does not have to be a word. A token may be a whole word, part of a word, punctuation, or a single byte. Spoiler: the tokenizer decides what counts as a token.
As you may already have guessed, the tokenizer is a translator between us and the Language Model. Following a fixed set of rules, it turns ordinary text, including words, letters, spaces, and punctuation, into a sequence of tokens: numbers the model can work with. When the model finishes generating, the tokenizer translates in the other direction and assembles those numbers back into text we can read.
One tokenizer might represent Hello as a single token, while another might use Hel + lo. Both decode back into the same text, but the sequence lengths differ. That matters to the model: it counts tokens, not human words or letters.
3.1. Why I used byte-level BPE
Okay, now we understand the job: somehow compress that long sequence of bytes. But how? Several algorithms exist for this. In my tokenizer I used one of the best-known approaches: Byte Pair Encoding, or BPE. The name sounds much grander than the idea. The algorithm looks for pieces of text that often appear next to each other and merges them into a new token.
Start with the bytes of Manas:
77 97 110 97 115
Suppose the pair 77 97, or Ma, appears often in the corpus. IDs from 0 through 255 are already reserved for every possible byte, so the first new token gets ID 256. Why 256? Because every earlier slot is taken. If that did not quite click yet, do not worry. Just remember that learned tokens begin at 256.
new token: 256 = [77, 97] together = Ma
Now we replace the two neighboring numbers 77 97 in the original sequence with the single new token 256:
before: 77 97 110 97 115
after: 256 110 97 115
See? No information disappeared: token 256 still means Ma. But the sequence is shorter. It now contains four tokens instead of five.
Then we repeat the same process. If 256 110, or Man, also occurs frequently, we create another token:
new token: 257 = [256, 110] together = Man
And once again, we replace two neighboring tokens with one:
before: 256 110 97 115
after: 257 97 115
Only three tokens remain from the original five. If Manas appears often enough, BPE can continue merging its pieces and eventually represent it even more compactly.
But how does BPE know which pairs are frequent? It has to be trained too. Yes, even the tokenizer learns, although it is not a neural network. It takes a large text corpus, counts every adjacent pair, merges the most frequent one, counts again, and repeats until the vocabulary reaches the chosen size. Frequent words and word pieces gradually become larger tokens. Rare words remain assembled from smaller pieces.
The main advantage of the byte-level approach is that unknown text is almost impossible. If BPE has never seen Avada Kedavra, at worst it breaks the phrase into bytes and still reads it. Not very compact, but no Voldemort is going to break it. That is the entire basic idea of BPE: if pieces appear together often, merge them together. Hey! We did it!
Before Tiny Manas, I had already built my own Kyrgyz byte-level BPE tokenizer and published the implementation on GitHub. In that project I implemented training, encoding, and decoding myself, assembled the corpus, and compared several designs. For Tiny Manas I froze one specific kyrgyz-byte-bpe-v1 artifact with a vocabulary of 32,768 tokens. This means that during training BPE added 32,512 learned tokens to the original 256 byte tokens: common pairs, word pieces, and entire words.
I also kept the tokenizer unchanged between model experiments. Otherwise I could not tell whether the model itself had improved or whether I had simply changed the way the text was packed.
An ordinary sentence can now become a short sequence of IDs:
Манас кыргыз элинин баатыры
↓
[2499, 772, 3953, 15726]
To us these are four pieces of text. To the model they are four addresses in a vocabulary. At last, we have a language in which we can speak to it.
4. Choosing the world Tiny Manas would learn
Now things become more interesting. We have a tokenizer and can turn text into numbers. Can we train a Language Model? Not yet. Do you know why? We still have nothing to train it on. We need a dataset: real text whose patterns the model can search for.
In theory, the text can be anything. Want a Harry Potter-like model? Train it only on the Harry Potter books. There is so little text, though, that the model would mostly learn names and familiar phrases rather than develop a broad command of language.
Educational projects in English often use Tiny Shakespeare: one compact corpus, one recognizable voice, and output that is easy to judge by ear. There are more literal experiments too. The author of Vintage-LLM collected texts published before 1900 to build a historically bounded model. Spoiler: the first 14M-parameter version mostly produced Victorian-sounding nonsense. Increasing both the data and the model helped. Historical atmosphere does not repeal the basic rule that little data plus a small model produces limited ability.
My goal was not to create an expert on all of Kyrgyz culture or compete with modern LLMs. I wanted to implement the full Transformer path with my own hands and produce at least locally meaningful generation. I needed a narrow, coherent, and personally interesting world. The Kyrgyz epic Manas became my counterpart to Tiny Shakespeare.
I used Manas01 from the Manas-UdS corpus, performed by Sayakbai Karalaev and distributed under CC BY-NC-SA 4.0. The digital document began with roughly 30,000 characters of scholarly introduction. In an earlier bigram experiment, I had accidentally left that material in and ended up with a model that learned academic vocabulary alongside the epic. For Tiny Manas I removed 30,877 leading characters and began with the heading Манастын туула элегиндеги бабалары.
After cleaning, the corpus contained:
- 1,794,194 characters;
- 3,249,652 UTF-8 bytes;
- 465,069 BPE tokens;
- 9,593 distinct token IDs that actually occur in the epic.
I preserved narrative order and split the sequence chronologically: 418,562 tokens for training, 23,253 for validation, and 23,254 for testing. If I had randomly mixed neighboring windows, the model could have trained immediately beside a validation passage. The score might have looked better, but the exam would have been dishonest.
5. What exactly does a Language Model learn?
Okay. We have a clean epic and its tokens. But what task do we actually give the model? It turns out to be the simplest possible one: predict the next token.
Suppose tokenization gives us this sequence:
Манас | көтөрдү | өз | кылычын
From it, we automatically create both the questions and the correct answers:
input: Манас | көтөрдү | өз
target: көтөрдү | өз | кылычын
The targets are the same sequence shifted forward by one position. After Манас, the correct answer is көтөрдү. After көтөрдү, it is өз. After өз, it is кылычын.
Sounds suspiciously simple, right? Yet to predict the next token consistently, the model has to learn the structure of the language, recurring facts, style, characters, and relationships between distant words. We never program the concepts of a hero or a sword by hand. We simply ask the same question over and over: “What comes next?” The model is forced to discover everything else as a useful way of answering it.
6. Token embeddings: how an ID gains internal meaning
Okay, the data is ready and so is the task. Now we can get to the good part. Or almost. First we have another problem to solve: a token ID means nothing by itself. The number 2499 is no closer to Manas in meaning than 2500. It is just an address. We need to give every token an internal numerical representation that the model can change during training.
For this we create a token embedding table. In Tiny Manas it has 32,768 rows, one for every token in the vocabulary, with 384 numbers in each row. If the token Манас happens to have ID 2499, looking it up returns its current vector:
2499 → [0.10, 1.20, 0.70, -0.50, ... 380 more numbers]
At first these numbers are almost random. During training they change along with all the other weights. Tokens used in similar situations gradually develop useful internal properties. Do not imagine that one number literally means “hero” and another means “weapon.” Meaning is spread across the whole vector and is mainly useful to the model's later layers.
If we return to the muscle analogy, each row in the embedding table is the first set of trainable muscles belonging to one token. But meaning alone is not enough. The model must also know where the token appears. It turns out that we need one more table...
7. Position embeddings: the same word in different places
The attention mechanism we will discuss shortly does not know token order by itself. Without extra information, a set of words looks like people seated around a table with no seat numbers. So we add a second vector to the token's meaning vector, one that describes its position.
Tiny Manas uses a learned position embedding table. Its size is determined not by the vocabulary, but by the maximum context: 256 positions, each with its own 384 numbers.
numbers describing the token “Manas”
+ numbers describing the third position
= one new vector: “Manas is in the third position”
The token embedding tells the model roughly what this piece is. The position embedding adds where it appears in the current window. The same token therefore gets a different final representation in the first and the hundredth position.
The number 256 is called the block size, or the model's context size. It is the maximum number of tokens Tiny Manas can use at once for one prediction. If the current fragment is shorter, its actual length is usually written as T. So T can range from 1 to 256, while block size remains the architectural ceiling.
After adding token and position embeddings, we get X with shape (B, T, 384). B is the number of independent sequences in one batch, T is their current length, and 384 is the width of each token vector. This X is what finally enters the Transformer.
8. Transformer: where tokens acquire context
What is a Transformer, anyway? No, not Optimus Prime, although the diagram can look just as intimidating at first.
The Transformer is the main computational body of Tiny Manas. It consists of eight nearly identical blocks arranged one after another. The text representation passes through the first block, its result enters the second, then the third, and so on. Each block does two large things: first, it lets tokens collect information from one another through Causal Multi-Head Self-Attention; then it processes what they collected inside a Feed-Forward Network.
The word he tells us almost nothing on its own. But in Manas raised the sword, and he..., its representation needs information from the preceding text. After passing through the Transformer, the token describes not only itself but also the context in which it appeared.
By context, I mean the previous tokens available to the model in its current window. Tiny Manas can see up to 256 tokens. That does not mean it understands the whole epic at once. During long generation, older tokens gradually fall off the left edge of the window.
All eight blocks have the same structure, so there is no reason to dissect every one. Let us take a single Transformer block, cut it open, and see what is inside. This is where the model becomes most densely packed. So, as Dante Alighieri wrote: “Abandon all hope, ye who enter here.”
9. LayerNorm: keeping numbers in a workable range
The first thing we meet inside a Transformer block is LayerNorm. Do not let the name scare you; its job is simple. It keeps the 384 numbers representing each token in a range that is comfortable for the model.
Imagine a sound engineer receiving a recording with some sections far too loud and others barely audible. Before doing anything else, they bring the volume into a sensible working range. The music does not change. It simply becomes easier to work with.
LayerNorm does the same thing to numbers. It does not make them identical or erase the token's meaning. The largest value is still the largest, and the smallest is still the smallest. LayerNorm only moves the whole set into a more convenient “weight class” so the model can learn more steadily.
That is all we need here. LayerNorm is a small preparation step: the following calculations become easier, and the parameters learn more steadily because they do not have to keep adapting to wildly changing scales.
10. Self-Attention: how a token decides where to look
Still with me? Good. Then please pay attention. The pun is fully intentional, because we have reached the most important part of the Transformer: Self-Attention.
Until now, each token knew its own meaning and its place in the sentence, but almost nothing about its neighbors. Take the word he. By itself it says very little. In Manas raised the sword, and he..., the model needs to look back and understand what this word refers to. Attention gives it that ability.
Attention rests on three properties: Query, Key, and Value. They are the three horsemen of the apocalypse our holy trinity.
Imagine that every token in the current sentence has all three.
Query means: “What information am I looking for from the other tokens?”
Key means: “Here is the kind of information I can offer.” But the Key is not the information itself. It is only the label attached to it.
Value contains the actual useful information another token can take.
Think of an ordinary package. The Key is the label describing what is inside: we use it to decide whether we need the box. The Value is the contents. First we read the label, and only if it matches our request do we open the package and take what is inside.
Okay, we understand the three properties. Now let us seat the tokens around a table.
The table does not contain all 32,768 tokens in the vocabulary, and it certainly does not contain the whole epic. Only the tokens from the current fragment are there. For example:
Manas | raised | his | sword
Every token shows the others its Query and Key. Roughly speaking, his says: “Here is the information I am looking for.” Manas, raised, and even his itself answer: “And here is what I can offer.”
The model compares the request from his with the label on every available token. If the Query matches a Key well, the connection is strong. If they barely fit, the connection is weak. In this way, every token gradually decides whom around the table it should pay more attention to.
In mathematical language, the whole conversation is written very briefly:
Q @ Kᵀ
Do not be intimidated by the notation. It means only one thing: “Compare the Query of every token with the Key of every other available token.” The result is a set of internal connection scores. The higher the score, the more useful one token appears to another.
10.1. Causal mask: hiding the answers during an exam
But now we run into a serious problem. During training the whole sentence is already loaded into the computer. When the model is standing on his, the correct answer, sword, is already sitting to its right. Without a restriction, it could simply peek over and say, “Oh, obviously. The next token is sword. Easiest training session of my life.”
That would not be learning. It would be plain cheating, like taking an exam with all the correct answers lying open in front of you.
So we cover all future tokens with a curtain called the causal mask. For the token his, only these positions remain visible:
Manas | raised | his
It is not allowed to look here:
sword
Each token can therefore look at itself and everything to its left, but never to its right. This is why the mechanism is called causal, not casual: it preserves the proper order of cause and effect.
10.2. Softmax: distributing 100% of attention
After the round-table discussion, we have internal connection scores. On their own, though, they do not tell us much. What does a score of 2.0 mean? Is that high or low? This is where Softmax enters.
Imagine that his has exactly 100% of its attention to distribute among the available participants. Softmax turns opaque scores into understandable shares. For example:
Manas: 67%
raised: 24%
his: 9%
sword: 0% ← hidden by the causal mask
Now the idea is much clearer. The token his decided that most of the useful information should come from Manas, a little less from raised, still less from itself, and none from the future word sword.
These percentages are called attention weights. They are not probabilities of the next token. They only specify how much information to take from each available participant at the table.
10.3. Value: finally taking the information
Okay. We found the relevant tokens and decided how much attention each one deserves. But we have not received the useful information yet. So far, we have only read the labels on the boxes.
This is where Value finally appears.
Query helped us state what we wanted. Key helped us find suitable sources. Softmax decided how much attention to assign to each one. Value contains the information we now take.
In our example, his takes 67% of the information from the Value of Manas, 24% from raised, 9% from its own Value, and nothing from the future sword. The collected information is then mixed into one new vector.
The token his is no longer sitting in a vacuum. Its representation now contains information that Manas appeared earlier and that he raised something. The token inspected the available context, decided who mattered, took the useful information, and updated itself. That is Self-Attention.
One important note: Query, Key, and Value are not permanently attached to the token Manas. They are created again for every specific sentence. In another context, the token Manas will look for and offer different information.
11. Why Attention needs several heads
So far, I have described Attention as if one observer were handling the whole round table. But one observer cannot easily notice every possible relationship in the text at once.
Imagine a complex case assigned to a single detective. They may identify who performed an action but miss who owns an object. Or they may understand the nearby words and overlook an important clue from the beginning of the text.
Tiny Manas therefore uses eight detectives instead of one. Each receives the entire same text but examines it in a different way. One may learn to notice acting characters, another relationships between objects, a third more distant context. We do not assign these roles by hand. At the beginning of training, none of the detectives understands much of anything. They gradually discover useful ways to examine the text.
Each detective is called an Attention Head. Inside every head, the same familiar process happens independently: Query is compared with Key, the causal mask hides the future, Softmax distributes attention, and Value delivers information.
At the end, each head writes a small numerical report about what it noticed. We do not need to choose the smartest detective. The model collects all eight reports and mixes them into one updated representation of the token.
That is Causal Multi-Head Self-Attention. Self means the tokens communicate within the same sequence. Multi-Head means several attention systems inspect it at once. Causal means none of them can look into the future.
12. Residual connections: do not make every layer rewrite everything
After Attention, a token receives new information from the context. But if we completely replaced everything it knew before, some useful information could disappear.
So the model preserves the token's original representation and simply adds the new information to it. This is called a Residual Connection.
Imagine an editor who does not rewrite an entire document from scratch. They keep the original text and add comments to it. Even if one comment turns out to be useless, the document itself has not disappeared.
The same thing happens to a token:
what the token knew before
+ what it learned now
= the updated token representation
Each Transformer block therefore does not recreate the token representation from nothing. It gradually adds to what is already there.
13. Feed-Forward Network: time to think about what was collected
Remember our round table? During Attention, the token spoke with earlier tokens and collected useful information from them. It may have learned that Manas appears in the sentence, that some action is taking place, and that a sword is mentioned nearby.
For now, though, these are only scattered notes. Attention helped collect them, but not properly think them through. This is where the Feed-Forward Network, or FFN, enters.
Imagine a bench where the token sits after the large meeting. It takes a heavy backpack off its shoulders. Inside are all the notes it just received from the other tokens.
The token opens the backpack and spreads the notes across a large field. Now it can connect them and explore different possibilities:
“If Manas appears here and an action is nearby, perhaps Manas is the one performing it.”
“If a sword appears after the action, perhaps Manas raised, took, or used it.”
Of course, the model does not phrase these thoughts in human language. It still works only with numbers. But the general idea is the same: the FFN mixes the information it has already collected in different ways and looks for useful combinations.
To give the token enough room to spread out all these possibilities, we temporarily provide much more space. The small backpack opens onto a large field, where many more combinations can be examined.
But another problem appears. Some possibilities may be useful; others will be complete nonsense. So another character sits down on the bench: the inner critic.
The critic lets a useful signal pass almost unchanged. It weakens a poor one. If it is unsure, it seems to say, “All right, that sounds suspicious, but let us not throw it away entirely just yet.”
After this check, the token gathers the remaining conclusions, packs them back into a backpack of the original size, and continues through the Transformer. That is the entire point of the Feed-Forward Network in our little scene.
Now let us translate the scene into numbers. The token's original backpack contains 384 numbers. The first Linear layer mixes them and expands them to 1,536 numbers. That is our large field.
The 1,536 numbers then pass through GELU. This is the inner critic: it lets some signals through almost unchanged, weakens others, and nearly zeros out some of them.
The second Linear layer gathers the result back into 384 numbers. The token arrived with a backpack of 384 numbers and must leave with a backpack of the same size. The difference is that the information inside has now been processed.
384 → Linear 1 → 1536 → GELU → Linear 2 → 384
The division of labor is simple. Attention sends the token to a meeting where it receives information from other tokens. The FFN seats it on a bench, lets it think through what it heard, and packs the conclusions back into the backpack.
14. Dropout: sometimes it helps to take a few notes away
Before the token carries its packed backpack onward, it meets one more character: Dropout.
During training, Dropout randomly pulls some finished notes out of the backpack and temporarily throws them away. In Tiny Manas, Dropout is 20%. This means that roughly 20% of the numbers become zeros on each step.
That sounds like sabotage. The model just did all that thinking, and now we throw away part of its conclusions. We do it deliberately so that the model cannot always depend on the same convenient features. One set of notes disappears today; another set disappears on the next step. The model has to find several ways to reach the right answer, which makes it less likely to memorize the training text.
In numerical terms, Dropout creates a random mask of zeros and ones and applies it to the contents of the backpack:
vector: [0.4, -0.2, 0.7, 0.1]
mask: [1, 0, 1, 0]
result: [0.4, 0, 0.7, 0]
The zeros mean those notes have temporarily disappeared. A new mask is created every time, so the model cannot know in advance which information will remain available. The surviving numbers are scaled up slightly so the overall signal does not become weaker.
When training ends and ordinary text generation begins, Dropout turns off. The model opens the entire backpack and uses every note it has learned.
The token then enters the next Transformer block, where the whole path repeats: the Attention meeting, the FFN bench, and a new Dropout mask.
15. LM Head: turning internal thoughts into candidates
Okay. Our model has passed through eight Transformer blocks, spoken with earlier tokens through Attention, sat on the FFN bench, and packed its conclusions back into a backpack of 384 numbers. But we have a problem: it still has not said a single word.
Those 384 numbers already contain information about the token itself, its position, and the available context. But they are still not text. We need a translator from the model's internal space back into the vocabulary.
The final Linear layer performs this job. It is called the Language Model Head, or LM Head. Imagine a contest in which all 32,768 vocabulary tokens are competing. The LM Head examines the final vector at the current position and gives every candidate one score:
Manas: 1.2
sword: 4.7
raised: -0.8
and: 2.1
These scores are called logits. The larger the logit, the more strongly the model currently favors that token. But 4.7 does not mean a probability of 47%. For now, these are only internal scores.
Mathematically, the LM Head performs one transformation:
384 numbers → 32,768 logits
In Tiny Manas, the LM Head shares its weights with the token embedding table. This technique is called weight tying. The same matrix helps translate token IDs into internal vectors at the beginning and score vocabulary tokens at the end. The model uses its parameters more efficiently this way.
At this point, the road splits. During training, the logits are compared with correct answers. During inference, they are used to choose the next token.
16. Training: how one error changes 26.9 million numbers
Let us return to the shifted correct answers:
input: Manas | raised | his
target: raised | his | sword
For every position, the LM Head produces 32,768 logits. Cross-Entropy looks at the probability the model assigned to the correct next token.
Think of Cross-Entropy as a strict teacher. It does not only care whether the model was wrong. It also cares how confidently it was wrong. If the model hesitated, the punishment is moderate. But if it gave the correct answer a probability of one percent, the teacher is fully entitled to say, “You were not only wrong. You were absolutely certain about that nonsense.”
Inside Cross-Entropy, Softmax turns the logits into probabilities. Then the probability of the correct token is selected and its negative logarithm is calculated:
probability of the correct token = 0.90 → loss ≈ 0.105
probability of the correct token = 0.10 → loss ≈ 2.303
probability of the correct token = 0.01 → loss ≈ 4.605
Cross-Entropy punishes both error and confidence in that error. Giving the correct answer 1% costs much more than giving it 10%. The errors from all positions in all sequences of the batch are averaged into one number called the loss. The smaller it is, the better the current predictions.
But if only one number remains at the end, how does the model know which of its millions of parameters is responsible? During the forward pass, our implementation retains a computational graph: a history of which operations and parameters affected each local error and, eventually, their average.
Then backpropagation begins. It walks backward through the history of calculations and determines how every parameter affected the final error.
A gradient answers the question: “If this particular parameter changed slightly, how would the total loss change?” Each parameter receives a small hint about the direction in which it should move. The model then updates its parameters using those hints.
The model receives a new batch, and the entire path repeats:
text → tokens → embeddings → Transformer blocks
→ LM Head → logits → Cross-Entropy → loss
→ backpropagation → parameter update
That is how almost-random parameters gradually become a system capable of continuing text.
17. Inference: the teacher goes home
After training, the correct answers are no longer available. The teacher really has gone home, and the model is on its own. It receives the beginning of a text, produces logits for the final position, and must choose its continuation.
Softmax turns the logits into probabilities:
sword: 70%
head: 15%
voice: 8%
hand: 4%
all others: 3%
We could always take the most likely token, but the text would quickly become monotonous. Instead, the next token is usually sampled at random while still respecting the probability distribution.
Language Models have many generation settings that influence how the next token is selected. To begin, we only need the two most basic ones: temperature and top-k.
Temperature controls how sharp the distribution becomes. A low temperature makes the leading choices even more likely: the text becomes cautious but tends to repeat itself. A high temperature gives rare choices more of a chance: variety increases, and so do mistakes.
Top-k keeps only the k most likely candidates before sampling and removes the long tail of nearly impossible options.
Suppose the model selects sword. We append it to the original line:
Manas raised his sword
The updated sequence passes through the model again, and it predicts one more token. Then another. Then another. Text generation is one operation repeated over and over: predict a token, append it, predict again. Once the text grows beyond 256 tokens, Tiny Manas continues to see only the latest 256.
Part II. How I Built and Trained Tiny Manas
Until now, we have been studying a Language Model in theory: taking each piece on its own, asking what problem it solves, and gradually assembling the whole chain in our heads. The theory ends here. What follows is my actual experiment: the model I wrote, how I verified that it genuinely learned, the versions I trained on Manas, the hypotheses that failed, and what I ultimately managed to run on a single Mac.
1. The Tiny Manas architecture
Before building a house, you need a blueprint. A model is no different. We now know which parts belong in a Transformer, but we still need to give them real dimensions and connect them into one working system.
Tiny Manas is a decoder-only Transformer. Its only job is to look at the tokens already written and predict the next one. It does not need a separate encoder: we are not translating one text into another or giving the model a second source of information. For the same reason, there is no cross-attention. There is one sequence and Causal Multi-Head Self-Attention that works only with that sequence and never looks into the future.
The first 26.9M version used this blueprint:
- a vocabulary of 32,768 tokens;
- a context window of 256 tokens;
- an internal representation of 384 numbers for every token;
- 8 Transformer blocks;
- 8 attention heads with 48 numbers each;
- an FFN that expands
384 → 1536 → 384with GELU; - LayerNorm before Attention and the FFN;
- residual connections around Attention and the FFN;
- Dropout 0.2;
- learned absolute position embeddings;
- shared weights between the token embedding table and LM Head;
- 26,877,696 trainable parameters.
Now let us walk through this blueprint technically. The input is a matrix of IDs with shape (B, T), where B is the number of independent fragments in one batch and T is the length of each fragment. The embedding table replaces every ID with a vector of 384 numbers. A position vector is added, producing X with shape (B, T, 384).
This X passes through eight identical blocks. In each attention head, the representation size is:
384 / 8 heads = 48 numbers per head
Each head computes its own Q, K, and V, builds attention scores as Q @ Kᵀ / √48, hides future positions with the causal mask, and mixes the available Value vectors. The outputs of all eight heads are concatenated back into 384 numbers. The FFN then expands each vector to 1,536 numbers, applies GELU, and compresses it back to 384.
After the eighth block, a final LayerNorm brings the numbers into a stable range one more time. The LM Head then turns every position into 32,768 logits, one score for each token in the vocabulary:
token IDs (B, T)
token + position embeddings → (B, T, 384)
8 Transformer blocks → (B, T, 384)
LM Head → (B, T, 32,768)
During training, the logits are flattened into a matrix of shape (B × T, 32,768), while the correct answers become a list of shape (B × T). Cross-Entropy can then evaluate every position in the batch in one operation, even though the final error is still averaged into a single number.
I used the clarity of nanoGPT as a tested reference, but wrote the compact implementation myself. I did not want to merely run someone else's finished code. I wanted to understand where every tensor shape came from, what each parameter stored, and why the data moved through the model in exactly this way.
2. First, the model had to prove that it could learn at all
The architecture was written. Could I send it straight into full training? Technically, yes. In practice, that would be like taking a long road trip in a car whose engine I had just assembled in the garage without ever starting it. Half an hour later something would begin to smoke, and I would have no idea where the mistake was.
The targets might be shifted incorrectly. Evaluation might read a different batch. A saved checkpoint might fail to load. Computation on the Mac might quietly move from MPS to the CPU. Loss could even decrease and create a pleasant illusion that everything was working.
So the first check was to intentionally overfit one fixed batch. Sounds strange, right? In Part I, I described overfitting as a problem, and now I was deliberately causing it. But the purpose here was different. We give the model the same small answer sheet over and over. If it cannot memorize even that, something in the chain between input, loss, and parameter updates is broken.
I disabled Dropout for this check. Randomly pulling notes out of the backpack makes no sense when we are testing whether the model can memorize one example exactly. The result was:
loss: 10.4385 → 0.02673
perplexity: ~34,150 → 1.027
top-1: 100%
top-5: 100%
time: 1.89 seconds
Three technical metrics appear here. Top-1 measures how often the correct token ranked first among all 32,768 candidates. Top-5 checks whether it appeared in the first five. Perplexity is calculated as exp(loss) and, very roughly, indicates how many plausible continuations the model seems to be choosing between. The closer it is to one, the more certain the model is. For a deliberately memorized batch, perplexity 1.027 and 100% accuracy mean exactly what we wanted: the engine started.
The check paid for itself immediately. I found a bug not in training, but in measurement. The optimizer trained on one “fixed” batch, while evaluation created another because the random number generator was in a different state. The model honestly memorized one answer sheet, while the examiner quietly handed it another.
I created the batch once and passed the same object into both training and evaluation. After the rerun, the model reached 100%. A useful lesson: a beautiful metric is worthless until you have verified exactly which data went into it.
3. A small pilot: rehearsing the full training loop
One batch proved that the main pieces were connected. But memorizing one sheet and learning from a full text are not the same thing. I needed a dress rehearsal: small enough to expose a problem quickly, but complete enough to include real training and validation splits.
I used 10,000 tokens. The first 9,000 were used for training and the final 1,000 for validation. The model in this run had 8.1 million parameters.
The best validation result appeared around step 200. After that, training loss continued its beautiful decline while validation loss turned around and began climbing. At step 1,000, the result was:
training loss: 0.1235
validation loss: 10.4687
Imagine a student who memorized every comma in a textbook but falls apart when asked the same question in different words. On the familiar 9,000 tokens, the model became nearly perfect. On the held-out thousand, it only got worse.
This time overfitting was a real problem again. The run was not supposed to produce beautiful prose. It was supposed to verify the loop training → validation → selecting the best state. It also showed exactly why training loss alone cannot judge a model: training can continue, the number can keep falling, and the ability to handle unfamiliar text can already be deteriorating.
4. The first full model: establishing a baseline
After those two checks, I could move to the full corpus. But I still had nothing against which to measure future improvements. “The generation feels a little more coherent” sounds nice, but it is not enough for an experiment. I needed a baseline that every later hypothesis would have to beat on the same data.
The first full model had:
parameters: 13,193,216
Transformer blocks: 6
embedding size: 256
attention heads: 8
context: 256 tokens
training steps: 3,000
positions processed: 12,288,000
time on M5: 15.1 minutes
By a “processed position,” I mean one question of the form “which token comes next?” The model answers many such questions in each batch. Across 3,000 steps, they added up to more than twelve million.
After training, I did not evaluate the model on the same text that changed its parameters. We had reserved validation and test splits for this. Validation helps compare decisions during the experiment. The test split is the final independent exam, one that should not be used to tune every new idea.
validation loss: 4.6384
validation perplexity: 103.38
validation top-1: 26.64%
validation top-5: 45.24%
test loss: 5.0201
test perplexity: 151.42
test top-1: 23.81%
test top-5: 41.09%
These numbers described a real working Language Model. On the test split, the correct next token ranked first almost one time in four and appeared in the top five roughly two times in five. For a small model trained from scratch on a single epic, this was a reasonable starting point.
The output had also stopped sounding like random Kyrgyz fragments. The model picked up names, forms of address, rhythm, and common patterns from Manas. But it lost the thread quickly, and repetition remained obvious. Good. I now had not only a working model but also a concrete problem for the next experiment.
5. The hypothesis that failed: more context
If the model forgets what happened earlier, the first idea seems obvious: give it more memory. The baseline context was 256 tokens. I increased it to 512 while leaving the rest of the architecture almost unchanged.
More memory means a smarter model. Beautiful. Logical. And, as it turned out, wrong.
First, we need to separate two ideas. The context window determines how many previous tokens the model can see. It does not guarantee that a small model can use everything it sees well. You can put twice as many documents on a detective's desk, but that does not make the detective twice as smart. More likely, they will spend more time sorting through the papers.
That is exactly what happens inside Attention. With context 256, each head builds a comparison table of size 256 × 256, or 65,536 cells. With context 512, the table becomes 512 × 512, or 262,144 cells. The sequence became twice as long, while the number of pairwise comparisons grew by roughly four times.
The parameter count barely changed: 13,193,216 became 13,258,752. Exactly 65,536 numbers were added, all new rows in the position embedding table for positions 256 through 511. The model had much more computation to perform, but almost no additional ability to reason about the text.
The run took 17.2 minutes, throughput fell by 12.3%, and quality did not improve:
validation perplexity: 112.94 (was 103.38)
test perplexity: 171.85 (was 151.42)
The longer window gave the model a harder problem without giving it enough new capacity to solve it. Under the same compute budget, it did less useful work and generalized worse. I did not keep defending a beautiful hypothesis after the data disagreed. I returned to a context of 256 tokens.
6. What actually helped: making the model larger
All right, more history did not help. So I changed the question: perhaps the model could already see enough text, but lacked the internal capacity to process it properly.
To continue the detective analogy, the previous experiment simply placed more documents on the same desk. This time I kept the old number of documents but expanded the team and gave it more room to work. The context stayed at 256, while each token representation became wider, the Transformer became deeper, and the FFN received more space in which to process features.
The final Tiny Manas looked like this:
parameters: 26,877,696
embedding size: 384
Transformer blocks: 8
attention heads: 8 × 48 numbers
FFN: 384 → 1536 → 384
context: 256 tokens
Dropout: 0.2
best checkpoint: step 2900
Training took 24.4 minutes. Every operation ran through PyTorch MPS on the Apple GPU. I disabled automatic CPU fallback: otherwise an unsupported operation might have quietly run more slowly on the processor and made the timing comparison dishonest.
The saved PyTorch samples reached about 959 MB of live MPS allocation. The driver reported roughly 4.33 GB in use. These are sampled readings, not an exact measurement of whole-process peak memory. The actual run fit within the MacBook Pro's 16 GB of unified memory.
The results improved on both held-out splits:
validation loss: 4.3457
validation perplexity: 77.15
validation top-1: 31.55%
validation top-5: 49.83%
test loss: 4.7575
test perplexity: 116.45
test top-1: 27.74%
test top-5: 45.25%
Compared with the first full model, perplexity fell by 25.4% on validation and 23.1% on test. Top-1 and top-5 also rose. This was not one lucky metric moving in isolation: the larger model genuinely ranked the correct continuation higher more often.
The improvement was not free, of course. Throughput fell from 13,565 to 8,378 processed positions per second. But this time the extra time bought measurable quality. The context experiment made the model slower and worse. Increasing width and depth made it slower but better. That is a useful engineering trade.
7. What the model learned, and what it did not
The metrics improved. But a reader can fairly ask: “What kind of text does it actually write?”
The final Tiny Manas sometimes produces passages that are immediately recognizable as imitations of the epic:
Манас! — Деп, ошонтүп, Арслан Манас кабылан
Көзүнүн жашы төгүлүп...
It learned names, forms of address, poetic formulas, line structure, and some local chains of action. Several lines can remain centered on one character or event. This is clearly better than random Kyrgyz fragments.
This is also where it becomes very easy to fall in love with a beautiful rhythm and declare victory. But sounding similar and understanding what is happening are not the same thing. The model can produce a plausible line with an impossible meaning, repeat one name over and over, invent a broken ending, or forget the reason for a battle a few sentences later.
There is another trap: show the single best sample out of a hundred and pretend the model always writes that way. Instead, I saved twenty generations with identical settings for every accepted version. I then measured the share of repeated trigrams, sequences of three words that the model used more than once.
first full model: 4.77% repeated trigrams on average
final Tiny Manas: 4.21% repeated trigrams on average
Repetition decreased on average, but variation between individual samples remained large. In one final output, more than a quarter of all trigrams were repeated. The problem improved, but it did not disappear.
I also checked whether the model copied long passages from training. The first matcher reported a maximum of seven words for the 26.9M model. I later found that it missed matches across punctuation and line breaks. Normalizing both sides in the same way raised the maximum to nine words in those same twenty saved outputs. The measurement correction report records the change. This is a matching word sequence with case, punctuation, and spacing ignored, not byte-for-byte copying. Such a small sample cannot prove that the model never memorized a longer passage elsewhere.
8. What this experiment does not prove
After a successful result, it is especially important to stop before assigning the model abilities we never tested.
The training, validation, and test splits all come from one edition and one performer. I split them chronologically, so the model was genuinely evaluated on later passages that never changed its weights. But it is still one long text performed by Sayakbai Karalaev. The experiment does not show that Tiny Manas can continue another narrator's version equally well.
The tokenizer had already been trained on a broader Kyrgyz corpus. Tiny Manas's neural parameters began with random values, but the way Kyrgyz text was divided into tokens was already useful. The result therefore cannot be attributed to the Transformer alone while completely separating out the tokenizer's contribution.
The expensive full runs used one fixed random seed. Final metrics were measured on 204,800 randomly selected positions in each held-out split rather than on every possible window. That is enough for an engineering comparison within this project. It is not enough to claim a new general benchmark.
Most importantly, Tiny Manas is not a general Kyrgyz model, an instruction-following model, or a chatbot. It knows one narrow world. If you ask about weather, programming, or modern life, its training data provides no basis for a good answer.
A stronger next test would use legally available text from another narrator or edition, held out entirely at the document level. That would tell us whether the model learned broader patterns of the epic or mainly adapted to the formulas of one performance. Simply training longer on the same text would be a weak idea. It would probably deepen memorization instead.
9. After the first run: what was worth changing?
The runs above gave me a working 26.9M model. The next question was whether the same Mac could train it more cheaply and produce better predictions without endlessly increasing its size.
I separated changes into two groups. Some should do the same work faster, such as avoiding a result that gets discarded immediately. Others change the model itself and may change its quality. The first group needs matching predictions and timing measurements. For the second, a fast forward pass is not enough: the model needs training and comparison on held-out text.
Consider the LM Head. During generation, we need the next token after the final position. Previously, the model still projected every position into 32,768 logits. A 256-token window produced 256 sets of answers when only the last was used. I kept every position inside the Transformer, where it supplies context, but applied the final projection only to the last position. The output tensor shrank from 32 MiB to 128 KiB. On that particular input, forward time fell from 7.307 to 5.894 ms while the checked predictions stayed the same.
Another useful change was BF16, a lower-precision number format used for part of the training computation. In two fresh full runs with the same initial weights and training windows, it reduced training-loop time by 20.58%. Validation loss changed by only 0.0000464. Parameters and optimizer state remained in FP32, the usual 32-bit format; I did not convert the entire training state to smaller numbers.
The largest prediction improvement came from changing how the model represents positions. Instead of a separately learned table for places in a sequence, I tried RoPE, rotary position embeddings. It rotates pairs of Query and Key coordinates according to their positions. Comparing Q and K then incorporates information about the tokens' relative distance. Value is not rotated.
I initially stopped this idea because each training step cost more time. That was premature: I had measured the price without checking the benefit. In the reopened experiment, a modest slowdown was allowed if validation showed an improvement. RoPE passed the intermediate checks and completed all 3,000 updates. On the same additional evaluation, loss fell from 4.34578 to 4.11584 and perplexity from 77.15 to 61.30. The post-selection test measured perplexity 92.88. Those numbers belong to the new version; they do not replace the historical results above.
The other ideas stayed in the record too:
| Change | What the experiment showed | Decision |
|---|---|---|
| KV cache: retain computed Keys and Values | Short generation became about 1.16 times faster; the benefit nearly vanished around the context limit | Use it, rebuilding the window after a crop |
| torch.compile: compile the computation | The first attempt broke gradient agreement; the corrected version was 3.02% slower | Keep ordinary eager execution |
| Activation checkpointing: recompute intermediate states instead of retaining them | 38.46% less sampled live memory, but 30.63% longer updates | Keep it as an option, disabled by default |
| A 16k or 8k vocabulary instead of 32k | Text throughput improved, but validation bits per byte worsened relative to the accepted model | Keep 32k; compare the same text, not counts of differently sized tokens |
| RMSNorm instead of LayerNorm | No material advantage appeared within 900 updates | Stop the pilot; full-budget performance remains unknown |
| SwiGLU instead of GELU in the FFN | A full 3,000 updates produced worse validation and more repetition | Keep GELU |
| GQA: share Keys and Values across several heads | The cache became four times smaller, but loss rose substantially after matched short adaptation | Keep eight independent KV heads |
| Specialized output-loss computation | The current workload fits in memory; no measured bottleneck justified this path | Do not spend another experiment without a reason |
The experiment records contain the configurations, limitations, and negative outcomes. RoPE left the model with 26,779,392 parameters because the learned position table was no longer needed. A better score did not repair the plot, though. Across twenty continuations, repeated trigrams averaged about 4.13%; malformed words and abrupt changes of character remained. I accepted the prediction improvement without declaring coherent storytelling solved.
10. New books: an improvement I did not promote
After changing the model's internals, an uncomfortable limit remained: it had learned from one edition of the epic. Rearranging the engine would not put new stories into its training data. The next experiment therefore began with books rather than another Transformer layer.
I prepared texts attributed to Sagymbay Orozbakov and Jusup Mamay from the Bizdin collection. Permission to use them for this research was confirmed. The complete books and extracted corpus are not redistributed on GitHub.
The PDFs needed work. Ordinary extraction sometimes split words into individual letters and pulled scholarly commentary into the verse. I selected the epic's page ranges, reconstructed words from their coordinates, and removed small footnotes and line numbers. Verse line breaks were preserved. Recognition errors did not disappear completely; several more problematic scans were excluded from this run.
Three Orozbakov volumes added 343,402 training tokens to the original 418,562. A separate fourth volume, containing 99,630 tokens, became validation. The Mamay text, with 408,359 tokens, was reserved for the final test. These assignments were fixed before inspecting model scores. I also checked exact tokenizer round trips and long matching word sequences between training and held-out books.
The comparison could then begin. Two models received the same RoPE architecture, identical initial weights, and 3,000 updates each: 12.288 million training positions. The first saw only the original text. For the second, half the training windows came from that text and half from the additional volumes. Windows never crossed book boundaries.
I added a safeguard. Winning on the new book would not be enough if the model became noticeably worse at continuing its familiar source. Alongside the new evaluation, I retained the original validation text and set the permitted loss increase beforehand: no more than 0.02.
| Model | Loss on the held-out Orozbakov volume | Loss on the familiar source | |---|---:|---:| | Deployed RoPE | 11.847 | 4.088 | | Expanded-corpus model | 4.208 | 4.134 |
The new-book difference is enormous. Familiar-source loss, however, increased by 0.0457, beyond the permitted 0.02. I did not promote the new weights. That describes this mixture and budget; it does not prove that expanding the data is pointless.
Why was the first-column gap so large? Line breaks explained part of it. The original source had been flattened into continuous text, while the new books retained verse lines. A diagnostic on the same target positions found that about 47% of the aggregate loss reduction occurred directly on newline-bearing tokens. Prediction improved on the other tokens too. Their contexts also contain line breaks, though, so the remaining benefit cannot be attributed entirely to new stories or another narrator.
This table uses a new fixed set of scored positions. Subtracting its 4.088 from the earlier 4.11584 would not establish another improvement: those are the same deployed weights evaluated differently. Within a comparison, the models need to answer the same questions.
Revisiting context 512
The expanded corpus did not pass the promotion requirement, so the next arm trained on the original text again. This time I tested a 512-token window with RoPE rather than the original learned position table. Parameter count and initial weights stayed the same. Four 512-token windows replaced eight 256-token windows in each microbatch, keeping the number of training positions per update unchanged.
The run passed intermediate checks at 900 and 1,200 updates but stopped at 1,500. Its final three loss differences from the control were +0.1123, -0.5095, and +0.4115. There was no consistent advantage of at least 0.02. I retained context 256 and left the remaining half of the budget unspent. This is a pilot stopped under a declared rule, not proof that a fully trained 512-token model would always be worse.
BF16 generation: a different workload from training
If BF16 accelerated training, why not use it to generate text too? I tested that separately on the deployed checkpoint without changing its weights. Here the model processes a short prompt and then produces one token at a time, so the training speedup need not carry over.
Prediction barely changed: validation loss increased by just 0.000024. Short generation nevertheless became 18.73% slower. Near the context limit, latency fell by only 3.14%. The persistent cache did shrink from 6 to 3 MiB, but total sampled live allocation did not show the required reduction. Training therefore stayed in BF16 and generation in FP32. The number format follows the measured workload, rather than being one choice made for the entire model.
This series ultimately left the deployed model unchanged. After closing selection, I evaluated it on the held-out Mamay text: loss was 12.371. That high value shows weak transfer to the other source and its formatting. It is not a test score for the expanded-corpus model, which I did not evaluate on the final test.
The O14–O17 report records the full protocol and outcomes. The paper sources provide the technical account, plots, and limitations.
11. Try Tiny Manas
After all the tables and caveats, you can finally try the result yourself. The interface below runs the selected model with roughly 26.8 million parameters, not another LLM pretending to be Tiny Manas. Before startup, the service verifies the checkpoint and tokenizer.
Prompts are limited to 300 characters and continuations to 128 tokens. Requests are rate-limited so one visitor cannot occupy the whole server. Temperature controls how adventurous the sampling becomes, while top-k limits the number of candidates at each step.
The model is tiny and was trained only on the epic. Broken words, repetition, and a lost plot are therefore not interface bugs. They are honest boundaries of the experiment.
Try Tiny Manas
Give the model a short Kyrgyz opening. It will continue one token at a time.
The continuation will appear here.
Experimental model, not an assistant. Four generations per visitor every ten minutes.
12. What I learned from Tiny Manas
I began this path wanting to understand what happens between an ordinary line of text and the next generated token. By the end, I had not only a working model but a complete chain of reasons.
Part I assembled that chain in theory. Bytes provide a universal representation of text. BPE shortens the sequence. Embeddings turn IDs into trainable vectors and add order. Attention collects information from context, the FFN processes it, residual connections preserve the foundation, and LayerNorm and Dropout help the model train steadily. The LM Head produces logits, Cross-Entropy measures error, backpropagation assigns responsibility, and the parameters gradually change.
Part II tested the whole mechanism in a real experiment. Overfitting one batch proved that the implementation could learn at all and helped uncover a measurement bug. The small pilot exposed the difference between memorization and generalization. The first full model established a baseline. Doubling context was slower and worse, although it sounded like the obvious improvement. Making the model itself larger, by contrast, reduced perplexity and increased accuracy on both held-out splits.
If you have made it this far, you now know that there is no single magical algorithm hiding inside the term “LLM.” There is a long chain of fairly understandable decisions. Each solves one small problem, and together they turn a sequence of numbers into a prediction of the next token. Magic sounds more exciting, of course. But I find the mechanics much more interesting.
Tiny Manas did not become a smart general-purpose LLM, and that was never the goal. It became my first complete achievement: a model whose major parts I can open, explain, trace through their tensor shapes, train myself, and evaluate honestly all the way to the edge of its abilities. That is why I began digging deeper in the first place.