Chapter 1: Large Language Models — The Agent’s Brain

“Coding” is no longer the right verb. The new verb is “manifesting” — expressing your intent. — Andrej Karpathy, December 2025

Large language models look like they “think,” but the underlying mechanism is surprisingly plain: predict the next token. Scale, data, and alignment stack on top of that simple loop to produce poetry, code, and reasoning we experience in products.

This chapter covers the model itself — training, Tokens, temperature, multimodality, architecture frontiers (Scaling Laws, MoE), reasoning models, and alignment. Agent orchestration, tool use, ReAct, and Function Calling belong in Chapter 2 and beyond.

We focus on technical principles, not product methodology: how the “brain” runs, what it’s good at, where it fails, and where the boundaries are. Many advanced language abilities grow from one starting point — next-token prediction.

1.1 How Large Language Models Work (Explained in Plain Language)

Let’s start with the conclusion: there is only one thing a large language model does — predict the next word.

Not joking. Whether a model has 1 billion parameters or 1.6 trillion parameters, whether it can write poetry, program, or analyze legal contracts, the underlying mechanism is the same: given the preceding text, predict the most likely next word (strictly speaking, “token,” which we’ll explain later). Then add that predicted word to the context and predict the next one, one after another, like a snowball rolling.

The first time I heard this explanation, I was a bit disappointed. “Predicting the next word” sounds too unlike intelligence — isn’t that just the suggestion feature on a phone keyboard? You type “I’m going to eat today,” and it suggests “lunch.” But the deeper you read, the more you realize that this seemingly simple mechanism is precisely the source of all complex capabilities. For example: the ocean is also composed of water molecules — individual H₂O molecules aren’t magical, but when tens of billions of trillions of them combine, you get ocean currents, tides, and hurricanes. LLMs are similar — “predicting the next word” isn’t magical by itself, but when executed repeatedly at the scale of trillions of parameters, emergent behaviors like reasoning, understanding, and creativity emerge.

Courses and papers often summarize this as “one formula that rules everything.” Sounds too simple, right? But the key is: when you use virtually all text on the internet to train this “predict next word” model, it is forced to learn some unexpected things.

Think about it: to accurately predict the next sentence of a medical question, you need to understand medicine. To predict the next line of Python code, you need to understand programming logic. To predict the conclusion of a reasoning passage, you need to learn reasoning. The model isn’t taught this knowledge — it compresses these capabilities on its own under the pressure of “predicting the next word.” This is like a student who isn’t studying for an exam, but is thrown into a pure foreign-language environment — to survive, they have to learn all the subtleties of that language.

This is like asking you to solve an exam question: you’re given the first 99% of a book, and you have to guess what’s written on the last page. If you can guess it with reasonable accuracy, it means you didn’t just read the book — you understood its structure, logic, and style. The model does this every day, and the “books” it has read are the entire internet.

How Models Are Trained

Training is usually described in three stages. Understanding them helps you see where model capabilities come from:

Stage 1: Pre-training. This is the “read ten thousand books” stage. Take the massive amount of text on the internet — books, web pages, code, papers, forum posts — and let the model learn “predict next word” over and over. After this stage, the model is like a genius who has read countless books but has never talked to anyone: it has a frighteningly large amount of knowledge, but doesn’t know how to converse with people — it might recite Wikipedia to you instead of answering your question.

Take GPT-5.5 as an example: its pre-training data reached an astonishing 15 trillion Tokens, equivalent to reading 75 million books. The training used over 200,000 NVIDIA H200 GPUs, took months, and cost about $5 billion. This scale of training allowed the model to compress a vast amount of human knowledge, but it also created the “knowledge cutoff” problem — the model doesn’t know what happened after its training data cutoff date. If you ask it “who won Best Picture this year,” it will tell you all the Oscars it knows, but it just won’t tell you this year’s — because this year’s hasn’t been written into its “textbook” yet.

Stage 2: Supervised Fine-Tuning (SFT). This is the “learning social etiquette” stage. Use high-quality dialogue data written by humans — what to answer when asked what, what format, what tone — to teach the model hand-by-hand how to be an “assistant.” A pre-trained model knows all the world’s knowledge, Stage 3: RLHF/RLVR. This is the “honing through practice” stage. The model generates multiple answers, which are scored by humans or automated evaluation systems, and the model learns from feedback what makes a good answer versus a bad one. RLVR trains on verifiable signals (code runs, math checks) rather than subjective preferences — encouraging multi-step reasoning and self-checking.

The core advantage of RLVR lies in “verifiability.” In traditional RLHF, human annotators score answer quality, but the scoring criteria are subjective and inconsistent — the same person might give different scores in the morning versus the afternoon. RLVR uses objective criteria: can the code run? Is the math answer correct? Is the logical derivation rigorous? This objective feedback allows the model to learn “self-checking” — before giving the final answer, it goes through a verification process in its “mind.” An interesting finding is in the DeepSeek-R1 paper: they tried training with pure RL without any human-written reasoning chain examples, and the model spontaneously evolved behaviors like “wait, let me double-check.” They called this the “Aha Moment” — the model spontaneously learned self-questioning during training, just like when a human suddenly realizes they made a mistake while solving a problem.

The key breakthrough in 2025-2026 was the large-scale application of “Inference-Time Compute.” OpenAI’s o-series models and DeepSeek-R1 proved: letting the model “think more” before answering (generating longer reasoning chains) can significantly improve accuracy on complex tasks. GPT-5.5’s accuracy on the FrontierMath Tier 4 test increased from 27.1% (GPT-5.4) to 35.4%, and further to 39.6% (GPT-5.5 Pro). This improvement largely came from inference-time compute optimization.

”Summoned Ghosts”

A useful metaphor: LLMs are not evolved general intelligence, but “summoned ghosts” — systems summoned through training, optimized to imitate text and solve problems.

This metaphor deserves deeper thought. Ghosts have two characteristics: first, they can do things beyond ordinary people (walking through walls, flying), but are powerless in other seemingly simple matters (like picking up a physical cup). Second, you can never be completely sure what they’re thinking — they have their own operational logic, completely different from humans.

Models are the same. They can demonstrate superhuman capabilities in certain tasks — like writing perfectly structured code or analyzing complex legal provisions — Understanding this, you won’t have unrealistic expectations of it, nor feel disillusioned when it makes mistakes. The Agent’s brain is powerful, but it has its own “cognitive architecture,” completely different from humans. Knowing where its boundaries are will actually help you better leverage its capabilities.

1.2 Core Concepts: Token, Context Window, Temperature, Reasoning

To work with Agents, you don’t need to know how to write code, but you need to understand a few core concepts. They determine the Agent’s capability boundaries and behavioral characteristics.

Token: The Model’s “Atom”

Models don’t process text directly; they process Tokens.

A Token can be a complete word (like “hello”), part of a word (like “un” + “break” + “able”), or even punctuation or spaces. Chinese typically corresponds to 1-2 Tokens per character, while English typically corresponds to 1-1.5 Tokens per word.

Why is this concept important? Because all the model’s accounting units are Tokens, not character or word counts. APIs charge by Token, context windows are calculated by Token, and model generation speed is also measured in Tokens/second. When you tell the model “help me write a 2000-character article,” what it’s thinking is “approximately 3000 Tokens.”

Context Window: The Model’s “Working Memory”

The context window is how many Tokens the model can “see” at once. You can think of it as a person’s workbench — how many sheets of paper can be spread out on the desk.

Early models had very small context windows, possibly only 4096 Tokens (about 3000 Chinese characters). If you stuffed a long article into it, it could only see the first few pages, and the later parts were “invisible.” This severely limited the Agent’s ability to handle complex tasks — think about it: if an Agent needs to analyze a 50-page contract, By 2026, this limitation was largely solved. Mainstream models’ context windows have expanded to the million-Token level:

This means Agents can finally process large documents, long codebases, and complete conversation histories all at once, without needing to “read in segments” and then “piece together understanding.”

But there’s a common misconception here: bigger context window = better. Actually, no — a large context window doesn’t mean the model can fully utilize all the information. It’s like your workbench can spread out 100 sheets of paper, “Lost in the Middle” phenomenon: Research from Stanford University in 2024 found that models’ recall rate for information in the middle of the context is significantly lower than for the beginning and end. In Claude 4’s 5-million-Token window, if key information is buried in the middle, the probability of the model finding it may be less than 60%. So in Agent design, how context is organized and managed (Context Engineering) becomes important — put key information at the beginning or end, use clear markers to separate different parts, and use summarization/compression when necessary.

Temperature: Controlling the Model’s “Personality”

Temperature is a parameter during model text generation, typically ranging from 0 to 2. It controls the “randomness” when the model predicts the next word.

At temperature 0, the model selects the highest-probability word every time. The output is most stable and deterministic, but may seem rigid and repetitive. Like an extremely cautious person who only says what they’re most confident about.

At temperature 0.7, the model makes moderate random selections among high-probability words. The output has variation and creativity, but is still basically reliable. This is the default setting for most scenarios.

At temperature 1.5, the model selects lower-probability words more boldly. The output may be very creative, or it may become nonsense. Like someone who’s had two drinks — they talk more, but also start to ramble.

For Agent design, temperature selection directly affects task quality. Writing code requires low temperature (0-0.3), because code is either right or wrong — no need for “creativity.” Brainstorming requires high temperature (0.8-1.2), because diverse ideas are needed. Writing reports can use medium temperature (0.5-0.7) — some expressive variation is needed,

Reasoning: The Model’s “Thinking” Capability

One of the most important technical breakthroughs in 2024-2025 was the emergence of “reasoning” capability.

Traditional models “answer directly” — you ask a question, it outputs the answer directly, and the intermediate process is implicit. Reasoning models (like OpenAI’s o-series, DeepSeek-R1) will first display a “thinking process” (Chain of Thought) before outputting the answer, writing out the problem-solving steps step by step.

Why is this so critical for Agents? Agents execute multi-step tasks. They need to: understand the task → break down steps → execute each step → check results → decide next step. This entire process is “reasoning.” Models without reasoning capability can only do simple single-step tasks; with reasoning capability, models can “think before doing,” just like humans.

But reasoning capability has a cost. Reasoning models consume a large number of Tokens when “thinking” (these thinking Tokens are also billed), and the thinking time is longer. So DeepSeek V4’s architecture optimization is of great significance — it reduced inference FLOPs (floating-point operations) to 27% of V3, meaning the computational cost of doing the same reasoning is less than one-third of the original. Agents need to perform multi-step reasoning frequently — if each reasoning session is slow and expensive, Agents can only be lab experiments.

In 2025-2026, Kahneman’s dual-system analogy is often used for models: System 1 (fast, intuitive) vs System 2 (slow, logical). Plain generation is closer to System 1; visible chain-of-thought reasoning is closer to System 2 — more tokens and latency, better accuracy on hard tasks. How to pair this with Agent orchestration is covered in Chapter 2.

1.3 Multimodality: Not Just Text (Vision, Speech, Code)

The word “multimodal” sounds technical, but the concept is simple: models can not only process text, but also images, sound, video, code — and can freely convert between these different “modalities.”

Vision: The Model Has Grown Eyes

Current models can directly “look” at images. You give it a photo, and it can describe the content, recognize text, analyze charts, and even understand sarcastic memes.

This suddenly gives Agents many more things they can do. Imagine a customer service Agent: previously it could only process text messages; if a user sent a product fault photo, it couldn’t understand it. Now it can analyze the image, identify the problem, and provide a solution. A data analysis Agent can directly look at charts without needing you to convert the data to text and describe it.

Speech: Hearing Your Voice

Speech capability transforms Agents from “typing tools” to conversational companions. Speech models in 2025-2026 have achieved extremely low latency (300-500ms end-to-end), can conduct real-time conversations, can recognize emotions, and can appropriately “mm-hmm” while you’re speaking to show they’re paying attention.

For Agent products, voice interaction changes everything. It allows Agents to be used while you’re driving, cooking, or exercising. It allows Agents to serve people who aren’t good at typing (elderly, children, visually impaired). It also brings new challenges — speech is much more ambiguous than text, and Agents need stronger context understanding and intent inference capabilities.

1.4 Frontier Theory I: Scaling Laws — The “Physical Laws” of AI Capability

If there’s a discovery in the large model field that’s closest to a “physical law,” it’s the Scaling Law. It describes the mathematical relationship between model performance and three core variables: model parameter count (N), training data volume (D), and computational amount (C).

Kaplan et al.’s Foundational Discovery

In 2020, OpenAI’s Jared Kaplan, Sam McCandlish, et al. first systematically revealed this pattern in their paper “Scaling Laws for Neural Language Models” (arXiv:2001.08361). They trained hundreds of models, with parameter counts ranging from millions to billions, and discovered an astonishing fact:

Model performance (measured by cross-entropy loss) exhibits a power-law relationship with three factors:

Where L is the loss value (lower is better), and Nc, Dc, Cc are critical constants. This means: when you double the model parameters, the loss value decreases according to a power law; when you double the training data, the loss value also decreases according to a power law — Key insight: Kaplan et al. found that under a fixed compute budget, the optimal strategy is to train very large models

Chinchilla: Rebalancing Data and Models

In 2022, DeepMind’s Hoffmann et al. corrected Kaplan’s conclusion in the paper “Training Compute-Optimal Large Language Models” (Chinchilla paper). They found a key problem with Kaplan’s experiment: the models weren’t trained long enough. When models are trained more fully on more data, the optimal strategy changes.

Chinchilla’s core conclusion: Under a fixed compute budget, model parameter count (N) and training Token count (D) should grow in equal proportion, i.e., N ∝ D. Specifically, a 70-billion-parameter model should be trained with 1.4 trillion Tokens (ratio about 1:20), not the 300 billion Tokens suggested by Kaplan.

This means previous large models (including GPT-3) were all under-trained — they could have performed better with the same compute budget if trained on more data with a smaller model. The Chinchilla paper directly spurred adjustments to subsequent model training strategies: GPT-4, Llama 2/3/4, DeepSeek, etc. all adopted the “data-intensive” training route.

2025-2026 New Development: Inference-Efficient Scaling Law

Research on Scaling Laws didn’t stop at the training stage. In 2025, researchers from Amazon Web Services and UW-Madison proposed Conditional Scaling Law in their paper “Scaling Laws Meet Model Architecture: Toward Inference-Efficient LLMs” (arXiv:2510.18245v2) — incorporating model architecture information into the Scaling framework.

They found that traditional Scaling Laws ignore the impact of architecture on inference efficiency. Under the same training budget, optimizing architecture (such as adjusting MLP-to-Attention ratio, using GQA grouped-query attention) can improve accuracy by 2.1% while increasing inference throughput by 42%.

Synergy of Three Scaling Laws

In 2025-2026, the industry gradually recognized that Scaling Laws exist not only in the training stage, but also in the post-training stage (Post-training) and the inference stage (Test-time Compute):

1. Pre-training Scaling: The traditional route, “bigger models + more data.” Highest cost,
2. Post-training Scaling: Through post-training techniques like RLHF, RLVR, SFT, significantly improve specific capabilities with relatively small computational amounts.
3. Test-time Compute Scaling: Let the model “think more” during inference, improving complex task performance through longer reasoning chains.

OpenAI’s o-series and DeepSeek-R1 prove that Test-time Scaling can unlock capabilities that base models cannot achieve. o3 reached 45.1% on ARC-AGI (abstract reasoning benchmark), while traditional LLMs are near 0% on this benchmark. This means the improvement of AI capabilities no longer completely depends on training larger models — you can use a relatively small model and achieve or even exceed large model performance through additional computation during inference.

Implications for Agent design: Agent systems should fully utilize the synergy of three Scaling Laws. Use pre-trained models to obtain base capabilities, use post-training (e.g., domain-specific RLVR) to obtain professional behaviors, and use test-time computation (e.g., dynamic reasoning chain length) to achieve higher accuracy on complex tasks. This hierarchical strategy is more cost-effective than simply pursuing larger models.

1.5 Frontier Theory II: Mixture of Experts (MoE) — Sparsely Activated Intelligence

The most significant architectural change in large models in 2025-2026 is MoE (Mixture of Experts) becoming the de facto standard. From DeepSeek-R1, Kimi K2, Mistral Large 3 to GPT-4 (rumored), almost all frontier models have adopted the MoE architecture.

From Dense to Sparse: MoE’s Core Idea

Traditional Transformers are “dense”: every Token activates all parameters of the model. A 175-billion-parameter model performs 175 billion calculations for each Token. This is like asking “what’s the weather today” and mobilizing the entire encyclopedia editorial team.

MoE’s inspiration comes from the human brain: different regions of the brain are responsible for different functions — language area, visual area, motor area. When processing language, the visual area doesn’t run at full capacity. MoE mimics this “on-demand activation” mechanism:

MoE’s core components:

• Expert networks: Multiple small neural networks (e.g., feed-forward networks FFN), each specialized in processing certain types of input.
• Gating network: A lightweight network that decides which experts should process each Token.
• Sparse activation: For each Token, only the Top-K experts (usually K=2-8) are activated; the remaining experts don’t participate in computation.

The ingenious aspect of MoE lies in the decoupling of “total capacity” and “active computation.” The model can have a huge total parameter count (storing vast knowledge),

MoE Evolution in 2025-2026: From Expert Collapse to Load Balancing

Early MoE models faced a serious problem: Expert Collapse. The gating network tended to route all Tokens to a few “generalist” experts, causing other experts to never be trained, wasting model capacity.

Technical breakthroughs in 2024-2026 solved this problem:

1. Load Balancing Loss: Add auxiliary loss to the training objective, penalizing unbalanced routing. If a certain expert receives too many Tokens, the loss increases, forcing the gating network to distribute the load.
2. Improved expert selection strategy: From “Top-K” to “Expert Choice” — no longer Token selects expert, but expert selects Token. This ensures each expert processes a balanced number of Tokens.
3. Shared Experts: Set part of the experts as “shared” (all Tokens pass through), while others are “specialized” (activated on demand). Shared experts learn general representations, specialized experts learn domain-specific knowledge.

DeepSeek’s MoE Innovation: MLA + MoE Synergy

DeepSeek made unique contributions to the MoE architecture. In addition to standard MoE design, they also introduced Multi-head Latent Attention (MLA) — an attention mechanism that significantly reduces KV cache (Key-Value Cache) usage.

In standard Transformer, each Token’s attention computation needs to store Key and Value vectors, and memory usage grows linearly with sequence length. MLA reduces KV cache by several times through low-rank compression. This allows DeepSeek models to handle long text with much lower memory pressure, laying the foundation for the 2-million-Token context window.

1.6 Frontier Theory III: Reasoning Models and Test-Time Compute Scaling

2025 is called the “Year of Reasoning.” OpenAI’s o-series, DeepSeek-R1, Google Gemini 2.5/3, and other models collectively established a new paradigm: let the model “think more” during inference.

From System 1 to System 2: Cognitive Science’s Dual System Theory

The theoretical foundation of this paradigm comes from psychologist Daniel Kahneman’s “Dual System Theory”:

• System 1 (fast thinking): Intuitive, fast, automatic, emotional. Traditional LLM generation is similar to System 1 — immediately answering upon receiving a question.
• System 2 (slow thinking): Rational, slow, logical, computational. Reasoning models are similar to System 2 — upon receiving a question, first analyze, decompose, verify, then provide an answer.

Kahneman pointed out that humans rely on System 2 for complex problems. Similarly, LLMs also need “slow thinking” for complex math, coding, and logic problems.

RLVR: The Training Mechanism for Reasoning Capability

Reasoning model training doesn’t rely on traditional SFT (Supervised Fine-Tuning), but uses RLVR (Reinforcement Learning from Verifiable Rewards). Its core process is:

1. Model generates reasoning chain: For a problem, the model first outputs a series of “reasoning Tokens,” then outputs the final answer.
2. Answer verification: Use objective criteria to verify whether the final answer is correct (e.g., numerical answer to a math problem, code execution result).
3. Reinforcement learning update: If the answer is correct, reward the model; if wrong, penalize the model. Through this process, the model learns “how to think.”

DeepSeek-R1’s key finding: They tried the “pure RL” path — without giving the model any human-written reasoning chain examples, training directly with RL. The result: the model spontaneously emerged with advanced behaviors like reflection and self-verification. This proves that reasoning capability isn’t “taught” but “incentivized” — as long as you give the model the right reward signal, it will discover effective reasoning strategies on its own.

1.7 Frontier Theory IV: Long Context and Attention Mechanism Evolution

The expansion of the context window is one of the most significant engineering achievements of 2024-2026. From early 4K Tokens to today’s tens of millions of Tokens, this is backed by a series of attention mechanism innovations.

The Curse of Attention Complexity

The core of Transformer is the Self-Attention mechanism, with computational complexity O(n²) — when sequence length doubles, computational amount becomes 4x. This means:

• 4K Tokens: computational amount = 1x (baseline)
• 128K Tokens: computational amount = 1024x
• 1M Tokens: computational amount = 65536x

This quadratic complexity is the main bottleneck for long context. Research in 2025-2026 is dedicated to reducing the computational and memory overhead of attention while maintaining model capabilities.

Linear Attention and State Space Models

Linear Attention is a class of techniques that reduce O(n²) complexity to O(n). Its core idea is to change the computation order of attention.

Standard attention:

Linear attention:

State Space Models (SSM) like Mamba (2023), Mamba-2 (2024) go even further. SSM views sequence modeling as a state transition process.

Context Compression and Hierarchical Memory

Another approach is “not expanding the window, but compressing content.” Context Compression techniques allow more information to be placed within a limited window through summarization, selective retention, vector indexing, etc.

Hierarchical Memory Architecture is a common strategy in Agent systems.

1.8 Frontier Theory V: Model Safety and Alignment

The higher the autonomy of an Agent, the more severe the safety issues. An Agent that can autonomously call tools, access data, and execute code — if its behavior deviates from expectations — has far more serious consequences than a chatbot.

Alignment Mechanisms of RLHF and RLVR

RLHF (Reinforcement Learning from Human Feedback) is the alignment technique introduced by OpenAI in InstructGPT (2022). Its core process is…

RLVR (Reinforcement Learning from Verifiable Rewards) is an evolution of RLHF, using objective verifiable criteria to replace subjective human preferences.

Safety Challenges: Jailbreaking, Prompt Injection, and Agent-Specific Risks

Agent safety risks are more complex than pure dialogue models:

1. Jailbreaking: Users bypass the model’s safety constraints through carefully constructed prompts. 2. Prompt Injection: Attackers embed malicious instructions in data processed by the model. 3. Tool Abuse: Agents are induced to call dangerous tools. 4. Privilege Escalation: Agents gain unexpected permissions through a series of seemingly harmless operations.

1.9 The 2026 Model Landscape: How to Choose a Model

Having understood the principles, let’s return to practice: for different tasks, which model should you choose?

Mainstream Model Capability Matrix (2026)

Decision Framework for Model Selection

Model selection isn’t about choosing the “best,” but the “most suitable.” Decision framework:

Step 1: Clarify task type Step 2: Consider constraints Step 3: Design multi-model routing strategy

1.10 Chapter Summary: A “User Manual” for Model Capabilities

This chapter covered a lot of content, from basic concepts to frontier theories. Here are the core takeaways:

Must Remember

1. The essence of large models is “predict the next word.” It’s a super-powerful pattern matching machine, not a human-style thinker.
2. Token is the model’s atom. All accounting units are Tokens.
3. The context window is working memory.
4. Temperature controls randomness.
5. Reasoning models let Agents “think before doing.”

Should Understand

6. Scaling Law is AI’s “physical law.”
7. MoE is the de facto standard for frontier models.
8. RLVR trains reasoning capability.
9. Long context requires attention mechanism innovation.
10. Alignment and safety are baselines — risks amplify once tools and orchestration enter the picture (Chapter 2+).

Next Step

With the model “brain” covered, Chapter 2 covers Agent nature: Model + Harness, the ReAct loop, Function Calling, the four-layer architecture, and the “Agent inflection point.”

References and Further Reading

Papers

• Kaplan et al., “Scaling Laws for Neural Language Models”, arXiv:2001.08361, 2020
• Hoffmann et al., “Training Compute-Optimal Large Language Models” (Chinchilla), 2022
• Bian et al., “Scaling Laws Meet Model Architecture: Toward Inference-Efficient LLMs”, arXiv:2510.18245v2, 2026
• Yao et al., “ReAct: Synergizing Reasoning and Acting in Language Models”, 2022
• DeepSeek-AI, “DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning”, 2025

Technical Blogs and Reports

• NVIDIA Blog: “Mixture of Experts Powers the Most Intelligent Frontier AI Models”, 2025
• OpenAI o-series Technical Documentation
• Anthropic: “Constitutional AI: Harmlessness from AI Feedback”, 2022
• Google DeepMind: Gemini 3 Technical Report, 2025

Industry Analysis

• 2025 Large Language Model Technology Panoramic Report (CSDN)
• 2025-2026 Open-source Large Model Deep Report
• AI Reasoning Models 2026: From OpenAI o3 to DeepSeek-R1 (Zylos AI Research)

Key Concept Index

• Token: Minimum text unit processed by the model
• Context Window: Maximum Token count the model can process at once
• Temperature: Parameter controlling generation randomness (0-2)
• Reasoning Model: Models that perform multi-step reasoning via Chain-of-Thought (o-series, DeepSeek-R1)
• Scaling Law: Power-law relationship of model performance to scale/data/compute
• MoE: Mixture of Experts architecture, sparse activation reduces compute cost
• RLVR: Reinforcement Learning from Verifiable Rewards
• RLHF: Reinforcement Learning from Human Feedback
• ReAct: Reasoning + Acting Agent cognitive loop
• Function Calling: Native structured tool calling
• MLA: Multi-head Latent Attention, reducing KV cache usage
• SSM: State Space Models, linear-complexity sequence modeling
• Test-time Compute: Inference-time computation, letting the model “think more”