ByteDance is betting 100 trillion yuan (over $14 billion) on seed model development, rejecting distillation shortcuts to compete with OpenAI's foundation models.

Recently, The Information broke news that at a Seed all-hands meeting two weeks prior, founder Zhang Yiming made a rare public statement, explicitly rejecting the use of distillation techniques to catch up with cutting-edge large language models. His position was unambiguous: "ByteDance should be willing to sacrifice some short-term interests for long-term goals."

In the latest Artificial Analysis global LLM rankings, Moonshot's Kimi K3 Max ranks second, Alibaba's Qwen 3.8 Max ranks fourth, Zhipu's GLM 5.2 Max and DeepSeek V4 Flash rank seventh and eighth respectively—Chinese models now occupy nearly half of the top ten positions globally. By contrast, ByteDance's Seed 2.1 Pro ranks only 21st, significantly lower than other Chinese models.

ByteDance possesses formidable resources. In terms of capital, the company is among the internet giants by annual profit. In talent, the Seed team comprises top-tier algorithmic researchers grown through ByteDance's recommendation and advertising systems. In compute capacity, tens of thousands of GPU clusters across intelligent computing centers are being rolled out nationwide. In data, Douyin and TikTok sit atop the world's largest native content pool.

According to ByteDance employees, the company's internal assessment is that ByteDance's unwillingness to use distillation is a key reason why its large language models lag behind other Chinese AI labs.

Distillation, in simple terms, means training your own model using answers generated by other large models. It's an industry-recognized "shortcut." Yet according to public reports, ByteDance has experienced at least three internal disputes over whether to pursue distillation.

The first conflict occurred in January 2025. When DeepSeek-R1 emerged and its reasoning style swept the globe, researchers within Seed proposed following suit with distillation. Zhang Yiming rejected the idea. He wanted Seed to learn "how to think" like humans, not merely mimic R1's "stream of consciousness."

The second conflict emerged after NVIDIA's Blackwell chip deployment. As the compute gap widened and competitors gained access to NVIDIA's latest offerings, while ByteDance had to make do with H20s, distillation advocates pressed their case again. Yet Zhang Yiming said no once more, and ByteDance committed 200 billion yuan to building additional intelligent computing centers in Ulanqab and Huailai.

Then came the pressure from Kimi K3's success in reaching the ranks of globally top-tier models. Each release of a new open-source model domestically became a stress test. Internal anxiety peaked, but in Zhang Yiming's view, rankings are superficial—commercial sovereignty is real.

Zhang Yiming held firm each time. According to sources close to Seed, the internal prohibition on distilling from open-source models is enforced through technical means like API detection, with explicit bans on mixing GPT-generated data into training sets dating back to April 2023.

"ByteDance initially tried distillation too, but Zhang Yiming later reflected deeply and said we need to take the long road. For companies at Alibaba's, Tencent's, and ByteDance's scale, we can't keep living off distilling from others," one major domestic tech executive told AI Technology Review.

Distillation in large models can be compared to "copying the teacher's answer key." This path delivers results fast. Feeding in a few million high-quality instruction samples, a model's benchmark scores can jump dramatically within mere weeks, rapidly approaching the distilled model's level. In the early stages of the LLM race, virtually all players tried this approach.

But as the competition entered deeper waters, controversy erupted. Since 2026, Anthropic has repeatedly and publicly accused Minimax, Moonshot, DeepSeek, and Alibaba Tongyi Lab of distilling its models, directly attributing China's rapid rise in large models to "cutting corners."

Ironically, Anthropic itself has been exposed for multiple instances of distilling OpenAI's models and training on pirated book data.

What truly sustained Zhang Yiming's commitment to "not cutting corners" is a deeper technical judgment: at a stage where data peaks, compute is scarce, and global model competition intensifies, distillation can only keep you trailing others, and might even leave you completely behind in the next leap in intelligence.

From a technical perspective, ByteDance Seed's commitment to training from scratch rather than joining the "distillation faction" represents an interconnected technical evolution logic. Training from scratch, though painful, is the only way to break through the ceiling.

Distillation's fundamental nature is using outputs from others' models as "correct answers" to train your own. It appears efficient but harbors an irreversible fatal flaw: model collapse.

In July 2024, joint research from Oxford, Cambridge, and other institutions graced the cover of Nature, systematically proving for the first time that if models repeatedly train on AI-generated data, tail information in the original data distribution gradually disappears, ultimately causing irreversible capability degradation.

Like inbreeding, the first generation appears healthy and normal, but genetic diversity is rapidly eroding. After several generations, recessive defects concentrate and suddenly manifest.

Meta's ICLR 2025 research further confirmed that even just 0.1% synthetic data mixed into training data suffices to trigger model collapse; even lower proportions retain toxic effects. Moreover, the larger the model parameters, the more the collapse effect gets amplified at specific thresholds—simply stacking parameters cannot solve the problem.

Knowledge in the real world isn't uniformly distributed. Common questions and standard answers occupy only the "head" of the information distribution, while rare edge cases, counterintuitive questions, subtle linguistic nuances in dialects and slang, and niche domain expertise all lurk in the "long tail." These long-tail elements determine a model's ceiling for handling unknown problems and represent the true boundary of intelligence.

AI-generated "standard answers" inherently flatten these tails. Model outputs gravitate toward the highest probability responses, automatically filtering out niche, anomalous, and uncertain content, leaving only the most "correct" and most mediocre results. Repeatedly training on such data, models become ever more confident and ever more narrow-minded. Benchmark scores may still climb, but their "worldview" quietly collapses.

ByteDance's greatest confidence, precisely, is data. Douyin generates over 80 million short videos daily; Toutiao's daily recommended content numbers in the hundreds of millions; overseas TikTok covers over 150 countries and territories—this is the world's largest native human content pool. Cleaning and denoising PB-scale real video, text, comment, and interaction data is far harder than simply taking a few million clean GPT-generated instructions, but only native real data preserves the complete distribution of reality, including the strange, vibrant, and unpredictable tails.

Scaling Law's essence is pushing intelligence's ceiling higher with more high-quality real data. Synthetic data can only smooth within existing boundaries, never transcend them. ByteDance chose a path that defines its own ceiling.

Tokenization is the model's "eyes"—it decides how many basic units to split text into, each unit corresponding to a vector representation. Using someone else's vocabulary is accepting their language preferences and cognitive granularity.

For instance, Llama's vocabulary is English-optimized. Encountering Chinese idioms, internet slang, or danmaku abbreviations, it often splinters into four or five scattered characters. This not only drastically reduces inference efficiency and inflates costs, but more critically, semantic relationships are shattered—the model never truly grasped Chinese expression logic from the input layer onward.

ByteDance Seed's core battleground was never pure-text large models, but native multimodal capabilities. Seedance 2.0 already achieves native audio-visual synchronized video generation, supporting 60-second 2K-resolution videos and 8-language lip-sync alignment. This capability's prerequisite is the model achieving atom-level mapping of text, video frames, and audio signals at the foundation—every semantic unit must align with corresponding visual and acoustic units in the same vector space.

If you distill someone else's model, you inherit their encoding foundation. How they tokenize video, you must tokenize video the same way; how they align audio-text timing, you must align it identically. Post-hoc fine-tuning can adjust parameters and effects, but cannot alter the underlying token space and semantic framework.

Not distilling means ByteDance defines the rules from scratch: what granularity optimally tokenizes video frames, how to encode audio rhythm into language models, how to perform joint attention across multimodal signals—all foundational decisions remain in-house.

For ByteDance pursuing video generation and multimodal interaction, control over foundational architecture matters far more than short-term text leaderboard rankings.

Training models through distillation is essentially lightweight "microtinkering": standing on someone else's ready-made foundation making adjustments—a few hundred GPUs running for weeks show results. Fast effect, low cost, it's many vendors' chosen path for quick catch-up.

But ByteDance's full-training-from-scratch path operates by entirely different rules. Training a 200B+ parameter MoE model from zero compares not single algorithmic points but the hard engineering muscle of an entire system. This is genuine AI "heavy industry."

In ultra-large-scale pretraining, the most critical metric is MTBF—mean time between failures. A complete pretraining cycle often spans months, coordinating tens of thousands of GPUs in parallel computation. A single GPU memory error, any high-speed link communication dropout, any operator precision overflow during this period can interrupt the entire task, even rendering all prior compute investment worthless.

The larger the cluster, the higher failure probability. Simple math: assume single GPU mean time between failures of 100,000 hours—a cluster of 10,000 cards encounters a hardware failure on average every 10 hours. Without millisecond-level fault detection, precise checkpoint-restart, and complete fault tolerance recovery mechanisms, large-scale full training is impossible.

For ByteDance, this challenge escalates further. Constrained by U.S. chip export restrictions, ByteDance cannot procure NVIDIA's latest Blackwell flagship, only performance-limited H20s, with single-card training efficiency lagging far behind overseas leading platforms. While others leverage stronger single-card performance to support training efficiency, ByteDance must bridge the gap through larger cluster scale, higher resource utilization, and extreme engineering optimization.

Public information shows ByteDance's training clusters span multiple intelligent computing centers in Inner Mongolia's Ulanqab, Shanxi's Datong, Hebei's Huailai and Zhangbei, and Anhui's Wuhu, establishing a distributed compute network spanning tens of thousands of cards. ByteDance's 2026 AI infrastructure capital expenditure has increased to over 200 billion yuan; the company's net profit shows notable year-over-year decline, with compute purchases and data center construction in Q3-Q4 as the core reason.

Zhang Yiming himself devotes half his attention to the Seed team. This multi-hundred-billion-yuan funding commitment, plus founder-level attention allocation, bets not merely on any single model's performance, but on an entire full-stack engineering system stably supporting ultra-large-scale training: bottom-up operator optimization, cluster communication scheduling, training fault recovery, compute resource management—all self-developed.

This training framework and infrastructure capable of steadily supporting "trillion-parameter training from scratch" constitutes ByteDance's true technological moat. Vendors pursuing distillation routes never need to crack the engineering problems of ten-thousand-card-scale stable training, naturally never accumulating this heavy-industrial-grade systems capability.

As Scaling Law continues advancing and model parameter scale keeps climbing, competition's threshold shifts from algorithms to engineering. By then, only players controlling complete infrastructure have standing at the table.

Training on others' model outputs, your model learns their output distribution—including speaking tone, answering patterns, even RLHF-phase-injected value preferences. It knows what to say about which questions, but not why; it can imitate results, not the reasoning chains behind them.

More critically, the model's "soul" bears others' imprint. What constitutes a good answer, what to refuse, what style for which context—these foundational preferences are inherited wholesale during distillation. Subsequent RLHF or reward model tuning is merely decoration atop someone else's foundation, rarely fundamentally altering model behavior patterns.

Douyin's recommendations measure completion rate and engagement, e-commerce measures click-to-purchase conversion, short-video content emphasizes pacing and visual impact—these unique commercial logics must be deeply embedded in the Reward Model, letting models align with ByteDance ecosystem product goals during RLHF.

Had you distilled GPT-4, the model's "soul" carries OpenAI's preference logic. However you tune in post-training, you're decorating someone else's foundation. Only training from scratch, full chain from pretraining through RLHF under your own control, lets ByteDance genuinely align models to its own extreme commercial logic. This concerns not merely effectiveness but controllability.

According to Financial Times reporting, ByteDance is discussing training a 100-trillion-parameter large model, significantly exceeding Alibaba Qwen 3.8-Max's 24 trillion parameters and Moonshot K3's 28 trillion parameters, the largest parameter scale model plan known domestically. The project remains in early exploration, led by Seed Foundation head Xiang Liang and coordinated with pretraining data head Shen Ke, with final official release uncertain.

This is a quintessentially "ByteDance" choice: rather than grinding to catch up at equal scale, directly scale to multiples of competitors' size, seeking leadership position through a magnitude leap.

Scaling Law remains the most reliable current advancement path for large models. Given adequate data quality and training efficiency, parameter scale increases produce predictable capability gains. While mainstream players compete within the 2-3 trillion parameter range, ByteDance directly targets 100 trillion, essentially trading scale for time, using larger investment to skip intermediate chase steps, directly reaching the next tier's threshold.

Doubling parameter scale, training difficulty doesn't grow linearly but exponentially. Data supply, compute stability, alignment difficulty, inference cost—each poses enormous challenges. 200 billion yuan capital expenditure, 70% profit decline, founder's half attention—all chips rest on this "slower, harder" path.

When DeepSeek faces formal OpenAI distillation accusations, when American legislation incorporates model data theft into national security scope, ByteDance's choice suddenly carries preemptive risk-avoidance strategic significance. Fully self-developed full-chain models carry no intellectual property disputes, no compliance baggage, standing safer ground for both global expansion and long-term technical competition.

Over the next three years, large model technology's paradigm window is rapidly narrowing. Today's technology path choices by leading players will determine competitive landscape for the next decade or beyond.

Distillation's allure lies in certainty: you know the destination, how to get there fastest, returns on investment clear to see. But its price is permanently losing the right to define the destination.

At the meeting, Zhang Yiming made clear that everyone should pursue intelligence's upper bound as their objective, expecting Seed model capabilities to reach the world's first tier.