Synthetic Data Changed Everything and Nobody Noticed

There's a dirty secret at the heart of every frontier AI model released in the past 18 months. Not a scandal-level secret. More like a structural truth that nobody in the industry wants to emphasize because it undermines the narrative they've been selling. The secret: the best AI models are trained, in significant part, on data generated by other AI models. Synthetic data. AI-generated text, code, math proofs, reasoning chains, and conversations, created specifically to train the next generation of models. Not scraped from the web. Not labeled by human annotators. Generated by machines, for machines. This isn't an edge case or a last resort. Microsoft's Phi-4 technical report says it plainly: the model "strategically incorporates synthetic data throughout the training process." And Phi-4 — a 14-billion-parameter model — outperforms GPT-4 on STEM reasoning. The student surpassed the teacher, trained partly on data the teacher generated. That should make you sit up straight. Because it means the "data wall" everyone was panicking about two years ago? It might not exist. ## The Data Wall That Wasn't The panic started around 2023. Several influential papers and industry analysts argued that we were running out of high-quality training data. Epoch AI published estimates showing that the stock of high-quality text on the internet — books, articles, academic papers, curated datasets — would be exhausted by 2026 if models kept scaling at the current rate. The implications were dire. If you can't get more data, you can't train bigger models. If you can't train bigger models, the scaling laws break down. If the scaling laws break down, the entire premise of the AI boom — that more compute plus more data always equals better models — collapses. OpenAI reportedly began conversations with major publishers, news organizations, and content platforms about licensing deals. They struck agreements with the Associated Press, Axel Springer, and others. Google signed deals with Reddit and various news outlets. The scramble for data looked like a land grab, and the implication was that whoever controlled the most training data would win the AI race. Then something surprising happened. Labs started training models on synthetic data, and the models got better, not worse. ## How It Actually Works Synthetic data generation for AI training isn't just "have GPT write some stuff." That naive approach leads to what researchers call "model collapse" — a degenerative process where each generation of model trained on the previous generation's output gets progressively worse, like a photocopy of a photocopy. The key insight, developed across multiple labs between 2023 and 2025, is that synthetic data needs to be structured, targeted, and diverse. You don't generate random text. You generate specific types of reasoning exercises, edge cases, and problem-solution pairs that fill gaps in the organic training data. Microsoft's Phi team pioneered this approach. For Phi-4, they didn't just ask GPT-4 to write text. They designed multi-step synthetic data pipelines: **Seed-based generation.** Start with real-world prompts, problems, or scenarios as seeds. Use a teacher model to generate detailed solutions, explanations, and reasoning chains. The seeds ensure the synthetic data is grounded in real-world topics. **Revision and refinement.** Generate a first-pass answer, then use the same or a different model to critique it, identify errors, and produce a refined version. This self-correction process produces cleaner training data than a single-pass generation. **Diversity injection.** Deliberately vary the style, difficulty level, domain, and format of generated data. If your math training data is all algebra, generate some geometry, some statistics, some number theory. If your code data is all Python, generate Rust, TypeScript, Go. Diversity prevents the model from overfitting to a narrow distribution. **Quality filtering.** Run generated data through automated quality checks — correctness verification for math problems, unit tests for code, consistency checks for logical reasoning. Discard anything that doesn't pass. This is expensive but essential. The result isn't "AI slop." It's carefully engineered training data that's often higher quality than what you'd scrape from the internet, because internet data includes a massive amount of noise — wrong answers on Stack Overflow, poorly written blog posts, contradictory information across sources. ## The Companies Building the Infrastructure While the big labs were figuring out synthetic data for their own models, a parallel industry was forming to provide synthetic data as a service. **Scale AI** has positioned itself as the dominant player in the broader AI data market, though its original business was human annotation. Scale's shift toward synthetic data reflects the industry's trajectory. The company offers synthetic data generation tools through its GenAI Platform, letting customers create domain-specific training datasets without relying entirely on human labelers. Scale was valued at $14 billion as of its 2024 funding round, and the synthetic data business is an increasingly large share of revenue. **Gretel** was arguably the first company built specifically around synthetic data. Founded in 2020, Gretel provides tools for generating tabular, text, and time-series synthetic data that preserves the statistical properties of real datasets without containing any actual real data points. Their core use case was initially privacy — letting companies share data that looked like their real data without exposing sensitive records. But the application has expanded to model training, where Gretel's synthetic datasets are used to augment limited real-world data. Gretel was acquired by NVIDIA in a deal that underscores how seriously the GPU maker takes synthetic data. The combined offering — NVIDIA's compute hardware plus Gretel's data generation software — targets enterprises that need to train custom AI models but don't have enough proprietary data to do it well. **Tonic.ai** carved out a niche in regulated industries. Their synthetic data platform generates realistic but fake data for healthcare, financial services, and other sectors where privacy regulations make it difficult or impossible to use real data for development and testing. A hospital can generate synthetic patient records that match the statistical distribution of their real patient data, train an AI model on the synthetic records, and deploy it without ever exposing actual patient information. **Mostly AI**, based in Vienna, focuses on synthetic data for analytics and machine learning. Their platform generates synthetic versions of structured datasets — customer records, transaction data, sensor readings — that can be shared, analyzed, and used for model training without privacy concerns. ## The Research Behind the Revolution The academic research on synthetic data has been prolific and surprisingly positive. A March 2024 paper from researchers at Microsoft and Carnegie Mellon, "Textbooks Are All You Need II," showed that small models trained on synthetic "textbook-quality" data could match or exceed much larger models on coding benchmarks. The insight: if you generate training data that looks like a well-written textbook — clear explanations, worked examples, progressive difficulty — the model learns more efficiently than if you feed it raw internet text. A key 2024 study published in Nature by researchers at Rice and Stanford examined whether AI-generated training data inevitably leads to model collapse. Their finding: it does, but only under specific conditions. If you maintain a core of high-quality real data and supplement it with synthetic data — rather than replacing all real data — the degenerative cycle doesn't occur. The ratio matters. Most researchers now target 50-70% real data and 30-50% synthetic data as a reasonable mix. Google's work on "Self-Play Fine-Tuning" (SPIN) demonstrated another approach: models can improve by generating their own training data through an adversarial process. The model generates answers, then learns to distinguish between its own generated answers and human-written answers, then generates better answers. After several rounds, the model's outputs improve measurably without any new human data. Anthropic has been notably quiet about its use of synthetic data, but its constitutional AI approach is, at its core, a synthetic data technique. You generate model outputs, have another model (or the same model with different prompts) evaluate those outputs against constitutional principles, and use the evaluations to train the model to produce better outputs. The "data" being generated is the evaluation itself. ## Why This Matters More Than GPT-5 Here's my actual thesis, and I'll state it directly: synthetic data is more important than any single model release. GPT-5 will come out. It'll be impressive. Benchmarks will go up. Demos will be cool. The internet will lose its mind for 72 hours. And then we'll go back to waiting for GPT-6. Synthetic data, by contrast, is a permanent capability shift. It means: **The data wall doesn't exist.** If you can generate high-quality training data, you're never limited by the stock of internet text. The constraint shifts from "do we have enough data?" to "can we generate the right data?" That's a fundamentally easier problem. **Small labs can compete.** You don't need a $500 million data licensing deal with every publisher on earth. You need a good teacher model, a smart data generation pipeline, and enough compute to run the pipeline. A 20-person startup can generate training data competitive with what Google scraped from the entire internet. That changes the competitive dynamics of the industry. **Domain-specific AI becomes practical.** Training a model for legal reasoning used to require a massive corpus of legal documents, many of which are proprietary or privileged. Now you can generate synthetic legal reasoning examples using a general-purpose model and train a specialist on the output. Same for medical, financial, scientific, and industrial domains. **Privacy regulations stop being a barrier.** You can't train a model on patient health records. But you can generate synthetic health records with the same statistical properties and train on those. The EU's GDPR and the US HIPAA framework both become much less burdensome when your training data contains zero real personal information. **Model improvement becomes recursive.** Each generation of model is better at generating training data for the next generation. Phi-4 was trained partly on GPT-4-generated data and outperformed GPT-4 in some areas. The next model will be trained on data generated by Phi-4's successors. The flywheel spins. ## The Risks Nobody Talks About I'd be dishonest if I painted this as purely positive. Synthetic data has real risks that the industry is mostly ignoring. **Homogenization.** If every model is trained on data generated by other models, the outputs converge. Diversity of perspective, style, and approach decreases. The internet already has a "GPT voice" problem — generic, hedging, overly structured prose that all sounds the same. Synthetic training data could amplify that problem. **Garbage amplification.** Quality filters aren't perfect. If a teacher model has systematic biases or errors, those get baked into the synthetic training data and amplified in the student model. A model that's subtly wrong about history will generate subtly wrong history training data, training a model that's confidently wrong about history. **Verification challenges.** How do you audit training data that was generated by an AI? You can check individual examples, but you can't check millions. The provenance of synthetic data is inherently less traceable than data scraped from identifiable sources. When something goes wrong, it's harder to figure out why. **Legal uncertainty.** Copyright law around synthetic data is largely untested. If GPT-4 generates training data that's stylistically similar to copyrighted text it was trained on, is the synthetic data a derivative work? Nobody knows. The courts haven't decided. The major AI companies are betting that synthetic data is sufficiently transformed to avoid copyright claims, but that bet hasn't been tested at scale. ## The Quiet Revolution Gartner projected that by 2026, 75% of businesses would use generative AI to create synthetic data, up from less than 5% in 2023. That prediction is playing out. The companies building the best models aren't the ones with the most data. They're the ones with the best data generation pipelines. The competitive advantage isn't hoarding — it's synthesis. Five years from now, I suspect we'll look back at the "data wall" panic of 2023-2024 the way we look back at predictions that the internet would run out of IP addresses. It was a real constraint, briefly. Then smart people engineered around it. And the solution they found — AI generating its own training data, in a controlled, quality-filtered, recursive process — turned out to be not just adequate but superior to what came before. Nobody held a press conference to announce it. There was no keynote demo. The revolution happened in training pipelines and research papers, quietly, while everyone was arguing about whether GPT-5 would be "AGI." Synthetic data changed everything. And almost nobody noticed.