The CODEW · Special Report · October 11, 2026
For most of the generative-AI buildout, the dominant constraint story has been compute: GPUs, data centers, power, and packaging. That story is incomplete. Access to accelerators does not automatically produce reliable systems. Data quality, expert feedback, domain coverage, and evaluation determine whether additional compute turns into capability—or into expensive noise.
This Special Report tests the thesis that data and evaluation are becoming structural infrastructure bottlenecks alongside compute. It does not claim they always dominate. It asks where they matter most, where compute or deployment still bind first, and who captures value as the constraints shift. Scale AI is the primary company lens; the argument is industry-wide.
How to read this report
[REPORTED] credible press or company disclosure via reporting · [COMPANY CLAIM] Scale or peer statements · [ANALYSIS] The CODEW interpretation/Author's analysis. Companion pieces: Startup Spotlight · Company Analysis · Company Deep Dive · The Term Sheet.
1. The Limits of the Compute-Centric Narrative
Compute is necessary. Training and serving large models require accelerators, high-bandwidth memory, networking, and power. Hyperscaler and lab capital expenditure on AI infrastructure remains enormous [REPORTED]. That does not make compute sufficient.
Models fail in production because of distribution shift, weak instruction-following, unsafe behavior, missing domain knowledge, or evaluation that never tested the failure mode. Those are data and measurement problems as much as FLOPs problems. More compute on poorly specified or poorly evaluated objectives yields diminishing returns [ANALYSIS].
The practical implication: organizations that only track GPU availability under-invest in the pipeline that makes those GPUs productive—sourcing, annotation, preference data, expert review, and evaluation design (see the companion Deep Dive).
2. From Pretraining to Post-Training and Evaluation
Early large-model progress emphasized scale of pretraining data and parameters. As systems moved into products, the binding work shifted toward post-training—supervised fine-tuning, preference optimization, safety tuning—and toward evaluation that reflects real tasks rather than only static benchmarks.
That shift increases demand for human preference data, domain-specialist input, and task-specific test suites. It also raises the cost of quality failures: a bad preference batch can teach the wrong behavior at scale. Providers such as Scale AI grew by serving this demand for frontier labs and, increasingly, enterprises and government [REPORTED / COMPANY CLAIM]. The market is not Scale alone; peers compete aggressively on expert feedback and neutrality [REPORTED].
3. Data Quality, Expert Feedback, and Domain Expertise
Capabilities that benefit most from expert-generated or expert-reviewed examples include medical and legal reasoning, advanced code, scientific tasks, safety-critical judgments, and any domain where generalist raters systematically miss edge cases. Bulk labeling still matters for volume; it does not substitute for calibrated expertise on hard subsets.
Data provenance—rights, consent, traceability—affects legal exposure and reproducibility. Enterprises and regulated buyers care about chain of custody as much as accuracy. That constraint favors vendors and internal teams that can operate under audit, not only those that can recruit the cheapest clickwork [ANALYSIS].
4. Synthetic Data: Opportunity and Limitations
Synthetic data can expand volume, balance rare classes, and reduce collection cost. It can also amplify model errors, narrow diversity, and create circular evaluation when generators and judges share the same failure modes. Validation against independent ground truth and human sample review remains necessary for high-stakes use.
Synthetic data therefore changes the shape of the bottleneck—it does not erase it. The scarce resource moves toward verification, coverage design, and human judgment on the residual hard cases [ANALYSIS].
5. Evaluation Infrastructure and Production Reliability
Static benchmarks are useful and gamed. Model-generated evaluations can scale and can be biased. Subjective human ratings need rubrics and calibration. Production reliability requires task suites that mirror real workflows, adversarial testing, and ongoing monitoring after deployment.
Organizations that treat evaluation as infrastructure—versioned, owned, tied to business outcomes—can tell capability gains from evaluation artifacts. Those that only chase leaderboard movement discover failures in production. Evaluation is therefore not a late QA step; it is part of the development system [ANALYSIS].
6. The Emerging AI Data Supply Chain
The supply chain now includes data platforms and labelers, expert marketplaces, evaluation specialists, model labs, cloud and accelerator vendors, and enterprise deployment layers. Scale AI occupies the data-and-evaluation services segment with expansion into applications; peers such as Surge AI and Mercor compete on frontier feedback; platform vendors compete on tooling; hyperscalers bundle capacity with their own stacks [REPORTED].
Ownership changes alter trust. Meta’s 2025 stake in Scale and the subsequent reduction of work by some rival labs illustrate how strategic capital can both fund a supplier and shrink its addressable customer set [REPORTED]. Neutrality became a product feature for competitors overnight.
7. Economics, Competition, and Potential Bottlenecks
Four layers structure the bottleneck analysis:
| Layer | Key question | Example focus |
|---|---|---|
| Compute | Can models be trained and served at required scale? | Accelerators, inference cost, capacity |
| Data | Are examples useful, diverse, rights-compliant? | Sourcing, annotation, expert input, synthetic |
| Evaluation | Can developers measure the failures that matter? | Benchmarks, human review, production monitoring |
| Deployment | Can the model deliver reliable value in workflows? | Integration, governance, cost, outcomes |
Value accrues to whoever is hardest to replace at the binding layer. Commodity labeling is easier to replace than calibrated expert networks and production-grade evaluation design. Tools and automation can lower cost without eliminating the need for judgment on residual hard cases. Specialized evaluation and proprietary real-world feedback loops are more defensible than undifferentiated annotation [ANALYSIS].
8. Enterprise Adoption and Governance Implications
Before deploying AI in high-consequence or domain-specific workflows, enterprises need data governance, security and privacy controls, auditability, evaluation suites tied to operational metrics, and clarity on make-versus-buy for each pipeline stage (Company Analysis). Procurement increasingly asks who touched the data, under what rights, and how quality was measured—not only unit price per label.
Measurable operational value from better data quality shows up as fewer production incidents, higher task success rates, lower rework, and faster iteration—not as a single industry-wide ROI formula. Organizations that cannot measure those outcomes will under- or over-spend on data vendors indiscriminately [ANALYSIS].
9. What Could Change the Thesis
Counterarguments that would weaken “data and evaluation as next bottlenecks”:
- Architectural or training-method advances that sharply reduce data requirements for a given capability level
- Synthetic data and automated evaluation that close the quality gap without human residual judgment
- Use cases where compute, power, or deployment integration remain clearly more binding than data
- Customer adoption and change management—not model quality—as the primary limiter of enterprise value
Honest application of the thesis is context-dependent: data and evaluation bind more tightly for post-training, domain systems, and production reliability than for pure pretraining scale races. Compute remains the headline constraint for raw training capacity. Both can be true [ANALYSIS].
10. The CODEW Intelligence Conclusion
AI infrastructure is not only silicon and megawatts. As models move from demos to products, the binding constraints increasingly include data quality, expert feedback, and evaluation systems that measure what matters in the real world.
Scale AI’s rise shows that those services can support large revenue and strategic capital. The Meta chapter shows that ownership and neutrality reshape who will buy them. Peers competing on expert data and independence show the market is contestable.
The durable insight for executives and investors: track the four layers—compute, data, evaluation, deployment—and ask which one limits progress for the specific system you are building. Betting only on GPUs is as incomplete as betting only on labels. Capability is the product of all four.
Special Report — The CODEW Intelligence. Cross-industry strategic analysis. Not investment advice.
Editorial Note
This report synthesizes the Scale AI cluster and tests a broader infrastructure thesis. Company references (Scale and peers) support evidence; the argument is not a collection of company summaries. Counterarguments are stated explicitly. Metrics and ownership facts are drawn from the Term Sheet and Spotlight source base and labeled accordingly.
Reviewed by Erwin Castro
on
Sunday, October 11, 2026
Rating:
No comments: