Arena, the crowdsourced AI leaderboard that emerged from a UC Berkeley research project in early 2023, has achieved a remarkable financial milestone. Just eight months after launching its first commercial product, the platform has reached $100 million in annualized revenue. This rapid growth underscores the soaring demand for reliable, human-driven evaluation of AI models.
The platform is best known for its simple yet powerful approach: it lets users compare responses from two anonymous AI models side by side and vote on which output is better. Over 10 million such pairwise evaluations have been submitted to date, creating a massive dataset of human preferences that companies now pay to access.
The Rise of Crowdsourced AI Evaluation
When the Chatbot Arena (as it was originally called) launched in 2023, the landscape of AI evaluation was dominated by static benchmarks like MMLU, HumanEval, and GLUE. These tests measure a model's performance on specific tasks, but they often fail to capture nuanced aspects of quality like helpfulness, creativity, and tone. Arena introduced a dynamic, crowd-powered alternative: Elo ratings derived from head-to-head voting.
The methodology gained traction quickly because it aligned closely with real user experience. Instead of relying on top-down metrics defined by researchers, Arena aggregated thousands of individual judgments. This approach proved especially valuable for generative models, where output quality is subjective. Labs from OpenAI, Anthropic, and Google soon began citing Arena rankings in their own product announcements, turning the leaderboard into the de facto scorecard for frontier AI.
The underlying technology evolved rapidly. Arena expanded from text-only comparisons to include coding, vision, and image generation tasks. In early 2025, it introduced Agent Mode, a feature designed to evaluate complex multi-step AI agent workflows. This move recognized that the next generation of AI systems would be judged on their ability to plan, use tools, and complete extended tasks, not just generate isolated outputs.
Business Model and Revenue Growth
The commercial product, called AI Evaluations, launched in September 2024. It provides model labs and enterprises with detailed performance analytics drawn from Arena's community. Instead of offering standard subscription pricing, the service charges customers based on consumption — meaning revenue is tied directly to the volume of evaluations requested.
CEO Anastasios Angelopoulos clarified that while the company reports annualized revenue, the model is not traditional SaaS recurring revenue. “A lot of people don’t even understand that our business is making any money at all — they still see us as like an open-source project,” he told TechCrunch. Despite this perception, the financial numbers speak for themselves. By December 2024, AI Evaluations had already reached $30 million in annualized revenue. The figure more than tripled over the next few months, hitting $100 million by May 2025.
The revenue surge reflects a broader market trend. Handshake, another player in the AI training data space, saw its annualized revenue nearly double from $550 million in January to almost $1 billion by April 2025, according to The Information. Mercor, a competitor in human labeling, also topped $1 billion ARR earlier this year, though it faced challenges from a supply chain breach that complicated its relationship with clients like Meta. A16z-backed Yupp, which attempted to build a similar crowdsourced evaluation platform, shut down in March after raising $33 million. Angelopoulos noted that Arena competes “for the same dollar” as Mercor, Surge, and Scale AI — companies that help model makers refine AI during post-training through human feedback.
Founding Team and Financing
Arena was co-founded by Angelopoulos and Wei-Lin Chiang, both postdoctoral researchers at UC Berkeley. Ion Stoica, the UC Berkeley professor and co-founder of Databricks, served as an advisor before the project formally incorporated in April 2025. The team's academic roots gave the initiative early credibility and helped attract a community of evaluators who valued impartiality.
In January 2025, Arena raised $150 million in a Series A round at a valuation of nearly $2 billion. The round brought total funding to $250 million, with participation from Felicis, Andreessen Horowitz, Kleiner Perkins, and Lightspeed. The capital allowed the company to scale its infrastructure, hire engineering talent, and develop new evaluation capabilities like Agent Mode. The rapid progression from research project to commercial powerhouse reflects a defining characteristic of the AI industry: the boundary between academic contribution and business opportunity is increasingly thin.
The Competitive Landscape of AI Evaluation
Arena's success highlights a growing realization among AI developers: unbiased, human-driven evaluation is a critical component of model development. Traditional benchmarks become saturated quickly — many models now achieve near-perfect scores on MMLU, making them less useful for differentiation. At the same time, subjective quality matters more than ever as models are deployed in customer-facing applications. Companies need to understand not just what a model can do, but how it is perceived by actual users.
The demand for this kind of data has created a vibrant ecosystem. Scale AI, one of the largest human labeling companies, reported billions in valuation but focuses on training data creation, not crowdsourced evaluation. Mercor and Surge similarly provide outsourced human labeling services for post-training refinement. Arena's unique position lies in combining a public, transparent leaderboard with a paid analytics layer — an approach that gives it both community trust and commercial viability.
The absence of a direct competitor with a comparable crowdsourcing model has worked to Arena's advantage. Yupp's shutdown eliminated the only other startup that attempted to aggregate human preferences for model comparison. The remaining alternatives are either closed, proprietary evaluation systems built by individual labs or generic data labeling services that lack the competitive voting dynamic.
Looking at the broader market, the numbers are staggering. Handshake's annualized revenue from AI training nearly doubled from $550 million to $1 billion in just three months. Mercor's revenue also crossed the billion-dollar threshold. These figures suggest that the market for human-in-the-loop AI evaluation and training could grow even larger as more enterprises deploy customized models. Arena, with its $100 million annualized run rate, is carving out a valuable niche by specializing in comparative evaluation rather than generic data labeling.
Implications for the AI Industry
The fact that a crowdsourced leaderboard can generate $100 million in less than a year from a paid service speaks volumes about changing priorities in AI development. Model makers no longer rely solely on static benchmarks or internal testing. They need to understand how their systems perform relative to competitors in a live, dynamic environment where user preferences shift over time. Arena's data offers that real-world signal.
Moreover, the platform's expansion into Agent Mode indicates that evaluation is evolving alongside AI capabilities. Early LLM comparisons focused on text generation quality. Now, with multi-step agent tasks becoming more common, evaluators need to assess planning, memory, tool use, and goal completion. Human judges can provide nuanced feedback that automated metrics struggle to capture — such as whether an agent's reasoning process was sound, not just whether it produced a correct final answer.
The growth also attracts investors. The $250 million raised by Arena, including a $150 million Series A at a nearly $2 billion valuation, shows that venture capital sees evaluation infrastructure as a defensible business. Unlike model training, which requires enormous compute and capital, evaluation can be built on a relatively lean model once a community and methodology are established. As Angelopoulos noted, many people still view Arena as an open-source project, but the revenue numbers reveal a sustainable commercial entity underneath.
For AI labs and enterprises, the cost of evaluation is a small fraction of the total investment in model development, yet it can heavily influence product direction. Rankings from Arena have been cited in blog posts, research papers, and product launches, making the platform a gatekeeper of sorts. Turning that influence into a $100 million business in under a year suggests that evaluating AI may be nearly as lucrative as building it.
The next milestones for Arena will likely involve expanding its evaluation categories further — perhaps into multimodal reasoning, real-time dialogue, or safety alignment testing. The company may also need to address questions about representativeness and bias in its evaluator pool. For now, though, the numbers demonstrate that the market for high-quality, human-driven AI evaluation is large and growing fast, and Arena is leading the charge.