Investment professionals have spent two years asking whether AI is ready to handle real research workflows. Today, the first practitioner-validated open benchmark designed specifically for that question delivered its first answer: not yet, and not close.<\/p>\n
Samaya AI, a Mountain View, California startup focused exclusively on AI for investment professionals, published evaluation results today<\/a> on FrontierFinance \u2014 its open benchmark measuring AI agent performance across the full investment workflow. The results set a ceiling: Anthropic’s Claude Fable 5, the top-performing frontier model in the evaluation, cleared just 49.2%<\/a> of the benchmark’s tasks. OpenAI’s GPT-5.6 Sol followed at 46.8%, and Claude Opus 4.8 at 45%. Google’s Gemini 3.1 Pro also participated; all clustered well below the halfway mark.<\/p>\n Samaya’s own proprietary agentic system, designed for exactly these tasks, cleared the bar \u2014 but only just, scoring 50.8% accuracy<\/a> at roughly four times lower inference cost than Fable 5 in its “low effort” configuration, and 56% in a “high effort” mode at roughly two times lower cost.<\/p>\n What FrontierFinance Actually Measures \u2014 and Why Prior Finance Benchmarks Fell Short<\/p>\n The benchmark’s design philosophy begins with a diagnosis of what has made existing finance AI evaluations misleading. Prior benchmarks focused almost exclusively on financial data extraction<\/a> \u2014 pulling a figure from a filing, answering a factual question about a balance sheet item. Those tasks are real but narrow, and models trained on large text corpora can handle them reasonably well.<\/p>\n FrontierFinance covers the tasks that come before and after that extraction: synthesizing a sector-wide picture from multiple sources, screening for opportunities across an idea universe, tracking a covenant through earnings commentary, updating a model when a macro catalyst breaks. These are the tasks that drive capital allocation decisions, not the retrieval steps that support them.<\/p>\n The benchmark comprises 220 open-ended queries spanning six use cases: Screening and Discovery, Company Research, Sector and Macro Analysis, Earnings and Events, Coverage and Catalyst Monitoring, and Financial Data and Modeling. Each query is paired with 11,543 expert-authored rubric items in total<\/a> \u2014 atomic, verifiable criteria that a complete answer must satisfy. Rubric items are classified as either “must-have” (required for an acceptable response) or “informational” (enriching but not essential). Scoring is the rubric qualification rate: the percentage of criteria satisfied, macro-averaged across queries, with each rubric scored by majority vote from three independent judge models.<\/p>\n This evaluation design was built by Samaya’s finance experts from a larger internal annotation set of approximately 5,000 complex examples accumulated over multiple years of building production AI systems for investment professionals. The full dataset and evaluation code are publicly available on Hugging Face and GitHub<\/a> respectively, making independent replication possible.<\/p>\n How Three Different Harnesses Explain the Cost-Quality Tradeoff<\/p>\n One of the technically significant findings in today’s release is not just the scores but how differently the same model performs depending on what it is connected to. Samaya evaluated models across three distinct harness architectures:<\/p>\n A web search harness pairs each frontier model with its built-in web search API \u2014 the way most enterprise teams currently deploy general-purpose AI on research tasks. A Finance Agent v2 harness, developed by the independent evaluation firm Vals AI and open-sourced on GitHub<\/a>, connects a model to six specialized finance tools: the SEC’s EDGAR API, a market price data API, web search, HTML parsing, long-document content search, and a calculator. Samaya’s in-house harness adds its own custom models, proprietary data index, and retrieval engines optimized for investment-specific content.<\/p>\n The data shows the harness matters as much as the base model. The Finance Agent v2 harness consistently outperformed web search alone for the same model. The quality-cost tradeoff within the harness-plus-model combinations was steep: meaningful gains in rubric qualification rate came with sharply higher per-query inference cost \u2014 until Samaya’s own system, which the company says breaks that tradeoff through custom model architecture and domain-specific retrieval, not through using a smaller general model on the same stack.<\/p>\n For the open-source and lower-cost models evaluated, DeepSeek V4 Pro reached 40.5%<\/a> on the Finance Agent v2 harness at roughly 26% of Fable 5’s per-query inference cost \u2014 the best cost-quality tradeoff among models without a proprietary harness. That finding is relevant for institutions with data-residency or confidentiality requirements that preclude sending research workflows to third-party APIs.<\/p>\n Data sourcing within the benchmark<\/a> reflects the actual distribution of information investment professionals work across: SEC filings account for roughly 39% of rubric grounding, with the rest drawn from earnings transcripts, investor presentations, professional knowledge, and market data \u2014 and that mix shifts meaningfully depending on which of the six use cases is being evaluated.<\/p>\n What the Sub-50% Ceiling Actually Means for Investment Workflows<\/p>\n “Investment use cases are uniquely hard for AI because being almost correct is still a loss,” said Maithra Raghu, CEO and Founder of Samaya AI<\/a>, who founded the company in 2022 alongside researchers from Google Brain, Meta, and Amazon. “Producing an expert-level response requires accuracy on every datapoint and every step of reasoning across a long, complex workflow, not just a plausible-looking output.”<\/p>\n That framing captures a structural problem the benchmark makes quantitative. In a multi-step investment workflow \u2014 building a relative value model across peer companies, synthesizing sector catalysts against a portfolio, flagging a covenant-relevant phrase buried in an earnings footnote \u2014 an error at any point in the reasoning chain can invalidate every downstream conclusion. A model that gets 49% of rubrics right is not delivering half-useful output; for many high-stakes tasks, it is delivering work that requires full human verification to catch the errors, negating much of the productivity gain.<\/p>\n The context matters here. Finance AI hallucination is no longer a theoretical risk: FINRA’s 2026 Annual Oversight Report<\/a> explicitly flagged AI hallucinations as a compliance risk that financial firms must manage. The EU AI Act’s requirements<\/a> for human oversight and accuracy guarantees for high-risk financial AI systems take effect imminently. In this environment, a domain-specific benchmark that names a concrete accuracy ceiling gives procurement teams something public leaderboards never have: a score tied to tasks their analysts actually perform.<\/p>\n FrontierFinance’s Credibility Problem \u2014 and Why It Still Matters<\/p>\n Samaya AI built this benchmark. Samaya AI’s system won it. That conflict of interest is real, and any organization evaluating whether to adopt FrontierFinance as a reference standard should weigh it seriously.<\/p>\n The benchmark gaming problem in AI is well-documented. In January 2026, Meta’s CEO confirmed to the Financial Times that Llama 4’s Chatbot Arena submission had been “fudged” \u2014 a purpose-built variant optimized for blind comparison voting rather than real-world performance. OpenAI secretly funded the FrontierMath benchmark while retaining exclusive early access to its questions. The phenomenon has a name in the research community: benchmaxxing<\/a> \u2014 optimizing for leaderboard position because that position drives press coverage, fundraising, and enterprise procurement, regardless of whether the score reflects genuine capability.<\/p>\n Samaya attempts to address the self-evaluation problem structurally: the full dataset of 220 queries and all 11,543 rubrics are publicly released, and the evaluation code is open-sourced on GitHub. Any institution can independently run frontier models against the same rubrics using the standardized harness. That is a meaningful architectural choice; most proprietary benchmarks offer neither.<\/p>\n The more important point for readers deciding whether to trust today’s results is this: Samaya’s own score \u2014 50.8% in low-effort mode, 56% in high-effort mode \u2014 must be read as a company’s claim about its own system, not as independently verified performance. But the frontier models’ scores tell a different story. Claude Fable 5, GPT-5.6 Sol, Claude Opus 4.8, and Gemini 3.1 Pro all scored between 45% and 49.2%. Those results were produced by external frontier systems running against a fixed, publicly available rubric set \u2014 not by Samaya’s own system, on Samaya’s own proprietary rubrics, graded by Samaya’s own judge. The frontier model ceiling is the finding that stands most independently of who built the benchmark.<\/p>\n That finding has independent corroboration. Vals AI’s Finance Agent v2 benchmark \u2014 developed by a separate evaluation firm using its own proprietary rubric methodology \u2014 has shown frontier models clustering in the high-40% to low-50% range<\/a> on realistic investment tasks throughout 2026. The convergent evidence from two independently designed benchmarks makes the sub-50% ceiling less a Samaya promotional claim and more a structural observation about where frontier AI currently stands relative to professional-grade investment research.<\/p>\n How Is Difficulty Measured \u2014 and What the Methodology Reveals About Finance Tasks<\/p>\n FrontierFinance’s difficulty measurement uses a statistical tool borrowed from tournament ranking: the Bradley-Terry model<\/a>, originally developed in 1952 for pairwise comparison data and now widely used in AI evaluation contexts including the underlying structure of Chatbot Arena’s Elo system. The benchmark’s difficulty scores were produced by having an AI agent perform pairwise comparisons between queries from FrontierFinance and from prior finance benchmarks, asking which query is harder to answer completely and correctly. Bradley-Terry coefficients were then fitted to those pairwise judgments to produce a relative difficulty ranking across the full query pool.<\/p>\n The output shows FrontierFinance covering a broader range of difficulty and a higher mean difficulty than any existing public finance benchmark \u2014 and the breakdown by use case reveals where the gaps are largest. Screening and Discovery tasks and Sector, Industry and Macro Analysis<\/a> were the hardest use cases across every system evaluated, with substantial room for improvement even for the top-performing models. Format and Presentation rubrics were where systems performed best \u2014 suggesting that what current AI handles most reliably is the packaging of answers, not their analytical depth.<\/p>\n One methodological note worth flagging: the difficulty scoring itself depends on an AI judge making pairwise comparisons. That judge is an AI system that may have its own systematic biases about what makes a finance question analytically demanding \u2014 biases that could skew difficulty scores toward tasks the judge happens to find confusing rather than tasks practitioners find most consequential. Samaya’s team is transparent about the four-stage expert-driven data collection process<\/a>, which includes finance-expert query drafting, expert rubric authoring, expert review and cleanup, and taxonomy-based rebalancing. But the difficulty scores themselves are AI-judged, and that is a limitation independent researchers evaluating the benchmark should examine.<\/p>\n Does This Tell Us AI Cannot Replace Analysts?<\/p>\n Not exactly. The benchmark was deliberately designed to test the hardest end of the investment workflow, not the full distribution of tasks analysts perform. Summarization, extraction, first-pass screening, and synthesis of structured earnings data are tasks where AI performs substantially better \u2014 and where it is already delivering productivity gains at hedge funds and asset managers using Samaya’s and competing platforms.<\/p>\n What today’s results do say clearly is that autonomous AI handling of end-to-end investment research workflows \u2014 without human verification at the output stage \u2014 is not yet supported by evidence from any of the most capable frontier models available. Samaya is backed by a roster of investors including NVIDIA, NEA, Databricks, Eric Schmidt, and Yann LeCun<\/a>, and says the platform serves tens of thousands of investment professionals globally at hedge funds, asset managers, and banks. That commercial scale makes the benchmark release more than a marketing exercise: a platform embedded at that depth in professional workflows has genuine incentives to define accuracy requirements at the right level.<\/p>\n The company said it plans to release subsequent, harder versions of FrontierFinance<\/a> as model capabilities evolve and the evaluation methodology matures through community feedback. The full benchmark dataset, rubrics, and evaluation code are publicly available at the Samaya Research site<\/a>.<\/p>\n Frequently Asked QuestionsWhat is the FrontierFinance benchmark and who built it?<\/p>\n FrontierFinance is an open benchmark for measuring how well AI agent systems handle real investment research tasks \u2014 not data extraction quizzes, but the full arc of an analyst’s work including idea screening, company research, sector and macro analysis, earnings monitoring, and catalyst tracking. It was built by Samaya AI, a Mountain View startup specializing in AI for investment professionals, using 220 expert-crafted queries and 11,543 rubric items derived from approximately 5,000 internal examples accumulated over multiple years of production deployment. The full dataset and evaluation code are publicly available on Hugging Face and GitHub.<\/p>\n How accurate are the best AI models at financial analyst tasks in 2026?<\/p>\n On FrontierFinance, the best-performing frontier model \u2014 Anthropic’s Claude Fable 5 \u2014 cleared 49.2% of tasks, with GPT-5.6 Sol at 46.8% and Claude Opus 4.8 at 45%. All evaluated frontier models scored below 50%. The independent Vals AI Finance Agent v2 benchmark, which uses a separate methodology, has produced similar results: frontier models cluster in the high-40% to low-50% range on realistic investment analyst tasks. The consistent finding across two independently designed evaluations suggests the sub-50% ceiling reflects a real structural capability gap, not an artifact of a single methodology.<\/p>\n Can Samaya AI’s own score on its own benchmark be trusted?<\/p>\n With caution. Samaya built FrontierFinance and reported its own system scoring 50.8% (low effort) and 56% (high effort) \u2014 results that have not been independently verified. That creator-conflict is a genuine limitation: in 2025 and 2026, multiple prominent AI systems were caught gaming benchmark evaluations, and self-reported scores on creator-owned benchmarks carry a well-documented credibility problem. The more independently reliable findings from today’s release are the frontier model scores, which were produced by external systems running against a fixed, publicly released rubric set. Those scores can be independently verified by any organization with the time and compute to run the evaluation themselves.<\/p>\n What practical steps should investment firms take based on these results?<\/p>\n Do not use any current frontier AI model for autonomous, unverified investment research workflows where errors carry material financial or compliance consequences. The benchmark confirms that even the strongest general-purpose models fail roughly half the tasks in a real analyst’s workflow. AI delivers clear value as a productivity accelerant for well-defined, human-reviewed subtasks \u2014 extraction, summarization, first-pass screening, and formatting \u2014 but today’s results do not support delegating end-to-end research decisions without human verification at the output stage. For procurement decisions, demand domain-specific benchmark evidence run against tasks representative of your actual workflow, not general leaderboard scores. FINRA’s 2026 Annual Oversight Report treats AI hallucination risk as a compliance obligation, not a theoretical concern.<\/p>\n","protected":false},"excerpt":{"rendered":"Investment professionals have spent two years asking whether AI is ready to handle real research workflows. Today, the…\n","protected":false},"author":2,"featured_media":123315,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[8],"tags":[24,53782,53,3154,182,38051,18297,62170,62171,62167],"class_list":["post-123314","post","type-post","status-publish","format-standard","has-post-thumbnail","category-anthropic","tag-ai","tag-ai-benchmark","tag-anthropic","tag-anthropic-claude","tag-claude","tag-claude-fable-5","tag-finance-ai","tag-frontierfinance","tag-investment-ai","tag-samaya-ai"],"_links":{"self":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/posts\/123314","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/comments?post=123314"}],"version-history":[{"count":0,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/posts\/123314\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/media\/123315"}],"wp:attachment":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/media?parent=123314"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/categories?post=123314"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/tags?post=123314"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}