500명 투자은행가 평가, 고객 전달 가능한 AI 결과물은 0%

500 investment bankers review AI outputs and find none ready for client delivery Jonathan Kemper View the LinkedIn Profile of Jonathan Kemper Apr 26, 2026 Nano Banana Pro prompted by THE DECODER A new benchmark puts top models like GPT-5.4 and Claude Opus 4.6 to work on the kinds of tasks junior investment bankers handle every day. Not a single output was deemed ready to send to a client. Still, more than half of the bankers said they'd use it as a starting point. A research team at Handshake AI and McGill University has released BankerToolBench, an open-source benchmark that tests AI agents against the typical workflows of junior investment bankers. Handshake AI is the business arm of the career platform Handshake, which places vetted academics and professionals inside AI labs to help train and evaluate their models. After running nine current top models through the test, the verdict from the bankers involved is blunt: none of the outputs are fit for client use. The team enlisted around 500 current and former investment bankers from firms including Goldman Sachs, JPMorgan, Evercore, Morgan Stanley, and Lazard. Of those, 172 designed the tasks themselves, logging more than 5,700 hours of work. Each of the 100 tasks took a human banker an average of five hours, with some running up to 21 hours. Real Excel models, not just text answers BankerToolBench grades the actual deliverables a junior banker would hand to a supervisor: Excel financial models with working formulas, PowerPoint decks for client meetings, PDF reports, and Word memos. The agents have to dig through data rooms, pull from market data platforms like FactSet and Capital IQ, and parse SEC filings. According to the paper, a single task can trigger up to 539 calls to the language model, with 97 percent tied to tool use or code execution. Each deliverable is checked against a banker-designed rubric averaging 150 individual criteria. The criteria span six areas, including technical correctness, client readiness, compliance, auditability, and consistency across files. Grading is handled by an AI verifier the authors built called Gandalf, based on Gemini 3 Flash Preview. It agrees with human reviewers 88.2 percent of the time, slightly above the 84.6 percent agreement rate between two human reviewers. GPT-5.4 leads, but it's not close to passing The team tested GPT-5.2, GPT-5.4, Claude Opus 4.5 and 4.6, Gemini 2.5 Pro, Gemini 3.1 Pro Preview, Grok 4, and the open-source models Qwen-3.5-397B and GLM-5. GPT-5.4 came out on top but still failed nearly half the criteria. Just 16 percent of its outputs cleared the bar where bankers would accept them as a useful starting point. Require three consistent runs, and that figure drops to 13 percent. Not a single output from any model was deemed ready to submit as is. With GPT-5.4, just 2 percent of tasks cleared every critically weighted criterion. With Gemini 2.5 Pro, that figure was zero. Pretty on the outside, broken underneath Claude Opus 4.6's outputs look polished at first glance, according to the researchers. But the Excel models reveal a fundamental flaw: most of the key numbers are hardcoded as fixed values rather than calculated through formulas. That's a dealbreaker in investment banking, the paper notes, because it makes scenario analysis impossible. Change the purchase price in the model, and nothing updates. Claude Opus 4.5 had the same problem. An analysis of GPT-5.4's agent trajectories surfaces four recurring failure modes. The most common, at 41 percent, are bugs in code and formula generation. The agents call python-pptx functions that don't exist, and rather than fixing the underlying issue, they simply delete the broken line. In 27 percent of cases, the business logic breaks down, such as adding cost synergies to the revenue line instead of to costs. Another 18 percent of errors stem from aborted data queries. And in 13 percent of cases, agents fabricate missing numbers and pass them off as sourced. Subtle errors that slip through The examples in the paper illustrate how subtle these failures can be. In one generated deck, the verifier flags a revenue figure of $189.5 billion on one slide and $201.0 billion on the next, both covering the same period. In another case, the agent uses Netflix red as an accent color even though the bank's style guide mandates a uniform blue. In a competitive analysis for a pharma deal, an agent fabricated specific clinical trial data after coming up empty in the SEC database. The models generally perform better on PowerPoint tasks than on Excel work. The toughest tasks fall in debt capital markets, merger models, and capital structure tables. The team attributes some of the shortfall to missing domain knowledge. When tasks are enriched with the kind of context bankers take for granted, scores rise significantly. A training tool, too BankerToolBench can also be used for reinforcement learning, according to the authors. In experiments with Qwen-3-4B and 32B, the Dr. GRPO and DPO methods boosted benchmark performance by a factor of five to thirteen, though from a very low baseline. The team flags several limits: the benchmark is US-focused, lacks confidential deal information, and doesn't capture the iterative teamwork inside a real bank. Even so, the authors call it one of the most detailed tests yet of whether AI agents can handle demanding knowledge work. For now, the answer is no. The full benchmark, including data, rubrics, and verifier, is publicly available . The findings line up with other recent research. A Vals.ai study conducted with a globally systemic bank found that OpenAI's o3 hit just 48.3 percent accuracy on financial analysis tasks. UC Berkeley research concluded that the teams getting agents to work in production are relying on simple, tightly controlled setups with few steps. And an analysis from Carnegie Mellon and Stanford argues that agent development has focused too narrowly on coding tasks, leaving economically important fields like management, law, and finance largely absent from benchmarks. Meanwhile, AI labs like Anthropic are working on exactly the weaknesses BankerToolBench exposes. Anthropic recently introduced a feature that lets Claude switch on its own between Excel and PowerPoint , and Cowork plugins now pipe market data services like FactSet, MSCI, and LSEG directly into the workflow. AI News Without the Hype – Curated by Humans Subscribe to THE DECODER for ad-free reading, a weekly AI newsletter, our exclusive "AI Radar" frontier report six times a year, full archive access, and access to our comment section. Subscribe now --> AI news without the hype Curated by humans. More than 16% discount. Read without distractions – no Google ads. Access to comments and community discussions. Weekly AI newsletter. 6 times a year: “AI Radar” – deep dives on key AI topics. Up to 25 % off on KI Pro online events. Access to our full ten-year archive. Get the latest AI news from The Decoder. Subscribe to The Decoder -->