마이크로소프트, 텍스트 설명으로 AI 행동 테스트 생성 도구 공개

AI researchers and labs have advanced by leaps and bounds in evaluating AI models for everything from safety and compliance to sycophancy and alignment . But it appears companies and developers are faced with a new, specific need: making sure that their AI system behaves as intended for their specific product or service. In a bid to make that testing process simpler, Microsoft on Tuesday took the wraps off ASSERT , short for Adaptive Spec-driven Scoring for Evaluation and Regression Testing. The open-source framework, Microsoft says, makes evaluating application-specific AI behavior easy by using AI to turn high-level, natural-language descriptions of goals, policies, or intended behaviors into thorough, scored tests that can be investigated. ASSERT takes plain-language descriptions of an AI model's expected behavior and policies, turns them into a structured set of acceptable and unacceptable behaviors, generates problem scenarios and test cases, runs them against the target system, and scores the results. It can also record the paths the AI system takes, including intermediate actions and tool calls, so developers can inspect where failures happen. Devs can provide system context, tools, and constraints, too, if they want to further customize what the evaluations cover. For example, a developer could specify that a document research AI agent shouldn't send emails to people outside the company, limit confidential information to C-level executives, and provide concise summaries with prior context in mind. ASSERT will use those rules to generate test cases that check whether the system follows those rules on an ongoing basis. The framework, according to Microsoft, fills a gap that broader, more general evaluations cannot when AI models are intended to behave in a manner that is shaped by an application or product's context, policies, and tools. "One of the things we've learned is that evaluations are absolutely critical to making good decisions," said Sarah Bird , chief product officer of Responsible AI at Microsoft. "Because if you don't understand the behavior of the AI system, it's really hard to know if it's meeting your organization's bar […] What we found is that if you really want to have a trustworthy system, you should evaluate many more dimensions that are application-specific." Bird said ASSERT can be used to evaluate systems when they're being built, after deployment, and even for continuous monitoring. The release comes amidst a gradual but broader shift in the AI industry. As models grow more capable, researchers are focusing on repeatable testing and regression checks, with Stanford’s HELM , MLCommons’ AILuminate , and evaluation groups like METR rolling out benchmarks to measure how models behave under different conditions. Topics AI , ai evaluations , AI regression testing , Microsoft When you purchase through links in our articles, we may earn a small commission . This doesn’t affect our editorial independence. Ram Iyer Editor Ram is a financial and tech reporter and editor. He covered North American and European M&A, equity, regulatory news and debt markets at Reuters and Acuris Global, and has also written about travel, tourism, entertainment and books. You can contact or verify outreach from Ram by emailing ram.iyer@techcrunch.com . View Bio June 18 Los Angeles Get an inside look at what it takes to scale and succeed from leaders at Mach Industries, Founders Fund, and Shinkei Systems. Through candid fireside chats and high-impact networking, you'll walk away with valuable insights and new connections. REGISTER NOW Most Popular ‘What a joke': GitHub Copilot's new token-based billing spurs consternation among devs Lucas Ropek I put Google's 24/7 AI assistant Gemini Spark to work, and it's actually pretty useful Sarah Perez Blue Origin's New Glenn rocket explodes during testing in Florida Sean O'Kane Anthropic releases Opus 4.8 with new ‘dynamic workflow' tool Russell Brandom Google engineer charged with insider trading after making $1.2M on Polymarket Amanda Silberling Meta launches Instagram, Facebook, and WhatsApp subscriptions, with more to come, including AI plans Sarah Perez Tech CEOs are apparently suffering from AI psychosis Julie Bort