See our AI Robotic system in action at ProMat (Booth #N6951) Book a personalized demo
Industry topics
March 17, 2021

The need for performance-based assessments

Tianhao Zhang

From eCommerce to apparel and grocery, organizations are urgently looking to maximize warehouse utilization while reducing costs. The COVID-19 pandemic has intensified these strains with an added concern about operational resilience.

Consequently, many leaders are moving rapidly to adopt AI-enabled robotic automation within their warehouses.

Robots that can learn and adapt - AI Robotics - have emerged as a powerful supply chain solution. AI Robotics can unlock a previously impossible level of performance and efficiency for a wide variety of use cases, with the leading applications currently being robotic warehouse picking.

The Complexity of Evaluating New AI Solutions

Technology innovation such as AI Robotics is a catalyst for meaningful transformation – enabling organizations to adapt, move forward, and differentiate. But it also requires a new and different evaluation and buying process.

In every conversation with supply chain executives and technical leaders, there’s one question our team is always asked: “How do we know which is the best brain, i.e. AI, for my robots?”

Understandable question.

Assessing baseline AI capabilities and comparing that across offerings is impossible relying on 90-second marketing videos and product data sheets. That’s because the key factor to consider when evaluating these systems is the AI’s ability to handle the scale and change over time of your own unique operations.

The Need for Performance-Based Assessments

There is simply no credible way to understand the critical aspects of AI capabilities without a rigorous assessment designed to mimic the scale, complexity, and change of your operations.

Through the nearly one hundred technology assessments we’ve participated in, our team’s recommendation is a performance-based assessment.

Economics drives the necessity of performance-based assessment. Here’s why.

A robot achieving 90% success – or one problem every ten times - requires operator intervention approximately once every 72 seconds. Effectively, that means you need a full-time person to oversee that one robot. 90% success, which is often an acceptable business outcome, actually renders a robotic station economically impractical.

Autonomy is defined by a robotic system achieving levels equivalent to manual performance, or 99%+ success rates. In dynamic and changing operations, achieving that level of autonomy is a significant AI challenge to achieve and maintain.

Underlying autonomy is three core AI capabilities that must be evaluated in any assessment:

  • Out-of-the-box performance. “Out-of-the-box” or "native" ability. Success the first time a new item or unknown scene is encountered.
  • Learning speed. The time required to adapt to and master new experiences, objects, or scenarios.
  • Learning potential. Ability to master new experiences. In some cases, the complexity, change, and scale are simply out of reach of the AI. I can spend the rest of my life training to dunk a basketball, but...

A well-designed and executed performance-based assessment – an AI Benchmark – tests for all three of these AI capabilities in the context of your own operations.

How to Design, Execute and Analyze an AI Benchmark

A well-executed AI Benchmark provides a performance-based approach to evaluating AI capabilities across multiple providers in an efficient and reliable way.

To learn more about designing, executing, and analyzing an objective AI Benchmark that enables you to effectively evaluate mission-critical AI capabilities, download the How to Run an AI Benchmark for Robotic Warehouse Picking guide by filling out the form below.

Download the AI Benchmark Guide