The Need for Performance Based Assessments

From eCommerce to apparel and grocery, organizations are urgently looking to maximize warehouse utilization while reducing cost. The COVID-19 pandemic has intensified these strains with an added concern about operational resilience.

Consequently, many leaders are moving rapidly to adopt AI-enabled robotic automation within their warehouses. 

Robots that can learn and adapt - AI Robotics - have emerged as a powerful supply chain solution. AI Robotics can unlock a previously impossible level of performance and efficiency for a wide variety of use cases, with the leading applications currently being robotic warehouse picking. 

The Complexity of Evaluating New AI Solutions 

Technology innovation such as AI Robotics is a catalyst for meaningful transformation -- enabling organizations to adapt, move forward and differentiate. But it also requires a new and different evaluation and buying process.

In every conversation with supply chain executives and technical leaders, there’s one question our team is always asked: “How do we know which is the best brain - i.e. AI - for my robots?”

Understandable question. 

Assessing baseline AI capabilities and comparing that across offerings is impossible relying on 90-second marketing videos and product data sheets. That’s because the key factor to consider when evaluating these systems is the AI’s ability to handle the scale and change over time of your own unique operations.

The Need for Performance Based Assessments

There is simply no credible way to understand the critical aspects of AI capabilities without a rigorous assessment designed to mimic the scale, complexity, and change of your operations.

Through the nearly one hundred technology assessments we’ve participated in, our team’s recommendation is a performance based assessment.

Economics drive the necessity of performance based assessment. Here’s why. 

A robot achieving 90% success - or one problem every ten times - requires operator intervention approximately once every 72 seconds. Effectively, that means you need a full time person to oversee that one robot. 90% success, which is often an acceptable business outcome, actually renders a robotic station economically impractical.  

Autonomy is defined by a robotic system achieving levels equivalent to manual performance, or 99%+ success rates. In dynamic and changing operations, achieving that level of autonomy is a significant AI challenge to achieve and maintain. 

Underlying autonomy are three core AI capabilities that must be evaluated in any assessment:

  • Out of the box performance. “Out of the box” or "native" ability. Success the first time a new item or unknown scene is encountered.

  • Learning speed. The time required to adapt to and master new experiences, objects or scenarios.

  • Learning potential. Ability to master new experiences. In some cases, the complexity, change and scale is simply out of reach of the AI. I can spend the rest of my life training to dunk a basketball, but....

A well designed and executed performance based assessment - an AI Benchmark - tests for all three of these AI capabilities in the context of your own operations. 

How to Design, Execute and Analyze an AI Benchmark

A well executed AI Benchmark provides a performance based approach to evaluating AI capabilities across multiple providers in an efficient and reliable way.

To learn more about designing, executing and analyzing an objective AI Benchmark that enables you to effectively evaluate mission critical AI capabilities, download How to Run an AI Benchmark for Robotic Warehouse Picking.

Download PDF