Covariant Research

Background of research at Covariant

Covariant was founded on research that fundamentally changed the world of AI. Today we continue to perform active research on new and novel approaches that we apply to solve the challenges that our robotic systems see in warehouses and fulfillment centers around the world. What normally can take 5-8 years to make it from the lab to the real world, we do in a matter of weeks and months. Learn more about our research here.

While most of our R&D efforts are kept private, giving us a leading edge in providing autonomous capabilities, we have decided to also start sharing a small fraction of our advances for public consumption.

We will showcase how AI-powered trajectory optimization increases the speed and reliability of robotic object manipulation. This work was developed by Haoran Tang (me), Peter (Xi) Chen, Yan Duan, Nikhil Mishra, Philipp Wu, Maximilian Sieb, and Yide Shentu. See the pending patent for reference: Trajectory optimization using neural networks.

Fast and reliable

Fast AND reliable

When automating your warehouses with robotic arms, motion speed is the key to efficiency. While industrial robots are fast by design, object manipulation poses further challenges. A trajectory should be both fast and reliable, i.e., protecting the object from collision and maintaining a stable grasp.

Moving too fast could cause the object to drop
Moving too fast could cause the object to drop

Unreliable trajectories cost expensive operational downtime, as typically, a human needs to intervene. For example, walk to the station, pause the robot, pick up the dropped object, and restart the system. The process can cost minutes.

Handcrafted motions

Handcrafted motions - inflexible and suboptimal

Traditionally, engineers design fixed robot motions through ad-hoc heuristics and trial-and-error. However, it cannot handle the exponential variation in real-world logistics. Objects are diverse in geometry, weight, deformation, packaging, and even detachability! Grasp points differ in poses and cup selections. The start and end locations change dynamically. It would be simply impossible to handcraft all millions of cases.

Rapid motion causes the box lid to detach, resulting in a collision
Rapid motion causes the box lid to detach, resulting in a collision
Grasping the lid is unstable, which must be accounted for when moving the object to its destination
Grasping the lid is unstable, which must be accounted for when moving the object to its destination

Perhaps more surprisingly, humans don’t always design the best trajectories. A poorly designed one could cause an unnecessary slowdown without anyone noticing, until a mathematically optimized solution reveals the truth.

Handcrafted trajectory (left) vs. mathematically-optimized trajectory (right)
Handcrafted trajectory (left) vs. mathematically-optimized trajectory (right)
Modern approach

Modern approach — AI-powered trajectory optimization

Trajectory optimization can be formulated as constrained optimization - minimizing the total motion time, subject to collision, suction stability, and other constraints. Despite many existing methods, none of them tackle two important problems in production: (1) fast (millisecond-level) inference; (2) accurate suction-object dynamics. We defer (2) to a later blog post. For now, let’s assume a good physics model and focus on (1).

Handcrafted trajectory (left) vs. AI-powered trajectory optimization (right)
Handcrafted trajectory (left) vs. AI-powered trajectory optimization (right)
Robot adjusts its motion and path based on the object characteristics and grasp quality, taking into account the physical boundaries
Robot adjusts its motion and path based on the object characteristics and grasp quality, taking into account the physical boundaries

Traditional optimizers need several seconds to find the solution. They are too slow for fast-paced warehouses, where the speed requirement can be over 1,000 PPH (picks per hour). Our solution is behavioral cloning (BC), i.e., learning the optimal solutions with a neural network. In addition, we further finetune the network on the original objectives with reinforcement learning (RL). The resulting network generates fast and successful trajectories within merely 10ms. Interestingly, this BC + RL paradigm also powers many of the biggest AI breakthroughs, such as AlphaGo and ChatGPT.

Powering warehouses

Power your warehouses with agile and smart robots

When we deployed trajectory optimization on our robotic putwall solution, it improved PPH by 10~15% and reduced the object drop rate (one of the reliability metrics) by 1~5%. It is an important part of the that makes our deployments successful.

Robot adjusting its path, speed, and acceleration to successfully pick and place objects into the destination cubbies
AI-first research

Continuous innovation with AI-first research

Driven by the need to solve real-world problems seen in the deployment of piece-picking robots in warehouses, the Covariant research and engineering teams continue to invent new methodologies and approaches. This directly impacts our customers in optimizing their warehouse operations to drive higher efficiency and lower costs.

Learn more about our patent-pending research on trajectory optimization using neural networks.

Related RFM-1 update: In-context learning to improve grasping