Constructing a computer vision system for aligning the contents of packages by a delta manipulator in food production

The article focuses on using computer vision to automate the process of aligning the contents of food packages in cartons on a conveyor line. This process is necessary to prevent the appearance of loosely closed packages due to uneven distribution of the contents inside the package. The research object is a combination of a stereo vision system and artificial intelligence algorithms that solve the problem of detecting and positioning an object in a 2D image for alignment with a delta manipulator designed specifically for the task of automating the process of aligning the package contents. The authors have obtained the following results: 1) For the positioning problem, a real-time object detection algorithm was implemented in the image; it broadcasts the detected 2D image sections to the stereo system forming partial depth maps. As a result, a fast and accurate algorithm for obtaining the 3D coordinates of an object was obtained. 2) To determine the current state of the package contents and optimize the number of alignment operations, the authors have implemented a binary classifier that informs the system by two types of statuses: “an object is aligned”, “an object is not aligned”. 3) For this task, the authors have collected a special data set consisting of a sequence of images with a frequency of 30 frames per second during 1 hour. Food contents were marked with rectangular boxes in each image and their state was noted, whether the package was aligned or not. Labeling was done using the open source LabelImg program, which provides a graphical interface for labeling images used in further training. 4) There was a program developed that implements the proposed algorithm in Python 3.6 using the Jupyter Lab integrated environment in Ubuntu 18.04 operating system. The paper presents the results of an experiment on using the proposed algorithm for estimating the 2D position of an object and the current state of the detected object. To evaluate the algorithm quality, the authors used the metric of the mean average precision (mAP) of object detection, as well as classification metrics - accuracy and recall for the task of determining the status of the package contents.