WP3 Flexible Processing Automation

Work Package Leader Dr. Ekrem Misimi, SINTEF Ocean (SO)

Participating partners SFA, NMBU, SRM, KU Leuven, INRIA, Dynatec, Food Processing Industry partners


To develop innovative processing concepts that are:
a) flexible to cope with small production volume series; adaptive to biological variation of raw material;
b) flexible to cope with market demand fluctuations and production supply;
c) effective in use of the existing raw material leading to minimization of food loss/waste;
d) increase automation degree for higher profitability.


T3.1 Multimodal machine vision for raw material analysis:
Use of multimodal 3D machine vision in multiple optical ranges and image processing algorithms for accurate identification, recognition and localization of raw material features from sufficient number of raw material images in multispectral bands.

T3.2: Model- and 3D vision recognition of compliant objects:
3D CAD modeling of the anatomy of raw material (e.g. chicken, fish, vegetable) by X-ray CT imaging and image processing. Use of the 3D CAD models with 3D vision measurements for adapted geometrical/anatomical internal and external characterization of individual items. In the development of concepts for adaptive cutting/processing, anatomical models are needed so that e.g. the borderline between meat and bone is accurately determined leading thus to maximum raw material utilization.

T3.3 3D vision for localization, recognition of raw material and dense visual servoing based on depth maps and 3D point clouds:
a) Development of novel methods for automatic localization of raw material objects in unstructured 3D environment by using e.g. Kinect v2 camera that combines visual (RGB-color) and 3D (depth) shape information features;
b) Automated visual servoing (guidance) of robot arms based on visual depth maps and dense 3D point clouds generated from RGB-D (Kinect v2) camera.

T3.4 Optimal machine learning algorithms for recognition and localization of compliant objects from 3D images in T3.3:
Vision concepts (eye) developed in previous tasks need optimal algorithms (brain) to interpret and understand the data. Robust machine learning algorithms such as Support Vector Classifier (SVM), Deep Learning Neural Nets implemented in GPU processors specifically tailored for food processing applications will be implemented to process the 3D visual data for localization of raw material objects/features.

T3.5 Multi-functional robotic end-effector tools for processing of food raw material:
Development of multipurpose end-effector (gripper/cutting) tools (hand) mounted on robot arm for processing of raw material.

T3.6 Adaptive trajectory generation and robot imitation learning for flexible processing of raw material and subsystem integration:
a) Developing methods for adaptive trajectory generation for processing (for example cutting), optimized for accuracy and speed based on the 3D images and anatomical 3D CAD models;
b) Teaching robot by demonstration (mimicking human motion patterns e.g. during cutting operations) to enable flexible automated processing of raw material;
c) integration of all subsystems developedin previous tasks into one flexible processing system to adaptively process food raw material for the selected cases.

Interrelation with other WPs:

Close interrelation with WP1 and WP2, and WP4. Concepts will be evaluated in WP5 regarding economic and environmental sustainability.


M3.1: Lab models of 3D machine vision system for characterization and 3D localization of raw material finished (Q8).
M3.2: Integration of sensors, tools and robot for selected cases completed (Q12).
M3.3: Robust 3D localization concepts based on 3D vision completed (Q12).
M3.4: 3D CAD anatomical models completed (Q10).
M3.5:Dense 3D visual servoing and adaptive trajectory generation for selected cases completed (Q14).


D3.1: Conceptsfor raw material characterization based on multimodal vision
D3.2: 3D recognition of raw materialobjects as a combination of 3D CAD anatomical models and 3D vision.
D3.3: Multifunctional grippers for an adaptive handling/processing of raw material.
D3.4: Flexible concepts for processing of raw material based on adaptive robot trajectory generation and visual servoing for selected cases.
D3.5: PhD thesis.
D3.6: 5 manuscripts submitted to journals.
D3.7: 2-3 MSc theses.