Virtual Image Processing – Paradigm Shift Through Simulation

Factories are becoming increasingly automated. Production facilities are becoming more flexible, so that no new plants have to be built when switching to new products. Visual inspection system hardware configuration – the last nail preventing the inspection systems to be truly purpose-flexible and thus ready for the implementation as a part of Industry 4.0 process chain. Production lines are more and more versatile, and products are changing rapidly, confronting inspection systems with more complex surfaces and materials. Every step of the production is controlled and digitalized to be as flexible as possible. And yet, when it comes to inspection, months of pre study are required, and no off-the shelf solution is available which can be easily adapted to different use cases and surfaces of different complexity.

Making Processes More Flexible & Efficient

However, quality control is often neglected. Inspection systems are still rigid and have to be designed for specific products. An inspection system consists of many hardware components, typically selected and parameterized by experienced engineers on the basis of physical tests. New systems are developed iteratively. Experts design an initial system, which is then modified until it can inspect the product with sufficient accuracy. These tests of different hardware solutions cost a lot of time and effort – several hours per test run. Therefore, a configuration is often chosen that works but is not optimal. The resulting sub-optimal image quality must be algorithmically compensated later.

To make this process more flexible and efficient, we are developing an adaptive, simulation-based framework that will revolutionize the development process for inspection systems. In the future, industrial inspection systems will be completely virtual designed and tested for reliability using this framework.

Close-up of the pivot points for the camera viewpoint candidates. A few points cover flat areas, while more points are generated in strongly curved regions.

Framework Development

The Virtual Image Processing research group at the Fraunhofer Institute for Industrial Mathematics ITWM is set on changing the paradigm by developing a modular framework fully capable of planning the acquisition requirements to completely inspect any given product. Using computer vision, computer graphics, machine learning and robotics it is possible to develop a framework offering tools for design optimization, allowing the assumption of a flexible image acquisition setup. Currently, very little or no research is focused on inspection system design and optimization.

A virtual image processing framework can overcome this gap, by thoroughly testing the acquisition hardware of choice and simulating the result. Most importantly, it makes optimization of component positioning possible, without requiring the engineer to remount the equipment repeatedly. Furthermore, computer vision algorithms can be developed and tested on simulated images, along with the acquired ones, overcoming a frequent problem of defect sample acquisition. Such problems are often found in industries where defects occur rarely but are critical when they do – airplane blisks (Blade Integrated Disk) and car brakes are two such examples.

Virtualization Core in Focus

The key to virtual image processing lies in the virtualization core, consisting of two interconnected components: planning and simulation. Simulating what the camera sees can be used to evaluate the design plan of an inspection system.

The core is fed by a CAD model – the geometry – of a product, along with different inspection parameters, for example the types of defects, product material, and inspection speed. Based on these parameters, the core will output a set of possible solutions and parameters, which an engineer can then use to adapt an inspection system, as well as the expected results, for example sensing viewpoints, light positions, and simulated inspection images.

The framework is currently being researched and developed on several fronts in parallel:

  • Parametric surface estimation
  • Active model-based position planning and optimization
  • Camera lens modelling
  • Position-based defect augmentation
  • Surface light response modelling

The focus is primarily on position planning ̶, the backbone of the overall system. This can then be extended modularly and supplemented by new functionality in order to be adapted to product-specific requirements.

For more information: