ExRealis
ExRealis becomes a pole of the GAIA ICube Platform (Graphics, Artificial Intelligence and data Analysis)
ExRealis - A digitization framework
In many industrial sectors, an increasing demand for tools intended to the production of digital content can be observed, either for video game or movie industry, simulation applications or rapid prototyping. Nowadays, many off-the-shelf 3D scanners are available, whose metrological accuracy guarantees reliable acquisitions of real objects and the creation of faithful digital surrogates. This partially leads to turn the digital content production task, so far exclusively granted to computer graphics artists, into a technical act of data gathering, often cheaper.
Despite the improvement of shape measurement technologies, processing measured data in order to create digital content directly usable by computer graphics applications still requires many intermediate tasks. Moreover, for specific fields like cultural heritage and artistic domains, shape only is far from being sufficient: measuring the appearance is of major importance too for the display of realistic-looking digital models.
The goal of ExRealis is to provide a unified framework that answers in an efficient manner to all of these needs. It offers several tools, devices and softwares that give the possibility to produce digital content of various nature from real data, with the purpose of supporting research works conducted by the IGG team around themes like realistic rendering and texture synthesis, as well as enabling service provisions in digitization with our partners. ExRealis proposes:
- a range of devices and software tools covering the whole digitization processing pipeline, from data acquisition to the creation of textured 3D models;
- some advanced methods for texture reconstruction and visualization accounting for complex lighting environments or material characteristics.
Our devices
The ExRealis framework contains several devices, detailed below, that enable the measurement of geometry and colour from real scenes or real objects.
Short range scanner
Medium range scanner
Photographic equipment
Short range scanner
This optical device based on structured light allows the 3D scanning of objects whose size ranges from 20cm to 1m. Depth measurement ranges from 80 to 120cm and the size of the produced range images is of 1280x960 pixels, which leads to an average resolution of 600 to 700 points per square cm.
The controlling software has entirely been developed by our team, which thus offers a high degree of flexibility and the possibility of a complete reconfiguration, so as to adjust, for instance, the device resolution to specific needs.
Medium range scanner
The time of flight laser scanner ScanStation2 from Leica Geosystems allows depth measurement ranging from 0.2 to 300m, at a speed rate peaking to 50 000 points/sec. Moreover, a set of targets provided by the manufacturer enables the registration of point clouds acquired from different locations.
Please see the technical documentation from Leica Geosystems for more details.
Photographic equipment
Our team also has at its disposal a semi-professional photographic equipment, including two cameras Canon EOS 5D Mark ii as well as lenses ranging from 25mm to 125mm. We use it for the acquisition of real object appearance (colour, texture), or for photogrammetry based 3D scanning.
Our software solution
ExRealis proposes a software for the processing of scanning data. We manage a continuous development process in order to integrate every tool which is useful to the creation of realistic digital models from physical real objects. The algorithms proposed allow the processing of geometry (reconstruction of 3D meshes from scanning device data) as well as appearance (colour or texture synthesis over those meshes from pictures).
Dedicated to a practical usage, this software has been designed in a modular way, with a constant care for ergonomics. It is intended to manage huge amounts of data, thanks to out-of-core data structures and mechanisms, and most of its algorithms are optimized thanks to parallel or GPU programming.
Geometry acquisition and processing
After a 3D scanning session (namely the data gathering by itself), many processings have to be applied to the raw data produced by the scanner before obtaining a usable 3D model.
Our software covers every step of this processing pipeline, thanks either to fully automatic algorithmic solutions, or to a suitable user interface whenever user intervention is needed, like in the case of some of the cleaning tasks, for instance. We are then in position to produce, from real objects, 3D models that are correct in terms of geometry and topology, at different level of details, and that can be exported to many file formats compatible with off-the-shelf CAD softwares.
|
Geometry registration. Devices requiring multiple 3D acquisitions from different viewpoints (structured light range scanners, lidars, etc) provide for each acquisition an independent point cloud, which needs to be realigned with the others, similarly to a big 3D puzzle. We developed an algorithm that enabled us to automate this process for the specific case of structured light range scanners, but ExRealis also provides manual registration thanks to a dedicated user interface, as well as registration refinement by iterative minimization algorithms. |
|
Cleaning. Since no scanning technology is now noise free, it is not uncommon to recover erroneous measured points, which risk to disturb the following processing. Here again, a dedicated user interface enables manual cleaning. However, this is often a tedious task. That’s why heuristics have been implemented in order to automate this process, using confidence values computed from geometric or colorimetric criteria. |
|
Surface reconstruction. Point clouds provided by shape measurement technologies constitute a representation which is not adapted to many usages to which 3D models are intended (realistic rendering, physical simulation, etc). It is thus often necessary to obtain a closed description of the surface, in the form of a 3D mesh. To do so, we integrated to ExRealis some methods derived from scientific literature to extract triangulated surfaces from sets of measured points. |
|
Surface simplification. To display 3D models on devices with low graphical performances (smartphones, tablets, autonomous virtual reality head mounted devices), it is important to be able to reduce geometry complexity. ExRealis incorporates different algorithms allowing this reduction while limiting the loss of geometry details, so as to preserve as most as possible the object shape during the simplification step. |
ExRealis also provides many other tools for the processing or the improvement of geometric data: re-sampling of point clouds, surface smoothing, hole filling, distance and volume measurement, etc.
Appearance acquisition and processing
Acquiring the appearance of a real object starts from a photographic campaign, so as to measure its photometric properties. Reconstructing a texture from these pictures then requires to solve the following key points:
|
Surface parametrisation. Giving an appearance to a 3D object is classically made through the use of textures, which are images that may contain various informations (albedo, reflectivity, roughness, details, and so on). To make each portion of the surface corresponding to an image part, the 3D object first need to be unfolded and flattened onto a plane, similarly to the earth globe onto a planisphere. Several algorithms to solve this so-called parametrisation step are available in the software. |
|
Photography registration. To reconstruct the colour of a model from pictures, a first step consists in recovering the viewpoints from which they have been shot, so as to be able to project their content onto the 3D model, similarly to what a video-projector would do. Camera calibration methods have then been integrated to ExRealis for this purpose, as a preliminary step to appearance reconstruction. |
|
Colour map reconstruction. The use of multiple pictures for colour map reconstruction requires to manage adequately the surface areas where several images are projected. Indeed, in these overlapping regions, a naive blending may result in visual artefacts. ExRealis provides more elaborated blending strategies based on visibility criteria which allow, for instance, to reduce the impact or reprojection errors or to eliminate the focus blur that may appear in some of the source images. |
|
Normal map reconstruction. Beyond a certain threshold, surface simplification may lead to the loss of geometry details. By storing those details into a normal map, it thus becomes possible to modify illumination during the rendering, in order to simulate geometry complexity over simplified 3D meshes. This approach is much cheaper than directly displaying highly detailed meshes. ExRealis allows the reconstruction of normal maps for coarse meshes from more detailed versions of those. On the image showed here, the model on the left is made of 10K triangles, the one in the middle of 4M triangles, and the one on the right of 10K triangles + a normal map to store the details. |
The produced textures may consist either in a simple colour information, or in more advanced representation models that give not only a clue on the object's hue, but also on other properties of its material, like shininess, for instance.
The IGG team has worked on the fitting and the real-time rendering of such models, considering especially light fields. Light fields capture the appearance of an object within a given lighting environment: the one corresponding to the acquisition moment. This includes, among other things, all illumination effets related to the observer's displacement around the object (specular peaks, inter-reflections, etc.) and allows afterwards the free inspection of the digital copy with the same lighting conditions, while maintaining a high degree or realism with respect to more basic texture models.
Our expertise
Working on 3D + appearance digitization since 2005, the IGG team has acquired a good expertise of this scientific field and a good knowledge on how to apply it to the practical cases of digitization campaigns. Within the framework of various collaborations, we had many occasions to apply these expertise and knowledge through service provisions, as illustrated below.
The Voodoo Museum of Strasbourg
The company Virtual Journey has launched a project to propose to the visitors of the Voodoo Musem in Strasbourg to live the peculiar rituals related to this secret religion still unknown in Europe through the use of a virtual reality head mounted device. The experience allows to be immersed at the heart of ceremonies thanks to 360° 3D videos shot in Benin, as well as to bring to life some of the museum artefacts so as to explain their usage by means of digital copies derived from 3D scanning. Our team has been involved in this project for that purpose.
We thus performed a photogrammetry scanning campaign during two and a half days so as to digitize about fifteen artefacts in relation to the presented rituals. After processing in lab, in particular with the ExRealis software that we develop, digital copies have been delivered to Virtual Journey for integration into its virtual reality application.
Officially deployed in October 2019, this project has been welcomed in a very enthusiastic way by visitors and by the press, which has offered us a fantastic media coverage:
- article and reportage on the website Alsace20.
- article in 20 Minutes.
- article in DNA.
- article on the blog Pokaa.
- presentation at the digital festival Bizz&Buzz.
Œuvre Notre Dame foundation
Within the framework of the Eveil3D project, and in partnership with the technology transfer center Holo3 and the Œuvre Notre Dame foundation, the IGG team has performed in october 2013 the digitization of two statues of the cathedral of Strasbourg. The goal was to use these statues in an immersive 3D environment for a language learning serious game. About 50cm high, they've been digitized with our short range structured light scanner. A photographic campaign has also been done so as to produce colour textures for both models.
Inter-university House of Human Sciences (MISHA), Strasbourg
MISHA has at its disposal a huge collection of ethnographic artefacts, and has launched some years ago a project based on the OpenSIM game engine that provides a virtual world to make it possible the study and the contextualization of these artefacts. Obviously, this requires to have digital copies of them, and so to have digitized them beforehand.
For this purpose, they contacted us in autumn 2013. They needed to produce digital copies of about ten african ethnographic objects from the Dogon tribe. Their geometry has been digitized with the short range structured light scanner, and their colour thanks to a photographic campaign.
Bois l'Abbé fortifications
Below are presented some renderings of a digitization we made of a part of the Bois l'Abbé fortifications (48°12'16.3"N 6°24'00.8"E), located at Uxegney near Epinal, France. The final model is made of about 63M points acquired with the Leica Scanstation 2 scanner from 20 different locations. Colour assigned to the points comes from the internal scanner camera, dedicated to data handling conveniency but not to a faithful appearance acquisition.
Gypsothèque, university of Strasbourg
We had the occasion to get in touch with the Gypsothèque of the Strasbourg university (plaster copy museum of cultural artefacts) to perform the digitization of one of their statues.
Some digitizations made at home
Here are some models digitized in our lab in order to produce data sets for illustrating the research works of our team. Some of these 3D models are provided with light field textures, that enable to simulate illumination variations related to the observer's displacement for more realism, as explained before. It has to be noted that basic colour textures can also be exported using standard image file formats.
Elephant
Geometry: 4.9M triangles
Texture: light field + colour
Dragon
Geometry: 18.2M triangles
Texture: light field + colour
Mask1
Geometry: 7.7M triangles
Texture: light field + colour
Mask2
Geometry: 8.7M triangles
Texture: colour
Venus at the bath
Geometry: 3.6M triangles
No texture
Collaborations
Several national and regional projects are supporting this framework:
- Projet Interreg EVEIL3D,
- ANR ATROCO,
- Ministère RIAM AMI3D,
- Région Pôle Image.
Contacts
If you have needs either for service provisions or for technology transfer, or if you are just interested by one of the models presented in this page, please contact Frédéric Larue, research engineer in charge of the ExRealis framework.