VISIRE Project IST-1999-10756 (2000-2003).
Keywords: EC V Framework. IST. Computer Vision. 3D. Autocalibration. Stereoscopy. 

Rationale

The objective of the current project is to develop an innovative three-dimensional Computer Vision based simulation environment for mass-market applications.

The main objective of the project is then to develop new 3D computer vision acquisition techniques and tools that makes possible the construction of realistic 3D scenarios for a great variety of applications; including simulations, multimedia, TV broadcast, etc. The intention is to achieve new levels of realism in building scenarios for computer simulations and new possibilities of man machine interaction for a fully integrated environment where synthetic models can be as close to the real world as one desires or maybe as far as imagination can think about.

One can easily imagine that such levels of performance can not be achieved with traditional techniques. Modelling of complex 3D environments from the synthesis point of view has been lagged by technology shortcomings that prevented its widespread use due to cost and limitations on the realistic quality of the results. While every small improvement on the rendering algorithms is automatically traduced in exponential increments in computing power and rendering time, cost and man hours required even for the most simple simulations makes the technology only valid for very selective applications.

The present project on the other hand, will base its approach on novel Computer Vision technologies that will not only acquire the 3D geometry information directly from the world scenario but will also map the required textures. In fact Computer Vision will become the core of development activities in the project.

The building procedure is also very important for a successful product. The 3D model assembly scheme can not be so complex or time consuming that it makes the system unusable. For that reason complex calibration requirements or cumbersome acquisition procedures are banned in this type of applications. In fact the objective is to achieve image acquisition requirements as easy as taking a video shot around the scenario being reconstructed. The inherent complexity of the problem will be solved internally by sophisticated computer vision algorithms without the user being aware of such complexity.

The foreseen sequence of activities to be performed for a 3D reconstruction session is as follows:

• The user takes one or several video shots around the scenario he desires to reconstruct.
• The acquired images are taken to the laboratory where sophisticated 3D reconstruction tools builds the real-like synthetic 3D environment. Images from the video stream are analysed, the 3D geometry information of the objects appearing in the scenario is recovered and textures are mapped after appropriate geometry correction.
• As result of processing the video data, the 3D modelisation of the site is obtained.

The target application for the project will consist of the simulation of a famous museum, but the techniques to be developed within the project can also be applied with little extra effort to a great variety of applications like 3D simulations of buildings, archaeological sites, public works (bridges, motorways), topographic 3D reconstruction of terrain, tele-shopping services, etc.

The final result will consist of a synthetic real-like 3D world where any computer manipulation will be possible. The 3D model of the museum will be digital and will support all the possibilities of digitally built environments but it will also support real-like textures taken from the original images so that it will be difficult to notice it is not a real model.

The user will be able to navigate through the different parts of the museum, watch the exhibition cabinets, focus his attention on special details, choose his point of view, etc. It will even be possible to look from positions and angles not usually possible in real museums What about looking at the objects in the exhibition cabinets from inside the cabinet itself or watch the exhibition room from the ceiling for a better perspective. Almost anything imaginable is now possible.

In that sense, it will also be possible to modify the environment. It is feasible for example to change the aspect of the elements in the building of the museum, add synthetic characters to the scenario (maybe a friendly synthetic guide will help you on your visit), add or delete elements (either synthetic or real-like) in the decoration, add new objects to the contents of the exhibition, etc.

But perhaps the most interesting possibility lies in the so-called hot spots or sensible areas. By means of hot spots facilities the 3D environment can be directly linked with external events. Imagine for example coming to an exhibition cabinet, touching the door handle with your mouse and see how the door cabinet opens and show its contents. Or select an object in the exhibition and listen to the object's story or watch a video clip about the manufacturing process of some jewel. All at your disposal in your computer window.

In spite of being reiterative, it is important to mention that the 3D models created by the system are not complex man-made 3D models generated by graphics Cad tools. The models obtained by the system are the result of a sophisticated Computer Vision process on real video streams that reflects with reliability the real aspect of the environment to be reconstructed.

The models will not show computer-like appearance or machine-like lighting. They will not have the appearance of brand new objects usually found in traditional computer graphic simulations but will demonstrate the real texture obtained from the video images. This means the 3D-reconstruction process will show most trustworthy state of the objects being reconstructed including possible time degradation or broken parts.

Objectives List

The specific objectives of the present project can be summarised on the following items:

• It is necessary to build a computer vision 3D-reconstruction system based on video image streams.
• The 3D reconstruction will be based on self-calibration techniques. No calibration apparatus will be allowed.
• The 3D reconstruction process will be semi-automatic. An intuitive user interface must be provided for short building times.
• The system must be able to map textures obtained from images in the video stream.
• The system must support hot spot facilities. Hot spots will allow triggering of external events such as: audio, video, text and links to other 3D simulations.
• The system must feature an interactive user interface that allow 3D navigation on the scenario, zoom and hot spots.
• The 3D scenarios obtained by the system should be seamlessly integrated into most commercial 3D packages or applications thanks to the compliance of the system with the VRML 2.0 de-facto standard.
• The system must support either domestic cameras or professional camcorders.
• The system must be suitable for high end PC like personal computers.
• The system must be suitable for the following environments (suitability for the different environments will be assured by means of VRML compatibility. No specific development is foreseen concerning Internet or communications infrastructure): Internet, CD-ROM, Information desks, Set-top boxes.

Actually, it is possible to display VRML models in commercial Internet browsers, last generation set-top boxes and information desks. Of course VRML is also suitable for CD-ROMS and most popular 3D modelling packages.

Project Outcomes

The project is expected to provide the following results:

• The 3D reconstruction tools. A complete software environment that allows Multimedia professionals to build realistic 3D models of the interior of buildings. The tools use as input a set of video sequences and produces the finished 3D models in VRML format.
• The simulation of a famous museum in Florence. As part of the work foreseen in the project the 3D reconstruction tools will be tested by building the 3D simulation of a famous museum in Florence. The resulting multimedia production will be used for promotion and dissemination. It could be considered also the possibility to commercialise the finished model. Current alternatives for the museum include Museum of Science in Florence, Museum of Technology in Milan, Uffizi Gallery and Palazzo Vechio.
• New computer vision algorithms that will hopefully advance the state of the art in the field.