projects people publications

Illumination Modeling for Computer Vision & Computer Graphics

We are researching problems related with the interaction of illumination and 3D shape in images for Computer Vision (shape estimation, tracking, recognition) and Computer Graphics (image relighting, augmented reality), in which object appearance is significantly influenced by the illumination conditions of the scene. Optical images are formed by measurements of the light that reflects from surfaces of various materials and orientations in 3D space. Effective approaches to numerous problems related to visual appearance, from photo-realistic image synthesis, to convincing Augmented Reality environments to object recognition, rely on the accuracy and efficiency of models of visual appearance of objects, thus motivating the study of the image formation process. In addition to knowing the geometry of the scene and of the cameras, we also need to model the reflectance properties of the surfaces being imaged and the configuration of illumination sources in the scene. In the general case it is impossible to determine simultaneously shape, reflectance and illumination without having any prior knowledge of the scene, so we have been working on a number of subproblems where different priors are used.

We have also developed novel solutions for 3D Shape estimation from shading from single images and stereo sets, multiple light source estimation, visual tracking of deformable objects with little texture, and face recognition under arbitrary illumination.

In Dr. Samaras's Ph.D. thesis, he proposed a method for the incorporation of any type of illumination constraints in deformable model frameworks. If the light source direction is unknown, both light source and object shape can be recovered, in . The applicability of the above methods was extended with the integration of shape from shading (SFS) and stereo, for non-constant albedo and non-uniformly Lambertian surfaces. In the case of moving objects, optical flow becomes part of a generalized illumination constraint, which was used for visual tracking of deformable objects with little texture.

Often it is impossible to know exactly either the reflectance properties or the shape of an object, but we can have a statistical prior for them, especially nowadays that image databases are ubiquitous. This led us to study the statistical properties of shape and illumination for the case of particular object categories. When applied to faces this allows for significant improvements to face recognition technology. More details can be found in the Face Recognition section.

Multiple Directional Illuminant Estimation from a Single Image

[publications]

We present a new method for the detection and estimation of multiple directional illuminants, using only one single image of an object of arbitrary known geometry. The surface is not assumed to be pure Lambertian, instead, it can have both Lambertian and specular properties. We propose a novel methodology that integrates information from shadows, shading and specularities in the presence of strong directional sources of illumination, even when significant non-directional sources exist in the scene. Since the specular spots have much sharper intensity changes than the Lambertian part, we can locate them in the image of the sphere by first down-sampling the image and then applying a region growing algorithm. Once the specularities have been roughly segmented, the remaining regions of the image are mostly Lambertian, and can be segmented into regions, with each region illuminated by a different set of sources, in a robust way. The regions are separated by boundaries consisting of critical points (points where one illuminant is perpendicular to the normal). Our region-based recursive least-squares method is impervious to noise and missing data. The illuminant estimation can be further refined by making use of shadow information when available, in a novel, integrated single-pass estimation. The method is generalized to objects of arbitrary known geometry, by mapping their normals to a sphere. Furthermore, we introduce a hybrid approach that combines our method with spherical harmonic representations of non-directional light sources, when such sources are present in the scene. We demonstrate experimentally the accuracy of our method, both in detecting the number of light sources and in estimating their directions, by testing on synthetic and real images.

Publications