Researchers Thiemo Alldieck, Mihai Zanfir and Cristian Sminchisescu from Google Research are developing a neural network for scanning 2D images into usable 3D objects.
The project is called PHORHUM, is specifically focused on scanning people — just from a frontally shot image, the AI system can create usable 3D models. Points that are not visible in the image — such as the back of the person — are automatically added.
Apart from that, the AI generates light reflections and colors from the pixels, which can then be placed as a texture on a figure. Two different networks are used for this. A feature extractor generates distance values and points within a given 3D space. A shading network then calculates shading and color values based on the illumination and normal vector of all points.
According to the researchers, PHORHUM is, therefore, the only neural network that outputs brightness values and shades in addition to colors and surface normals instead of integrating them firmly and as unchangeable constants.
Google shows the first results in a presentation video. The system also seems to work well on non-visible areas, such as the backs of the models.
However, a constructed model visibly loses detail there. With further research, however, such models could eventually be useful in the gaming industry, for example, when concept art needs to be turned into a 3D model. Until now, this has been a manual and time-consuming process.
