Unified Underwater Structure-from-Motion
Recently, demands on underwater 3D reconstruction are rapidly growing in the field of robotics, marine biology, sports science, and so on. Structure-from-Motion (SfM) is known to be a powerful technique for underwater 3D reconstruction, attracting wide attention in that it should assume non-centraol projection model considering light refraction on water surface.

On the other hand, underwater SfM research is biased toward the premise of geometry that flat refractive interface is fixed facing the front of camera. There have been few studies assuming other case where refractive interface moves or surface does not satisfy planarity (such as rippling water surface) (Table 1).
Table 1. Related work on underwater SfM
In this work, our goal is to represent such variety of scenarios with an unified model, and solve each scenarios as specialized cases of it.

Unified Underwater SfM Model

Figure 1. Unified Underwater SfM Model
Figure 1 shows the unified underwater SfM model we propose in this work. We define a geometry where underwater scene is static while refractive interface and camera move freely and separately, and it is also equivalent where underwater scene moves. Although the refractive interface is planar, it is assumed that the normal of the surface is different for each position in order to explain large deviation of the projected position due to refraction.

All the scenarios on Table 1 can be represented as specialized cases of this model. For example, it is equivalent to general underwater SfM geometry assumption when distance between camera and refractive interface is constant, and all the normal on refractive interface is facing the front of camera.

Optimization Algorithm

In many optimization algorithms for 3D reconstruction such as bundle adjustment, generally reprojection error by forward projection is minimized. However, analytical forward projection is not possible in this model since refractive interface has different normal for each position (incontinuous). Instead, we use a constraint that muptiple light rays projected from each 2D points on image intersect at certain 3D point.
Figure 2. Hard and soft normal constraints on refractive interface
As a specialized case of this model, assuming that the refractive interface is a perfect plane without individual normals, it can be optimized with forward projection in the same way as normal bundle adjustment (Figure 2 Left). When complete planarity of refractive interface cannot be assumed, the system will degenerate as any ray can be explained with non-constrained normals, thus reconstruction will not success. Therefore, smoothness constraint is introduced which assume nearby normals have close directions each other.

Underwater SfM Reconstruction Results

Followings are reconstruction results obtained from our unified underwater SfM model and optimization algorithm.
Figure 3. Reconstruction of underwater mannequin (a scenario with moving flat refractive interface, fixed camera and scene)

Figure 4. Reconstruction of stacked desk in the pool equipped with wave generator (a scenario with moving camera, dynamic refractive interface and fixed scene)

Publications
Kawasaki Laboratory