Refraction-free image synthesis using light field camera
Recently, there is a growing demand on underwater 3D reconstruction in the field of underwater robot vision, sports science, and so on. However, cameras are usually contained in the waterproof housing in underwater measurement, which causes refraction on the surface and makes the system non-central projection. Thus, many 3D reconstruction techniques assuming central projection cannot be applied directly. Therefore, we propose an algorithm to synthesize refraction-free image, where refraction effect is removed, from underwater image captured by light field camera. Because light field camera can collect and decompose non-central projection rays unlike normal camera, it is possible to synthesize refraction-free image by computing the rays bent by refraction and moving their projection points. Since the proposed algorithm is irrelavant to the following 3D reconstruction algorithm, various techniques can be applied such as passive stereo, Structured-Light, and so on.

Refraction-free image synthesis using light field camera

Algorithm for refraction-free image synthesis using light field camera

Light field camera can be modeled with main lens, Micro Lens Array (MLA), and image plane, as shown in the figure. We assume main lens follows Thin-lens model, and MLA follows central-projection model. In addition, there is a refractive interface in front of the camera, and the transformation is assumed to be calibrated. When a ray shown as a red line is observed, the ray reaches from a 3D point in the water, and refracts on the refractive interface. If we assume the red ray is central-projection, a ray shown as an orange solid line must be observed. However, actually the orange line refracts and observed at the another point on the image plane shown as orange dotted lines. Therefore, we can synthesize refraction-free image by computing the projection point geometrically, and moving the pixel to the projection point of the red ray.
In practice, it is impossible to decompose the orange ray which does not pass through the center of MLA, instead we use the closest ray which passes through the center of MLA, shown as a green ray. When selecting the closest ray, we define a metric considering ray direction and passing point on the refractive interface, and select a ray which minimize the metric.
We show examples of refraction-free images by the proposed algorithm.
Refraction-free images synthesized by the proposed algorithm

Application to underwater 3D reconstrucion

Since the proposed algorithm is irrelavant to the following 3D reconstruction algorithm, various techniques can be applied. We show the examples of application of passive stereo and Structured-Light.
Underwater 3D reconstruction by the poposed algorithm and passive stereo (Left: Captured images, Middle: Conventional method using approximation, Right: Porposed algorithm)

Underwater 3D reconstruction by the proposed algorithm and Structured-Light
Resources
Publications
Kawasaki Laboratory