Deep Recurrent Network for Fast and Full-Resolution Light Field Deblurring

Deep Recurrent Network for Fast and Full-Resolution Light Field Deblurring

accepted in IEEE Signal Processing Letters, vol. 26, no. 10, pp. 1788-1792, December 2019.

Jonathan Samuel Lumentut Tae Hyun Kim Ravi Ramamoorthi In Kyu Park
Inha University, Korea Hanyang University, Korea UC San Diego, USA Inha University, Korea

Abstract

The popularity of parallax-based image processing is increasing while in contrast early works on recovering sharp light field from its blurry input (deblurring) remain stagnant. State-of-the-art blind light field deblurring methods suffer from several problem such as slow processing, reduced spatial size, and simplified motion blur model. In this paper, we solve these challenging problems by proposing a novel light field recurrent deblurring network that is trained under 6 degree-of-freedom camera motion-blur model. By combining the real light field captured using Lytro Illum and synthetic light field rendering of 3D scenes from UnrealCV, we provide a large-scale blurry light field dataset to train the network. The proposed method outperforms the state-of-the-art methods in term of deblurring quality, the capability of handling full-resolution, and a fast runtime.

Paper

Download paper: [link]

Video Demo

Download video: [mp4]

Deblurring Results

Real motion blur case from [8] dataset

Krishnan et al. [1]

Pan et al. [6]

Srinivasan et al. [8]

Ours

Synthesized 3-DOF motion blur case

Krishnan et al. [1]

Pan et al. [6]

Srinivasan et al. [8]

Ours

Synthesized 6-DOF motion blur case

Krishnan et al. [1]

Pan et al. [6]

Srinivasan et al. [8]

Ours

Quantitative Results

Ablation Study

Ablation Study on Variated Angular Sample ( n ) and Neighbor ( j )
j PSNR/SSIM n PSNR/SSIM
( n = 10 ) ( j = 3 )
1 24.68 / .819 5 24.69 / .824
3 24.72 / .824 10 24.72 / .824
5 24.68 / .823 15 24.62 / .822

Comparison on 3-DOF Test Set [8]

PSNR/SSIM/RMSE Result Between Algorithms on 3-DOF Motion
Method Cropped Resolution Full Resolution
( 6 x 5 x 5 x 200 x 200 x 3 ) ( 6 x 5 x 5 x 320 x 512 x 3 )
[1] 24.50 / .787 / 0.0717 25.08 / .809 / 0.0622
[6] 21.98 / .731 / 0.0866 22.86 / .767 / 0.0769
[8] 24.75 / .781 / 0.0673 N/A (Memory Overflow)
Ours 27.57 / .855 / 0.0453 27.21 / .871 / 0.0459

Comparison on 6-DOF Test Set

PSNR/SSIM/RMSE Result Between Algorithms on 6-DOF Motion
Method Cropped Resolution Full Resolution
( 40 x 5 x 5 x 200 x 200 x 3 ) ( 40 x 5 x 5 x 320 x 512 x 3 )
[1] 23.41 / .778 / 0.0732 23.41 / .774 / 0.0703
[6] 21.70 / .732 / 0.0875 21.58 / .730 / 0.0857
[8] 23.61 / .765 / 0.0703 N/A (Memory Overflow)
Ours 26.57 / .846 / 0.0490 25.73 / .840 / 0.0531

Comparison on Execution Time

Comparison of Execution Time
Method Cropped Resolution [sec.] Full Resolution [sec.]
( 5 x 5 x 200 x 200 x 3 ) ( 5 x 5 x 320 x 512 x 3 )
[1] ~100 (CPU) ~445 (CPU)
[6] ~240 (CPU) ~1000 (CPU)
[8] ~8,000 (GPU) N/A (Memory Overflow)
Ours ~0.5 (GPU) ~1.7 (GPU)

References

  1. D. Krishnan, T. Tay, and R. Fergus, “Blind deconvolution using a normalized sparsity measure,” in Proc. of IEEE Conference on Computer Vision and Pattern Recognition, 2011, pp. 233–240.
  2. T. H. Kim, K. M. Lee, B. Scholkopf, and M. Hirsch, “Online video deblurring via dynamic temporal blending network,” in Proc. of IEEE International Conference on Computer Vision, 2017, pp. 4058–4067.
  3. D. Lee, H. Park, I. K. Park and K. M. Lee, “Joint blind motion deblurring and depth estimation of light field,” in Proc. of European Conference on Computer Vision, 2018, pp. 233–240.
  4. M. Mahesh Mohan and A. N. Rajagopalan, “Divide and conquer for full-resolution light field deblurring,” in Proc. of IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 6421–6429.
  5. S. Nah, T. H. Kim, and K. M. Lee, “Deep multi-scale convolutional neural network for dynamic scene deblurring,” in Proc. of IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 3.
  6. J. Pan, Z. Hu, Z. Su, and M. H. Yang, “Deblurring text images via L0-regularized intensity and gradien prior,” in Proc. of IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 2901-2908.
  7. W. Qiu, F. Zhong, Y. Zhang, S. Qiao, Z. Xiao, T. S. Kim, and Y. Yang, “UnrealCV: Virtual worlds for computer vision.,” in Proc. of the 25th ACM International Conference on Multimedia, 2017, pp. 1221-1224.
  8. P. P. Srinivasan, R. Ng, and R. Ramamoorthi, “Light field blind motion deblurring,” in Proc. of IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 2262-2270.
  9. O. Whyte, J. Sivic, A. Zisserman, and J. Ponce, “Non-uniform deblurring for shaken images,” in Proc. of IEEE Conference on Computer Vision and Pattern Recognition, 2010, pp. 168-186.
  10. Williem, I. K. Park, and K. M. Lee, “Robust light field depth estimation using occlusion-noise aware data costs,” in Proc. of IEEE Trans. on Pattern Analysis and Machine Intelligence, vol. 40, no. 10, 2018, pp. 2484–2497.