Computational imaging without a computer: seeing through random diffusers at the speed of light

Yi Luo; Yifan Zhao; Jingxi Li; Ege Çetintaş; Yair Rivenson; Mona Jarrahi; Aydogan Ozcan

doi:10.1186/s43593-022-00012-4

Your Location：

Home >

Browse articles >

Computational imaging without a computer: seeing through random diffusers at the speed of light

eLight Vol. 2, Issue 1, (2022)

RESEARCH ARTICLE | Updated：2024-09-28

- Computational imaging without a computer: seeing through random diffusers at the speed of light
- Affiliations：
  
  Electrical and Computer Engineering Department, University of California, Los Angeles, Los Angeles, CA 90095, USA
  Bioengineering Department, University of California, Los Angeles, Los Angeles, CA 90095, USA
  California NanoSystems Institute, University of California, Los Angeles, Los Angeles, CA 90095, USA
- Author bio：
  
  *ozcan@ucla.edu
- Funds：
- DOI：10.1186/s43593-022-00012-4
  CLC：
- Published：2022-12，
  
  Published Online：26 January 2022，
  
  Received：03 December 2021，
  
  Revised：31 December 2021，
  
  Accepted：08 January 2022
- Accepted：
Scan QR Code
Yi Luo, Yifan Zhao, Jingxi Li, et al. Computational imaging without a computer: seeing through random diffusers at the speed of light. [J]. eLight, 2022,2(1).
DOI：

Yi Luo, Yifan Zhao, Jingxi Li, et al. Computational imaging without a computer: seeing through random diffusers at the speed of light. [J]. eLight, 2022,2(1). DOI： 10.1186/s43593-022-00012-4.

摘要

Abstract

Imaging through diffusers presents a challenging problem with various digital image reconstruction solutions demonstrated to date using computers. Here

we present a computer-free

all-optical image reconstruction method to see through random diffusers at the speed of light. Using deep learning

a set of transmissive diffractive surfaces are trained to all-optically reconstruct images of arbitrary objects that are completely covered by unknown

random phase diffusers. After the training stage

which is a one-time effort

the resulting diffractive surfaces are fabricated and form a passive optical network that is physically positioned between the unknown object and the image plane to all-optically reconstruct the object pattern through an unknown

new phase diffuser. We experimentally demonstrated this concept using coherent THz illumination and all-optically reconstructed objects distorted by unknown

random diffusers

never used during training. Unlike digital methods

all-optical diffractive reconstructions do not require power except for the illumination light. This diffractive solution to see through diffusers can be extended to other wavelengths

and might fuel various applications in biomedical imaging

astronomy

atmospheric sciences

oceanography

security

robotics

autonomous vehicles

among many others.

关键词

Keywords

Imaging through diffusersComputational imagingDiffractive neural networkDeep learning

references

Eckart, A. & Genzel, R. Stellar proper motions in the central 0.1 pc of the Galaxy. Monthly Notices of the Royal Astronomical Society 284, 576-598 (1997).

M. Solan et al., Towards a greater understanding of pattern, scale and process in marine benthic systems: a picture is worth a thousand worms. J. Exp. Mar. Biol. Ecol. 285-286, 313-338 (2003)

Tan, R. T.Visibility in bad weather from a single image. in 2008 IEEE Conference on Computer Vision and Pattern Recognition 1-8 (2008). https://doi.org/10.1109/CVPR.2008.4587643https://doi.org/10.1109/CVPR.2008.4587643.

V. Ntziachristos, Going deeper than microscopy: the optical imaging frontier in biology. Nat. Methods 7, 603-614 (2010)

N. Ji, D.E. Milkie, E. Betzig, Adaptive optics via pupil segmentation for high-resolution imaging in biological tissues. Nat. Methods 7, 141-147(2010)

K. He, J. Sun, X. Tang, Single Image Haze Removal Using Dark Channel Prior. IEEE Trans. Pattern Anal. Mach. Intell. 33, 2341-2353 (2011)

J. Bertolotti et al., Non-invasive imaging through opaque scattering layers. Nature 491, 232-234 (2012)

A.P. Mosk, A. Lagendijk, G. Lerosey, M. Fink, Controlling waves in space and time for imaging and focusing in complex media. Nat. Photonics 6, 283-292 (2012)

O. Katz, P. Heidmann, M. Fink, S. Gigan, Non-invasive single-shot imaging through scattering layers and around corners via speckle correlations. Nature Photon 8, 784-790 (2014)

S.-C. Huang, B.-H. Chen, Y.-J. Cheng, An Efficient Visibility Enhancement Algorithm for Road Scenes Captured by Intelligent Transportation Systems. IEEE Trans. Intell. Transp. Syst. 15, 2321-2332 (2014)

S. Li, M. Deng, J. Lee, A. Sinha, G. Barbastathis, Imaging through glass diffusers using densely connected convolutional networks. Optica, OPTICA 5, 803-813 (2018)

Y. Li, Y. Xue, L. Tian, Deep speckle correlation: a deep learning approach toward scalable imaging through scattering media. Optica 5, 1181 (2018)

D.B. Lindell, G. Wetzstein, Three-dimensional imaging through scattering media based on confocal diffuse tomography. Nat Commun 11, 4517 (2020)

J.W. Goodman, W.H. Huntley, D.W. Jackson, M. Lehmann, Wavefront-reconstruction imaging through random media. Appl. Phys. Lett. 8, 311-313 (1966)

H. Kogelnik, K.S. Pennington, Holographic Imaging Through a Random Medium. J. Opt. Soc. Am. 58, 273 (1968)

S. Popoff, G. Lerosey, M. Fink, A.C. Boccara, S. Gigan, Image transmission through an opaque material. Nat Commun 1, 81 (2010)

J. Li et al., Conjugate adaptive optics in widefield microscopy with an extended-source wavefront sensor. Optica 2, 682 (2015)

E. Edrei, G. Scarcelli, Optical imaging through dynamic turbid media using the Fourier-domain shower-curtain effect. Optica 3, 71 (2016)

X. Li, J.A. Greenberg, M.E. Gehm, Single-shot multispectral imaging through a thin scatterer. Optica,OPTICA 6, 864-871 (2019)

M. Jang et al., Relation between speckle decorrelation and optical phase conjugation (OPC)-based turbidity suppression through dynamic scattering media: a study on in vivo mouse skin. Biomed. Opt. Express 6, 72 (2015)

S.G. Narasimhan, S.K. Nayar, Contrast restoration of weather degraded images. IEEE Trans. Pattern Anal. Mach. Intell. 25, 713-724 (2003)

E.A. Bucher, Computer Simulation of Light Pulse Propagation for Communication Through Thick Clouds. Appl. Opt. 12, 2391 (1973)

A. Lopez, E. Nezry, R. Touzi, H. Laur, Structure detection and statistical adaptive speckle filtering in SAR images. Int. J. Remote Sens. 14, 1735-1758 (1993)

Lohmann, A. W., Weigelt, G. & Wirnitzer, B. Speckle masking in astronomy: triple correlation theory and applications. Appl. Opt., AO 22, 4028-4037 (1983).

Roggemann, M. C., Welsh, B. M. & Hunt, R. B.. Imaging Through Turbulence. (CRC Press, 1996).

J.S. Jaffe, K.D. Moore, J. Mclean, M.R. Strand, Underwater optical imaging: Status and prospects. Oceanography 14, 64-66 (2001)

Schettini, R. & Corchs, S. Underwater Image Processing: State of the Art of Restoration and Image Enhancement Methods. EURASIP Journal on Advances in Signal Processing 2010, (2010).

Z. Jia et al., A two-step approach to see-through bad weather for surveillance video quality enhancement. Mach. Vis. Appl. 23, 1059-1082 (2012)

Tarel, J.-P. & Hautière, N.Fast visibility restoration from a single color or gray level image. in 2009 IEEE 12th International Conference on Computer Vision 2201-2208 (2009). https://doi.org/10.1109/ICCV.2009.5459251https://doi.org/10.1109/ICCV.2009.5459251.

M. Johnson-Roberson et al., High-Resolution Underwater Robotic Vision-Based Mapping and Three-Dimensional Reconstruction for Archaeology. Journal of Field Robotics 34, 625-643 (2017)

Hao, Z., You, S., Li, Y., Li, K. & Lu, F.Learning From Synthetic Photorealistic Raindrop for Single Image Raindrop Removal. in Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops 0-0 (2019).

Majer, F., Yan, Z., Broughton, G., Ruichek, Y. & Krajník, T.Learning to see through haze: Radar-based Human Detection for Adverse Weather Conditions. in 2019 European Conference on Mobile Robots (ECMR) 1-7 (2019). doi:https://doi.org/10.1109/ECMR.2019.8870954https://doi.org/10.1109/ECMR.2019.8870954.

Popoff, S. M. et al. Measuring the Transmission Matrix in Optics: An Approach to the Study and Control of Light Propagation in Disordered Media. Physical Review Letters 104, (2010).

W J. Goodman,. Speckle Phenomena in Optics: Theory and Applications. (Roberts and Company Publishers, 2007).

D.B. Conkey, A.M. Caravaca-Aguirre, R. Piestun, High-speed scattering medium characterization with application to focusing light through turbid media. Opt. Express 20, 1733 (2012)

Wang, K. et al. Direct wavefront sensing for high-resolution in vivo imaging in scattering tissue. Nature Communications 6, (2015).

I.M. Vellekoop, A.P. Mosk, Focusing coherent light through opaque strongly scattering media. Opt. Lett. 32, 2309 (2007)

I.M. Vellekoop, A. Lagendijk, A.P. Mosk, Exploiting disorder for perfect focusing. Nat. Photonics 4, 320-322 (2010)

R. Horstmeyer, H. Ruan, C. Yang, Guidestar-assisted wavefront-shaping methods for focusing light into biological tissue. Nat. Photonics 9, 563-571 (2015)

M. Nixon et al., Real-time wavefront shaping through scattering media by all-optical feedback. Nat. Photonics 7, 919-924 (2013)

O. Katz, E. Small, Y. Silberberg, Looking around corners and through thin turbid layers in real time with scattered incoherent light. Nat. Photonics 6, 549-553 (2012)

S. Feng, C. Kane, P.A. Lee, A.D. Stone, Correlations and Fluctuations of Coherent Wave Transmission through Disordered Media. Phys. Rev. Lett. 61, 834-837 (1988)

I. Freund, M. Rosenbluh, S. Feng, Memory Effects in Propagation of Optical Waves through Disordered Media. Phys. Rev. Lett. 61, 2328-2331 (1988)

Edrei, E. & Scarcelli, G. Memory-effect based deconvolution microscopy for super-resolution imaging through scattering media. Scientific Reports 6, (2016).

W. Yang, G. Li, G. Situ, Imaging through scattering media with the auxiliary of a known reference object. Sci. Rep. 8, 9614 (2018)

He, H., Guan, Y. & Zhou, J. Image restoration through thin turbid layers by correlation with a known object. Opt. Express, OE 21, 12539-12545 (2013).

X. Wang et al., Prior-information-free single-shot scattering imaging beyond the memory effect. Opt. Lett. 44, 1423 (2019)

Yang, M. et al. Deep hybrid scattering image learning.J. Phys. D: Appl. Phys. 52, 115105 (2019).

Lyu, M., Wang, H., Li, G., Zheng, S. & Situ, G. Learning-based lensless imaging through optically thick scattering media. AP 1, 036002 (2019).

Y. Rivenson et al., Deep learning microscopy. Optica, OPTICA 4, 1437-1443 (2017)

Y. Rivenson et al., Deep Learning Enhanced Mobile-Phone Microscopy. ACS Photonics (2018). https://doi.org/10.1021/acsphotonics.8b00146https://doi.org/10.1021/acsphotonics.8b00146

E. Nehme, L.E. Weiss, T. Michaeli, Y. Shechtman, Deep-STORM: super-resolution single-molecule microscopy by deep learning. Optica, OPTICA 5, 458-464 (2018)

H. Wang et al., Deep learning enables cross-modality super-resolution in fluorescence microscopy. Nat. Methods 16, 103-110 (2019)

Y. Rivenson et al., Virtual histological staining of unlabelled tissue-autofluorescence images via deep learning. Nat Biomed Eng 3, 466-477 (2019)

Y. Wu et al., Three-dimensional virtual refocusing of fluorescence microscopy images using deep learning. Nat. Methods 16, 1323-1331 (2019)

Y. Wu et al., Extended depth-of-field in holographic imaging using deeplearning-based autofocusing and phase recovery. Optica 5, 704 (2018)

Wu, Y. et al. Bright-field holography: cross-modality deep learning enables snapshot 3D imaging with bright-field contrast using a single hologram. Light: Science & Applications 8, 1-7 (2019).

T. Liu et al., Deep learning-based super-resolution in coherent imaging systems. Sci. Rep. 9, 1-13 (2019)

Liu, T. et al. Deep learning-based color holographic microscopy.Journal of Biophotonics 12, e201900107 (2019).

G. Barbastathis, A. Ozcan, G. Situ, On the use of deep learning for computational imaging. Optica,OPTICA 6, 921-943 (2019)

Wang, F. et al. Phase imaging with an untrained neural network. Light: Science & Applications 9, 77 (2020).

Malkiel, I. et al. Plasmonic nanostructure design and characterization via Deep Learning. Light: Science & Applications 7, (2018).

D. Liu, Y. Tan, E. Khoram, Z. Yu, Training Deep Neural Networks for the Inverse Design of Nanophotonic Structures. ACS Photonics 5, 1365-1369 (2018)

Peurifoy, J. et al. Nanophotonic particle simulation and inverse design using artificial neural networks. Science Advances 4, eaar4206 (2018).

W. Ma, F. Cheng, Y. Liu, Deep-Learning-Enabled On-Demand Design of Chiral Metamaterials. ACS Nano 12, 6326-6334 (2018)

Luo, Y. et al. Design of task-specific optical systems using broadband diffractive neural networks.Light: Science & Applications 8, 1-14 (2019).

M. Veli et al., Terahertz pulse shaping using diffractive surfaces. Nat. Commun. 12, 37 (2021)

D. Psaltis, D. Brady, X.G. Gu, S. Lin, Holography in artificial neural networks. Nature 343, 325-330 (1990)

Y. Shen et al., Deep learning with coherent nanophotonic circuits. Nat. Photonics 11, 441-446 (2017)

Chang, J., Sitzmann, V., Dun, X., Heidrich, W. & Wetzstein, G. Hybrid optical-electronic convolutional neural networks with optimized diffractive optics for image classification. Scientific Reports 8, (2018).

X. Lin et al., All-optical machine learning using diffractive deep neural networks. Science 361, 1004 (2018)

N.M. Estakhri, B. Edwards, N. Engheta, Inverse-designed metastructures that solve equations. Science 363, 1333-1338 (2019)

J. Li, D. Mengu, Y. Luo, Y. Rivenson, A. Ozcan, Class-specific differential detection in diffractive optical neural networks improves inference accuracy. Adv. Photon. 1, 1 (2019)

D. Mengu, Y. Luo, Y. Rivenson, A. Ozcan, Analysis of Diffractive Optical Neural Networks and Their Integration With Electronic Neural Networks. IEEE J. Sel. Top. Quantum Electron. 26, 1-14 (2020)

Mengu, D. et al. Misalignment resilient diffractive optical networks. Nanophotonics 0, (2020).

Li, J. et al. Spectrally encoded single-pixel machine vision using diffractive networks. Science Advances 7, eabd7690 (2021).

O. Kulce, D. Mengu, Y. Rivenson, A. Ozcan, All-optical information-processing capacity of diffractive surfaces. Light Sci Appl 10, 25 (2021)

B. Rahmani, D. Loterie, G. Konstantinou, D. Psaltis, C. Moser, Multimode optical fiber transmission with a deep learning network. Light Sci Appl 7, 1-11 (2018)

Bai, B. et al. Pathological crystal imaging with single-shot computational polarized light microscopy. Journal of Biophotonics 13, e201960036 (2020).

T. Liu et al., Deep Learning-Based Holographic Polarization Microscopy. ACS Photonics 7, 3023-3034 (2020)

LeCun, Y. et al. Handwritten Digit Recognition with a Back-Propagation Network. in Advances in Neural Information Processing Systems 2 (ed.Touretzky, D. S.) 396-404 (Morgan-Kaufmann, 1990).

Benesty, J., Chen, J., Huang, Y. & Cohen, I. Pearson Correlation Coefficient. in Noise Reduction in Speech Processing vol. 21-4 (Springer Berlin Heidelberg, 2009).

Wu, T., Dong, J., Shao, X. & Gigan, S.Imaging through a thin scattering layer and jointly retrieving the point-spread-function using phase-diversity. Opt. Express, OE 25, 27182-27194 (2017).

X. Xu et al., Imaging of objects through a thin scattering layer using a spectrally and spatially separated reference. Opt. Express 26, 15073 (2018)

Hofer, M., Soeller, C., Brasselet, S. & Bertolotti, J.Wide field fluorescence epi-microscopy behind a scattering medium enabled by speckle correlations. Opt. Express, OE 26, 9866-9881 (2018).

S. Lowenthal, D. Joyeux, Speckle Removal by a Slowly Moving Diffuser Associated with a Motionless Diffuser. J. Opt. Soc. Am. 61, 847 (1971)

Kingma, D. P. & Ba, J. Adam: A Method for Stochastic Optimization. arXiv:1412.6980. [cs] (2014).

Rahman, M. S. S., Li, J., Mengu, D., Rivenson, Y. & Ozcan, A. Ensemble learning of diffractive optical networks. Light: Science & Applications 10, 14 (2021).

Views

Downloads

CSCD

Alert me when the article has been cited

Submit

Publicity Resources

Computational coherent Raman scattering imaging: breaking physical barriers by fusion of advanced instrumentation and data science

Microtaper leaky-mode spectrometer with picometer resolution

Three-dimensional nanoscale reduced-angle ptycho-tomographic imaging with deep learning (RAPID)

Polarization-based underwater geolocalization with deep learning

Large-scale phase retrieval

Related Author

Haonan Lin

Ji-Xin Cheng

Yaoguang Ma

Xinying He

Yuwei Tang

Xinhang Liu

Sijie Pian

Qingqing Cen

Related Institution

Department of Biomedical Engineering, Department of Electrical and Computer Engineering, Photonics Center, Boston University

State Key Laboratory of Extreme Photonics and Instrumentation, College of Optical Science and Engineering; International Research Center for Advanced Photonics, Zhejiang University

ZJU-Hangzhou Global Scientific and Technological Innovation Center

Jiaxing Key Laboratory of Photonic Sensing & Intelligent Imaging, Intelligent Optics & Photonics Research Center, Jiaxing Research Institute, Zhejiang University

Department of Mechanical Engineering, Massachusetts Institute of Technology

⁰