Three-dimensional nanoscale reduced-angle ptycho-tomographic imaging with deep learning (RAPID)

Ziling Wu; Iksung Kang; Yudong Yao; Yi Jiang; Junjing Deng; Jeffrey Klug; Stefan Vogt; George Barbastathis

doi:10.1186/s43593-022-00037-9

Your Location：

Home >

Browse articles >

Three-dimensional nanoscale reduced-angle ptycho-tomographic imaging with deep learning (RAPID)

eLight Vol. 3, Issue 1, (2023)

RESEARCH ARTICLE | Updated：2024-09-25

- Three-dimensional nanoscale reduced-angle ptycho-tomographic imaging with deep learning (RAPID)
- Affiliations：
  
  Department of Mechanical Engineering, Massachusetts Institute of Technology, Cambridge, MA 02139, USA
  Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology, Cambridge, MA 02139, USA
  Argonne National Laboratory, Lemont, IL 60439, USA
  Singapore-MIT Alliance for Research and Technology (SMART) Centre, Singapore 138602, Singapore
  Present Address: Singapore-MIT Alliance for Research and Technology (SMART) Centre, Singapore 138602, Singapore
  Present Address: University of California, Berkeley, CA 94720, USA
- Author bio：
  
  *Ziling Wuzilingwu@mit.edu
  George Barbastathisgbarb@mit.edu
- Funds：
- DOI：10.1186/s43593-022-00037-9
  CLC：
- Published：2023，
  
  Published Online：03 April 2023，
  
  Received：21 August 2022，
  
  Revised：28 September 2022，
  
  Accepted：11 November 2022
- Accepted：
Scan QR Code
Ziling Wu, Iksung Kang, Yudong Yao, et al. Three-dimensional nanoscale reduced-angle ptycho-tomographic imaging with deep learning (RAPID). [J]. eLight, 2023,3(1).
DOI：

Ziling Wu, Iksung Kang, Yudong Yao, et al. Three-dimensional nanoscale reduced-angle ptycho-tomographic imaging with deep learning (RAPID). [J]. eLight, 2023,3(1). DOI： 10.1186/s43593-022-00037-9.

摘要

Abstract

X-ray ptychographic tomography is a nondestructive method for three dimensional (3D) imaging with nanometer-sized resolvable features. The size of the volume that can be imaged is almost arbitrary

limited only by the penetration depth and the available scanning time. Here we present a method that rapidly accelerates the imaging operation over a given volume through acquiring a limited set of data via large angular reduction and compensating for the resulting ill-posedness through deeply learned priors. The proposed 3D reconstruction method "RAPID" relies initially on a subset of the object measured with the nominal number of required illumination angles and treats the reconstructions from the conventional two-step approach as ground truth. It is then trained to reproduce equal fidelity from much fewer angles. After training

it performs with similar fidelity on the hitherto unexamined portions of t

he object

previously not shown during training

with a limited set of acquisitions. In our experimental demonstration

the nominal number of angles was 349 and the reduced number of angles was 21

resulting in a ×140 aggregate speedup over a volume of 4.48×93.18×3.92µm

and with (14 nm)

feature size

i.e. ~10

voxels. RAPID's key distinguishing feature over earlier attempts is the incorporation of atrous spatial pyramid pooling modules into the deep neural network framework in an anisotropic way. We found that adjusting the atrous rate improves reconstruction fidelity because it expands the convolutional kernels' range to match the physics of multi-slice ptychography without significantly increasing the number of parameters.

关键词

Keywords

X-ray ptychographic tomographyDeep learningReduced-angleRapid imagingAtrous spatial pyramid poolingAnisotropic

references

K. Grunewald, P. Desai, D.C. Winkler, J.B. Heymann, D.M. Belnap, W. Baumeister, A.C. Steven, Three-dimensional structure of herpes simplex virus from cryo-electron tomography. Science302(5649), 1396-1398(2003)

P. Ercius, O. Alaidi, M.J. Rames, G. Ren, Electron tomography: a three-dimensional analytic tool for hard and soft materials research. Adv. Mater. 27(38), 5638-5663 (2015)

M. Holler, M. Guizar-Sicairos, E.H. Tsai, R. Dinapoli, E. Müller, O. Bunk, J. Raabe, G. Aeppli, High-resolution non-destructive three-dimensional imaging of integrated circuits. Nature543(7645), 402-406 (2017)

B. Parkinson, Layer-by-layer nanometer scale etching of two-dimensional substrates using the scanning tunneling microscope. J. Am. Chem. Soc. 112(21), 7498-7502 (1990)

C. Harrison, M. Park, P. Chaikin, R.A. Register, D.H. Adamson, N. Yao, Layer by layer imaging of diblock copolymer films with a scanning electron microscope. Polymer39(13), 2733-2744 (1998)

S.H. Lee, K.-N. Chen, J.J.-Q. Lu, Wafer-to-wafer alignment for three-dimensional integration: a review. J. Microelectromech. Syst. 20(4), 885-898(2011)

R.R. Schaller, Moore's law: past, present and future. IEEE Spectr.34(6), 52-59 (1997)

R. Hegerl, W. Hoppe, Dynamische theorie der kristallstrukturanalyse durch elektronenbeugung im inhomogenen primärstrahlwellenfeld. Ber. Bunsenges. Phys. Chem. 74(11), 1148-1154 (1970)

M. Dierolf, A. Menzel, P. Thibault, P. Schneider, C.M. Kewish, R. Wepf, O. Bunk, F. Pfeiffer, Ptychographic x-ray computed tomography at the nanoscale. Nature467(7314), 436-439 (2010)

H. Takano, Y. Wu, J. Irwin, S. Maderych, M. Leibowitz, A. Tkachuk, A. Kumar, B. Hornberger, A. Momose, Comparison of image properties in full-field phase x-ray microscopes based on grating interferometry and Zernike's phase contrast optics. Appl. Phys. Lett. 113(6), 063105 (2018)

M. Guizar-Sicairos, A. Diaz, M. Holler, M.S. Lucas, A. Menzel, R.A. Wepf, O. Bunk, Phase tomography from x-ray coherent diffractive imaging projections. Opt. Express19(22), 21345-21357 (2011)

H.M.L. Faulkner, J. Rodenburg, Movable aperture lensless transmission microscopy: a novel phase retrieval algorithm. Phys. Rev. Lett. 93(2), 023903 (2004)

A.M. Maiden, J.M. Rodenburg, An improved ptychographical phase retrieval algorithm for diffractive imaging. Ultramicroscopy109(10), 1256-1262 (2009)

F. Pfeiffer, X-ray ptychography. Nat. Photonics12(1), 9-17 (2018)

M.J. Willemink, P.B. Noël, The evolution of image reconstruction for ctfrom filtered back projection to artificial intelligence. Eur. Radiol. 29(5), 2185-2195 (2019)

P. Gilbert, Iterative methods for the three-dimensional reconstruction of an object from projections. J. Theo. Biol. 36(1), 105-117 (1972)

A.H. Andersen, A.C. Kak, Simultaneous algebraic reconstruction technique (sart): a superior implementation of the art algorithm. Ultrason. Imaging6(1), 81-94 (1984)

C. Bouman, K. Sauer, A generalized Gaussian image model for edge-preserving map estimation. IEEE Trans. Image Process. 2(3), 296-310 (1993)

A. Schropp, P. Boye, A. Goldschmidt, S. Hönig, R. Hoppe, J. Patommel, C. Rakete, D. Samberg, S. Stephan, S. Schöder, Non-destructive and quantitative imaging of a nano-structured microchip by ptychographic hard x-ray scanning microscopy. J. Microsc. 241(1), 9-12 (2011)

A. Diaz, B. Malkova, M. Holler, M. Guizar-Sicairos, E. Lima, V. Panneels, G. Pigino, A.G. Bittermann, L. Wettstein, T. Tomizaki, Three-dimensional mass density mapping of cellular ultrastructure by ptychographic x-ray nanotomography. J. Struct. Biol. 192(3), 461-469 (2015)

P. Trtik, A. Diaz, M. Guizar-Sicairos, A. Menzel, O. Bunk, Density mapping of hardened cement paste using ptychographic x-ray computed tomography. Cement Concr. Compos. 36, 71-77 (2013)

B. Chen, M. Guizar-Sicairos, G. Xiong, L. Shemilt, A. Diaz, J. Nutter, N. Burdet, S. Huo, J. Mancuso, A. Monteith, Three-dimensional structure analysis and percolation properties of a barrier marine coating. Sci. Rep. 3(1), 1-5(2013)

M. Esmaeili, J.B. Fløystad, A. Diaz, K. Høydalsvik, M. Guizar-Sicairos, J.W. Andreasen, D.W. Breiby, Ptychographic x-ray tomography of silk fiber hydration. Macromolecules46(2), 434-439 (2013)

O. Bunk, M. Dierolf, S. Kynde, I. Johnson, O. Marti, F. Pfeiffer, Influence of the overlap parameter on the convergence of the ptychographical iterative engine. Ultramicroscopy108(5), 481-487 (2008)

R.A. Crowther, D. DeRosier, A. Klug, The reconstruction of a three-dimensional structure from projections and its application to electron microscopy. Proc. R. Soc. London A Math. Phys. Sci. 317(1530), 319-340(1970)

P.M. Pelz, M. Guizar-Sicairos, P. Thibault, I. Johnson, M. Holler, A. Menzel, On-the-fly scans for x-ray ptychography. Appl. Phys. Lett. 105(25), 251101(2014)

J.N. Clark, X. Huang, R.J. Harder, I.K. Robinson, Continuous scanning mode for ptychography. Opt. Lett. 39(20), 6066-6069 (2014)

Y. Yao, Y. Jiang, J.A. Klug, M. Wojcik, E.R. Maxey, N.S. Sirica, C. Roehrig, Z. Cai, S. Vogt, B. Lai, Multi-beam x-ray ptychography for high-throughput coherent diffraction imaging. Sci. Rep. 10(1), 1-8 (2020)

J. Deng, Y.S. Nashed, S. Chen, N.W. Phillips, T. Peterka, R. Ross, S. Vogt, C. Jacobsen, D.J. Vine, Continuous motion scan ptychography: characterization for increased speed in coherent x-ray imaging. Opt. Express23(5), 5438-5451 (2015)

J. Deng, C. Preissner, J.A. Klug, S. Mashrafi, C. Roehrig, Y. Jiang, Y. Yao, M. Wojcik, M.D. Wyman, D. Vine, The velociprobe: an ultrafast hard x-ray nanoprobe for high-resolution ptychographic imaging. Rev. Sci. Instrum. 90(8), 083701 (2019)

D. Gürsoy, Direct coupling of tomography and ptychography. Opt. Lett. 42(16), 3169-3172 (2017)

M. Kahnt, J. Becher, D. Brückner, Y. Fam, T. Sheppard, T. Weissenberger, F. Wittwer, J.-D. Grunwaldt, W. Schwieger, C.G. Schroer, Coupled ptychography and tomography algorithm improves reconstruction of experimental data. Optica6(10), 1282-1289 (2019)

H. Chang, P. Enfedaque, S. Marchesini, Iterative joint ptychography-tomography with total variation regularization. in 2019 IEEE International Conference on Image Processing (ICIP), pp. 2931-2935 (2019). IEEE

S. Aslan, V. Nikitin, D.J. Ching, T. Bicer, S. Leyffer, D. Gürsoy, Joint ptycho-tomography reconstruction through alternating direction method of multipliers. Opt. Express27(6), 9128-9143 (2019)

A. Suzuki, S. Furutaku, K. Shimomura, K. Yamauchi, Y. Kohmura, T. Ishikawa, Y. Takahashi, High-resolution multislice x-ray ptychography of extended thick objects. Phys. Rev. Lett. 112(5), 053903 (2014)

K. Shimomura, A. Suzuki, M. Hirose, Y. Takahashi, Precession x-ray ptychography with multislice approach. Phys. Rev. B. 91(21), 214114 (2015)

K. Shimomura, M. Hirose, T. Higashino, Y. Takahashi, Three-dimensional iterative multislice reconstruction for ptychographic x-ray computed tomography. Opt. Express26(24), 31199-31208 (2018)

P. Li, A. Maiden, Multi-slice ptychographic tomography. Sci. Rep. 8(1), 1-10 (2018)

C. Jacobsen, Relaxation of the crowther criterion in multislice tomography. Opt. Lett. 43(19), 4811-4814 (2018)

A.M. Maiden, M.J. Humphry, J. Rodenburg, Ptychographic transmission microscopy in three dimensions using a multi-slice approach. JOSA A29(8), 1606-1614 (2012)

T. Ramos, B.E. Grønager, M.S. Andersen, J.W. Andreasen, Direct three-dimensional tomographic reconstruction and phase retrieval of far-field coherent diffraction patterns. Phys. Rev. A99(2), 023801 (2019)

S. Barutcu, P. Ruiz, F. Schiffers, S. Aslan, D. Gursoy, O. Cossairt, A.K. Katsag-gelos, Simultaneous 3d x-ray ptycho-tomography with gradient descent. in 2020 IEEE International Conference on Image Processing (ICIP), pp. 96-100 (2020). IEEE

Z. Fabian, J. Haldar, R. Leahy, M. Soltanolkotabi, 3d phase retrieval at nano-scale via accelerated wirtinger flow. in 2020 28th European Signal Processing Conference (EUSIPCO), pp. 2080-2084 (2021). IEEE

V. Nikitin, S. Aslan, Y. Yao, T. Biçer, S. Leyffer, R. Mokso, D. Gürsoy, Photon-limited ptychography of 3d objects via Bayesian reconstruction. OSA Continuum2(10), 2948-2968 (2019)

S. Li, M. Deng, J. Lee, A. Sinha, G. Barbastathis, Imaging through glass diffusers using densely connected convolutional networks. Optica5(7), 803-813 (2018)

Y. Li, Y. Xue, L. Tian, Deep speckle correlation: a deep learning approach toward scalable imaging through scattering media. Optica5(10), 1181-1190 (2018)

A. Goy, K. Arthur, S. Li, G. Barbastathis, Low photon count phase retrieval using deep learning. Phys. Rev. Lett. 121(24), 243902 (2018)

M. Deng, S. Li, A. Goy, I. Kang, G. Barbastathis, Learning to synthesize: robust phase retrieval at low photon counts. Light Sci. Appl. 9(1), 1-16(2020)

Y. Rivenson, T. Liu, Z. Wei, Y. Zhang, K. de Haan, A. Ozcan, Phasestain: the digital staining of label-free quantitative phase microscopy images using deep learning. Light Sci. Appl. 8(1), 1-11 (2019)

H. Majeed, A. Keikhosravi, M.E. Kandel, T.H. Nguyen, Y. Liu, A. Kajdacsy-Balla, K. Tangella, K.W. Eliceiri, G. Popescu, Quantitative histopathology of stained tissues using color spatial light interference microscopy (cslim). Sci. Rep. 9(1), 1-14 (2019)

A. Goy, G. Rughoobur, S. Li, K. Arthur, A.I. Akinwande, G. Barbastathis, High-resolution limited-angle phase tomography of dense layered objects using deep neural networks. Proc. Natl. Acad. Sci. 116(40), 19848-19856 (2019)

I. Kang, A. Goy, G. Barbastathis, Dynamical machine learning volumetric reconstruction of objects' interiors from limited angular views. Light Sci. Appl. 10(1), 1-21 (2021)

F. Wang, Y. Bian, H. Wang, M. Lyu, G. Pedrini, W. Osten, G. Barbastathis, G. Situ, Phase imaging with an untrained neural network. Light Sci. Appl. 9(1), 1-7 (2020)

H. Chung, J. Huh, G. Kim, Y.K. Park, J.C. Ye, Missing cone artifact removal in odt using unsupervised deep learning in the projection domain. IEEE Trans. Comput. Imag. 7, 747-758 (2021)

S. Aslan, Z. Liu, V. Nikitin, T. Bicer, S. Leyffer, D. Gürsoy, Joint ptycho-tomography with deep generative priors. Mach. Learn. Sci. Technol. 2(4), 045017(2021)

L. Alzubaidi, J. Zhang, A.J. Humaidi, A. Al-Dujaili, Y. Duan, O. Al-Shamma, J. Santamaría, M.A. Fadhel, M. Al-Amidie, L. Farhan, Review of deep learning: Concepts, cnn architectures, challenges, applications, future directions. J. Big Data8(1), 1-74 (2021)

K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition. in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778 (2016)

S. Mallat, A Wavelet Tour of Signal Processing (Elsevier, Amsterdam, 1999)

L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A.L. Yuille, Semantic image segmentation with deep convolutional nets and fully connected crfs. arXiv preprint arXiv: 1412. 7062 (2014)

K. He, X. Zhang, S. Ren, J. Sun, Spatial pyramid pooling in deep convolutional networks for visual recognition. IEEE Trans. Pattern Anal. Mach. Intell. 37(9), 1904-1916 (2015)

L.-C. Chen, G. Papandreou, F. Schroff, H. Adam, Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv: 1706. 05587(2017)

U.S. Kamilov, I.N. Papadopoulos, M.H. Shoreh, A. Goy, C. Vonesch, M. Unser, D. Psaltis, Learning approach to optical tomography. Optica2(6), 517-522(2015)

O. Ronneberger, P. Fischer, T. Brox, U-net: convolutional networks for biomedical image segmentation. in International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 234-241(2015). Springer

Ö. Çiçek, A. Abdulkadir, S.S. Lienkamp, T. Brox, O. Ronneberger, 3D U-Net: learning dense volumetric segmentation from sparse annotation. in International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 424-432 (2016). Springer

Z. Wang, E.P. Simoncelli, A.C. Bovik, Multiscale structural similarity for image quality assessment. In: The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003, vol. 2, pp. 1398-1402 (2003). IEEE

J. Lim, Is ber the bit error ratio or the bit error rate?EDN. Retrieved, 02-16(2015)

L.R. Dice, Measures of the amount of ecologic association between species. Ecology26(3), 297-302 (1945)

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A.N. Gomez, Ł Kaiser, I. Polosukhin, Attention is all you need. Adv. Neural Informat. Process. Syst. 30, (2017)

A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unter-thiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv: 2010. 11929 (2020)

I. Kang, Z. Wu, Y. Jiang, Y. Yao, J. Deng, J. Klug, N. Weisse-Bernstein, S. Vogt, G. Barbastathis, Attentional Ptycho-Tomography (APT) for three-dimensional nanoscale X-ray imaging with minimal data acquisition and computation time (Submitted)

L. Helfen, T. Baumbach, P. Mikulik, D. Kiel, P. Pernot, P. Cloetens, J. Baruchel, High-resolution three-dimensional imaging of flat objects by synchrotron-radiation computed laminography. Appl. Phys. Lett. 86(7), 071915(2005)

M. Holler, M. Odstrcil, M. Guizar-Sicairos, M. Lebugle, E. Müller, S. Finizio, G. Tinti, C. David, J. Zusman, W. Unglaub, Three-dimensional imaging of integrated circuits with macro-to nanoscale zoom. Nat. Electron. 2(10), 464-470 (2019)

Z. Wu, I. Kang, T. Zhou, V. Coykendall, B. Ge, M.J. Cherukara, G. Barbasta-this, Photon-starved x-ray ptychographic imaging using spatial pyramid atrous convolution end-to-end reconstruction (ptychospacer). in Computational Optical Sensing and Imaging, pp. 1-6 (2022). Optica Publishing Group

M.J. Cherukara, T. Zhou, Y. Nashed, P. Enfedaque, A. Hexemer, R.J. Harder, M.V. Holt, Ai-enabled high-resolution scanning coherent diffraction imaging. Appl. Phys. Lett. 117(4), 044103 (2020)

K. Weiss, T.M. Khoshgoftaar, D. Wang, A survey of transfer learning. J. Big data3(1), 1-40 (2016)

E.H. Tsai, I. Usov, A. Diaz, A. Menzel, M. Guizar-Sicairos, X-ray ptychography with extended depth of field. Opt. Express24(25), 29089-29108 (2016)

D.P. Kingma, J. Ba, Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412. 6980 (2014)

M. Odstrčil, A. Menzel, M. Guizar-Sicairos, Iterative least-squares solver for generalized maximum-likelihood ptychography. Opt. Express26(3), 3108-3123 (2018)

K. Wakonig, H.-C. Stadler, M. Odstrčil, E.H. Tsai, A. Diaz, M. Holler, I. Usov, J. Raabe, A. Menzel, M. Guizar-Sicairos, Ptychoshelves, a versatile high-level framework for high-performance analysis of ptychographic data. J. Appl. Crystallogr. 53(2), (2020)

D. Gürsoy, F. De Carlo, X. Xiao, C. Jacobsen, Tomopy: a framework for the analysis of synchrotron tomographic data. J. Synchrotron Radiat. 21(5), 1188-1193 (2014)

D.M. Pelt, D. Gürsoy, W.J. Palenstijn, J. Sijbers, F. De Carlo, K.J. Batenburg, Integration of tomopy and the astra toolbox for advanced processing and reconstruction of tomographic synchrotron data. J. Synchrotron Radiat. 23(3), 842-849 (2016)

Views

Downloads

CSCD

Alert me when the article has been cited

Submit

Publicity Resources

Microtaper leaky-mode spectrometer with picometer resolution

Computational coherent Raman scattering imaging: breaking physical barriers by fusion of advanced instrumentation and data science

Polarization-based underwater geolocalization with deep learning

Computational imaging without a computer: seeing through random diffusers at the speed of light

Related Author

Yaoguang Ma

Xinying He

Yuwei Tang

Xinhang Liu

Sijie Pian

Qingqing Cen

Haonan Lin

Ji-Xin Cheng

Related Institution

State Key Laboratory of Extreme Photonics and Instrumentation, College of Optical Science and Engineering; International Research Center for Advanced Photonics, Zhejiang University

ZJU-Hangzhou Global Scientific and Technological Innovation Center

Jiaxing Key Laboratory of Photonic Sensing & Intelligent Imaging, Intelligent Optics & Photonics Research Center, Jiaxing Research Institute, Zhejiang University

Department of Biomedical Engineering, Department of Electrical and Computer Engineering, Photonics Center, Boston University

Department of Electrical and Computer Engineering, University of Illinois at Urbana-Champaign

⁰