LiberTEM-live: High-performance live data processing

A. Clausen1*, D. Weber1, M. Bryan2, R. Dunin-Borkowsi1

1 Forschungszentrum Jülich 2CEA-LETI

*a.clausen@fz-juelich.de

Modern direct electron detectors generate data at high rates through their fast frame rates. Frame-based detectors like the DECTRIS Arina can read out at frame rates of 120 kHz, and event-based detectors like the Timepix3 can even reach the MHz range. This creates opportunities for new scientific applications that are only practical with sufficient detector speed, such as 4D STEM, but also requires suitable software support.

For innovative computational applications and methods, users should be able to quickly develop a combination of instrument control, data processing, visualization and user interaction, as opposed to relying on monolithic pre-made solutions. A modular architecture that separates concerns of data source, processing and visualization can help to re-use and apply code across different applications and instruments. We contribute to the LiberTEM framework [1] for this purpose.

We show how LiberTEM-live, the live processing extension of the LiberTEM framework, can support some of the fastest detectors currently available using only moderate computer hardware. This allows practical live processing for previewing data and adjusting parameters while acquiring. Users can develop their own implementations for high-speed live data processing using only Python. Furthermore, we demonstrate how sparse arrays allow native support for both frame-based and event-based detectors with the same implementation of an algorithm.

References

[1] A. Clausen et al. JOSS 2020, 10.21105/joss.02006.