Atomic-resolution in situ electron microscopy provides a huge number of parameters that allow new insights into the behavior of materials in reduced dimensions and their material dynamics. We perform in-situ experiments with graphene and metal atom clusters on graphene with our Cc/Cs-corrected SALVE (Sub-Angstrom Low-Voltage Electron Microscopy) instrument. Due to the fast dynamics of graphene atoms under electron bombardment and on the other hand the formation of metal atomic clusters on graphene an immense amount of data is generated both in terms of the number of frames acquired and the amount of information per frame; just as an example, in a 10 nm × 10 nm image, the trajectories of more than 4000 carbon atoms must already be described. This makes the manual evaluation and the application of conventional image analysis methods, such as by hand-made filter kernels, untenable due to their enormous time consumption and the possibility of user bias. Deep learning in the form of convolutional neural networks offers a reliable and effective way to handle large amounts of complex image data. Using simulated training data results in a flexible training pipeline that is robust to user bias. By coupling multiple networks, we have incorporated multi-tasking capability into the workflow, including segmentation of multilayers, contaminants, and micropores and subtraction of non-uniform illumination intensity. To further investigate the generality of our deep learning pipeline, the robustness of the NN was demonstrated on various tasks in different sample systems, including identification of defects and dopants in graphene, shape and positions detection of adsorbed metal clusters, and statistical analysis of amorphous polymeric membrane. By the application of carefully trained neural networks, it becomes possible to fully exploit the richness of data captured in TEM images.