This study presents the design and development of a Convolutional Neural Network (CNN) for the semantic segmentation of microstructural metallographic images of steel, identifying four key phases: pearlite, ferrite, martensite, and retained austenite. The proposed workflow covers each step from metallographic sample preparation to model training and evaluation. The chosen CNN architecture, U-Net, was selected for its robustness and generalizability, especially when trained on limited datasets. This study furthermore investigates the feasibility of using U-Net to distinguish retained austenite from martensite—a task traditionally reliant on X-ray Diffraction and Electron Backscatter Diffraction (EBSD) rather than light microscopy. Additionally, this work addresses common challenges in semantic segmentation, such as border distortions and artifacts that arise during patch-based image segmentation and reassembly, proposing a solution to mitigate these issues for enhanced image fidelity and model accuracy.