Atomistic simulation using classical semi-empirical potentials is an important way to gain insight into the microscopic deformation mechanisms, particularly interactions of crystallographic defects in materials. However, the selection of inappropriate potentials can lead to uncertainties in the observed mechanisms due to the artefacts associated with them. Therefore, a systematic assessment of the relevant interatomic potentials for a particular material system is essential. In the present study, we systematically evaluated interatomic potentials of Mg and its alloys, including embedded-atom method (EAM) and modified embedded-atom method (MEAM) potentials, by means of different material characteristics, such as elastic properties, stacking fault energies, grain boundary (GB) energies, per-site segregation energies, as well as dislocation motion on basal, prismatic, and pyramidal planes. The results are compared with experiments and density functional theory (DFT) calculations from the literatures. We found that the EAM potentials, although more efficient for modeling fundamental properties, differed significantly from experimental observations in describing dislocation properties. In addition, we found a strong correlation between the per-site segregation energies of alloying elements at GBs and the local GB site volumes.