To rationalize and improve the performance of all-solid-state Li-ion batteries with state-of-the-art nickel-rich layered oxide cathodes, it is crucial to understand the ion intercalation dynamics, lithium heterogeneities, crystallographic defect generation, local stress responses and degradation mechanisms occurring during battery operation. In this work, we developed a microstructure-resolved physics-based battery model to investigate the electro-chemo-mechanical behavior for both single- and polycrystal composite cathodes with sub-grain-level resolution. The reaction at the cathode|electrolyte interface and Li-ion intercalation inside the cathode is described by the Butler-Volmer and Cahn-Hillard models, respectively. The formulations are combined with a crystal plasticity constitutive model to consider the anisotropic a-axis and c-axis lattice strain and dislocation gliding on the ab-plane. State of charge-dependent material parameters of the multi-physics model (a and c lattice parameters, open circuit voltage, Li-ion diffusion coefficients, shear strength) are quantified by various experiments, e.g. operando diffraction analysis, GITT experiments, solid-state nuclear magnetic resonance spectroscopy, and nano-indentation experiments. This model was then used to study the interplay between the microstructure and electro-chemical performance, in order to elucidate the role of heterogeneity in voltage degradation. Finally, we discussed the implications of lithium heterogeneities and dislocation accumulation for the irreversible structure reconstruction from the initially layered structure to spinel and/or rock-salt-like structure.