The effect of intercritical annealing (IA) time on the Lüders strain and Portevin-Le Chatelier (PLC) bands of 3.5Mn TRIP steel was study with digital image correlation (DIC) and electron backscattered diffraction (EBSD) techniques in this study. The results show that the IA time has a great influence on the yielding behavior of the steel, and both the yield strength (YS) and yield point elongation (YPE) decrease with the prolonging of annealing time. When IA time is less than 3 h, the decrease of YPE is mainly attributed to the increasing retained austenite (RA) volume fraction, while the IA time exceeds 3 h, it is mainly caused by the formation of α’-martensite during the cooling stage of the IA treatment, which increases the dislocation density in the surrounding ferrite matrix of fresh martensite. However, only when the ferrite grain satisfies (110)α close packed plane parallel (CPPP) relationship with its adjacent martensite during the austenite to martensite transformation process, the transformation induced stresses can be effectively relaxed by relatively easy deformation of the surrounding ferrite neighbors and results in the dislocation density increasing in ferrite grains. Otherwise, only a short-range deformation in the form of a deformed ferrite layer at the ferrite/martensite interface will occur.