For the harmonic instability problem in AC-DC interconnected transmission system, this paper deduces the criterion to determine whether the DC system occurs core saturation harmonic instability. From HVDC harmonic instability mechanism and considering the DC bias magnetic core saturation effects, the criterion makes use of the characteristic that positive sequence second harmonic amplitude is higher than the other harmonic generated in DC magnetic bias of converter transformer, takes into account the whole process that the second harmonic transfers between AC and DC systems back and forth through inverter overall. In the case analysis, electromagnetic transient model is established and simulated by PSCAD/EMTDC, the result of electromagnetic transient fault simulation is consistent with the result of frequency scanning method based on the new criterion, so the correctness and applicability of engineering criteria is verified.