Superconductor magnetic energy storage (SMES) system has rapid response and controls active and reactive power independently, so it is applied more broadly. But its charge or discharge processes will exchange power with the power system and may affect relay protection of transmission line. This paper builds an electromagnetic transient simulation model for SMES and studies characteristics of the polygon distance relay of transmission line in one-machine-infinite-bus (OMIB) system with SMES in PSCAD. The simulation on the polygon distance relay is conducted under different faults and at different installation sites of SMES. Results show that when the SMES is connected to the power grid, the measured impedance is changed and the relay action is affected. At last, a scheme for improving the relay protection characteristic curve is proposed and proved to be feasible by simulation.