In order to reduce the error of the design of relay protection, setting calculation of transformer, a method is proposed based on the combination of internal leakage reactance parameters calculation and short-circuit impedance parameters determination. By using curve fitting technique, this method obtains the exact equivalent short circuit impedance parameters of the transformer in different fault conditions. On this basis, it establishes the custom model of the short-circuit impedance of primary protection and backup protection combining the real digital transformer protection device in the PSCAD/EMTDC by using FORTRAN language interface technology. Besides, using the equivalent system model, the short circuit to ground short circuit, inter-turn short circuit, phase to phase short circuit within the different positions of the transformer coil are simulated and calculated. The results show that, using the curve fitting method model and custom element can accurately simulate and analyze the internal fault of transformer and relay protection action, which provides a scientific basis for the design of relay protection principle and application.