Anti-polarity of zero sequence CT leads to failure of fault line selection in extinction coil grounding system. In order to overcome the drawback, a new approach using compensation characteristics of extinction coil is proposed. The healthy line mathematical model for each line is built and the transient component and steady component of zero sequence voltage and current are used to get corresponding admittance parameters. To solve the component annihilation, this paper presents an improved matrix pencil algorithm for the extraction of high frequency component of small amplitude. The rule of the compensation effect of extinction coil is obtained that in set frequency band two component parameters of fault line are opposite in sign or decrease significantly in numerical values and the calculation results for two parameters of healthy line are similar, the fault line selection is realized. Simulation results verify the correctness of the algorithm without the impact of anti-polarity of zero sequence CT, which has highly adaptive capacity for practical engineering.