Large-scale integration of new energy through AC interconnection lines has become a typical scenario in modern power systems. In the present situation, because of the influence of the operation mode of new energy systems, power frequency variation-based distance protection faces the risk of incorrect operation. To further investigate the performance degradation mechanism of this protection method, voltage plane analysis reveals that under the control strategy of photovoltaic (PV) power stations, the positive-sequence current at the PV station output exhibits phase- and magnitude-controlled nonlinear characteristics. This causes the protection compensation voltage to vary with fault conditions, with its vector direction undergoing uncertain deviations, leading to an unclear operating region for protection operation. Consequently, the performance of impedance relays based on traditional setting criteria is severely degraded. On this basis, leveraging the relationship between the boundary conditions for correct operation in distance protection and the PV station’s integration capacity, a method for rapid assessment of protection applicability and calculation of the integration capacity value at the boundary of protection failure is proposed. A PSCAD/EMTDC simulation model of the distribution network is established, and the correctness of the theoretical analysis are validated through extensive simulations.