The existing converter station protection of high-voltage direct current transmission (HVDC) system has the problem of misoperation when external faults occur. Based on the integral ratio of characteristic harmonic currents, a protection optimization scheme for HVDC rectifier stations is proposed. In this scheme, the characteristic harmonic current ratio between the primary side of the converter transformer and the AC line at the transmission end is used to identify the internal and external faults. The ratio is greater than 1 when the internal fault occurs, and the ratio is less than 1 when the external fault occurs. Then the time coordination between the protection of the rectifier station and the outside AC system is designed to prevent misoperation of rectifier station protection in the case of AC system failure outside the zone. Simulation results show that the proposed protection optimization scheme can accurately identify internal and external faults, and is not affected by fault type, fault distance, transition resistance, fault initial angle, and noise interference. This work is supported by Key Research and Development Project of Sichuan Province ( No. 2020YFG0063).