To explore the adaptability of existing traveling wave protection for the DC line to Baihetan-Jiangsu serial hybrid ultra-high voltage direct current (UHVDC) transmission system (Bai-Jiang hybrid system for short), this paper deduces the boundary propagation characteristics of the fault traveling wave at both ends of the DC line during faults inside and outside the DC line zone based on the fault additional network of the Bai-Jiang hybrid system. The analysis finds that the boundary propagation characteristics of the forward zone of the DC line faults differ from those of the conventional line commutated converter (LCC) HVDC system. The changes for the traveling wave protection criterion are analyzed, and this analysis shows that the traveling wave protection criterion is mainly affected by the reflection coefficient on the rectifier side and the refraction coefficient on the inverter side. Comparing the Bai-Jiang hybrid system with a conventional LCC-HVDC system, it is found that when a short-circuit fault occurs outside the forward zone of the DC line, the voltage change amount and change rate, the polar wave change amount and change rate of the Bai-Jiang hybrid system change more than those of the conventional LCC-HVDC system. That leads to an increased risk of false operation of the traveling wave protection. Finally, a simulation model of the Bai-Jiang hybrid system is built based on PSCAD, and the results verify the correctness of the theoretical analysis. This work is supported by the National Natural Science Foundation of China (No. 51977183).