In ultra high voltage transmission system, series compensated capacitor has many advantages such as enhancing the transmission capacity and improving the stability of power system. But series compensated capacitor also generates many problems in protective relaying of transmission lines. In this paper, according to the theoretical analysis of the distance protection and zero-sequence direction protection affected by series capacitor, it is presented that a microprocessor-based protection algorithm based on TCSC capacitor. The algorithm uses the operation parameters of TCSC, which considers the effect of the transient process of series compensated capacitor and the state of capacitor operation to protection parameters setting. So it can avoid protective relaying rejection or malfunction. At last the results of computer simulation confirm that this algorithm has high adaptability and reliability in different TCSC operation, which can satisfy the basic requirements of protective relaying.