There are many forms of network structures in the process of digital substation layer. It is significant to research the process layer’s reliability under different network structures of relay protection system for the network structure selection of process layer and engineering application. Based on the digitized substation relay protection system structure characteristic, this paper puts forward a reliability analysis method for protection system. The protection system is divided into sampling subsystem and trip subsystem and their fault tree models are established respectively. The method comprehensively considers the equipment failure and repair process and makes use of the fault tree and Monte Carlo simulation to solve the reliability protection of system. Finally, the proposed method is used to evaluate the reliability of the protection system with different network structures. The main elements affecting the reliability of protection system are given by the analysis of the component probability importance.