The structure and theory of relay protection system in smart substation have changed greatly. How to analyze the reliability of relay systems objectively and properly is an important problem. The information can be divided into three parts and the topology is analyzed rationally by minimal path sets algorithm, then the theory of intuitionistic trapezoidal fuzzy numbers is used to analyze and calculate the fuzzy and uncertain failure probabilities of devices. With the reliability indexes of relay protection system and the software of model, every minimal path can be analyzed by the decision-making method based on intuitionistic trapezoidal fuzzy numbers and the weakness of relay system is identified. This method proposed is proved correct and practical by the sample model.