In order to solve the problems of time consuming and manual work in fault location of the traditional relay protection system, the research of the intelligent positioning method of the relay protection is carried out. First of all, the main types of faults in the relay protection system of smart substation are analyzed, providing theoretical support for fault simulation experiments. Secondly, the faults in the relay protection are simulated by using PCS-978, and the data of experiments are collected to build a database, providing data support for intelligent positioning method of the relay protection. Then, based on the experiments, the fault location matrix and table are designed. Finally, an intelligent positioning method based on deep neural network is proposed, incorporating with SMOTE and Random Forest algorithm. The empirical results demonstrate that the intelligent location method can improve the efficiency of fault diagnosis in smart substation protection system. The method which can help relay protection technicians locate faults quickly and accurately is intensely practical.