Increasing unstructured data of big data in electric system puts forward a new challenge to traditional manual processing mode. As a typical kind of unstructured data, the infrared image is very important for the research of big data in electric system. In order to automatically processing massive infrared fault images, this paper presents a convolutional recursive network based current transformer infrared fault image diagnosis method. The infrared fault images are first segmented by super pixel segmentation method and then we take advantage of the hue information to extract the temperature anomaly area; secondly, a two-level joint convolution recursive neural network is adopted, the fault device position can be identified by training a large number of samples; finally, the fault information is confirmed according to the location information of fault classification. The experimental results show that, this algorithm has better robustness, higher accuracy, and can improve the efficiency of infrared diagnosis, which is also the foundation for the feature representation of unstructured data.