To address the issues of sample scarcity and difficulty in identifying unknown categories in distribution transformer fault diagnosis, this paper proposes a joint small-sample and unknown-class recognition method that integrates an improved convolutional variational autoencoder (IM‑Conv VAE) with a probability of inclusion support vector machine (PISVM). First, IM‑Conv VAE is used to augment small-sample data and expand the sample size. Then, feature extraction is performed under the joint loss function constraints of core margin loss. Finally, PISVM is constructed by introducing class probability boundaries to perform open-set recognition on the extracted features, thereby achieving classification of known categories and marking of unknown categories. Experimental results show that under limited sample conditions, the proposed model can identify known faults with high accuracy and effectively distinguish unknown faults. Even when the number of training samples is further reduced, the model maintains good stability and strong recognition capability. The model exhibits excellent fault recognition performance and has high potential for engineering application and promotion.