To address issues such as overfitting and poor generalization caused by small-sample data in wind turbine gearbox fault detection, this paper proposes a fault detection model based on the reinforcement learning Mahalanobis-Taguchi system (RLMTS). First, features filtered by orthogonal array and signal-to-noise ratio analysis are used to construct the initial Mahalanobis space. Then, reinforcement learning and predefined rules are used to explore and optimize this space. Finally, an antlion optimizer is employed to improve the threshold determination of the traditional Mahalanobis Taguchi system. Experimental results show that RLMTS is effective for fault detection in different small-sample scenarios. Compared with 17 other methods, RLMTS has better diagnostic performance, greater robustness, and broader applicability, making it particularly suitable for small-sample wind turbine gearbox fault detection. It is conducive to improving the reliability, efficiency and safety of the gearbox operation, while also reducing maintenance cost and ensuring stable and high efficiency wind power generation.