The characteristic analysis of low-frequency oscillation based on disturbed measured trajectory gradually becomes an important means of modern large grid stability analysis. When empirical mode decomposition (EMD) method is used to analyze the low-frequency oscillation signal, the problems, such as end effect, frequency drift, mode mixing and damping loss, will affect the analysis accuracy of the method. So, the ARMA model-based endpoint extension method, the B spline interpolation method, the fine complex wavelet analysis and the energy difference factor control method are proposed separately to solve the above problems. An improved algorithm is designed by the hybrid application of the above method, which effectively improves the accuracy of detecting signal non-stationary characteristics. Finally, through the testing and analysis of EPRI-36 simulation case and RTDS real-time case, it is verified that the proposed algorithm has high accuracy and effectiveness to detect the modes information of complex low-frequency oscillation signal.