Interharmonic is a kind of harmonic signal which is not an integer of the fundamental frequency component. It is harmful to power quality and reduces power system reliability, so measuring inter-harmonics accurately is important to power system safety and stability. This paper introduces the atomic decomposition based on particle swarm optimization to the measurement and analysis of inter-harmonics in power system for the first time, which can overcome the shortcoming such as long computing time of traditional matching pursuit algorithm. This method applies the particle swarm optimization to the atom decomposition iterative optimization process, selects the best matched atoms reflecting the signal feature by comparing the particles’ best fitness values, extracts harmonics and inter-harmonics component, obtains the time-frequency distribution which has high time-frequency resolution and is independent of the cross-term interference, and finally analyzes the power system harmonic accurately. The two simulation examples indicate the method has high accuracy, better adaptivity and anti-noise abilities. It also provides a new way for the accurate analysis of power system inter-harmonics.