With the continuous increase of the penetration rate of new sources, it cannot meet the frequency modulation needs of new power systems relying only on a traditional thermal power unit. Therefore, multi-type unit joint frequency modulation has become the main measure to alleviate the current grid frequency fluctuation problem. This paper proposes a multi-source optimal combined frequency modulation strategy based on improved multi-agent Q learning. First, it analyzes the frequency modulation characteristics of various types of energy and designs a control strategy for the joint frequency modulation system. Secondly, the multi-agent Q learning algorithm is improved, the pre-learning result as the initial matrix of the algorithm is selected, and a search factor is introduced on the basis of the greedy strategy, which greatly improves the optimization effect and shortens the running time. Finally, the dynamic decision-making ability of the algorithm and the PSCAD/EMTDC model are used for joint simulation and verified under the two load disturbance conditions. The method in this paper can minimize the system frequency fluctuation and shorten the time required for frequency modulation, providing favorable conditions for primary frequency modulation. This work is supported by the National Key Research and Development Program of China (No. 2019YFB1505400).