The integration of a significant number of distributed generators has altered the structure and control methods in distribution networks. To address the voltage stability issues caused by the intermittency and fluctuation of distributed generators, this paper proposes the stabilization of the distribution network voltage by adjusting the distribution of reactive and active power flows within the system. A distribution network voltage coordinated optimization method is proposed based on the counterfactual multi-agent policy gradients (COMA) algorithm. The proposed method can use a counterfactual baseline to resolve the “credit assignment” challenge in multi-agent reinforcement learning, enabling the joint optimization scheduling of active power generation and reactive power compensation devices. Agents select actions based on local observations, thereby reducing the system’s communication load and eliminating the dependency on precise flow models, to achieve real-time optimization control of distribution networks. The feasibility and effectiveness of the proposed algorithm are demonstrated by using the improved IEEE33-node system and 141-node system. Compared with the classic control algorithms, the proposed method has further performance advantages in the voltage optimization and control problems for distribution networks.