With the “dual-carbon” goal, the high penetration of distributed energy and intensified coupling of heterogeneous energy sources have made it difficult to solve the optimal dispatch problem in integrated energy systems (IES). Deep reinforcement learning provides an effective means to address this challenge. However, traditional deep reinforcement learning usually weights the safety constraints to the reward function in the form of penalty terms, and the weighting coefficients are usually determined manually and remain fixed during iterations, affecting the convergence performance and constraint handling capability of the algorithm to some extent. To address this issue, this paper proposes an IES optimal dispatch method based on constrained reinforcement learning. First, a safety value network based on IES unit operation and system power flow constraints is constructed. The safety and an economic value of agent decisions are evaluated respectively through the dynamic parallel synergy of Lagrange multipliers and an economic value network. Second, the primal-dual approach is used to update the agent policy and Lagrange multipliers alternately to circumvent the influence of subjective bias caused by manually set penalty coefficients in IES scheduling decisions. Additionally, expert knowledge is leveraged to guide the training process to prevent computational resource waste due to blind optimization. Finally, simulation case studies are carried out based on an electric-thermal coupled system to verify the safety and efficiency of the proposed method.