The AC/DC hybrid transmission system is developing fast, and it plays an important role in region connection. At the same time, it raises out new demands. A new method based on simulated annealing genetic algorithm is proposed to optimize reactive power and realize voltage control. Firstly, it builds the model of AC/DC transmission system, whose objective is the minimum of cost of power loss and reactive power generated by generators and reactive power compensation, the constraints consideration of DC converter, controlling angle, node voltage and power flow balance. Then it gives out the step to solve the model by simulated annealing genetic algorithms. At last, it uses IEEE30, which is added to DC transmission system, to prove that this paper’s method is more effective and the results is better.