To solve the optimal power flow problem of VSC-HVDC system, the primal dual interior point method or intelligent algorithm is often used. However, the primal dual interior point method can not solve the OPF (such as reactive power optimization) with discrete variables. And intelligent algorithm is easy to fall into local optimal solution when solving this kind of problem and the computation time is too long. Therefore, a simplified null space interior point algorithm with discrete penalty function is proposed. The main idea of the algorithm is to optimize continuous variables within the framework of S-NSIPM. When the convergence function is less than a certain value, it adds penalty function into the calculation of discrete quantity, at the same time adjusts the penalty factor as the iteration between the iteration quantity changes. Numerical examples show that the algorithm has high stability, good adaptability and strong adaptability, and can solve the optimization problem of VSC-HVDC system with discrete variables.