In an AC/DC hybrid power system, converter and on-load-tap-changer (OLTC) models have a large number of non-smooth characteristics such as limits and a dead zone. How to deal with the piecewise function constraints introduced by a converter is a difficult problem in existing state estimation. A mixed integer nonlinear programming (MINLP) model of OLTC is proposed and tap position is estimated by controlling the secondary voltage level. Given that the converter uses piecewise functions to describe droop control characteristics including limits, dead zone and so on, the piecewise functions are smoothed by approximate fitting functions. The proposed model significantly improves the computational efficiency compared with the method of using MINLP to describe piecewise functions. The correctness and practicability of the proposed SE model are verified on a 3-bus system and IEEE14-bus AC/DC hybrid systems, and the influence of the fitting coefficient on state estimation is analyzed. Numerical examples show that the proposed model can enhance the ability of state estimation to deal with bad data and improve the adaptability to non-smooth characteristics. The convergence of state estimation is improved as well as ensuring calculation accuracy. This work is supported by the National Natural Science Foundation of China (No. 52177125 and No. 51707196).