Commutation failure is one of the most common faults in High Voltage Direct Current (HVDC) transmission system. In order to reduce the probability of commutation failure, a suppression strategy for continuous commutation failure based on nonlinear VDCOL is proposed in this paper. Considering the low sensitivity of DC current command generated by conventional VDCOL, the proposed nonlinear dynamic VDCOL control strategy takes severity of the system failure into account and dynamically adjusts control curve of the VDCOL controller according to voltage level, which improves response speed of the DC current command and limits DC current to reduce the probability of continuous commutation failure. The proposed control method is tested by CIGRE model of HVDC in PSCAD/EMTDC. Simulations show that the proposed nonlinear dynamic VDCOL controller can suppress continuous commutation failure during short circuit fault, and it is beneficial to restore DC voltage and transmission power of the system and ensure the normal operation of HVDC transmission system. This work is supported by the Project of State Grid Hunan Electric Power Company Limited (No. SGTYHT/17-JS-201), the Postdoctoral Innovative Talent Support Program of China (No. BX20180095) and China Postdoctoral Science Foundation (No. 2018M640747).