Given the problem of an Open-Circuit (OC) fault of power devices in Voltage Source Converters (VSC), a novel OC fault detection method for VSC using instantaneous frequency characteristics is proposed. First, the modal components contained in the three-phase AC current of the VSC are analyzed, and the Empirical Mode Decomposition (EMD) method is used to extract the switching modal components in the three-phase current. Secondly, the instantaneous frequency of the switching modal component is obtained through Hilbert transform so as to detect the OC fault of the power device by the change of the instantaneous frequency characteristics of the switching modal component before and after the OC fault. Finally, the simulation model of a three-terminal VSC interconnection system is built in PSCAD/EMTDC to verify the proposed method. The simulation results show that the proposed method can quickly detect the single or multiple OC fault of a power module in VSC within several switching periods. The method has robust detection performance during power adjustment and power flow reversal. This work is supported by the Science and Technology Project of State Grid Corporation of China (No. 5442SH190006) “Research on Typical Design and Operation Service of Urban Smart Energy System” and the Science and Technology Project of State Grid Shanghai Electric Power Company (No. 5209211900M6).