The nonlinearity in conversion between electric signal and light signal will make terrible effect on the performance of radio-over-fiber (RoF) system. As a result, how to decrease the nonlinearity and improve the performance of the link becomes the focal point. Based on the former researches, this paper aims at studying the method to inhibit the third order inter-modulation distortion (IMD3), caused by the nonlinearity of the modulator, and then to get higher linearity of the system. Firstly, the basic plan to achieve RoF system is proposed. The linearity of the system is not good resulting from the serious nonlinear distortion of the modulator, and the result shows that Spurious free dynamic range (SFDR) of this plan can only reach 111.32 dB.Hz^2/3. Based on the first plan, an improved scheme is proposed. In order to improve the linearity, this scheme adds a Mach–Zehnder Modulator (MZM) and a phase shifter to generate an unmodulated optical carrier, so the main carrier from the unmodulated signal and the main carrier caused by the modulated signal can inhibit each other. Under this plan, SFDR can reach 121.8 dB.Hz^2/3. The experimental results show that SFDR of the second scheme is 10.48 dB.Hz^2/3 higher than that of the basic one and the linearity of the system has been improved successfully. This work is supported by National Natural Science Foundation of China (No. 61405142).