Renewable energy power generation systems usually connect the LCL filter at the outlet of the inverter to suppress high-frequency harmonics. However, in the parameter design process of the LCL filter, unreasonable parameter design will aggravate the problem of harmonic instability in the system. Therefore, by combining the external characteristic requirements of the LCL filter and considering the constraints of the system stability on the filter, the multi-objective optimization model of LCL filter parameters with the largest harmonic attenuation rate, the lowest inductance cost and the best system stability is established. In the process of solving the model, for the single target optimization algorithm and the traditional multi-objective optimization algorithm, there are problems such as single design goal, poor convergence and poor optimization effect, the MOEA/D multi-objective optimization algorithm is proposed. The algorithm utilizes the evolutionary orientation of weight vectors and the dominant sharing characteristics of subproblem neighborhoods to guarantee the ability of the algorithm to find the optimal solution and the uniform distribution in the target space, the purpose of which is to get the ideal pareto optimal frontier, then the fuzzy membership function is used to quantitatively analyze its optimization effect. Finally, the voltage source converter (VSC) in the photovoltaic grid-connected system is simulated by Matlab/Simulink software. The simulation results fully demonstrate the effectiveness and applicability of the proposed design. This work issupported by Youth Science Fund Project of National Natural Science Foundation of China (No. 51807114).