To address the problem of power fluctuations caused by direct grid connection of wind power, this paper proposes a capacity optimization algorithm based on improved Archimedes optimization algorithm with complete ensemble empirical mode decomposition with adaptive noise (CEEMDAN). A weighted filtering algorithm, combining amplitude limiting and moving average, is used to smooth wind power output and reduce the lag in the smoothed signal, thus generating the reference power for grid-connected wind power and the hybrid energy storage system (HESS). To allocate HESS’s internal power, the reference power is decomposed into high- and low-frequency components using CEEMDAN. Taking into account factors such as HESS power and capacity, state of charge (SOC), and load defect rate, a capacity optimization model aiming at the minimum annual comprehensive cost is constructed and solved by improved Archimedes optimization algorithm. Simulation analysis based on real case data shows that, compared with the original grid-connected wind power, the proposed HESS configuration scheme reduces power fluctuation by 13.538% and increases smoothness by 16.057%. Compared with the traditional single energy storage, the proposed method achieves better fluctuation mitigation and reduces the required capacity. Moreover, it lowers investment cost by 15.325% compared to the conventional Archimedes optimization algorithm.