In subsynchronous frequency range, usually there are multiple modes of torsional vibration for turbine generator shaft. At present, the effective damping strategy is deficient for multi-mode subsynchronous oscillation induced by high voltage direct current. In power system, the phenomena have happened several times that obvious shaft oscillations still exist after subsynchronous damping controller has been applied. In this paper, state space method is used to analyze shaft mechanical system, and the concept of mode observability and controllability degree of subsynchronous oscillation is established. In the design of subsynchronous damping controller, the parameters are optimized based on observability and controllability degree of subsynchronous oscillation modes. Simulation results of PSCAD/EMTDC indicate that traditional subsynchronous damping controller is not very effective for the oscillation mode with relative low observability and controllability degree and the subsynchronous damping controller based on observability and controllability degree proposed is very effective for all subsynchronous oscillations in different modes. The product of this paper has important reference value for damping multi-modes subsynchronous oscillation induced by high voltage direct current.