Strong nonlinear IV characteristics existing in PV array may cause the maximum power point tracking voltage of photovoltaic system in different nonlinear section.The irrational design of the system controller may cause system instability or even collapse.This paper aims at the unstable region in photovoltaic system to analyze the phenomenon of instability in the MPPT control through establishing system state equation and transfer function.The influencing parameters are determined via simulation and experiment.It is concluded that the stability of the system can be enhanced by adjusting the system control parameters and properly selecting bus capacitance.