The nonlinear frequency response characteristics of a spur gear pair with fractional-order derivative under combined internal and external excitations are investigated based on the incremental harmonic balance (IHB) method. First, a pure torsional vibration model is proposed that contains various complex factors, such as the time-varying mesh stiffness, transmission error, the fluctuation of input torque, backlash. Then, the IHB method is developed to calculate the higher-order approximate solution of the system, and the correctness of the results is verified by comparing with numerical simulation results obtained by the power series expansion (PSE) method. Furthermore, the types of various impact situations and their judgment conditions are discussed, and the different impact behaviors are analyzed in detail when by using phase diagrams and amplitude–frequency response curves. The influence of important parameters on the dynamic characteristics of gear pair is analyzed at last. The results indicate that the analytical solution derived by IHB method is sufficiently precise. Significantly, the dynamic characteristics of the system could be effectively controlled by adjusting time-varying mesh stiffness coefficient, the order and coefficient of fractional-order term, and the amplitude of internal excitation or external excitation. As a part of the theory of fractional-order mechanical system, the impact performance of fractional-order gear pair is approached for the first time by analytical method.