The unsteady flow in the compressor at small mass flow rate has an important impact on the safety and efficiency of the compressor. Rotating instability was found in the experiment at near stall condition. Through URANS simulation, the origin of unsteady flow in an isolated subsonic rotor is studied. And the relationship between unsteadiness of tip leakage flow and rotating instability is revealed. With the deepening of the throttle, the flow field in the rotor changes from steady to unsteady. The intermittent spiral type breakdown of tip leakage vortex is considered to be the origin of the unsteady flow. Quantitative analysis of the tip leakage vortex shows the breakdown cycle caused by the interaction of the tip leaked vortex with the adjacent blade. When the tangential velocity and axial velocity of the leakage vortex reach a critical value, the tip leakage vortex will break. A radial vortex called back flow vortex will appear periodically after breakdown happens, which plays an important in rotating instability. The back flow vortex at upstream causes an overflow at adjacent blade leading edge, which results the next breakdown happens at downstream. Due to such feedback, the tip leakage vortex breakdown at two location alternately. A possible cause of RI was proposed: The spiral breakdown of the tip leakage vortex at different positions resulted in a cross-passage structure, which propagates into circumferential direction.