Binocular stereo measurement system can obtain accurate three-dimensional information from two-dimensional images. It has been widely applied in many fields such as vehicle tracking, robot navigating, automatic crane lifting, as well as other fields. The crucial step of binocular stereo measurement is image matching. For the image matching, it is a great challenge to ensure both real-time and matching accuracy simultaneously. The image matching algorithm has a great influence on the image matching time and accuracy. In this paper, a real-time image matching algorithm for binocular stereo measurement system is proposed based on Speedup Robust Features (SURF) algorithm. In the proposed algorithm, firstly, the key feature points are identified by the original SURF algorithm method. Secondly, the main direction of the key feature point is determined by intensity centroid method. Then, the feature descriptor is calculated by the BRIEF binary method so that the time of feature description can be shortened. Finally, RANSAC (Random Sample Consensus) method is adopted to remove mismatching points. The experiments results show that the proposed algorithm can shorten image matching time obviously and improve the accuracy of matching points.