In this paper, an efficient multi-scale level set method is proposed to track a non-rigid object contour in real time. The proposed algorithm consists of two cycles which are carried out in different scale domains. In the first cycle on the coarse scale, the algorithm evolves the contour depending on the observed data. In the other cycle on the fine scale, the smoothness regularization is imposed on the curve. In order to maintain the curve data during the scale transition, we propose a simple pattern to interpolate empty grids effectively. In addition, we propose a position prediction scheme so as to accelerate the convergence rate of the algorithm. Experimental results show that dramatic speedup is achieved on video tracking experiments without the performance degradation.