Deburring is a machining process which involves much noise, vibration and dust, so it can be harmful to workers. Therefore, much research has been done to use a manipulator to perform deburring instead of human workers. The precise tracking of the contour of an arbitrary-shaped part is of major concern in robotic deburring. In this study, a tool path generation method based on the CAD model and the direct teaching method is proposed to minimize the position and orientation errors of the workpieces. Without knowledge of the position and orientation of the workpiece, which is often hard to obtain, the optimal deburring trajectory can be generated by matching the extracted tool path from the CAD model to the teaching points. Furthermore, impedance control is used to avoid applying excessive contact force. From a series of experiments on robotic deburring, the performance of the proposed algorithm is verified.