Recently, Nguyen proposed a method for tracking a nonparameterized object (subject) contour in a single video stream. Nguyen 's approach combined outputs of two steps: creating a predicted contour and removing background edges. In this paper, we propose a method to increase object tracking accuracy by improving the background edge removal process. Nguyen 's background edge removal method of leaving many irrelevant edges is subject to inaccurate contour tracking. Our accurate tracking is based on reducing affects from irrelevant edges by selecting the boundary edge only. We select high-valued edge pixels of average image intensity gradients in the contour normal direction. Our experimental results show that our tracking approach is robust enough to handle a complex-textured scene.