When the sine-wave grating of a Gabor patch drifts to the left or right, the perceived position of the entire object is shifted in the direction of local motion. In the current paper, we explored whether active control of the physical position of the patch can overcome such motion induced illusory displacement. We created a simple computer game and asked participants to continuously guide a Gabor patch along a randomly curving path. When the grating inside the Gabor patch was stationary, participants could perform this task without error. When the grating drifted to either left or right, we observed systematic errors consistent with previous reports of motion-induced illusory displacement. Specifically, when the grating drifted to the right, participants adjusted the global position of the patch to the left of the target line, and when it drifted to the left, errors were to the right of the line. The magnitude of the errors was consistent with previously reported perceptual judgements for centrally presented items, and scaled systematically with the speed of local drift. Importantly, we found no evidence that participants could adapt or compensate for illusory displacement given active control of the target. The current findings could have important implications for interface design, suggesting that local dynamic components of a display could affect perception and action within the more global application environment.