Stable haptic interaction has been studied extensively by an energy-based approach. However, the energy in the haptic system is not directly measurable, but estimated from some measured quantities such as force and velocity; therefore, the estimated energy is occasionally inaccurate. To resolve this problem, a new observer, working in the frequency domain, is proposed in this research. The observer quantifies the degree of instability of a haptic system, and a proposed controller generates variable damping in proportion to this quantitative instability. Especially, for a double layered virtual wall, the proposed methods were much faster in detecting haptic instability than other schemes, and successfully reduced unstable behavior.