Stable haptic interaction has been studied extensively by an energy-based approach. However, the energy in the haptic system is not directly measurable, but estimated from some measured quantities such as force and velocity; therefore, the estimated energy is occasionally inaccurate. Furthermore, haptic stability is difficult to observe by the energy-based approach for complicated virtual environments. To resolve this problem, a new observer, called a haptic stability observer, is proposed in this research. The empirical investigation of unstable haptic interaction in the frequency domain showed that the typical unstable behavior of a haptic system is different from random oscillations introduced by the hand motion of a human operator. The haptic stability observer working in the frequency domain quantifies the degree of instability of a haptic system. A double layered wall was implemented in the virtual environment to verify the validity of the HSO. Experimental results of the double layered wall show that the HSO is much faster than the passivity observer in detecting haptic instability.