The security of current key exchange protocols such as Diffie-Hellman key exchange is based on the hardness of number theoretic problems. However, these key exchange protocols are threatened by weak random number generators, advances to CPU power, a new attack from the eavesdropper, and the emergence of a quantum computer. Quantum Key Distribution (QKD) addresses these challenges by using quantum properties to exchange a secret key without the risk of being intercepted. Recent developments on the QKD system resulted in a stable key generation with fewer errors so that the QKD system is rapidly becoming a solid commercial proposition. However, although the security of the QKD system is guaranteed by quantum physics, its careless implementation could make the system vulnerable. In this paper, we proposed the first side-channel attack on plug-and-play QKD system. Through a single electromagnetic trace obtained from the phase modulator on Alice's side, we were able to classify the electromagnetic trace into four classes, which corresponds to the number of bit and basis combination in the BB84 protocol. We concluded that the plug-and-play QKD system is vulnerable to side-channel attack so that the countermeasure must be considered.