Anomaly detection in the video is a challenging problem in computer vision tasks. Deep networks recently have been successfully applied and achieved competitive performance in anomaly detection. Modern deep networks employ many modules which extract important features. The anomaly detection approaches just developed network architecture and inserted additional networks to improve performance, however, these methods generally require a tremendous amount of computational load and training parameters. Because of limitations in the real world such as field equipment, mobile system, etc., reducing the number of trainable parameters and model capacity is an important issue in anomaly detection. Moreover, the method, which improves the performance of the anomaly detection algorithm, should be developed without additional trainable parameters. In this paper, we propose a sparsity injecting module which reinforces the feature representation of the existing model and presents the abnormality score function using sparsity. In experimental results, our sparsity injecting module improves the performance of state-of-the-art methods without additional trainable parameters.