For patients who are suffering from epilepsy, how quickly and accurately detect seizures is an important issue. Electroencephalography (EEG) is one of the most widely-used measures for the seizure detection and thus has been used in many linear model/deep neural network-based methods. However, those existing EEG-based seizure detection methods have been hindered by limitations such as high latency and/or inconstant seizure detection ability. In this work, we propose an attention-based deep learning algorithm to handle these limitations. Further, the algorithm is learned in an end-to-end manner by combining a seizure EEG representation and a classification stage. To be specific, the proposed network exploits two encoder networks to represent seizure EEG. Then, with the attention mechanism, our network captures temporal interactions from the learned features. Finally, the proposed method efficiently and effectively identifies seizures. We demonstrate the validity of our proposed work by conducting classification of seizures using a publicly available CHB-MIT dataset. Further, we also compare the proposed network to other competitive state-of-the-art methods with an appropriate statistical analysis. Last but not least, we inspect the real-world usability of our method by estimating latency time.