Accurate animal sound classification is an important task in automated animal monitoring system. Such monitoring system is essential for preventing epidemics caused by animal disease. Based on such needs, there has been a variety of efforts to develop an accurate system performing animal sound classification in deep learning framework. Although many research issues and methods to address the issues were introduced, no one has yet to address overcoming the machine learning barriers induced by a single objective function. As learnable parameters only consider a single penalty at the output prediction for training, they cannot capture other characteristics contained in the dataset to extract more generalized prediction. This paper proposes a deep learning based multi-task learning framework for animal sound classification. Both animal species and group classification are performed in an end-to-end learning process. Experimental results show that the proposed multi-task method outperforms single-task method in our recorded animal sound dataset.