Brain tumor grading is pivotal in treatment planning. Contrast-enhanced T1-weighted MR image is commonly used for grading. However, the classification of different types of high-grade gliomas using T1-weighted MR images is still challenging, due to the lack of imaging biomarkers. Previous studies only focused on simple visual features, ignoring rich information provided by MR images. In this paper, we propose an automatic classification pipeline using random forest to differentiate the WHO Grade III and Grade IV gliomas, by extracting discriminative features based on 3D patches. The proposed pipeline consists of three main steps in both the training and the testing stages. First, we select numerous 3D patches in and around the tumor regions of the given MR images. This can suppress the intensity information from the normal region, which is trivial for the classification process. Second, we extract features based on both patch-wise information and subject-wise clinical information, and then we refine this step to optimize the performance of malignant tumor classification. Third, we incorporate the classification forest for training/testing the classifier. We validate the proposed framework on 96 malignant brain tumor patients that consist of both Grade III (N = 38) and Grade IV gliomas (N = 58). The experiments show that the proposed framework has demonstrated its validity in the application of high-grade gliomas classification, which may help improve the poor prognosis of high-grade gliomas.