Automatic parcellation of cortical surfaces into anatomically meaningful regions of interest (ROIs) is of great importance in brain analysis. Due to the complex shape of the convoluted cerebral cortex, conventional methods generally require three steps to obtain the parcellations. First, the original cortical surface is iteratively inflated and mapped onto a spherical surface with minimal metric distortion, for providing a simpler shape for analysis. Then, a registration or learning-based labeling method is adopted to parcellate ROIs on the mapped spherical surface. Finally, parcellation labels on the spherical surface are mapped back to the original cortical surface. Despite great success, spherical mapping of the original cortical surface is inherently sensitive to topological noise and cannot deal with the impaired brains that violate spherical topology. To address these issues, in this paper, we propose to directly parcellate the cerebral cortex on the original cortical surface manifold without requiring spherical mapping, by leveraging the strong learning ability of the graph convolutional neural networks. Also, we extend the convolution to the surface manifold using the kernel strategy, which enables us to overcome the notorious shape difference issue (e.g., different vertex number and connections) across different subjects. Our method aims to learn the highly nonlinear mapping between cortical attribute patterns (on local intrinsic surface patches) and parcellation labels. We have validated our method on a normal cortical surface dataset and a synthetic dataset with impaired brains, which shows that our method achieves comparable accuracy to the methods using spherical mapping, and works well on cortical surfaces violating the spherical topology.