Vessel segmentation and anatomical labeling are of great significance for vascular disease analysis. Because vessels in 3D images are the tree-like tubular structures with diverse shapes and sizes, and direct use of convolutional neural networks (CNNs, based on spatial convolutional kernels) for vessel segmentation often encounters great challenges. To tackle this problem, we propose a graph convolutional network (GCN)-based point cloud approach to improve vessel segmentation over the conventional CNN-based method and further conduct semantic labeling on 13 major head and neck vessels. The proposed method can not only learn the global shape representation but also precisely adapt to local vascular shapes by utilizing the prior knowledge of tubular structures to explicitly learn anatomical shape. Specifically, starting from rough segmentation using V-Net, our approach further refines the segmentation and performs labeling on the refined segmentations, with two steps. First, a point cloud network is applied to the points formed by initial vessel voxels to refine vessel segmentation. Then, GCN is employed on the point cloud to further label vessels into 13 major segments. To evaluate the performance of our proposed method, CT angiography images (covering heads and necks) of 72 subjects are used in our experiment. Using four-fold cross-validation, an average Dice coefficient of 0.965 can be achieved for vessel segmentation compared to that of 0.885 obtained by the conventional V-Net based segmentation. Also, for vessel labeling, our proposed algorithm achieves an average Dice coefficient of 0.899 for 13 vessel segments compared to that of 0.829 by V-Net. These results show that our proposed method could facilitate head and neck vessel analysis by providing automatic and accurate vessel segmentation and labeling results.