Quantitative analysis of perivascular spaces (PVSs) is important to reveal the correlations between cerebrovascular lesions and neurodegenerative diseases. In this study, we propose a learning-based segmentation framework to extract the PVSs from high-resolution 7T MR images. Specifically, we integrate three types of vascular filter responses into a structured random forest for classifying voxels into PVS and background. In addition, we also propose a novel entropy-based sampling strategy to extract informative samples in the background for training the classification model. Since various vascular features can be extracted by the three vascular filters, even thin and low-contrast structures can be effectively extracted from the noisy background. Moreover, continuous and smooth segmentation results can be obtained by utilizing the patch-based structured labels. The segmentation performance is evaluated on 19 subjects with 7T MR images, and the experimental results demonstrate that the joint use of entropy-based sampling strategy, vascular features and structured learning improves the segmentation accuracy, with the Dice similarity coefficient reaching 66%.