The standard Active Shape Model (ASM) generally uses a whole population to train a single PCA-based shape model for segmentation of all testing samples. Since some testing samples can be similar to only sub-population of training samples, it will be more effective if particular shape statistics extracted from the respective sub-population can be used for guiding image segmentation. Accordingly, we design a set of hierarchical shape statistical models, including a whole-population shape model and a series of sub-population models. The whole-population shape model is used to guide the initial segmentation of the testing sample, and the initial segmentation result is then used to select a suitable sub-population shape model according to the shape similarity between the testing sample and each sub-population. By using the selected sub-population shape model, the segmentation result can be further refined. To achieve this segmentation process, several particular steps are designed next. First, all linearly aligned samples in the whole population are used to generate a whole-population shape model. Second, an affinity propagation method is used to cluster all linearly aligned samples into several clusters, to determine the samples belonging to the same sub-populations. Third, the original samples of each sub-population are linearly aligned to their own mean shape, and the respective sub-population shape model is built using the newly aligned samples in this sub-population. By using all these three steps, we can generate hierarchical shape statistical models to guide image segmentation. Experimental results show that the proposed method can significantly improve the segmentation performance, compared to conventional ASM.