The early postnatal period witnesses rapid and dynamic brain development. Understanding the cognitive development patterns can help identify various disorders at early ages of life and is essential for the health and well-being of children. This inspires us to investigate the relation between cognitive ability and the cerebral cortex by exploiting brain images in a longitudinal study. Specifically, we aim to predict the infant brain development status based on the morphological features of the cerebral cortex. For this goal, we introduce a multi-view multi-task learning approach to dexterously explore complementary information from different time points and handle the missing data simultaneously. Specifically, we establish a novel model termed as Latent Partial Multi-view Representation Learning. The approach regards data of different time points as different views, and constructs a latent representation to capture the complementary underlying information from different and even incomplete time points. It uncovers the latent representation that can be jointly used to learn the prediction model. This formulation elegantly explores the complementarity, effectively reduces the redundancy of different views, and improves the accuracy of prediction. The minimization problem is solved by the Alternating Direction Method of Multipliers (ADMM). Experimental results on real data validate the proposed method.