Several geographical latent representation models that capture geographical influences among points-of-interest (POIs) have been proposed. Although the models improve POI recommendation performance, they depend on shallow methods that cannot effectively capture highly non-linear geographical influences from complex user-POI networks. In this paper, we propose a new graph-based geographical latent representation model (GGLR) which can capture highly non-linear geographical influences from complex user-POI networks. Our proposed GGLR considers two types of geographical influences: ingoing influences and outgoing influences. Based on a graph auto-encoder, geographical latent representations of ingoing and outgoing influences are trained to increase geographical influences between two consecutive POIs that frequently appear in check-in histories. Furthermore, we propose a graph neural network-based POI recommendation model (GPR) that uses the trained geographical latent representations of ingoing and outgoing influences for the estimation of user preferences. In the experimental evaluation on real-world datasets, we show that GGLR effectively captures highly non-linear geographical influences and GPR achieves state-of-the-art performance.