The development of the fourth industrial revolution has influenced various industrial fields worldwide. The Unmanned Aerial Vehicle (UAV) is expected to be one of the most commercialized areas in the world for the next decade. Since the importance of building safety checks in South Korea is growing, this study focuses on developing a preliminary safety inspection process with UAV, specifically for cracks and deformation. The process includes various methods to improve surveying efficiency, such as thermography, machine learning algorithms, and 3D point cloud modeling. After proposing the inspection process, detailed structural evaluating methods are verified against a conventional method. The image processing method is performed to examine exterior defects, and quantitative defect evaluation methods are applied using 3D point cloud models of buildings. As a result, a crack map, crack depth, and deformations of structures are obtained. Error rates are compared between the proposed method and the conventional method. This paper presents an aerial image-based approach using the UAV for inspecting cracks and deformations of buildings.