Unlike simple images processed by the existing image-based search engines, flowers have wider and more irregular range of shapes and patterns. In this paper we present an automatic recognition system of flowers for smartphone users. After a user transmits a flower image to the server, the image processing and searching is performed only by the server, eliminating the user interaction from the recognition process. The server detects the contour of a flower image by using both color-based and edge-based contour detection. Then, we classify its color groups and contour shapes by using k-means clustering and history matching. After comparing the input image with the reference images stored on the server, the server sends the most similar image to the user. We also address the image recognition failure issue caused by the light and the camera angle by partial recognition and image recovery. We have obtained the success rate of 94.8 % for 500 images from 100 species.