In this paper we introduce an influenza surveillance and forecast system that can track the proliferation of influenza and predict potential infections by analyzing smartphone sensor readings. While previous studies focus on the social connectivity to deduce proliferation paths, we investigate physical contacts and their surrounding features including the staying time of these contacts, the human density and the openness of the space, and the infection status of each individual. By using a smartphone equipped with various sensors we can estimate the infection status of its owner as well as the surrounding features based on the analysis of the envelope of incoming sound and the mobility history of the owner. A surveillance server, which aggregates the information from multiple smartphones, monitors the infection status of influenza and ranks both the high risk persons and the influential persons that have to be vaccinated promptly. To evaluate the performance of our system we model the proliferation of influenza by applying both an influenza infection model and a community mobility model to mobile agents in NS-2 simulator. The simulation results show that the forecast accuracy of our system is 90.2% while the accuracy of forecast based on the social connectivity alone is 75.3%. By using the proliferation forecast our system can reveal influential persons, reducing 33.5% of infections by vaccinating only 6% of the entire group.