More and more applications are being discovered for scalable video coding (SVC) in network-adaptive video streaming and storage systems. In the SVC encoder, an exhaustive search technique is employed to select the best coding mode for each macro block. This technique achieves the highest possible coding efficiency, but it results in extremely large encoding time. In this paper, we propose an adaptive fast mode decision algorithm for inter layer coding for spatial scalability and coarse grain quality scalability (CGS). In the proposed method, we select candidate modes adaptively for estimation in enhancement layer using the information of base layer and rate distortion (RD) cost of base layer skip mode. RD cost of base layer skip mode gives the correlation information of enhancement layer with base layer and base layer mode is used to reduce the number of candidate modes. We verify that the overall encoding time can be reduced up to 63 % through the analysis of experimental results.