This study was aimed to analyze the influence of the cutting parameters (spindle speed, feed rare and cutting depth) on the surface roughness of the machined parts with the influence of the machining stability of the cutter. To consider the chattering effect, the machining stabilities were calculated based on the measured tool tip frequency response functions. A series of machining tests were conducted on aluminum workpiece under different cutting parameters. Then the surface roughness prediction models in the form of nonlinear quadratic and power-law functions were established based on the multivariable regression method, in which the input parameters, cutting depth and spindle speed, were respectively defined in stable and unstable regions according to the stability lobes diagram. Current results show that both models built with cutting parameters defined in stable regions demonstrate higher prediction accuracy of the surface roughness, about 90%, when compared with the models defined in full regions with accuracy about 80%. In particular, the power-law model is proven to have 90 % prediction accuracy when validated with the cutting parameters in stable region. As a conclusion, the mathematical models based on the cutting parameters with well defined machining stability are proven to show more accurate prediction ability of the surface roughness. It could be expected that the prediction model can further be applied to optimize the machining conditions in low speed roughing and high speed finishing process with desirable surface quality.