In this paper a novel optimal approach of control strategy is introduced by applying fractional calculus in structure of sliding mode control for a range of dynamics system liable to uncertainties. So, a fractional sliding mode control was designed for dynamics of the two-link rigid-flexible manipulator. Furthermore a multi-objective genetic algorithm was proposed in order to find the ideal variable structure of the sliding mode control. Optimal variables achieved by optimization of the conventional sliding mode control. Then the performance of the both conventional and the fractional sliding mode control were compared with respect to optimal variables. Results indicated that by applying the optimized fractional sliding mode control, system’s error was significantly reduced and, so as tracking the desired value was done with a high accuracy and also a smooth control action was achieved.