An effective method of minimizing vibrations during milling of small flexible workpieces is presented, using a new modified approach to the problem of vibration reduction, based on a workpiece holder with adjustable clamping stiffness. A completely new formulation of a mechanistic model of cutting dynamics for milling processes was developed. An important novelty is also the discrete modeling of a flexible workpiece in the convention of the rigid finite element method, as well as the inclusion of the rotational speed of a slender cutting tool in the structural model. It is expected to confirm the research hypothesis that the use of dedicated mechatronic design techniques optimizes the effectiveness of the vibration reduction system during milling, taking into account the dynamic conditions of the cutting process and the use of clamping the workpiece on the milling table in a special holder. A three-dimensional mechanistic description of the milling process will improve the behavior and effectiveness of the workpiece vibration reduction system. The research methodology based on selected mechatronic design techniques assumes computer simulation (virtual prototyping – VP) of the tool-workpiece vibration suppression system, experimentally aided virtual prototyping (EAVP) and implementation of vibration suppression in the target system (ITS). An original method of simulation of a computational model of the milling process in hybrid coordinates was developed, the effectiveness of which was confirmed by the results of machining workpieces of bronze CC331G and aluminum alloy EN AW-6101.