This paper introduces a new methodology for probabilistic optimal design of multibody systems. Specifically, the effects of dimensional uncertainties on the behavior of a system are considered. The proposed reliability-based optimization method addresses difficulties such as high computational effort and non-smoothness of the system's responses, for example, as a result of contact events. The approach is based on decomposition of the design space into regions, corresponding to either acceptable or non-acceptable system performance. The boundaries of these regions are defined using Support Vector Machines (SVMs), which are explicit in terms of the design parameters. A SVM can be trained based on a limited number of samples, obtained from a design of experiments, and allows a very efficient estimation of probability of failure, even when Monte Carlo Simulation (MCS) is used. A modularly structured tolerance analysis scheme for automatic estimation of system production cost and probability of system failure is presented. In this scheme, detection of failure is based on multibody system simulation, yielding high computational demand. A S VM-based replication of the failure detection process is derived, which ultimately allows for automatic optimization of tolerance assignments. A simple multibody system, whose performance usually shows high tolerance sensitivity, is chosen as an exemplary system for illustration of the proposed approach. The system is optimally designed for minimum manufacturing cost while satisfying a target performance level with a given probability.