Resting state functional magnetic resonance imaging (fMRI) aims to measure baseline neuronal connectivity independent of functional tasks. Most existing network detection methods rely on a fixed threshold to identify functionally connected voxels under the resting state. Due to fMRI nonstationarity, a fixed threshold cannot adapt to inter-session and inter-subject variation. In this work, a new method is proposed for resting state fMRI data analysis. Specifically, the resting state network mapping is formulated as an outlier detection process that is implemented using one-class support vector machine (SVM). The results are refined by using a spatial-feature domain prototype selection method and two-class SVM reclassification. The final decision on each voxel is made by comparing its probabilities of functionally connected and unconnected. The proposed method was evaluated using synthetic and experimental fMRI data. A comparison study was also performed with independent component analysis (ICA) and correlation analysis. The experimental results show that the proposed method can provide comparable or better network detection performance than ICA and correlation analysis. The method is potentially applicable to various resting state quantitative fMRI studies.