This paper presents a new high-throughput, low-complexity Bit Flipping (BF) decoder for Low-Density Parity-Check (LDPC) codes on the Binary Symmetric Channel (BSC), called Probabilistic Parallel Bit Flipping (PPBF). The advantage of PPBF comes from the fact that, no global operation is required during the decoding process and from that, all of the computations could be parallelized and localized at the computing units. Also in PPBF, the probabilistic feature in flipping the Variable Node (VN) is incorporated for all satisfaction level of its CN neighbors. This type of probabilistic incorporation makes PPBF more dynamic to correct some error patterns which are unsolvable by other BF decoders. PPBF offers a faster decoding process with an equivalent error correction performance to the Probabilistic Gradient Descent Bit Flipping (PGDBF) decoder, which is better than all so-far introduced BF decoders in BSC channel. A hardware implementation architecture of PPBF is also presented in this paper with detailed circuits for the probabilistic signal generator and processing units. The implementation of PPBF on FPGA confirms that, the PPBF complexity is much lower than that of the PGDBF and even lower than the one of the deterministic Gradient Descent Bit Flipping (GDBF) decoder. The good decoding performance along with the high throughput and low complexity lead PPBF decoder to become a brilliant candidate for the next generation of communication and storage standards.