Semaphore is a means of long-distance communication using semaphore flags as tools. This communication method has been used since ancient times to convey information. In Indonesia, semaphore communication is practiced in scouting activities and by mariners to convey specific messages. In these activities, communication using semaphores involves transmitting important information through specific gestures performed by demonstrators. Transmitting messages using semaphore letter codes can be challenging for beginners. With the increasing use of semaphores in specific fields, there is a need for a system that can automatically recognize semaphore letter gestures in real-time. The designed system can be used in learning processes and implementing semaphore letter code in reading devices. This research aims to design a real-time semaphore letter code recognition system using soft computing methods. Digital image processing is chosen for image recognition in the designed system. Image segmentation is employed to obtain the object parts of the image, followed by wavelet as a feature extraction method. Back propagation neural network (BPNN) is used for semaphore gesture classification. 910 image data were used to design the semaphore gesture recognition system. The success rate in sequential recognition is 94% at a distance of 3 meters, 90% at 4 meters, 88% at 5 meters, 86% at 6 meters, and 83% at 7 meters. The test results demonstrate the positive potential of the system for use in learning processes and the implementation of semaphore letter code reading devices.