Power flow optimization is a critical aspect of power and control system design, aiming to minimize running costs while ensuring control variable tolerances and maximizing power delivery. The utilization of flexible alternating current transmission system (FACTS) particularly unified power flow controllers (UPFCs), enhances a system's power transfer capacity. UPFCs offer versatile adjustments for active and reactive power lines and voltage levels simultaneously, contributing to improved system performance and stability. A novel approach for enhancing flexibility and control in a doubly fed induction generator (DFIG)-based wind energy conversion system (WECS) was proposed. The technique involved the integration of a fuzzy-based control strategy for UPFC and the utilization of a cascaded adaptive neuro fuzzy inference system (ANFIS) to manage UPFC's series and shunt converters. Additionally, an improved butterfly optimization algorithm (IBOA) was introduced to ensure stability for DFIG-WECS. The methods were implemented and validated through simulations using MATLAB software. The effectiveness of the proposed approach was demonstrated through experimental results. The integration of cascaded ANFIS-based UPFC control and IBOA with WECS resulted in increased system power quality and stability. The total harmonic distortion (THD) of the system is reduced by 1.8%, indicating an enhancement in power quality. The application of UPFC and innovative control methodologies results in optimal power flow (OPF), maintaining stringent control variable tolerances in power systems. This research presents a systematic approach to enhancing flexibility and control in a DFIG-based WECS through the innovative use of UPFC and advanced control techniques. The proposed fuzzy-based UPFC control, in conjunction IBOA, significantly improves power quality and stability, thereby enhancing the overall efficiency of the system.