A shaft is a crucial component of a rotating machine that transfers energy from its source to the necessary part. In the past two decades, composite materials have been extensively investigated and utilized for rotating shafts due to their relatively high modulus and specific strength without compromising the component's dependability. Carbon/epoxy and boron/epoxy composite materials are often employed and compared to conventional steel shafts. In this study, a finite element analysis tool, engineering simulation software (ANSYS) was utilized for the three dimensions (3D) rotor dynamic examination of a material shaft, which allows for an accurate dynamics analysis strategy of the rotor behavior, such as shear stress, torque capacity, and resonance. The static structural, modal analysis, and the harmonic response of the rotor shaft made up of steel, titanium alloy, carbon/epoxy, and boron/epoxy were investigated. The results of the study indicate that boron/epoxy has the highest value of shear stress, torque capacity, and minimum resonance speed point up to 300 Hz. The weight of the carbon/epoxy material is also relatively low when compared to the results of other materials like steel and titanium alloy. Furthermore, the study found that the weight reduction is between 49% to 76% for the boron/epoxy and carbon/epoxy materials compared to a traditional steel shaft. In conclusion, the designed composite shaft, particularly the boron/epoxy and carbon/epoxy materials, has been shown to be the most effective replacement for traditional steel shafts, as they possess superior mechanical properties and result in a significant weight reduction.