(Publisher of Peer Reviewed Open Access Journals)

International Journal of Advanced Technology and Engineering Exploration (IJATEE)

ISSN (Print):2394-5443    ISSN (Online):2394-7454
Volume-10 Issue-105 August-2023
Full-Text PDF
Paper Title : Localization for self-driving vehicles based on deep learning networks and RGB cameras
Author Name : Shahad S. Ghintab and Mohammed Y. Hassan
Abstract :

Autonomous vehicles (AVs) have emerged as captivating engineering ventures in the twenty-first century, capturing the interest of numerous academics and engineers across multiple generations. The world looks forward to leveraging AVs for reducing accidents caused by human errors and optimizing parking space utilization, particularly in urban areas. Accurate localization is pivotal for effective AV navigation, enabling the vehicle to pinpoint its precise position. While global positioning system (GPS) coordinates are widely used, their inherent errors and limitations can render them inadequate for determining precise location information, particularly in urban settings. Furthermore, drifting errors can undermine the efficacy of simultaneous localization and mapping (SLAM) algorithms. The proposed approach involves the utilization of a deep neural network, specifically a modified AlexNet architecture, which is a convolutional neural network (CNN), for localizing AVs in well-lit urban driving environments. The CNN enhances accuracy while reducing computational complexity and training time. Instead of relying on costly light detection and ranging (LiDAR) or radar sensors, a more affordable red green blue (RGB) camera sensor is employed. During testing, depth images are combined with RGB images using the intensity hue saturation (IHS) algorithm to enhance precision. Simulation results demonstrate an impressive accuracy rate of 95.49%, affirming the effectiveness of the proposed strategy. This study introduces a lightweight, precise, and reliable CNN architecture that significantly improves the accuracy of AV localization, simultaneously reducing predicted position errors by a considerable margin. The network's superiority is evidenced by mean square error (MSE) values of 0.039, 0.0099, and 0.0047 for position x, y, and orientation predictions, respectively. To validate real-time performance, the trained CNN was implemented in Python and integrated into the car learning to act (CARLA) simulator, enabling the online localization of a self-driving vehicle. This application successfully showcases the feasibility and efficacy of the proposed method.

Keywords : Autonomous vehicle, Localization, Deep learning, Convolutional neural networks CNN, Intensity hue saturation IHS, K-mean algorithm.
Cite this article : Ghintab SS, Hassan MY. Localization for self-driving vehicles based on deep learning networks and RGB cameras. International Journal of Advanced Technology and Engineering Exploration. 2023; 10(105):1016-1036. DOI:10.19101/IJATEE.2023.10101118.
References :
[1]Karur K, Sharma N, Dharmatti C, Siegel JE. A survey of path planning algorithms for mobile robots. Vehicles. 2021; 3(3):448-68.
[Crossref] [Google Scholar]
[2]Chen S, Liu B, Feng C, Vallespi-gonzalez C, Wellington C. 3d point cloud processing and learning for autonomous driving: impacting map creation, localization, and perception. IEEE Signal Processing Magazine. 2020; 38(1):68-86.
[Crossref] [Google Scholar]
[3]Fayyad J, Jaradat MA, Gruyer D, Najjaran H. Deep learning sensor fusion for autonomous vehicle perception and localization: a review. Sensors. 2020; 20(15):1-35.
[Crossref] [Google Scholar]
[4]Reid TG, Houts SE, Cammarata R, Mills G, Agarwal S, Vora A, et al. Localization requirements for autonomous vehicles. SAE International Journal of Connected and Automated Vehicles. 2019:1-16.
[Crossref] [Google Scholar]
[5]Rublee E, Rabaud V, Konolige K, Bradski G. ORB: an efficient alternative to SIFT or SURF. In international conference on computer vision 2011 (pp. 2564-71). IEEE.
[Crossref] [Google Scholar]
[6]Lowe DG. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision. 2004; 60:91-110.
[Crossref] [Google Scholar]
[7]Bay H, Tuytelaars T, Van GL. Surf: speeded up robust features. In computer vision–ECCV: 9th European conference on computer vision, Graz, Austria, 2006. Proceedings, Part I, 2006 (pp. 404-17). Springer Berlin Heidelberg.
[Crossref] [Google Scholar]
[8]Fischler MA, Bolles RC. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM. 1981; 24(6):381-95.
[Google Scholar]
[9]Mur-artal R, Tardós JD. Orb-slam2: an open-source slam system for monocular, stereo, and RGB-d cameras. IEEE Transactions on Robotics. 2017; 33(5):1255-62.
[Crossref] [Google Scholar]
[10]Engel J, Koltun V, Cremers D. Direct sparse odometry. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2017; 40(3):611-25.
[Crossref] [Google Scholar]
[11]Shotton J, Glocker B, Zach C, Izadi S, Criminisi A, Fitzgibbon A. Scene coordinate regression forests for camera relocalization in RGB-D images. In proceedings of the conference on computer vision and pattern recognition 2013 (pp. 2930-7).
[Google Scholar]
[12]Kendall A, Gal Y, Cipolla R. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In proceedings of the IEEE conference on computer vision and pattern recognition 2018 (pp. 7482-91).
[Google Scholar]
[13]Ballardini AL, Fontana S, Cattaneo D, Matteucci M, Sorrenti DG. Vehicle localization using 3D building models and point cloud matching. Sensors. 2021; 21(16):1-19.
[Google Scholar]
[14]Zghair NA, Al-araji AS. A one decade survey of autonomous mobile robot systems. International Journal of Electrical and Computer Engineering. 2021; 11(6):4891-906.
[Crossref] [Google Scholar]
[15]Stenborg E, Toft C, Hammarstrand L. Long-term visual localization using semantically segmented images. In international conference on robotics and automation 2018 (pp. 6484-90). IEEE.
[Crossref] [Google Scholar]
[16]Parisotto E, Chaplot D, Zhang J, Salakhutdinov R. Global pose estimation with an attention-based recurrent network. In proceedings of the conference on computer vision and pattern recognition workshops 2018 (pp. 237-46). IEEE.
[Google Scholar]
[17]Heng L, Choi B, Cui Z, Geppert M, Hu S, Kuan B, et al. Project autovision: localization and 3d scene perception for an autonomous vehicle with a multi-camera system. In international conference on robotics and automation 2019 (pp. 4695-702). IEEE.
[Crossref] [Google Scholar]
[18]Amini A, Rosman G, Karaman S, Rus D. Variational end-to-end navigation and localization. In international conference on robotics and automation 2019 (pp. 8958-64). IEEE.
[Crossref] [Google Scholar]
[19]Ma WC, Tartavull I, Bârsan IA, Wang S, Bai M, Mattyus G, et al. Exploiting sparse semantic HD maps for self-driving vehicle localization. In IEEE/RSJ international conference on intelligent robots and systems 2019(pp. 5304-11). IEEE.
[Crossref] [Google Scholar]
[20]Yin H, Wang Y, Ding X, Tang L, Huang S, Xiong R. 3D LiDAR-based global localization using siamese neural network. IEEE Transactions on Intelligent Transportation Systems. 2019; 21(4):1380-92.
[Crossref] [Google Scholar]
[21]Wan G, Yang X, Cai R, Li H, Zhou Y, Wang H, et al. Robust and precise vehicle localization based on multi-sensor fusion in diverse city scenes. In IEEE international conference on robotics and automation 2018 (pp. 4670-77). IEEE.
[Crossref] [Google Scholar]
[22]Chen X, Vizzo I, Läbe T, Behley J, Stachniss C. Range image-based LiDAR localization for autonomous vehicles. In international conference on robotics and automation 2021 (pp. 5802-8). IEEE.
[Crossref] [Google Scholar]
[23]Héry E, Xu P, Bonnifait P. Consistent decentralized cooperative localization for autonomous vehicles using LiDAR, GNSS, and HD maps. Journal of Field Robotics. 2021; 38(4):552-71.
[Crossref] [Google Scholar]
[24]Qin T, Zheng Y, Chen T, Chen Y, Su Q. A light-weight semantic map for visual localization towards autonomous driving. In international conference on robotics and automation 2021 (pp. 11248-54). IEEE.
[Crossref] [Google Scholar]
[25]Chu X, Lu Z, Gesbert D, Wang L, Wen X. Vehicle localization via cooperative channel mapping. IEEE Transactions on Vehicular Technology. 2021; 70(6):5719-33.
[Crossref] [Google Scholar]
[26]Li Y, Cai Y, Malekian R, Wang H, Sotelo MA, Li Z. Creating navigation map in semi-open scenarios for intelligent vehicle localization using multi-sensor fusion. Expert Systems with Applications. 2021; 184:115543.
[Crossref] [Google Scholar]
[27]Liu J, Guo G. Vehicle localization during GPS outages with extended Kalman filter and deep learning. IEEE Transactions on Instrumentation and Measurement. 2021; 70:1-10.
[Crossref] [Google Scholar]
[28]Guo C, Lin M, Guo H, Liang P, Cheng E. Coarse-to-fine semantic localization with HD map for autonomous driving in structural scenes. In IEEE/RSJ international conference on intelligent robots and systems 2021 (pp. 1146-53). IEEE.
[Crossref] [Google Scholar]
[29]Ren R, Fu H, Xue H, Li X, Hu X, Wu M. LiDAR‐based robust localization for field autonomous vehicles in off‐road environments. Journal of Field Robotics. 2021; 38(8):1059-77.
[Crossref] [Google Scholar]
[30]Yanase R, Hirano D, Aldibaja M, Yoneda K, Suganuma N. LiDAR-and radar-based robust vehicle localization with confidence estimation of matching results. Sensors. 2022; 22(9):1-20.
[Crossref] [Google Scholar]
[31]Peng B, Xie H, Chen W. ROLL: long-term robust LiDAR-based localization with temporary mapping in changing environments. In IEEE/RSJ international conference on intelligent robots and systems 2022 (pp. 2841-7). IEEE.
[Crossref] [Google Scholar]
[32]Dauptain X, Koné A, Grolleau D, Cerezo V, Gennesseaux M, Do MT. Conception of a high-level perception and localization system for autonomous driving. Sensors. 2022; 22(24):9661.
[Crossref] [Google Scholar]
[33]Lee J, Back M, Hwang SS, Chun IY. Improved real-time monocular SLAM using semantic segmentation on selective frames. IEEE Transactions on Intelligent Transportation Systems. 2022; 24(3):2800-13.
[Crossref] [Google Scholar]
[34]Kang MS, Ahn JH, Im JU, Won JH. LiDAR-and V2X-based cooperative localization technique for autonomous driving in a GNSS-denied environment. Remote Sensing. 2022; 14(22):1-16.
[Crossref] [Google Scholar]
[35]Han L, Shi Z, Wang H. A localization and mapping algorithm based on improved LVI-SAM for vehicles in field environments. Sensors. 2023; 23(7):1-14.
[Crossref] [Google Scholar]
[36]Raheem F, Abdulwahhab AA. Deep learning convolution neural networks analysis and comparative study for static alphabet ASL hand gesture recognition. Journal of Xidian University. 2020; 14(4):1871-81.
[Crossref] [Google Scholar]
[37]He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. In proceedings of the IEEE conference on computer vision and pattern recognition 2016 (pp. 770-8). IEEE.
[Google Scholar]
[38]Fujiyoshi H, Hirakawa T, Yamashita T. Deep learning-based image recognition for autonomous driving. IATSS Research. 2019; 43(4):244-52.
[Crossref] [Google Scholar]
[39]Sandler M, Howard A, Zhu M, Zhmoginov A, Chen LC. Mobilenetv2: inverted residuals and linear bottlenecks. In proceedings of the conference on computer vision and pattern recognition 2018 (pp. 4510-20). IEEE.
[Google Scholar]
[40]Wani MA, Bhat FA, Afzal S, Khan AI. Advances in deep learning. Springer; 2020.
[Crossref] [Google Scholar]
[41]Alzubaidi L, Zhang J, Humaidi AJ, Al-dujaili A, Duan Y, Al-shamma O, et al. Review of deep learning: concepts, CNN architectures, challenges, applications, future directions. Journal of Big Data. 2021; 8:1-74.
[Crossref] [Google Scholar]
[42]Abdulhussein AA, Raheem FA. Hand gesture recognition of static letters American sign language (ASL) using deep learning. Engineering and Technology Journal. 2020; 38(6):926-37.
[Crossref] [Google Scholar]
[43]Krizhevsky A, Sutskever I, Hinton GE. Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems. 2012:1-9.
[Google Scholar]
[44]Gómez VV. LiDAR-based scene understanding for autonomous driving using deep learning (Doctoral Dissertation, Universitat Politècnica de Catalunya (UPC). 2020.
[Google Scholar]
[45]Mishra D, Palkar B. Image fusion techniques: a review. International Journal of Computer Applications. 2015; 130(9):7-13.
[Google Scholar]
[46]Zhao X. Image fusion based on IHS transform and principal component analysis (PCA) transform. In international conference on computer technology, electronics and communication (ICCTEC) 2017(pp. 304-7). IEEE.
[Crossref] [Google Scholar]
[47]Atiyah HA, Hassan MY. Outdoor localization in mobile robot with 3D LiDAR based on principal component analysis and K-Nearest neighbors algorithm. Engineering and Technology Journal. 2021; 39(6):965-76.
[Google Scholar]
[48]Hassan MY, Kothapalli G. Comparison between neural network based PI and PID controllers. In 7th international multi-conference on systems, signals and devices 2010 (pp. 1-6). IEEE.
[Crossref] [Google Scholar]
[49]Zhou B, Liu J, Sun W, Chen R, Tomlin CJ, Yuan Y. PBSGD: powered stochastic gradient descent methods for accelerated non-convex optimization. In proceedings of the twenty-ninth international joint conference on artificial intelligence 2020 (pp. 3258-66).
[Google Scholar]
[50]Dosovitskiy A, Ros G, Codevilla F, Lopez A, Koltun V. CARLA: an open urban driving simulator. In conference on robot learning 2017 (pp. 1-16). PMLR.
[Google Scholar]
[51]Ghintab SS, Hassan MY. CNN-based visual localization for autonomous vehicles under different weather conditions. Engineering and Technology Journal. 2023; 41(2):375-86.
[Crossref] [Google Scholar]
[52]Wu Y, Li Y, Ge X, Gao Y, Qian W. An efficient method for calculating the error statistics of block-based approximate adders. IEEE Transactions on Computers. 2018; 68(1):21-38.
[Crossref] [Google Scholar]