Automatic handwritten digit recognition provides significant contributions towards many real-time applications starting from the vehicle’s number plate to doctor’s prescription. However, the real challenge in these applications highly depends on the factors such as accuracy rate and time. Considering this significance, a novel handwritten digit recognition method is proposed without the adoption of any pre-processing steps like noise prediction, segmentation, and feature selection/extraction. The purpose of eliminating these preliminary steps is to reduce the computational complexity as the utilization of the Deep Learning (DL) approach helps to reduce the computational complexity of directly performing classification. Here, a novel Layered Convolutional Neural Networks (LCNN) model with an efficient Squirrel Optimizer (LCNN-SO) is modeled to attain better classification and global solution during the handwritten digit recognition. This cascaded model is a simple emerging DL-based one with multiple layers. The proposed LCNN-SO model adds multiple layers over the CNN model to focus on accurate classification and optimizing the layers to achieve better results using squirrel optimizer. The layered stages of CNN with the optimizer are trained and constructed to recognize the various kinds of digit isolation over the input data. Here, Special Database 1 and Special Database 2 are used to analyze and classify the input data for providing non-segregated digits for further processing in real-time applications. The simulation is carried out in MATLAB 2018 environment and metrics like accuracy, elapsed time, precision, recall, and F-measure are evaluated. The outcomes of these metrics are 98.5%, 99%, 99.5%, and 99.50% respectively. The anticipated LCNN-SO model gives better prediction accuracy when compared to existing approaches like Convolutional Neural Networks + Long-Short Term Memory (CNN+LSTM), pre-trained Convolutional Neural Networks + Multi-Layered Perceptron (CNN+MLP), pre-trained CNN+LSTM, pre-trained Convolutional Neural Networks +Support Vector Machine (CNN+SVM), Dense trajectories with Histogram of Gradients (HoG), and Convolutional Neural Networks + Orthogonal Learning Particle Swarm Optimization (OLPSO) respectively.