In the real-world modern environment, intelligent transportation systems and intelligent surveillance systems are considered to play an anchor role in facilitating security and safety to the human society. These surveillance systems are diversely utilized in most of the places ranging from the application of border security to a street monitoring system that closely observes the abnormal event occurrence on the road. The core aim of the work is to present a rich set of abnormal event videos for intelligent surveillance systems. Hence, potential abnormal event detection and considerable video summarization mechanism is needed with maximum accuracy and reduced complexities. In this paper, hybrid convolution neural network (CNN) and bi-directional long short-term memory (Bi-LSTM) based abnormal event detection model is employed with reduced complexity. This model includes convolutional neural network with a pre-trained model for extracting spatio-temporal features from each individual frame selected from a series of frames, which is then passed to multi-layer Bi-LSTM that possesses the capability of accurately classifying the abnormal events in complex surveillance scenes of the road. Hierarchical temporal attention-based long short-term memory (LSTM) encoder-decoder model is included in the fine grain process in order to attain a better video summarization that preserves video key information and attains an optimal storage. The experimental results with the functional parameters confirmed the maximized abnormal event frames with minimum complexity using video summarization method and the accuracy of the abnormal detection scheme, on par with the benchmarked approaches considered for investigation.