Tezislər / Theses

THE 3
rd
 INTERNATIONAL SCIENTIFIC CONFERENCES OF STUDENTS AND YOUNG RESEARCHERS 
dedicated to the 99
th
anniversary of the National Leader of Azerbaijan Heydar Aliyev
128
Taking the complex morphological structure into account, the character 
tokenization of the sentences is preferred over word tokenization to consider 
all possible word occurrences in the sentences. Then, padding is applied to 
define maximum length for the input sentences. 
Recurrent Neural Networks (RNN) is implemented with encoder and 
decoder layers which provides an advantage of holding result from previous 
step in its hidden state and using it predicting the next character in the word. 
Encoder transforms input sentence into feature vectors that were used by 
decoder as an initial hidden state with the combination of current word or 
character and its own hidden state. Both encoder and decoder are built by 
defining embedding and LSTM layers. Encoder-decoder-based model can 
result in performance bottleneck when a long sentence is inputted which 
makes it difficult for RNN to transmit all the required information from the 
previous step. To prevent this problem, Bahdanau attention mechanism is 
implemented between encoder and decoder to keep and pass all information 
over full length of input sentence efficiently. Attention mechanism has 3 parts, 
namely Feed forward neural network, softmax calculation and context vector 
that contain information about attention should be allotted for each element 
in the sequence. 
Finally, the model is evaluated on test dataset that contains total of 1000 
misspelled words and the accuracy of 94% is achieved as a result. Moreover, 
it can be further improved by expanding the training dataset and continue the 
study of typing behaviour of Azerbaijani people. The platform can be created 
where the help can be acquired from the community by developing the game 
for labelling the correct and wrong sentence pairs as true or false which can 
provide us with the advantage of utilizing feedback mechanism in the 
supervised learning to minimize the loss. As a future work, the software 
extension as Grammarly can be developed to ease the usage of the model 
for community. 
References
[1] Haldar, Rishin & Mukhopadhyay, Debajyot, “Levenshtein Distance Technique in Dictionary 
Lookup Methods: An Improved” 2011.
[2] Minjoon Seol, Aniruddha Kembhavi, Ali Farhadi, Hananneh Hajishirzi, bi-directional 
attention flow for machine comprehension, conference paper at ICLR, 2017
[3] Samir Mammadov, Neural Spelling Correction for Azerbaijani Language, IEEE Xplore, 
2019 

---