Recurrent neural networks factor computation along the symbol positions of the input and output sequences. Aligning positions to steps in computation time, they generate a sequence of hidden states h_t. This inherently sequential nature precludes parallelization within training examples, which becomes critical at longer sequence lengths.
An attention function can be described as mapping a query and a set of key-value pairs to an output. Instead of performing a single attention function of dimension d_model, we found it beneficial to linearly project queries, keys, and values h = 8 times with different, learned linear projections.
Dividing by √d_k prevents the dot products from growing excessively large for large key dimensions, avoiding saturation in regions of vanishing gradients.