Attention cannot tell word order on its own. Sinusoidal encodings add a vector of sines and cosines, RoPE turns queries and keys by an angle that grows with position, and ALiBi pushes down distant scores.
This interactive visualization needs JavaScript.