The attention mechanism in higher dimensions

This is the Addendum to the and last video of a trilogy of videos titled "The world's smallest transformer model". In this video, we work out a calculation that we left in the original video, which is a pair of heads of attention (K and Q matrices) that require a more complex architecture, more specifically, a 4-3-4 architecture which will embed the dataset in a 3-dimensional space instead of 2-dimensions. This extra freedom allows us to build the two new heads of attention. Video 1: How neural networks bend space https://www.youtube.com/watch?v=3THzlSIzfug Video 2: The world's smallest tra