An interesting thing about transformers is that they are simpler than the LSTMs that came before them. Problems like vanishing gradients set limits on how complex a neural network can be.

I think neural nets are not the final answer. A lot of research has been done on neural nets lately because computer hardware has evolved enough to make it possible, but we will have better technologies in the future.

I see AI with neural nets as airplanes with flapping wings. We don't imitate nature because we don't have the same limitations. Eventually, we will develop more efficient algorithms that perform the same operations as neural nets.