Switch Transformers

Switch Transformers introduce a sparse Mixture of Experts architecture that routes each input to a single expert, reducing communication overhead while sc