The Promise of Scaling AI with Mixture of Experts
Artificial intelligence (AI) models have seen remarkable progress in recent years, with models like GPT-3 and PaLM showing human-level performance on many language tasks. However, the compute required to train these massive models is prohibitive, making further progress difficult. Enter…