Why is it so hard for AI models to work equally well in every language?
Most large language models were trained on internet text — and the internet is overwhelmingly English. A model that sees billions of English sentences but only millions of Tamil sentences will be much better at English, not because the architecture cares, but because practice makes perfect in training just as in humans.
The problem runs deeper than data volume. Tokenization — how a model chops text into chunks before processing — is usually designed around Latin scripts. A single English word might become one token; the same concept in Hindi or Arabic might fragment into many more, making those languages effectively more expensive and less fluent for the model to handle.
Closing the gap means rethinking how tokens are defined, how training budgets are allocated across languages, and how models are evaluated where human raters are scarce. For the majority of the world's speakers, today's best AI still speaks their language as a second one.