The Code-Switching Problem: Why Multilingual AI Is Harder Than It Looks

I remember sitting in a meeting room in Taxila, listening to our Pakistani banking partners discuss their customer verification systems. The conversation flowed naturally between Urdu, English, and occasional Hindi phrases. Someone would start a sentence in English, switch to Urdu mid-thought, then throw in a technical term in English again. Nobody blinked. This was simply how people spoke.

Then one of the bankers asked a question that would shape the next two years of our MIMER project: "Can your AI understand this?" He gestured at the air, encompassing the multilingual conversation we had just been having. "Because this is how our customers actually talk."

That was the moment I realized we had been solving the wrong problem.

The Invisible Majority

We had spent months building sophisticated deepfake detection systems for voice authentication. Our technology could distinguish real voices from artificially generated ones with impressive accuracy on benchmark datasets. We were achieving results that would make any researcher proud.

But those benchmark datasets were almost entirely monolingual. People spoke English, or Mandarin, or Spanish. They did not switch between languages mid-sentence. They certainly did not exhibit the complex, fluid multilingualism that characterized actual human conversation in Pakistan and India.

The numbers tell a sobering story. Over half a billion people speak Hindi and Urdu. Hindi is the third most spoken language globally. Yet when we surveyed existing voice authentication and deepfake detection datasets, we found nothing that captured how these populations actually communicate.

Code-switching, the practice of alternating between languages within a single conversation or even a single sentence, is not a bug in human communication. It is a feature. It conveys social meaning, establishes identity, and enables nuanced expression that monolingual speech cannot achieve.

But for AI systems, code-switching is a nightmare.

Building Reality Into Data

During the development of our Multilingual Spoofed Speech dataset, we made a deliberate choice to preserve natural code-switching behavior. We did not ask speakers to compartmentalize their languages. We captured them as they were, switching between Urdu, Hindi, and English as context and comfort dictated.

The result was 472,486 utterances from 154 speakers, featuring the actual conversational dynamics of bilingual South Asian speakers. Some samples had minimal code-switching. Others alternated languages multiple times within seconds. The dataset reflected reality, which meant it reflected complexity.

When we analyzed the data, approximately 50% of audio samples contained significant code-switching. In the Hindi subset, speakers switched languages even more frequently than in Urdu. This was not random variation. It was systematic behavior shaped by social context, speaker background, and the pragmatics of bilingual communication.

For our spoofing detection systems, this created a compound challenge. We were not just trying to distinguish real voices from synthetic ones. We were trying to do so in an environment where the linguistic ground was constantly shifting beneath the model's feet.

The Prosody Problem

One of the subtler challenges relates to prosody, the rhythm and melody of speech. When speakers switch languages, they do not simply swap out words. They often shift prosodic patterns as well.

A speaker might start a sentence with the intonation patterns of Urdu, switch to an English phrase with English stress patterns, then return to Urdu prosody for the remainder. For human listeners embedded in these linguistic communities, this sounds completely natural. For machine learning models trained on clean, monolingual data, it is deeply confusing.

Voice conversion systems, the spoofing attacks we were defending against, had to grapple with this same complexity. In our testing, sophisticated techniques like Seed-VC and HierSpeech++ produced spoofed samples that were nearly indistinguishable from genuine ones, even in code-switched contexts. These systems had learned to preserve the natural flow of multilingual speech, making them far more dangerous than simpler text-to-speech approaches.

The Performance Paradox

When we evaluated our detection framework, WavSpeech-AASIST, across the three language subsets, we encountered a puzzling asymmetry. The system performed well on Urdu, achieving an equal error rate of just 0.43%. English was slightly more challenging at 1.39%. But Hindi proved dramatically harder, with an error rate of 5.55%.

Why was Hindi so much more difficult than Urdu?

The answer lay in the frequency and nature of code-switching. Hindi speakers in our dataset switched between languages more often and more abruptly. These rapid transitions created prosodic and phonetic shifts that our models struggled to follow. The phonetic boundaries blurred. The acoustic features that indicated genuine versus spoofed speech became less reliable.

Moreover, many Hindi recordings contained background music and overlapping voices, common in South Asian broadcast environments. This environmental noise was preserved in the spoofed samples, creating an additional layer of acoustic complexity. The model had to distinguish between noise patterns associated with genuine recordings and those associated with the spoofing process itself, all while tracking rapid language switches.

This revealed something important. It is not just that multilingual speech is more variable. It is that the variability itself interferes with the acoustic signatures we rely on for detecting synthesis and manipulation.

The Accent Layer

There is yet another dimension that we encountered repeatedly. The English spoken by native Urdu and Hindi speakers carries distinct phonetic characteristics. Pronunciation patterns, intonation contours, and rhythm differ from native English speakers in predictable ways.

When someone switches from Urdu to English, they do not switch accents. The English phrases carry prosodic and phonetic markers of the speaker's first language. This is not a deficiency. It is a characteristic of genuine bilingual speech.

But for AI models trained primarily on native English speakers, these accented productions of English words look anomalous. When we tested our systems on supposedly multilingual datasets that were actually just monolingual English with different speakers, they performed well but failed to generalize to the English-with-Urdu-accent that our target users actually produced.

This highlighted a problematic assumption underlying much multilingual AI research. We often treat languages as discrete categories, assuming that once a model learns Language A and Language B separately, it can handle both together. But bilingual speech is not two monolingual modes spliced together. It is a distinct communicative system with its own phonetic, prosodic, and pragmatic rules.

The Data Hunger

One frustrating aspect of code-switching research is the perpetual scarcity of training data. Monolingual datasets are plentiful. English speech corpora contain hundreds or thousands of hours of annotated audio. High-resource languages like Mandarin and Spanish have substantial resources too.

But code-switched data? Vanishingly rare.

During MIMER, we explored various data augmentation strategies. We used voice conversion not just to generate spoofed samples but to increase the diversity of genuine multilingual speech. We employed automatic transcription on unannotated multilingual broadcasts, trading manual annotation costs for noisier but more abundant data.

These approaches helped. They increased training volume and improved model performance. But they could not fully replicate the complexity of natural code-switching. Synthetic code-switches lack the pragmatic motivations of real ones. They do not capture the social meanings embedded in language choice.

The Cost of Complexity

Our WavSpeech-AASIST framework combined multiple self-supervised learning models, specifically wav2vec and UniSpeech, to capture the complementary representations needed for multilingual spoofing detection.

Interestingly, this multi-embedding approach did not uniformly improve performance. On English and on standard monolingual benchmarks like ASVspoof-2019, fusing representations from both models yielded the best results. But on Urdu and Hindi, using wav2vec features alone sometimes performed better.

This suggests that straightforward concatenation of multilingual representations can introduce interference rather than synergy. The features that help with one linguistic context may add noise in another. What we need are adaptive fusion strategies that can weight different feature streams based on the linguistic context the model currently inhabits.

What This Means for Deployment

These challenges have direct implications for real-world systems. Consider voice authentication used by South Asian banks, our original use case. If these systems are trained on monolingual data or artificial multilingual data without code-switching, they will either reject genuine customers who code-switch naturally or fail to detect sophisticated spoofing attacks that exploit code-switching patterns.

The same applies to voice assistants, automated customer service systems, and any AI that processes spoken language in genuinely multilingual environments. A system that cannot follow code-switched speech is essentially useless to billions of people. A fraud detection system that fails to model code-switching opens exploitable vulnerabilities.

This is not a peripheral concern affecting edge cases. It is a fundamental limitation affecting how AI systems interact with the majority of the world's population. Most of the world is multilingual. Code-switching is normal, widespread practice. Building AI that cannot handle it means building AI that cannot serve most people.

The Path Forward

MIMER taught me that the hardest problems in AI are often the ones we do not notice because they live outside the linguistic and cultural contexts where most AI research happens.

Code-switching is invisible if you work primarily in monolingual English environments. It becomes a curiosity, a special case, something to address once the main problems are solved. But for billions of people, it is simply how language works.

Building AI systems that genuinely serve global populations requires centering these supposedly marginal cases. It means starting with the complexity of real human communication rather than hoping that systems optimized on simplified data will somehow generalize.

The technical challenges are substantial. We need new theoretical frameworks for thinking about multilingual speech that do not reduce it to parallel monolingual channels. We need architectures that can adapt to linguistic context shifts rather than treating them as noise. We need evaluation metrics that capture not just accuracy but also linguistic appropriateness and social acceptability.

I do not know how to fully solve the code-switching problem. Neither does anyone else, yet. But I do know that solving it requires more than better models or bigger datasets. It requires rethinking our assumptions about what language is and how AI should engage with it.

The conversation in that Taxila meeting room continues to echo. "Can your AI understand this?"

Not yet. But we are beginning to understand why the question is harder than we thought.

---

This reflects on work from the MIMER project (Mitigating Presentation Attacks in Remote Identity Proofing: Pakistan in Focus), funded by EPSRC, developing the Multilingual Spoofed Speech (MSS) dataset for deepfake detection in code-switched environments.