ai4bharat

AI4Bharat is a research lab at the Indian Institute of Technology Madras (IIT Madras) dedicated to developing open-source datasets, tools, models, and applications for Indian languages. Here's an in-depth look at their work, focus areas, and contributions:

Mission and Vision:

• Mission: To bring parity with respect to English in AI technologies for Indian languages through open-source contributions in datasets, models, and applications, while fostering an innovation ecosystem.

• Vision: To enable equitable access to information and content in India's diverse linguistic landscape, thereby promoting digital inclusion.

Key Focus Areas:

• Machine Translation (MT):

  • Samanantar Corpus: The largest publicly available parallel dataset for Indian languages with around 230 million bitext pairs across all 22 scheduled languages.
  • IndicTrans2: A high-quality, multilingual NMT model supporting translations across all 22 scheduled Indic languages.

• Automatic Speech Recognition (ASR):

  • Extensive Data Collection: Over 300,000 hours of raw speech data, aiming to cover all 22 languages with a focus on linguistic diversity across India.

• Text-to-Speech (TTS):

  • IndicVoices: Initiatives like AI4BTTS to create natural-sounding synthetic voices for Indian languages.

• Transliteration and Language Identification:

  • Aksharantar: The largest transliteration dataset for Indian languages, enhancing transliteration accuracy.

• Large Language Models (LLMs):

  • IndicLLM Suite: A suite of resources for 22 Indic languages, including a massive pre-training dataset (Sangraha) and instruction-response pairs for fine-tuning (IndicAlign-Instruct).

• Dataset Building:

  • Bharat Parallel Corpus Collection (BPCC): Focuses on parallel text for translation, released under CC0 license.
  • Data Collection Efforts: Supported by the Digital India Bhashini Mission, involving workshops and community-driven data collection across various Indian languages.

     

Tools and Models:

• IndicBERT, IndicBART, Airavata: Multilingual LLMs tailored for Indian languages, emphasizing cultural and linguistic nuances.
• Setu: A tool for large-scale data crawling and cleaning.

   

Community and Collaboration:

• Open-Source Commitment: All tools and datasets are made available under permissive licenses to encourage wide adoption and contribution.
• Collaborations: Works closely with initiatives like Bhashini under the Digital India program.

   

Impact and Applications:

• Real-world Use Cases: From enhancing NPCI Payments with speech recognition to supporting the Supreme Court with translation services.
• Academic and Research Contributions: Publications in top-tier conferences, showcasing their work's global recognition.

Recent Developments:

• Indic Parler-TTS: An open-source TTS system in collaboration with Hugging Face for over a billion Indic speakers.
• Airavata: An instruction-tuned LLM specifically for Hindi, indicating ongoing work in language-specific AI models.

AI4Bharat's work is pivotal for the digital empowerment of India through AI, focusing on inclusivity across its linguistic diversity. Their dedication to open-source development not only aids academic research but also has practical applications in various sectors across India.