Dataset Card for Aksharantar

Table of Contents

• Table of Contents
• Dataset Description
  • Dataset Summary
  • Supported Tasks and Leaderboards
  • Languages
• Dataset Structure
  • Data Instances
  • Data Fields
  • Data Splits
• Dataset Creation
  • Curation Rationale
  • Source Data
  • Annotations
  • Personal and Sensitive Information
• Considerations for Using the Data
  • Social Impact of Dataset
  • Discussion of Biases
  • Other Known Limitations
• Additional Information
  • Dataset Curators
  • Licensing Information
  • Citation Information
  • Contributions

Dataset Description

• Homepage: https://indicnlp.ai4bharat.org/indic-xlit/
• Repository: https://github.com/AI4Bharat/IndicXlit/
• Paper: Aksharantar: Towards building open transliteration tools for the next billion users
• Leaderboard:
• Point of Contact:

Dataset Summary

Aksharantar is the largest publicly available transliteration dataset for 20 Indic languages. The corpus has 26M Indic language-English transliteration pairs.

Supported Tasks and Leaderboards

[More Information Needed]

Languages

Dataset Structure

Data Instances

A random sample from Hindi (hin) Train dataset.

{
'unique_identifier': 'hin1241393',
'native word': 'स्वाभिमानिक',
'english word': 'swabhimanik',
'source': 'IndicCorp',
'score': -0.1028788579
}

Data Fields

• unique_identifier (string): 3-letter language code followed by a unique number in each set (Train, Test, Val).

• native word (string): A word in Indic language.

• english word (string): Transliteration of native word in English (Romanised word).

• source (string): Source of the data.

• score (num): Character level log probability of indic word given roman word by IndicXlit (model). Pairs with average threshold of the 0.35 are considered.

  For created data sources, depending on the destination/sampling method of a pair in a language, it will be one of:

  • Dakshina Dataset
  • IndicCorp
  • Samanantar
  • Wikidata
  • Existing sources
  • Named Entities Indian (AK-NEI)
  • Named Entities Foreign (AK-NEF)
  • Data from Uniform Sampling method. (Ak-Uni)
  • Data from Most Frequent words sampling method. (Ak-Freq)

Data Splits

Dataset Creation

Information in the paper. Aksharantar: Towards building open transliteration tools for the next billion users

Curation Rationale

[More Information Needed]

Source Data

Initial Data Collection and Normalization

Information in the paper. Aksharantar: Towards building open transliteration tools for the next billion users

Who are the source language producers?

[More Information Needed]

Annotations

Information in the paper. Aksharantar: Towards building open transliteration tools for the next billion users

Annotation process

Information in the paper. Aksharantar: Towards building open transliteration tools for the next billion users

Who are the annotators?

Information in the paper. Aksharantar: Towards building open transliteration tools for the next billion users

Personal and Sensitive Information

[More Information Needed]

Considerations for Using the Data

Social Impact of Dataset

[More Information Needed]

Discussion of Biases

[More Information Needed]

Other Known Limitations

[More Information Needed]

Additional Information

Dataset Curators

[More Information Needed]

Licensing Information

This data is released under the following licensing scheme:

• Manually collected data: Released under CC-BY license.
• Mined dataset (from Samanantar and IndicCorp): Released under CC0 license.
• Existing sources: Released under CC0 license.

CC-BY License

CC0 License Statement

• We do not own any of the text from which this data has been extracted.
• We license the actual packaging of the mined data under the Creative Commons CC0 license (“no rights reserved”).
• To the extent possible under law, AI4Bharat has waived all copyright and related or neighboring rights to Aksharantar manually collected data and existing sources.
• This work is published from: India.

Citation Information

@misc{madhani2022aksharantar,
      title={Aksharantar: Towards Building Open Transliteration Tools for the Next Billion Users},
      author={Yash Madhani and Sushane Parthan and Priyanka Bedekar and Ruchi Khapra and Anoop Kunchukuttan and Pratyush Kumar and Mitesh Shantadevi Khapra},
      year={2022},
      eprint={},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

Contributions