Tutorial

• Sample structure of manifest file

Applicable to svarah_manifest.json & saa_l1_manifest.json

{"audio_filepath": <path to audio file 1>, "duration": <seconds>, "text": <transcript 1>}
{"audio_filepath": <path to audio file 2>, "duration": <seconds>, "text": <transcript 2>}

• Running evaluation scripts

For azure and google cloud evaluations, you will be required to add your key associated with the services offered by each. For others, you can run the following :

python eval_<hf_model>.py  --manifest <manifest path>

For processing audio filepaths, kindly change them as per your directory structure in the scripts.

• Meta statistics of speakers

The meta_speaker_stats.csv file consists of 11 columns which describes some meta statistics of speakers involved in Svarah:

• speaker_id — unique speaker identifier
• duration — duration of audio recorded (seconds)
• text — transcript of audio
• gender — “Male” / “Female”
• age-group — speaker’s age group (18-30, 30-45, 45-60 & 60+ )
• primary_language — speaker’s primary language
• native_place_state — speaker’s native state
• native_place_district — speaker’s native district
• highest_qualification — speaker’s highest education qualification
• job_category — speakers’s job category (Part Time, Full Time, Other)
• occupation_domain — speaker’s domain of occupation (Education and Research, Healthcare [Medical & Pharma], Government, Technology and Services, Information and Media, Financial Services [Banking and Insurance], Transportation and Logistics, Entertainment, Social service, Manufacturing & Retail )

• Svarah folder tree

    Svarah
        ├── audio
        │   ├── <filename>.wav
        │   └── <filename>.txt
        │    .
        │    .
        │    .
        ├── svarah_manifest.json
        ├── saa_l1_manifest.json
        └── meta_speaker_stats.csv
  

Table 1: WER comparison

Table 1 depicts WER’s of different models on (i) Svarah that contains data from Indian speakers and (ii) SAA_L1, LibriSpeech Clean (Libri) which contain data from native English speakers.

Table 2: Accent-wise split of Svarah

Table 2: Number of hours and Number of tokens in each accent

Citation

If you benefit from this dataset, kindly cite as follows:

@misc{javed2023svarah,
      title={Svarah: Evaluating English ASR Systems on Indian Accents},
      author={Tahir Javed and Sakshi Joshi and Vignesh Nagarajan and Sai Sundaresan and Janki Nawale and Abhigyan Raman and Kaushal Bhogale and Pratyush Kumar and Mitesh M. Khapra},
      year={2023},
      eprint={2305.15760},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}