{TURNA}: A {T}urkish Encoder-Decoder Language Model for Enhanced Understanding and Generation

Gökçe Uludo\ugan, Zeynep Balal, Furkan Akkurt, Meliksah Turker, Onur Gungor, Susan Üsküdarlı

Findings of the Association for Computational Linguistics ACL 2024

Abstract

The recent advances in natural language processing have predominantly favored well-resourced English-centric models, resulting in a significant gap with low-resource languages. In this work, we introduce TURNA, a language model developed for the low-resource language Turkish and is capable of both natural language understanding and generation tasks. TURNA is pretrained with an encoder-decoder architecture based on the unified framework UL2 with a diverse corpus that we specifically curated for this purpose. We evaluated TURNA with three generation tasks and five understanding tasks for Turkish. The results show that TURNA outperforms several multilingual models in both understanding and generation tasks and competes with monolingual Turkish models in understanding tasks.

BibTeX

@inproceedings{uludoganTURNA2024,
    title = "{TURNA}: A {T}urkish Encoder-Decoder Language Model for Enhanced Understanding and Generation",
    author = {Uludo{\u{g}}an, G{\"o}k{\c{c}}e  and
      Balal, Zeynep  and
      Akkurt, Furkan  and
      Turker, Meliksah  and
      Gungor, Onur  and
      {\"U}sk{\"u}darl{\i}, Susan},
    editor = "Ku, Lun-Wei  and
      Martins, Andre  and
      Srikumar, Vivek",
    booktitle = "Findings of the Association for Computational Linguistics ACL 2024",
    month = 8,
    year = "2024",
    day    = {11--16},
    address = "Bangkok, Thailand",
    publisher = "Association for Computational Linguistics",
    link= "https://aclanthology.org/2024.findings-acl.600",
    pages = "10103--10117",
    group        = {conference},
    abstract = {The recent advances in natural language processing have predominantly favored well-resourced English-centric models, resulting in a significant gap with low-resource languages. In this work, we introduce TURNA, a language model developed for the low-resource language Turkish and is capable of both natural language understanding and generation tasks. TURNA is pretrained with an encoder-decoder architecture based on the unified framework UL2 with a diverse corpus that we specifically curated for this purpose. We evaluated TURNA with three generation tasks and five understanding tasks for Turkish. The results show that TURNA outperforms several multilingual models in both understanding and generation tasks and competes with monolingual Turkish models in understanding tasks.}
}