首页    期刊浏览 2024年11月29日 星期五
登录注册

文章基本信息

  • 标题:Evaluation of Language Models over Croatian Newspaper Texts
  • 其他标题:Evaluation of Language Models over Croatian Newspaper Texts
  • 本地全文:下载
  • 作者:Beliga, Slobodan ; Ipšić, Ivo ; Martinčić-Ipšić, Sanda
  • 期刊名称:Engineering Economics
  • 印刷版ISSN:2029-5839
  • 出版年度:2017
  • 卷号:46
  • 期号:4
  • 页码:425-444
  • DOI:10.5755/j01.itc.46.4.18367
  • 语种:English
  • 出版社:Kaunas University of Technology
  • 摘要:Statistical language modeling involves techniques and procedures that assign probabilities to word sequences or, said in other words, estimate the regularity of the language. This paper presents basic characteristics of statistical language models, reviews their use in the large set of speech and language applications, explains their formal definition and shows different types of language models. Detailed overview of n-gram and class-based models (as well as their combinations) is given chronologically, by type and complexity of models, and in aspect of their use in different NLP applications for different natural languages. The proposed experimental procedure compares three different types of statistical language models: n-gram models based on words, categorical models based on automatically determined categories and categorical models based on POS tags. In the paper, we propose a language model for contemporary Croatian texts, a procedure how to determine the best n-gram and the optimal number of categories, which leads to significant decrease of language model perplexity, estimated from the Croatian News Agency articles (HINA) corpus. Using different language models estimated from the HINA corpus, we show experimentally that models based on categories contribute to a better description of the natural language than those based on words. These findings of the proposed experiment are applicable, except for Croatian, for similar highly inflectional languages with rich morphology and non-mandatory sentence word order.DOI: http://dx.doi.org/10.5755/j01.itc.46.4.18367
  • 关键词:Statistical language model; Natural language regularity; Word-based language model; Category-based language model; Brown algorithm; POS class; N-gram; Perplexity; Croatian corpora
国家哲学社会科学文献中心版权所有