# documents

url: http://essay.utwente.nl/102964/
scrape_date: 2025-03-11T15:39:26.242368
content_type: markdown
hash: f4404e8ce8dbbe112800b2abc673506005b8e59fe56e362a9d6961538f33f546
title:
 Retrofitted: Enhancing Immersion in SoloRPGs through Large Language Models  - University of Twente Student Theses

content:
[](https://www.utwente.nl/en/)

  * [Home](http://essay.utwente.nl/)
  * [About](http://essay.utwente.nl/information.html)
  * [Latest Additions](http://essay.utwente.nl/cgi/latest)
  * [Upload student thesis](http://essay.utwente.nl/upload.html)
  * [Create a publication list](http://essay.utwente.nl/publist.html)
  * [Statistics](http://essay.utwente.nl/statistics.html)
  * [Contact](http://essay.utwente.nl/contact.html)
  

  * [UT Research Information](https://www.utwente.nl/researchinformation)
  

  * [Search](http://essay.utwente.nl/cgi/search/advanced)



[University of Twente](https://www.utwente.nl/en/)

[Student Theses ](http://essay.utwente.nl/)

# University of Twente Student Theses

[Login](http://essay.utwente.nl/cgi/users/home)

##  Retrofitted: Enhancing Immersion in SoloRPGs through Large Language Models 

Jordan, Matthew (2024) _Retrofitted: Enhancing Immersion in SoloRPGs through Large Language Models._

[![\[img\]](http://essay.utwente.nl/102964/1.hassmallThumbnailVersion/Jordan_BA_EEMCS.pdf)](http://essay.utwente.nl/102964/1/Jordan_BA_EEMCS.pdf)|  [PDF](http://essay.utwente.nl/102964/1/Jordan_BA_EEMCS.pdf)   
1MB  
---|---  
Abstract:| This thesis explores enhancing player immersion in Solo Role-Playing Games (SoloRPGs) through integrating Large Language Models (LLMs). Traditional SoloRPGs, often suffer from complex rules and setups that can hinder player immersion. Attempts to transform SoloRPGs into digital formats have had some success but at the cost of reducing player choice due to pre-coded limitations. This project investigates whether LLMs, known for their human-like conversational capabilities can provide a solution that enhances immersion while maintaining player freedom. The research is guided by the primary question: "To what extent can player immersion in a SoloRPG be enhanced by a natural language interface using large language models?" Supporting this inquiry are sub-questions defining immersion, measurement techniques, and applying LLMs in creating natural language interfaces. The methodology involved creating a modified version of the SoloRPG "Quill," incorporating OpenAI's GPT-3.5 Turbo model. This version was tested against the traditional game through user evaluations that measured flow, presence, and cognitive absorption. Findings from 36 participants indicated no significant difference in flow and presence between the traditional and modified versions. However, a significant increase in cognitive absorption was observed in the LLM-enhanced version, suggesting that natural language interfaces can indeed enhance certain aspects of immersion in SoloRPGs.  
---|---  
Item Type:| Essay (Bachelor)  
Faculty:| EEMCS: Electrical Engineering, Mathematics and Computer Science  
Subject:| 01 general works, 18 languages and literature, 54 computer science  
Programme:| Creative Technology BSc (50447)  
Link to this item:| <https://purl.utwente.nl/essays/102964>  
Export this item as:| [BibTeX](/cgi/export/102964/BibTeX/essay-eprint-102964.bib)  
[EndNote](/cgi/export/102964/EndNote/essay-eprint-102964.enw)  
[HTML Citation](/cgi/export/102964/HTML/essay-eprint-102964.html)  
[Reference Manager](/cgi/export/102964/RIS/essay-eprint-102964.ris)  
  
  
![-](/style/images/minus.png) Show download statistics for this publication

**Daily downloads in the past month**

![]()

**Monthly downloads in the past 12 months**

![]()

[More statistics for this item...](/cgi/irstats.cgi?IRS_epchoice=EPrint&page=dashboard&eprint=102964)

![+](/style/images/plus.png) Show download statistics for this publication

Repository Staff Only: [item control page](http://essay.utwente.nl/cgi/users/home?screen=EPrint::View&eprintid=102964)


url: https://www.reddit.com/r/gaming/comments/1j2fde2/ai_playing_npcs_why_this_isnt_a_thing/
title: Reddit - Dive into anything
scrape_date: 2025-03-11T15:39:26.305346
content_type: markdown
hash: 2dfbb4a83cf3b27cb461156c6cde394f418023fb304141b37289ae486dab0322
content:
Skip to main content

Open menu Open navigation [ ](/)Go to Reddit Home

r/gaming  A chip A close button

Get App Get the Reddit app  [ Log In ](https://www.reddit.com/login/)Log in to Reddit

Expand user menu Open settings menu

[ ![r/gaming icon](https://styles.redditmedia.com/t5_2qh03/styles/communityIcon_1isvxgkk7hw51.png) Go to gaming  ](/r/gaming/)

[r/gaming](/r/gaming/)

![A banner for the subreddit](https://styles.redditmedia.com/t5_2qh03/styles/bannerBackgroundImage_j84aqdq3eyd91.png)

![r/gaming icon](https://styles.redditmedia.com/t5_2qh03/styles/communityIcon_1isvxgkk7hw51.png) [r/gaming](/r/gaming/)

The Number One Gaming forum on the Internet. 

* * *

Members Online

•

[Libero03](/user/Libero03/)

#  AI playing NPCs - why this isn't a thing? 

Large Language Models (LLM) are quite accessible and light nowadays, you can even launch one on any PC. You can easily put an instruction for such AI to roleplay any character or personality with background story and lore. Why RPG games aren't using this technology? 

Imagine RPG where NPCs dialogues are unique every time you play, where they interact with each other and simulate daily life in their environment. That would boost replayability, immersion and emergent fun through the roof. I can't wait to see it. 

Read more 

New to Reddit? 

Create your account and connect with a world of communities. 

Continue with Email 

Continue With Phone Number 

By continuing, you agree to our [User Agreement](https://www.redditinc.com/policies/user-agreement) and acknowledge that you understand the [Privacy Policy](https://www.redditinc.com/policies/privacy-policy). 

** ** [ Rank by size  ](https://www.reddit.com/best/communities/1/#t5_2qh03/)

Public 

Anyone can view, post, and comment to this community

[Reddit Rules](https://www.redditinc.com/policies/content-policy) [Privacy Policy](https://www.reddit.com/policies/privacy-policy) [User Agreement](https://www.redditinc.com/policies/user-agreement) [Reddit, Inc. © 2025. All rights reserved.](https://redditinc.com)

Expand Navigation Collapse Navigation

* &nbsp;

* &nbsp;

* * *

* TOPICS

* Internet Culture (Viral)

  * [ Amazing ](https://reddit.com/t/amazing/)
  * [ Animals & Pets ](https://reddit.com/t/animals_and_pets/)
  * [ Cringe & Facepalm ](https://reddit.com/t/cringe_and_facepalm/)
  * [ Funny ](https://reddit.com/t/funny/)
  * [ Interesting ](https://reddit.com/t/interesting/)
  * [ Memes ](https://reddit.com/t/memes/)
  * [ Oddly Satisfying ](https://reddit.com/t/oddly_satisfying/)
  * [ Reddit Meta ](https://reddit.com/t/reddit_meta/)
  * [ Wholesome & Heartwarming ](https://reddit.com/t/wholesome_and_heartwarming/)



* Games

  * [ Action Games ](https://reddit.com/t/action_games/)
  * [ Adventure Games ](https://reddit.com/t/adventure_games/)
  * [ Esports ](https://reddit.com/t/esports/)
  * [ Gaming Consoles & Gear ](https://reddit.com/t/gaming_consoles_and_gear/)
  * [ Gaming News & Discussion ](https://reddit.com/t/gaming_news_and_discussion/)
  * [ Mobile Games ](https://reddit.com/t/mobile_games/)
  * [ Other Games ](https://reddit.com/t/other_games/)
  * [ Role-Playing Games ](https://reddit.com/t/role_playing_games/)
  * [ Simulation Games ](https://reddit.com/t/simulation_games/)
  * [ Sports & Racing Games ](https://reddit.com/t/sports_and_racing_games/)
  * [ Strategy Games ](https://reddit.com/t/strategy_games/)
  * [ Tabletop Games ](https://reddit.com/t/tabletop_games/)



* Q&As

  * [ Q&As ](https://reddit.com/t/q_and_as/)
  * [ Stories & Confessions ](https://reddit.com/t/stories_and_confessions/)



* Technology

  * [ 3D Printing ](https://reddit.com/t/3d_printing/)
  * [ Artificial Intelligence & Machine Learning ](https://reddit.com/t/artificial_intelligence_and_machine_learning/)
  * [ Computers & Hardware ](https://reddit.com/t/computers_and_hardware/)
  * [ Consumer Electronics ](https://reddit.com/t/consumer_electronics/)
  * [ DIY Electronics ](https://reddit.com/t/diy_electronics/)
  * [ Programming ](https://reddit.com/t/programming/)
  * [ Software & Apps ](https://reddit.com/t/software_and_apps/)
  * [ Streaming Services ](https://reddit.com/t/streaming_services/)
  * [ Tech News & Discussion ](https://reddit.com/t/tech_news_and_discussion/)
  * [ Virtual & Augmented Reality ](https://reddit.com/t/virtual_and_augmented_reality/)



* Pop Culture

  * [ Celebrities ](https://reddit.com/t/celebrities/)
  * [ Creators & Influencers ](https://reddit.com/t/creators_and_influencers/)
  * [ Generations & Nostalgia ](https://reddit.com/t/generations_and_nostalgia/)
  * [ Podcasts ](https://reddit.com/t/podcasts/)
  * [ Streamers ](https://reddit.com/t/streamers/)
  * [ Tarot & Astrology ](https://reddit.com/t/tarot_and_astrology/)



* Movies & TV

  * [ Action Movies & Series ](https://reddit.com/t/action_movies_and_series/)
  * [ Animated Movies & Series ](https://reddit.com/t/animated_movies_and_series/)
  * [ Comedy Movies & Series ](https://reddit.com/t/comedy_movies_and_series/)
  * [ Crime, Mystery, & Thriller Movies & Series ](https://reddit.com/t/crime_mystery_and_thriller_movies_and_series/)
  * [ Documentary Movies & Series ](https://reddit.com/t/documentary_movies_and_series/)
  * [ Drama Movies & Series ](https://reddit.com/t/drama_movies_and_series/)
  * [ Fantasy Movies & Series ](https://reddit.com/t/fantasy_movies_and_series/)
  * [ Horror Movies & Series ](https://reddit.com/t/horror_movies_and_series/)
  * [ Movie News & Discussion ](https://reddit.com/t/movie_news_and_discussion/)
  * [ Reality TV ](https://reddit.com/t/reality_tv/)
  * [ Romance Movies & Series ](https://reddit.com/t/romance_movies_and_series/)
  * [ Sci-Fi Movies & Series ](https://reddit.com/t/scifi_movies_and_series/)
  * [ Superhero Movies & Series ](https://reddit.com/t/superhero_movies_and_series/)
  * [ TV News & Discussion ](https://reddit.com/t/tv_news_and_discussion/)



* * *

* RESOURCES

* [ About Reddit ](https://www.redditinc.com)
* [ Advertise ](https://ads.reddit.com/register?utm_source=web3x_consumer&utm_name=left_nav_cta&utm_content=default)
* [ Reddit Pro BETA  ](/reddit-pro?utm_source=reddit&utm_medium=left_nav_resources)
* [ Help ](https://support.reddithelp.com/hc?utm_source=reddit&utm_medium=footer&utm_campaign=evergreen)
* [ Blog ](https://redditblog.com/)
* [ Careers ](https://www.redditinc.com/careers)
* [ Press ](https://www.redditinc.com/press)

* * *

* [ Communities ](https://www.reddit.com/best/communities/1/)
* [ Best of Reddit ](https://www.reddit.com/posts/2025/global/)
* [ Topics ](https://www.reddit.com/topics/a-1/)

[ ![u/FinNewsNow avatar](https://styles.redditmedia.com/t5_44c9vd/styles/profileIcon_coec293gr9p61.png?width=48&height=48&frame=1&auto=webp&crop=48:48,smart&s=4ca1d5b22fbe2b13f5220d10f39fabb8f294cde6) FinNewsNow ](/user/FinNewsNow/) • Promoted

The Artificial Intelligence BOOM Means Continued Strong Demand for This Critical Element — Pushing Prices Higher!

Learn More

financial-news-now.com 

![Thumbnail image: The Artificial Intelligence BOOM Means Continued Strong Demand for This Critical Element — Pushing Prices Higher!](https://preview.redd.it/k8b9ehx0t35e1.jpeg?auto=webp&s=dc3e403debd3b17ffb2579142b922d4c61eb92e5)

[ ![u/PayPalOfficial avatar](https://styles.redditmedia.com/t5_outfg/styles/profileIcon_2vli30095bne1.jpeg?width=48&height=48&frame=1&auto=webp&crop=48:48,smart&s=72fa688f1c063da072640af4a294ddad09105f3e) PayPalOfficial ](/user/PayPalOfficial/) • Promoted

30 days of crypto prizes. Enter for a chance to win. Click to learn more and opt in. NO PURCH. NEC. See paypal.com/cryptosweepstakesrules

Learn More

paypal.com 

![Thumbnail image: 30 days of crypto prizes. Enter for a chance to win. Click to learn more and opt in. NO PURCH. NEC. See paypal.com/cryptosweepstakesrules](https://preview.redd.it/id71zifzbwne1.jpeg?auto=webp&s=98f1e3236a46056d0090a395b9ae0ae03bfb6529)

![](https://id.rlcdn.com/472486.gif)


url: https://hal.science/hal-04637685/
title: AI and the Future of Entertainment Technology - Archive ouverte HAL
scrape_date: 2025-03-11T15:39:26.429371
content_type: markdown
hash: 59496ad7207ff516d28e36af934cd3b5d69fc20b33642b070ec0975e44df5fc7
content:
  * Recherche
  * Accéder directement au contenu
  * Pied de page



[ ![Logo](/assets/img/hal-logo-header.png) ![Logo](/assets/img/hal-logo-text.png) ](https://hal.science)

  * [ __ Documentation ](https://about.hal.science)
  * FR

Français (FR)

Anglais (EN)

  * Se connecter




  * [HAL science ouverte **** ](/)
  * Recherche 

Loading...

###  Recherche avancée

Information de documents

Titres 
    * [Titres](javascript:void\(0\);)
    * [Sous-titre](javascript:void\(0\);)
    * [Titre de l'ouvrage](javascript:void\(0\);)
    * [Titre du volume (Série)](javascript:void\(0\);)
    * [Champ de recherche par défaut (multicritères) + texte intégral des PDF](javascript:void\(0\);)
    * [Résumé](javascript:void\(0\);)
    * [Texte intégral indexé des documents PDF](javascript:void\(0\);)
    * [Mots-clés](javascript:void\(0\);)
    * [Type de document](javascript:void\(0\);)
    * [Sous-type de document](javascript:void\(0\);)
    * [Tous les identifiants du document](javascript:void\(0\);)
    * [Identifiant HAL du dépôt](javascript:void\(0\);)
    * [Langue du document (texte)](javascript:void\(0\);)
    * [Pays (Texte)](javascript:void\(0\);)
    * [Ville](javascript:void\(0\);)
    * [À paraître (true ou false)](javascript:void\(0\);)
__ Ajouter

Auteur

Auteur (multicritères) 
    * [Auteur (multicritères)](javascript:void\(0\);)
    * [Auteur : Nom complet](javascript:void\(0\);)
    * [Auteur : Nom de famille](javascript:void\(0\);)
    * [Auteur : Prénom](javascript:void\(0\);)
    * [Auteur : Complément de nom, deuxième prénom](javascript:void\(0\);)
    * [Auteur : Organisme payeur](javascript:void\(0\);)
    * [Auteur : IdHal (chaîne de caractères)](javascript:void\(0\);)
    * [Auteur : Fonction](javascript:void\(0\);)
    * [Auteur : personID (entier)](javascript:void\(0\);)
    * [Auteur : Identifiant de l'organisme payeur](javascript:void\(0\);)
    * [Auteur : Identifiant de la structure](javascript:void\(0\);)
    * [Directeur de thèse](javascript:void\(0\);)
    * [Éditeur](javascript:void\(0\);)
    * [Éditeur scientifique](javascript:void\(0\);)
    * [Éditeur de la série](javascript:void\(0\);)
__ Ajouter

Structure

Structure (multicritères) 
    * [Structure (multicritères)](javascript:void\(0\);)
    * [Structure : Acronyme](javascript:void\(0\);)
    * [Structure : Nom](javascript:void\(0\);)
    * [Structure : Code](javascript:void\(0\);)
    * [Structure : Pays](javascript:void\(0\);)
    * [Structure : Type](javascript:void\(0\);)
    * [Structure : État dans le référentiel](javascript:void\(0\);)
    * [Structure : Identifiant HAL de la structure](javascript:void\(0\);)
    * [Structure/équipe de recherche : Pays](javascript:void\(0\);)
    * [Structure/regroupement d'équipes : Pays](javascript:void\(0\);)
    * [Structure/laboratoire : Pays](javascript:void\(0\);)
    * [Structure/regroupement de laboratoires : Pays](javascript:void\(0\);)
    * [Structure/institution : Pays](javascript:void\(0\);)
    * [Structure/regroupement d'institutions : Pays](javascript:void\(0\);)
__ Ajouter

Autres

Champ de recherche par défaut (multicritères) 
    * [Champ de recherche par défaut (multicritères)](javascript:void\(0\);)
    * [](javascript:void\(0\);)
    * [Revues (multicritères)](javascript:void\(0\);)
    * [Revue : Éditeur](javascript:void\(0\);)
    * [Revue : Titre abrégé](javascript:void\(0\);)
    * [Revue : Titre](javascript:void\(0\);)
    * [Revue : Identifiant interne](javascript:void\(0\);)
    * [Revue : Date de début de publication](javascript:void\(0\);)
    * [Revue : ISSN électronique](javascript:void\(0\);)
    * [Revue : ISSN](javascript:void\(0\);)
    * [Revue : Éditeur](javascript:void\(0\);)
    * [Revue : Couleur dans SHERPA/RoMEO](javascript:void\(0\);)
    * [Revue : État dans le référentiel](javascript:void\(0\);)
    * [](javascript:void\(0\);)
    * [Colloque (multicritères)](javascript:void\(0\);)
    * [Colloque : Titre](javascript:void\(0\);)
    * [Colloque : Organisateur](javascript:void\(0\);)
    * [Colloque : date de début (Année)](javascript:void\(0\);)
    * [Colloque : date de fin (Année)](javascript:void\(0\);)
    * [Conférence invitée (oui/non)](javascript:void\(0\);)
    * [](javascript:void\(0\);)
    * [Projets ANR (multicritères)](javascript:void\(0\);)
    * [Projet ANR : Acronyme](javascript:void\(0\);)
    * [Projet ANR : Acronyme du programme](javascript:void\(0\);)
    * [Projet ANR : Code décision (référence)](javascript:void\(0\);)
    * [Projet ANR : Nom](javascript:void\(0\);)
    * [Projet ANR : Identifiant interne](javascript:void\(0\);)
    * [Projet ANR : État dans le référentiel](javascript:void\(0\);)
    * [Projet ANSES: Acronyme](javascript:void\(0\);)
    * [Projet ANSES: Code decision (reference)](javascript:void\(0\);)
    * [Projet ANSES: Nom](javascript:void\(0\);)
    * [Projet ADEME: Acronyme](javascript:void\(0\);)
    * [Projet ADEME: Code decision (reference)](javascript:void\(0\);)
    * [Projet ADEME: Nom](javascript:void\(0\);)
    * [](javascript:void\(0\);)
    * [Projets européens (multicritères)](javascript:void\(0\);)
    * [Projet européen : Acronyme](javascript:void\(0\);)
    * [Projet européen : Identifiant de l'appel à projet](javascript:void\(0\);)
    * [Projet européen : Référence](javascript:void\(0\);)
    * [Projet européen : Nom](javascript:void\(0\);)
    * [Projet européen : Date de fin](javascript:void\(0\);)
    * [Projet européen : Financement](javascript:void\(0\);)
    * [Projet européen : Date de début](javascript:void\(0\);)
    * [Projet européen : État dans le référentiel](javascript:void\(0\);)
    * [Projet européen : Identifiant interne](javascript:void\(0\);)
    * [](javascript:void\(0\);)
    * [Date de production : année](javascript:void\(0\);)
    * [Date de mise en ligne : année](javascript:void\(0\);)
    * [Date de publication : année](javascript:void\(0\);)
    * [Date d'écriture : année](javascript:void\(0\);)
    * [Date de modification du dépôt : année](javascript:void\(0\);)
    * [Date de dépôt : année](javascript:void\(0\);)
    * [Date de publication électronique : année](javascript:void\(0\);)
    * [](javascript:void\(0\);)
    * [Collection HAL (multicritères)](javascript:void\(0\);)
    * [Collection HAL : catégorie](javascript:void\(0\);)
    * [Collection HAL : Code](javascript:void\(0\);)
    * [Collection HAL : Nom](javascript:void\(0\);)
    * [Collection HAL : Identifiant interne](javascript:void\(0\);)
    * [](javascript:void\(0\);)
    * [Identifiant interne du contributeur/déposant](javascript:void\(0\);)
    * [Nom complet du contributeur/déposant](javascript:void\(0\);)
    * [](javascript:void\(0\);)
    * [Domaines](javascript:void\(0\);)
    * [Domaine primaire](javascript:void\(0\);)
    * [Domaine racine](javascript:void\(0\);)
    * [Sous-domaine niveau 1](javascript:void\(0\);)
    * [Sous-domaine niveau 2](javascript:void\(0\);)
    * [Sous-domaine niveau 3](javascript:void\(0\);)
    * [](javascript:void\(0\);)
    * [Statut du document](javascript:void\(0\);)
    * [Version du document](javascript:void\(0\);)
    * [Type de dépôt](javascript:void\(0\);)
    * [Type de document](javascript:void\(0\);)
    * [ISBN](javascript:void\(0\);)
    * [Numéro - référence](javascript:void\(0\);)
    * [Identifiant DOI](javascript:void\(0\);)
    * [](javascript:void\(0\);)
    * [Classification](javascript:void\(0\);)
    * [Audience](javascript:void\(0\);)
    * [Vulgarisation](javascript:void\(0\);)
    * [Comité de lecture - texte (oui ou non)](javascript:void\(0\);)
    * [Actes de colloque](javascript:void\(0\);)
    * [Référence interne](javascript:void\(0\);)
    * [Financement](javascript:void\(0\);)
    * [Collaborations](javascript:void\(0\);)
__ Ajouter

     Lancer la recherche

Recherche experte (SolR)

###  Recherche experte (SolR)

Lancer la recherche

Vers la recherche avancée

  * [ Déposer  ](/submit/index)



×

×

×

Loading...

×

Pré-Publication, Document De Travail [ ](https://doc.hal.science/faq/#quest-ce-quun-preprint) Année : 2024

## AI and the Future of Entertainment Technology

Markus Schatten (1) 

Afficher plus de détails

1 AI Lab - Artificial Intelligence Laboratory    


[Markus Schatten](/search/index/q/*/authIdHal_i/1398385)

  * Fonction : **Auteur**
  * PersonId : **1398385**



[Artificial Intelligence Laboratory](/search/index/q/*/structId_i/1202478)

Réduire la vue détaillée

####  Résumé 

en 

Dive into the future of entertainment where artificial intelligence (AI) elevates viewers from passive audiences to active creators and directors of their own media experiences. This article explores the transformative impact of Generative Artificial Intelligence Applications (GAIAs) across various media, including text, images, audio, and video. Envision creating personalized video games, movies, and narratives with mere spoken commands, tailoring the content to your individual preferences for an unprecedented level of customization. While showcasing AI’s potential to blur the lines between consumer and creator, we also discuss the significant challenges and ethical considerations that accompany these technologies. Engage with this critical analysis of how generative AI is redefining our interaction with media, signaling a new era in entertainment technology, but also prompting a need for careful consideration of privacy, security, and the impact on traditional creative roles. 

####  Mots clés 

en 

[Generative artificial intelligence](/search/index/q/*/keyword_t/Generative%20artificial%20intelligence) [Future](/search/index/q/*/keyword_t/Future) [Entertainment Industry](/search/index/q/*/keyword_t/Entertainment%20Industry) [Personalization](/search/index/q/*/keyword_t/Personalization)

####  Domaines 

[Intelligence artificielle [cs.AI]](/search/index/q/*/level1_domain_s/info.info-ai) [Ordinateur et société [cs.CY]](/search/index/q/*/level1_domain_s/info.info-cy)

Liste complète des métadonnées 

Fichiers et aperçu 

Fichier principal

![Vignette du fichier](//thumb.ccsd.cnrs.fr/9476574/medium)

ms_ai_future_entertainment.pdf (7)  [Télécharger le fichier ](https://hal.science/hal-04637685v1/file/ms_ai_future_entertainment.pdf)

Origine |  Fichiers produits par l'(les) auteur(s)   
---|---  
Licence |  ![](/assets/img/licences/cc.png) ![](/assets/img/licences/by.png) ![](/assets/img/licences/nd.png) [ Paternité - Pas de modifications](http://creativecommons.org/licenses/by-nd/4.0/)  
  
[Markus Schatten](/search/index/q/*/contributorId_i/1559698)  : [Connectez-vous pour contacter le contributeur](javascript:void\(0\);)

https://hal.science/hal-04637685

Soumis le : dimanche 7 juillet 2024-11:16:54

Dernière modification le : mardi 9 juillet 2024-03:12:18

[ __ Télécharger pour visualiser ](https://hal.science/hal-04637685v1/document)

###  Dates et versions

hal-04637685 , [ version 1](https://hal.science/hal-04637685v1) (07-07-2024)

###  Licence

![](/assets/img/licences/cc.png) ![](/assets/img/licences/by.png) ![](/assets/img/licences/nd.png) [ Paternité - Pas de modifications](http://creativecommons.org/licenses/by-nd/4.0/)

###  Identifiants

  * HAL Id :  hal-04637685 , version 1 



###  Citer

Markus Schatten. AI and the Future of Entertainment Technology. 2024. [⟨hal-04637685⟩](https://hal.science/hal-04637685v1)

###  Exporter

[BibTeX](https://hal.science/hal-04637685v1/bibtex) [XML-TEI](https://hal.science/hal-04637685v1/tei) [Dublin Core](https://hal.science/hal-04637685v1/dc) [DC Terms](https://hal.science/hal-04637685v1/dcterms) [EndNote](https://hal.science/hal-04637685v1/endnote) [DataCite](https://hal.science/hal-04637685v1/datacite)

###  Collections

[ TDS-MACS](/TDS-MACS)

855  Consultations

715  Téléchargements

###  Partager

[](https://www.addtoany.com/share) More 

[![Ministère de l'enseignement supérieur et de la recherche](/public/mesr.svg)](https://www.enseignementsup-recherche.gouv.fr/)

[![CNRS](/public/cnrs.png)](https://www.cnrs.fr/)

[![Inria](/public/inria.svg)](https://www.inria.fr/)

[![INRAE](/public/inrae.svg)](https://www.inrae.fr/)

Contact 

  * [Support](//doc.archives-ouvertes.fr/support)
  * [ ![CoreTrustSeal logo](/assets/img/CoreTrustSeal.png) ](https://doi.org/10.34894/VQ1DJA)



Ressources

  * [ Documentation ](//doc.archives-ouvertes.fr/)
  * [ Formation ](//learning.hal.science/)
  * [ FAQ ](//doc.archives-ouvertes.fr/faq)
  * [ API ](https://api.archives-ouvertes.fr/docs)
  * [ OAI-PMH ](https://api.archives-ouvertes.fr/docs/oai)
  * [ AuréHAL ](https://aurehal.archives-ouvertes.fr)



Informations

  * [ À propos ](https://about.hal.science)
  * [ Données personnelles ](https://about.hal.science/privacy-policy)
  * [ Mentions légales ](https://about.hal.science/legal-notice/)
  * [ Accessibilité ](https://about.hal.science/web-accessibility/)
  * [ Conformité RGAA ](https://about.hal.science/RGAA-compliance/)



Questions juridiques 

  * [ Je publie, quels sont mes droits ? ](https://www.ouvrirlascience.fr/je-publie-quels-sont-mes-droits/)
  * [ Loi pour une République numérique ](https://www.ouvrirlascience.fr/guide-application-loi-republique-numerique-article-30-ecrits-scientifiques-version-courte/)
  * [ Stratégie de non-cession des droits ](https://www.ouvrirlascience.fr/mettre-en-oeuvre-la-strategie-de-non-cession-des-droits-sur-les-publications-scientifiques/)
  * [ SHERPA/RoMEO ](https://v2.sherpa.ac.uk/romeo/)



Portails 

  * [ Portails institutionnels ](https://hal.science/browse/portal)
  * [HAL](https://hal.science)
  * [HAL SHS](//shs.hal.science)
  * [HAL Thèses](//theses.hal.science)
  * [MédiHAL](//media.hal.science)



CCSD 

  * [ CCSD ](//www.ccsd.cnrs.fr/)
  * [ Episciences ](//www.episciences.org/)
  * [ Sciencesconf ](//www.sciencesconf.org/)


url: https://upcommons.upc.edu/handle/2117/415440
title: Improving realism and interactivity in games: a study of AI-integrated non player characters (NPCs)
scrape_date: 2025-03-11T15:39:26.575194
content_type: markdown
hash: 450ead399047a6dec65512d08a285add9db80203efea7cf0f504764f92e03788
content:
Ir al contenido (pulsa Retorno)

#  [![Universitat Politècnica de Catalunya](/themes/UPCommons//images/logoUPC.png)](https://www.upc.edu)

  * **   **
    * Catala
    * Castellano 
    * English 
  * **   ** [Inicia la sessio](/login)[Registre (usuaris no UPC)](/register)[Entrada (usuaris no UPC)](/ldap-login)
  * Intercanvia la navegacio   



  * [mailContacta](/contact)


  * world Catala 
    * Catala
    * Castellano 
    * English 


  * userInicia sessio    [Inicia la sessio](/login)[Registre (usuaris no UPC)](/register)[Entrada (usuaris no UPC)](/ldap-login)

Intercanvia la navegacio   

#  [**UPCommons.**  Portal del coneixement obert de la UPC](/)

![Banner header](/themes/UPCommons//images/upcommons09.png)

75.141 Treballs academics UPC

You are here:

Visualitza l'item ****

  * [__  Pagina inicial de UPCommons](/)
  * [Treballs academics](/handle/2117/28592)
  * [Masters oficials](/handle/2099.1/4237)
  * [Master in Artificial Intelligence - MAI](/handle/2099.1/20413)
  * Visualitza l'item



  * __  [Pagina inicial de UPCommons](/)
  * [Treballs academics](/handle/2117/28592)
  * [Masters oficials](/handle/2099.1/4237)
  * [Master in Artificial Intelligence - MAI](/handle/2099.1/20413)
  * Visualitza l'item



JavaScript is disabled for your browser. Some features of this site may not work without it.

## Improving realism and interactivity in games: a study of AI-integrated non player characters (NPCs)

![Thumbnail](/bitstream/id/fc8d39f6-6d3e-4acf-a812-13c3011fa2b2/188203.pdf.jpg?sequence=4&isAllowed=y)

##### Visualitza/Obre

[__188203.pdf (23,02Mb)](/bitstream/handle/2117/415440/188203.pdf?sequence=2&isAllowed=y)

[__  Veure estadistiques d'us d'UPCommons](/handle/2117/415440/es)

[__  Estadistiques de LA Referencia / Recolecta](javascript:toggleStatsLaReferencia\(\))

Inclou dades d'us des de 2022

Cita com:

hdl:2117/415440

  
[Mostra el registre d'item complet](/handle/2117/415440?show=full)

[Nagarkar, Saurabh Bharat](/browse?value=Nagarkar, Saurabh Bharat&type=author)

Tutor / director[Panizzi, Emanuele](/browse?value=Panizzi, Emanuele&type=contributor)

Realitzat a/amb[Università degli studi di Roma "La Sapienza"](/browse?value=Universita%CC%80 degli studi di Roma %22La Sapienza%22&type=contributor)

Tipus de documentProjecte Final de Master Oficial

Data2024-07-10

Condicions d'accesAcces obert

**Tots els drets reservats.** Aquesta obra esta protegida pels drets de propietat intel*lectual i industrial corresponents. Sense perjudici de les exempcions legals existents, queda prohibida la seva reproduccio, distribucio, comunicacio publica o transformacio sense l'autoritzacio de la persona titular dels drets 

Abstract

This thesis presents the development and integration of a conversational AI within Non-Player Characters (NPCs) in video games. The main objective is to elevate player-NPC dialogues, making them more dynamic and reflective of human-like interactions. Utilizing OpenAI's GPT-4 API, the project pushes the boundaries of NPC interaction by enabling complex conversation and emotional engagement throughout the conversation. A comprehensive dataset was constructed, including environment descriptions, game context, and a broad range of human interactions such as humor, empathy, and problem-solving. These elements inform the AI's dialogue generation process, enhancing the NPCs' ability to provide tailored and context-aware responses. The technical implementation integrates these advanced conversational capabilities within the gaming environment, ensuring seamless interactions without disrupting the game's narrative flow. Initial testing phases focused on refining the AI's response mechanisms and reducing latency to improve real-time interaction quality. Subsequent evaluations involved user surveys to gauge the AI's impact on game experience and narrative engagement. Findings indicate a significant improvement in user satisfaction, confirming the potential of advanced conversational AIs to change NPC interactions in video games. The findings of this research not only showcase the potential enhancements to gameplay but also provide a framework for future advancements in the application of AI within interactive digital storytelling and virtual character development.

Materies[Artificial intelligence](/browse?value=Artificial intelligence&type=subject), [Video games](/browse?value=Video games&type=subject), [Intel*ligencia artificial](/browse?value=Intel%C2%B7lig%C3%A8ncia artificial&type=subject), [Videojocs](/browse?value=Videojocs&type=subject)

TitulacioMÀSTER UNIVERSITARI EN INTEL*LIGÈNCIA ARTIFICIAL (Pla 2017)

URI<http://hdl.handle.net/2117/415440>

Col*leccions

  * [Masters oficials - Master in Artificial Intelligence - MAI](/handle/2099.1/20413) [315]



[__  Veure estadistiques d'us d'UPCommons](/handle/2117/415440/es)

  
[Mostra el registre d'item complet](/handle/2117/415440?show=full)

  
Fitxers| Descripcio| Mida| Format| Visualitza  
---|---|---|---|---  
[188203.pdf](/bitstream/handle/2117/415440/188203.pdf?sequence=2&isAllowed=y "188203.pdf")| | 23,02Mb| PDF| [Visualitza/Obre](/bitstream/handle/2117/415440/188203.pdf?sequence=2&isAllowed=y)  
  
Cerca a UPCommons

Aquesta col*leccio

## Explora

Aquesta col*leccio[Per data d'edicio](/handle/2099.1/20413/browse?type=dateissued)[Autors](/handle/2099.1/20413/browse?type=author)[Altres contribucions](/handle/2099.1/20413/browse?type=contributor)[Titols](/handle/2099.1/20413/browse?type=title)[Temes](/handle/2099.1/20413/browse?type=subject)Aquest diposit[Comunitats i col*leccions](/community-list-pfc)[Per data d'edicio](/handle/2117/28592/browse?type=dateissued)[Autors](/handle/2117/28592/browse?type=author)[Altres contribucions](/handle/2117/28592/browse?type=contributor)[Titols](/handle/2117/28592/browse?type=title)[Temes](/handle/2117/28592/browse?type=subject)

(C) [ UPC ![Obrir en finestra nova](/themes/UPCommons//images/blank_blau.png)](https://www.upc.edu) . Servei de Biblioteques, Publicacions i Arxius 

[info.biblioteques@upc.edu](mailto:info.biblioteques@upc.edu)

  * [Sobre aquest web](/page/about)
  * [Metadades subjectes a:![Metadades subjectes a CC0](/themes/UPCommons//images/creativecommons/cc-zero.png)](https://creativecommons.org/publicdomain/zero/1.0/)
  * [Contacta](/contact)
  * [Envia comentaris](/feedback)
  * Configuracio de privadesa
  * ![Inici de la pàgina](/themes/UPCommons//images/amunt.png)



[ ](/htmlmap)
  *[
                                    UPC
                                ]: Universitat Politècnica de Catalunya . BarcelonaTech


url: https://arxiv.org/abs/2411.00308
title: [2411.00308] GPT for Games: An Updated Scoping Review (2020-2024)
scrape_date: 2025-03-11T15:43:15.064159
content_type: markdown
hash: 1df03d69422ad43941fd9cb5e99dc7d205d02f4d2ed48d60293069050a96ceb9
content:
Skip to main content

[![Cornell University](/static/browse/0.3.4/images/icons/cu/cornell-reduced-white-SMALL.svg)](https://www.cornell.edu/)

We gratefully acknowledge support from the Simons Foundation, [member institutions](https://info.arxiv.org/about/ourmembers.html), and all contributors. [Donate](https://info.arxiv.org/about/donate.html)

[](/IgnoreMe)

[![arxiv logo](/static/browse/0.3.4/images/arxiv-logo-one-color-white.svg)](/) > [cs](/list/cs/recent) > arXiv:2411.00308 

[Help](https://info.arxiv.org/help) | [Advanced Search](https://arxiv.org/search/advanced)

All fields Title Author Abstract Comments Journal reference ACM classification MSC classification Report number arXiv identifier DOI ORCID arXiv author ID Help pages Full text

Search

[![arXiv logo](/static/browse/0.3.4/images/arxiv-logomark-small-white.svg)](https://arxiv.org/)

[ ![Cornell University Logo](/static/browse/0.3.4/images/icons/cu/cornell-reduced-white-SMALL.svg) ](https://www.cornell.edu/)

open search

GO

open navigation menu

## quick links

  * [Login](https://arxiv.org/login)
  * [Help Pages](https://info.arxiv.org/help)
  * [About](https://info.arxiv.org/about)



# Computer Science > Artificial Intelligence

**arXiv:2411.00308** (cs) 

[Submitted on 1 Nov 2024]

# Title:GPT for Games: An Updated Scoping Review (2020-2024)

Authors:[Daijin Yang](https://arxiv.org/search/cs?searchtype=author&query=Yang,+D), [Erica Kleinman](https://arxiv.org/search/cs?searchtype=author&query=Kleinman,+E), [Casper Harteveld](https://arxiv.org/search/cs?searchtype=author&query=Harteveld,+C)

View a PDF of the paper titled GPT for Games: An Updated Scoping Review (2020-2024), by Daijin Yang and 2 other authors

[View PDF](/pdf/2411.00308) [HTML (experimental)](https://arxiv.org/html/2411.00308v1)

> Abstract:Due to GPT's impressive generative capabilities, its applications in games are expanding rapidly. To offer researchers a comprehensive understanding of the current applications and identify both emerging trends and unexplored areas, this paper introduces an updated scoping review of 131 articles, 76 of which were published in 2024, to explore GPT's potential for games. By coding and synthesizing the papers, we identify five prominent applications of GPT in current game research: procedural content generation, mixed-initiative game design, mixed-initiative gameplay, playing games, and game user research. Drawing on insights from these application areas and emerging research, we propose future studies should focus on expanding the technical boundaries of the GPT models and exploring the complex interaction dynamics between them and users. This review aims to illustrate the state of the art in innovative GPT applications in games, offering a foundation to enrich game development and enhance player experiences through cutting-edge AI innovations. 

Comments: | Submitted to IEEE Transactions on Games  
---|---  
Subjects: |  Artificial Intelligence (cs.AI)  
ACM classes: | A.1  
Cite as: | [arXiv:2411.00308](https://arxiv.org/abs/2411.00308) [cs.AI]  
  | (or  [arXiv:2411.00308v1](https://arxiv.org/abs/2411.00308v1) [cs.AI] for this version)   
  |  <https://doi.org/10.48550/arXiv.2411.00308> Focus to learn more arXiv-issued DOI via DataCite  
  
## Submission history

From: Daijin Yang [[view email](/show-email/223adfea/2411.00308)]   
**[v1]** Fri, 1 Nov 2024 02:00:25 UTC (923 KB)  


Full-text links:

## Access Paper:

View a PDF of the paper titled GPT for Games: An Updated Scoping Review (2020-2024), by Daijin Yang and 2 other authors

  * [View PDF](/pdf/2411.00308)
  * [HTML (experimental)](https://arxiv.org/html/2411.00308v1)
  * [TeX Source](/src/2411.00308)
  * [Other Formats](/format/2411.00308)



[ ![license icon](https://arxiv.org/icons/licenses/by-sa-4.0.png) view license ](http://creativecommons.org/licenses/by-sa/4.0/ "Rights to this article")

Current browse context: 

cs.AI

[< prev](/prevnext?id=2411.00308&function=prev&context=cs.AI "previous in cs.AI \(accesskey p\)")   |   [next >](/prevnext?id=2411.00308&function=next&context=cs.AI "next in cs.AI \(accesskey n\)")   


[new](/list/cs.AI/new) |  [recent](/list/cs.AI/recent) | [2024-11](/list/cs.AI/2024-11)

Change to browse by: 

[cs](/abs/2411.00308?context=cs)  


### References & Citations

  * [NASA ADS](https://ui.adsabs.harvard.edu/abs/arXiv:2411.00308)
  * [Google Scholar](https://scholar.google.com/scholar_lookup?arxiv_id=2411.00308)
  * [Semantic Scholar](https://api.semanticscholar.org/arXiv:2411.00308)



[a](/static/browse/0.3.4/css/cite.css) export BibTeX citation Loading...

## BibTeX formatted citation

×

loading...

Data provided by: 

### Bookmark

[ ![BibSonomy logo](/static/browse/0.3.4/images/icons/social/bibsonomy.png) ](http://www.bibsonomy.org/BibtexHandler?requTask=upload&url=https://arxiv.org/abs/2411.00308&description=GPT for Games: An Updated Scoping Review \(2020-2024\) "Bookmark on BibSonomy") [ ![Reddit logo](/static/browse/0.3.4/images/icons/social/reddit.png) ](https://reddit.com/submit?url=https://arxiv.org/abs/2411.00308&title=GPT for Games: An Updated Scoping Review \(2020-2024\) "Bookmark on Reddit")

Bibliographic Tools

# Bibliographic and Citation Tools

Bibliographic Explorer Toggle

Bibliographic Explorer _([What is the Explorer?](https://info.arxiv.org/labs/showcase.html#arxiv-bibliographic-explorer))_

Connected Papers Toggle

Connected Papers _([What is Connected Papers?](https://www.connectedpapers.com/about))_

Litmaps Toggle

Litmaps _([What is Litmaps?](https://www.litmaps.co/))_

scite.ai Toggle

scite Smart Citations _([What are Smart Citations?](https://www.scite.ai/))_

Code, Data, Media

# Code, Data and Media Associated with this Article

alphaXiv Toggle

alphaXiv _([What is alphaXiv?](https://alphaxiv.org/))_

Links to Code Toggle

CatalyzeX Code Finder for Papers _([What is CatalyzeX?](https://www.catalyzex.com))_

DagsHub Toggle

DagsHub _([What is DagsHub?](https://dagshub.com/))_

GotitPub Toggle

Gotit.pub _([What is GotitPub?](http://gotit.pub/faq))_

Huggingface Toggle

Hugging Face _([What is Huggingface?](https://huggingface.co/huggingface))_

Links to Code Toggle

Papers with Code _([What is Papers with Code?](https://paperswithcode.com/))_

ScienceCast Toggle

ScienceCast _([What is ScienceCast?](https://sciencecast.org/welcome))_

Demos

# Demos

Replicate Toggle

Replicate _([What is Replicate?](https://replicate.com/docs/arxiv/about))_

Spaces Toggle

Hugging Face Spaces _([What is Spaces?](https://huggingface.co/docs/hub/spaces))_

Spaces Toggle

TXYZ.AI _([What is TXYZ.AI?](https://txyz.ai))_

Related Papers

# Recommenders and Search Tools

Link to Influence Flower

Influence Flower _([What are Influence Flowers?](https://influencemap.cmlab.dev/))_

Core recommender toggle

CORE Recommender _([What is CORE?](https://core.ac.uk/services/recommender))_

  * Author
  * Venue
  * Institution
  * Topic



About arXivLabs 

# arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? [**Learn more about arXivLabs**](https://info.arxiv.org/labs/index.html).

[Which authors of this paper are endorsers?](/auth/show-endorsers/2411.00308) | [Disable MathJax](javascript:setMathjaxCookie\(\)) ([What is MathJax?](https://info.arxiv.org/help/mathjax.html)) 

  * [About](https://info.arxiv.org/about)
  * [Help](https://info.arxiv.org/help)



  * contact arXivClick here to contact arXiv [ Contact](https://info.arxiv.org/help/contact.html)
  * subscribe to arXiv mailingsClick here to subscribe [ Subscribe](https://info.arxiv.org/help/subscribe)



  * [Copyright](https://info.arxiv.org/help/license/index.html)
  * [Privacy Policy](https://info.arxiv.org/help/policies/privacy_policy.html)



  * [Web Accessibility Assistance](https://info.arxiv.org/help/web_accessibility.html)
  * [arXiv Operational Status ](https://status.arxiv.org)  
Get status notifications via [email](https://subscribe.sorryapp.com/24846f03/email/new) or [slack](https://subscribe.sorryapp.com/24846f03/slack/new)



  *[ACM]: Association of Computing Machinery Classification


url: https://arxiv.org/html/2402.18659v1
title: Large Language Models and Games: A Survey and Roadmap
scrape_date: 2025-03-11T15:43:15.182906
content_type: markdown
hash: 80ad1eec88ab56b45407fcc5b0d32bca1e5e18d9d698b2c209d09097edac77b8
content:
  1. [1 Introduction](https://arxiv.org/html/2402.18659v1#S1 "1 Introduction ‣ Large Language Models and Games: A Survey and Roadmap")
  2. [2 A Note on Terminology](https://arxiv.org/html/2402.18659v1#S2 "2 A Note on Terminology ‣ Large Language Models and Games: A Survey and Roadmap")
  3. [3 Roles of LLMs in Games](https://arxiv.org/html/2402.18659v1#S3 "3 Roles of LLMs in Games ‣ Large Language Models and Games: A Survey and Roadmap")
    1. [3.1 Player](https://arxiv.org/html/2402.18659v1#S3.SS1 "3.1 Player ‣ 3 Roles of LLMs in Games ‣ Large Language Models and Games: A Survey and Roadmap")
    2. [3.2 Non-Player Characters](https://arxiv.org/html/2402.18659v1#S3.SS2 "3.2 Non-Player Characters ‣ 3 Roles of LLMs in Games ‣ Large Language Models and Games: A Survey and Roadmap")
    3. [3.3 Player Assistant](https://arxiv.org/html/2402.18659v1#S3.SS3 "3.3 Player Assistant ‣ 3 Roles of LLMs in Games ‣ Large Language Models and Games: A Survey and Roadmap")
    4. [3.4 Commentator/Reteller](https://arxiv.org/html/2402.18659v1#S3.SS4 "3.4 Commentator/Reteller ‣ 3 Roles of LLMs in Games ‣ Large Language Models and Games: A Survey and Roadmap")
    5. [3.5 Game Master](https://arxiv.org/html/2402.18659v1#S3.SS5 "3.5 Game Master ‣ 3 Roles of LLMs in Games ‣ Large Language Models and Games: A Survey and Roadmap")
    6. [3.6 Game Mechanic](https://arxiv.org/html/2402.18659v1#S3.SS6 "3.6 Game Mechanic ‣ 3 Roles of LLMs in Games ‣ Large Language Models and Games: A Survey and Roadmap")
    7. [3.7 Automated Designer](https://arxiv.org/html/2402.18659v1#S3.SS7 "3.7 Automated Designer ‣ 3 Roles of LLMs in Games ‣ Large Language Models and Games: A Survey and Roadmap")
    8. [3.8 Design Assistant](https://arxiv.org/html/2402.18659v1#S3.SS8 "3.8 Design Assistant ‣ 3 Roles of LLMs in Games ‣ Large Language Models and Games: A Survey and Roadmap")
  4. [4 A Roadmap for Future Applications of LLMs in Games](https://arxiv.org/html/2402.18659v1#S4 "4 A Roadmap for Future Applications of LLMs in Games ‣ Large Language Models and Games: A Survey and Roadmap")
  5. [5 Limitations of LLMs in games](https://arxiv.org/html/2402.18659v1#S5 "5 Limitations of LLMs in games ‣ Large Language Models and Games: A Survey and Roadmap")
  6. [6 Ethical Issues with LLMs in games](https://arxiv.org/html/2402.18659v1#S6 "6 Ethical Issues with LLMs in games ‣ Large Language Models and Games: A Survey and Roadmap")
  7. [7 Conclusions](https://arxiv.org/html/2402.18659v1#S7 "7 Conclusions ‣ Large Language Models and Games: A Survey and Roadmap")



License: arXiv.org perpetual non-exclusive license

arXiv:2402.18659v1 [cs.CL] 28 Feb 2024

# Large Language Models and Games:   
A Survey and Roadmap

Roberto Gallotta11{}^{1}start_FLOATSUPERSCRIPT 1 end_FLOATSUPERSCRIPT, Graham Todd22{}^{2}start_FLOATSUPERSCRIPT 2 end_FLOATSUPERSCRIPT, Marvin Zammit11{}^{1}start_FLOATSUPERSCRIPT 1 end_FLOATSUPERSCRIPT, Sam Earle22{}^{2}start_FLOATSUPERSCRIPT 2 end_FLOATSUPERSCRIPT, Antonios Liapis11{}^{1}start_FLOATSUPERSCRIPT 1 end_FLOATSUPERSCRIPT, Julian Togelius22{}^{2}start_FLOATSUPERSCRIPT 2 end_FLOATSUPERSCRIPT and Georgios N. Yannakakis11{}^{1}start_FLOATSUPERSCRIPT 1 end_FLOATSUPERSCRIPT   
1: Institute of Digital Games, University of Malta, Msida, Malta   
2: Tandon School of Engineering, New York University, New York, USA   
roberto.gallotta@um.edu.mt, gdrtodd@nyu.edu, marvin.zammit@um.edu.mt, se2161@nyu.edu, antonios.liapis@um.edu.mt, julian@togelius.com, georgios.yannakakis@um.edu.mt 

###### Abstract

Recent years have seen an explosive increase in research on large language models (LLMs), and accompanying public engagement on the topic. While starting as a niche area within natural language processing, LLMs have shown remarkable potential across a broad range of applications and domains, including games. This paper surveys the current state of the art across the various applications of LLMs _in_ and _for_ games, and identifies the different roles LLMs can take within a game. Importantly, we discuss underexplored areas and promising directions for future uses of LLMs in games and we reconcile the potential and limitations of LLMs within the games domain. As the first comprehensive survey and roadmap at the intersection of LLMs and games, we are hopeful that this paper will serve as the basis for groundbreaking research and innovation in this exciting new field.

###### Index Terms: 

Large Language Models, Digital Games, Video Games, Survey, Generative Text, Gameplaying, Procedural Content Generation, Generative AI. 

##  1 Introduction

Five years ago, autoregressive language modeling was a somewhat niche topic within natural language processing. Training models to simply predict text based on existing text was considered something of primarily theoretical interest, though it might have applications as writing support systems. This changed drastically in 2019 when the GPT-2 model was released [[1](https://arxiv.org/html/2402.18659v1#bib.bib1)]. GPT-2 demonstrated convincingly that transformer models trained on large text corpora could not only generate surprisingly high-quality and coherent text, but also that the text generation could be controlled by carefully prompting the model. Subsequent developments, including larger models, instruction fine-tuning, reinforcement learning from human feedback [[2](https://arxiv.org/html/2402.18659v1#bib.bib2)], and the combination of these features in ChatGPT in late 2022, turbocharged interest in large language models (LLMs). It suddenly appeared that LLMs could do almost anything—as long as both problem and solution could be formulated as text.

LLMs are currently a very active research field, where researchers are both focused on improving the capabilities of LLMs while reducing their compute and memory footprint, and on understanding and learning to harness the capabilities of existing LLMs. Informed opinions on the ultimate capabilities of LLM technology vary widely, from those who see such models as “sparks of AGI” [[3](https://arxiv.org/html/2402.18659v1#bib.bib3)] to those who see them as largely doing approximate retrieval from a lossy compression of the internet [[4](https://arxiv.org/html/2402.18659v1#bib.bib4)].

Games, including board games and video games, serve both as a source of important benchmarks for AI research and as an important application area for AI techniques [[5](https://arxiv.org/html/2402.18659v1#bib.bib5)]. Almost any game utilizes some kind of AI technology, and we are currently in an exploratory phase where the developers and researchers try to figure out how to best make use of recent advances in AI [[6](https://arxiv.org/html/2402.18659v1#bib.bib6)]. One might also make the argument that video game design and video game technology might underlie much of the development of future human-computer interfaces and, for lack of a better word, the _metaverse_.

In this paper, we set out to chart the impact LLMs have had on games and games research, and the impact they are likely to have in the near- to mid-term future. We survey existing work from both academia and (mostly independent) game creators that use LLMs _with_ and _for_ games. This paper does not set out to capture modern advances in LLM technology or algorithms for training LLMs. Not only do such resources exist [[7](https://arxiv.org/html/2402.18659v1#bib.bib7)], but the breakneck speed of technical advances in this field will likely make our writeup obsolete in a year or so. Instead, we focus on work that leverages LLMs in games and propose a range of roles that the LLM can take in the broader ecosystem of games (both within the game and beyond). We lay out promising future directions for efforts to use LLMs in games, and discuss limitations (both technical and ethical) that should be addressed for a brighter future of LLM research in games.

##  2 A Note on Terminology

This paper concerns the intersection between games, including board games, video games, and other types of games people play, and large language models. But what exactly is an LLM?

Broadly speaking, an LLM is a model that is trained on text so as to be able to reproduce text in response to other text. But this definition is overly broad, as it would include Shannon’s original n𝑛nitalic_n-gram models from 1946 [[8](https://arxiv.org/html/2402.18659v1#bib.bib8)], rudimentary recurrent neural nets from the early 1990s [[9](https://arxiv.org/html/2402.18659v1#bib.bib9)], and the Tegic T9 text prediction system that would help you write text messages with your Nokia 3210. For readers who are unfamiliar with such technologies, they were ubiquitous in 90’s mobile phones.

What distinguishes LLMs from other text generative models is that they are _large_. But which model size is considered large enough? LLMs became a well-recognized term with the introduction of OpenAI’s GPT-2 network, released in 2019 [[1](https://arxiv.org/html/2402.18659v1#bib.bib1)]. GPT stands for Generative Pretrained Transformer, where transformer is a type of neural network introduced in 2017 [[10](https://arxiv.org/html/2402.18659v1#bib.bib10)]. This model became very influential because of what was perceived as a quantum leap in output quality compared to previous models. The various versions of GPT-2 have between 117 millions and 1.5 billions of parameters. Because of the association between the term LLM and the GPT-class of models, we will use the size of GPT-2 as a soft cutoff on the type of models we consider as LLMs; we are concerned with models of few hundred million parameters or more. While language models could in principle be based on various architectures, including LSTM networks, the current LLM landscape is dominated by variants of the transformer architecture; in this survey we rely primarily on this type of model architecture.

It is important to note that LLMs are by no means limited to the GPT family of models. There is by now a large variety of LLMs of varying size and capabilities, including open-source models such as Mistral [[11](https://arxiv.org/html/2402.18659v1#bib.bib11)] and the Llama [[12](https://arxiv.org/html/2402.18659v1#bib.bib12)] family, which can be fine-tuned and analyzed in various ways and run locally, and even be embedded in games’ runtimes.

One could also argue that the definition is somewhat narrow, as many modern LLMs are multimodal models, meaning that they can take as input and/or produce as output modalities other than text. In particular, many modern LLMs can process and produce images. This is often achieved through combining the core transformer network with a convolutional network for input and a diffusion model for output. Examples include GPT-4V [[13](https://arxiv.org/html/2402.18659v1#bib.bib13)] and the open-source Llava [[14](https://arxiv.org/html/2402.18659v1#bib.bib14)]. In this paper, we consider large multimodal models (LMMs) [[13](https://arxiv.org/html/2402.18659v1#bib.bib13)] as long as they retain their ability to both consume and produce text.

This survey will not concern itself with AI and machine learning techniques that are not LLMs as defined above. In particular, we will not be covering the large literature on game playing and content generation using machine learning methods [[15](https://arxiv.org/html/2402.18659v1#bib.bib15)] that is not using textual input and output. We will, however, occasionally mention some of that work when relevant, in particular to help provide historical context.

##  3 Roles of LLMs in Games

Past attempts at a typology for AI in games focused on three roles the AI can take in a game: to play a game, to design a game, or to model the (human) players [[5](https://arxiv.org/html/2402.18659v1#bib.bib5)]. LLMs are typically presented as conversational agents, which often lead the public to give them anthropomorphic qualities—such as reasoning and creativity. Therefore, we follow these trends when considering the roles an LLM can be called to play within the game or within the game development process. An LLM can operate within the game as a player (replacing a human player while imitating their goals), as a non-player character such as an enemy or interlocutor, as an assistant providing hints or handling menial tasks for a human player, as a Game Master controlling the flow of the game, or hidden within the games’ ruleset (controlling a minor or major mechanic of the game). There are however other roles an LLM can play outside of the game’s runtime, such as a designer for the game (replacing a human designer) or as an assistant to a human designer. Finally, the LLM can interface with a player or an audience in different ways, acting as a commentator of an ongoing play session (during runtime) or a reteller of past game events in some narrative form (outside runtime). Some of these roles (autonomous player, autonomous designer) are prominent in the broader AI and games research [[5](https://arxiv.org/html/2402.18659v1#bib.bib5)] and LLM research has targeted them extensively, while some of the other roles have been toyed with in exploratory research. The following sections present the roles themselves, surveying research undertaken for each role, while we identify gaps and opportunities for future research in Section [4](https://arxiv.org/html/2402.18659v1#S4 "4 A Roadmap for Future Applications of LLMs in Games ‣ Large Language Models and Games: A Survey and Roadmap").

###  3.1 Player

How can an LLM play a game? Fundamentally, language model players require some transformation from their typical output space (i.e. sequences of tokens) into the input space of the game. In addition, aspects of the game and its current state must be provided to the LLM in some form in order for it to play at any kind of reasonable level. Depending on the game itself, these mappings might be intuitive or complex. We identify three general classes of games to which LLM players are well suited: (a) games where states and actions can be compactly represented as sequences of abstract tokens, (b) games where the main input and output modalities are natural language, and (c) games for which external programs can control player actions via an API.

The first class of games mostly includes turn-based board games (e.g. Chess), since the discrete set of board positions and moves is more easily transformed into a compact representation (e.g. Portable Game Notation) than, for instance, a first-person shooter. By tokenizing sequences of moves taken from a game database, the problem of action selection can then be mapped to the standard autoregressive learning objective on which LLMs are trained – predicting the next move given the context of those that preceded it. _Chess_ [[16](https://arxiv.org/html/2402.18659v1#bib.bib16), [17](https://arxiv.org/html/2402.18659v1#bib.bib17), [18](https://arxiv.org/html/2402.18659v1#bib.bib18)], Go [[19](https://arxiv.org/html/2402.18659v1#bib.bib19)], and Othello [[20](https://arxiv.org/html/2402.18659v1#bib.bib20)] have all been used as testbeds for LLM players in this way. However, board games are not the only kind of game that can be represented as token sequences: the generalist GATO [[21](https://arxiv.org/html/2402.18659v1#bib.bib21)] agent can play a variety of Atari games at human or near-human levels by processing visual inputs as sequences of pixel values in raster order. Pixel values are interleaved with separator tokens and previous actions, allowing the model to accurately predict the appropriate game action in a dataset of human play traces. It is possible that continued improvement in Transformer models that capture both spatial and visual dynamics [[22](https://arxiv.org/html/2402.18659v1#bib.bib22), [23](https://arxiv.org/html/2402.18659v1#bib.bib23)] could allow for a similar approach to scale to even more complex games. However, such approaches require large datasets of gameplay video that may be comparatively more difficult to collect. In addition, we note that the reliance on human gameplay traces as the basis for learning may make it more difficult for an LLM player to reach super-human performance without leaps in terms of reasoning and generalization, which we revisit in the limitations of LLMs (see Section [5](https://arxiv.org/html/2402.18659v1#S5 "5 Limitations of LLMs in games ‣ Large Language Models and Games: A Survey and Roadmap")).

The second class of games most obviously includes text adventure games such as Zork (Infocom, 1977), where game states are presented as natural language descriptions and the game is already equipped with a parser to handle natural language responses. This means that LLMs can be queried for game actions in a way that still leverages their large-scale pretraining on natural language text. The earliest application of LLMs to these kinds of text games is CALM [[24](https://arxiv.org/html/2402.18659v1#bib.bib24)], a GPT-2 system finetuned on a dataset of human gameplay transcripts collected from a variety of text-based adventure games. The model is trained to predict the natural language string provided by human players given the context of previous states and actions and information about the avatar (e.g. their inventory). To actually play a game, the trained language model generates multiple candidate actions and deep reinforcement learning (RL) is used to optimize a policy that selects actions from among the candidates. At the time of its publication, this RL component was necessary because the LLM alone was not capable of generalizing well to unseen games or situations [[24](https://arxiv.org/html/2402.18659v1#bib.bib24)]. However, a more recent investigation of ChatGPT as a Zork player has indicated that LLM performance is increasing [[25](https://arxiv.org/html/2402.18659v1#bib.bib25)]. In a preliminary experiment, Yao et al. [[24](https://arxiv.org/html/2402.18659v1#bib.bib24)] show that the performance of ChatGPT can approach that of existing algorithms for text game playing, as long as a human interlocutor remains in the loop to assist the model (e.g. by reminding it of actions it has already tried). However, there is obviously much room for improvement in directly applying LLMs to text games in this way. Additionally, the ability for LLMs to play entirely novel, niche, or unseen text games (especially important given the likelihood that such systems encounter walkthroughs or playtraces of popular text games during their training) remains largely unexplored.

Beyond text adventure games, the most well-known application of LLM players in board games is CICERO [[26](https://arxiv.org/html/2402.18659v1#bib.bib26)] for playing the deal-making and subterfuge game _Diplomacy_ (Avalon Hill Games, 1976). _Diplomacy_ is well-positioned for natural language instruction, as players can freely converse in private messages in order to arrange plans, negotiate, or deceive. CICERO builds from a pre-trained LLM and is fine-tuned on a large corpus of _Diplomacy_ transcripts. Throughout the game, samples from the model are sent to other players and the various dialogue transcripts are collected to condition the potential action. CICERO is further trained to condition its outputs on specific game intents (inferred from the transcripts and added as additional context during training). In order to select an action, CICERO uses as “strategic reasoning module” that predicts the actions of other players, using a value and policy function learned from self-play. Diplomacy is an interesting game in part because the action space is split between natural language utterances and a more standard set of moves on a discrete game board, and CICERO demonstrates how an LLM can be integrated as part of a larger system for high-level play.

Finally, we consider games for which a robust API exists. This is less of a kind of game in the sense of its style or mechanics, and more a fact about its popularity or its ease of implementation. An API is an important attribute because it allows LLMs to act as players not by directly generating actions, but by producing programs that act as policies. Improvements in the code-generation abilities of LLMs have allowed them to write small programs that can produce actions given game states without further intervention from the model. For instance, the VOYAGER system [[27](https://arxiv.org/html/2402.18659v1#bib.bib27)] leverages the code-generation abilities of GPT-4 to play Minecraft (Mojang Studios, 2011) by interacting with the popular Mineflayer API. Using a sophisticated chain of prompts, VOYAGER generates blocks of code that leverage calls to the API in order to execute high-level “skills” (e.g. “Attack nearest zombie”) that are automatically converted into low-level game inputs (e.g. mouse movements and key presses). GPT-4 is also used as a high-level goal generator and planner, which in turn informs the code generation. This approach has proved very successful, with VOYAGER being the first automated system to complete a variety of in-game _Minecraft_ challenges. The results are impressive and indicate that generating action-producing programs may be a more efficient way to leverage latent LLM knowledge than direct action sampling. However, VOYAGER does benefit substantially from the availability of a robust API and vast amounts of internet discussion for its target game of _Minecraft_. As with the analysis of ChatGPT on _Zork_ , the ability of this approach to generalize to less popular or entirely unseen games remains to be seen.

###  3.2 Non-Player Characters

![Refer to caption](extracted/5438418/graphics/aipeople_2.png) Figure 1: Screenshot of the promotional video for AI people, where a player can interact via their avatar’s chat with other NPCs and watch how their text has consequences between NPC relationships, in this case. Used with permission from [[28](https://arxiv.org/html/2402.18659v1#bib.bib28)].

Non-player characters (NPCs) are agents which exist in virtual game worlds but whose actions are not directly controlled by the players. NPCs exist to enrich the player’s experience and deepen immersion by adding to the world’s ambiance and making it more believable [[29](https://arxiv.org/html/2402.18659v1#bib.bib29)]. NPCs may serve as pets, allies, enemies111We use enemies and allies here for in-game agents with different skills and ways to affect the world than e.g. gameplaying AI such as opponents in _Chess_., merchants, quest givers, or bystanders. Therefore, they have different agency even from AI-controlled players, and their goal is never to win. This makes designing AI for NPCs interesting [[5](https://arxiv.org/html/2402.18659v1#bib.bib5)], while LLMs can provide unique advantages in this task. Their ability to “understand” game-world settings allows LLMs to adapt their responses to match these settings. It has been shown that LLMs are able to role-play through different scenarios [[30](https://arxiv.org/html/2402.18659v1#bib.bib30)], thereby highlighting their potential to provide a more flexible and apt tool to emulate human behavior. We identify two ways in which LLMs can control NPCs: (a) through their _dialogue_ , and (b) through their _behavior_. Behavior relates to in-game action selection, discussed in Section [3.1](https://arxiv.org/html/2402.18659v1#S3.SS1 "3.1 Player ‣ 3 Roles of LLMs in Games ‣ Large Language Models and Games: A Survey and Roadmap"); however, we note that the heuristics and goals of such behavior is different than an AI player trying to win the game.

LLMs are naturally suited for natural language conversation, and as NPC dialogue systems they can generate dynamic and contextually appropriate responses based on player input. This makes interactions with NPCs more engaging and realistic, reduces repetitive discourse and provides a more explorative experience within the game [[31](https://arxiv.org/html/2402.18659v1#bib.bib31)]. LLMs can engage the players in the gameworld’s narrative as _foreground_ NPCs, _background_ NPCs, or _narrator_ NPCs. We discuss narrator LLMs as commentators in Section [3.4](https://arxiv.org/html/2402.18659v1#S3.SS4 "3.4 Commentator/Reteller ‣ 3 Roles of LLMs in Games ‣ Large Language Models and Games: A Survey and Roadmap") whereas we cover the other two NPC types here. Foreground NPCs form part of the overarching narrative of the game, or one of its sub-narratives. They may be enemies, allies, information-givers, quest-givers, or item-providers. Their dialogue is heavily constrained by the scope of the narrative, their role within it, and the player actions. Foreground NPCs’ text generation process via LLMs must consider the overall context of the game and the interaction with the player and keep track of events transpiring in the playthrough. This raises concerns regarding the memory capacity of LLMs, as well as the impact of possible hallucinations, i.e. plausible but false statements [[32](https://arxiv.org/html/2402.18659v1#bib.bib32)]; we revisit these limitations in Section [5](https://arxiv.org/html/2402.18659v1#S5 "5 Limitations of LLMs in games ‣ Large Language Models and Games: A Survey and Roadmap").

Background NPCs populate the majority of the virtual world but are not part of any specific task the player is carrying out. Their purpose is to make the environment more believable and act independently of the players [[31](https://arxiv.org/html/2402.18659v1#bib.bib31)]. Since such NPCs’ presence is purely decorative and their dialogue is essentially small talk, their dialogue generation is less constrained, perhaps bound only by the identity of the speaker and their background. That said, their believability hinges on their ability to maintain the illusion that they have their own agency in the world and can interact with it [[33](https://arxiv.org/html/2402.18659v1#bib.bib33)].

Studies have shown that multiple agents are able to follow game rules and engage in game playing [[34](https://arxiv.org/html/2402.18659v1#bib.bib34), [35](https://arxiv.org/html/2402.18659v1#bib.bib35)], with different models consistently exhibiting their own aptitudes and weaknesses when applied to specific roles. This ability to interact within constraints is useful to instill believable behaviors in foreground and background NPCs, grounding their actions and dialogue within the rules of the game environment. Other work has focused more on the conversational and story-writing abilities of LLMs, such as the creation of dialogue between multiple characters, each having their unique personality, whilst following a consistent plot. One such example is the use of LLMs to generate a _South Park_ (Comedy Central, 1997) episode [[36](https://arxiv.org/html/2402.18659v1#bib.bib36)] with multiple characters within a well-known setting. There are limitations to this approach, primarily that LLMs perform something like a theatrical improvisation, rather than acting as an actor studying a part [[30](https://arxiv.org/html/2402.18659v1#bib.bib30)]. Through this unconstrained process, the LLM is prone to hallucinations which do not fit the desired scenario. This volatility can be mitigated by providing the LLM not only with the conversation history but also with the current state of the environment, such as the items within it and their affordances, as well as the other characters and their corresponding actions. This method has been successfully employed to create conversational NPCs in text-based games [[37](https://arxiv.org/html/2402.18659v1#bib.bib37)], but may also be extended to other scenarios or to cover the use of LLMs as active or interactive narrators.

###  3.3 Player Assistant

A somewhat less explored role for LLMs in games is that of a player assistant: an interactive agent intended to enrich or guide the player experience in some way. This could be in the form of a sequence of tutorial-style tips, a character that does not causally interact with the game world at all, or an agent that is able to interact with the game world at a similar level as the player. Existing games make use of player assistants in different ways. For example, in _The Sims_ (Electronic Arts, 2000) a disembodied assistant provides tips specific to the game context via dialogue boxes. _Civilization VI_ (Firaxis Games, 2016) uses different assistants with a visual depiction to provide similar advice suggesting the best build option according to their idiosyncratic heuristic; they may thus alleviate some decision-making from the player. In management games, AI may automate menial tasks such as assigning jobs to a planet’s population in _Stellaris_ (Paradox Interactive, 2016); this assistance reduces cognitive load from the player, but the player can always micro-manage this task if they wish.

LLMs will likely be appealing as player assistants given their potential expressive and conversational capacities. Coupled with either LLM-powered or heuristic-based methods for finding the best policy or action given the current game context (see Section [3.1](https://arxiv.org/html/2402.18659v1#S3.SS1 "3.1 Player ‣ 3 Roles of LLMs in Games ‣ Large Language Models and Games: A Survey and Roadmap")), an LLM-powered player assistant can construct the suggested action and the explanation as a natural language utterance delivered by a disembodied or embodied agent, expressed with a corresponding sentiment, and manifested through its body stance, gestures and facial expression. Similarly, LLMs may assist the player by undertaking some minor tasks in the game via a tailored smaller role as “player” within that smaller task description (see Section [3.1](https://arxiv.org/html/2402.18659v1#S3.SS1 "3.1 Player ‣ 3 Roles of LLMs in Games ‣ Large Language Models and Games: A Survey and Roadmap")). The potential of LLM-powered player assistants is not explored in current research, to the best of our knowledge. We highlight the potential of this future application in Section [4](https://arxiv.org/html/2402.18659v1#S4 "4 A Roadmap for Future Applications of LLMs in Games ‣ Large Language Models and Games: A Survey and Roadmap").

###  3.4 Commentator/Reteller

We view LLMs to be ideally suited as commentators or retellers. Here, we identify these roles as an agent that produces and narrates a sequence of events, for the benefit of either human players or spectators. Such an agent may consider only in-game events and in-game context, acting as an in-game entity such as a sports commentator in _FIFA_ (EA Sports, 1993), or also consider out-of-game events and context such as the player (their actions, strategies, motivations, etc.). The reteller [[38](https://arxiv.org/html/2402.18659v1#bib.bib38)] exclusively narrates past events—often grouped into a concise “chunk” such as a game session (i.e. based on out-of-game context) or a quest (i.e. based only on in-game context). The commentator may be narrating current, ongoing events which have not been concluded, similar to a streamer concurrently discussing their current actions (including out-of-game context) or a sportscaster in an in-progress sports game such as _FIFA_.

The vision of automated “let’s play”-style commentary generation is not new. It was proposed in [[39](https://arxiv.org/html/2402.18659v1#bib.bib39)] and implemented via classical machine learning methods, to limited success. In [[40](https://arxiv.org/html/2402.18659v1#bib.bib40)], an LSTM with text, vision and game-state input was trained to generate characters for a commentary script in a racing game. Results of this approach featured repetitive and context-irrelevant generated text. LSTMs were also used by [[41](https://arxiv.org/html/2402.18659v1#bib.bib41)] to generate text, at a character level, for _Getting Over It With Bennett Foddy_ (Bennett Foddy, 2017), a challenging side-scrolling climbing game.

LLMs for commentary are also explored by Renella and Eger [[42](https://arxiv.org/html/2402.18659v1#bib.bib42)], who argue that LLMs could assist games streamers (e.g. on Twitch) while the streamer multitasks gameplay with audience interaction. The authors develop a pipeline for automatically commenting upon _League of Legends_ (Riot Games, 2009) games. They take a multi-phased approach, training a model on hand-annotated data to recognize key events, then prompting ChatGPT to generate zero-shot commentary on these events in the style of a particular (known) fictional character, and finally sending the generated text through the FakeYou222<https://fakeyou.com/> API to be voiced in the timbre of this same character. For example, once the event detection model has identified an enemy double kill in a particular frame, ChatGPT responds in the style of Rick Sanchez, from _Rick and Morty_ (Cartoon Network, 2013): “What the heck?! That enemy team just got a double kill! I can’t believe it! They must be pretty good! I better watch out for them!” An additional loop buffers detected events—for example, delaying commentary on a double kill in case it should escalate into a triple kill, or prioritizing among a quick barrage of events—and prompts ChatGPT to generate random fillers, such as thanking (fictional) new subscribers. 

Despite the existence of the aforementioned studies, research on LLMs as game commentators remains rather limited. The appeal is obvious: simulation-based games of emergent narrative already generate rich narrative histories, and are remixed by human players to produce secondary content that is often popular in its own right. In principle, LLMs could be used to generate more succinct retellings or highlight reels of these game events. Prompting current LLMs for stories, without any further specification of style or substance, tends to produce output that feels generic. Past events recorded in simulation games could ultimately provide specificity and narrative coherence to these outputs. Exploring more concepts beyond automating streamer commentary, such as assisting streamers via LLM commentary of the audiences’ reactions rather than the in-game actions, remains unexplored. We revisit this along other future applications in Section [4](https://arxiv.org/html/2402.18659v1#S4 "4 A Roadmap for Future Applications of LLMs in Games ‣ Large Language Models and Games: A Survey and Roadmap").

###  3.5 Game Master

A Game Master (GM) in table-top role-playing games (TTRPGs) is the person who creates the plot of a game, its characters, and narrative. GMs wear many hats during the course of the game session [[43](https://arxiv.org/html/2402.18659v1#bib.bib43)], but also before and after the session by preparing and adapting adventures and stories and checking in with the group [[44](https://arxiv.org/html/2402.18659v1#bib.bib44)]. Digital games have mostly prescripted stories or level progressions, and their players have a restricted range of affordances, compared to TTRPG players whose actions are only limited by their imagination. Similarly, the story told around the table can take any direction. Since human GMs mostly communicate about the gameworld, story, game state and action resolutions via natural language (although props such as maps, miniatures, hand-outs are also common), the potential of LLMs as a GM is often mentioned both in research circles and TTRPG discussion boards [[45](https://arxiv.org/html/2402.18659v1#bib.bib45)]. LLMs as GMs also open the potential for solo play, while a TTRPG requires at least one player and the human GM.

One of the first notable text-based adventures managed by a fine-tuned version of GPT-2 is AI Dungeon [[46](https://arxiv.org/html/2402.18659v1#bib.bib46)]. AI Dungeon333<https://play.aidungeon.com/> is an online interactive chat-based storytelling application where the player takes actions through semantic input alone. The LLM continues the story based on the player’s input, in the fashion of a human GM. The game has evolved since its creation to make use of more recent LLM models, which the player can choose from before starting a play session. Different game world settings are also offered, and players are also able to share the stories they create. Similar games have emerged online ever since444<https://koboldai.net>, <https://www.hiddendoor.co>, and a freely available code repository, _Kobold AI Client_ 555<https://github.com/KoboldAI/KoboldAI-Client>, allows a local or remote installation of a client for such LLM-run games. Some of these games also use Stable Diffusion text-to-image models [[47](https://arxiv.org/html/2402.18659v1#bib.bib47)] to generate visuals accompanying different parts of the narrative.

In lieu of replacing a human GM, LLMs have also been employed as GM assistants. CALYPSO [[48](https://arxiv.org/html/2402.18659v1#bib.bib48)] is a set of tools running on a Discord server which the GM can query either to generate random encounters, brainstorm ideas, or alternatively chat with a fictional character in a _Dungeons & Dragons_ (TSR, 1974) TTRPG setting. CALYPSO highlights that hallucinations of GPT-3 can have both positive effects when it generates plausible details not included in descriptions published in the original game manual (e.g. the shapes of creatures’ eyes) and also negative effects when the created details are outright incorrect (e.g. describing the wings of a canonically wingless creature). In addition, the model’s preconditioning to avoid racial bias was found to occasionally prevent it from generating racial details of fantasy creatures in the game. Other work used smaller GPT models to improvise in-game conversations [[49](https://arxiv.org/html/2402.18659v1#bib.bib49)] by monitoring and transcribing verbal exchanges between the GM and the players, and attempting to generate appropriate responses. This example was integrated into “Shoelace” [[50](https://arxiv.org/html/2402.18659v1#bib.bib50)], which is itself a GM assisting tool helping with content lookup by creating a node-based plan of the game narrative and encounters. The versatility of LLMs given their ability to rapidly process text input paves the way for their integration into the multitude of existing tools and aids for human GMs.

![Refer to caption](extracted/5438418/graphics/1001nights_steam_screenshot.jpg) Figure 2: In 1001 Nights [[51](https://arxiv.org/html/2402.18659v1#bib.bib51)], the player uses free-form text to trick the king (role-played by an LLM) into uttering the name of a particular weapon (which will then materialize, allowing the player to defeat him). Image used with permission.

###  3.6 Game Mechanic

Games can also be built around a specific mechanic that relies on LLMs, similar to the AI-based game design patterns identified in [[52](https://arxiv.org/html/2402.18659v1#bib.bib52)]. An obvious mechanic revolves around the social interactions facilitated by LLM-powered conversational NPCs. In this vein, [[53](https://arxiv.org/html/2402.18659v1#bib.bib53)] has employed LLMs to populate a virtual village with 25 characters, enabling them to communicate and engage in social behavior within a sandbox environment. Players were able to interact with these agents using a text-based interface. The environment state and actions of each agent were stored in a language-based format and summarized in order to retain knowledge for each agent when prompting for its actions. This led to emerging believable social interactions, such as the agents spontaneously inviting other agents to a party which one of them was organizing. Similarly, _GoodAI_ are developing the AI people video game which operates as a sandbox simulation where LLM-powered NPCs “interact with each other and their environment, forming relationships and displaying emotions” [[28](https://arxiv.org/html/2402.18659v1#bib.bib28)]. The player can interact with the agents via natural language chat, triggering reactions and potentially disrupting the relationship between NPCs (see Fig. [1](https://arxiv.org/html/2402.18659v1#S3.F1 "Figure 1 ‣ 3.2 Non-Player Characters ‣ 3 Roles of LLMs in Games ‣ Large Language Models and Games: A Survey and Roadmap")).

Natural language interactions form a natural pool of mechanics to build games around, such as gamifying users’ attempts at jailbreaking LLMs [[54](https://arxiv.org/html/2402.18659v1#bib.bib54)]. The game 1001 nights, depicted in [Figure 2](https://arxiv.org/html/2402.18659v1#S3.F2 "Figure 2 ‣ 3.5 Game Master ‣ 3 Roles of LLMs in Games ‣ Large Language Models and Games: A Survey and Roadmap"), exemplifies this by having an LLM co-create a story from human prompts, where the player’s objective is to try and steer the story to include specific keywords in order for the main character, Scheherazade, to turn these into tangible items in aid of her escape [[51](https://arxiv.org/html/2402.18659v1#bib.bib51)]. Similarly, _Gandalf_ 666<https://gandalf.lakera.ai/> challenges the player to trick an LLM into revealing a password. The game increases the difficulty of the task as levels progress by adjusting the prompt specifications, such as forcing the LLM to re-examine its generated response to ensure it does not include the password.

![Refer to caption](extracted/5438418/graphics/infinite-craft.jpg) Figure 3: In _Infinite Craft_ , the player combines what begins as a simple set of atomic elements into increasingly complex entities, with an LLM dictating the product resulting from arbitrary combinations. Image used with permission.

Another strength of LLMs is language synthesis, which is leveraged by _Infinite Craft_ 777<https://neal.fun/infinite-craft/>. Infinite Craft is an “alchemy” game, in which the player combines elements to produce new ones (see Fig. [3](https://arxiv.org/html/2402.18659v1#S3.F3 "Figure 3 ‣ 3.6 Game Mechanic ‣ 3 Roles of LLMs in Games ‣ Large Language Models and Games: A Survey and Roadmap")). In _Infinite Craft_ , the player begins with a set of core elements (water, fire, wind and earth). But while the former have a set of interactions defined manually by the designer, _Infinite Craft_ prompts LLaMA 2 [[12](https://arxiv.org/html/2402.18659v1#bib.bib12)] to imagine the product of the combination of these elements [[55](https://arxiv.org/html/2402.18659v1#bib.bib55)]. Judging from gameplay, it appears that for each distinct combination, LLaMA is prompted to produce the result only once, with the product stored in a database for future reference. Thus seemingly anything in the language model’s vocabulary might “emerge” from the combination of these elements, including all 50 states888<https://x.com/FeralFlex/status/1758332430136615298>, “Dream”999<https://x.com/slutzsmp/status/1760394123243135169>, and the fictional “Super Stonedosaurus Tacosaurus Rex”101010<https://x.com/pvtspicy/status/1759316982984237139>. On occasion, the model can choose to return one of the combined elements, or refuse to combine (in particular very lengthy or complex) elements.

###  3.7 Automated Designer

A key role of AI in games [[5](https://arxiv.org/html/2402.18659v1#bib.bib5)] is the algorithmic generation of game content such as levels and visuals, or even entire games. Unlike a Game Master who creates a game via natural language—meant to exist in the “theater of the mind” of the players—the aim of procedural content generation (PCG) is to create content intended for use in a digital game and thus it is required to satisfy certain constraints such as playability and aesthetic quality.

Any PCG method that is trained on available content corpora fits under the Procedural Content Generation via Machine Learning (PCGML) paradigm [[15](https://arxiv.org/html/2402.18659v1#bib.bib15)]. Strictly speaking the original PCGML framework of 2018 did not consider LLMs; instead it relied on machine learning methods such as autoencoders and LSTMs. However, important challenges of PCGML remain when considering LLMs for PCG: notably, the reliance on high-quality, machine-readable datasets from human-authored levels. While some datasets exist for arcade game levels [[56](https://arxiv.org/html/2402.18659v1#bib.bib56)], for most games the content remains both unavailable and protected by intellectual property (IP) laws. We revisit this issue in Section [6](https://arxiv.org/html/2402.18659v1#S6 "6 Ethical Issues with LLMs in games ‣ Large Language Models and Games: A Survey and Roadmap").

Prior work in PCG has demonstrated that tile-based game levels can be reliably generated with sequence-based prediction models (e.g. LSTMs) from a modest set of examples, by treating such levels as linear sequences of tile types in raster order [[57](https://arxiv.org/html/2402.18659v1#bib.bib57)]. More recently, modern LLMs have been leveraged in a similar way to generally greater success. In [[58](https://arxiv.org/html/2402.18659v1#bib.bib58)], a GPT-2 model was fine-tuned on a large dataset of _Sokoban_ (Thinking Rabbit, 1982) levels and, at test time, samples from the model to produce novel puzzles (see Figure [4](https://arxiv.org/html/2402.18659v1#S3.F4 "Figure 4 ‣ 3.7 Automated Designer ‣ 3 Roles of LLMs in Games ‣ Large Language Models and Games: A Survey and Roadmap")). Interestingly, their results indicate that while the GPT-2 model struggles when the fine-tuning dataset is restricted in size, GPT-3 (and, presumably, larger models released since then) are better able to accommodate limited training sets.

![Refer to caption](extracted/5438418/graphics/gpt3_sokoban.png) Figure 4: A level for the puzzle game Sokoban generated by GPT-3, visualized with the Griddly tileset [[59](https://arxiv.org/html/2402.18659v1#bib.bib59)]. Image used with permission.

A similar approach, MarioGPT, trains a GPT-2 model on a relatively small dataset of Super Mario Bros (Nintendo, 1985) levels [[60](https://arxiv.org/html/2402.18659v1#bib.bib60)]. MarioGPT overcomes the issue of data sparsity by using the initial dataset as the starting point for an evolutionary algorithm. Existing levels are selected and then sections of the level are mutated by sampling from the GPT-model and then correcting the border between the re-generated section and the rest of the level with a similarly-trained BERT (i.e. bi-directional) model [[61](https://arxiv.org/html/2402.18659v1#bib.bib61)]. This approach produces a large and diverse set of playable levels, despite starting from less than 20 levels.

Both aforementioned GPT-based level generation approaches also show the promise of incorporating natural language instructions to produce conditional level generators, either by prefixing game levels in the training dataset with desired level characteristics [[58](https://arxiv.org/html/2402.18659v1#bib.bib58)] or by embedding user instructions and allowing the model to attend to the embedding during generation [[60](https://arxiv.org/html/2402.18659v1#bib.bib60)]. It seems likely that more sophisticated techniques such as reinforcement learning with human feedback [[2](https://arxiv.org/html/2402.18659v1#bib.bib2)] could produce level generators that are even more capable of accommodating user guidance.

###  3.8 Design Assistant

An AI for design assistance can provide several benefits to the creative process. Depending on the type of tool, type of AI, and type of creative process, the AI can minimize development time and cost, reduce human effort, support collaboration among members of a design team or elicit a user’s creativity [[62](https://arxiv.org/html/2402.18659v1#bib.bib62)]. So far, in games, most of the AI-powered design assistant tools focus on autocompleting a human’s in-progress design [[63](https://arxiv.org/html/2402.18659v1#bib.bib63)] or providing many possible suggestions for the designer to consider [[64](https://arxiv.org/html/2402.18659v1#bib.bib64), [65](https://arxiv.org/html/2402.18659v1#bib.bib65), [66](https://arxiv.org/html/2402.18659v1#bib.bib66), [67](https://arxiv.org/html/2402.18659v1#bib.bib67)]. Based on the level of control the AI co-creator [[68](https://arxiv.org/html/2402.18659v1#bib.bib68)] has on the process, we identify three levels of assistance:

  * •

Conceptual assistance where the AI provides high-level guidance which is not game-ready, requiring that the designer adapts and curates the AI output in a way that fits their own vision and the constraints of the game.

  * •

Procedural assistance where the AI is part of the creative process and through interaction with the user can produce increasingly more final versions of the intended artifact. Here, AI is expected to understand the context of the game for which the content is intended, in order to provide meaningful assistance. However, the AI does not need to produce a final, playable artifact but could instead simply provide the next creative step for discussion with the designer [[69](https://arxiv.org/html/2402.18659v1#bib.bib69)]. Moreover, the designer is ultimately responsible for curating and adapting the generated content, as well as deciding when the co-creative process is completed [[70](https://arxiv.org/html/2402.18659v1#bib.bib70)].

  * •

Production assistance where the AI must create the final artifact based on user requirements. This is the closest level to PCG (see Section [3.7](https://arxiv.org/html/2402.18659v1#S3.SS7 "3.7 Automated Designer ‣ 3 Roles of LLMs in Games ‣ Large Language Models and Games: A Survey and Roadmap")), but is different in that the designer remains in control and can refine their specification or reject a created artifact (versus an autonomous generator which directly sends content to the player). As expected, however, the AI operates in a much more constrained space in this scenario as it must account for all other game mechanics (the design of which are presumed finalized) and designer goals which are somehow encoded or presumed via learned designer models [[71](https://arxiv.org/html/2402.18659v1#bib.bib71)].




One can argue that existing interfaces with LLMs, Foundation Models and Large Multimodal Models act as design assistants. The designer provides their specifications and receives one (in LLMs) or multiple (in AI image generators) suggestions that they can refine further. Many creatives report using such interfaces for brainstorming and concept development [[72](https://arxiv.org/html/2402.18659v1#bib.bib72)], including game developers [[73](https://arxiv.org/html/2402.18659v1#bib.bib73)]. However, the applicability of LLMs as design assistants is somewhat limited, reverting only to conceptual assistance. Similarly, their potential for refining an existing idea (i.e. offering procedural assistance) is underexplored, as we discuss in Section [4](https://arxiv.org/html/2402.18659v1#S4 "4 A Roadmap for Future Applications of LLMs in Games ‣ Large Language Models and Games: A Survey and Roadmap").

Conceptual assistance is thus the easiest for LLMs, and is the first case explored in games. In [[74](https://arxiv.org/html/2402.18659v1#bib.bib74)], design assistance is envisioned in a tool that combines the game description provided by the user with existing knowledge of similar games to suggest possible game features to a designer. The features suggested are fairly generic, few-word guidelines (e.g.: “learn new combat”) which would need extensive design effort and creativity to transform into an implementable and coherent game design.

Since production assistance is also close to traditional PCG pipelines, it is also understandably explored for games. In [[75](https://arxiv.org/html/2402.18659v1#bib.bib75)], GPT-3 generates levels from a prompt that describes the level’s features (e.g. width and height) while a human curates and edits the results to ensure playability. This curated set of levels is then used for further fine-tuning, potentially automating the generative process. Instead of generating the final artifact directly, in [[76](https://arxiv.org/html/2402.18659v1#bib.bib76)] the LLM is employed as a parser that extrapolates from user requests, expressed in natural language, the high level parameters (level difficulty and size, type of sustainability problem, and simulation goal metrics) required by the PCG system to generate the final game. A collection of candidate game levels are generated from a single user request. Each candidate is then evaluated by an RL agent, and the best game is presented to the user via the Unity game engine.

As noted above, so far research has focused on either LLMs for conceptual assistance (putting significant onus on a human designer) or as production assistance (leveraging a human designer as curator). The conversational nature of LLMs, however, seems particularly well-suited for procedural assistance when designing content; we revisit this missed opportunity in Section [4](https://arxiv.org/html/2402.18659v1#S4 "4 A Roadmap for Future Applications of LLMs in Games ‣ Large Language Models and Games: A Survey and Roadmap").

##  4 A Roadmap for Future Applications of LLMs in Games

The previous section attempted to group current research in LLMs and games into a typology focused on the roles an LLM is asked to play. As part of this exercise, we identified a number of roles that have been heavily researched. Unsurprisingly, the role of player and automated designer have received attention: this matches the general trends within AI and Games research more broadly [[5](https://arxiv.org/html/2402.18659v1#bib.bib5)]. Following general trends in Game AI for playing or generating content, LLM-based approaches are likely to flourish via community events, benchmarks and competitions, with first steps already being taken in this direction [[77](https://arxiv.org/html/2402.18659v1#bib.bib77)]. Based on the roles listed in Section [3](https://arxiv.org/html/2402.18659v1#S3 "3 Roles of LLMs in Games ‣ Large Language Models and Games: A Survey and Roadmap"), we identify below some gaps found in the literature, and lay out possible research directions that leverage the power of LLMs in new ways. 

While academic interest in design assistance within games has blossomed in the last decade, we find that the potential of LLMs has so far been underutilized. LLM design assistants either ask too much of a human designer in terms of creative interpretation and actual development [[74](https://arxiv.org/html/2402.18659v1#bib.bib74)] or too little, demoting them to content curator [[75](https://arxiv.org/html/2402.18659v1#bib.bib75)]. Past research in mixed-initiative systems [[68](https://arxiv.org/html/2402.18659v1#bib.bib68)] assumes a more co-creative initiative from both human and machine, and the power of LLMs as conversational agents matches the original vision of a creative dialogue between initiatives [[70](https://arxiv.org/html/2402.18659v1#bib.bib70)]. Therefore, a promising unexplored direction lies in a more procedural assistance (see Section [3.8](https://arxiv.org/html/2402.18659v1#S3.SS8 "3.8 Design Assistant ‣ 3 Roles of LLMs in Games ‣ Large Language Models and Games: A Survey and Roadmap")) where the LLM not only produces output but also reasons about it to the human designer. LLMs seem especially well-suited for this task, as the context is retained and the designer can iteratively refine past products that the LLM has generated. However, concerns of LLMs’ limited memory may arise (see Section [5](https://arxiv.org/html/2402.18659v1#S5 "5 Limitations of LLMs in games ‣ Large Language Models and Games: A Survey and Roadmap")) in long-term design process. On the other hand, iterative refining is not as straightforward for other state-of-the-art technology such as LMMs, despite some promising results via e.g. InstructPix2Pix [[78](https://arxiv.org/html/2402.18659v1#bib.bib78)]. It is expected that such applications will raise new challenges in terms of hallucinations, explainability [[79](https://arxiv.org/html/2402.18659v1#bib.bib79)], capturing or modeling designer intent [[71](https://arxiv.org/html/2402.18659v1#bib.bib71)], and more. We discuss such challenges further in Section [6](https://arxiv.org/html/2402.18659v1#S6 "6 Ethical Issues with LLMs in games ‣ Large Language Models and Games: A Survey and Roadmap").

While we identified player assistance as an important role that LLMs can play, we have not yet found any work that targets any aspect of this. The conversational ability of LLMs make them ideally suited for tutorial writing or hint-giving, especially in short snippets as provided e.g. by a conversational agent. However, it is important to note that LLMs often hallucinate or overfit to the corpus they have been trained on, and may be challenged, for instance, to summarize or lookup specific rules given a game manual. Similar limitations were identified when using an LLM as assistant to a human Game Master [[49](https://arxiv.org/html/2402.18659v1#bib.bib49)], where the LLM could not find elements in the pre-written adventure to highlight when asked a question about the scene. Other technologies (as simple as a database search query) could be used instead, with the LLM undertaking only the task of converting the found information into a natural language utterance. Beyond mere hint-giving, however, an LLM could also act as a more hands-on player assistant, taking over more trivial tasks (such as managing minutiae of one city in a strategy game). This is also powerful for Game Master assistance, as the LLM can keep track of locations visited and NPCs met, or looking up rules. In both cases, addressing the issue of hallucinations and consistency will need to be addressed, which we review in Section [5](https://arxiv.org/html/2402.18659v1#S5 "5 Limitations of LLMs in games ‣ Large Language Models and Games: A Survey and Roadmap").

Another role seemingly well-suited for LLMs that has received limited attention is that of commentator or reteller. Work so far has focused on automating the commentary of streamers or eSport casters [[42](https://arxiv.org/html/2402.18659v1#bib.bib42)]. While this direction is still largely uncharted, there are more directions that could leverage LLMs for streamer assistance rather than automation (and replacement). Rather than narrate events occurring within the game (or video stream), LLMs can summarize the audience interactions and engagement levels—thus acting as a commentator not of the game but of the audience watching it. This could allow a human streamer to better keep track of topics discussed in the chat, and engage as needed without having to read every comment. While this has been identified as a research direction for AI already [[80](https://arxiv.org/html/2402.18659v1#bib.bib80)], it has not yet been implemented. Under the role of streamer assistance, issues of explainability of the LLM’s commentary would become pertinent (e.g. to address one audience member by name); we revisit this in Section [5](https://arxiv.org/html/2402.18659v1#S5 "5 Limitations of LLMs in games ‣ Large Language Models and Games: A Survey and Roadmap").

Finally, it is worth noting that one pillar of AI in games research is missing altogether from LLM research and applications surveyed: that of modeling players [[5](https://arxiv.org/html/2402.18659v1#bib.bib5)]. LLMs appear to offer promising, yet admittedly unexplored, methods for affective game computing [[81](https://arxiv.org/html/2402.18659v1#bib.bib81)]. At first glance, this is not particularly surprising: while player modeling often relies on supervised machine learning algorithms, it is not as clear how this can be achieved via text input or output. In principle, an LLM could predict affective state transitions such as “the game is more engaging now” and thereby adapt the game environment to elicit a supposedly more engaging experience for the player. Learning such transitions builds on the experience-driven procedural content generation paradigm [[82](https://arxiv.org/html/2402.18659v1#bib.bib82)] but with an LLM acting as the player experience model. We therefore envision the fine tuning of LLMs so that they learn to represent and infer player experience transitions based on in-game observations and demonstrations of experience. As highlighted in Section [5](https://arxiv.org/html/2402.18659v1#S5 "5 Limitations of LLMs in games ‣ Large Language Models and Games: A Survey and Roadmap"), however, current LLMs struggle to capture user intent during conversation—let alone more ill-defined concepts such as players’ emotion or engagement [[83](https://arxiv.org/html/2402.18659v1#bib.bib83)]. Current datasets on affect in games are formatted as continuous or categorical variables, often fluctuating over time [[84](https://arxiv.org/html/2402.18659v1#bib.bib84)], which would be challenging to format as text without processing. While perhaps using language as input or output for the player model requires some innovative pre-processing or more advanced LLM technologies, the underlying transformer architecture and attention-based algorithms show promise already. We expect more research on player modeling powered by transformers, such as leveraging behavior transformers [[85](https://arxiv.org/html/2402.18659v1#bib.bib85)] to imitate human playtraces grouped by playstyle [[86](https://arxiv.org/html/2402.18659v1#bib.bib86)].

To wrap up, we believe that every role an LLM could be called to play in (or around) a game identified in Section [3](https://arxiv.org/html/2402.18659v1#S3 "3 Roles of LLMs in Games ‣ Large Language Models and Games: A Survey and Roadmap") could benefit from additional attention. This technology remains nascent, and changes are forthcoming which may address several limitations we identified both in the above paragraphs and, more extensively, in Section [5](https://arxiv.org/html/2402.18659v1#S5 "5 Limitations of LLMs in games ‣ Large Language Models and Games: A Survey and Roadmap"). The natural language capabilities (especially for text generation) make LLMs ideal conversational assistants (for a player, a designer, a game master, or a streamer). The ability of LLMs to consume and reason from text corpora also opens new possibilities for automated design moving beyond tile-based level generation (which needs carefully crafted corpora) and more towards open-ended content such as game narratives [[87](https://arxiv.org/html/2402.18659v1#bib.bib87), [88](https://arxiv.org/html/2402.18659v1#bib.bib88), [89](https://arxiv.org/html/2402.18659v1#bib.bib89), [90](https://arxiv.org/html/2402.18659v1#bib.bib90)] or even game design documents. The potential of LLMs in that regard is already voiced by many evangelists in the field, but research on actual implementation of such ideas and on addressing the IP concerns they may raise (see Section [6](https://arxiv.org/html/2402.18659v1#S6 "6 Ethical Issues with LLMs in games ‣ Large Language Models and Games: A Survey and Roadmap")) are still forthcoming.

##  5 Limitations of LLMs in games

Large language models have exciting potential for video games, but they also come with inherent limitations. Mainly, LLMs suffer from hallucinations [[32](https://arxiv.org/html/2402.18659v1#bib.bib32), [91](https://arxiv.org/html/2402.18659v1#bib.bib91)], meaning that they will output plausible but false statements simply because they are a probable sequence of words. Hallucinations are inevitable given how the world is described to the machine [[92](https://arxiv.org/html/2402.18659v1#bib.bib92)]; LLMs lack grounding, so the text they generate is detached from constraints of reality. Yet LLMs always “act” confidently in their responses, even when wholly mistaken. Beyond hallucinations, LLMs suffer from factual errors [[93](https://arxiv.org/html/2402.18659v1#bib.bib93), [94](https://arxiv.org/html/2402.18659v1#bib.bib94), [95](https://arxiv.org/html/2402.18659v1#bib.bib95)], outputting responses that are wrong even though the LLM has access to information that proves otherwise. In the context of video games, these limitations affect certain applications of LLMs more than others, for example NPCs may hallucinate quests that do not exist in the game, or a player assistant may provide suggestions to the user based on wrong assumptions.

Another limitation of using LLMs in video games is that LLMs sometimes struggle to capture user intent. This is especially evident with expressions of sarcasm [[96](https://arxiv.org/html/2402.18659v1#bib.bib96)]. The ability to capture user intent is important for applications of LLMs that converse directly with the player. An LLM that correctly understands user intent then also correctly understands the context of the conversation, which is not always the case for current LLMs. Many LLMs make errors in correctly understanding user requests [[97](https://arxiv.org/html/2402.18659v1#bib.bib97)], and clarifying to the LLM multiple times results in a frustrating experience for the user. This limitation is most relevant to cases where the LLM is in direct conversation with the user, e.g. as design assistant, player assistant or game master. Depending on how much the LLM output controls the user experience (e.g. as game master or offering production assistance to a human designer), the inability to capture user intent can be a source of frustration.

On a larger scale, LLMs suffer from losing context and struggling with continuity. This is because the “memory” of an LLM is constrained by its context size, which limits the extent of its inputs and outputs, as well as its response time due to the attention mechanism [[10](https://arxiv.org/html/2402.18659v1#bib.bib10)]. The longer the conversation, the less likely it is that the LLM will recall early events [[98](https://arxiv.org/html/2402.18659v1#bib.bib98)]. In video games, it is possible to separately summarize the game events (see [Section 3.4](https://arxiv.org/html/2402.18659v1#S3.SS4 "3.4 Commentator/Reteller ‣ 3 Roles of LLMs in Games ‣ Large Language Models and Games: A Survey and Roadmap")) and process them as part of the input to the LLM. As a game progresses past a few game sessions, however, this summary may still be too long or details of increasing significance will be omitted, leading to a degraded performance. This is especially relevant for roles requiring a long-term engagement, such as LLM-powered retellers or game masters. In _Infinite Craft_ (see Section [3.6](https://arxiv.org/html/2402.18659v1#S3.SS6 "3.6 Game Mechanic ‣ 3 Roles of LLMs in Games ‣ Large Language Models and Games: A Survey and Roadmap")), this is handled by an external database that stores and lookups past combination rules—ensuring consistency in future uses of the same mechanic. However, LLMs could theoretically tackle this issue directly. A Retrieval-Augmented Generation (RAG) system [[99](https://arxiv.org/html/2402.18659v1#bib.bib99)] could address this limitation, drawing from a database containing vector representations or other latent representations of pertinent text or data. When the text generator processes a sequence, the RAG system would retrieve similar entries from this external data source. This would hypothetically provide a streamlined archive of game events and actions for the LLMs to consult in order to generate a consistent narrative progression.

Another challenge is that currently LLMs are trained to be highly compliant to the users’ requests. For an LLM-as-assistant, this is not a cause for concern, but in the role of a game master this can create issues. Human game masters frequently curb the more exotic player requests, which could drastically diverge from the game narrative, or which would result in an unrecoverable disruption of a required sequence of game events. An LLM game master would try to accommodate for even the most bizarre requests, with little consideration for the consequential impacts to any predetermined game events.

Finally, the implementation and deployment of LLMs to video game applications is still very limited. A video game is a domain where responsiveness is vital for players, so it follows that LLMs should also be able to provide their responses quickly. Unfortunately, while research on more efficient and faster architectures is being carried out [[100](https://arxiv.org/html/2402.18659v1#bib.bib100)], the real-time application of LLMs is still not plausible. This is especially evident in other domains such as design applications, where “real time” responses are generated in around 30 seconds to over a minute [[101](https://arxiv.org/html/2402.18659v1#bib.bib101)].

##  6 Ethical Issues with LLMs in games

With the improvement of AI methods applied to video games over the recent years, many questions regarding their ethics and real-world impact have been raised [[102](https://arxiv.org/html/2402.18659v1#bib.bib102)]. Using LLMs raises ethical issues regarding sustainability, copyright, explainability, and biases. Naturally, each of these issues has serious implications in the field of video games.

The reliance of LLMs on training data and training time raises concerns regarding their carbon footprint. Beyond training costs, inference over the model’s lifespan has a greater environmental impact due to constant querying [[103](https://arxiv.org/html/2402.18659v1#bib.bib103), [104](https://arxiv.org/html/2402.18659v1#bib.bib104)]. Factors like renewable and local energy, better model architectures, and more meaningful (and thus less wasteful) training data can mitigate this. In the context of LLMs for digital games, sustainability remains crucial, considering the carbon footprint of frequent queries during gameplay (e.g. for Game Master or NPC responses, or for LLM-powered players). This is especially pertinent if the LLM is intended to run locally, on consumer-level hardware which is usually powered by non-renewable sources.

When it comes to copyright, issues apply to the input data, the output data, and the model itself. LLMs trained on data under copyright is an unfortunate common practice [[105](https://arxiv.org/html/2402.18659v1#bib.bib105)], deservedly raising public outrage [[106](https://arxiv.org/html/2402.18659v1#bib.bib106), [107](https://arxiv.org/html/2402.18659v1#bib.bib107)]. The models themselves have different copyright licenses applied, which can also lead to artifacts they generate to fall under the public domain [[11](https://arxiv.org/html/2402.18659v1#bib.bib11), [108](https://arxiv.org/html/2402.18659v1#bib.bib108)]. For the game industry, matters of IP and copyright are extremely important. This is as much a concern regarding having the company’s copyrighted content somehow used as training by competitors, as it is about using LLMs that can produce material that the company cannot copyright. It is important to note here that, at least when it comes to the latter concern, the role the LLM takes is very pertinent. If an LLM or LMM produces content automatically (see Section [3.7](https://arxiv.org/html/2402.18659v1#S3.SS7 "3.7 Automated Designer ‣ 3 Roles of LLMs in Games ‣ Large Language Models and Games: A Survey and Roadmap")), past legal consensus in the US indicates that the material can not be copyrighted [[109](https://arxiv.org/html/2402.18659v1#bib.bib109)]. If an LLM or LMM acts as an “assistive tool” [[110](https://arxiv.org/html/2402.18659v1#bib.bib110)] to a designer (especially for conceptual assistance, see Section [3.8](https://arxiv.org/html/2402.18659v1#S3.SS8 "3.8 Design Assistant ‣ 3 Roles of LLMs in Games ‣ Large Language Models and Games: A Survey and Roadmap")) then the extensive and impactful human effort needed to transform these concepts into game design and game art likely makes the final product copyrightable [[110](https://arxiv.org/html/2402.18659v1#bib.bib110)]. The limited rulings in copyright courts regarding this, however, and the “likely” caveat we include in our own text, understandably would make game companies hesitant to tread in untested waters for major game IPs beyond e.g. small-scale indie productions [[45](https://arxiv.org/html/2402.18659v1#bib.bib45)]. For researchers, however, the ethical issues of copyright breach and exploitation by large corporations, and the public outcry for the above, leave a bad taste and make research in LLMs less palatable [[111](https://arxiv.org/html/2402.18659v1#bib.bib111)].

In applications, understanding how a final result or product is reached is a extremely crucial, particularly when a product is iteratively refined as with design assistants (see Section [3.8](https://arxiv.org/html/2402.18659v1#S3.SS8 "3.8 Design Assistant ‣ 3 Roles of LLMs in Games ‣ Large Language Models and Games: A Survey and Roadmap")). This is a problem of explainability [[79](https://arxiv.org/html/2402.18659v1#bib.bib79)], whereas LLMs are inherently opaque in their generation process. In [[112](https://arxiv.org/html/2402.18659v1#bib.bib112)], the authors highlight different methods to improve the explainability of language models, such as concept-based explanations or saliency maps. Particularly for LLMs, the self-explanation applied via the chain-of-thought (CoT) [[113](https://arxiv.org/html/2402.18659v1#bib.bib113)] reasoning has received attention by the research community [[114](https://arxiv.org/html/2402.18659v1#bib.bib114), [115](https://arxiv.org/html/2402.18659v1#bib.bib115)]. While this method adds a layer of explained reasoning to the generated output, there are multiple examples in the literature that demonstrate how this reasoning may just be an illusion of reasoning capabilities. Such examples include disregarding the provided reasoning in the final output [[116](https://arxiv.org/html/2402.18659v1#bib.bib116)], or reaching the correct solution via incorrect steps in math problems [[117](https://arxiv.org/html/2402.18659v1#bib.bib117)]. In video-games, explainability is paramount across roles, ensuring gameplay coherence and user engagement.

Finally, biases emerge as LLMs are trained on a large corpus, usually scraped from the (Western-focused part of the) internet. This allows models to capture a current reality snapshot, which is advantageous for a conversational or questions answering model, though it requires curating this data from different kinds of biases. Some biases, such as social stereotypes, could be targeted and alleviated; others, such as exclusionary norms, pose greater challenges. In video games, we identify two main concerns when interacting with an LLM: toxic behavior, and stereotypes or incorrect notions. Toxic behavior is a harmful property that a language model may learn from its training corpus, which often contains text from community-based fora or social platforms. Tools that combat toxic language in video games are constantly evolving, with some even blocking chat messages before they are even delivered to the user [[118](https://arxiv.org/html/2402.18659v1#bib.bib118), [119](https://arxiv.org/html/2402.18659v1#bib.bib119)]. Therefore, similar applications could theoretically be developed to target toxic outputs from language models. Unlike human players, however, when an LLM plays the role of an NPC, it should align with the game themes and avoid any kind of toxic language or racial slurs. This requires developers to ensure proper behavior of the model through data cleaning, if the model is trained from scratch, or supplying tailored data if finetuning it to their needs. Addressing _prejudices_ such as stereotypes and incorrect notions is complex, as they are not necessarily related to single words or expressions, but instead present themselves as a collection of ideals that can be wrong at best, and harmful at worst. An NPC LLM may exhibit real world stereotypes that can impact negatively the player experience, although we argue that the impact of prejudices from a player commentator or game master is much stronger and disturbing due to their perceived authority.

##  7 Conclusions

As discussed in this paper, LLMs can take up many different roles that can improve the experience of players in video games, or enhance the ability of game designers to bring their ideas to life. However, we also highlighted many different challenges specific to the applications of LLMs and intrinsic to the nature of LLMs and the ecosystem that surrounds them. Despite technical, ethical, and legal challenges posed by LLMs, it is not realistic to ignore the impact that this research will likely have on both Game AI research and the game industry. We expect to see many new technical innovations from LLM researchers and corporations. Anticipating this, we propose promising directions where LLMs could be applied to games in the future.

## Acknowledgments

This work has been supported by the European Union’s Horizon 2020 research and innovation programme from the AI4media project (Grant Agreement No. 951911). 

## References

  * [1] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever _et al._ , “Language models are unsupervised multitask learners,” _OpenAI blog_ , vol. 1, no. 8, 2019. 
  * [2] T. Kaufmann, P. Weng, V. Bengs, and E. Hüllermeier, “A survey of reinforcement learning from human feedback,” _arXiv preprint arXiv:2312.14925_ , 2023. 
  * [3] S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg, H. Nori, H. Palangi, M. T. Ribeiro, and Y. Zhang, “Sparks of artificial general intelligence: Early experiments with GPT-4,” _arXiv preprint arXiv:2303.12712_ , 2023. 
  * [4] R. T. McCoy, S. Yao, D. Friedman, M. Hardy, and T. L. Griffiths, “Embers of autoregression: Understanding large language models through the problem they are trained to solve,” _arXiv preprint arXiv:2309.13638_ , 2023. 
  * [5] G. N. Yannakakis and J. Togelius, _Artificial Intelligence and Games_. Springer, 2018. 
  * [6] J. Gwertzman and J. Soslow, “The generative AI revolution in games,” <https://a16z.com/the-generative-ai-revolution-in-games/>, 2022, accessed 25 Feb 2024. 
  * [7] S. Minaee, T. Mikolov, N. Nikzad, M. Chenaghlu, R. Socher, X. Amatriain, and J. Gao, “Large language models: A survey,” _arXiv preprint arXiv:2402.06196_ , 2024. 
  * [8] C. E. Shannon, “A mathematical theory of communication,” _The Bell system technical journal_ , vol. 27, no. 3, 1948. 
  * [9] J. L. Elman, “Finding structure in time,” _Cognitive science_ , vol. 14, no. 2, 1990. 
  * [10] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Proceedings of the Conference on Neural Information Processing Systems_ , 2017. 
  * [11] A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M.-A. Lachaux, P. Stock, T. L. Scao, T. Lavril, T. Wang, T. Lacroix, and W. E. Sayed, “Mistral 7B,” _arXiv preprint arXiv:2310.06825_ , 2023. 
  * [12] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar _et al._ , “Llama: Open and efficient foundation language models,” _arXiv preprint arXiv:2302.13971_ , 2023. 
  * [13] Z. Yang, L. Li, K. Lin, J. Wang, C.-C. Lin, Z. Liu, and L. Wang, “The dawn of LMMs: Preliminary explorations with GPT-4V(ision),” _arXiv preprint arXiv:2309.17421_ , 2023. 
  * [14] H. Liu, C. Li, Q. Wu, and Y. J. Lee, “Visual instruction tuning,” in _Proceedings of the Neural Information Processing Systems Conference_ , 2023\. 
  * [15] A. Summerville, S. Snodgrass, M. Guzdial, C. Holmgård, A. K. Hoover, A. Isaksen, A. Nealen, and J. Togelius, “Procedural content generation via machine learning (PCGML),” _IEEE Transactions on Games_ , vol. 10, no. 3, 2018. 
  * [16] D. Noever, M. Ciolino, and J. Kalin, “The chess transformer: Mastering play using generative language models,” _arXiv preprint arXiv:2008.04057_ , 2020\. 
  * [17] A. Stöckl, “Watching a language model learning chess,” in _Proceedings of the Recent Advances in Natural Language Processing International Conference_ , 2021. 
  * [18] S. Toshniwal, S. Wiseman, K. Livescu, and K. Gimpel, “Chess as a testbed for language model state tracking,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 36, no. 10, 2022. 
  * [19] M. Ciolino, J. Kalin, and D. Noever, “The Go transformer: Natural language modeling for game play,” in _Proceedings of the Artificial Intelligence for Industries Internation Conference_ , 2020. 
  * [20] K. Li, A. K. Hopkins, D. Bau, F. Viégas, H. Pfister, and M. Wattenberg, “Emergent world representations: Exploring a sequence model trained on a synthetic task,” in _Proceedings of the International Conference on Learning Representations_ , 2023. 
  * [21] S. Reed, K. Zolna, E. Parisotto, S. G. Colmenarejo, A. Novikov, G. Barth-Maron, M. Gimenez, Y. Sulsky, J. Kay, J. T. Springenberg _et al._ , “A generalist agent,” _arXiv preprint arXiv:2205.06175_ , 2022. 
  * [22] M. Xu, W. Dai, C. Liu, X. Gao, W. Lin, G.-J. Qi, and H. Xiong, “Spatial-temporal transformer networks for traffic flow forecasting,” _arXiv preprint arXiv:2001.02908_ , 2020. 
  * [23] H. Chang, H. Zhang, L. Jiang, C. Liu, and W. T. Freeman, “MaskGIT: Masked generative image transformer,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022. 
  * [24] S. Yao, R. Rao, M. Hausknecht, and K. Narasimhan, “Keep CALM and explore: Language models for action generation in text-based games,” in _Proceedings of the Empirical Methods in Natural Language Processing Conference_ , 2020. 
  * [25] C. F. Tsai, X. Zhou, S. S. Liu, J. Li, M. Yu, and H. Mei, “Can large language models play text games well? current state-of-the-art and open questions,” _arXiv preprint arXiv:2304.02868_ , 2023. 
  * [26] Meta Fundamental AI Research Diplomacy Team (FAIR), A. Bakhtin, N. Brown, E. Dinan, G. Farina, C. Flaherty, D. Fried, A. Goff, J. Gray, H. Hu _et al._ , “Human-level play in the game of diplomacy by combining language models with strategic reasoning,” _Science_ , vol. 378, no. 6624, 2022. 
  * [27] G. Wang, Y. Xie, Y. Jiang, A. Mandlekar, C. Xiao, Y. Zhu, L. Fan, and A. Anandkumar, “Voyager: An open-ended embodied agent with large language models,” _arXiv preprint arXiv:2305.16291_ , 2023. 
  * [28] GoodAI, “Introducing our work on general-purpose llm agents,” <https://www.goodai.com/introducing-general-purpose-llm-agents/>, 2023, accessed 24 Feb 2024. 
  * [29] M. Ç. Uludağlı and K. Oğuz, “Non-player character decision-making in computer games,” _Artificial Intelligence Review_ , vol. 56, no. 12, 2023. 
  * [30] M. Shanahan, K. McDonell, and L. Reynolds, “Role play with large language models,” _Nature_ , vol. 623, p. 493–498, 2023. 
  * [31] H. Warpefelt and H. Verhagen, “A model of non-player character believability,” _Journal of Gaming & Virtual Worlds_, vol. 9, no. 1, 2017\. 
  * [32] J. Duan, H. Cheng, S. Wang, A. Zavalny, C. Wang, R. Xu, B. Kailkhura, and K. Xu, “Shifting attention to relevance: Towards the uncertainty estimation of large language models,” _arXiv preprint arXiv:2307.01379_ , 2023. 
  * [33] A. Mehta, Y. Kunjadiya, A. Kulkarni, and M. Nagar, “Exploring the viability of conversational AI for Non-Playable Characters: A comprehensive survey,” in _Proceedings of the 4th International Conference on Recent Trends in Computer Science and Technology_ , 2022. 
  * [34] E. Akata, L. Schulz, J. Coda-Forno, S. J. Oh, M. Bethge, and E. Schulz, “Playing repeated games with large language models,” _arXiv preprint arXiv:2305.16867_ , 2023. 
  * [35] Y. Xu, S. Wang, P. Li, F. Luo, X. Wang, W. Liu, and Y. Liu, “Exploring large language models for communication games: An empirical study on Werewolf,” _arXiv preprint arXiv:2309.04658_ , 2023. 
  * [36] P. Maas, F. Carey, C. Wheeler, E. Saatchi, P. Billington, and J. Yaffa Shamash, “To infinity and beyond: SHOW-1 and showrunner agents in multi-agent simulations,” <https://fablestudio.github.io/showrunner-agents/>, 2023, accessed 27 Feb 2024. 
  * [37] J. Urbanek, A. Fan, S. Karamcheti, S. Jain, S. Humeau, E. Dinan, T. Rocktäschel, D. Kiela, A. Szlam, and J. Weston, “Learning to speak and act in a fantasy text adventure game,” in _Proceedings of the Empirical Methods in Natural Language Processing Conference and the Natural Language Processing International Joint Conference_ , 2019. 
  * [38] M. P. Eladhari, “Re-tellings: The fourth layer of narrative as an instrument for critique,” in _Proceedings of the Interactive Digital Storytelling International Conference_ , 2018. 
  * [39] M. Guzdial, S. Shah, and M. Riedl, “Towards automated let’s play commentary,” in _Proceedings of the AIIDE workshop on Experimental AI in Games_ , 2018\. 
  * [40] T. Ishigaki, G. Topić, Y. Hamazono, H. Noji, I. Kobayashi, Y. Miyao, and H. Takamura, “Generating racing game commentary from vision, language, and structured data,” in _Proceedings of the Natural Language Generation International Conference_ , 2021. 
  * [41] C. Li, S. Gandhi, and B. Harrison, “End-to-end let’s play commentary generation using multi-modal video representations,” in _Proceedings of the Foundations of Digital Games International Conference_ , 2019. 
  * [42] N. Renella and M. Eger, “Towards automated video game commentary using generative AI,” in _Proceedings of the AIIDE workshop on Experimental AI in Games_ , 2023. 
  * [43] A. Tychsen, M. Hitchens, T. Brolund, and M. Kavakli, “The game master,” in _Proceedings of the Australasian conference on Interactive entertainment_ , 2005. 
  * [44] A. Liapis and A. Denisova, “The challenge of evaluating player experience in tabletop role-playing games,” in _Proceedings of the Foundations of Digital Games International Conference_ , 2023. 
  * [45] A. Liapis, “The discourse on computational creativity in indie tabletop role-playing games,” in preparation. 
  * [46] M. Hua and R. Raley, “Playing with unicorns: AI dungeon and citizen NLP,” _Digital Humanities Quarterly_ , vol. 14, no. 4, 2020. 
  * [47] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022. 
  * [48] A. Zhu, L. Martin, A. Head, and C. Callison-Burch, “CALYPSO: LLMs as Dungeon Masters’ assistants,” in _Proceedings of the Nineteenth AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment_ , 2023\. 
  * [49] J. Kelly, M. Mateas, and N. Wardrip-Fruin, “Towards computational support with language models for TTRPG game masters,” in _Proceedings of the FDG workshop on Human-AI Interaction through Play_ , 2023. 
  * [50] D. Acharya, J. Kelly, W. Tate, M. Joslyn, M. Mateas, and N. Wardrip-Fruin, “Shoelace: A storytelling assistant for GUMSHOE One-2-One,” in _Proceedings of the Foundations of Digital Games International Conference_ , 2023. 
  * [51] Y. Sun, Z. Li, K. Fang, C. H. Lee, and A. Asadipour, “Language as reality: A co-creative storytelling game experience in 1001 nights using generative AI,” in _Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment_ , 2023. 
  * [52] M. Treanor, A. Zook, M. P. Eladhari, J. Togelius, G. Smith, M. Cook, T. Thompson, B. Magerko, J. Levine, and A. Smith, “AI-based game design patterns,” in _Proceedings of the Foundations of Digital Games International Conference_ , 2015. 
  * [53] J. S. Park, J. O’Brien, C. J. Cai, M. R. Morris, P. Liang, and M. S. Bernstein, “Generative agents: Interactive simulacra of human behavior,” in _Proceedings of the Annual ACM Symposium on User Interface Software and Technology_ , 2023. 
  * [54] Y. Liu, G. Deng, Z. Xu, Y. Li, Y. Zheng, Y. Zhang, L. Zhao, T. Zhang, and Y. Liu, “Jailbreaking ChatGPT via prompt engineering: An empirical study,” _arXiv preprint arXiv:2305.13860_ , 2023. 
  * [55] T. Litchfield, “This browser-based ‘endless crafting game’ starts you off with fire and water, but it quickly escalates to God, the Big Bang, and ‘Yin-Yoda’,” <https://www.pcgamer.com/this-browser-based-endless-crafting-game-starts-you-off-with-fire-and-water-but-it-quickly-escalates-to-god-the-big-bang-and-yin-yoda/>, 2024, accessed 28 Feburary 2024. 
  * [56] A. J. Summerville, S. Snodgrass, M. Mateas, and S. Ontanon, “The VGLC: The video game level corpus,” in _Proceedings of the FDG workshop on Procedural Content Generation_ , 2016. 
  * [57] A. Summerville and M. Mateas, “Super Mario as a string: Platformer level generation via LSTMs,” in _Proceedings of the Joint Conference of DIGRA and FDG_ , 2016. 
  * [58] G. Todd, S. Earle, M. U. Nasir, M. C. Green, and J. Togelius, “Level generation through large language models,” in _Proceedings of the Foundations of Digital Games International Conference_ , 2023. 
  * [59] C. Bamford, “Griddly: A platform for AI research in games,” _Software Impacts_ , vol. 8, 2021. 
  * [60] S. Sudhakaran, M. González-Duque, M. Freiberger, C. Glanois, E. Najarro, and S. Risi, “MarioGPT: Open-ended text2level generation through large language models,” _Advances in Neural Information Processing Systems_ , vol. 36, 2024. 
  * [61] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in _Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics_ , 2019. 
  * [62] A. Liapis, “Searching for sentient design tools for game development,” Ph.D. dissertation, Center for Computer Games, IT University of Copenhagen, Copenhagen, Denmark, 2014. 
  * [63] G. Smith, J. Whitehead, and M. Mateas, “Tanagra: Reactive planning and constraint solving for mixed-initiative level design,” _IEEE Transactions on Computational Intelligence and AI in Games_ , vol. 3, no. 3, 2011\. 
  * [64] A. Liapis, G. N. Yannakakis, and J. Togelius, “Sentient sketchbook: Computer-aided game level authoring,” in _Proceedings of the Foundations of Digital Games Conference_ , 2013. 
  * [65] P. Migkotzidis and A. Liapis, “SuSketch: Surrogate models of gameplay as a design assistant,” _IEEE Transactions on Games_ , vol. 14, no. 2, 2021. 
  * [66] M. Charity, I. Dave, A. Khalifa, and J. Togelius, “Baba is Y’all 2.0: Design and investigation of a collaborative mixed-initiative system,” _IEEE Transactions on Games_ , 2022, early Access. 
  * [67] R. Gallotta, K. Arulkumaran, and L. B. Soros, “Preference-learning emitters for mixed-initiative Quality-Diversity algorithms,” _IEEE Transactions on Games_ , 2023, early Access. 
  * [68] G. N. Yannakakis, A. Liapis, and C. Alexopoulos, “Mixed-initiative co-creativity,” in _Proceedings of the Foundations of Digital Games Conference_ , 2014. 
  * [69] A. Liapis, G. N. Yannakakis, and J. Togelius, “Sentient world: Human-based procedural cartography,” in _Proceedings of Evolutionary and Biologically Inspired Music, Sound, Art and Design Conference_ , 2013, vol. 7834, LNCS. 
  * [70] D. Novick and S. Sutton, “What is mixed-initiative interaction?” in _Proceedings of the AAAI Spring Symposium on Computational Models for Mixed Initiative Interaction_ , 1997. 
  * [71] A. Liapis, G. N. Yannakakis, and J. Togelius, “Designer modeling for personalized game content creation tools,” in _Proceedings of the AIIDE Workshop on Artificial Intelligence & Game Aesthetics_, 2013. 
  * [72] V. Vimpari, A. Kultima, P. Hämäläinen, and C. Guckelsberger, ““An adapt-or-die type of situation”: Perception, adoption, and use of text-to-image-generation AI by game industry professionals,” in _Proceedings of the ACM on Human-Computer Interaction_ , vol. 7, 2023. 
  * [73] J. Boucher, G. Smith, and Y. Telliel, “Examining early professionals’ use of generative AI in the game development process,” in _Proceedings of the AIIDE Workshop on Experimental Artificial Intelligence in Games_ , 2023. 
  * [74] M. Charity, Y. Bhartia, D. Zhang, A. Khalifa, and J. Togelius, “A Preliminary Study on a Conceptual Game Feature Generation and Recommendation System,” in _Proceedings of the IEEE Conference on Games_ , 2023. 
  * [75] M. U. Nasir and J. Togelius, “Practical PCG through large language models,” in _Proceedings of the IEEE Conference on Games_ , 2023. 
  * [76] V. Kumaran, D. Carpenter, J. Rowe, B. Mott, and J. Lester, “End-to-end procedural level generation in educational games with natural language instruction,” in _Proceedings of the IEEE Conference on Games_ , 2023. 
  * [77] P. Taveekitworachai, F. Abdullah, M. F. Dewantoro, R. Thawonmas, J. Togelius, and J. Renz, “ChatGPT4PCG competition: Character-like level generation for science birds,” _arXiv preprint arXiv:2303.15662_ , 2023. 
  * [78] T. Brooks, A. Holynski, and A. A. Efros, “InstructPix2Pix: Learning to follow image editing instructions,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023. 
  * [79] J. Zhu, A. Liapis, S. Risi, R. Bidarra, and G. M. Youngblood, “Explainable AI for designers: A human-centered perspective on mixed-initiative co-creation,” in _Proceedings of the IEEE Conference on Computational Intelligence and Games_ , 2018. 
  * [80] A. Liapis, M. Awiszus, A. J. Champandard, M. Cook, A. Denisova, A. Dockhorn, T. Thompson, and J. Zhu, “Artificial intelligence for audiences,” in _Human-Game AI Interaction (Dagstuhl Seminar 22251)_ , D. Ashlock, S. Maghsudi, D. P. Liebana, P. Spronck, and M. Eberhardinger, Eds., 2023. 
  * [81] G. N. Yannakakis and D. Melhart, “Affective game computing: A survey,” _Proceedings of the IEEE_ , 2023. 
  * [82] G. N. Yannakakis and J. Togelius, “Experience-driven procedural content generation,” _IEEE Transactions on Affective Computing_ , vol. 2, no. 3, pp. 147–161, 2011. 
  * [83] K. Pinitas, D. Renaudie, M. Thomsen, M. Barthet, K. Makantasis, A. Liapis, and G. N. Yannakakis, “Predicting player engagement in Tom Clancy’s The Division 2: A multimodal approach via pixels and gamepad actions,” in _Proceedings of the ACM International Conference on Multimodal Interaction_ , 2023. 
  * [84] D. Melhart, A. Liapis, and G. N. Yannakakis, “The Arousal video Game AnnotatIoN (AGAIN) dataset,” _IEEE Transactions on Affective Computing_ , vol. 13, no. 4, 2022. 
  * [85] N. M. M. Shafiullah, Z. J. Cui, A. Altanzaya, and L. Pinto, “Behavior transformers: Cloning k𝑘kitalic_k modes with one stone,” in _Proceedings of the Neural Information Processing Systems Conference_ , 2022. 
  * [86] J. Pfau, A. Liapis, G. N. Yannakakis, and R. Malaka, “Dungeons & Replicants II: Automated game balancing across multiple difficulty dimensions via deep player behavior modeling,” _IEEE Transactions on Games_ , 2022. 
  * [87] S. Johnson-Bey, M. Mateas, and N. Wardrip-Fruin, “Toward using ChatGPT to generate theme-relevant simulated storyworlds,” in _Proceedings of the AIIDE Workshop on Experimental Artificial Intelligence in Games_ , 2023. 
  * [88] M. Kreminski, D. Acharya, N. Junius, E. Oliver, K. Compton, M. Dickinson, C. Focht, S. Mason, S. Mazeika, and N. Wardrip-Fruin, “Cozy mystery construction kit: Prototyping toward an AI-assisted collaborative storytelling mystery game,” in _Proceedings of the Foundations of Digital Games International Conference_ , 2019. 
  * [89] C. Fernández-Vara and A. Thomson, “Procedural generation of narrative puzzles in adventure games: The puzzle-dice system,” in _Proceedings of the Workshop on Procedural Content Generation in Games_ , 2012. 
  * [90] J. Dormans, “Adventures in level design: generating missions and spaces for action adventure games,” in _Proceedings of the Workshop on Procedural Content Generation in Games_ , 2010. 
  * [91] P. Manakul, A. Liusie, and M. J. F. Gales, “SelfCheckGPT: Zero-resource black-box hallucination detection for generative large language models,” in _Proceedings of the Empirical Methods in Natural Language Processing Conference_ , 2023. 
  * [92] A. Strasser, “On pitfalls (and advantages) of sophisticated large language models,” _arXiv preprint arXiv:2303.17511_ , 2023. 
  * [93] N. Bian, H. Lin, P. Liu, Y. Lu, C. Zhang, B. He, X. Han, and L. Sun, “Influence of external information on large language models mirrors social cognitive patterns,” _arXiv preprint arXiv:2305.04812_ , 2023. 
  * [94] M. Karpinska and M. Iyyer, “Large language models effectively leverage document-level context for literary translation, but critical errors persist,” in _Proceedings of the Machine Translation Conference_ , 2023. 
  * [95] Z. Gekhman, J. Herzig, R. Aharoni, C. Elkind, and I. Szpektor, “TrueTeacher: Learning factual consistency evaluation with large language models,” in _Proceedings of the Empirical Methods in Natural Language Processing Conference_ , 2023. 
  * [96] J. Zhou, “An evaluation of state-of-the-art large language models for sarcasm detection,” _arXiv preprint arXiv:2312.03706_ , 2023. 
  * [97] Y. Liu, T. Han, S. Ma, J. Zhang, Y. Yang, J. Tian, H. He, A. Li, M. He, Z. Liu, Z. Wu, L. Zhao, D. Zhu, X. Li, N. Qiang, D. Shen, T. Liu, and B. Ge, “Summary of chatgpt-related research and perspective towards the future of large language models,” _Meta-Radiology_ , vol. 1, no. 2, 2023. 
  * [98] D. Li, A. S. Rawat, M. Zaheer, X. Wang, M. Lukasik, A. Veit, F. Yu, and S. Kumar, “Large language models with controllable working memory,” in _Findings of the Association for Computational Linguistics_ , 2023. 
  * [99] P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rocktäschel, S. Riedel, and D. Kiela, “Retrieval-augmented generation for knowledge-intensive nlp tasks,” in _Proceedings of the Conference on Neural Information Processing Systems_ , 2020. 
  * [100] X. Miao, G. Oliaro, Z. Zhang, X. Cheng, H. Jin, T. Chen, and Z. Jia, “Towards efficient generative large language model serving: A survey from algorithms to systems,” _arXiv preprint arXiv:2312.15234_ , 2023. 
  * [101] F. de la Torre, C. M. Fang, H. Huang, A. Banburski-Fahey, J. A. Fernandez, and J. Lanier, “LLMR: Real-time prompting of interactive worlds using large language models,” _arXiv preprint arXiv:2309.12276_ , 2023. 
  * [102] D. Melhart, J. Togelius, B. Mikkelsen, C. Holmgård, and G. N. Yannakakis, “The ethics of AI in games,” _IEEE Transactions on Affective Computing_ , 2023. 
  * [103] A. A. Chien, L. Lin, H. Nguyen, V. Rao, T. Sharma, and R. Wijayawardana, “Reducing the carbon impact of generative AI inference (today and in 2035),” in _Proceedings of the SCS Workshop on Sustainable Computer Systems_ , 2023. 
  * [104] S. A. Khowaja, P. Khuwaja, and K. Dev, “ChatGPT needs SPADE (sustainability, PrivAcy, digital divide, and ethics) evaluation: A review,” _arXiv preprint arXiv:2305.03123_ , 2023. 
  * [105] Free Law Project, “Authors Guild v. OpenAI Inc.” <https://www.courtlistener.com/docket/67810584/authors-guild-v-openai-inc/>, accessed 27 Feb 2024. 
  * [106] J. A. Rothchild, “Copyright implications of the use of code repositories to train a machine learning model,” _Free Software Foundation_ , 2022. 
  * [107] HackerNews, “Github copilot,” <https://news.ycombinator.com/item?id=27676266>, 2021, accessed 27 Feb 2024\. 
  * [108] P. Zhang, G. Zeng, T. Wang, and W. Lu, “Tinyllama: An open-source small language model,” 2024. 
  * [109] U.S. Copyright Office Review Board, “Decision affirming refusal of registration of A Recent Entrance to Paradise,” 2022. 
  * [110] ——, “Registration decision on Zarya of the Dawn,” 2023. 
  * [111] C. E. Lamb and D. G. Brown, “Should we have seen the coming storm? Transformers, society, and CC,” in _Proceedings of the International Conference on Computational Creativity_ , 2023. 
  * [112] Y. Liu, Y. Yao, J.-F. Ton, X. Zhang, R. Guo, H. Cheng, Y. Klochkov, M. F. Taufiq, and H. Li, “Trustworthy LLMs: a survey and guideline for evaluating large language models’ alignment,” in _Proceedings of the NeurIPS Workshop on Socially Responsible Language Modelling Research_ , 2023. 
  * [113] T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa, “Large language models are zero-shot reasoners,” in _Proceedings of the Advances in Neural Information Processing Systems Conference_ , 2022. 
  * [114] S. Chen, B. Li, and D. Niu, “Boosting of thoughts: Trial-and-error problem solving with large language models,” _arXiv preprint arXiv:2402.11140_ , 2024\. 
  * [115] D. Mondal, S. Modi, S. Panda, R. Singh, and G. S. Rao, “KAM-CoT: Knowledge augmented multimodal Chain-of-Thoughts reasoning,” _arXiv preprint arXiv:2401.12863_ , 2024. 
  * [116] M. Turpin, J. Michael, E. Perez, and S. R. Bowman, “Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting,” in _Proceedings of the Neural Information Processing Systems Conference_ , 2023. 
  * [117] S. Frieder, L. Pinchetti, A. Chevalier, R.-R. Griffiths, T. Salvatori, T. Lukasiewicz, P. C. Petersen, and J. Berner, “Mathematical Capabilities of ChatGPT,” in _Proceedings of the Neural Information Processing Systems Conference_ , 2023. 
  * [118] Y. Jia, W. Wu, F. Cao, and S. C. Han, “In-game toxic language detection: Shared task and attention residuals,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2022. 
  * [119] J. Thomas, N. Chorakhalikar, A. Dantrey, R. R. Nalla, and P. Mehta, “Automatic classification and reporting of inappropriate language in online applications,” US Patent 20 210 370 188, 2021. 





url: https://arxiv.org/html/2402.18659
title: Large Language Models and Games: A Survey and Roadmap
scrape_date: 2025-03-11T15:43:15.183176
content_type: markdown
hash: 8ed828ba9e2b13497e52ee06a68c69022e4aa5854cfde044ea8714b16c86b745
content:
  1. [1 Introduction](https://arxiv.org/html/2402.18659v5#S1 "In Large Language Models and Games: A Survey and Roadmap")
  2. [2 A Note on Terminology](https://arxiv.org/html/2402.18659v5#S2 "In Large Language Models and Games: A Survey and Roadmap")
  3. [3 Roles of LLMs in Games](https://arxiv.org/html/2402.18659v5#S3 "In Large Language Models and Games: A Survey and Roadmap")
    1. [3.1 Player](https://arxiv.org/html/2402.18659v5#S3.SS1 "In 3 Roles of LLMs in Games ‣ Large Language Models and Games: A Survey and Roadmap")
    2. [3.2 Non-Player Characters](https://arxiv.org/html/2402.18659v5#S3.SS2 "In 3 Roles of LLMs in Games ‣ Large Language Models and Games: A Survey and Roadmap")
    3. [3.3 Player Assistant](https://arxiv.org/html/2402.18659v5#S3.SS3 "In 3 Roles of LLMs in Games ‣ Large Language Models and Games: A Survey and Roadmap")
    4. [3.4 Commentator/Reteller](https://arxiv.org/html/2402.18659v5#S3.SS4 "In 3 Roles of LLMs in Games ‣ Large Language Models and Games: A Survey and Roadmap")
    5. [3.5 Analyst](https://arxiv.org/html/2402.18659v5#S3.SS5 "In 3 Roles of LLMs in Games ‣ Large Language Models and Games: A Survey and Roadmap")
    6. [3.6 Game Master](https://arxiv.org/html/2402.18659v5#S3.SS6 "In 3 Roles of LLMs in Games ‣ Large Language Models and Games: A Survey and Roadmap")
    7. [3.7 Game Mechanic](https://arxiv.org/html/2402.18659v5#S3.SS7 "In 3 Roles of LLMs in Games ‣ Large Language Models and Games: A Survey and Roadmap")
    8. [3.8 Automated Designer](https://arxiv.org/html/2402.18659v5#S3.SS8 "In 3 Roles of LLMs in Games ‣ Large Language Models and Games: A Survey and Roadmap")
    9. [3.9 Design Assistant](https://arxiv.org/html/2402.18659v5#S3.SS9 "In 3 Roles of LLMs in Games ‣ Large Language Models and Games: A Survey and Roadmap")
  4. [4 A Roadmap for Future Applications of LLMs in Games](https://arxiv.org/html/2402.18659v5#S4 "In Large Language Models and Games: A Survey and Roadmap")
  5. [5 Limitations of LLMs in games](https://arxiv.org/html/2402.18659v5#S5 "In Large Language Models and Games: A Survey and Roadmap")
  6. [6 Ethical Issues with LLMs in games](https://arxiv.org/html/2402.18659v5#S6 "In Large Language Models and Games: A Survey and Roadmap")
  7. [7 Conclusions](https://arxiv.org/html/2402.18659v5#S7 "In Large Language Models and Games: A Survey and Roadmap")



# Large Language Models and Games:   
A Survey and Roadmap

Roberto Gallotta1 , Graham Todd2 , Marvin Zammit1 , Sam Earle2 , Antonios Liapis1 Julian Togelius2 and Georgios N. Yannakakis1,   
1: Institute of Digital Games, University of Malta, Msida, Malta   
2: Tandon School of Engineering, New York University, New York, USA   
roberto.gallotta@um.edu.mt, gdrtodd@nyu.edu, marvin.zammit@um.edu.mt, se2161@nyu.edu, antonios.liapis@um.edu.mt, julian@togelius.com, georgios.yannakakis@um.edu.mt 

###### Abstract

Recent years have seen an explosive increase in research on large language models (LLMs), and accompanying public engagement on the topic. While starting as a niche area within natural language processing, LLMs have shown remarkable potential across a broad range of applications and domains, including games. This paper surveys the current state of the art across the various applications of LLMs _in_ and _for_ games, and identifies the different roles LLMs can take within a game. Importantly, we discuss underexplored areas and promising directions for future uses of LLMs in games and we reconcile the potential and limitations of LLMs within the games domain. As the first comprehensive survey and roadmap at the intersection of LLMs and games, we are hopeful that this paper will serve as the basis for groundbreaking research and innovation in this exciting new field.

###### Index Terms: 

Large Language Models, Digital Games, Video Games, Survey, Generative Text, Gameplaying, Procedural Content Generation, Generative AI. 

##  1 Introduction

Five years ago, autoregressive language modeling was a somewhat niche topic within natural language processing. Training models to simply predict text based on existing text was considered of primarily theoretical interest, although it might have applications as writing support systems. This changed drastically in 2019 when the Generative Pre-trained Transformer 2 (GPT-2) model was released [[1](https://arxiv.org/html/2402.18659v5#bib.bib1)]. GPT-2 demonstrated convincingly that transformer models trained on large text corpora could not only generate surprisingly high-quality and coherent text, but also that text generation could be controlled by carefully prompting the model. While not the first autoregressive model [[2](https://arxiv.org/html/2402.18659v5#bib.bib2), [3](https://arxiv.org/html/2402.18659v5#bib.bib3)], GPT-2 was the first of the “large” models, and as such we use it here as cutoff mark (see also Section [2](https://arxiv.org/html/2402.18659v5#S2 "2 A Note on Terminology ‣ Large Language Models and Games: A Survey and Roadmap")). Subsequent developments, including larger models, instruction fine-tuning, reinforcement learning from human feedback [[4](https://arxiv.org/html/2402.18659v5#bib.bib4)], and the combination of these features in ChatGPT in late 2022, turbocharged interest in large language models (LLMs). Capabilities of LLMs were seemingly unbounded, as long as both problem and solution could be formulated as text.

LLMs are currently a very active research field. Researchers are focused on improving the capabilities of LLMs while reducing their compute and memory footprint, but also on understanding and learning to harness the capabilities of existing LLMs. Informed opinions on the ultimate capabilities of LLM technology vary widely, from the enthusiastic [[5](https://arxiv.org/html/2402.18659v5#bib.bib5)] to the pessimistic [[6](https://arxiv.org/html/2402.18659v5#bib.bib6), [7](https://arxiv.org/html/2402.18659v5#bib.bib7)]. Our aim is to approach the topic from somewhere in-between these two perspectives: optimistic with respect to the potential of LLMs and realistic with respect to their technical, theoretical, and ethical shortcomings.

Games, including board games and video games, serve both as a source of important benchmarks for research in Artificial Intelligence (AI) and as an important application area for AI techniques [[8](https://arxiv.org/html/2402.18659v5#bib.bib8)]. Almost every game utilizes some kind of AI technology, and we are currently in an exploratory phase where both developers and researchers try to figure out how to best make use of recent advances in this field [[9](https://arxiv.org/html/2402.18659v5#bib.bib9)].

In this paper, we set out to chart the impact LLMs have had on games and games research, and the impact they are likely to have in the near- to mid-term future. We survey existing work from both academia and (mostly independent) game creators that use LLMs _with_ and _for_ games. This paper does not set out to capture modern advances in LLM technology or algorithms for training LLMs. Not only do such resources exist [[10](https://arxiv.org/html/2402.18659v5#bib.bib10)], but the breakneck speed of technical advances in this field will likely make our writeup obsolete in a year or so. Instead, we focus on work that leverages LLMs in games and propose a range of roles that the LLM can take in the broader ecosystem of games (both within the game and beyond). We lay out promising future directions for efforts to use LLMs in games, and discuss limitations (both technical and ethical) that should be addressed for a brighter future of LLM research in games.

It is important to note that this survey emerges from the top down, based on our expertise in AI and games [[8](https://arxiv.org/html/2402.18659v5#bib.bib8)], and extensive work on most topics covered by this paper. The focus of the paper, in Section [3](https://arxiv.org/html/2402.18659v5#S3 "3 Roles of LLMs in Games ‣ Large Language Models and Games: A Survey and Roadmap"), is built from our own typology and supported where possible by academic and non-academic work. While a bottom-up approach via e.g. keyword search through general paper repositories is valuable [[11](https://arxiv.org/html/2402.18659v5#bib.bib11), [12](https://arxiv.org/html/2402.18659v5#bib.bib12), [13](https://arxiv.org/html/2402.18659v5#bib.bib13)], the process would lead to a very different type of paper. Indicatively, this approach was followed by Yang et al. [[14](https://arxiv.org/html/2402.18659v5#bib.bib14)] who investigated current uses of GPT models in video games by searching articles in ACM, IEEE Xplore, Springer, and AAAI with keywords “game” and “GPT”. Instead, we have attempted to conduct a comprehensive manual review of all recent proceedings from the major conferences in AI and games111IEEE Conference on Games, Foundations of Digital Games conference, Artificial Intelligence and Interactive Digital Entertainment conference., and the IEEE Transactions on Games for work relevant to the themes of this paper.

##  2 A Note on Terminology

This paper concerns the intersection between games (board games, video games, or other), and large language models. But what exactly is an LLM?

Broadly speaking, an LLM is a model that is trained on text in order to be able to reproduce text in response to other text. But this definition is overly broad, as it would include Shannon’s original n𝑛nitalic_n-gram models from 1946 [[15](https://arxiv.org/html/2402.18659v5#bib.bib15)], rudimentary recurrent neural networks from the early 1990s [[16](https://arxiv.org/html/2402.18659v5#bib.bib16)], and the Tegic T9 text prediction system that would help you write text messages on your Nokia 3210.

What distinguishes LLMs from other text generative models is mainly that they are _large_. But which model size is considered large enough? In 2019 emerging models such as BERT [[2](https://arxiv.org/html/2402.18659v5#bib.bib2)] and ERNIE [[3](https://arxiv.org/html/2402.18659v5#bib.bib3)] showcased significant advances in language modeling, but LLMs became a well-recognized term with the introduction of OpenAI’s GPT-2 [[1](https://arxiv.org/html/2402.18659v5#bib.bib1)], whose various versions have between 117 millions and 1.5 billions of parameters. Because of the association between the term LLM and the GPT-class of models, we will use the size of GPT-2 as a soft cutoff on the type of models we consider LLMs; we are concerned with models of few hundred million parameters or more. Each subsequent iteration of the GPT family features an increased number of parameters, and larger and more diverse training corpora.

Another distinct trait of LLMs is their architecture. While language models could in principle be based on various architectures, including Long Short-Term Memory (LSTM) networks [[17](https://arxiv.org/html/2402.18659v5#bib.bib17)], the current LLM landscape is dominated by variants of the transformer architecture, a type of neural network introduced in 2017 [[18](https://arxiv.org/html/2402.18659v5#bib.bib18)]. This model became very influential because of what was perceived as a quantum leap in output quality compared to previous models. In this survey we rely primarily on LLMs employing this architectural basis.

The last feature of LLMs we consider is their versatility across a wide range of tasks with minimal or no fine-tuning or retraining. This capability represents a significant shift; since the release of GPT-3.5, LLMs have evolved from primarily autoregressive predictive text models to pre-trained, general-purpose conversational models.

It is important to note that LLMs are by no means limited to the GPT family of models. There is by now a large variety of LLMs of varying size and capabilities, including open-source models such as Mistral [[19](https://arxiv.org/html/2402.18659v5#bib.bib19)] and the Llama [[20](https://arxiv.org/html/2402.18659v5#bib.bib20)] family, which can be fine-tuned, run locally, and even be embedded in games’ runtimes.

One could also argue that the definition is somewhat narrow, as many modern LLMs are multimodal models, meaning that they can take as input and/or produce as output modalities other than text. In particular, many modern LLMs can process and produce images. This is often achieved through combining the core transformer network with a visual encoder network for input and a latent diffusion model for output. Examples include GPT-4V [[21](https://arxiv.org/html/2402.18659v5#bib.bib21)] and the open-source Llava [[22](https://arxiv.org/html/2402.18659v5#bib.bib22)]. In this paper, we consider large multimodal models (LMMs) [[21](https://arxiv.org/html/2402.18659v5#bib.bib21)] as long as they retain their ability to both consume and produce text.

This survey will not concern itself with AI and machine learning techniques that are not LLMs as defined above. In particular, we will not be covering the large literature on game playing and content generation using machine learning methods [[23](https://arxiv.org/html/2402.18659v5#bib.bib23)] that does not use textual input and output. We will, however, occasionally mention some of that work where relevant, in particular to help provide historical context.

##  3 Roles of LLMs in Games

Past attempts at a typology for AI in games focused on three roles the AI can take in a game: to play a game, to design a game, or to model the (human) players [[8](https://arxiv.org/html/2402.18659v5#bib.bib8)]. LLMs are typically presented as conversational agents, which often invites the public to give them anthropomorphic qualities —such as reasoning and creativity. We follow these trends when considering the roles an LLM can be called to play within the game or within the game development process. An LLM can operate within the game as a player (replacing a human player while imitating their goals), as a non-player character such as an enemy or interlocutor, as an assistant providing hints or handling menial tasks for a human player, as a Game Master controlling the flow of the game, or hidden within the games’ ruleset (controlling a mechanic of the game). There are however other roles an LLM can play outside of the game’s runtime, such as a designer for the game (replacing or assisting a human designer) or as analyst of the gameplay data of the playerbase. Finally, the LLM can interface with a player or an audience in different ways, acting as a commentator of an ongoing play session (during runtime) or a reteller of past game events in some narrative form (outside runtime). Some of these roles (autonomous player, autonomous designer) are prominent in the broader AI and games research [[8](https://arxiv.org/html/2402.18659v5#bib.bib8)] and LLM research has targeted them extensively, while some of the other roles have been toyed with in exploratory research. The following sections present the roles themselves, surveying research undertaken for each role, while we identify gaps and opportunities for future research in Section [4](https://arxiv.org/html/2402.18659v5#S4 "4 A Roadmap for Future Applications of LLMs in Games ‣ Large Language Models and Games: A Survey and Roadmap").

###  3.1 Player

How can an LLM play a game? Fundamentally, LLM players require some transformation from their typical output space (i.e. sequences of tokens) into the input space of the game. In addition, aspects of the game and its current state must be provided to the LLM in some form in order for it to play at a reasonable level. Depending on the game itself, these mappings might be intuitive or complex. We identify three general classes of games to which LLM players are well suited: (a) games where states and actions can be compactly represented as sequences of abstract tokens, (b) games where the main input and output modalities are natural language, and (c) games for which external programs can control player actions via an API.

The first class of games mostly includes turn-based board games (e.g. Chess), since the discrete set of board positions and moves is more easily transformed into a compact representation (e.g. Portable Game Notation [[24](https://arxiv.org/html/2402.18659v5#bib.bib24)]) than, for instance, a first-person shooter. By tokenizing sequences of moves taken from a game database, the problem of action selection can then be mapped to the standard autoregressive learning objective on which LLMs are trained —predicting the next move given the context of those that preceded it. _Chess_ [[25](https://arxiv.org/html/2402.18659v5#bib.bib25), [26](https://arxiv.org/html/2402.18659v5#bib.bib26), [27](https://arxiv.org/html/2402.18659v5#bib.bib27)], Go [[28](https://arxiv.org/html/2402.18659v5#bib.bib28)], and Othello [[29](https://arxiv.org/html/2402.18659v5#bib.bib29)] have all been used as testbeds for LLM players in this way. This approach allows even more complex game states to be reasoned upon by an LLM player. In Bateni and Whitehead’s work [[30](https://arxiv.org/html/2402.18659v5#bib.bib30)], for example, the LLM plays the popular video game _Slay the Spire_ (Mega Crit, 2017), understanding synergies between cards based solely on their description and adapting to changes in gameplay rules. However, board games are not the only kind of game that can be represented as token sequences: the generalist GATO [[31](https://arxiv.org/html/2402.18659v5#bib.bib31)] agent can play a variety of Atari games at human or near-human levels by processing visual inputs as sequences of pixel values in raster order. Pixel values are interleaved with separator tokens and previous actions, allowing the model to accurately predict the appropriate game action in a dataset of human play traces. It is possible that continued improvement in transformer models that capture both spatial and visual dynamics [[32](https://arxiv.org/html/2402.18659v5#bib.bib32), [33](https://arxiv.org/html/2402.18659v5#bib.bib33)] could allow for a similar approach to scale to even more complex games. However, such approaches require large datasets of gameplay videos that may be comparatively more difficult to collect. In addition, we note that reliance on human gameplay traces as the basis for learning may make it more difficult for an LLM player to reach super-human performance without leaps in terms of reasoning and generalization (see Section [5](https://arxiv.org/html/2402.18659v5#S5 "5 Limitations of LLMs in games ‣ Large Language Models and Games: A Survey and Roadmap")).

The second class of games most obviously includes text adventure games such as Zork (Infocom, 1977), where game states are presented as natural language descriptions and the game is already equipped with a parser to handle natural language responses. This means that LLMs can be queried for game actions in a way that still leverages their large-scale pre-training on natural language text. The earliest application of LLMs to these kinds of text games is CALM [[34](https://arxiv.org/html/2402.18659v5#bib.bib34)], a GPT-2 system finetuned on a dataset of human gameplay transcripts collected from a variety of text adventure games. The model is trained to predict the natural language string provided by human players given the context of previous states, actions and information about the avatar (e.g. their inventory). To actually play a game, the trained language model generates multiple candidate actions and deep reinforcement learning (RL) is used to optimize a policy that selects actions from among the candidates. At the time of its publication, this RL component was necessary because the LLM alone was not capable of generalizing well to unseen games or situations [[34](https://arxiv.org/html/2402.18659v5#bib.bib34)]. However, a more recent investigation of ChatGPT as a Zork player has indicated that LLM performance is improving [[35](https://arxiv.org/html/2402.18659v5#bib.bib35)]. In a preliminary experiment, Yao et al. [[34](https://arxiv.org/html/2402.18659v5#bib.bib34)] show that the performance of ChatGPT can approach that of existing algorithms for text game playing, as long as a human interlocutor remains in the loop to assist the model (e.g. by reminding it of actions it has already tried). However, there is obviously much room for improvement in directly applying LLMs to text games in this way. Additionally, the ability for LLMs to play entirely novel, niche, or unseen text games (especially important given the likelihood that such systems encounter walkthroughs or playtraces of popular text games during their training) remains largely unexplored.

In a similar vein, inductive biases from large language models can be applied to help guide the policies of agents trained with other methods. For instance, the GALAD system [[36](https://arxiv.org/html/2402.18659v5#bib.bib36)] uses a pre-trained LLM to guide an agent towards morally acceptable actions in text games from the Jericho suite [[37](https://arxiv.org/html/2402.18659v5#bib.bib37)], while the MOTIF system [[38](https://arxiv.org/html/2402.18659v5#bib.bib38)] learns an intrinsic reward function for NetHack NetHack, 1987) by mining preferences between game states from an LLM.

Text adventure games are not the only cases where natural language input and output are used for playing: many board games operate via player negotiation. _CICERO_ [[39](https://arxiv.org/html/2402.18659v5#bib.bib39)] leveraged LLMs for playing the deal-making and subterfuge game _Diplomacy_ (Avalon Hill Games, 1976). CICERO builds from a pre-trained LLM and is fine-tuned on a large corpus of _Diplomacy_ transcripts. Throughout the game, samples from the model are sent to other players and the various dialogue transcripts are collected to condition the potential action. CICERO is further trained to condition its outputs on specific game intents (inferred from the transcripts and added as additional context during training). In order to select an action, CICERO uses a “strategic reasoning module” that predicts the actions of other players, using a value and policy function learned from self-play. Diplomacy is an interesting game in part because the action space is split between natural language utterances and a more standard set of moves on a discrete game board, and CICERO demonstrates how an LLM can be integrated as part of a larger system for high-level play.

Finally, we consider games for which a robust API exists. This is less of a kind of game in the sense of its style or mechanics, and more a fact about its popularity or its ease of implementation. An API is an important attribute because it allows LLMs to act as players not by directly generating actions, but by producing programs that act as policies. Improvements in the code generation abilities of LLMs have allowed them to write small programs that can produce actions given game states without further intervention from the model. For instance, the VOYAGER system [[40](https://arxiv.org/html/2402.18659v5#bib.bib40)] leverages the code generation abilities of GPT-4 to play Minecraft (Mojang Studios, 2011) by interacting with the popular Mineflayer API. Using a sophisticated chain of prompts, VOYAGER generates blocks of code that leverage calls to the API in order to execute high-level “skills” (e.g. “Attack nearest zombie”) that are automatically converted into low-level game inputs (e.g. mouse movements and key presses). GPT-4 is also used as a high-level goal generator and planner, which in turn informs the code generation. This approach proved to be very successful, with VOYAGER being the first automated system to complete a variety of in-game _Minecraft_ challenges. The results are impressive and indicate that generating action-producing programs may be a more efficient way to leverage latent LLM knowledge than direct action sampling. However, VOYAGER does benefit substantially from the availability of a robust API and vast amounts of internet discussions around _Minecraft_. As with the analysis of ChatGPT on _Zork_ , the ability of this approach to generalize to less popular or entirely unseen games remains to be seen.

###  3.2 Non-Player Characters

![Refer to caption](extracted/6056081/graphics/1.png) Figure 1: Screenshot of the promotional video for AI people, where a player can interact via their avatar’s chat with other NPCs and watch how their text has consequences between NPC relationships, in this case. Used with permission from [[41](https://arxiv.org/html/2402.18659v5#bib.bib41)].

Non-player characters (NPCs) are agents which exist in virtual game worlds but whose actions are not directly controlled by the players. NPCs exist to enrich the player’s experience and deepen immersion by adding to the world’s ambiance and making it more believable [[42](https://arxiv.org/html/2402.18659v5#bib.bib42)]. NPCs may serve as pets, allies, enemies222We use enemies and allies here for in-game agents with different skills and ways to affect the world than e.g. game playing AI such as opponents in _Chess_., merchants, quest givers, or bystanders. Therefore, they have different agency even from AI-controlled players, and their goal is never to win. This makes designing AI for NPCs interesting [[8](https://arxiv.org/html/2402.18659v5#bib.bib8)], and LLMs are uniquely advantaged in this task since they can “understand” gameworld settings and adapt their responses accordingly. It has been shown that LLMs are able to role-play through different scenarios [[43](https://arxiv.org/html/2402.18659v5#bib.bib43), [44](https://arxiv.org/html/2402.18659v5#bib.bib44)], thereby highlighting their potential to provide a more flexible and apt tool to emulate human behavior. We identify two ways in which LLMs can control NPCs: (a) through their _dialogue_ , and (b) through their _behavior_. Behavior relates to in-game action selection, discussed in Section [3.1](https://arxiv.org/html/2402.18659v5#S3.SS1 "3.1 Player ‣ 3 Roles of LLMs in Games ‣ Large Language Models and Games: A Survey and Roadmap"); however, we note that the heuristics and goals of such behavior is different than an AI player trying to win the game.

LLMs are naturally suited for natural language conversation, and as NPC dialogue systems they can generate dynamic and contextually appropriate responses based on player input [[45](https://arxiv.org/html/2402.18659v5#bib.bib45), [46](https://arxiv.org/html/2402.18659v5#bib.bib46), [47](https://arxiv.org/html/2402.18659v5#bib.bib47), [48](https://arxiv.org/html/2402.18659v5#bib.bib48)]. This makes interactions with NPCs more engaging and realistic, reduces repetitive discourse and provides a more explorative experience within the game [[49](https://arxiv.org/html/2402.18659v5#bib.bib49)]. LLMs can engage the players in the gameworld’s narrative as _foreground_ NPCs, _background_ NPCs, or _narrator_ NPCs. We discuss narrator LLMs as commentators in Section [3.4](https://arxiv.org/html/2402.18659v5#S3.SS4 "3.4 Commentator/Reteller ‣ 3 Roles of LLMs in Games ‣ Large Language Models and Games: A Survey and Roadmap") whereas we cover the other two NPC types here. Foreground NPCs form part of the overarching narrative of the game, or one of its sub-narratives. They may be enemies, allies, information-givers, quest-givers, or item-providers. Their dialogue is heavily constrained by the scope of the narrative, their role within it, and the player actions. Foreground NPCs’ text generation process via LLMs must consider the overall context of the game and the interaction with the player, and keep track of events transpiring in the playthrough. This raises concerns regarding the memory capacity of LLMs, as well as the impact of possible hallucinations, i.e. plausible but false statements [[50](https://arxiv.org/html/2402.18659v5#bib.bib50)]. We revisit these limitations in Section [5](https://arxiv.org/html/2402.18659v5#S5 "5 Limitations of LLMs in games ‣ Large Language Models and Games: A Survey and Roadmap"). The purpose of background NPCs is to make the environment more believable and act independently of the players [[49](https://arxiv.org/html/2402.18659v5#bib.bib49)]. Such NPCs’ presence is purely decorative and their dialogue is essentially small talk. Thus their dialogue generation is less constrained, perhaps bound only by the identity of the speaker and their background. That said, their believability hinges on their ability to maintain the illusion that they have their own agency in the world and can interact with it [[51](https://arxiv.org/html/2402.18659v5#bib.bib51)]. Park et al. [[45](https://arxiv.org/html/2402.18659v5#bib.bib45)] explored NPC interactions within limited environments, where a number of LLM-based agents simulated social interactions in a sandbox environment. Within the constraints of their environment and the social affordances, the agents behaved in a believable manner, following their goals, planning for new ones, and even recalling past events as they interacted.

Other studies have shown that multiple LLM-based agents are able to follow game rules and engage in game playing [[52](https://arxiv.org/html/2402.18659v5#bib.bib52), [53](https://arxiv.org/html/2402.18659v5#bib.bib53)], with different models consistently exhibiting their own aptitudes and weaknesses when applied to specific roles. This ability to interact within constraints is useful to instill believable behaviors in foreground and background NPCs, grounding their actions and dialogue within the rules of the game environment. Some work has focused more on the conversational and story-writing abilities of LLMs, such as the creation of dialogue between multiple characters, each having their unique personality, whilst following a consistent plot. One such example is the use of LLMs to generate a _South Park_ (Comedy Central, 1997) episode [[54](https://arxiv.org/html/2402.18659v5#bib.bib54)] with multiple characters within a well-known setting. There are limitations to this approach, primarily that LLMs perform something like a theatrical improvisation, rather than acting as an actor studying a part [[43](https://arxiv.org/html/2402.18659v5#bib.bib43)]. Through this unconstrained process the LLM is prone to hallucinations which do not fit the desired scenario. This volatility can be mitigated by providing the LLM not only with the conversation history but also with the current state of the environment, such as the items within it and their affordances, as well as the other characters and their corresponding actions. Urbanek et al. [[55](https://arxiv.org/html/2402.18659v5#bib.bib55)] used a configurable multi-user dungeon text-based environment in a fantasy setting to allow for both human and language model-based players. The latter were able to use an updated game state, including descriptions of local environments, objects, and characters, to take better actions and engage in coherent dialogue. This approach may also be extended to other scenarios or to cover the use of LLMs as active or interactive narrators. Ubisoft showcased LLM-based NPCs in their Neo NPC demo [[48](https://arxiv.org/html/2402.18659v5#bib.bib48)], where the player can freely converse with the in-game characters. Each NPC was given a carefully hand-crafted persona, and all NPCs could respond within the constraints of the game narrative and their prescribed personality whilst generating realistic responses. The players were able to engage in game-specific activities with these NPCs, such as planning a heist, or even attempt an unrelated course of dialogue altogether. Care was taken to minimize toxicity and social biases inherent in the LLMs through the prompt defining each NPC, which also had a bearing on how the latter would react to any player’s offensive or unruly discourse.

###  3.3 Player Assistant

A somewhat less explored role for LLMs in games is that of a player assistant: an interactive agent intended to enrich or guide the player experience in some way. This could be in the form of a sequence of tutorial-style tips, a character that does not causally interact with the game world at all, or an agent able to interact within the game environment at a similar level as the player. Existing games make use of player assistants in different ways. For example, in _The Sims_ (Electronic Arts, 2000) a disembodied assistant provides tips specific to the game context via dialogue boxes. _Civilization VI_ (Firaxis Games, 2016) uses different assistants (with different portraits) to suggest the best build option according to their idiosyncratic heuristic, alleviating some decision-making from the player. In management games, AI may automate menial tasks such as assigning jobs to a planet’s population in _Stellaris_ (Paradox Interactive, 2016); this assistance reduces cognitive load from the player, but the player can always micro-manage this task if they wish.

LLMs are appealing as player assistants given their expressive and conversational capacities. An LLM-based player assistant could plausibly choose an action to suggest to the player, and—more importantly—form the explanation for this suggestion as a natural language utterance delivered by a disembodied or embodied agent. This utterance could even be accompanied by a corresponding sentiment, and manifested through the assistant’s body stance, gestures and facial expression (in the case of an embodied assistant). The choice of action to suggest to the player could be based on either LLM-powered or heuristic-based methods for finding the best policy or action given the current game context (see Section [3.1](https://arxiv.org/html/2402.18659v5#S3.SS1 "3.1 Player ‣ 3 Roles of LLMs in Games ‣ Large Language Models and Games: A Survey and Roadmap")). While not intended as a player assistant, this type of exposition is realized by embodied agents in _AI people_ [[41](https://arxiv.org/html/2402.18659v5#bib.bib41)] (see Section [3.7](https://arxiv.org/html/2402.18659v5#S3.SS7 "3.7 Game Mechanic ‣ 3 Roles of LLMs in Games ‣ Large Language Models and Games: A Survey and Roadmap")). Similarly, LLMs may assist the player by undertaking some minor tasks in the game via a tailored smaller role as “player” within that smaller task description (see Section [3.1](https://arxiv.org/html/2402.18659v5#S3.SS1 "3.1 Player ‣ 3 Roles of LLMs in Games ‣ Large Language Models and Games: A Survey and Roadmap")). The LLM could extrapolate the policy for such a minor task through conversation with the player, parsing the natural language chat similar to [[56](https://arxiv.org/html/2402.18659v5#bib.bib56)].

A special case of player assistant comes from an “inner voice”, uttered by the player’s own avatar. Hints given in the player avatar’s own voice are a trope in classic point-and-click adventure games, such as Guybrush Threepwood commenting “A rubber chicken with a pulley in the middle…What possible use could that have?” when picking up the item in _The Secret of Monkey Island_ (Lucasfilm Games, 1990). This specific concept of “inner voice” was explored as an application of LLMs in the work of Rist [[57](https://arxiv.org/html/2402.18659v5#bib.bib57)], although the LLM’s freedom was limited. Using a hand-crafted game environment with pre-scripted events and location-based triggers meant to occur when the player would either interact with the world or observe something, Rist authored where the LLM is prompted to generate short comments in different styles (e.g. in neutral or sarcastic tone) when certain predefined in-game events are met. These comments are generated based on text descriptions of what the user sees and can do in that moment. The LLM still requires human-authored knowledge (e.g. where the commentary occurs, what the hints are, and why in terms of designer goals), and is thus not a fully-autonomous player assistant.

Despite the work of Rist [[57](https://arxiv.org/html/2402.18659v5#bib.bib57)], which focused more on immersion than assistance, the potential of LLM-powered player assistants is not explored in current research. We highlight the potential of this application in Section [4](https://arxiv.org/html/2402.18659v5#S4 "4 A Roadmap for Future Applications of LLMs in Games ‣ Large Language Models and Games: A Survey and Roadmap").

###  3.4 Commentator/Reteller

LLMs are also ideally suited as commentators or retellers. Here, we identify these roles as an agent that produces and narrates a sequence of events, for the benefit of either human players or spectators. Such an agent may consider only in-game events and in-game context, acting as an in-game entity such as a sports commentator in _FIFA_ (EA Sports, 1993) or also consider out-of-game events and context such as the player (their actions, strategies, motivations, etc.). The reteller [[58](https://arxiv.org/html/2402.18659v5#bib.bib58)] exclusively narrates past events —often grouped into a concise “chunk” such as a game session (i.e. based on out-of-game context) or a quest (i.e. based only on in-game context). The commentator may be narrating current, ongoing events which have not been concluded, similar to a streamer concurrently discussing their current actions (including out-of-game context) or a sportscaster in an in-progress sports game such as _FIFA_.

The vision of automated “let’s play”-style commentary generation is not new. It was proposed by Guzdial, Shah and Riedl et al. [[59](https://arxiv.org/html/2402.18659v5#bib.bib59)] and implemented via classical machine learning methods, with limited success. Ishigaki et al. [[60](https://arxiv.org/html/2402.18659v5#bib.bib60)] trained an LSTM with text, vision and game state input to generate characters for a commentary script in the racing game Assetto Corsa (Kunos Simulazioni, 2013). Results of this approach featured repetitive and context-irrelevant generated text. LSTMs were also used by Li, Gandhi and Harrison [[61](https://arxiv.org/html/2402.18659v5#bib.bib61)] to generate text, at a character level, for _Getting Over It With Bennett Foddy_ (Bennett Foddy, 2017), a challenging side-scrolling climbing game.

LLMs for commentary were also explored by Renella and Eger [[62](https://arxiv.org/html/2402.18659v5#bib.bib62)], who argue that LLMs could assist game streamers (e.g. on Twitch) while the streamer multitasks gameplay with audience interaction. The authors developed a pipeline for automatically commenting upon _League of Legends_ (Riot Games, 2009) games. They took a multi-phased approach, training a model on hand-annotated data to recognize key events, then prompting ChatGPT to generate zero-shot commentary on these events in the style of a particular (known) fictional character, and finally sending the generated text through the FakeYou333<https://fakeyou.com/> API to be voiced in the timbre of this same character. For example, once the event detection model has identified an enemy double kill in a particular frame, ChatGPT responds in the style of Rick Sanchez from _Rick and Morty_ (Cartoon Network, 2013): “What the heck?! That enemy team just got a double kill! I can’t believe it! They must be pretty good! I better watch out for them!” An additional loop buffers detected events —for example, delaying commentary on a double kill in case it should escalate into a triple kill, or prioritizing among a quick barrage of events —and prompts ChatGPT to generate random fillers, such as thanking (fictional) new subscribers.

Despite the existence of the aforementioned studies, research on LLMs as game commentators remains rather limited. The appeal is obvious: simulation-based games of emergent narrative already generate rich narrative histories, and are remixed by human players to produce secondary content that is often popular in its own right. In principle, LLMs could be used to generate more succinct retellings or highlight reels of these game events. Prompting current LLMs for stories, without any further specification of style or substance, tends to produce output that feels generic. Past events recorded in simulation games could ultimately provide specificity and narrative coherence to these outputs. Exploring more concepts beyond automating streamer commentary, such as assisting streamers via LLM commentary of the audiences’ reactions rather than the in-game actions, remains unexplored. We revisit this along other future applications in Section [4](https://arxiv.org/html/2402.18659v5#S4 "4 A Roadmap for Future Applications of LLMs in Games ‣ Large Language Models and Games: A Survey and Roadmap").

###  3.5 Analyst

Another role for LLMs is that of a data analyst. Here, we take this role to primarily analyze player experience and behavior, rather than the broader big-data job title for which some LLMs are naturally suited for [[63](https://arxiv.org/html/2402.18659v5#bib.bib63), [64](https://arxiv.org/html/2402.18659v5#bib.bib64), [65](https://arxiv.org/html/2402.18659v5#bib.bib65)]. While player modeling is an important aspect of AI research [[8](https://arxiv.org/html/2402.18659v5#bib.bib8)] and game development practice [[66](https://arxiv.org/html/2402.18659v5#bib.bib66)], LLMs have not received much attention for this purpose. However, the ability of LLMs to make sense of structured data such as code [[67](https://arxiv.org/html/2402.18659v5#bib.bib67), [68](https://arxiv.org/html/2402.18659v5#bib.bib68)] can be a boon for this type of work.

So far, language models (both large and small) have been used for clustering player behaviors. Player2vec [[69](https://arxiv.org/html/2402.18659v5#bib.bib69)] trained the Longformer transformer architecture [[70](https://arxiv.org/html/2402.18659v5#bib.bib70)], with up to 121 million parameters, on a corpus of game events stored in JSON format which captured player interactions with a casual mobile game. Using dimensionality reduction methods on the latent vectors of the last layer of a pre-trained transformer, the authors discovered eight clusters with player traits that could be useful for market research, such as “lean-in casual economy aware” [[69](https://arxiv.org/html/2402.18659v5#bib.bib69)]. In such work, the strength of LLMs is their ability to process structured or unstructured data without the need for extensive data pre-processing. However, as with previous experiments in bottom-up clustering of players [[71](https://arxiv.org/html/2402.18659v5#bib.bib71), [72](https://arxiv.org/html/2402.18659v5#bib.bib72)], the resulting clusters need to be interpreted by expert game analysts (or the game’s designers) to derive some meaningful player types. In this form, current research points to LLMs acting as analyst assistants rather than independent analysts; we revisit future directions in this vein in Section [4](https://arxiv.org/html/2402.18659v5#S4 "4 A Roadmap for Future Applications of LLMs in Games ‣ Large Language Models and Games: A Survey and Roadmap").

Representing game logs as text interpreted by an LLM need not be the end-goal for analysis. The LLM representations of game logs can also be used to find common patterns that can be used for other purposes such as gameplay (or gameplay footage) similarity. Indicatively, Rasajski et al. [[73](https://arxiv.org/html/2402.18659v5#bib.bib73)] used LLMs on recorded gameplay action logs to establish action similarity. This similarity was in turn applied to train video encodings on gameplay footage of these recorded actions, in order to align the visual latent vectors to the action logs. While the work focused on the computer vision task of representation learning for gameplay footage, better (learned) representations of gameplay pixels would be more generalizable across games in downstream tasks. Such latent representations would be useful for general action recognition or general affect modeling [[74](https://arxiv.org/html/2402.18659v5#bib.bib74)]; we expand on these downstream tasks in Section [4](https://arxiv.org/html/2402.18659v5#S4 "4 A Roadmap for Future Applications of LLMs in Games ‣ Large Language Models and Games: A Survey and Roadmap").

###  3.6 Game Master

A Game Master (GM) in tabletop role-playing games (TTRPGs) is the person who creates the plot of a game, its characters, and narrative. GMs wear many hats during the course of the game session [[75](https://arxiv.org/html/2402.18659v5#bib.bib75)]; they prepare and adapt stories before sessions, guide gameplay during, and follow up with players afterward [[76](https://arxiv.org/html/2402.18659v5#bib.bib76)]. Digital games have mostly pre-scripted stories or level progressions and their players have a restricted range of affordances, compared to TTRPG players whose actions are only limited by their imagination. Similarly, the story told around the table can take any direction. Since human GMs mostly communicate about the gameworld, story, game state and action resolutions via natural language (although props such as maps, miniatures, hand-outs are also common), the potential of LLMs as a GM is often mentioned both in research circles and TTRPG discussion boards. LLMs as GMs also open the potential for solo play, while a TTRPG requires at least one player and a human GM.

One of the first notable text adventures managed by a fine-tuned version of GPT-2 is AI Dungeon [[77](https://arxiv.org/html/2402.18659v5#bib.bib77)]. It is an online444<https://play.aidungeon.com/> interactive chat-based storytelling application where the player takes actions through language input alone. The LLM continues the story based on the player’s input, in the fashion of a human GM. The game has evolved since its creation to make use of more recent LLM models, which the player can choose from before starting a play session. Different gameworld settings are also offered, and players are also able to share the stories they create. Similar games have emerged online ever since555<https://koboldai.net>, <https://www.hiddendoor.co>, and a freely available code repository, _Kobold AI Client_ 666<https://github.com/KoboldAI/KoboldAI-Client>, allows a local or remote installation of a client for such LLM-run games. Some of these games also use Stable Diffusion text-to-image models [[78](https://arxiv.org/html/2402.18659v5#bib.bib78)] to generate visuals accompanying different parts of the narrative. More recent work [[79](https://arxiv.org/html/2402.18659v5#bib.bib79)] investigates how different characterizations of the GM and their way of presenting events to the players impacts the overall experience in TTRPGs. This study presented an online interface for a custom game based on a _Dungeons & Dragons_ (TSR, 1974) fantasy setting. LLMs were ascribed different GM roles through a number of prompts, and human players participated in 45-minute games (split into 3 sub-sessions), with an overall positive response to the automated GM.

In lieu of replacing a human GM, LLMs have also been employed as GM assistants. CALYPSO [[80](https://arxiv.org/html/2402.18659v5#bib.bib80)] is a set of tools running on a Discord server which the GM can query either to generate random encounters, brainstorm ideas, or alternatively chat with a fictional character in a _Dungeons & Dragons_ setting. CALYPSO highlights that hallucinations of GPT-3 can have both positive effects when it generates plausible details not included in descriptions published in the original game manual (e.g. the shapes of creatures’ eyes) and also negative effects when the created details are outright incorrect (e.g. describing the wings of a canonically wingless creature). In addition, the model’s preconditioning to avoid racial bias was found to occasionally prevent it from generating racial details of fantasy creatures in the game. Other work used smaller GPT models to improvise in-game conversations [[81](https://arxiv.org/html/2402.18659v5#bib.bib81)] by monitoring and transcribing verbal exchanges between the GM and the players, and attempting to generate appropriate responses. This example was integrated into Shoelace [[82](https://arxiv.org/html/2402.18659v5#bib.bib82)], which is itself a GM assisting tool helping with content lookup by creating a node-based plan of the game narrative and encounters. The versatility of LLMs given their ability to rapidly process text input paves the way for their integration into the multitude of existing tools and aids for human GMs.

![Refer to caption](extracted/6056081/graphics/2.png) Figure 2: In 1001 Nights [[83](https://arxiv.org/html/2402.18659v5#bib.bib83)], the player uses free-form text to trick the king (role-played by an LLM) into uttering the name of a particular weapon (which will then materialize, allowing the player to defeat him). Image used with permission.

###  3.7 Game Mechanic

Games can also be built around a specific mechanic that relies on LLMs, similar to the AI-based game design patterns identified by Treanor et al. [[84](https://arxiv.org/html/2402.18659v5#bib.bib84)]. An obvious mechanic revolves around the social interactions facilitated by LLM-powered conversational NPCs. In this vein, the Generative Agents project [[45](https://arxiv.org/html/2402.18659v5#bib.bib45)] has employed LLMs to populate a virtual village with 25 characters, enabling them to communicate and engage in social behavior within a sandbox environment. Players were able to interact with these agents via text. The environment state and actions of each agent were stored in a language-based format and summarized in order to retain knowledge for each agent when prompting for its actions. This led to emerging believable social interactions, such as the agents spontaneously inviting other agents to a party which one of them was organizing. Similarly, GoodAI are developing the AI people video game which operates as a sandbox simulation where LLM-powered NPCs “interact with each other and their environment, forming relationships and displaying emotions” [[41](https://arxiv.org/html/2402.18659v5#bib.bib41)]. The player can interact with the agents via natural language chat, triggering reactions and potentially disrupting the relationship between NPCs (see Fig. [1](https://arxiv.org/html/2402.18659v5#S3.F1 "Figure 1 ‣ 3.2 Non-Player Characters ‣ 3 Roles of LLMs in Games ‣ Large Language Models and Games: A Survey and Roadmap")).

Natural language interactions form a natural pool of mechanics to build games around, such as gamifying users’ attempts at jailbreaking LLMs [[85](https://arxiv.org/html/2402.18659v5#bib.bib85)]. The game 1001 nights, depicted in [Figure 2](https://arxiv.org/html/2402.18659v5#S3.F2 "In 3.6 Game Master ‣ 3 Roles of LLMs in Games ‣ Large Language Models and Games: A Survey and Roadmap"), exemplifies this by having an LLM co-create a story from human prompts, where the player’s objective is to try and steer the story to include specific keywords in order for the main character, Scheherazade, to turn these into tangible items in aid of her escape [[83](https://arxiv.org/html/2402.18659v5#bib.bib83)]. Similarly, _Gandalf_ 777<https://gandalf.lakera.ai/> challenges the player to trick an LLM into revealing a password. The game increases the difficulty of the task as levels progress by adjusting the prompt specifications, such as forcing the LLM to re-examine its generated response to ensure it does not include the password.

![Refer to caption](extracted/6056081/graphics/3.png) Figure 3: In _Infinite Craft_ , the player combines what begins as a simple set of atomic elements into increasingly complex entities, with an LLM dictating the product resulting from arbitrary combinations. Image used with permission.

Another strength of LLMs is language synthesis. Huang and Sun [[86](https://arxiv.org/html/2402.18659v5#bib.bib86)] used GPT-3 to generate new words from the combination of two user-selected words. These are used to progress in a text-based game scenario, where the player wins by unlocking goal words. Similarly, language synthesis is leveraged by _Infinite Craft_ 888<https://neal.fun/infinite-craft/>, an “alchemy” game, in which the player combines elements to produce new ones (see Fig. [3](https://arxiv.org/html/2402.18659v5#S3.F3 "Figure 3 ‣ 3.7 Game Mechanic ‣ 3 Roles of LLMs in Games ‣ Large Language Models and Games: A Survey and Roadmap")). In _Infinite Craft_ , the player begins with a set of core elements (water, fire, wind and earth). But while the former have a set of interactions defined manually by the designer, _Infinite Craft_ prompts Llama 2 [[20](https://arxiv.org/html/2402.18659v5#bib.bib20)] to imagine the product of the combination of these elements [[87](https://arxiv.org/html/2402.18659v5#bib.bib87)]. Judging from gameplay, it appears that for each distinct combination, Llama is prompted to produce the result only once, with the product stored in a database for future reference. Thus seemingly anything in the language model’s vocabulary might “emerge” from the combination of these elements, including all 50 states999<https://x.com/FeralFlex/status/1758332430136615298>, “Dream”101010<https://x.com/slutzsmp/status/1760394123243135169>, and the fictional “Super Stonedosaurus Tacosaurus Rex”111111<https://x.com/pvtspicy/status/1759316982984237139>. On occasion, the model can choose to return one of the combined elements, or refuse to combine (e.g. very lengthy or complex) elements.

###  3.8 Automated Designer

A key role of AI in games [[8](https://arxiv.org/html/2402.18659v5#bib.bib8)] is the algorithmic generation of game content such as levels and visuals, or even entire games. Unlike a Game Master who creates a game via natural language—meant to exist in the “theater of the mind” of the players—the aim of procedural content generation (PCG) is to create content intended for use in a digital game and thus it is required to satisfy certain constraints such as playability and aesthetic quality.

Any PCG method that is trained on available content corpora fits under the Procedural Content Generation via Machine Learning (PCGML) paradigm [[23](https://arxiv.org/html/2402.18659v5#bib.bib23)]. Strictly speaking the original PCGML framework of 2018 did not consider LLMs; instead it relied on machine learning methods such as autoencoders and LSTMs. However, important challenges of PCGML remain when considering LLMs for PCG: notably, the reliance on high-quality, machine-readable datasets from human-authored levels. While some datasets exist for arcade game levels [[88](https://arxiv.org/html/2402.18659v5#bib.bib88)], for most games the content remains both unavailable and protected by intellectual property (IP) laws. We revisit this issue in Section [6](https://arxiv.org/html/2402.18659v5#S6 "6 Ethical Issues with LLMs in games ‣ Large Language Models and Games: A Survey and Roadmap"). Prior work in PCG has demonstrated that tile-based game levels can be reliably generated with sequence-based prediction models (e.g. LSTMs) from a modest set of examples, by treating such levels as linear sequences of tile types in raster order [[89](https://arxiv.org/html/2402.18659v5#bib.bib89), [90](https://arxiv.org/html/2402.18659v5#bib.bib90)].

More recently, this approach has been extended to modern LLMs pre-trained on natural language (instead of sequence models trained from the ground up). Todd et al. [[91](https://arxiv.org/html/2402.18659v5#bib.bib91)] fine-tuned a GPT-2 model on a large dataset of _Sokoban_ (Thinking Rabbit, 1982) levels and, at test time, sampled from the model to produce novel puzzles (see Figure [4](https://arxiv.org/html/2402.18659v5#S3.F4 "Figure 4 ‣ 3.8 Automated Designer ‣ 3 Roles of LLMs in Games ‣ Large Language Models and Games: A Survey and Roadmap")). Interestingly, their results indicate that while the GPT-2 model struggles when the fine-tuning dataset is restricted in size, GPT-3 (and, presumably, larger models released since then) are better able to accommodate limited training sets.

![Refer to caption](extracted/6056081/graphics/4.png) Figure 4: A level for the puzzle game Sokoban generated by GPT-3, visualized with the Griddly tileset [[92](https://arxiv.org/html/2402.18659v5#bib.bib92)]. Image used with permission.

Using a similar approach, MarioGPT trains a GPT-2 model on a relatively small dataset of Super Mario Bros (Nintendo, 1985) levels [[93](https://arxiv.org/html/2402.18659v5#bib.bib93)]. MarioGPT overcomes the issue of data scarcity by using the initial dataset as the starting point for an evolutionary algorithm. Existing levels are selected and then sections of the level are mutated by sampling from the GPT model and then correcting the border between the re-generated section and the rest of the level with a similarly-trained BERT (i.e. bi-directional) model [[2](https://arxiv.org/html/2402.18659v5#bib.bib2)]. This approach produces a large and diverse set of playable levels, despite starting from less than 20 levels.

The above GPT-based level generation approaches also show the promise of incorporating natural language instructions to produce conditional level generators, either by prefixing game levels in the training dataset with desired level characteristics [[91](https://arxiv.org/html/2402.18659v5#bib.bib91)] or by embedding user instructions and allowing the model to attend to the embedding during generation [[93](https://arxiv.org/html/2402.18659v5#bib.bib93)]. A recent example of the latter is Cardistry [[94](https://arxiv.org/html/2402.18659v5#bib.bib94)], which used GPT-3.5 to transform a short personal narrative into a set of playing cards. Along with text information, the LLM creates prompts for DALL-E [[95](https://arxiv.org/html/2402.18659v5#bib.bib95)], completing the generation pipeline with playing card art.

User requests however are typically fuzzy and can be misinterpreted by the LLM. Hu et al. [[56](https://arxiv.org/html/2402.18659v5#bib.bib56)] first “refine” user requests for the generation of Minecraft structures using an LLM, adding knowledge specific to the domain (such as block palette and building dimensions). Then, a second LLM is employed to generate the description of the structure that can be interpreted and placed in the game.

While most applications of LLMs as automated designers involve some mapping from the LLM’s natural language output to other game-specific formats, LLMs can naturally be applied to games where the content is itself just natural language. In this vein, LLMs have been leveraged for the generation of New York Times Connections puzzles [[96](https://arxiv.org/html/2402.18659v5#bib.bib96)], which involve having a user group 16 words into 4 groups of 4 words each, with the challenge being for the user to infer what common semantic theme unites each group.

![Refer to caption](extracted/6056081/graphics/5.png) Figure 5: A screenshot of a CrawLLM game instance, generated for the theme of Ancient Egypt [[97](https://arxiv.org/html/2402.18659v5#bib.bib97)]. In CrawLLM, the LLM generates themes, stories, characters, and locations for a card-based dungeon crawler game while Stable Diffusion generates the visuals. Image used with permission.

Unlike the above examples, LLMs can also produce instructions for other LLMs or Foundation Models (FMs) without a user request in the first place. In CrawLLM [[97](https://arxiv.org/html/2402.18659v5#bib.bib97)], the Mixtral 8x7B LLM [[98](https://arxiv.org/html/2402.18659v5#bib.bib98)] generates the theme, visual style, and even enemy descriptions which act as blueprints to guide additional LLM queries for producing player-facing narrative (e.g. introductory text) or textures and animations via Stable Diffusion [[78](https://arxiv.org/html/2402.18659v5#bib.bib78)] for re-theming a dungeon crawler game with card-based combat mechanics. In CrawLLM, the game design (code and card details) are pre-authored by the human, while LLMs and foundation models re-theme the assets to provide a visually and narratively novel experience every time (see Fig. [5](https://arxiv.org/html/2402.18659v5#S3.F5 "Figure 5 ‣ 3.8 Automated Designer ‣ 3 Roles of LLMs in Games ‣ Large Language Models and Games: A Survey and Roadmap")).

Finally, LLMs can generate new games by writing game code directly. For instance, the GAVEL system [[99](https://arxiv.org/html/2402.18659v5#bib.bib99)] combines evolution with LLMs to generate board games. In GAVEL, a language model fine-tuned on a dataset of board game programs written in the Ludii description language [[100](https://arxiv.org/html/2402.18659v5#bib.bib100)] acts as the mutation operator within the evolutionary loop.

###  3.9 Design Assistant

An AI for design assistance can provide several benefits to the creative process. Depending on the type of tool, type of AI, and type of creative process, the AI can minimize development time and cost, reduce human effort, support collaboration among members of a design team, or elicit a user’s creativity [[101](https://arxiv.org/html/2402.18659v5#bib.bib101)]. So far, in games, most of the AI-powered design assistant tools focus on autocompleting a human’s in-progress design [[102](https://arxiv.org/html/2402.18659v5#bib.bib102)] or providing many possible suggestions for the designer to consider [[103](https://arxiv.org/html/2402.18659v5#bib.bib103), [104](https://arxiv.org/html/2402.18659v5#bib.bib104), [105](https://arxiv.org/html/2402.18659v5#bib.bib105), [106](https://arxiv.org/html/2402.18659v5#bib.bib106), [107](https://arxiv.org/html/2402.18659v5#bib.bib107), [108](https://arxiv.org/html/2402.18659v5#bib.bib108)]. Ideally, we would want an LLM that can act as a human colleague that we can bounce off ideas to and collaborate with. Such LLMs are still beyond the current state of the art [[109](https://arxiv.org/html/2402.18659v5#bib.bib109)]. Existing tools implement co-creating [[110](https://arxiv.org/html/2402.18659v5#bib.bib110)] LLMs at different levels of control, which we can explore under the existing Co-Creative Framework for Interaction Design [[111](https://arxiv.org/html/2402.18659v5#bib.bib111)]. Focusing on the interaction over the artifact, the LLM can be of conceptual assistance, providing high-level guidance which is not game-ready. This would require that the designer adapts and curates the AI output in a way that fits their own vision and the constraints of the game. This implies that the LLM contributes very little in the actual artifact generation, merely providing new suggestions. By allowing the LLM to also refine or transform the artifact, we see a procedural assistance by the LLM. Interacting with the designer, the LLM can produce increasingly more final versions of the intended artifact. The LLM is expected to understand the context of the game for which the content is intended, in order to provide meaningful assistance. However, the LLM does not need to produce a final, playable artifact but could instead simply provide the next creative step for discussion with the designer [[112](https://arxiv.org/html/2402.18659v5#bib.bib112)]. Moreover, the designer is ultimately responsible for curating and adapting the generated content, as well as deciding when the co-creative process is completed [[113](https://arxiv.org/html/2402.18659v5#bib.bib113), [114](https://arxiv.org/html/2402.18659v5#bib.bib114), [115](https://arxiv.org/html/2402.18659v5#bib.bib115)]. Finally, if the LLM is allowed to _directly_ create and alter the artifact based on user requirements, we say it provides production assistance. This is the closest level to PCG (see Section [3.8](https://arxiv.org/html/2402.18659v5#S3.SS8 "3.8 Automated Designer ‣ 3 Roles of LLMs in Games ‣ Large Language Models and Games: A Survey and Roadmap")), but is different in that the designer remains in control and can refine their specification or reject a created artifact (versus an autonomous generator which directly sends content to the player). As expected, however, the AI operates in a much more constrained space in this scenario as it must account for all other game mechanics (the design of which are presumed finalized) and designer goals which are somehow encoded or presumed via learned designer models [[116](https://arxiv.org/html/2402.18659v5#bib.bib116)].

One can argue that existing interfaces with LLMs and Large Multimodal Models (LMMs) act as design assistants. The designer provides their specifications and receives one (in LLMs) or multiple (in AI image generators) suggestions that they can further refine. Many creatives report using such interfaces for brainstorming and concept development [[117](https://arxiv.org/html/2402.18659v5#bib.bib117)], including game developers [[118](https://arxiv.org/html/2402.18659v5#bib.bib118)]. However, the applicability of LLMs as design assistants is somewhat limited, reverting only to conceptual assistance. Similarly, their potential for refining an existing idea (i.e. offering procedural assistance) is underexplored, as we discuss in Section [4](https://arxiv.org/html/2402.18659v5#S4 "4 A Roadmap for Future Applications of LLMs in Games ‣ Large Language Models and Games: A Survey and Roadmap").

Conceptual assistance is thus the easiest for LLMs, and is the first case explored in games. Charity et al. [[107](https://arxiv.org/html/2402.18659v5#bib.bib107)] envision design assistance as a tool that combines the game description provided by the user with existing knowledge of similar games to suggest possible game features back to the user. The suggested features are fairly generic, few-word guidelines (e.g.: “learn new combat”) which would need extensive design effort and creativity to transform into an implementable and coherent game design. When asking an LLM for specific game features to implement in a digital game, players found them less compelling than human-designed ones [[119](https://arxiv.org/html/2402.18659v5#bib.bib119)]. The suggestions, however, were still useful for game designers, as they provided a different perspective that could kickstart their creation process. A thorough analysis of strengths and weaknesses of LLMs in this role can be drawn from Project AVA [[120](https://arxiv.org/html/2402.18659v5#bib.bib120)], a non-commercial digital game developed at Keywords Studios with the assistance of LLMs and LMMs for multiple aspects of the typical game development pipeline. LLMs are shown to help greatly in giving inspiration to the designers (albeit not being creative themselves), provide simple starting code for the game logic, and even assist developers by revealing errors in the code during development. LLMs however often fall short on anything more involved or requiring further domain knowledge, such as requesting code for specific game logic or understanding functionality in UI elements. Similarly, LMMs required a lot of tuning by human artists, but provided a solid foundation for concept art and proof-of-concept user interfaces.

Since production assistance is also close to traditional PCG pipelines, it is also understandably explored for games. Nasir and Togelius [[121](https://arxiv.org/html/2402.18659v5#bib.bib121)] used GPT-3 to generate levels for the Metavoidal (Yellow Lab Games, 2022) brawler game from a prompt that describes the level’s features (e.g. width and height) while a human curates and edits the results to ensure playability. This curated set of levels is then used for further fine-tuning, potentially automating the generative process. Kelly et al. [[122](https://arxiv.org/html/2402.18659v5#bib.bib122)] instead use GPT-4 to generate stories in natural language while abiding to logic constraints, assisting story writers. Instead of generating the final artifact directly, Kumaran et al. [[123](https://arxiv.org/html/2402.18659v5#bib.bib123)] use the LLM to parse game level parameters from the user request. The natural language input to the LLM contains information about both what the user is looking for and properties that the level of Future Worlds—a game-based learning exhibit about environmental sustainability [[124](https://arxiv.org/html/2402.18659v5#bib.bib124)]—should posses, such as difficulty, size, type of sustainability problem, and simulation goal metrics. A collection of candidate game levels are generated from a single user request. Each candidate is then evaluated by a RL agent, and the best game is presented to the designer via the Unity game engine.

![Refer to caption](extracted/6056081/graphics/6.png) Figure 6: A screenshot of the LLMaker digital game content design assistant tool [[125](https://arxiv.org/html/2402.18659v5#bib.bib125)]. In LLMaker, a human designer makes requests in natural language to alter a dungeon level for a video game, editing both the level layout and the entities within. The LLM satisfies the designer requests by calling the appropriate function, ensuring that the content adheres to domain constraints. Image used with permission.

Examples of procedural assistance are very few: LLMaker [[125](https://arxiv.org/html/2402.18659v5#bib.bib125)] (see [Figure 6](https://arxiv.org/html/2402.18659v5#S3.F6 "In 3.9 Design Assistant ‣ 3 Roles of LLMs in Games ‣ Large Language Models and Games: A Survey and Roadmap")) is a chat-only level editor where the user can chat with the LLM and request changes to be made to the game level and its content. Natural conversation with the LLM allows the designer to explore the current state of the level (for example, the user may ask “What are the enemies in the entrance room?”) or draw inspiration for the next changes to make (for example, by asking “What kind of enemies should I add to this corridor?”). When the user requests a change to be made to the level, the LLM translates their request into a valid function call, ensuring content consistency and adherence to domain constraints [[126](https://arxiv.org/html/2402.18659v5#bib.bib126)]. Parameters for the function call that have not been specified by the user are generated by the LLM itself, biased by the existing level and overall user preferences. For example, if the user asks “Change the name of the banshee to Scary Ghost” the LLM will not only change the enemy’s name but also their description to reflect the change, even though this was not specified explicitly by the user (see Fig. [6](https://arxiv.org/html/2402.18659v5#S3.F6 "Figure 6 ‣ 3.9 Design Assistant ‣ 3 Roles of LLMs in Games ‣ Large Language Models and Games: A Survey and Roadmap")) In addition to the level and its content, LLMaker employs Stable Diffusion models to generate the graphical assets for the artifacts, based on descriptions generated by the LLM itself. As noted above, so far research has mostly focused on either LLMs for conceptual assistance (putting significant onus on a human designer) or as production assistance (leveraging a human designer as curator). The conversational nature of LLMs, however, seems particularly well-suited for procedural assistance when designing content; we revisit this underexplored area of research in Section [4](https://arxiv.org/html/2402.18659v5#S4 "4 A Roadmap for Future Applications of LLMs in Games ‣ Large Language Models and Games: A Survey and Roadmap").

##  4 A Roadmap for Future Applications of LLMs in Games

The previous section attempted to group current research in LLMs and games into a typology focused on the roles an LLM is asked to play. As part of this exercise, we identified a number of roles that have been heavily researched. Unsurprisingly, the role of player and automated designer have received most attention: this matches the general trends within AI and games research more broadly [[8](https://arxiv.org/html/2402.18659v5#bib.bib8)]. Following general trends in Game AI for playing or generating content, LLM-based approaches are likely to flourish via community events, benchmarks and competitions, with first steps already being taken in this direction [[127](https://arxiv.org/html/2402.18659v5#bib.bib127)]. Based on the roles listed in Section [3](https://arxiv.org/html/2402.18659v5#S3 "3 Roles of LLMs in Games ‣ Large Language Models and Games: A Survey and Roadmap"), we identify below some gaps found in the literature, and lay out possible research directions that leverage the power of LLMs in new ways.

While academic interest in design assistance within games has blossomed in the last decade, we find that the potential of LLMs has so far been underutilized. LLM design assistants either ask too much of a human designer in terms of creative interpretation and actual development [[107](https://arxiv.org/html/2402.18659v5#bib.bib107)] or too little, demoting them to content curator [[121](https://arxiv.org/html/2402.18659v5#bib.bib121)]. Past research in mixed-initiative systems [[110](https://arxiv.org/html/2402.18659v5#bib.bib110)] assumes a more co-creative initiative from both human and machine, and the power of LLMs as conversational agents matches the original vision of a creative dialogue between initiatives [[115](https://arxiv.org/html/2402.18659v5#bib.bib115)]. Therefore, a promising unexplored direction lies in a more procedural assistance (see Section [3.9](https://arxiv.org/html/2402.18659v5#S3.SS9 "3.9 Design Assistant ‣ 3 Roles of LLMs in Games ‣ Large Language Models and Games: A Survey and Roadmap")) where the LLM not only produces output but also reasons about it to the human designer. LLMs seem especially well-suited for this task, as the context is retained and the designer can iteratively refine past products that the LLM has generated. However, concerns of LLMs’ limited memory may arise (see Section [5](https://arxiv.org/html/2402.18659v5#S5 "5 Limitations of LLMs in games ‣ Large Language Models and Games: A Survey and Roadmap")) in long-term design processes. On the other hand, iterative refining is not as straightforward for other state-of-the-art technologies such as LMMs, despite some promising results via e.g. InstructPix2Pix [[128](https://arxiv.org/html/2402.18659v5#bib.bib128)]. It is expected that such applications will raise new challenges in terms of hallucinations, explainability [[129](https://arxiv.org/html/2402.18659v5#bib.bib129)], capturing or modeling designer intent [[116](https://arxiv.org/html/2402.18659v5#bib.bib116)], and more. We discuss such challenges further in Section [6](https://arxiv.org/html/2402.18659v5#S6 "6 Ethical Issues with LLMs in games ‣ Large Language Models and Games: A Survey and Roadmap").

While we identified player assistance as an important role that LLMs can play, we have found little work that targets any aspect of this beyond re-theming designer-defined hints [[57](https://arxiv.org/html/2402.18659v5#bib.bib57)]. The conversational ability of LLMs make them ideally suited for tutorial writing or hint-giving, especially in short snippets as provided e.g. by a conversational agent. However, it is important to note that LLMs often hallucinate or overfit to the corpus they have been trained on, and may be challenged, for instance, to summarize or lookup specific rules given a game manual. Similar limitations were identified when using an LLM as assistant to a human Game Master [[81](https://arxiv.org/html/2402.18659v5#bib.bib81)], where the LLM could not find information in the pre-written adventure when asked a question about the scene. Other technologies (as simple as a database search query) could be used instead, with the LLM undertaking only the task of converting the found information into a natural language utterance. Beyond mere hint-giving, however, an LLM could also act as a more hands-on player assistant, taking over more trivial tasks (such as managing minutiae of one city in a strategy game). This is also powerful for Game Master assistance, as the LLM can keep track of locations visited and NPCs met, or look up rules. In both cases, addressing the issue of hallucinations and consistency will need to be addressed, which we review in Section [5](https://arxiv.org/html/2402.18659v5#S5 "5 Limitations of LLMs in games ‣ Large Language Models and Games: A Survey and Roadmap").

Another role seemingly well-suited for LLMs that has received limited attention is that of commentator or reteller. Work so far has focused on automating the commentary of streamers or eSport casters [[62](https://arxiv.org/html/2402.18659v5#bib.bib62)]. While this direction is still largely uncharted, there are more directions that could leverage LLMs for streamer assistance rather than automation (and replacement). Rather than narrate events occurring within the game (or video stream), LLMs can summarize the audience interactions and engagement levels —thus acting as a commentator not of the game but of the audience watching it. This could allow a human streamer to better keep track of topics discussed in the chat, and engage as needed without having to read every comment. While this has been identified as a research direction for AI already [[130](https://arxiv.org/html/2402.18659v5#bib.bib130)], it has yet to be implemented. Under the role of streamer assistance, issues of explainability of the LLM’s commentary would become pertinent (e.g. to address one audience member by name); we revisit this in Section [5](https://arxiv.org/html/2402.18659v5#S5 "5 Limitations of LLMs in games ‣ Large Language Models and Games: A Survey and Roadmap"). It is worth noting that streamers have already begun to explore AI assistance to their streams. To the best of our knowledge, the most relevant example is YouTube user Criken who plays alongside an AI assistant as a conversational agent, having a dialogue with an otherwise out-of-the-box LLM either embedded within the game121212<https://youtu.be/dQ-7-r5aM1U> or as part of the stream131313<https://youtu.be/KhE9NhUqtBc>. This example is not explicitly targeted for reacting to in-game events or audience interaction (and acts more as NPC than commentator) but indicates that some streamers are open to the use of such technology for their craft.

Despite a few attempts to leverage LLMs for games user research (see Section [3.5](https://arxiv.org/html/2402.18659v5#S3.SS5 "3.5 Analyst ‣ 3 Roles of LLMs in Games ‣ Large Language Models and Games: A Survey and Roadmap")), there is much unexplored potential in this direction. LLMs so far are used to cluster gameplay logs [[69](https://arxiv.org/html/2402.18659v5#bib.bib69), [73](https://arxiv.org/html/2402.18659v5#bib.bib73)], but they could also explain the groupings in the form of e.g. play personas [[131](https://arxiv.org/html/2402.18659v5#bib.bib131), [71](https://arxiv.org/html/2402.18659v5#bib.bib71)] described in natural language. Such a task would raise issues of explainability and privacy more broadly (see Section [5](https://arxiv.org/html/2402.18659v5#S5 "5 Limitations of LLMs in games ‣ Large Language Models and Games: A Survey and Roadmap")), and would likely still involve a game designer or user researcher in the loop for quality assurance. More importantly, moving from these log-based clusters towards capturing the player’s experience or emotional state [[132](https://arxiv.org/html/2402.18659v5#bib.bib132)] remains an open challenge. In principle, an LLM could predict affective state transitions such as “the game is more engaging now” and thereby adapt the game environment to elicit a supposedly more engaging experience for the player. Learning such transitions builds on the experience-driven procedural content generation paradigm [[133](https://arxiv.org/html/2402.18659v5#bib.bib133)] but with an LLM acting as the player experience model. Future research could explore how LLMs can be fine-tuned to represent and infer player experience transitions based on in-game observations and demonstrations of experience. Two challenges need to be addressed for this: (a) representations of game states as natural language and (b) hallucinations of human experience. For the former challenge, potentially leveraging work of LLM players (see Section [3.1](https://arxiv.org/html/2402.18659v5#S3.SS1 "3.1 Player ‣ 3 Roles of LLMs in Games ‣ Large Language Models and Games: A Survey and Roadmap")) and how they pass the game state to the LLM seems a promising first step. For the latter challenge, however, current LLMs struggle to capture user intent during conversation —let alone more ill-defined concepts such as players’ emotion or engagement [[134](https://arxiv.org/html/2402.18659v5#bib.bib134)]. Current datasets on affect in games are formatted as continuous or categorical variables, often fluctuating over time [[135](https://arxiv.org/html/2402.18659v5#bib.bib135)], which would be challenging to format as text without processing. While perhaps using language as input or output for the player model requires some innovative pre-processing or more advanced LLM technologies, the underlying GPT architecture shows promise already. Broekens _et al._ showed that ChatGPT could detect emotion in English text [[136](https://arxiv.org/html/2402.18659v5#bib.bib136)], although admittedly games include many more modalities (e.g. visuals, audio) than pure text, which is mainly relegated to narrative [[137](https://arxiv.org/html/2402.18659v5#bib.bib137)]. We expect more research on player modeling powered by transformers, if not LLMs directly, such as leveraging behavior transformers [[138](https://arxiv.org/html/2402.18659v5#bib.bib138)] to imitate human playtraces grouped by playstyle [[139](https://arxiv.org/html/2402.18659v5#bib.bib139)].

To wrap up, we believe that every role an LLM could be called to play in (or around) a game identified in Section [3](https://arxiv.org/html/2402.18659v5#S3 "3 Roles of LLMs in Games ‣ Large Language Models and Games: A Survey and Roadmap") could benefit from additional attention. This technology remains nascent and changes are forthcoming which may address several limitations we identified above and more extensively in Section [5](https://arxiv.org/html/2402.18659v5#S5 "5 Limitations of LLMs in games ‣ Large Language Models and Games: A Survey and Roadmap"). The natural language capabilities (especially for text generation) make LLMs ideal conversational assistants (for a player, a designer, a GM, or a streamer). The ability of LLMs to consume and reason from text corpora also opens new possibilities for automated design moving beyond tile-based level generation (which needs carefully crafted corpora) and more towards open-ended content such as game narratives [[140](https://arxiv.org/html/2402.18659v5#bib.bib140), [141](https://arxiv.org/html/2402.18659v5#bib.bib141), [142](https://arxiv.org/html/2402.18659v5#bib.bib142), [143](https://arxiv.org/html/2402.18659v5#bib.bib143), [144](https://arxiv.org/html/2402.18659v5#bib.bib144), [145](https://arxiv.org/html/2402.18659v5#bib.bib145)] or even game design documents. The potential of LLMs in that regard is already voiced by many evangelists in the field, but research on actual implementation of such ideas and on addressing the IP concerns they may raise (see Section [6](https://arxiv.org/html/2402.18659v5#S6 "6 Ethical Issues with LLMs in games ‣ Large Language Models and Games: A Survey and Roadmap")) are still forthcoming.

While the focus of this paper is on what LLMs can do for games, we do not underestimate what games can do for LLMs. One of the watershed moments for AI and games research was the article by Laird and Van Lent naming games as the “killer app” for human-level AI [[146](https://arxiv.org/html/2402.18659v5#bib.bib146)]. This remains true for LLMs today: games are ideally poised for LLM research. Not only do games produce rich multimodal data (ideal for e.g. LMMs), but there also exist rich corpora of text and multimodal data produced by players, viewers, fans, etc. Game text data, such as transcripts, have already been used to train LLM players [[39](https://arxiv.org/html/2402.18659v5#bib.bib39), [40](https://arxiv.org/html/2402.18659v5#bib.bib40)]. On the other hand, LLMs struggle with both spatial reasoning and planning by their very nature, while most games rely heavily on both aspects. From strategy board games and digital games (where long-term planning is crucial) to first-person shooters (which hinge on precision in spatial reasoning and a reactive plan for reaching the enemy base), such games remain state-of-the-art testbeds for gameplaying AI [[147](https://arxiv.org/html/2402.18659v5#bib.bib147), [148](https://arxiv.org/html/2402.18659v5#bib.bib148)] and will likely be fraught arenas for LLM research. Games also hinge on long-term interactions, especially in the case of LLM-based GMs (see Section [3.6](https://arxiv.org/html/2402.18659v5#S3.SS6 "3.6 Game Master ‣ 3 Roles of LLMs in Games ‣ Large Language Models and Games: A Survey and Roadmap")). Games can thus form testbeds or benchmarks to explore the limits of recollection under different context lengths, a critical limitation of LLMs detailed in Section [5](https://arxiv.org/html/2402.18659v5#S5 "5 Limitations of LLMs in games ‣ Large Language Models and Games: A Survey and Roadmap"). In terms of game design tasks, we also note that games are complex constrained problems, with hard constraints on e.g. levels that can be completed [[149](https://arxiv.org/html/2402.18659v5#bib.bib149)], but also soft constraints regarding game balance between competing players in multi-player games [[150](https://arxiv.org/html/2402.18659v5#bib.bib150), [139](https://arxiv.org/html/2402.18659v5#bib.bib139)], or the progression and pacing of a single-player experience [[102](https://arxiv.org/html/2402.18659v5#bib.bib102)]. While some LLMs can handle some hard constraints via, say, function calling [[126](https://arxiv.org/html/2402.18659v5#bib.bib126)], this may not be possible for more complex or more constrained game domains. Moreover, soft constraints would need to be conveyed to the LLM in more nuanced ways. Game benchmarks specific to LLMs have already started to emerge [[127](https://arxiv.org/html/2402.18659v5#bib.bib127)], but identifying critical game-based challenges for LLMs, appropriate and interesting benchmarks, and (ethically sourced) data for training or fine-tuning LLMs remains an open question.

##  5 Limitations of LLMs in games

Large language models have exciting potential for games, but they also come with inherent limitations. Mainly, LLMs suffer from hallucinations [[50](https://arxiv.org/html/2402.18659v5#bib.bib50), [151](https://arxiv.org/html/2402.18659v5#bib.bib151)], meaning that they will output plausible but false statements simply because they are a probable sequence of words. Hallucinations are inevitable, given how the world is described to the machine [[152](https://arxiv.org/html/2402.18659v5#bib.bib152)]; LLMs lack grounding, so the text they generate is detached from constraints of reality. Yet LLMs always “act” confidently in their responses, even when wholly mistaken. Indeed, Hicks et al. argue that the term AI hallucinations misrepresents how “the models are in an important way indifferent to the truth of their outputs” [[7](https://arxiv.org/html/2402.18659v5#bib.bib7)]. LLMs are shown to also output responses that are wrong even though the LLM has access to information that proves otherwise [[153](https://arxiv.org/html/2402.18659v5#bib.bib153), [154](https://arxiv.org/html/2402.18659v5#bib.bib154), [155](https://arxiv.org/html/2402.18659v5#bib.bib155)]. In the context of digital games, these limitations affect certain applications of LLMs more than others, for example NPCs may hallucinate quests that do not exist in the game, or a player assistant may provide suggestions to the user based on wrong assumptions.

Another limitation is that LLMs sometimes struggle to capture user intent. This is especially evident with expressions of sarcasm [[156](https://arxiv.org/html/2402.18659v5#bib.bib156)]. The ability to capture user intent is important for applications of LLMs that converse directly with the player. Many LLMs misunderstand user requests [[157](https://arxiv.org/html/2402.18659v5#bib.bib157)], and clarifying to the LLM multiple times leads to frustration. This limitation is most relevant to cases where the LLM is in direct conversation with the user, e.g. as design assistant, player assistant, or Game Master. Depending on how much the LLM output controls the user experience (e.g. as Game Master or offering production assistance to a human designer), the inability to capture user intent can be a frustrating experience.

On a larger scale, LLMs suffer from losing context, and struggle with continuity. This is because the “memory” of an LLM is constrained by its context size, which limits the extent of its inputs and outputs, as well as its response time due to the attention mechanism [[18](https://arxiv.org/html/2402.18659v5#bib.bib18)]. The longer the conversation, the less likely it is that the LLM will recall early events [[158](https://arxiv.org/html/2402.18659v5#bib.bib158)]. In digital games, it is possible to separately summarize the game events (see [Section 3.4](https://arxiv.org/html/2402.18659v5#S3.SS4 "3.4 Commentator/Reteller ‣ 3 Roles of LLMs in Games ‣ Large Language Models and Games: A Survey and Roadmap")) and process them as part of the input to the LLM. As a game progresses past a few game sessions, however, this summary may still be too long, or details of increasing significance will be omitted, thus leading to a degraded performance. This is especially relevant for roles requiring long-term engagement, such as LLM-powered retellers or Game Masters. In _Infinite Craft_ (see Section [3.7](https://arxiv.org/html/2402.18659v5#S3.SS7 "3.7 Game Mechanic ‣ 3 Roles of LLMs in Games ‣ Large Language Models and Games: A Survey and Roadmap")), this is handled by an external database that stores and looks up past combination rules, ensuring consistency in future uses of the same mechanic. However, LLMs could theoretically tackle this issue directly.

Recent models have tried to address this recollection issue by increasing the context length, with some of the larger models encompassing 128K or even 10M tokens [[159](https://arxiv.org/html/2402.18659v5#bib.bib159)]. Despite this being adequate for a wide range of applications, it may still fall short when applied to long-term tracking of game states. In particular, massive multiplayer online games offer a simulation space with a large intricate domain of actions and interactions, which scales exponentially with the number of agents (players or otherwise) participating. Researchers have also tried to address the context limit by including compressive memory into the attention mechanism of the LLM [[160](https://arxiv.org/html/2402.18659v5#bib.bib160)], in an attempt to create a seemingly infinite context length. The authors of [[160](https://arxiv.org/html/2402.18659v5#bib.bib160)], however, acknowledge its current limitation, partly due to the difficulty of selecting and compressing the data which should be “memorized”. A different approach proposed by Fountas et al. [[161](https://arxiv.org/html/2402.18659v5#bib.bib161)] draws inspiration from cognitive science to equip LLMs with episodic memory, greatly reducing context length limitations during information retrieval.

A Retrieval-Augmented Generation (RAG) system [[162](https://arxiv.org/html/2402.18659v5#bib.bib162)] could address this limitation, drawing from a database containing vector representations or other latent representations of pertinent text or data. When the text generator processes a sequence, the RAG system would retrieve similar entries from this external data source. This would hypothetically provide a streamlined archive of game events and actions for the LLMs to consult in order to generate a consistent narrative progression.

Another challenge is that currently LLMs are trained to be highly compliant to the users’ requests. For an LLM assistant, this is not a cause for concern, but in the role of a Game Master this can create issues. Human GMs frequently curb the more exotic player requests which could drastically diverge from the game narrative or which would result in an unrecoverable disruption of a required sequence of game events. An LLM Game Master would try to accommodate for even the most bizarre requests, with little consideration for the consequential impacts to any predetermined game events.

Yet another limitation of LLMs that prevents their application in mainstream media is their cost. Running AAA games and LLMs in parallel on consumer hardware is infeasible [[163](https://arxiv.org/html/2402.18659v5#bib.bib163)] due to their computation requirements. If one wants to integrate LLMs in games, they would have to host the models on their own servers or access existing models via APIs. Additionally, the cost of querying LLMs is a recurring cost, and cannot be properly estimated beforehand. This kind of problem is also affected by the scale of LLMs-powered games or tools. Similarly to how server costs increase with the number of active players in massive multiplayer online games, the more players use a LLM over multiple play sessions, the more the game developers or publishers will have to bear the financial burden. The monetary cost of this approach can be prohibitive or difficult to estimate for real-world applications. The game need not even be played by other players: to evaluate the performance of their simulations with multiple LLM-based NPCs, Park et al. ran simulations for several days with a cost of “thousands of dollars in token credits” [[45](https://arxiv.org/html/2402.18659v5#bib.bib45)]. While promising techniques to reduce the costs of running LLMs exist [[164](https://arxiv.org/html/2402.18659v5#bib.bib164), [165](https://arxiv.org/html/2402.18659v5#bib.bib165)], these are not yet widespread and require further engineering to set them up properly.

Perhaps due to the above limitations, the implementation and deployment of LLMs in digital game applications is still very limited. A digital game is a domain where responsiveness is vital for players, so it follows that LLMs should also be able to provide their responses quickly. Unfortunately, while research on more efficient and faster architectures is being carried out [[166](https://arxiv.org/html/2402.18659v5#bib.bib166)], the real-time application of LLMs is still not plausible. This is especially evident in other domains such as design applications, where “real time” responses are generated in around 30 seconds to over a minute [[167](https://arxiv.org/html/2402.18659v5#bib.bib167)].

##  6 Ethical Issues with LLMs in games

With the improvement of AI methods applied to games over the recent years, many questions regarding their ethics and real-world impact have been raised [[168](https://arxiv.org/html/2402.18659v5#bib.bib168)]. Using LLMs raises ethical issues regarding sustainability, copyright, explainability, privacy, and biases. Naturally, each of these issues has serious implications in the field of games.

The reliance of LLMs on training data and training time raises concerns regarding their carbon footprint. Beyond training costs, inference over the model’s lifespan has a greater environmental impact due to constant querying [[169](https://arxiv.org/html/2402.18659v5#bib.bib169), [170](https://arxiv.org/html/2402.18659v5#bib.bib170)]. Factors like renewable and local energy, better model architectures, and more meaningful (and thus less wasteful) training data can mitigate this. In the context of LLMs for digital games, sustainability remains crucial, considering the carbon footprint of frequent queries during gameplay (e.g. for Game Master or NPC responses, or for LLM-powered players). This is especially pertinent if the LLM is intended to run locally, on consumer-level hardware which are usually powered by non-renewable sources.

When it comes to copyright, issues apply to the input data, the output data, and the model itself. LLMs trained on data under copyright is an unfortunate common practice [[171](https://arxiv.org/html/2402.18659v5#bib.bib171)], deservedly raising public outrage [[172](https://arxiv.org/html/2402.18659v5#bib.bib172), [173](https://arxiv.org/html/2402.18659v5#bib.bib173)]. The models themselves have different copyright licenses applied, which can also lead to artifacts they generate to fall under the public domain [[19](https://arxiv.org/html/2402.18659v5#bib.bib19), [174](https://arxiv.org/html/2402.18659v5#bib.bib174)]. For the game industry, matters of IP and copyright are extremely important. This is as much a concern regarding having the company’s copyrighted content somehow used as training data by competitors, as it is about LLMs producing material that the company cannot copyright. It is important to note here that, at least when it comes to the latter concern, the role the LLM takes is very pertinent. If an LLM or LMM produces content automatically (see Section [3.8](https://arxiv.org/html/2402.18659v5#S3.SS8 "3.8 Automated Designer ‣ 3 Roles of LLMs in Games ‣ Large Language Models and Games: A Survey and Roadmap")), past legal consensus in the USA indicates that the material can not be copyrighted [[175](https://arxiv.org/html/2402.18659v5#bib.bib175)]. If an LLM or LMM acts as an “assistive tool” [[176](https://arxiv.org/html/2402.18659v5#bib.bib176)] to a designer (especially for conceptual assistance, see Section [3.9](https://arxiv.org/html/2402.18659v5#S3.SS9 "3.9 Design Assistant ‣ 3 Roles of LLMs in Games ‣ Large Language Models and Games: A Survey and Roadmap")) then the extensive and impactful human effort needed to transform these concepts into game design and game art likely makes the final product eligible for copyright [[176](https://arxiv.org/html/2402.18659v5#bib.bib176)]. The limited rulings in copyright courts regarding this, however, and the “likely” caveat we include in our own text, understandably would make game companies hesitant to tread in untested waters for major game IPs beyond e.g. small-scale indie productions [[120](https://arxiv.org/html/2402.18659v5#bib.bib120), [83](https://arxiv.org/html/2402.18659v5#bib.bib83)]. For researchers, however, the ethical issues of copyright breach and exploitation by large corporations, and the public outcry for the above, leave a bad taste and make research in LLMs less palatable [[177](https://arxiv.org/html/2402.18659v5#bib.bib177)].

In applications, understanding how a final result or product is reached is extremely crucial, particularly when a product is iteratively refined as with design assistants (see Section [3.9](https://arxiv.org/html/2402.18659v5#S3.SS9 "3.9 Design Assistant ‣ 3 Roles of LLMs in Games ‣ Large Language Models and Games: A Survey and Roadmap")). This is a problem of explainability [[129](https://arxiv.org/html/2402.18659v5#bib.bib129)], whereas LLMs are inherently opaque in their generation process. Liu et al. [[178](https://arxiv.org/html/2402.18659v5#bib.bib178)] highlight different methods to improve the explainability of language models, such as concept-based explanations or saliency maps. Particularly for LLMs, the self-explanation applied via the chain-of-thought [[179](https://arxiv.org/html/2402.18659v5#bib.bib179)] reasoning has received attention by the research community [[180](https://arxiv.org/html/2402.18659v5#bib.bib180), [181](https://arxiv.org/html/2402.18659v5#bib.bib181)]. While this method adds a layer of explained reasoning to the generated output, there are multiple examples in the literature that demonstrate how this reasoning may just be an illusion of reasoning capabilities. Such examples include disregarding the provided reasoning in the final output [[182](https://arxiv.org/html/2402.18659v5#bib.bib182)], or reaching the correct solution via incorrect steps in math problems [[183](https://arxiv.org/html/2402.18659v5#bib.bib183)]. In the domain of games, explainability is paramount across roles, ensuring gameplay coherence and user engagement.

Replicability of an application’s behavior is equally crucial. When applying LLMs to digital games (especially as game mechanics, NPCs, or automated designers) one would expect their output quality to not change over time. This is not the case for closed-source LLMs: even when using the same model name, the same request at one time can generate content that is vastly different from a past iteration [[184](https://arxiv.org/html/2402.18659v5#bib.bib184)]. In this case, developers may have to consider switching to open-source models with open weights, such as those hosted on the popular HuggingFace Transformers library [[185](https://arxiv.org/html/2402.18659v5#bib.bib185)]. An additional benefit of switching to self-hosted models, model size notwithstanding, is the additional level of privacy that is guaranteed to the users. Querying local models ensures all messages remain within the application, whereas interacting with models hosted via APIs entails that conversations are exchanged over third-party websites. A developer might be willing to share conversation logs with an API provider141414Such as ChatGPT, which shares conversations by default unless opted out for model improvement. However, users may not be aware of this practice or its implications. Local deployment of LLMs has been democratized by making models more accessible on lower-end hardware, relying on the widespread adoption of the GGUF format151515Details of this file format are available at <https://github.com/ggerganov/ggml/blob/master/docs/gguf.md> and the release of different versions of the same model at varying degrees of quantization [[186](https://arxiv.org/html/2402.18659v5#bib.bib186)]. The quantization of an LLM usually results in a loss of performance, but this is usually considered a valid trade-off for the reduced model size to load on VRAM. Combined with friendly APIs for running LLMs locally, such as Open WebUI161616<https://openwebui.com/> and LM Studio171717<https://lmstudio.ai/>, it is possible to run LLMs in a more controlled fashion. The more pertinent technological breakthroughs lie in compacting size and carbon footprint while retaining high-quality LLM outputs.

Finally, biases emerge as LLMs are trained on a large corpus, usually scraped from the (Western-focused part of the) internet. This allows models to capture a current reality snapshot, which is advantageous for a conversational or question-answering model, though it requires curating this data from different kinds of biases. Some biases, such as social stereotypes, could be targeted and alleviated; others, such as exclusionary norms, pose greater challenges. In games, we identify two main concerns when interacting with an LLM: toxic behavior, and stereotypes or incorrect notions. Toxic behavior is a harmful property that a language model may learn from its training corpus, which often contains text from community-based fora or social platforms. Tools that combat toxic language in digital games are constantly evolving, with some even blocking chat messages before they are delivered to the user [[187](https://arxiv.org/html/2402.18659v5#bib.bib187), [188](https://arxiv.org/html/2402.18659v5#bib.bib188)]. Therefore, similar applications could theoretically be developed to target toxic outputs from language models. Unlike human players, however, when an LLM plays the role of an NPC, it should align with the game themes and avoid any kind of toxic language or racial slurs. This requires developers to ensure proper behavior of the model through data cleaning, if the model is trained from scratch, or supplying tailored data if finetuning it to their needs. Addressing _prejudices_ such as stereotypes and incorrect notions is complex, as they are not necessarily related to single words or expressions, but instead present themselves as a collection of ideals that can be wrong at best, and harmful at worst. An NPC LLM may exhibit real world stereotypes that can negatively impact the player experience, although we argue that the impact of prejudices from an LLM commentator or Game Master is much stronger and disturbing due to their perceived authority.

##  7 Conclusions

As discussed in this paper, LLMs can take up many different roles that can improve the experience of players in digital games, or enhance the ability of game designers to bring their ideas to life. However, we also highlighted many different challenges specific to the applications of LLMs and intrinsic to the nature of LLMs and the ecosystem that surrounds them. Despite technical, ethical, and legal challenges posed by LLMs, it is not realistic to ignore the impact that this research will likely have on both Game AI research and the game industry. We expect to see many new technical innovations from LLM researchers and corporations. Anticipating this, we propose promising directions where LLMs could be applied to games in the future.

## Acknowledgments

This work has been supported by the European Union’s Horizon 2020 research and innovation programme from the AI4media project (Grant Agreement No. 951911), and by the US National Science Foundation under the Graduate Research Fellowship Program.

## References

  * [1] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever _et al._ , “Language models are unsupervised multitask learners,” _OpenAI blog_ , vol. 1, no. 8, 2019. 
  * [2] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in _Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics_ , 2019. 
  * [3] Z. Zhang, X. Han, Z. Liu, X. Jiang, M. Sun, and Q. Liu, “ERNIE: Enhanced language representation with informative entities,” in _Proceedings of the Annual Meeting of the Association for Computational Linguistics_ , 2019. 
  * [4] T. Kaufmann, P. Weng, V. Bengs, and E. Hüllermeier, “A survey of reinforcement learning from human feedback,” _arXiv preprint arXiv:2312.14925_ , 2023. 
  * [5] S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg, H. Nori, H. Palangi, M. T. Ribeiro, and Y. Zhang, “Sparks of artificial general intelligence: Early experiments with GPT-4,” _arXiv preprint arXiv:2303.12712_ , 2023. 
  * [6] R. T. McCoy, S. Yao, D. Friedman, M. Hardy, and T. L. Griffiths, “Embers of autoregression: Understanding large language models through the problem they are trained to solve,” _arXiv preprint arXiv:2309.13638_ , 2023. 
  * [7] M. T. Hicks, J. Humphries, and J. Slater, “Correction: ChatGPT is bullshit,” _Ethics and Information Technology_ , vol. 26, no. 46, 2024. 
  * [8] G. N. Yannakakis and J. Togelius, _Artificial Intelligence and Games_. Springer, 2018. 
  * [9] J. Gwertzman and J. Soslow, “The generative AI revolution in games,” <https://a16z.com/the-generative-ai-revolution-in-games/>, 2022, accessed 25 Feb 2024. 
  * [10] S. Minaee, T. Mikolov, N. Nikzad, M. Chenaghlu, R. Socher, X. Amatriain, and J. Gao, “Large language models: A survey,” _arXiv preprint arXiv:2402.06196_ , 2024. 
  * [11] P. García-Sánchez, A. M. García, P. A. Castillo, and I. J. Pérez, “A bibliometric study of the research area of videogames using Dimensions.ai database,” in _Proceedings of the International Conference on Information Technology and Quantitative Management_ , 2019. 
  * [12] A. Liapis, “10 years of the PCG workshop: Past and future trends,” in _Proceedings of the FDG Workshop on Procedural Content Generation_ , 2020\. 
  * [13] A. Jordanous and B. Keller, “Modelling creativity: Identifying key components through a corpus-based approach,” _Public Library of Science_ , vol. 11, pp. 1–27, 2016. 
  * [14] D. Yang, E. Kleinman, and C. Harteveld, “GPT for games: A scoping review (2020-2023),” in _Proceedings of the IEEE Conference on Games_ , 2024. 
  * [15] C. E. Shannon, “A mathematical theory of communication,” _The Bell system technical journal_ , vol. 27, no. 3, 1948. 
  * [16] J. L. Elman, “Finding structure in time,” _Cognitive science_ , vol. 14, no. 2, 1990. 
  * [17] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” _Neural Computation_ , vol. 9, pp. 1735–1780, 1997. 
  * [18] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Proceedings of the Conference on Neural Information Processing Systems_ , 2017. 
  * [19] A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M.-A. Lachaux, P. Stock, T. L. Scao, T. Lavril, T. Wang, T. Lacroix, and W. E. Sayed, “Mistral 7B,” _arXiv preprint arXiv:2310.06825_ , 2023. 
  * [20] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar _et al._ , “Llama: Open and efficient foundation language models,” _arXiv preprint arXiv:2302.13971_ , 2023. 
  * [21] Z. Yang, L. Li, K. Lin, J. Wang, C.-C. Lin, Z. Liu, and L. Wang, “The dawn of LMMs: Preliminary explorations with GPT-4V(ision),” _arXiv preprint arXiv:2309.17421_ , 2023. 
  * [22] H. Liu, C. Li, Q. Wu, and Y. J. Lee, “Visual instruction tuning,” in _Proceedings of the Neural Information Processing Systems Conference_ , 2023\. 
  * [23] A. Summerville, S. Snodgrass, M. Guzdial, C. Holmgård, A. K. Hoover, A. Isaksen, A. Nealen, and J. Togelius, “Procedural content generation via machine learning (PCGML),” _IEEE Transactions on Games_ , vol. 10, no. 3, 2018. 
  * [24] S. J. Edwards, “Standard: Portable game notation specification and implementation guide,” <https://ia802908.us.archive.org/26/items/pgn-standard-1994-03-12/PGN_standard_1994-03-12.txt>, 1993, accessed 12 June 2024. 
  * [25] D. Noever, M. Ciolino, and J. Kalin, “The chess transformer: Mastering play using generative language models,” _arXiv preprint arXiv:2008.04057_ , 2020\. 
  * [26] A. Stöckl, “Watching a language model learning chess,” in _Proceedings of the Recent Advances in Natural Language Processing International Conference_ , 2021. 
  * [27] S. Toshniwal, S. Wiseman, K. Livescu, and K. Gimpel, “Chess as a testbed for language model state tracking,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 36, no. 10, 2022. 
  * [28] M. Ciolino, J. Kalin, and D. Noever, “The Go transformer: Natural language modeling for game play,” in _Proceedings of the Artificial Intelligence for Industries International Conference_ , 2020. 
  * [29] K. Li, A. K. Hopkins, D. Bau, F. Viégas, H. Pfister, and M. Wattenberg, “Emergent world representations: Exploring a sequence model trained on a synthetic task,” in _Proceedings of the International Conference on Learning Representations_ , 2023. 
  * [30] B. Bateni and J. Whitehead, “Language-driven play: Large language models as game-playing agents in Slay the Spire,” in _Proceedings of the International Conference on the Foundations of Digital Games_ , 2024. 
  * [31] S. Reed, K. Zolna, E. Parisotto, S. G. Colmenarejo, A. Novikov, G. Barth-Maron, M. Gimenez, Y. Sulsky, J. Kay, J. T. Springenberg _et al._ , “A generalist agent,” _arXiv preprint arXiv:2205.06175_ , 2022. 
  * [32] M. Xu, W. Dai, C. Liu, X. Gao, W. Lin, G.-J. Qi, and H. Xiong, “Spatial-temporal transformer networks for traffic flow forecasting,” _arXiv preprint arXiv:2001.02908_ , 2020. 
  * [33] H. Chang, H. Zhang, L. Jiang, C. Liu, and W. T. Freeman, “MaskGIT: Masked generative image transformer,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022. 
  * [34] S. Yao, R. Rao, M. Hausknecht, and K. Narasimhan, “Keep CALM and explore: Language models for action generation in text-based games,” in _Proceedings of the Empirical Methods in Natural Language Processing Conference_ , 2020. 
  * [35] C. F. Tsai, X. Zhou, S. S. Liu, J. Li, M. Yu, and H. Mei, “Can large language models play text games well? current state-of-the-art and open questions,” _arXiv preprint arXiv:2304.02868_ , 2023. 
  * [36] P. Ammanabrolu, L. Jiang, M. Sap, H. Hajishirzi, and Y. Choi, “Aligning to social norms and values in interactive narratives,” in _Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , 2022. 
  * [37] M. Hausknecht, P. Ammanabrolu, M.-A. Côté, and X. Yuan, “Interactive fiction games: A colossal adventure,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2020. 
  * [38] M. Klissarov, P. D’Oro, S. Sodhani, R. Raileanu, P.-L. Bacon, P. Vincent, A. Zhang, and M. Henaff, “Motif: Intrinsic motivation from artificial intelligence feedback,” _arXiv preprint arXiv:2310.00166_ , 2023. 
  * [39] Meta Fundamental AI Research Diplomacy Team (FAIR), A. Bakhtin, N. Brown, E. Dinan, G. Farina, C. Flaherty, D. Fried, A. Goff, J. Gray, H. Hu _et al._ , “Human-level play in the game of diplomacy by combining language models with strategic reasoning,” _Science_ , vol. 378, no. 6624, 2022. 
  * [40] G. Wang, Y. Xie, Y. Jiang, A. Mandlekar, C. Xiao, Y. Zhu, L. Fan, and A. Anandkumar, “Voyager: An open-ended embodied agent with large language models,” in _Proceedings of the NeurIPS Workshop on Foundation Models for Decision Making_ , 2023. 
  * [41] GoodAI, “Introducing our work on general-purpose LLM agents,” <https://www.goodai.com/introducing-general-purpose-llm-agents/>, 2023, accessed 24 Feb 2024. 
  * [42] M. Ç. Uludağlı and K. Oğuz, “Non-player character decision-making in computer games,” _Artificial Intelligence Review_ , vol. 56, no. 12, 2023. 
  * [43] M. Shanahan, K. McDonell, and L. Reynolds, “Role play with large language models,” _Nature_ , vol. 623, 2023. 
  * [44] F. Gao, K. Fang, and W. K. Victor Chan, “Chemical life: Knowledge-based personality, emotion and action cues in educational games,” in _Proceedings of the IEEE Conference on Games_ , 2023. 
  * [45] J. S. Park, J. O’Brien, C. J. Cai, M. R. Morris, P. Liang, and M. S. Bernstein, “Generative agents: Interactive simulacra of human behavior,” in _Proceedings of the Annual ACM Symposium on User Interface Software and Technology_ , 2023. 
  * [46] M. Kreminski, “Toward better gossip simulation in emergent narrative systems,” in _Proceedings of the IEEE Conference on Games_ , 2023. 
  * [47] M. Müller-Brockhausen, G. Barbero, and M. Preuss, “Chatter generation through language models,” in _Proceedings of the IEEE Conference on Games_ , 2023\. 
  * [48] L. O’Brien, “How Ubisoft’s New Generative AI Prototype Changes the Narrative for NPCs,” <https://news.ubisoft.com/en-us/article/5qXdxhshJBXoanFZApdG3L/how-ubisofts-new-generative-ai-prototype-changes-the-narrative-for-npcs>, 2024, accessed 12 June 2024. 
  * [49] H. Warpefelt and H. Verhagen, “A model of non-player character believability,” _Journal of Gaming & Virtual Worlds_, vol. 9, no. 1, 2017\. 
  * [50] J. Duan, H. Cheng, S. Wang, A. Zavalny, C. Wang, R. Xu, B. Kailkhura, and K. Xu, “Shifting attention to relevance: Towards the uncertainty estimation of large language models,” _arXiv preprint arXiv:2307.01379_ , 2023. 
  * [51] A. Mehta, Y. Kunjadiya, A. Kulkarni, and M. Nagar, “Exploring the viability of conversational AI for non-playable characters: A comprehensive survey,” in _Proceedings of the International Conference on Recent Trends in Computer Science and Technology_ , 2022. 
  * [52] E. Akata, L. Schulz, J. Coda-Forno, S. J. Oh, M. Bethge, and E. Schulz, “Playing repeated games with large language models,” _arXiv preprint arXiv:2305.16867_ , 2023. 
  * [53] Y. Xu, S. Wang, P. Li, F. Luo, X. Wang, W. Liu, and Y. Liu, “Exploring large language models for communication games: An empirical study on Werewolf,” _arXiv preprint arXiv:2309.04658_ , 2023. 
  * [54] P. Maas, F. Carey, C. Wheeler, E. Saatchi, P. Billington, and J. Yaffa Shamash, “To infinity and beyond: SHOW-1 and Showrunner agents in multi-agent simulations,” <https://fablestudio.github.io/showrunner-agents/>, 2023, accessed 27 Feb 2024. 
  * [55] J. Urbanek, A. Fan, S. Karamcheti, S. Jain, S. Humeau, E. Dinan, T. Rocktäschel, D. Kiela, A. Szlam, and J. Weston, “Learning to speak and act in a fantasy text adventure game,” in _Proceedings of the Empirical Methods in Natural Language Processing Conference and the Natural Language Processing International Joint Conference_ , 2019. 
  * [56] S. Hu, Z. Huang, C. Hu, and J. Liu, “3D building generation in Minecraft via large language models,” in _Proceedings of the IEEE Conference on Games_ , 2024. 
  * [57] T. Rist, “Using a large language model to turn explorations of virtual 3d-worlds into interactive narrative experiences,” in _Proceedings of the IEEE Conference on Games_ , 2024. 
  * [58] M. P. Eladhari, “Re-tellings: The fourth layer of narrative as an instrument for critique,” in _Proceedings of the Interactive Digital Storytelling International Conference_ , 2018. 
  * [59] M. Guzdial, S. Shah, and M. Riedl, “Towards automated let’s play commentary,” in _Proceedings of the AIIDE workshop on Experimental AI in Games_ , 2018\. 
  * [60] T. Ishigaki, G. Topić, Y. Hamazono, H. Noji, I. Kobayashi, Y. Miyao, and H. Takamura, “Generating racing game commentary from vision, language, and structured data,” in _Proceedings of the Natural Language Generation International Conference_ , 2021. 
  * [61] C. Li, S. Gandhi, and B. Harrison, “End-to-end let’s play commentary generation using multi-modal video representations,” in _Proceedings of the International Conference on the Foundations of Digital Games_ , 2019. 
  * [62] N. Renella and M. Eger, “Towards automated video game commentary using generative AI,” in _Proceedings of the AIIDE workshop on Experimental AI in Games_ , 2023. 
  * [63] G. Jaimovitch-López, C. Ferri, J. Hernández-Orallo, F. Martínez-Plumed, and M. J. Ramírez-Quintana, “Can language models automate data wrangling?” _Machine Learning_ , 2023. 
  * [64] L. Cheng, X. Li, and L. Bing, “Is GPT-4 a good data analyst?” in _Findings of the Association for Computational Linguistics_ , 2023. 
  * [65] P. Maddigan and T. Susnjak, “Chat2VIS: Generating data visualisations via natural language using ChatGPT, Codex and GPT-3 large language models,” _IEEE Access_ , vol. 11, 2023. 
  * [66] M. S. El-Nasr, A. Drachen, and A. Canossa, _Game Analytics: Maximizing the Value of Player Data 2013th Edition_. Springer, 2013. 
  * [67] Z. Cheng, T. Xie, P. Shi, C. Li, R. Nadkarni, Y. Hu, C. Xiong, D. Radev, M. Ostendorf, L. Zettlemoyer, N. A. Smith, and T. Yu, “Binding language models in symbolic languages,” in _Proceedings of the International Conference on Learning Representations_ , 2023. 
  * [68] Z. Chen, H. Mao, H. Li, W. Jin, H. Wen, X. Wei, S. Wang, D. Yin, W. Fan, H. Liu, and J. Tang, “Exploring the potential of large language models (LLMs) in learning on graph,” in _Proceedings of the NeurIPS Workshop on New Frontiers in Graph Learning_ , 2023. 
  * [69] T. Wang, M. Honari-Jahromi, S. Katsarou, O. Mikheeva, T. Panagiotakopoulos, S. Asadi, and O. Smirnov, “player2vec: A language modeling approach to understand player behavior in games,” _arXiv preprint arXiv:2404.04234_ , 2024. 
  * [70] I. Beltagy, M. E. Peters, and A. Cohan, “Longformer: The long-document transformer,” _arXiv preprint arXiv:2004.05150_ , 2020. 
  * [71] A. Drachen, A. Canossa, and G. N. Yannakakis, “Player modeling using self-organization in Tomb Raider: Underworld,” in _Proceedings of the IEEE Symposium on Computational Intelligence and Games_ , 2009. 
  * [72] A. Canossa, J. B. Martinez, and J. Togelius, “Give me a reason to dig: Minecraft and psychology of motivation,” in _Proceedings of the IEEE Conference on Computational Intelligence and Games_ , 2013. 
  * [73] N. Rasajski, C. Trivedi, K. Makantasis, A. Liapis, and G. N. Yannakakis, “BehAVE: Behaviour alignment of video game encodings,” in _Proceedings of the ECCV Workshop on Computer Vision For Videogames_ , 2024\. 
  * [74] J. Togelius and G. N. Yannakakis, “General general game AI,” in _Proceedings of the IEEE Conference on Computational Intelligence and Games_ , 2016. 
  * [75] A. Tychsen, M. Hitchens, T. Brolund, and M. Kavakli, “The game master,” in _Proceedings of the Australasian conference on Interactive entertainment_ , 2005. 
  * [76] A. Liapis and A. Denisova, “The challenge of evaluating player experience in tabletop role-playing games,” in _Proceedings of the International Conference on the Foundations of Digital Games_ , 2023. 
  * [77] M. Hua and R. Raley, “Playing with unicorns: AI dungeon and citizen NLP,” _Digital Humanities Quarterly_ , vol. 14, no. 4, 2020. 
  * [78] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022. 
  * [79] X. You, P. Taveekitworachai, S. Chen, M. C. Gursesli, X. Li, Y. Xia, and R. Thawonmas, “Dungeons, Dragons, and Emotions: A preliminary study of player sentiment in LLM-driven TTRPGs,” in _Proceedings of the International Conference on the Foundations of Digital Games_ , 2024. 
  * [80] A. Zhu, L. Martin, A. Head, and C. Callison-Burch, “CALYPSO: LLMs as Dungeon Masters’ assistants,” in _Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment_ , 2023. 
  * [81] J. Kelly, M. Mateas, and N. Wardrip-Fruin, “Towards computational support with language models for TTRPG game masters,” in _Proceedings of the FDG Workshop on Human-AI Interaction through Play_ , 2023. 
  * [82] D. Acharya, J. Kelly, W. Tate, M. Joslyn, M. Mateas, and N. Wardrip-Fruin, “Shoelace: A storytelling assistant for GUMSHOE One-2-One,” in _Proceedings of the International Conference on the Foundations of Digital Games_ , 2023. 
  * [83] Y. Sun, Z. Li, K. Fang, C. H. Lee, and A. Asadipour, “Language as reality: A co-creative storytelling game experience in 1001 Nights using generative AI,” in _Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment_ , 2023. 
  * [84] M. Treanor, A. Zook, M. P. Eladhari, J. Togelius, G. Smith, M. Cook, T. Thompson, B. Magerko, J. Levine, and A. Smith, “AI-based game design patterns,” in _Proceedings of the International Conference on the Foundations of Digital Games_ , 2015. 
  * [85] Y. Liu, G. Deng, Z. Xu, Y. Li, Y. Zheng, Y. Zhang, L. Zhao, T. Zhang, and Y. Liu, “Jailbreaking ChatGPT via prompt engineering: An empirical study,” _arXiv preprint arXiv:2305.13860_ , 2023. 
  * [86] L. Huang and X. Sun, “Create ice cream: Real-time creative element synthesis framework based on GPT3.0,” in _Proceedings of the IEEE Conference on Games_ , 2023. 
  * [87] T. Litchfield, “This browser-based ‘endless crafting game’ starts you off with fire and water, but it quickly escalates to God, the Big Bang, and ‘Yin-Yoda’,” <https://www.pcgamer.com/this-browser-based-endless-crafting-game-starts-you-off-with-fire-and-water-but-it-quickly-escalates-to-god-the-big-bang-and-yin-yoda/>, 2024, accessed 28 Feburary 2024. 
  * [88] A. J. Summerville, S. Snodgrass, M. Mateas, and S. Ontanon, “The VGLC: The video game level corpus,” in _Proceedings of the FDG Workshop on Procedural Content Generation_ , 2016. 
  * [89] A. Summerville and M. Mateas, “Super Mario as a string: Platformer level generation via LSTMs,” in _Proceedings of the Joint Conference of DIGRA and FDG_ , 2016. 
  * [90] Y. Zakaria, M. Fayek, and M. Hadhoud, “Procedural level generation for Sokoban via deep learning: An experimental study,” _IEEE Transactions on Games_ , vol. 15, no. 1, 2022. 
  * [91] G. Todd, S. Earle, M. U. Nasir, M. C. Green, and J. Togelius, “Level generation through large language models,” in _Proceedings of the International Conference on the Foundations of Digital Games_ , 2023. 
  * [92] C. Bamford, “Griddly: A platform for AI research in games,” _Software Impacts_ , vol. 8, 2021. 
  * [93] S. Sudhakaran, M. González-Duque, M. Freiberger, C. Glanois, E. Najarro, and S. Risi, “MarioGPT: Open-ended text2level generation through large language models,” _Advances in Neural Information Processing Systems_ , vol. 36, 2024. 
  * [94] B. Lyman, A. Ebrahimi, J. E. C. III, S. Chan, C. Barney, and B. D. Schutter, “Cardistry: Exploring a GPT model workflow as an adapted method of gaminiscing,” in _Proceedings of the International Conference on the Foundations of Digital Games_ , 2024. 
  * [95] J. Betker, G. Goh, L. Jing, T. Brooks, J. Wang, L. Li, L. Ouyang, J. Zhuang, J. Lee, Y. Guo, W. Manassra, P. Dhariwal, C. Chu, YunxinJiao, and A. Ramesh, “Improving image generation with better captions,” <https://cdn.openai.com/papers/dall-e-3.pdf>, 2023, accessed 13 Jun 2024. 
  * [96] T. Merino, S. Earle, R. Sudhakaran, S. Sudhakaran, and J. Togelius, “Making new connections: Llms as puzzle generators for the new york times’ connections word game,” _arXiv preprint arXiv:2407.11240_ , 2024. 
  * [97] M. Zammit, A. Liapis, and G. N. Yannakakis, “CrawLLM: Theming games with large language models,” in _Proceedings of the IEEE Conference on Games_ , 2024. 
  * [98] A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford _et al._ , “Mixtral of experts,” _arXiv preprint arXiv:2401.04088_ , 2024. 
  * [99] G. Todd, A. Padula, M. Stephenson, É. Piette, D. J. Soemers, and J. Togelius, “GAVEL: Generating games via evolution and language models,” _arXiv preprint arXiv:2407.09388_ , 2024. 
  * [100] C. Browne, M. Stephenson, É. Piette, and D. J. N. J. Soemers, “A practical introduction to the Ludii general game system,” in _Proceedings of the International Conference on Advances in Computer Games_ , 2019. 
  * [101] A. Liapis, “Searching for sentient design tools for game development,” Ph.D. dissertation, Center for Computer Games, IT University of Copenhagen, Copenhagen, Denmark, 2014. 
  * [102] G. Smith, J. Whitehead, and M. Mateas, “Tanagra: Reactive planning and constraint solving for mixed-initiative level design,” _IEEE Transactions on Computational Intelligence and AI in Games_ , vol. 3, no. 3, 2011\. 
  * [103] A. Liapis, G. N. Yannakakis, and J. Togelius, “Sentient Sketchbook: Computer-aided game level authoring,” in _Proceedings of the International Conference on the Foundations of Digital Games_ , 2013. 
  * [104] P. Migkotzidis and A. Liapis, “SuSketch: Surrogate models of gameplay as a design assistant,” _IEEE Transactions on Games_ , vol. 14, no. 2, 2021. 
  * [105] M. Charity, I. Dave, A. Khalifa, and J. Togelius, “Baba is Y’all 2.0: Design and investigation of a collaborative mixed-initiative system,” _IEEE Transactions on Games_ , 2022. 
  * [106] R. Gallotta, K. Arulkumaran, and L. B. Soros, “Preference-learning emitters for mixed-initiative quality-diversity algorithms,” _IEEE Transactions on Games_ , 2023. 
  * [107] M. Charity, Y. Bhartia, D. Zhang, A. Khalifa, and J. Togelius, “A preliminary study on a conceptual game feature generation and recommendation system,” in _Proceedings of the IEEE Conference on Games_ , 2023. 
  * [108] M. G. Torii, T. Murakami, and Y. Ochiai, “Lottery and sprint: Generate a board game with design sprint method on AutoGPT,” in _Companion Proceedings of the Annual Symposium on Computer-Human Interaction in Play_ , 2023. 
  * [109] D. Dhamani and M. L. Maher, “The tyranny of possibilities in the design of task-oriented LLM systems: A scoping survey,” _arXiv preprint arXiv:2312.17601_ , 2023. 
  * [110] G. N. Yannakakis, A. Liapis, and C. Alexopoulos, “Mixed-initiative co-creativity,” in _Proceedings of the International Conference on the Foundations of Digital Games_ , 2014. 
  * [111] J. Rezwana and M. L. Maher, “Designing creative AI partners with COFI: A framework for modeling interaction in human-AI co-creative systems,” _ACM Transactions on Computer-Human Interaction_ , vol. 30, no. 5, 2023. 
  * [112] A. Liapis, G. N. Yannakakis, and J. Togelius, “Sentient World: Human-based procedural cartography,” in _Proceedings of Evolutionary and Biologically Inspired Music, Sound, Art and Design Conference_ , 2013. 
  * [113] R. Barth, “The convergence of AI and creativity: Introducing Ghostwriter,” <https://news.ubisoft.com/en-us/article/7Cm07zbBGy4Xml6WgYi25d/the-convergence-of-ai-and-creativity-introducing-ghostwriter>, accessed 12 June 2024. 
  * [114] M. Guzdial and M. Riedl, “An interaction framework for studying co-creative AI,” _arXiv preprint arXiv:1903.09709_ , 2019. 
  * [115] D. Novick and S. Sutton, “What is mixed-initiative interaction?” in _Proceedings of the AAAI Spring Symposium on Computational Models for Mixed Initiative Interaction_ , 1997. 
  * [116] A. Liapis, G. N. Yannakakis, and J. Togelius, “Designer modeling for personalized game content creation tools,” in _Proceedings of the AIIDE Workshop on Artificial Intelligence & Game Aesthetics_, 2013. 
  * [117] V. Vimpari, A. Kultima, P. Hämäläinen, and C. Guckelsberger, ““An adapt-or-die type of situation”: Perception, adoption, and use of text-to-image-generation AI by game industry professionals,” in _Proceedings of the ACM on Human-Computer Interaction_ , vol. 7, 2023. 
  * [118] J. Boucher, G. Smith, and Y. Telliel, “Examining early professionals’ use of generative AI in the game development process,” in _Proceedings of the AIIDE Workshop on Experimental Artificial Intelligence in Games_ , 2023. 
  * [119] A. Anjum, Y. Li, N. Law, M. Charity, and J. Togelius, “The ink splotch effect: A case study on ChatGPT as a co-creative game designer,” in _Proceedings of the International Conference on the Foundations of Digital Games_ , 2024. 
  * [120] L. W. Stephen Peacock, “Unravelling Project AVA: Insights from our research on GenAI in game development (presented by Keywords Studios),” <https://www.gdcvault.com/play/1034841/Unravelling-Project-AVA-Insights-from>, 2024, accessed 13 Jun 2024. 
  * [121] M. U. Nasir and J. Togelius, “Practical PCG through large language models,” in _Proceedings of the IEEE Conference on Games_ , 2023. 
  * [122] J. Kelly, A. Calderwood, N. Wardrip-Fruin, and M. Mateas, “There and back again: Extracting formal domains for controllable neurosymbolic story authoring,” in _Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment_ , 2023. 
  * [123] V. Kumaran, D. Carpenter, J. Rowe, B. Mott, and J. Lester, “End-to-End procedural level generation in educational games with natural language instruction,” in _Proceedings of the IEEE Conference on Games_ , 2023. 
  * [124] J. Rowe, E. Lobene, B. Mott, and J. Lester, “Play in the museum: Design and development of a game-based learning exhibit for informal science education,” _International Journal of Gaming and Computer-Mediated Simulations_ , 2017. 
  * [125] R. Gallotta, A. Liapis, and G. N. Yannakakis, “LLMaker: A game level design interface using (only) natural language,” in _Proceedings of the IEEE Conference on Games_ , 2024. 
  * [126] ——, “Consistent game content creation via function calling for large language models,” in _Proceedings of the IEEE Conference on Games_ , 2024\. 
  * [127] P. Taveekitworachai, F. Abdullah, M. F. Dewantoro, R. Thawonmas, J. Togelius, and J. Renz, “ChatGPT4PCG competition: Character-like level generation for science birds,” in _Proceedings of the IEEE Conference on Games_ , 2023. 
  * [128] T. Brooks, A. Holynski, and A. A. Efros, “InstructPix2Pix: Learning to follow image editing instructions,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023. 
  * [129] J. Zhu, A. Liapis, S. Risi, R. Bidarra, and G. M. Youngblood, “Explainable AI for designers: A human-centered perspective on mixed-initiative co-creation,” in _Proceedings of the IEEE Conference on Computational Intelligence and Games_ , 2018. 
  * [130] A. Liapis, M. Awiszus, A. J. Champandard, M. Cook, A. Denisova, A. Dockhorn, T. Thompson, and J. Zhu, “Artificial intelligence for audiences,” in _Human-Game AI Interaction (Dagstuhl Seminar 22251)_ , D. Ashlock, S. Maghsudi, D. P. Liebana, P. Spronck, and M. Eberhardinger, Eds. Schloss Dagstuhl – Leibniz-Zentrum für Informatik, 2023. 
  * [131] A. Canossa and A. Drachen, “Patterns of play: Play-personas in user-centred game development,” in _Proceedings of DiGRA_ , 2009. 
  * [132] G. N. Yannakakis and D. Melhart, “Affective game computing: A survey,” _Proceedings of the IEEE_ , 2023. 
  * [133] G. N. Yannakakis and J. Togelius, “Experience-driven procedural content generation,” _IEEE Transactions on Affective Computing_ , 2011. 
  * [134] K. Pinitas, D. Renaudie, M. Thomsen, M. Barthet, K. Makantasis, A. Liapis, and G. N. Yannakakis, “Predicting player engagement in Tom Clancy’s The Division 2: A multimodal approach via pixels and gamepad actions,” in _Proceedings of the ACM International Conference on Multimodal Interaction_ , 2023. 
  * [135] D. Melhart, A. Liapis, and G. N. Yannakakis, “The Arousal video Game AnnotatIoN (AGAIN) dataset,” _IEEE Transactions on Affective Computing_ , vol. 13, no. 4, 2022. 
  * [136] J. Broekens, B. Hilpert, S. Verberne, K. Baraka, P. Gebhard, and A. Plaat, “Fine-grained affective processing capabilities emerging from large language models,” in _Proceedings of the International Conference on Affective Computing and Intelligent Interaction_ , 2023. 
  * [137] A. Liapis, “Artificial intelligence for designing games,” in _The Handbook of Artificial Intelligence and the Arts_ , P. Machado, J. Romero, and G. Greenfield, Eds. Springer, 2021, in print. 
  * [138] N. M. M. Shafiullah, Z. J. Cui, A. Altanzaya, and L. Pinto, “Behavior transformers: Cloning k𝑘kitalic_k modes with one stone,” in _Proceedings of the Neural Information Processing Systems Conference_ , 2022. 
  * [139] J. Pfau, A. Liapis, G. N. Yannakakis, and R. Malaka, “Dungeons & Replicants II: Automated game balancing across multiple difficulty dimensions via deep player behavior modeling,” _IEEE Transactions on Games_ , 2023. 
  * [140] S. Johnson-Bey, M. Mateas, and N. Wardrip-Fruin, “Toward using ChatGPT to generate theme-relevant simulated storyworlds,” in _Proceedings of the AIIDE Workshop on Experimental Artificial Intelligence in Games_ , 2023. 
  * [141] M. Kreminski, D. Acharya, N. Junius, E. Oliver, K. Compton, M. Dickinson, C. Focht, S. Mason, S. Mazeika, and N. Wardrip-Fruin, “Cozy mystery construction kit: Prototyping toward an AI-assisted collaborative storytelling mystery game,” in _Proceedings of the International Conference on the Foundations of Digital Games_ , 2019. 
  * [142] C. Fernández-Vara and A. Thomson, “Procedural generation of narrative puzzles in adventure games: The Puzzle-Dice system,” in _Proceedings of the Workshop on Procedural Content Generation in Games_ , 2012. 
  * [143] J. Dormans, “Adventures in level design: generating missions and spaces for action adventure games,” in _Proceedings of the Workshop on Procedural Content Generation in Games_ , 2010. 
  * [144] S. Al-Nassar, A. Schaap, M. V. D. Zwart, M. Preuss, and M. A. Gómez-Maureira, “QuestVille: Procedural quest generation using NLP models,” in _Proceedings of the International Conference on the Foundations of Digital Games_ , 2023. 
  * [145] V. Kumaran, J. Rowe, B. Mott, and J. Lester, “SceneCraft: Automating interactive narrative scene generation in digital games with large language models,” _Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment_ , 2023. 
  * [146] J. E. Laird and M. van Lent, “Human-level AI’s killer application,” _AI Magazine_ , 2001. 
  * [147] The AlphaStar team, “AlphaStar: Mastering the real-time strategy game StarCraft II,” <https://deepmind.google/discover/blog/alphastar-mastering-the-real-time-strategy-game-starcraft-ii/>, 2019, accessed 10 July 2024. 
  * [148] S. Milani _et al._ , “Towards solving fuzzy tasks with human feedback: A retrospective of the MineRL BASALT 2022 competition,” in _Proceedings of the NeurIPS 2022 Competitions Track_ , 2022. 
  * [149] A. M. Smith, E. Andersen, M. Mateas, and Z. Popovic, “A case study of expressively constrainable level design automation tools for a puzzle game,” in _Proceedings of the International Conference on the Foundations of Digital Games_ , 2012. 
  * [150] D. Karavolos, A. Liapis, and G. N. Yannakakis, “Learning the patterns of balance in a multi-player shooter game,” in _Proceedings of the FDG Workshop on Procedural Content Generation in Games_ , 2017. 
  * [151] P. Manakul, A. Liusie, and M. J. F. Gales, “SelfCheckGPT: Zero-resource black-box hallucination detection for generative large language models,” in _Proceedings of the Empirical Methods in Natural Language Processing Conference_ , 2023. 
  * [152] A. Strasser, “On pitfalls (and advantages) of sophisticated large language models,” _arXiv preprint arXiv:2303.17511_ , 2023. 
  * [153] N. Bian, H. Lin, P. Liu, Y. Lu, C. Zhang, B. He, X. Han, and L. Sun, “Influence of external information on large language models mirrors social cognitive patterns,” _arXiv preprint arXiv:2305.04812_ , 2023. 
  * [154] M. Karpinska and M. Iyyer, “Large language models effectively leverage document-level context for literary translation, but critical errors persist,” in _Proceedings of the Machine Translation Conference_ , 2023. 
  * [155] Z. Gekhman, J. Herzig, R. Aharoni, C. Elkind, and I. Szpektor, “TrueTeacher: Learning factual consistency evaluation with large language models,” in _Proceedings of the Empirical Methods in Natural Language Processing Conference_ , 2023. 
  * [156] J. Zhou, “An evaluation of state-of-the-art large language models for sarcasm detection,” _arXiv preprint arXiv:2312.03706_ , 2023. 
  * [157] Y. Liu, T. Han, S. Ma, J. Zhang, Y. Yang, J. Tian, H. He, A. Li, M. He, Z. Liu, Z. Wu, L. Zhao, D. Zhu, X. Li, N. Qiang, D. Shen, T. Liu, and B. Ge, “Summary of ChatGPT-related research and perspective towards the future of large language models,” _Meta-Radiology_ , vol. 1, no. 2, 2023. 
  * [158] D. Li, A. S. Rawat, M. Zaheer, X. Wang, M. Lukasik, A. Veit, F. Yu, and S. Kumar, “Large language models with controllable working memory,” in _Findings of the Association for Computational Linguistics_ , 2023. 
  * [159] Gemini Team, “Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context,” _arXiv preprint arXiv:2403.05530_ , 2024\. 
  * [160] T. Munkhdalai, M. Faruqui, and S. Gopal, “Leave no context behind: Efficient infinite context transformers with infini-attention,” _arXiv preprint arXiv:2404.07143_ , 2024. 
  * [161] Z. Fountas, M. A. Benfeghoul, A. Oomerjee, F. Christopoulou, G. Lampouras, H. Bou-Ammar, and J. Wang, “Human-like episodic memory for infinite context LLMs,” _arXiv preprint arXiv:2407.09450_ , 2024. 
  * [162] P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rocktäschel, S. Riedel, and D. Kiela, “Retrieval-augmented generation for knowledge-intensive NLP tasks,” in _Proceedings of the Conference on Neural Information Processing Systems_ , 2020. 
  * [163] Valve Corporation, “Steam Hardware & Software Survey: May 2024,” <https://store.steampowered.com/hwsurvey/steam-hardware-software-survey-welcome-to-steam>, accessed 12 June 2024. 
  * [164] M. Yue, J. Zhao, M. Zhang, L. Du, and Z. Yao, “Large language model cascades with mixture of thoughts representations for cost-efficient reasoning,” _arXiv preprint arXiv:2310.0309_ , 2024. 
  * [165] Y. Yu, Q. Zhang, J. Li, Q. Fu, and D. Ye, “Affordable generative agents,” _arXiv preprint arXiv:2402.02053_ , 2024. 
  * [166] X. Miao, G. Oliaro, Z. Zhang, X. Cheng, H. Jin, T. Chen, and Z. Jia, “Towards efficient generative large language model serving: A survey from algorithms to systems,” _arXiv preprint arXiv:2312.15234_ , 2023. 
  * [167] F. de la Torre, C. M. Fang, H. Huang, A. Banburski-Fahey, J. A. Fernandez, and J. Lanier, “LLMR: Real-time prompting of interactive worlds using large language models,” _arXiv preprint arXiv:2309.12276_ , 2023. 
  * [168] D. Melhart, J. Togelius, B. Mikkelsen, C. Holmgård, and G. N. Yannakakis, “The ethics of AI in games,” _IEEE Transactions on Affective Computing_ , vol. 15, no. 1, 2023. 
  * [169] A. A. Chien, L. Lin, H. Nguyen, V. Rao, T. Sharma, and R. Wijayawardana, “Reducing the carbon impact of generative AI inference (today and in 2035),” in _Proceedings of the SCS Workshop on Sustainable Computer Systems_ , 2023. 
  * [170] S. A. Khowaja, P. Khuwaja, and K. Dev, “ChatGPT needs SPADE (sustainability, PrivAcy, Digital divide, and Ethics) evaluation: A review,” _arXiv preprint arXiv:2305.03123_ , 2023. 
  * [171] Free Law Project, “Authors Guild v. OpenAI Inc.” <https://www.courtlistener.com/docket/67810584/authors-guild-v-openai-inc/>, accessed 27 Feb 2024. 
  * [172] J. A. Rothchild, “Copyright implications of the use of code repositories to train a machine learning model,” _Free Software Foundation_ , 2022. 
  * [173] HackerNews, “Github Copilot,” <https://news.ycombinator.com/item?id=27676266>, 2021, accessed 27 Feb 2024\. 
  * [174] P. Zhang, G. Zeng, T. Wang, and W. Lu, “TinyLlama: An open-source small language model,” _arXiv preprint arXiv:2401.02385_ , 2024. 
  * [175] U.S. Copyright Office Review Board, “Decision affirming refusal of registration of “A Recent Entrance to Paradise”,” 2022. 
  * [176] ——, “Registration decision on “Zarya of the Dawn”,” 2023. 
  * [177] C. E. Lamb and D. G. Brown, “Should we have seen the coming storm? Transformers, society, and CC,” in _Proceedings of the International Conference on Computational Creativity_ , 2023. 
  * [178] Y. Liu, Y. Yao, J.-F. Ton, X. Zhang, R. Guo, H. Cheng, Y. Klochkov, M. F. Taufiq, and H. Li, “Trustworthy LLMs: a survey and guideline for evaluating large language models’ alignment,” in _Proceedings of the NeurIPS Workshop on Socially Responsible Language Modelling Research_ , 2023. 
  * [179] T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa, “Large language models are zero-shot reasoners,” in _Proceedings of the Advances in Neural Information Processing Systems Conference_ , 2022. 
  * [180] S. Chen, B. Li, and D. Niu, “Boosting of thoughts: Trial-and-error problem solving with large language models,” _arXiv preprint arXiv:2402.11140_ , 2024\. 
  * [181] D. Mondal, S. Modi, S. Panda, R. Singh, and G. S. Rao, “KAM-CoT: Knowledge augmented multimodal chain-of-thoughts reasoning,” _arXiv preprint arXiv:2401.12863_ , 2024. 
  * [182] M. Turpin, J. Michael, E. Perez, and S. R. Bowman, “Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting,” in _Proceedings of the Neural Information Processing Systems Conference_ , 2023. 
  * [183] S. Frieder, L. Pinchetti, A. Chevalier, R.-R. Griffiths, T. Salvatori, T. Lukasiewicz, P. C. Petersen, and J. Berner, “Mathematical capabilities of ChatGPT,” in _Proceedings of the Neural Information Processing Systems Conference_ , 2023. 
  * [184] A. M. Karkaj, M. J. Nelson, I. Koutis, and A. K. Hoover, “Prompt wrangling: On replication and generalization in large language models for PCG levels,” in _Proceedings of the International Conference on the Foundations of Digital Games_ , 2024. 
  * [185] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. Le Scao, S. Gugger, M. Drame, Q. Lhoest, and A. Rush, “Transformers: State-of-the-art natural language processing,” in _Proceedings of the Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , 2020. 
  * [186] B. Jacob, S. Kligys, B. Chen, M. Zhu, M. Tang, A. Howard, H. Adam, and D. Kalenichenko, “Quantization and training of neural networks for efficient integer-arithmetic-only inference,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018. 
  * [187] Y. Jia, W. Wu, F. Cao, and S. C. Han, “In-game toxic language detection: Shared task and attention residuals,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2022. 
  * [188] J. Thomas, N. Chorakhalikar, A. Dantrey, R. R. Nalla, and P. Mehta, “Automatic classification and reporting of inappropriate language in online applications,” US Patent 20 210 370 188, 2021. 



![\[Uncaptioned image\]](extracted/6056081/bios/roberto.jpg) |  Roberto Gallotta is a second-year Ph.D. student in Game Research at the Institute of Digital Games, University of Malta, researching dynamic evolutionary computation and mixed-initiative co-creation for the aide of video game designers. Prior to joining the Institute of Digital Games, he was a junior researcher at Araya Inc., Tokyo, for a year. He has published multiple papers, mostly focused on procedural content generation and evolutionary computation with games as a domain, and co-organized the “ALife for and from video games” workshop at ALIFE2023.  
---|---  
![\[Uncaptioned image\]](extracted/6056081/bios/graham.jpg) |  Graham Todd is a fourth-year PhD student in Computer Science in the Game Innovation Lab at NYU Tandon. His research focuses on the intersection of language games, and the ways in which people and algorithms can generate novel games and goals. He is particularly interested in what games can teach us about their players. He has published a number of papers on topics ranging from modeling the process of game generation with evolutionary algorithms to interrogating the ways that LLMs understand language through word games.  
---|---  
![\[Uncaptioned image\]](extracted/6056081/bios/marvin.png) |  Marvin Zammit is a researcher at the Institute of Digital Games, University of Malta, where he is currently reading for a Ph.D. in Game Research. His research revolves around computational creativity, procedural content generation, evolutionary computation, quality diversity, and educational games. His primary interest lies in the application of machine learning algorithms in building practical pipelines for game development. He is also an experienced developer of games and interactive installations.  
---|---  
![\[Uncaptioned image\]](extracted/6056081/bios/sam.png) |  Sam Earle is a fifth-year PhD student in Computer Science in the Game Innovation Lab at NYU Tandon. His research focuses on open-ended learning in terms of the automatic generation of diverse playable game environments, and the training of robust embodied agents within these games. He has also investigated using foundation models to enable text-guided environment generation, with an eye toward steering ever-complexifying environment- and agent-generation loops toward human-relevant content.  
---|---  
![\[Uncaptioned image\]](extracted/6056081/bios/liapis.jpg) |  Antonios Liapis is an Associate Professor at the Institute of Digital Games, University of Malta, where he bridges the gap between game technology and game design in courses focusing on human-computer creativity, digital prototyping and game development. He received the Ph.D. degree in Information Technology from the IT University of Copenhagen in 2014. His research focuses on Artificial Intelligence in games, human-computer interaction, computational creativity, and user modeling. He has published over 140 papers in the aforementioned fields, and has received several awards for his research contributions and reviewing effort. He serves as Associate Editor for the IEEE Transactions on Games, and has served as general chair in four international conferences, as guest editor in five special issues in international journals, and has co-organized 15 workshops.  
---|---  
![\[Uncaptioned image\]](extracted/6056081/bios/mini-julian.jpg) |  Julian Togelius is an Associate Professor in the Department of Computer Science and Engineering at New York University, and director of the NYU Game Innovation Lab. He is also a co-founder and research directo of the game AI company modl.ai. Julian’s research focuses on games for AI and AI for games; for example, procedural content generation via reinforcement learning, open-ended learning in generative environments, and LLM-guided game creation. Julian got his PhD in Computer Science in 2007 from the University of Essex in England, and he has also worked in Switzerland and Denmark despite being Swedish. He was Editor-in-Chief of IEEE Transactions on Games from 2018 to 2021.  
---|---  
![\[Uncaptioned image\]](extracted/6056081/bios/yannakakis_cf.png) |  Georgios N. Yannakakis (F’24) is a Professor at the Institute of Digital Games, University of Malta (UM) and a co-founder of modl.ai. He received the PhD degree in Informatics from the University of Edinburgh in 2006. Prior to joining the Institute of Digital Games, UM, in 2012 he was an Associate Professor at the Center for Computer Games Research at the IT University of Copenhagen. He does research at the crossroads of artificial intelligence, computational creativity, affective computing, advanced game technology, and human-computer interaction. He has published more than 350 papers in the aforementioned fields and his work has been cited broadly. His research has been supported by numerous national and European grants (including a Marie Skłodowska-Curie Fellowship) and has appeared in _Science Magazine_ and _New Scientist_ among other venues. He is currently the Editor-in-Chief of the IEEE Transactions on Games and an Associate Editor of the IEEE Transactions on Evolutionary Computation. Georgios is an IEEE Fellow.  
---|---  
  


url: https://upcommons.upc.edu/handle/2117/410162
title: Development of a PCG with a LLM as an inference engine and an ontology to aid the creative process
scrape_date: 2025-03-11T15:43:15.746107
content_type: markdown
hash: 1aefbb4edfcef7677bff63c64e9bc409f5b829efb3897d0833f6ed9fe8eff566
content:
Ir al contenido (pulsa Retorno)

#  [![Universitat Politècnica de Catalunya](/themes/UPCommons//images/logoUPC.png)](https://www.upc.edu)

  * **   **
    * Catala
    * Castellano 
    * English 
  * **   ** [Inicia la sessio](/login)[Registre (usuaris no UPC)](/register)[Entrada (usuaris no UPC)](/ldap-login)
  * Intercanvia la navegacio   



  * [mailContacta](/contact)


  * world Catala 
    * Catala
    * Castellano 
    * English 


  * userInicia sessio    [Inicia la sessio](/login)[Registre (usuaris no UPC)](/register)[Entrada (usuaris no UPC)](/ldap-login)

Intercanvia la navegacio   

#  [**UPCommons.**  Portal del coneixement obert de la UPC](/)

![Banner header](/themes/UPCommons//images/upcommons03.png)

75.141 Treballs academics UPC

You are here:

Visualitza l'item ****

  * [__  Pagina inicial de UPCommons](/)
  * [Treballs academics](/handle/2117/28592)
  * [Facultat d'Informatica de Barcelona](/handle/2099.1/1441)
  * [Grau en Enginyeria Informatica (Pla 2010)](/handle/2099.1/18595)
  * Visualitza l'item



  * __  [Pagina inicial de UPCommons](/)
  * [Treballs academics](/handle/2117/28592)
  * [Facultat d'Informatica de Barcelona](/handle/2099.1/1441)
  * [Grau en Enginyeria Informatica (Pla 2010)](/handle/2099.1/18595)
  * Visualitza l'item



JavaScript is disabled for your browser. Some features of this site may not work without it.

## Development of a PCG with a LLM as an inference engine and an ontology to aid the creative process

![Thumbnail](/bitstream/id/9e855b55-df0d-4dab-b632-24ea6afa41a3/183224.pdf.jpg?sequence=4&isAllowed=y)

##### Visualitza/Obre

[__183224.pdf (1,523Mb)](/bitstream/handle/2117/410162/183224.pdf?sequence=2&isAllowed=y)

[__  Veure estadistiques d'us d'UPCommons](/handle/2117/410162/es)

[__  Estadistiques de LA Referencia / Recolecta](javascript:toggleStatsLaReferencia\(\))

Inclou dades d'us des de 2022

Cita com:

hdl:2117/410162

  
[Mostra el registre d'item complet](/handle/2117/410162?show=full)

[Margineda Mangues, Xavier](/browse?value=Margineda Mangues, Xavier&type=author)

Tutor / director[Gimenez Ábalos, Victor](/browse?authority=9d382a7c-250a-4e9b-b8f8-c5dabc362a56;drac:30434890;gauss:1202032&type=contributor)[![Més informació](/themes/UPCommons//images/drac.png)](https://futur.upc.edu/30434890 "Futur Profile")[![Més informació](/themes/UPCommons//images/apren.png)](https://apren.upc.edu/ca/professorat/1202032 "Aprèn Profile"); [Gnatyshak, Dmitry](/browse?authority=39ac1028-9e72-4bb2-ad45-4ae1c5acecfb;orcid:0000-0001-6779-6283;drac:27618537;gauss:1223629&type=contributor)[![Més informació](/themes/UPCommons//images/orcid.png)](http://orcid.org/0000-0001-6779-6283 "ORCID Profile")[![Més informació](/themes/UPCommons//images/drac.png)](https://futur.upc.edu/27618537 "Futur Profile")[![Més informació](/themes/UPCommons//images/apren.png)](https://apren.upc.edu/ca/professorat/1223629 "Aprèn Profile")

Tipus de documentTreball Final de Grau

Data2024-01-25

Condicions d'accesAcces obert

**Tots els drets reservats.** Aquesta obra esta protegida pels drets de propietat intel*lectual i industrial corresponents. Sense perjudici de les exempcions legals existents, queda prohibida la seva reproduccio, distribucio, comunicacio publica o transformacio sense l'autoritzacio de la persona titular dels drets 

Abstract

When creating a story, there is a valuable world-building process to be engaged with. This project presents the development of a Procedural Content Generator to aid in this process. This is done with the use of quantized LLM model, hosted in a local computer (with limited capacity), as an inference engine of our structure, based on Knowledge-Based Systems (KBS). We also make use of an ontology to represent the various elements of a story, and act as the permanence and knowledge of our KBS. The Procedural Content Generator obtained starts from an initial ontology, which represents a story in some point of it's course. From there it starts a looping structure in which the program has the ability to choose a sub-pipeline, which are the means through which the program expands on the story. It keeps adding content onto the ontology until the program considers that it sufficiently expanded the original ontology, after which it stops the execution. To achieve results gotten, we first made a series of experiments with the LLM to see the capacities and limitations of the model. This experiments include tasks of categorization in personalized simple categories and contextual content generation in relation to one or two pre-existing elements of a story, in between others. This document presents both the experiments done with the LLM to achieve useful results and the finally designed pipeline that would be developed.

 

A l'hora de crear una historia, hi ha un valuos proces de construccio del mon en el que un es pot involucrar. Aquest projecte presenta el desenvolupament d'un Generador de Contingut Procedimental per ajudar en aquest proces. Aixo es fa amb l'us d'un model LLM quantitzat, utilitzat en un ordinador local (amb capacitat limitada), com a motor d'inferencia de la nostra estructura, que esta basada en Sistemes Basats en el Coneixement (SBC). Tambe fem servir una ontologia per representar els diversos elements d'una historia i actuar com la permanencia i coneixement del nostre SBC. El Generador de Contingut Procedimental obtingut parteix d'una ontologia inicial, que representa una historia en algun moment del seu recorregut. Des d'aqui comença una estructura amb bucle en que el programa te la capacitat de triar una sub-linea, que son els mitjans a traves dels quals el programa amplia la historia. Aquesta continua afegint contingut a l'ontologia fins que el programa considera que ha expandit prou l'ontologia original, moment en que atura l'execucio. Per obtenir els resultats, primer vam realitzar una serie d'experiments amb el LLM per veure les capacitats i limitacions del model. Aquests experiments inclouen tasques de categoritzacio en categories simples personalitzades i generacio de contingut contextual en relacio amb un o dos elements preexistents d'una historia, entre d'altres. Aquest document presenta tant els experiments realitzats amb el LLM per obtenir resultats com el disseny resultant del programa que es desenvolupa.

Materies[Ontologies (Information retrieval)](/browse?value=Ontologies \(Information retrieval\)&type=subject), [Expert systems (Computer science))](/browse?value=Expert systems \(Computer science\)\)&type=subject), [Ontologies (Informatica)](/browse?value=Ontologies \(Inform%C3%A0tica\)&type=subject), [Sistemes experts (Informatica)](/browse?value=Sistemes experts \(Inform%C3%A0tica\)&type=subject)

TitulacioGRAU EN ENGINYERIA INFORMÀTICA (Pla 2010)

URI<http://hdl.handle.net/2117/410162>

Col*leccions

  * [Facultat d'Informatica de Barcelona - Grau en Enginyeria Informatica (Pla 2010)](/handle/2099.1/18595) [2.826]



[__  Veure estadistiques d'us d'UPCommons](/handle/2117/410162/es)

  
[Mostra el registre d'item complet](/handle/2117/410162?show=full)

  
Fitxers| Descripcio| Mida| Format| Visualitza  
---|---|---|---|---  
[183224.pdf](/bitstream/handle/2117/410162/183224.pdf?sequence=2&isAllowed=y "183224.pdf")| | 1,523Mb| PDF| [Visualitza/Obre](/bitstream/handle/2117/410162/183224.pdf?sequence=2&isAllowed=y)  
  
Cerca a UPCommons

Aquesta col*leccio

## Explora

Aquesta col*leccio[Per data d'edicio](/handle/2099.1/18595/browse?type=dateissued)[Autors](/handle/2099.1/18595/browse?type=author)[Altres contribucions](/handle/2099.1/18595/browse?type=contributor)[Titols](/handle/2099.1/18595/browse?type=title)[Temes](/handle/2099.1/18595/browse?type=subject)Aquest diposit[Comunitats i col*leccions](/community-list-pfc)[Per data d'edicio](/handle/2117/28592/browse?type=dateissued)[Autors](/handle/2117/28592/browse?type=author)[Altres contribucions](/handle/2117/28592/browse?type=contributor)[Titols](/handle/2117/28592/browse?type=title)[Temes](/handle/2117/28592/browse?type=subject)

(C) [ UPC ![Obrir en finestra nova](/themes/UPCommons//images/blank_blau.png)](https://www.upc.edu) . Servei de Biblioteques, Publicacions i Arxius 

[info.biblioteques@upc.edu](mailto:info.biblioteques@upc.edu)

  * [Sobre aquest web](/page/about)
  * [Metadades subjectes a:![Metadades subjectes a CC0](/themes/UPCommons//images/creativecommons/cc-zero.png)](https://creativecommons.org/publicdomain/zero/1.0/)
  * [Contacta](/contact)
  * [Envia comentaris](/feedback)
  * Configuracio de privadesa
  * ![Inici de la pàgina](/themes/UPCommons//images/amunt.png)



[ ](/htmlmap)
  *[
                                    UPC
                                ]: Universitat Politècnica de Catalunya . BarcelonaTech


url: http://arxiv.org/abs/2412.21022v1
scrape_date: 2025-03-12T11:17:33.191831
content_type: markdown
hash: 2e4769c2cc2313415966a307204818bcb71cf9b0f19f84d025ef72640799409c
title:
[2412.21022v1] Text Classification: Neural Networks VS Machine Learning Models VS Pre-trained Models

content:
Skip to main content

[![Cornell University](/static/browse/0.3.4/images/icons/cu/cornell-reduced-white-SMALL.svg)](https://www.cornell.edu/)

We gratefully acknowledge support from the Simons Foundation, [member institutions](https://info.arxiv.org/about/ourmembers.html), and all contributors. [Donate](https://info.arxiv.org/about/donate.html)

[](/IgnoreMe)

[![arxiv logo](/static/browse/0.3.4/images/arxiv-logo-one-color-white.svg)](/) > [cs](/list/cs/recent) > arXiv:2412.21022v1 

[Help](https://info.arxiv.org/help) | [Advanced Search](https://arxiv.org/search/advanced)

All fields Title Author Abstract Comments Journal reference ACM classification MSC classification Report number arXiv identifier DOI ORCID arXiv author ID Help pages Full text

Search

[![arXiv logo](/static/browse/0.3.4/images/arxiv-logomark-small-white.svg)](https://arxiv.org/)

[ ![Cornell University Logo](/static/browse/0.3.4/images/icons/cu/cornell-reduced-white-SMALL.svg) ](https://www.cornell.edu/)

open search

GO

open navigation menu

## quick links

  * [Login](https://arxiv.org/login)
  * [Help Pages](https://info.arxiv.org/help)
  * [About](https://info.arxiv.org/about)



# Computer Science > Machine Learning

**arXiv:2412.21022v1** (cs) 

[Submitted on 30 Dec 2024]

# Title:Text Classification: Neural Networks VS Machine Learning Models VS Pre-trained Models

Authors:[Christos Petridis](https://arxiv.org/search/cs?searchtype=author&query=Petridis,+C)

View a PDF of the paper titled Text Classification: Neural Networks VS Machine Learning Models VS Pre-trained Models, by Christos Petridis

[View PDF](/pdf/2412.21022v1) [HTML (experimental)](https://arxiv.org/html/2412.21022v1)

> Abstract:Text classification is a very common task nowadays and there are many efficient methods and algorithms that we can employ to accomplish it. Transformers have revolutionized the field of deep learning, particularly in Natural Language Processing (NLP) and have rapidly expanded to other domains such as computer vision, time-series analysis and more. The transformer model was firstly introduced in the context of machine translation and its architecture relies on self-attention mechanisms to capture complex relationships within data sequences. It is able to handle long-range dependencies more effectively than traditional neural networks (such as Recurrent Neural Networks and Multilayer Perceptrons). In this work, we present a comparison between different techniques to perform text classification. We take into consideration seven pre-trained models, three standard neural networks and three machine learning models. For standard neural networks and machine learning models we also compare two embedding techniques: TF-IDF and GloVe, with the latter consistently outperforming the former. Finally, we demonstrate the results from our experiments where pre-trained models such as BERT and DistilBERT always perform better than standard models/algorithms. 

Subjects: |  Machine Learning (cs.LG)  
---|---  
Cite as: | [arXiv:2412.21022](https://arxiv.org/abs/2412.21022) [cs.LG]  
  | (or  [arXiv:2412.21022v1](https://arxiv.org/abs/2412.21022v1) [cs.LG] for this version)   
  |  <https://doi.org/10.48550/arXiv.2412.21022> Focus to learn more arXiv-issued DOI via DataCite  
  
## Submission history

From: Christos Petridis [[view email](/show-email/f698d679/2412.21022)]   
**[v1]** Mon, 30 Dec 2024 15:44:05 UTC (1,746 KB)  


Full-text links:

## Access Paper:

View a PDF of the paper titled Text Classification: Neural Networks VS Machine Learning Models VS Pre-trained Models, by Christos Petridis

  * [View PDF](/pdf/2412.21022v1)
  * [HTML (experimental)](https://arxiv.org/html/2412.21022v1)
  * [TeX Source](/src/2412.21022v1)
  * [Other Formats](/format/2412.21022v1)



[ ![license icon](https://arxiv.org/icons/licenses/by-nc-nd-4.0.png) view license ](http://creativecommons.org/licenses/by-nc-nd/4.0/ "Rights to this article")

Current browse context: 

cs.LG

[< prev](/prevnext?id=2412.21022&function=prev&context=cs.LG "previous in cs.LG \(accesskey p\)")   |   [next >](/prevnext?id=2412.21022&function=next&context=cs.LG "next in cs.LG \(accesskey n\)")   


[new](/list/cs.LG/new) |  [recent](/list/cs.LG/recent) | [2024-12](/list/cs.LG/2024-12)

Change to browse by: 

[cs](/abs/2412.21022?context=cs)  


### References & Citations

  * [NASA ADS](https://ui.adsabs.harvard.edu/abs/arXiv:2412.21022)
  * [Google Scholar](https://scholar.google.com/scholar_lookup?arxiv_id=2412.21022)
  * [Semantic Scholar](https://api.semanticscholar.org/arXiv:2412.21022)



[a](/static/browse/0.3.4/css/cite.css) export BibTeX citation Loading...

## BibTeX formatted citation

×

loading...

Data provided by: 

### Bookmark

[ ![BibSonomy logo](/static/browse/0.3.4/images/icons/social/bibsonomy.png) ](http://www.bibsonomy.org/BibtexHandler?requTask=upload&url=https://arxiv.org/abs/2412.21022&description=Text Classification: Neural Networks VS Machine Learning Models VS Pre-trained Models "Bookmark on BibSonomy") [ ![Reddit logo](/static/browse/0.3.4/images/icons/social/reddit.png) ](https://reddit.com/submit?url=https://arxiv.org/abs/2412.21022&title=Text Classification: Neural Networks VS Machine Learning Models VS Pre-trained Models "Bookmark on Reddit")

Bibliographic Tools

# Bibliographic and Citation Tools

Bibliographic Explorer Toggle

Bibliographic Explorer _([What is the Explorer?](https://info.arxiv.org/labs/showcase.html#arxiv-bibliographic-explorer))_

Connected Papers Toggle

Connected Papers _([What is Connected Papers?](https://www.connectedpapers.com/about))_

Litmaps Toggle

Litmaps _([What is Litmaps?](https://www.litmaps.co/))_

scite.ai Toggle

scite Smart Citations _([What are Smart Citations?](https://www.scite.ai/))_

Code, Data, Media

# Code, Data and Media Associated with this Article

alphaXiv Toggle

alphaXiv _([What is alphaXiv?](https://alphaxiv.org/))_

Links to Code Toggle

CatalyzeX Code Finder for Papers _([What is CatalyzeX?](https://www.catalyzex.com))_

DagsHub Toggle

DagsHub _([What is DagsHub?](https://dagshub.com/))_

GotitPub Toggle

Gotit.pub _([What is GotitPub?](http://gotit.pub/faq))_

Huggingface Toggle

Hugging Face _([What is Huggingface?](https://huggingface.co/huggingface))_

Links to Code Toggle

Papers with Code _([What is Papers with Code?](https://paperswithcode.com/))_

ScienceCast Toggle

ScienceCast _([What is ScienceCast?](https://sciencecast.org/welcome))_

Demos

# Demos

Replicate Toggle

Replicate _([What is Replicate?](https://replicate.com/docs/arxiv/about))_

Spaces Toggle

Hugging Face Spaces _([What is Spaces?](https://huggingface.co/docs/hub/spaces))_

Spaces Toggle

TXYZ.AI _([What is TXYZ.AI?](https://txyz.ai))_

Related Papers

# Recommenders and Search Tools

Link to Influence Flower

Influence Flower _([What are Influence Flowers?](https://influencemap.cmlab.dev/))_

Core recommender toggle

CORE Recommender _([What is CORE?](https://core.ac.uk/services/recommender))_

IArxiv recommender toggle

IArxiv Recommender _([What is IArxiv?](https://iarxiv.org/about))_

  * Author
  * Venue
  * Institution
  * Topic



About arXivLabs 

# arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? [**Learn more about arXivLabs**](https://info.arxiv.org/labs/index.html).

[Which authors of this paper are endorsers?](/auth/show-endorsers/2412.21022) | [Disable MathJax](javascript:setMathjaxCookie\(\)) ([What is MathJax?](https://info.arxiv.org/help/mathjax.html)) 

  * [About](https://info.arxiv.org/about)
  * [Help](https://info.arxiv.org/help)



  * contact arXivClick here to contact arXiv [ Contact](https://info.arxiv.org/help/contact.html)
  * subscribe to arXiv mailingsClick here to subscribe [ Subscribe](https://info.arxiv.org/help/subscribe)



  * [Copyright](https://info.arxiv.org/help/license/index.html)
  * [Privacy Policy](https://info.arxiv.org/help/policies/privacy_policy.html)



  * [Web Accessibility Assistance](https://info.arxiv.org/help/web_accessibility.html)
  * [arXiv Operational Status ](https://status.arxiv.org)  
Get status notifications via [email](https://subscribe.sorryapp.com/24846f03/email/new) or [slack](https://subscribe.sorryapp.com/24846f03/slack/new)


url: http://arxiv.org/abs/1611.06639v1
scrape_date: 2025-03-12T11:17:33.193313
content_type: markdown
hash: 09f9cd37cc3a75df0efe9dd9e86b17cef25f0a82924fa9b3dc3ce9d3b70380f3
title:
[1611.06639v1] Text Classification Improved by Integrating Bidirectional LSTM with Two-dimensional Max Pooling

content:
Skip to main content

[![Cornell University](/static/browse/0.3.4/images/icons/cu/cornell-reduced-white-SMALL.svg)](https://www.cornell.edu/)

We gratefully acknowledge support from the Simons Foundation, [member institutions](https://info.arxiv.org/about/ourmembers.html), and all contributors. [Donate](https://info.arxiv.org/about/donate.html)

[](/IgnoreMe)

[![arxiv logo](/static/browse/0.3.4/images/arxiv-logo-one-color-white.svg)](/) > [cs](/list/cs/recent) > arXiv:1611.06639v1 

[Help](https://info.arxiv.org/help) | [Advanced Search](https://arxiv.org/search/advanced)

All fields Title Author Abstract Comments Journal reference ACM classification MSC classification Report number arXiv identifier DOI ORCID arXiv author ID Help pages Full text

Search

[![arXiv logo](/static/browse/0.3.4/images/arxiv-logomark-small-white.svg)](https://arxiv.org/)

[ ![Cornell University Logo](/static/browse/0.3.4/images/icons/cu/cornell-reduced-white-SMALL.svg) ](https://www.cornell.edu/)

open search

GO

open navigation menu

## quick links

  * [Login](https://arxiv.org/login)
  * [Help Pages](https://info.arxiv.org/help)
  * [About](https://info.arxiv.org/about)



# Computer Science > Computation and Language

**arXiv:1611.06639v1** (cs) 

[Submitted on 21 Nov 2016]

# Title:Text Classification Improved by Integrating Bidirectional LSTM with Two-dimensional Max Pooling

Authors:[Peng Zhou](https://arxiv.org/search/cs?searchtype=author&query=Zhou,+P), [Zhenyu Qi](https://arxiv.org/search/cs?searchtype=author&query=Qi,+Z), [Suncong Zheng](https://arxiv.org/search/cs?searchtype=author&query=Zheng,+S), [Jiaming Xu](https://arxiv.org/search/cs?searchtype=author&query=Xu,+J), [Hongyun Bao](https://arxiv.org/search/cs?searchtype=author&query=Bao,+H), [Bo Xu](https://arxiv.org/search/cs?searchtype=author&query=Xu,+B)

View a PDF of the paper titled Text Classification Improved by Integrating Bidirectional LSTM with Two-dimensional Max Pooling, by Peng Zhou and 5 other authors

[View PDF](/pdf/1611.06639v1)

> Abstract:Recurrent Neural Network (RNN) is one of the most popular architectures used in Natural Language Processsing (NLP) tasks because its recurrent structure is very suitable to process variable-length text. RNN can utilize distributed representations of words by first converting the tokens comprising each text into vectors, which form a matrix. And this matrix includes two dimensions: the time-step dimension and the feature vector dimension. Then most existing models usually utilize one-dimensional (1D) max pooling operation or attention-based operation only on the time-step dimension to obtain a fixed-length vector. However, the features on the feature vector dimension are not mutually independent, and simply applying 1D pooling operation over the time-step dimension independently may destroy the structure of the feature representation. On the other hand, applying two-dimensional (2D) pooling operation over the two dimensions may sample more meaningful features for sequence modeling tasks. To integrate the features on both dimensions of the matrix, this paper explores applying 2D max pooling operation to obtain a fixed-length representation of the text. This paper also utilizes 2D convolution to sample more meaningful information of the matrix. Experiments are conducted on six text classification tasks, including sentiment analysis, question classification, subjectivity classification and newsgroup classification. Compared with the state-of-the-art models, the proposed models achieve excellent performance on 4 out of 6 tasks. Specifically, one of the proposed models achieves highest accuracy on Stanford Sentiment Treebank binary classification and fine-grained classification tasks. 

Comments: | 11 pages  
---|---  
Subjects: |  Computation and Language (cs.CL)  
Cite as: | [arXiv:1611.06639](https://arxiv.org/abs/1611.06639) [cs.CL]  
  | (or  [arXiv:1611.06639v1](https://arxiv.org/abs/1611.06639v1) [cs.CL] for this version)   
  |  <https://doi.org/10.48550/arXiv.1611.06639> Focus to learn more arXiv-issued DOI via DataCite  
  
## Submission history

From: Peng Zhou [[view email](/show-email/0a6957f8/1611.06639)]   
**[v1]** Mon, 21 Nov 2016 03:26:29 UTC (125 KB)  


Full-text links:

## Access Paper:

View a PDF of the paper titled Text Classification Improved by Integrating Bidirectional LSTM with Two-dimensional Max Pooling, by Peng Zhou and 5 other authors

  * [View PDF](/pdf/1611.06639v1)
  * [TeX Source](/src/1611.06639v1)
  * [Other Formats](/format/1611.06639v1)



[view license](http://arxiv.org/licenses/nonexclusive-distrib/1.0/ "Rights to this article")

Current browse context: 

cs.CL

[< prev](/prevnext?id=1611.06639&function=prev&context=cs.CL "previous in cs.CL \(accesskey p\)")   |   [next >](/prevnext?id=1611.06639&function=next&context=cs.CL "next in cs.CL \(accesskey n\)")   


[new](/list/cs.CL/new) |  [recent](/list/cs.CL/recent) | [2016-11](/list/cs.CL/2016-11)

Change to browse by: 

[cs](/abs/1611.06639?context=cs)  


### References & Citations

  * [NASA ADS](https://ui.adsabs.harvard.edu/abs/arXiv:1611.06639)
  * [Google Scholar](https://scholar.google.com/scholar_lookup?arxiv_id=1611.06639)
  * [Semantic Scholar](https://api.semanticscholar.org/arXiv:1611.06639)



### [DBLP](https://dblp.uni-trier.de) \- CS Bibliography

[listing](https://dblp.uni-trier.de/db/journals/corr/corr1611.html#ZhouQZXBX16 "listing on DBLP") | [bibtex](https://dblp.uni-trier.de/rec/bibtex/journals/corr/ZhouQZXBX16 "DBLP bibtex record")

[Peng Zhou](https://dblp.uni-trier.de/search/author?author=Peng%20Zhou "DBLP author search")  
[Zhenyu Qi](https://dblp.uni-trier.de/search/author?author=Zhenyu%20Qi "DBLP author search")  
[Suncong Zheng](https://dblp.uni-trier.de/search/author?author=Suncong%20Zheng "DBLP author search")  
[Jiaming Xu](https://dblp.uni-trier.de/search/author?author=Jiaming%20Xu "DBLP author search")  
[Hongyun Bao](https://dblp.uni-trier.de/search/author?author=Hongyun%20Bao "DBLP author search")

…

[a](/static/browse/0.3.4/css/cite.css) export BibTeX citation Loading...

## BibTeX formatted citation

×

loading...

Data provided by: 

### Bookmark

[ ![BibSonomy logo](/static/browse/0.3.4/images/icons/social/bibsonomy.png) ](http://www.bibsonomy.org/BibtexHandler?requTask=upload&url=https://arxiv.org/abs/1611.06639&description=Text Classification Improved by Integrating Bidirectional LSTM with Two-dimensional Max Pooling "Bookmark on BibSonomy") [ ![Reddit logo](/static/browse/0.3.4/images/icons/social/reddit.png) ](https://reddit.com/submit?url=https://arxiv.org/abs/1611.06639&title=Text Classification Improved by Integrating Bidirectional LSTM with Two-dimensional Max Pooling "Bookmark on Reddit")

Bibliographic Tools

# Bibliographic and Citation Tools

Bibliographic Explorer Toggle

Bibliographic Explorer _([What is the Explorer?](https://info.arxiv.org/labs/showcase.html#arxiv-bibliographic-explorer))_

Connected Papers Toggle

Connected Papers _([What is Connected Papers?](https://www.connectedpapers.com/about))_

Litmaps Toggle

Litmaps _([What is Litmaps?](https://www.litmaps.co/))_

scite.ai Toggle

scite Smart Citations _([What are Smart Citations?](https://www.scite.ai/))_

Code, Data, Media

# Code, Data and Media Associated with this Article

alphaXiv Toggle

alphaXiv _([What is alphaXiv?](https://alphaxiv.org/))_

Links to Code Toggle

CatalyzeX Code Finder for Papers _([What is CatalyzeX?](https://www.catalyzex.com))_

DagsHub Toggle

DagsHub _([What is DagsHub?](https://dagshub.com/))_

GotitPub Toggle

Gotit.pub _([What is GotitPub?](http://gotit.pub/faq))_

Huggingface Toggle

Hugging Face _([What is Huggingface?](https://huggingface.co/huggingface))_

Links to Code Toggle

Papers with Code _([What is Papers with Code?](https://paperswithcode.com/))_

ScienceCast Toggle

ScienceCast _([What is ScienceCast?](https://sciencecast.org/welcome))_

Demos

# Demos

Replicate Toggle

Replicate _([What is Replicate?](https://replicate.com/docs/arxiv/about))_

Spaces Toggle

Hugging Face Spaces _([What is Spaces?](https://huggingface.co/docs/hub/spaces))_

Spaces Toggle

TXYZ.AI _([What is TXYZ.AI?](https://txyz.ai))_

Related Papers

# Recommenders and Search Tools

Link to Influence Flower

Influence Flower _([What are Influence Flowers?](https://influencemap.cmlab.dev/))_

Core recommender toggle

CORE Recommender _([What is CORE?](https://core.ac.uk/services/recommender))_

  * Author
  * Venue
  * Institution
  * Topic



About arXivLabs 

# arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? [**Learn more about arXivLabs**](https://info.arxiv.org/labs/index.html).

[Which authors of this paper are endorsers?](/auth/show-endorsers/1611.06639) | [Disable MathJax](javascript:setMathjaxCookie\(\)) ([What is MathJax?](https://info.arxiv.org/help/mathjax.html)) 

  * [About](https://info.arxiv.org/about)
  * [Help](https://info.arxiv.org/help)



  * contact arXivClick here to contact arXiv [ Contact](https://info.arxiv.org/help/contact.html)
  * subscribe to arXiv mailingsClick here to subscribe [ Subscribe](https://info.arxiv.org/help/subscribe)



  * [Copyright](https://info.arxiv.org/help/license/index.html)
  * [Privacy Policy](https://info.arxiv.org/help/policies/privacy_policy.html)



  * [Web Accessibility Assistance](https://info.arxiv.org/help/web_accessibility.html)
  * [arXiv Operational Status ](https://status.arxiv.org)  
Get status notifications via [email](https://subscribe.sorryapp.com/24846f03/email/new) or [slack](https://subscribe.sorryapp.com/24846f03/slack/new)


url: https://arxiv.org/html/2406.16893v1
title: A Survey on Transformers in NLP with Focus on Efficiency
scrape_date: 2025-03-12T11:17:33.464709
content_type: markdown
hash: 09360e0a9f29cefaa7fa3ab2a5a6b38cb20936609ed3e80e08e9304340be43b8
content:
  1. [1 Introduction](https://arxiv.org/html/2406.16893v1#S1 "In A Survey on Transformers in NLP with Focus on Efficiency")
  2. [2 Survey Methodology](https://arxiv.org/html/2406.16893v1#S2 "In A Survey on Transformers in NLP with Focus on Efficiency")
  3. [3 Overview of NLP](https://arxiv.org/html/2406.16893v1#S3 "In A Survey on Transformers in NLP with Focus on Efficiency")
    1. [3.1 Applications of NLP](https://arxiv.org/html/2406.16893v1#S3.SS1 "In 3 Overview of NLP ‣ A Survey on Transformers in NLP with Focus on Efficiency")
      1. [3.1.1 Sentiment Analysis](https://arxiv.org/html/2406.16893v1#S3.SS1.SSS1 "In 3.1 Applications of NLP ‣ 3 Overview of NLP ‣ A Survey on Transformers in NLP with Focus on Efficiency")
      2. [3.1.2 Misinformation Detection](https://arxiv.org/html/2406.16893v1#S3.SS1.SSS2 "In 3.1 Applications of NLP ‣ 3 Overview of NLP ‣ A Survey on Transformers in NLP with Focus on Efficiency")
      3. [3.1.3 Machine Translation](https://arxiv.org/html/2406.16893v1#S3.SS1.SSS3 "In 3.1 Applications of NLP ‣ 3 Overview of NLP ‣ A Survey on Transformers in NLP with Focus on Efficiency")
      4. [3.1.4 Question Answering](https://arxiv.org/html/2406.16893v1#S3.SS1.SSS4 "In 3.1 Applications of NLP ‣ 3 Overview of NLP ‣ A Survey on Transformers in NLP with Focus on Efficiency")
      5. [3.1.5 Text Summarization](https://arxiv.org/html/2406.16893v1#S3.SS1.SSS5 "In 3.1 Applications of NLP ‣ 3 Overview of NLP ‣ A Survey on Transformers in NLP with Focus on Efficiency")
    2. [3.2 Evolution of NLP](https://arxiv.org/html/2406.16893v1#S3.SS2 "In 3 Overview of NLP ‣ A Survey on Transformers in NLP with Focus on Efficiency")
      1. [3.2.1 Rule-Based Approaches](https://arxiv.org/html/2406.16893v1#S3.SS2.SSS1 "In 3.2 Evolution of NLP ‣ 3 Overview of NLP ‣ A Survey on Transformers in NLP with Focus on Efficiency")
      2. [3.2.2 Traditional Machine Learning Approaches](https://arxiv.org/html/2406.16893v1#S3.SS2.SSS2 "In 3.2 Evolution of NLP ‣ 3 Overview of NLP ‣ A Survey on Transformers in NLP with Focus on Efficiency")
      3. [3.2.3 Deep Learning Approaches](https://arxiv.org/html/2406.16893v1#S3.SS2.SSS3 "In 3.2 Evolution of NLP ‣ 3 Overview of NLP ‣ A Survey on Transformers in NLP with Focus on Efficiency")
  4. [4 Transformers in NLP](https://arxiv.org/html/2406.16893v1#S4 "In A Survey on Transformers in NLP with Focus on Efficiency")
    1. [4.1 The Evolution of Transformers](https://arxiv.org/html/2406.16893v1#S4.SS1 "In 4 Transformers in NLP ‣ A Survey on Transformers in NLP with Focus on Efficiency")
    2. [4.2 Stages of Modeling](https://arxiv.org/html/2406.16893v1#S4.SS2 "In 4 Transformers in NLP ‣ A Survey on Transformers in NLP with Focus on Efficiency")
      1. [4.2.1 Pre-Training](https://arxiv.org/html/2406.16893v1#S4.SS2.SSS1 "In 4.2 Stages of Modeling ‣ 4 Transformers in NLP ‣ A Survey on Transformers in NLP with Focus on Efficiency")
      2. [4.2.2 Fine-Tuning](https://arxiv.org/html/2406.16893v1#S4.SS2.SSS2 "In 4.2 Stages of Modeling ‣ 4 Transformers in NLP ‣ A Survey on Transformers in NLP with Focus on Efficiency")
      3. [4.2.3 Prompt Engineering](https://arxiv.org/html/2406.16893v1#S4.SS2.SSS3 "In 4.2 Stages of Modeling ‣ 4 Transformers in NLP ‣ A Survey on Transformers in NLP with Focus on Efficiency")
  5. [5 Efficient Modeling Considerations](https://arxiv.org/html/2406.16893v1#S5 "In A Survey on Transformers in NLP with Focus on Efficiency")
    1. [5.1 Efficiency Measures](https://arxiv.org/html/2406.16893v1#S5.SS1 "In 5 Efficient Modeling Considerations ‣ A Survey on Transformers in NLP with Focus on Efficiency")
    2. [5.2 Software Designing](https://arxiv.org/html/2406.16893v1#S5.SS2 "In 5 Efficient Modeling Considerations ‣ A Survey on Transformers in NLP with Focus on Efficiency")
      1. [5.2.1 Data Curation](https://arxiv.org/html/2406.16893v1#S5.SS2.SSS1 "In 5.2 Software Designing ‣ 5 Efficient Modeling Considerations ‣ A Survey on Transformers in NLP with Focus on Efficiency")
      2. [5.2.2 Representation of Text](https://arxiv.org/html/2406.16893v1#S5.SS2.SSS2 "In 5.2 Software Designing ‣ 5 Efficient Modeling Considerations ‣ A Survey on Transformers in NLP with Focus on Efficiency")
      3. [5.2.3 Model Design](https://arxiv.org/html/2406.16893v1#S5.SS2.SSS3 "In 5.2 Software Designing ‣ 5 Efficient Modeling Considerations ‣ A Survey on Transformers in NLP with Focus on Efficiency")
      4. [5.2.4 Model Compression](https://arxiv.org/html/2406.16893v1#S5.SS2.SSS4 "In 5.2 Software Designing ‣ 5 Efficient Modeling Considerations ‣ A Survey on Transformers in NLP with Focus on Efficiency")
    3. [5.3 Hardware Designing](https://arxiv.org/html/2406.16893v1#S5.SS3 "In 5 Efficient Modeling Considerations ‣ A Survey on Transformers in NLP with Focus on Efficiency")
      1. [5.3.1 Customized Hardware](https://arxiv.org/html/2406.16893v1#S5.SS3.SSS1 "In 5.3 Hardware Designing ‣ 5 Efficient Modeling Considerations ‣ A Survey on Transformers in NLP with Focus on Efficiency")
      2. [5.3.2 Software-Hardware Co-Design](https://arxiv.org/html/2406.16893v1#S5.SS3.SSS2 "In 5.3 Hardware Designing ‣ 5 Efficient Modeling Considerations ‣ A Survey on Transformers in NLP with Focus on Efficiency")
  6. [6 Results and Discussion](https://arxiv.org/html/2406.16893v1#S6 "In A Survey on Transformers in NLP with Focus on Efficiency")
    1. [6.1 Statistical Insights](https://arxiv.org/html/2406.16893v1#S6.SS1 "In 6 Results and Discussion ‣ A Survey on Transformers in NLP with Focus on Efficiency")
    2. [6.2 Trend of Research](https://arxiv.org/html/2406.16893v1#S6.SS2 "In 6 Results and Discussion ‣ A Survey on Transformers in NLP with Focus on Efficiency")
    3. [6.3 Future Avenues](https://arxiv.org/html/2406.16893v1#S6.SS3 "In 6 Results and Discussion ‣ A Survey on Transformers in NLP with Focus on Efficiency")
  7. [7 Conclusion](https://arxiv.org/html/2406.16893v1#S7 "In A Survey on Transformers in NLP with Focus on Efficiency")



# A Survey on Transformers in NLP with Focus on Efficiency

Wazib Ansar   
A. K. Choudhury School of IT   
University of Calcutta, Kolkata, India   
waakcs_rs@caluniv.ac.in   
&Saptarsi Goswami   
Department of Computer Science   
Bangabasi Morning College   
Kolkata, India   
sgakc@caluniv.ac.in   
&Amlan Chakrabarti   
A. K. Choudhury School of IT   
University of Calcutta   
Kolkata, India   
acakcs@caluniv.ac.in   


###### Abstract

The advent of transformers with attention mechanisms and associated pre-trained models have revolutionized the field of Natural Language Processing (NLP). However, such models are resource-intensive due to highly complex architecture. This limits their application to resource-constrained environments. While choosing an appropriate NLP model, a major trade-off exists over choosing accuracy over efficiency and vice versa. This paper presents a commentary on the evolution of NLP and its applications with emphasis on their accuracy as-well-as efficiency. Following this, a survey of research contributions towards enhancing the efficiency of transformer-based models at various stages of model development along with hardware considerations has been conducted. The goal of this survey is to determine how current NLP techniques contribute towards a sustainable society and to establish a foundation for future research.

_K_ eywords Attention Mechanism ⋅⋅\cdot⋅ Efficiency Considerations ⋅⋅\cdot⋅ LLM ⋅⋅\cdot⋅ Natural Language Processing (NLP) ⋅⋅\cdot⋅ Transformers

##  1 Introduction

In recent years, there has been phenomenal evolution in the field of Natural Language Processing (NLP). This has been based upon a lot of research works over time upon tasks ranging from sentiment analysis [[1](https://arxiv.org/html/2406.16893v1#bib.bib1)], misinformation detection [[2](https://arxiv.org/html/2406.16893v1#bib.bib2), [20](https://arxiv.org/html/2406.16893v1#bib.bib20), [27](https://arxiv.org/html/2406.16893v1#bib.bib27)], machine translation [[3](https://arxiv.org/html/2406.16893v1#bib.bib3), [4](https://arxiv.org/html/2406.16893v1#bib.bib4)], text summarization [[5](https://arxiv.org/html/2406.16893v1#bib.bib5)] to question-answering [[6](https://arxiv.org/html/2406.16893v1#bib.bib6)]. These works have contributed towards addressing the limitations posed by preceding works as-well-as increasing their performance. The driving force behind this progress has been deep learning techniques, particularly Transformers [[7](https://arxiv.org/html/2406.16893v1#bib.bib7)] and associated pre-trained models like Bidirectional Encoder Representations from Transformers (BERT) [[8](https://arxiv.org/html/2406.16893v1#bib.bib8)], XLNet [[9](https://arxiv.org/html/2406.16893v1#bib.bib9)], Bidirectional and Auto-Regressive Transformers (BART) [[10](https://arxiv.org/html/2406.16893v1#bib.bib10)], Generative-Pre-trained Transformer (GPT) [[11](https://arxiv.org/html/2406.16893v1#bib.bib11)] along with its successors i.e. GPT-2 [[12](https://arxiv.org/html/2406.16893v1#bib.bib12)] and GPT-3 [[13](https://arxiv.org/html/2406.16893v1#bib.bib13)]. These advancements have empowered NLP models to perform complex linguistic tasks relating to understanding natural language and even generating responses as a human would provide [[14](https://arxiv.org/html/2406.16893v1#bib.bib14)].

The current research accomplishments in NLP have been enabled through the availability of voluminous textual data, sophisticated deep learning models, and high-end computing resources. As the complexity of the models rises, computing resources requirement for such models surge exponentially [[15](https://arxiv.org/html/2406.16893v1#bib.bib15)]. With the apparent deceleration of Moore’s Law111https://www.nature.com/news/the-chips-are-down-for-moore-s-law-1.19338, increasing the performance of algorithms comes at the cost of straining the computing resources along with faltering efficiency. This leads to high energy requirements translating into a hike in carbon emissions [[16](https://arxiv.org/html/2406.16893v1#bib.bib16), [17](https://arxiv.org/html/2406.16893v1#bib.bib17)]. Therefore, the need of the hour is to think out of the box and devise sustainable methodologies that can keep the performance growth rate steady while at the same time being efficient enough to be practically applicable to resource-constrained environments like mobile and edge devices [[18](https://arxiv.org/html/2406.16893v1#bib.bib18)].

The term "efficiency" of a deep learning model in NLP can be generically defined as the trade-off between the performance and the cost factors. Thus, the goal of efficient modeling lies in achieving pareto-improvement by reducing the training as-well-as inference cost for a model to achieve a benchmark level of performance [[108](https://arxiv.org/html/2406.16893v1#bib.bib108)]. The cost factors include the number of Floating-point Operations (FlOps) [[19](https://arxiv.org/html/2406.16893v1#bib.bib19)], inference time [[141](https://arxiv.org/html/2406.16893v1#bib.bib141)], model size [[18](https://arxiv.org/html/2406.16893v1#bib.bib18)], speed-up ratio [[141](https://arxiv.org/html/2406.16893v1#bib.bib141)], number of model parameters [[18](https://arxiv.org/html/2406.16893v1#bib.bib18)], energy consumption [[16](https://arxiv.org/html/2406.16893v1#bib.bib16)], and carbon emissions [[17](https://arxiv.org/html/2406.16893v1#bib.bib17)]. The efforts to enhance efficiency can be directed at various stages of model development, i.e. data curation, text representation, model design, and model compression. Efficiency can also be achieved by designing optimal hardware and maximizing its utilization. Thus, achieving efficiency improvement of a language model in NLP is a nuanced task full of challenges.

Despite the challenges, plenty of developments towards efficiency improvement of models in NLP have taken place. Besides, numerous surveys have tried to summarize these developments to serve as a stepping stone for prospective researchers who wish to contribute to this domain. Bannour and Ligozat [[151](https://arxiv.org/html/2406.16893v1#bib.bib151)] performed a systematic review of the carbon footprint by NLP models. They identified the tools and studied their accuracy as-well-as applicability for assessing the energy consumption and carbon emissions of contemporary NLP models. However, the study was limited only to assessing the environmental impact of NLP models for a single task of named entity recognition. Khadivi and Sato [[150](https://arxiv.org/html/2406.16893v1#bib.bib150)] performed bibliometric analysis on NLP papers published between 2002 to 2021 based on factors like growth-rate, doubling-time, and collaboration among authors. Given the developments, they predicted the research trend and future directions. Koubaa et al. [[153](https://arxiv.org/html/2406.16893v1#bib.bib153)] performed a critical review on ChatGPT by discussing the supporting concepts, competing technologies along with its applications. Treviso et al. [[19](https://arxiv.org/html/2406.16893v1#bib.bib19)] performed a literature review on efficient approaches in NLP focusing on data processing, model design as-well-as hardware utilization. The paper primarily confers the theoretical narratives with a comparative analysis of results from the viewpoint of efficiency is missing. Xu and McAuley [[141](https://arxiv.org/html/2406.16893v1#bib.bib141)] presented a review of model compression and acceleration techniques with a discussion on associated metrics for efficiency evaluation. Their study was limited only to pre-trained models and did not account for data-efficiency, parameter-efficiency, or hardware-design considerations. Tay et al. [[155](https://arxiv.org/html/2406.16893v1#bib.bib155)] presented a taxonomy of efficient transformer models in NLP in the form of a literature review. Even though, the survey is extensive, a comprehensive coverage of the works in the given domain is lacking. Xipeng et al. [[119](https://arxiv.org/html/2406.16893v1#bib.bib119)] survey pre-trained models in NLP with emphasis on model categories, pre-training objectives, fine-tuning, and downstream tasks. However, they do not focus on the efficiency considerations of the models.

To address the shortcomings of the previous surveys, we augment the body of knowledge with a first-of-its-kind systematic literature review on efficient transformer-based models in NLP. Firstly, it presents a primer on NLP, its applications, and the evolution of NLP techniques. Then, it performs an extensive as-well-as exhaustive study focusing on all stages of model development to achieve efficiency ranging from data curation to model design involving pre-training, fine-tuning, prompt engineering to inferencing. Not only does it explore software improvements but also efficient hardware developments accompanied with software-hardware co-designing approaches. It furnishes qualitative as-well-as quantitative evolution of NLP models and weighs the efficacy of the models in terms of their efficiency. Lastly, it analyzes the trend of research and presents the future directions. This survey paper targets researchers, professionals as well as scholars interested in the NLP particularly transformer-based Large Language Models (LLMs), and wish to design efficient leaner models to bring down the overall computational budget or for deployment on devices with low computational resources. The key contributions of this paper are as follows:

  1. 1.

We conduct a comprehensive study on transformer-based models in NLP emphasizing all stages of model development.

  2. 2.

This paper presents a qualitative and quantitative analysis of software as-well-as hardware-based contributions related to transformers in NLP from the perspective of efficiency.

  3. 3.

Finally, based on the review of existing works, we perceive the trend of developments in NLP and present a road-map to achieve pareto-optimality.




The remainder of this paper has been organized in the following manner. Section [2](https://arxiv.org/html/2406.16893v1#S2 "2 Survey Methodology ‣ A Survey on Transformers in NLP with Focus on Efficiency") puts forth the methodology adopted for this survey. Section [3](https://arxiv.org/html/2406.16893v1#S3 "3 Overview of NLP ‣ A Survey on Transformers in NLP with Focus on Efficiency") enunciates the domain of NLP, its applications, and the evolution of NLP based on the programming paradigms. Section [4](https://arxiv.org/html/2406.16893v1#S4 "4 Transformers in NLP ‣ A Survey on Transformers in NLP with Focus on Efficiency") elucidates the concepts associated with transformers and various modeling stages in transformer-based LLMs. Section [5](https://arxiv.org/html/2406.16893v1#S5 "5 Efficient Modeling Considerations ‣ A Survey on Transformers in NLP with Focus on Efficiency") showcases the developments towards efficient modeling. Section [6](https://arxiv.org/html/2406.16893v1#S6 "6 Results and Discussion ‣ A Survey on Transformers in NLP with Focus on Efficiency") presents the results of this survey in the form of statistical insights, trends of research, and ushers the future scope. Finally, in section [7](https://arxiv.org/html/2406.16893v1#S7 "7 Conclusion ‣ A Survey on Transformers in NLP with Focus on Efficiency"), the conclusions are drawn.

![Refer to caption](extracted/5597951/SurveyProcess.jpg) Figure 1: PRISMA for the survey methodology

##  2 Survey Methodology

The foundation of research lies in the critical review of existing literature and analysis of the previous results obtained through related works. It can serve several purposes, including presenting the information that is currently available about a term or concept, mapping the history of developments, determining connections between related concepts, assessing the evidence supporting any proposition, or demonstrating why a problem merits more investigation [[147](https://arxiv.org/html/2406.16893v1#bib.bib147)]. Irrespective of the field of study, there have been various typologies of surveys distinguished by certain characteristics [[149](https://arxiv.org/html/2406.16893v1#bib.bib149)]. Bibliometric analysis is a kind of survey utilizing article details like journal/ conference name, publication date, and citations as-well-as author details like name, affiliation, and collaborations to assess the developments and trends in a field of study from a statistical perspective [[150](https://arxiv.org/html/2406.16893v1#bib.bib150)]. A systematic review gathers and summarizes the findings of research works on a given subject that satisfy the standards of scientific credibility and pertinence to form a set of research questions and answer them [[151](https://arxiv.org/html/2406.16893v1#bib.bib151)]. The Preferred reporting items for systematic reviews and meta-analyses, i.e. the PRISMA statement comprises a 27-item checklist for systematic reviews [[148](https://arxiv.org/html/2406.16893v1#bib.bib148)]. A systematic mapping describes and catalogs the existing information on a topic or question of interest rather than attempting to provide a response to a particular question [[152](https://arxiv.org/html/2406.16893v1#bib.bib152)]. A literature review seeks to uncover important ideas, hypotheses, and research findings as well as knowledge gaps. It makes an effort to go over the claims and conclusions from earlier research in a narrative, chronological order [[154](https://arxiv.org/html/2406.16893v1#bib.bib154)]. A critical review examines certain concepts, themes, or theoretical viewpoints found in the body of contemporary works. It provides more of a reflection and critique of the concept under consideration [[153](https://arxiv.org/html/2406.16893v1#bib.bib153)]. However, it often inculcates bias due to the contextualization of the previous works by authors concerning their propositions. After examining the merits and demerits of the survey typologies, we conclude that given the topic of our survey and the developments in the given field, a combination of literature review and systematic review would be the best option. Hence, we adopt a systematic literature review as the survey methodology for this paper. This would enable us to elucidate the vital concepts related to transformers NLP with associated developments from the perspective of efficiency in a systematic manner. We formulate the following research questions and attempt to address them in this survey:

  * RQ1:

What are the applications of NLP, and what kinds of techniques are used to perform such applications?

  * RQ2:

What are transformer-based models and how transformers are utilized in NLP LLMs?

  * RQ3:

What is the efficiency vs efficacy trade-off in transformer-based LLMs?

  * RQ4:

What efficiency measures are present for NLP models?

  * RQ5:

What efficiency considerations are there for transformer-based NLP models?

  * RQ6:

Which stages of model development can be targeted for efficiency enhancement?

  * RQ7:

What is the current research trend in NLP and to what extent efficiency considerations are going to be prevalent in the near future?




The survey comprised of original as-well-as review articles written in the English language published in digital libraries like Google Scholar222https://scholar.google.co.in, ACM Digital Library333https://dl.acm.org, IEEE Xplore, Semantic Scholar444https://www.semanticscholar.org and Science Direct555https://www.sciencedirect.com. The articles were retrieved using keywords related to NLP and its associated terms like "NLP", "pre-trained models", "LLM", "transformers", "embedding", "pre-training", "fine-tuning", "prompt engineering", "sustainability in NLP" and "efficiency modeling in NLP". Articles published between 2000 to 2023 were included with a prime focus on articles published since 2016. The papers that were recently published were given preference. Besides, some pioneering works were included irrespective of the year in which they were published. Journal publications were chosen over conference papers where two or more articles were found to share the same subject or methodology. When choosing the papers, the journal’s impact factor and citations were taken into account. The duplicate articles were removed if it was observed that the writers had written similar works. At first, 3,210 articles were retrieved. Out of which 214 duplicates were removed. A spreadsheet application was utilized to store and process article metadata like "title", "abstract", "reference", "author list", "year of publication", "name of journal or conference" and "number of citations". Then, 1,990 articles were filtered out through statistical examination based on the above-mentioned criteria. From the remaining 1,006 articles, 312 articles were short-listed after going through the title and abstract. Finally, the full-text screening of the short-listed articles was performed leading to 151 articles to be included in the survey for this paper. To ensure a scientific and systematic structure of content, the study has been separated into several coherent pieces depending on significance.

![Refer to caption](extracted/5597951/ApplicationsNLP.jpg) Figure 2: Applications of NLP

##  3 Overview of NLP

NLP is a research domain concerned with providing computing devices the ability to comprehend and process input text in natural language understood by human beings. Using various NLP approaches one can extract meaningful information from an unstructured text corpus and even synthesize outputs in natural language [[14](https://arxiv.org/html/2406.16893v1#bib.bib14)]. The two complimentary facets of NLP are Natural Language Understanding (NLU) and Natural Language Generation (NLG) as illustrated in Figure [2](https://arxiv.org/html/2406.16893v1#S2.F2 "Figure 2 ‣ 2 Survey Methodology ‣ A Survey on Transformers in NLP with Focus on Efficiency"). NLU is the process of enabling computers to understand and derive meaning from natural language. By bridging the gap between unstructured text data and representations that are understood by machines, NLU enables machines to comprehend and process natural language input. Instances include sentiment analysis [[1](https://arxiv.org/html/2406.16893v1#bib.bib1)], opinion spam classification [[2](https://arxiv.org/html/2406.16893v1#bib.bib2)], fake news detection [[20](https://arxiv.org/html/2406.16893v1#bib.bib20)] and rumor verification [[21](https://arxiv.org/html/2406.16893v1#bib.bib21)]. On the other hand, NLG enables computers to produce natural language from structured data or other unstructured text inputs. The primary objective of NLG is to communicate information in a way that is comprehensible to human beings and appropriate as per the given situation. Instances include question answering [[22](https://arxiv.org/html/2406.16893v1#bib.bib22)], machine translation [[3](https://arxiv.org/html/2406.16893v1#bib.bib3)] and text summarization [[23](https://arxiv.org/html/2406.16893v1#bib.bib23), [24](https://arxiv.org/html/2406.16893v1#bib.bib24)].

###  3.1 Applications of NLP

Some of the renowned applications of NLP have been elucidated as follows:

####  3.1.1 Sentiment Analysis

Sentiment analysis is an application of NLP concerned with the extraction and evaluation of expressions, feelings, and orientations of people regarding a certain physical or abstract subject [[1](https://arxiv.org/html/2406.16893v1#bib.bib1)]. It has evolved over a period of time with primarily three tiers of analysis- document-based, sentence-based, and aspect-based. While Document-Based methods provide the overall sentiment for the entire document, they fail to capture the sentiments expressed in individual sentiments [[25](https://arxiv.org/html/2406.16893v1#bib.bib25), [58](https://arxiv.org/html/2406.16893v1#bib.bib58)]. Sentence-based methods provide the sentiment polarity associated with individual sentences in a document. They are an improvement over Document-based methods but falter to capture sentiments associated with the aspects present in a sentence [[26](https://arxiv.org/html/2406.16893v1#bib.bib26)]. Aspect-Based Sentiment Analysis (ABSA) redresses the impediments of Document-Based methods as-well-as Sentence-Based methods with its ability to associate sentiments to individual aspects [[1](https://arxiv.org/html/2406.16893v1#bib.bib1), [55](https://arxiv.org/html/2406.16893v1#bib.bib55)].

####  3.1.2 Misinformation Detection

Misinformation detection deals with identifying fake, biased, or propaganda-based content posted through online platforms. In contrast to classifying the polarity of opinions as in sentiment analysis, it detects fraudulent opinions. The detection methods may be based upon the content, the meta-data, or through learning some patterns present in the content [[2](https://arxiv.org/html/2406.16893v1#bib.bib2)]. It can be extended to fake news as-well-as rumor verification tasks. On one hand, fake news consists of news articles with delusive content to misinform the readers [[20](https://arxiv.org/html/2406.16893v1#bib.bib20)]. On the other hand, a rumor can be attributed to information that is rapidly disseminated without ascertaining its authenticity. Thus, a rumor might be true, false, or even unverified [[21](https://arxiv.org/html/2406.16893v1#bib.bib21)]. The approaches to detect fake news may exploit information present in the content of the post, user profile as-well-as social context [[27](https://arxiv.org/html/2406.16893v1#bib.bib27)].

####  3.1.3 Machine Translation

With increasing globalization, the entire world is becoming a single community. Therefore, it is becoming increasingly important to overcome linguistic barriers so that seamless transmission of knowledge and information can take place. Machine translation enables automatic conversion of a given piece of text from one language to another. This field is full of challenges due to multiple possible translations of a word depending upon the context and difficulty in understanding idiomatic phrases [[28](https://arxiv.org/html/2406.16893v1#bib.bib28)]. The advent of neural networks and encoder-decoder architectures for sequence-to-sequence models [[3](https://arxiv.org/html/2406.16893v1#bib.bib3), [4](https://arxiv.org/html/2406.16893v1#bib.bib4)] mitigated the impediments to a large extent. Subsequent transformer-based approaches [[115](https://arxiv.org/html/2406.16893v1#bib.bib115)] and associated LLMs [[18](https://arxiv.org/html/2406.16893v1#bib.bib18), [120](https://arxiv.org/html/2406.16893v1#bib.bib120)] have helped achieve SOTA performance.

####  3.1.4 Question Answering

Question Answering (QA) is an application of NLP that focuses on inventing and developing models and algorithms to automatically produce human-like responses to user queries or questions [[6](https://arxiv.org/html/2406.16893v1#bib.bib6)]. The objective is to make it possible for computers to comprehend natural language input and produce pertinent, correct responses in a conversational style. The existing QA systems can be grouped into extractive QA and generative QA. The former selects the span of text from a document termed context which serves as the answer to a given question [[29](https://arxiv.org/html/2406.16893v1#bib.bib29)]. While the latter produces automatically generated nuanced answers on the basis of the comprehended information [[22](https://arxiv.org/html/2406.16893v1#bib.bib22)].

####  3.1.5 Text Summarization

Text summarization is another acclaimed application of NLP. It follows a similar approach as machine translation, with a difference in the fact that instead of producing output text for each sentence, it condenses the input to a concise form representing the vital information present in the text. The two main branches of text summarization are - extractive-summarization and abstractive-summarization. The earlier methods focused upon extractive-summarization, in which the entire sentences are extracted based upon their significance [[23](https://arxiv.org/html/2406.16893v1#bib.bib23), [30](https://arxiv.org/html/2406.16893v1#bib.bib30)]. Recent methods focus on the more critical approach of summarizing text by paraphrasing the information learned without using sentences from the original text. Such methods generally deploy deep learning-based sequence-to-sequence models [[24](https://arxiv.org/html/2406.16893v1#bib.bib24), [31](https://arxiv.org/html/2406.16893v1#bib.bib31)].

###  3.2 Evolution of NLP

Over the years, NLP has made considerable strides as a result of ground-breaking research, rising processing capacity, and the creation of complex language models. The development of NLP is evidence of the persistent effort to close the language and artificial intelligence gap. The major advancements from rule-based systems, conventional machine learning techniques, deep learning, and pre-trained language models have been outlined in this section with Table [1](https://arxiv.org/html/2406.16893v1#S3.T1 "Table 1 ‣ 3.2 Evolution of NLP ‣ 3 Overview of NLP ‣ A Survey on Transformers in NLP with Focus on Efficiency") presenting a comparative review of the notable contributions.

Table 1: An Overview of Notable Contributions in NLP Year |  Task Description |  Methods Used |  Author(s)  
---|---|---|---  
2008 |  Analysis of opinion spamming |  LR |  Jindal et al. [[2](https://arxiv.org/html/2406.16893v1#bib.bib2)]  
2010 |  Sentiment analysis based on dependency-tree |  CRF |  Nakagawa et al. [[32](https://arxiv.org/html/2406.16893v1#bib.bib32)]  
2013 |  CBoW and SG for vector representation of text |  CBoW and SG |  Mikolov et al. [[35](https://arxiv.org/html/2406.16893v1#bib.bib35)]  
2014 |  GloVe for word-vector representation |  GloVe |  Pennington et al. [[36](https://arxiv.org/html/2406.16893v1#bib.bib36)]  
2014 |  Doc2Vec representation of text |  Doc2Vec |  Le and Mikolov [[38](https://arxiv.org/html/2406.16893v1#bib.bib38)]  
2014 |  Machine translation using encoder-decoder architecture |  RNN |  Cho et al. [[3](https://arxiv.org/html/2406.16893v1#bib.bib3)]  
2014 |  Machine translation using neural networks |  RNN |  Bahdanau et al. [[4](https://arxiv.org/html/2406.16893v1#bib.bib4)]  
2014 |  Aspect-extraction and mining of opinions |  RNN |  Irsoy and Cardie [[39](https://arxiv.org/html/2406.16893v1#bib.bib39)]  
2015 |  Aspect-extraction and opinion mining |  Rule Based |  Liu et al. [[40](https://arxiv.org/html/2406.16893v1#bib.bib40)]  
2015 |  Targeting efficiency through Knowledge Distillation |  Neural Networks |  Hinton, Geoffrey [[41](https://arxiv.org/html/2406.16893v1#bib.bib41)]  
2016 |  FastText for word-vector representation |  FastText |  Joulin et al. [[42](https://arxiv.org/html/2406.16893v1#bib.bib42)]  
2016 |  Aspect extraction and opinion mining using multi-layer CNN |  CNN |  Poria et al. [[43](https://arxiv.org/html/2406.16893v1#bib.bib43)]  
2016 |  Abstractive text summarization using sequence-to-sequence architecture |  RNN |  Nallapati et al. [[24](https://arxiv.org/html/2406.16893v1#bib.bib24)]  
2017 |  Extractive text summarization using deep learning |  RNN |  Nallapati et al. [[30](https://arxiv.org/html/2406.16893v1#bib.bib30)]  
2017 |  Abstractive text summarization using pointer-generator networks |  LSTM |  See et al. [[31](https://arxiv.org/html/2406.16893v1#bib.bib31)]  
2017 |  A survey of fake news detection techniques |  - |  Shu et al. [[20](https://arxiv.org/html/2406.16893v1#bib.bib20)]  
2018 |  Universal Language Model Fine-tuning (ULMFit) |  LSTM |  Howard and Ruder [[44](https://arxiv.org/html/2406.16893v1#bib.bib44)]  
2018 |  Embeddings from Language-Models (ELMo) |  LSTM |  Peters et al. [[45](https://arxiv.org/html/2406.16893v1#bib.bib45)]  
2018 |  Generative-Pre-trained Transformer (GPT) |  Transformers |  Radford et al. [[11](https://arxiv.org/html/2406.16893v1#bib.bib11)]  
2018 |  Bidirectional Encoder Representations from Transformers (BERT) |  Transformers |  Devlin et al. [[8](https://arxiv.org/html/2406.16893v1#bib.bib8)]  
2019 |  Autoregressive pre-training using XLNet |  Transformers |  Yang et al. [[9](https://arxiv.org/html/2406.16893v1#bib.bib9)]  
2019 |  GPT-2 for unsupervised multi-task learning |  Transformers |  Radford et al. [[12](https://arxiv.org/html/2406.16893v1#bib.bib12)]  
2019 |  Recurrence-based chunking approach (Transformer-XL) |  Transformers |  Dai et al. [[46](https://arxiv.org/html/2406.16893v1#bib.bib46)]  
2019 |  Cross-layer parameter sharing with matrix decomposition |  Transformers |  Lan et al.[[47](https://arxiv.org/html/2406.16893v1#bib.bib47)]  
2020 |  Few-shot learning using GPT-3 |  Transformers |  Brown et al. [[13](https://arxiv.org/html/2406.16893v1#bib.bib13)]  
2020 |  Utilizing Hash similarity for token clustering (Reformer) |  Transformers |  Kitaev et al.[[48](https://arxiv.org/html/2406.16893v1#bib.bib48)]  
2020 |  Low-rank approximations of the self-attention matrix (Linformer) |  Transformers |  Wang et al. [[49](https://arxiv.org/html/2406.16893v1#bib.bib49)]  
2021 |  Utilizing K-means clustering for capturing patterns (Routing Transformer) |  Transformers |  Roy et al.[[50](https://arxiv.org/html/2406.16893v1#bib.bib50)]  
2021 |  Parameter sharing with downsampling (Perceiver) |  Transformers |  Jaegle et al. [[51](https://arxiv.org/html/2406.16893v1#bib.bib51)]  
2021 |  Representing model parameters with reduced precision through Quantization |  Transformers |  Radford et al. [[52](https://arxiv.org/html/2406.16893v1#bib.bib52)]  
2022 |  Effective data curation for efficient pre-trained models |  Transformers |  Zhang et al. [[53](https://arxiv.org/html/2406.16893v1#bib.bib53)]  
2023 |  Structured pruning for efficient pre-trained models |  Transformers |  Sajjad et al. [[54](https://arxiv.org/html/2406.16893v1#bib.bib54)]  
  
![Refer to caption](extracted/5597951/RuleBased.jpg) (a) Rule-Based

![Refer to caption](extracted/5597951/MachineLearning.jpg) (b) Traditional Machine Learning

![Refer to caption](extracted/5597951/DeepLearning.jpg) (c) Deep Learning

![Refer to caption](extracted/5597951/TransferLearning.jpg) (d) Transfer Learning

Figure 3: Prevalent Modeling Approaches in NLP

####  3.2.1 Rule-Based Approaches

The rule-based approaches utilize the grammatical rules behind sentence construction to extract the features and process a text. Even though these approaches are currently losing their charm, they are somewhat suitable for less intensive applications as they are unsupervised, domain-independent, and efficient with considerable accuracy. However, the accuracy of the rule-based approaches is dependent upon the grammatical construction of text [[40](https://arxiv.org/html/2406.16893v1#bib.bib40)]. The rule-based approaches can also be combined with other methods such as Convolutional Neural Networks (CNN) [[55](https://arxiv.org/html/2406.16893v1#bib.bib55)] or even transformer-based models like BERT [[56](https://arxiv.org/html/2406.16893v1#bib.bib56)]to give the dual benefit of satisfactory performance with efficiency.

####  3.2.2 Traditional Machine Learning Approaches

In traditional machine learning approaches, after pre-processing operations, the features are defined followed by extraction of the features. After that, using traditional machine learning techniques the processing is carried out. Some of the notable machine learning techniques are Naive Bayes (NB) Classifier [[57](https://arxiv.org/html/2406.16893v1#bib.bib57)], Support Vector Machines (SVM) [[58](https://arxiv.org/html/2406.16893v1#bib.bib58)], Logistic Regression (LR) [[2](https://arxiv.org/html/2406.16893v1#bib.bib2)], Maximum Entropy (ME) Classifier [[25](https://arxiv.org/html/2406.16893v1#bib.bib25)], k-Nearest Neighbour (k-NN) [[58](https://arxiv.org/html/2406.16893v1#bib.bib58)] and Conditional Random Fields (CRF) [[32](https://arxiv.org/html/2406.16893v1#bib.bib32)]. The efficiency of such models is considerably less than the rule-based approaches due to the requirement of a huge training-set resulting in exorbitant training time with accuracy governed by the choice of features [[55](https://arxiv.org/html/2406.16893v1#bib.bib55)].

####  3.2.3 Deep Learning Approaches

The advent of deep learning approaches enabled a model to learn the features by itself which was earlier not possible using traditional machine learning approaches. This led to the development of models that could understand the context better than traditional machine learning approaches like Recurrent Neural Networks (RNN) [[39](https://arxiv.org/html/2406.16893v1#bib.bib39)] and Convolutional Neural Networks (CNN) [[55](https://arxiv.org/html/2406.16893v1#bib.bib55), [43](https://arxiv.org/html/2406.16893v1#bib.bib43)]. However, RNNs failed to capture the relationship among words beyond a certain length [[15](https://arxiv.org/html/2406.16893v1#bib.bib15)] due to inherent “vanishing-gradient” and “exploding-gradient” issues [[59](https://arxiv.org/html/2406.16893v1#bib.bib59)]. The use of Long Short Term Memory (LSTM) ameliorated this issue with a gated mechanism having the gradients pruned leading to improved accuracy [[60](https://arxiv.org/html/2406.16893v1#bib.bib60)]. Further developments on LSTM led to pre-trained models like Embeddings from Language-Models (ELMo) [[45](https://arxiv.org/html/2406.16893v1#bib.bib45)] and Universal Language Model Fine-tuning (ULMFit) [[44](https://arxiv.org/html/2406.16893v1#bib.bib44)]. Subsequent developments led to transformer-based models as discussed in Section [4](https://arxiv.org/html/2406.16893v1#S4 "4 Transformers in NLP ‣ A Survey on Transformers in NLP with Focus on Efficiency"). Concerning sustainability, a point to ponder is that these deep learning models have better accuracy compared to traditional machine learning approaches, but are domain-dependent, and training them from scratch requires considerable computing resources [[15](https://arxiv.org/html/2406.16893v1#bib.bib15)].

##  4 Transformers in NLP

The Transformer-based approaches come under the purview of deep learning. However, due to the revolution in NLP brought about by them and the immense developments carried out, they deserve to be discussed separately in this section. The evolution of transformers, the concepts associated with it accompanied with the stages of modeling have been enunciated herein-below.

![Refer to caption](extracted/5597951/Transformer_Architecture.jpg) Figure 4: Illustration of the transformer architecture

###  4.1 The Evolution of Transformers

A series of developments paved the way for the transformers. Earlier works on NLG tasks like machine translation devised sequence-to-sequence models comprising two RNN blocks namely, the encoder and the decoder [[3](https://arxiv.org/html/2406.16893v1#bib.bib3), [110](https://arxiv.org/html/2406.16893v1#bib.bib110)]. Given an input sequence X=(x1,x2,…,xn)𝑋subscript𝑥1subscript𝑥2…subscript𝑥𝑛X=(x_{1},x_{2},...,x_{n})italic_X = ( italic_x start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT , italic_x start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT , … , italic_x start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT ), the RNN-based encoder derives a hidden representation H=(h1,h2,…,hn)𝐻subscriptℎ1subscriptℎ2…subscriptℎ𝑛H=(h_{1},h_{2},...,h_{n})italic_H = ( italic_h start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT , italic_h start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT , … , italic_h start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT ). Subsequently, a few other non-linear functions can also be applied to obtain the final H𝐻Hitalic_H. For tt⁢hsuperscript𝑡𝑡ℎt^{th}italic_t start_POSTSUPERSCRIPT italic_t italic_h end_POSTSUPERSCRIPT time-step, htsubscriptℎ𝑡h_{t}italic_h start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT is calculated from xtsubscript𝑥𝑡x_{t}italic_x start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT and ht−1subscriptℎ𝑡1h_{t-1}italic_h start_POSTSUBSCRIPT italic_t - 1 end_POSTSUBSCRIPT as shown in equation (1).

ht=R⁢N⁢N⁢(xt,ht−1)subscriptℎ𝑡𝑅𝑁𝑁subscript𝑥𝑡subscriptℎ𝑡1h_{t}=RNN(x_{t},h_{t-1})italic_h start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT = italic_R italic_N italic_N ( italic_x start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT , italic_h start_POSTSUBSCRIPT italic_t - 1 end_POSTSUBSCRIPT ) (1)

The decoder predicts one output token at each time-step on the basis of the previously predicted tokens y1,y2,…,yt−1subscript𝑦1subscript𝑦2…subscript𝑦𝑡1{y_{1},y_{2},...,y_{t-1}}italic_y start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT , italic_y start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT , … , italic_y start_POSTSUBSCRIPT italic_t - 1 end_POSTSUBSCRIPT and H𝐻Hitalic_H as a joint probability distribution shown in equation (2).

yt=p⁢(yt|y1,y2,…,yt−1,H)subscript𝑦𝑡𝑝conditionalsubscript𝑦𝑡subscript𝑦1subscript𝑦2…subscript𝑦𝑡1𝐻y_{t}=p(y_{t}|{y_{1},y_{2},...,y_{t-1}},H)italic_y start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT = italic_p ( italic_y start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT | italic_y start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT , italic_y start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT , … , italic_y start_POSTSUBSCRIPT italic_t - 1 end_POSTSUBSCRIPT , italic_H ) (2)

However, the above approach leads to loss of information as the length of the input sequence grows due to compression of information into a fixed-length vector. To ameliorate this issue, Bahdanau et al. [[4](https://arxiv.org/html/2406.16893v1#bib.bib4)] deployed a soft-search mechanism for identifying the significant tokens from the input sequence for the prediction of the output at a given time-step. For this, they introduce the term context vector ctsubscript𝑐𝑡c_{t}italic_c start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT derived from H𝐻Hitalic_H which weighs the significance of the token hidden states as shown in equation (3).

ct=∑i=1nαt⁢i⋅hisubscript𝑐𝑡superscriptsubscript𝑖1𝑛⋅subscript𝛼𝑡𝑖subscriptℎ𝑖c_{t}=\sum_{i=1}^{n}\alpha_{ti}\cdot h_{i}italic_c start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT = ∑ start_POSTSUBSCRIPT italic_i = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT italic_α start_POSTSUBSCRIPT italic_t italic_i end_POSTSUBSCRIPT ⋅ italic_h start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT (3)

where αt⁢isubscript𝛼𝑡𝑖\alpha_{ti}italic_α start_POSTSUBSCRIPT italic_t italic_i end_POSTSUBSCRIPT is a distribution (often a softmax) function as follows:

αt⁢i=e⁢x⁢p⁢(et⁢i)∑j=1ne⁢x⁢p⁢(et⁢j)subscript𝛼𝑡𝑖𝑒𝑥𝑝subscript𝑒𝑡𝑖superscriptsubscript𝑗1𝑛𝑒𝑥𝑝subscript𝑒𝑡𝑗\alpha_{ti}=\frac{exp(e_{ti})}{\sum_{j=1}^{n}exp(e_{tj})}italic_α start_POSTSUBSCRIPT italic_t italic_i end_POSTSUBSCRIPT = divide start_ARG italic_e italic_x italic_p ( italic_e start_POSTSUBSCRIPT italic_t italic_i end_POSTSUBSCRIPT ) end_ARG start_ARG ∑ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT italic_e italic_x italic_p ( italic_e start_POSTSUBSCRIPT italic_t italic_j end_POSTSUBSCRIPT ) end_ARG (4)

given that,

e⁢x⁢p⁢(et⁢i)=fa⁢(st−1,hi)𝑒𝑥𝑝subscript𝑒𝑡𝑖subscript𝑓𝑎subscript𝑠𝑡1subscriptℎ𝑖exp(e_{ti})=f_{a}(s_{t-1},h_{i})italic_e italic_x italic_p ( italic_e start_POSTSUBSCRIPT italic_t italic_i end_POSTSUBSCRIPT ) = italic_f start_POSTSUBSCRIPT italic_a end_POSTSUBSCRIPT ( italic_s start_POSTSUBSCRIPT italic_t - 1 end_POSTSUBSCRIPT , italic_h start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) (5)

Here, e⁢x⁢p⁢(et⁢i)𝑒𝑥𝑝subscript𝑒𝑡𝑖exp(e_{ti})italic_e italic_x italic_p ( italic_e start_POSTSUBSCRIPT italic_t italic_i end_POSTSUBSCRIPT ) evaluates the alignment between the output at position t𝑡titalic_t and the input tokens around position i𝑖iitalic_i. The fa⁢(∗)subscript𝑓𝑎f_{a}(*)italic_f start_POSTSUBSCRIPT italic_a end_POSTSUBSCRIPT ( ∗ ) function takes the previous hidden state st−1subscript𝑠𝑡1s_{t-1}italic_s start_POSTSUBSCRIPT italic_t - 1 end_POSTSUBSCRIPT of the RNN decoder and it⁢hsuperscript𝑖𝑡ℎi^{th}italic_i start_POSTSUPERSCRIPT italic_t italic_h end_POSTSUPERSCRIPT time-step hidden representation hisubscriptℎ𝑖h_{i}italic_h start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT. Finally, the decoder applies a non-linear function fd⁢(∗)subscript𝑓𝑑f_{d}(*)italic_f start_POSTSUBSCRIPT italic_d end_POSTSUBSCRIPT ( ∗ ) to generate the output ytsubscript𝑦𝑡y_{t}italic_y start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT for time-step t𝑡titalic_t as follows:

yt=fd⁢(yt−1,st,ct)subscript𝑦𝑡subscript𝑓𝑑subscript𝑦𝑡1subscript𝑠𝑡subscript𝑐𝑡y_{t}=f_{d}(y_{t-1},s_{t},c_{t})italic_y start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT = italic_f start_POSTSUBSCRIPT italic_d end_POSTSUBSCRIPT ( italic_y start_POSTSUBSCRIPT italic_t - 1 end_POSTSUBSCRIPT , italic_s start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT , italic_c start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ) (6)

This led to the foundation of the attention mechanism, an indispensable component of modern transformer architecture. To compute attention, the input is transformed into an embedded sequence Z∈ℝL×ℝD𝑍superscriptℝ𝐿superscriptℝ𝐷Z\in\mathbb{R}^{L}\times\mathbb{R}^{D}italic_Z ∈ blackboard_R start_POSTSUPERSCRIPT italic_L end_POSTSUPERSCRIPT × blackboard_R start_POSTSUPERSCRIPT italic_D end_POSTSUPERSCRIPT comprising token and positional embeddings where L𝐿Litalic_L is the sequence length and D𝐷Ditalic_D is embedding dimension. Then, key Kssubscript𝐾𝑠K_{s}italic_K start_POSTSUBSCRIPT italic_s end_POSTSUBSCRIPT, query Qssubscript𝑄𝑠Q_{s}italic_Q start_POSTSUBSCRIPT italic_s end_POSTSUBSCRIPT, and value Vssubscript𝑉𝑠V_{s}italic_V start_POSTSUBSCRIPT italic_s end_POSTSUBSCRIPT are calculated through linear transformations on the sequence Z𝑍Zitalic_Z as follows:

Qs,Ks,Vs=Wq⁢Z,Wk⁢Z,Wv⁢Zformulae-sequencesubscript𝑄𝑠subscript𝐾𝑠subscript𝑉𝑠superscript𝑊𝑞𝑍superscript𝑊𝑘𝑍superscript𝑊𝑣𝑍Q_{s},K_{s},V_{s}=W^{q}Z,W^{k}Z,W^{v}Zitalic_Q start_POSTSUBSCRIPT italic_s end_POSTSUBSCRIPT , italic_K start_POSTSUBSCRIPT italic_s end_POSTSUBSCRIPT , italic_V start_POSTSUBSCRIPT italic_s end_POSTSUBSCRIPT = italic_W start_POSTSUPERSCRIPT italic_q end_POSTSUPERSCRIPT italic_Z , italic_W start_POSTSUPERSCRIPT italic_k end_POSTSUPERSCRIPT italic_Z , italic_W start_POSTSUPERSCRIPT italic_v end_POSTSUPERSCRIPT italic_Z (7)

where, Wq,Wksuperscript𝑊𝑞superscript𝑊𝑘W^{q},W^{k}italic_W start_POSTSUPERSCRIPT italic_q end_POSTSUPERSCRIPT , italic_W start_POSTSUPERSCRIPT italic_k end_POSTSUPERSCRIPT and Wvsuperscript𝑊𝑣W^{v}italic_W start_POSTSUPERSCRIPT italic_v end_POSTSUPERSCRIPT ∈ℝD×DHabsentsuperscriptℝ𝐷𝐷𝐻\in\mathbb{R}^{D\times\frac{D}{H}}∈ blackboard_R start_POSTSUPERSCRIPT italic_D × divide start_ARG italic_D end_ARG start_ARG italic_H end_ARG end_POSTSUPERSCRIPT denote the weight matrices corresponding to Kssubscript𝐾𝑠K_{s}italic_K start_POSTSUBSCRIPT italic_s end_POSTSUBSCRIPT, Qssubscript𝑄𝑠Q_{s}italic_Q start_POSTSUBSCRIPT italic_s end_POSTSUBSCRIPT and Vssubscript𝑉𝑠V_{s}italic_V start_POSTSUBSCRIPT italic_s end_POSTSUBSCRIPT. The key K𝐾Kitalic_K represents the input features. These features might be at character-level, word-level, document-level, or a combination of multiple features. Qssubscript𝑄𝑠Q_{s}italic_Q start_POSTSUBSCRIPT italic_s end_POSTSUBSCRIPT is the vector whose relationship with K𝐾Kitalic_K is computed during attention computation. This is accomplished through a compatibility function fc⁢(∗)subscript𝑓𝑐f_{c}(*)italic_f start_POSTSUBSCRIPT italic_c end_POSTSUBSCRIPT ( ∗ ) as follows:

ea=fc⁢(Qs,Ks)subscript𝑒𝑎subscript𝑓𝑐subscript𝑄𝑠subscript𝐾𝑠e_{a}=f_{c}(Q_{s},K_{s})italic_e start_POSTSUBSCRIPT italic_a end_POSTSUBSCRIPT = italic_f start_POSTSUBSCRIPT italic_c end_POSTSUBSCRIPT ( italic_Q start_POSTSUBSCRIPT italic_s end_POSTSUBSCRIPT , italic_K start_POSTSUBSCRIPT italic_s end_POSTSUBSCRIPT ) (8)

One might notice the similarity between equation (8) and the alignment function in equation (5) wherein the alignment between the previous decoded token and the hidden states is computed. Furthermore, the fc⁢(∗)subscript𝑓𝑐f_{c}(*)italic_f start_POSTSUBSCRIPT italic_c end_POSTSUBSCRIPT ( ∗ ) can have varied forms as summarized in Table [2](https://arxiv.org/html/2406.16893v1#S4.T2 "Table 2 ‣ 4.1 The Evolution of Transformers ‣ 4 Transformers in NLP ‣ A Survey on Transformers in NLP with Focus on Efficiency"). Following this, the attention weights awsubscript𝑎𝑤a_{w}italic_a start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT are obtained after being fed into a distribution function fδ⁢(∗)subscript𝑓𝛿f_{\delta}(*)italic_f start_POSTSUBSCRIPT italic_δ end_POSTSUBSCRIPT ( ∗ ) to normalize the alignment scores and transform it into a probability distribution as follows:

aw=fδ⁢(ea)subscript𝑎𝑤subscript𝑓𝛿subscript𝑒𝑎a_{w}=f_{\delta}(e_{a})italic_a start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT = italic_f start_POSTSUBSCRIPT italic_δ end_POSTSUBSCRIPT ( italic_e start_POSTSUBSCRIPT italic_a end_POSTSUBSCRIPT ) (9)

Here too, the distribution functions can have varied forms with softmax activation being the most widely used [[7](https://arxiv.org/html/2406.16893v1#bib.bib7)]. To obtain the attention-weighted representation of the input Z′superscript𝑍′Z^{\prime}italic_Z start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT, pairwise inner product between Vssubscript𝑉𝑠V_{s}italic_V start_POSTSUBSCRIPT italic_s end_POSTSUBSCRIPT and awsubscript𝑎𝑤a_{w}italic_a start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT is computed as follows:

Z′=aw⋅Vssuperscript𝑍′⋅subscript𝑎𝑤subscript𝑉𝑠Z^{\prime}=a_{w}\cdot V_{s}italic_Z start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT = italic_a start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT ⋅ italic_V start_POSTSUBSCRIPT italic_s end_POSTSUBSCRIPT (10)

Vssubscript𝑉𝑠V_{s}italic_V start_POSTSUBSCRIPT italic_s end_POSTSUBSCRIPT represents the sequence vector upon which the attention weights are applied to determine the significant tokens. In most of the studies, Vssubscript𝑉𝑠V_{s}italic_V start_POSTSUBSCRIPT italic_s end_POSTSUBSCRIPT is considered identical to kssubscript𝑘𝑠k_{s}italic_k start_POSTSUBSCRIPT italic_s end_POSTSUBSCRIPT. Finally, the attention-based context vector Casubscript𝐶𝑎C_{a}italic_C start_POSTSUBSCRIPT italic_a end_POSTSUBSCRIPT is obtained as the element-wise sum of Z′superscript𝑍′Z^{\prime}italic_Z start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT such that elements with higher attention weights have more significance compared to lower attention weights as shown in equation (11).

Ca=∑zj′⁢, ⁢∀zj′∈Z′subscript𝐶𝑎subscriptsuperscript𝑧′𝑗, for-allsubscriptsuperscript𝑧′𝑗superscript𝑍′C_{a}=\sum z^{\prime}_{j}\textit{, }\forall z^{\prime}_{j}\in Z^{\prime}italic_C start_POSTSUBSCRIPT italic_a end_POSTSUBSCRIPT = ∑ italic_z start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT , ∀ italic_z start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT ∈ italic_Z start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT (11)

Table 2: Cost Behind Training Transformer-based Models Type | Representation | Reference  
---|---|---  
Similarity | fc⁢(Qs,Ks)=s⁢i⁢m⁢(Qs,Ks)subscript𝑓𝑐subscript𝑄𝑠subscript𝐾𝑠𝑠𝑖𝑚subscript𝑄𝑠subscript𝐾𝑠f_{c}(Q_{s},K_{s})=sim(Q_{s},K_{s})italic_f start_POSTSUBSCRIPT italic_c end_POSTSUBSCRIPT ( italic_Q start_POSTSUBSCRIPT italic_s end_POSTSUBSCRIPT , italic_K start_POSTSUBSCRIPT italic_s end_POSTSUBSCRIPT ) = italic_s italic_i italic_m ( italic_Q start_POSTSUBSCRIPT italic_s end_POSTSUBSCRIPT , italic_K start_POSTSUBSCRIPT italic_s end_POSTSUBSCRIPT ) | Graves et al. [[114](https://arxiv.org/html/2406.16893v1#bib.bib114)]  
Multiplicative or Dot | fc⁢(Qs,Ks)=QsT⋅Kssubscript𝑓𝑐subscript𝑄𝑠subscript𝐾𝑠⋅superscriptsubscript𝑄𝑠𝑇subscript𝐾𝑠f_{c}(Q_{s},K_{s})=Q_{s}^{T}\cdot K_{s}italic_f start_POSTSUBSCRIPT italic_c end_POSTSUBSCRIPT ( italic_Q start_POSTSUBSCRIPT italic_s end_POSTSUBSCRIPT , italic_K start_POSTSUBSCRIPT italic_s end_POSTSUBSCRIPT ) = italic_Q start_POSTSUBSCRIPT italic_s end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_T end_POSTSUPERSCRIPT ⋅ italic_K start_POSTSUBSCRIPT italic_s end_POSTSUBSCRIPT | Luong et al. [[115](https://arxiv.org/html/2406.16893v1#bib.bib115)]  
Scaled Multiplicative | fc⁢(Qs,Ks)=QsT⋅KsDsubscript𝑓𝑐subscript𝑄𝑠subscript𝐾𝑠⋅superscriptsubscript𝑄𝑠𝑇subscript𝐾𝑠𝐷f_{c}(Q_{s},K_{s})=\frac{Q_{s}^{T}\cdot K_{s}}{\sqrt{D}}italic_f start_POSTSUBSCRIPT italic_c end_POSTSUBSCRIPT ( italic_Q start_POSTSUBSCRIPT italic_s end_POSTSUBSCRIPT , italic_K start_POSTSUBSCRIPT italic_s end_POSTSUBSCRIPT ) = divide start_ARG italic_Q start_POSTSUBSCRIPT italic_s end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_T end_POSTSUPERSCRIPT ⋅ italic_K start_POSTSUBSCRIPT italic_s end_POSTSUBSCRIPT end_ARG start_ARG square-root start_ARG italic_D end_ARG end_ARG | Vaswani et al. [[7](https://arxiv.org/html/2406.16893v1#bib.bib7)]  
Bilinear | fc⁢(Qs,Ks)=QsT⋅W⋅Kssubscript𝑓𝑐subscript𝑄𝑠subscript𝐾𝑠⋅superscriptsubscript𝑄𝑠𝑇𝑊subscript𝐾𝑠f_{c}(Q_{s},K_{s})=Q_{s}^{T}\cdot W\cdot K_{s}italic_f start_POSTSUBSCRIPT italic_c end_POSTSUBSCRIPT ( italic_Q start_POSTSUBSCRIPT italic_s end_POSTSUBSCRIPT , italic_K start_POSTSUBSCRIPT italic_s end_POSTSUBSCRIPT ) = italic_Q start_POSTSUBSCRIPT italic_s end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_T end_POSTSUPERSCRIPT ⋅ italic_W ⋅ italic_K start_POSTSUBSCRIPT italic_s end_POSTSUBSCRIPT | Luong et al. [[115](https://arxiv.org/html/2406.16893v1#bib.bib115)]  
Additive | fc⁢(Qs,Ks)=WT⁢g⁢(W1⁢Ks+W2⁢Qs+b)subscript𝑓𝑐subscript𝑄𝑠subscript𝐾𝑠superscript𝑊𝑇𝑔subscript𝑊1subscript𝐾𝑠subscript𝑊2subscript𝑄𝑠𝑏f_{c}(Q_{s},K_{s})=W^{T}g(W_{1}K_{s}+W_{2}Q_{s}+b)italic_f start_POSTSUBSCRIPT italic_c end_POSTSUBSCRIPT ( italic_Q start_POSTSUBSCRIPT italic_s end_POSTSUBSCRIPT , italic_K start_POSTSUBSCRIPT italic_s end_POSTSUBSCRIPT ) = italic_W start_POSTSUPERSCRIPT italic_T end_POSTSUPERSCRIPT italic_g ( italic_W start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT italic_K start_POSTSUBSCRIPT italic_s end_POSTSUBSCRIPT + italic_W start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT italic_Q start_POSTSUBSCRIPT italic_s end_POSTSUBSCRIPT + italic_b ) | Bahdanau et al. [[4](https://arxiv.org/html/2406.16893v1#bib.bib4)]  
Source: Based on a study by Galassi et al. [[113](https://arxiv.org/html/2406.16893v1#bib.bib113)]  
Note: W,W1,W2𝑊subscript𝑊1subscript𝑊2W,W_{1},W_{2}italic_W , italic_W start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT , italic_W start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT and b𝑏bitalic_b are learnable parameters  
  
Often, there is only one input sequence and attention is computed solely based on it. It gave rise to self-attention or intra-attention, a concept refined in many later works [[111](https://arxiv.org/html/2406.16893v1#bib.bib111), [112](https://arxiv.org/html/2406.16893v1#bib.bib112)]. It is achieved by having the same vector for both Kssubscript𝐾𝑠K_{s}italic_K start_POSTSUBSCRIPT italic_s end_POSTSUBSCRIPT and Qssubscript𝑄𝑠Q_{s}italic_Q start_POSTSUBSCRIPT italic_s end_POSTSUBSCRIPT. In this manner, it helps to capture the relevance of a particular token in a sequence concerning other tokens in it. Furthermore, to accommodate parallel computation of attention at diverse positions, Multi-Head Attention (MHA) awmsubscriptsuperscript𝑎𝑚𝑤a^{m}_{w}italic_a start_POSTSUPERSCRIPT italic_m end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT was devised which concatenates awsubscript𝑎𝑤a_{w}italic_a start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT computations from all the Dhsubscript𝐷ℎD_{h}italic_D start_POSTSUBSCRIPT italic_h end_POSTSUBSCRIPT attention heads and projects them through Wo∈ℝD×ℝDsuperscript𝑊𝑜superscriptℝ𝐷superscriptℝ𝐷W^{o}\in\mathbb{R}^{D}\times\mathbb{R}^{D}italic_W start_POSTSUPERSCRIPT italic_o end_POSTSUPERSCRIPT ∈ blackboard_R start_POSTSUPERSCRIPT italic_D end_POSTSUPERSCRIPT × blackboard_R start_POSTSUPERSCRIPT italic_D end_POSTSUPERSCRIPT as depicted hereinbelow.

awm=C⁢o⁢n⁢c⁢a⁢t⁢e⁢n⁢a⁢t⁢e⁢(aw⁢[i])⁢Wo,∀i∈Dhformulae-sequencesubscriptsuperscript𝑎𝑚𝑤𝐶𝑜𝑛𝑐𝑎𝑡𝑒𝑛𝑎𝑡𝑒subscript𝑎𝑤delimited-[]𝑖superscript𝑊𝑜for-all𝑖subscript𝐷ℎa^{m}_{w}=Concatenate(a_{w}[i])W^{o},\forall{i\in D_{h}}italic_a start_POSTSUPERSCRIPT italic_m end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT = italic_C italic_o italic_n italic_c italic_a italic_t italic_e italic_n italic_a italic_t italic_e ( italic_a start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT [ italic_i ] ) italic_W start_POSTSUPERSCRIPT italic_o end_POSTSUPERSCRIPT , ∀ italic_i ∈ italic_D start_POSTSUBSCRIPT italic_h end_POSTSUBSCRIPT (12)

A milestone achievement was the transformer architecture with multi-head scaled inner-product attention mechanism by Vaswani et al. [[7](https://arxiv.org/html/2406.16893v1#bib.bib7)] as shown in Figure [4](https://arxiv.org/html/2406.16893v1#S4.F4 "Figure 4 ‣ 4 Transformers in NLP ‣ A Survey on Transformers in NLP with Focus on Efficiency"). This was the first time a sequence-to-sequence model entirely based on self-attention without any CNN or RNN units was proposed. The transformers with attention mechanism provide high performance with exceptional sequence representation abilities and support parallel training unlike the LSTM-based sequential methods [[7](https://arxiv.org/html/2406.16893v1#bib.bib7)]. Moreover, the genesis of transformer-based pre-trained models or LLMs has transformed the field of NLP providing relief from training the model from scratch. These models are pre-trained on large data-sets and just need to be fine-tuned as per the application. This helps to provide high accuracy with computational efficiency and robustness when applied in various domains thereby making them an apt choice in the current scenario [[7](https://arxiv.org/html/2406.16893v1#bib.bib7)]. One of the foundational LLM was OpenAI’s Generative-Pre-trained Transformer (OpenAI GPT) based upon transformer-decoder architecture with unidirectional context parsing. To overcome this limitation, Bidirectional Encoder Representations from Transformers (BERT) [[8](https://arxiv.org/html/2406.16893v1#bib.bib8)] adopted bidirectional context-parsing deploying a transformer-encoder architecture. However, BERT suffers from drawbacks like the exclusion of a "Mask" token during fine-tuning and parallel predictions without dependency consideration. These drawbacks have been resolved by its successor XLNet through "permutation language modeling" in which the prediction tokens are permuted randomly [[9](https://arxiv.org/html/2406.16893v1#bib.bib9)]. The successors of OpenAI GPT i.e. GPT-2 [[12](https://arxiv.org/html/2406.16893v1#bib.bib12)] and GPT-3 [[13](https://arxiv.org/html/2406.16893v1#bib.bib13)] further enhance the performance, efficiency, and reusability with the concept of "in-context learning". This feature further eliminates the need to fine-tune the model and the model just needs to be conditioned with the instances or description of the application. Apart from this, LLMs have been devised utilizing the entire transformer encoder-decoder architecture. The T5 transformer [[120](https://arxiv.org/html/2406.16893v1#bib.bib120)] is one such LLM that is pre-trained by predicting a span of tokens corresponding to a mask. Another variation in the form of PEGASUS [[146](https://arxiv.org/html/2406.16893v1#bib.bib146)] enforces masking of entire sentences as a pre-training objective and termed it Gap-Sentence Generation. Similarly, BART [[10](https://arxiv.org/html/2406.16893v1#bib.bib10)] comprises encoder-decoder blocks and applies noise to corrupt the input text and then attempts reconstruction through denoising. These are just a few examples and the rest of the paper presents several other transformer-based models supported with an interpretation of their efficiency.

###  4.2 Stages of Modeling

####  4.2.1 Pre-Training

Creating an LLM does not only revolve around devising a complex architecture with millions of parameters. Rather, models need to be trained on data-sets proportionate to the model size to deliver optimum performance [[116](https://arxiv.org/html/2406.16893v1#bib.bib116)]. Thus, large models need large data-sets. But, high-quality annotated data-sets are scarcely available for training a model in a supervised fashion. This is due to annotation being expensive, and requiring expertise in understanding the syntax, semantics as well as domain knowledge. However, there exists plenty of unannotated textual content that can be utilized to make LLMs learn vital representations through unsupervised or self-supervised learning. Previously training or Pre-Training LLMs on these tasks groom the model towards discerning linguistic intricacies, significantly enhancing the performance at downstream tasks with to faster convergence even with limited data. The inception of pre-training can be attributed to the surge in the development of deep convolutional models following the ImageNet666https://image-net.org/challenges/LSVRC/ challenge in the early 2010s. In NLP, Collobert et al. [[117](https://arxiv.org/html/2406.16893v1#bib.bib117)] first demonstrated the concept of pre-trained word embeddings generated from large unannotated corpora. Subsequently, the pre-trained versions of word embeddings like GloVe [[36](https://arxiv.org/html/2406.16893v1#bib.bib36)] and Word2Vec [[35](https://arxiv.org/html/2406.16893v1#bib.bib35)] were devised. In context to the Language Model, Dai and Le [[118](https://arxiv.org/html/2406.16893v1#bib.bib118)] became the torchbearer followed by other models like ELMo [[45](https://arxiv.org/html/2406.16893v1#bib.bib45)], ULMFit [[44](https://arxiv.org/html/2406.16893v1#bib.bib44)], GPT [[11](https://arxiv.org/html/2406.16893v1#bib.bib11)] and BERT [[8](https://arxiv.org/html/2406.16893v1#bib.bib8)]. Since then, a plethora of LLMs have been developed with an upward trend in associated research. There exist quite a few strategies for pre-training LLMs [[119](https://arxiv.org/html/2406.16893v1#bib.bib119)]. Out of them, a few significant ones have been mentioned herein-below.

  * •

Causal Language Modeling (CLM):  It relies on self-supervised language modeling to predict the next token in a sequence maximizing the likelihood of the conditional probability distribution over all the unique tokens based on the context. CLM works in a unidirectional manner, i.e. left-to-right manner. This implies that the context only includes the tokens to its left. CLM is more suited for NLG applications. A prominent example of an LLM using CLM is GPT [[11](https://arxiv.org/html/2406.16893v1#bib.bib11)]. For a given sequence X=(x1,…,x2,xn)𝑋subscript𝑥1…subscript𝑥2subscript𝑥𝑛X=(x_{1},...,x_{2},x_{n})italic_X = ( italic_x start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT , … , italic_x start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT , italic_x start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT ), the loss function of CLM is computed as follows:

ℒC⁢L⁢M=−∑t=1Tl⁢o⁢g⁢p⁢(xt|X<t)subscriptℒ𝐶𝐿𝑀superscriptsubscript𝑡1𝑇𝑙𝑜𝑔𝑝conditionalsubscript𝑥𝑡subscript𝑋absent𝑡\mathcal{L}_{CLM}=-\sum_{t=1}^{T}{logp(x_{t}|X_{<t})}caligraphic_L start_POSTSUBSCRIPT italic_C italic_L italic_M end_POSTSUBSCRIPT = - ∑ start_POSTSUBSCRIPT italic_t = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_T end_POSTSUPERSCRIPT italic_l italic_o italic_g italic_p ( italic_x start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT | italic_X start_POSTSUBSCRIPT < italic_t end_POSTSUBSCRIPT ) (12)

  * •

Masked Language Modeling (MLM): To ameliorate the limitation of CLM to attend only to tokens leftwards, MLM was devised where the context was constructed in a bidirectional fashion, i.e. allowing it to infer from tokens present in both right as well as left direction. This makes MLM the apt choice for NLU applications. An MLM usually works by masking out some random percentage of tokens in the sequence and then predicting those tokens based on the context. One of the famous LLM utilizing MLM is BERT [[8](https://arxiv.org/html/2406.16893v1#bib.bib8)]. For a given sequence X=(x1,…,x2,xn)𝑋subscript𝑥1…subscript𝑥2subscript𝑥𝑛X=(x_{1},...,x_{2},x_{n})italic_X = ( italic_x start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT , … , italic_x start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT , italic_x start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT ), the loss function of MLM is computed as follows:

ℒM⁢L⁢M=−∑x′∈m⁢(X)l⁢o⁢g⁢p⁢(x′|X\⁢m⁢(X))subscriptℒ𝑀𝐿𝑀subscriptsuperscript𝑥′𝑚𝑋𝑙𝑜𝑔𝑝conditionalsuperscript𝑥′subscript𝑋\𝑚𝑋\mathcal{L}_{MLM}=-\sum_{x^{\prime}\in m(X)}{logp(x^{\prime}|X_{\texttt{% \textbackslash}m(X)})}caligraphic_L start_POSTSUBSCRIPT italic_M italic_L italic_M end_POSTSUBSCRIPT = - ∑ start_POSTSUBSCRIPT italic_x start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT ∈ italic_m ( italic_X ) end_POSTSUBSCRIPT italic_l italic_o italic_g italic_p ( italic_x start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT | italic_X start_POSTSUBSCRIPT \ italic_m ( italic_X ) end_POSTSUBSCRIPT ) (13)

where, m⁢(X)𝑚𝑋m(X)italic_m ( italic_X ), X\⁢m⁢(X)subscript𝑋\𝑚𝑋X_{\texttt{\textbackslash}m(X)}italic_X start_POSTSUBSCRIPT \ italic_m ( italic_X ) end_POSTSUBSCRIPT denote the masked tokens, and the remaining tokens in the sequence X𝑋Xitalic_X respectively. Vanilla MLM deals with replacing single tokens which can reduce their effectiveness at sequence-to-sequence NLG tasks. A sequence-to-sequence variation of MLM solves this by predicting a span of tokens corresponding to a mask as can be seen in T5 transformer [[120](https://arxiv.org/html/2406.16893v1#bib.bib120)]. Subsequently, even entire sentences have been masked in LLMs like PEGASUS [[146](https://arxiv.org/html/2406.16893v1#bib.bib146)] to make the pre-training objective related to the downstream task of abstractive summarization. LLMs like BART [[10](https://arxiv.org/html/2406.16893v1#bib.bib10)] apply noise to corrupt the input text and then perform denoising by reconstructing the span of text. This allows pre-training on shorter sequences with equivalent efficacy contributing towards enhanced efficiency. A limitation of MLM is that the masked tokens are restricted to pre-training and are not available at the fine-tuning phase leading to a discrepancy.

  * •

Permutation Language Modeling (PLM): To mitigate the drawback of MLM related to the unavailability of the mask token during the fine-tuning stage, PLM was proposed [[9](https://arxiv.org/html/2406.16893v1#bib.bib9)]. PLM generates a random permutation of the input sequence wherein a permutation defines the order of token predictions (not to be confused with the order of tokens in the sequence). During pre-training, the model tries to predict some of the tokens selected as the target considering its position and the remaining tokens. To achieve faster convergence, the endmost tokens are often predicted. A popular LLM formulated on this pre-training objective is XLNet [[9](https://arxiv.org/html/2406.16893v1#bib.bib9)]. Given an input sequence X𝑋Xitalic_X with S𝑆Sitalic_S being its random permutation sequence, the equation for the loss function of PLM is as follows:

ℒP⁢L⁢M=−∑t=1Tl⁢o⁢g⁢p⁢(st|S<t)subscriptℒ𝑃𝐿𝑀superscriptsubscript𝑡1𝑇𝑙𝑜𝑔𝑝conditionalsubscript𝑠𝑡subscript𝑆absent𝑡\mathcal{L}_{PLM}=-\sum_{t=1}^{T}{logp(s_{t}|S_{<t})}caligraphic_L start_POSTSUBSCRIPT italic_P italic_L italic_M end_POSTSUBSCRIPT = - ∑ start_POSTSUBSCRIPT italic_t = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_T end_POSTSUPERSCRIPT italic_l italic_o italic_g italic_p ( italic_s start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT | italic_S start_POSTSUBSCRIPT < italic_t end_POSTSUBSCRIPT ) (14)

  * •

Contrastive Learning (CL): Contrastive learning aims to capture linguistic contextual information by distinguishing (contrasting) between valid and invalid samples by means of similarity evaluation. Next-Sentence Prediction (NSP) is an example of CL utilized in BERT [[8](https://arxiv.org/html/2406.16893v1#bib.bib8)]. Here, the objective is to identify whether a pair of sentences are next to each other given a set of contiguous and non-contiguous sentences. However, a few works have stated that although NSP focuses on the topic as well as coherence prediction, it is found to be ineffective and unreliable in coherence prediction even demonstrating performance drop due to NSP [[121](https://arxiv.org/html/2406.16893v1#bib.bib121)]. To resolve this issue Sentence-Order Prediction (SOP) was proposed to predict the order of sentences instead of predicting whether a given sentence is the next sentence to another sentence. The LLM ALBERT showcases superior performance by modeling the inter-sentence coherence through SOP [[47](https://arxiv.org/html/2406.16893v1#bib.bib47)]. The loss functions for both SOP and NSP aim to determine the constructiveness of two sentences X𝑋Xitalic_X and Y𝑌Yitalic_Y as follows:

ℒN⁢S⁢P/S⁢O⁢P=−l⁢o⁢g⁢p⁢(k|X,Y)⁢, ⁢∀p∈0,1formulae-sequencesubscriptℒ𝑁𝑆𝑃𝑆𝑂𝑃𝑙𝑜𝑔𝑝conditional𝑘𝑋𝑌, for-all𝑝01\mathcal{L}_{NSP/SOP}=-logp(k|X,Y)\textit{, }\forall p\in{0,1}caligraphic_L start_POSTSUBSCRIPT italic_N italic_S italic_P / italic_S italic_O italic_P end_POSTSUBSCRIPT = - italic_l italic_o italic_g italic_p ( italic_k | italic_X , italic_Y ) , ∀ italic_p ∈ 0 , 1 (15)

Regarding efficiency considerations of LLMs, it can be said that pre-training requires the maximum computational resources among all the stages of modeling. Although the pre-training strategies contribute to a great extent towards the performance, the model design along with the quality and size of the data upon which pre-training is performed plays a crucial role in efficiency [[19](https://arxiv.org/html/2406.16893v1#bib.bib19)]. The efficient data curation as well as model design considerations have been discussed in Section [5.2.1](https://arxiv.org/html/2406.16893v1#S5.SS2.SSS1 "5.2.1 Data Curation ‣ 5.2 Software Designing ‣ 5 Efficient Modeling Considerations ‣ A Survey on Transformers in NLP with Focus on Efficiency") and Section [5.2.3](https://arxiv.org/html/2406.16893v1#S5.SS2.SSS3 "5.2.3 Model Design ‣ 5.2 Software Designing ‣ 5 Efficient Modeling Considerations ‣ A Survey on Transformers in NLP with Focus on Efficiency").




####  4.2.2 Fine-Tuning

As seen above, pre-training an LLM serves as an effective model initialization strategy and aids in generalization with faster convergence on limited annotated data. However, to make a pre-trained model excel at a domain-specific task, additional training effort is required to exploit annotated samples specific to the downstream task. It is known as fine-tuning. It underlies the concept of transfer learning wherein a model pre-trained on a certain task having large data is trained again (fine-tuned) on a related task with significantly fewer data. There are various fine-tuning approaches. The first approach is to unfreeze a few layers of the model and retain the weights of the other layers calculated during pre-training. Usually, the output layer is customized as per the output representation format and fine-tuned with a few other unfrozen layers upon the task-specific data. The second approach is to fine-tune the frozen model with limited data during initiation and unfreeze other layers in due course.

The efficiency considerations for fine-tuning lie in minimizing the number of layers to unfreeze, i.e. number of parameters of the pre-trained LLM to fine-tune. Unfreezing more layers increases the computational requirements of fine-tuning but can enhance the accuracy of the downstream task. This applies only if abundant data is available to perform FT. In most cases, fine-tuning only the last few layers can obtain desirable results [[122](https://arxiv.org/html/2406.16893v1#bib.bib122)]. This is due to the fact the lower layers capture low-level, local features primarily related to the syntax. Whereas, the higher layers capture the global information involving high-level semantic abstractions specific to the task at hand. The efficiency can also be improved through adapter modules, i.e. an isolated network that is fine-tuned and combined with the pre-trained model having all the parameters intact [[123](https://arxiv.org/html/2406.16893v1#bib.bib123)]. Further variations include utilizing Kronecker product of low-rank matrices for the construction of parameter matrices for the adapter [[124](https://arxiv.org/html/2406.16893v1#bib.bib124)]. Another variation involves reparameterization to low-dimensional subspaces for fine-tuning, enhancing efficiency by reducing the number of parameter updates [[125](https://arxiv.org/html/2406.16893v1#bib.bib125)]. There lies one drawback of the adapter approach- it raises the overall model parameters leading to more computations during inference. This hindrance was resolved through Adaptable Adapters which applies differing activations specific to each layer as well as data-sets accompanied with a switch trained to select appropriate layers of the adapter module [[126](https://arxiv.org/html/2406.16893v1#bib.bib126)]. Furthermore, AdaMix combined various parameter-efficient adapters to provide SOTA results with an efficiency equivalent to fine-tuning with a single adapter module [[127](https://arxiv.org/html/2406.16893v1#bib.bib127)].

![Refer to caption](extracted/5597951/PromptEngineering.jpg) Figure 5: Illustration of prompt engineering process

####  4.2.3 Prompt Engineering

GPT-2 [[12](https://arxiv.org/html/2406.16893v1#bib.bib12)] first demonstrated the multi-task learning ability of generative LLMs. It was capable of performing various tasks out of the box minimizing manual effort during inference. Its subsequent version, GPT-3 [[13](https://arxiv.org/html/2406.16893v1#bib.bib13)] was further able to perform few-shot or in-context learning, i.e. it could generate the required predictions just by providing the task description along with priming with a few use-cases. This led to the emergence of the term Prompting or Prompt Engineering associated with a series of developments. Schick and Schutze [[128](https://arxiv.org/html/2406.16893v1#bib.bib128)] utilized a pre-trained LLM for in-context learning which excelled at tasks such as predicting the next sentence given the first sentence, generating the second sentence replacing "__" in the prompt or even generating both the sentences through providing the description. Reynolds and McDonell [[129](https://arxiv.org/html/2406.16893v1#bib.bib129)] demonstrated the mathematical reasoning ability of pre-trained LLMs where given a mathematical problem, the LLM could generate the solution with detailed steps. Wei et al. [[130](https://arxiv.org/html/2406.16893v1#bib.bib130)] further demonstrated that an LLM’s multi-task learning abilities can be enhanced by prompt-learning on several supervised data-sets concerning various tasks. From the current state of developments [[130](https://arxiv.org/html/2406.16893v1#bib.bib130), [131](https://arxiv.org/html/2406.16893v1#bib.bib131), [132](https://arxiv.org/html/2406.16893v1#bib.bib132), [133](https://arxiv.org/html/2406.16893v1#bib.bib133), [134](https://arxiv.org/html/2406.16893v1#bib.bib134), [135](https://arxiv.org/html/2406.16893v1#bib.bib135)], it can be inferred that the approaches for prompt engineering can be primarily clustered into the following categories:

  * •

Instruction-based Learning:  Also known as Priming, it involves providing the instructions related to the task description optionally with a few samples of the inputs and their corresponding outputs [[130](https://arxiv.org/html/2406.16893v1#bib.bib130), [132](https://arxiv.org/html/2406.16893v1#bib.bib132)]. For instance, providing the instruction to perform translation accompanied with a few examples in the prompt to prime the LLM to generate a translation for any new sentence.

  * •

Template-based Learning:  It deals with exploiting predefined structures, known as templates to construct prompts. The templates can be designed as cloze styled\- inserting placeholders in the prompt text and attempting to fill in the blanks [[133](https://arxiv.org/html/2406.16893v1#bib.bib133)], multiple-choice type\- providing multiple hypotheses in the template and asking the model to choose the correct one [[134](https://arxiv.org/html/2406.16893v1#bib.bib134)] or prefix-type\- adding special prefixes before the input to denote the task to be performed on the input [[131](https://arxiv.org/html/2406.16893v1#bib.bib131), [132](https://arxiv.org/html/2406.16893v1#bib.bib132)].

  * •

Proxy-Task-based Learning:  It involves probing an LLM with a proxy-task, i.e. a related task sharing some attributes of the original task, to obtain the output of the original task through transferring the inference to the desired form. This enhances the efficiency and eases inference due to utilizing simpler tasks closer to those upon which the model has been previously trained to obtain outputs for tasks leveraging rigorous linguistic comprehension. Instances include applying textual entailment for topic detection [[135](https://arxiv.org/html/2406.16893v1#bib.bib135)] or achieving coreference resolution through question answering [[136](https://arxiv.org/html/2406.16893v1#bib.bib136)].




Regarding the efficiency of prompt engineering approaches, it can be commented that in-context learning significantly reduces the computational complexity due to zero parameter updates in the pre-trained LLM. For a multi-task LLM, prompting can yield results at par with fine-tuning the model with several data samples [[131](https://arxiv.org/html/2406.16893v1#bib.bib131)]. Apart from these, certain prompt engineering practices also enhance efficiency. Firstly, optimizing the length of the prompt and its textual complexity improves the response-time by requiring less computations. Secondly, designing prompts considering the resources available and allowing batch processing can improve efficiency. Thirdly, caching the intermediate outputs can reduce the amount of processing required leading to faster response. Finally, the selection of the LLM for prompt-engineering plays a crucial role. The selection must be done considering the desired performance given the availability of computational resources.

##  5 Efficient Modeling Considerations

The capability of transformer-based models to deliver high accuracy at tasks might seem undisputed. But, they are highly complex requiring humongous parameters. The first factor behind this is the O⁢(L2⋅D)𝑂⋅superscript𝐿2𝐷O(L^{2}\cdot D)italic_O ( italic_L start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ⋅ italic_D ) complexity of attention computation [[7](https://arxiv.org/html/2406.16893v1#bib.bib7)]. The second factor is the number of attention heads involved in MHA. Even if MHA supports parallel computation, it raises the complexity of the model and requires sophisticated hardware for implementation. The third factor is the number of transformer blocks involved. This increases the number of sequential layers scaling up the execution time. The fourth factor is the number of embedding dimensions D𝐷Ditalic_D. It can be observed that self-attention is directly proportional to D𝐷Ditalic_D and thus determining the optimal embedding dimension is very crucial as very high embedding dimensions might incur redundancy while lower values of D𝐷Ditalic_D might lead to information loss. The fifth factor is the size of the data being trained on. The potency of current SOTA models to offer zero-shot or few-shot learning is due to the humongous data they are being trained upon. This results in gigantic model sizes. A single training instance might be as expensive as $40,000 [[16](https://arxiv.org/html/2406.16893v1#bib.bib16)] or even more. The remainder of this section discusses the efficiency measures along with the developments to improve the efficiency of transformer-based models both from a software as-well-as hardware perspective.

###  5.1 Efficiency Measures

The developments to achieve better performance at tasks are at the cost of increased model complexity translating to escalated training costs and carbon emissions. Given the complexity of SOTA NLP models, the cost of training might even exceed the annual energy requirements of certain cities. Strubell et al. [[16](https://arxiv.org/html/2406.16893v1#bib.bib16)] performed a study in which they calculated the power consumption, carbon emissions along with the monetary cost associated with the training of a set of NLP models. In their study, it was found that training a NLP model cost as much as a trans-Atlantic flight. They also reflected on the percentage of energy coming from renewable sources from countries all over the world. To measure the efficiency η𝜂\etaitalic_η, the trade-off between the model performance and cost factors needs to be calculated as shown in equation (1). The cost factors can be defined concerning various metrics as follows:

  1. 1.

Floating-point Operations (FlOps) define the number of floating-point operations needed for a single instance computation [[19](https://arxiv.org/html/2406.16893v1#bib.bib19)]. This can serve as a consistent benchmark irrespective of the hardware of the application. However, existing HPCs with support for parallel processing might lead to non-uniform execution times even with the same number of FlOps.

  2. 2.

Inference Time denotes the time required by the model to process a test input and generate a suitable response [[141](https://arxiv.org/html/2406.16893v1#bib.bib141)]. Unlike FlOps, it is hardware-dependent, i.e. it depends upon the configuration of the HPC and support for parallel execution. From the evaluation perspective, it enables a real-time measure of various algorithms based on execution upon identical HPC.

  3. 3.

Speed-up Ratio helps to perform comparison of a model concerning another model [[141](https://arxiv.org/html/2406.16893v1#bib.bib141)]. Here, one model is taken as the baseline and the improvement in efficiency of the other model is measured compared to it. In context to transformer-based models, speed-up can be calculated based on the number of transformer blocks, attention heads, or overall number of layers in the model.

  4. 4.

Model Size and Number of Parameters are internal indicators of the computational requirements [[18](https://arxiv.org/html/2406.16893v1#bib.bib18)]. Some models might be more efficient despite the same or even more number of layers and FlOps due to the sharing of parameters [[47](https://arxiv.org/html/2406.16893v1#bib.bib47)]. In such cases, the number of model parameters provides an indicator to the overall model size and serves as an efficiency evaluation metric.

  5. 5.

Carbon Footprint is the most significant indicator of the environmental impact due to an LLM. However, it is an uphill task to precisely report the carbon emissions due to the involvement of multiple factors for its computation [[16](https://arxiv.org/html/2406.16893v1#bib.bib16), [18](https://arxiv.org/html/2406.16893v1#bib.bib18)]. The preliminary approaches involve tools to calculate the energy consumption and carbon footprint relying on the execution time, number of cores, memory requirements, and platform information supplied by the user [[105](https://arxiv.org/html/2406.16893v1#bib.bib105), [106](https://arxiv.org/html/2406.16893v1#bib.bib106)]. Further developments led to packages being deployed on systems to directly access the CPU, GPU and DRAM statistics and calculate power consumption777https://github.com/epfl-iglobalhealth/cumulator [[107](https://arxiv.org/html/2406.16893v1#bib.bib107)]. However, most of these studies only account for the computing resources and do not consider the cooling, networking and other operational costs.




η=P⁢e⁢r⁢f⁢o⁢r⁢m⁢a⁢n⁢c⁢eC⁢o⁢s⁢t⁢F⁢a⁢c⁢t⁢o⁢r⁢s𝜂𝑃𝑒𝑟𝑓𝑜𝑟𝑚𝑎𝑛𝑐𝑒𝐶𝑜𝑠𝑡𝐹𝑎𝑐𝑡𝑜𝑟𝑠\eta=\frac{Performance}{CostFactors}italic_η = divide start_ARG italic_P italic_e italic_r italic_f italic_o italic_r italic_m italic_a italic_n italic_c italic_e end_ARG start_ARG italic_C italic_o italic_s italic_t italic_F italic_a italic_c italic_t italic_o italic_r italic_s end_ARG (1)

For performance, it is necessary to discover the pareto-improvement by comparing it with a benchmark, i.e. attaining higher accuracy at lower cost [[108](https://arxiv.org/html/2406.16893v1#bib.bib108)]. Schwarts et al. [[18](https://arxiv.org/html/2406.16893v1#bib.bib18)] formulated cost factors proportional to the time and resources for execution on a single sample Essubscript𝐸𝑠E_{s}italic_E start_POSTSUBSCRIPT italic_s end_POSTSUBSCRIPT, data size Dssubscript𝐷𝑠D_{s}italic_D start_POSTSUBSCRIPT italic_s end_POSTSUBSCRIPT and the number of epochs n𝑛nitalic_n required for training as depicted in equation (2).

C⁢o⁢s⁢t⁢F⁢a⁢c⁢t⁢o⁢r⁢s∝Es⋅Ds⋅nproportional-to𝐶𝑜𝑠𝑡𝐹𝑎𝑐𝑡𝑜𝑟𝑠⋅subscript𝐸𝑠subscript𝐷𝑠𝑛CostFactors\propto E_{s}\cdot D_{s}\cdot nitalic_C italic_o italic_s italic_t italic_F italic_a italic_c italic_t italic_o italic_r italic_s ∝ italic_E start_POSTSUBSCRIPT italic_s end_POSTSUBSCRIPT ⋅ italic_D start_POSTSUBSCRIPT italic_s end_POSTSUBSCRIPT ⋅ italic_n (2)

Despite the research developments, the current approaches for measuring efficiency are not fool-proof. There lies a disparity in the carbon emissions reported by various monitoring applications. The majority of the studies focus on only model training or do not differentiate between fine-tuning or prompt-engineering stages. Furthermore, the cost of production of hardware and infrastructure for deployment of these models is often unaccounted for. A study by Gupta et al. [[109](https://arxiv.org/html/2406.16893v1#bib.bib109)] reveals that the environmental impact due to setting up infrastructure and hardware equipment is maximum compared to other life-cycle stages for data-centers.

###  5.2 Software Designing

To achieve efficiency in NLP models, numerous software design considerations have been devised to target various stages of model development as highlighted in Figure [6](https://arxiv.org/html/2406.16893v1#S5.F6 "Figure 6 ‣ 5.2 Software Designing ‣ 5 Efficient Modeling Considerations ‣ A Survey on Transformers in NLP with Focus on Efficiency"). In this section, commentary on such techniques based on the modeling stages such as data curation, text representation, model design, and model compression have been presented.

![Refer to caption](extracted/5597951/SoftwareDesign.jpg) Figure 6: Efficiency considerations through software designing

####  5.2.1 Data Curation

Data curation plays a vital role in determining the efficiency of the Language Model (LM). A data set with reduced sequence lengths or less number of training samples minimizes the model complexity and reduces the training effort significantly [[61](https://arxiv.org/html/2406.16893v1#bib.bib61)]. Duplicate removal from the data set can enhance the efficiency of LM and might also enhance or improve its performance compared to the entire corpus [[62](https://arxiv.org/html/2406.16893v1#bib.bib62)]. In the case of pre-trained LMs, such filtering can be applied both during the pre-training [[53](https://arxiv.org/html/2406.16893v1#bib.bib53)] as-well-as the fine-tuning stages [[63](https://arxiv.org/html/2406.16893v1#bib.bib63)]. Although filtering eliminates biases inherent in the data set, their application is restricted to cases with abundant data as the performance reduces when insufficient data is available [[64](https://arxiv.org/html/2406.16893v1#bib.bib64)].

While duplicate removal applies to already available data sets, Active Learning comes into play while collecting data. It aims to reduce the training data while retaining model performance by labeling the most informative samples and selecting them for training [[65](https://arxiv.org/html/2406.16893v1#bib.bib65)]. For the identification of informative samples, various approaches have been adopted such as selecting samples with high uncertainty [[66](https://arxiv.org/html/2406.16893v1#bib.bib66)], maximum diversity [[67](https://arxiv.org/html/2406.16893v1#bib.bib67)] or both [[68](https://arxiv.org/html/2406.16893v1#bib.bib68)]. However, determining the usefulness of the samples and annotating them is a challenging task [[69](https://arxiv.org/html/2406.16893v1#bib.bib69)]. Its efficacy in diverse downstream tasks cannot be ascertained and can include outliers [[70](https://arxiv.org/html/2406.16893v1#bib.bib70), [71](https://arxiv.org/html/2406.16893v1#bib.bib71)].

Another perspective to data curation can be to order the samples in the data-set to improve utilization also known as Curriculum Learning. The ordering approach deploys heuristics capturing the complexity of sequences and determines a pace to progressively move from simpler sequences to complex sequences [[72](https://arxiv.org/html/2406.16893v1#bib.bib72)]. However, the pace has to be monitored to guarantee efficiency, and automation of the pace serves to be beneficial [[73](https://arxiv.org/html/2406.16893v1#bib.bib73)].

Establishing a balance between the size of training data and the model parameters is also important to achieve pareto-improvement as mentioned in section [5.1](https://arxiv.org/html/2406.16893v1#S5.SS1 "5.1 Efficiency Measures ‣ 5 Efficient Modeling Considerations ‣ A Survey on Transformers in NLP with Focus on Efficiency"). Hoffmann et al. [[116](https://arxiv.org/html/2406.16893v1#bib.bib116)] state that the number of model parameters and the size of the training set must be in the same ratio. They showcased that their model named Chinchilla based on this theory outperformed several SOTA models with a significantly higher number of parameters.

Thus it can be inferred that determining the quality of samples in the corpus and selecting high-quality samples devoid of repetitive information, outliers and incorrectly ordered sequences can boost the modeling efficiency. Moreover, this can be extended to decomposing the individual text sequences into smaller sub-sequences with essential information and discarding the irrelevant portions leading to efficient representation of context [[56](https://arxiv.org/html/2406.16893v1#bib.bib56)]. This significantly enhances the efficiency of transformer-based models with attention mechanisms having complexity quadratically proportional to sequence length [[7](https://arxiv.org/html/2406.16893v1#bib.bib7)].

####  5.2.2 Representation of Text

After data curation, the next step is to target an efficient representation of text. Earlier methods like Bag-of-Words [[33](https://arxiv.org/html/2406.16893v1#bib.bib33)] produced sparse vectors with enormous dimensions equal to the vocabulary size and did not account for statistical relationships and ordinal information [[33](https://arxiv.org/html/2406.16893v1#bib.bib33)]. The quest for the advancement of text representation and amelioration of the shortcomings of previous works led to the bifurcation of approaches into categories like matrix-factorization and context-window-based models. Among the matrix-factorization approaches, Latent Semantic Analysis (LSA) [[34](https://arxiv.org/html/2406.16893v1#bib.bib34)] could comprehend the statistical information but failed to capture term relationships. On the other hand, Global Vectors for Word Representation (GloVe) [[36](https://arxiv.org/html/2406.16893v1#bib.bib36)] used log-bilinear regression and addressed the inability to capture term relationships by LSA. The context-window family of approaches can be attributed to Neural-Network Language Model (NNLM) [[37](https://arxiv.org/html/2406.16893v1#bib.bib37)]. It was succeeded by Word2Vec approaches like Continuous Bag-of-Words (CBoW) and Skip-Gram (SG) [[35](https://arxiv.org/html/2406.16893v1#bib.bib35)] which effectively captured local-context information and its features like hierarchical-softmax, negative-sampling and frequent word sub-sampling boosted its efficiency. This was further improved by the fastText approach [[42](https://arxiv.org/html/2406.16893v1#bib.bib42)] with a manifold rise in efficiency using a constraint upon rank and fast-loss estimation.

While these embeddings were successful in capturing semantic information, they failed to do so for context-specific meaning. When meeting polysemous words, or words whose meanings change depending on the situation, this issue is exacerbated. As a result, contextualized word representations such as Context2Vec [[74](https://arxiv.org/html/2406.16893v1#bib.bib74)], ELMo [[45](https://arxiv.org/html/2406.16893v1#bib.bib45)], BERT [[8](https://arxiv.org/html/2406.16893v1#bib.bib8)], and others were developed. Although these representations are state-of-the-art (SOTA), each token in the sequence has multiple dimensions, which contributes to the "curse of dimensionality" problem with a high memory cost and more complex model parameters.

Sentence-level representations have been found to achieve a better performance vs efficiency tradeoff compared to word-level embeddings [[75](https://arxiv.org/html/2406.16893v1#bib.bib75)]. The Doc2Vec is one of the pioneering approaches in this domain that attempted to represent a document as a vector through supervised learning [[38](https://arxiv.org/html/2406.16893v1#bib.bib38)]. The Universal Sentence Encoder consisting of a deep averaging-network version and a transformer-based model exhibited appreciable ability for embedding sentences at the sentence level [[76](https://arxiv.org/html/2406.16893v1#bib.bib76)]. Sentence-BERT, which has expanded on the BERT architecture, is another noteworthy strategy [[77](https://arxiv.org/html/2406.16893v1#bib.bib77)]. Cross-lingual variations leveraging BERT [[78](https://arxiv.org/html/2406.16893v1#bib.bib78)] were produced as a result of further developments. Compared to word embeddings, these methods produce optimized fixed-length representations with minimal memory usage.

To ameliorate the curse of dimensionality, a few studies have been conducted to determine the optimal embedding dimensions to reduce the excessive memory consumption retaining the semantic and syntactic characteristics in the data [[79](https://arxiv.org/html/2406.16893v1#bib.bib79)]. Instances include determining the embedding dimensions based on corpus statistics like count of pairwise equidistant words [[80](https://arxiv.org/html/2406.16893v1#bib.bib80)], reducing the dimensionality of embedding vector applying Principal Component Analysis (PCA) [[81](https://arxiv.org/html/2406.16893v1#bib.bib81)] and compressed image representations equivalent to a given text [[82](https://arxiv.org/html/2406.16893v1#bib.bib82)].

####  5.2.3 Model Design

To achieve desirable performance on the text representations obtained, the model design should be such that it delivers optimal results with minimum complexity. The contributions towards model design can be grouped under the following categories:

  * •

Chunking: It deals with chunking into several blocks, processing each block individually, and connecting the representations of these blocks through recurrence or by some other mechanism. ABSA BERT [[56](https://arxiv.org/html/2406.16893v1#bib.bib56)] breaks down each sequence based on significant phrases contained in it while filtering out irrelevant chunks of tokens before being fed into the BERT model. An extension to the chunking approach has been proposed in the case of Transformer-XL [[46](https://arxiv.org/html/2406.16893v1#bib.bib46)] wherein multiple blocks are connected through a recurrence mechanism. This helps to efficiently compute attention for long sequences by breaking them down into multiple blocks.

![Refer to caption](extracted/5597951/GloabalAttention.jpg) (a) Global

![Refer to caption](extracted/5597951/BandAttention.jpg) (b) Band

![Refer to caption](extracted/5597951/DilatedAttention.jpg) (c) Dilated

![Refer to caption](extracted/5597951/RandomAttention.jpg) (d) Random

![Refer to caption](extracted/5597951/BlockAttention.jpg) (e) Block

Figure 7: Common types of sparse attention patterns
  * •

Sparse Attention: A few contributions attempt sparsification of the attention matrix to reduce the complexity of computing attention in transformer-based models. This implies limiting the count of keys to be attended by queries based either on certain pre-defined patterns or input-conditioned connections. Some common patterns might be global attention, band attention, dilated attention, random attention, and block attention as illustrated in Figure [7](https://arxiv.org/html/2406.16893v1#S5.F7 "Figure 7 ‣ 1st item ‣ 5.2.3 Model Design ‣ 5.2 Software Designing ‣ 5 Efficient Modeling Considerations ‣ A Survey on Transformers in NLP with Focus on Efficiency"). This technique exploits the inherent sparsity in the attention matrix in real-life applications even after computing attention on all possible query-key pairs. Sparse Transformer [[87](https://arxiv.org/html/2406.16893v1#bib.bib87)] factorizes the attention matrix to attain sparse patterns where connectivity is established between a pre-defined set of tokens. This reduces the complexity of attention to O⁢(n⁢n)𝑂𝑛𝑛O(n\sqrt{n})italic_O ( italic_n square-root start_ARG italic_n end_ARG ). Longformer [[84](https://arxiv.org/html/2406.16893v1#bib.bib84)] employs attention at fixed intervals in a strided fashion. It adopts a blend of band attention, dilated attention, and global attention to achieve a near linear scaling factor with respect to the sequence length. Extended Transformer Construction (ETC) [[156](https://arxiv.org/html/2406.16893v1#bib.bib156)] follows a similar approach agglomerating global attention and local band attention with relative positional encoding. Additionally, it employs masking through Contrastive Predictive Coding as a pre-training objective. BigBird [[157](https://arxiv.org/html/2406.16893v1#bib.bib157)] builds upon the ETC model by applying random patterns of sparse-attention. It can handle sequences 8 times the length and achieve linear complexity compared to the conventional attention mechanism. Selective Learn Forget Network (SLFN) [[86](https://arxiv.org/html/2406.16893v1#bib.bib86)] adopts a gated mechanism upon multi-head attention in a single-block transformer architecture for selective retention of attention weights. This aided in filtering out insignificant information while retaining long range dependencies. Memory Compressed Transformer [[85](https://arxiv.org/html/2406.16893v1#bib.bib85)] reduces the number of query-key pairs applying strided convolution. BlockBERT [[83](https://arxiv.org/html/2406.16893v1#bib.bib83)] proposes an efficient version of BERT by incorporating block-wise patterns in the attention matrix for sparsity.

  * •

Mixture-of-Experts (MOE): The concept of sparsification for efficient computation has been taken forward with the notion of Mixture-of-Experts (MOE). In this, the input is routed through multiple sub-networks replacing the single feed-forward layer. Models such as GLaM [[88](https://arxiv.org/html/2406.16893v1#bib.bib88)] demonstrate that it helps to attain high accuracy along with efficient use of resources. FasterMoE [[102](https://arxiv.org/html/2406.16893v1#bib.bib102)] further tackled the load-imbalance in MOE models through fine-grained concurrent scheduling for distributed computing.

  * •

Low-Rank Approximation: To reduce the computational complexity of the attention mechanism, low-rank approximation aims to approximate the attention matrix with a lower-rank matrix. Recently, techniques like Linformer [[49](https://arxiv.org/html/2406.16893v1#bib.bib49)] have been devised to perform low-rank approximations of the self-attention matrix to enhance efficiency. Similarly, the application of kernels for approximation of the computation of self-attention has gained popularity as it reduces the effort required to compute self-attention for the entire sequence matrix. A prominent example of this is the Performers [[89](https://arxiv.org/html/2406.16893v1#bib.bib89)].

  * •

Clustering: It refers to grouping related elements, features in a sequence, or even attention heads to achieve efficient computation of attention. Some other works learn patterns in the data by capturing relevant tokens and clustering them together into buckets. Based on the similarity metric applied for clustering, various models have been devised. For instance, the Reformer [[48](https://arxiv.org/html/2406.16893v1#bib.bib48)] utilizes a hashing-based similarity measure while the Routing Transformer [[50](https://arxiv.org/html/2406.16893v1#bib.bib50)] deploys a K-means clustering algorithm.

  * •

Parameter Sharing: The complexity of a model is proportional to the number of parameters present. Hence reducing the number of parameters can be beneficial towards model efficiency. This can be achieved through the sharing of parameters across the layers in the transformer network. Perceiver [[51](https://arxiv.org/html/2406.16893v1#bib.bib51)] is one such model which performs downsampling apart from sharing weights among layers for efficient computation. ALBERT [[47](https://arxiv.org/html/2406.16893v1#bib.bib47)] on the other hand applies matrix decomposition upon the embedding layer along with cross-layer parameter sharing.




####  5.2.4 Model Compression

Once the model designing process is over, the model can be further compressed to reduce the computational requirements for attaining efficiency. A variety of techniques have been discussed herein for model compression. The first technique is Pruning [[90](https://arxiv.org/html/2406.16893v1#bib.bib90)] which deals with removing the undesired weight parameters from the transformer-based neural network. This brings down the computational complexity, and memory footprint as-well-as the bandwidth leading to a rise in model efficiency. It also serves as a regularizer and prevents overfitting. Pruning can be applied during pre-training [[91](https://arxiv.org/html/2406.16893v1#bib.bib91)], fine-tuning [[92](https://arxiv.org/html/2406.16893v1#bib.bib92)], or at the time of inference [[93](https://arxiv.org/html/2406.16893v1#bib.bib93)]. Pruning is usually structured or unstructured. While the former prunes a section of the neural network such as entire hidden layers or attention heads [[54](https://arxiv.org/html/2406.16893v1#bib.bib54)], the latter prunes individual weights [[91](https://arxiv.org/html/2406.16893v1#bib.bib91)]. Although structured pruning significantly contributes towards enhancing efficiency, unstructured pruning retains the model performance in a better way. Pruning although beneficial can even have adverse effects on model performance if done excessively.

Knowledge Distillation is another concept based on condensing the knowledge from a highly complex model termed as "teacher model" to a less complex model termed as "student model" based on a custom loss function [[41](https://arxiv.org/html/2406.16893v1#bib.bib41)]. The predictions from the teacher model are converted into a probabilistic distribution having soft labels and fed into the student model. The soft labels accelerate the learning process of the student model due to more variance imbibed in the soft labels. A temperature T is applied to obtain the soft labels from the logits of the teacher model. The higher the value of T, the softer the probability distribution. Whereas, the loss between the soft labels and the hard targets is computed and minimized. Although knowledge distillation enhances efficiency, it hinders the overall model performance and generalizability and hyperparameter tuning of the student model increases computational cost [[94](https://arxiv.org/html/2406.16893v1#bib.bib94)].

Quantization is the process of representing model parameters in memory with reduced precision to bring down the memory requirements of the model and subsequently enable efficient computation. Instances include converting values in 32-bit floating-point notation ’float-32’ to 8-bit integer notation ’int-8’ [[82](https://arxiv.org/html/2406.16893v1#bib.bib82), [52](https://arxiv.org/html/2406.16893v1#bib.bib52)]. Despite enhancing the model’s efficiency, it might lead to a loss of information degrading the model’s performance. It has been observed that while quantization performs well in the case of linear functions, for non-linear functions reducing the precision provides unsatisfactory results [[52](https://arxiv.org/html/2406.16893v1#bib.bib52)]. Hence, a mixed quantization approach specific to the model components can prove effective.

###  5.3 Hardware Designing

Apart from software design, hardware considerations for efficiency in deploying LLMs are a vital yet comparatively less explored domain. Figure [8](https://arxiv.org/html/2406.16893v1#S5.F8 "Figure 8 ‣ 5.3 Hardware Designing ‣ 5 Efficient Modeling Considerations ‣ A Survey on Transformers in NLP with Focus on Efficiency") summarizes the developments in efficient hardware design whereas the remainder of this section explains them in detail.

![Refer to caption](extracted/5597951/HardwareDesign.jpg) Figure 8: Efficiency considerations through hardware designing

####  5.3.1 Customized Hardware

Hardware accelerators with parallel processing can speed up the processing of LLMs. Instances include Graphical Processing Units (GPU) having several Arithmetic Logic Units (ALU) and high-bandwidth memory. Besides, Tensor Processing Units (TPU) with matrix processors supporting mixed-precision computing for neural network computations contribute to fast and efficient processing with reduced bottlenecks. Specialized HPCs utilizing Field Programmable Gate Arrays (FPGA) tailor-made as per the application enhances the performance along with efficiency. Xilinx AI [[96](https://arxiv.org/html/2406.16893v1#bib.bib96)] is one such FPGA hardware with separate a Artificial Intelligence (AI) compute unit. The Intel Stratix [[95](https://arxiv.org/html/2406.16893v1#bib.bib95)] has tensor cores for AI-related computing integrated inside the FPGA to accelerate the throughput. Lu et al. [[99](https://arxiv.org/html/2406.16893v1#bib.bib99)] put forth a hardware accelerator for a transformer with MHA. Liu et al. [[100](https://arxiv.org/html/2406.16893v1#bib.bib100)] further implemented a hardware accelerator of the quantized version of BERT on FPGA. However, the majority of these developments are limited to certain applications only is that they are not suited to handle sparse data, high-precision arithmetic operations, or certain linear algebra problems. Moreover, re-configurable hardware like FPGA is noted to have higher FlOps compared to fixed hardware. Nevertheless, this can be seen as a viable option considering the production costs of fixed hardware for short-term applications [[101](https://arxiv.org/html/2406.16893v1#bib.bib101)].

####  5.3.2 Software-Hardware Co-Design

Co-design is a school of thought realizing the contribution of hardware for the efficacy of a given software and deals with joint optimization of hardware with the software to reap improved efficiency along with performance [[101](https://arxiv.org/html/2406.16893v1#bib.bib101)]. Ham et al. [[98](https://arxiv.org/html/2406.16893v1#bib.bib98)] presented a lightweight co-design mechanism for approximation of attention computation by filtering out insignificant relations. Wang et al. [[97](https://arxiv.org/html/2406.16893v1#bib.bib97)] presented a co-design architecture with token pruning as well and head pruning to remove the unnecessary tokens as well as attention heads along with a quantization approach to extract the most significant bits (MSB) based on confidence scores. Qu et al. [[104](https://arxiv.org/html/2406.16893v1#bib.bib104)] devised a detector module for identifying and eliminating weak attention connections to reduce the computational overhead and jointly optimized it with the main transformer module. The detector module applied low-rank transformations with low-precision computation to make the overall architecture lightweight. Rajbhandari et al. [[103](https://arxiv.org/html/2406.16893v1#bib.bib103)] attempted model compression along with an optimized inference strategy to reduce latency and the model size through an MoE-based co-design approach.

Apart from developing High-Performance Computing (HPC) architectures, distributed frameworks like DeepSpeed888https://www.microsoft.com/en-us/research/project/deepspeed/ can accommodate training of LLMs in an efficient manner. The distributed computing versions of popular libraries like PyTorch Distributed999https://pytorch.org/tutorials/beginner/dist_overview.html and TensorFlow101010https://www.tensorflow.org/guide/distributed_training have also been developed to facilitate training of LLMs. Besides, parallelism in data can also lead to higher throughput by splitting the data into smaller chunks and distributing it among several HPCs to train the model using tools like Horovod111111https://horovod.ai/.

##  6 Results and Discussion

###  6.1 Statistical Insights

![Refer to caption](extracted/5597951/YearDistribution.jpg) Figure 9: Year-wise distribution of papers

Figure [9](https://arxiv.org/html/2406.16893v1#S6.F9 "Figure 9 ‣ 6.1 Statistical Insights ‣ 6 Results and Discussion ‣ A Survey on Transformers in NLP with Focus on Efficiency") presents the year-wise distribution of surveyed papers. As we have focused more on recently published works, it can be perceived that the major share of papers is from the last five years. The maximum number of papers is between 2019 and 2021 following a rising trend. However, for the years 2022 and 2023 a declining trend is observed. This can be attributed to the fact that to assess the quality of papers the number of citations is considered to be one of the important indicators. But, it is tough for a paper to get cited by a significant number of papers in such a short span of time.

![Refer to caption](extracted/5597951/ArticleShareP.jpg) Figure 10: Percentage share of article-types

Apart from observing the year-wise distribution of papers, the share of various article types, i.e. journals, conferences, books and pre-print papers. Moreover, the journals have been segregated into regular papers and review papers. From the pie-chart shown in Figure [10](https://arxiv.org/html/2406.16893v1#S6.F10 "Figure 10 ‣ 6.1 Statistical Insights ‣ 6 Results and Discussion ‣ A Survey on Transformers in NLP with Focus on Efficiency"), it can be observed that conference papers account for 56% of the total share of articles. While journals have a 31% share further subdivided into 23% being regular papers and 8% being review papers. The reason behind this is the presence of a variety of prestigious conferences on NLP which are considered to be more reputed than several journals. Thus, such conferences are preferred over journals by prominent researchers in NLP. Interestingly, 11% share of articles are from pre-print platforms like arXiv121212https://arxiv.org/. Further review of the high-quality pre-print articles shows that a major share of such articles are milestone achievements authored by eminent researchers and scientists belonging to reputed institutions. As pre-prints offer recognition for contributions in a couple of days, it has become the apt avenue to claim authorship for a novel contribution. Lastly, 2% of articles reviewed are books. This is because NLP is a rapidly evolving field of research while books are typically considered permanent sources of knowledge presenting persistent concepts that remain relevant for many years. Hence, a book on NLP might lose its significance in just a few years due to rapid technological developments.

![Refer to caption](extracted/5597951/CategoryDistribution.jpg) Figure 11: Category-wise distribution of articles

For a more detailed analysis, the distribution of articles comprising of significant terms related to NLP has been illustrated in Figure [11](https://arxiv.org/html/2406.16893v1#S6.F11 "Figure 11 ‣ 6.1 Statistical Insights ‣ 6 Results and Discussion ‣ A Survey on Transformers in NLP with Focus on Efficiency"). The top-noted terms belong to the following categories131313The related terms have been grouped and categorized into topics as shown in Figure [11](https://arxiv.org/html/2406.16893v1#S6.F11 "Figure 11 ‣ 6.1 Statistical Insights ‣ 6 Results and Discussion ‣ A Survey on Transformers in NLP with Focus on Efficiency"). in descending order "Transformers NLP", "Efficiency Considerations", "Pre-Trained Models", "Deep Learning", "Hardware Design" and "Machine Learning". This ascertains that the topics discussed in the reviewed papers align with the objective of this survey. It is to be noted that, although transformers are a subset of deep learning techniques in NLP, it has been treated separately for more transparency given the magnanimous volume of articles based on transformers. From the low share of articles on NLP focusing on the efficient use of hardware, it can be inferred that the current research trend majorly emphasizes on software to formulate pareto-optimal solutions with almost no consideration for hardware. Whereas, the state of developments necessitates the inclusion of hardware design considerations while formulating new models to achieve optimal efficacy accompanied by efficiency.

![Refer to caption](extracted/5597951/TrendNLP.jpg) Figure 12: Comparison of the trend of NLP vs Transformers. Source: Google Trends

To further validate our research, we compare the trend of transformer with NLP based on the number of web searches by people throughout the world over the last five years utilizing Google Trends141414https://trends.google.com/trends/. Figure [12](https://arxiv.org/html/2406.16893v1#S6.F12 "Figure 12 ‣ 6.1 Statistical Insights ‣ 6 Results and Discussion ‣ A Survey on Transformers in NLP with Focus on Efficiency"), portrays an overall rising trend for both the terms, i.e. "Transformers" and "NLP". The popularity of transformers started rising after 2019 and since then there has been a steady growth. Furthermore, the growth-rate for both "Transformers" and "NLP" are almost similar with "Transformers" having a slightly steeper growth rate in recent years. This shows the significance of transformer-based models in the evolution of NLP. Moreover, the trend confirms the statistical analysis of the surveyed papers mentioned above.

Table 3: Evaluation of High Performance Models Model |  Year |  Pre-training Dataset |  #Parameters |  GLUE |  LAMBADA |  PTWL  
---|---|---|---|---|---|---  
BERT_large [[8](https://arxiv.org/html/2406.16893v1#bib.bib8)] |  2018 |  WikiEn+Book Corpus |  340M |  81.9 |  |  31.3  
GPT [[11](https://arxiv.org/html/2406.16893v1#bib.bib11)] |  2018 |  BookCorpus |  117M |  72.8 |  - |  -  
ROBERTA [[121](https://arxiv.org/html/2406.16893v1#bib.bib121)] |  2019 |  BookCorpus + CC-News + OpenWebText + STORIES |  340M |  88.5 |  - |  -  
XLNET [[9](https://arxiv.org/html/2406.16893v1#bib.bib9)] |  2019 |  WikiEn + BookCorpus + Giga5 + ClueWeb + Common Crawl |  340M |  90.5 |  - |  -  
GPT-2 [[12](https://arxiv.org/html/2406.16893v1#bib.bib12)] |  2019 |  Web Crawl Text |  1.5B |  - |  - |  35.76  
BART [[10](https://arxiv.org/html/2406.16893v1#bib.bib10)] |  2019 |  BookCorpus + CC-News + OpenWebText + STORIES |  370M |  88.4 |  - |  -  
Transformer-XL [[46](https://arxiv.org/html/2406.16893v1#bib.bib46)] |  2019 |  Wikipedia |  24M |  - |  - |  54.55  
GPT-3 [[13](https://arxiv.org/html/2406.16893v1#bib.bib13)] |  2020 |  Web Crawl Text + Book Corpus |  175B |  |  86.4 |  20.5  
T5 [[120](https://arxiv.org/html/2406.16893v1#bib.bib120)] |  2020 |  Colossal Clean Crawled Corpus (C4) |  11B |  89.7 |  - |  -  
XLM-R [[145](https://arxiv.org/html/2406.16893v1#bib.bib145)] |  2020 |  CommonCrawl |  10.7B |  91.8 |  - |  -  
XLM-R [[145](https://arxiv.org/html/2406.16893v1#bib.bib145)] |  2020 |  CommonCrawl |  10.7B |  91.8 |  - |  -  
Megatron Turing NLG [[142](https://arxiv.org/html/2406.16893v1#bib.bib142)] |  2022 |  CommonCrawl + Realnews + Github + Wikipedia+ Gutenberg + Books3 + ArXiv + PubMed Abstracts + Stack Exchange + Pile-CC + OpenWebText2 |  530B |  - |  87.2 |  -  
PaLM [[143](https://arxiv.org/html/2406.16893v1#bib.bib143)] |  2022 |  Public Forums + Source Codes + WikiEn + Web Documents + News + Books |  540B |  - |  89.7 |  -  
Turing ULRv6 [[144](https://arxiv.org/html/2406.16893v1#bib.bib144)] |  2022 |  CommonCrawl |  4.6B |  91.3 |  - |  -  
Chinchilla [[116](https://arxiv.org/html/2406.16893v1#bib.bib116)] |  2022 |  MassiveText |  70B |  - |  77.7 |  -  
LLaMA [[137](https://arxiv.org/html/2406.16893v1#bib.bib137)] |  2023 |  CommonCrawl + C4 + Github + Wikipedia + Gutenberg + Books3 + ArXiv + Stack Exchange |  65B |  - |  84 |  -  
Note: #Parameters- No. of model parameters, LAMBADA- LAMBADA (Accuracy), PTWL- Penn Treebank (Word Level Perplexity), ’-’ indicates non-avalability of data  
Table 4: Cost Behind Training Models Model | GPU/ TPU | GPU/ TPU Hours | Energy | Emissions | Source  
---|---|---|---|---|---  
Transformerbase | GPU-P100 x 8 | 96 | 1.416 | 0.0117 | Strubell et al. [[16](https://arxiv.org/html/2406.16893v1#bib.bib16)]  
Transformerbig | GPU-P100 x 8 | 672 | 1.515 | 0.0864 | Strubell et al. [[16](https://arxiv.org/html/2406.16893v1#bib.bib16)]  
ELMo | GPU-P100 x 3 | 1008 | 0.51766 | 0.118 | Strubell et al. [[16](https://arxiv.org/html/2406.16893v1#bib.bib16)]  
BERTbase | GPU-V100 x 64 | 5056 | 12.04151 | 0.647 | Strubell et al. [[16](https://arxiv.org/html/2406.16893v1#bib.bib16)]  
GPT-2 | TPU-v3 x 32 | 5376 | - | - | Strubell et al. [[16](https://arxiv.org/html/2406.16893v1#bib.bib16)]  
Gopher | GPU-A100 x 16 | 5725 | 1,066 | 352 | Luccioni et al. [[139](https://arxiv.org/html/2406.16893v1#bib.bib139)]  
GPT-3 | GPU-A100 x 16 | 6912 | 1,287 | 502 | Luccioni et al. [[139](https://arxiv.org/html/2406.16893v1#bib.bib139)]  
NAS | TPU-v2 x 1 | 32623 | - | - | Strubell et al. [[16](https://arxiv.org/html/2406.16893v1#bib.bib16)]  
GShard | TPU-v3 x 1024 | 76,185.600 | 24.100 | 4.8 | Patterson et al. [[17](https://arxiv.org/html/2406.16893v1#bib.bib17)]  
LLaMA-7B | GPU-A100 | 82,432 | 36 | 14 | Touvron et al. [[137](https://arxiv.org/html/2406.16893v1#bib.bib137)]  
LLaMA-13B | GPU-A100 | 135,168 | 59 | 23 | Touvron et al. [[137](https://arxiv.org/html/2406.16893v1#bib.bib137)]  
T5 | TPU-v3 x 512 | 245,760 | 85.7 | 46.7 | Patterson et al. [[17](https://arxiv.org/html/2406.16893v1#bib.bib17)]  
XLM | GPU-V100 x 512 | 250,675.2 | 167.443 | 39 | Faiz et al. [[140](https://arxiv.org/html/2406.16893v1#bib.bib140)]  
LLaMA-33B | GPU-A100 | 530,432 | 233 | 90 | Touvron et al. [[137](https://arxiv.org/html/2406.16893v1#bib.bib137)]  
Switch | TPU-v3 x 1024 | 663,552 | 179 | 72.200 | Patterson et al. [[17](https://arxiv.org/html/2406.16893v1#bib.bib17)]  
OPT-175B | GPU-A100 | 809,472 | 356 | 137 | Touvron et al. [[137](https://arxiv.org/html/2406.16893v1#bib.bib137)]  
LLaMA-65B | GPU-A100 | 1,022,362 | 449 | 173 | Touvron et al. [[137](https://arxiv.org/html/2406.16893v1#bib.bib137)]  
BLOOM-175B | GPU-A100 | 1,082,880 | 475 | 183 | Touvron et al. [[137](https://arxiv.org/html/2406.16893v1#bib.bib137)]  
LaMDA | TPU-V3 | 1418035 | 451 | 25.200 | Thoppilan et al. [[138](https://arxiv.org/html/2406.16893v1#bib.bib138)]  
GPT3 | GPU-V100 x 10000 | 3,552,000 | 1,287 | 552.100 | Patterson et al. [[17](https://arxiv.org/html/2406.16893v1#bib.bib17)]  
Note: Emissions- Co2 emitted (metric tons), Energy- Power Consumption (MWh)  
![Refer to caption](extracted/5597951/DataVsParameters.jpg) Figure 13: Relationship between the size of pre-training data and number of model parameters in LLMs

###  6.2 Trend of Research

From the recent developments, it is evident that transformer-based pre-trained models have excelled in terms of accuracy compared to other conventional machine learning and deep learning algorithms. Table [3](https://arxiv.org/html/2406.16893v1#S6.T3 "Table 3 ‣ 6.1 Statistical Insights ‣ 6 Results and Discussion ‣ A Survey on Transformers in NLP with Focus on Efficiency") presents a comparison of various renowned LLMs based on their year of release, number of parameters, accuracy and pre-training data. This shows that the current trend is towards designing powerful pre-trained models that only need to be fine-tuned as per the requirements of a particular task. However, these models have tremendous computational complexity and for each new task, they need to be fine-tuned on a sufficiently large data-set. Recently, some efforts have been directed towards "task agnostic models" as in subsequent versions of GPT promoting few-shot or even zero-shot learning through "prompting". However, such models could be termed as "multi-task learners" rather than task agnostic models as their generalizability is significantly inferior to human cognition. Moreover, to achieve multi-task generalizability, fine-tuning upon several tasks (i.e. various large-scale data-sets) is required. Overall, this limits research on such models in resource-constrained environments and also aggravates carbon emissions. This can be visualized from Table [4](https://arxiv.org/html/2406.16893v1#S6.T4 "Table 4 ‣ 6.1 Statistical Insights ‣ 6 Results and Discussion ‣ A Survey on Transformers in NLP with Focus on Efficiency") which shows that training SOTA NLP models can lead to several tonnes of carbon emissions. Figure [13](https://arxiv.org/html/2406.16893v1#S6.F13 "Figure 13 ‣ 6.1 Statistical Insights ‣ 6 Results and Discussion ‣ A Survey on Transformers in NLP with Focus on Efficiency"), captures this relationship among the model parameters and the size of the data used for pre-training the model. It can be observed that there has been an overall rising trend in the size of the pre-training data as-well-as the model parameter count. The earlier LLMs incrementally raised the model parameters and pre-training corpus obeying a linear relationship among both. Subsequent LLMs focused majorly on increasing the model parameters without much rise in the size of the pre-training data. Besides, the recent LLMs are focusing on striking a balance between the size of data and the model size if not reducing the model size compared to the volume of pre-training data. This signals a ray of hope that the awareness of efficient LLMs in the NLP research community is proliferating.

###  6.3 Future Avenues

To make transformer-based pre-trained models efficient, a significant effort has been directed toward reducing the complexity of the attention mechanism as-well-as decreasing the number of model parameters. In this paper, several such techniques as-well-as associated models have been discussed. From these studies, it can be inferred that efficiency can be achieved at multiple stages of model development. However, the goal of formulating an efficient pre-trained model is still far from being achieved. For this, efforts towards devising efficient model design need to be consolidated with efficient pre-training and fine-tuning strategies along with effective prompt-based learning approaches (if applicable). Also, the data-sets play a major role in the trade-off between performance and efficiency. Determining the optimal size of the data-sets, distribution of sequence length distribution as-well-as the quality of training samples is of utmost importance to restrict training costs and prevent over-training.

##  7 Conclusion

NLP empowers computing devices to decipher and process natural language text. Various applications in NLP include sentiment analysis, misinformation detection, machine translation, and text summarization. It can be observed that NLP has evolved considerably from rule-based approaches, followed by machine learning and deep learning models to the advent of transformer-based pre-trained models. Although the efficacy of NLP models at tasks has increased manifold over time, the sustainability of the models based on factors like efficiency, task-agnosticism, and domain-independence is a matter of concern. It motivates directing research towards formulating sustainable NLP models. In this paper, a survey of research works aimed at enhancing the efficiency of NLP models has been conducted. These works have been systematically presented targeting the various stages of model development. It highlights the efforts towards devising practical models with appreciable performance for implementation in resource-constrained environments with a significantly low carbon footprint. It ushers a paradigm shift in devising NLP models keeping sustainability in mind.

## References

  * [1] Liu, Bing. Sentiment analysis: Mining opinions, sentiments, and emotions. Cambridge University Press, 2015. 
  * [2] Jindal, Nitin, and Bing Liu. “Opinion spam and analysis." In Proceedings of the 2008 international conference on web search and data mining, pp. 219-230. 2008. 
  * [3] Cho, Kyunghyun, Bart van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. "On the Properties of Neural Machine Translation: Encoder–Decoder Approaches." In Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation, pp. 103-111. 2014. 
  * [4] Bahdanau, Dzmitry, Kyung Hyun Cho, and Yoshua Bengio. "Neural machine translation by jointly learning to align and translate." In 3rd International Conference on Learning Representations, ICLR 2015. 2015. 
  * [5] El-Kassas, Wafaa S., Cherif R. Salama, Ahmed A. Rafea, and Hoda K. Mohamed. "Automatic text summarization: A comprehensive survey." Expert systems with applications 165 (2021): 113679. 
  * [6] Soares, Marco Antonio Calijorne, and Fernando Silva Parreiras. "A literature review on question answering techniques, paradigms and systems." Journal of King Saud University-Computer and Information Sciences 32, no. 6 (2020): 635-646. 
  * [7] Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. “Attention is all you need." In Advances in neural information processing systems, pp. 5998-6008. 2017. 
  * [8] Kenton, Jacob Devlin Ming-Wei Chang, and Lee Kristina Toutanova. "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding." In Proceedings of NAACL-HLT, pp. 4171-4186. 2019. 
  * [9] Yang, Zhilin, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R. Salakhutdinov, and Quoc V. Le. “Xlnet: Generalized autoregressive pretraining for language understanding." In Advances in neural information processing systems, pp. 5753-5763. 2019. 
  * [10] Lewis, Mike, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. "BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension." In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7871-7880. 2020. 
  * [11] Radford, Alec, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. “Improving language understanding by generative pre-training." URL https://s3-us-west-2.amazonaws.com/openai-assets/researchcovers/languageunsupervised/language understanding paper. pdf (2018). 
  * [12] Radford, Alec, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. “Language models are unsupervised multitask learners." OpenAI Blog 1, no. 8 (2019): 9. 
  * [13] Brown, Tom B., Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan et al. "Language models are few-shot learners." In Proceedings of the 34th International Conference on Neural Information Processing Systems, pp. 1877-1901. 2020. 
  * [14] Chowdhary, K. R. "Natural language processing." In Fundamentals of Artificial Intelligence, pp. 603-649. Springer, New Delhi, 2020. 
  * [15] Gers, Felix A., and Jürgen Schmidhuber. "Recurrent nets that time and count." In Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks. IJCNN 2000. Neural Computing: New Challenges and Perspectives for the New Millennium, vol. 3, pp. 189-194. IEEE, 2000. 
  * [16] Strubell, Emma, Ananya Ganesh, and Andrew McCallum. "Energy and Policy Considerations for Deep Learning in NLP." In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 3645-3650. 2019. 
  * [17] Patterson, David, Joseph Gonzalez, Quoc Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. "Carbon emissions and large neural network training." arXiv preprint arXiv:2104.10350 (2021). 
  * [18] Schwartz, Roy, Jesse Dodge, Noah A. Smith, and Oren Etzioni. "Green ai." Communications of the ACM 63, no. 12 (2020): 54-63. 
  * [19] Treviso, Marcos, Ji-Ung Lee, Tianchu Ji, Betty van Aken, Qingqing Cao, Manuel R. Ciosici, Michael Hassid et al. "Efficient methods for natural language processing: A survey." Transactions of the Association for Computational Linguistics 11 (2023): 826-860. 
  * [20] Shu, Kai, Amy Sliva, Suhang Wang, Jiliang Tang, and Huan Liu. “Fake news detection on social media: A data mining perspective." ACM SIGKDD Explorations Newsletter 19, no. 1 (2017): 22-36. 
  * [21] Qazvinian, Vahed, Emily Rosengren, Dragomir Radev, and Qiaozhu Mei. "Rumor has it: Identifying misinformation in microblogs." In Proceedings of the 2011 conference on empirical methods in natural language processing, pp. 1589-1599. 2011. 
  * [22] Lewis, Mike, and Angela Fan. "Generative question answering: Learning to answer the whole question." In International Conference on Learning Representations. 2018. 
  * [23] Kupiec, Julian, Jan Pedersen, and Francine Chen. "A trainable document summarizer." In Proceedings of the 18th annual international ACM SIGIR conference on Research and development in information retrieval, pp. 68-73. 1995. 
  * [24] Nallapati, Ramesh, Bing Xiang, and Bowen Zhou. "Sequence-to-sequence rnns for text summarization." (2016). 
  * [25] Pang, Bo, Lillian Lee, and Shivakumar Vaithyanathan. “Thumbs up?: sentiment classification using machine learning techniques." In Proceedings of the ACL-02 conference on Empirical methods in natural language processing-Volume 10, pp. 79-86. Association for Computational Linguistics, 2002. 
  * [26] Wilson, Theresa, Janyce Wiebe, and Paul Hoffmann. “Recognizing contextual polarity in phrase-level sentiment analysis." In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing. 2005. 
  * [27] Ansar, Wazib, and Saptarsi Goswami. "Combating the menace: A survey on characterization and detection of fake news from a data science perspective." International Journal of Information Management Data Insights 1, no. 2 (2021): 100052. 
  * [28] Bird, Steven, Ewan Klein, and Edward Loper. Natural language processing with Python: analyzing text with the natural language toolkit. “ O’Reilly Media, Inc.", 2009. 
  * [29] Xu, Peng, Davis Liang, Zhiheng Huang, and Bing Xiang. "Attention-guided generative models for extractive question answering." arXiv preprint arXiv:2110.06393 (2021). 
  * [30] Nallapati, Ramesh, Feifei Zhai, and Bowen Zhou. "Summarunner: A recurrent neural network based sequence model for extractive summarization of documents." In Thirty-first AAAI conference on artificial intelligence. 2017. 
  * [31] See, Abigail, Peter J. Liu, and Christopher D. Manning. "Get To The Point: Summarization with Pointer-Generator Networks." In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1073-1083. 2017. 
  * [32] Nakagawa, Tetsuji, Kentaro Inui, and Sadao Kurohashi. "Dependency tree-based sentiment classification using CRFs with hidden variables." In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pp. 786-794. Association for Computational Linguistics, 2010. 
  * [33] Harris, Zellig S. "Distributional structure." Word 10, no. 2-3 (1954): 146-162. 
  * [34] Deerwester, Scott, Susan T. Dumais, George W. Furnas, Thomas K. Landauer, and Richard Harshman. “Indexing by latent semantic analysis." Journal of the American society for information science 41, no. 6 (1990): 391-407. 
  * [35] Mikolov, Tomas, Ilya Sutskever, Kai Chen, Greg S. Corrado, and Jeff Dean. "Distributed representations of words and phrases and their compositionality." In Advances in neural information processing systems, pp. 3111-3119. 2013. 
  * [36] Pennington, Jeffrey, Richard Socher, and Christopher Manning. “Glove: Global vectors for word representation." In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532-1543. 2014. 
  * [37] Y. Bengio, R. Ducharme, P. Vincent. A neural probabilistic language model. Journal of Machine Learning Research, 3:1137-1155, 2003. 
  * [38] Le, Quoc, and Tomas Mikolov. "Distributed representations of sentences and documents." In International conference on machine learning, pp. 1188-1196. PMLR, 2014. 
  * [39] Irsoy, Ozan, and Claire Cardie. "Opinion mining with deep recurrent neural networks." In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 720-728. 2014. 
  * [40] Liu, Qian, Zhiqiang Gao, Bing Liu, and Yuanlin Zhang. “Automated rule selection for aspect extraction in opinion mining." In Twenty-Fourth International Joint Conference on Artificial Intelligence. 2015. 
  * [41] Hinton, Geoffrey, Oriol Vinyals, and Jeff Dean. "Distilling the Knowledge in a Neural Network." stat 1050 (2015): 9. 
  * [42] Joulin, Armand, Édouard Grave, Piotr Bojanowski, and Tomáš Mikolov. "Bag of Tricks for Efficient Text Classification." In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pp. 427-431. 2017. 
  * [43] Poria, Soujanya, Erik Cambria, and Alexander Gelbukh. “Aspect extraction for opinion mining with a deep convolutional neural network." Knowledge-Based Systems 108 (2016): 42-49. 
  * [44] Howard, Jeremy, and Sebastian Ruder. "Universal Language Model Fine-tuning for Text Classification." In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 328-339. 2018. 
  * [45] Peters, Matthew E., Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. "Deep contextualized word representations." In Proceedings of NAACL-HLT, pp. 2227-2237. 2018. 
  * [46] Dai, Zihang, Zhilin Yang, Yiming Yang, Jaime G. Carbonell, Quoc Le, and Ruslan Salakhutdinov. "Transformer-XL: Attentive Language Models beyond a Fixed-Length Context." In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 2978-2988. 2019. 
  * [47] Lan, Zhenzhong, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. "ALBERT: A Lite BERT for Self-supervised Learning of Language Representations." In International Conference on Learning Representations. 2019. 
  * [48] Kitaev, Nikita, Lukasz Kaiser, and Anselm Levskaya. "Reformer: The Efficient Transformer." In International Conference on Learning Representations. 2019. 
  * [49] Wang, Sinong, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. "Linformer: Self-attention with linear complexity." arXiv preprint arXiv:2006.04768 (2020). 
  * [50] Roy, Aurko, Mohammad Saffar, Ashish Vaswani, and David Grangier. "Efficient content-based sparse attention with routing transformers." Transactions of the Association for Computational Linguistics 9 (2021): 53-68. 
  * [51] Jaegle, Andrew, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. "Perceiver: General perception with iterative attention." In International conference on machine learning, pp. 4651-4664. PMLR, 2021. 
  * [52] Radford, Alec, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry et al. "Learning transferable visual models from natural language supervision." In International conference on machine learning, pp. 8748-8763. PMLR, 2021. 
  * [53] Zhang, Susan, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan et al. "Opt: Open pre-trained transformer language models." arXiv preprint arXiv:2205.01068 (2022). 
  * [54] Sajjad, Hassan, Fahim Dalvi, Nadir Durrani, and Preslav Nakov. "On the effect of dropping layers of pre-trained transformer models." Computer Speech & Language 77 (2023): 101429. 
  * [55] Ray, Paramita, and Amlan Chakrabarti. "A Mixed approach of Deep Learning method and Rule-Based method to improve Aspect Level Sentiment Analysis." Applied Computing and Informatics (2019). 
  * [56] Ansar, Wazib, Saptarsi Goswami, Amlan Chakrabarti, and Basabi Chakraborty. "An efficient methodology for aspect-based sentiment analysis using BERT through refined aspect extraction." Journal of Intelligent & Fuzzy Systems 40, no. 5 (2021): 9627-9644. 
  * [57] Malik, Vikas, and Amit Kumar. "Sentiment Analysis of Twitter Data Using Naive Bayes Algorithm." International Journal on Recent and Innovation Trends in Computing and Communication 6, no. 4 (2018): 120-125. 
  * [58] Huq, Mohammad Rezwanul, Ahmad Ali, and Anika Rahman. “Sentiment analysis on Twitter data using KNN and SVM." IJACSA) International Journal of Advanced Computer Science and Applications 8, no. 6 (2017): 19-25. 
  * [59] Bengio, Yoshua, Patrice Simard, and Paolo Frasconi. "Learning long-term dependencies with gradient descent is difficult." IEEE transactions on neural networks 5, no. 2 (1994): 157-166. 
  * [60] Hochreiter, Sepp, and Jürgen Schmidhuber. "Long short-term memory." Neural computation 9, no. 8 (1997): 1735-1780. 
  * [61] Hoffmann, Jordan, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas et al. "An empirical analysis of compute-optimal large language model training." Advances in Neural Information Processing Systems 35 (2022): 30016-30030. 
  * [62] Lee, Katherine, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. "Deduplicating Training Data Makes Language Models Better." In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8424-8445. 2022. 
  * [63] Mishra, Swaroop, and Bhavdeep Singh Sachdeva. "Do we need to create big datasets to learn a task?." In Proceedings of SustaiNLP: Workshop on Simple and Efficient Natural Language Processing, pp. 169-173. 2020. 
  * [64] Le Bras, Ronan, Swabha Swayamdipta, Chandra Bhagavatula, Rowan Zellers, Matthew Peters, Ashish Sabharwal, and Yejin Choi. "Adversarial filters of dataset biases." In International conference on machine learning, pp. 1078-1088. PMLR, 2020. 
  * [65] Ren, Pengzhen, Yun Xiao, Xiaojun Chang, Po-Yao Huang, Zhihui Li, Brij B. Gupta, Xiaojiang Chen, and Xin Wang. "A survey of deep active learning." ACM computing surveys (CSUR) 54, no. 9 (2021): 1-40. 
  * [66] Yuan, Michelle, Hsuan-Tien Lin, and Jordan Boyd-Graber. "Cold-start Active Learning through Self-supervised Language Modeling." In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 7935-7948. 2020. 
  * [67] Sener, Ozan, and Silvio Savarese. "Active Learning for Convolutional Neural Networks: A Core-Set Approach." In International Conference on Learning Representations. 2018. 
  * [68] Margatina, Katerina, Giorgos Vernikos, Loïc Barrault, and Nikolaos Aletras. "Active Learning by Acquiring Contrastive Examples." In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 650-663. 2021. 
  * [69] Settles, Burr, Mark Craven, and Lewis Friedland. "Active learning with real annotation costs." In Proceedings of the NIPS workshop on cost-sensitive learning, vol. 1. 2008. 
  * [70] Lowell, David, Zachary C. Lipton, and Byron C. Wallace. "Practical Obstacles to Deploying Active Learning." In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 21-30. 2019. 
  * [71] Karamcheti, Siddharth, Ranjay Krishna, Li Fei-Fei, and Christopher D. Manning. "Mind Your Outliers! Investigating the Negative Impact of Outliers on Active Learning for Visual Question Answering." In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 7265-7281. 2021. 
  * [72] Press, Ofir, Noah A. Smith, and Mike Lewis. "Shortformer: Better Language Modeling using Shorter Inputs." In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 5493-5505. 2021. 
  * [73] Kumar, M., Benjamin Packer, and Daphne Koller. "Self-paced learning for latent variable models." Advances in neural information processing systems 23 (2010). 
  * [74] Melamud, Oren, Jacob Goldberger, and Ido Dagan. "context2vec: Learning generic context embedding with bidirectional lstm." In Proceedings of the 20th SIGNLL conference on computational natural language learning, pp. 51-61. 2016. 
  * [75] Conneau, A., D. Kiela, H. Schwenk, L. Barrault, and A. Bordes. "Supervised learning of universal sentence representations from natural language inference data." In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 670-680. Association for Computational Linguistics, 2017. 
  * [76] Cer, Daniel, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant et al. "Universal sentence encoder." arXiv preprint arXiv:1803.11175 (2018). 
  * [77] Reimers, Nils, and Iryna Gurevych. "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks." In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). Association for Computational Linguistics, 2019. 
  * [78] Feng, Fangxiaoyu, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. "Language-agnostic BERT Sentence Embedding." In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 878-891. 2022. 
  * [79] Del Giudice, Marco. "Effective dimensionality: A tutorial." Multivariate behavioral research 56, no. 3 (2021): 527-542. 
  * [80] Patel, Kevin, and Pushpak Bhattacharyya. "Towards lower bounds on number of dimensions for word embeddings." In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pp. 31-36. 2017. 
  * [81] Raunak, Vikas, Vivek Gupta, and Florian Metze. "Effective dimensionality reduction for word embeddings." In Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019), pp. 235-243. 2019. 
  * [82] Ansar, Wazib, Saptarsi Goswami, Amlan Chakrabarti, and Basabi Chakraborty. "TexIm: A Novel Text-to-Image Encoding Technique Using BERT." In Computer Vision and Machine Intelligence: Proceedings of CVMI 2022, pp. 123-139. Singapore: Springer Nature Singapore, 2023. 
  * [83] Qiu, Jiezhong, Hao Ma, Omer Levy, Wen-tau Yih, Sinong Wang, and Jie Tang. "Blockwise Self-Attention for Long Document Understanding." In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 2555-2565. 2020. 
  * [84] Beltagy, Iz, Matthew E. Peters, and Arman Cohan. "Longformer: The long-document transformer." arXiv preprint arXiv:2004.05150 (2020). 
  * [85] Liu, Peter J., Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. "Generating Wikipedia by Summarizing Long Sequences." In International Conference on Learning Representations. 2018. 
  * [86] Ansar, Wazib, Saptarsi Goswami, Amlan Chakrabarti, and Basabi Chakraborty. "A novel selective learning based transformer encoder architecture with enhanced word representation." Applied Intelligence 53, no. 8 (2023): 9424-9443. 
  * [87] Child, Rewon, Scott Gray, Alec Radford, and Ilya Sutskever. "Generating long sequences with sparse transformers." arXiv preprint arXiv:1904.10509 (2019). 
  * [88] Du, Nan, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun et al. "Glam: Efficient scaling of language models with mixture-of-experts." In International Conference on Machine Learning, pp. 5547-5569. PMLR, 2022. 
  * [89] Choromanski, Krzysztof, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins et al. "Masked language modeling for proteins via linearly scalable long-context transformers." arXiv preprint arXiv:2006.03555 (2020). 
  * [90] LeCun, Yann, John Denker, and Sara Solla. "Optimal brain damage." Advances in neural information processing systems 2 (1989). 
  * [91] Louizos, Christos, Max Welling, and Diederik P. Kingma. "Learning Sparse Neural Networks through L0 Regularization." In International Conference on Learning Representations. 2018. 
  * [92] Sanh, Victor, Thomas Wolf, and Alexander Rush. "Movement pruning: Adaptive sparsity by fine-tuning." Advances in Neural Information Processing Systems 33 (2020): 20378-20389. 
  * [93] Fan, Angela, Edouard Grave, and Armand Joulin. "Reducing Transformer Depth on Demand with Structured Dropout." In International Conference on Learning Representations. 2019. 
  * [94] Stanton, Samuel, Pavel Izmailov, Polina Kirichenko, Alexander A. Alemi, and Andrew G. Wilson. "Does knowledge distillation really work?." Advances in Neural Information Processing Systems 34 (2021): 6906-6919. 
  * [95] Boutros, Andrew, Eriko Nurvitadhi, Rui Ma, Sergey Gribok, Zhipeng Zhao, James C. Hoe, Vaughn Betz, and Martin Langhammer. "Beyond peak performance: Comparing the real performance of AI-optimized FPGAs and GPUs." In 2020 International Conference on Field-Programmable Technology (ICFPT), pp. 10-19. IEEE, 2020. 
  * [96] Gaide, Brian, Dinesh Gaitonde, Chirag Ravishankar, and Trevor Bauer. "Xilinx adaptive compute acceleration platform: VersalTM architecture." In Proceedings of the 2019 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays, pp. 84-93. 2019. 
  * [97] Wang, Hanrui, Zhekai Zhang, and Song Han. "Spatten: Efficient sparse attention architecture with cascade token and head pruning." In 2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA), pp. 97-110. IEEE, 2021. 
  * [98] Ham, Tae Jun, Yejin Lee, Seong Hoon Seo, Soosung Kim, Hyunji Choi, Sung Jun Jung, and Jae W. Lee. "ELSA: Hardware-software co-design for efficient, lightweight self-attention mechanism in neural networks." In 2021 ACM/IEEE 48th Annual International Symposium on Computer Architecture (ISCA), pp. 692-705. IEEE, 2021. 
  * [99] Lu, Siyuan, Meiqi Wang, Shuang Liang, Jun Lin, and Zhongfeng Wang. "Hardware accelerator for multi-head attention and position-wise feed-forward in the transformer." In 2020 IEEE 33rd International System-on-Chip Conference (SOCC), pp. 84-89. IEEE, 2020. 
  * [100] Liu, Zejian, Gang Li, and Jian Cheng. "Hardware acceleration of fully quantized bert for efficient natural language processing." In 2021 Design, Automation & Test in Europe Conference & Exhibition (DATE), pp. 513-516. IEEE, 2021. 
  * [101] Hooker, Sara. "The hardware lottery." Communications of the ACM 64, no. 12 (2021): 58-65. 
  * [102] He, Jiaao, Jidong Zhai, Tiago Antunes, Haojie Wang, Fuwen Luo, Shangfeng Shi, and Qin Li. "FasterMoE: modeling and optimizing training of large-scale dynamic pre-trained models." In Proceedings of the 27th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, pp. 120-134. 2022. 
  * [103] Rajbhandari, Samyam, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Ammar Ahmad Awan, Jeff Rasley, and Yuxiong He. "Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation ai scale." In International Conference on Machine Learning, pp. 18332-18346. PMLR, 2022. 
  * [104] Qu, Zheng, Liu Liu, Fengbin Tu, Zhaodong Chen, Yufei Ding, and Yuan Xie. "Dota: detect and omit weak attentions for scalable transformer acceleration." In Proceedings of the 27th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, pp. 14-26. 2022. 
  * [105] Lannelongue, Loïc, Jason Grealey, and Michael Inouye. "Green algorithms: quantifying the carbon footprint of computation." Advanced science 8, no. 12 (2021): 2100707. 
  * [106] Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. "Quantifying the carbon emissions of machine learning." In Climate Change workshop, NeurIPS 2019. 2019. 
  * [107] Lasse F. Wolff Anthony, Benjamin Kanding, and Raghavendra Selvan. "Carbontracker: Tracking and predicting the carbon footprint of training deep learning models". In ICML Workshop on "Challenges in Deploying and monitoring Machine Learning Systems". 2020. 
  * [108] Dürlich, Luise, Evangelia Gogoulou, and Joakim Nivre. "On the Concept of Resource-Efficiency in NLP." In Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa), pp. 135-145. 2023. 
  * [109] Gupta, Udit, Young Geun Kim, Sylvia Lee, Jordan Tse, Hsien-Hsin S. Lee, Gu-Yeon Wei, David Brooks, and Carole-Jean Wu. "Chasing carbon: The elusive environmental footprint of computing." In 2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA), pp. 854-867. IEEE, 2021. 
  * [110] Sutskever, Ilya, Oriol Vinyals, and Quoc V. Le. "Sequence to sequence learning with neural networks." Advances in neural information processing systems 27 (2014). 
  * [111] Lin, Zhouhan, Minwei Feng, Cicero dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. "A structured self-attentive sentence embedding." In International Conference on Learning Representations. International Conference on Learning Representations, ICLR, 2017. 
  * [112] Kim, Yoon, Carl Denton, Luong Hoang, and Alexander M. Rush. "Structured Attention Networks." In International Conference on Learning Representations. 2016. 
  * [113] Galassi, Andrea, Marco Lippi, and Paolo Torroni. "Attention in natural language processing." IEEE transactions on neural networks and learning systems 32, no. 10 (2020): 4291-4308. 
  * [114] Graves, Alex, Greg Wayne, and Ivo Danihelka. "Neural turing machines." arXiv preprint arXiv:1410.5401 (2014). 
  * [115] Luong, Minh-Thang, Hieu Pham, and Christopher D. Manning. "Effective Approaches to Attention-based Neural Machine Translation." In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 1412-1421. 2015.(2015). 
  * [116] Hoffmann, Jordan, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas et al. "Training compute-optimal large language models." arXiv preprint arXiv:2203.15556 (2022). 
  * [117] Collobert, Ronan, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. "Natural language processing (almost) from scratch." Journal of machine learning research 12, no. ARTICLE (2011): 2493-2537. 
  * [118] Dai, Andrew M., and Quoc V. Le. "Semi-supervised sequence learning." Advances in neural information processing systems 28 (2015). 
  * [119] Qiu, Xipeng, Tianxiang Sun, Yige Xu, Yunfan Shao, Ning Dai, and Xuanjing Huang. "Pre-trained models for natural language processing: A survey." Science China Technological Sciences 63, no. 10 (2020): 1872-1897. 
  * [120] Raffel, Colin, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. "Exploring the limits of transfer learning with a unified text-to-text transformer." The Journal of Machine Learning Research 21, no. 1 (2020): 5485-5551. 
  * [121] Liu, Yinhan, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. "Roberta: A robustly optimized bert pretraining approach." arXiv preprint arXiv:1907.11692 (2019). 
  * [122] Rogers, Anna, Olga Kovaleva, and Anna Rumshisky. "A primer in BERTology: What we know about how BERT works." Transactions of the Association for Computational Linguistics 8 (2021): 842-866. 
  * [123] Houlsby, Neil, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. "Parameter-efficient transfer learning for NLP." In International Conference on Machine Learning, pp. 2790-2799. PMLR, 2019. 
  * [124] Karimi Mahabadi, Rabeeh, James Henderson, and Sebastian Ruder. "Compacter: Efficient low-rank hypercomplex adapter layers." Advances in Neural Information Processing Systems 34 (2021): 1022-1035. 
  * [125] Aghajanyan, Armen, Sonal Gupta, and Luke Zettlemoyer. "Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning." In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 7319-7328. 2021. 
  * [126] Moosavi, Nafise Sadat, Quentin Delfosse, Kristian Kersting, and Iryna Gurevych. "Adaptable Adapters." In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 3742-3753. 2022. 
  * [127] Wang, Yaqing, and Sahaj Agarwal. "AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning." In The 2022 Conference on Empirical Methods in Natural Language Processing. 2022. 
  * [128] Schick, Timo, and Hinrich Schütze. "Generating Datasets with Pretrained Language Models." In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 6943-6951. 2021. 
  * [129] Reynolds, Laria, and Kyle McDonell. "Prompt programming for large language models: Beyond the few-shot paradigm." In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1-7. 2021. 
  * [130] Wei, Jason, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. "Finetuned Language Models are Zero-Shot Learners." In International Conference on Learning Representations. 2021. 
  * [131] Liu, Pengfei, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. "Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing." ACM Computing Surveys 55, no. 9 (2023): 1-35. 
  * [132] Schick, Timo, and Hinrich Schütze. "Few-shot text generation with natural language instructions." In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 390-402. 2021. 
  * [133] Schick, Timo, and Hinrich Schütze. "Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference." In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 255-269. 2021. 
  * [134] Trinh, Trieu H., and Quoc V. Le. "A simple method for commonsense reasoning." arXiv preprint arXiv:1806.02847 (2018). 
  * [135] Yin, Wenpeng, Jamaal Hay, and Dan Roth. "Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach." In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3914-3923. 2019. 
  * [136] Wu, Wei, Fei Wang, Arianna Yuan, Fei Wu, and Jiwei Li. "CorefQA: Coreference resolution as query-based span prediction." In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 6953-6963. 2020. 
  * [137] Touvron, Hugo, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière et al. "Llama: Open and efficient foundation language models." arXiv preprint arXiv:2302.13971 (2023). 
  * [138] Thoppilan, Romal, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin et al. "Lamda: Language models for dialog applications." arXiv preprint arXiv:2201.08239 (2022). 
  * [139] Luccioni, Alexandra Sasha, Sylvain Viguier, and Anne-Laure Ligozat. "Estimating the carbon footprint of bloom, a 176b parameter language model." Journal of Machine Learning Research 24, no. 253 (2023): 1-15. 
  * [140] Faiz, Ahmad, Sotaro Kaneda, Ruhan Wang, Rita Osi, Parteek Sharma, Fan Chen, and Lei Jiang. "LLMCarbon: Modeling the end-to-end Carbon Footprint of Large Language Models." arXiv preprint arXiv:2309.14393 (2023). 
  * [141] Xu, Canwen, and Julian McAuley. "A survey on model compression and acceleration for pretrained language models." In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 9, pp. 10566-10575. 2023. 
  * [142] Smith, Shaden, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu et al. "Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language model." arXiv preprint arXiv:2201.11990 (2022). 
  * [143] Chowdhery, Aakanksha, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham et al. "Palm: Scaling language modeling with pathways." Journal of Machine Learning Research 24, no. 240 (2023): 1-113. 
  * [144] Patra, Barun, Saksham Singhal, Shaohan Huang, Zewen Chi, Li Dong, Furu Wei, Vishrav Chaudhary, and Xia Song. "Beyond english-centric bitexts for better multilingual language representation learning." arXiv preprint arXiv:2210.14867 (2022). 
  * [145] Conneau, Alexis, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Édouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. "Unsupervised Cross-lingual Representation Learning at Scale." In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8440-8451. 2020. 
  * [146] Zhang, Jingqing, Yao Zhao, Mohammad Saleh, and Peter Liu. "Pegasus: Pre-training with extracted gap-sentences for abstractive summarization." In International Conference on Machine Learning, pp. 11328-11339. PMLR, 2020. 
  * [147] Aromataris, Edoardo, and Alan Pearson. "The systematic review: an overview." AJN The American Journal of Nursing 114, no. 3 (2014): 53-58. 
  * [148] Moher, David, Alessandro Liberati, Jennifer Tetzlaff, Douglas G. Altman, and Prisma Group. "Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement." International journal of surgery 8, no. 5 (2010): 336-341. 
  * [149] Grant, Maria J., and Andrew Booth. "A typology of reviews: an analysis of 14 review types and associated methodologies." Health information & libraries journal 26, no. 2 (2009): 91-108. 
  * [150] Khadivi, Nasim, and Sho Sato. "A Bibliometric Study of Natural Language Processing Using Dimensions Database: Development, Research Trend, and Future Research Directions." Journal of Data Science, Informetrics, and Citation Studies 2, no. 2 (2023): 77-89. 
  * [151] Bannour, Nesrine, Sahar Ghannay, Aurélie Névéol, and Anne-Laure Ligozat. "Evaluating the carbon footprint of NLP methods: a survey and analysis of existing tools." In Proceedings of the Second Workshop on Simple and Efficient Natural Language Processing, pp. 11-21. 2021. 
  * [152] Petersen, Kai, Sairam Vakkalanka, and Ludwik Kuzniarz. "Guidelines for conducting systematic mapping studies in software engineering: An update." Information and software technology 64 (2015): 1-18. 
  * [153] Koubaa, Anis, Wadii Boulila, Lahouari Ghouti, Ayyub Alzahem, and Shahid Latif. "Exploring ChatGPT Capabilities and Limitations: A Survey." IEEE Access (2023). 
  * [154] Denney, Andrew S., and Richard Tewksbury. "How to write a literature review." Journal of criminal justice education 24, no. 2 (2013): 218-234. 
  * [155] Tay, Yi, Mostafa Dehghani, Dara Bahri, and Donald Metzler. "Efficient Transformers: A Survey." ACM Computing Surveys 55, no. 6 (2023): 1-28. https://doi.org/10.1145/3530811 
  * [156] Ainslie, Joshua, Santiago Ontanon, Chris Alberti, Vaclav Cvicek, Zachary Fisher, Philip Pham, Anirudh Ravula, Sumit Sanghai, Qifan Wang, and Li Yang. "ETC: Encoding Long and Structured Inputs in Transformers." In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 268-284. 2020. 
  * [157] Zaheer, Manzil, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham et al. "Big bird: Transformers for longer sequences." Advances in neural information processing systems 33 (2020): 17283-17297. 





