# documents

scrape_date: 2025-03-17T13:26:49.795568
content_type: markdown
hash: b05a4c9ff23cf178e5558bee650ec0f16b252c3ab9ca426983e517ec87207aae
url:
https://github.com/akitenkrad/akitenkrad-blog/blob/e493935c054a5ba82143a5b8b5a0daeebcafb376/src/content/posts/arxiv/202307/20230728000000/index.md

title:
akitenkrad-blog/src/content/posts/arxiv/202307/20230728000000/index.md at e493935c054a5ba82143a5b8b5a0daeebcafb376 · akitenkrad/akitenkrad-blog · GitHub

content:
Skip to content

## Navigation Menu

Toggle navigation

[ ](/)

[ Sign in ](/login?return_to=https%3A%2F%2Fgithub.com%2Fakitenkrad%2Fakitenkrad-blog%2Fblob%2Fe493935c054a5ba82143a5b8b5a0daeebcafb376%2Fsrc%2Fcontent%2Fposts%2Farxiv%2F202307%2F20230728000000%2Findex.md)

  * Product 

    * [ GitHub Copilot Write better code with AI  ](https://github.com/features/copilot)
    * [ Security Find and fix vulnerabilities  ](https://github.com/features/security)
    * [ Actions Automate any workflow  ](https://github.com/features/actions)
    * [ Codespaces Instant dev environments  ](https://github.com/features/codespaces)
    * [ Issues Plan and track work  ](https://github.com/features/issues)
    * [ Code Review Manage code changes  ](https://github.com/features/code-review)
    * [ Discussions Collaborate outside of code  ](https://github.com/features/discussions)
    * [ Code Search Find more, search less  ](https://github.com/features/code-search)

Explore
    * [ All features ](https://github.com/features)
    * [ Documentation  ](https://docs.github.com)
    * [ GitHub Skills  ](https://skills.github.com)
    * [ Blog  ](https://github.blog)

  * Solutions 

By company size
    * [ Enterprises ](https://github.com/enterprise)
    * [ Small and medium teams ](https://github.com/team)
    * [ Startups ](https://github.com/enterprise/startups)
    * [ Nonprofits ](/solutions/industry/nonprofits)

By use case
    * [ DevSecOps ](/solutions/use-case/devsecops)
    * [ DevOps ](/solutions/use-case/devops)
    * [ CI/CD ](/solutions/use-case/ci-cd)
    * [ View all use cases ](/solutions/use-case)

By industry
    * [ Healthcare ](/solutions/industry/healthcare)
    * [ Financial services ](/solutions/industry/financial-services)
    * [ Manufacturing ](/solutions/industry/manufacturing)
    * [ Government ](/solutions/industry/government)
    * [ View all industries ](/solutions/industry)

[ View all solutions  ](/solutions)

  * Resources 

Topics
    * [ AI ](/resources/articles/ai)
    * [ DevOps ](/resources/articles/devops)
    * [ Security ](/resources/articles/security)
    * [ Software Development ](/resources/articles/software-development)
    * [ View all ](/resources/articles)

Explore
    * [ Learning Pathways  ](https://resources.github.com/learn/pathways)
    * [ Events & Webinars  ](https://resources.github.com)
    * [ Ebooks & Whitepapers ](https://github.com/resources/whitepapers)
    * [ Customer Stories ](https://github.com/customer-stories)
    * [ Partners  ](https://partner.github.com)
    * [ Executive Insights ](https://github.com/solutions/executive-insights)

  * Open Source 

    * [ GitHub Sponsors Fund open source developers  ](/sponsors)

    * [ The ReadME Project GitHub community articles  ](https://github.com/readme)

Repositories
    * [ Topics ](https://github.com/topics)
    * [ Trending ](https://github.com/trending)
    * [ Collections ](https://github.com/collections)

  * Enterprise 

    * [ Enterprise platform AI-powered developer platform  ](/enterprise)

Available add-ons
    * [ Advanced Security Enterprise-grade security features  ](https://github.com/enterprise/advanced-security)
    * [ Copilot for business Enterprise-grade AI features  ](/features/copilot/copilot-business)
    * [ Premium Support Enterprise-grade 24/7 support  ](/premium-support)

  * [Pricing](https://github.com/pricing)



Search or jump to...

# Search code, repositories, users, issues, pull requests...

Search 

Clear




[Search syntax tips](https://docs.github.com/search-github/github-code-search/understanding-github-code-search-syntax)

#  Provide feedback 

We read every piece of feedback, and take your input very seriously.

Include my email address so I can be contacted

Cancel  Submit feedback 

#  Saved searches 

## Use saved searches to filter your results more quickly

Name

Query

To see all available qualifiers, see our [documentation](https://docs.github.com/search-github/github-code-search/understanding-github-code-search-syntax). 

Cancel  Create saved search 

[ Sign in ](/login?return_to=https%3A%2F%2Fgithub.com%2Fakitenkrad%2Fakitenkrad-blog%2Fblob%2Fe493935c054a5ba82143a5b8b5a0daeebcafb376%2Fsrc%2Fcontent%2Fposts%2Farxiv%2F202307%2F20230728000000%2Findex.md)

[ Sign up ](/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fblob%2Fshow&source=header-repo&source_repo=akitenkrad%2Fakitenkrad-blog) Reseting focus

You signed in with another tab or window. [Reload]() to refresh your session. You signed out in another tab or window. [Reload]() to refresh your session. You switched accounts on another tab or window. [Reload]() to refresh your session. Dismiss alert

{{ message }}

[ akitenkrad ](/akitenkrad) / **[akitenkrad-blog](/akitenkrad/akitenkrad-blog) ** Public

  * [ Notifications ](/login?return_to=%2Fakitenkrad%2Fakitenkrad-blog) You must be signed in to change notification settings
  * [ Fork 0 ](/login?return_to=%2Fakitenkrad%2Fakitenkrad-blog)
  * [ Star  2 ](/login?return_to=%2Fakitenkrad%2Fakitenkrad-blog)




  * [ Code ](/akitenkrad/akitenkrad-blog)
  * [ Issues 0 ](/akitenkrad/akitenkrad-blog/issues)
  * [ Pull requests 0 ](/akitenkrad/akitenkrad-blog/pulls)
  * [ Actions ](/akitenkrad/akitenkrad-blog/actions)
  * [ Projects 0 ](/akitenkrad/akitenkrad-blog/projects)
  * [ Security ](/akitenkrad/akitenkrad-blog/security)
  * [ Insights ](/akitenkrad/akitenkrad-blog/pulse)



Additional navigation options

  * [ Code  ](/akitenkrad/akitenkrad-blog)
  * [ Issues  ](/akitenkrad/akitenkrad-blog/issues)
  * [ Pull requests  ](/akitenkrad/akitenkrad-blog/pulls)
  * [ Actions  ](/akitenkrad/akitenkrad-blog/actions)
  * [ Projects  ](/akitenkrad/akitenkrad-blog/projects)
  * [ Security  ](/akitenkrad/akitenkrad-blog/security)
  * [ Insights  ](/akitenkrad/akitenkrad-blog/pulse)



## Files

e493935

## Breadcrumbs

  1. [akitenkrad-blog](/akitenkrad/akitenkrad-blog/tree/e493935c054a5ba82143a5b8b5a0daeebcafb376)
  2. /[src](/akitenkrad/akitenkrad-blog/tree/e493935c054a5ba82143a5b8b5a0daeebcafb376/src)
  3. /[content](/akitenkrad/akitenkrad-blog/tree/e493935c054a5ba82143a5b8b5a0daeebcafb376/src/content)
  4. /[posts](/akitenkrad/akitenkrad-blog/tree/e493935c054a5ba82143a5b8b5a0daeebcafb376/src/content/posts)
  5. /[arxiv](/akitenkrad/akitenkrad-blog/tree/e493935c054a5ba82143a5b8b5a0daeebcafb376/src/content/posts/arxiv)
  6. /[202307](/akitenkrad/akitenkrad-blog/tree/e493935c054a5ba82143a5b8b5a0daeebcafb376/src/content/posts/arxiv/202307)
  7. /[20230728000000](/akitenkrad/akitenkrad-blog/tree/e493935c054a5ba82143a5b8b5a0daeebcafb376/src/content/posts/arxiv/202307/20230728000000)



/

# index.md

Copy path

Blame

Blame

## Latest commit

## History

[History](/akitenkrad/akitenkrad-blog/commits/e493935c054a5ba82143a5b8b5a0daeebcafb376/src/content/posts/arxiv/202307/20230728000000/index.md)

[](/akitenkrad/akitenkrad-blog/commits/e493935c054a5ba82143a5b8b5a0daeebcafb376/src/content/posts/arxiv/202307/20230728000000/index.md)

2185 lines (1279 loc) · 167 KB

e493935

## Breadcrumbs

  1. [akitenkrad-blog](/akitenkrad/akitenkrad-blog/tree/e493935c054a5ba82143a5b8b5a0daeebcafb376)
  2. /[src](/akitenkrad/akitenkrad-blog/tree/e493935c054a5ba82143a5b8b5a0daeebcafb376/src)
  3. /[content](/akitenkrad/akitenkrad-blog/tree/e493935c054a5ba82143a5b8b5a0daeebcafb376/src/content)
  4. /[posts](/akitenkrad/akitenkrad-blog/tree/e493935c054a5ba82143a5b8b5a0daeebcafb376/src/content/posts)
  5. /[arxiv](/akitenkrad/akitenkrad-blog/tree/e493935c054a5ba82143a5b8b5a0daeebcafb376/src/content/posts/arxiv)
  6. /[202307](/akitenkrad/akitenkrad-blog/tree/e493935c054a5ba82143a5b8b5a0daeebcafb376/src/content/posts/arxiv/202307)
  7. /[20230728000000](/akitenkrad/akitenkrad-blog/tree/e493935c054a5ba82143a5b8b5a0daeebcafb376/src/content/posts/arxiv/202307/20230728000000)



/

# index.md

Top

## File metadata and controls

  * Preview

  * Code

  * Blame




2185 lines (1279 loc) · 167 KB

[Raw](https://github.com/akitenkrad/akitenkrad-blog/raw/e493935c054a5ba82143a5b8b5a0daeebcafb376/src/content/posts/arxiv/202307/20230728000000/index.md)

draft | title | date | author | description | tags | menu | math  
---|---|---|---|---|---|---|---  
false | arXiv @ 2023.07.28 | 2023-07-28 | akitenkrad |  |  | arXiv | Published:2023  
---|---  
  
| sidebar  
---  
| name | identifier | parent | weight  
---|---|---|---  
arXiv @ 2023.07.28 | arxiv_20230728 | 202307_arxiv | 10  
  
true  
<iframe src="pie.html" width=900 height=620 style="border:none"></iframe>

## Primary Categories

  * cs.CL (18)
  * cs.LG (11)
  * cs.CV (20)
  * cs.HC (5)
  * cs.RO (4)
  * cs.MM (3)
  * eess.IV (4)
  * math.CT (1)
  * cs.IR (6)
  * cs.SD (2)
  * q-fin.CP (1)
  * eess.SY (1)
  * cs.AI (3)
  * cs.NI (1)
  * cs.CY (6)
  * cs.CR (4)
  * math.OC (1)
  * cs.IT (2)
  * cs.DC (1)



## cs.CL (18)

### (1/94) Speed Reading Tool Powered by Artificial Intelligence for Students with ADHD, Dyslexia, or Short Attention Span (Megat Irfan Zackry Bin Ismail Ahmad Nazran bin Yusri Muhammad Hafizzul Bin Abdul Manap Muhammad Muizzuddin Bin Kamarozaman, 2023)

{{}}

Megat Irfan Zackry Bin Ismail Ahmad Nazran bin Yusri Muhammad Hafizzul Bin Abdul Manap Muhammad Muizzuddin Bin Kamarozaman. (2023)  
**Speed Reading Tool Powered by Artificial Intelligence for Students with ADHD, Dyslexia, or Short Attention Span**

* * *

Primary Category: cs.CL  
Categories: cs-AI, cs-CL, cs.CL  
Keywords: AI, Attention, NLP, T5, Transformer  
[Paper Link](http://arxiv.org/abs/2307.14544v1)

* * *

**ABSTRACT**  
This paper presents a novel approach to assist students with dyslexia, ADHD, and short attention span in digesting any text-based information more efficiently. The proposed solution utilizes the Multilayer Perceptron (MLP) algorithm for complex text processing and summarization tasks. The tool leverages the T5 (Text-to-Text Transfer Transformer) model from Hugging Face, which treats every NLP task as a text generation task. The model is fine-tuned on specific tasks using a smaller dataset. The NLTK's Punkt Sentence Tokenizer is used to divide a text into a list of sentences. The application is served using Flask, a lightweight web server and framework. The tool also applies principles from Bionic Reading to enhance readability, which includes a bolding function and adjustments to line, word, and character spacing. The paper discusses the methodology, implementation, and results of the AI-based speed reading tool.

{{}}

### (2/94) CliniDigest: A Case Study in Large Language Model Based Large-Scale Summarization of Clinical Trial Descriptions (Renee D. White et al., 2023)

{{}}

Renee D. White, Tristan Peng, Pann Sripitak, Alexander Rosenberg Johansen, Michael Snyder, Stanford University. (2023)  
**CliniDigest: A Case Study in Large Language Model Based Large-Scale Summarization of Clinical Trial Descriptions**

* * *

Primary Category: cs.CL  
Categories: cs-CL, cs.CL  
Keywords: Clinical, GPT, GPT-3.5, Language Model, Summarization  
[Paper Link](http://arxiv.org/abs/2307.14522v1)

* * *

**ABSTRACT**  
A clinical trial is a study that evaluates new biomedical interventions. To design new trials, researchers draw inspiration from those current and completed. In 2022, there were on average more than 100 clinical trials submitted to ClinicalTrials.gov every day, with each trial having a mean of approximately 1500 words [1]. This makes it nearly impossible to keep up to date. To mitigate this issue, we have created a batch clinical trial summarizer called CliniDigest using GPT-3.5. CliniDigest is, to our knowledge, the first tool able to provide real-time, truthful, and comprehensive summaries of clinical trials. CliniDigest can reduce up to 85 clinical trial descriptions (approximately 10,500 words) into a concise 200-word summary with references and limited hallucinations. We have tested CliniDigest on its ability to summarize 457 trials divided across 27 medical subdomains. For each field, CliniDigest generates summaries of $\mu=153,\ \sigma=69 $ words, each of which utilizes $\mu=54%,\ \sigma=30% $ of the sources. A more comprehensive evaluation is planned and outlined in this paper.

{{}}

### (3/94) Controllable Generation of Dialogue Acts for Dialogue Systems via Few-Shot Response Generation and Ranking (Angela Ramirez et al., 2023)

{{}}

Angela Ramirez, Karik Agarwal, Juraj Juraska, Utkarsh Garg, Marilyn A. Walker. (2023)  
**Controllable Generation of Dialogue Acts for Dialogue Systems via Few-Shot Response Generation and Ranking**

* * *

Primary Category: cs.CL  
Categories: cs-CL, cs.CL  
Keywords: Dialog, Dialogue, Few-Shot  
[Paper Link](http://arxiv.org/abs/2307.14440v1)

* * *

**ABSTRACT**  
Dialogue systems need to produce responses that realize multiple types of dialogue acts (DAs) with high semantic fidelity. In the past, natural language generators (NLGs) for dialogue were trained on large parallel corpora that map from a domain-specific DA and its semantic attributes to an output utterance. Recent work shows that pretrained language models (LLMs) offer new possibilities for controllable NLG using prompt-based learning. Here we develop a novel few-shot overgenerate-and-rank approach that achieves the controlled generation of DAs. We compare eight few-shot prompt styles that include a novel method of generating from textual pseudo-references using a textual style transfer approach. We develop six automatic ranking functions that identify outputs with both the correct DA and high semantic accuracy at generation time. We test our approach on three domains and four LLMs. To our knowledge, this is the first work on NLG for dialogue that automatically ranks outputs using both DA and attribute accuracy. For completeness, we compare our results to fine-tuned few-shot models trained with 5 to 100 instances per DA. Our results show that several prompt settings achieve perfect DA accuracy, and near perfect semantic accuracy (99.81%) and perform better than few-shot fine-tuning.

{{}}

### (4/94) Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Models (Mayee F. Chen et al., 2023)

{{}}

Mayee F. Chen, Nicholas Roberts, Kush Bhatia, Jue Wang, Ce Zhang, Frederic Sala, Christopher Ré. (2023)  
**Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Models**

* * *

Primary Category: cs.CL  
Categories: cs-CL, cs-LG, cs.CL  
Keywords: Language Model  
[Paper Link](http://arxiv.org/abs/2307.14430v1)

* * *

**ABSTRACT**  
The quality of training data impacts the performance of pre-trained large language models (LMs). Given a fixed budget of tokens, we study how to best select data that leads to good downstream model performance across tasks. We develop a new framework based on a simple hypothesis: just as humans acquire interdependent skills in a deliberate order, language models also follow a natural order when learning a set of skills from their training data. If such an order exists, it can be utilized for improved understanding of LMs and for data-efficient training. Using this intuition, our framework formalizes the notion of a skill and of an ordered set of skills in terms of the associated data. First, using both synthetic and real data, we demonstrate that these ordered skill sets exist, and that their existence enables more advanced skills to be learned with less data when we train on their prerequisite skills. Second, using our proposed framework, we introduce an online data sampling algorithm, Skill-It, over mixtures of skills for both continual pre-training and fine-tuning regimes, where the objective is to efficiently learn multiple skills in the former and an individual skill in the latter. On the LEGO synthetic in the continual pre-training setting, Skill-It obtains 36.5 points higher accuracy than random sampling. On the Natural Instructions dataset in the fine-tuning setting, Skill-It reduces the validation loss on the target skill by 13.6% versus training on data associated with the target skill itself. We apply our skills framework on the recent RedPajama dataset to continually pre-train a 3B-parameter LM, achieving higher accuracy on the LM Evaluation Harness with 1B tokens than the baseline approach of sampling uniformly over data sources with 3B tokens.

{{}}

### (5/94) Towards Generalist Biomedical AI (Tao Tu et al., 2023)

{{}}

Tao Tu, Shekoofeh Azizi, Danny Driess, Mike Schaekermann, Mohamed Amin, Pi-Chuan Chang, Andrew Carroll, Chuck Lau, Ryutaro Tanno, Ira Ktena, Basil Mustafa, Aakanksha Chowdhery, Yun Liu, Simon Kornblith, David Fleet, Philip Mansfield, Sushant Prakash, Renee Wong, Sunny Virmani, Christopher Semturs, S Sara Mahdavi, Bradley Green, Ewa Dominowska, Blaise Aguera y Arcas, Joelle Barral, Dale Webster, Greg S. Corrado, Yossi Matias, Karan Singhal, Pete Florence, Alan Karthikesalingam, Vivek Natarajan. (2023)  
**Towards Generalist Biomedical AI**

* * *

Primary Category: cs.CL  
Categories: cs-CL, cs-CV, cs.CL  
Keywords: AI, PaLM  
[Paper Link](http://arxiv.org/abs/2307.14334v1)

* * *

**ABSTRACT**  
Medicine is inherently multimodal, with rich data modalities spanning text, imaging, genomics, and more. Generalist biomedical artificial intelligence (AI) systems that flexibly encode, integrate, and interpret this data at scale can potentially enable impactful applications ranging from scientific discovery to care delivery. To enable the development of these models, we first curate MultiMedBench, a new multimodal biomedical benchmark. MultiMedBench encompasses 14 diverse tasks such as medical question answering, mammography and dermatology image interpretation, radiology report generation and summarization, and genomic variant calling. We then introduce Med-PaLM Multimodal (Med-PaLM M), our proof of concept for a generalist biomedical AI system. Med-PaLM M is a large multimodal generative model that flexibly encodes and interprets biomedical data including clinical language, imaging, and genomics with the same set of model weights. Med-PaLM M reaches performance competitive with or exceeding the state of the art on all MultiMedBench tasks, often surpassing specialist models by a wide margin. We also report examples of zero-shot generalization to novel medical concepts and tasks, positive transfer learning across tasks, and emergent zero-shot medical reasoning. To further probe the capabilities and limitations of Med-PaLM M, we conduct a radiologist evaluation of model-generated (and human) chest X-ray reports and observe encouraging performance across model scales. In a side-by-side ranking on 246 retrospective chest X-rays, clinicians express a pairwise preference for Med-PaLM M reports over those produced by radiologists in up to 40.50% of cases, suggesting potential clinical utility. While considerable work is needed to validate these models in real-world use cases, our results represent a milestone towards the development of generalist biomedical AI systems.

{{}}

### (6/94) Utilizing Large Language Models for Natural Interface to Pharmacology Databases (Hong Lu et al., 2023)

{{}}

Hong Lu, Chuan Li, Yinheng Li, Jie Zhao. (2023)  
**Utilizing Large Language Models for Natural Interface to Pharmacology Databases**

* * *

Primary Category: cs.CL  
Categories: cs-AI, cs-CL, cs-DB, cs.CL  
Keywords: Language Model  
[Paper Link](http://arxiv.org/abs/2307.15717v1)

* * *

**ABSTRACT**  
The drug development process necessitates that pharmacologists undertake various tasks, such as reviewing literature, formulating hypotheses, designing experiments, and interpreting results. Each stage requires accessing and querying vast amounts of information. In this abstract, we introduce a Large Language Model (LLM)-based Natural Language Interface designed to interact with structured information stored in databases. Our experiments demonstrate the feasibility and effectiveness of the proposed framework. This framework can generalize to query a wide range of pharmaceutical data and knowledge bases.

{{}}

### (7/94) Comparative Analysis of Libraries for the Sentimental Analysis (Wendy Ccoya et al., 2023)

{{}}

Wendy Ccoya, Edson Pinto. (2023)  
**Comparative Analysis of Libraries for the Sentimental Analysis**

* * *

Primary Category: cs.CL  
Categories: cs-CL, cs.CL  
Keywords: BERT, GPT, NLP, Transformer, Transformers  
[Paper Link](http://arxiv.org/abs/2307.14311v1)

* * *

**ABSTRACT**  
This study is main goal is to provide a comparative comparison of libraries using machine learning methods. Experts in natural language processing (NLP) are becoming more and more interested in sentiment analysis (SA) of text changes. The objective of employing NLP text analysis techniques is to recognize and categorize feelings related to twitter users utterances. In this examination, issues with SA and the libraries utilized are also looked at. provides a number of cooperative methods to classify emotional polarity. The Naive Bayes Classifier, Decision Tree Classifier, Maxent Classifier, Sklearn Classifier, Sklearn Classifier MultinomialNB, and other conjoint learning algorithms, according to recent research, are very effective. In the project will use Five Python and R libraries NLTK, TextBlob, Vader, Transformers (GPT and BERT pretrained), and Tidytext will be used in the study to apply sentiment analysis techniques. Four machine learning models Tree of Decisions (DT), Support Vector Machine (SVM), Naive Bayes (NB), and K-Nearest Neighbor (KNN) will also be used. To evaluate how well libraries for SA operate in the social network environment, comparative study was also carried out. The measures to assess the best algorithms in this experiment, which used a single data set for each method, were precision, recall, and F1 score. We conclude that the BERT transformer method with an Accuracy: 0.973 is recommended for sentiment analysis.

{{}}

### (8/94) Automatically Evaluating Opinion Prevalence in Opinion Summarization (Christopher Malon, 2023)

{{}}

Christopher Malon. (2023)  
**Automatically Evaluating Opinion Prevalence in Opinion Summarization**

* * *

Primary Category: cs.CL  
Categories: cs-CL, cs.CL  
Keywords: Amazon, Summarization  
[Paper Link](http://arxiv.org/abs/2307.14305v1)

* * *

**ABSTRACT**  
When faced with a large number of product reviews, it is not clear that a human can remember all of them and weight opinions representatively to write a good reference summary. We propose an automatic metric to test the prevalence of the opinions that a summary expresses, based on counting the number of reviews that are consistent with each statement in the summary, while discrediting trivial or redundant statements. To formulate this opinion prevalence metric, we consider several existing methods to score the factual consistency of a summary statement with respect to each individual source review. On a corpus of Amazon product reviews, we gather multiple human judgments of the opinion consistency, to determine which automatic metric best expresses consistency in product reviews. Using the resulting opinion prevalence metric, we show that a human authored summary has only slightly better opinion prevalence than randomly selected extracts from the source reviews, and previous extractive and abstractive unsupervised opinion summarization methods perform worse than humans. We demonstrate room for improvement with a greedy construction of extractive summaries with twice the opinion prevalence achieved by humans. Finally, we show that preprocessing source reviews by simplification can raise the opinion prevalence achieved by existing abstractive opinion summarization systems to the level of human performance.

{{}}

### (9/94) Developing and Evaluating Tiny to Medium-Sized Turkish BERT Models (Himmet Toprak Kesgin et al., 2023)

{{}}

Himmet Toprak Kesgin, Muzaffer Kaan Yuce, Mehmet Fatih Amasyali. (2023)  
**Developing and Evaluating Tiny to Medium-Sized Turkish BERT Models**

* * *

Primary Category: cs.CL  
Categories: cs-AI, cs-CL, cs-LG, cs.CL  
Keywords: BERT  
[Paper Link](http://arxiv.org/abs/2307.14134v1)

* * *

**ABSTRACT**  
This study introduces and evaluates tiny, mini, small, and medium-sized uncased Turkish BERT models, aiming to bridge the research gap in less-resourced languages. We trained these models on a diverse dataset encompassing over 75GB of text from multiple sources and tested them on several tasks, including mask prediction, sentiment analysis, news classification, and, zero-shot classification. Despite their smaller size, our models exhibited robust performance, including zero-shot task, while ensuring computational efficiency and faster execution times. Our findings provide valuable insights into the development and application of smaller language models, especially in the context of the Turkish language.

{{}}

### (10/94) Leveraging Implicit Feedback from Deployment Data in Dialogue (Richard Yuanzhe Pang et al., 2023)

{{}}

Richard Yuanzhe Pang, Stephen Roller, Kyunghyun Cho, He He, Jason Weston. (2023)  
**Leveraging Implicit Feedback from Deployment Data in Dialogue**

* * *

Primary Category: cs.CL  
Categories: cs-CL, cs.CL  
Keywords: Dialog, Dialogue  
[Paper Link](http://arxiv.org/abs/2307.14117v1)

* * *

**ABSTRACT**  
We study improving social conversational agents by learning from natural dialogue between users and a deployed model, without extra annotations. To implicitly measure the quality of a machine-generated utterance, we leverage signals like user response length, sentiment and reaction of the future human utterances in the collected dialogue episodes. Our experiments use the publicly released deployment data from BlenderBot (Xu et al., 2023). Human evaluation indicates improvements in our new models over baseline responses; however, we find that some proxy signals can lead to more generations with undesirable properties as well. For example, optimizing for conversation length can lead to more controversial or unfriendly generations compared to the baseline, whereas optimizing for positive sentiment or reaction can decrease these behaviors.

{{}}

### (11/94) Decoding ChatGPT: A Taxonomy of Existing Research, Current Challenges, and Possible Future Directions (Shahab Saquib Sohail et al., 2023)

{{}}

Shahab Saquib Sohail, Faiza Farhat, Yassine Himeur, Mohammad Nadeem, Dag Øivind Madsen, Yashbir Singh, Shadi Atalla, Wathiq Mansoor. (2023)  
**Decoding ChatGPT: A Taxonomy of Existing Research, Current Challenges, and Possible Future Directions**

* * *

Primary Category: cs.CL  
Categories: cs-CL, cs-CY, cs.CL  
Keywords: AI, ChatGPT, GPT, Transformer  
[Paper Link](http://arxiv.org/abs/2307.14107v1)

* * *

**ABSTRACT**  
Chat Generative Pre-trained Transformer (ChatGPT) has gained significant interest and attention since its launch in November 2022. It has shown impressive performance in various domains, including passing exams and creative writing. However, challenges and concerns related to biases and trust persist. In this work, we present a comprehensive review of over 100 Scopus-indexed publications on ChatGPT, aiming to provide a taxonomy of ChatGPT research and explore its applications. We critically analyze the existing literature, identifying common approaches employed in the studies. Additionally, we investigate diverse application areas where ChatGPT has found utility, such as healthcare, marketing and financial services, software engineering, academic and scientific writing, research and education, environmental science, and natural language processing. Through examining these applications, we gain valuable insights into the potential of ChatGPT in addressing real-world challenges. We also discuss crucial issues related to ChatGPT, including biases and trustworthiness, emphasizing the need for further research and development in these areas. Furthermore, we identify potential future directions for ChatGPT research, proposing solutions to current challenges and speculating on expected advancements. By fully leveraging the capabilities of ChatGPT, we can unlock its potential across various domains, leading to advancements in conversational AI and transformative impacts in society.

{{}}

### (12/94) Multi3WOZ: A Multilingual, Multi-Domain, Multi-Parallel Dataset for Training and Evaluating Culturally Adapted Task-Oriented Dialog Systems (Songbo Hu et al., 2023)

{{}}

Songbo Hu, Han Zhou, Mete Hergul, Milan Gritta, Guchun Zhang, Ignacio Iacobacci, Ivan Vulić, Anna Korhonen. (2023)  
**Multi3WOZ: A Multilingual, Multi-Domain, Multi-Parallel Dataset for Training and Evaluating Culturally Adapted Task-Oriented Dialog Systems**

* * *

Primary Category: cs.CL  
Categories: cs-CL, cs.CL  
Keywords: Dialog, Multilingual  
[Paper Link](http://arxiv.org/abs/2307.14031v1)

* * *

**ABSTRACT**  
Creating high-quality annotated data for task-oriented dialog (ToD) is known to be notoriously difficult, and the challenges are amplified when the goal is to create equitable, culturally adapted, and large-scale ToD datasets for multiple languages. Therefore, the current datasets are still very scarce and suffer from limitations such as translation-based non-native dialogs with translation artefacts, small scale, or lack of cultural adaptation, among others. In this work, we first take stock of the current landscape of multilingual ToD datasets, offering a systematic overview of their properties and limitations. Aiming to reduce all the detected limitations, we then introduce Multi3WOZ, a novel multilingual, multi-domain, multi-parallel ToD dataset. It is large-scale and offers culturally adapted dialogs in 4 languages to enable training and evaluation of multilingual and cross-lingual ToD systems. We describe a complex bottom-up data collection process that yielded the final dataset, and offer the first sets of baseline scores across different ToD-related tasks for future reference, also highlighting its challenging nature.

{{}}

### (13/94) Affective Natural Language Generation of Event Descriptions through Fine-grained Appraisal Conditions (Yarik Menchaca Resendiz et al., 2023)

{{}}

Yarik Menchaca Resendiz, Roman Klinger. (2023)  
**Affective Natural Language Generation of Event Descriptions through Fine-grained Appraisal Conditions**

* * *

Primary Category: cs.CL  
Categories: cs-CL, cs.CL  
Keywords: Attention, Natural Language Generation, T5  
[Paper Link](http://arxiv.org/abs/2307.14004v1)

* * *

**ABSTRACT**  
Models for affective text generation have shown a remarkable progress, but they commonly rely only on basic emotion theories or valance/arousal values as conditions. This is appropriate when the goal is to create explicit emotion statements ("The kid is happy."). Emotions are, however, commonly communicated implicitly. For instance, the emotional interpretation of an event ("Their dog died.") does often not require an explicit emotion statement. In psychology, appraisal theories explain the link between a cognitive evaluation of an event and the potentially developed emotion. They put the assessment of the situation on the spot, for instance regarding the own control or the responsibility for what happens. We hypothesize and subsequently show that including appraisal variables as conditions in a generation framework comes with two advantages. (1) The generation model is informed in greater detail about what makes a specific emotion and what properties it has. This leads to text generation that better fulfills the condition. (2) The variables of appraisal allow a user to perform a more fine-grained control of the generated text, by stating properties of a situation instead of only providing the emotion category. Our Bart and T5-based experiments with 7 emotions (Anger, Disgust, Fear, Guilt, Joy, Sadness, Shame), and 7 appraisals (Attention, Responsibility, Control, Circumstance, Pleasantness, Effort, Certainty) show that (1) adding appraisals during training improves the accurateness of the generated texts by 10 pp in F1. Further, (2) the texts with appraisal variables are longer and contain more details. This exemplifies the greater control for users.

{{}}

### (14/94) This is not correct! Negation-aware Evaluation of Language Generation Systems (Miriam Anschütz et al., 2023)

{{}}

Miriam Anschütz, Diego Miguel Lozano, Georg Groh. (2023)  
**This is not correct! Negation-aware Evaluation of Language Generation Systems**

* * *

Primary Category: cs.CL  
Categories: cs-CL, cs-LG, cs.CL  
Keywords: BLEU  
[Paper Link](http://arxiv.org/abs/2307.13989v1)

* * *

**ABSTRACT**  
Large language models underestimate the impact of negations on how much they change the meaning of a sentence. Therefore, learned evaluation metrics based on these models are insensitive to negations. In this paper, we propose NegBLEURT, a negation-aware version of the BLEURT evaluation metric. For that, we designed a rule-based sentence negation tool and used it to create the CANNOT negation evaluation dataset. Based on this dataset, we fine-tuned a sentence transformer and an evaluation metric to improve their negation sensitivity. Evaluating these models on existing benchmarks shows that our fine-tuned models outperform existing metrics on the negated sentences by far while preserving their base models' performances on other perturbations.

{{}}

### (15/94) How Does Diffusion Influence Pretrained Language Models on Out-of-Distribution Data? (Huazheng Wang et al., 2023)

{{}}

Huazheng Wang, Daixuan Cheng, Haifeng Sun, Jingyu Wang, Qi Qi, Jianxin Liao, Jing Wang, Cong Liu. (2023)  
**How Does Diffusion Influence Pretrained Language Models on Out-of-Distribution Data?**

* * *

Primary Category: cs.CL  
Categories: cs-AI, cs-CL, cs.CL  
Keywords: Language Model, NLP, Pretrained Language Models, Transformer  
[Paper Link](http://arxiv.org/abs/2307.13949v1)

* * *

**ABSTRACT**  
Transformer-based pretrained language models (PLMs) have achieved great success in modern NLP. An important advantage of PLMs is good out-of-distribution (OOD) robustness. Recently, diffusion models have attracted a lot of work to apply diffusion to PLMs. It remains under-explored how diffusion influences PLMs on OOD data. The core of diffusion models is a forward diffusion process which gradually applies Gaussian noise to inputs, and a reverse denoising process which removes noise. The noised input reconstruction is a fundamental ability of diffusion models. We directly analyze OOD robustness by measuring the reconstruction loss, including testing the abilities to reconstruct OOD data, and to detect OOD samples. Experiments are conducted by analyzing different training parameters and data statistical features on eight datasets. It shows that finetuning PLMs with diffusion degrades the reconstruction ability on OOD data. The comparison also shows that diffusion models can effectively detect OOD samples, achieving state-of-the-art performance in most of the datasets with an absolute accuracy improvement up to 18%. These results indicate that diffusion reduces OOD robustness of PLMs.

{{}}

### (16/94) GrammarGPT: Exploring Open-Source LLMs for Native Chinese Grammatical Error Correction with Supervised Fine-Tuning (Yaxin Fan et al., 2023)

{{}}

Yaxin Fan, Feng Jiang, Peifeng Li, Haizhou Li. (2023)  
**GrammarGPT: Exploring Open-Source LLMs for Native Chinese Grammatical Error Correction with Supervised Fine-Tuning**

* * *

Primary Category: cs.CL  
Categories: cs-CL, cs.CL  
Keywords: ChatGPT, GPT, Language Model, NLP  
[Paper Link](http://arxiv.org/abs/2307.13923v1)

* * *

**ABSTRACT**  
Grammatical error correction aims to correct ungrammatical sentences automatically. Recently, some work has demonstrated the excellent capabilities of closed-source Large Language Models (LLMs, e.g., ChatGPT) in grammatical error correction. However, the potential of open-source LLMs remains unexplored. In this paper, we introduced GrammarGPT, an open-source LLM, to preliminary explore its potential for native Chinese grammatical error correction. The core recipe of GrammarGPT is to leverage the hybrid dataset of ChatGPT-generated and human-annotated. For grammatical errors with clues, we proposed a heuristic method to guide ChatGPT to generate ungrammatical sentences by providing those clues. For grammatical errors without clues, we collected ungrammatical sentences from publicly available websites and manually corrected them. In addition, we employed an error-invariant augmentation method to enhance the ability of the model to correct native Chinese grammatical errors. We ultimately constructed about 1k parallel data and utilized these data to fine-tune open-source LLMs (e.g., Phoenix, released by The Chinese University of Hong Kong, Shenzhen) with instruction tuning. The experimental results show that GrammarGPT outperforms the existing SOTA system significantly. Although model parameters are 20x larger than the SOTA baseline, the required amount of data for instruction tuning is 1200x smaller, illustrating the potential of open-source LLMs on native CGEC. Our GrammarGPT ranks $3^{rd}$ on NLPCC2023 SharedTask1, demonstrating our approach's effectiveness. The code and data are available at \url{[https://github.com/FreedomIntelligence/GrammarGPT}](https://github.com/FreedomIntelligence/GrammarGPT%7D).

{{}}

### (17/94) Data Augmentation for Neural Machine Translation using Generative Language Model (Seokjin Oh et al., 2023)

{{}}

Seokjin Oh, Su ah Lee, Woohwan Jung. (2023)  
**Data Augmentation for Neural Machine Translation using Generative Language Model**

* * *

Primary Category: cs.CL  
Categories: cs-AI, cs-CL, cs.CL  
Keywords: Augmentation, BLEU, ChatGPT, GPT, Language Model, Machine Translation  
[Paper Link](http://arxiv.org/abs/2307.16833v1)

* * *

**ABSTRACT**  
Despite the rapid growth in model architecture, the scarcity of large parallel corpora remains the main bottleneck in Neural Machine Translation. Data augmentation is a technique that enhances the performance of data-hungry models by generating synthetic data instead of collecting new ones. We explore prompt-based data augmentation approaches that leverage large-scale language models such as ChatGPT. To create a synthetic parallel corpus, we compare 3 methods using different prompts. We employ two assessment metrics to measure the diversity of the generated synthetic data. This approach requires no further model training cost, which is mandatory in other augmentation methods like back-translation. The proposed method improves the unaugmented baseline by 0.68 BLEU score.

{{}}

### (18/94) FinTree: Financial Dataset Pretrain Transformer Encoder for Relation Extraction (Hyunjong Ok, 2023)

{{}}

Hyunjong Ok. (2023)  
**FinTree: Financial Dataset Pretrain Transformer Encoder for Relation Extraction**

* * *

Primary Category: cs.CL  
Categories: cs-AI, cs-CL, cs.CL  
Keywords: Financial, Relation Extraction, Transformer  
[Paper Link](http://arxiv.org/abs/2307.13900v1)

* * *

**ABSTRACT**  
We present FinTree, Financial Dataset Pretrain Transformer Encoder for Relation Extraction. Utilizing an encoder language model, we further pretrain FinTree on the financial dataset, adapting the model in financial domain tasks. FinTree stands out with its novel structure that predicts a masked token instead of the conventional [CLS] token, inspired by the Pattern Exploiting Training methodology. This structure allows for more accurate relation predictions between two given entities. The model is trained with a unique input pattern to provide contextual and positional information about the entities of interest, and a post-processing step ensures accurate predictions in line with the entity types. Our experiments demonstrate that FinTree outperforms on the REFinD, a large-scale financial relation extraction dataset. The code and pretrained models are available at <https://github.com/HJ-Ok/FinTree>.

{{}}

## cs.LG (11)

### (19/94) Controlling the Inductive Bias of Wide Neural Networks by Modifying the Kernel's Spectrum (Amnon Geifman et al., 2023)

{{}}

Amnon Geifman, Daniel Barzilai, Ronen Basri, Meirav Galun. (2023)  
**Controlling the Inductive Bias of Wide Neural Networks by Modifying the Kernel's Spectrum**

* * *

Primary Category: cs.LG  
Categories: cs-LG, cs.LG  
Keywords: Bias  
[Paper Link](http://arxiv.org/abs/2307.14531v1)

* * *

**ABSTRACT**  
Wide neural networks are biased towards learning certain functions, influencing both the rate of convergence of gradient descent (GD) and the functions that are reachable with GD in finite training time. As such, there is a great need for methods that can modify this bias according to the task at hand. To that end, we introduce Modified Spectrum Kernels (MSKs), a novel family of constructed kernels that can be used to approximate kernels with desired eigenvalues for which no closed form is known. We leverage the duality between wide neural networks and Neural Tangent Kernels and propose a preconditioned gradient descent method, which alters the trajectory of GD. As a result, this allows for a polynomial and, in some cases, exponential training speedup without changing the final solution. Our method is both computationally efficient and simple to implement.

{{}}

### (20/94) HUGE: Huge Unsupervised Graph Embeddings with TPUs (Brandon Mayer et al., 2023)

{{}}

Brandon Mayer, Anton Tsitsulin, Hendrik Fichtenberger, Jonathan Halcrow, Bryan Perozzi. (2023)  
**HUGE: Huge Unsupervised Graph Embeddings with TPUs**

* * *

Primary Category: cs.LG  
Categories: cs-DC, cs-LG, cs-SI, cs.LG  
Keywords: Embedding  
[Paper Link](http://arxiv.org/abs/2307.14490v1)

* * *

**ABSTRACT**  
Graphs are a representation of structured data that captures the relationships between sets of objects. With the ubiquity of available network data, there is increasing industrial and academic need to quickly analyze graphs with billions of nodes and trillions of edges. A common first step for network understanding is Graph Embedding, the process of creating a continuous representation of nodes in a graph. A continuous representation is often more amenable, especially at scale, for solving downstream machine learning tasks such as classification, link prediction, and clustering. A high-performance graph embedding architecture leveraging Tensor Processing Units (TPUs) with configurable amounts of high-bandwidth memory is presented that simplifies the graph embedding problem and can scale to graphs with billions of nodes and trillions of edges. We verify the embedding space quality on real and synthetic large-scale datasets.

{{}}

### (21/94) Reinforcement Learning by Guided Safe Exploration (Qisong Yang et al., 2023)

{{}}

Qisong Yang, Thiago D. Simão, Nils Jansen, Simon H. Tindemans, Matthijs T. J. Spaan. (2023)  
**Reinforcement Learning by Guided Safe Exploration**

* * *

Primary Category: cs.LG  
Categories: cs-AI, cs-LG, cs.LG  
Keywords: Reinforcement Learning  
[Paper Link](http://arxiv.org/abs/2307.14316v1)

* * *

**ABSTRACT**  
Safety is critical to broadening the application of reinforcement learning (RL). Often, we train RL agents in a controlled environment, such as a laboratory, before deploying them in the real world. However, the real-world target task might be unknown prior to deployment. Reward-free RL trains an agent without the reward to adapt quickly once the reward is revealed. We consider the constrained reward-free setting, where an agent (the guide) learns to explore safely without the reward signal. This agent is trained in a controlled environment, which allows unsafe interactions and still provides the safety signal. After the target task is revealed, safety violations are not allowed anymore. Thus, the guide is leveraged to compose a safe behaviour policy. Drawing from transfer learning, we also regularize a target policy (the student) towards the guide while the student is unreliable and gradually eliminate the influence of the guide as training progresses. The empirical analysis shows that this method can achieve safe transfer learning and helps the student solve the target task faster.

{{}}

### (22/94) Unraveling the Complexity of Splitting Sequential Data: Tackling Challenges in Video and Time Series Analysis (Diego Botache et al., 2023)

{{}}

Diego Botache, Kristina Dingel, Rico Huhnstock, Arno Ehresmann, Bernhard Sick. (2023)  
**Unraveling the Complexity of Splitting Sequential Data: Tackling Challenges in Video and Time Series Analysis**

* * *

Primary Category: cs.LG  
Categories: cs-AI, cs-CV, cs-LG, cs.LG  
Keywords: Time Series  
[Paper Link](http://arxiv.org/abs/2307.14294v1)

* * *

**ABSTRACT**  
Splitting of sequential data, such as videos and time series, is an essential step in various data analysis tasks, including object tracking and anomaly detection. However, splitting sequential data presents a variety of challenges that can impact the accuracy and reliability of subsequent analyses. This concept article examines the challenges associated with splitting sequential data, including data acquisition, data representation, split ratio selection, setting up quality criteria, and choosing suitable selection strategies. We explore these challenges through two real-world examples: motor test benches and particle tracking in liquids.

{{}}

### (23/94) A comparison of machine learning surrogate models of street-scale flooding in Norfolk, Virginia (Diana McSpadden et al., 2023)

{{}}

Diana McSpadden, Steven Goldenberg, Binata Roy, Malachi Schram, Jonathan L. Goodall, Heather Richter. (2023)  
**A comparison of machine learning surrogate models of street-scale flooding in Norfolk, Virginia**

* * *

Primary Category: cs.LG  
Categories: cs-LG, cs.LG  
Keywords: LSTM  
[Paper Link](http://arxiv.org/abs/2307.14185v1)

* * *

**ABSTRACT**  
Low-lying coastal cities, exemplified by Norfolk, Virginia, face the challenge of street flooding caused by rainfall and tides, which strain transportation and sewer systems and can lead to property damage. While high-fidelity, physics-based simulations provide accurate predictions of urban pluvial flooding, their computational complexity renders them unsuitable for real-time applications. Using data from Norfolk rainfall events between 2016 and 2018, this study compares the performance of a previous surrogate model based on a random forest algorithm with two deep learning models: Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU). This investigation underscores the importance of using a model architecture that supports the communication of prediction uncertainty and the effective integration of relevant, multi-modal features.

{{}}

### (24/94) Actions Speak What You Want: Provably Sample-Efficient Reinforcement Learning of the Quantal Stackelberg Equilibrium from Strategic Feedbacks (Siyu Chen et al., 2023)

{{}}

Siyu Chen, Mengdi Wang, Zhuoran Yang. (2023)  
**Actions Speak What You Want: Provably Sample-Efficient Reinforcement Learning of the Quantal Stackelberg Equilibrium from Strategic Feedbacks**

* * *

Primary Category: cs.LG  
Categories: cs-AI, cs-LG, cs.LG, math-ST, stat-ML, stat-TH  
Keywords: Reinforcement Learning  
[Paper Link](http://arxiv.org/abs/2307.14085v1)

* * *

**ABSTRACT**  
We study reinforcement learning (RL) for learning a Quantal Stackelberg Equilibrium (QSE) in an episodic Markov game with a leader-follower structure. In specific, at the outset of the game, the leader announces her policy to the follower and commits to it. The follower observes the leader's policy and, in turn, adopts a quantal response policy by solving an entropy-regularized policy optimization problem induced by leader's policy. The goal of the leader is to find her optimal policy, which yields the optimal expected total return, by interacting with the follower and learning from data. A key challenge of this problem is that the leader cannot observe the follower's reward, and needs to infer the follower's quantal response model from his actions against leader's policies. We propose sample-efficient algorithms for both the online and offline settings, in the context of function approximation. Our algorithms are based on (i) learning the quantal response model via maximum likelihood estimation and (ii) model-free or model-based RL for solving the leader's decision making problem, and we show that they achieve sublinear regret upper bounds. Moreover, we quantify the uncertainty of these estimators and leverage the uncertainty to implement optimistic and pessimistic algorithms for online and offline settings. Besides, when specialized to the linear and myopic setting, our algorithms are also computationally efficient. Our theoretical analysis features a novel performance-difference lemma which incorporates the error of quantal response model, which might be of independent interest.

{{}}

### (25/94) Are Transformers with One Layer Self-Attention Using Low-Rank Weight Matrices Universal Approximators? (Tokio Kajitsuka et al., 2023)

{{}}

Tokio Kajitsuka, Issei Sato. (2023)  
**Are Transformers with One Layer Self-Attention Using Low-Rank Weight Matrices Universal Approximators?**

* * *

Primary Category: cs.LG  
Categories: 68T07, I-2-0, cs-LG, cs.LG  
Keywords: Attention, Self-Attention, Transformer, Transformers  
[Paper Link](http://arxiv.org/abs/2307.14023v1)

* * *

**ABSTRACT**  
Existing analyses of the expressive capacity of Transformer models have required excessively deep layers for data memorization, leading to a discrepancy with the Transformers actually used in practice. This is primarily due to the interpretation of the softmax function as an approximation of the hardmax function. By clarifying the connection between the softmax function and the Boltzmann operator, we prove that a single layer of self-attention with low-rank weight matrices possesses the capability to perfectly capture the context of an entire input sequence. As a consequence, we show that single-layer Transformer has a memorization capacity for finite samples, and that Transformers consisting of one self-attention layer with two feed-forward neural networks are universal approximators for continuous functions on a compact domain.

{{}}

### (26/94) Controlling the Latent Space of GANs through Reinforcement Learning: A Case Study on Task-based Image-to-Image Translation (Mahyar Abbasian et al., 2023)

{{}}

Mahyar Abbasian, Taha Rajabzadeh, Ahmadreza Moradipari, Seyed Amir Hossein Aqajari, Hongsheng Lu, Amir Rahmani. (2023)  
**Controlling the Latent Space of GANs through Reinforcement Learning: A Case Study on Task-based Image-to-Image Translation**

* * *

Primary Category: cs.LG  
Categories: cs-AI, cs-CV, cs-LG, cs.LG  
Keywords: AI, Reinforcement Learning  
[Paper Link](http://arxiv.org/abs/2307.13978v1)

* * *

**ABSTRACT**  
Generative Adversarial Networks (GAN) have emerged as a formidable AI tool to generate realistic outputs based on training datasets. However, the challenge of exerting control over the generation process of GANs remains a significant hurdle. In this paper, we propose a novel methodology to address this issue by integrating a reinforcement learning (RL) agent with a latent-space GAN (l-GAN), thereby facilitating the generation of desired outputs. More specifically, we have developed an actor-critic RL agent with a meticulously designed reward policy, enabling it to acquire proficiency in navigating the latent space of the l-GAN and generating outputs based on specified tasks. To substantiate the efficacy of our approach, we have conducted a series of experiments employing the MNIST dataset, including arithmetic addition as an illustrative task. The outcomes of these experiments serve to validate our methodology. Our pioneering integration of an RL agent with a GAN model represents a novel advancement, holding great potential for enhancing generative networks in the future.

{{}}

### (27/94) Entropy Neural Estimation for Graph Contrastive Learning (Yixuan Ma et al., 2023)

{{}}

Yixuan Ma, Xiaolin Zhang, Peng Zhang, Kun Zhan. (2023)  
**Entropy Neural Estimation for Graph Contrastive Learning**

* * *

Primary Category: cs.LG  
Categories: cs-AI, cs-LG, cs.LG  
Keywords: Contrastive Learning  
[Paper Link](http://arxiv.org/abs/2307.13944v1)

* * *

**ABSTRACT**  
Contrastive learning on graphs aims at extracting distinguishable high-level representations of nodes. In this paper, we theoretically illustrate that the entropy of a dataset can be approximated by maximizing the lower bound of the mutual information across different views of a graph, \ie, entropy is estimated by a neural network. Based on this finding, we propose a simple yet effective subset sampling strategy to contrast pairwise representations between views of a dataset. In particular, we randomly sample nodes and edges from a given graph to build the input subset for a view. Two views are fed into a parameter-shared Siamese network to extract the high-dimensional embeddings and estimate the information entropy of the entire graph. For the learning process, we propose to optimize the network using two objectives, simultaneously. Concretely, the input of the contrastive loss function consists of positive and negative pairs. Our selection strategy of pairs is different from previous works and we present a novel strategy to enhance the representation ability of the graph encoder by selecting nodes based on cross-view similarities. We enrich the diversity of the positive and negative pairs by selecting highly similar samples and totally different data with the guidance of cross-view similarity scores, respectively. We also introduce a cross-view consistency constraint on the representations generated from the different views. This objective guarantees the learned representations are consistent across views from the perspective of the entire graph. We conduct extensive experiments on seven graph benchmarks, and the proposed approach achieves competitive performance compared to the current state-of-the-art methods. The source code will be publicly released once this paper is accepted.

{{}}

### (28/94) Graph Neural Networks-based Hybrid Framework For Predicting Particle Crushing Strength (Tongya Zheng et al., 2023)

{{}}

Tongya Zheng, Tianli Zhang, Qingzheng Guan, Wenjie Huang, Zunlei Feng, Mingli Song, Chun Chen. (2023)  
**Graph Neural Networks-based Hybrid Framework For Predicting Particle Crushing Strength**

* * *

Primary Category: cs.LG  
Categories: cs-CE, cs-LG, cs.LG  
Keywords: GNN, Graph Neural Network, Graph Neural Networks  
[Paper Link](http://arxiv.org/abs/2307.13909v1)

* * *

**ABSTRACT**  
Graph Neural Networks have emerged as an effective machine learning tool for multi-disciplinary tasks such as pharmaceutical molecule classification and chemical reaction prediction, because they can model non-euclidean relationships between different entities. Particle crushing, as a significant field of civil engineering, describes the breakage of granular materials caused by the breakage of particle fragment bonds under the modeling of numerical simulations, which motivates us to characterize the mechanical behaviors of particle crushing through the connectivity of particle fragments with Graph Neural Networks (GNNs). However, there lacks an open-source large-scale particle crushing dataset for research due to the expensive costs of laboratory tests or numerical simulations. Therefore, we firstly generate a dataset with 45,000 numerical simulations and 900 particle types to facilitate the research progress of machine learning for particle crushing. Secondly, we devise a hybrid framework based on GNNs to predict particle crushing strength in a particle fragment view with the advances of state of the art GNNs. Finally, we compare our hybrid framework against traditional machine learning methods and the plain MLP to verify its effectiveness. The usefulness of different features is further discussed through the gradient attribution explanation w.r.t the predictions. Our data and code are released at <https://github.com/doujiang-zheng/GNN-For-Particle-Crushing>.

{{}}

### (29/94) Robustness Verification of Deep Neural Networks using Star-Based Reachability Analysis with Variable-Length Time Series Input (Neelanjana Pal et al., 2023)

{{}}

Neelanjana Pal, Diego Manzanas Lopez, Taylor T Johnson. (2023)  
**Robustness Verification of Deep Neural Networks using Star-Based Reachability Analysis with Variable-Length Time Series Input**

* * *

Primary Category: cs.LG  
Categories: cs-AI, cs-CY, cs-LG, cs-NE, cs.LG  
Keywords: Time Series  
[Paper Link](http://arxiv.org/abs/2307.13907v1)

* * *

**ABSTRACT**  
Data-driven, neural network (NN) based anomaly detection and predictive maintenance are emerging research areas. NN-based analytics of time-series data offer valuable insights into past behaviors and estimates of critical parameters like remaining useful life (RUL) of equipment and state-of-charge (SOC) of batteries. However, input time series data can be exposed to intentional or unintentional noise when passing through sensors, necessitating robust validation and verification of these NNs. This paper presents a case study of the robustness verification approach for time series regression NNs (TSRegNN) using set-based formal methods. It focuses on utilizing variable-length input data to streamline input manipulation and enhance network architecture generalizability. The method is applied to two data sets in the Prognostics and Health Management (PHM) application areas: (1) SOC estimation of a Lithium-ion battery and (2) RUL estimation of a turbine engine. The NNs' robustness is checked using star-based reachability analysis, and several performance measures evaluate the effect of bounded perturbations in the input on network outputs, i.e., future outcomes. Overall, the paper offers a comprehensive case study for validating and verifying NN-based analytics of time-series data in real-world applications, emphasizing the importance of robustness testing for accurate and reliable predictions, especially considering the impact of noise on future outcomes.

{{}}

## cs.CV (20)

### (30/94) Open Problems in Computer Vision for Wilderness SAR and The Search for Patricia Wu-Murad (Thomas Manzini et al., 2023)

{{}}

Thomas Manzini, Robin Murphy. (2023)  
**Open Problems in Computer Vision for Wilderness SAR and The Search for Patricia Wu-Murad**

* * *

Primary Category: cs.CV  
Categories: cs-AI, cs-CV, cs-LG, cs.CV  
Keywords: Computer Vision  
[Paper Link](http://arxiv.org/abs/2307.14527v1)

* * *

**ABSTRACT**  
This paper details the challenges in applying two computer vision systems, an EfficientDET supervised learning model and the unsupervised RX spectral classifier, to 98.9 GB of drone imagery from the Wu-Murad wilderness search and rescue (WSAR) effort in Japan and identifies 3 directions for future research. There have been at least 19 proposed approaches and 3 datasets aimed at locating missing persons in drone imagery, but only 3 approaches (2 unsupervised and 1 of an unknown structure) are referenced in the literature as having been used in an actual WSAR operation. Of these proposed approaches, the EfficientDET architecture and the unsupervised spectral RX classifier were selected as the most appropriate for this setting. The EfficientDET model was applied to the HERIDAL dataset and despite achieving performance that is statistically equivalent to the state-of-the-art, the model fails to translate to the real world in terms of false positives (e.g., identifying tree limbs and rocks as people), and false negatives (e.g., failing to identify members of the search team). The poor results in practice for algorithms that showed good results on datasets suggest 3 areas of future research: more realistic datasets for wilderness SAR, computer vision models that are capable of seamlessly handling the variety of imagery that can be collected during actual WSAR operations, and better alignment on performance measures.

{{}}

### (31/94) SuperInpaint: Learning Detail-Enhanced Attentional Implicit Representation for Super-resolutional Image Inpainting (Canyu Zhang et al., 2023)

{{}}

Canyu Zhang, Qing Guo, Xiaoguang Li, Renjie Wan, Hongkai Yu, Ivor Tsang, Song Wang. (2023)  
**SuperInpaint: Learning Detail-Enhanced Attentional Implicit Representation for Super-resolutional Image Inpainting**

* * *

Primary Category: cs.CV  
Categories: cs-CV, cs.CV  
Keywords: Attention  
[Paper Link](http://arxiv.org/abs/2307.14489v1)

* * *

**ABSTRACT**  
In this work, we introduce a challenging image restoration task, referred to as SuperInpaint, which aims to reconstruct missing regions in low-resolution images and generate completed images with arbitrarily higher resolutions. We have found that this task cannot be effectively addressed by stacking state-of-the-art super-resolution and image inpainting methods as they amplify each other's flaws, leading to noticeable artifacts. To overcome these limitations, we propose the detail-enhanced attentional implicit representation (DEAR) that can achieve SuperInpaint with a single model, resulting in high-quality completed images with arbitrary resolutions. Specifically, we use a deep convolutional network to extract the latent embedding of an input image and then enhance the high-frequency components of the latent embedding via an adaptive high-pass filter. This leads to detail-enhanced semantic embedding. We further feed the semantic embedding into an unmask-attentional module that suppresses embeddings from ineffective masked pixels. Additionally, we extract a pixel-wise importance map that indicates which pixels should be used for image reconstruction. Given the coordinates of a pixel we want to reconstruct, we first collect its neighboring pixels in the input image and extract their detail-enhanced semantic embeddings, unmask-attentional semantic embeddings, importance values, and spatial distances to the desired pixel. Then, we feed all the above terms into an implicit representation and generate the color of the specified pixel. To evaluate our method, we extend three existing datasets for this new task and build 18 meaningful baselines using SOTA inpainting and super-resolution methods. Extensive experimental results demonstrate that our method outperforms all existing methods by a significant margin on four widely used metrics.

{{}}

### (32/94) Self-supervised Few-shot Learning for Semantic Segmentation: An Annotation-free Approach (Sanaz Karimijafarbigloo et al., 2023)

{{}}

Sanaz Karimijafarbigloo, Reza Azad, Dorit Merhof. (2023)  
**Self-supervised Few-shot Learning for Semantic Segmentation: An Annotation-free Approach**

* * *

Primary Category: cs.CV  
Categories: cs-CV, cs.CV  
Keywords: Semantic Segmentation  
[Paper Link](http://arxiv.org/abs/2307.14446v1)

* * *

**ABSTRACT**  
Few-shot semantic segmentation (FSS) offers immense potential in the field of medical image analysis, enabling accurate object segmentation with limited training data. However, existing FSS techniques heavily rely on annotated semantic classes, rendering them unsuitable for medical images due to the scarcity of annotations. To address this challenge, multiple contributions are proposed: First, inspired by spectral decomposition methods, the problem of image decomposition is reframed as a graph partitioning task. The eigenvectors of the Laplacian matrix, derived from the feature affinity matrix of self-supervised networks, are analyzed to estimate the distribution of the objects of interest from the support images. Secondly, we propose a novel self-supervised FSS framework that does not rely on any annotation. Instead, it adaptively estimates the query mask by leveraging the eigenvectors obtained from the support images. This approach eliminates the need for manual annotation, making it particularly suitable for medical images with limited annotated data. Thirdly, to further enhance the decoding of the query image based on the information provided by the support image, we introduce a multi-scale large kernel attention module. By selectively emphasizing relevant features and details, this module improves the segmentation process and contributes to better object delineation. Evaluations on both natural and medical image datasets demonstrate the efficiency and effectiveness of our method. Moreover, the proposed approach is characterized by its generality and model-agnostic nature, allowing for seamless integration with various deep architectures. The code is publicly available at \href{[https://github.com/mindflow-institue/annotation_free_fewshot}{\textcolor{magenta}{GitHub}}](https://github.com/mindflow-institue/annotation_free_fewshot%7D%7B%5Ctextcolor%7Bmagenta%7D%7BGitHub%7D%7D).

{{}}

### (33/94) MAMo: Leveraging Memory and Attention for Monocular Video Depth Estimation (Rajeev Yasarla et al., 2023)

{{}}

Rajeev Yasarla, Hong Cai, Jisoo Jeong, Yunxiao Shi, Risheek Garrepalli, Fatih Porikli. (2023)  
**MAMo: Leveraging Memory and Attention for Monocular Video Depth Estimation**

* * *

Primary Category: cs.CV  
Categories: cs-CV, cs.CV  
Keywords: Attention  
[Paper Link](http://arxiv.org/abs/2307.14336v1)

* * *

**ABSTRACT**  
We propose MAMo, a novel memory and attention frame-work for monocular video depth estimation. MAMo can augment and improve any single-image depth estimation networks into video depth estimation models, enabling them to take advantage of the temporal information to predict more accurate depth. In MAMo, we augment model with memory which aids the depth prediction as the model streams through the video. Specifically, the memory stores learned visual and displacement tokens of the previous time instances. This allows the depth network to cross-reference relevant features from the past when predicting depth on the current frame. We introduce a novel scheme to continuously update the memory, optimizing it to keep tokens that correspond with both the past and the present visual information. We adopt attention-based approach to process memory features where we first learn the spatio-temporal relation among the resultant visual and displacement memory tokens using self-attention module. Further, the output features of self-attention are aggregated with the current visual features through cross-attention. The cross-attended features are finally given to a decoder to predict depth on the current frame. Through extensive experiments on several benchmarks, including KITTI, NYU-Depth V2, and DDAD, we show that MAMo consistently improves monocular depth estimation networks and sets new state-of-the-art (SOTA) accuracy. Notably, our MAMo video depth estimation provides higher accuracy with lower latency, when omparing to SOTA cost-volume-based video depth models.

{{}}

### (34/94) Event-based Vision for Early Prediction of Manipulation Actions (Daniel Deniz et al., 2023)

{{}}

Daniel Deniz, Cornelia Fermuller, Eduardo Ros, Manuel Rodriguez-Alvarez, Francisco Barranco. (2023)  
**Event-based Vision for Early Prediction of Manipulation Actions**

* * *

Primary Category: cs.CV  
Categories: cs-AI, cs-CV, cs.CV  
Keywords: Transformer  
[Paper Link](http://arxiv.org/abs/2307.14332v1)

* * *

**ABSTRACT**  
Neuromorphic visual sensors are artificial retinas that output sequences of asynchronous events when brightness changes occur in the scene. These sensors offer many advantages including very high temporal resolution, no motion blur and smart data compression ideal for real-time processing. In this study, we introduce an event-based dataset on fine-grained manipulation actions and perform an experimental study on the use of transformers for action prediction with events. There is enormous interest in the fields of cognitive robotics and human-robot interaction on understanding and predicting human actions as early as possible. Early prediction allows anticipating complex stages for planning, enabling effective and real-time interaction. Our Transformer network uses events to predict manipulation actions as they occur, using online inference. The model succeeds at predicting actions early on, building up confidence over time and achieving state-of-the-art classification. Moreover, the attention-based transformer architecture allows us to study the role of the spatio-temporal patterns selected by the model. Our experiments show that the Transformer network captures action dynamic features outperforming video-based approaches and succeeding with scenarios where the differences between actions lie in very subtle cues. Finally, we release the new event dataset, which is the first in the literature for manipulation action recognition. Code will be available at <https://github.com/DaniDeniz/EventVisionTransformer>.

{{}}

### (35/94) Sparse Double Descent in Vision Transformers: real or phantom threat? (Victor Quétu et al., 2023)

{{}}

Victor Quétu, Marta Milovanovic, Enzo Tartaglione. (2023)  
**Sparse Double Descent in Vision Transformers: real or phantom threat?**

* * *

Primary Category: cs.CV  
Categories: cs-CV, cs.CV  
Keywords: Transformer, Transformers  
[Paper Link](http://arxiv.org/abs/2307.14253v1)

* * *

**ABSTRACT**  
Vision transformers (ViT) have been of broad interest in recent theoretical and empirical works. They are state-of-the-art thanks to their attention-based approach, which boosts the identification of key features and patterns within images thanks to the capability of avoiding inductive bias, resulting in highly accurate image analysis. Meanwhile, neoteric studies have reported a ``sparse double descent'' phenomenon that can occur in modern deep-learning models, where extremely over-parametrized models can generalize well. This raises practical questions about the optimal size of the model and the quest over finding the best trade-off between sparsity and performance is launched: are Vision Transformers also prone to sparse double descent? Can we find a way to avoid such a phenomenon? Our work tackles the occurrence of sparse double descent on ViTs. Despite some works that have shown that traditional architectures, like Resnet, are condemned to the sparse double descent phenomenon, for ViTs we observe that an optimally-tuned $\ell_2$ regularization relieves such a phenomenon. However, everything comes at a cost: optimal lambda will sacrifice the potential compression of the ViT.

{{}}

### (36/94) ADAPT: Efficient Multi-Agent Trajectory Prediction with Adaptation (Görkay Aydemir et al., 2023)

{{}}

Görkay Aydemir, Adil Kaan Akan, Fatma Güney. (2023)  
**ADAPT: Efficient Multi-Agent Trajectory Prediction with Adaptation**

* * *

Primary Category: cs.CV  
Categories: cs-CV, cs-RO, cs.CV  
Keywords: AI  
[Paper Link](http://arxiv.org/abs/2307.14187v1)

* * *

**ABSTRACT**  
Forecasting future trajectories of agents in complex traffic scenes requires reliable and efficient predictions for all agents in the scene. However, existing methods for trajectory prediction are either inefficient or sacrifice accuracy. To address this challenge, we propose ADAPT, a novel approach for jointly predicting the trajectories of all agents in the scene with dynamic weight learning. Our approach outperforms state-of-the-art methods in both single-agent and multi-agent settings on the Argoverse and Interaction datasets, with a fraction of their computational overhead. We attribute the improvement in our performance: first, to the adaptive head augmenting the model capacity without increasing the model size; second, to our design choices in the endpoint-conditioned prediction, reinforced by gradient stopping. Our analyses show that ADAPT can focus on each agent with adaptive prediction, allowing for accurate predictions efficiently. <https://KUIS-AI.github.io/adapt>

{{}}

### (37/94) Resolution-Aware Design of Atrous Rates for Semantic Segmentation Networks (Bum Jun Kim et al., 2023)

{{}}

Bum Jun Kim, Hyeyeon Choi, Hyeonah Jang, Sang Woo Kim. (2023)  
**Resolution-Aware Design of Atrous Rates for Semantic Segmentation Networks**

* * *

Primary Category: cs.CV  
Categories: cs-CV, cs.CV  
Keywords: AI, Semantic Segmentation  
[Paper Link](http://arxiv.org/abs/2307.14179v1)

* * *

**ABSTRACT**  
DeepLab is a widely used deep neural network for semantic segmentation, whose success is attributed to its parallel architecture called atrous spatial pyramid pooling (ASPP). ASPP uses multiple atrous convolutions with different atrous rates to extract both local and global information. However, fixed values of atrous rates are used for the ASPP module, which restricts the size of its field of view. In principle, atrous rate should be a hyperparameter to change the field of view size according to the target task or dataset. However, the manipulation of atrous rate is not governed by any guidelines. This study proposes practical guidelines for obtaining an optimal atrous rate. First, an effective receptive field for semantic segmentation is introduced to analyze the inner behavior of segmentation networks. We observed that the use of ASPP module yielded a specific pattern in the effective receptive field, which was traced to reveal the module's underlying mechanism. Accordingly, we derive practical guidelines for obtaining the optimal atrous rate, which should be controlled based on the size of input image. Compared to other values, using the optimal atrous rate consistently improved the segmentation results across multiple datasets, including the STARE, CHASE_DB1, HRF, Cityscapes, and iSAID datasets.

{{}}

### (38/94) LOIS: Looking Out of Instance Semantics for Visual Question Answering (Siyu Zhang et al., 2023)

{{}}

Siyu Zhang, Yeming Chen, Yaoru Sun, Fang Wang, Haibo Shi, Haoran Wang. (2023)  
**LOIS: Looking Out of Instance Semantics for Visual Question Answering**

* * *

Primary Category: cs.CV  
Categories: cs-AI, cs-CL, cs-CV, cs.CV  
Keywords: QA, Question Answering  
[Paper Link](http://arxiv.org/abs/2307.14142v1)

* * *

**ABSTRACT**  
Visual question answering (VQA) has been intensively studied as a multimodal task that requires effort in bridging vision and language to infer answers correctly. Recent attempts have developed various attention-based modules for solving VQA tasks. However, the performance of model inference is largely bottlenecked by visual processing for semantics understanding. Most existing detection methods rely on bounding boxes, remaining a serious challenge for VQA models to understand the causal nexus of object semantics in images and correctly infer contextual information. To this end, we propose a finer model framework without bounding boxes in this work, termed Looking Out of Instance Semantics (LOIS) to tackle this important issue. LOIS enables more fine-grained feature descriptions to produce visual facts. Furthermore, to overcome the label ambiguity caused by instance masks, two types of relation attention modules: 1) intra-modality and 2) inter-modality, are devised to infer the correct answers from the different multi-view features. Specifically, we implement a mutual relation attention module to model sophisticated and deeper visual semantic relations between instance objects and background information. In addition, our proposed attention model can further analyze salient image regions by focusing on important word-related questions. Experimental results on four benchmark VQA datasets prove that our proposed method has favorable performance in improving visual reasoning capability.

{{}}

### (39/94) Creative Birds: Self-Supervised Single-View 3D Style Transfer (Renke Wang et al., 2023)

{{}}

Renke Wang, Guimin Que, Shuo Chen, Xiang Li, Jun Li, Jian Yang. (2023)  
**Creative Birds: Self-Supervised Single-View 3D Style Transfer**

* * *

Primary Category: cs.CV  
Categories: cs-CV, cs.CV  
Keywords: Self-Supervised, Style Transfer  
[Paper Link](http://arxiv.org/abs/2307.14127v2)

* * *

**ABSTRACT**  
In this paper, we propose a novel method for single-view 3D style transfer that generates a unique 3D object with both shape and texture transfer. Our focus lies primarily on birds, a popular subject in 3D reconstruction, for which no existing single-view 3D transfer methods have been developed.The method we propose seeks to generate a 3D mesh shape and texture of a bird from two single-view images. To achieve this, we introduce a novel shape transfer generator that comprises a dual residual gated network (DRGNet), and a multi-layer perceptron (MLP). DRGNet extracts the features of source and target images using a shared coordinate gate unit, while the MLP generates spatial coordinates for building a 3D mesh. We also introduce a semantic UV texture transfer module that implements textural style transfer using semantic UV segmentation, which ensures consistency in the semantic meaning of the transferred regions. This module can be widely adapted to many existing approaches. Finally, our method constructs a novel 3D bird using a differentiable renderer. Experimental results on the CUB dataset verify that our method achieves state-of-the-art performance on the single-view 3D style transfer task. Code is available in <https://github.com/wrk226/creative_birds>.

{{}}

### (40/94) Memory-Efficient Graph Convolutional Networks for Object Classification and Detection with Event Cameras (Kamil Jeziorek et al., 2023)

{{}}

Kamil Jeziorek, Andrea Pinna, Tomasz Kryjak. (2023)  
**Memory-Efficient Graph Convolutional Networks for Object Classification and Detection with Event Cameras**

* * *

Primary Category: cs.CV  
Categories: cs-CV, cs.CV, eess-IV  
Keywords: Graph Convolutional Network  
[Paper Link](http://arxiv.org/abs/2307.14124v1)

* * *

**ABSTRACT**  
Recent advances in event camera research emphasize processing data in its original sparse form, which allows the use of its unique features such as high temporal resolution, high dynamic range, low latency, and resistance to image blur. One promising approach for analyzing event data is through graph convolutional networks (GCNs). However, current research in this domain primarily focuses on optimizing computational costs, neglecting the associated memory costs. In this paper, we consider both factors together in order to achieve satisfying results and relatively low model complexity. For this purpose, we performed a comparative analysis of different graph convolution operations, considering factors such as execution time, the number of trainable model parameters, data format requirements, and training outcomes. Our results show a 450-fold reduction in the number of parameters for the feature extraction module and a 4.5-fold reduction in the size of the data representation while maintaining a classification accuracy of 52.3%, which is 6.3% higher compared to the operation used in state-of-the-art approaches. To further evaluate performance, we implemented the object detection architecture and evaluated its performance on the N-Caltech101 dataset. The results showed an accuracy of 53.7 % mAP@0.5 and reached an execution rate of 82 graphs per second.

{{}}

### (41/94) A semantics-driven methodology for high-quality image annotation (Fausto Giunchiglia et al., 2023)

{{}}

Fausto Giunchiglia, Mayukh Bagchi, Xiaolei Diao. (2023)  
**A semantics-driven methodology for high-quality image annotation**

* * *

Primary Category: cs.CV  
Categories: cs-AI, cs-CV, cs-MM, cs.CV  
Keywords: Computer Vision, ImageNet, Natural Language Processing  
[Paper Link](http://arxiv.org/abs/2307.14119v1)

* * *

**ABSTRACT**  
Recent work in Machine Learning and Computer Vision has highlighted the presence of various types of systematic flaws inside ground truth object recognition benchmark datasets. Our basic tenet is that these flaws are rooted in the many-to-many mappings which exist between the visual information encoded in images and the intended semantics of the labels annotating them. The net consequence is that the current annotation process is largely under-specified, thus leaving too much freedom to the subjective judgment of annotators. In this paper, we propose vTelos, an integrated Natural Language Processing, Knowledge Representation, and Computer Vision methodology whose main goal is to make explicit the (otherwise implicit) intended annotation semantics, thus minimizing the number and role of subjective choices. A key element of vTelos is the exploitation of the WordNet lexico-semantic hierarchy as the main means for providing the meaning of natural language labels and, as a consequence, for driving the annotation of images based on the objects and the visual properties they depict. The methodology is validated on images populating a subset of the ImageNet hierarchy.

{{}}

### (42/94) ECO: Ensembling Context Optimization for Vision-Language Models (Lorenzo Agnolucci et al., 2023)

{{}}

Lorenzo Agnolucci, Alberto Baldrati, Francesco Todino, Federico Becattini, Marco Bertini, Alberto Del Bimbo. (2023)  
**ECO: Ensembling Context Optimization for Vision-Language Models**

* * *

Primary Category: cs.CV  
Categories: cs-CV, cs.CV  
Keywords: Language Model  
[Paper Link](http://arxiv.org/abs/2307.14063v1)

* * *

**ABSTRACT**  
Image recognition has recently witnessed a paradigm shift, where vision-language models are now used to perform few-shot classification based on textual prompts. Among these, the CLIP model has shown remarkable capabilities for zero-shot transfer by matching an image and a custom textual prompt in its latent space. This has paved the way for several works that focus on engineering or learning textual contexts for maximizing CLIP's classification capabilities. In this paper, we follow this trend by learning an ensemble of prompts for image classification. We show that learning diverse and possibly shorter contexts improves considerably and consistently the results rather than relying on a single trainable prompt. In particular, we report better few-shot capabilities with no additional cost at inference time. We demonstrate the capabilities of our approach on 11 different benchmarks.

{{}}

### (43/94) ESSAformer: Efficient Transformer for Hyperspectral Image Super-resolution (Mingjin Zhang et al., 2023)

{{}}

Mingjin Zhang, Chi Zhang, Qiming Zhang, Jie Guo, Xinbo Gao, Jing Zhang. (2023)  
**ESSAformer: Efficient Transformer for Hyperspectral Image Super-resolution**

* * *

Primary Category: cs.CV  
Categories: cs-AI, cs-CV, cs.CV  
Keywords: Transformer  
[Paper Link](http://arxiv.org/abs/2307.14010v1)

* * *

**ABSTRACT**  
Single hyperspectral image super-resolution (single-HSI-SR) aims to restore a high-resolution hyperspectral image from a low-resolution observation. However, the prevailing CNN-based approaches have shown limitations in building long-range dependencies and capturing interaction information between spectral features. This results in inadequate utilization of spectral information and artifacts after upsampling. To address this issue, we propose ESSAformer, an ESSA attention-embedded Transformer network for single-HSI-SR with an iterative refining structure. Specifically, we first introduce a robust and spectral-friendly similarity metric, \ie, the spectral correlation coefficient of the spectrum (SCC), to replace the original attention matrix and incorporates inductive biases into the model to facilitate training. Built upon it, we further utilize the kernelizable attention technique with theoretical support to form a novel efficient SCC-kernel-based self-attention (ESSA) and reduce attention computation to linear complexity. ESSA enlarges the receptive field for features after upsampling without bringing much computation and allows the model to effectively utilize spatial-spectral information from different scales, resulting in the generation of more natural high-resolution images. Without the need for pretraining on large-scale datasets, our experiments demonstrate ESSA's effectiveness in both visual quality and quantitative results.

{{}}

### (44/94) Learning Snippet-to-Motion Progression for Skeleton-based Human Motion Prediction (Xinshun Wang et al., 2023)

{{}}

Xinshun Wang, Qiongjie Cui, Chen Chen, Shen Zhao, Mengyuan Liu. (2023)  
**Learning Snippet-to-Motion Progression for Skeleton-based Human Motion Prediction**

* * *

Primary Category: cs.CV  
Categories: cs-CV, cs.CV  
Keywords: Graph Convolutional Network  
[Paper Link](http://arxiv.org/abs/2307.14006v1)

* * *

**ABSTRACT**  
Existing Graph Convolutional Networks to achieve human motion prediction largely adopt a one-step scheme, which output the prediction straight from history input, failing to exploit human motion patterns. We observe that human motions have transitional patterns and can be split into snippets representative of each transition. Each snippet can be reconstructed from its starting and ending poses referred to as the transitional poses. We propose a snippet-to-motion multi-stage framework that breaks motion prediction into sub-tasks easier to accomplish. Each sub-task integrates three modules: transitional pose prediction, snippet reconstruction, and snippet-to-motion prediction. Specifically, we propose to first predict only the transitional poses. Then we use them to reconstruct the corresponding snippets, obtaining a close approximation to the true motion sequence. Finally we refine them to produce the final prediction output. To implement the network, we propose a novel unified graph modeling, which allows for direct and effective feature propagation compared to existing approaches which rely on separate space-time modeling. Extensive experiments on Human 3.6M, CMU Mocap and 3DPW datasets verify the effectiveness of our method which achieves state-of-the-art performance.

{{}}

### (45/94) Analysis of Video Quality Datasets via Design of Minimalistic Video Quality Models (Wei Sun et al., 2023)

{{}}

Wei Sun, Wen Wen, Xiongkuo Min, Long Lan, Guangtao Zhai, Kede Ma. (2023)  
**Analysis of Video Quality Datasets via Design of Minimalistic Video Quality Models**

* * *

Primary Category: cs.CV  
Categories: cs-CV, cs-MM, cs.CV, eess-IV  
Keywords: QA  
[Paper Link](http://arxiv.org/abs/2307.13981v1)

* * *

**ABSTRACT**  
Blind video quality assessment (BVQA) plays an indispensable role in monitoring and improving the end-users' viewing experience in various real-world video-enabled media applications. As an experimental field, the improvements of BVQA models have been measured primarily on a few human-rated VQA datasets. Thus, it is crucial to gain a better understanding of existing VQA datasets in order to properly evaluate the current progress in BVQA. Towards this goal, we conduct a first-of-its-kind computational analysis of VQA datasets via designing minimalistic BVQA models. By minimalistic, we restrict our family of BVQA models to build only upon basic blocks: a video preprocessor (for aggressive spatiotemporal downsampling), a spatial quality analyzer, an optional temporal quality analyzer, and a quality regressor, all with the simplest possible instantiations. By comparing the quality prediction performance of different model variants on eight VQA datasets with realistic distortions, we find that nearly all datasets suffer from the easy dataset problem of varying severity, some of which even admit blind image quality assessment (BIQA) solutions. We additionally justify our claims by contrasting our model generalizability on these VQA datasets, and by ablating a dizzying set of BVQA design choices related to the basic building blocks. Our results cast doubt on the current progress in BVQA, and meanwhile shed light on good practices of constructing next-generation VQA datasets and models.

{{}}

### (46/94) Improving Semi-Supervised Semantic Segmentation with Dual-Level Siamese Structure Network (Zhibo Tain et al., 2023)

{{}}

Zhibo Tain, Xiaolin Zhang, Peng Zhang, Kun Zhan. (2023)  
**Improving Semi-Supervised Semantic Segmentation with Dual-Level Siamese Structure Network**

* * *

Primary Category: cs.CV  
Categories: cs-CV, cs-LG, cs.CV  
Keywords: Semantic Segmentation, Semi-Supervised  
[Paper Link](http://arxiv.org/abs/2307.13938v1)

* * *

**ABSTRACT**  
Semi-supervised semantic segmentation (SSS) is an important task that utilizes both labeled and unlabeled data to reduce expenses on labeling training examples. However, the effectiveness of SSS algorithms is limited by the difficulty of fully exploiting the potential of unlabeled data. To address this, we propose a dual-level Siamese structure network (DSSN) for pixel-wise contrastive learning. By aligning positive pairs with a pixel-wise contrastive loss using strong augmented views in both low-level image space and high-level feature space, the proposed DSSN is designed to maximize the utilization of available unlabeled data. Additionally, we introduce a novel class-aware pseudo-label selection strategy for weak-to-strong supervision, which addresses the limitations of most existing methods that do not perform selection or apply a predefined threshold for all classes. Specifically, our strategy selects the top high-confidence prediction of the weak view for each class to generate pseudo labels that supervise the strong augmented views. This strategy is capable of taking into account the class imbalance and improving the performance of long-tailed classes. Our proposed method achieves state-of-the-art results on two datasets, PASCAL VOC 2012 and Cityscapes, outperforming other SSS algorithms by a significant margin.

{{}}

### (47/94) AIDE: A Vision-Driven Multi-View, Multi-Modal, Multi-Tasking Dataset for Assistive Driving Perception (Dingkang Yang et al., 2023)

{{}}

Dingkang Yang, Shuai Huang, Zhi Xu, Zhenpeng Li, Shunli Wang, Mingcheng Li, Yuzheng Wang, Yang Liu, Kun Yang, Zhaoyu Chen, Yan Wang, Jing Liu, Peixuan Zhang, Peng Zhai, Lihua Zhang. (2023)  
**AIDE: A Vision-Driven Multi-View, Multi-Modal, Multi-Tasking Dataset for Assistive Driving Perception**

* * *

Primary Category: cs.CV  
Categories: cs-CV, cs.CV  
Keywords: AI  
[Paper Link](http://arxiv.org/abs/2307.13933v1)

* * *

**ABSTRACT**  
Driver distraction has become a significant cause of severe traffic accidents over the past decade. Despite the growing development of vision-driven driver monitoring systems, the lack of comprehensive perception datasets restricts road safety and traffic security. In this paper, we present an AssIstive Driving pErception dataset (AIDE) that considers context information both inside and outside the vehicle in naturalistic scenarios. AIDE facilitates holistic driver monitoring through three distinctive characteristics, including multi-view settings of driver and scene, multi-modal annotations of face, body, posture, and gesture, and four pragmatic task designs for driving understanding. To thoroughly explore AIDE, we provide experimental benchmarks on three kinds of baseline frameworks via extensive methods. Moreover, two fusion strategies are introduced to give new insights into learning effective multi-stream/modal representations. We also systematically investigate the importance and rationality of the key components in AIDE and benchmarks. The project link is <https://github.com/ydk122024/AIDE>.

{{}}

### (48/94) EasyNet: An Easy Network for 3D Industrial Anomaly Detection (Ruitao Chen et al., 2023)

{{}}

Ruitao Chen, Guoyang Xie, Jiaqi Liu, Jinbao Wang, Ziqi Luo, Jinfan Wang, Feng Zheng. (2023)  
**EasyNet: An Easy Network for 3D Industrial Anomaly Detection**

* * *

Primary Category: cs.CV  
Categories: cs-CV, cs.CV  
Keywords: Anomaly Detection  
[Paper Link](http://arxiv.org/abs/2307.13925v2)

* * *

**ABSTRACT**  
3D anomaly detection is an emerging and vital computer vision task in industrial manufacturing (IM). Recently many advanced algorithms have been published, but most of them cannot meet the needs of IM. There are several disadvantages: i) difficult to deploy on production lines since their algorithms heavily rely on large pre-trained models; ii) hugely increase storage overhead due to overuse of memory banks; iii) the inference speed cannot be achieved in real-time. To overcome these issues, we propose an easy and deployment-friendly network (called EasyNet) without using pre-trained models and memory banks: firstly, we design a multi-scale multi-modality feature encoder-decoder to accurately reconstruct the segmentation maps of anomalous regions and encourage the interaction between RGB images and depth images; secondly, we adopt a multi-modality anomaly segmentation network to achieve a precise anomaly map; thirdly, we propose an attention-based information entropy fusion module for feature fusion during inference, making it suitable for real-time deployment. Extensive experiments show that EasyNet achieves an anomaly detection AUROC of 92.6% without using pre-trained models and memory banks. In addition, EasyNet is faster than existing methods, with a high frame rate of 94.55 FPS on a Tesla V100 GPU.

{{}}

### (49/94) AViT: Adapting Vision Transformers for Small Skin Lesion Segmentation Datasets (Siyi Du et al., 2023)

{{}}

Siyi Du, Nourhan Bayasi, Ghassan Harmarneh, Rafeef Garbi. (2023)  
**AViT: Adapting Vision Transformers for Small Skin Lesion Segmentation Datasets**

* * *

Primary Category: cs.CV  
Categories: cs-CV, cs.CV  
Keywords: Transformer, Transformers  
[Paper Link](http://arxiv.org/abs/2307.13897v1)

* * *

**ABSTRACT**  
Skin lesion segmentation (SLS) plays an important role in skin lesion analysis. Vision transformers (ViTs) are considered an auspicious solution for SLS, but they require more training data compared to convolutional neural networks (CNNs) due to their inherent parameter-heavy structure and lack of some inductive biases. To alleviate this issue, current approaches fine-tune pre-trained ViT backbones on SLS datasets, aiming to leverage the knowledge learned from a larger set of natural images to lower the amount of skin training data needed. However, fully fine-tuning all parameters of large backbones is computationally expensive and memory intensive. In this paper, we propose AViT, a novel efficient strategy to mitigate ViTs' data-hunger by transferring any pre-trained ViTs to the SLS task. Specifically, we integrate lightweight modules (adapters) within the transformer layers, which modulate the feature representation of a ViT without updating its pre-trained weights. In addition, we employ a shallow CNN as a prompt generator to create a prompt embedding from the input image, which grasps fine-grained information and CNN's inductive biases to guide the segmentation task on small datasets. Our quantitative experiments on 4 skin lesion datasets demonstrate that AViT achieves competitive, and at times superior, performance to SOTA but with significantly fewer trainable parameters. Our code is available at <https://github.com/siyi-wind/AViT>.

{{}}

## cs.HC (5)

### (50/94) Words That Stick: Predicting Decision Making and Synonym Engagement Using Cognitive Biases and Computational Linguistics (Nimrod Dvir et al., 2023)

{{}}

Nimrod Dvir, Elaine Friedman, Suraj Commuri, Fan Yang, Jennifer Romano. (2023)  
**Words That Stick: Predicting Decision Making and Synonym Engagement Using Cognitive Biases and Computational Linguistics**

* * *

Primary Category: cs.HC  
Categories: 03B65, H-5; I-7, cs-AI, cs-CL, cs-HC, cs.HC  
Keywords: Bias, NLP  
[Paper Link](http://arxiv.org/abs/2307.14511v1)

* * *

**ABSTRACT**  
This research draws upon cognitive psychology and information systems studies to anticipate user engagement and decision-making on digital platforms. By employing natural language processing (NLP) techniques and insights from cognitive bias research, we delve into user interactions with synonyms within digital content. Our methodology synthesizes four cognitive biasesRepresentativeness, Ease-of-use, Affect, and Distributioninto the READ model. Through a comprehensive user survey, we assess the model's ability to predict user engagement, discovering that synonyms that accurately represent core ideas, are easy to understand, elicit emotional responses, and are commonly encountered, promote greater user engagement. Crucially, our work offers a fresh lens on human-computer interaction, digital behaviors, and decision-making processes. Our results highlight the promise of cognitive biases as potent indicators of user engagement, underscoring their significance in designing effective digital content across fields like education and marketing.

{{}}

### (51/94) A Predictive Model of Digital Information Engagement: Forecasting User Engagement With English Words by Incorporating Cognitive Biases, Computational Linguistics and Natural Language Processing (Nimrod Dvir et al., 2023)

{{}}

Nimrod Dvir, Elaine Friedman, Suraj Commuri, Fan yang, Jennifer Romano. (2023)  
**A Predictive Model of Digital Information Engagement: Forecasting User Engagement With English Words by Incorporating Cognitive Biases, Computational Linguistics and Natural Language Processing**

* * *

Primary Category: cs.HC  
Categories: 68U15, H-5; H-5-1; H-5-2; D-3-2, cs-CL, cs-HC, cs-LG, cs.HC  
Keywords: AI, Bias, Natural Language Processing  
[Paper Link](http://arxiv.org/abs/2307.14500v1)

* * *

**ABSTRACT**  
This study introduces and empirically tests a novel predictive model for digital information engagement (IE) - the READ model, an acronym for the four pivotal attributes of engaging information: Representativeness, Ease-of-use, Affect, and Distribution. Conceptualized within the theoretical framework of Cumulative Prospect Theory, the model integrates key cognitive biases with computational linguistics and natural language processing to develop a multidimensional perspective on information engagement. A rigorous testing protocol was implemented, involving 50 randomly selected pairs of synonymous words (100 words in total) from the WordNet database. These words' engagement levels were evaluated through a large-scale online survey (n = 80,500) to derive empirical IE metrics. The READ attributes for each word were then computed and their predictive efficacy examined. The findings affirm the READ model's robustness, accurately predicting a word's IE level and distinguishing the more engaging word from a pair of synonyms with an 84% accuracy rate. The READ model's potential extends across various domains, including business, education, government, and healthcare, where it could enhance content engagement and inform AI language model development and generative text work. Future research should address the model's scalability and adaptability across different domains and languages, thereby broadening its applicability and efficacy.

{{}}

### (52/94) AI and Education: An Investigation into the Use of ChatGPT for Systems Thinking (Holger Arndt, 2023)

{{}}

Holger Arndt. (2023)  
**AI and Education: An Investigation into the Use of ChatGPT for Systems Thinking**

* * *

Primary Category: cs.HC  
Categories: 91, cs-AI, cs-CY, cs-HC, cs.HC  
Keywords: AI, ChatGPT, GPT  
[Paper Link](http://arxiv.org/abs/2307.14206v1)

* * *

**ABSTRACT**  
This exploratory study investigates the potential of the artificial intelligence tool, ChatGPT, to support systems thinking (ST) in various subjects. Using both general and subject specific prompts, the study assesses the accuracy, helpfulness, and reliability of ChatGPT's responses across different versions of the tool. The results indicate that ChatGPT can provide largely correct and very helpful responses in various subjects, demonstrating its potential as a tool for enhancing ST skills. However, occasional inaccuracies highlight the need for users to remain critical of ChatGPT's responses. Despite some limitations, this study suggests that with careful use and attention to its idiosyncrasies, ChatGPT can be a valuable tool for teaching and learning ST.

{{}}

### (53/94) Leveraging Large Language Models for Mental Health Prediction via Online Text Data (Xuhai Xu et al., 2023)

{{}}

Xuhai Xu, Bingshen Yao, Yuanzhe Dong, Hong Yu, James Hendler, Anind K. Dey, Dakuo Wang. (2023)  
**Leveraging Large Language Models for Mental Health Prediction via Online Text Data**

* * *

Primary Category: cs.HC  
Categories: 68U35, H-5-2; I-2-m, cs-CL, cs-HC, cs.HC  
Keywords: GPT, GPT-3.5, Language Model  
[Paper Link](http://arxiv.org/abs/2307.14385v1)

* * *

**ABSTRACT**  
The recent technology boost of large language models (LLMs) has empowered a variety of applications. However, there is very little research on understanding and improving LLMs' capability for the mental health domain. In this work, we present the first comprehensive evaluation of multiple LLMs, including Alpaca, Alpaca-LoRA, and GPT-3.5, on various mental health prediction tasks via online text data. We conduct a wide range of experiments, covering zero-shot prompting, few-shot prompting, and instruction finetuning. The results indicate the promising yet limited performance of LLMs with zero-shot and few-shot prompt designs for mental health tasks. More importantly, our experiments show that instruction finetuning can significantly boost the performance of LLMs for all tasks simultaneously. Our best-finetuned model, Mental-Alpaca, outperforms GPT-3.5 (25 times bigger) by 16.7% on balanced accuracy and performs on par with the state-of-the-art task-specific model. We summarize our findings into a set of action guidelines for future researchers, engineers, and practitioners on how to empower LLMs with better mental health domain knowledge and become an expert in mental health prediction tasks.

{{}}

### (54/94) Embedding Democratic Values into Social Media AIs via Societal Objective Functions (Chenyan Jia et al., 2023)

{{}}

Chenyan Jia, Michelle S. Lam, Minh Chau Mai, Jeff Hancock, Michael S. Bernstein. (2023)  
**Embedding Democratic Values into Social Media AIs via Societal Objective Functions**

* * *

Primary Category: cs.HC  
Categories: cs-AI, cs-HC, cs.HC  
Keywords: AI, Embedding, Social Media  
[Paper Link](http://arxiv.org/abs/2307.13912v1)

* * *

**ABSTRACT**  
Can we design artificial intelligence (AI) systems that rank our social media feeds to consider democratic values such as mitigating partisan animosity as part of their objective functions? We introduce a method for translating established, vetted social scientific constructs into AI objective functions, which we term societal objective functions, and demonstrate the method with application to the political science construct of anti-democratic attitudes. Traditionally, we have lacked observable outcomes to use to train such models, however, the social sciences have developed survey instruments and qualitative codebooks for these constructs, and their precision facilitates translation into detailed prompts for large language models. We apply this method to create a democratic attitude model that estimates the extent to which a social media post promotes anti-democratic attitudes, and test this democratic attitude model across three studies. In Study 1, we first test the attitudinal and behavioral effectiveness of the intervention among US partisans (N=1,380) by manually annotating (alpha=.895) social media posts with anti-democratic attitude scores and testing several feed ranking conditions based on these scores. Removal (d=.20) and downranking feeds (d=.25) reduced participants' partisan animosity without compromising their experience and engagement. In Study 2, we scale up the manual labels by creating the democratic attitude model, finding strong agreement with manual labels (rho=.75). Finally, in Study 3, we replicate Study 1 using the democratic attitude model instead of manual labels to test its attitudinal and behavioral impact (N=558), and again find that the feed downranking using the societal objective function reduced partisan animosity (d=.25). This method presents a novel strategy to draw on social science theory and methods to mitigate societal harms in social media AIs.

{{}}

## cs.RO (4)

### (55/94) Attention of Robot Touch: Tactile Saliency Prediction for Robust Sim-to-Real Tactile Control (Yijiong Lin et al., 2023)

{{}}

Yijiong Lin, Mauro Comi, Alex Church, Dandan Zhang, Nathan F. Lepora. (2023)  
**Attention of Robot Touch: Tactile Saliency Prediction for Robust Sim-to-Real Tactile Control**

* * *

Primary Category: cs.RO  
Categories: cs-AI, cs-RO, cs.RO  
Keywords: Attention  
[Paper Link](http://arxiv.org/abs/2307.14510v1)

* * *

**ABSTRACT**  
High-resolution tactile sensing can provide accurate information about local contact in contact-rich robotic tasks. However, the deployment of such tasks in unstructured environments remains under-investigated. To improve the robustness of tactile robot control in unstructured environments, we propose and study a new concept: \textit{tactile saliency} for robot touch, inspired by the human touch attention mechanism from neuroscience and the visual saliency prediction problem from computer vision. In analogy to visual saliency, this concept involves identifying key information in tactile images captured by a tactile sensor. While visual saliency datasets are commonly annotated by humans, manually labelling tactile images is challenging due to their counterintuitive patterns. To address this challenge, we propose a novel approach comprised of three interrelated networks: 1) a Contact Depth Network (ConDepNet), which generates a contact depth map to localize deformation in a real tactile image that contains target and noise features; 2) a Tactile Saliency Network (TacSalNet), which predicts a tactile saliency map to describe the target areas for an input contact depth map; 3) and a Tactile Noise Generator (TacNGen), which generates noise features to train the TacSalNet. Experimental results in contact pose estimation and edge-following in the presence of distractors showcase the accurate prediction of target features from real tactile images. Overall, our tactile saliency prediction approach gives robust sim-to-real tactile control in environments with unknown distractors. Project page: <https://sites.google.com/view/tactile-saliency/>.

{{}}

### (56/94) Sim-to-Real Model-Based and Model-Free Deep Reinforcement Learning for Tactile Pushing (Max Yang et al., 2023)

{{}}

Max Yang, Yijiong Lin, Alex Church, John Lloyd, Dandan Zhang, David A. W. Barton, Nathan F. Lepora. (2023)  
**Sim-to-Real Model-Based and Model-Free Deep Reinforcement Learning for Tactile Pushing**

* * *

Primary Category: cs.RO  
Categories: cs-RO, cs.RO  
Keywords: Reinforcement Learning  
[Paper Link](http://arxiv.org/abs/2307.14272v1)

* * *

**ABSTRACT**  
Object pushing presents a key non-prehensile manipulation problem that is illustrative of more complex robotic manipulation tasks. While deep reinforcement learning (RL) methods have demonstrated impressive learning capabilities using visual input, a lack of tactile sensing limits their capability for fine and reliable control during manipulation. Here we propose a deep RL approach to object pushing using tactile sensing without visual input, namely tactile pushing. We present a goal-conditioned formulation that allows both model-free and model-based RL to obtain accurate policies for pushing an object to a goal. To achieve real-world performance, we adopt a sim-to-real approach. Our results demonstrate that it is possible to train on a single object and a limited sample of goals to produce precise and reliable policies that can generalize to a variety of unseen objects and pushing scenarios without domain randomization. We experiment with the trained agents in harsh pushing conditions, and show that with significantly more training samples, a model-free policy can outperform a model-based planner, generating shorter and more reliable pushing trajectories despite large disturbances. The simplicity of our training environment and effective real-world performance highlights the value of rich tactile information for fine manipulation. Code and videos are available at <https://sites.google.com/view/tactile-rl-pushing/>.

{{}}

### (57/94) MorphoLander: Reinforcement Learning Based Landing of a Group of Drones on the Adaptive Morphogenetic UAV (Sausar Karaf et al., 2023)

{{}}

Sausar Karaf, Aleksey Fedoseev, Mikhail Martynov, Zhanibek Darush, Aleksei Shcherbak, Dzmitry Tsetserukou. (2023)  
**MorphoLander: Reinforcement Learning Based Landing of a Group of Drones on the Adaptive Morphogenetic UAV**

* * *

Primary Category: cs.RO  
Categories: cs-RO, cs.RO  
Keywords: Drone, Reinforcement Learning  
[Paper Link](http://arxiv.org/abs/2307.14147v2)

* * *

**ABSTRACT**  
This paper focuses on a novel robotic system MorphoLander representing heterogeneous swarm of drones for exploring rough terrain environments. The morphogenetic leader drone is capable of landing on uneven terrain, traversing it, and maintaining horizontal position to deploy smaller drones for extensive area exploration. After completing their tasks, these drones return and land back on the landing pads of MorphoGear. The reinforcement learning algorithm was developed for a precise landing of drones on the leader robot that either remains static during their mission or relocates to the new position. Several experiments were conducted to evaluate the performance of the developed landing algorithm under both even and uneven terrain conditions. The experiments revealed that the proposed system results in high landing accuracy of 0.5 cm when landing on the leader drone under even terrain conditions and 2.35 cm under uneven terrain conditions. MorphoLander has the potential to significantly enhance the efficiency of the industrial inspections, seismic surveys, and rescue missions in highly cluttered and unstructured environments.

{{}}

### (58/94) Research on Inertial Navigation Technology of Unmanned Aerial Vehicles with Integrated Reinforcement Learning Algorithm (Longcheng Guo, 2023)

{{}}

Longcheng Guo. (2023)  
**Research on Inertial Navigation Technology of Unmanned Aerial Vehicles with Integrated Reinforcement Learning Algorithm**

* * *

Primary Category: cs.RO  
Categories: cs-RO, cs.RO  
Keywords: Reinforcement Learning  
[Paper Link](http://arxiv.org/abs/2307.14038v1)

* * *

**ABSTRACT**  
We first define appropriate state representation and action space, and then design an adjustment mechanism based on the actions selected by the intelligent agent. The adjustment mechanism outputs the next state and reward value of the agent. Additionally, the adjustment mechanism calculates the error between the adjusted state and the unadjusted state. Furthermore, the intelligent agent stores the acquired experience samples containing states and reward values in a buffer and replays the experiences during each iteration to learn the dynamic characteristics of the environment. We name the improved algorithm as the DQM algorithm. Experimental results demonstrate that the intelligent agent using our proposed algorithm effectively reduces the accumulated errors of inertial navigation in dynamic environments. Although our research provides a basis for achieving autonomous navigation of unmanned aerial vehicles, there is still room for significant optimization. Further research can include testing unmanned aerial vehicles in simulated environments, testing unmanned aerial vehicles in real-world environments, optimizing the design of reward functions, improving the algorithm workflow to enhance convergence speed and performance, and enhancing the algorithm's generalization ability.

{{}}

## cs.MM (3)

### (59/94) Modality-Agnostic Audio-Visual Deepfake Detection (Cai Yu et al., 2023)

{{}}

Cai Yu, Peng Chen, Jiahe Tian, Jin Liu, Jiao Dai, Xi Wang, Yesheng Chai, Jizhong Han. (2023)  
**Modality-Agnostic Audio-Visual Deepfake Detection**

* * *

Primary Category: cs.MM  
Categories: cs-MM, cs-SD, cs.MM, eess-AS  
Keywords: AI  
[Paper Link](http://arxiv.org/abs/2307.14491v1)

* * *

**ABSTRACT**  
As AI-generated content (AIGC) thrives, Deepfakes have expanded from single-modality falsification to cross-modal fake content creation, where either audio or visual components can be manipulated. While using two unimodal detectors can detect audio-visual deepfakes, cross-modal forgery clues could be overlooked. Existing multimodal deepfake detection methods typically establish correspondence between the audio and visual modalities for binary real/fake classification, and require the co-occurrence of both modalities. However, in real-world multi-modal applications, missing modality scenarios may occur where either modality is unavailable. In such cases, audio-visual detection methods are less practical than two independent unimodal methods. Consequently, the detector can not always obtain the number or type of manipulated modalities beforehand, necessitating a fake-modality-agnostic audio-visual detector. In this work, we propose a unified fake-modality-agnostic scenarios framework that enables the detection of multimodal deepfakes and handles missing modalities cases, no matter the manipulation hidden in audio, video, or even cross-modal forms. To enhance the modeling of cross-modal forgery clues, we choose audio-visual speech recognition (AVSR) as a preceding task, which effectively extracts speech correlation across modalities, which is difficult for deepfakes to reproduce. Additionally, we propose a dual-label detection approach that follows the structure of AVSR to support the independent detection of each modality. Extensive experiments show that our scheme not only outperforms other state-of-the-art binary detection methods across all three audio-visual datasets but also achieves satisfying performance on detection modality-agnostic audio/video fakes. Moreover, it even surpasses the joint use of two unimodal methods in the presence of missing modality cases.

{{}}

### (60/94) Neural-based Cross-modal Search and Retrieval of Artwork (Yan Gong et al., 2023)

{{}}

Yan Gong, Georgina Cosma, Axel Finke. (2023)  
**Neural-based Cross-modal Search and Retrieval of Artwork**

* * *

Primary Category: cs.MM  
Categories: cs-MM, cs.MM  
Keywords: Embedding  
[Paper Link](http://arxiv.org/abs/2307.14244v1)

* * *

**ABSTRACT**  
Creating an intelligent search and retrieval system for artwork images, particularly paintings, is crucial for documenting cultural heritage, fostering wider public engagement, and advancing artistic analysis and interpretation. Visual-Semantic Embedding (VSE) networks are deep learning models used for information retrieval, which learn joint representations of textual and visual data, enabling 1) cross-modal search and retrieval tasks, such as image-to-text and text-to-image retrieval; and 2) relation-focused retrieval to capture entity relationships and provide more contextually relevant search results. Although VSE networks have played a significant role in cross-modal information retrieval, their application to painting datasets, such as ArtUK, remains unexplored. This paper introduces BoonArt, a VSE-based cross-modal search engine that allows users to search for images using textual queries, and to obtain textual descriptions along with the corresponding images when using image queries. The performance of BoonArt was evaluated using the ArtUK dataset. Experimental evaluations revealed that BoonArt achieved 97% Recall@10 for image-to-text retrieval, and 97.4% Recall@10 for text-to-image Retrieval. By bridging the gap between textual and visual modalities, BoonArt provides a much-improved search performance compared to traditional search engines, such as the one provided by the ArtUK website. BoonArt can be utilised to work with other artwork datasets.

{{}}

### (61/94) Boon: A Neural Search Engine for Cross-Modal Information Retrieval (Yan Gong et al., 2023)

{{}}

Yan Gong, Georgina Cosma. (2023)  
**Boon: A Neural Search Engine for Cross-Modal Information Retrieval**

* * *

Primary Category: cs.MM  
Categories: cs-MM, cs.MM  
Keywords: AI, Embedding, GPT, GPT-3.5, Google, Information Retrieval, Transformer, Transformers  
[Paper Link](http://arxiv.org/abs/2307.14240v1)

* * *

**ABSTRACT**  
Visual-Semantic Embedding (VSE) networks can help search engines better understand the meaning behind visual content and associate it with relevant textual information, leading to more accurate search results. VSE networks can be used in cross-modal search engines to embed image and textual descriptions in a shared space, enabling image-to-text and text-to-image retrieval tasks. However, the full potential of VSE networks for search engines has yet to be fully explored. This paper presents Boon, a novel cross-modal search engine that combines two state-of-the-art networks: the GPT-3.5-turbo large language model, and the VSE network VITR (VIsion Transformers with Relation-focused learning) to enhance the engine's capabilities in extracting and reasoning with regional relationships in images. VITR employs encoders from CLIP that were trained with 400 million image-description pairs and it was fine-turned on the RefCOCOg dataset. Boon's neural-based components serve as its main functionalities: 1) a 'cross-modal search engine' that enables end-users to perform image-to-text and text-to-image retrieval. 2) a 'multi-lingual conversational AI' component that enables the end-user to converse about one or more images selected by the end-user. Such a feature makes the search engine accessible to a wide audience, including those with visual impairments. 3) Boon is multi-lingual and can take queries and handle conversations about images in multiple languages. Boon was implemented using the Django and PyTorch frameworks. The interface and capabilities of the Boon search engine are demonstrated using the RefCOCOg dataset, and the engine's ability to search for multimedia through the web is facilitated by Google's API.

{{}}

## eess.IV (4)

### (62/94) Role of Image Acquisition and Patient Phenotype Variations in Automatic Segmentation Model Generalization (Timothy L. Kline et al., 2023)

{{}}

Timothy L. Kline, Sumana Ramanathan, Harrison C. Gottlich, Panagiotis Korfiatis, Adriana V. Gregory. (2023)  
**Role of Image Acquisition and Patient Phenotype Variations in Automatic Segmentation Model Generalization**

* * *

Primary Category: eess.IV  
Categories: cs-CV, cs-LG, eess-IV, eess.IV  
Keywords: AI  
[Paper Link](http://arxiv.org/abs/2307.14482v1)

* * *

**ABSTRACT**  
Purpose: This study evaluated the out-of-domain performance and generalization capabilities of automated medical image segmentation models, with a particular focus on adaptation to new image acquisitions and disease type. Materials: Datasets from both non-contrast and contrast-enhanced abdominal CT scans of healthy patients and those with polycystic kidney disease (PKD) were used. A total of 400 images (100 non-contrast controls, 100 contrast controls, 100 non-contrast PKD, 100 contrast PKD) were utilized for training/validation of models to segment kidneys, livers, and spleens, and the final models were then tested on 100 non-contrast CT images of patients affected by PKD. Performance was evaluated using Dice, Jaccard, TPR, and Precision. Results: Models trained on a diverse range of data showed no worse performance than models trained exclusively on in-domain data when tested on in-domain data. For instance, the Dice similarity of the model trained on 25% from each dataset was found to be non-inferior to the model trained purely on in-domain data. Conclusions: The results indicate that broader training examples significantly enhances model generalization and out-of-domain performance, thereby improving automated segmentation tools' applicability in clinical settings. The study's findings provide a roadmap for future research to adopt a data-centric approach in medical image AI model development.

{{}}

### (63/94) Artifact Restoration in Histology Images with Diffusion Probabilistic Models (Zhenqi He et al., 2023)

{{}}

Zhenqi He, Junjun He, Jin Ye, Yiqing Shen. (2023)  
**Artifact Restoration in Histology Images with Diffusion Probabilistic Models**

* * *

Primary Category: eess.IV  
Categories: cs-CV, eess-IV, eess.IV  
Keywords: Transformer  
[Paper Link](http://arxiv.org/abs/2307.14262v1)

* * *

**ABSTRACT**  
Histological whole slide images (WSIs) can be usually compromised by artifacts, such as tissue folding and bubbles, which will increase the examination difficulty for both pathologists and Computer-Aided Diagnosis (CAD) systems. Existing approaches to restoring artifact images are confined to Generative Adversarial Networks (GANs), where the restoration process is formulated as an image-to-image transfer. Those methods are prone to suffer from mode collapse and unexpected mistransfer in the stain style, leading to unsatisfied and unrealistic restored images. Innovatively, we make the first attempt at a denoising diffusion probabilistic model for histological artifact restoration, namely ArtiFusion.Specifically, ArtiFusion formulates the artifact region restoration as a gradual denoising process, and its training relies solely on artifact-free images to simplify the training complexity.Furthermore, to capture local-global correlations in the regional artifact restoration, a novel Swin-Transformer denoising architecture is designed, along with a time token scheme. Our extensive evaluations demonstrate the effectiveness of ArtiFusion as a pre-processing method for histology analysis, which can successfully preserve the tissue structures and stain style in artifact-free regions during the restoration. Code is available at <https://github.com/zhenqi-he/ArtiFusion>.

{{}}

### (64/94) Non-Linear Self Augmentation Deep Pipeline for Cancer Treatment outcome Prediction (Francesco Rundo et al., 2023)

{{}}

Francesco Rundo, Concetto Spampinato, Michael Rundo. (2023)  
**Non-Linear Self Augmentation Deep Pipeline for Cancer Treatment outcome Prediction**

* * *

Primary Category: eess.IV  
Categories: cs-AI, eess-IV, eess.IV  
Keywords: Augmentation  
[Paper Link](http://arxiv.org/abs/2307.14398v1)

* * *

**ABSTRACT**  
Immunotherapy emerges as promising approach for treating cancer. Encouraging findings have validated the efficacy of immunotherapy medications in addressing tumors, resulting in prolonged survival rates and notable reductions in toxicity compared to conventional chemotherapy methods. However, the pool of eligible patients for immunotherapy remains relatively small, indicating a lack of comprehensive understanding regarding the physiological mechanisms responsible for favorable treatment response in certain individuals while others experience limited benefits. To tackle this issue, the authors present an innovative strategy that harnesses a non-linear cellular architecture in conjunction with a deep downstream classifier. This approach aims to carefully select and enhance 2D features extracted from chest-abdomen CT images, thereby improving the prediction of treatment outcomes. The proposed pipeline has been meticulously designed to seamlessly integrate with an advanced embedded Point of Care system. In this context, the authors present a compelling case study focused on Metastatic Urothelial Carcinoma (mUC), a particularly aggressive form of cancer. Performance evaluation of the proposed approach underscores its effectiveness, with an impressive overall accuracy of approximately 93%

{{}}

### (65/94) Hybrid Representation-Enhanced Sampling for Bayesian Active Learning in Musculoskeletal Segmentation of Lower Extremities (Ganping Li et al., 2023)

{{}}

Ganping Li, Yoshito Otake, Mazen Soufi, Masashi Taniguchi, Masahide Yagi, Noriaki Ichihashi, Keisuke Uemura, Masaki Takao, Nobuhiko Sugano, Yoshinobu Sato. (2023)  
**Hybrid Representation-Enhanced Sampling for Bayesian Active Learning in Musculoskeletal Segmentation of Lower Extremities**

* * *

Primary Category: eess.IV  
Categories: cs-CV, eess-IV, eess.IV  
Keywords: Active Learning  
[Paper Link](http://arxiv.org/abs/2307.13986v1)

* * *

**ABSTRACT**  
Purpose: Obtaining manual annotations to train deep learning (DL) models for auto-segmentation is often time-consuming. Uncertainty-based Bayesian active learning (BAL) is a widely-adopted method to reduce annotation efforts. Based on BAL, this study introduces a hybrid representation-enhanced sampling strategy that integrates density and diversity criteria to save manual annotation costs by efficiently selecting the most informative samples. Methods: The experiments are performed on two lower extremity (LE) datasets of MRI and CT images by a BAL framework based on Bayesian U-net. Our method selects uncertain samples with high density and diversity for manual revision, optimizing for maximal similarity to unlabeled instances and minimal similarity to existing training data. We assess the accuracy and efficiency using Dice and a proposed metric called reduced annotation cost (RAC), respectively. We further evaluate the impact of various acquisition rules on BAL performance and design an ablation study for effectiveness estimation. Results: The proposed method showed superiority or non-inferiority to other methods on both datasets across two acquisition rules, and quantitative results reveal the pros and cons of the acquisition rules. Our ablation study in volume-wise acquisition shows that the combination of density and diversity criteria outperforms solely using either of them in musculoskeletal segmentation. Conclusion: Our sampling method is proven efficient in reducing annotation costs in image segmentation tasks. The combination of the proposed method and our BAL framework provides a semi-automatic way for efficient annotation of medical image datasets.

{{}}

## math.CT (1)

### (66/94) Obstructions to Compositionality (Caterina Puca et al., 2023)

{{}}

Caterina Puca, Amar Hadzihasanovic, Fabrizio Genovese, Bob Coecke. (2023)  
**Obstructions to Compositionality**

* * *

Primary Category: math.CT  
Categories: cs-LO, math-CT, math.CT  
Keywords: AI  
[Paper Link](http://arxiv.org/abs/2307.14461v1)

* * *

**ABSTRACT**  
Compositionality is at the heart of computer science and several other areas of applied category theory such as computational linguistics, categorical quantum mechanics, interpretable AI, dynamical systems, compositional game theory, and Petri nets. However, the meaning of the term seems to vary across the many different applications. This work contributes to understanding, and in particular qualifying, different kinds of compositionality. Formally, we introduce invariants of categories that we call zeroth and first homotopy posets, generalising in a precise sense the $\pi_0$ and $\pi_1$ of a groupoid. These posets can be used to obtain a qualitative description of how far an object is from being terminal and a morphism is from being iso. In the context of applied category theory, this formal machinery gives us a way to qualitatively describe the "failures of compositionality", seen as failures of certain (op)lax functors to be strong, by classifying obstructions to the (op)laxators being isomorphisms. Failure of compositionality, for example for the interpretation of a categorical syntax in a semantic universe, can both be a bad thing and a good thing, which we illustrate by respective examples in graph theory and quantum theory.

{{}}

## cs.IR (6)

### (67/94) Integrating Offline Reinforcement Learning with Transformers for Sequential Recommendation (Xumei Xi et al., 2023)

{{}}

Xumei Xi, Yuke Zhao, Quan Liu, Liwen Ouyang, Yang Wu. (2023)  
**Integrating Offline Reinforcement Learning with Transformers for Sequential Recommendation**

* * *

Primary Category: cs.IR  
Categories: cs-IR, cs.IR  
Keywords: Reinforcement Learning, Transformer, Transformers  
[Paper Link](http://arxiv.org/abs/2307.14450v1)

* * *

**ABSTRACT**  
We consider the problem of sequential recommendation, where the current recommendation is made based on past interactions. This recommendation task requires efficient processing of the sequential data and aims to provide recommendations that maximize the long-term reward. To this end, we train a farsighted recommender by using an offline RL algorithm with the policy network in our model architecture that has been initialized from a pre-trained transformer model. The pre-trained model leverages the superb ability of the transformer to process sequential information. Compared to prior works that rely on online interaction via simulation, we focus on implementing a fully offline RL framework that is able to converge in a fast and stable way. Through extensive experiments on public datasets, we show that our method is robust across various recommendation regimes, including e-commerce and movie suggestions. Compared to state-of-the-art supervised learning algorithms, our algorithm yields recommendations of higher quality, demonstrating the clear advantage of combining RL and transformers.

{{}}

### (68/94) ChatGPT and Persuasive Technologies for the Management and Delivery of Personalized Recommendations in Hotel Hospitality (Manolis Remountakis et al., 2023)

{{}}

Manolis Remountakis, Konstantinos Kotis, Babis Kourtzis, George E. Tsekouras. (2023)  
**ChatGPT and Persuasive Technologies for the Management and Delivery of Personalized Recommendations in Hotel Hospitality**

* * *

Primary Category: cs.IR  
Categories: 68T01, I-2-1, cs-AI, cs-CL, cs-IR, cs-LG, cs.IR  
Keywords: ChatGPT, GPT  
[Paper Link](http://arxiv.org/abs/2307.14298v1)

* * *

**ABSTRACT**  
Recommender systems have become indispensable tools in the hotel hospitality industry, enabling personalized and tailored experiences for guests. Recent advancements in large language models (LLMs), such as ChatGPT, and persuasive technologies, have opened new avenues for enhancing the effectiveness of those systems. This paper explores the potential of integrating ChatGPT and persuasive technologies for automating and improving hotel hospitality recommender systems. First, we delve into the capabilities of ChatGPT, which can understand and generate human-like text, enabling more accurate and context-aware recommendations. We discuss the integration of ChatGPT into recommender systems, highlighting the ability to analyze user preferences, extract valuable insights from online reviews, and generate personalized recommendations based on guest profiles. Second, we investigate the role of persuasive technology in influencing user behavior and enhancing the persuasive impact of hotel recommendations. By incorporating persuasive techniques, such as social proof, scarcity and personalization, recommender systems can effectively influence user decision-making and encourage desired actions, such as booking a specific hotel or upgrading their room. To investigate the efficacy of ChatGPT and persuasive technologies, we present a pilot experi-ment with a case study involving a hotel recommender system. We aim to study the impact of integrating ChatGPT and persua-sive techniques on user engagement, satisfaction, and conversion rates. The preliminary results demonstrate the potential of these technologies in enhancing the overall guest experience and business performance. Overall, this paper contributes to the field of hotel hospitality by exploring the synergistic relationship between LLMs and persuasive technology in recommender systems, ultimately influencing guest satisfaction and hotel revenue.

{{}}

### (69/94) Large Language Models are Competitive Near Cold-start Recommenders for Language- and Item-based Preferences (Scott Sanner et al., 2023)

{{}}

Scott Sanner, Krisztian Balog, Filip Radlinski, Ben Wedin, Lucas Dixon. (2023)  
**Large Language Models are Competitive Near Cold-start Recommenders for Language- and Item-based Preferences**

* * *

Primary Category: cs.IR  
Categories: cs-IR, cs-LG, cs.IR  
Keywords: Language Model  
[Paper Link](http://arxiv.org/abs/2307.14225v1)

* * *

**ABSTRACT**  
Traditional recommender systems leverage users' item preference history to recommend novel content that users may like. However, modern dialog interfaces that allow users to express language-based preferences offer a fundamentally different modality for preference input. Inspired by recent successes of prompting paradigms for large language models (LLMs), we study their use for making recommendations from both item-based and language-based preferences in comparison to state-of-the-art item-based collaborative filtering (CF) methods. To support this investigation, we collect a new dataset consisting of both item-based and language-based preferences elicited from users along with their ratings on a variety of (biased) recommended items and (unbiased) random items. Among numerous experimental results, we find that LLMs provide competitive recommendation performance for pure language-based preferences (no item preferences) in the near cold-start case in comparison to item-based CF methods, despite having no supervised training for this specific task (zero-shot) or only a few labels (few-shot). This is particularly promising as language-based preference representations are more explainable and scrutable than item-based or vector-based representations.

{{}}

### (70/94) A Probabilistic Position Bias Model for Short-Video Recommendation Feeds (Olivier Jeunen, 2023)

{{}}

Olivier Jeunen. (2023)  
**A Probabilistic Position Bias Model for Short-Video Recommendation Feeds**

* * *

Primary Category: cs.IR  
Categories: cs-IR, cs.IR  
Keywords: Bias  
[Paper Link](http://arxiv.org/abs/2307.14059v1)

* * *

**ABSTRACT**  
Modern web-based platforms show ranked lists of recommendations to users, attempting to maximise user satisfaction or business metrics. Typically, the goal of such systems boils down to maximising the exposure probability for items that are deemed "reward-maximising" according to a metric of interest. This general framing comprises streaming applications, as well as e-commerce or job recommendations, and even web search. Position bias or user models can be used to estimate exposure probabilities for each use-case, specifically tailored to how users interact with the presented rankings. A unifying factor in these diverse problem settings is that typically only one or several items will be engaged with (clicked, streamed,...) before a user leaves the ranked list. Short-video feeds on social media platforms diverge from this general framing in several ways, most notably that users do not tend to leave the feed after e.g. liking a post. Indeed, seemingly infinite feeds invite users to scroll further down the ranked list. For this reason, existing position bias or user models tend to fall short in such settings, as they do not accurately capture users' interaction modalities. In this work, we propose a novel and probabilistically sound personalised position bias model for feed recommendations. We focus on a 1st-level feed in a hierarchical structure, where users may enter a 2nd-level feed via any given 1st-level item. We posit that users come to the platform with a scrolling budget drawn according to some distribution, and show how the survival function of said distribution can be used to obtain closed-form estimates for personalised exposure probabilities. Empirical insights from a large-scale social media platform show how our probabilistic position bias model more accurately captures empirical exposure than existing models, and paves the way for unbiased evaluation and learning-to-rank.

{{}}

### (71/94) Multi-view Hypergraph Contrastive Policy Learning for Conversational Recommendation (Sen Zhao et al., 2023)

{{}}

Sen Zhao, Wei Wei, Xian-Ling Mao, Shuai Zhu, Minghui Yang, Zujie Wen, Dangyang Chen, Feida Zhu. (2023)  
**Multi-view Hypergraph Contrastive Policy Learning for Conversational Recommendation**

* * *

Primary Category: cs.IR  
Categories: cs-IR, cs.IR  
Keywords: Conversational Recommendation  
[Paper Link](http://arxiv.org/abs/2307.14024v1)

* * *

**ABSTRACT**  
Conversational recommendation systems (CRS) aim to interactively acquire user preferences and accordingly recommend items to users. Accurately learning the dynamic user preferences is of crucial importance for CRS. Previous works learn the user preferences with pairwise relations from the interactive conversation and item knowledge, while largely ignoring the fact that factors for a relationship in CRS are multiplex. Specifically, the user likes/dislikes the items that satisfy some attributes (Like/Dislike view). Moreover social influence is another important factor that affects user preference towards the item (Social view), while is largely ignored by previous works in CRS. The user preferences from these three views are inherently different but also correlated as a whole. The user preferences from the same views should be more similar than that from different views. The user preferences from Like View should be similar to Social View while different from Dislike View. To this end, we propose a novel model, namely Multi-view Hypergraph Contrastive Policy Learning (MHCPL). Specifically, MHCPL timely chooses useful social information according to the interactive history and builds a dynamic hypergraph with three types of multiplex relations from different views. The multiplex relations in each view are successively connected according to their generation order.

{{}}

### (72/94) Domain Disentanglement with Interpolative Data Augmentation for Dual-Target Cross-Domain Recommendation (Jiajie Zhu et al., 2023)

{{}}

Jiajie Zhu, Yan Wang, Feng Zhu, Zhu Sun. (2023)  
**Domain Disentanglement with Interpolative Data Augmentation for Dual-Target Cross-Domain Recommendation**

* * *

Primary Category: cs.IR  
Categories: 68T07 (Primary), H-3-3; I-2-6, cs-IR, cs.IR  
Keywords: Augmentation  
[Paper Link](http://arxiv.org/abs/2307.13910v1)

* * *

**ABSTRACT**  
The conventional single-target Cross-Domain Recommendation (CDR) aims to improve the recommendation performance on a sparser target domain by transferring the knowledge from a source domain that contains relatively richer information. By contrast, in recent years, dual-target CDR has been proposed to improve the recommendation performance on both domains simultaneously. However, to this end, there are two challenges in dual-target CDR: (1) how to generate both relevant and diverse augmented user representations, and (2) how to effectively decouple domain-independent information from domain-specific information, in addition to domain-shared information, to capture comprehensive user preferences. To address the above two challenges, we propose a Disentanglement-based framework with Interpolative Data Augmentation for dual-target Cross-Domain Recommendation, called DIDA-CDR. In DIDA-CDR, we first propose an interpolative data augmentation approach to generating both relevant and diverse augmented user representations to augment sparser domain and explore potential user preferences. We then propose a disentanglement module to effectively decouple domain-specific and domain-independent information to capture comprehensive user preferences. Both steps significantly contribute to capturing more comprehensive user preferences, thereby improving the recommendation performance on each domain. Extensive experiments conducted on five real-world datasets show the significant superiority of DIDA-CDR over the state-of-the-art methods.

{{}}

## cs.SD (2)

### (73/94) WavJourney: Compositional Audio Creation with Large Language Models (Xubo Liu et al., 2023)

{{}}

Xubo Liu, Zhongkai Zhu, Haohe Liu, Yi Yuan, Meng Cui, Qiushi Huang, Jinhua Liang, Yin Cao, Qiuqiang Kong, Mark D. Plumbley, Wenwu Wang. (2023)  
**WavJourney: Compositional Audio Creation with Large Language Models**

* * *

Primary Category: cs.SD  
Categories: cs-AI, cs-MM, cs-SD, cs.SD, eess-AS  
Keywords: AI, Language Model  
[Paper Link](http://arxiv.org/abs/2307.14335v1)

* * *

**ABSTRACT**  
Large Language Models (LLMs) have shown great promise in integrating diverse expert models to tackle intricate language and vision tasks. Despite their significance in advancing the field of Artificial Intelligence Generated Content (AIGC), their potential in intelligent audio content creation remains unexplored. In this work, we tackle the problem of creating audio content with storylines encompassing speech, music, and sound effects, guided by text instructions. We present WavJourney, a system that leverages LLMs to connect various audio models for audio content generation. Given a text description of an auditory scene, WavJourney first prompts LLMs to generate a structured script dedicated to audio storytelling. The audio script incorporates diverse audio elements, organized based on their spatio-temporal relationships. As a conceptual representation of audio, the audio script provides an interactive and interpretable rationale for human engagement. Afterward, the audio script is fed into a script compiler, converting it into a computer program. Each line of the program calls a task-specific audio generation model or computational operation function (e.g., concatenate, mix). The computer program is then executed to obtain an explainable solution for audio generation. We demonstrate the practicality of WavJourney across diverse real-world scenarios, including science fiction, education, and radio play. The explainable and interactive design of WavJourney fosters human-machine co-creation in multi-round dialogues, enhancing creative control and adaptability in audio production. WavJourney audiolizes the human imagination, opening up new avenues for creativity in multimedia content creation.

{{}}

### (74/94) Say Goodbye to RNN-T Loss: A Novel CIF-based Transducer Architecture for Automatic Speech Recognition (Tian-Hao Zhang et al., 2023)

{{}}

Tian-Hao Zhang, Dinghao Zhou, Guiping Zhong, Baoxiang Li. (2023)  
**Say Goodbye to RNN-T Loss: A Novel CIF-based Transducer Architecture for Automatic Speech Recognition**

* * *

Primary Category: cs.SD  
Categories: cs-CL, cs-SD, cs.SD, eess-AS  
Keywords: AI, Speech Recognition  
[Paper Link](http://arxiv.org/abs/2307.14132v2)

* * *

**ABSTRACT**  
RNN-T models are widely used in ASR, which rely on the RNN-T loss to achieve length alignment between input audio and target sequence. However, the implementation complexity and the alignment-based optimization target of RNN-T loss lead to computational redundancy and a reduced role for predictor network, respectively. In this paper, we propose a novel model named CIF-Transducer (CIF-T) which incorporates the Continuous Integrate-and-Fire (CIF) mechanism with the RNN-T model to achieve efficient alignment. In this way, the RNN-T loss is abandoned, thus bringing a computational reduction and allowing the predictor network a more significant role. We also introduce Funnel-CIF, Context Blocks, Unified Gating and Bilinear Pooling joint network, and auxiliary training strategy to further improve performance. Experiments on the 178-hour AISHELL-1 and 10000-hour WenetSpeech datasets show that CIF-T achieves state-of-the-art results with lower computational overhead compared to RNN-T models.

{{}}

## q-fin.CP (1)

### (75/94) Modeling Inverse Demand Function with Explainable Dual Neural Networks (Zhiyu Cao et al., 2023)

{{}}

Zhiyu Cao, Zihan Chen, Prerna Mishra, Hamed Amini, Zachary Feinstein. (2023)  
**Modeling Inverse Demand Function with Explainable Dual Neural Networks**

* * *

Primary Category: q-fin.CP  
Categories: J-1; I-2-6, cs-CE, cs-NE, q-fin-CP, q-fin.CP  
Keywords: Financial  
[Paper Link](http://arxiv.org/abs/2307.14322v1)

* * *

**ABSTRACT**  
Financial contagion has been widely recognized as a fundamental risk to the financial system. Particularly potent is price-mediated contagion, wherein forced liquidations by firms depress asset prices and propagate financial stress, enabling crises to proliferate across a broad spectrum of seemingly unrelated entities. Price impacts are currently modeled via exogenous inverse demand functions. However, in real-world scenarios, only the initial shocks and the final equilibrium asset prices are typically observable, leaving actual asset liquidations largely obscured. This missing data presents significant limitations to calibrating the existing models. To address these challenges, we introduce a novel dual neural network structure that operates in two sequential stages: the first neural network maps initial shocks to predicted asset liquidations, and the second network utilizes these liquidations to derive resultant equilibrium prices. This data-driven approach can capture both linear and non-linear forms without pre-specifying an analytical structure; furthermore, it functions effectively even in the absence of observable liquidation data. Experiments with simulated datasets demonstrate that our model can accurately predict equilibrium asset prices based solely on initial shocks, while revealing a strong alignment between predicted and true liquidations. Our explainable framework contributes to the understanding and modeling of price-mediated contagion and provides valuable insights for financial authorities to construct effective stress tests and regulatory policies.

{{}}

## eess.SY (1)

### (76/94) A Constraint Enforcement Deep Reinforcement Learning Framework for Optimal Energy Storage Systems Dispatch (Shengren Hou et al., 2023)

{{}}

Shengren Hou, Edgar Mauricio Salazar Duque, Peter Palensky, Pedro P. Vergara. (2023)  
**A Constraint Enforcement Deep Reinforcement Learning Framework for Optimal Energy Storage Systems Dispatch**

* * *

Primary Category: eess.SY  
Categories: cs-LG, cs-SY, eess-SY, eess.SY, math-OC  
Keywords: Reinforcement Learning  
[Paper Link](http://arxiv.org/abs/2307.14304v1)

* * *

**ABSTRACT**  
The optimal dispatch of energy storage systems (ESSs) presents formidable challenges due to the uncertainty introduced by fluctuations in dynamic prices, demand consumption, and renewable-based energy generation. By exploiting the generalization capabilities of deep neural networks (DNNs), deep reinforcement learning (DRL) algorithms can learn good-quality control models that adaptively respond to distribution networks' stochastic nature. However, current DRL algorithms lack the capabilities to enforce operational constraints strictly, often even providing unfeasible control actions. To address this issue, we propose a DRL framework that effectively handles continuous action spaces while strictly enforcing the environments and action space operational constraints during online operation. Firstly, the proposed framework trains an action-value function modeled using DNNs. Subsequently, this action-value function is formulated as a mixed-integer programming (MIP) formulation enabling the consideration of the environment's operational constraints. Comprehensive numerical simulations show the superior performance of the proposed MIP-DRL framework, effectively enforcing all constraints while delivering high-quality dispatch decisions when compared with state-of-the-art DRL algorithms and the optimal solution obtained with a perfect forecast of the stochastic variables.

{{}}

## cs.AI (3)

### (77/94) General Purpose Artificial Intelligence Systems (GPAIS): Properties, Definition, Taxonomy, Open Challenges and Implications (Isaac Triguero et al., 2023)

{{}}

Isaac Triguero, Daniel Molina, Javier Poyatos, Javier Del Ser, Francisco Herrera. (2023)  
**General Purpose Artificial Intelligence Systems (GPAIS): Properties, Definition, Taxonomy, Open Challenges and Implications**

* * *

Primary Category: cs.AI  
Categories: cs-AI, cs-LG, cs.AI  
Keywords: AI  
[Paper Link](http://arxiv.org/abs/2307.14283v1)

* * *

**ABSTRACT**  
Most applications of Artificial Intelligence (AI) are designed for a confined and specific task. However, there are many scenarios that call for a more general AI, capable of solving a wide array of tasks without being specifically designed for them. The term General-Purpose Artificial Intelligence Systems (GPAIS) has been defined to refer to these AI systems. To date, the possibility of an Artificial General Intelligence, powerful enough to perform any intellectual task as if it were human, or even improve it, has remained an aspiration, fiction, and considered a risk for our society. Whilst we might still be far from achieving that, GPAIS is a reality and sitting at the forefront of AI research. This work discusses existing definitions for GPAIS and proposes a new definition that allows for a gradual differentiation among types of GPAIS according to their properties and limitations. We distinguish between closed-world and open-world GPAIS, characterising their degree of autonomy and ability based on several factors such as adaptation to new tasks, competence in domains not intentionally trained for, ability to learn from few data, or proactive acknowledgment of their own limitations. We then propose a taxonomy of approaches to realise GPAIS, describing research trends such as the use of AI techniques to improve another AI or foundation models. As a prime example, we delve into generative AI, aligning them with the terms and concepts presented in the taxonomy. Through the proposed definition and taxonomy, our aim is to facilitate research collaboration across different areas that are tackling general-purpose tasks, as they share many common aspects. Finally, we discuss the current state of GPAIS, its challenges and prospects, implications for our society, and the need for responsible and trustworthy AI systems and regulation, with the goal of providing a holistic view of GPAIS.

{{}}

### (78/94) A New Perspective on Evaluation Methods for Explainable Artificial Intelligence (XAI) (Timo Speith et al., 2023)

{{}}

Timo Speith, Markus Langer. (2023)  
**A New Perspective on Evaluation Methods for Explainable Artificial Intelligence (XAI)**

* * *

Primary Category: cs.AI  
Categories: cs-AI, cs-SE, cs.AI  
Keywords: AI  
[Paper Link](http://arxiv.org/abs/2307.14246v1)

* * *

**ABSTRACT**  
Within the field of Requirements Engineering (RE), the increasing significance of Explainable Artificial Intelligence (XAI) in aligning AI-supported systems with user needs, societal expectations, and regulatory standards has garnered recognition. In general, explainability has emerged as an important non-functional requirement that impacts system quality. However, the supposed trade-off between explainability and performance challenges the presumed positive influence of explainability. If meeting the requirement of explainability entails a reduction in system performance, then careful consideration must be given to which of these quality aspects takes precedence and how to compromise between them. In this paper, we critically examine the alleged trade-off. We argue that it is best approached in a nuanced way that incorporates resource availability, domain characteristics, and considerations of risk. By providing a foundation for future research and best practices, this work aims to advance the field of RE for AI.

{{}}

### (79/94) Revisiting the Performance-Explainability Trade-Off in Explainable Artificial Intelligence (XAI) (Barnaby Crook et al., 2023)

{{}}

Barnaby Crook, Maximilian Schlüter, Timo Speith. (2023)  
**Revisiting the Performance-Explainability Trade-Off in Explainable Artificial Intelligence (XAI)**

* * *

Primary Category: cs.AI  
Categories: cs-AI, cs-SE, cs.AI  
Keywords: AI  
[Paper Link](http://arxiv.org/abs/2307.14239v1)

* * *

**ABSTRACT**  
Within the field of Requirements Engineering (RE), the increasing significance of Explainable Artificial Intelligence (XAI) in aligning AI-supported systems with user needs, societal expectations, and regulatory standards has garnered recognition. In general, explainability has emerged as an important non-functional requirement that impacts system quality. However, the supposed trade-off between explainability and performance challenges the presumed positive influence of explainability. If meeting the requirement of explainability entails a reduction in system performance, then careful consideration must be given to which of these quality aspects takes precedence and how to compromise between them. In this paper, we critically examine the alleged trade-off. We argue that it is best approached in a nuanced way that incorporates resource availability, domain characteristics, and considerations of risk. By providing a foundation for future research and best practices, this work aims to advance the field of RE for AI.

{{}}

## cs.NI (1)

### (80/94) A Clustering Strategy for Enhanced FL-Based Intrusion Detection in IoT Networks (Jacopo Talpini et al., 2023)

{{}}

Jacopo Talpini, Fabio Sartori, Marco Savi. (2023)  
**A Clustering Strategy for Enhanced FL-Based Intrusion Detection in IoT Networks**

* * *

Primary Category: cs.NI  
Categories: cs-NI, cs.NI  
Keywords: Intrusion Detection  
[Paper Link](http://arxiv.org/abs/2307.14268v1)

* * *

**ABSTRACT**  
The Internet of Things (IoT) is growing rapidly and so the need of ensuring protection against cybersecurity attacks to IoT devices. In this scenario, Intrusion Detection Systems (IDSs) play a crucial role and data-driven IDSs based on machine learning (ML) have recently attracted more and more interest by the research community. While conventional ML-based IDSs are based on a centralized architecture where IoT devices share their data with a central server for model training, we propose a novel approach that is based on federated learning (FL). However, conventional FL is ineffective in the considered scenario, due to the high statistical heterogeneity of data collected by IoT devices. To overcome this limitation, we propose a three-tier FL-based architecture where IoT devices are clustered together based on their statistical properties. Clustering decisions are taken by means of a novel entropy-based strategy, which helps improve model training performance. We tested our solution on the CIC-ToN-IoT dataset: our clustering strategy increases intrusion detection performance with respect to a conventional FL approach up to +17% in terms of F1-score, along with a significant reduction of the number of training rounds.

{{}}

## cs.CY (6)

### (81/94) Improving International Climate Policy via Mutually Conditional Binding Commitments (Jobst Heitzig et al., 2023)

{{}}

Jobst Heitzig, Jörg Oechssler, Christoph Pröschel, Niranjana Ragavan, Yat Long Lo. (2023)  
**Improving International Climate Policy via Mutually Conditional Binding Commitments**

* * *

Primary Category: cs.CY  
Categories: cs-AI, cs-CY, cs.CY  
Keywords: AI  
[Paper Link](http://arxiv.org/abs/2307.14267v1)

* * *

**ABSTRACT**  
The Paris Agreement, considered a significant milestone in climate negotiations, has faced challenges in effectively addressing climate change due to the unconditional nature of most Nationally Determined Contributions (NDCs). This has resulted in a prevalence of free-riding behavior among major polluters and a lack of concrete conditionality in NDCs. To address this issue, we propose the implementation of a decentralized, bottom-up approach called the Conditional Commitment Mechanism. This mechanism, inspired by the National Popular Vote Interstate Compact, offers flexibility and incentives for early adopters, aiming to formalize conditional cooperation in international climate policy. In this paper, we provide an overview of the mechanism, its performance in the AI4ClimateCooperation challenge, and discuss potential real-world implementation aspects. Prior knowledge of the climate mitigation collective action problem, basic economic principles, and game theory concepts are assumed.

{{}}

### (82/94) Improving International Climate Policy via Mutually Conditional Binding Commitments (Jobst Heitzig et al., 2023)

{{}}

Jobst Heitzig, Jörg Oechssler, Christoph Pröschel, Niranjana Ragavan, Richie YatLong Lo. (2023)  
**Improving International Climate Policy via Mutually Conditional Binding Commitments**

* * *

Primary Category: cs.CY  
Categories: cs-AI, cs-CY, cs.CY  
Keywords: Reinforcement Learning  
[Paper Link](http://arxiv.org/abs/2307.14266v1)

* * *

**ABSTRACT**  
This paper proposes enhancements to the RICE-N simulation and multi-agent reinforcement learning framework to improve the realism of international climate policy negotiations. Acknowledging the framework's value, we highlight the necessity of significant enhancements to address the diverse array of factors in modeling climate negotiations. Building upon our previous work on the "Conditional Commitments Mechanism" (CCF mechanism) we discuss ways to bridge the gap between simulation and reality. We suggest the inclusion of a recommender or planner agent to enhance coordination, address the Real2Sim gap by incorporating social factors and non-party stakeholder sub-agents, and propose enhancements to the underlying Reinforcement Learning solution algorithm. These proposed improvements aim to advance the evaluation and formulation of negotiation protocols for more effective international climate policy decision-making in Rice-N. However, further experimentation and testing are required to determine the implications and effectiveness of these suggestions.

{{}}

### (83/94) AI4GCC - Team: Below Sea Level: Critiques and Improvements (Bram Renting et al., 2023)

{{}}

Bram Renting, Phillip Wozny, Robert Loftin, Claudia Wieners, Erman Acar. (2023)  
**AI4GCC - Team: Below Sea Level: Critiques and Improvements**

* * *

Primary Category: cs.CY  
Categories: cs-AI, cs-CY, cs.CY  
Keywords: AI  
[Paper Link](http://arxiv.org/abs/2307.13894v1)

* * *

**ABSTRACT**  
We present a critical analysis of the simulation framework RICE-N, an integrated assessment model (IAM) for evaluating the impacts of climate change on the economy. We identify key issues with RICE-N, including action masking and irrelevant actions, and suggest improvements such as utilizing tariff revenue and penalizing overproduction. We also critically engage with features of IAMs in general, namely overly optimistic damage functions and unrealistic abatement cost functions. Our findings contribute to the ongoing efforts to further develop the RICE-N framework in an effort to improve the simulation, making it more useful as an inspiration for policymakers.

{{}}

### (84/94) Dynamic Grouping for Climate Change Negotiation: Facilitating Cooperation and Balancing Interests through Effective Strategies (Yu Qin et al., 2023)

{{}}

Yu Qin, Duo Zhang, Yuren Pang. (2023)  
**Dynamic Grouping for Climate Change Negotiation: Facilitating Cooperation and Balancing Interests through Effective Strategies**

* * *

Primary Category: cs.CY  
Categories: cs-AI, cs-CY, cs.CY  
Keywords: AI  
[Paper Link](http://arxiv.org/abs/2307.13893v1)

* * *

**ABSTRACT**  
In this paper, we propose a dynamic grouping negotiation model for climate mitigation based on real-world business and political negotiation protocols. Within the AI4GCC competition framework, we develop a three-stage process: group formation and updates, intra-group negotiation, and inter-group negotiation. Our model promotes efficient and effective cooperation between various stakeholders to achieve global climate change objectives. By implementing a group-forming method and group updating strategy, we address the complexities and imbalances in multi-region climate negotiations. Intra-group negotiations ensure that all members contribute to mitigation efforts, while inter-group negotiations use the proposal-evaluation framework to set mitigation and savings rates. We demonstrate our negotiation model within the RICE-N framework, illustrating a promising approach for facilitating international cooperation on climate change mitigation.

{{}}

### (85/94) AI4GCC - Team: Below Sea Level: Score and Real World Relevance (Phillip Wozny et al., 2023)

{{}}

Phillip Wozny, Bram Renting, Robert Loftin, Claudia Wieners, Erman Acar. (2023)  
**AI4GCC - Team: Below Sea Level: Score and Real World Relevance**

* * *

Primary Category: cs.CY  
Categories: cs-AI, cs-CY, cs.CY  
Keywords: AI  
[Paper Link](http://arxiv.org/abs/2307.13892v1)

* * *

**ABSTRACT**  
As our submission for track three of the AI for Global Climate Cooperation (AI4GCC) competition, we propose a negotiation protocol for use in the RICE-N climate-economic simulation. Our proposal seeks to address the challenges of carbon leakage through methods inspired by the Carbon Border Adjustment Mechanism (CBAM) and Climate Clubs (CC). We demonstrate the effectiveness of our approach by comparing simulated outcomes to representative concentration pathways (RCP) and shared socioeconomic pathways (SSP). Our protocol results in a temperature rise comparable to RCP 3.4/4.5 and SSP 2. Furthermore, we provide an analysis of our protocol's World Trade Organization compliance, administrative and political feasibility, and ethical concerns. We recognize that our proposal risks hurting the least developing countries, and we suggest specific corrective measures to avoid exacerbating existing inequalities, such as technology sharing and wealth redistribution. Future research should improve the RICE-N tariff mechanism and implement actions allowing for the aforementioned corrective measures.

{{}}

### (86/94) Human Culture: A History Irrelevant and Predictable Experience (Hao Wang, 2023)

{{}}

Hao Wang. (2023)  
**Human Culture: A History Irrelevant and Predictable Experience**

* * *

Primary Category: cs.CY  
Categories: cs-CY, cs.CY  
Keywords: AI  
[Paper Link](http://arxiv.org/abs/2307.13882v1)

* * *

**ABSTRACT**  
Human culture research has witnessed an opportunity of revolution thanks to the big data and social network revolution. Websites such as Douban.com, Goodreads.com, Pandora and IMDB become the new gold mine for cultural researchers. In 2021 and 2022, the author of this paper invented 2 data-free recommender systems for AI cold-start problem. The algorithms can recommend cultural and commercial products to users without reference to users' past preferences. The social implications of the new inventions are human cultural tastes can be predicted very precisely without any information related to human individuals. In this paper, we analyze the AI technologies and its cultural implications together with other AI algorithms. We show that human culture is (mostly) a history irrelevant and predictable experience.

{{}}

## cs.CR (4)

### (87/94) Unveiling Security, Privacy, and Ethical Concerns of ChatGPT (Xiaodong Wu et al., 2023)

{{}}

Xiaodong Wu, Ran Duan, Jianbing Ni. (2023)  
**Unveiling Security, Privacy, and Ethical Concerns of ChatGPT**

* * *

Primary Category: cs.CR  
Categories: cs-AI, cs-CR, cs.CR  
Keywords: AI, ChatGPT, GPT, GPT-4, Security  
[Paper Link](http://arxiv.org/abs/2307.14192v1)

* * *

**ABSTRACT**  
This paper delves into the realm of ChatGPT, an AI-powered chatbot that utilizes topic modeling and reinforcement learning to generate natural responses. Although ChatGPT holds immense promise across various industries, such as customer service, education, mental health treatment, personal productivity, and content creation, it is essential to address its security, privacy, and ethical implications. By exploring the upgrade path from GPT-1 to GPT-4, discussing the model's features, limitations, and potential applications, this study aims to shed light on the potential risks of integrating ChatGPT into our daily lives. Focusing on security, privacy, and ethics issues, we highlight the challenges these concerns pose for widespread adoption. Finally, we analyze the open problems in these areas, calling for concerted efforts to ensure the development of secure and ethically sound large language models.

{{}}

### (88/94) Enhanced Security against Adversarial Examples Using a Random Ensemble of Encrypted Vision Transformer Models (Ryota Iijima et al., 2023)

{{}}

Ryota Iijima, Miki Tanaka, Sayaka Shiota, Hitoshi Kiya. (2023)  
**Enhanced Security against Adversarial Examples Using a Random Ensemble of Encrypted Vision Transformer Models**

* * *

Primary Category: cs.CR  
Categories: cs-CR, cs-CV, cs.CR  
Keywords: Security, Transformer  
[Paper Link](http://arxiv.org/abs/2307.13985v1)

* * *

**ABSTRACT**  
Deep neural networks (DNNs) are well known to be vulnerable to adversarial examples (AEs). In addition, AEs have adversarial transferability, which means AEs generated for a source model can fool another black-box model (target model) with a non-trivial probability. In previous studies, it was confirmed that the vision transformer (ViT) is more robust against the property of adversarial transferability than convolutional neural network (CNN) models such as ConvMixer, and moreover encrypted ViT is more robust than ViT without any encryption. In this article, we propose a random ensemble of encrypted ViT models to achieve much more robust models. In experiments, the proposed scheme is verified to be more robust against not only black-box attacks but also white-box ones than convention methods.

{{}}

### (89/94) Dual-Space Attacks against Random-Walk-based Anomaly Detection (Yuni Lai et al., 2023)

{{}}

Yuni Lai, Marcin Waniek, Yulin Zhu, Liying Li, Jingwen Wu, Tomasz P. Michalak, Talal Rahwan, Kai Zhou. (2023)  
**Dual-Space Attacks against Random-Walk-based Anomaly Detection**

* * *

Primary Category: cs.CR  
Categories: cs-AI, cs-CR, cs.CR  
Keywords: Anomaly Detection  
[Paper Link](http://arxiv.org/abs/2307.14387v1)

* * *

**ABSTRACT**  
Random Walks-based Anomaly Detection (RWAD) is commonly used to identify anomalous patterns in various applications. An intriguing characteristic of RWAD is that the input graph can either be pre-existing or constructed from raw features. Consequently, there are two potential attack surfaces against RWAD: graph-space attacks and feature-space attacks. In this paper, we explore this vulnerability by designing practical dual-space attacks, investigating the interplay between graph-space and feature-space attacks. To this end, we conduct a thorough complexity analysis, proving that attacking RWAD is NP-hard. Then, we proceed to formulate the graph-space attack as a bi-level optimization problem and propose two strategies to solve it: alternative iteration (alterI-attack) or utilizing the closed-form solution of the random walk model (cf-attack). Finally, we utilize the results from the graph-space attacks as guidance to design more powerful feature-space attacks (i.e., graph-guided attacks). Comprehensive experiments demonstrate that our proposed attacks are effective in enabling the target nodes from RWAD with a limited attack budget. In addition, we conduct transfer attack experiments in a black-box setting, which show that our feature attack significantly decreases the anomaly scores of target nodes. Our study opens the door to studying the dual-space attack against graph anomaly detection in which the graph space relies on the feature space.

{{}}

### (90/94) Security Weaknesses in IoT Management Platforms (Bhaskar Tejaswi et al., 2023)

{{}}

Bhaskar Tejaswi, Mohammad Mannan, Amr Youssef. (2023)  
**Security Weaknesses in IoT Management Platforms**

* * *

Primary Category: cs.CR  
Categories: cs-CR, cs.CR  
Keywords: Security  
[Paper Link](http://arxiv.org/abs/2307.13952v1)

* * *

**ABSTRACT**  
A diverse set of Internet of Things (IoT) devices are becoming an integrated part of daily lives, and playing an increasingly vital role in various industry, enterprise and agricultural settings. The current IoT ecosystem relies on several IoT management platforms to manage and operate a large number of IoT devices, their data, and their connectivity. Considering their key role, these platforms must be properly secured against cyber attacks. In this work, we first explore the core operations/features of leading platforms to design a framework to perform a systematic security evaluation of these platforms. Subsequently, we use our framework to analyze a representative set of 52 IoT management platforms, including 42 web-hosted and 10 locally-deployable platforms. We discover a number of high severity unauthorized access vulnerabilities in 9/52 evaluated IoT management platforms, which could be abused to perform attacks such as remote IoT SIM deactivation, IoT SIM overcharging and IoT device data forgery. More seriously, we also uncover instances of broken authentication in 13/52 platforms, including complete account takeover on 8/52 platforms along with remote code execution on 2/52 platforms. In effect, 17/52 platforms were affected by vulnerabilities that could lead to platform-wide attacks. Overall, vulnerabilities were uncovered in 33 platforms, out of which 28 platforms responded to our responsible disclosure. We were also assigned 11 CVEs and awarded bounty for our findings.

{{}}

## math.OC (1)

### (91/94) Improving Conflict Analysis in MIP Solvers by Pseudo-Boolean Reasoning (Gioni Mexi et al., 2023)

{{}}

Gioni Mexi, Timo Berthold, Ambros Gleixner, Jakob Nordström. (2023)  
**Improving Conflict Analysis in MIP Solvers by Pseudo-Boolean Reasoning**

* * *

Primary Category: math.OC  
Categories: cs-DM, math-OC, math.OC  
Keywords: Reasoning  
[Paper Link](http://arxiv.org/abs/2307.14166v1)

* * *

**ABSTRACT**  
Conflict analysis has been successfully generalized from Boolean satisfiability (SAT) solving to mixed integer programming (MIP) solvers, but although MIP solvers operate with general linear inequalities, the conflict analysis in MIP has been limited to reasoning with the more restricted class of clausal constraint. This is in contrast to how conflict analysis is performed in so-called pseudo-Boolean solving, where solvers can reason directly with 0-1 integer linear inequalities rather than with clausal constraints extracted from such inequalities. In this work, we investigate how pseudo-Boolean conflict analysis can be integrated in MIP solving, focusing on 0-1 integer linear programs (0-1 ILPs). Phrased in MIP terminology, conflict analysis can be understood as a sequence of linear combinations and cuts. We leverage this perspective to design a new conflict analysis algorithm based on mixed integer rounding (MIR) cuts, which theoretically dominates the state-of-the-art division-based method in pseudo-Boolean solving. We also report results from a first proof-of-concept implementation of different pseudo-Boolean conflict analysis methods in the open-source MIP solver SCIP. When evaluated on a large and diverse set of 0-1 ILP instances from MIPLIB 2017, our new MIR-based conflict analysis outperforms both previous pseudo-Boolean methods and the clause-based method used in MIP. Our conclusion is that pseudo-Boolean conflict analysis in MIP is a promising research direction that merits further study, and that it might also make sense to investigate the use of such conflict analysis to generate stronger no-goods in constraint programming.

{{}}

## cs.IT (2)

### (92/94) Is the Performance of NOMA-aided Integrated Sensing and Multicast-Unicast Communications Improved by IRS? (Yang Gou et al., 2023)

{{}}

Yang Gou, Yinghui Ye, Guangyue Lu, Lu Lv, Rose Qingyang Hu. (2023)  
**Is the Performance of NOMA-aided Integrated Sensing and Multicast-Unicast Communications Improved by IRS?**

* * *

Primary Category: cs.IT  
Categories: cs-IT, cs.IT, eess-SP, math-IT  
Keywords: OCR  
[Paper Link](http://arxiv.org/abs/2307.14050v1)

* * *

**ABSTRACT**  
In this paper, we consider intelligent reflecting surface (IRS) in a non-orthogonal multiple access (NOMA)-aided Integrated Sensing and Multicast-Unicast Communication (ISMUC) system, where the multicast signal is used for sensing and communications while the unicast signal is used only for communications. Our goal is to depict whether the IRS improves the performance of NOMA-ISMUC system or not under the imperfect/perfect successive interference cancellation (SIC) scenario. Towards this end, we formulate a non-convex problem to maximize the unicast rate while ensuring the minimum target illumination power and multicast rate. To settle this problem, we employ the Dinkelbach method to transform this original problem into an equivalent one, which is then solved via alternating optimization algorithm and semidefinite relaxation (SDR) with Sequential Rank-One Constraint Relaxation (SROCR). Based on this, an iterative algorithm is devised to obtain a near-optimal solution. Computer simulations verify the quick convergence of the devised iterative algorithm, and provide insightful results. Compared to NOMA-ISMUC without IRS, IRS-aided NOMA-ISMUC achieves a higher rate with perfect SIC but keeps the almost same rate in the case of imperfect SIC.

{{}}

### (93/94) Reinforcement Learning for Sequential Decoding of Generalized LDPC Codes (Salman Habib et al., 2023)

{{}}

Salman Habib, David G. M. Mitchell. (2023)  
**Reinforcement Learning for Sequential Decoding of Generalized LDPC Codes**

* * *

Primary Category: cs.IT  
Categories: cs-IT, cs.IT, math-IT  
Keywords: Reinforcement Learning  
[Paper Link](http://arxiv.org/abs/2307.13905v1)

* * *

**ABSTRACT**  
In this work, we propose reinforcement learning (RL) for sequential decoding of moderate length generalized low-density parity-check (GLDPC) codes. Here, sequential decoding refers to scheduling all the generalized constraint nodes (GCNs) and single parity-check nodes (SPCNs) of a GLDPC code serially in each iteration. A GLDPC decoding environment is modeled as a finite Markov decision process (MDP) in which the state-space comprises of all possible sequences of hard-decision values of the variables nodes (VNs) connected to the scheduled GCN or SPCN, and the action-space of the MDP consists of all possible actions (GCN and SPCN scheduling). The goal of RL is to determine an optimized scheduling policy, i.e., one that results in a decoded codeword by minimizing the complexity of the belief propagation (BP) decoder. For training, we consider the proportion of correct bits at the output of the GCN or SPCN as a reward once it is scheduled. The expected rewards for scheduling all the GCNs/SPCNs in the code's Tanner graph are earned via BP decoding during the RL phase. The proposed RL-based decoding scheme is shown to significantly outperform the standard BP flooding decoder, as well as a sequential decoder in which the GCNs/SPCNs are scheduled randomly.

{{}}

## cs.DC (1)

### (94/94) Low-Parameter Federated Learning with Large Language Models (Jingang Jiang et al., 2023)

{{}}

Jingang Jiang, Xiangyang Liu, Chenyou Fan. (2023)  
**Low-Parameter Federated Learning with Large Language Models**

* * *

Primary Category: cs.DC  
Categories: cs-DC, cs.DC  
Keywords: Language Model, NLU, Natural Language Understanding  
[Paper Link](http://arxiv.org/abs/2307.13896v1)

* * *

**ABSTRACT**  
We study few-shot Natural Language Understanding (NLU) tasks with Large Language Models (LLMs) in federated learning (FL) scenarios. It is a challenging task due to limited labeled data and communication capacities in FL, especially with mobile devices. Recent studies show LLMs can be prompted to perform few-shot NLU tasks like sentiment analysis and arithmetic reasoning. However, the huge sizes of LLMs result in high computation and communication costs, making classical FL schemes impractical. To address these challenges, we propose Low-Parameter Federated Learning (LP-FL). LP-FL combines few-shot prompt learning from LLMs with efficient communication and federating techniques. Our approach enables federated clients to assign soft labels to unlabeled data using gradually learned knowledge from the global model. Through iterative soft-label assigning, we continually expand the labeled set during the FL process. Additionally, to reduce computation and communication costs, LP-FL utilizes the Low-Rank Adaptation (LoRA) technique for compact learnable parameter construction, efficient local model fine-tuning, and affordable global model federation. LP-FL consistently outperforms Full-Parameter Federated Learning (FP-FL) in sentiment analysis tasks across various FL settings. Its resistance to overfitting allows LP-FL to equal or surpass centralized training in few-shot scenarios.

{{}}

## Footer

[ ](https://github.com "GitHub") (C) 2025 GitHub, Inc. 

### Footer navigation

  * [Terms](https://docs.github.com/site-policy/github-terms/github-terms-of-service)
  * [Privacy](https://docs.github.com/site-policy/privacy-policies/github-privacy-statement)
  * [Security](https://github.com/security)
  * [Status](https://www.githubstatus.com/)
  * [Docs](https://docs.github.com/)
  * [Contact](https://support.github.com?tags=dotcom-footer)
  * Manage cookies 
  * Do not share my personal information 



You can’t perform that action at this time. 


title: Prompt Engineering: Strategies for Optimizing AI-Language Models | Synoptek
scrape_date: 2025-03-17T13:26:50.821258
content_type: markdown
hash: 2e6816b1b385b53f19bc43ec77f70b51f192f87518228b852baa8f8d44f60c74
url:
https://synoptek.com/insights/it-blogs/data-insights/prompt-engineering-strategies-for-optimizing-ai-language-models/

content:
×



  * [Sales: (303) 728-3335](tel:30372833535)
  * [Support: (888) 796-6783](tel:8887966783)
  * [Client Portal](https://support.synoptek.com/csm)
  * [Contact](https://synoptek.com/contact-us/)



[ ![Synoptek Logo](https://synoptekmark.b-cdn.net/wp-content/themes/synoptek/images/imagesv2/synoptek-logo.svg) ](https://synoptek.com/)

[__](/contact-us)

__

__

  * [INDUSTRIES __](https://synoptek.com/industries/)

    *       * #### OUR CORE EXPERTISE

      * * * *

      * [ Architecture, Engineering & Construction](https://synoptek.com/industries/architecture-engineering-and-construction/)
      * * * *

      * [Energy](https://synoptek.com/industries/energy/)
      * * * *

      * [Healthcare](https://synoptek.com/industries/healthcare/)
      * * * *

      * [Manufacturing](https://synoptek.com/industries/manufacturing/)
      * * * *

      * [Transportation & Logistics](https://synoptek.com/industries/transportation-and-logistics/)
    *       * #### We Serve

      * * * *

      * [Financial Services](https://synoptek.com/industries/financial-services/)
      * [Higher Education](https://synoptek.com/industries/higher-education/)
      * [Life Sciences](https://synoptek.com/industries/life-sciences/)
      * [Nonprofit](https://synoptek.com/industries/nonprofit/)
      * [Retail](https://synoptek.com/industries/retail/)
      * [Software](https://synoptek.com/industries/software-product/)

  * [SERVICES __](https://synoptek.com/services/)
    *       * #### [Digital Customer Experience](https://synoptek.com/services/digital-customer-experience/)

Transform digital strategies and improve customer engagement with comprehensive consulting, design, and innovative execution.

      * * * *

      * [Customer Experience Management](https://synoptek.com/services/digital-customer-experience/customer-experience-management/)
      * [App and Product Development](https://synoptek.com/services/digital-customer-experience/application-and-product-development/)
      * [E-commerce](https://synoptek.com/services/digital-customer-experience/e-commerce/)
      * [Digital Marketing](https://synoptek.com/services/digital-customer-experience/digital-marketing/)
      * * * *

    *       * #### [Digital Enterprise](https://synoptek.com/services/digital-enterprise/)

Seamlessly integrate technology across the business to drive efficiency, enhance productivity, and foster growth.

      * * * *

      * [Technology Consulting & Advisory](https://synoptek.com/services/digital-enterprise/technology-consulting-and-advisory/)
      * [Business Applications](https://synoptek.com/services/digital-enterprise/business-applications/)
      * [AI and Data Insights](https://synoptek.com/services/digital-enterprise/ai-and-data-insights/)
      * [Workforce Productivity](https://synoptek.com/services/digital-enterprise/workforce-productivity/)
      * * * *

    *       * #### [Cloud & Agile Infrastructure](https://synoptek.com/services/cloud-and-agile-infrastructure/)

Build an agile and flexible organization to effortlessly scale your operations, enhance performance, and stay ahead.

      * * * *

      * [Cloud Advancement](https://synoptek.com/services/cloud-and-agile-infrastructure/cloud-advancement/)
      * [Cybersecurity](https://synoptek.com/services/cloud-and-agile-infrastructure/cybersecurity/)
      * [Infrastructure Performance](https://synoptek.com/services/cloud-and-agile-infrastructure/infrastructure-performance/)
      * * * *

  * [SynoptekMANAGE](https://synoptek.com/synoptek-manage/)
  * [SynoptekPLATFORM](https://synoptek.com/the-synoptek-platform/)
  * [INSIGHTS __](https://synoptek.com/insights/)
    *       * #### Topic

      * * * *

      * [ Advisory & Strategy](https://synoptek.com/insights-topic/?tab=advisory-strategy)
      * [Business Resilience](https://synoptek.com/insights-topic/?tab=business-resilience)
      * [Cost Optimization](https://synoptek.com/insights-topic/?tab=cost-optimization)
      * [Mergers & Acquisitions](https://synoptek.com/insights-topic/?tab=mergers-acquisitions)
      * [Agile Organizations](https://synoptek.com/insights-topic/?tab=agile-organizations)
      * [Al & Business](https://synoptek.com/insights-topic/?tab=ai-business)
    *       * #### Function

      * * * *

      * [Executive](https://synoptek.com/insights-function/?tab=executive)
      * [Finance](https://synoptek.com/insights-function/?tab=finance)
      * [Technology](https://synoptek.com/insights-function/?tab=technology)
      * [Operations](https://synoptek.com/insights-function/?tab=operations)
      * [Revenue](https://synoptek.com/insights-function/?tab=revenue)
    *       * #### Asset Type

      * * * *

      * [Thought Leadership](https://synoptek.com/insights?tab=leader)
      * [Blogs](https://synoptek.com/insights?tab=it)
      * [White Papers](https://synoptek.com/insights?tab=white)
      * [Case Studies](https://synoptek.com/insights?tab=case)
      * [Datasheets](https://synoptek.com/insights?tab=data)
      * [Videos](https://synoptek.com/insights?tab=videos)
      * [On-demand Webinars](https://synoptek.com/insights?tab=demand)
  * [ABOUT US __](https://synoptek.com/about-us/)
    *       * [Company](https://synoptek.com/about-us/company/)
      * * * *

      * [ Leadership](https://synoptek.com/leadership/?type=Executive%20Leaders)
      * * * *

      * [Careers](https://synoptek.com/career/)
      * * * *

      * [Partners](https://synoptek.com/partners/)
      * * * *

      * [Social Impact](https://synoptek.com/social-impact/)
      * * * *

    *       * [News](https://synoptek.com/about-us/news/)
      * * * *

      * [Events and Webinars](https://synoptek.com/about-us/events-and-webinars/)
      * * * *

      * [Locations](https://synoptek.com/location/irvine-headquarter/)
      * * * *

      * [FAQ](https://synoptek.com/about-us/faq/)
  * [CLIENT PORTAL](https://ww5.autotask.net/clientportal/Login.aspx?ci=7849)
  * __

  * [Contact Us](https://synoptek.com/contact-us/)
  * [Support: (888) 796-6783](tel:8887966783)
  * [Sales: (303) 728-3335](tel:30372833535)



__




#### Blog: Data Insights

# Prompt Engineering: Strategies for Optimizing AI-Language Models

[Home](https://synoptek.com/) / [Insights](https://synoptek.com/insights/) / [Blog](https://synoptek.com/./it-blogs/) / Prompt Engineering: Strategies for Optimizing AI-Language Models

July 20, 2023 - **by Anish Purohit**

[ ![Twitter](https://synoptekmark.b-cdn.net/wp-content/themes/synoptek/images/twitter.svg) ](https://twitter.com/intent/tweet?text=Prompt Engineering: Strategies for Optimizing AI-Language Models https://synoptek.com/insights/it-blogs/data-insights/prompt-engineering-strategies-for-optimizing-ai-language-models/&related=Synoptek) [ ![Facebook](https://synoptekmark.b-cdn.net/wp-content/themes/synoptek/images/facebook.svg) ](https://www.facebook.com/sharer/sharer.php?u=https://synoptek.com/insights/it-blogs/data-insights/prompt-engineering-strategies-for-optimizing-ai-language-models/&t=Prompt Engineering: Strategies for Optimizing AI-Language Models"e&ext=) [ ![Linkedin](https://synoptekmark.b-cdn.net/wp-content/themes/synoptek/images/linkedin.svg) ](https://www.linkedin.com/sharing/share-offsite/?url=https://synoptek.com/insights/it-blogs/data-insights/prompt-engineering-strategies-for-optimizing-ai-language-models/)

The surge in popularity of prominent Open AI language models like ChatGPT, GPT-3, DALL-E, and Google Bard has sparked a significant buzz around the term "prompt engineering" in the market. Although its concept existed even before the emergence of these [large language models](https://synoptek.com/insights/it-blogs/data-insights/artificial-intelligence-tailor-solutions-with-large-language-models/), the interest has grown manifold post-ChatGPT.

This blog aims to delve into the importance of prompt engineering in the context of generative AI models. We will explore strategies that can yield optimal results in a limited number of trials, saving time and enhancing the quality of search outcomes.

## What is Prompt Engineering?

It can be defined as providing explicit instructions or cues to AI models, helping them generate more accurate and relevant results. Prompt engineering is being utilized, either directly or indirectly, even before the term itself gained recognition. For instance, organizations use various strategies to optimize results when utilizing Google search. From using quotation marks to ensuring specific keywords appear on the first or second page.

![What is Prompt Engineering?](https://synoptekmark.b-cdn.net/wp-content/uploads/2023/07/prompt-engineering.webp)

## What Are the 6 Pillars of Prompt Engineering?

A concept in [Artificial Intelligence](https://synoptek.com/business-process-and-software-solutions/data-insights/artificial-intelligence-consulting/), prompt engineering is increasingly being used to develop and optimize language models (LMs) for a wide variety of applications and research topics. Developers can use it to better understand the capabilities (and limitations) of large language models (LLMs). They can also increase the capacity of LLMs on a wide range of common and complex tasks and design robust techniques for AI-based applications. The six pillars of prompt engineering include precision, relevance, optimization, model, performance, and customization. Let’s go into further detail below:

![Pillars of Prompt Engineering](https://synoptekmark.b-cdn.net/wp-content/uploads/2023/07/pillars-of-prompt-engineering.webp)

### 1\. Precision

Open AI language models mimic human behavior; therefore, the precision of results is an important aspect of prompt engineering. Precision ensures that the generated responses are reliable, error-free, and meet the specific requirements of the prompt engineering task.

### 2\. Relevance

Open AI LMs are trained on vast datasets, so whenever the user asks a question, the answer must have coherence. Relevance in prompt engineering ensures that the instructions provided to the AI model align with the intended task or goal at hand.

### 3\. Optimization

Open AI responses should be efficient and fast, especially in scenarios where real-time or near-real-time responses are required. Optimized prompts can help reduce the computational resources needed and improve the inference speed, enabling more efficient deployment of AI systems.

### 4\. Model

The term Model in prompt engineering is a core component responsible for generating responses based on the input prompts. The model learns from data to understand language patterns and context, enabling the AI system to produce accurate and relevant outputs. Prompt engineers ensure the model's performance, adaptability, and continuous improvement through fine-tuning and optimization, leading to effective response generation in AI language models.

### 5\. Performance

Performance is crucial in prompt engineering as it directly impacts the quality and effectiveness of AI language models. A well-performing model generates accurate, relevant, and contextually appropriate responses, enhancing user satisfaction and trust.

### 6\. Customization

Any AI system needs to be customizable and adaptable to specific tasks or domains. Customization ensures that the generated responses are highly relevant, accurate, and aligned with the specific needs and requirements of the prompt engineering application.

## What Are the Top Strategies for Effective Prompt Engineering?

Strategies are crucial in prompt engineering as they provide a systematic approach to achieving desired outcomes and optimizing AI language models. Effective strategies ensure precision, control, and relevance in generating responses, leading to accurate and contextually appropriate outputs. By following well-defined strategies, prompt engineers can save time, enhance performance, and address ethical considerations.

Here are a few steps or strategies that can help in getting the desired results:

![Top Strategies for Effective Prompt Engineering](https://synoptekmark.b-cdn.net/wp-content/uploads/2023/07/strategies-for-effective-prompt-engineering.webp)

### 1\. Define Your Goals

Before you embark on the prompt engineering journey, it is crucial to have a clear understanding of the task. Identify what specific tasks you want the AI model to perform and specify the format and desired results. For example, you can use DALL-E to get image-generated models in ChatGPT to give text-based outputs.

### 2\. Provide Clear and Specific Instructions

After having a blueprint of the output, emphasize providing clear and specific instructions to the model. Discuss all possible techniques for crafting unambiguous prompts that leave no room for misinterpretation. Provide examples of effective instruction design that yields desired model behavior. The emphasis should be on using meaningful keywords with appropriate significance rather than merely the number of words employed.

### 3\. Experiment With Multiple Prompts

Each AI language model application may have distinct requirements. By experimenting with multiple prompts, you can explore different approaches and formats to find the most effective and impactful prompts. These steps help prompt engineers to optimize model performance, improve response quality, and achieve the desired outcomes.

### 4\. Be Cognizant of Bias and Plan Mitigation Steps

Bias awareness and mitigation are essential to ensure fair and ethical AI language model outputs. By actively addressing biases, prompt engineers can strive for equitable and inclusive responses, promoting the ethical use of AI technology and avoiding potential harm or discrimination.

### 5\. Ensure Domain-Specific Prompt Design

Domain-specific prompts are essential in prompt engineering as they provide customized instructions and context that align with a specific field or industry. By incorporating domain-specific terminology and constraints, these prompts enhance the accuracy and relevance of AI model outputs within the designated domain. This approach ensures that the AI models possess specialized knowledge, enabling them to produce more precise and contextually appropriate responses for domain-specific tasks.

### 6\. Enable Iterative Refinement and Error Analysis

Iterative refinement and error analysis paves the way for continuous improvement of AI language models. Through iterative refinement, prompt design can be adjusted based on the analysis of model outputs, leading to enhanced performance and accuracy. Error analysis helps identify common errors, biases, and limitations, allowing prompt engineers to make necessary corrections and improvements to optimize model behavior.

## What Are the Key Industries Where Prompt Engineering Can Be Used?

Prompt engineering is increasingly being used for developing LLMs and augmenting their capabilities with domain knowledge and external tools. Key industries include eCommerce, healthcare, market and advertising, education, and customer service. Here’s how prompt engineering is used across each of these industries:

### 1\. E-commerce

Prompt engineering can be utilized in the e-commerce industry for personalized product recommendations, chatbots for customer support, and content generation for product descriptions or marketing materials.

### 2\. Healthcare

In the healthcare industry, prompt engineering can be applied for patient interaction and support, medical data analysis, and generating personalized treatment plans or medical reports.

### 3\. Marketing and Advertising

Prompt engineering can play a role in generating compelling ad copies, personalized marketing campaigns, sentiment analysis for brand perception, and chatbots for customer engagement.

### 4\. Education

Prompt engineering has applications in education for intelligent tutoring systems, automated essay grading, language learning support, and generating educational content or quizzes.

### 5\. Customer Service

In the realm of customer service, prompt engineering can be employed to enhance experience through chatbots, automated email responses, and personalized support.

## Improve Model Performance With Prompt Engineering

As the interest in open AI language models increases, prompt engineering is helping improve model performance, enhance response quality, and enable better user experiences. But prompting techniques vary depending upon LLM models such as GPT-3, DALL-E, and ChatGPT used.

To drive the best prompt engineering outcomes, engaging with expert engineers who possess in-depth knowledge of the underlying mechanisms of large-scale Open AI models is critical. Skilled resources have an intricate understanding of these models, allowing for efficient troubleshooting and resolution of issues.

* * *

## About the Author

![Anish Purohit](https://synoptekmark.b-cdn.net/wp-content/uploads/2023/05/anish-purohit-author.png)

#### Anish Purohit

##### Data Science Manager

Anish Purohit is a certified Azure Data Science Associate with over 11 years of industry experience. With a strong analytical mindset, Anish excels in architecting, designing, and deploying data products using a combination of statistics and technologies. He is proficient in AL/ML/DL and has extensive knowledge of automation, having built and deployed solutions on Microsoft Azure and AWS and led and mentored teams of data engineers and scientists.

### Related Insights

Blog

[Augmented Analytics in Business Intelligence Tools: A New Era](https://synoptek.com/insights/it-blogs/data-insights/augmented-analytics-in-business-intelligence-tools/)

Case Study

[Data Modernization Boosts Query Execution by 50% for Building Manufacturer](https://synoptek.com/insights/case-studies/data-insights/data-modernization-boosts-query-execution-for-building-manufacturer/)

Case Study

[Copilot Enhances IT Support Efficiency for Oil and Gas Leader](https://synoptek.com/insights/case-studies/data-insights/copilot-enhances-it-support-efficiency-for-oil-and-gas-leader/)

Thought Leadership

[GenAI Trends 2025: 5 Key Developments to Watch out for](https://synoptek.com/insights/thought-leadership/data-insights/genai-trends-key-developments-to-watch-out-for/)

Case Study

[Gen AI Boosts Customer Service Metrics for IT Provider](https://synoptek.com/insights/case-studies/data-insights/gen-ai-boosts-customer-service-metrics-for-it-provider/)

[View more Insights __](https://synoptek.com/insights/)

### Talk to Experts

## Learn More

![Data Science and Artificial Intelligence](https://synoptekmark.b-cdn.net/wp-content/uploads/2021/03/thumb-artificial-Intelligence-consulting.jpg)

####  Artificial Intelligence Services 

[Read More](https://synoptek.com/business-process-and-software-solutions/data-insights/artificial-intelligence-consulting/)

![Business Intelligence Consulting](https://synoptekmark.b-cdn.net/wp-content/uploads/2021/03/thumb-business-intelligence-consulting.jpg)

####  Business Intelligence and Analytics Consulting 

[Read More](https://synoptek.com/business-process-and-software-solutions/data-insights/business-intelligence-consulting/)

![Analytics as a Service \(AaaS\)](https://synoptekmark.b-cdn.net/wp-content/uploads/2021/03/thumb-managed-analytics-and-reporting-services.jpg)

####  Analytics as a Service (AaaS) 

[Read More](https://synoptek.com/offerings/data-insights/analytics-as-a-service/)

  * [INDUSTRIES](https://synoptek.com/industries/)
  * [AEC](https://synoptek.com/industries/architecture-engineering-and-construction/)
  * [Energy](https://synoptek.com/industries/energy/)
  * [Healthcare](https://synoptek.com/industries/healthcare/)
  * [Manufacturing](https://synoptek.com/industries/manufacturing/)
  * [Transportation & Logistics](https://synoptek.com/industries/transportation-and-logistics/)



  * [Financial Services](https://synoptek.com/industries/financial-services/)
  * [Higher Education](https://synoptek.com/industries/higher-education/)
  * [Life Sciences](https://synoptek.com/industries/life-sciences/)
  * [Nonprofit](https://synoptek.com/industries/nonprofit/)
  * [Retail](https://synoptek.com/industries/retail/)
  * [Software](https://synoptek.com/industries/software-product/)



  * [Services](https://synoptek.com/services/)
  * [Digital Customer Experience](https://synoptek.com/services/digital-customer-experience/)
  * [Digital Enterprise](https://synoptek.com/services/digital-enterprise/)
  * [Cloud & Agile Infrastructre](https://synoptek.com/services/cloud-and-agile-infrastructure/)
  * [SynoptekMANAGE](https://synoptek.com/synoptek-manage/)
  * [SynoptekPLATFORM](https://synoptek.com/the-synoptek-platform/)



  * [About Us](https://synoptek.com/about-us/)
  * [Company](https://synoptek.com/about-us/company/)
  * [Leadership](https://synoptek.com/leadership/?type=Executive%20Leaders)
  * [Careers](https://synoptek.com/career/)
  * [Partners](https://synoptek.com/about-us/partners/)
  * [Social Impact](https://synoptek.com/social-impact/)
  * [News](https://synoptek.com/about-us/news/)
  * [Events and Webinars](https://synoptek.com/about-us/events-and-webinars/)
  * [FAQ](https://synoptek.com/about-us/faq/)



  * LOCATIONS
  * [LOCATIONS](https://synoptek.com/attachment/airlines-and-aviation-banner/)
  * [USA](https://synoptek.com/location/irvine-headquarter/)
  * [Canada](https://synoptek.com/location/saint-john/)
  * [India](https://synoptek.com/location/ahmedabad/)
  * [Areas We Serve](https://synoptek.com/locations/areas-we-serve/)
  * [Contact Us](https://synoptek.com/contact-us/)



###### Sign up for our Monthly Newsletter: Tech-Arc

![Synoptek Logo](https://synoptekmark.b-cdn.net/wp-content/themes/synoptek/images/imagesv2/synoptek-logo.svg)

Synoptek delivers accelerated business results through advisory led transformative systems integration and managed services. We partner with organizations worldwide to help them navigate the ever-changing business and technology landscape, build solid foundations for their business, and achieve their business goals.

(C) 2025 Synoptek, LLC. All Rights Reserved.

  * [Privacy Policy](https://synoptek.com/about-us/privacy-policy/)



  * [__](http://www.linkedin.com/company/synoptek)
  * [__](http://www.facebook.com/pages/Synoptek/443683439023289?ref=stream)
  * [__](https://twitter.com/Synoptek)
  * [__](https://www.youtube.com/user/synoptekservice)



×

[__](javascript:)

![](https://rdcdn.com/rt?aid=21968&e=1&img=1) ![zoominfo](https://ws.zoominfo.com/pixel/vePqIOeAwy0zFGDvZjoP)


scrape_date: 2025-03-17T13:26:50.821249
content_type: markdown
hash: 6b63744eb250ecba6df54f79f6fa649f8c26b40cb3f89649c3eb7ba46239ad92
url:
https://github.com/wwd29/arxiv-daily/blob/1a3074665786adbe2140d325f14a577e8297e535/html/user_0/2023-11-17.html

title:
arxiv-daily/html/user_0/2023-11-17.html at 1a3074665786adbe2140d325f14a577e8297e535 · wwd29/arxiv-daily · GitHub

content:
Skip to content

## Navigation Menu

Toggle navigation

[ ](/)

[ Sign in ](/login?return_to=https%3A%2F%2Fgithub.com%2Fwwd29%2Farxiv-daily%2Fblob%2F1a3074665786adbe2140d325f14a577e8297e535%2Fhtml%2Fuser_0%2F2023-11-17.html)

  * Product 

    * [ GitHub Copilot Write better code with AI  ](https://github.com/features/copilot)
    * [ Security Find and fix vulnerabilities  ](https://github.com/features/security)
    * [ Actions Automate any workflow  ](https://github.com/features/actions)
    * [ Codespaces Instant dev environments  ](https://github.com/features/codespaces)
    * [ Issues Plan and track work  ](https://github.com/features/issues)
    * [ Code Review Manage code changes  ](https://github.com/features/code-review)
    * [ Discussions Collaborate outside of code  ](https://github.com/features/discussions)
    * [ Code Search Find more, search less  ](https://github.com/features/code-search)

Explore
    * [ All features ](https://github.com/features)
    * [ Documentation  ](https://docs.github.com)
    * [ GitHub Skills  ](https://skills.github.com)
    * [ Blog  ](https://github.blog)

  * Solutions 

By company size
    * [ Enterprises ](https://github.com/enterprise)
    * [ Small and medium teams ](https://github.com/team)
    * [ Startups ](https://github.com/enterprise/startups)
    * [ Nonprofits ](/solutions/industry/nonprofits)

By use case
    * [ DevSecOps ](/solutions/use-case/devsecops)
    * [ DevOps ](/solutions/use-case/devops)
    * [ CI/CD ](/solutions/use-case/ci-cd)
    * [ View all use cases ](/solutions/use-case)

By industry
    * [ Healthcare ](/solutions/industry/healthcare)
    * [ Financial services ](/solutions/industry/financial-services)
    * [ Manufacturing ](/solutions/industry/manufacturing)
    * [ Government ](/solutions/industry/government)
    * [ View all industries ](/solutions/industry)

[ View all solutions  ](/solutions)

  * Resources 

Topics
    * [ AI ](/resources/articles/ai)
    * [ DevOps ](/resources/articles/devops)
    * [ Security ](/resources/articles/security)
    * [ Software Development ](/resources/articles/software-development)
    * [ View all ](/resources/articles)

Explore
    * [ Learning Pathways  ](https://resources.github.com/learn/pathways)
    * [ Events & Webinars  ](https://resources.github.com)
    * [ Ebooks & Whitepapers ](https://github.com/resources/whitepapers)
    * [ Customer Stories ](https://github.com/customer-stories)
    * [ Partners  ](https://partner.github.com)
    * [ Executive Insights ](https://github.com/solutions/executive-insights)

  * Open Source 

    * [ GitHub Sponsors Fund open source developers  ](/sponsors)

    * [ The ReadME Project GitHub community articles  ](https://github.com/readme)

Repositories
    * [ Topics ](https://github.com/topics)
    * [ Trending ](https://github.com/trending)
    * [ Collections ](https://github.com/collections)

  * Enterprise 

    * [ Enterprise platform AI-powered developer platform  ](/enterprise)

Available add-ons
    * [ Advanced Security Enterprise-grade security features  ](https://github.com/enterprise/advanced-security)
    * [ Copilot for business Enterprise-grade AI features  ](/features/copilot/copilot-business)
    * [ Premium Support Enterprise-grade 24/7 support  ](/premium-support)

  * [Pricing](https://github.com/pricing)



Search or jump to...

# Search code, repositories, users, issues, pull requests...

Search 

Clear




[Search syntax tips](https://docs.github.com/search-github/github-code-search/understanding-github-code-search-syntax)

#  Provide feedback 

We read every piece of feedback, and take your input very seriously.

Include my email address so I can be contacted

Cancel  Submit feedback 

#  Saved searches 

## Use saved searches to filter your results more quickly

Name

Query

To see all available qualifiers, see our [documentation](https://docs.github.com/search-github/github-code-search/understanding-github-code-search-syntax). 

Cancel  Create saved search 

[ Sign in ](/login?return_to=https%3A%2F%2Fgithub.com%2Fwwd29%2Farxiv-daily%2Fblob%2F1a3074665786adbe2140d325f14a577e8297e535%2Fhtml%2Fuser_0%2F2023-11-17.html)

[ Sign up ](/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fblob%2Fshow&source=header-repo&source_repo=wwd29%2Farxiv-daily) Reseting focus

You signed in with another tab or window. [Reload]() to refresh your session. You signed out in another tab or window. [Reload]() to refresh your session. You switched accounts on another tab or window. [Reload]() to refresh your session. Dismiss alert

{{ message }}

[ wwd29 ](/wwd29) / **[arxiv-daily](/wwd29/arxiv-daily) ** Public

  * [ Notifications ](/login?return_to=%2Fwwd29%2Farxiv-daily) You must be signed in to change notification settings
  * [ Fork 4 ](/login?return_to=%2Fwwd29%2Farxiv-daily)
  * [ Star  15 ](/login?return_to=%2Fwwd29%2Farxiv-daily)




  * [ Code ](/wwd29/arxiv-daily)
  * [ Issues 0 ](/wwd29/arxiv-daily/issues)
  * [ Pull requests 0 ](/wwd29/arxiv-daily/pulls)
  * [ Actions ](/wwd29/arxiv-daily/actions)
  * [ Projects 0 ](/wwd29/arxiv-daily/projects)
  * [ Security ](/wwd29/arxiv-daily/security)
  * [ Insights ](/wwd29/arxiv-daily/pulse)



Additional navigation options

  * [ Code  ](/wwd29/arxiv-daily)
  * [ Issues  ](/wwd29/arxiv-daily/issues)
  * [ Pull requests  ](/wwd29/arxiv-daily/pulls)
  * [ Actions  ](/wwd29/arxiv-daily/actions)
  * [ Projects  ](/wwd29/arxiv-daily/projects)
  * [ Security  ](/wwd29/arxiv-daily/security)
  * [ Insights  ](/wwd29/arxiv-daily/pulse)



## Files

1a30746

## Breadcrumbs

  1. [arxiv-daily](/wwd29/arxiv-daily/tree/1a3074665786adbe2140d325f14a577e8297e535)
  2. /[html](/wwd29/arxiv-daily/tree/1a3074665786adbe2140d325f14a577e8297e535/html)
  3. /[user_0](/wwd29/arxiv-daily/tree/1a3074665786adbe2140d325f14a577e8297e535/html/user_0)



/

# 2023-11-17.html

Copy path

Blame

Blame

## Latest commit

## History

[History](/wwd29/arxiv-daily/commits/1a3074665786adbe2140d325f14a577e8297e535/html/user_0/2023-11-17.html)

[](/wwd29/arxiv-daily/commits/1a3074665786adbe2140d325f14a577e8297e535/html/user_0/2023-11-17.html)

3714 lines (3567 loc) · 250 KB

1a30746

## Breadcrumbs

  1. [arxiv-daily](/wwd29/arxiv-daily/tree/1a3074665786adbe2140d325f14a577e8297e535)
  2. /[html](/wwd29/arxiv-daily/tree/1a3074665786adbe2140d325f14a577e8297e535/html)
  3. /[user_0](/wwd29/arxiv-daily/tree/1a3074665786adbe2140d325f14a577e8297e535/html/user_0)



/

# 2023-11-17.html

Top

## File metadata and controls

  * Code

  * Blame




3714 lines (3567 loc) · 250 KB

[Raw](https://github.com/wwd29/arxiv-daily/raw/1a3074665786adbe2140d325f14a577e8297e535/html/user_0/2023-11-17.html)

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465

466

467

468

469

470

471

472

473

474

475

476

477

478

479

480

481

482

483

484

485

486

487

488

489

490

491

492

493

494

495

496

497

498

499

500

501

502

503

504

505

506

507

508

509

510

511

512

513

514

515

516

517

518

519

520

521

522

523

524

525

526

527

528

529

530

531

532

533

534

535

536

537

538

539

540

541

542

543

544

545

546

547

548

549

550

551

552

553

554

555

556

557

558

559

560

561

562

563

564

565

566

567

568

569

570

571

572

573

574

575

576

577

578

579

580

581

582

583

584

585

586

587

588

589

590

591

592

593

594

595

596

597

598

599

600

601

602

603

604

605

606

607

608

609

610

611

612

613

614

615

616

617

618

619

620

621

622

623

624

625

626

627

628

629

630

631

632

633

634

635

636

637

638

639

640

641

642

643

644

645

646

647

648

649

650

651

652

653

654

655

656

657

658

659

660

661

662

663

664

665

666

667

668

669

670

671

672

673

674

675

676

677

678

679

680

681

682

683

684

685

686

687

688

689

690

691

692

693

694

695

696

697

698

699

700

701

702

703

704

705

706

707

708

709

710

711

712

713

714

715

716

717

718

719

720

721

722

723

724

725

726

727

728

729

730

731

732

733

734

735

736

737

738

739

740

741

742

743

744

745

746

747

748

749

750

751

752

753

754

755

756

757

758

759

760

761

762

763

764

765

766

767

768

769

770

771

772

773

774

775

776

777

778

779

780

781

782

783

784

785

786

787

788

789

790

791

792

793

794

795

796

797

798

799

800

801

802

803

804

805

806

807

808

809

810

811

812

813

814

815

816

817

818

819

820

821

822

823

824

825

826

827

828

829

830

831

832

833

834

835

836

837

838

839

840

841

842

843

844

845

846

847

848

849

850

851

852

853

854

855

856

857

858

859

860

861

862

863

864

865

866

867

868

869

870

871

872

873

874

875

876

877

878

879

880

881

882

883

884

885

886

887

888

889

890

891

892

893

894

895

896

897

898

899

900

901

902

903

904

905

906

907

908

909

910

911

912

913

914

915

916

917

918

919

920

921

922

923

924

925

926

927

928

929

930

931

932

933

934

935

936

937

938

939

940

941

942

943

944

945

946

947

948

949

950

951

952

953

954

955

956

957

958

959

960

961

962

963

964

965

966

967

968

969

970

971

972

973

974

975

976

977

978

979

980

981

982

983

984

985

986

987

988

989

990

991

992

993

994

995

996

997

998

999

1000

<link rel="stylesheet" href="../../css/markdown.css" />

<article class="markdown-body">

<h2>secure</h2>

<h3>Title: MirrorNet: A TEE-Friendly Framework for Secure On-device DNN Inference. (arXiv:2311.09489v1 [cs.CR])</h3>

<ul>

<li>Paper URL: <a href="http://arxiv.org/abs/2311.09489">http://arxiv.org/abs/2311.09489</a></li>

<li>Code URL: null</li>

<li>Copy Paste: <code><input type="checkbox">[[2311.09489]] MirrorNet: A TEE-Friendly Framework for Secure On-device DNN Inference(http://arxiv.org/abs/2311.09489)</code></li>

<li>Summary: <p>Deep neural network (DNN) models have become prevalent in edge devices for

real-time inference. However, they are vulnerable to model extraction attacks

and require protection. Existing defense approaches either fail to fully

safeguard model confidentiality or result in significant latency issues. To

overcome these challenges, this paper presents MirrorNet, which leverages

Trusted Execution Environment (TEE) to enable secure on-device DNN inference.

It generates a TEE-friendly implementation for any given DNN model to protect

the model confidentiality, while meeting the stringent computation and storage

constraints of TEE. The framework consists of two key components: the backbone

model (BackboneNet), which is stored in the normal world but achieves lower

inference accuracy, and the Companion Partial Monitor (CPM), a lightweight

mirrored branch stored in the secure world, preserving model confidentiality.

During inference, the CPM monitors the intermediate results from the

BackboneNet and rectifies the classification output to achieve higher accuracy.

To enhance flexibility, MirrorNet incorporates two modules: the CPM Strategy

Generator, which generates various protection strategies, and the Performance

Emulator, which estimates the performance of each strategy and selects the most

optimal one. Extensive experiments demonstrate the effectiveness of MirrorNet

in providing security guarantees while maintaining low computation latency,

making MirrorNet a practical and promising solution for secure on-device DNN

inference. For the evaluation, MirrorNet can achieve a 18.6% accuracy gap

between authenticated and illegal use, while only introducing 0.99% hardware

overhead.

</p></li>

</ul>

<h3>Title: LightEMU: Hardware Assisted Fuzzing of Trusted Applications. (arXiv:2311.09532v1 [cs.CR])</h3>

<ul>

<li>Paper URL: <a href="http://arxiv.org/abs/2311.09532">http://arxiv.org/abs/2311.09532</a></li>

<li>Code URL: null</li>

<li>Copy Paste: <code><input type="checkbox">[[2311.09532]] LightEMU: Hardware Assisted Fuzzing of Trusted Applications(http://arxiv.org/abs/2311.09532)</code></li>

<li>Summary: <p>Trusted Execution Environments (TEEs) are deployed in many CPU designs

because of the confidentiality and integrity guarantees they provide. ARM

TrustZone is a TEE extensively deployed on smart phones, IoT devices, and

notebooks. Specifically, TrustZone is used to separate code execution and data

into two worlds, normal world and secure world. However, this separation

inherently prevents traditional fuzzing approaches which rely upon

coverage-guided feedback and existing fuzzing research is, therefore, extremely

limited. In this paper, we present a native and generic method to perform

efficient and scalable feedback-driven fuzzing on Trusted Applications (TAs)

using ARM CoreSight. We propose LightEMU, a novel fuzzing framework that allows

us to fuzz TAs by decoupling them from relied TEE. We argue that LightEMU is a

promising first-stage approach for rapidly discovering TA vulnerabilities prior

to investing effort in whole system TEE evaluation precisely because the

majority of publicly disclosed TrustZone bugs reside in the TA code itself. We

implement LightEMU and adapt it to Teegris, Trusty, OP-TEE and QSEE and

evaluate 8 real-world TAs while triggering 3 unique crashes and achieving x10

time speedup when fuzzing TAs using the state-of-the-art TrustZone fuzzing

framework.

</p></li>

</ul>

<h3>Title: Scalable and Adaptively Secure Any-Trust Distributed Key Generation and All-hands Checkpointing. (arXiv:2311.09592v1 [cs.CR])</h3>

<ul>

<li>Paper URL: <a href="http://arxiv.org/abs/2311.09592">http://arxiv.org/abs/2311.09592</a></li>

<li>Code URL: null</li>

<li>Copy Paste: <code><input type="checkbox">[[2311.09592]] Scalable and Adaptively Secure Any-Trust Distributed Key Generation and All-hands Checkpointing(http://arxiv.org/abs/2311.09592)</code></li>

<li>Summary: <p>The classical distributed key generation protocols (DKG) are resurging due to

their widespread applications in blockchain. While efforts have been made to

improve DKG communication, practical large scale deployments are still yet to

come, due to various challenges including broadcast channel scalability and

worst-case complaint phase. In this paper, we propose a practical DKG for

DL-based cryptosystems, with only (quasi-)linear computation/communication cost

per participant, with the help of a public ledger, and beacon; Notably, our DKG

only incurs constant-size blockchain storage cost for broadcast, even in the

face of worst-case complaints. Moreover, our protocol satisfies adaptive

security. The key to our improvements lies in delegating the most costly

operations to an Any-Trust group. This group is randomly sampled and consists

of a small number of individuals. The population only trusts that at least one

member in the group is honest, without knowing which one. Additionally, we

introduce an extended broadcast channel based on a blockchain and data

dispersal network (such as IPFS), enabling reliable broadcasting of

arbitrary-size messages at the cost of constant-size blockchain storage, which

may be of independent interest.

</p>

<p>Our DKG leads to a fully practical instantiation of Filecoin's checkpointing

mechanism, in which all validators of a Proof-of-Stake (PoS) blockcahin

periodically run DKG and threshold signing to create checkpoints on Bitcoin,

thereby enhancing the security of the PoS chain. In comparison with another

checkpointing approach of Babylon (Oakland, 2023), ours enjoys a significally

smaller monetary cost of Bitcoin transaction fees. For a PoS chain with

$2^{12}$ validators, our cost is merely 0.6\% of that incurred by Babylon's

approach.

</p></li>

</ul>

<h2>security</h2>

<h3>Title: HAL 9000: Skynet's Risk Manager. (arXiv:2311.09449v1 [cs.CR])</h3>

<ul>

<li>Paper URL: <a href="http://arxiv.org/abs/2311.09449">http://arxiv.org/abs/2311.09449</a></li>

<li>Code URL: null</li>

<li>Copy Paste: <code><input type="checkbox">[[2311.09449]] HAL 9000: Skynet's Risk Manager(http://arxiv.org/abs/2311.09449)</code></li>

<li>Summary: <p>Intrusion Tolerant Systems (ITSs) are a necessary component for

cyber-services/infrastructures. Additionally, as cyberattacks follow a

multi-domain attack surface, a similar defensive approach should be applied,

namely, the use of an evolving multi-disciplinary solution that combines ITS,

cybersecurity and Artificial Intelligence (AI). With the increased popularity

of AI solutions, due to Big Data use-case scenarios and decision support and

automation scenarios, new opportunities to apply Machine Learning (ML)

algorithms have emerged, namely ITS empowerment. Using ML algorithms, an ITS

can augment its intrusion tolerance capability, by learning from previous

attacks and from known vulnerabilities. As such, this work's contribution is

twofold: (1) an ITS architecture (Skynet) based on the state-of-the-art and

incorporates new components to increase its intrusion tolerance capability and

its adaptability to new adversaries; (2) an improved Risk Manager design that

leverages AI to improve ITSs by automatically assessing OS risks to intrusions,

and advise with safer configurations. One of the reasons that intrusions are

successful is due to bad configurations or slow adaptability to new threats.

This can be caused by the dependency that systems have for human intervention.

One of the characteristics in Skynet and HAL 9000 design is the removal of

human intervention. Being fully automatized lowers the chance of successful

intrusions caused by human error. Our experiments using Skynet, shows that HAL

is able to choose 15% safer configurations than the state-of-the-art risk

manager.

</p></li>

</ul>

<h3>Title: Market Research on IIoT Standard Compliance Monitoring Providers and deriving Attributes for IIoT Compliance Monitoring. (arXiv:2311.09991v1 [cs.CR])</h3>

<ul>

<li>Paper URL: <a href="http://arxiv.org/abs/2311.09991">http://arxiv.org/abs/2311.09991</a></li>

<li>Code URL: null</li>

<li>Copy Paste: <code><input type="checkbox">[[2311.09991]] Market Research on IIoT Standard Compliance Monitoring Providers and deriving Attributes for IIoT Compliance Monitoring(http://arxiv.org/abs/2311.09991)</code></li>

<li>Summary: <p>Adapting security architectures to common standards like IEC 62443 or ISO

27000 in the Industrial Internet of Things (IIoT) involves complex processes

and compliance reports. Automatic monitoring of compliance status would enhance

this process. Despite limited research, practical applications exist. This

paper conducts a market study on providers implementing IEC 62443 in IIoT,

aiming to formulate a catalog of monitorable attributes aligned with the

standard. The study reveals challenges, such as a lack of formal separation in

security architectures, limiting visibility. Despite these challenges,

practical implementations share commonalities, providing insights into viable

monitoring properties. The research serves as a crucial entry point into

developing a comprehensive catalog of monitorable attributes for IEC 62443

standards in IIoT.

</p>

<p>Aligned with the IEC 62443 SR catalog of document 3-3, monitorable attributes

are derived based on current research about IIoT security and Expert Knowledge.

The provided tables serve as an exemplary extract, not exhaustive, defining

three types of attributes based on their origin of creation.

</p></li>

</ul>

<h3>Title: Towards more Practical Threat Models in Artificial Intelligence Security. (arXiv:2311.09994v1 [cs.CR])</h3>

<ul>

<li>Paper URL: <a href="http://arxiv.org/abs/2311.09994">http://arxiv.org/abs/2311.09994</a></li>

<li>Code URL: null</li>

<li>Copy Paste: <code><input type="checkbox">[[2311.09994]] Towards more Practical Threat Models in Artificial Intelligence Security(http://arxiv.org/abs/2311.09994)</code></li>

<li>Summary: <p>Recent works have identified a gap between research and practice in

artificial intelligence security: threats studied in academia do not always

reflect the practical use and security risks of AI. For example, while models

are often studied in isolation, they form part of larger ML pipelines in

practice. Recent works also brought forward that adversarial manipulations

introduced by academic attacks are impractical. We take a first step towards

describing the full extent of this disparity. To this end, we revisit the

threat models of the six most studied attacks in AI security research and match

them to AI usage in practice via a survey with \textbf{271} industrial

practitioners. On the one hand, we find that all existing threat models are

indeed applicable. On the other hand, there are significant mismatches:

research is often too generous with the attacker, assuming access to

information not frequently available in real-world settings. Our paper is thus

a call for action to study more practical threat models in artificial

intelligence security.

</p></li>

</ul>

<h2>privacy</h2>

<h3>Title: Privacy Threats in Stable Diffusion Models. (arXiv:2311.09355v1 [cs.CV])</h3>

<ul>

<li>Paper URL: <a href="http://arxiv.org/abs/2311.09355">http://arxiv.org/abs/2311.09355</a></li>

<li>Code URL: null</li>

<li>Copy Paste: <code><input type="checkbox">[[2311.09355]] Privacy Threats in Stable Diffusion Models(http://arxiv.org/abs/2311.09355)</code></li>

<li>Summary: <p>This paper introduces a novel approach to membership inference attacks (MIA)

targeting stable diffusion computer vision models, specifically focusing on the

highly sophisticated Stable Diffusion V2 by StabilityAI. MIAs aim to extract

sensitive information about a model's training data, posing significant privacy

concerns. Despite its advancements in image synthesis, our research reveals

privacy vulnerabilities in the stable diffusion models' outputs. Exploiting

this information, we devise a black-box MIA that only needs to query the victim

model repeatedly. Our methodology involves observing the output of a stable

diffusion model at different generative epochs and training a classification

model to distinguish when a series of intermediates originated from a training

sample or not. We propose numerous ways to measure the membership features and

discuss what works best. The attack's efficacy is assessed using the ROC AUC

method, demonstrating a 60\% success rate in inferring membership information.

This paper contributes to the growing body of research on privacy and security

in machine learning, highlighting the need for robust defenses against MIAs.

Our findings prompt a reevaluation of the privacy implications of stable

diffusion models, urging practitioners and developers to implement enhanced

security measures to safeguard against such attacks.

</p></li>

</ul>

<h3>Title: How Trustworthy are Open-Source LLMs? An Assessment under Malicious Demonstrations Shows their Vulnerabilities. (arXiv:2311.09447v1 [cs.CL])</h3>

<ul>

<li>Paper URL: <a href="http://arxiv.org/abs/2311.09447">http://arxiv.org/abs/2311.09447</a></li>

<li>Code URL: null</li>

<li>Copy Paste: <code><input type="checkbox">[[2311.09447]] How Trustworthy are Open-Source LLMs? An Assessment under Malicious Demonstrations Shows their Vulnerabilities(http://arxiv.org/abs/2311.09447)</code></li>

<li>Summary: <p>The rapid progress in open-source Large Language Models (LLMs) is

significantly driving AI development forward. However, there is still a limited

understanding of their trustworthiness. Deploying these models at scale without

sufficient trustworthiness can pose significant risks, highlighting the need to

uncover these issues promptly. In this work, we conduct an assessment of

open-source LLMs on trustworthiness, scrutinizing them across eight different

aspects including toxicity, stereotypes, ethics, hallucination, fairness,

sycophancy, privacy, and robustness against adversarial demonstrations. We

propose an enhanced Chain of Utterances-based (CoU) prompting strategy by

incorporating meticulously crafted malicious demonstrations for trustworthiness

attack. Our extensive experiments encompass recent and representative series of

open-source LLMs, including Vicuna, MPT, Falcon, Mistral, and Llama 2. The

empirical outcomes underscore the efficacy of our attack strategy across

diverse aspects. More interestingly, our result analysis reveals that models

with superior performance in general NLP tasks do not always have greater

trustworthiness; in fact, larger models can be more vulnerable to attacks.

Additionally, models that have undergone instruction tuning, focusing on

instruction following, tend to be more susceptible, although fine-tuning LLMs

for safety alignment proves effective in mitigating adversarial trustworthiness

attacks.

</p></li>

</ul>

<h3>Title: Reducing Privacy Risks in Online Self-Disclosures with Language Models. (arXiv:2311.09538v1 [cs.CL])</h3>

<ul>

<li>Paper URL: <a href="http://arxiv.org/abs/2311.09538">http://arxiv.org/abs/2311.09538</a></li>

<li>Code URL: null</li>

<li>Copy Paste: <code><input type="checkbox">[[2311.09538]] Reducing Privacy Risks in Online Self-Disclosures with Language Models(http://arxiv.org/abs/2311.09538)</code></li>

<li>Summary: <p>Self-disclosure, while being common and rewarding in social media

interaction, also poses privacy risks. In this paper, we take the initiative to

protect the user-side privacy associated with online self-disclosure through

identification and abstraction. We develop a taxonomy of 19 self-disclosure

categories, and curate a large corpus consisting of 4.8K annotated disclosure

spans. We then fine-tune a language model for identification, achieving over

75% in Token F$_1$. We further conduct a HCI user study, with 82\% of

participants viewing the model positively, highlighting its real world

applicability. Motivated by the user feedback, we introduce the task of

self-disclosure abstraction. We experiment with both one-span abstraction and

three-span abstraction settings, and explore multiple fine-tuning strategies.

Our best model can generate diverse abstractions that moderately reduce privacy

risks while maintaining high utility according to human evaluation.

</p></li>

</ul>

<h3>Title: Exploring the Privacy-Energy Consumption Tradeoff for Split Federated Learning. (arXiv:2311.09441v1 [cs.LG])</h3>

<ul>

<li>Paper URL: <a href="http://arxiv.org/abs/2311.09441">http://arxiv.org/abs/2311.09441</a></li>

<li>Code URL: null</li>

<li>Copy Paste: <code><input type="checkbox">[[2311.09441]] Exploring the Privacy-Energy Consumption Tradeoff for Split Federated Learning(http://arxiv.org/abs/2311.09441)</code></li>

<li>Summary: <p>Split Federated Learning (SFL) has recently emerged as a promising

distributed learning technology, leveraging the strengths of both federated

learning and split learning. It emphasizes the advantages of rapid convergence

while addressing privacy concerns. As a result, this innovation has received

significant attention from both industry and academia. However, since the model

is split at a specific layer, known as a cut layer, into both client-side and

server-side models for the SFL, the choice of the cut layer in SFL can have a

substantial impact on the energy consumption of clients and their privacy, as

it influences the training burden and the output of the client-side models.

Moreover, the design challenge of determining the cut layer is highly

intricate, primarily due to the inherent heterogeneity in the computing and

networking capabilities of clients. In this article, we provide a comprehensive

overview of the SFL process and conduct a thorough analysis of energy

consumption and privacy. This analysis takes into account the influence of

various system parameters on the cut layer selection strategy. Additionally, we

provide an illustrative example of the cut layer selection, aiming to minimize

the risk of clients from reconstructing the raw data at the server while

sustaining energy consumption within the required energy budget, which involve

trade-offs. Finally, we address open challenges in this field including their

applications to 6G technology. These directions represent promising avenues for

future research and development.

</p></li>

</ul>

<h2>protect</h2>

<h2>defense</h2>

<h2>attack</h2>

<h3>Title: Backdoor Activation Attack: Attack Large Language Models using Activation Steering for Safety-Alignment. (arXiv:2311.09433v1 [cs.CR])</h3>

<ul>

<li>Paper URL: <a href="http://arxiv.org/abs/2311.09433">http://arxiv.org/abs/2311.09433</a></li>

<li>Code URL: null</li>

<li>Copy Paste: <code><input type="checkbox">[[2311.09433]] Backdoor Activation Attack: Attack Large Language Models using Activation Steering for Safety-Alignment(http://arxiv.org/abs/2311.09433)</code></li>

<li>Summary: <p>To ensure AI safety, instruction-tuned Large Language Models (LLMs) are

specifically trained to ensure alignment, which refers to making models behave

in accordance with human intentions. While these models have demonstrated

commendable results on various safety benchmarks, the vulnerability of their

safety alignment has not been extensively studied. This is particularly

troubling given the potential harm that LLMs can inflict. Existing attack

methods on LLMs often rely on poisoned training data or the injection of

malicious prompts. These approaches compromise the stealthiness and

generalizability of the attacks, making them susceptible to detection.

Additionally, these models often demand substantial computational resources for

implementation, making them less practical for real-world applications. In this

work, we introduce a novel attack framework, called Backdoor Activation Attack,

which injects trojan steering vectors into the activation layers of LLMs. These

malicious steering vectors can be triggered at inference time to steer the

models toward attacker-desired behaviors by manipulating their activations. In

particular, the steering vectors are generated by taking the difference between

benign and malicious activations. Then, the most effective steering vector is

selected and added to the forward passes of the LLMs. Our experiment results on

four primary alignment tasks show that our proposed method is highly effective

and adds little or no overhead to attack efficiency. Additionally, we discuss

potential countermeasures against such activation attacks. Our code and data

are available at https://email-haoran-for-link. Warning: this paper contains

content that can be offensive or upsetting.

</p></li>

</ul>

<h2>robust</h2>

<h3>Title: Reading Between the Mud: A Challenging Motorcycle Racer Number Dataset. (arXiv:2311.09256v1 [cs.CV])</h3>

<ul>

<li>Paper URL: <a href="http://arxiv.org/abs/2311.09256">http://arxiv.org/abs/2311.09256</a></li>

<li>Code URL: https://github.com/jacobtyo/swintextspotter</li>

<li>Copy Paste: <code><input type="checkbox">[[2311.09256]] Reading Between the Mud: A Challenging Motorcycle Racer Number Dataset(http://arxiv.org/abs/2311.09256)</code></li>

<li>Summary: <p>This paper introduces the off-road motorcycle Racer number Dataset (RnD), a

new challenging dataset for optical character recognition (OCR) research. RnD

contains 2,411 images from professional motorsports photographers that depict

motorcycle racers in off-road competitions. The images exhibit a wide variety

of factors that make OCR difficult, including mud occlusions, motion blur,

non-standard fonts, glare, complex backgrounds, etc. The dataset has 5,578

manually annotated bounding boxes around visible motorcycle numbers, along with

transcribed digits and letters. Our experiments benchmark leading OCR

algorithms and reveal an end-to-end F1 score of only 0.527 on RnD, even after

fine-tuning. Analysis of performance on different occlusion types shows mud as

the primary challenge, degrading accuracy substantially compared to normal

conditions. But the models struggle with other factors including glare, blur,

shadows, and dust. Analysis exposes substantial room for improvement and

highlights failure cases of existing models. RnD represents a valuable new

benchmark to drive innovation in real-world OCR capabilities. The authors hope

the community will build upon this dataset and baseline experiments to make

progress on the open problem of robustly recognizing text in unconstrained

natural environments. The dataset is available at

https://github.com/JacobTyo/SwinTextSpotter.

</p></li>

</ul>

<h3>Title: NormNet: Scale Normalization for 6D Pose Estimation in Stacked Scenarios. (arXiv:2311.09269v1 [cs.CV])</h3>

<ul>

<li>Paper URL: <a href="http://arxiv.org/abs/2311.09269">http://arxiv.org/abs/2311.09269</a></li>

<li>Code URL: https://github.com/shuttlet/normnet</li>

<li>Copy Paste: <code><input type="checkbox">[[2311.09269]] NormNet: Scale Normalization for 6D Pose Estimation in Stacked Scenarios(http://arxiv.org/abs/2311.09269)</code></li>

<li>Summary: <p>Existing Object Pose Estimation (OPE) methods for stacked scenarios are not

robust to changes in object scale. This paper proposes a new 6DoF OPE network

(NormNet) for different scale objects in stacked scenarios. Specifically, each

object's scale is first learned with point-wise regression. Then, all objects

in the stacked scenario are normalized into the same scale through semantic

segmentation and affine transformation. Finally, they are fed into a shared

pose estimator to recover their 6D poses. In addition, we introduce a new

Sim-to-Real transfer pipeline, combining style transfer and domain

randomization. This improves the NormNet's performance on real data even if we

only train it on synthetic data. Extensive experiments demonstrate that the

proposed method achieves state-of-the-art performance on public benchmarks and

the MultiScale dataset we constructed. The real-world experiments show that our

method can robustly estimate the 6D pose of objects at different scales.

</p></li>

</ul>

<h3>Title: Leveraging Citizen Science for Flood Extent Detection using Machine Learning Benchmark Dataset. (arXiv:2311.09276v1 [cs.CV])</h3>

<ul>

<li>Paper URL: <a href="http://arxiv.org/abs/2311.09276">http://arxiv.org/abs/2311.09276</a></li>

<li>Code URL: null</li>

<li>Copy Paste: <code><input type="checkbox">[[2311.09276]] Leveraging Citizen Science for Flood Extent Detection using Machine Learning Benchmark Dataset(http://arxiv.org/abs/2311.09276)</code></li>

<li>Summary: <p>Accurate detection of inundated water extents during flooding events is

crucial in emergency response decisions and aids in recovery efforts. Satellite

Remote Sensing data provides a global framework for detecting flooding extents.

Specifically, Sentinel-1 C-Band Synthetic Aperture Radar (SAR) imagery has

proven to be useful in detecting water bodies due to low backscatter of water

features in both co-polarized and cross-polarized SAR imagery. However,

increased backscatter can be observed in certain flooded regions such as

presence of infrastructure and trees - rendering simple methods such as pixel

intensity thresholding and time-series differencing inadequate. Machine

Learning techniques has been leveraged to precisely capture flood extents in

flooded areas with bumps in backscatter but needs high amounts of labelled data

to work desirably. Hence, we created a labeled known water body extent and

flooded area extents during known flooding events covering about 36,000 sq.

kilometers of regions within mainland U.S and Bangladesh. Further, We also

leveraged citizen science by open-sourcing the dataset and hosting an open

competition based on the dataset to rapidly prototype flood extent detection

using community generated models. In this paper we present the information

about the dataset, the data processing pipeline, a baseline model and the

details about the competition, along with discussion on winning approaches. We

believe the dataset adds to already existing datasets based on Sentinel-1C SAR

data and leads to more robust modeling of flood extents. We also hope the

results from the competition pushes the research in flood extent detection

further.

</p></li>

</ul>

<h3>Title: Synthetically Enhanced: Unveiling Synthetic Data's Potential in Medical Imaging Research. (arXiv:2311.09402v1 [cs.CV])</h3>

<ul>

<li>Paper URL: <a href="http://arxiv.org/abs/2311.09402">http://arxiv.org/abs/2311.09402</a></li>

<li>Code URL: null</li>

<li>Copy Paste: <code><input type="checkbox">[[2311.09402]] Synthetically Enhanced: Unveiling Synthetic Data's Potential in Medical Imaging Research(http://arxiv.org/abs/2311.09402)</code></li>

<li>Summary: <p>Chest X-rays (CXR) are the most common medical imaging study and are used to

diagnose multiple medical conditions. This study examines the impact of

synthetic data supplementation, using diffusion models, on the performance of

deep learning (DL) classifiers for CXR analysis. We employed three datasets:

CheXpert, MIMIC-CXR, and Emory Chest X-ray, training conditional denoising

diffusion probabilistic models (DDPMs) to generate synthetic frontal

radiographs. Our approach ensured that synthetic images mirrored the

demographic and pathological traits of the original data. Evaluating the

classifiers' performance on internal and external datasets revealed that

synthetic data supplementation enhances model accuracy, particularly in

detecting less prevalent pathologies. Furthermore, models trained on synthetic

data alone approached the performance of those trained on real data. This

suggests that synthetic data can potentially compensate for real data shortages

in training robust DL models. However, despite promising outcomes, the

superiority of real data persists.

</p></li>

</ul>

<h3>Title: On the Quantification of Image Reconstruction Uncertainty without Training Data. (arXiv:2311.09639v1 [cs.CV])</h3>

<ul>

<li>Paper URL: <a href="http://arxiv.org/abs/2311.09639">http://arxiv.org/abs/2311.09639</a></li>

<li>Code URL: null</li>

<li>Copy Paste: <code><input type="checkbox">[[2311.09639]] On the Quantification of Image Reconstruction Uncertainty without Training Data(http://arxiv.org/abs/2311.09639)</code></li>

<li>Summary: <p>Computational imaging plays a pivotal role in determining hidden information

from sparse measurements. A robust inverse solver is crucial to fully

characterize the uncertainty induced by these measurements, as it allows for

the estimation of the complete posterior of unrecoverable targets. This, in

turn, facilitates a probabilistic interpretation of observational data for

decision-making. In this study, we propose a deep variational framework that

leverages a deep generative model to learn an approximate posterior

distribution to effectively quantify image reconstruction uncertainty without

the need for training data. We parameterize the target posterior using a

flow-based model and minimize their Kullback-Leibler (KL) divergence to achieve

accurate uncertainty estimation. To bolster stability, we introduce a robust

flow-based model with bi-directional regularization and enhance expressivity

through gradient boosting. Additionally, we incorporate a space-filling design

to achieve substantial variance reduction on both latent prior space and target

posterior space. We validate our method on several benchmark tasks and two

real-world applications, namely fastMRI and black hole image reconstruction.

Our results indicate that our method provides reliable and high-quality image

reconstruction with robust uncertainty estimation.

</p></li>

</ul>

<h3>Title: Event-based Motion-Robust Accurate Shape Estimation for Mixed Reflectance Scenes. (arXiv:2311.09652v1 [cs.CV])</h3>

<ul>

<li>Paper URL: <a href="http://arxiv.org/abs/2311.09652">http://arxiv.org/abs/2311.09652</a></li>

<li>Code URL: null</li>

<li>Copy Paste: <code><input type="checkbox">[[2311.09652]] Event-based Motion-Robust Accurate Shape Estimation for Mixed Reflectance Scenes(http://arxiv.org/abs/2311.09652)</code></li>

<li>Summary: <p>Event-based structured light systems have recently been introduced as an

exciting alternative to conventional frame-based triangulation systems for the

3D measurements of diffuse surfaces. Important benefits include the fast

capture speed and the high dynamic range provided by the event camera - albeit

at the cost of lower data quality. So far, both low-accuracy event-based as

well as high-accuracy frame-based 3D imaging systems are tailored to a specific

surface type, such as diffuse or specular, and can not be used for a broader

class of object surfaces ("mixed reflectance scenes"). In this paper, we

present a novel event-based structured light system that enables fast 3D

imaging of mixed reflectance scenes with high accuracy. On the captured events,

we use epipolar constraints that intrinsically enable decomposing the measured

reflections into diffuse, two-bounce specular, and other multi-bounce

reflections. The diffuse objects in the scene are reconstructed using

triangulation. Eventually, the reconstructed diffuse scene parts are used as a

"display" to evaluate the specular scene parts via deflectometry. This novel

procedure allows us to use the entire scene as a virtual screen, using only a

scanning laser and an event camera. The resulting system achieves fast and

motion-robust (14Hz) reconstructions of mixed reflectance scenes with &lt; 500

$\mu$m accuracy. Moreover, we introduce a "superfast" capture mode (250Hz) for

the 3D measurement of diffuse scenes.

</p></li>

</ul>

<h3>Title: Robust Contrastive Learning With Theory Guarantee. (arXiv:2311.09671v1 [cs.LG])</h3>

<ul>

<li>Paper URL: <a href="http://arxiv.org/abs/2311.09671">http://arxiv.org/abs/2311.09671</a></li>

<li>Code URL: null</li>

<li>Copy Paste: <code><input type="checkbox">[[2311.09671]] Robust Contrastive Learning With Theory Guarantee(http://arxiv.org/abs/2311.09671)</code></li>

<li>Summary: <p>Contrastive learning (CL) is a self-supervised training paradigm that allows

us to extract meaningful features without any label information. A typical CL

framework is divided into two phases, where it first tries to learn the

features from unlabelled data, and then uses those features to train a linear

classifier with the labeled data. While a fair amount of existing theoretical

works have analyzed how the unsupervised loss in the first phase can support

the supervised loss in the second phase, none has examined the connection

between the unsupervised loss and the robust supervised loss, which can shed

light on how to construct an effective unsupervised loss for the first phase of

CL. To fill this gap, our work develops rigorous theories to dissect and

identify which components in the unsupervised loss can help improve the robust

supervised loss and conduct proper experiments to verify our findings.

</p></li>

</ul>

<h3>Title: Redefining the Laparoscopic Spatial Sense: AI-based Intra- and Postoperative Measurement from Stereoimages. (arXiv:2311.09744v1 [cs.CV])</h3>

<ul>

<li>Paper URL: <a href="http://arxiv.org/abs/2311.09744">http://arxiv.org/abs/2311.09744</a></li>

<li>Code URL: null</li>

<li>Copy Paste: <code><input type="checkbox">[[2311.09744]] Redefining the Laparoscopic Spatial Sense: AI-based Intra- and Postoperative Measurement from Stereoimages(http://arxiv.org/abs/2311.09744)</code></li>

<li>Summary: <p>A significant challenge in image-guided surgery is the accurate measurement

task of relevant structures such as vessel segments, resection margins, or

bowel lengths. While this task is an essential component of many surgeries, it

involves substantial human effort and is prone to inaccuracies. In this paper,

we develop a novel human-AI-based method for laparoscopic measurements

utilizing stereo vision that has been guided by practicing surgeons. Based on a

holistic qualitative requirements analysis, this work proposes a comprehensive

measurement method, which comprises state-of-the-art machine learning

architectures, such as RAFT-Stereo and YOLOv8. The developed method is assessed

in various realistic experimental evaluation environments. Our results outline

the potential of our method achieving high accuracies in distance measurements

with errors below 1 mm. Furthermore, on-surface measurements demonstrate

robustness when applied in challenging environments with textureless regions.

Overall, by addressing the inherent challenges of image-guided surgery, we lay

the foundation for a more robust and accurate solution for intra- and

postoperative measurements, enabling more precise, safe, and efficient surgical

procedures.

</p></li>

</ul>

<h3>Title: Rusty Detection Using Image Processing For Maintenance Of Stations. (arXiv:2311.09849v1 [cs.CV])</h3>

<ul>

<li>Paper URL: <a href="http://arxiv.org/abs/2311.09849">http://arxiv.org/abs/2311.09849</a></li>

<li>Code URL: null</li>

<li>Copy Paste: <code><input type="checkbox">[[2311.09849]] Rusty Detection Using Image Processing For Maintenance Of Stations(http://arxiv.org/abs/2311.09849)</code></li>

<li>Summary: <p>This study addresses the challenge of accurately seg-menting rusted areas on

painted construction surfaces. A method leveraging digital image processing is

explored to calculate the percentage of rust present on painted coatings. The

proposed segmentation approach is based on the HSV color model. To equalize

luminosity and mitigate the influence of illumination, a fundamental model of

single-scale Retinex is applied specifically to the saturation component.

</p>

<p>Subsequently, the image undergoes further processing, involv-ing manual color

filtering. This step is crucial for refining the identification of rusted

regions. To enhance precision and filter out noise, the pixel areas selected

through color filtering are subjected to the DBScan algorithm. This multi-step

process aims to achieve a robust segmentation of rusted areas on painted

construction surfaces, providing a valuable contribution to the field of

corrosion detection and analysis.

</p></li>

</ul>

<h3>Title: Neural machine translation for automated feedback on children's early-stage writing. (arXiv:2311.09389v1 [cs.CL])</h3>

<ul>

<li>Paper URL: <a href="http://arxiv.org/abs/2311.09389">http://arxiv.org/abs/2311.09389</a></li>

<li>Code URL: null</li>

<li>Copy Paste: <code><input type="checkbox">[[2311.09389]] Neural machine translation for automated feedback on children's early-stage writing(http://arxiv.org/abs/2311.09389)</code></li>

<li>Summary: <p>In this work, we address the problem of assessing and constructing feedback

for early-stage writing automatically using machine learning. Early-stage

writing is typically vastly different from conventional writing due to phonetic

spelling and lack of proper grammar, punctuation, spacing etc. Consequently,

early-stage writing is highly non-trivial to analyze using common linguistic

metrics. We propose to use sequence-to-sequence models for "translating"

early-stage writing by students into "conventional" writing, which allows the

translated text to be analyzed using linguistic metrics. Furthermore, we

propose a novel robust likelihood to mitigate the effect of noise in the

dataset. We investigate the proposed methods using a set of numerical

experiments and demonstrate that the conventional text can be predicted with

high accuracy.

</p></li>

</ul>

<h3>Title: To Translate or Not to Translate: A Systematic Investigation of Translation-Based Cross-Lingual Transfer to Low-Resource Languages. (arXiv:2311.09404v1 [cs.CL])</h3>

<ul>

<li>Paper URL: <a href="http://arxiv.org/abs/2311.09404">http://arxiv.org/abs/2311.09404</a></li>

<li>Code URL: null</li>

<li>Copy Paste: <code><input type="checkbox">[[2311.09404]] To Translate or Not to Translate: A Systematic Investigation of Translation-Based Cross-Lingual Transfer to Low-Resource Languages(http://arxiv.org/abs/2311.09404)</code></li>

<li>Summary: <p>Perfect machine translation (MT) would render cross-lingual transfer (XLT) by

means of multilingual language models (LMs) superfluous. Given, on one hand,

the large body of work on improving XLT with multilingual LMs and, on the other

hand, recent advances in massively multilingual MT, in this work, we

systematically evaluate existing and propose new translation-based XLT

approaches for transfer to low-resource languages. We show that all

translation-based approaches dramatically outperform zero-shot XLT with

multilingual LMs, rendering the approach that combines the round-trip

translation of the source-language training data with the translation of the

target-language test instances the most effective. We next show that one can

obtain further empirical gains by adding reliable translations to other

high-resource languages to the training data. Moreover, we propose an effective

translation-based XLT strategy even for languages not supported by the MT

system. Finally, we show that model selection for XLT based on target-language

validation data obtained with MT outperforms model selection based on the

source-language data. We hope that our findings encourage adoption of more

robust translation-based baselines in XLT research.

</p></li>

</ul>

<h3>Title: Clarify When Necessary: Resolving Ambiguity Through Interaction with LMs. (arXiv:2311.09469v1 [cs.CL])</h3>

<ul>

<li>Paper URL: <a href="http://arxiv.org/abs/2311.09469">http://arxiv.org/abs/2311.09469</a></li>

<li>Code URL: null</li>

<li>Copy Paste: <code><input type="checkbox">[[2311.09469]] Clarify When Necessary: Resolving Ambiguity Through Interaction with LMs(http://arxiv.org/abs/2311.09469)</code></li>

<li>Summary: <p>Resolving ambiguities through interaction is a hallmark of natural language,

and modeling this behavior is a core challenge in crafting AI assistants. In

this work, we study such behavior in LMs by proposing a task-agnostic framework

for resolving ambiguity by asking users clarifying questions. Our framework

breaks down this objective into three subtasks: (1) determining when

clarification is needed, (2) determining what clarifying question to ask, and

(3) responding accurately with the new information gathered through

clarification. We evaluate systems across three NLP applications: question

answering, machine translation and natural language inference. For the first

subtask, we present a novel uncertainty estimation approach, intent-sim, that

determines the utility of querying for clarification by estimating the entropy

over user intents. Our method consistently outperforms existing uncertainty

estimation approaches at identifying predictions that will benefit from

clarification. When only allowed to ask for clarification on 10% of examples,

our system is able to double the performance gains over randomly selecting

examples to clarify. Furthermore, we find that intent-sim is robust,

demonstrating improvements across a wide range of NLP tasks and LMs. Together,

our work lays foundation for studying clarifying interactions with LMs.

</p></li>

</ul>

<h3>Title: Show Your Work with Confidence: Confidence Bands for Tuning Curves. (arXiv:2311.09480v1 [cs.CL])</h3>

<ul>

<li>Paper URL: <a href="http://arxiv.org/abs/2311.09480">http://arxiv.org/abs/2311.09480</a></li>

<li>Code URL: https://github.com/nalourie/opda</li>

<li>Copy Paste: <code><input type="checkbox">[[2311.09480]] Show Your Work with Confidence: Confidence Bands for Tuning Curves(http://arxiv.org/abs/2311.09480)</code></li>

<li>Summary: <p>The choice of hyperparameters greatly impacts performance in natural language

processing. Often, it is hard to tell if a method is better than another or

just better tuned. Tuning curves fix this ambiguity by accounting for tuning

effort. Specifically, they plot validation performance as a function of the

number of hyperparameter choices tried so far. While several estimators exist

for these curves, it is common to use point estimates, which we show fail

silently and give contradictory results when given too little data.

</p>

<p>Beyond point estimates, confidence bands are necessary to rigorously

establish the relationship between different approaches. We present the first

method to construct valid confidence bands for tuning curves. The bands are

exact, simultaneous, and distribution-free, thus they provide a robust basis

for comparing methods.

</p>

<p>Empirical analysis shows that while bootstrap confidence bands, which serve

as a baseline, fail to approximate their target confidence, ours achieve it

exactly. We validate our design with ablations, analyze the effect of sample

size, and provide guidance on comparing models with our method. To promote

confident comparisons in future work, we release a library implementing the

method at https://github.com/nalourie/opda .

</p></li>

</ul>

<h3>Title: SegMix: A Simple Structure-Aware Data Augmentation Method. (arXiv:2311.09505v1 [cs.CL])</h3>

<ul>

<li>Paper URL: <a href="http://arxiv.org/abs/2311.09505">http://arxiv.org/abs/2311.09505</a></li>

<li>Code URL: null</li>

<li>Copy Paste: <code><input type="checkbox">[[2311.09505]] SegMix: A Simple Structure-Aware Data Augmentation Method(http://arxiv.org/abs/2311.09505)</code></li>

<li>Summary: <p>Interpolation-based Data Augmentation (DA) methods (Mixup) linearly

interpolate the inputs and labels of two or more training examples. Mixup has

more recently been adapted to the field of Natural Language Processing (NLP),

mainly for sequence labeling tasks. However, such a simple adoption yields

mixed or unstable improvements over the baseline models. We argue that the

direct-adoption methods do not account for structures in NLP tasks. To this

end, we propose SegMix, a collection of interpolation-based DA algorithms that

can adapt to task-specific structures. SegMix poses fewer constraints on data

structures, is robust to various hyperparameter settings, applies to more task

settings, and adds little computational overhead. In the algorithm's core, we

apply interpolation methods on task-specific meaningful segments, in contrast

to applying them on sequences as in prior work. We find SegMix to be a flexible

framework that combines rule-based DA methods with interpolation-based methods,

creating interesting mixtures of DA techniques. We show that SegMix

consistently improves performance over strong baseline models in Named Entity

Recognition (NER) and Relation Extraction (RE) tasks, especially under

data-scarce settings. Furthermore, this method is easy to implement and adds

negligible training overhead.

</p></li>

</ul>

<h3>Title: SCORE: A framework for Self-Contradictory Reasoning Evaluation. (arXiv:2311.09603v1 [cs.CL])</h3>

<ul>

<li>Paper URL: <a href="http://arxiv.org/abs/2311.09603">http://arxiv.org/abs/2311.09603</a></li>

<li>Code URL: null</li>

<li>Copy Paste: <code><input type="checkbox">[[2311.09603]] SCORE: A framework for Self-Contradictory Reasoning Evaluation(http://arxiv.org/abs/2311.09603)</code></li>

<li>Summary: <p>Large language models (LLMs) have demonstrated impressive reasoning ability

in various language-based tasks. Despite many proposed reasoning methods aimed

at enhancing performance in downstream tasks, two fundamental questions

persist: Does reasoning genuinely support predictions, and how reliable is the

quality of reasoning? In this paper, we propose a framework \textsc{SCORE} to

analyze how well LLMs can reason. Specifically, we focus on self-contradictory

reasoning, where reasoning does not support the prediction. We find that LLMs

often contradict themselves when performing reasoning tasks that involve

contextual information and commonsense. The model may miss evidence or use

shortcuts, thereby exhibiting self-contradictory behaviors. We also employ the

Point-of-View (POV) method, which probes models to generate reasoning from

multiple perspectives, as a diagnostic tool for further analysis. We find that

though LLMs may appear to perform well in one-perspective settings, they fail

to stabilize such behavior in multi-perspectives settings. Even for correct

predictions, the reasoning may be messy and incomplete, and LLMs can easily be

led astray from good reasoning. \textsc{SCORE}'s results underscore the lack of

robustness required for trustworthy reasoning and the urgency for further

research to establish best practices for a comprehensive evaluation of

reasoning beyond accuracy-based metrics.

</p></li>

</ul>

<h3>Title: Take One Step at a Time to Know Incremental Utility of Demonstration: An Analysis on Reranking for Few-Shot In-Context Learning. (arXiv:2311.09619v1 [cs.CL])</h3>

<ul>

<li>Paper URL: <a href="http://arxiv.org/abs/2311.09619">http://arxiv.org/abs/2311.09619</a></li>

<li>Code URL: null</li>

<li>Copy Paste: <code><input type="checkbox">[[2311.09619]] Take One Step at a Time to Know Incremental Utility of Demonstration: An Analysis on Reranking for Few-Shot In-Context Learning(http://arxiv.org/abs/2311.09619)</code></li>

<li>Summary: <p>In-Context Learning (ICL) is an emergent capability of Large Language Models

(LLMs). Only a few demonstrations enable LLMs to be used as blackbox for new

tasks. Previous studies have shown that using LLMs' outputs as labels is

effective in training models to select demonstrations. Such a label is expected

to estimate utility of a demonstration in ICL; however, it has not been well

understood how different labeling strategies affect results on target tasks.

This paper presents an analysis on different utility functions by focusing on

LLMs' output probability given ground-truth output, and task-specific reward

given LLMs' prediction. Unlike the previous work, we introduce a novel labeling

method, incremental utility, which estimates how much incremental knowledge is

brought into the LLMs by a demonstration. We conduct experiments with

instruction-tuned LLMs on binary/multi-class classification, segmentation, and

translation across Arabic, English, Finnish, Japanese, and Spanish. Our results

show that (1) the probability is effective when the probability values are

distributed across the whole value range (on the classification tasks), and (2)

the downstream metric is more robust when nuanced reward values are provided

with long outputs (on the segmentation and translation tasks). We then show

that the proposed incremental utility further helps ICL by contrasting how the

LLMs perform with and without the demonstrations.

</p></li>

</ul>

<h3>Title: Online Continual Knowledge Learning for Language Models. (arXiv:2311.09632v1 [cs.CL])</h3>

<ul>

<li>Paper URL: <a href="http://arxiv.org/abs/2311.09632">http://arxiv.org/abs/2311.09632</a></li>

<li>Code URL: null</li>

<li>Copy Paste: <code><input type="checkbox">[[2311.09632]] Online Continual Knowledge Learning for Language Models(http://arxiv.org/abs/2311.09632)</code></li>

<li>Summary: <p>Large Language Models (LLMs) serve as repositories of extensive world

knowledge, enabling them to perform tasks such as question-answering and

fact-checking. However, this knowledge can become obsolete as global contexts

change. In this paper, we introduce a novel problem in the realm of continual

learning: Online Continual Knowledge Learning (OCKL). This problem formulation

aims to manage the dynamic nature of world knowledge in LMs under real-time

constraints. We propose a new benchmark and evaluation metric designed to

measure both the rate of new knowledge acquisition and the retention of

previously learned knowledge. Our empirical evaluation, conducted using a

variety of state-of-the-art methods, establishes robust base-lines for OCKL.

Our results reveal that existing continual learning approaches are

unfortunately insufficient for tackling the unique challenges posed by OCKL. We

identify key factors that influence the trade-off between knowledge acquisition

and retention, thereby advancing our understanding of how to train LMs in a

continually evolving environment.

</p></li>

</ul>

<h3>Title: Evolving Domain Adaptation of Pretrained Language Models for Text Classification. (arXiv:2311.09661v1 [cs.CL])</h3>

<ul>

<li>Paper URL: <a href="http://arxiv.org/abs/2311.09661">http://arxiv.org/abs/2311.09661</a></li>

<li>Code URL: null</li>

<li>Copy Paste: <code><input type="checkbox">[[2311.09661]] Evolving Domain Adaptation of Pretrained Language Models for Text Classification(http://arxiv.org/abs/2311.09661)</code></li>

<li>Summary: <p>Adapting pre-trained language models (PLMs) for time-series text

classification amidst evolving domain shifts (EDS) is critical for maintaining

accuracy in applications like stance detection. This study benchmarks the

effectiveness of evolving domain adaptation (EDA) strategies, notably

self-training, domain-adversarial training, and domain-adaptive pretraining,

with a focus on an incremental self-training method. Our analysis across

various datasets reveals that this incremental method excels at adapting PLMs

to EDS, outperforming traditional domain adaptation techniques. These findings

highlight the importance of continually updating PLMs to ensure their

effectiveness in real-world applications, paving the way for future research

into PLM robustness against the natural temporal evolution of language.

</p></li>

</ul>

<h3>Title: Whispers of Doubt Amidst Echoes of Triumph in NLP Robustness. (arXiv:2311.09694v1 [cs.CL])</h3>

<ul>

<li>Paper URL: <a href="http://arxiv.org/abs/2311.09694">http://arxiv.org/abs/2311.09694</a></li>

<li>Code URL: null</li>

<li>Copy Paste: <code><input type="checkbox">[[2311.09694]] Whispers of Doubt Amidst Echoes of Triumph in NLP Robustness(http://arxiv.org/abs/2311.09694)</code></li>

<li>Summary: <p>Are the longstanding robustness issues in NLP resolved by today's larger and

more performant models? To address this question, we conduct a thorough

investigation using 19 models of different sizes spanning different

architectural choices and pretraining objectives. We conduct evaluations using

(a) OOD and challenge test sets, (b) CheckLists, (c) contrast sets, and (d)

adversarial inputs. Our analysis reveals that not all OOD tests provide further

insight into robustness. Evaluating with CheckLists and contrast sets shows

significant gaps in model performance; merely scaling models does not make them

sufficiently robust. Finally, we point out that current approaches for

adversarial evaluations of models are themselves problematic: they can be

easily thwarted, and in their current forms, do not represent a sufficiently

deep probe of model robustness. We conclude that not only is the question of

robustness in NLP as yet unresolved, but even some of the approaches to measure

robustness need to be reassessed.

</p></li>

</ul>

<h3>Title: Towards Robust Temporal Reasoning of Large Language Models via a Multi-Hop QA Dataset and Pseudo-Instruction Tuning. (arXiv:2311.09821v1 [cs.CL])</h3>

<ul>

<li>Paper URL: <a href="http://arxiv.org/abs/2311.09821">http://arxiv.org/abs/2311.09821</a></li>

<li>Code URL: null</li>

<li>Copy Paste: <code><input type="checkbox">[[2311.09821]] Towards Robust Temporal Reasoning of Large Language Models via a Multi-Hop QA Dataset and Pseudo-Instruction Tuning(http://arxiv.org/abs/2311.09821)</code></li>

<li>Summary: <p>Knowledge in the real world is being updated constantly. However, it is

costly to frequently update large language models (LLMs). Therefore, it is

crucial for LLMs to understand the concept of temporal knowledge. However,

prior works on temporal question answering did not emphasize multi-answer and

multi-hop types of temporal reasoning. In this paper, we propose a complex

temporal question-answering (QA) dataset Complex-TR that focuses on

multi-answer and multi-hop temporal reasoning. Besides, we also propose a novel

data augmentation strategy to improve the complex temporal reasoning capability

and robustness of LLMs. We conducted experiments on multiple temporal QA

datasets. Experimental results show that our method is able to improve LLMs'

performance on temporal QA benchmarks by significant margins.

</p></li>

</ul>

<h3>Title: Breaking Boundaries: Balancing Performance and Robustness in Deep Wireless Traffic Forecasting. (arXiv:2311.09790v1 [cs.LG])</h3>

<ul>

<li>Paper URL: <a href="http://arxiv.org/abs/2311.09790">http://arxiv.org/abs/2311.09790</a></li>

<li>Code URL: null</li>

<li>Copy Paste: <code><input type="checkbox">[[2311.09790]] Breaking Boundaries: Balancing Performance and Robustness in Deep Wireless Traffic Forecasting(http://arxiv.org/abs/2311.09790)</code></li>

<li>Summary: <p>Balancing the trade-off between accuracy and robustness is a long-standing

challenge in time series forecasting. While most of existing robust algorithms

have achieved certain suboptimal performance on clean data, sustaining the same

performance level in the presence of data perturbations remains extremely hard.

% In this paper, we study a wide array of perturbation scenarios and propose

novel defense mechanisms against adversarial attacks using real-world telecom

data. We compare our strategy against two existing adversarial training

algorithms under a range of maximal allowed perturbations, defined using

$\ell_{\infty}$-norm, $\in [0.1,0.4]$. % Our findings reveal that our hybrid

strategy, which is composed of a classifier to detect adversarial examples, a

denoiser to eliminate noise from the perturbed data samples, and a standard

forecaster, achieves the best performance on both clean and perturbed data. %

Our optimal model can retain up to $92.02\%$ the performance of the original

forecasting model in terms of Mean Squared Error (MSE) on clean data, while

being more robust than the standard adversarially trained models on perturbed

data. Its MSE is 2.71$\times$ and 2.51$\times$ lower than those of comparing

methods on normal and perturbed data, respectively. In addition, the components

of our models can be trained in parallel, resulting in better computational

efficiency. % Our results indicate that we can optimally balance the trade-off

between the performance and robustness of forecasting models by improving the

classifier and denoiser, even in the presence of sophisticated and destructive

poisoning attacks.

</p></li>

</ul>

<h2>biometric</h2>

<h2>steal</h2>

<h2>extraction</h2>

<h3>Title: LOKE: Linked Open Knowledge Extraction for Automated Knowledge Graph Construction. (arXiv:2311.09366v1 [cs.CL])</h3>

<ul>

<li>Paper URL: <a href="http://arxiv.org/abs/2311.09366">http://arxiv.org/abs/2311.09366</a></li>

<li>Code URL: null</li>

<li>Copy Paste: <code><input type="checkbox">[[2311.09366]] LOKE: Linked Open Knowledge Extraction for Automated Knowledge Graph Construction(http://arxiv.org/abs/2311.09366)</code></li>

<li>Summary: <p>While the potential of Open Information Extraction (Open IE) for Knowledge

Graph Construction (KGC) may seem promising, we find that the alignment of Open

IE extraction results with existing knowledge graphs to be inadequate. The

advent of Large Language Models (LLMs), especially the commercially available

OpenAI models, have reset expectations for what is possible with deep learning

models and have created a new field called prompt engineering. We investigate

the use of GPT models and prompt engineering for knowledge graph construction

with the Wikidata knowledge graph to address a similar problem to Open IE,

which we call Open Knowledge Extraction (OKE) using an approach we call the

Linked Open Knowledge Extractor (LOKE, pronounced like "Loki"). We consider the

entity linking task essential to construction of real world knowledge graphs.

We merge the CaRB benchmark scoring approach with data from the TekGen dataset

for the LOKE task. We then show that a well engineered prompt, paired with a

naive entity linking approach (which we call LOKE-GPT), outperforms AllenAI's

OpenIE 4 implementation on the OKE task, although it over-generates triples

compared to the reference set due to overall triple scarcity in the TekGen set.

Through an analysis of entity linkability in the CaRB dataset, as well as

outputs from OpenIE 4 and LOKE-GPT, we see that LOKE-GPT and the "silver"

TekGen triples show that the task is significantly different in content from

OIE, if not structure. Through this analysis and a qualitative analysis of

sentence extractions via all methods, we found that LOKE-GPT extractions are of

high utility for the KGC task and suitable for use in semi-automated extraction

settings.

</p></li>

</ul>

<h3>Title: A Survey on Online User Aggression: Content Detection and Behavioural Analysis on Social Media Platforms. (arXiv:2311.09367v1 [cs.CL])</h3>

<ul>

<li>Paper URL: <a href="http://arxiv.org/abs/2311.09367">http://arxiv.org/abs/2311.09367</a></li>

<li>Code URL: null</li>

<li>Copy Paste: <code><input type="checkbox">[[2311.09367]] A Survey on Online User Aggression: Content Detection and Behavioural Analysis on Social Media Platforms(http://arxiv.org/abs/2311.09367)</code></li>

<li>Summary: <p>The rise of social media platforms has led to an increase in cyber-aggressive

behavior, encompassing a broad spectrum of hostile behavior, including

cyberbullying, online harassment, and the dissemination of offensive and hate

speech. These behaviors have been associated with significant societal

consequences, ranging from online anonymity to real-world outcomes such as

depression, suicidal tendencies, and, in some instances, offline violence.

Recognizing the societal risks associated with unchecked aggressive content,

this paper delves into the field of Aggression Content Detection and Behavioral

Analysis of Aggressive Users, aiming to bridge the gap between disparate

studies. In this paper, we analyzed the diversity of definitions and proposed a

unified cyber-aggression definition. We examine the comprehensive process of

Aggression Content Detection, spanning from dataset creation, feature selection

and extraction, and detection algorithm development. Further, we review studies

on Behavioral Analysis of Aggression that explore the influencing factors,

consequences, and patterns associated with cyber-aggressive behavior. This

systematic literature review is a cross-examination of content detection and

behavioral analysis in the realm of cyber-aggression. The integrated

investigation reveals the effectiveness of incorporating sociological insights

into computational techniques for preventing cyber-aggressive behavior.

Finally, the paper concludes by identifying research gaps and encouraging

further progress in the unified domain of socio-computational aggressive

behavior analysis.

</p></li>

</ul>

<h3>Title: SQATIN: Supervised Instruction Tuning Meets Question Answering for Improved Dialogue NLU. (arXiv:2311.09502v1 [cs.CL])</h3>

<ul>

<li>Paper URL: <a href="http://arxiv.org/abs/2311.09502">http://arxiv.org/abs/2311.09502</a></li>

<li>Code URL: null</li>

<li>Copy Paste: <code><input type="checkbox">[[2311.09502]] SQATIN: Supervised Instruction Tuning Meets Question Answering for Improved Dialogue NLU(http://arxiv.org/abs/2311.09502)</code></li>

<li>Summary: <p>Task-oriented dialogue (ToD) systems help users execute well-defined tasks

across a variety of domains (e.g., $\textit{flight booking}$ or $\textit{food

ordering}$), with their Natural Language Understanding (NLU) components being

dedicated to the analysis of user utterances, predicting users' intents

($\textit{Intent Detection}$, ID) and extracting values for informational slots

($\textit{Value Extraction}$, VE). In most domains, labelled NLU data is

scarce, making sample-efficient learning -- enabled with effective transfer

paradigms -- paramount. In this work, we introduce SQATIN, a new framework for

dialog NLU based on (i) instruction tuning and (ii) question-answering-based

formulation of ID and VE tasks. According to the evaluation on established NLU

benchmarks, SQATIN sets the new state of the art in dialogue NLU, substantially

surpassing the performance of current models based on standard fine-tuning

objectives in both in-domain training and cross-domain transfer. SQATIN yields

particularly large performance gains in cross-domain transfer, owing to the

fact that our QA-based instruction tuning leverages similarities between

natural language descriptions of classes (i.e., slots and intents) across

domains.

</p></li>

</ul>

<h3>Title: A Reevaluation of Event Extraction: Past, Present, and Future Challenges. (arXiv:2311.09562v1 [cs.CL])</h3>

<ul>

<li>Paper URL: <a href="http://arxiv.org/abs/2311.09562">http://arxiv.org/abs/2311.09562</a></li>

<li>Code URL: https://github.com/ej0cl6/textee</li>

<li>Copy Paste: <code><input type="checkbox">[[2311.09562]] A Reevaluation of Event Extraction: Past, Present, and Future Challenges(http://arxiv.org/abs/2311.09562)</code></li>

<li>Summary: <p>Event extraction has attracted much attention in recent years due to its

potential for many applications. However, recent studies observe some

evaluation challenges, suggesting that reported scores might not reflect the

true performance. In this work, we first identify and discuss these evaluation

challenges, including the unfair comparisons resulting from different

assumptions about data or different data preprocessing steps, the

incompleteness of the current evaluation framework leading to potential dataset

bias or data split bias, and low reproducibility of prior studies. To address

these challenges, we propose TextEE, a standardized, fair, and reproducible

benchmark for event extraction. TextEE contains standardized data preprocessing

scripts and splits for more than ten datasets across different domains. In

addition, we aggregate and re-implement over ten event extraction approaches

published in recent years and conduct a comprehensive reevaluation. Finally, we

explore the capability of large language models in event extraction and discuss

some future challenges. We expect TextEE will serve as a reliable benchmark for

event extraction, facilitating future research in the field.

</p></li>

</ul>

<h3>Title: A Self-enhancement Multitask Framework for Unsupervised Aspect Category Detection. (arXiv:2311.09708v1 [cs.CL])</h3>

<ul>

<li>Paper URL: <a href="http://arxiv.org/abs/2311.09708">http://arxiv.org/abs/2311.09708</a></li>

<li>Code URL: null</li>

<li>Copy Paste: <code><input type="checkbox">[[2311.09708]] A Self-enhancement Multitask Framework for Unsupervised Aspect Category Detection(http://arxiv.org/abs/2311.09708)</code></li>

<li>Summary: <p>Our work addresses the problem of unsupervised Aspect Category Detection

using a small set of seed words. Recent works have focused on learning

embedding spaces for seed words and sentences to establish similarities between

sentences and aspects. However, aspect representations are limited by the

quality of initial seed words, and model performances are compromised by noise.

To mitigate this limitation, we propose a simple framework that automatically

enhances the quality of initial seed words and selects high-quality sentences

for training instead of using the entire dataset. Our main concepts are to add

a number of seed words to the initial set and to treat the task of noise

resolution as a task of augmenting data for a low-resource task. In addition,

we jointly train Aspect Category Detection with Aspect Term Extraction and

Aspect Term Polarity to further enhance performance. This approach facilitates

shared representation learning, allowing Aspect Category Detection to benefit

from the additional guidance offered by other tasks. Extensive experiments

demonstrate that our framework surpasses strong baselines on standard datasets.

</p></li>

</ul>

<h3>Title: MOKA: Moral Knowledge Augmentation for Moral Event Extraction. (arXiv:2311.09733v1 [cs.CL])</h3>

<ul>

<li>Paper URL: <a href="http://arxiv.org/abs/2311.09733">http://arxiv.org/abs/2311.09733</a></li>

<li>Code URL: null</li>

<li>Copy Paste: <code><input type="checkbox">[[2311.09733]] MOKA: Moral Knowledge Augmentation for Moral Event Extraction(http://arxiv.org/abs/2311.09733)</code></li>

<li>Summary: <p>News media employ moral language to create memorable stories, and readers

often engage with the content that align with their values. Moral theories have

been applied to news analysis studying moral values in isolation, while the

intricate dynamics among participating entities in shaping moral events have

been overlooked. This is mainly due to the use of obscure language to conceal

evident ideology and values, coupled with the insufficient moral reasoning

capability in most existing NLP systems, where LLMs are no exception. To study

this phenomenon, we first annotate a new dataset, MORAL EVENTS, consisting of

5,494 structured annotations on 474 news articles by diverse US media across

the political spectrum. We further propose MOKA, a moral event extraction

framework with MOral Knowledge Augmentation, that leverages knowledge derived

from moral words and moral scenarios. Experimental results show that MOKA

outperforms competitive baselines across three moral event understanding tasks.

Further analyses illuminate the selective reporting of moral events by media

outlets of different ideological leanings, suggesting the significance of

event-level morality analysis in news. Our datasets and codebase are available

at https://github.com/launchnlp/MOKA.

</p></li>

</ul>

<h3>Title: CARE: Extracting Experimental Findings From Clinical Literature. (arXiv:2311.09736v1 [cs.CL])</h3>

<ul>

<li>Paper URL: <a href="http://arxiv.org/abs/2311.09736">http://arxiv.org/abs/2311.09736</a></li>

<li>Code URL: null</li>

<li>Copy Paste: <code><input type="checkbox">[[2311.09736]] CARE: Extracting Experimental Findings From Clinical Literature(http://arxiv.org/abs/2311.09736)</code></li>

<li>Summary: <p>Extracting fine-grained experimental findings from literature can provide

massive utility for scientific applications. Prior work has focused on

developing annotation schemas and datasets for limited aspects of this problem,

leading to simpler information extraction datasets which do not capture the

real-world complexity and nuance required for this task. Focusing on

biomedicine, this work presents CARE (Clinical Aggregation-oriented Result

[View remainder of file in raw view](https://github.com/wwd29/arxiv-daily/raw/1a3074665786adbe2140d325f14a577e8297e535/html/user_0/2023-11-17.html)

## Footer

[ ](https://github.com "GitHub") (C) 2025 GitHub, Inc. 

### Footer navigation

  * [Terms](https://docs.github.com/site-policy/github-terms/github-terms-of-service)
  * [Privacy](https://docs.github.com/site-policy/privacy-policies/github-privacy-statement)
  * [Security](https://github.com/security)
  * [Status](https://www.githubstatus.com/)
  * [Docs](https://docs.github.com/)
  * [Contact](https://support.github.com?tags=dotcom-footer)
  * Manage cookies 
  * Do not share my personal information 



You can’t perform that action at this time. 


scrape_date: 2025-03-17T13:26:50.325675
content_type: markdown
hash: ffd4d3663993b23ba2e06ec42f9ebc1b48a1cb61c47908b0a1a1d0d61beaeef9
url:
https://github.com/jmzhoulab/ArxivPaperReader/blob/090b54f9e0c9c996b4c20f937e1fe5fe4b4384ea/docs/source/202404/240409.rst

title:
ArxivPaperReader/docs/source/202404/240409.rst at 090b54f9e0c9c996b4c20f937e1fe5fe4b4384ea · jmzhoulab/ArxivPaperReader · GitHub

content:
Skip to content

## Navigation Menu

Toggle navigation

[ ](/)

[ Sign in ](/login?return_to=https%3A%2F%2Fgithub.com%2Fjmzhoulab%2FArxivPaperReader%2Fblob%2F090b54f9e0c9c996b4c20f937e1fe5fe4b4384ea%2Fdocs%2Fsource%2F202404%2F240409.rst)

  * Product 

    * [ GitHub Copilot Write better code with AI  ](https://github.com/features/copilot)
    * [ Security Find and fix vulnerabilities  ](https://github.com/features/security)
    * [ Actions Automate any workflow  ](https://github.com/features/actions)
    * [ Codespaces Instant dev environments  ](https://github.com/features/codespaces)
    * [ Issues Plan and track work  ](https://github.com/features/issues)
    * [ Code Review Manage code changes  ](https://github.com/features/code-review)
    * [ Discussions Collaborate outside of code  ](https://github.com/features/discussions)
    * [ Code Search Find more, search less  ](https://github.com/features/code-search)

Explore
    * [ All features ](https://github.com/features)
    * [ Documentation  ](https://docs.github.com)
    * [ GitHub Skills  ](https://skills.github.com)
    * [ Blog  ](https://github.blog)

  * Solutions 

By company size
    * [ Enterprises ](https://github.com/enterprise)
    * [ Small and medium teams ](https://github.com/team)
    * [ Startups ](https://github.com/enterprise/startups)
    * [ Nonprofits ](/solutions/industry/nonprofits)

By use case
    * [ DevSecOps ](/solutions/use-case/devsecops)
    * [ DevOps ](/solutions/use-case/devops)
    * [ CI/CD ](/solutions/use-case/ci-cd)
    * [ View all use cases ](/solutions/use-case)

By industry
    * [ Healthcare ](/solutions/industry/healthcare)
    * [ Financial services ](/solutions/industry/financial-services)
    * [ Manufacturing ](/solutions/industry/manufacturing)
    * [ Government ](/solutions/industry/government)
    * [ View all industries ](/solutions/industry)

[ View all solutions  ](/solutions)

  * Resources 

Topics
    * [ AI ](/resources/articles/ai)
    * [ DevOps ](/resources/articles/devops)
    * [ Security ](/resources/articles/security)
    * [ Software Development ](/resources/articles/software-development)
    * [ View all ](/resources/articles)

Explore
    * [ Learning Pathways  ](https://resources.github.com/learn/pathways)
    * [ Events & Webinars  ](https://resources.github.com)
    * [ Ebooks & Whitepapers ](https://github.com/resources/whitepapers)
    * [ Customer Stories ](https://github.com/customer-stories)
    * [ Partners  ](https://partner.github.com)
    * [ Executive Insights ](https://github.com/solutions/executive-insights)

  * Open Source 

    * [ GitHub Sponsors Fund open source developers  ](/sponsors)

    * [ The ReadME Project GitHub community articles  ](https://github.com/readme)

Repositories
    * [ Topics ](https://github.com/topics)
    * [ Trending ](https://github.com/trending)
    * [ Collections ](https://github.com/collections)

  * Enterprise 

    * [ Enterprise platform AI-powered developer platform  ](/enterprise)

Available add-ons
    * [ Advanced Security Enterprise-grade security features  ](https://github.com/enterprise/advanced-security)
    * [ Copilot for business Enterprise-grade AI features  ](/features/copilot/copilot-business)
    * [ Premium Support Enterprise-grade 24/7 support  ](/premium-support)

  * [Pricing](https://github.com/pricing)



Search or jump to...

# Search code, repositories, users, issues, pull requests...

Search 

Clear




[Search syntax tips](https://docs.github.com/search-github/github-code-search/understanding-github-code-search-syntax)

#  Provide feedback 

We read every piece of feedback, and take your input very seriously.

Include my email address so I can be contacted

Cancel  Submit feedback 

#  Saved searches 

## Use saved searches to filter your results more quickly

Name

Query

To see all available qualifiers, see our [documentation](https://docs.github.com/search-github/github-code-search/understanding-github-code-search-syntax). 

Cancel  Create saved search 

[ Sign in ](/login?return_to=https%3A%2F%2Fgithub.com%2Fjmzhoulab%2FArxivPaperReader%2Fblob%2F090b54f9e0c9c996b4c20f937e1fe5fe4b4384ea%2Fdocs%2Fsource%2F202404%2F240409.rst)

[ Sign up ](/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fblob%2Fshow&source=header-repo&source_repo=jmzhoulab%2FArxivPaperReader) Reseting focus

You signed in with another tab or window. [Reload]() to refresh your session. You signed out in another tab or window. [Reload]() to refresh your session. You switched accounts on another tab or window. [Reload]() to refresh your session. Dismiss alert

{{ message }}

[ jmzhoulab ](/jmzhoulab) / **[ArxivPaperReader](/jmzhoulab/ArxivPaperReader) ** Public

generated from [readthedocs/tutorial-template](/readthedocs/tutorial-template)

  * [ Notifications ](/login?return_to=%2Fjmzhoulab%2FArxivPaperReader) You must be signed in to change notification settings
  * [ Fork 0 ](/login?return_to=%2Fjmzhoulab%2FArxivPaperReader)
  * [ Star  0 ](/login?return_to=%2Fjmzhoulab%2FArxivPaperReader)




  * [ Code ](/jmzhoulab/ArxivPaperReader)
  * [ Issues 0 ](/jmzhoulab/ArxivPaperReader/issues)
  * [ Pull requests 0 ](/jmzhoulab/ArxivPaperReader/pulls)
  * [ Actions ](/jmzhoulab/ArxivPaperReader/actions)
  * [ Projects 0 ](/jmzhoulab/ArxivPaperReader/projects)
  * [ Security ](/jmzhoulab/ArxivPaperReader/security)
  * [ Insights ](/jmzhoulab/ArxivPaperReader/pulse)



Additional navigation options

  * [ Code  ](/jmzhoulab/ArxivPaperReader)
  * [ Issues  ](/jmzhoulab/ArxivPaperReader/issues)
  * [ Pull requests  ](/jmzhoulab/ArxivPaperReader/pulls)
  * [ Actions  ](/jmzhoulab/ArxivPaperReader/actions)
  * [ Projects  ](/jmzhoulab/ArxivPaperReader/projects)
  * [ Security  ](/jmzhoulab/ArxivPaperReader/security)
  * [ Insights  ](/jmzhoulab/ArxivPaperReader/pulse)



## Files

090b54f

## Breadcrumbs

  1. [ArxivPaperReader](/jmzhoulab/ArxivPaperReader/tree/090b54f9e0c9c996b4c20f937e1fe5fe4b4384ea)
  2. /[docs](/jmzhoulab/ArxivPaperReader/tree/090b54f9e0c9c996b4c20f937e1fe5fe4b4384ea/docs)
  3. /[source](/jmzhoulab/ArxivPaperReader/tree/090b54f9e0c9c996b4c20f937e1fe5fe4b4384ea/docs/source)
  4. /[202404](/jmzhoulab/ArxivPaperReader/tree/090b54f9e0c9c996b4c20f937e1fe5fe4b4384ea/docs/source/202404)



/

# 240409.rst

Copy path

Blame

Blame

## Latest commit

## History

[History](/jmzhoulab/ArxivPaperReader/commits/090b54f9e0c9c996b4c20f937e1fe5fe4b4384ea/docs/source/202404/240409.rst)

[](/jmzhoulab/ArxivPaperReader/commits/090b54f9e0c9c996b4c20f937e1fe5fe4b4384ea/docs/source/202404/240409.rst)

2120 lines (1296 loc) · 261 KB

090b54f

## Breadcrumbs

  1. [ArxivPaperReader](/jmzhoulab/ArxivPaperReader/tree/090b54f9e0c9c996b4c20f937e1fe5fe4b4384ea)
  2. /[docs](/jmzhoulab/ArxivPaperReader/tree/090b54f9e0c9c996b4c20f937e1fe5fe4b4384ea/docs)
  3. /[source](/jmzhoulab/ArxivPaperReader/tree/090b54f9e0c9c996b4c20f937e1fe5fe4b4384ea/docs/source)
  4. /[202404](/jmzhoulab/ArxivPaperReader/tree/090b54f9e0c9c996b4c20f937e1fe5fe4b4384ea/docs/source/202404)



/

# 240409.rst

Top

## File metadata and controls

  * Preview

  * Code

  * Blame




2120 lines (1296 loc) · 261 KB

[Raw](https://github.com/jmzhoulab/ArxivPaperReader/raw/090b54f9e0c9c996b4c20f937e1fe5fe4b4384ea/docs/source/202404/240409.rst)

# 240409

[[2404.04298] SELF-[IN]CORRECT: LLMs Struggle with Refining Self-Generated Responses](https://arxiv.org/abs/2404.04298)
    
    
    Thu, 4 Apr 2024 20:27:37 GMT
    Dongwei Jiang, Jingyu Zhang, Orion Weller, Nathaniel Weir, Benjamin Van Durme, Daniel Khashabi
    

Can LLMs continually improve their previous outputs for better results? An affirmative answer would require LLMs to be better at discriminating among previously-generated alternatives, than generating initial responses. We explore the validity of this hypothesis in practice. We first introduce a unified framework that allows us to compare the generative and discriminative capability of any model on any task. Then, in our resulting experimental analysis of several LLMs, we do not observe the performance of those models on discrimination to be reliably better than generation. We hope these findings inform the growing literature on self-improvement AI systems.

* * *

[[2404.04326] Hypothesis Generation with Large Language Models](https://arxiv.org/abs/2404.04326)
    
    
    Fri, 5 Apr 2024 18:00:07 GMT
    Yangqiaoyu Zhou, Haokun Liu, Tejes Srivastava, Hongyuan Mei, and Chenhao Tan
    

Effective generation of novel hypotheses is instrumental to scientific progress. So far, researchers have been the main powerhouse behind hypothesis generation by painstaking data analysis and thinking (also known as the Eureka moment). In this paper, we examine the potential of large language models (LLMs) to generate hypotheses. We focus on hypothesis generation based on data (i.e., labeled examples). To enable LLMs to handle arbitrarily long contexts, we generate initial hypotheses from a small number of examples and then update them iteratively to improve the quality of hypotheses. Inspired by multi-armed bandits, we design a reward function to inform the exploitation-exploration tradeoff in the update process. Our algorithm is able to generate hypotheses that enable much better predictive performance than few-shot prompting in classification tasks, improving accuracy by 31.7% on a synthetic dataset and by 13.9%, 3.3% and, 24.9% on three real-world datasets. We also outperform supervised learning by 12.8% and 11.2% on two challenging real-world datasets. Furthermore, we find that the generated hypotheses not only corroborate human-verified theories but also uncover new insights for the tasks.

* * *

[[2404.04436] AI Knowledge and Reasoning: Emulating Expert Creativity in Scientific Research](https://arxiv.org/abs/2404.04436)
    
    
    Fri, 5 Apr 2024 22:30:47 GMT
    Anirban Mukherjee, Hannah Hanwen Chang
    

We investigate whether modern AI can emulate expert creativity in complex scientific endeavors. We introduce novel methodology that utilizes original research articles published after the AI's training cutoff, ensuring no prior exposure, mitigating concerns of rote memorization and prior training. The AI are tasked with redacting findings, predicting outcomes from redacted research, and assessing prediction accuracy against reported results. Analysis on 589 published studies in four leading psychology journals over a 28-month period, showcase the AI's proficiency in understanding specialized research, deductive reasoning, and evaluating evidentiary alignment--cognitive hallmarks of human subject matter expertise and creativity. These findings suggest the potential of general-purpose AI to transform academia, with roles requiring knowledge-based creativity become increasingly susceptible to technological substitution.

* * *

[[2404.04442] Exploring Autonomous Agents through the Lens of Large Language Models: A Review](https://arxiv.org/abs/2404.04442)
    
    
    Fri, 5 Apr 2024 22:59:02 GMT
    Saikat Barua
    

Large Language Models (LLMs) are transforming artificial intelligence, enabling autonomous agents to perform diverse tasks across various domains. These agents, proficient in human-like text comprehension and generation, have the potential to revolutionize sectors from customer service to healthcare. However, they face challenges such as multimodality, human value alignment, hallucinations, and evaluation. Techniques like prompting, reasoning, tool utilization, and in-context learning are being explored to enhance their capabilities. Evaluation platforms like AgentBench, WebArena, and ToolLLM provide robust methods for assessing these agents in complex scenarios. These advancements are leading to the development of more resilient and capable autonomous agents, anticipated to become integral in our digital lives, assisting in tasks from email responses to disease diagnosis. The future of AI, with LLMs at the forefront, is promising.

* * *

[[2404.04540] The Case for Developing a Foundation Model for Planning-like Tasks from Scratch](https://arxiv.org/abs/2404.04540)
    
    
    Sat, 6 Apr 2024 07:44:40 GMT
    Biplav Srivastava, Vishal Pallagani
    

Foundation Models (FMs) have revolutionized many areas of computing, including Automated Planning and Scheduling (APS). For example, a recent study found them useful for planning problems: plan generation, language translation, model construction, multi-agent planning, interactive planning, heuristics optimization, tool integration, and brain-inspired planning. Besides APS, there are many seemingly related tasks involving the generation of a series of actions with varying guarantees of their executability to achieve intended goals, which we collectively call planning-like (PL) tasks like business processes, programs, workflows, and guidelines, where researchers have considered using FMs. However, previous works have primarily focused on pre-trained, off-the-shelf FMs and optionally fine-tuned them. This paper discusses the need for a comprehensive FM for PL tasks from scratch and explores its design considerations. We argue that such an FM will open new and efficient avenues for PL problem-solving, just like LLMs are creating for APS.

* * *

[[2404.04619] Do We Really Need a Complex Agent System? Distill Embodied Agent into a Single Model](https://arxiv.org/abs/2404.04619)
    
    
    Sat, 6 Apr 2024 12:51:00 GMT
    Zhonghan Zhao, Ke Ma, Wenhao Chai, Xuan Wang, Kewei Chen, Dongxu Guo, Yanting Zhang, Hongwei Wang and Gaoang Wang
    

With the power of large language models (LLMs), open-ended embodied agents can flexibly understand human instructions, generate interpretable guidance strategies, and output executable actions. Nowadays, Multi-modal Language Models~(MLMs) integrate multi-modal signals into LLMs, further bringing richer perception to entity agents and allowing embodied agents to perceive world-understanding tasks more delicately. However, existing works: 1) operate independently by agents, each containing multiple LLMs, from perception to action, resulting in gaps between complex tasks and execution; 2) train MLMs on static data, struggling with dynamics in open-ended scenarios; 3) input prior knowledge directly as prompts, suppressing application flexibility. We propose STEVE-2, a hierarchical knowledge distillation framework for open-ended embodied tasks, characterized by 1) a hierarchical system for multi-granular task division, 2) a mirrored distillation method for parallel simulation data, and 3) an extra expert model for bringing additional knowledge into parallel simulation. After distillation, embodied agents can complete complex, open-ended tasks without additional expert guidance, utilizing the performance and knowledge of a versatile MLM. Extensive evaluations on navigation and creation tasks highlight the superior performance of STEVE-2 in open-ended tasks, with $1.4 times$ \- $7.3 times$ in performance.

* * *

[[2404.04667] Autonomous Artificial Intelligence Agents for Clinical Decision Making in Oncology](https://arxiv.org/abs/2404.04667)
    
    
    Sat, 6 Apr 2024 15:50:19 GMT
    Dyke Ferber, Omar S. M. El Nahhas, Georg W\"olflein, Isabella C. Wiest, Jan Clusmann, Marie-Elisabeth Le{\ss}man, Sebastian Foersch, Jacqueline Lammert, Maximilian Tschochohei, Dirk J\"ager, Manuel Salto-Tellez, Nikolaus Schultz, Daniel Truhn, Jakob Nikolas Kather
    

Multimodal artificial intelligence (AI) systems have the potential to enhance clinical decision-making by interpreting various types of medical data. However, the effectiveness of these models across all medical fields is uncertain. Each discipline presents unique challenges that need to be addressed for optimal performance. This complexity is further increased when attempting to integrate different fields into a single model. Here, we introduce an alternative approach to multimodal medical AI that utilizes the generalist capabilities of a large language model (LLM) as a central reasoning engine. This engine autonomously coordinates and deploys a set of specialized medical AI tools. These tools include text, radiology and histopathology image interpretation, genomic data processing, web searches, and document retrieval from medical guidelines. We validate our system across a series of clinical oncology scenarios that closely resemble typical patient care workflows. We show that the system has a high capability in employing appropriate tools (97%), drawing correct conclusions (93.6%), and providing complete (94%), and helpful (89.2%) recommendations for individual patient cases while consistently referencing relevant literature (82.5%) upon instruction. This work provides evidence that LLMs can effectively plan and execute domain-specific models to retrieve or synthesize new information when used as autonomous agents. This enables them to function as specialist, patient-tailored clinical assistants. It also simplifies regulatory compliance by allowing each component tool to be individually validated and approved. We believe, that our work can serve as a proof-of-concept for more advanced LLM-agents in the medical domain.

* * *

[[2404.04735] MACM: Utilizing a Multi-Agent System for Condition Mining in Solving Complex Mathematical Problems](https://arxiv.org/abs/2404.04735)
    
    
    Sat, 6 Apr 2024 21:39:01 GMT
    Bin Lei
    

Recent advancements in large language models, such as GPT-4, have demonstrated remarkable capabilities in processing standard queries. Despite these advancements, their performance substantially declines in textbf{advanced mathematical problems requiring complex, multi-step logical reasoning}. To enhance their inferential capabilities, current research has delved into textit{prompting engineering}, exemplified by methodologies such as the Tree of Thought and Graph of Thought. Nonetheless, these existing approaches encounter two significant limitations. Firstly, their effectiveness in tackling complex mathematical problems is somewhat constrained. Secondly, the necessity to design distinct prompts for individual problems hampers their generalizability. In response to these limitations, this paper introduces the textit{Multi-Agent System for conditional Mining} (textbf{MACM}) prompting method. It not only resolves intricate mathematical problems but also demonstrates strong generalization capabilities across various mathematical contexts. With the assistance of MACM, the accuracy of GPT-4 Turbo on the most challenging level five mathematical problems in the MATH dataset increase from $mathbf{54.68%} text{ to } mathbf{76.73%}$. The code is available in url{<https://github.com/bin123apple/MACM>}.

* * *

[[2404.04752] Challenges Faced by Large Language Models in Solving Multi-Agent Flocking](https://arxiv.org/abs/2404.04752)
    
    
    Sat, 6 Apr 2024 22:34:07 GMT
    Peihan Li, Vishnu Menon, Bhavanaraj Gudiguntla, Daniel Ting and Lifeng Zhou
    

Flocking is a behavior where multiple agents in a system attempt to stay close to each other while avoiding collision and maintaining a desired formation. This is observed in the natural world and has applications in robotics, including natural disaster search and rescue, wild animal tracking, and perimeter surveillance and patrol. Recently, large language models (LLMs) have displayed an impressive ability to solve various collaboration tasks as individual decision-makers. Solving multi-agent flocking with LLMs would demonstrate their usefulness in situations requiring spatial and decentralized decision-making. Yet, when LLM-powered agents are tasked with implementing multi-agent flocking, they fall short of the desired behavior. After extensive testing, we find that agents with LLMs as individual decision-makers typically opt to converge on the average of their initial positions or diverge from each other. After breaking the problem down, we discover that LLMs cannot understand maintaining a shape or keeping a distance in a meaningful way. Solving multi-agent flocking with LLMs would enhance their ability to understand collaborative spatial reasoning and lay a foundation for addressing more complex multi-agent tasks. This paper discusses the challenges LLMs face in multi-agent flocking and suggests areas for future improvement and research.

* * *

[[2404.04902] AI2Apps: A Visual IDE for Building LLM-based AI Agent Applications](https://arxiv.org/abs/2404.04902)
    
    
    Sun, 7 Apr 2024 10:02:09 GMT
    Xin Pang, Zhucong Li, Jiaxiang Chen, Yuan Cheng, Yinghui Xu, Yuan Qi
    

We introduce AI2Apps, a Visual Integrated Development Environment (Visual IDE) with full-cycle capabilities that accelerates developers to build deployable LLM-based AI agent Applications. This Visual IDE prioritizes both the Integrity of its development tools and the Visuality of its components, ensuring a smooth and efficient building experience.On one hand, AI2Apps integrates a comprehensive development toolkit ranging from a prototyping canvas and AI-assisted code editor to agent debugger, management system, and deployment tools all within a web-based graphical user interface. On the other hand, AI2Apps visualizes reusable front-end and back-end code as intuitive drag-and-drop components. Furthermore, a plugin system named AI2Apps Extension (AAE) is designed for Extensibility, showcasing how a new plugin with 20 components enables web agent to mimic human-like browsing behavior. Our case study demonstrates substantial efficiency improvements, with AI2Apps reducing token consumption and API calls when debugging a specific sophisticated multimodal agent by approximately 90% and 80%, respectively. The AI2Apps, including an online demo, open-source code, and a screencast video, is now publicly accessible.

* * *

[[2404.05074] On the Uniqueness of Solution for the Bellman Equation of LTL Objectives](https://arxiv.org/abs/2404.05074)
    
    
    Sun, 7 Apr 2024 21:06:52 GMT
    Zetong Xuan, Alper Kamil Bozkurt, Miroslav Pajic, Yu Wang
    

Surrogate rewards for linear temporal logic (LTL) objectives are commonly utilized in planning problems for LTL objectives. In a widely-adopted surrogate reward approach, two discount factors are used to ensure that the expected return approximates the satisfaction probability of the LTL objective. The expected return then can be estimated by methods using the Bellman updates such as reinforcement learning. However, the uniqueness of the solution to the Bellman equation with two discount factors has not been explicitly discussed. We demonstrate with an example that when one of the discount factors is set to one, as allowed in many previous works, the Bellman equation may have multiple solutions, leading to inaccurate evaluation of the expected return. We then propose a condition for the Bellman equation to have the expected return as the unique solution, requiring the solutions for states inside a rejecting bottom strongly connected component (BSCC) to be 0. We prove this condition is sufficient by showing that the solutions for the states with discounting can be separated from those for the states without discounting under this condition

* * *

[[2404.05569] 360{deg}REA: Towards A Reusable Experience Accumulation with 360{deg} Assessment for Multi-Agent System](https://arxiv.org/abs/2404.05569)
    
    
    Mon, 8 Apr 2024 14:43:13 GMT
    Shen Gao, Hao Li, Zhengliang Shi, Chengrui Huang, Quan Tu, Zhiliang Tian, Minlie Huang, Shuo Shang
    

Large language model agents have demonstrated remarkable advancements across various complex tasks. Recent works focus on optimizing the agent team or employing self-reflection to iteratively solve complex tasks. Since these agents are all based on the same LLM, only conducting self-evaluation or removing underperforming agents does not substantively enhance the capability of the agents. We argue that a comprehensive evaluation and accumulating experience from evaluation feedback is an effective approach to improving system performance. In this paper, we propose Reusable Experience Accumulation with 360{deg} Assessment (360{deg}REA), a hierarchical multi-agent framework inspired by corporate organizational practices. The framework employs a novel 360{deg} performance assessment method for multi-perspective performance evaluation with fine-grained assessment. To enhance the capability of agents in addressing complex tasks, we introduce dual-level experience pool for agents to accumulate experience through fine-grained assessment. Extensive experiments on complex task datasets demonstrate the effectiveness of 360{deg}REA.

* * *

[[2404.04281] Similar Data Points Identification with LLM: A Human-in-the-loop Strategy Using Summarization and Hidden State Insights](https://arxiv.org/abs/2404.04281)
    
    
    Wed, 3 Apr 2024 03:17:28 GMT
    Xianlong Zeng, Fanghao Song, Ang Liu
    

This study introduces a simple yet effective method for identifying similar data points across non-free text domains, such as tabular and image data, using Large Language Models (LLMs). Our two-step approach involves data point summarization and hidden state extraction. Initially, data is condensed via summarization using an LLM, reducing complexity and highlighting essential information in sentences. Subsequently, the summarization sentences are fed through another LLM to extract hidden states, serving as compact, feature-rich representations. This approach leverages the advanced comprehension and generative capabilities of LLMs, offering a scalable and efficient strategy for similarity identification across diverse datasets. We demonstrate the effectiveness of our method in identifying similar data points on multiple datasets. Additionally, our approach enables non-technical domain experts, such as fraud investigators or marketing operators, to quickly identify similar data points tailored to specific scenarios, demonstrating its utility in practical applications. In general, our results open new avenues for leveraging LLMs in data analysis across various domains.

* * *

[[2404.04285] MIMIR: A Streamlined Platform for Personalized Agent Tuning in Domain Expertise](https://arxiv.org/abs/2404.04285)
    
    
    Wed, 3 Apr 2024 23:42:38 GMT
    Chunyuan Deng, Xiangru Tang, Yilun Zhao, Hanming Wang, Haoran Wang, Wangchunshu Zhou, Arman Cohan, Mark Gerstein
    

Recently, large language models (LLMs) have evolved into interactive agents, proficient in planning, tool use, and task execution across a wide variety of tasks. However, without specific agent tuning, open-source models like LLaMA currently struggle to match the efficiency of GPT- 4, particularly given the scarcity of agent-tuning datasets for fine-tuning. In response, we introduce textsc{Mimir}: a streamlined platform offering a customizable pipeline that enables users to leverage both private knowledge and publicly available, legally compliant datasets at scale for textbf{personalized agent tuning}. Additionally, textsc{Mimir} supports the generation of general instruction-tuning datasets from the same input. This dual capability ensures that language agents developed through the platform possess both specific agent abilities and general competencies. textsc{Mimir} integrates these features into a cohesive end-to-end platform, facilitating everything from the uploading of personalized files to one-click agent fine-tuning.

* * *

[[2404.04286] Language Model Evolution: An Iterated Learning Perspective](https://arxiv.org/abs/2404.04286)
    
    
    Thu, 4 Apr 2024 02:01:25 GMT
    Yi Ren, Shangmin Guo, Linlu Qiu, Bailin Wang, Danica J. Sutherland
    

With the widespread adoption of Large Language Models (LLMs), the prevalence of iterative interactions among these models is anticipated to increase. Notably, recent advancements in multi-round self-improving methods allow LLMs to generate new examples for training subsequent models. At the same time, multi-agent LLM systems, involving automated interactions among agents, are also increasing in prominence. Thus, in both short and long terms, LLMs may actively engage in an evolutionary process. We draw parallels between the behavior of LLMs and the evolution of human culture, as the latter has been extensively studied by cognitive scientists for decades. Our approach involves leveraging Iterated Learning (IL), a Bayesian framework that elucidates how subtle biases are magnified during human cultural evolution, to explain some behaviors of LLMs. This paper outlines key characteristics of agents' behavior in the Bayesian-IL framework, including predictions that are supported by experimental verification with various LLMs. This theoretical framework could help to more effectively predict and guide the evolution of LLMs in desired directions.

* * *

[[2404.04287] CONFLARE: CONFormal LArge language model REtrieval](https://arxiv.org/abs/2404.04287)
    
    
    Thu, 4 Apr 2024 02:58:21 GMT
    Pouria Rouzrokh, Shahriar Faghani, Cooper U. Gamble, Moein Shariatnia, Bradley J. Erickson
    

Retrieval-augmented generation (RAG) frameworks enable large language models (LLMs) to retrieve relevant information from a knowledge base and incorporate it into the context for generating responses. This mitigates hallucinations and allows for the updating of knowledge without retraining the LLM. However, RAG does not guarantee valid responses if retrieval fails to identify the necessary information as the context for response generation. Also, if there is contradictory content, the RAG response will likely reflect only one of the two possible responses. Therefore, quantifying uncertainty in the retrieval process is crucial for ensuring RAG trustworthiness. In this report, we introduce a four-step framework for applying conformal prediction to quantify retrieval uncertainty in RAG frameworks. First, a calibration set of questions answerable from the knowledge base is constructed. Each question's embedding is compared against document embeddings to identify the most relevant document chunks containing the answer and record their similarity scores. Given a user-specified error rate ({alpha}), these similarity scores are then analyzed to determine a similarity score cutoff threshold. During inference, all chunks with similarity exceeding this threshold are retrieved to provide context to the LLM, ensuring the true answer is captured in the context with a (1-{alpha}) confidence level. We provide a Python package that enables users to implement the entire workflow proposed in our work, only using LLMs and without human intervention.

* * *

[[2404.04292] Conversational Disease Diagnosis via External Planner-Controlled Large Language Models](https://arxiv.org/abs/2404.04292)
    
    
    Thu, 4 Apr 2024 06:16:35 GMT
    Zhoujian Sun, Cheng Luo, Zhengxing Huang
    

The advancement of medical artificial intelligence (AI) has set the stage for the realization of conversational diagnosis, where AI systems mimic human doctors by engaging in dialogue with patients to deduce diagnoses. This study introduces an innovative approach using external planners augmented with large language models (LLMs) to develop a medical task-oriented dialogue system. This system comprises a policy module for information gathering, a LLM based module for natural language understanding and generation, addressing the limitations of previous AI systems in these areas. By emulating the two-phase decision-making process of doctors disease screening and differential diagnosis. we designed two distinct planners. The first focuses on collecting patient symptoms to identify potential diseases, while the second delves into specific inquiries to confirm or exclude these diseases. Utilizing reinforcement learning and active learning with LLMs, we trained these planners to navigate medical dialogues effectively. Our evaluation on the MIMIC-IV dataset demonstrated the system's capability to outperform existing models, indicating a significant step towards achieving automated conversational disease diagnostics and enhancing the precision and accessibility of medical diagnoses.

* * *

[[2404.04293] Reason from Fallacy: Enhancing Large Language Models' Logical Reasoning through Logical Fallacy Understanding](https://arxiv.org/abs/2404.04293)
    
    
    Thu, 4 Apr 2024 08:38:03 GMT
    Yanda Li, Dixuan Wang, Jiaqing Liang, Guochao Jiang, Qianyu He, Yanghua Xiao, Deqing Yang
    

Large Language Models (LLMs) have demonstrated good performance in many reasoning tasks, but they still struggle with some complicated reasoning tasks including logical reasoning. One non-negligible reason for LLMs' suboptimal performance on logical reasoning is their overlooking of understanding logical fallacies correctly. To evaluate LLMs' capability of logical fallacy understanding (LFU), we propose five concrete tasks from three cognitive dimensions of WHAT, WHY, and HOW in this paper. Towards these LFU tasks, we have successfully constructed a new dataset LFUD based on GPT-4 accompanied by a little human effort. Our extensive experiments justify that our LFUD can be used not only to evaluate LLMs' LFU capability, but also to fine-tune LLMs to obtain significantly enhanced performance on logical reasoning.

* * *

[[2404.04302] CBR-RAG: Case-Based Reasoning for Retrieval Augmented Generation in LLMs for Legal Question Answering](https://arxiv.org/abs/2404.04302)
    
    
    Thu, 4 Apr 2024 21:47:43 GMT
    Nirmalie Wiratunga, Ramitha Abeyratne, Lasal Jayawardena, Kyle Martin, Stewart Massie, Ikechukwu Nkisi-Orji, Ruvan Weerasinghe, Anne Liret, Bruno Fleisch
    

Retrieval-Augmented Generation (RAG) enhances Large Language Model (LLM) output by providing prior knowledge as context to input. This is beneficial for knowledge-intensive and expert reliant tasks, including legal question-answering, which require evidence to validate generated text outputs. We highlight that Case-Based Reasoning (CBR) presents key opportunities to structure retrieval as part of the RAG process in an LLM. We introduce CBR-RAG, where CBR cycle's initial retrieval stage, its indexing vocabulary, and similarity knowledge containers are used to enhance LLM queries with contextually relevant cases. This integration augments the original LLM query, providing a richer prompt. We present an evaluation of CBR-RAG, and examine different representations (i.e. general and domain-specific embeddings) and methods of comparison (i.e. inter, intra and hybrid similarity) on the task of legal question-answering. Our results indicate that the context provided by CBR's case reuse enforces similarity between relevant components of the questions and the evidence base leading to significant improvements in the quality of generated answers.

* * *

[[2404.04332] Scope Ambiguities in Large Language Models](https://arxiv.org/abs/2404.04332)
    
    
    Fri, 5 Apr 2024 18:01:02 GMT
    Gaurav Kamath, Sebastian Schuster, Sowmya Vajjala and Siva Reddy
    

Sentences containing multiple semantic operators with overlapping scope often create ambiguities in interpretation, known as scope ambiguities. These ambiguities offer rich insights into the interaction between semantic structure and world knowledge in language processing. Despite this, there has been little research into how modern large language models treat them. In this paper, we investigate how different versions of certain autoregressive language models -- GPT-2, GPT-3/3.5, Llama 2 and GPT-4 -- treat scope ambiguous sentences, and compare this with human judgments. We introduce novel datasets that contain a joint total of almost 1,000 unique scope-ambiguous sentences, containing interactions between a range of semantic operators, and annotated for human judgments. Using these datasets, we find evidence that several models (i) are sensitive to the meaning ambiguity in these sentences, in a way that patterns well with human judgments, and (ii) can successfully identify human-preferred readings at a high level of accuracy (over 90% in some cases).

* * *

[[2404.04351] Assisting humans in complex comparisons: automated information comparison at scale](https://arxiv.org/abs/2404.04351)
    
    
    Fri, 5 Apr 2024 18:44:54 GMT
    Truman Yuen, Graham A. Watt, Yuri Lawryshyn
    

Generative Large Language Models enable efficient analytics across knowledge domains, rivalling human experts in information comparisons. However, the applications of LLMs for information comparisons face scalability challenges due to the difficulties in maintaining information across large contexts and overcoming model token limitations. To address these challenges, we developed the novel Abstractive Summarization & Criteria-driven Comparison Endpoint (ASC$^2$End) system to automate information comparison at scale. Our system employs Semantic Text Similarity comparisons for generating evidence-supported analyses. We utilize proven data-handling strategies such as abstractive summarization and retrieval augmented generation to overcome token limitations and retain relevant information during model inference. Prompts were designed using zero-shot strategies to contextualize information for improved model reasoning. We evaluated abstractive summarization using ROUGE scoring and assessed the generated comparison quality using survey responses. Models evaluated on the ASC$^2$End system show desirable results providing insights on the expected performance of the system. ASC$^2$End is a novel system and tool that enables accurate, automated information comparison at scale across knowledge domains, overcoming limitations in context length and retrieval.

* * *

[[2404.04361] Deciphering Political Entity Sentiment in News with Large Language Models: Zero-Shot and Few-Shot Strategies](https://arxiv.org/abs/2404.04361)
    
    
    Fri, 5 Apr 2024 19:14:38 GMT
    Alapan Kuila, Sudeshna Sarkar
    

Sentiment analysis plays a pivotal role in understanding public opinion, particularly in the political domain where the portrayal of entities in news articles influences public perception. In this paper, we investigate the effectiveness of Large Language Models (LLMs) in predicting entity-specific sentiment from political news articles. Leveraging zero-shot and few-shot strategies, we explore the capability of LLMs to discern sentiment towards political entities in news content. Employing a chain-of-thought (COT) approach augmented with rationale in few-shot in-context learning, we assess whether this method enhances sentiment prediction accuracy. Our evaluation on sentiment-labeled datasets demonstrates that LLMs, outperform fine-tuned BERT models in capturing entity-specific sentiment. We find that learning in-context significantly improves model performance, while the self-consistency mechanism enhances consistency in sentiment prediction. Despite the promising results, we observe inconsistencies in the effectiveness of the COT prompting method. Overall, our findings underscore the potential of LLMs in entity-centric sentiment analysis within the political news domain and highlight the importance of suitable prompting strategies and model architectures.

* * *

[[2404.04510] IITK at SemEval-2024 Task 2: Exploring the Capabilities of LLMs for Safe Biomedical Natural Language Inference for Clinical Trials](https://arxiv.org/abs/2404.04510)
    
    
    Sat, 6 Apr 2024 05:44:53 GMT
    Shreyasi Mandal and Ashutosh Modi
    

Large Language models (LLMs) have demonstrated state-of-the-art performance in various natural language processing (NLP) tasks across multiple domains, yet they are prone to shortcut learning and factual inconsistencies. This research investigates LLMs' robustness, consistency, and faithful reasoning when performing Natural Language Inference (NLI) on breast cancer Clinical Trial Reports (CTRs) in the context of SemEval 2024 Task 2: Safe Biomedical Natural Language Inference for Clinical Trials. We examine the reasoning capabilities of LLMs and their adeptness at logical problem-solving. A comparative analysis is conducted on pre-trained language models (PLMs), GPT-3.5, and Gemini Pro under zero-shot settings using Retrieval-Augmented Generation (RAG) framework, integrating various reasoning chains. The evaluation yields an F1 score of 0.69, consistency of 0.71, and a faithfulness score of 0.90 on the test dataset.

* * *

[[2404.04514] Joint Visual and Text Prompting for Improved Object-Centric Perception with Multimodal Large Language Models](https://arxiv.org/abs/2404.04514)
    
    
    Sat, 6 Apr 2024 05:59:02 GMT
    Songtao Jiang, Yan Zhang, Chenyi Zhou, Yeying Jin, Yang Feng, Jian Wu, Zuozhu Liu
    

Multimodal Large Language Models (MLLMs) such as GPT-4V and Gemini Pro face challenges in achieving human-level perception in Visual Question Answering (VQA), particularly in object-oriented perception tasks which demand fine-grained understanding of object identities, locations or attributes, as indicated by empirical findings. This is mainly due to their limited capability to effectively integrate complex visual cues with textual information and potential object hallucinations. In this paper, we present a novel approach, Joint Visual and Text Prompting (VTPrompt), that employs fine-grained visual information to enhance the capability of MLLMs in VQA, especially for object-oriented perception. VTPrompt merges visual and text prompts to extract key concepts from textual questions and employs a detection model to highlight relevant objects as visual prompts in images. The processed images alongside text prompts are subsequently fed into MLLMs to produce more accurate answers. Our experiments with GPT-4V and Gemini Pro, on three benchmarks, i.e., MME , MMB and POPE, demonstrate significant improvements. Particularly, our method led to a score improvement of up to 183.5 for GPT-4V on MME and enhanced MMB performance by 8.17% for GPT-4V and 15.69% for Gemini Pro.

* * *

[[2404.04522] Q-PEFT: Query-dependent Parameter Efficient Fine-tuning for Text Reranking with Large Language Models](https://arxiv.org/abs/2404.04522)
    
    
    Sat, 6 Apr 2024 06:44:41 GMT
    Zhiyuan Peng, Xuyang Wu, Qifan Wang, Sravanthi Rajanala, Yi Fang
    

Parameter Efficient Fine-Tuning (PEFT) methods have been extensively utilized in Large Language Models (LLMs) to improve the down-streaming tasks without the cost of fine-tuing the whole LLMs. Recent studies have shown how to effectively use PEFT for fine-tuning LLMs in ranking tasks with convincing performance; there are some limitations, including the learned prompt being fixed for different documents, overfitting to specific tasks, and low adaptation ability. In this paper, we introduce a query-dependent parameter efficient fine-tuning (Q-PEFT) approach for text reranking to leak the information of the true queries to LLMs and then make the generation of true queries from input documents much easier. Specifically, we utilize the query to extract the top-$k$ tokens from concatenated documents, serving as contextual clues. We further augment Q-PEFT by substituting the retrieval mechanism with a multi-head attention layer to achieve end-to-end training and cover all the tokens in the documents, guiding the LLMs to generate more document-specific synthetic queries, thereby further improving the reranking performance. Extensive experiments are conducted on four public datasets, demonstrating the effectiveness of our proposed approach.

* * *

[[2404.04626] Towards Analyzing and Understanding the Limitations of DPO: A Theoretical Perspective](https://arxiv.org/abs/2404.04626)
    
    
    Sat, 6 Apr 2024 13:24:37 GMT
    Duanyu Feng, Bowen Qin, Chen Huang, Zheng Zhang, Wenqiang Lei
    

Direct Preference Optimization (DPO), which derives reward signals directly from pairwise preference data, has shown its effectiveness on aligning Large Language Models (LLMs) with human preferences. Despite its widespread use across various tasks, DPO has been criticized for its sensitivity to the SFT's effectiveness and its hindrance to the learning capacity towards human-preferred responses, leading to less satisfactory performance. To overcome those limitations, the theoretical understanding of DPO are indispensable but still lacking. To this end, we take a step towards theoretically analyzing and understanding the limitations of DPO. Specifically, we provide an analytical framework using the field theory to analyze the optimization process of DPO. By analyzing the gradient vector field of the DPO loss function, we find that the DPO loss function decreases the probability of producing human dispreferred data at a faster rate than it increases the probability of producing preferred data. This provides theoretical insights for understanding the limitations of DPO discovered in the related research experiments, thereby setting the foundation for its improvement.

* * *

[[2404.04631] On the Limitations of Large Language Models (LLMs): False Attribution](https://arxiv.org/abs/2404.04631)
    
    
    Sat, 6 Apr 2024 13:38:15 GMT
    Tosin Adewumi, Nudrat Habib, Lama Alkhaled and Elisa Barney
    

In this work, we provide insight into one important limitation of large language models (LLMs), i.e. false attribution, and introduce a new hallucination metric - Simple Hallucination Index (SHI). The task of automatic author attribution for relatively small chunks of text is an important NLP task but can be challenging. We empirically evaluate the power of 3 open SotA LLMs in zero-shot setting (LLaMA-2-13B, Mixtral 8x7B, and Gemma-7B), especially as human annotation can be costly. We collected the top 10 most popular books, according to Project Gutenberg, divided each one into equal chunks of 400 words, and asked each LLM to predict the author. We then randomly sampled 162 chunks for human evaluation from each of the annotated books, based on the error margin of 7% and a confidence level of 95% for the book with the most chunks (Great Expectations by Charles Dickens, having 922 chunks). The average results show that Mixtral 8x7B has the highest prediction accuracy, the lowest SHI, and a Pearson's correlation (r) of 0.737, 0.249, and -0.9996, respectively, followed by LLaMA-2-13B and Gemma-7B. However, Mixtral 8x7B suffers from high hallucinations for 3 books, rising as high as an SHI of 0.87 (in the range 0-1, where 1 is the worst). The strong negative correlation of accuracy and SHI, given by r, demonstrates the fidelity of the new hallucination metric, which is generalizable to other tasks. We publicly release the annotated chunks of data and our codes to aid the reproducibility and evaluation of other models.

* * *

[[2404.04659] Multilingual Pretraining and Instruction Tuning Improve Cross-Lingual Knowledge Alignment, But Only Shallowly](https://arxiv.org/abs/2404.04659)
    
    
    Sat, 6 Apr 2024 15:25:06 GMT
    Changjiang Gao, Hongda Hu, Peng Hu, Jiajun Chen, Jixing Li, Shujian Huang
    

Despite their strong ability to retrieve knowledge in English, current large language models show imbalance abilities in different languages. Two approaches are proposed to address this, i.e., multilingual pretraining and multilingual instruction tuning. However, whether and how do such methods contribute to the cross-lingual knowledge alignment inside the models is unknown. In this paper, we propose CLiKA, a systematic framework to assess the cross-lingual knowledge alignment of LLMs in the Performance, Consistency and Conductivity levels, and explored the effect of multilingual pretraining and instruction tuning on the degree of alignment. Results show that: while both multilingual pretraining and instruction tuning are beneficial for cross-lingual knowledge alignment, the training strategy needs to be carefully designed. Namely, continued pretraining improves the alignment of the target language at the cost of other languages, while mixed pretraining affect other languages less. Also, the overall cross-lingual knowledge alignment, especially in the conductivity level, is unsatisfactory for all tested LLMs, and neither multilingual pretraining nor instruction tuning can substantially improve the cross-lingual knowledge conductivity.

* * *

[[2404.04671] Inferring the Phylogeny of Large Language Models and Predicting their Performances in Benchmarks](https://arxiv.org/abs/2404.04671)
    
    
    Sat, 6 Apr 2024 16:16:30 GMT
    Nicolas Yax, Pierre-Yves Oudeyer, Stefano Palminteri
    

This paper introduces PhyloLM, a method applying phylogenetic algorithms to Large Language Models to explore their finetuning relationships, and predict their performance characteristics. By leveraging the phylogenetic distance metric, we construct dendrograms, which satisfactorily capture distinct LLM families (across a set of 77 open-source and 22 closed models). Furthermore, phylogenetic distance predicts performances in benchmarks (we test MMLU and ARC), thus enabling a time and cost-effective estimation of LLM capabilities. The approach translates genetic concepts to machine learning, offering tools to infer LLM development, relationships, and capabilities, even in the absence of transparent training information.

* * *

[[2404.04676] Order-Based Pre-training Strategies for Procedural Text Understanding](https://arxiv.org/abs/2404.04676)
    
    
    Sat, 6 Apr 2024 16:39:07 GMT
    Abhilash Nandy, Yash Kulkarni, Pawan Goyal, Niloy Ganguly
    

In this paper, we propose sequence-based pretraining methods to enhance procedural understanding in natural language processing. Procedural text, containing sequential instructions to accomplish a task, is difficult to understand due to the changing attributes of entities in the context. We focus on recipes, which are commonly represented as ordered instructions, and use this order as a supervision signal. Our work is one of the first to compare several 'order as-supervision' transformer pre-training methods, including Permutation Classification, Embedding Regression, and Skip-Clip, and shows that these methods give improved results compared to the baselines and SoTA LLMs on two downstream Entity-Tracking datasets: NPN-Cooking dataset in recipe domain and ProPara dataset in open domain. Our proposed methods address the non-trivial Entity Tracking Task that requires prediction of entity states across procedure steps, which requires understanding the order of steps. These methods show an improvement over the best baseline by 1.6% and 7-9% on NPN-Cooking and ProPara Datasets respectively across metrics.

* * *

[[2404.04722] PoLLMgraph: Unraveling Hallucinations in Large Language Models via State Transition Dynamics](https://arxiv.org/abs/2404.04722)
    
    
    Sat, 6 Apr 2024 20:02:20 GMT
    Derui Zhu, Dingfan Chen, Qing Li, Zongxiong Chen, Lei Ma, Jens Grossklags, Mario Fritz
    

Despite tremendous advancements in large language models (LLMs) over recent years, a notably urgent challenge for their practical deployment is the phenomenon of hallucination, where the model fabricates facts and produces non-factual statements. In response, we propose PoLLMgraph, a Polygraph for LLMs, as an effective model-based white-box detection and forecasting approach. PoLLMgraph distinctly differs from the large body of existing research that concentrates on addressing such challenges through black-box evaluations. In particular, we demonstrate that hallucination can be effectively detected by analyzing the LLM's internal state transition dynamics during generation via tractable probabilistic models. Experimental results on various open-source LLMs confirm the efficacy of PoLLMgraph, outperforming state-of-the-art methods by a considerable margin, evidenced by over 20% improvement in AUC-ROC on common benchmarking datasets like TruthfulQA. Our work paves a new way for model-based white-box analysis of LLMs, motivating the research community to further explore, understand, and refine the intricate dynamics of LLM behaviors.

* * *

[[2404.04728] Navigating the Landscape of Hint Generation Research: From the Past to the Future](https://arxiv.org/abs/2404.04728)
    
    
    Sat, 6 Apr 2024 20:42:46 GMT
    Anubhav Jangra, Jamshid Mozafari, Adam Jatowt, Smaranda Muresan
    

Digital education has gained popularity in the last decade, especially after the COVID-19 pandemic. With the improving capabilities of large language models to reason and communicate with users, envisioning intelligent tutoring systems (ITSs) that can facilitate self-learning is not very far-fetched. One integral component to fulfill this vision is the ability to give accurate and effective feedback via hints to scaffold the learning process. In this survey article, we present a comprehensive review of prior research on hint generation, aiming to bridge the gap between research in education and cognitive science, and research in AI and Natural Language Processing. Informed by our findings, we propose a formal definition of the hint generation task, and discuss the roadmap of building an effective hint generation system aligned with the formal definition, including open challenges, future directions and ethical considerations.

* * *

[[2404.04748] Multilingual Brain Surgeon: Large Language Models Can be Compressed Leaving No Language Behind](https://arxiv.org/abs/2404.04748)
    
    
    Sat, 6 Apr 2024 22:16:32 GMT
    Hongchuan Zeng, Hongshen Xu, Lu Chen, Kai Yu
    

Large Language Models (LLMs) have ushered in a new era in Natural Language Processing, but their massive size demands effective compression techniques for practicality. Although numerous model compression techniques have been investigated, they typically rely on a calibration set that overlooks the multilingual context and results in significant accuracy degradation for low-resource languages. This paper introduces Multilingual Brain Surgeon (MBS), a novel calibration data sampling method for multilingual LLMs compression. MBS overcomes the English-centric limitations of existing methods by sampling calibration data from various languages proportionally to the language distribution of the model training datasets. Our experiments, conducted on the BLOOM multilingual LLM, demonstrate that MBS improves the performance of existing English-centric compression methods, especially for low-resource languages. We also uncover the dynamics of language interaction during compression, revealing that the larger the proportion of a language in the training set and the more similar the language is to the calibration language, the better performance the language retains after compression. In conclusion, MBS presents an innovative approach to compressing multilingual LLMs, addressing the performance disparities and improving the language inclusivity of existing compression techniques.

* * *

[[2404.04809] Low-Resource Machine Translation through Retrieval-Augmented LLM Prompting: A Study on the Mambai Language](https://arxiv.org/abs/2404.04809)
    
    
    Sun, 7 Apr 2024 05:04:38 GMT
    Rapha\"el Merx, Aso Mahmudi, Katrina Langford, Leo Alberto de Araujo, Ekaterina Vylomova
    

This study explores the use of large language models (LLMs) for translating English into Mambai, a low-resource Austronesian language spoken in Timor-Leste, with approximately 200,000 native speakers. Leveraging a novel corpus derived from a Mambai language manual and additional sentences translated by a native speaker, we examine the efficacy of few-shot LLM prompting for machine translation (MT) in this low-resource context. Our methodology involves the strategic selection of parallel sentences and dictionary entries for prompting, aiming to enhance translation accuracy, using open-source and proprietary LLMs (LlaMa 2 70b, Mixtral 8x7B, GPT-4). We find that including dictionary entries in prompts and a mix of sentences retrieved through TF-IDF and semantic embeddings significantly improves translation quality. However, our findings reveal stark disparities in translation performance across test sets, with BLEU scores reaching as high as 21.2 on materials from the language manual, in contrast to a maximum of 4.4 on a test set provided by a native speaker. These results underscore the importance of diverse and representative corpora in assessing MT for low-resource languages. Our research provides insights into few-shot LLM prompting for low-resource MT, and makes available an initial corpus for the Mambai language.

* * *

[[2404.04817] FRACTAL: Fine-Grained Scoring from Aggregate Text Labels](https://arxiv.org/abs/2404.04817)
    
    
    Sun, 7 Apr 2024 05:54:28 GMT
    Yukti Makhija and Priyanka Agrawal and Rishi Saket and Aravindan Raghuveer
    

Large language models (LLMs) are being increasingly tuned to power complex generation tasks such as writing, fact-seeking, querying and reasoning. Traditionally, human or model feedback for evaluating and further tuning LLM performance has been provided at the response level, enabling faster and more cost-effective assessments. However, recent works (Amplayo et al. [2022], Wu et al. [2023]) indicate that sentence-level labels may provide more accurate and interpretable feedback for LLM optimization. In this work, we introduce methods to disaggregate response-level labels into sentence-level (pseudo-)labels. Our approach leverages multiple instance learning (MIL) and learning from label proportions (LLP) techniques in conjunction with prior information (e.g., document-sentence cosine similarity) to train a specialized model for sentence-level scoring. We also employ techniques which use model predictions to pseudo-label the train-set at the sentence-level for model training to further improve performance. We conduct extensive evaluations of our methods across six datasets and four tasks: retrieval, question answering, summarization, and math reasoning. Our results demonstrate improved performance compared to multiple baselines across most of these tasks. Our work is the first to develop response-level feedback to sentence-level scoring techniques, leveraging sentence-level prior information, along with comprehensive evaluations on multiple tasks as well as end-to-end finetuning evaluation showing performance comparable to a model trained on fine-grained human annotated labels.

* * *

[[2404.04838] Data Bias According to Bipol: Men are Naturally Right and It is the Role of Women to Follow Their Lead](https://arxiv.org/abs/2404.04838)
    
    
    Sun, 7 Apr 2024 07:24:45 GMT
    Irene Pagliai, Goya van Boven, Tosin Adewumi, Lama Alkhaled, Namrata Gurung, Isabella S\"odergren and Elisa Barney
    

We introduce new large labeled datasets on bias in 3 languages and show in experiments that bias exists in all 10 datasets of 5 languages evaluated, including benchmark datasets on the English GLUE/SuperGLUE leaderboards. The 3 new languages give a total of almost 6 million labeled samples and we benchmark on these datasets using SotA multilingual pretrained models: mT5 and mBERT. The challenge of social bias, based on prejudice, is ubiquitous, as recent events with AI and large language models (LLMs) have shown. Motivated by this challenge, we set out to estimate bias in multiple datasets. We compare some recent bias metrics and use bipol, which has explainability in the metric. We also confirm the unverified assumption that bias exists in toxic comments by randomly sampling 200 samples from a toxic dataset population using the confidence level of 95% and error margin of 7%. Thirty gold samples were randomly distributed in the 200 samples to secure the quality of the annotation. Our findings confirm that many of the datasets have male bias (prejudice against women), besides other types of bias. We publicly release our new datasets, lexica, models, and codes.

* * *

[[2404.04850] Lucky 52: How Many Languages Are Needed to Instruction Fine-Tune Large Language Models?](https://arxiv.org/abs/2404.04850)
    
    
    Sun, 7 Apr 2024 07:44:33 GMT
    Shaoxiong Ji and Pinzhen Chen
    

Fine-tuning large language models for multilingual downstream tasks requires a diverse set of languages to capture the nuances and structures of different linguistic contexts effectively. While the specific number varies depending on the desired scope and target languages, we argue that the number of languages, language exposure, and similarity that incorporate the selection of languages for fine-tuning are some important aspects to examine. By fine-tuning large multilingual models on 1 to 52 languages, this paper answers one question: How many languages are needed in instruction fine-tuning for multilingual tasks? We investigate how multilingual instruction fine-tuned models behave on multilingual benchmarks with an increasing number of languages and discuss our findings from the perspective of language exposure and similarity.

* * *

[[2404.04900] Radial Networks: Dynamic Layer Routing for High-Performance Large Language Models](https://arxiv.org/abs/2404.04900)
    
    
    Sun, 7 Apr 2024 09:52:31 GMT
    Jordan Dotzel, Yash Akhauri, Ahmed S. AbouElhamayed, Carly Jiang, Mohamed Abdelfattah, Zhiru Zhang
    

Large language models (LLMs) often struggle with strict memory, latency, and power demands. To meet these demands, various forms of dynamic sparsity have been proposed that reduce compute on an input-by-input basis. These methods improve over static methods by exploiting the variance across individual inputs, which has steadily grown with the exponential increase in training data. Yet, the increasing depth within modern models, currently with hundreds of layers, has opened opportunities for dynamic layer sparsity, which skips the computation for entire layers. In this work, we explore the practicality of layer sparsity by profiling residual connections and establish the relationship between model depth and layer sparsity. For example, the residual blocks in the OPT-66B model have a median contribution of 5% to its output. We then take advantage of this dynamic sparsity and propose Radial Networks, which perform token-level routing between layers guided by a trained router module. These networks can be used in a post-training distillation from sequential networks or trained from scratch to co-learn the router and layer weights. They enable scaling to larger model sizes by decoupling the number of layers from the dynamic depth of the network, and their design allows for layer reuse. By varying the compute token by token, they reduce the overall resources needed for generating entire sequences. Overall, this leads to larger capacity networks with significantly lower compute and serving costs for large language models.

* * *

[[2404.04925] Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontiers](https://arxiv.org/abs/2404.04925)
    
    
    Sun, 7 Apr 2024 11:52:44 GMT
    Libo Qin, Qiguang Chen, Yuhang Zhou, Zhi Chen, Yinghui Li, Lizi Liao, Min Li, Wanxiang Che, Philip S. Yu
    

Multilingual Large Language Models are capable of using powerful Large Language Models to handle and respond to queries in multiple languages, which achieves remarkable success in multilingual natural language processing tasks. Despite these breakthroughs, there still remains a lack of a comprehensive survey to summarize existing approaches and recent developments in this field. To this end, in this paper, we present a thorough review and provide a unified perspective to summarize the recent progress as well as emerging trends in multilingual large language models (MLLMs) literature. The contributions of this paper can be summarized: (1) First survey: to our knowledge, we take the first step and present a thorough review in MLLMs research field according to multi-lingual alignment; (2) New taxonomy: we offer a new and unified perspective to summarize the current progress of MLLMs; (3) New frontiers: we highlight several emerging frontiers and discuss the corresponding challenges; (4) Abundant resources: we collect abundant open-source resources, including relevant papers, data corpora, and leaderboards. We hope our work can provide the community with quick access and spur breakthrough research in MLLMs.

* * *

[[2404.04941] Prompting Large Language Models for Zero-shot Essay Scoring via Multi-trait Specialization](https://arxiv.org/abs/2404.04941)
    
    
    Sun, 7 Apr 2024 12:25:35 GMT
    Sanwoo Lee, Yida Cai, Desong Meng, Ziyang Wang, Yunfang Wu
    

Advances in automated essay scoring (AES) have traditionally relied on labeled essays, requiring tremendous cost and expertise for their acquisition. Recently, large language models (LLMs) have achieved great success in various tasks, but their potential is less explored in AES. In this paper, we propose Multi Trait Specialization (MTS), a zero-shot prompting framework to elicit essay scoring capabilities in LLMs. Specifically, we leverage ChatGPT to decompose writing proficiency into distinct traits and generate scoring criteria for each trait. Then, an LLM is prompted to extract trait scores from several conversational rounds, each round scoring one of the traits based on the scoring criteria. Finally, we derive the overall score via trait averaging and min-max scaling. Experimental results on two benchmark datasets demonstrate that MTS consistently outperforms straightforward prompting (Vanilla) in average QWK across all LLMs and datasets, with maximum gains of 0.437 on TOEFL11 and 0.355 on ASAP. Additionally, with the help of MTS, the small-sized Llama2-13b-chat substantially outperforms ChatGPT, facilitating an effective deployment in real applications.

* * *

[[2404.04949] SilverSight: A Multi-Task Chinese Financial Large Language Model Based on Adaptive Semantic Space Learning](https://arxiv.org/abs/2404.04949)
    
    
    Sun, 7 Apr 2024 13:02:21 GMT
    Yuhang Zhou and Zeping Li and Siyu Tian and Yuchen Ni and Sen Liu and Guangnan Ye and Hongfeng Chai
    

Large language models (LLMs) are increasingly being applied across various specialized fields, leveraging their extensive knowledge to empower a multitude of scenarios within these domains. However, each field encompasses a variety of specific tasks that require learning, and the diverse, heterogeneous data across these domains can lead to conflicts during model task transfer. In response to this challenge, our study introduces an Adaptive Semantic Space Learning (ASSL) framework, which utilizes the adaptive reorganization of data distributions within the semantic space to enhance the performance and selection efficacy of multi-expert models. Utilizing this framework, we trained a financial multi-task LLM named "SilverSight". Our research findings demonstrate that our framework can achieve results close to those obtained with full data training using only 10% of the data, while also exhibiting strong generalization capabilities.

* * *

[[2404.04963] SemEval-2024 Task 2: Safe Biomedical Natural Language Inference for Clinical Trials](https://arxiv.org/abs/2404.04963)
    
    
    Sun, 7 Apr 2024 13:58:41 GMT
    Mael Jullien, Marco Valentino, Andr\'e Freitas
    

Large Language Models (LLMs) are at the forefront of NLP achievements but fall short in dealing with shortcut learning, factual inconsistency, and vulnerability to adversarial inputs.These shortcomings are especially critical in medical contexts, where they can misrepresent actual model capabilities. Addressing this, we present SemEval-2024 Task 2: Safe Biomedical Natural Language Inference for ClinicalTrials. Our contributions include the refined NLI4CT-P dataset (i.e., Natural Language Inference for Clinical Trials - Perturbed), designed to challenge LLMs with interventional and causal reasoning tasks, along with a comprehensive evaluation of methods and results for participant submissions. A total of 106 participants registered for the task contributing to over 1200 individual submissions and 25 system overview papers. This initiative aims to advance the robustness and applicability of NLI models in healthcare, ensuring safer and more dependable AI assistance in clinical decision-making. We anticipate that the dataset, models, and outcomes of this task can support future research in the field of biomedical NLI. The dataset, competition leaderboard, and website are publicly available.

* * *

[[2404.04990] MLaKE: Multilingual Knowledge Editing Benchmark for Large Language Models](https://arxiv.org/abs/2404.04990)
    
    
    Sun, 7 Apr 2024 15:23:28 GMT
    Zihao Wei, Jingcheng Deng, Liang Pang, Hanxing Ding, Huawei Shen, Xueqi Cheng
    

The extensive utilization of large language models (LLMs) underscores the crucial necessity for precise and contemporary knowledge embedded within their intrinsic parameters. Existing research on knowledge editing primarily concentrates on monolingual scenarios, neglecting the complexities presented by multilingual contexts and multi-hop reasoning. To address these challenges, our study introduces MLaKE (Multilingual Language Knowledge Editing), a novel benchmark comprising 4072 multi-hop and 5360 single-hop questions designed to evaluate the adaptability of knowledge editing methods across five languages: English, Chinese, Japanese, French, and German. MLaKE aggregates fact chains from Wikipedia across languages and utilizes LLMs to generate questions in both free-form and multiple-choice. We evaluate the multilingual knowledge editing generalization capabilities of existing methods on MLaKE. Existing knowledge editing methods demonstrate higher success rates in English samples compared to other languages. However, their generalization capabilities are limited in multi-language experiments. Notably, existing knowledge editing methods often show relatively high generalization for languages within the same language family compared to languages from different language families. These results underscore the imperative need for advancements in multilingual knowledge editing and we hope MLaKE can serve as a valuable resource for benchmarking and solution development.

* * *

[[2404.05143] Plug and Play with Prompts: A Prompt Tuning Approach for Controlling Text Generation](https://arxiv.org/abs/2404.05143)
    
    
    Mon, 8 Apr 2024 01:54:28 GMT
    Rohan Deepak Ajwani, Zining Zhu, Jonathan Rose, Frank Rudzicz
    

Transformer-based Large Language Models (LLMs) have shown exceptional language generation capabilities in response to text-based prompts. However, controlling the direction of generation via textual prompts has been challenging, especially with smaller models. In this work, we explore the use of Prompt Tuning to achieve controlled language generation. Generated text is steered using prompt embeddings, which are trained using a small language model, used as a discriminator. Moreover, we demonstrate that these prompt embeddings can be trained with a very small dataset, with as low as a few hundred training examples. Our method thus offers a data and parameter efficient solution towards controlling language model outputs. We carry out extensive evaluation on four datasets: SST-5 and Yelp (sentiment analysis), GYAFC (formality) and JIGSAW (toxic language). Finally, we demonstrate the efficacy of our method towards mitigating harmful, toxic, and biased text generated by language models.

* * *

[[2404.05144] Enhancing Clinical Efficiency through LLM: Discharge Note Generation for Cardiac Patients](https://arxiv.org/abs/2404.05144)
    
    
    Mon, 8 Apr 2024 01:55:28 GMT
    HyoJe Jung, Yunha Kim, Heejung Choi, Hyeram Seo, Minkyoung Kim, JiYe Han, Gaeun Kee, Seohyun Park, Soyoung Ko, Byeolhee Kim, Suyeon Kim, Tae Joon Jun, Young-Hak Kim
    

Medical documentation, including discharge notes, is crucial for ensuring patient care quality, continuity, and effective medical communication. However, the manual creation of these documents is not only time-consuming but also prone to inconsistencies and potential errors. The automation of this documentation process using artificial intelligence (AI) represents a promising area of innovation in healthcare. This study directly addresses the inefficiencies and inaccuracies in creating discharge notes manually, particularly for cardiac patients, by employing AI techniques, specifically large language model (LLM). Utilizing a substantial dataset from a cardiology center, encompassing wide-ranging medical records and physician assessments, our research evaluates the capability of LLM to enhance the documentation process. Among the various models assessed, Mistral-7B distinguished itself by accurately generating discharge notes that significantly improve both documentation efficiency and the continuity of care for patients. These notes underwent rigorous qualitative evaluation by medical expert, receiving high marks for their clinical relevance, completeness, readability, and contribution to informed decision-making and care planning. Coupled with quantitative analyses, these results confirm Mistral-7B's efficacy in distilling complex medical information into concise, coherent summaries. Overall, our findings illuminate the considerable promise of specialized LLM, such as Mistral-7B, in refining healthcare documentation workflows and advancing patient care. This study lays the groundwork for further integrating advanced AI technologies in healthcare, demonstrating their potential to revolutionize patient documentation and support better care outcomes.

* * *

[[2404.05160] Linguistic Changes in Spontaneous Speech for Detecting Parkinsons Disease Using Large Language Models](https://arxiv.org/abs/2404.05160)
    
    
    Mon, 8 Apr 2024 03:00:10 GMT
    Jonathan Crawford
    

Parkinsons disease is the second most prevalent neurodegenerative disorder with over ten million active cases worldwide and one million new diagnoses per year. Detecting and subsequently diagnosing the disease is challenging because of symptom heterogeneity with respect to complexity, as well as the type and timing of phenotypic manifestations. Typically, language impairment can present in the prodromal phase and precede motor symptoms suggesting that a linguistic-based approach could serve as a diagnostic method for incipient Parkinsons disease. Additionally, improved linguistic models may enhance other approaches through ensemble techniques. The field of large language models is advancing rapidly, presenting the opportunity to explore the use of these new models for detecting Parkinsons disease and to improve on current linguistic approaches with high-dimensional representations of linguistics. We evaluate the application of state-of-the-art large language models to detect Parkinsons disease automatically from spontaneous speech with up to 73% accuracy.

* * *

[[2404.05221] LLM Reasoners: New Evaluation, Library, and Analysis of Step-by-Step Reasoning with Large Language Models](https://arxiv.org/abs/2404.05221)
    
    
    Mon, 8 Apr 2024 06:35:09 GMT
    Shibo Hao, Yi Gu, Haotian Luo, Tianyang Liu, Xiyan Shao, Xinyuan Wang, Shuhua Xie, Haodi Ma, Adithya Samavedhi, Qiyue Gao, Zhen Wang, Zhiting Hu
    

Generating accurate step-by-step reasoning is essential for Large Language Models (LLMs) to address complex problems and enhance robustness and interpretability. Despite the flux of research on developing advanced reasoning approaches, systematically analyzing the diverse LLMs and reasoning strategies in generating reasoning chains remains a significant challenge. The difficulties stem from the lack of two key elements: (1) an automatic method for evaluating the generated reasoning chains on different tasks, and (2) a unified formalism and implementation of the diverse reasoning approaches for systematic comparison. This paper aims to close the gap: (1) We introduce AutoRace for fully automated reasoning chain evaluation. Existing metrics rely on expensive human annotations or pre-defined LLM prompts not adaptable to different tasks. In contrast, AutoRace automatically creates detailed evaluation criteria tailored for each task, and uses GPT-4 for accurate evaluation following the criteria. (2) We develop LLM Reasoners, a library for standardized modular implementation of existing and new reasoning algorithms, under a unified formulation of the search, reward, and world model components. With the new evaluation and library, (3) we conduct extensive study of different reasoning approaches (e.g., CoT, ToT, RAP). The analysis reveals interesting findings about different factors contributing to reasoning, including the reward-guidance, breadth-vs-depth in search, world model, and prompt formats, etc.

* * *

[[2404.05337] Towards Objectively Benchmarking Social Intelligence for Language Agents at Action Level](https://arxiv.org/abs/2404.05337)
    
    
    Mon, 8 Apr 2024 09:25:32 GMT
    Chenxu Wang, Bin Dai, Huaping Liu, and Baoyuan Wang
    

Prominent large language models have exhibited human-level performance in many domains, even enabling the derived agents to simulate human and social interactions. While practical works have substantiated the practicability of grounding language agents in sandbox simulation or embodied simulators, current social intelligence benchmarks either stay at the language level or use subjective metrics. In pursuit of a more realistic and objective evaluation, we introduce the Social Tasks in Sandbox Simulation (STSS) benchmark, which assesses language agents textbf{objectively} at the textbf{action level} by scrutinizing the goal achievements within the multi-agent simulation. Additionally, we sample conversation scenarios to build a language-level benchmark to provide an economically prudent preliminary evaluation and align with prevailing benchmarks. To gauge the significance of agent architecture, we implement a target-driven planning (TDP) module as an adjunct to the existing agent. Our evaluative findings highlight that the STSS benchmark is challenging for state-of-the-art language agents. Furthermore, it effectively discriminates between distinct language agents, suggesting its usefulness as a benchmark for evaluating both language models and agent architectures.

* * *

[[2404.05399] SafetyPrompts: a Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safety](https://arxiv.org/abs/2404.05399)
    
    
    Mon, 8 Apr 2024 10:57:25 GMT
    Paul R\"ottger, Fabio Pernisi, Bertie Vidgen, Dirk Hovy
    

The last two years have seen a rapid growth in concerns around the safety of large language models (LLMs). Researchers and practitioners have met these concerns by introducing an abundance of new datasets for evaluating and improving LLM safety. However, much of this work has happened in parallel, and with very different goals in mind, ranging from the mitigation of near-term risks around bias and toxic content generation to the assessment of longer-term catastrophic risk potential. This makes it difficult for researchers and practitioners to find the most relevant datasets for a given use case, and to identify gaps in dataset coverage that future work may fill. To remedy these issues, we conduct a first systematic review of open datasets for evaluating and improving LLM safety. We review 102 datasets, which we identified through an iterative and community-driven process over the course of several months. We highlight patterns and trends, such as a a trend towards fully synthetic datasets, as well as gaps in dataset coverage, such as a clear lack of non-English datasets. We also examine how LLM safety datasets are used in practice -- in LLM release publications and popular LLM benchmarks -- finding that current evaluation practices are highly idiosyncratic and make use of only a small fraction of available datasets. Our contributions are based on SafetyPrompts.com, a living catalogue of open datasets for LLM safety, which we commit to updating continuously as the field of LLM safety develops.

* * *

[[2404.05406] PerkwE_COQA: enhance Persian Conversational Question Answering by combining contextual keyword extraction with Large Language Models](https://arxiv.org/abs/2404.05406)
    
    
    Mon, 8 Apr 2024 11:14:58 GMT
    Pardis Moradbeiki, Nasser Ghadiri
    

Smart cities need the involvement of their residents to enhance quality of life. Conversational query-answering is an emerging approach for user engagement. There is an increasing demand of an advanced conversational question-answering that goes beyond classic systems. Existing approaches have shown that LLMs offer promising capabilities for CQA, but may struggle to capture the nuances of conversational contexts. The new approach involves understanding the content and engaging in a multi-step conversation with the user to fulfill their needs. This paper presents a novel method to elevate the performance of Persian Conversational question-answering (CQA) systems. It combines the strengths of Large Language Models (LLMs) with contextual keyword extraction. Our method extracts keywords specific to the conversational flow, providing the LLM with additional context to understand the user's intent and generate more relevant and coherent responses. We evaluated the effectiveness of this combined approach through various metrics, demonstrating significant improvements in CQA performance compared to an LLM-only baseline. The proposed method effectively handles implicit questions, delivers contextually relevant answers, and tackles complex questions that rely heavily on conversational context. The findings indicate that our method outperformed the evaluation benchmarks up to 8% higher than existing methods and the LLM-only baseline.

* * *

[[2404.05411] Know When To Stop: A Study of Semantic Drift in Text Generation](https://arxiv.org/abs/2404.05411)
    
    
    Mon, 8 Apr 2024 11:25:30 GMT
    Ava Spataru, Eric Hambro, Elena Voita and Nicola Cancedda
    

In this work, we explicitly show that modern LLMs tend to generate correct facts first, then "drift away" and generate incorrect facts later: this was occasionally observed but never properly measured. We develop a semantic drift score that measures the degree of separation between correct and incorrect facts in generated texts and confirm our hypothesis when generating Wikipedia-style biographies. This correct-then-incorrect generation pattern suggests that factual accuracy can be improved by knowing when to stop generation. Therefore, we explore the trade-off between information quantity and factual accuracy for several early stopping methods and manage to improve factuality by a large margin. We further show that reranking with semantic similarity can further improve these results, both compared to the baseline and when combined with early stopping. Finally, we try calling external API to bring the model back to the right generation path, but do not get positive results. Overall, our methods generalize and can be applied to any long-form text generation to produce more reliable information, by balancing trade-offs between factual accuracy, information quantity and computational cost.

* * *

[[2404.05415] Relation Extraction Using Large Language Models: A Case Study on Acupuncture Point Locations](https://arxiv.org/abs/2404.05415)
    
    
    Mon, 8 Apr 2024 11:33:00 GMT
    Yiming Li, Xueqing Peng, Jianfu Li, Xu Zuo, Suyuan Peng, Donghong Pei, Cui Tao, Hua Xu, Na Hong
    

In acupuncture therapy, the accurate location of acupoints is essential for its effectiveness. The advanced language understanding capabilities of large language models (LLMs) like Generative Pre-trained Transformers (GPT) present a significant opportunity for extracting relations related to acupoint locations from textual knowledge sources. This study aims to compare the performance of GPT with traditional deep learning models (Long Short-Term Memory (LSTM) and Bidirectional Encoder Representations from Transformers for Biomedical Text Mining (BioBERT)) in extracting acupoint-related location relations and assess the impact of pretraining and fine-tuning on GPT's performance. We utilized the World Health Organization Standard Acupuncture Point Locations in the Western Pacific Region (WHO Standard) as our corpus, which consists of descriptions of 361 acupoints. Five types of relations ('direction_of,' 'distance_of,' 'part_of,' 'near_acupoint,' and 'located_near') (n= 3,174) between acupoints were annotated. Five models were compared: BioBERT, LSTM, pre-trained GPT-3.5, and fine-tuned GPT-3.5, as well as pre-trained GPT-4. Performance metrics included micro-average exact match precision, recall, and F1 scores. Our results demonstrate that fine-tuned GPT-3.5 consistently outperformed other models in F1 scores across all relation types. Overall, it achieved the highest micro-average F1 score of 0.92. This study underscores the effectiveness of LLMs like GPT in extracting relations related to acupoint locations, with implications for accurately modeling acupuncture knowledge and promoting standard implementation in acupuncture training and practice. The findings also contribute to advancing informatics applications in traditional and complementary medicine, showcasing the potential of LLMs in natural language processing.

* * *

[[2404.05446] XL$^2$Bench: A Benchmark for Extremely Long Context Understanding with Long-range Dependencies](https://arxiv.org/abs/2404.05446)
    
    
    Mon, 8 Apr 2024 12:29:07 GMT
    Xuanfan Ni, Hengyi Cai, Xiaochi Wei, Shuaiqiang Wang, Dawei Yin, Piji Li
    

Large Language Models (LLMs) have demonstrated remarkable performance across diverse tasks but are constrained by their small context window sizes. Various efforts have been proposed to expand the context window to accommodate even up to 200K input tokens. Meanwhile, building high-quality benchmarks with much longer text lengths and more demanding tasks to provide comprehensive evaluations is of immense practical interest to facilitate long context understanding research of LLMs. However, prior benchmarks create datasets that ostensibly cater to long-text comprehension by expanding the input of traditional tasks, which falls short to exhibit the unique characteristics of long-text understanding, including long dependency tasks and longer text length compatible with modern LLMs' context window size. In this paper, we introduce a benchmark for extremely long context understanding with long-range dependencies, XL$^2$Bench, which includes three scenarios: Fiction Reading, Paper Reading, and Law Reading, and four tasks of increasing complexity: Memory Retrieval, Detailed Understanding, Overall Understanding, and Open-ended Generation, covering 27 subtasks in English and Chinese. It has an average length of 100K+ words (English) and 200K+ characters (Chinese). Evaluating six leading LLMs on XL$^2$Bench, we find that their performance significantly lags behind human levels. Moreover, the observed decline in performance across both the original and enhanced datasets underscores the efficacy of our approach to mitigating data contamination.

* * *

[[2404.05449] RoT: Enhancing Large Language Models with Reflection on Search Trees](https://arxiv.org/abs/2404.05449)
    
    
    Mon, 8 Apr 2024 12:31:23 GMT
    Wenyang Hui, Yan Wang, Kewei Tu, Chengyue Jiang
    

Large language models (LLMs) have demonstrated impressive capability in reasoning and planning when integrated with tree-search-based prompting methods. However, since these methods ignore the previous search experiences, they often make the same mistakes in the search process. To address this issue, we introduce Reflection on search Trees (RoT), an LLM reflection framework designed to improve the performance of tree-search-based prompting methods. It uses a strong LLM to summarize guidelines from previous tree search experiences to enhance the ability of a weak LLM. The guidelines are instructions about solving this task through tree search which can prevent the weak LLMs from making similar mistakes in the past search process. In addition, we proposed a novel state selection method, which identifies the critical information from historical search processes to help RoT generate more specific and meaningful guidelines. In our extensive experiments, we find that RoT significantly improves the performance of LLMs in reasoning or planning tasks with various tree-search-based prompting methods (e.g., BFS and MCTS). Non-tree-search-based prompting methods such as Chain-of-Thought (CoT) can also benefit from RoT guidelines since RoT can provide task-specific knowledge collected from the search experience.

* * *

[[2404.05483] PetKaz at SemEval-2024 Task 8: Can Linguistics Capture the Specifics of LLM-generated Text?](https://arxiv.org/abs/2404.05483)
    
    
    Mon, 8 Apr 2024 13:05:02 GMT
    Kseniia Petukhova, Roman Kazakov, Ekaterina Kochmar
    

In this paper, we present our submission to the SemEval-2024 Task 8 "Multigenerator, Multidomain, and Multilingual Black-Box Machine-Generated Text Detection", focusing on the detection of machine-generated texts (MGTs) in English. Specifically, our approach relies on combining embeddings from the RoBERTa-base with diversity features and uses a resampled training set. We score 12th from 124 in the ranking for Subtask A (monolingual track), and our results show that our approach is generalizable across unseen models and domains, achieving an accuracy of 0.91.

* * *

[[2404.05502] PetKaz at SemEval-2024 Task 3: Advancing Emotion Classification with an LLM for Emotion-Cause Pair Extraction in Conversations](https://arxiv.org/abs/2404.05502)
    
    
    Mon, 8 Apr 2024 13:25:03 GMT
    Roman Kazakov, Kseniia Petukhova, Ekaterina Kochmar
    

In this paper, we present our submission to the SemEval-2023 Task~3 "The Competition of Multimodal Emotion Cause Analysis in Conversations", focusing on extracting emotion-cause pairs from dialogs. Specifically, our approach relies on combining fine-tuned GPT-3.5 for emotion classification and a BiLSTM-based neural network to detect causes. We score 2nd in the ranking for Subtask 1, demonstrating the effectiveness of our approach through one of the highest weighted-average proportional F1 scores recorded at 0.264.

* * *

[[2404.05590] MedExpQA: Multilingual Benchmarking of Large Language Models for Medical Question Answering](https://arxiv.org/abs/2404.05590)
    
    
    Mon, 8 Apr 2024 15:03:57 GMT
    I\~nigo Alonso, Maite Oronoz, Rodrigo Agerri
    

Large Language Models (LLMs) have the potential of facilitating the development of Artificial Intelligence technology to assist medical experts for interactive decision support, which has been demonstrated by their competitive performances in Medical QA. However, while impressive, the required quality bar for medical applications remains far from being achieved. Currently, LLMs remain challenged by outdated knowledge and by their tendency to generate hallucinated content. Furthermore, most benchmarks to assess medical knowledge lack reference gold explanations which means that it is not possible to evaluate the reasoning of LLMs predictions. Finally, the situation is particularly grim if we consider benchmarking LLMs for languages other than English which remains, as far as we know, a totally neglected topic. In order to address these shortcomings, in this paper we present MedExpQA, the first multilingual benchmark based on medical exams to evaluate LLMs in Medical Question Answering. To the best of our knowledge, MedExpQA includes for the first time reference gold explanations written by medical doctors which can be leveraged to establish various gold-based upper-bounds for comparison with LLMs performance. Comprehensive multilingual experimentation using both the gold reference explanations and Retrieval Augmented Generation (RAG) approaches show that performance of LLMs still has large room for improvement, especially for languages other than English. Furthermore, and despite using state-of-the-art RAG methods, our results also demonstrate the difficulty of obtaining and integrating readily available medical knowledge that may positively impact results on downstream evaluations for Medical Question Answering. So far the benchmark is available in four languages, but we hope that this work may encourage further development to other languages.

* * *

[[2404.05624] LTNER: Large Language Model Tagging for Named Entity Recognition with Contextualized Entity Marking](https://arxiv.org/abs/2404.05624)
    
    
    Mon, 8 Apr 2024 15:54:02 GMT
    Faren Yan and Peng Yu and Xin Chen
    

The use of LLMs for natural language processing has become a popular trend in the past two years, driven by their formidable capacity for context comprehension and learning, which has inspired a wave of research from academics and industry professionals. However, for certain NLP tasks, such as NER, the performance of LLMs still falls short when compared to supervised learning methods. In our research, we developed a NER processing framework called LTNER that incorporates a revolutionary Contextualized Entity Marking Gen Method. By leveraging the cost-effective GPT-3.5 coupled with context learning that does not require additional training, we significantly improved the accuracy of LLMs in handling NER tasks. The F1 score on the CoNLL03 dataset increased from the initial 85.9% to 91.9%, approaching the performance of supervised fine-tuning. This outcome has led to a deeper understanding of the potential of LLMs.

* * *

[[2404.05632] Fighting crime with Transformers: Empirical analysis of address parsing methods in payment data](https://arxiv.org/abs/2404.05632)
    
    
    Mon, 8 Apr 2024 16:04:26 GMT
    Haitham Hammami, Louis Baligand, Bojan Petrovski
    

In the financial industry, identifying the location of parties involved in payments is a major challenge in the context of various regulatory requirements. For this purpose address parsing entails extracting fields such as street, postal code, or country from free text message attributes. While payment processing platforms are updating their standards with more structured formats such as SWIFT with ISO 20022, address parsing remains essential for a considerable volume of messages. With the emergence of Transformers and Generative Large Language Models (LLM), we explore the performance of state-of-the-art solutions given the constraint of processing a vast amount of daily data. This paper also aims to show the need for training robust models capable of dealing with real-world noisy transactional data. Our results suggest that a well fine-tuned Transformer model using early-stopping significantly outperforms other approaches. Nevertheless, generative LLMs demonstrate strong zero-shot performance and warrant further investigations.

* * *

[[2404.05692] Evaluating Mathematical Reasoning Beyond Accuracy](https://arxiv.org/abs/2404.05692)
    
    
    Mon, 8 Apr 2024 17:18:04 GMT
    Shijie Xia, Xuefeng Li, Yixin Liu, Tongshuang Wu, Pengfei Liu
    

The leaderboard of Large Language Models (LLMs) in mathematical tasks has been continuously updated. However, the majority of evaluations focus solely on the final results, neglecting the quality of the intermediate steps. This oversight can mask underlying problems, such as logical errors or unnecessary steps in the reasoning process. To measure reasoning beyond final-answer accuracy, we introduce ReasonEval, a new methodology for evaluating the quality of reasoning steps. ReasonEval employs $textit{validity}$ and $textit{redundancy}$ to characterize the reasoning quality, as well as accompanying LLMs to assess them automatically. Instantiated by base models that possess strong mathematical knowledge and trained with high-quality labeled data, ReasonEval achieves state-of-the-art performance on human-labeled datasets and can accurately detect different types of errors generated by perturbation. When applied to evaluate LLMs specialized in math, we find that an increase in final-answer accuracy does not necessarily guarantee an improvement in the overall quality of the reasoning steps for challenging mathematical problems. Additionally, we observe that ReasonEval can play a significant role in data selection. We release the best-performing model, meta-evaluation script, and all evaluation results at <https://github.com/GAIR-NLP/ReasonEval>.

* * *

[[2404.04291] Investigating Regularization of Self-Play Language Models](https://arxiv.org/abs/2404.04291)
    
    
    Thu, 4 Apr 2024 05:38:44 GMT
    Reda Alami, Abdalgader Abubaker, Mastane Achab, Mohamed El Amine Seddik, Salem Lahlou
    

This paper explores the effects of various forms of regularization in the context of language model alignment via self-play. While both reinforcement learning from human feedback (RLHF) and direct preference optimization (DPO) require to collect costly human-annotated pairwise preferences, the self-play fine-tuning (SPIN) approach replaces the rejected answers by data generated from the previous iterate. However, the SPIN method presents a performance instability issue in the learning phase, which can be mitigated by playing against a mixture of the two previous iterates. In the same vein, we propose in this work to address this issue from two perspectives: first, by incorporating an additional Kullback-Leibler (KL) regularization to stay at the proximity of the reference policy; second, by using the idea of fictitious play which smoothens the opponent policy across all previous iterations. In particular, we show that the KL-based regularizer boils down to replacing the previous policy by its geometric mixture with the base policy inside of the SPIN loss function. We finally discuss empirical results on MT-Bench as well as on the Hugging Face Open LLM Leaderboard.

* * *

[[2404.04360] Prompt Public Large Language Models to Synthesize Data for Private On-device Applications](https://arxiv.org/abs/2404.04360)
    
    
    Fri, 5 Apr 2024 19:14:14 GMT
    Shanshan Wu, Zheng Xu, Yanxiang Zhang, Yuanbo Zhang, Daniel Ramage
    

Pre-training on public data is an effective method to improve the performance for federated learning (FL) with differential privacy (DP). This paper investigates how large language models (LLMs) trained on public data can improve the quality of pre-training data for the on-device language models trained with DP and FL. We carefully design LLM prompts to filter and transform existing public data, and generate new data to resemble the real user data distribution. The model pre-trained on our synthetic dataset achieves relative improvement of 19.0% and 22.8% in next word prediction accuracy compared to the baseline model pre-trained on a standard public dataset, when evaluated over the real user data in Gboard (Google Keyboard, a production mobile keyboard application). Furthermore, our method achieves evaluation accuracy better than or comparable to the baseline during the DP FL fine-tuning over millions of mobile devices, and our final model outperforms the baseline in production A/B testing. Our experiments demonstrate the strengths of LLMs in synthesizing data close to the private distribution even without accessing the private data, and also suggest future research directions to further reduce the distribution gap.

* * *

[[2404.04475] Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators](https://arxiv.org/abs/2404.04475)
    
    
    Sat, 6 Apr 2024 02:29:02 GMT
    Yann Dubois and Bal\'azs Galambosi and Percy Liang and Tatsunori B. Hashimoto
    

LLM-based auto-annotators have become a key component of the LLM development process due to their cost-effectiveness and scalability compared to human-based evaluation. However, these auto-annotators can introduce complex biases that are hard to remove. Even simple, known confounders such as preference for longer outputs remain in existing automated evaluation metrics. We propose a simple regression analysis approach for controlling biases in auto-evaluations. As a real case study, we focus on reducing the length bias of AlpacaEval, a fast and affordable benchmark for chat LLMs that uses LLMs to estimate response quality. Despite being highly correlated with human preferences, AlpacaEval is known to favor models that generate longer outputs. We introduce a length-controlled AlpacaEval that aims to answer the counterfactual question: "What would the preference be if the model's and baseline's output had the same length?". To achieve this, we first fit a generalized linear model to predict the biased output of interest (auto-annotator preferences) based on the mediators we want to control for (length difference) and other relevant features. We then obtain length-controlled preferences by predicting preferences while conditioning the GLM with a zero difference in lengths. Length-controlling not only improves the robustness of the metric to manipulations in model verbosity, we also find that it increases the Spearman correlation with LMSYS' Chatbot Arena from 0.94 to 0.98. We release the code and leaderboard at <https://tatsu-lab.github.io/alpaca_eval/> .

* * *

[[2404.04575] To Cool or not to Cool? Temperature Network Meets Large Foundation Models via DRO](https://arxiv.org/abs/2404.04575)
    
    
    Sat, 6 Apr 2024 09:55:03 GMT
    Zi-Hao Qiu, Siqi Guo, Mao Xu, Tuo Zhao, Lijun Zhang, Tianbao Yang
    

The temperature parameter plays a profound role during training and/or inference with large foundation models (LFMs) such as large language models (LLMs) and CLIP models. Particularly, it adjusts the logits in the softmax function in LLMs, which is crucial for next token generation, and it scales the similarities in the contrastive loss for training CLIP models. A significant question remains: Is it viable to learn a neural network to predict a personalized temperature of any input data for enhancing LFMs"? In this paper, we present a principled framework for learning a small yet generalizable temperature prediction network (TempNet) to improve LFMs. Our solution is composed of a novel learning framework with a robust loss underpinned by constrained distributionally robust optimization (DRO), and a properly designed TempNet with theoretical inspiration. TempNet can be trained together with a large foundation model from scratch or learned separately given a pretrained foundation model. It is not only useful for predicting personalized temperature to promote the training of LFMs but also generalizable and transferable to new tasks. Our experiments on LLMs and CLIP models demonstrate that TempNet greatly improves the performance of existing solutions or models, e.g. Table 1. The code to reproduce the experimental results in this paper can be found at <https://github.com/zhqiu/TempNet>.

* * *

[[2404.04656] Binary Classifier Optimization for Large Language Model Alignment](https://arxiv.org/abs/2404.04656)
    
    
    Sat, 6 Apr 2024 15:20:59 GMT
    Seungjae Jung, Gunsoo Han, Daniel Wontae Nam and Kyoung-Woon On
    

Aligning Large Language Models (LLMs) to human preferences through preference optimization has been crucial but labor-intensive, necessitating for each prompt a comparison of both a chosen and a rejected text completion by evaluators. Recently, Kahneman-Tversky Optimization (KTO) has demonstrated that LLMs can be aligned using merely binary "thumbs-up" or "thumbs-down" signals on each prompt-completion pair. In this paper, we present theoretical foundations to explain the successful alignment achieved through these binary signals. Our analysis uncovers a new perspective: optimizing a binary classifier, whose logit is a reward, implicitly induces minimizing the Direct Preference Optimization (DPO) loss. In the process of this discovery, we identified two techniques for effective alignment: reward shift and underlying distribution matching. Consequently, we propose a new algorithm, textit{Binary Classifier Optimization}, that integrates the techniques. We validate our methodology in two settings: first, on a paired preference dataset, where our method performs on par with DPO and KTO; and second, on binary signal datasets simulating real-world conditions with divergent underlying distributions between thumbs-up and thumbs-down data. Our model consistently demonstrates effective and robust alignment across two base LLMs and three different binary signal datasets, showcasing the strength of our approach to learning from binary feedback.

* * *

[[2404.04793] SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget](https://arxiv.org/abs/2404.04793)
    
    
    Sun, 7 Apr 2024 03:08:14 GMT
    Zihao Wang, Shaoduo Gan
    

Optimizing the Key-Value (KV) cache of the Large Language Model (LLM) has been considered critical to saving the cost of inference. Most of the existing KV-cache compression algorithms attempted to sparsify the sequence of tokens by taking advantage of the different importance of tokens. In this work, we found that by identifying the importance of attention layers, we could optimize the KV-cache jointly from two dimensions. Based on our observations regarding layer-wise importance in inference, we propose SqueezeAttention to precisely optimize the allocation of KV-cache budget among layers on-the-fly and then incorporate three representative token sparsification algorithms to compress the KV-cache for each layer with its very own budget. By optimizing the KV-cache from both sequence's and layer's dimensions, SqueezeAttention achieves around 30% to 70% of the memory reductions and up to 2.2 times of throughput improvements in a wide range of LLMs and benchmarks. The code is available at <https://github.com/hetailang/SqueezeAttention>.

* * *

[[2404.04854] Contextual Chart Generation for Cyber Deception](https://arxiv.org/abs/2404.04854)
    
    
    Sun, 7 Apr 2024 07:56:14 GMT
    David D. Nguyen, David Liebowitz, Surya Nepal, Salil S. Kanhere, Sharif Abuadbba
    

Honeyfiles are security assets designed to attract and detect intruders on compromised systems. Honeyfiles are a type of honeypot that mimic real, sensitive documents, creating the illusion of the presence of valuable data. Interaction with a honeyfile reveals the presence of an intruder, and can provide insights into their goals and intentions. Their practical use, however, is limited by the time, cost and effort associated with manually creating realistic content. The introduction of large language models has made high-quality text generation accessible, but honeyfiles contain a variety of content including charts, tables and images. This content needs to be plausible and realistic, as well as semantically consistent both within honeyfiles and with the real documents they mimic, to successfully deceive an intruder. In this paper, we focus on an important component of the honeyfile content generation problem: document charts. Charts are ubiquitous in corporate documents and are commonly used to communicate quantitative and scientific data. Existing image generation models, such as DALL-E, are rather prone to generating charts with incomprehensible text and unconvincing data. We take a multi-modal approach to this problem by combining two purpose-built generative models: a multitask Transformer and a specialized multi-head autoencoder. The Transformer generates realistic captions and plot text, while the autoencoder generates the underlying tabular data for the plot. To advance the field of automated honeyplot generation, we also release a new document-chart dataset and propose a novel metric Keyword Semantic Matching (KSM). This metric measures the semantic consistency between keywords of a corpus and a smaller bag of words. Extensive experiments demonstrate excellent performance against multiple large language models, including ChatGPT and GPT4.

* * *

[[2404.04885] TimeGPT in Load Forecasting: A Large Time Series Model Perspective](https://arxiv.org/abs/2404.04885)
    
    
    Sun, 7 Apr 2024 09:05:09 GMT
    Wenlong Liao, Fernando Porte-Agel, Jiannong Fang, Christian Rehtanz, Shouxiang Wang, Dechang Yang, and Zhe Yang
    

Machine learning models have made significant progress in load forecasting, but their forecast accuracy is limited in cases where historical load data is scarce. Inspired by the outstanding performance of large language models (LLMs) in computer vision and natural language processing, this paper aims to discuss the potential of large time series models in load forecasting with scarce historical data. Specifically, the large time series model is constructed as a time series generative pre-trained transformer (TimeGPT), which is trained on massive and diverse time series datasets consisting of 100 billion data points (e.g., finance, transportation, banking, web traffic, weather, energy, healthcare, etc.). Then, the scarce historical load data is used to fine-tune the TimeGPT, which helps it to adapt to the data distribution and characteristics associated with load forecasting. Simulation results show that TimeGPT outperforms the benchmarks (e.g., popular machine learning models and statistical models) for load forecasting on several real datasets with scarce training samples, particularly for short look-ahead times. However, it cannot be guaranteed that TimeGPT is always superior to benchmarks for load forecasting with scarce data, since the performance of TimeGPT may be affected by the distribution differences between the load data and the training data. In practical applications, we can divide the historical data into a training set and a validation set, and then use the validation set loss to decide whether TimeGPT is the best choice for a specific dataset.

* * *

[[2404.04997] Adapting LLMs for Efficient Context Processing through Soft Prompt Compression](https://arxiv.org/abs/2404.04997)
    
    
    Sun, 7 Apr 2024 15:44:20 GMT
    Cangqing Wang, Yutian Yang, Ruisi Li, Dan Sun, Ruicong Cai, Yuzhu Zhang, Chengqian Fu and Lillian Floyd
    

The rapid advancement of Large Language Models (LLMs) has inaugurated a transformative epoch in natural language processing, fostering unprecedented proficiency in text generation, comprehension, and contextual scrutiny. Nevertheless, effectively handling extensive contexts, crucial for myriad applications, poses a formidable obstacle owing to the intrinsic constraints of the models' context window sizes and the computational burdens entailed by their operations. This investigation presents an innovative framework that strategically tailors LLMs for streamlined context processing by harnessing the synergies among natural language summarization, soft prompt compression, and augmented utility preservation mechanisms. Our methodology, dubbed SoftPromptComp, amalgamates natural language prompts extracted from summarization methodologies with dynamically generated soft prompts to forge a concise yet semantically robust depiction of protracted contexts. This depiction undergoes further refinement via a weighting mechanism optimizing information retention and utility for subsequent tasks. We substantiate that our framework markedly diminishes computational overhead and enhances LLMs' efficacy across various benchmarks, while upholding or even augmenting the caliber of the produced content. By amalgamating soft prompt compression with sophisticated summarization, SoftPromptComp confronts the dual challenges of managing lengthy contexts and ensuring model scalability. Our findings point towards a propitious trajectory for augmenting LLMs' applicability and efficiency, rendering them more versatile and pragmatic for real-world applications. This research enriches the ongoing discourse on optimizing language models, providing insights into the potency of soft prompts and summarization techniques as pivotal instruments for the forthcoming generation of NLP solutions.

* * *

[[2404.05047] Initial Exploration of Zero-Shot Privacy Utility Tradeoffs in Tabular Data Using GPT-4](https://arxiv.org/abs/2404.05047)
    
    
    Sun, 7 Apr 2024 19:02:50 GMT
    Bishwas Mandal, George Amariucai, Shuangqing Wei
    

We investigate the application of large language models (LLMs), specifically GPT-4, to scenarios involving the tradeoff between privacy and utility in tabular data. Our approach entails prompting GPT-4 by transforming tabular data points into textual format, followed by the inclusion of precise sanitization instructions in a zero-shot manner. The primary objective is to sanitize the tabular data in such a way that it hinders existing machine learning models from accurately inferring private features while allowing models to accurately infer utility-related attributes. We explore various sanitization instructions. Notably, we discover that this relatively simple approach yields performance comparable to more complex adversarial optimization methods used for managing privacy-utility tradeoffs. Furthermore, while the prompts successfully obscure private features from the detection capabilities of existing machine learning models, we observe that this obscuration alone does not necessarily meet a range of fairness metrics. Nevertheless, our research indicates the potential effectiveness of LLMs in adhering to these fairness metrics, with some of our experimental results aligning with those achieved by well-established adversarial optimization techniques.

* * *

[[2404.05086] A Note on LoRA](https://arxiv.org/abs/2404.05086)
    
    
    Sun, 7 Apr 2024 22:00:50 GMT
    Vlad Fomenko, Han Yu, Jongho Lee, Stanley Hsieh, Weizhu Chen
    

LoRA (Low-Rank Adaptation) has emerged as a preferred method for efficiently adapting Large Language Models (LLMs) with remarkable simplicity and efficacy. This note extends the original LoRA paper by offering new perspectives that were not initially discussed and presents a series of insights for deploying LoRA at scale. Without introducing new experiments, we aim to improve the understanding and application of LoRA.

* * *

[[2404.05182] DLoRA: Distributed Parameter-Efficient Fine-Tuning Solution for Large Language Model](https://arxiv.org/abs/2404.05182)
    
    
    Mon, 8 Apr 2024 04:14:02 GMT
    Chao Gao and Sai Qian Zhang
    

To enhance the performance of large language models (LLM) on downstream tasks, one solution is to fine-tune certain LLM parameters and make it better align with the characteristics of the training dataset. This process is commonly known as parameter-efficient fine-tuning (PEFT). Due to the scale of LLM, PEFT operations are usually executed in the public environment (e.g., cloud server). This necessitates the sharing of sensitive user data across public environments, thereby raising potential privacy concerns. To tackle these challenges, we propose a distributed PEFT framework called DLoRA. DLoRA enables scalable PEFT operations to be performed collaboratively between the cloud and user devices. Coupled with the proposed Kill and Revive algorithm, the evaluation results demonstrate that DLoRA can significantly reduce the computation and communication workload over the user devices while achieving superior accuracy and privacy protection.

* * *

[[2404.05545] Evaluating Interventional Reasoning Capabilities of Large Language Models](https://arxiv.org/abs/2404.05545)
    
    
    Mon, 8 Apr 2024 14:15:56 GMT
    Tejas Kasetty, Divyat Mahajan, Gintare Karolina Dziugaite, Alexandre Drouin, Dhanya Sridhar
    

Numerous decision-making tasks require estimating causal effects under interventions on different parts of a system. As practitioners consider using large language models (LLMs) to automate decisions, studying their causal reasoning capabilities becomes crucial. A recent line of work evaluates LLMs ability to retrieve commonsense causal facts, but these evaluations do not sufficiently assess how LLMs reason about interventions. Motivated by the role that interventions play in causal inference, in this paper, we conduct empirical analyses to evaluate whether LLMs can accurately update their knowledge of a data-generating process in response to an intervention. We create benchmarks that span diverse causal graphs (e.g., confounding, mediation) and variable types, and enable a study of intervention-based reasoning. These benchmarks allow us to isolate the ability of LLMs to accurately predict changes resulting from their ability to memorize facts or find other shortcuts. Our analysis on four LLMs highlights that while GPT- 4 models show promising accuracy at predicting the intervention effects, they remain sensitive to distracting factors in the prompts.

* * *

[[2404.05567] Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models](https://arxiv.org/abs/2404.05567)
    
    
    Mon, 8 Apr 2024 14:39:49 GMT
    Bowen Pan, Yikang Shen, Haokun Liu, Mayank Mishra, Gaoyuan Zhang, Aude Oliva, Colin Raffel, Rameswar Panda
    

Mixture-of-Experts (MoE) language models can reduce computational costs by 2-4$times$ compared to dense models without sacrificing performance, making them more efficient in computation-bounded scenarios. However, MoE models generally require 2-4$times$ times more parameters to achieve comparable performance to a dense model, which incurs larger GPU memory requirements and makes MoE models less efficient in I/O-bounded scenarios like autoregressive generation. In this work, we propose a hybrid dense training and sparse inference framework for MoE models (DS-MoE) which achieves strong computation and parameter efficiency by employing dense computation across all experts during training and sparse computation during inference. Our experiments on training LLMs demonstrate that our DS-MoE models are more parameter-efficient than standard sparse MoEs and are on par with dense models in terms of total parameter size and performance while being computationally cheaper (activating 30-40% of the model's parameters). Performance tests using vLLM show that our DS-MoE-6B model runs up to $1.86times$ faster than similar dense models like Mistral-7B, and between $1.50times$ and $1.71times$ faster than comparable MoEs, such as DeepSeekMoE-16B and Qwen1.5-MoE-A2.7B.

* * *

[[2404.04264] Logic Query of Thoughts: Guiding Large Language Models to Answer Complex Logic Queries with Knowledge Graphs](https://arxiv.org/abs/2404.04264)
    
    
    Sun, 17 Mar 2024 17:01:45 GMT
    Lihui Liu, Zihao Wang, Ruizhong Qiu, Yikun Ban, Hanghang Tong
    

Despite the superb performance in many tasks, large language models (LLMs) bear the risk of generating hallucination or even wrong answers when confronted with tasks that demand the accuracy of knowledge. The issue becomes even more noticeable when addressing logic queries that require multiple logic reasoning steps. On the other hand, knowledge graph (KG) based question answering methods are capable of accurately identifying the correct answers with the help of knowledge graph, yet its accuracy could quickly deteriorate when the knowledge graph itself is sparse and incomplete. It remains a critical challenge on how to integrate knowledge graph reasoning with LLMs in a mutually beneficial way so as to mitigate both the hallucination problem of LLMs as well as the incompleteness issue of knowledge graphs. In this paper, we propose 'Logic-Query-of-Thoughts' (LGOT) which is the first of its kind to combine LLMs with knowledge graph based logic query reasoning. LGOT seamlessly combines knowledge graph reasoning and LLMs, effectively breaking down complex logic queries into easy to answer subquestions. Through the utilization of both knowledge graph reasoning and LLMs, it successfully derives answers for each subquestion. By aggregating these results and selecting the highest quality candidate answers for each step, LGOT achieves accurate results to complex questions. Our experimental findings demonstrate substantial performance enhancements, with up to 20% improvement over ChatGPT.

* * *

[[2404.04268] The Use of Generative Search Engines for Knowledge Work and Complex Tasks](https://arxiv.org/abs/2404.04268)
    
    
    Tue, 19 Mar 2024 18:17:46 GMT
    Siddharth Suri, Scott Counts, Leijie Wang, Chacha Chen, Mengting Wan, Tara Safavi, Jennifer Neville, Chirag Shah, Ryen W. White, Reid Andersen, Georg Buscher, Sathish Manivannan, Nagu Rangan, Longqi Yang
    

Until recently, search engines were the predominant method for people to access online information. The recent emergence of large language models (LLMs) has given machines new capabilities such as the ability to generate new digital artifacts like text, images, code etc., resulting in a new tool, a generative search engine, which combines the capabilities of LLMs with a traditional search engine. Through the empirical analysis of Bing Copilot (Bing Chat), one of the first publicly available generative search engines, we analyze the types and complexity of tasks that people use Bing Copilot for compared to Bing Search. Findings indicate that people use the generative search engine for more knowledge work tasks that are higher in cognitive complexity than were commonly done with a traditional search engine.

* * *

[[2404.04306] AuditGPT: Auditing Smart Contracts with ChatGPT](https://arxiv.org/abs/2404.04306)
    
    
    Fri, 5 Apr 2024 07:19:13 GMT
    Shihao Xia, Shuai Shao, Mengting He, Tingting Yu, Linhai Song, Yiying Zhang
    

To govern smart contracts running on Ethereum, multiple Ethereum Request for Comment (ERC) standards have been developed, each containing a set of rules to guide the behaviors of smart contracts. Violating the ERC rules could cause serious security issues and financial loss, signifying the importance of verifying smart contracts follow ERCs. Today's practices of such verification are to either manually audit each single contract or use expert-developed, limited-scope program-analysis tools, both of which are far from being effective in identifying ERC rule violations. This paper presents a tool named AuditGPT that leverages large language models (LLMs) to automatically and comprehensively verify ERC rules against smart contracts. To build AuditGPT, we first conduct an empirical study on 222 ERC rules specified in four popular ERCs to understand their content, their security impacts, their specification in natural language, and their implementation in Solidity. Guided by the study, we construct AuditGPT by separating the large, complex auditing process into small, manageable tasks and design prompts specialized for each ERC rule type to enhance LLMs' auditing performance. In the evaluation, AuditGPT successfully pinpoints 418 ERC rule violations and only reports 18 false positives, showcasing its effectiveness and accuracy. Moreover, AuditGPT beats an auditing service provided by security experts in effectiveness, accuracy, and cost, demonstrating its advancement over state-of-the-art smart-contract auditing practices.

* * *

[[2404.04376] ClickDiffusion: Harnessing LLMs for Interactive Precise Image Editing](https://arxiv.org/abs/2404.04376)
    
    
    Fri, 5 Apr 2024 19:38:18 GMT
    Alec Helbling, Seongmin Lee, Polo Chau
    

Recently, researchers have proposed powerful systems for generating and manipulating images using natural language instructions. However, it is difficult to precisely specify many common classes of image transformations with text alone. For example, a user may wish to change the location and breed of a particular dog in an image with several similar dogs. This task is quite difficult with natural language alone, and would require a user to write a laboriously complex prompt that both disambiguates the target dog and describes the destination. We propose ClickDiffusion, a system for precise image manipulation and generation that combines natural language instructions with visual feedback provided by the user through a direct manipulation interface. We demonstrate that by serializing both an image and a multi-modal instruction into a textual representation it is possible to leverage LLMs to perform precise transformations of the layout and appearance of an image. Code available at <https://github.com/poloclub/ClickDiffusion>.

* * *

[[2404.04392] Increased LLM Vulnerabilities from Fine-tuning and Quantization](https://arxiv.org/abs/2404.04392)
    
    
    Fri, 5 Apr 2024 20:31:45 GMT
    Divyanshu Kumar, Anurakt Kumar, Sahil Agarwal and Prashanth Harshangi
    

Large Language Models (LLMs) have become very popular and have found use cases in many domains, such as chatbots, auto-task completion agents, and much more. However, LLMs are vulnerable to different types of attacks, such as jailbreaking, prompt injection attacks, and privacy leakage attacks. Foundational LLMs undergo adversarial and alignment training to learn not to generate malicious and toxic content. For specialized use cases, these foundational LLMs are subjected to fine-tuning or quantization for better performance and efficiency. We examine the impact of downstream tasks such as fine-tuning and quantization on LLM vulnerability. We test foundation models like Mistral, Llama, MosaicML, and their fine-tuned versions. Our research shows that fine-tuning and quantization reduces jailbreak resistance significantly, leading to increased LLM vulnerabilities. Finally, we demonstrate the utility of external guardrails in reducing LLM vulnerabilities.

* * *

[[2404.04763] GenEARL: A Training-Free Generative Framework for Multimodal Event Argument Role Labeling](https://arxiv.org/abs/2404.04763)
    
    
    Sun, 7 Apr 2024 00:28:13 GMT
    Hritik Bansal, Po-Nien Kung, P. Jeffrey Brantingham, Kai-Wei Chang, Nanyun Peng
    

Multimodal event argument role labeling (EARL), a task that assigns a role for each event participant (object) in an image is a complex challenge. It requires reasoning over the entire image, the depicted event, and the interactions between various objects participating in the event. Existing models heavily rely on high-quality event-annotated training data to understand the event semantics and structures, and they fail to generalize to new event types and domains. In this paper, we propose GenEARL, a training-free generative framework that harness the power of the modern generative models to understand event task descriptions given image contexts to perform the EARL task. Specifically, GenEARL comprises two stages of generative prompting with a frozen vision-language model (VLM) and a frozen large language model (LLM). First, a generative VLM learns the semantics of the event argument roles and generates event-centric object descriptions based on the image. Subsequently, a LLM is prompted with the generated object descriptions with a predefined template for EARL (i.e., assign an object with an event argument role). We show that GenEARL outperforms the contrastive pretraining (CLIP) baseline by 9.4% and 14.2% accuracy for zero-shot EARL on the M2E2 and SwiG datasets, respectively. In addition, we outperform CLIP-Event by 22% precision on M2E2 dataset. The framework also allows flexible adaptation and generalization to unseen domains.

* * *

[[2404.04849] Hidden You Malicious Goal Into Benigh Narratives: Jailbreak Large Language Models through Logic Chain Injection](https://arxiv.org/abs/2404.04849)
    
    
    Sun, 7 Apr 2024 07:42:12 GMT
    Zhilong Wang, Yebo Cao, and Peng Liu
    

Jailbreak attacks on Language Model Models (LLMs) entail crafting prompts aimed at exploiting the models to generate malicious content. Existing jailbreak attacks can successfully deceive the LLMs, however they cannot deceive the human. This paper proposes a new type of jailbreak attacks which can deceive both the LLMs and human (i.e., security analyst). The key insight of our idea is borrowed from the social psychology - that is human are easily deceived if the lie is hidden in truth. Based on this insight, we proposed the logic-chain injection attacks to inject malicious intention into benign truth. Logic-chain injection attack firstly dissembles its malicious target into a chain of benign narrations, and then distribute narrations into a related benign article, with undoubted facts. In this way, newly generate prompt cannot only deceive the LLMs, but also deceive human.

* * *

[[2404.04869] Prompting Multi-Modal Tokens to Enhance End-to-End Autonomous Driving Imitation Learning with LLMs](https://arxiv.org/abs/2404.04869)
    
    
    Sun, 7 Apr 2024 08:31:12 GMT
    Yiqun Duan, Qiang Zhang, Renjing Xu
    

The utilization of Large Language Models (LLMs) within the realm of reinforcement learning, particularly as planners, has garnered a significant degree of attention in recent scholarly literature. However, a substantial proportion of existing research predominantly focuses on planning models for robotics that transmute the outputs derived from perception models into linguistic forms, thus adopting a `pure-language' strategy. In this research, we propose a hybrid End-to-End learning framework for autonomous driving by combining basic driving imitation learning with LLMs based on multi-modality prompt tokens. Instead of simply converting perception results from the separated train model into pure language input, our novelty lies in two aspects. 1) The end-to-end integration of visual and LiDAR sensory input into learnable multi-modality tokens, thereby intrinsically alleviating description bias by separated pre-trained perception models. 2) Instead of directly letting LLMs drive, this paper explores a hybrid setting of letting LLMs help the driving model correct mistakes and complicated scenarios. The results of our experiments suggest that the proposed methodology can attain driving scores of 49.21%, coupled with an impressive route completion rate of 91.34% in the offline evaluation conducted via CARLA. These performance metrics are comparable to the most advanced driving models.

* * *

[[2404.05101] StockGPT: A GenAI Model for Stock Prediction and Trading](https://arxiv.org/abs/2404.05101)
    
    
    Sun, 7 Apr 2024 22:53:43 GMT
    Dat Mai
    

This paper introduces StockGPT, an autoregressive "number" model pretrained directly on the history of daily U.S. stock returns. Treating each return series as a sequence of tokens, the model excels at understanding and predicting the highly intricate stock return dynamics. Instead of relying on handcrafted trading patterns using historical stock prices, StockGPT automatically learns the hidden representations predictive of future returns via its attention mechanism. On a held-out test sample from 2001 to 2023, a daily rebalanced long-short portfolio formed from StockGPT predictions earns an annual return of 119% with a Sharpe ratio of 6.5. The StockGPT-based portfolio completely explains away momentum and long-/short-term reversals, eliminating the need for manually crafted price-based strategies and also encompasses most leading stock market factors. This highlights the immense promise of generative AI in surpassing human in making complex financial investment decisions and illustrates the efficacy of the attention mechanism of large language models when applied to a completely different domain.

* * *

[[2404.05188] Have You Merged My Model? On The Robustness of Large Language Model IP Protection Methods Against Model Merging](https://arxiv.org/abs/2404.05188)
    
    
    Mon, 8 Apr 2024 04:30:33 GMT
    Tianshuo Cong and Delong Ran and Zesen Liu and Xinlei He and Jinyuan Liu and Yichen Gong and Qi Li and Anyu Wang and Xiaoyun Wang
    

Model merging is a promising lightweight model empowerment technique that does not rely on expensive computing devices (e.g., GPUs) or require the collection of specific training data. Instead, it involves editing different upstream model parameters to absorb their downstream task capabilities. However, uncertified model merging can infringe upon the Intellectual Property (IP) rights of the original upstream models. In this paper, we conduct the first study on the robustness of IP protection methods in model merging scenarios. We investigate two state-of-the-art IP protection techniques: Quantization Watermarking and Instructional Fingerprint, along with various advanced model merging technologies, such as Task Arithmetic, TIES-MERGING, and so on. Experimental results indicate that current Large Language Model (LLM) watermarking techniques cannot survive in the merged models, whereas model fingerprinting techniques can. Our research aims to highlight that model merging should be an indispensable consideration in the robustness assessment of model IP protection techniques, thereby promoting the healthy development of the open-source LLM community.

* * *

[[2404.05213] Evaluation of an LLM in Identifying Logical Fallacies: A Call for Rigor When Adopting LLMs in HCI Research](https://arxiv.org/abs/2404.05213)
    
    
    Mon, 8 Apr 2024 06:00:14 GMT
    Gionnieve Lim, Simon T. Perrault
    

There is increasing interest in the adoption of LLMs in HCI research. However, LLMs may often be regarded as a panacea because of their powerful capabilities with an accompanying oversight on whether they are suitable for their intended tasks. We contend that LLMs should be adopted in a critical manner following rigorous evaluation. Accordingly, we present the evaluation of an LLM in identifying logical fallacies that will form part of a digital misinformation intervention. By comparing to a labeled dataset, we found that GPT-4 achieves an accuracy of 0.79, and for our intended use case that excludes invalid or unidentified instances, an accuracy of 0.90. This gives us the confidence to proceed with the application of the LLM while keeping in mind the areas where it still falls short. The paper describes our evaluation approach, results and reflections on the use of the LLM for our intended task.

* * *

[[2404.05427] AutoCodeRover: Autonomous Program Improvement](https://arxiv.org/abs/2404.05427)
    
    
    Mon, 8 Apr 2024 11:55:09 GMT
    Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, Abhik Roychoudhury
    

Researchers have made significant progress in automating the software development process in the past decades. Automated techniques for issue summarization, bug reproduction, fault localization, and program repair have been built to ease the workload of developers. Recent progress in Large Language Models (LLMs) has significantly impacted the development process, where developers can use LLM-based programming assistants to achieve automated coding. Nevertheless software engineering involves the process of program improvement apart from coding, specifically to enable software maintenance (e.g. program repair to fix bugs) and software evolution (e.g. feature additions). In this paper, we propose an automated approach for solving Github issues to autonomously achieve program improvement. In our approach called AutoCodeRover, LLMs are combined with sophisticated code search capabilities, ultimately leading to a program modification or patch. In contrast to recent LLM agent approaches from AI researchers and practitioners, our outlook is more software engineering oriented. We work on a program representation (abstract syntax tree) as opposed to viewing a software project as a mere collection of files. Our code search exploits the program structure in the form of classes/methods to enhance LLM's understanding of the issue's root cause, and effectively retrieve a context via iterative search. The use of spectrum based fault localization using tests, further sharpens the context. Experiments on the recently proposed SWE-bench-lite which consists of 300 real-life Github issues involving bug fixing and feature additions show increased efficacy (resolving more than 20% on SWE-bench-lite), as compared to recent efforts from the AI community. We posit that our workflow enables autonomous software engineering, where, in future, auto-generated code from LLMs can be autonomously improved.

* * *

[[2404.05499] Constraining Large Language Model for Generating Computer-Parsable Content](https://arxiv.org/abs/2404.05499)
    
    
    Mon, 8 Apr 2024 13:22:24 GMT
    Jiaye Wang
    

We propose a method to guide Large Language Models (LLMs) in generating structured content adhering to specific conventions without fine-tuning. By utilizing coroutine-based content generation constraints through a pre-agreed context-free grammar (CFG), LLMs are directed during decoding to produce formal language compliant outputs. This enhances stability and consistency in generating target data structures, types, or instructions, reducing application development complexities. Experimentally, error rates of GPT-2 and Gemma exceed 95% for DSLs longer than 36 and 282 tokens, respectively. We introduce YieldLang, a coroutine-based DSL generation framework, and evaluate it with LLMs on various tasks including JSON and Mermaid flowchart generation. Compared to benchmarks, our approach improves accuracy by 1.09 to 11.6 times, with LLMs requiring only about 16.5% of the samples to generate JSON effectively. This enhances usability of LLM-generated content for computer programs.

* * *

[[2404.05508] Synergy of Large Language Model and Model Driven Engineering for Automated Development of Centralized Vehicular Systems](https://arxiv.org/abs/2404.05508)
    
    
    Mon, 8 Apr 2024 13:28:11 GMT
    Nenad Petrovic, Fengjunjie Pan, Krzysztof Lebioda, Vahid Zolfaghari, Sven Kirchner, Nils Purschke, Muhammad Aqib Khan, Viktor Vorobev, Alois Knoll
    

We present a prototype of a tool leveraging the synergy of model driven engineering (MDE) and Large Language Models (LLM) for the purpose of software development process automation in the automotive industry. In this approach, the user-provided input is free form textual requirements, which are first translated to Ecore model instance representation using an LLM, which is afterwards checked for consistency using Object Constraint Language (OCL) rules. After successful consistency check, the model instance is fed as input to another LLM for the purpose of code generation. The generated code is evaluated in a simulated environment using CARLA simulator connected to an example centralized vehicle architecture, in an emergency brake scenario.

* * *

[[2404.05083] HaVTR: Improving Video-Text Retrieval Through Augmentation Using Large Foundation Models](https://arxiv.org/abs/2404.05083)
    
    
    Sun, 7 Apr 2024 21:46:47 GMT
    Yimu Wang, Shuai Yuan, Xiangru Jian, Wei Pang, Mushi Wang, Ning Yu
    

While recent progress in video-text retrieval has been driven by the exploration of powerful model architectures and training strategies, the representation learning ability of video-text retrieval models is still limited due to low-quality and scarce training data annotations. To address this issue, we present a novel video-text learning paradigm, HaVTR, which augments video and text data to learn more generalized features. Specifically, we first adopt a simple augmentation method, which generates self-similar data by randomly duplicating or dropping subwords and frames. In addition, inspired by the recent advancement in visual and language generative models, we propose a more powerful augmentation method through textual paraphrasing and video stylization using large language models (LLMs) and visual generative models (VGMs). Further, to bring richer information into video and text, we propose a hallucination-based augmentation method, where we use LLMs and VGMs to generate and add new relevant information to the original data. Benefiting from the enriched data, extensive experiments on several video-text retrieval benchmarks demonstrate the superiority of HaVTR over existing methods.

* * *

[[2404.05225] LayoutLLM: Layout Instruction Tuning with Large Language Models for Document Understanding](https://arxiv.org/abs/2404.05225)
    
    
    Mon, 8 Apr 2024 06:40:28 GMT
    Chuwei Luo, Yufan Shen, Zhaoqing Zhu, Qi Zheng, Zhi Yu, Cong Yao
    

Recently, leveraging large language models (LLMs) or multimodal large language models (MLLMs) for document understanding has been proven very promising. However, previous works that employ LLMs/MLLMs for document understanding have not fully explored and utilized the document layout information, which is vital for precise document understanding. In this paper, we propose LayoutLLM, an LLM/MLLM based method for document understanding. The core of LayoutLLM is a layout instruction tuning strategy, which is specially designed to enhance the comprehension and utilization of document layouts. The proposed layout instruction tuning strategy consists of two components: Layout-aware Pre-training and Layout-aware Supervised Fine-tuning. To capture the characteristics of document layout in Layout-aware Pre-training, three groups of pre-training tasks, corresponding to document-level, region-level and segment-level information, are introduced. Furthermore, a novel module called layout chain-of-thought (LayoutCoT) is devised to enable LayoutLLM to focus on regions relevant to the question and generate accurate answers. LayoutCoT is effective for boosting the performance of document understanding. Meanwhile, it brings a certain degree of interpretability, which could facilitate manual inspection and correction. Experiments on standard benchmarks show that the proposed LayoutLLM significantly outperforms existing methods that adopt open-source 7B LLMs/MLLMs for document understanding. The training data of the LayoutLLM is publicly available at <https://github.com/AlibabaResearch/AdvancedLiterateMachinery/tree/main/DocumentUnderstanding/LayoutLLM>

* * *

[[2404.05719] Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs](https://arxiv.org/abs/2404.05719)
    
    
    Mon, 8 Apr 2024 17:55:44 GMT
    Keen You, Haotian Zhang, Eldon Schoop, Floris Weers, Amanda Swearngin, Jeffrey Nichols, Yinfei Yang, Zhe Gan
    

Recent advancements in multimodal large language models (MLLMs) have been noteworthy, yet, these general-domain MLLMs often fall short in their ability to comprehend and interact effectively with user interface (UI) screens. In this paper, we present Ferret-UI, a new MLLM tailored for enhanced understanding of mobile UI screens, equipped with referring, grounding, and reasoning capabilities. Given that UI screens typically exhibit a more elongated aspect ratio and contain smaller objects of interest (e.g., icons, texts) than natural images, we incorporate "any resolution" on top of Ferret to magnify details and leverage enhanced visual features. Specifically, each screen is divided into 2 sub-images based on the original aspect ratio (i.e., horizontal division for portrait screens and vertical division for landscape screens). Both sub-images are encoded separately before being sent to LLMs. We meticulously gather training samples from an extensive range of elementary UI tasks, such as icon recognition, find text, and widget listing. These samples are formatted for instruction-following with region annotations to facilitate precise referring and grounding. To augment the model's reasoning ability, we further compile a dataset for advanced tasks, including detailed description, perception/interaction conversations, and function inference. After training on the curated datasets, Ferret-UI exhibits outstanding comprehension of UI screens and the capability to execute open-ended instructions. For model evaluation, we establish a comprehensive benchmark encompassing all the aforementioned tasks. Ferret-UI excels not only beyond most open-source UI MLLMs, but also surpasses GPT-4V on all the elementary UI tasks.

* * *

[[2404.04689] Multicalibration for Confidence Scoring in LLMs](https://arxiv.org/abs/2404.04689)
    
    
    Sat, 6 Apr 2024 17:33:37 GMT
    Gianluca Detommaso, Martin Bertran, Riccardo Fogliato, Aaron Roth
    

This paper proposes the use of "multicalibration" to yield interpretable and reliable confidence scores for outputs generated by large language models (LLMs). Multicalibration asks for calibration not just marginally, but simultaneously across various intersecting groupings of the data. We show how to form groupings for prompt/completion pairs that are correlated with the probability of correctness via two techniques: clustering within an embedding space, and "self-annotation" - querying the LLM by asking it various yes-or-no questions about the prompt. We also develop novel variants of multicalibration algorithms that offer performance improvements by reducing their tendency to overfit. Through systematic benchmarking across various question answering datasets and LLMs, we show how our techniques can yield confidence scores that provide substantial improvements in fine-grained measures of both calibration and accuracy compared to existing methods.

* * *

[[2404.05183] Progressive Alignment with VLM-LLM Feature to Augment Defect Classification for the ASE Dataset](https://arxiv.org/abs/2404.05183)
    
    
    Mon, 8 Apr 2024 04:17:27 GMT
    Chih-Chung Hsu, Chia-Ming Lee, Chun-Hung Sun, and Kuang-Ming Wu
    

Traditional defect classification approaches are facing with two barriers. (1) Insufficient training data and unstable data quality. Collecting sufficient defective sample is expensive and time-costing, consequently leading to dataset variance. It introduces the difficulty on recognition and learning. (2) Over-dependence on visual modality. When the image pattern and texture is monotonic for all defect classes in a given dataset, the performance of conventional AOI system cannot be guaranteed. In scenarios where image quality is compromised due to mechanical failures or when defect information is inherently difficult to discern, the performance of deep models cannot be guaranteed. A main question is, "how to solve those two problems when they occur at the same time?" The feasible strategy is to explore another feature within dataset and combine an eminent vision-language model (VLM) and Large-Language model (LLM) with their astonishing zero-shot capability. In this work, we propose the special ASE dataset, including rich data description recorded on image, for defect classification, but the defect feature is uneasy to learn directly. Secondly, We present the prompting for VLM-LLM against defect classification with the proposed ASE dataset to activate extra-modality feature from images to enhance performance. Then, We design the novel progressive feature alignment (PFA) block to refine image-text feature to alleviate the difficulty of alignment under few-shot scenario. Finally, the proposed Cross-modality attention fusion (CMAF) module can effectively fuse different modality feature. Experiment results have demonstrated our method's effectiveness over several defect classification methods for the ASE dataset.

* * *

[[2404.05185] Convergence analysis of controlled particle systems arising in deep learning: from finite to infinite sample size](https://arxiv.org/abs/2404.05185)
    
    
    Mon, 8 Apr 2024 04:22:55 GMT
    Huafu Liao, Alp\'ar R. M\'esz\'aros, Chenchen Mou, Chao Zhou
    

This paper deals with a class of neural SDEs and studies the limiting behavior of the associated sampled optimal control problems as the sample size grows to infinity. The neural SDEs with N samples can be linked to the N-particle systems with centralized control. We analyze the Hamilton--Jacobi--Bellman equation corresponding to the N-particle system and establish regularity results which are uniform in N. The uniform regularity estimates are obtained by the stochastic maximum principle and the analysis of a backward stochastic Riccati equation. Using these uniform regularity results, we show the convergence of the minima of objective functionals and optimal parameters of the neural SDEs as the sample size N tends to infinity. The limiting objects can be identified with suitable functions defined on the Wasserstein space of Borel probability measures. Furthermore, quantitative algebraic convergence rates are also obtained.

* * *

[[2404.05465] HAMMR: HierArchical MultiModal React agents for generic VQA](https://arxiv.org/abs/2404.05465)
    
    
    Mon, 8 Apr 2024 12:43:32 GMT
    Lluis Castrejon, Thomas Mensink, Howard Zhou, Vittorio Ferrari, Andre Araujo, Jasper Uijlings
    

Combining Large Language Models (LLMs) with external specialized tools (LLMs+tools) is a recent paradigm to solve multimodal tasks such as Visual Question Answering (VQA). While this approach was demonstrated to work well when optimized and evaluated for each individual benchmark, in practice it is crucial for the next generation of real-world AI systems to handle a broad range of multimodal problems. Therefore we pose the VQA problem from a unified perspective and evaluate a single system on a varied suite of VQA tasks including counting, spatial reasoning, OCR-based reasoning, visual pointing, external knowledge, and more. In this setting, we demonstrate that naively applying the LLM+tools approach using the combined set of all tools leads to poor results. This motivates us to introduce HAMMR: HierArchical MultiModal React. We start from a multimodal ReAct-based system and make it hierarchical by enabling our HAMMR agents to call upon other specialized agents. This enhances the compositionality of the LLM+tools approach, which we show to be critical for obtaining high accuracy on generic VQA. Concretely, on our generic VQA suite, HAMMR outperforms the naive LLM+tools approach by 19.5%. Additionally, HAMMR achieves state-of-the-art results on this task, outperforming the generic standalone PaLI-X VQA model by 5.0%.

* * *

[[2310.00280] Corex: Pushing the Boundaries of Complex Reasoning through Multi-Model Collaboration](https://arxiv.org/abs/2310.00280)
    
    
    replaced with revised version Sun, 7 Apr 2024 16:20:59 GMT
    Submission history From: Qiushi Sun [view email]
    [v1] Sat, 30 Sep 2023 07:11:39 UTC (1,003 KB)
    [v2] Sun, 7 Apr 2024 16:20:59 UTC (816 KB)
    Qiushi Sun, Zhangyue Yin, Xiang Li, Zhiyong Wu, Xipeng Qiu, Lingpeng Kong
    

Large Language Models (LLMs) are evolving at an unprecedented pace and have exhibited considerable capability in the realm of natural language processing (NLP) with world knowledge. Benefiting from ultra-large-scale training corpora, a single LLM can manage typical NLP tasks competently. However, its performance in executing reasoning tasks is still confined by the limitations of its internal representations. To push this boundary further, we introduce Corex in this paper, a suite of novel general-purpose strategies that transform LLMs into autonomous agents pioneering multi-model collaborations for complex task-solving. Inspired by human behaviors, Corex is constituted by diverse collaboration paradigms including Debate, Review, and Retrieve modes, which collectively work towards enhancing the factuality, faithfulness, and reliability of the reasoning process. These paradigms foster task-agnostic approaches that enable LLMs to ''think outside the box,'' thereby overcoming hallucinations and providing better solutions. Through extensive experiments across four different types of reasoning tasks, we demonstrate that orchestrating multiple LLMs to work in concert yields substantially better performance compared to existing methods. Further results and in-depth analysis demonstrate the cost-effectiveness of our method, facilitating collaboration among different LLMs and promoting annotation efficiency.

* * *

[[2310.06174] Cost-Efficient Prompt Engineering for Unsupervised Entity Resolution](https://arxiv.org/abs/2310.06174)
    
    
    replaced with revised version Sat, 6 Apr 2024 22:59:54 GMT
    Submission history From: Navapat Nananukul [view email]
    [v1] Mon, 9 Oct 2023 21:57:07 UTC (248 KB)
    [v2] Sat, 6 Apr 2024 22:59:54 UTC (13,848 KB)
    Navapat Nananukul, Khanin Sisaengsuwanchai, Mayank Kejriwal
    

Entity Resolution (ER) is the problem of semi-automatically determining when two entities refer to the same underlying entity, with applications ranging from healthcare to e-commerce. Traditional ER solutions required considerable manual expertise, including domain-specific feature engineering, as well as identification and curation of training data. Recently released large language models (LLMs) provide an opportunity to make ER more seamless and domain-independent. However, it is also well known that LLMs can pose risks, and that the quality of their outputs can depend on how prompts are engineered. Unfortunately, a systematic experimental study on the effects of different prompting methods for addressing unsupervised ER, using LLMs like ChatGPT, has been lacking thus far. This paper aims to address this gap by conducting such a study. We consider some relatively simple and cost-efficient ER prompt engineering methods and apply them to ER on two real-world datasets widely used in the community. We use an extensive set of experimental results to show that an LLM like GPT3.5 is viable for high-performing unsupervised ER, and interestingly, that more complicated and detailed (and hence, expensive) prompting methods do not necessarily outperform simpler approaches. We provide brief discussions on qualitative and error analysis, including a study of the inter-consistency of different prompting methods to determine whether they yield stable outputs. Finally, we consider some limitations of LLMs when applied to ER.

* * *

[[2402.05359] Prompting with Divide-and-Conquer Program Makes Large Language Models Discerning to Hallucination and Deception](https://arxiv.org/abs/2402.05359)
    
    
    replaced with revised version Fri, 5 Apr 2024 23:22:07 GMT
    Submission history From: Yizhou Zhang [view email]
    [v1] Thu, 8 Feb 2024 02:37:30 UTC (585 KB)
    [v2] Mon, 11 Mar 2024 23:15:10 UTC (585 KB)
    [v3] Thu, 14 Mar 2024 21:12:42 UTC (585 KB)
    [v4] Fri, 5 Apr 2024 23:22:07 UTC (586 KB)
    Yizhou Zhang, Lun Du, Defu Cao, Qiang Fu, Yan Liu
    

Foundation models, such as Large language Models (LLMs), have attracted significant amount of interest due to their large number of applications. Existing works show that appropriate prompt design, such as Chain-of-Thoughts, can unlock LLM's powerful capacity in diverse areas. However, when handling tasks involving repetitive sub-tasks and/or deceptive contents, such as arithmetic calculation and article-level fake news detection, existing prompting strategies either suffers from insufficient expressive power or intermediate errors triggered by hallucination. To make LLM more discerning to such intermediate errors, we propose to guide LLM with a Divide-and-Conquer program that simultaneously ensures superior expressive power and disentangles task decomposition, sub-task resolution, and resolution assembly process. Theoretic analysis reveals that our strategy can guide LLM to extend the expressive power of fixed-depth Transformer. Experiments indicate that our proposed method can achieve better performance than typical prompting strategies in tasks bothered by intermediate errors and deceptive contents, such as large integer multiplication, hallucination detection and misinformation detection.

* * *

[[2304.06875] nanoLM: an Affordable LLM Pre-training Benchmark via Accurate Loss Prediction across Scales](https://arxiv.org/abs/2304.06875)
    
    
    replaced with revised version Sat, 6 Apr 2024 05:50:39 GMT
    Submission history From: Yiqun Yao [view email]
    [v1] Fri, 14 Apr 2023 00:45:01 UTC (538 KB)
    [v2] Sat, 29 Apr 2023 03:14:58 UTC (242 KB)
    [v3] Sun, 3 Sep 2023 06:55:28 UTC (259 KB)
    [v4] Sat, 6 Apr 2024 05:50:39 UTC (928 KB)
    Yiqun Yao, Siqi fan, Xiusheng Huang, Xuezhi Fang, Xiang Li, Ziyi Ni, Xin Jiang, Xuying Meng, Peng Han, Shuo Shang, Kang Liu, Aixin Sun, Yequan Wang
    

As language models scale up, it becomes increasingly expensive to verify research ideas because conclusions on small models do not trivially transfer to large ones. A possible solution is to establish a generic system that accurately predicts certain metrics for large models without training them. Existing scaling laws require hyperparameter search on the largest models, limiting their predicative capability. In this paper, we present an approach (namely {mu}Scaling) to predict the pre-training loss, based on our observations that Maximal Update Parametrization ({mu}P) enables accurate fitting of scaling laws close to common loss basins in hyperparameter space. With {mu}Scaling, different model designs can be compared on large scales by training only their smaller counterparts. Further, we introduce nanoLM: an affordable LLM pre-training benchmark that facilitates this new research paradigm. With around 14% of the one-time pre-training cost, we can accurately forecast the loss for models up to 52B. Our goal with nanoLM is to empower researchers with limited resources to reach meaningful conclusions on large models. We also aspire for our benchmark to serve as a bridge between the academic community and the industry. Code for {mu}Scaling is available at this https URL. Code for nanoLLM will be available later.

* * *

[[2305.02869] Masked Structural Growth for 2x Faster Language Model Pre-training](https://arxiv.org/abs/2305.02869)
    
    
    replaced with revised version Sat, 6 Apr 2024 06:18:26 GMT
    Submission history From: Yiqun Yao [view email]
    [v1] Thu, 4 May 2023 14:28:39 UTC (316 KB)
    [v2] Fri, 8 Mar 2024 08:54:08 UTC (555 KB)
    [v3] Sat, 6 Apr 2024 06:18:26 UTC (555 KB)
    Yiqun Yao, Zheng Zhang, Jing Li, and Yequan Wang
    

Accelerating large language model pre-training is a critical issue in present research. In this paper, we focus on speeding up pre-training by progressively growing from a small Transformer structure to a large one. There are two main research problems associated with progressive growth: determining the optimal growth schedule, and designing efficient growth operators. In terms of growth schedule, the impact of each single dimension on a schedule's efficiency is under-explored by existing work. Regarding the growth operators, existing methods rely on the initialization of new weights to inherit knowledge, and achieve only non-strict function preservation, limiting further improvements on training dynamics. To address these issues, we propose Masked Structural Growth (MSG), including (i) growth schedules involving all possible dimensions and (ii) strictly function-preserving growth operators that is independent of the initialization of new weights. Experiments show that MSG is significantly faster than related work: we achieve up to 2.2x speedup in pre-training different types of language models while maintaining comparable or better downstream performances. Code is publicly available at this https URL.

* * *

[[2305.09620] AI-Augmented Surveys: Leveraging Large Language Models and Surveys for Opinion Prediction](https://arxiv.org/abs/2305.09620)
    
    
    replaced with revised version Sun, 7 Apr 2024 02:10:04 GMT
    Submission history From: Byungkyu Lee [view email]
    [v1] Tue, 16 May 2023 17:13:07 UTC (6,333 KB)
    [v2] Sun, 26 Nov 2023 16:25:49 UTC (6,277 KB)
    [v3] Sun, 7 Apr 2024 02:10:04 UTC (6,266 KB)
    Junsol Kim, Byungkyu Lee
    

Large language models (LLMs) that produce human-like responses have begun to revolutionize research practices in the social sciences. We develop a novel methodological framework that fine-tunes LLMs with repeated cross-sectional surveys to incorporate the meaning of survey questions, individual beliefs, and temporal contexts for opinion prediction. We introduce two new emerging applications of the AI-augmented survey: retrodiction (i.e., predict year-level missing responses) and unasked opinion prediction (i.e., predict entirely missing responses). Among 3,110 binarized opinions from 68,846 Americans in the General Social Survey from 1972 to 2021, our models based on Alpaca-7b excel in retrodiction (AUC = 0.86 for personal opinion prediction, $rho$ = 0.98 for public opinion prediction). These remarkable prediction capabilities allow us to fill in missing trends with high confidence and pinpoint when public attitudes changed, such as the rising support for same-sex marriage. On the other hand, our fine-tuned Alpaca-7b models show modest success in unasked opinion prediction (AUC = 0.73, $rho$ = 0.67). We discuss practical constraints and ethical concerns regarding individual autonomy and privacy when using LLMs for opinion prediction. Our study demonstrates that LLMs and surveys can mutually enhance each other's capabilities: LLMs can broaden survey potential, while surveys can improve the alignment of LLMs.

* * *

[[2305.14970] Getting Sick After Seeing a Doctor? Diagnosing and Mitigating Knowledge Conflicts in Event Temporal Reasoning](https://arxiv.org/abs/2305.14970)
    
    
    replaced with revised version Mon, 8 Apr 2024 14:59:53 GMT
    Submission history From: Tianqing Fang [view email]
    [v1] Wed, 24 May 2023 10:04:06 UTC (161 KB)
    [v2] Mon, 8 Apr 2024 14:59:53 UTC (593 KB)
    Tianqing Fang, Zhaowei Wang, Wenxuan Zhou, Hongming Zhang, Yangqiu Song, Muhao Chen
    

Event temporal reasoning aims at identifying the temporal relations between two or more events from narratives. However, knowledge conflicts arise when there is a mismatch between the actual temporal relations of events in the context and the prior knowledge or biases learned by the model. In this paper, we propose to detect knowledge-conflict examples in event temporal reasoning using bias indicators, which include event relation prior bias, tense bias, narrative bias, and dependency bias. We define conflict examples as those where event relations are opposite to biased or prior relations. To mitigate event-related knowledge conflicts, we introduce a Counterfactual Data Augmentation (CDA) based method that can be applied to both Pre-trained Language Models (PLMs) and Large Language Models (LLMs) either as additional training data or demonstrations for In-Context Learning. Experiments suggest both PLMs and LLMs suffer from knowledge conflicts in event temporal reasoning, and CDA has the potential for reducing hallucination and improving model performance.

* * *

[[2305.15047] Ghostbuster: Detecting Text Ghostwritten by Large Language Models](https://arxiv.org/abs/2305.15047)
    
    
    replaced with revised version Fri, 5 Apr 2024 23:57:35 GMT
    Submission history From: Nicholas Tomlin [view email]
    [v1] Wed, 24 May 2023 11:37:10 UTC (158 KB)
    [v2] Mon, 13 Nov 2023 18:43:11 UTC (898 KB)
    [v3] Fri, 5 Apr 2024 23:57:35 UTC (688 KB)
    Vivek Verma, Eve Fleisig, Nicholas Tomlin, Dan Klein
    

We introduce Ghostbuster, a state-of-the-art system for detecting AI-generated text. Our method works by passing documents through a series of weaker language models, running a structured search over possible combinations of their features, and then training a classifier on the selected features to predict whether documents are AI-generated. Crucially, Ghostbuster does not require access to token probabilities from the target model, making it useful for detecting text generated by black-box models or unknown model versions. In conjunction with our model, we release three new datasets of human- and AI-generated text as detection benchmarks in the domains of student essays, creative writing, and news articles. We compare Ghostbuster to a variety of existing detectors, including DetectGPT and GPTZero, as well as a new RoBERTa baseline. Ghostbuster achieves 99.0 F1 when evaluated across domains, which is 5.9 F1 higher than the best preexisting model. It also outperforms all previous approaches in generalization across writing domains (+7.5 F1), prompting strategies (+2.1 F1), and language models (+4.4 F1). We also analyze the robustness of our system to a variety of perturbations and paraphrasing attacks and evaluate its performance on documents written by non-native English speakers.

* * *

[[2307.04642] TRAQ: Trustworthy Retrieval Augmented Question Answering via Conformal Prediction](https://arxiv.org/abs/2307.04642)
    
    
    replaced with revised version Fri, 5 Apr 2024 20:08:57 GMT
    Submission history From: Shuo Li [view email]
    [v1] Fri, 7 Jul 2023 02:42:06 UTC (410 KB)
    [v2] Fri, 5 Apr 2024 20:08:57 UTC (8,945 KB)
    Shuo Li, Sangdon Park, Insup Lee, Osbert Bastani
    

When applied to open-domain question answering, large language models (LLMs) frequently generate incorrect responses based on made-up facts, which are called $textit{hallucinations}$. Retrieval augmented generation (RAG) is a promising strategy to avoid hallucinations, but it does not provide guarantees on its correctness. To address this challenge, we propose the Trustworthy Retrieval Augmented Question Answering, or $textit{TRAQ}$, which provides the first end-to-end statistical correctness guarantee for RAG. TRAQ uses conformal prediction, a statistical technique for constructing prediction sets that are guaranteed to contain the semantically correct response with high probability. Additionally, TRAQ leverages Bayesian optimization to minimize the size of the constructed sets. In an extensive experimental evaluation, we demonstrate that TRAQ provides the desired correctness guarantee while reducing prediction set size by 16.2% on average compared to an ablation. The implementation is available at $href{this https URL}{TRAQ}$.

* * *

[[2308.05361] WeaverBird: Empowering Financial Decision-Making with Large Language Model, Knowledge Base, and Search Engine](https://arxiv.org/abs/2308.05361)
    
    
    replaced with revised version Sun, 7 Apr 2024 03:26:38 GMT
    Submission history From: Siqiao Xue [view email]
    [v1] Thu, 10 Aug 2023 06:08:20 UTC (1,008 KB)
    [v2] Wed, 30 Aug 2023 03:11:56 UTC (1,015 KB)
    [v3] Sat, 2 Dec 2023 12:24:14 UTC (22,130 KB)
    [v4] Sun, 7 Apr 2024 03:26:38 UTC (22,129 KB)
    Siqiao Xue, Fan Zhou, Yi Xu, Ming Jin, Qingsong Wen, Hongyan Hao, Qingyang Dai, Caigao Jiang, Hongyu Zhao, Shuo Xie, Jianshan He, James Zhang, Hongyuan Mei
    

We present WeaverBird, an intelligent dialogue system designed specifically for the finance domain. Our system harnesses a large language model of GPT architecture that has been tuned using extensive corpora of finance-related text. As a result, our system possesses the capability to understand complex financial queries, such as "How should I manage my investments during inflation?", and provide informed responses. Furthermore, our system incorporates a local knowledge base and a search engine to retrieve relevant information. The final responses are conditioned on the search results and include proper citations to the sources, thus enjoying an enhanced credibility. Through a range of finance-related questions, we have demonstrated the superior performance of our system compared to other models. To experience our system firsthand, users can interact with our live demo at this https URL, as well as watch our 2-min video illustration at this https URL.

* * *

[[2308.12032] From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning](https://arxiv.org/abs/2308.12032)
    
    
    replaced with revised version Sat, 6 Apr 2024 03:52:04 GMT
    Submission history From: Ming Li [view email]
    [v1] Wed, 23 Aug 2023 09:45:29 UTC (8,121 KB)
    [v2] Fri, 8 Sep 2023 05:11:01 UTC (11,510 KB)
    [v3] Fri, 15 Sep 2023 20:33:44 UTC (11,508 KB)
    [v4] Tue, 20 Feb 2024 02:26:47 UTC (11,605 KB)
    [v5] Sat, 6 Apr 2024 03:52:04 UTC (11,610 KB)
    Ming Li, Yong Zhang, Zhitao Li, Jiuhai Chen, Lichang Chen, Ning Cheng, Jianzong Wang, Tianyi Zhou, Jing Xiao
    

In the realm of Large Language Models (LLMs), the balance between instruction data quality and quantity is a focal point. Recognizing this, we introduce a self-guided methodology for LLMs to autonomously discern and select cherry samples from open-source datasets, effectively minimizing manual curation and potential cost for instruction tuning an LLM. Our key innovation, the Instruction-Following Difficulty (IFD) metric, emerges as a pivotal metric to identify discrepancies between a model's expected responses and its intrinsic generation capability. Through the application of IFD, cherry samples can be pinpointed, leading to a marked uptick in model training efficiency. Empirical validations on datasets like Alpaca and WizardLM underpin our findings; with a mere $10%$ of original data input, our strategy showcases improved results. This synthesis of self-guided cherry-picking and the IFD metric signifies a transformative leap in the instruction tuning of LLMs, promising both efficiency and resource-conscious advancements. Codes, data, and models are available: this https URL

* * *

[[2309.10182] Positive and Risky Message Assessment for Music Products](https://arxiv.org/abs/2309.10182)
    
    
    replaced with revised version Mon, 8 Apr 2024 07:31:28 GMT
    Submission history From: Yigeng Zhang [view email]
    [v1] Mon, 18 Sep 2023 22:20:13 UTC (473 KB)
    [v2] Mon, 8 Apr 2024 07:31:28 UTC (1,295 KB)
    Yigeng Zhang, Mahsa Shafaei, Fabio A. Gonz\'alez, Thamar Solorio
    

In this work, we introduce a pioneering research challenge: evaluating positive and potentially harmful messages within music products. We initiate by setting a multi-faceted, multi-task benchmark for music content assessment. Subsequently, we introduce an efficient multi-task predictive model fortified with ordinality-enforcement to address this challenge. Our findings reveal that the proposed method not only significantly outperforms robust task-specific alternatives but also possesses the capability to assess multiple aspects simultaneously. Furthermore, through detailed case studies, where we employed Large Language Models (LLMs) as surrogates for content assessment, we provide valuable insights to inform and guide future research on this topic. The code for dataset creation and model implementation is publicly available at this https URL.

* * *

[[2309.17453] Efficient Streaming Language Models with Attention Sinks](https://arxiv.org/abs/2309.17453)
    
    
    replaced with revised version Sun, 7 Apr 2024 00:56:53 GMT
    Submission history From: Guangxuan Xiao [view email]
    [v1] Fri, 29 Sep 2023 17:59:56 UTC (9,548 KB)
    [v2] Tue, 21 Nov 2023 05:04:49 UTC (13,230 KB)
    [v3] Tue, 12 Dec 2023 00:33:07 UTC (13,230 KB)
    [v4] Sun, 7 Apr 2024 00:56:53 UTC (13,231 KB)
    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, Mike Lewis
    

Deploying Large Language Models (LLMs) in streaming applications such as multi-round dialogue, where long interactions are expected, is urgently needed but poses two major challenges. Firstly, during the decoding stage, caching previous tokens' Key and Value states (KV) consumes extensive memory. Secondly, popular LLMs cannot generalize to longer texts than the training sequence length. Window attention, where only the most recent KVs are cached, is a natural approach -- but we show that it fails when the text length surpasses the cache size. We observe an interesting phenomenon, namely attention sink, that keeping the KV of initial tokens will largely recover the performance of window attention. In this paper, we first demonstrate that the emergence of attention sink is due to the strong attention scores towards initial tokens as a "sink" even if they are not semantically important. Based on the above analysis, we introduce StreamingLLM, an efficient framework that enables LLMs trained with a finite length attention window to generalize to infinite sequence lengths without any fine-tuning. We show that StreamingLLM can enable Llama-2, MPT, Falcon, and Pythia to perform stable and efficient language modeling with up to 4 million tokens and more. In addition, we discover that adding a placeholder token as a dedicated attention sink during pre-training can further improve streaming deployment. In streaming settings, StreamingLLM outperforms the sliding window recomputation baseline by up to 22.2x speedup. Code and datasets are provided at this https URL.

* * *

[[2310.00322] Red Teaming Game: A Game-Theoretic Framework for Red Teaming Language Models](https://arxiv.org/abs/2310.00322)
    
    
    replaced with revised version Sat, 6 Apr 2024 16:48:20 GMT
    Submission history From: Chengdong Ma [view email]
    [v1] Sat, 30 Sep 2023 09:35:50 UTC (1,186 KB)
    [v2] Tue, 10 Oct 2023 09:00:40 UTC (2,526 KB)
    [v3] Mon, 1 Apr 2024 09:09:16 UTC (2,517 KB)
    [v4] Sat, 6 Apr 2024 16:48:20 UTC (2,519 KB)
    Chengdong Ma, Ziran Yang, Minquan Gao, Hai Ci, Jun Gao, Xuehai Pan, Yaodong Yang
    

Deployable Large Language Models (LLMs) must conform to the criterion of helpfulness and harmlessness, thereby achieving consistency between LLMs outputs and human values. Red-teaming techniques constitute a critical way towards this criterion. Existing work rely solely on manual red team designs and heuristic adversarial prompts for vulnerability detection and optimization. These approaches lack rigorous mathematical formulation, thus limiting the exploration of diverse attack strategy within quantifiable measure and optimization of LLMs under convergence guarantees. In this paper, we present Red-teaming Game (RTG), a general game-theoretic framework without manual annotation. RTG is designed for analyzing the multi-turn attack and defense interactions between Red-team language Models (RLMs) and Blue-team Language Model (BLM). Within the RTG, we propose Gamified Red-teaming Solver (GRTS) with diversity measure of the semantic space. GRTS is an automated red teaming technique to solve RTG towards Nash equilibrium through meta-game analysis, which corresponds to the theoretically guaranteed optimization direction of both RLMs and BLM. Empirical results in multi-turn attacks with RLMs show that GRTS autonomously discovered diverse attack strategies and effectively improved security of LLMs, outperforming existing heuristic red-team designs. Overall, RTG has established a foundational framework for red teaming tasks and constructed a new scalable oversight technique for alignment.

* * *

[[2310.02168] Editing Personality for Large Language Models](https://arxiv.org/abs/2310.02168)
    
    
    replaced with revised version Sat, 6 Apr 2024 14:32:40 GMT
    Submission history From: Ningyu Zhang [view email]
    [v1] Tue, 3 Oct 2023 16:02:36 UTC (9,645 KB)
    [v2] Tue, 21 Nov 2023 18:18:49 UTC (2,471 KB)
    [v3] Sat, 6 Apr 2024 14:32:40 UTC (9,688 KB)
    Shengyu Mao, Xiaohan Wang, Mengru Wang, Yong Jiang, Pengjun Xie, Fei Huang, Ningyu Zhang
    

This paper introduces an innovative task focused on editing the personality traits of Large Language Models (LLMs). This task seeks to adjust the models' responses to opinion-related questions on specified topics since an individual's personality often manifests in the form of their expressed opinions, thereby showcasing different personality traits. Specifically, we construct a new benchmark dataset PersonalityEdit to address this task. Drawing on the theory in Social Psychology, we isolate three representative traits, namely Neuroticism, Extraversion, and Agreeableness, as the foundation for our benchmark. We then gather data using GPT-4, generating responses that not only align with a specified topic but also embody the targeted personality trait. We conduct comprehensive experiments involving various baselines and discuss the representation of personality behavior in LLMs. Our intriguing findings uncover potential challenges of the proposed task, illustrating several remaining issues. We anticipate that our work can provide the NLP community with insights. Code and datasets are available at this https URL.

* * *

[[2310.05046] FakeGPT: Fake News Generation, Explanation and Detection of Large Language Models](https://arxiv.org/abs/2310.05046)
    
    
    replaced with revised version Sat, 6 Apr 2024 08:10:43 GMT
    Submission history From: Yue Huang [view email]
    [v1] Sun, 8 Oct 2023 07:01:07 UTC (3,475 KB)
    [v2] Sat, 6 Apr 2024 08:10:43 UTC (1,254 KB)
    Yue Huang and Lichao Sun
    

The rampant spread of fake news has adversely affected society, resulting in extensive research on curbing its spread. As a notable milestone in large language models (LLMs), ChatGPT has gained significant attention due to its exceptional natural language processing capabilities. In this study, we present a thorough exploration of ChatGPT's proficiency in generating, explaining, and detecting fake news as follows. Generation -- We employ four prompt methods to generate fake news samples and prove the high quality of these samples through both self-assessment and human evaluation. Explanation -- We obtain nine features to characterize fake news based on ChatGPT's explanations and analyze the distribution of these factors across multiple public datasets. Detection -- We examine ChatGPT's capacity to identify fake news. We explore its detection consistency and then propose a reason-aware prompt method to improve its performance. Although our experiments demonstrate that ChatGPT shows commendable performance in detecting fake news, there is still room for its improvement. Consequently, we further probe into the potential extra information that could bolster its effectiveness in detecting fake news.

* * *

[[2310.08041] QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Models](https://arxiv.org/abs/2310.08041)
    
    
    replaced with revised version Sat, 6 Apr 2024 10:22:57 GMT
    Submission history From: Bohan Zhuang [view email]
    [v1] Thu, 12 Oct 2023 05:25:49 UTC (1,424 KB)
    [v2] Wed, 21 Feb 2024 06:40:49 UTC (1,656 KB)
    [v3] Sat, 6 Apr 2024 10:22:57 UTC (1,659 KB)
    Jing Liu, Ruihao Gong, Xiuying Wei, Zhiwei Dong, Jianfei Cai, Bohan Zhuang
    

Large Language Models (LLMs) excel in NLP, but their demands hinder their widespread deployment. While Quantization-Aware Training (QAT) offers a solution, its extensive training costs make Post-Training Quantization (PTQ) a more practical approach for LLMs. In existing studies, activation outliers in particular channels are identified as the bottleneck to PTQ accuracy. They propose to transform the magnitudes from activations to weights, which however offers limited alleviation or suffers from unstable gradients, resulting in a severe performance drop at low-bitwidth. In this paper, we propose QLLM, an accurate and efficient low-bitwidth PTQ method designed for LLMs. QLLM introduces an adaptive channel reassembly technique that reallocates the magnitude of outliers to other channels, thereby mitigating their impact on the quantization range. This is achieved by channel disassembly and channel assembly, which first breaks down the outlier channels into several sub-channels to ensure a more balanced distribution of activation magnitudes. Then similar channels are merged to maintain the original channel number for efficiency. Additionally, an adaptive strategy is designed to autonomously determine the optimal number of sub-channels for channel disassembly. To further compensate for the performance loss caused by quantization, we propose an efficient tuning method that only learns a small number of low-rank weights while freezing the pre-trained quantized model. After training, these low-rank parameters can be fused into the frozen weights without affecting inference. Extensive experiments on LLaMA-1 and LLaMA-2 show that QLLM can obtain accurate quantized models efficiently. For example, QLLM quantizes the 4-bit LLaMA-2-70B within 10 hours on a single A100-80G GPU, outperforming the previous state-of-the-art method by 7.89% on the average accuracy across five zero-shot tasks.

* * *

[[2310.10118] Learning to Rank Context for Named Entity Recognition Using a Synthetic Dataset](https://arxiv.org/abs/2310.10118)
    
    
    replaced with revised version Mon, 8 Apr 2024 13:53:48 GMT
    Submission history From: Arthur Amalvy [view email] [via CCSD proxy]
    [v1] Mon, 16 Oct 2023 06:53:12 UTC (499 KB)
    [v2] Mon, 6 Nov 2023 10:09:22 UTC (600 KB)
    [v3] Mon, 8 Apr 2024 13:53:48 UTC (600 KB)
    Arthur Amalvy (LIA), Vincent Labatut (LIA), Richard Dufour (LS2N - \'equipe TALN)
    

While recent pre-trained transformer-based models can perform named entity recognition (NER) with great accuracy, their limited range remains an issue when applied to long documents such as whole novels. To alleviate this issue, a solution is to retrieve relevant context at the document level. Unfortunately, the lack of supervision for such a task means one has to settle for unsupervised approaches. Instead, we propose to generate a synthetic context retrieval training dataset using Alpaca, an instructiontuned large language model (LLM). Using this dataset, we train a neural context retriever based on a BERT model that is able to find relevant context for NER. We show that our method outperforms several retrieval baselines for the NER task on an English literary dataset composed of the first chapter of 40 books.

* * *

[[2310.10648] Bridging the Novice-Expert Gap via Models of Decision-Making: A Case Study on Remediating Math Mistakes](https://arxiv.org/abs/2310.10648)
    
    
    replaced with revised version Sat, 6 Apr 2024 16:15:27 GMT
    Submission history From: Rose Wang [view email]
    [v1] Mon, 16 Oct 2023 17:59:50 UTC (1,957 KB)
    [v2] Mon, 25 Mar 2024 16:56:39 UTC (1,863 KB)
    [v3] Sat, 6 Apr 2024 16:15:27 UTC (1,797 KB)
    Rose E. Wang, Qingyang Zhang, Carly Robinson, Susanna Loeb, Dorottya Demszky
    

Scaling high-quality tutoring remains a major challenge in education. Due to growing demand, many platforms employ novice tutors who, unlike experienced educators, struggle to address student mistakes and thus fail to seize prime learning opportunities. Our work explores the potential of large language models (LLMs) to close the novice-expert knowledge gap in remediating math mistakes. We contribute Bridge, a method that uses cognitive task analysis to translate an expert's latent thought process into a decision-making model for remediation. This involves an expert identifying (A) the student's error, (B) a remediation strategy, and (C) their intention before generating a response. We construct a dataset of 700 real tutoring conversations, annotated by experts with their decisions. We evaluate state-of-the-art LLMs on our dataset and find that the expert's decision-making model is critical for LLMs to close the gap: responses from GPT4 with expert decisions (e.g., "simplify the problem") are +76% more preferred than without. Additionally, context-sensitive decisions are critical to closing pedagogical gaps: random decisions decrease GPT4's response quality by -97% than expert decisions. Our work shows the potential of embedding expert thought processes in LLM generations to enhance their capability to bridge novice-expert knowledge gaps. Our dataset and code can be found at: url{this https URL}.

* * *

[[2310.15147] S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models](https://arxiv.org/abs/2310.15147)
    
    
    replaced with revised version Sat, 6 Apr 2024 15:20:18 GMT
    Submission history From: Fangyu Lei [view email]
    [v1] Mon, 23 Oct 2023 17:52:06 UTC (1,274 KB)
    [v2] Sat, 6 Apr 2024 15:20:18 UTC (1,592 KB)
    Fangyu Lei, Qian Liu, Yiming Huang, Shizhu He, Jun Zhao, Kang Liu
    

The rapid development of Large Language Models (LLMs) has led to great strides in model capabilities like long-context understanding and reasoning. However, as LLMs are able to process longer contexts, it becomes more challenging to evaluate whether they have acquired certain capabilities, since the length of text (e.g., 200K tokens) they can process far exceeds what humans can reliably assess in a reasonable duration. In this paper, we propose using complex synthetic tasks as a proxy evaluation method, and present S3Eval, a Synthetic, Scalable, Systematic evaluation suite for LLMs evaluation. The synthetic nature of S3Eval provides users full control over the dataset, allowing them to systematically probe LLM capabilities by scaling text length and varying task difficulty across diverse scenarios. The strong correlation between S3Eval and real-world benchmarks demonstrates the soundness of using S3Eval for evaluation of LLMs. S3Eval provides a flexible and infinite long-context data generation method. We have generated a comprehensive dataset called S3Eval-Standard, and experimental results have shown that it poses significant challenges for all existing LLMs.

* * *

[[2311.03839] Aspects of human memory and Large Language Models](https://arxiv.org/abs/2311.03839)
    
    
    replaced with revised version Mon, 8 Apr 2024 13:47:49 GMT
    Submission history From: Romuald A. Janik [view email]
    [v1] Tue, 7 Nov 2023 09:39:12 UTC (536 KB)
    [v2] Thu, 9 Nov 2023 18:16:24 UTC (537 KB)
    [v3] Mon, 8 Apr 2024 13:47:49 UTC (537 KB)
    Romuald A. Janik
    

Large Language Models (LLMs) are huge artificial neural networks which primarily serve to generate text, but also provide a very sophisticated probabilistic model of language use. Since generating a semantically consistent text requires a form of effective memory, we investigate the memory properties of LLMs and find surprising similarities with key characteristics of human memory. We argue that the human-like memory properties of the Large Language Model do not follow automatically from the LLM architecture but are rather learned from the statistics of the training textual data. These results strongly suggest that the biological features of human memory leave an imprint on the way that we structure our textual narratives.

* * *

[[2311.05553] Removing RLHF Protections in GPT-4 via Fine-Tuning](https://arxiv.org/abs/2311.05553)
    
    
    replaced with revised version Fri, 5 Apr 2024 23:30:56 GMT
    Submission history From: Qiusi Zhan [view email]
    [v1] Thu, 9 Nov 2023 17:54:59 UTC (110 KB)
    [v2] Fri, 10 Nov 2023 21:17:17 UTC (111 KB)
    [v3] Fri, 5 Apr 2024 23:30:56 UTC (374 KB)
    Qiusi Zhan, Richard Fang, Rohan Bindu, Akul Gupta, Tatsunori Hashimoto, Daniel Kang
    

As large language models (LLMs) have increased in their capabilities, so does their potential for dual use. To reduce harmful outputs, produces and vendors of LLMs have used reinforcement learning with human feedback (RLHF). In tandem, LLM vendors have been increasingly enabling fine-tuning of their most powerful models. However, concurrent work has shown that fine-tuning can remove RLHF protections. We may expect that the most powerful models currently available (GPT-4) are less susceptible to fine-tuning attacks. In this work, we show the contrary: fine-tuning allows attackers to remove RLHF protections with as few as 340 examples and a 95% success rate. These training examples can be automatically generated with weaker models. We further show that removing RLHF protections does not decrease usefulness on non-censored outputs, providing evidence that our fine-tuning strategy does not decrease usefulness despite using weaker models to generate training data. Our results show the need for further research on protections on LLMs.

* * *

[[2311.07092] To Tell The Truth: Language of Deception and Language Models](https://arxiv.org/abs/2311.07092)
    
    
    replaced with revised version Mon, 8 Apr 2024 05:54:40 GMT
    Submission history From: Bodhisattwa Prasad Majumder [view email]
    [v1] Mon, 13 Nov 2023 05:40:11 UTC (8,273 KB)
    [v2] Wed, 21 Feb 2024 06:53:12 UTC (8,223 KB)
    [v3] Mon, 8 Apr 2024 05:54:40 UTC (8,227 KB)
    Sanchaita Hazra, Bodhisattwa Prasad Majumder
    

Text-based misinformation permeates online discourses, yet evidence of people's ability to discern truth from such deceptive textual content is scarce. We analyze a novel TV game show data where conversations in a high-stake environment between individuals with conflicting objectives result in lies. We investigate the manifestation of potentially verifiable language cues of deception in the presence of objective truth, a distinguishing feature absent in previous text-based deception datasets. We show that there exists a class of detectors (algorithms) that have similar truth detection performance compared to human subjects, even when the former accesses only the language cues while the latter engages in conversations with complete access to all potential sources of cues (language and audio-visual). Our model, built on a large language model, employs a bottleneck framework to learn discernible cues to determine truth, an act of reasoning in which human subjects often perform poorly, even with incentives. Our model detects novel but accurate language cues in many cases where humans failed to detect deception, opening up the possibility of humans collaborating with algorithms and ameliorating their ability to detect the truth.

* * *

[[2311.08268] A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easily](https://arxiv.org/abs/2311.08268)
    
    
    replaced with revised version Sun, 7 Apr 2024 03:04:10 GMT
    Submission history From: Peng Ding [view email]
    [v1] Tue, 14 Nov 2023 16:02:16 UTC (9,188 KB)
    [v2] Sun, 18 Feb 2024 07:06:51 UTC (9,191 KB)
    [v3] Wed, 27 Mar 2024 13:29:31 UTC (9,519 KB)
    [v4] Sun, 7 Apr 2024 03:04:10 UTC (9,519 KB)
    Peng Ding, Jun Kuang, Dan Ma, Xuezhi Cao, Yunsen Xian, Jiajun Chen, Shujian Huang
    

Large Language Models (LLMs), such as ChatGPT and GPT-4, are designed to provide useful and safe responses. However, adversarial prompts known as 'jailbreaks' can circumvent safeguards, leading LLMs to generate potentially harmful content. Exploring jailbreak prompts can help to better reveal the weaknesses of LLMs and further steer us to secure them. Unfortunately, existing jailbreak methods either suffer from intricate manual design or require optimization on other white-box models, which compromises either generalization or efficiency. In this paper, we generalize jailbreak prompt attacks into two aspects: (1) Prompt Rewriting and (2) Scenario Nesting. Based on this, we propose ReNeLLM, an automatic framework that leverages LLMs themselves to generate effective jailbreak prompts. Extensive experiments demonstrate that ReNeLLM significantly improves the attack success rate while greatly reducing the time cost compared to existing baselines. Our study also reveals the inadequacy of current defense methods in safeguarding LLMs. Finally, we analyze the failure of LLMs defense from the perspective of prompt execution priority, and propose corresponding defense strategies. We hope that our research can catalyze both the academic community and LLMs developers towards the provision of safer and more regulated LLMs. The code is available at this https URL.

* * *

[[2311.09214] Mind's Mirror: Distilling Self-Evaluation Capability and Comprehensive Thinking from Large Language Models](https://arxiv.org/abs/2311.09214)
    
    
    replaced with revised version Sun, 7 Apr 2024 19:17:53 GMT
    Submission history From: Weize Liu [view email]
    [v1] Wed, 15 Nov 2023 18:56:23 UTC (7,964 KB)
    [v2] Mon, 25 Mar 2024 17:52:30 UTC (8,364 KB)
    [v3] Sun, 7 Apr 2024 19:17:53 UTC (8,364 KB)
    Weize Liu, Guocong Li, Kai Zhang, Bang Du, Qiyuan Chen, Xuming Hu, Hongxia Xu, Jintai Chen, Jian Wu
    

Large language models (LLMs) have achieved remarkable advancements in natural language processing. However, the massive scale and computational demands of these models present formidable challenges when considering their practical deployment in resource-constrained environments. While techniques such as chain-of-thought (CoT) distillation have displayed promise in distilling LLMs into small language models (SLMs), there is a risk that distilled SLMs may still inherit flawed reasoning and hallucinations from LLMs. To address these issues, we propose a twofold methodology: First, we introduce a novel method for distilling the self-evaluation capability from LLMs into SLMs, aiming to mitigate the adverse effects of flawed reasoning and hallucinations inherited from LLMs. Second, we advocate for distilling more comprehensive thinking by incorporating multiple distinct CoTs and self-evaluation outputs, to ensure a more thorough and robust knowledge transfer into SLMs. Experiments on three NLP benchmarks demonstrate that our method significantly improves the performance of distilled SLMs, offering a new perspective for developing more effective and efficient SLMs in resource-constrained environments.

* * *

[[2311.09702] Deceptive Semantic Shortcuts on Reasoning Chains: How Far Can Models Go without Hallucination?](https://arxiv.org/abs/2311.09702)
    
    
    replaced with revised version Fri, 5 Apr 2024 18:08:51 GMT
    Submission history From: Bangzheng Li [view email]
    [v1] Thu, 16 Nov 2023 09:27:36 UTC (7,940 KB)
    [v2] Wed, 13 Mar 2024 09:11:15 UTC (7,842 KB)
    [v3] Fri, 5 Apr 2024 18:08:51 UTC (7,999 KB)
    Bangzheng Li, Ben Zhou, Fei Wang, Xingyu Fu, Dan Roth, Muhao Chen
    

Despite the recent advancement in large language models (LLMs) and their high performances across numerous benchmarks, recent research has unveiled that LLMs suffer from hallucinations and unfaithful reasoning. This work studies a specific type of hallucination induced by semantic associations. Specifically, we investigate to what extent LLMs take shortcuts from certain keyword/entity biases in the prompt instead of following the correct reasoning path. To quantify this phenomenon, we propose a novel probing method and benchmark called EureQA. We start from questions that LLMs will answer correctly with utmost certainty, and mask the important entity with evidence sentence recursively, asking models to find masked entities according to a chain of evidence before answering the question. During the construction of the evidence, we purposefully replace semantic clues (entities) that may lead to the correct answer with distractor clues (evidence) that will not directly lead to the correct answer but require a chain-like reasoning process. We evaluate if models can follow the correct reasoning chain instead of short-cutting through distractor clues. We find that existing LLMs lack the necessary capabilities to follow correct reasoning paths and resist the attempt of greedy shortcuts. We show that the distractor semantic associations often lead to model hallucination, which is strong evidence that questions the validity of current LLM reasoning.

* * *

[[2312.09203] Measurement in the Age of LLMs: An Application to Ideological Scaling](https://arxiv.org/abs/2312.09203)
    
    
    replaced with revised version Mon, 8 Apr 2024 00:33:54 GMT
    Submission history From: Aaron Schein [view email]
    [v1] Thu, 14 Dec 2023 18:34:06 UTC (4,025 KB)
    [v2] Mon, 8 Apr 2024 00:33:54 UTC (3,952 KB)
    Sean O'Hagan and Aaron Schein
    

Much of social science is centered around terms like ``ideology'' or ``power'', which generally elude precise definition, and whose contextual meanings are trapped in surrounding language. This paper explores the use of large language models (LLMs) to flexibly navigate the conceptual clutter inherent to social scientific measurement tasks. We rely on LLMs' remarkable linguistic fluency to elicit ideological scales of both legislators and text, which accord closely to established methods and our own judgement. A key aspect of our approach is that we elicit such scores directly, instructing the LLM to furnish numeric scores itself. This approach affords a great deal of flexibility, which we showcase through a variety of different case studies. Our results suggest that LLMs can be used to characterize highly subtle and diffuse manifestations of political ideology in text.

* * *

[[2312.11779] Tokenization Matters: Navigating Data-Scarce Tokenization for Gender Inclusive Language Technologies](https://arxiv.org/abs/2312.11779)
    
    
    replaced with revised version Sat, 6 Apr 2024 09:32:53 GMT
    Submission history From: Anaelia Ovalle [view email]
    [v1] Tue, 19 Dec 2023 01:28:46 UTC (8,413 KB)
    [v2] Thu, 21 Dec 2023 11:45:55 UTC (9,972 KB)
    [v3] Sat, 6 Apr 2024 09:32:53 UTC (7,517 KB)
    Anaelia Ovalle, Ninareh Mehrabi, Palash Goyal, Jwala Dhamala, Kai-Wei Chang, Richard Zemel, Aram Galstyan, Yuval Pinter, Rahul Gupta
    

Gender-inclusive NLP research has documented the harmful limitations of gender binary-centric large language models (LLM), such as the inability to correctly use gender-diverse English neopronouns (e.g., xe, zir, fae). While data scarcity is a known culprit, the precise mechanisms through which scarcity affects this behavior remain underexplored. We discover LLM misgendering is significantly influenced by Byte-Pair Encoding (BPE) tokenization, the tokenizer powering many popular LLMs. Unlike binary pronouns, BPE overfragments neopronouns, a direct consequence of data scarcity during tokenizer training. This disparate tokenization mirrors tokenizer limitations observed in multilingual and low-resource NLP, unlocking new misgendering mitigation strategies. We propose two techniques: (1) pronoun tokenization parity, a method to enforce consistent tokenization across gendered pronouns, and (2) utilizing pre-existing LLM pronoun knowledge to improve neopronoun proficiency. Our proposed methods outperform finetuning with standard BPE, improving neopronoun accuracy from 14.1% to 58.4%. Our paper is the first to link LLM misgendering to tokenization and deficient neopronoun grammar, indicating that LLMs unable to correctly treat neopronouns as pronouns are more prone to misgender.

* * *

[[2312.13871] Evaluating Task-oriented Dialogue Systems: A Systematic Review of Measures, Constructs and their Operationalisations](https://arxiv.org/abs/2312.13871)
    
    
    replaced with revised version Mon, 8 Apr 2024 07:36:48 GMT
    Submission history From: Anouck Braggaar [view email]
    [v1] Thu, 21 Dec 2023 14:15:46 UTC (389 KB)
    [v2] Mon, 8 Apr 2024 07:36:48 UTC (413 KB)
    Anouck Braggaar, Christine Liebrecht, Emiel van Miltenburg, Emiel Krahmer
    

This review gives an extensive overview of evaluation methods for task-oriented dialogue systems, paying special attention to practical applications of dialogue systems, for example for customer service. The review (1) provides an overview of the used constructs and metrics in previous work, (2) discusses challenges in the context of dialogue system evaluation and (3) develops a research agenda for the future of dialogue system evaluation. We conducted a systematic review of four databases (ACL, ACM, IEEE and Web of Science), which after screening resulted in 122 studies. Those studies were carefully analysed for the constructs and methods they proposed for evaluation. We found a wide variety in both constructs and methods. Especially the operationalisation is not always clearly reported. Newer developments concerning large language models are discussed in two contexts: to power dialogue systems and to use in the evaluation process. We hope that future work will take a more critical approach to the operationalisation and specification of the used constructs. To work towards this aim, this review ends with recommendations for evaluation and suggestions for outstanding questions.

* * *

[[2401.15656] LLsM: Generative Linguistic Steganography with Large Language Model](https://arxiv.org/abs/2401.15656)
    
    
    replaced with revised version Mon, 8 Apr 2024 03:50:39 GMT
    Submission history From: Yihao Wang [view email]
    [v1] Sun, 28 Jan 2024 13:21:44 UTC (205 KB)
    [v2] Tue, 6 Feb 2024 14:55:56 UTC (241 KB)
    [v3] Mon, 8 Apr 2024 03:50:39 UTC (748 KB)
    Yihao Wang and Ruiqi Song and Ru Zhang and Jianyi Liu and Lingxiao Li
    

Linguistic Steganography (LS) tasks aim to generate steganographic text (stego) based on secret information. Only authorized recipients can perceive the existence of the stegos and extract secrets, thereby preserving privacy. However, existing LS methods do not consider the controllable generation of stegos containing specific discourses such as style, genre, and theme. And they are difficult to simulate high-quality natural texts. As a result, the stegos are easily perceived and detectable, compromising covert communication. This paper proposes the LLsM, the first LS work with the Large Language Model (LLM). Regarding open-source LLMs, we reconstruct the token generator of LLM to the "stego generator" so that it can control the generation of stego based on the secret. In this "stego generator", the candidate pool is encoded by range coding, and the adjustment factor for the interval length is also given. The secret determines the interval, thereby determining the next token. This better simulates the distribution of natural texts and controls the adjustment of the embedding rate. In addition, we preliminarily built an LLsM-c architecture for closed-source LLMs. It encodes discourse to obtain high-quality prompts containing discourse based on secrets, and generates pure natural texts containing discourse. Experiments show that LLsM performs superior to prevalent LS and related-task baselines regarding various kinds of concealment and anti-steganalysis. LLsM's MAUVE surpasses baselines by 60%-80% and anti-steganalysis exceeds baselines by 20%-30%. Notably, LLsM can also generate longer stegos with high quality, showing its advantages in understanding and coherence.

* * *

[[2401.17390] Customizing Language Model Responses with Contrastive In-Context Learning](https://arxiv.org/abs/2401.17390)
    
    
    replaced with revised version Mon, 8 Apr 2024 05:22:51 GMT
    Submission history From: Xiang Gao [view email]
    [v1] Tue, 30 Jan 2024 19:13:12 UTC (650 KB)
    [v2] Mon, 8 Apr 2024 05:22:51 UTC (650 KB)
    Xiang Gao, Kamalika Das
    

Large language models (LLMs) are becoming increasingly important for machine learning applications. However, it can be challenging to align LLMs with our intent, particularly when we want to generate content that is preferable over others or when we want the LLM to respond in a certain style or tone that is hard to describe. To address this challenge, we propose an approach that uses contrastive examples to better describe our intent. This involves providing positive examples that illustrate the true intent, along with negative examples that show what characteristics we want LLMs to avoid. The negative examples can be retrieved from labeled data, written by a human, or generated by the LLM itself. Before generating an answer, we ask the model to analyze the examples to teach itself what to avoid. This reasoning step provides the model with the appropriate articulation of the user's need and guides it towards generting a better answer. We tested our approach on both synthesized and real-world datasets, including StackExchange and Reddit, and found that it significantly improves performance compared to standard few-shot prompting

* * *

[[2402.05125] Zero-Shot Clinical Trial Patient Matching with LLMs](https://arxiv.org/abs/2402.05125)
    
    
    replaced with revised version Mon, 8 Apr 2024 01:43:43 GMT
    Submission history From: Michael Wornow [view email]
    [v1] Mon, 5 Feb 2024 00:06:08 UTC (1,046 KB)
    [v2] Mon, 8 Apr 2024 01:43:43 UTC (1,531 KB)
    [v3] Wed, 10 Apr 2024 05:37:26 UTC (1,531 KB)
    Michael Wornow, Alejandro Lozano, Dev Dash, Jenelle Jindal, Kenneth W. Mahaffey, Nigam H. Shah
    

Matching patients to clinical trials is a key unsolved challenge in bringing new drugs to market. Today, identifying patients who meet a trial's eligibility criteria is highly manual, taking up to 1 hour per patient. Automated screening is challenging, however, as it requires understanding unstructured clinical text. Large language models (LLMs) offer a promising solution. In this work, we explore their application to trial matching. First, we design an LLM-based system which, given a patient's medical history as unstructured clinical text, evaluates whether that patient meets a set of inclusion criteria (also specified as free text). Our zero-shot system achieves state-of-the-art scores on the n2c2 2018 cohort selection benchmark. Second, we improve the data and cost efficiency of our method by identifying a prompting strategy which matches patients an order of magnitude faster and more cheaply than the status quo, and develop a two-stage retrieval pipeline that reduces the number of tokens processed by up to a third while retaining high performance. Third, we evaluate the interpretability of our system by having clinicians evaluate the natural language justifications generated by the LLM for each eligibility decision, and show that it can output coherent explanations for 97% of its correct decisions and 75% of its incorrect ones. Our results establish the feasibility of using LLMs to accelerate clinical trial operations.

* * *

[[2402.14207] Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Models](https://arxiv.org/abs/2402.14207)
    
    
    replaced with revised version Mon, 8 Apr 2024 05:38:50 GMT
    Submission history From: Yijia Shao [view email]
    [v1] Thu, 22 Feb 2024 01:20:17 UTC (8,683 KB)
    [v2] Mon, 8 Apr 2024 05:38:50 UTC (8,683 KB)
    Yijia Shao, Yucheng Jiang, Theodore A. Kanell, Peter Xu, Omar Khattab, Monica S. Lam
    

We study how to apply large language models to write grounded and organized long-form articles from scratch, with comparable breadth and depth to Wikipedia pages. This underexplored problem poses new challenges at the pre-writing stage, including how to research the topic and prepare an outline prior to writing. We propose STORM, a writing system for the Synthesis of Topic Outlines through Retrieval and Multi-perspective Question Asking. STORM models the pre-writing stage by (1) discovering diverse perspectives in researching the given topic, (2) simulating conversations where writers carrying different perspectives pose questions to a topic expert grounded on trusted Internet sources, (3) curating the collected information to create an outline. For evaluation, we curate FreshWiki, a dataset of recent high-quality Wikipedia articles, and formulate outline assessments to evaluate the pre-writing stage. We further gather feedback from experienced Wikipedia editors. Compared to articles generated by an outline-driven retrieval-augmented baseline, more of STORM's articles are deemed to be organized (by a 25% absolute increase) and broad in coverage (by 10%). The expert feedback also helps identify new challenges for generating grounded long articles, such as source bias transfer and over-association of unrelated facts.

* * *

[[2402.14710] IEPile: Unearthing Large-Scale Schema-Based Information Extraction Corpus](https://arxiv.org/abs/2402.14710)
    
    
    replaced with revised version Mon, 8 Apr 2024 06:01:42 GMT
    Submission history From: Ningyu Zhang [view email]
    [v1] Thu, 22 Feb 2024 17:11:38 UTC (1,480 KB)
    [v2] Mon, 8 Apr 2024 06:01:42 UTC (1,475 KB)
    Honghao Gui, Lin Yuan, Hongbin Ye, Ningyu Zhang, Mengshu Sun, Lei Liang, Huajun Chen
    

Large Language Models (LLMs) demonstrate remarkable potential across various domains; however, they exhibit a significant performance gap in Information Extraction (IE). Note that high-quality instruction data is the vital key for enhancing the specific capabilities of LLMs, while current IE datasets tend to be small in scale, fragmented, and lack standardized schema. To this end, we introduce IEPile, a comprehensive bilingual (English and Chinese) IE instruction corpus, which contains approximately 0.32B tokens. We construct IEPile by collecting and cleaning 33 existing IE datasets, and introduce schema-based instruction generation to unearth a large-scale corpus. Experimental results on LLaMA, Baichuan and Qwen demonstrate that using IEPile can enhance the performance of LLMs for IE, especially the zero-shot generalization. We open-source the resource and pre-trained models, hoping to provide valuable support to the NLP community.

* * *

[[2402.15755] Dental Severity Assessment through Few-shot Learning and SBERT Fine-tuning](https://arxiv.org/abs/2402.15755)
    
    
    replaced with revised version Sat, 6 Apr 2024 08:18:57 GMT
    Submission history From: Mohammad Dehghani [view email]
    [v1] Sat, 24 Feb 2024 08:02:19 UTC (417 KB)
    [v2] Sat, 6 Apr 2024 08:18:57 UTC (420 KB)
    Mohammad Dehghani
    

Dental diseases have a significant impact on a considerable portion of the population, leading to various health issues that can detrimentally affect individuals' overall well-being. The integration of automated systems in oral healthcare has become increasingly crucial. Machine learning approaches offer a viable solution to address challenges such as diagnostic difficulties, inefficiencies, and errors in oral disease diagnosis. These methods prove particularly useful when physicians struggle to predict or diagnose diseases at their early stages. In this study, thirteen different machine learning, deep learning, and large language models were employed to determine the severity level of oral health issues based on radiologists' reports. The results revealed that the Few-shot learning with SBERT and Multi-Layer Perceptron model outperformed all other models across various experiments, achieving an impressive accuracy of 94.1% as the best result. Consequently, this model exhibits promise as a reliable tool for evaluating the severity of oral diseases, enabling patients to receive more effective treatment and aiding healthcare professionals in making informed decisions regarding resource allocation and the management of high-risk patients.

* * *

[[2402.16311] Cross-domain Chinese Sentence Pattern Parsing](https://arxiv.org/abs/2402.16311)
    
    
    replaced with revised version Mon, 8 Apr 2024 03:22:41 GMT
    Submission history From: Jingsi Yu [view email]
    [v1] Mon, 26 Feb 2024 05:30:48 UTC (3,568 KB)
    [v2] Tue, 27 Feb 2024 08:21:35 UTC (3,568 KB)
    [v3] Mon, 8 Apr 2024 03:22:41 UTC (3,567 KB)
    Jingsi Yu, Cunliang Kong, Liner Yang, Meishan Zhang, Lin Zhu, Yujie Wang, Haozhe Lin, Maosong Sun, Erhong Yang
    

Sentence Pattern Structure (SPS) parsing is a syntactic analysis method primarily employed in language teaching.Existing SPS parsers rely heavily on textbook corpora for training, lacking cross-domain this http URL overcome this constraint, this paper proposes an innovative approach leveraging large language models (LLMs) within a self-training framework. Partial syntactic rules from a source domain are combined with target domain sentences to dynamically generate training data, enhancing the adaptability of the parser to diverse domains.Experiments conducted on textbook and news domains demonstrate the effectiveness of the proposed method, outperforming rule-based baselines by 1.68 points on F1 metrics.

* * *

[[2403.03689] General2Specialized LLMs Translation for E-commerce](https://arxiv.org/abs/2403.03689)
    
    
    replaced with revised version Sat, 6 Apr 2024 04:07:49 GMT
    Submission history From: Kaidi Chen [view email]
    [v1] Wed, 6 Mar 2024 13:15:21 UTC (175 KB)
    [v2] Sat, 6 Apr 2024 04:07:49 UTC (175 KB)
    Kaidi Chen, Ben Chen, Dehong Gao, Huangyu Dai, Wen Jiang, Wei Ning, Shanqing Yu, Libin Yang, Xiaoyan Cai
    

Existing Neural Machine Translation (NMT) models mainly handle translation in the general domain, while overlooking domains with special writing formulas, such as e-commerce and legal documents. Taking e-commerce as an example, the texts usually include amounts of domain-related words and have more grammar problems, which leads to inferior performances of current NMT methods. To address these problems, we collect two domain-related resources, including a set of term pairs (aligned Chinese-English bilingual terms) and a parallel corpus annotated for the e-commerce domain. Furthermore, we propose a two-step fine-tuning paradigm (named G2ST) with self-contrastive semantic enhancement to transfer one general NMT model to the specialized NMT model for e-commerce. The paradigm can be used for the NMT models based on Large language models (LLMs). Extensive evaluations on real e-commerce titles demonstrate the superior translation quality and robustness of our G2ST approach, as compared with state-of-the-art NMT models such as LLaMA, Qwen, GPT-3.5, and even GPT-4.

* * *

[[2403.03888] FaaF: Facts as a Function for the evaluation of generated text](https://arxiv.org/abs/2403.03888)
    
    
    replaced with revised version Mon, 8 Apr 2024 14:49:52 GMT
    Submission history From: Vasileios Katranidis [view email]
    [v1] Wed, 6 Mar 2024 17:48:06 UTC (573 KB)
    [v2] Mon, 8 Apr 2024 14:49:52 UTC (609 KB)
    Vasileios Katranidis and Gabor Barany
    

The demand for accurate and efficient verification of information in texts generated by large language models (LMs) is at an all-time high, but remains unresolved. Recent efforts have focused on extracting and verifying atomic facts from these texts via prompting LM evaluators. However, we demonstrate that this method of prompting is unreliable when faced with incomplete or inaccurate reference information. We introduce Facts as a Function (FaaF), a new approach to the fact verification task that leverages the function-calling capabilities of LMs. FaaF significantly enhances the ability of LMs to identify unsupported facts in texts, while also improving efficiency and significantly lowering costs compared to prompt-based methods. Additionally, we propose a framework for evaluating factual recall in Retrieval Augmented Generation (RAG) systems, which we employ to compare prompt-based and FaaF methods using various LMs under challenging conditions.

* * *

[[2403.04787] Ever-Evolving Memory by Blending and Refining the Past](https://arxiv.org/abs/2403.04787)
    
    
    replaced with revised version Sun, 7 Apr 2024 04:31:30 GMT
    Submission history From: Seo Hyun Kim [view email]
    [v1] Sun, 3 Mar 2024 08:12:59 UTC (11,153 KB)
    [v2] Sun, 7 Apr 2024 04:31:30 UTC (4,195 KB)
    Seo Hyun Kim, Keummin Ka, Yohan Jo, Seung-won Hwang, Dongha Lee, Jinyoung Yeo
    

For a human-like chatbot, constructing a long-term memory is crucial. However, current large language models often lack this capability, leading to instances of missing important user information or redundantly asking for the same information, thereby diminishing conversation quality. To effectively construct memory, it is crucial to seamlessly connect past and present information, while also possessing the ability to forget obstructive information. To address these challenges, we propose CREEM, a novel memory system for long-term conversation. Improving upon existing approaches that construct memory based solely on current sessions, CREEM blends past memories during memory formation. Additionally, we introduce a refining process to handle redundant or outdated information. Unlike traditional paradigms, we view responding and memory construction as inseparable tasks. The blending process, which creates new memories, also serves as a reasoning step for response generation by informing the connection between past and present. Through evaluation, we demonstrate that CREEM enhances both memory and response qualities in multi-session personalized dialogues.

* * *

[[2403.07865] Exploring Safety Generalization Challenges of Large Language Models via Code](https://arxiv.org/abs/2403.07865)
    
    
    replaced with revised version Sun, 7 Apr 2024 15:39:24 GMT
    Submission history From: Qibing Ren [view email]
    [v1] Tue, 12 Mar 2024 17:55:38 UTC (8,284 KB)
    [v2] Thu, 14 Mar 2024 16:57:37 UTC (8,284 KB)
    [v3] Sun, 7 Apr 2024 15:39:24 UTC (8,288 KB)
    Qibing Ren, Chang Gao, Jing Shao, Junchi Yan, Xin Tan, Yu Qiao, Wai Lam, Lizhuang Ma
    

The rapid advancement of Large Language Models (LLMs) has brought about remarkable generative capabilities but also raised concerns about their potential misuse. While strategies like supervised fine-tuning and reinforcement learning from human feedback have enhanced their safety, these methods primarily focus on natural languages, which may not generalize to other domains. This paper introduces CodeAttack, a framework that transforms natural language inputs into code inputs, presenting a novel environment for testing the safety generalization of LLMs. Our comprehensive studies on state-of-the-art LLMs including GPT-4, Claude-2, and Llama-2 series reveal a common safety vulnerability of these models against code input: CodeAttack bypasses the safety guardrails of all models more than 80% of the time. We find that a larger distribution gap between CodeAttack and natural language leads to weaker safety generalization, such as encoding natural language input with data structures. Furthermore, we give two hypotheses about the success of CodeAttack: (1) the misaligned bias acquired by LLMs during code training, prioritizing code completion over avoiding the potential safety risk; (2) the limited self-evaluation capability regarding the safety of their code outputs. Finally, we analyze potential mitigation measures. These findings highlight new safety risks in the code domain and the need for more robust safety alignment algorithms to match the code capabilities of LLMs.

* * *

[[2403.11169] Correcting misinformation on social media with a large language model](https://arxiv.org/abs/2403.11169)
    
    
    replaced with revised version Sat, 6 Apr 2024 08:49:31 GMT
    Submission history From: Xinyi Zhou [view email]
    [v1] Sun, 17 Mar 2024 10:59:09 UTC (28,311 KB)
    [v2] Sat, 6 Apr 2024 08:49:31 UTC (32,292 KB)
    Xinyi Zhou, Ashish Sharma, Amy X. Zhang, Tim Althoff
    

Real-world misinformation can be partially correct and even factual but misleading. It undermines public trust in science and democracy, particularly on social media, where it can spread rapidly. High-quality and timely correction of misinformation that identifies and explains its (in)accuracies has been shown to effectively reduce false beliefs. Despite the wide acceptance of manual correction, it is difficult to promptly correct newly created misinformation and to scale this approach, a concern as technologies like large language models (LLMs) make misinformation easier to produce. LLMs also have versatile capabilities that could accelerate misinformation correction--however, they struggle due to a lack of recent information, a tendency to produce false content, and limitations in addressing multimodal information. We propose MUSE, an LLM augmented with access to and credibility evaluation of up-to-date information. By retrieving evidence as refutations or contexts, MUSE identifies and explains (in)accuracies in a piece of content--not presupposed to be misinformation--with references. It also describes images and conducts multimodal searches to verify and correct multimodal content. Fact-checking experts evaluate responses to social media content that are not presupposed to be (non-)misinformation but broadly include incorrect, partially correct, and correct posts, that may or may not be misleading. We propose and evaluate 13 dimensions of misinformation correction quality, ranging from the accuracy of identifications and factuality of explanations to the relevance and credibility of references. The results demonstrate MUSE's ability to promptly write high-quality responses to potential misinformation on social media--overall, MUSE outperforms GPT-4 by 37% and even high-quality responses from laypeople by 29%.

* * *

[[2403.14171] MMIDR: Teaching Large Language Model to Interpret Multimodal Misinformation via Knowledge Distillation](https://arxiv.org/abs/2403.14171)
    
    
    replaced with revised version Mon, 8 Apr 2024 08:30:06 GMT
    Submission history From: Longzheng Wang [view email]
    [v1] Thu, 21 Mar 2024 06:47:28 UTC (2,312 KB)
    [v2] Thu, 4 Apr 2024 05:13:56 UTC (2,312 KB)
    [v3] Mon, 8 Apr 2024 08:30:06 UTC (2,312 KB)
    Longzheng Wang, Xiaohan Xu, Lei Zhang, Jiarui Lu, Yongxiu Xu, Hongbo Xu, Minghao Tang, Chuang Zhang
    

Automatic detection of multimodal misinformation has gained a widespread attention recently. However, the potential of powerful Large Language Models (LLMs) for multimodal misinformation detection remains underexplored. Besides, how to teach LLMs to interpret multimodal misinformation in cost-effective and accessible way is still an open question. To address that, we propose MMIDR, a framework designed to teach LLMs in providing fluent and high-quality textual explanations for their decision-making process of multimodal misinformation. To convert multimodal misinformation into an appropriate instruction-following format, we present a data augmentation perspective and pipeline. This pipeline consists of a visual information processing module and an evidence retrieval module. Subsequently, we prompt the proprietary LLMs with processed contents to extract rationales for interpreting the authenticity of multimodal misinformation. Furthermore, we design an efficient knowledge distillation approach to distill the capability of proprietary LLMs in explaining multimodal misinformation into open-source LLMs. To explore several research questions regarding the performance of LLMs in multimodal misinformation detection tasks, we construct an instruction-following multimodal misinformation dataset and conduct comprehensive experiments. The experimental findings reveal that our MMIDR exhibits sufficient detection performance and possesses the capacity to provide compelling rationales to support its assessments.

* * *

[[2403.18349] Rejection Improves Reliability: Training LLMs to Refuse Unknown Questions Using RL from Knowledge Feedback](https://arxiv.org/abs/2403.18349)
    
    
    replaced with revised version Sun, 7 Apr 2024 07:41:28 GMT
    Submission history From: Hongshen Xu [view email]
    [v1] Wed, 27 Mar 2024 08:39:56 UTC (3,955 KB)
    [v2] Sun, 7 Apr 2024 07:41:28 UTC (3,696 KB)
    Hongshen Xu, Zichen Zhu, Situo Zhang, Da Ma, Shuai Fan, Lu Chen, Kai Yu
    

Large Language Models (LLMs) often generate erroneous outputs, known as hallucinations, due to their limitations in discerning questions beyond their knowledge scope. While addressing hallucination has been a focal point in research, previous efforts primarily concentrate on enhancing correctness without giving due consideration to the significance of rejection mechanisms. In this paper, we conduct a comprehensive examination of the role of rejection, introducing the notion of model reliability along with corresponding metrics. These metrics measure the model's ability to provide accurate responses while adeptly rejecting questions exceeding its knowledge boundaries, thereby minimizing hallucinations. To improve the inherent reliability of LLMs, we present a novel alignment framework called Reinforcement Learning from Knowledge Feedback (RLKF). RLKF leverages knowledge feedback to dynamically determine the model's knowledge boundary and trains a reliable reward model to encourage the refusal of out-of-knowledge questions. Experimental results on mathematical questions affirm the substantial efficacy of RLKF in significantly enhancing LLM reliability.

* * *

[[2404.01129] Structured Information Matters: Incorporating Abstract Meaning Representation into LLMs for Improved Open-Domain Dialogue Evaluation](https://arxiv.org/abs/2404.01129)
    
    
    replaced with revised version Sat, 6 Apr 2024 16:44:37 GMT
    Submission history From: Bohao Yang [view email]
    [v1] Mon, 1 Apr 2024 14:11:45 UTC (1,112 KB)
    [v2] Sat, 6 Apr 2024 16:44:37 UTC (1,112 KB)
    Bohao Yang, Kun Zhao, Chen Tang, Liang Zhan, Chenghua Lin
    

Automatic open-domain dialogue evaluation has attracted increasing attention. Trainable evaluation metrics are commonly trained with true positive and randomly selected negative responses, resulting in a tendency for them to assign a higher score to the responses that share higher content similarity with a given context. However, adversarial negative responses possess high content similarity with the contexts whilst being semantically different. Therefore, existing evaluation metrics are not robust enough to evaluate such responses, resulting in low correlations with human judgments. While recent studies have shown some efficacy in utilizing Large Language Models (LLMs) for open-domain dialogue evaluation, they still encounter challenges in effectively handling adversarial negative examples. In this paper, we propose a simple yet effective framework for open-domain dialogue evaluation, which combines domain-specific language models (SLMs) with LLMs. The SLMs can explicitly incorporate Abstract Meaning Representation (AMR) graph information of the dialogue through a gating mechanism for enhanced semantic representation learning. The evaluation result of SLMs and AMR graph information are plugged into the prompt of LLM, for the enhanced in-context learning performance. Experimental results on open-domain dialogue evaluation tasks demonstrate the superiority of our method compared to a wide range of state-of-the-art baselines, especially in discriminating adversarial negative responses. Our code is available at this https URL.

* * *

[[2404.02491] Measuring Social Norms of Large Language Models](https://arxiv.org/abs/2404.02491)
    
    
    replaced with revised version Sun, 7 Apr 2024 05:02:39 GMT
    Submission history From: Ye Yuan [view email]
    [v1] Wed, 3 Apr 2024 05:58:57 UTC (9,299 KB)
    [v2] Sun, 7 Apr 2024 05:02:39 UTC (9,299 KB)
    Ye Yuan, Kexin Tang, Jianhao Shen, Ming Zhang, Chenguang Wang
    

We present a new challenge to examine whether large language models understand social norms. In contrast to existing datasets, our dataset requires a fundamental understanding of social norms to solve. Our dataset features the largest set of social norm skills, consisting of 402 skills and 12,383 questions covering a wide set of social norms ranging from opinions and arguments to culture and laws. We design our dataset according to the K-12 curriculum. This enables the direct comparison of the social understanding of large language models to humans, more specifically, elementary students. While prior work generates nearly random accuracy on our benchmark, recent large language models such as GPT3.5-Turbo and LLaMA2-Chat are able to improve the performance significantly, only slightly below human performance. We then propose a multi-agent framework based on large language models to improve the models' ability to understand social norms. This method further improves large language models to be on par with humans. Given the increasing adoption of large language models in real-world applications, our finding is particularly important and presents a unique direction for future improvements.

* * *

[[2404.03304] Concept -- An Evaluation Protocol on Conversation Recommender Systems with System-centric and User-centric Factors](https://arxiv.org/abs/2404.03304)
    
    
    replaced with revised version Sat, 6 Apr 2024 07:04:35 GMT
    Submission history From: Chen Huang [view email]
    [v1] Thu, 4 Apr 2024 08:56:48 UTC (12,010 KB)
    [v2] Sat, 6 Apr 2024 07:04:35 UTC (12,014 KB)
    Chen Huang, Peixin Qin, Yang Deng, Wenqiang Lei, Jiancheng Lv, Tat-Seng Chua
    

The conversational recommendation system (CRS) has been criticized regarding its user experience in real-world scenarios, despite recent significant progress achieved in academia. Existing evaluation protocols for CRS may prioritize system-centric factors such as effectiveness and fluency in conversation while neglecting user-centric aspects. Thus, we propose a new and inclusive evaluation protocol, Concept, which integrates both system- and user-centric factors. We conceptualise three key characteristics in representing such factors and further divide them into six primary abilities. To implement Concept, we adopt a LLM-based user simulator and evaluator with scoring rubrics that are tailored for each primary ability. Our protocol, Concept, serves a dual purpose. First, it provides an overview of the pros and cons in current CRS models. Second, it pinpoints the problem of low usability in the "omnipotent" ChatGPT and offers a comprehensive reference guide for evaluating CRS, thereby setting the foundation for CRS improvement.

* * *

[[2404.03471] The Impact of Unstated Norms in Bias Analysis of Language Models](https://arxiv.org/abs/2404.03471)
    
    
    replaced with revised version Sun, 7 Apr 2024 21:55:38 GMT
    Submission history From: Faiza Khattak Dr. [view email]
    [v1] Thu, 4 Apr 2024 14:24:06 UTC (884 KB)
    [v2] Sun, 7 Apr 2024 21:55:38 UTC (884 KB)
    Farnaz Kohankhaki, Jacob-Junqi Tian, David Emerson, Laleh Seyyed-Kalantari, Faiza Khan Khattak
    

Large language models (LLMs), trained on vast datasets, can carry biases that manifest in various forms, from overt discrimination to implicit stereotypes. One facet of bias is performance disparities in LLMs, often harming underprivileged groups, such as racial minorities. A common approach to quantifying bias is to use template-based bias probes, which explicitly state group membership (e.g. White) and evaluate if the outcome of a task, sentiment analysis for instance, is invariant to the change of group membership (e.g. change White race to Black). This approach is widely used in bias quantification. However, in this work, we find evidence of an unexpectedly overlooked consequence of using template-based probes for LLM bias quantification. We find that in doing so, text examples associated with White ethnicities appear to be classified as exhibiting negative sentiment at elevated rates. We hypothesize that the scenario arises artificially through a mismatch between the pre-training text of LLMs and the templates used to measure bias through reporting bias, unstated norms that imply group membership without explicit statement. Our finding highlights the potential misleading impact of varying group membership through explicit mention in bias quantification

* * *

[[2404.03887] SAAS: Solving Ability Amplification Strategy for Enhanced Mathematical Reasoning in Large Language Models](https://arxiv.org/abs/2404.03887)
    
    
    replaced with revised version Mon, 8 Apr 2024 03:05:08 GMT
    Submission history From: Chanjun Park [view email]
    [v1] Fri, 5 Apr 2024 04:25:47 UTC (735 KB)
    [v2] Mon, 8 Apr 2024 03:05:08 UTC (736 KB)
    Hyeonwoo Kim, Gyoungjin Gim, Yungi Kim, Jihoo Kim, Byungju Kim, Wonseok Lee, Chanjun Park
    

This study presents a novel learning approach designed to enhance both mathematical reasoning and problem-solving abilities of Large Language Models (LLMs). We focus on integrating the Chain-of-Thought (CoT) and the Program-of-Thought (PoT) learning, hypothesizing that prioritizing the learning of mathematical reasoning ability is helpful for the amplification of problem-solving ability. Thus, the initial learning with CoT is essential for solving challenging mathematical problems. To this end, we propose a sequential learning approach, named SAAS (Solving Ability Amplification Strategy), which strategically transitions from CoT learning to PoT learning. Our empirical study, involving an extensive performance comparison using several benchmarks, demonstrates that our SAAS achieves state-of-the-art (SOTA) performance. The results underscore the effectiveness of our sequential learning approach, marking a significant advancement in the field of mathematical reasoning in LLMs.

* * *

[[2404.04167] Chinese Tiny LLM: Pretraining a Chinese-Centric Large Language Model](https://arxiv.org/abs/2404.04167)
    
    
    replaced with revised version Mon, 8 Apr 2024 16:33:57 GMT
    Submission history From: Zheng Tianyu [view email]
    [v1] Fri, 5 Apr 2024 15:20:02 UTC (2,398 KB)
    [v2] Mon, 8 Apr 2024 16:33:57 UTC (2,385 KB)
    [v3] Tue, 9 Apr 2024 10:48:19 UTC (2,385 KB)
    Xinrun Du, Zhouliang Yu, Songyang Gao, Ding Pan, Yuyang Cheng, Ziyang Ma, Ruibin Yuan, Xingwei Qu, Jiaheng Liu, Tianyu Zheng, Xinchen Luo, Guorui Zhou, Binhang Yuan, Wenhu Chen, Jie Fu, and Ge Zhang
    

In this study, we introduce CT-LLM, a 2B large language model (LLM) that illustrates a pivotal shift towards prioritizing the Chinese language in developing LLMs. Uniquely initiated from scratch, CT-LLM diverges from the conventional methodology by primarily incorporating Chinese textual data, utilizing an extensive corpus of 1,200 billion tokens, including 800 billion Chinese tokens, 300 billion English tokens, and 100 billion code tokens. This strategic composition facilitates the model's exceptional proficiency in understanding and processing Chinese, a capability further enhanced through alignment techniques. Demonstrating remarkable performance on the CHC-Bench, CT-LLM excels in Chinese language tasks, and showcases its adeptness in English through SFT. This research challenges the prevailing paradigm of training LLMs predominantly on English corpora and then adapting them to other languages, broadening the horizons for LLM training methodologies. By open-sourcing the full process of training a Chinese LLM, including a detailed data processing procedure with the obtained Massive Appropriate Pretraining Chinese Corpus (MAP-CC), a well-chosen multidisciplinary Chinese Hard Case Benchmark (CHC-Bench), and the 2B-size Chinese Tiny LLM (CT-LLM), we aim to foster further exploration and innovation in both academia and industry, paving the way for more inclusive and versatile language models.

* * *

[[2312.15159] Understanding the Potential of FPGA-Based Spatial Acceleration for Large Language Model Inference](https://arxiv.org/abs/2312.15159)
    
    
    replaced with revised version Sun, 7 Apr 2024 06:03:02 GMT
    Submission history From: Hongzheng Chen [view email]
    [v1] Sat, 23 Dec 2023 04:27:06 UTC (944 KB)
    [v2] Sun, 7 Apr 2024 06:03:02 UTC (2,498 KB)
    Hongzheng Chen, Jiahao Zhang, Yixiao Du, Shaojie Xiang, Zichao Yue, Niansong Zhang, Yaohui Cai, Zhiru Zhang
    

Recent advancements in large language models (LLMs) boasting billions of parameters have generated a significant demand for efficient deployment in inference workloads. The majority of existing approaches rely on temporal architectures that reuse hardware units for different network layers and operators. However, these methods often encounter challenges in achieving low latency due to considerable memory access overhead. This paper investigates the feasibility and potential of model-specific spatial acceleration for LLM inference on FPGAs. Our approach involves the specialization of distinct hardware units for specific operators or layers, facilitating direct communication between them through a dataflow architecture while minimizing off-chip memory accesses. We introduce a comprehensive analytical model for estimating the performance of a spatial LLM accelerator, taking into account the on-chip compute and memory resources available on an FPGA. Through our analysis, we can determine the scenarios in which FPGA-based spatial acceleration can outperform its GPU-based counterpart. To enable more productive implementations of an LLM model on FPGAs, we further provide a library of high-level synthesis (HLS) kernels that are composable and reusable. This library will be made available as open-source. To validate the effectiveness of both our analytical model and HLS library, we have implemented BERT and GPT2 on an AMD Alveo U280 FPGA device. Experimental results demonstrate our approach can achieve up to 13.4x speedup when compared to previous FPGA-based accelerators for the BERT model. For GPT generative inference, we attain a 2.2x speedup compared to DFX, an FPGA overlay, in the prefill stage, while achieving a 1.9x speedup and a 5.7x improvement in energy efficiency compared to the NVIDIA A100 GPU in the decode stage.

* * *

[[2401.03955] Tiny Time Mixers (TTMs): Fast Pre-trained Models for Enhanced Zero/Few-Shot Forecasting of Multivariate Time Series](https://arxiv.org/abs/2401.03955)
    
    
    replaced with revised version Sat, 6 Apr 2024 17:16:18 GMT
    Submission history From: Vijay Ekambaram [view email]
    [v1] Mon, 8 Jan 2024 15:21:21 UTC (2,236 KB)
    [v2] Thu, 11 Jan 2024 18:21:27 UTC (2,241 KB)
    [v3] Wed, 17 Jan 2024 16:27:24 UTC (2,249 KB)
    [v4] Sat, 6 Apr 2024 17:16:18 UTC (2,250 KB)
    [v5] Tue, 9 Apr 2024 07:19:41 UTC (2,250 KB)
    Vijay Ekambaram, Arindam Jati, Nam H. Nguyen, Pankaj Dayama, Chandra Reddy, Wesley M. Gifford, Jayant Kalagnanam
    

Large pre-trained models for zero/few-shot learning excel in language and vision domains but encounter challenges in multivariate time series (TS) due to the diverse nature and scarcity of publicly available pre-training data. Consequently, there has been a recent surge in utilizing pre-trained large language models (LLMs) with token adaptations for TS forecasting. These approaches employ cross-domain transfer learning and surprisingly yield impressive results. However, these models are typically very slow and large (~billion parameters) and do not consider cross-channel correlations. To address this, we present Tiny Time Mixers (TTM), a significantly small model based on the lightweight TSMixer architecture. TTM marks the first success in developing fast and tiny general pre-trained models (<1M parameters), exclusively trained on public TS datasets, with effective transfer learning capabilities for forecasting. To tackle the complexity of pre-training on multiple datasets with varied temporal resolutions, we introduce several novel enhancements such as adaptive patching, dataset augmentation via downsampling, and resolution prefix tuning. Moreover, we employ a multi-level modeling strategy to effectively model channel correlations and infuse exogenous signals during fine-tuning, a crucial capability lacking in existing benchmarks. TTM shows significant accuracy gains (12-38%) over popular benchmarks in few/zero-shot forecasting. It also drastically reduces the compute needs as compared to LLM-TS methods, with a 14X cut in learnable parameters, 106X less total parameters, and substantial reductions in fine-tuning (65X) and inference time (54X). In fact, TTM's zero-shot often surpasses the few-shot results in many popular benchmarks, highlighting the efficacy of our approach. Models and source code are available at this https URL

* * *

[[2402.05964] A Survey on Transformer Compression](https://arxiv.org/abs/2402.05964)
    
    
    replaced with revised version Sun, 7 Apr 2024 13:03:58 GMT
    Submission history From: Yehui Tang [view email]
    [v1] Mon, 5 Feb 2024 12:16:28 UTC (458 KB)
    [v2] Sun, 7 Apr 2024 13:03:58 UTC (459 KB)
    Yehui Tang, Yunhe Wang, Jianyuan Guo, Zhijun Tu, Kai Han, Hailin Hu, and Dacheng Tao
    

Transformer plays a vital role in the realms of natural language processing (NLP) and computer vision (CV), specially for constructing large language models (LLM) and large vision models (LVM). Model compression methods reduce the memory and computational cost of Transformer, which is a necessary step to implement large language/vision models on practical devices. Given the unique architecture of Transformer, featuring alternative attention and feedforward neural network (FFN) modules, specific compression techniques are usually required. The efficiency of these compression methods is also paramount, as retraining large models on the entire training dataset is usually impractical. This survey provides a comprehensive review of recent compression methods, with a specific focus on their application to Transformer-based models. The compression methods are primarily categorized into pruning, quantization, knowledge distillation, and efficient architecture design (Mamba, RetNet, RWKV, etc.). In each category, we discuss compression methods for both language and vision tasks, highlighting common underlying principles. Finally, we delve into the relation between various compression methods, and discuss further directions in this domain.

* * *

[[2402.10184] Rethinking Information Structures in RLHF: Reward Generalization from a Graph Theory Perspective](https://arxiv.org/abs/2402.10184)
    
    
    replaced with revised version Mon, 8 Apr 2024 07:50:17 GMT
    Submission history From: Tianyi Qiu [view email]
    [v1] Thu, 15 Feb 2024 18:39:24 UTC (706 KB)
    [v2] Sat, 17 Feb 2024 03:26:47 UTC (705 KB)
    [v3] Tue, 20 Feb 2024 18:37:31 UTC (706 KB)
    [v4] Mon, 8 Apr 2024 07:50:17 UTC (747 KB)
    Tianyi Qiu, Fanzhi Zeng, Jiaming Ji, Dong Yan, Kaile Wang, Jiayi Zhou, Yang Han, Josef Dai, Xuehai Pan, Yaodong Yang
    

There is a trilemma in reinforcement learning from human feedback (RLHF): the incompatibility between highly diverse contexts, low labeling cost, and reliable alignment performance. We mitigate such incompatibility through the design of dataset information structures during reward modeling, and introduce the Induced Bayesian Network (IBN), the first theory of reward generalization capable of generating substantial verified predictions on large language models (LLMs). Specifically, we first reexamine the RLHF process and propose a theoretical framework portraying it as an autoencoding process over text distributions. Our framework formalizes the RLHF objective of ensuring distributional consistency between human preference and LLM behavior. Then, based on this framework, we introduce the IBN to analyze generalization in the reward modeling stage of RLHF. Drawing from random graph theory and causal analysis, it enables empirically grounded derivation of generalization error bounds, a key improvement over classical theories of generalization. Finally, an insight from our analysis is the superiority of the tree-based information structure in reward modeling, compared to chain-based baselines in conventional RLHF methods. With IBN, we derive that in complex contexts with limited data, the tree-based reward model (RM), trained on a tree-structured preference dataset, induces up to $Theta(log n/loglog n)$ times less variance than the baseline, where $n$ is the dataset size. As validation, we demonstrate that on three NLP tasks, the tree-based RM achieves 65% win rate on average against chain-based baselines. It shows that alignment performance can be gained for free via the design of dataset information structure, without the need for any other changes.

* * *

[[2402.17453] DS-Agent: Automated Data Science by Empowering Large Language Models with Case-Based Reasoning](https://arxiv.org/abs/2402.17453)
    
    
    replaced with revised version Sat, 6 Apr 2024 12:28:57 GMT
    Submission history From: Siyuan Guo [view email]
    [v1] Tue, 27 Feb 2024 12:26:07 UTC (370 KB)
    [v2] Wed, 13 Mar 2024 12:02:25 UTC (370 KB)
    [v3] Sat, 6 Apr 2024 12:28:57 UTC (370 KB)
    Siyuan Guo, Cheng Deng, Ying Wen, Hechang Chen, Yi Chang, Jun Wang
    

In this work, we investigate the potential of large language models (LLMs) based agents to automate data science tasks, with the goal of comprehending task requirements, then building and training the best-fit machine learning models. Despite their widespread success, existing LLM agents are hindered by generating unreasonable experiment plans within this scenario. To this end, we present DS-Agent, a novel automatic framework that harnesses LLM agent and case-based reasoning (CBR). In the development stage, DS-Agent follows the CBR framework to structure an automatic iteration pipeline, which can flexibly capitalize on the expert knowledge from Kaggle, and facilitate consistent performance improvement through the feedback mechanism. Moreover, DS-Agent implements a low-resource deployment stage with a simplified CBR paradigm to adapt past successful solutions from the development stage for direct code generation, significantly reducing the demand on foundational capabilities of LLMs. Empirically, DS-Agent with GPT-4 achieves an unprecedented 100% success rate in the development stage, while attaining 36% improvement on average one pass rate across alternative LLMs in the deployment stage. In both stages, DS-Agent achieves the best rank in performance, costing $1.60 and $0.13 per run with GPT-4, respectively. Our code is open-sourced at this https URL.

* * *

[[2403.09054] Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inference](https://arxiv.org/abs/2403.09054)
    
    
    replaced with revised version Sat, 6 Apr 2024 00:22:37 GMT
    Submission history From: Prashant Jayaprakash Nair [view email]
    [v1] Thu, 14 Mar 2024 02:42:42 UTC (1,504 KB)
    [v2] Sat, 6 Apr 2024 00:22:37 UTC (1,516 KB)
    Muhammad Adnan and Akhil Arunkumar and Gaurav Jain and Prashant J. Nair and Ilya Soloveychik and Purushotham Kamath
    

Transformers have emerged as the underpinning architecture for Large Language Models (LLMs). In generative language models, the inference process involves two primary phases: prompt processing and token generation. Token generation, which constitutes the majority of the computational workload, primarily entails vector-matrix multiplications and interactions with the Key-Value (KV) Cache. This phase is constrained by memory bandwidth due to the overhead of transferring weights and KV cache values from the memory system to the computing units. This memory bottleneck becomes particularly pronounced in applications that require long-context and extensive text generation, both of which are increasingly crucial for LLMs. This paper introduces "Keyformer", an innovative inference-time approach, to mitigate the challenges associated with KV cache size and memory bandwidth utilization. Keyformer leverages the observation that approximately 90% of the attention weight in generative inference focuses on a specific subset of tokens, referred to as "key" tokens. Keyformer retains only the key tokens in the KV cache by identifying these crucial tokens using a novel score function. This approach effectively reduces both the KV cache size and memory bandwidth usage without compromising model accuracy. We evaluate Keyformer's performance across three foundational models: GPT-J, Cerebras-GPT, and MPT, which employ various positional embedding algorithms. Our assessment encompasses a variety of tasks, with a particular emphasis on summarization and conversation tasks involving extended contexts. Keyformer's reduction of KV cache reduces inference latency by 2.1x and improves token generation throughput by 2.4x, while preserving the model's accuracy.

* * *

[[2403.18742] Understanding the Learning Dynamics of Alignment with Human Feedback](https://arxiv.org/abs/2403.18742)
    
    
    replaced with revised version Mon, 8 Apr 2024 15:51:17 GMT
    Submission history From: Shawn Im [view email]
    [v1] Wed, 27 Mar 2024 16:39:28 UTC (2,295 KB)
    [v2] Wed, 3 Apr 2024 15:30:03 UTC (2,293 KB)
    [v3] Mon, 8 Apr 2024 15:51:17 UTC (2,293 KB)
    Shawn Im, Yixuan Li
    

Aligning large language models (LLMs) with human intentions has become a critical task for safely deploying models in real-world systems. While existing alignment approaches have seen empirical success, theoretically understanding how these methods affect model behavior remains an open question. Our work provides an initial attempt to theoretically analyze the learning dynamics of human preference alignment. We formally show how the distribution of preference datasets influences the rate of model updates and provide rigorous guarantees on the training accuracy. Our theory also reveals an intricate phenomenon where the optimization is prone to prioritizing certain behaviors with higher preference distinguishability. We empirically validate our findings on contemporary LLMs and alignment tasks, reinforcing our theoretical insights and shedding light on considerations for future alignment approaches. Disclaimer: This paper contains potentially offensive text; reader discretion is advised.

* * *

[[2403.20208] Unleashing the Potential of Large Language Models for Predictive Tabular Tasks in Data Science](https://arxiv.org/abs/2403.20208)
    
    
    replaced with revised version Sun, 7 Apr 2024 02:21:25 GMT
    Submission history From: Yazheng Yang [view email]
    [v1] Fri, 29 Mar 2024 14:41:21 UTC (712 KB)
    [v2] Wed, 3 Apr 2024 12:23:27 UTC (712 KB)
    [v3] Thu, 4 Apr 2024 06:28:25 UTC (712 KB)
    [v4] Sun, 7 Apr 2024 02:21:25 UTC (712 KB)
    Yazheng Yang, Yuqi Wang, Sankalok Sen, Lei Li, Qi Liu
    

In the domain of data science, the predictive tasks of classification, regression, and imputation of missing values are commonly encountered challenges associated with tabular data. This research endeavors to apply Large Language Models (LLMs) towards addressing these predictive tasks. Despite their proficiency in comprehending natural language, LLMs fall short in dealing with structured tabular data. This limitation stems from their lacking exposure to the intricacies of tabular data during their foundational training. Our research aims to mitigate this gap by compiling a comprehensive corpus of tables annotated with instructions and executing large-scale training of Llama-2 on this enriched dataset. Furthermore, we investigate the practical application of applying the trained model to zero-shot prediction, few-shot prediction, and in-context learning scenarios. Through extensive experiments, our methodology has shown significant improvements over existing benchmarks. These advancements highlight the efficacy of tailoring LLM training to solve table-related problems in data science, thereby establishing a new benchmark in the utilization of LLMs for enhancing tabular intelligence.

* * *

[[2404.02937] Explainable Traffic Flow Prediction with Large Language Models](https://arxiv.org/abs/2404.02937)
    
    
    replaced with revised version Mon, 8 Apr 2024 15:52:00 GMT
    Submission history From: Xusen Guo [view email]
    [v1] Wed, 3 Apr 2024 07:14:15 UTC (5,136 KB)
    [v2] Mon, 8 Apr 2024 15:52:00 UTC (5,136 KB)
    Xusen Guo, Qiming Zhang, Mingxing Peng, Meixin Zhu, and Hao (Frank) Yang
    

Traffic flow prediction is crucial for urban planning, transportation management, and infrastructure development. However, achieving both accuracy and interpretability in prediction models remains challenging due to the complexity of traffic data and the inherent opacity of deep learning methodologies. In this paper, we propose a novel approach, Traffic Flow Prediction LLM (TF-LLM), which leverages large language models (LLMs) to generate interpretable traffic flow predictions. By transferring multi-modal traffic data into natural language descriptions, TF-LLM captures complex spatial-temporal patterns and external factors such as weather conditions, Points of Interest (PoIs), date, and holidays. We fine-tune the LLM framework using language-based instructions to align with spatial-temporal traffic flow data. Our comprehensive multi-modal traffic flow dataset (CATraffic) in California enables the evaluation of TF-LLM against state-of-the-art deep learning baselines. Results demonstrate TF-LLM's competitive accuracy while providing intuitive and interpretable predictions. We discuss the spatial-temporal and input dependencies for explainable future flow forecasting, showcasing TF-LLM's potential for diverse city prediction tasks. This paper contributes to advancing explainable traffic prediction models and lays a foundation for future exploration of LLM applications in transportation.

* * *

[[2308.12574] Modeling Uncertainty and Using Post-fusion as Fallback Improves Retrieval Augmented Generation with LLMs](https://arxiv.org/abs/2308.12574)
    
    
    replaced with revised version Mon, 8 Apr 2024 03:37:00 GMT
    Submission history From: Ye Liu [view email]
    [v1] Thu, 24 Aug 2023 05:26:54 UTC (1,026 KB)
    [v2] Mon, 8 Apr 2024 03:37:00 UTC (1,223 KB)
    Ye Liu, Semih Yavuz, Rui Meng, Meghana Moorthy, Shafiq Joty, Caiming Xiong, Yingbo Zhou
    

The integration of retrieved passages and large language models (LLMs), such as ChatGPTs, has significantly contributed to improving open-domain question answering. However, there is still a lack of exploration regarding the optimal approach for incorporating retrieved passages into the answer generation process. This paper aims to fill this gap by investigating different methods of combining retrieved passages with LLMs to enhance answer generation. We begin by examining the limitations of a commonly-used concatenation approach. Surprisingly, this approach often results in generating "unknown" outputs, even when the correct document is among the top-k retrieved passages. To address this issue, we explore four alternative strategies for integrating the retrieved passages with the LLMs. These strategies include two single-round methods that utilize chain-of-thought reasoning and two multi-round strategies that incorporate feedback loops. Through comprehensive analyses and experiments, we provide insightful observations on how to effectively leverage retrieved passages to enhance the answer generation capability of LLMs.

* * *

[[2403.12488] DetToolChain: A New Prompting Paradigm to Unleash Detection Ability of MLLM](https://arxiv.org/abs/2403.12488)
    
    
    replaced with revised version Sun, 7 Apr 2024 11:38:48 GMT
    Submission history From: Yixuan Wu [view email]
    [v1] Tue, 19 Mar 2024 06:54:33 UTC (2,306 KB)
    [v2] Sun, 7 Apr 2024 11:38:48 UTC (2,207 KB)
    Yixuan Wu, Yizhou Wang, Shixiang Tang, Wenhao Wu, Tong He, Wanli Ouyang, Jian Wu, Philip Torr
    

We present DetToolChain, a novel prompting paradigm, to unleash the zero-shot object detection ability of multimodal large language models (MLLMs), such as GPT-4V and Gemini. Our approach consists of a detection prompting toolkit inspired by high-precision detection priors and a new Chain-of-Thought to implement these prompts. Specifically, the prompts in the toolkit are designed to guide the MLLM to focus on regional information (e.g., zooming in), read coordinates according to measure standards (e.g., overlaying rulers and compasses), and infer from the contextual information (e.g., overlaying scene graphs). Building upon these tools, the new detection chain-of-thought can automatically decompose the task into simple subtasks, diagnose the predictions, and plan for progressive box refinements. The effectiveness of our framework is demonstrated across a spectrum of detection tasks, especially hard cases. Compared to existing state-of-the-art methods, GPT-4V with our DetToolChain improves state-of-the-art object detectors by +21.5% AP50 on MS COCO Novel class set for open-vocabulary detection, +24.23% Acc on RefCOCO val set for zero-shot referring expression comprehension, +14.5% AP on D-cube describe object detection FULL setting.

* * *

[[2403.13801] Natural Language as Policies: Reasoning for Coordinate-Level Embodied Control with LLMs](https://arxiv.org/abs/2403.13801)
    
    
    replaced with revised version Sat, 6 Apr 2024 04:12:47 GMT
    Submission history From: Yusuke Mikami [view email]
    [v1] Wed, 20 Mar 2024 17:58:12 UTC (585 KB)
    [v2] Sat, 6 Apr 2024 04:12:47 UTC (644 KB)
    Yusuke Mikami, Andrew Melnik, Jun Miura, Ville Hautam\"aki
    

We demonstrate experimental results with LLMs that address robotics task planning problems. Recently, LLMs have been applied in robotics task planning, particularly using a code generation approach that converts complex high-level instructions into mid-level policy codes. In contrast, our approach acquires text descriptions of the task and scene objects, then formulates task planning through natural language reasoning, and outputs coordinate level control commands, thus reducing the necessity for intermediate representation code as policies with pre-defined APIs. Our approach is evaluated on a multi-modal prompt simulation benchmark, demonstrating that our prompt engineering experiments with natural language reasoning significantly enhance success rates compared to its absence. Furthermore, our approach illustrates the potential for natural language descriptions to transfer robotics skills from known tasks to previously unseen tasks. The project website: this https URL

* * *

[[2403.16687] Investigation of the effectiveness of applying ChatGPT in Dialogic Teaching Using Electroencephalography](https://arxiv.org/abs/2403.16687)
    
    
    replaced with revised version Mon, 8 Apr 2024 09:23:43 GMT
    Submission history From: Jiayue Zhang [view email]
    [v1] Mon, 25 Mar 2024 12:23:12 UTC (454 KB)
    [v2] Mon, 8 Apr 2024 09:23:43 UTC (455 KB)
    Jiayue Zhang, Yiheng Liu, Wenqi Cai, Yali Peng, Jingjing Yu, Senqing Qi, Taotao Long, Bao Ge
    

In recent years, the rapid development of artificial intelligence technology, especially the emergence of large language models (LLMs) such as ChatGPT, has presented significant prospects for application in the field of education. LLMs possess the capability to interpret knowledge, answer questions, and consider context, thus providing support for dialogic teaching to students. Therefore, an examination of the capacity of LLMs to effectively fulfill instructional roles, thereby facilitating student learning akin to human educators within dialogic teaching scenarios, is an exceptionally valuable research topic. This research recruited 34 undergraduate students as participants, who were randomly divided into two groups. The experimental group engaged in dialogic teaching using ChatGPT, while the control group interacted with human teachers. Both groups learned the histogram equalization unit in the information-related course "Digital Image Processing". The research findings show comparable scores between the two groups on the retention test. However, students who engaged in dialogue with ChatGPT exhibited lower performance on the transfer test. Electroencephalography data revealed that students who interacted with ChatGPT exhibited higher levels of cognitive activity, suggesting that ChatGPT could help students establish a knowledge foundation and stimulate cognitive activity. However, its strengths on promoting students. knowledge application and creativity were insignificant. Based upon the research findings, it is evident that ChatGPT cannot fully excel in fulfilling teaching tasks in the dialogue teaching in information related courses. Combining ChatGPT with traditional human teachers might be a more ideal approach. The synergistic use of both can provide students with more comprehensive learning support, thus contributing to enhancing the quality of teaching.

* * *

[[2404.03204] RALL-E: Robust Codec Language Modeling with Chain-of-Thought Prompting for Text-to-Speech Synthesis](https://arxiv.org/abs/2404.03204)
    
    
    replaced with revised version Sat, 6 Apr 2024 04:35:50 GMT
    Submission history From: Detai Xin [view email]
    [v1] Thu, 4 Apr 2024 05:15:07 UTC (233 KB)
    [v2] Sat, 6 Apr 2024 04:35:50 UTC (233 KB)
    Detai Xin, Xu Tan, Kai Shen, Zeqian Ju, Dongchao Yang, Yuancheng Wang, Shinnosuke Takamichi, Hiroshi Saruwatari, Shujie Liu, Jinyu Li, Sheng Zhao
    

We present RALL-E, a robust language modeling method for text-to-speech (TTS) synthesis. While previous work based on large language models (LLMs) shows impressive performance on zero-shot TTS, such methods often suffer from poor robustness, such as unstable prosody (weird pitch and rhythm/duration) and a high word error rate (WER), due to the autoregressive prediction style of language models. The core idea behind RALL-E is chain-of-thought (CoT) prompting, which decomposes the task into simpler steps to enhance the robustness of LLM-based TTS. To accomplish this idea, RALL-E first predicts prosody features (pitch and duration) of the input text and uses them as intermediate conditions to predict speech tokens in a CoT style. Second, RALL-E utilizes the predicted duration prompt to guide the computing of self-attention weights in Transformer to enforce the model to focus on the corresponding phonemes and prosody features when predicting speech tokens. Results of comprehensive objective and subjective evaluations demonstrate that, compared to a powerful baseline method VALL-E, RALL-E significantly improves the WER of zero-shot TTS from $6.3%$ (without reranking) and $2.1%$ (with reranking) to $2.8%$ and $1.0%$, respectively. Furthermore, we demonstrate that RALL-E correctly synthesizes sentences that are hard for VALL-E and reduces the error rate from $68%$ to $4%$.

* * *

[[2404.03543] CodeEditorBench: Evaluating Code Editing Capability of Large Language Models](https://arxiv.org/abs/2404.03543)
    
    
    replaced with revised version Sat, 6 Apr 2024 04:29:25 GMT
    Submission history From: JiaWei Guo [view email]
    [v1] Thu, 4 Apr 2024 15:49:49 UTC (2,063 KB)
    [v2] Sat, 6 Apr 2024 04:29:25 UTC (2,064 KB)
    Jiawei Guo, Ziming Li, Xueling Liu, Kaijing Ma, Tianyu Zheng, Zhouliang Yu, Ding Pan, Yizhi LI, Ruibo Liu, Yue Wang, Shuyue Guo, Xingwei Qu, Xiang Yue, Ge Zhang, Wenhu Chen, Jie Fu
    

Large Language Models (LLMs) for code are rapidly evolving, with code editing emerging as a critical capability. We introduce CodeEditorBench, an evaluation framework designed to rigorously assess the performance of LLMs in code editing tasks, including debugging, translating, polishing, and requirement switching. Unlike existing benchmarks focusing solely on code generation, CodeEditorBench emphasizes real-world scenarios and practical aspects of software development. We curate diverse coding challenges and scenarios from five sources, covering various programming languages, complexity levels, and editing tasks. Evaluation of 19 LLMs reveals that closed-source models (particularly Gemini-Ultra and GPT-4), outperform open-source models in CodeEditorBench, highlighting differences in model performance based on problem types and prompt sensitivities. CodeEditorBench aims to catalyze advancements in LLMs by providing a robust platform for assessing code editing capabilities. We will release all prompts and datasets to enable the community to expand the dataset and benchmark emerging LLMs. By introducing CodeEditorBench, we contribute to the advancement of LLMs in code editing and provide a valuable resource for researchers and practitioners.

* * *

[[2310.12362] REMARK-LLM: A Robust and Efficient Watermarking Framework for Generative Large Language Models](https://arxiv.org/abs/2310.12362)
    
    
    replaced with revised version Mon, 8 Apr 2024 00:16:46 GMT
    Submission history From: Ruisi Zhang [view email]
    [v1] Wed, 18 Oct 2023 22:14:37 UTC (882 KB)
    [v2] Mon, 8 Apr 2024 00:16:46 UTC (904 KB)
    Ruisi Zhang, Shehzeen Samarah Hussain, Paarth Neekhara, Farinaz Koushanfar
    

We present REMARK-LLM, a novel efficient, and robust watermarking framework designed for texts generated by large language models (LLMs). Synthesizing human-like content using LLMs necessitates vast computational resources and extensive datasets, encapsulating critical intellectual property (IP). However, the generated content is prone to malicious exploitation, including spamming and plagiarism. To address the challenges, REMARK-LLM proposes three new components: (i) a learning-based message encoding module to infuse binary signatures into LLM-generated texts; (ii) a reparameterization module to transform the dense distributions from the message encoding to the sparse distribution of the watermarked textual tokens; (iii) a decoding module dedicated for signature extraction; Furthermore, we introduce an optimized beam search algorithm to guarantee the coherence and consistency of the generated content. REMARK-LLM is rigorously trained to encourage the preservation of semantic integrity in watermarked content, while ensuring effective watermark retrieval. Extensive evaluations on multiple unseen datasets highlight REMARK-LLM proficiency and transferability in inserting 2 times more signature bits into the same texts when compared to prior art, all while maintaining semantic integrity. Furthermore, REMARK-LLM exhibits better resilience against a spectrum of watermark detection and removal attacks.

* * *

[[2310.13289] SALMONN: Towards Generic Hearing Abilities for Large Language Models](https://arxiv.org/abs/2310.13289)
    
    
    replaced with revised version Mon, 8 Apr 2024 06:12:52 GMT
    Submission history From: Changli Tang [view email]
    [v1] Fri, 20 Oct 2023 05:41:57 UTC (1,372 KB)
    [v2] Mon, 8 Apr 2024 06:12:52 UTC (1,402 KB)
    Changli Tang and Wenyi Yu and Guangzhi Sun and Xianzhao Chen and Tian Tan and Wei Li and Lu Lu and Zejun Ma and Chao Zhang
    

Hearing is arguably an essential ability of artificial intelligence (AI) agents in the physical world, which refers to the perception and understanding of general auditory information consisting of at least three types of sounds: speech, audio events, and music. In this paper, we propose SALMONN, a speech audio language music open neural network, built by integrating a pre-trained text-based large language model (LLM) with speech and audio encoders into a single multimodal model. SALMONN enables the LLM to directly process and understand general audio inputs and achieve competitive performances on a number of speech and audio tasks used in training, such as automatic speech recognition and translation, auditory-information-based question answering, emotion recognition, speaker verification, and music and audio captioning etc. SALMONN also has a diverse set of emergent abilities unseen in the training, which includes but is not limited to speech translation to untrained languages, speech-based slot filling, spoken-query-based question answering, audio-based storytelling, and speech audio co-reasoning etc. The presence of cross-modal emergent abilities is studied, and a novel few-shot activation tuning approach is proposed to activate such abilities. To our knowledge, SALMONN is the first model of its type and can be regarded as a step towards AI with generic hearing abilities. The source code, model checkpoints and data are available at this https URL.

* * *

[[2404.00938] How Can Large Language Models Enable Better Socially Assistive Human-Robot Interaction: A Brief Survey](https://arxiv.org/abs/2404.00938)
    
    
    replaced with revised version Fri, 5 Apr 2024 21:55:47 GMT
    Submission history From: Ellen Landrum [view email]
    [v1] Mon, 1 Apr 2024 05:50:56 UTC (16 KB)
    [v2] Fri, 5 Apr 2024 21:55:47 UTC (16 KB)
    Zhonghao Shi, Ellen Landrum, Amy O' Connell, Mina Kian, Leticia Pinto-Alva, Kaleen Shrestha, Xiaoyuan Zhu, Maja J Matari\'c
    

Socially assistive robots (SARs) have shown great success in providing personalized cognitive-affective support for user populations with special needs such as older adults, children with autism spectrum disorder (ASD), and individuals with mental health challenges. The large body of work on SAR demonstrates its potential to provide at-home support that complements clinic-based interventions delivered by mental health professionals, making these interventions more effective and accessible. However, there are still several major technical challenges that hinder SAR-mediated interactions and interventions from reaching human-level social intelligence and efficacy. With the recent advances in large language models (LLMs), there is an increased potential for novel applications within the field of SAR that can significantly expand the current capabilities of SARs. However, incorporating LLMs introduces new risks and ethical concerns that have not yet been encountered, and must be carefully be addressed to safely deploy these more advanced systems. In this work, we aim to conduct a brief survey on the use of LLMs in SAR technologies, and discuss the potentials and risks of applying LLMs to the following three major technical challenges of SAR: 1) natural language dialog; 2) multimodal understanding; 3) LLMs as robot policies.

* * *

[[2404.01800] Sentiment Analysis of Citations in Scientific Articles Using ChatGPT: Identifying Potential Biases and Conflicts of Interest](https://arxiv.org/abs/2404.01800)
    
    
    replaced with revised version Sat, 6 Apr 2024 19:08:08 GMT
    Submission history From: Walid Hariri [view email]
    [v1] Tue, 2 Apr 2024 09:59:49 UTC (199 KB)
    [v2] Sat, 6 Apr 2024 19:08:08 UTC (200 KB)
    Walid Hariri
    

Scientific articles play a crucial role in advancing knowledge and informing research directions. One key aspect of evaluating scientific articles is the analysis of citations, which provides insights into the impact and reception of the cited works. This article introduces the innovative use of large language models, particularly ChatGPT, for comprehensive sentiment analysis of citations within scientific articles. By leveraging advanced natural language processing (NLP) techniques, ChatGPT can discern the nuanced positivity or negativity of citations, offering insights into the reception and impact of cited works. Furthermore, ChatGPT's capabilities extend to detecting potential biases and conflicts of interest in citations, enhancing the objectivity and reliability of scientific literature evaluation. This study showcases the transformative potential of artificial intelligence (AI)-powered tools in enhancing citation analysis and promoting integrity in scholarly research.

* * *

[[2302.07517] Versatile User Identification in Extended Reality using Pretrained Similarity-Learning](https://arxiv.org/abs/2302.07517)
    
    
    replaced with revised version Sun, 7 Apr 2024 09:34:03 GMT
    Submission history From: Christian Rack [view email]
    [v1] Wed, 15 Feb 2023 08:26:24 UTC (1,515 KB)
    [v2] Mon, 3 Apr 2023 09:46:40 UTC (1,558 KB)
    [v3] Wed, 28 Jun 2023 08:50:58 UTC (1,591 KB)
    [v4] Mon, 3 Jul 2023 13:33:21 UTC (1,591 KB)
    [v5] Sun, 7 Apr 2024 09:34:03 UTC (8,305 KB)
    Christian Rack, Konstantin Kobs, Tamara Fernando, Andreas Hotho, Marc Erich Latoschik
    

Different approaches in machine learning have proven useful for user verification and identification based on motion data in eXtended Reality (XR). However, their real-world application still faces significant challenges concerning versatility, i.e., in terms of extensibility and generalization capability. This article presents a solution that is both extensible to new users without expensive retraining, and that generalizes well across different sessions, devices, and user tasks. To this end, we developed a similarity-learning model and pretrained it on the "Who Is Alyx?" dataset. This dataset features a wide array of actions from users playing the VR game "Half-Life: Alyx". In contrast to previous works, we used a dedicated set of users for model validation and final evaluation. Furthermore, we extended this evaluation using an independent dataset that features completely different users, tasks, and three different XR devices. In comparison with a traditional classification-learning baseline, our model shows superior performance, especially in scenarios with limited enrollment data. The pretraining process allows easy deployment in a diverse range of XR applications while maintaining high versatility. Looking ahead, our methodology can facilitate the easy integration of pretrained motion-based identification models into 3D engines: dedicated plugins for Unreal and Unity promise straightforward and quick adoption of these advanced models in the future, even for those lacking expertise in motion analysis and machine learning.

* * *

[[2306.09610] CHORUS: Foundation Models for Unified Data Discovery and Exploration](https://arxiv.org/abs/2306.09610)
    
    
    replaced with revised version Fri, 5 Apr 2024 20:26:02 GMT
    Submission history From: Moe Kayali [view email]
    [v1] Fri, 16 Jun 2023 03:58:42 UTC (228 KB)
    [v2] Wed, 27 Sep 2023 02:02:56 UTC (348 KB)
    [v3] Fri, 5 Apr 2024 20:26:02 UTC (440 KB)
    Moe Kayali, Anton Lykov, Ilias Fountalis, Nikolaos Vasiloglou, Dan Olteanu, Dan Suciu
    

We apply foundation models to data discovery and exploration tasks. Foundation models include large language models (LLMs) that show promising performance on a range of diverse tasks unrelated to their training. We show that these models are highly applicable to the data discovery and data exploration domain. When carefully used, they have superior capability on three representative tasks: table-class detection, column-type annotation and join-column prediction. On all three tasks, we show that a foundation-model-based approach outperforms the task-specific models and so the state of the art. Further, our approach often surpasses human-expert task performance. We investigate the fundamental characteristics of this approach including generalizability to several foundation models and the impact of non-determinism on the outputs. All in all, this suggests a future direction in which disparate data management tasks can be unified under foundation models.

## Footer

[ ](https://github.com "GitHub") (C) 2025 GitHub, Inc. 

### Footer navigation

  * [Terms](https://docs.github.com/site-policy/github-terms/github-terms-of-service)
  * [Privacy](https://docs.github.com/site-policy/privacy-policies/github-privacy-statement)
  * [Security](https://github.com/security)
  * [Status](https://www.githubstatus.com/)
  * [Docs](https://docs.github.com/)
  * [Contact](https://support.github.com?tags=dotcom-footer)
  * Manage cookies 
  * Do not share my personal information 



You can’t perform that action at this time. 


