# documents

scrape_date: 2025-03-17T13:31:34.447734
content_type: markdown
hash: 5a0b765e2a4216eb6943608106a0bc21b6c27f61d8d396f70bb6c8a4d18faf92
url:
https://github.com/wwd29/arxiv-daily/blob/1a3074665786adbe2140d325f14a577e8297e535/html/user_0/2024-09-27.html

title:
arxiv-daily/html/user_0/2024-09-27.html at 1a3074665786adbe2140d325f14a577e8297e535 · wwd29/arxiv-daily · GitHub

content:
Skip to content

## Navigation Menu

Toggle navigation

[ ](/)

[ Sign in ](/login?return_to=https%3A%2F%2Fgithub.com%2Fwwd29%2Farxiv-daily%2Fblob%2F1a3074665786adbe2140d325f14a577e8297e535%2Fhtml%2Fuser_0%2F2024-09-27.html)

  * Product 

    * [ GitHub Copilot Write better code with AI  ](https://github.com/features/copilot)
    * [ Security Find and fix vulnerabilities  ](https://github.com/features/security)
    * [ Actions Automate any workflow  ](https://github.com/features/actions)
    * [ Codespaces Instant dev environments  ](https://github.com/features/codespaces)
    * [ Issues Plan and track work  ](https://github.com/features/issues)
    * [ Code Review Manage code changes  ](https://github.com/features/code-review)
    * [ Discussions Collaborate outside of code  ](https://github.com/features/discussions)
    * [ Code Search Find more, search less  ](https://github.com/features/code-search)

Explore
    * [ All features ](https://github.com/features)
    * [ Documentation  ](https://docs.github.com)
    * [ GitHub Skills  ](https://skills.github.com)
    * [ Blog  ](https://github.blog)

  * Solutions 

By company size
    * [ Enterprises ](https://github.com/enterprise)
    * [ Small and medium teams ](https://github.com/team)
    * [ Startups ](https://github.com/enterprise/startups)
    * [ Nonprofits ](/solutions/industry/nonprofits)

By use case
    * [ DevSecOps ](/solutions/use-case/devsecops)
    * [ DevOps ](/solutions/use-case/devops)
    * [ CI/CD ](/solutions/use-case/ci-cd)
    * [ View all use cases ](/solutions/use-case)

By industry
    * [ Healthcare ](/solutions/industry/healthcare)
    * [ Financial services ](/solutions/industry/financial-services)
    * [ Manufacturing ](/solutions/industry/manufacturing)
    * [ Government ](/solutions/industry/government)
    * [ View all industries ](/solutions/industry)

[ View all solutions  ](/solutions)

  * Resources 

Topics
    * [ AI ](/resources/articles/ai)
    * [ DevOps ](/resources/articles/devops)
    * [ Security ](/resources/articles/security)
    * [ Software Development ](/resources/articles/software-development)
    * [ View all ](/resources/articles)

Explore
    * [ Learning Pathways  ](https://resources.github.com/learn/pathways)
    * [ Events & Webinars  ](https://resources.github.com)
    * [ Ebooks & Whitepapers ](https://github.com/resources/whitepapers)
    * [ Customer Stories ](https://github.com/customer-stories)
    * [ Partners  ](https://partner.github.com)
    * [ Executive Insights ](https://github.com/solutions/executive-insights)

  * Open Source 

    * [ GitHub Sponsors Fund open source developers  ](/sponsors)

    * [ The ReadME Project GitHub community articles  ](https://github.com/readme)

Repositories
    * [ Topics ](https://github.com/topics)
    * [ Trending ](https://github.com/trending)
    * [ Collections ](https://github.com/collections)

  * Enterprise 

    * [ Enterprise platform AI-powered developer platform  ](/enterprise)

Available add-ons
    * [ Advanced Security Enterprise-grade security features  ](https://github.com/enterprise/advanced-security)
    * [ Copilot for business Enterprise-grade AI features  ](/features/copilot/copilot-business)
    * [ Premium Support Enterprise-grade 24/7 support  ](/premium-support)

  * [Pricing](https://github.com/pricing)



Search or jump to...

# Search code, repositories, users, issues, pull requests...

Search 

Clear




[Search syntax tips](https://docs.github.com/search-github/github-code-search/understanding-github-code-search-syntax)

#  Provide feedback 

We read every piece of feedback, and take your input very seriously.

Include my email address so I can be contacted

Cancel  Submit feedback 

#  Saved searches 

## Use saved searches to filter your results more quickly

Name

Query

To see all available qualifiers, see our [documentation](https://docs.github.com/search-github/github-code-search/understanding-github-code-search-syntax). 

Cancel  Create saved search 

[ Sign in ](/login?return_to=https%3A%2F%2Fgithub.com%2Fwwd29%2Farxiv-daily%2Fblob%2F1a3074665786adbe2140d325f14a577e8297e535%2Fhtml%2Fuser_0%2F2024-09-27.html)

[ Sign up ](/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fblob%2Fshow&source=header-repo&source_repo=wwd29%2Farxiv-daily) Reseting focus

You signed in with another tab or window. [Reload]() to refresh your session. You signed out in another tab or window. [Reload]() to refresh your session. You switched accounts on another tab or window. [Reload]() to refresh your session. Dismiss alert

{{ message }}

[ wwd29 ](/wwd29) / **[arxiv-daily](/wwd29/arxiv-daily) ** Public

  * [ Notifications ](/login?return_to=%2Fwwd29%2Farxiv-daily) You must be signed in to change notification settings
  * [ Fork 4 ](/login?return_to=%2Fwwd29%2Farxiv-daily)
  * [ Star  15 ](/login?return_to=%2Fwwd29%2Farxiv-daily)




  * [ Code ](/wwd29/arxiv-daily)
  * [ Issues 0 ](/wwd29/arxiv-daily/issues)
  * [ Pull requests 0 ](/wwd29/arxiv-daily/pulls)
  * [ Actions ](/wwd29/arxiv-daily/actions)
  * [ Projects 0 ](/wwd29/arxiv-daily/projects)
  * [ Security ](/wwd29/arxiv-daily/security)
  * [ Insights ](/wwd29/arxiv-daily/pulse)



Additional navigation options

  * [ Code  ](/wwd29/arxiv-daily)
  * [ Issues  ](/wwd29/arxiv-daily/issues)
  * [ Pull requests  ](/wwd29/arxiv-daily/pulls)
  * [ Actions  ](/wwd29/arxiv-daily/actions)
  * [ Projects  ](/wwd29/arxiv-daily/projects)
  * [ Security  ](/wwd29/arxiv-daily/security)
  * [ Insights  ](/wwd29/arxiv-daily/pulse)



## Files

1a30746

## Breadcrumbs

  1. [arxiv-daily](/wwd29/arxiv-daily/tree/1a3074665786adbe2140d325f14a577e8297e535)
  2. /[html](/wwd29/arxiv-daily/tree/1a3074665786adbe2140d325f14a577e8297e535/html)
  3. /[user_0](/wwd29/arxiv-daily/tree/1a3074665786adbe2140d325f14a577e8297e535/html/user_0)



/

# 2024-09-27.html

Copy path

Blame

Blame

## Latest commit

## History

[History](/wwd29/arxiv-daily/commits/1a3074665786adbe2140d325f14a577e8297e535/html/user_0/2024-09-27.html)

[](/wwd29/arxiv-daily/commits/1a3074665786adbe2140d325f14a577e8297e535/html/user_0/2024-09-27.html)

1778 lines (1617 loc) · 336 KB

1a30746

## Breadcrumbs

  1. [arxiv-daily](/wwd29/arxiv-daily/tree/1a3074665786adbe2140d325f14a577e8297e535)
  2. /[html](/wwd29/arxiv-daily/tree/1a3074665786adbe2140d325f14a577e8297e535/html)
  3. /[user_0](/wwd29/arxiv-daily/tree/1a3074665786adbe2140d325f14a577e8297e535/html/user_0)



/

# 2024-09-27.html

Top

## File metadata and controls

  * Code

  * Blame




1778 lines (1617 loc) · 336 KB

[Raw](https://github.com/wwd29/arxiv-daily/raw/1a3074665786adbe2140d325f14a577e8297e535/html/user_0/2024-09-27.html)

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465

466

467

468

469

470

471

472

473

474

475

476

477

478

479

480

481

482

483

484

485

486

487

488

489

490

491

492

493

494

495

496

497

498

499

500

501

502

503

504

505

506

507

508

509

510

511

512

513

514

515

516

517

518

519

520

521

522

523

524

525

526

527

528

529

530

531

532

533

534

535

536

537

538

539

540

541

542

543

544

545

546

547

548

549

550

551

552

553

554

555

556

557

558

559

560

561

562

563

564

565

566

567

568

569

570

571

572

573

574

575

576

577

578

579

580

581

582

583

584

585

586

587

588

589

590

591

592

593

594

595

596

597

598

599

600

601

602

603

604

605

606

607

608

609

610

611

612

613

614

615

616

617

618

619

620

621

622

623

624

625

626

627

628

629

630

631

632

633

634

635

636

637

638

639

640

641

642

643

644

645

646

647

648

649

650

651

652

653

654

655

656

657

658

659

660

661

662

663

664

665

666

667

668

669

670

671

672

673

674

675

676

677

678

679

680

681

682

683

684

685

686

687

688

689

690

691

692

693

694

695

696

697

698

699

700

701

702

703

704

705

706

707

708

709

710

711

712

713

714

715

716

717

718

719

720

721

722

723

724

725

726

727

728

729

730

731

732

733

734

735

736

737

738

739

740

741

742

743

744

745

746

747

748

749

750

751

752

753

754

755

756

757

758

759

760

761

762

763

764

765

766

767

768

769

770

771

772

773

774

775

776

777

778

779

780

781

782

783

784

785

786

787

788

789

790

791

792

793

794

795

796

797

798

799

800

801

802

803

804

805

806

807

808

809

810

811

812

813

814

815

816

817

818

819

820

821

822

823

824

825

826

827

828

829

830

831

832

833

834

835

836

837

838

839

840

841

842

843

844

845

846

847

848

849

850

851

852

853

854

855

856

857

858

859

860

861

862

863

864

865

866

867

868

869

870

871

872

873

874

875

876

877

878

879

880

881

882

883

884

885

886

887

888

889

890

891

892

893

894

895

896

897

898

899

900

901

902

903

904

905

906

907

908

909

910

911

912

913

914

915

916

917

918

919

920

921

922

923

924

925

926

927

928

929

930

931

932

933

934

935

936

937

938

939

940

941

942

943

944

945

946

947

948

949

950

951

952

953

954

955

956

957

958

959

960

961

962

963

964

965

966

967

968

969

970

971

972

973

974

975

976

977

978

979

980

981

982

983

984

985

986

987

988

989

990

991

992

993

994

995

996

997

998

999

1000

<link rel="stylesheet" href="../../css/markdown.css" />

<article class="markdown-body">

<h1>2024-09-27</h1>

<h3>Title: An Art-centric perspective on AI-based content moderation of nudity</h3>

<ul>

<li><strong>Authors: </strong>Piera Riccio, Georgina Curto, Thomas Hofmann, Nuria Oliver</a></li>

<li><strong>Subjects: </strong>cs.CV, cs.SI</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17156">https://arxiv.org/abs/2409.17156</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17156">https://arxiv.org/pdf/2409.17156</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17156]] An Art-centric perspective on AI-based content moderation of nudity(https://arxiv.org/abs/2409.17156)</code><input type="text"></li>

<li><strong>Keywords: </strong>generative</a></li>

<li><strong>Abstract: </strong>At a time when the influence of generative Artificial Intelligence on visual arts is a highly debated topic, we raise the attention towards a more subtle phenomenon: the algorithmic censorship of artistic nudity online. We analyze the performance of three "Not-Safe-For-Work'' image classifiers on artistic nudity, and empirically uncover the existence of a gender and a stylistic bias, as well as evident technical limitations, especially when only considering visual information. Hence, we propose a multi-modal zero-shot classification approach that improves artistic nudity classification. From our research, we draw several implications that we hope will inform future research on this topic.</li>

</ul>

<h3>Title: REAL: Response Embedding-based Alignment for LLMs</h3>

<ul>

<li><strong>Authors: </strong>Honggen Zhang, Igor Molybog, June Zhang, Xufeng Zhao</a></li>

<li><strong>Subjects: </strong>cs.CL, cs.AI</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17169">https://arxiv.org/abs/2409.17169</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17169">https://arxiv.org/pdf/2409.17169</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17169]] REAL: Response Embedding-based Alignment for LLMs(https://arxiv.org/abs/2409.17169)</code><input type="text"></li>

<li><strong>Keywords: </strong>large language model</a></li>

<li><strong>Abstract: </strong>Aligning large language models (LLMs) to human preferences is a crucial step in building helpful and safe AI tools, which usually involve training on supervised datasets. Popular algorithms such as Direct Preference Optimization rely on pairs of AI-generated responses ranked according to human feedback. The labeling process is the most labor-intensive and costly part of the alignment pipeline, and improving its efficiency would have a meaningful impact on AI development. We propose a strategy for sampling a high-quality training dataset that focuses on acquiring the most informative response pairs for labeling out of a set of AI-generated responses. Experimental results on synthetic HH-RLHF benchmarks indicate that choosing dissimilar response pairs enhances the direct alignment of LLMs while reducing inherited labeling errors. We also applied our method to the real-world dataset SHP2, selecting optimal pairs from multiple responses. The model aligned on dissimilar response pairs obtained the best win rate on the dialogue task. Our findings suggest that focusing on less similar pairs can improve the efficiency of LLM alignment, saving up to 65% of annotators' work.</li>

</ul>

<h3>Title: What Would You Ask When You First Saw $a^2+b^2=c^2$? Evaluating LLM on Curiosity-Driven Questioning</h3>

<ul>

<li><strong>Authors: </strong>Shashidhar Reddy Javaji, Zining Zhu</a></li>

<li><strong>Subjects: </strong>cs.CL, cs.AI, cs.LG</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17172">https://arxiv.org/abs/2409.17172</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17172">https://arxiv.org/pdf/2409.17172</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17172]] What Would You Ask When You First Saw $a^2+b^2=c^2$? Evaluating LLM on Curiosity-Driven Questioning(https://arxiv.org/abs/2409.17172)</code><input type="text"></li>

<li><strong>Keywords: </strong>large language model</a></li>

<li><strong>Abstract: </strong>Large language models (LLMs) can store a massive amount of knowledge, yet their potential to acquire new knowledge remains unknown. We propose a novel evaluation framework that evaluates this capability. This framework prompts LLMs to generate questions about a statement introducing scientific knowledge, simulating a curious person when facing the statement for the first time. We score the qualities of the generated questions, thereby evaluating the knowledge acquisition potential of the LLM. We apply controlled ablation studies to validate our scoring procedures. Additionally, we created a synthetic dataset consisting of 1101 statements in physics, chemistry, and maths with distinct levels of difficulties, 300 general knowledge statements, and 567 incorrect statements. Human evaluations were conducted to validate our model assessments, achieving an approximate weighted Cohen's kappa of 0.7 on all three metrics considered. We find that while large models like GPT-4 and Mistral 8x7b are adept at generating coherent and relevant questions, the smaller Phi-2 model is equally or more effective. This indicates that size does not solely determine a model's knowledge acquisition potential. The proposed framework quantifies a critical model capability that was commonly overlooked and opens up research opportunities for developing more knowledgeable AI systems</li>

</ul>

<h3>Title: A Multiple-Fill-in-the-Blank Exam Approach for Enhancing Zero-Resource Hallucination Detection in Large Language Models</h3>

<ul>

<li><strong>Authors: </strong>Satoshi Munakata, Taku Fukui, Takao Mohri</a></li>

<li><strong>Subjects: </strong>cs.CL, cs.AI</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17173">https://arxiv.org/abs/2409.17173</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17173">https://arxiv.org/pdf/2409.17173</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17173]] A Multiple-Fill-in-the-Blank Exam Approach for Enhancing Zero-Resource Hallucination Detection in Large Language Models(https://arxiv.org/abs/2409.17173)</code><input type="text"></li>

<li><strong>Keywords: </strong>large language model</a></li>

<li><strong>Abstract: </strong>Large language models (LLMs) often fabricate a hallucinatory text. Several methods have been developed to detect such text by semantically comparing it with the multiple versions probabilistically regenerated. However, a significant issue is that if the storyline of each regenerated text changes, the generated texts become incomparable, which worsen detection accuracy. In this paper, we propose a hallucination detection method that incorporates a multiple-fill-in-the-blank exam approach to address this storyline-changing issue. First, our method creates a multiple-fill-in-the-blank exam by masking multiple objects from the original text. Second, prompts an LLM to repeatedly answer this exam. This approach ensures that the storylines of the exam answers align with the original ones. Finally, quantifies the degree of hallucination for each original sentence by scoring the exam answers, considering the potential for \emph{hallucination snowballing} within the original text itself. Experimental results show that our method alone not only outperforms existing methods, but also achieves clearer state-of-the-art performance in the ensembles with existing methods.</li>

</ul>

<h3>Title: CSCE: Boosting LLM Reasoning by Simultaneous Enhancing of Casual Significance and Consistency</h3>

<ul>

<li><strong>Authors: </strong>Kangsheng Wang, Xiao Zhang, Zizheng Guo, Tianyu Hu, Huimin Ma</a></li>

<li><strong>Subjects: </strong>cs.CL, cs.AI</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17174">https://arxiv.org/abs/2409.17174</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17174">https://arxiv.org/pdf/2409.17174</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17174]] CSCE: Boosting LLM Reasoning by Simultaneous Enhancing of Casual Significance and Consistency(https://arxiv.org/abs/2409.17174)</code><input type="text"></li>

<li><strong>Keywords: </strong>robust, large language model</a></li>

<li><strong>Abstract: </strong>Chain-based reasoning methods like chain of thought (CoT) play a rising role in solving reasoning tasks for large language models (LLMs). However, the causal illusions between \textit{a step of reasoning} and \textit{corresponding state transitions} are becoming a significant obstacle to advancing LLMs' reasoning capabilities, especially in long-range reasoning tasks. This paper proposes a non-chain-based reasoning framework for simultaneous consideration of causal significance and consistency, i.e., the Causal Significance and Consistency Enhancer (CSCE). We customize LLM's loss function utilizing treatment effect assessments to enhance its reasoning ability from two aspects: causal significance and consistency. This ensures that the model captures essential causal relationships and maintains robust and consistent performance across various scenarios. Additionally, we transform the reasoning process from the cascading multiple one-step reasoning commonly used in Chain-Based methods, like CoT, to a causal-enhanced method that outputs the entire reasoning process in one go, further improving the model's reasoning efficiency. Extensive experiments show that our method improves both the reasoning success rate and speed. These improvements further demonstrate that non-chain-based methods can also aid LLMs in completing reasoning tasks.</li>

</ul>

<h3>Title: XDC Gasless Subnet: Gasless Subnet Staking dApp for XDC Network</h3>

<ul>

<li><strong>Authors: </strong>Mohuya Chakraborty, Atul Khekade</a></li>

<li><strong>Subjects: </strong>cs.CR</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17176">https://arxiv.org/abs/2409.17176</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17176">https://arxiv.org/pdf/2409.17176</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17176]] XDC Gasless Subnet: Gasless Subnet Staking dApp for XDC Network(https://arxiv.org/abs/2409.17176)</code><input type="text"></li>

<li><strong>Keywords: </strong>security</a></li>

<li><strong>Abstract: </strong>With a delegated proof-of-stake (XDPoS) consensus mechanism, the XDC Network is an enterprise-focused blockchain platform that combines the strength of public and private blockchains to provide quick transaction times, low energy consumption, and economical gas fees. XDC is designed for interoperability and supports decentralized apps (dApps) and integrates smoothly with financial systems. It is perfect for trade financing and tokenisation of physical assets because of its emphasis on security and scalability. However, there are a few critical issues that hamper wider acceptance and usability for certain high-frequency applications. This whitepaper introduces a novel and enthralling dApp for establishing a gasless subnet in which mainnet XDC can be staked to spin off a subnet that functions similarly to a non-crypto network, accepting currency fees on the XDC network. This would allow users to stake their tokens without incurring gas fees making the staking process more efficient, cost-effective, and simultaneously enhancing scalability. Performance evaluation of the dApp shows promising results in terms of throughput, latency, scalability, security, and cost efficiency. The use cases and applications of this approach along with challenges and ensuing solutions are included.</li>

</ul>

<h3>Title: Fully automatic extraction of morphological traits from the Web: utopia or reality?</h3>

<ul>

<li><strong>Authors: </strong>Diego Marcos, Robert van de Vlasakker, Ioannis N. Athanasiadis, Pierre Bonnet, Hervé Goeau, Alexis Joly, W. Daniel Kissling, César Leblanc, André S.J. van Proosdij, Konstantinos P. Panousis</a></li>

<li><strong>Subjects: </strong>cs.CL, cs.AI, cs.LG</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17179">https://arxiv.org/abs/2409.17179</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17179">https://arxiv.org/pdf/2409.17179</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17179]] Fully automatic extraction of morphological traits from the Web: utopia or reality?(https://arxiv.org/abs/2409.17179)</code><input type="text"></li>

<li><strong>Keywords: </strong>extraction, large language model</a></li>

<li><strong>Abstract: </strong>Plant morphological traits, their observable characteristics, are fundamental to understand the role played by each species within their ecosystem. However, compiling trait information for even a moderate number of species is a demanding task that may take experts years to accomplish. At the same time, massive amounts of information about species descriptions is available online in the form of text, although the lack of structure makes this source of data impossible to use at scale. To overcome this, we propose to leverage recent advances in large language models (LLMs) and devise a mechanism for gathering and processing information on plant traits in the form of unstructured textual descriptions, without manual curation. We evaluate our approach by automatically replicating three manually created species-trait matrices. Our method managed to find values for over half of all species-trait pairs, with an F1-score of over 75%. Our results suggest that large-scale creation of structured trait databases from unstructured online text is currently feasible thanks to the information extraction capabilities of LLMs, being limited by the availability of textual descriptions covering all the traits of interest.</li>

</ul>

<h3>Title: A Mobile Payment Scheme Using Biometric Identification with Mutual Authentication</h3>

<ul>

<li><strong>Authors: </strong>Jack Sturgess, Ivan Martinovic</a></li>

<li><strong>Subjects: </strong>cs.CR</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17181">https://arxiv.org/abs/2409.17181</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17181">https://arxiv.org/pdf/2409.17181</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17181]] A Mobile Payment Scheme Using Biometric Identification with Mutual Authentication(https://arxiv.org/abs/2409.17181)</code><input type="text"></li>

<li><strong>Keywords: </strong>protect, attack, biometric</a></li>

<li><strong>Abstract: </strong>Cashless payment systems offer many benefits over cash, but also have some drawbacks. Fake terminals, skimming, wireless connectivity, and relay attacks are persistent problems. Attempts to overcome one problem often lead to another - for example, some systems use QR codes to avoid skimming and connexion issues, but QR codes can be stolen at distance and relayed. In this paper, we propose a novel mobile payment scheme based on biometric identification that provides mutual authentication to protect the user from rogue terminals. Our scheme imposes only minimal requirements on terminal hardware, does not depend on wireless connectivity between the user and the verifier during the authentication phase, and does not require the user to trust the terminal until it has authenticated itself to the user. We show that our scheme is resistant against phishing, replay, relay, and presentation attacks.</li>

</ul>

<h3>Title: Enhancing Guardrails for Safe and Secure Healthcare AI</h3>

<ul>

<li><strong>Authors: </strong>Ananya Gangavarapu</a></li>

<li><strong>Subjects: </strong>cs.CR, cs.AI</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17190">https://arxiv.org/abs/2409.17190</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17190">https://arxiv.org/pdf/2409.17190</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17190]] Enhancing Guardrails for Safe and Secure Healthcare AI(https://arxiv.org/abs/2409.17190)</code><input type="text"></li>

<li><strong>Keywords: </strong>secure, security, robust, generative</a></li>

<li><strong>Abstract: </strong>Generative AI holds immense promise in addressing global healthcare access challenges, with numerous innovative applications now ready for use across various healthcare domains. However, a significant barrier to the widespread adoption of these domain-specific AI solutions is the lack of robust safety mechanisms to effectively manage issues such as hallucination, misinformation, and ensuring truthfulness. Left unchecked, these risks can compromise patient safety and erode trust in healthcare AI systems. While general-purpose frameworks like Llama Guard are useful for filtering toxicity and harmful content, they do not fully address the stringent requirements for truthfulness and safety in healthcare contexts. This paper examines the unique safety and security challenges inherent to healthcare AI, particularly the risk of hallucinations, the spread of misinformation, and the need for factual accuracy in clinical settings. I propose enhancements to existing guardrails frameworks, such as Nvidia NeMo Guardrails, to better suit healthcare-specific needs. By strengthening these safeguards, I aim to ensure the secure, reliable, and accurate use of AI in healthcare, mitigating misinformation risks and improving patient safety.</li>

</ul>

<h3>Title: An Effective, Robust and Fairness-aware Hate Speech Detection Framework</h3>

<ul>

<li><strong>Authors: </strong>Guanyi Mou, Kyumin Lee</a></li>

<li><strong>Subjects: </strong>cs.CL, cs.LG</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17191">https://arxiv.org/abs/2409.17191</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17191">https://arxiv.org/pdf/2409.17191</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17191]] An Effective, Robust and Fairness-aware Hate Speech Detection Framework(https://arxiv.org/abs/2409.17191)</code><input type="text"></li>

<li><strong>Keywords: </strong>attack, robust, fair</a></li>

<li><strong>Abstract: </strong>With the widespread online social networks, hate speeches are spreading faster and causing more damage than ever before. Existing hate speech detection methods have limitations in several aspects, such as handling data insufficiency, estimating model uncertainty, improving robustness against malicious attacks, and handling unintended bias (i.e., fairness). There is an urgent need for accurate, robust, and fair hate speech classification in online social networks. To bridge the gap, we design a data-augmented, fairness addressed, and uncertainty estimated novel framework. As parts of the framework, we propose Bidirectional Quaternion-Quasi-LSTM layers to balance effectiveness and efficiency. To build a generalized model, we combine five datasets collected from three platforms. Experiment results show that our model outperforms eight state-of-the-art methods under both no attack scenario and various attack scenarios, indicating the effectiveness and robustness of our model. We share our code along with combined dataset for better future research</li>

</ul>

<h3>Title: A random measure approach to reinforcement learning in continuous time</h3>

<ul>

<li><strong>Authors: </strong>Christian Bender, Nguyen Tran Thuan</a></li>

<li><strong>Subjects: </strong>cs.LG, math.PR, stat.ML</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17200">https://arxiv.org/abs/2409.17200</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17200">https://arxiv.org/pdf/2409.17200</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17200]] A random measure approach to reinforcement learning in continuous time(https://arxiv.org/abs/2409.17200)</code><input type="text"></li>

<li><strong>Keywords: </strong>diffusion</a></li>

<li><strong>Abstract: </strong>We present a random measure approach for modeling exploration, i.e., the execution of measure-valued controls, in continuous-time reinforcement learning (RL) with controlled diffusion and jumps. First, we consider the case when sampling the randomized control in continuous time takes place on a discrete-time grid and reformulate the resulting stochastic differential equation (SDE) as an equation driven by suitable random measures. The construction of these random measures makes use of the Brownian motion and the Poisson random measure (which are the sources of noise in the original model dynamics) as well as the additional random variables, which are sampled on the grid for the control execution. Then, we prove a limit theorem for these random measures as the mesh-size of the sampling grid goes to zero, which leads to the grid-sampling limit SDE that is jointly driven by white noise random measures and a Poisson random measure. We also argue that the grid-sampling limit SDE can substitute the exploratory SDE and the sample SDE of the recent continuous-time RL literature, i.e., it can be applied for the theoretical analysis of exploratory control problems and for the derivation of learning algorithms.</li>

</ul>

<h3>Title: Immersion and Invariance-based Coding for Privacy-Preserving Federated Learning</h3>

<ul>

<li><strong>Authors: </strong>Haleh Hayati, Carlos Murguia, Nathan van de Wouw</a></li>

<li><strong>Subjects: </strong>cs.CR, cs.LG</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17201">https://arxiv.org/abs/2409.17201</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17201">https://arxiv.org/pdf/2409.17201</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17201]] Immersion and Invariance-based Coding for Privacy-Preserving Federated Learning(https://arxiv.org/abs/2409.17201)</code><input type="text"></li>

<li><strong>Keywords: </strong>privacy, protect, federate</a></li>

<li><strong>Abstract: </strong>Federated learning (FL) has emerged as a method to preserve privacy in collaborative distributed learning. In FL, clients train AI models directly on their devices rather than sharing data with a centralized server, which can pose privacy risks. However, it has been shown that despite FL's partial protection of local data privacy, information about clients' data can still be inferred from shared model updates during training. In recent years, several privacy-preserving approaches have been developed to mitigate this privacy leakage in FL, though they often provide privacy at the cost of model performance or system efficiency. Balancing these trade-offs presents a significant challenge in implementing FL schemes. In this manuscript, we introduce a privacy-preserving FL framework that combines differential privacy and system immersion tools from control theory. The core idea is to treat the optimization algorithms used in standard FL schemes (e.g., gradient-based algorithms) as a dynamical system that we seek to immerse into a higher-dimensional system (referred to as the target optimization algorithm). The target algorithm's dynamics are designed such that, first, the model parameters of the original algorithm are immersed in its parameters; second, it operates on distorted parameters; and third, it converges to an encoded version of the true model parameters from the original algorithm. These encoded parameters can then be decoded at the server to retrieve the original model parameters. We demonstrate that the proposed privacy-preserving scheme can be tailored to offer any desired level of differential privacy for both local and global model parameters, while maintaining the same accuracy and convergence rate as standard FL algorithms.</li>

</ul>

<h3>Title: 2024 BRAVO Challenge Track 1 1st Place Report: Evaluating Robustness of Vision Foundation Models for Semantic Segmentation</h3>

<ul>

<li><strong>Authors: </strong>Tommie Kerssies, Daan de Geus, Gijs Dubbelman</a></li>

<li><strong>Subjects: </strong>cs.CV, cs.AI, cs.LG, cs.RO</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17208">https://arxiv.org/abs/2409.17208</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17208">https://arxiv.org/pdf/2409.17208</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17208]] 2024 BRAVO Challenge Track 1 1st Place Report: Evaluating Robustness of Vision Foundation Models for Semantic Segmentation(https://arxiv.org/abs/2409.17208)</code><input type="text"></li>

<li><strong>Keywords: </strong>robust, segmentation</a></li>

<li><strong>Abstract: </strong>In this report, we present our solution for Track 1 of the 2024 BRAVO Challenge, where a model is trained on Cityscapes and its robustness is evaluated on several out-of-distribution datasets. Our solution leverages the powerful representations learned by vision foundation models, by attaching a simple segmentation decoder to DINOv2 and fine-tuning the entire model. This approach outperforms more complex existing approaches, and achieves 1st place in the challenge. Our code is publicly available at this https URL.</li>

</ul>

<h3>Title: Mnemosyne: Parallelization Strategies for Efficiently Serving Multi-Million Context Length LLM Inference Requests Without Approximations</h3>

<ul>

<li><strong>Authors: </strong>Amey Agrawal, Junda Chen, Íñigo Goiri, Ramachandran Ramjee, Chaojie Zhang, Alexey Tumanov, Esha Choukse</a></li>

<li><strong>Subjects: </strong>cs.LG, cs.DC</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17264">https://arxiv.org/abs/2409.17264</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17264">https://arxiv.org/pdf/2409.17264</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17264]] Mnemosyne: Parallelization Strategies for Efficiently Serving Multi-Million Context Length LLM Inference Requests Without Approximations(https://arxiv.org/abs/2409.17264)</code><input type="text"></li>

<li><strong>Keywords: </strong>large language model</a></li>

<li><strong>Abstract: </strong>As large language models (LLMs) evolve to handle increasingly longer contexts, serving inference requests for context lengths in the range of millions of tokens presents unique challenges. While existing techniques are effective for training, they fail to address the unique challenges of inference, such as varying prefill and decode phases and their associated latency constraints - like Time to First Token (TTFT) and Time Between Tokens (TBT). Furthermore, there are no long context inference solutions that allow batching requests to increase the hardware utilization today. In this paper, we propose three key innovations for efficient interactive long context LLM inference, without resorting to any approximation: adaptive chunking to reduce prefill overheads in mixed batching, Sequence Pipeline Parallelism (SPP) to lower TTFT, and KV Cache Parallelism (KVP) to minimize TBT. These contributions are combined into a 3D parallelism strategy, enabling Mnemosyne to scale interactive inference to context lengths at least up to 10 million tokens with high throughput enabled with batching. To our knowledge, Mnemosyne is the first to be able to achieve support for 10 million long context inference efficiently, while satisfying production-grade SLOs on TBT (30ms) on contexts up to and including 10 million.</li>

</ul>

<h3>Title: Model aggregation: minimizing empirical variance outperforms minimizing empirical error</h3>

<ul>

<li><strong>Authors: </strong>Théo Bourdais, Houman Owhadi</a></li>

<li><strong>Subjects: </strong>cs.LG, cs.AI, math.NA, stat.ML</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17267">https://arxiv.org/abs/2409.17267</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17267">https://arxiv.org/pdf/2409.17267</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17267]] Model aggregation: minimizing empirical variance outperforms minimizing empirical error(https://arxiv.org/abs/2409.17267)</code><input type="text"></li>

<li><strong>Keywords: </strong>robust</a></li>

<li><strong>Abstract: </strong>Whether deterministic or stochastic, models can be viewed as functions designed to approximate a specific quantity of interest. We propose a data-driven framework that aggregates predictions from diverse models into a single, more accurate output. This aggregation approach exploits each model's strengths to enhance overall accuracy. It is non-intrusive - treating models as black-box functions - model-agnostic, requires minimal assumptions, and can combine outputs from a wide range of models, including those from machine learning and numerical solvers. We argue that the aggregation process should be point-wise linear and propose two methods to find an optimal aggregate: Minimal Error Aggregation (MEA), which minimizes the aggregate's prediction error, and Minimal Variance Aggregation (MVA), which minimizes its variance. While MEA is inherently more accurate when correlations between models and the target quantity are perfectly known, Minimal Empirical Variance Aggregation (MEVA), an empirical version of MVA - consistently outperforms Minimal Empirical Error Aggregation (MEEA), the empirical counterpart of MEA, when these correlations must be estimated from data. The key difference is that MEVA constructs an aggregate by estimating model errors, while MEEA treats the models as features for direct interpolation of the quantity of interest. This makes MEEA more susceptible to overfitting and poor generalization, where the aggregate may underperform individual models during testing. We demonstrate the versatility and effectiveness of our framework in various applications, such as data science and partial differential equations, showing how it successfully integrates traditional solvers with machine learning models to improve both robustness and accuracy.</li>

</ul>

<h3>Title: On the Vulnerability of Applying Retrieval-Augmented Generation within Knowledge-Intensive Application Domains</h3>

<ul>

<li><strong>Authors: </strong>Xun Xian, Ganghua Wang, Xuan Bi, Jayanth Srinivasa, Ashish Kundu, Charles Fleming, Mingyi Hong, Jie Ding</a></li>

<li><strong>Subjects: </strong>cs.CR, cs.AI, cs.CL, cs.DB, cs.ET, cs.IR, cs.LG</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17275">https://arxiv.org/abs/2409.17275</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17275">https://arxiv.org/pdf/2409.17275</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17275]] On the Vulnerability of Applying Retrieval-Augmented Generation within Knowledge-Intensive Application Domains(https://arxiv.org/abs/2409.17275)</code><input type="text"></li>

<li><strong>Keywords: </strong>defense, attack, robust, large language model</a></li>

<li><strong>Abstract: </strong>Retrieval-Augmented Generation (RAG) has been empirically shown to enhance the performance of large language models (LLMs) in knowledge-intensive domains such as healthcare, finance, and legal contexts. Given a query, RAG retrieves relevant documents from a corpus and integrates them into the LLMs' generation process. In this study, we investigate the adversarial robustness of RAG, focusing specifically on examining the retrieval system. First, across 225 different setup combinations of corpus, retriever, query, and targeted information, we show that retrieval systems are vulnerable to universal poisoning attacks in medical Q\&A. In such attacks, adversaries generate poisoned documents containing a broad spectrum of targeted information, such as personally identifiable information. When these poisoned documents are inserted into a corpus, they can be accurately retrieved by any users, as long as attacker-specified queries are used. To understand this vulnerability, we discovered that the deviation from the query's embedding to that of the poisoned document tends to follow a pattern in which the high similarity between the poisoned document and the query is retained, thereby enabling precise retrieval. Based on these findings, we develop a new detection-based defense to ensure the safe use of RAG. Through extensive experiments spanning various Q\&A domains, we observed that our proposed method consistently achieves excellent detection rates in nearly all cases.</li>

</ul>

<h3>Title: SHEATH: Defending Horizontal Collaboration for Distributed CNNs against Adversarial Noise</h3>

<ul>

<li><strong>Authors: </strong>Muneeba Asif, Mohammad Kumail Kazmi, Mohammad Ashiqur Rahman, Syed Rafay Hasan, Soamar Homsi</a></li>

<li><strong>Subjects: </strong>cs.CR, cs.DC</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17279">https://arxiv.org/abs/2409.17279</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17279">https://arxiv.org/pdf/2409.17279</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17279]] SHEATH: Defending Horizontal Collaboration for Distributed CNNs against Adversarial Noise(https://arxiv.org/abs/2409.17279)</code><input type="text"></li>

<li><strong>Keywords: </strong>secure, security, privacy, attack</a></li>

<li><strong>Abstract: </strong>As edge computing and the Internet of Things (IoT) expand, horizontal collaboration (HC) emerges as a distributed data processing solution for resource-constrained devices. In particular, a convolutional neural network (CNN) model can be deployed on multiple IoT devices, allowing distributed inference execution for image recognition while ensuring model and data privacy. Yet, this distributed architecture remains vulnerable to adversaries who want to make subtle alterations that impact the model, even if they lack access to the entire model. Such vulnerabilities can have severe implications for various sectors, including healthcare, military, and autonomous systems. However, security solutions for these vulnerabilities have not been explored. This paper presents a novel framework for Secure Horizontal Edge with Adversarial Threat Handling (SHEATH) to detect adversarial noise and eliminate its effect on CNN inference by recovering the original feature maps. Specifically, SHEATH aims to address vulnerabilities without requiring complete knowledge of the CNN model in HC edge architectures based on sequential partitioning. It ensures data and model integrity, offering security against adversarial attacks in diverse HC environments. Our evaluations demonstrate SHEATH's adaptability and effectiveness across diverse CNN configurations.</li>

</ul>

<h3>Title: Disco4D: Disentangled 4D Human Generation and Animation from a Single Image</h3>

<ul>

<li><strong>Authors: </strong>Hui En Pang, Shuai Liu, Zhongang Cai, Lei Yang, Tianwei Zhang, Ziwei Liu</a></li>

<li><strong>Subjects: </strong>cs.CV</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17280">https://arxiv.org/abs/2409.17280</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17280">https://arxiv.org/pdf/2409.17280</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17280]] Disco4D: Disentangled 4D Human Generation and Animation from a Single Image(https://arxiv.org/abs/2409.17280)</code><input type="text"></li>

<li><strong>Keywords: </strong>extraction, diffusion</a></li>

<li><strong>Abstract: </strong>We present \textbf{Disco4D}, a novel Gaussian Splatting framework for 4D human generation and animation from a single image. Different from existing methods, Disco4D distinctively disentangles clothings (with Gaussian models) from the human body (with SMPL-X model), significantly enhancing the generation details and flexibility. It has the following technical innovations. \textbf{1)} Disco4D learns to efficiently fit the clothing Gaussians over the SMPL-X Gaussians. \textbf{2)} It adopts diffusion models to enhance the 3D generation process, \textit{e.g.}, modeling occluded parts not visible in the input image. \textbf{3)} It learns an identity encoding for each clothing Gaussian to facilitate the separation and extraction of clothing assets. Furthermore, Disco4D naturally supports 4D human animation with vivid dynamics. Extensive experiments demonstrate the superiority of Disco4D on 4D human generation and animation tasks. Our visualizations can be found in \url{this https URL}.</li>

</ul>

<h3>Title: Investigating Privacy Attacks in the Gray-Box Setting to Enhance Collaborative Learning Schemes</h3>

<ul>

<li><strong>Authors: </strong>Federico Mazzone, Ahmad Al Badawi, Yuriy Polyakov, Maarten Everts, Florian Hahn, Andreas Peter</a></li>

<li><strong>Subjects: </strong>cs.CR</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17283">https://arxiv.org/abs/2409.17283</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17283">https://arxiv.org/pdf/2409.17283</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17283]] Investigating Privacy Attacks in the Gray-Box Setting to Enhance Collaborative Learning Schemes(https://arxiv.org/abs/2409.17283)</code><input type="text"></li>

<li><strong>Keywords: </strong>privacy, protect, attack</a></li>

<li><strong>Abstract: </strong>The notion that collaborative machine learning can ensure privacy by just withholding the raw data is widely acknowledged to be flawed. Over the past seven years, the literature has revealed several privacy attacks that enable adversaries to extract information about a model's training dataset by exploiting access to model parameters during or after training. In this work, we study privacy attacks in the gray-box setting, where the attacker has only limited access - in terms of view and actions - to the model. The findings of our investigation provide new insights for the development of privacy-preserving collaborative learning solutions. We deploy SmartCryptNN, a framework that tailors homomorphic encryption to protect the portions of the model posing higher privacy risks. Our solution offers a trade-off between privacy and efficiency, which varies based on the extent and selection of the model components we choose to protect. We explore it on dense neural networks, where through extensive evaluation of diverse datasets and architectures, we uncover instances where a favorable sweet spot in the trade-off can be achieved by safeguarding only a single layer of the network. In one of such instances, our approach trains ~4 times faster compared to fully encrypted solutions, while reducing membership leakage by 17.8 times compared to plaintext solutions.</li>

</ul>

<h3>Title: Blockchain-Enabled Variational Information Bottleneck for Data Extraction Based on Mutual Information in Internet of Vehicles</h3>

<ul>

<li><strong>Authors: </strong>Cui Zhang, Wenjun Zhang, Qiong Wu, Pingyi Fan, Nan Cheng, Wen Chen, Khaled B. Letaief</a></li>

<li><strong>Subjects: </strong>cs.CR, cs.LG</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17287">https://arxiv.org/abs/2409.17287</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17287">https://arxiv.org/pdf/2409.17287</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17287]] Blockchain-Enabled Variational Information Bottleneck for Data Extraction Based on Mutual Information in Internet of Vehicles(https://arxiv.org/abs/2409.17287)</code><input type="text"></li>

<li><strong>Keywords: </strong>secure, security, privacy, extraction</a></li>

<li><strong>Abstract: </strong>The Internet of Vehicles (IoV) network can address the issue of limited computing resources and data processing capabilities of individual vehicles, but it also brings the risk of privacy leakage to vehicle users. Applying blockchain technology can establish secure data links within the IoV, solving the problems of insufficient computing resources for each vehicle and the security of data transmission over the network. However, with the development of the IoV, the amount of data interaction between multiple vehicles and between vehicles and base stations, roadside units, etc., is continuously increasing. There is a need to further reduce the interaction volume, and intelligent data compression is key to solving this problem. The VIB technique facilitates the training of encoding and decoding models, substantially diminishing the volume of data that needs to be transmitted. This paper introduces an innovative approach that integrates blockchain with VIB, referred to as BVIB, designed to lighten computational workloads and reinforce the security of the network. We first construct a new network framework by separating the encoding and decoding networks to address the computational burden issue, and then propose a new algorithm to enhance the security of IoV networks. We also discuss the impact of the data extraction rate on system latency to determine the most suitable data extraction rate. An experimental framework combining Python and C++ has been established to substantiate the efficacy of our BVIB approach. Comprehensive simulation studies indicate that the BVIB consistently excels in comparison to alternative foundational methodologies.</li>

</ul>

<h3>Title: Consistent estimation of generative model representations in the data kernel perspective space</h3>

<ul>

<li><strong>Authors: </strong>Aranyak Acharyya, Michael W. Trosset, Carey E. Priebe, Hayden S. Helm</a></li>

<li><strong>Subjects: </strong>cs.LG, math.ST</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17308">https://arxiv.org/abs/2409.17308</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17308">https://arxiv.org/pdf/2409.17308</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17308]] Consistent estimation of generative model representations in the data kernel perspective space(https://arxiv.org/abs/2409.17308)</code><input type="text"></li>

<li><strong>Keywords: </strong>diffusion, generative, large language model</a></li>

<li><strong>Abstract: </strong>Generative models, such as large language models and text-to-image diffusion models, produce relevant information when presented a query. Different models may produce different information when presented the same query. As the landscape of generative models evolves, it is important to develop techniques to study and analyze differences in model behaviour. In this paper we present novel theoretical results for embedding-based representations of generative models in the context of a set of queries. We establish sufficient conditions for the consistent estimation of the model embeddings in situations where the query set and the number of models grow.</li>

</ul>

<h3>Title: KIPPS: Knowledge infusion in Privacy Preserving Synthetic Data Generation</h3>

<ul>

<li><strong>Authors: </strong>Anantaa Kotal, Anupam Joshi</a></li>

<li><strong>Subjects: </strong>cs.LG, cs.AI, cs.CR</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17315">https://arxiv.org/abs/2409.17315</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17315">https://arxiv.org/pdf/2409.17315</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17315]] KIPPS: Knowledge infusion in Privacy Preserving Synthetic Data Generation(https://arxiv.org/abs/2409.17315)</code><input type="text"></li>

<li><strong>Keywords: </strong>security, privacy, generative</a></li>

<li><strong>Abstract: </strong>The integration of privacy measures, including differential privacy techniques, ensures a provable privacy guarantee for the synthetic data. However, challenges arise for Generative Deep Learning models when tasked with generating realistic data, especially in critical domains such as Cybersecurity and Healthcare. Generative Models optimized for continuous data struggle to model discrete and non-Gaussian features that have domain constraints. Challenges increase when the training datasets are limited and not diverse. In such cases, generative models create synthetic data that repeats sensitive features, which is a privacy risk. Moreover, generative models face difficulties comprehending attribute constraints in specialized domains. This leads to the generation of unrealistic data that impacts downstream accuracy. To address these issues, this paper proposes a novel model, KIPPS, that infuses Domain and Regulatory Knowledge from Knowledge Graphs into Generative Deep Learning models for enhanced Privacy Preserving Synthetic data generation. The novel framework augments the training of generative models with supplementary context about attribute values and enforces domain constraints during training. This added guidance enhances the model's capacity to generate realistic and domain-compliant synthetic data. The proposed model is evaluated on real-world datasets, specifically in the domains of Cybersecurity and Healthcare, where domain constraints and rules add to the complexity of the data. Our experiments evaluate the privacy resilience and downstream accuracy of the model against benchmark methods, demonstrating its effectiveness in addressing the balance between privacy preservation and data accuracy in complex domains.</li>

</ul>

<h3>Title: Bi-TTA: Bidirectional Test-Time Adapter for Remote Physiological Measurement</h3>

<ul>

<li><strong>Authors: </strong>Haodong Li, Hao Lu, Ying-Cong Chen</a></li>

<li><strong>Subjects: </strong>cs.CV</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17316">https://arxiv.org/abs/2409.17316</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17316">https://arxiv.org/pdf/2409.17316</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17316]] Bi-TTA: Bidirectional Test-Time Adapter for Remote Physiological Measurement(https://arxiv.org/abs/2409.17316)</code><input type="text"></li>

<li><strong>Keywords: </strong>privacy</a></li>

<li><strong>Abstract: </strong>Remote photoplethysmography (rPPG) is gaining prominence for its non-invasive approach to monitoring physiological signals using only cameras. Despite its promise, the adaptability of rPPG models to new, unseen domains is hindered due to the environmental sensitivity of physiological signals. To address this, we pioneer the Test-Time Adaptation (TTA) in rPPG, enabling the adaptation of pre-trained models to the target domain during inference, sidestepping the need for annotations or source data due to privacy considerations. Particularly, utilizing only the user's face video stream as the accessible target domain data, the rPPG model is adjusted by tuning on each single instance it encounters. However, 1) TTA algorithms are designed predominantly for classification tasks, ill-suited in regression tasks such as rPPG due to inadequate supervision. 2) Tuning pre-trained models in a single-instance manner introduces variability and instability, posing challenges to effectively filtering domain-relevant from domain-irrelevant features while simultaneously preserving the learned information. To overcome these challenges, we present Bi-TTA, a novel expert knowledge-based Bidirectional Test-Time Adapter framework. Specifically, leveraging two expert-knowledge priors for providing self-supervision, our Bi-TTA primarily comprises two modules: a prospective adaptation (PA) module using sharpness-aware minimization to eliminate domain-irrelevant noise, enhancing the stability and efficacy during the adaptation process, and a retrospective stabilization (RS) module to dynamically reinforce crucial learned model parameters, averting performance degradation caused by overfitting or catastrophic forgetting. To this end, we established a large-scale benchmark for rPPG tasks under TTA protocol. The experimental results demonstrate the significant superiority of our approach over the state-of-the-art.</li>

</ul>

<h3>Title: The poison of dimensionality</h3>

<ul>

<li><strong>Authors: </strong>Lê-Nguyên Hoang</a></li>

<li><strong>Subjects: </strong>cs.LG, cs.CR, stat.ML</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17328">https://arxiv.org/abs/2409.17328</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17328">https://arxiv.org/pdf/2409.17328</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17328]] The poison of dimensionality(https://arxiv.org/abs/2409.17328)</code><input type="text"></li>

<li><strong>Keywords: </strong>defense, attack, robust</a></li>

<li><strong>Abstract: </strong>This paper advances the understanding of how the size of a machine learning model affects its vulnerability to poisoning, despite state-of-the-art defenses. Given isotropic random honest feature vectors and the geometric median (or clipped mean) as the robust gradient aggregator rule, we essentially prove that, perhaps surprisingly, linear and logistic regressions with $D \geq 169 H^2/P^2$ parameters are subject to arbitrary model manipulation by poisoners, where $H$ and $P$ are the numbers of honestly labeled and poisoned data points used for training. Our experiments go on exposing a fundamental tradeoff between augmenting model expressivity and increasing the poisoners' attack surface, on both synthetic data, and on MNIST & FashionMNIST data for linear classifiers with random features. We also discuss potential implications for source-based learning and neural nets.</li>

</ul>

<h3>Title: VL4AD: Vision-Language Models Improve Pixel-wise Anomaly Detection</h3>

<ul>

<li><strong>Authors: </strong>Liangyu Zhong, Joachim Sicking, Fabian Hüger, Hanno Gottschalk</a></li>

<li><strong>Subjects: </strong>cs.CV</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17330">https://arxiv.org/abs/2409.17330</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17330">https://arxiv.org/pdf/2409.17330</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17330]] VL4AD: Vision-Language Models Improve Pixel-wise Anomaly Detection(https://arxiv.org/abs/2409.17330)</code><input type="text"></li>

<li><strong>Keywords: </strong>segmentation</a></li>

<li><strong>Abstract: </strong>Semantic segmentation networks have achieved significant success under the assumption of independent and identically distributed data. However, these networks often struggle to detect anomalies from unknown semantic classes due to the limited set of visual concepts they are typically trained on. To address this issue, anomaly segmentation often involves fine-tuning on outlier samples, necessitating additional efforts for data collection, labeling, and model retraining. Seeking to avoid this cumbersome work, we take a different approach and propose to incorporate Vision-Language (VL) encoders into existing anomaly detectors to leverage the semantically broad VL pre-training for improved outlier awareness. Additionally, we propose a new scoring function that enables data- and training-free outlier supervision via textual prompts. The resulting VL4AD model, which includes max-logit prompt ensembling and a class-merging strategy, achieves competitive performance on widely used benchmark datasets, thereby demonstrating the potential of vision-language models for pixel-wise anomaly detection.</li>

</ul>

<h3>Title: ChatCam: Empowering Camera Control through Conversational AI</h3>

<ul>

<li><strong>Authors: </strong>Xinhang Liu, Yu-Wing Tai, Chi-Keung Tang</a></li>

<li><strong>Subjects: </strong>cs.CV</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17331">https://arxiv.org/abs/2409.17331</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17331">https://arxiv.org/pdf/2409.17331</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17331]] ChatCam: Empowering Camera Control through Conversational AI(https://arxiv.org/abs/2409.17331)</code><input type="text"></li>

<li><strong>Keywords: </strong>large language model</a></li>

<li><strong>Abstract: </strong>Cinematographers adeptly capture the essence of the world, crafting compelling visual narratives through intricate camera movements. Witnessing the strides made by large language models in perceiving and interacting with the 3D world, this study explores their capability to control cameras with human language guidance. We introduce ChatCam, a system that navigates camera movements through conversations with users, mimicking a professional cinematographer's workflow. To achieve this, we propose CineGPT, a GPT-based autoregressive model for text-conditioned camera trajectory generation. We also develop an Anchor Determinator to ensure precise camera trajectory placement. ChatCam understands user requests and employs our proposed tools to generate trajectories, which can be used to render high-quality video footage on radiance field representations. Our experiments, including comparisons to state-of-the-art approaches and user studies, demonstrate our approach's ability to interpret and execute complex instructions for camera operation, showing promising applications in real-world production settings.</li>

</ul>

<h3>Title: Block Expanded DINORET: Adapting Natural Domain Foundation Models for Retinal Imaging Without Catastrophic Forgetting</h3>

<ul>

<li><strong>Authors: </strong>Jay Zoellin, Colin Merk, Mischa Buob, Amr Saad, Samuel Giesser, Tahm Spitznagel, Ferhat Turgut, Rui Santos, Yukun Zhou, Sigfried Wagner, Pearse A. Keane, Yih Chung Tham, Delia Cabrera DeBuc, Matthias D. Becker, Gabor M. Somfai</a></li>

<li><strong>Subjects: </strong>cs.CV, cs.AI</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17332">https://arxiv.org/abs/2409.17332</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17332">https://arxiv.org/pdf/2409.17332</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17332]] Block Expanded DINORET: Adapting Natural Domain Foundation Models for Retinal Imaging Without Catastrophic Forgetting(https://arxiv.org/abs/2409.17332)</code><input type="text"></li>

<li><strong>Keywords: </strong>robust, transformer</a></li>

<li><strong>Abstract: </strong>Integrating deep learning into medical imaging is poised to greatly advance diagnostic methods but it faces challenges with generalizability. Foundation models, based on self-supervised learning, address these issues and improve data efficiency. Natural domain foundation models show promise for medical imaging, but systematic research evaluating domain adaptation, especially using self-supervised learning and parameter-efficient fine-tuning, remains underexplored. Additionally, little research addresses the issue of catastrophic forgetting during fine-tuning of foundation models. We adapted the DINOv2 vision transformer for retinal imaging classification tasks using self-supervised learning and generated two novel foundation models termed DINORET and BE DINORET. Publicly available color fundus photographs were employed for model development and subsequent fine-tuning for diabetic retinopathy staging and glaucoma detection. We introduced block expansion as a novel domain adaptation strategy and assessed the models for catastrophic forgetting. Models were benchmarked to RETFound, a state-of-the-art foundation model in ophthalmology. DINORET and BE DINORET demonstrated competitive performance on retinal imaging tasks, with the block expanded model achieving the highest scores on most datasets. Block expansion successfully mitigated catastrophic forgetting. Our few-shot learning studies indicated that DINORET and BE DINORET outperform RETFound in terms of data-efficiency. This study highlights the potential of adapting natural domain vision models to retinal imaging using self-supervised learning and block expansion. BE DINORET offers robust performance without sacrificing previously acquired capabilities. Our findings suggest that these methods could enable healthcare institutions to develop tailored vision models for their patient populations, enhancing global healthcare inclusivity.</li>

</ul>

<h3>Title: Non-asymptotic Convergence of Training Transformers for Next-token Prediction</h3>

<ul>

<li><strong>Authors: </strong>Ruiquan Huang, Yingbin Liang, Jing Yang</a></li>

<li><strong>Subjects: </strong>cs.LG, stat.ML</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17335">https://arxiv.org/abs/2409.17335</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17335">https://arxiv.org/pdf/2409.17335</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17335]] Non-asymptotic Convergence of Training Transformers for Next-token Prediction(https://arxiv.org/abs/2409.17335)</code><input type="text"></li>

<li><strong>Keywords: </strong>transformer</a></li>

<li><strong>Abstract: </strong>Transformers have achieved extraordinary success in modern machine learning due to their excellent ability to handle sequential data, especially in next-token prediction (NTP) tasks. However, the theoretical understanding of their performance in NTP is limited, with existing studies focusing mainly on asymptotic performance. This paper provides a fine-grained non-asymptotic analysis of the training dynamics of a one-layer transformer consisting of a self-attention module followed by a feed-forward layer. We first characterize the essential structural properties of training datasets for NTP using a mathematical framework based on partial orders. Then, we design a two-stage training algorithm, where the pre-processing stage for training the feed-forward layer and the main stage for training the attention layer exhibit fast convergence performance. Specifically, both layers converge sub-linearly to the direction of their corresponding max-margin solutions. We also show that the cross-entropy loss enjoys a linear convergence rate. Furthermore, we show that the trained transformer presents non-trivial prediction ability with dataset shift, which sheds light on the remarkable generalization performance of transformers. Our analysis technique involves the development of novel properties on the attention gradient and further in-depth analysis of how these properties contribute to the convergence of the training process. Our experiments further validate our theoretical findings.</li>

</ul>

<h3>Title: A vision-based framework for human behavior understanding in industrial assembly lines</h3>

<ul>

<li><strong>Authors: </strong>Konstantinos Papoutsakis, Nikolaos Bakalos, Konstantinos Fragkoulis, Athena Zacharia, Georgia Kapetadimitri, Maria Pateraki</a></li>

<li><strong>Subjects: </strong>cs.CV</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17356">https://arxiv.org/abs/2409.17356</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17356">https://arxiv.org/pdf/2409.17356</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17356]] A vision-based framework for human behavior understanding in industrial assembly lines(https://arxiv.org/abs/2409.17356)</code><input type="text"></li>

<li><strong>Keywords: </strong>robust</a></li>

<li><strong>Abstract: </strong>This paper introduces a vision-based framework for capturing and understanding human behavior in industrial assembly lines, focusing on car door manufacturing. The framework leverages advanced computer vision techniques to estimate workers' locations and 3D poses and analyze work postures, actions, and task progress. A key contribution is the introduction of the CarDA dataset, which contains domain-relevant assembly actions captured in a realistic setting to support the analysis of the framework for human pose and action analysis. The dataset comprises time-synchronized multi-camera RGB-D videos, motion capture data recorded in a real car manufacturing environment, and annotations for EAWS-based ergonomic risk scores and assembly activities. Experimental results demonstrate the effectiveness of the proposed approach in classifying worker postures and robust performance in monitoring assembly task progress.</li>

</ul>

<h3>Title: Improving satellite imagery segmentation using multiple Sentinel-2 revisits</h3>

<ul>

<li><strong>Authors: </strong>Kartik Jindgar, Grace W. Lindsay</a></li>

<li><strong>Subjects: </strong>cs.CV</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17363">https://arxiv.org/abs/2409.17363</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17363">https://arxiv.org/pdf/2409.17363</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17363]] Improving satellite imagery segmentation using multiple Sentinel-2 revisits(https://arxiv.org/abs/2409.17363)</code><input type="text"></li>

<li><strong>Keywords: </strong>transformer, segmentation</a></li>

<li><strong>Abstract: </strong>In recent years, analysis of remote sensing data has benefited immensely from borrowing techniques from the broader field of computer vision, such as the use of shared models pre-trained on large and diverse datasets. However, satellite imagery has unique features that are not accounted for in traditional computer vision, such as the existence of multiple revisits of the same location. Here, we explore the best way to use revisits in the framework of fine-tuning pre-trained remote sensing models. We focus on an applied research question of relevance to climate change mitigation -- power substation segmentation -- that is representative of applied uses of pre-trained models more generally. Through extensive tests of different multi-temporal input schemes across diverse model architectures, we find that fusing representations from multiple revisits in the model latent space is superior to other methods of using revisits, including as a form of data augmentation. We also find that a SWIN Transformer-based architecture performs better than U-nets and ViT-based models. We verify the generality of our results on a separate building density estimation task.</li>

</ul>

<h3>Title: The Overfocusing Bias of Convolutional Neural Networks: A Saliency-Guided Regularization Approach</h3>

<ul>

<li><strong>Authors: </strong>David Bertoin, Eduardo Hugo Sanchez, Mehdi Zouitine, Emmanuel Rachelson</a></li>

<li><strong>Subjects: </strong>cs.CV, cs.AI</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17370">https://arxiv.org/abs/2409.17370</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17370">https://arxiv.org/pdf/2409.17370</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17370]] The Overfocusing Bias of Convolutional Neural Networks: A Saliency-Guided Regularization Approach(https://arxiv.org/abs/2409.17370)</code><input type="text"></li>

<li><strong>Keywords: </strong>transformer</a></li>

<li><strong>Abstract: </strong>Despite transformers being considered as the new standard in computer vision, convolutional neural networks (CNNs) still outperform them in low-data regimes. Nonetheless, CNNs often make decisions based on narrow, specific regions of input images, especially when training data is limited. This behavior can severely compromise the model's generalization capabilities, making it disproportionately dependent on certain features that might not represent the broader context of images. While the conditions leading to this phenomenon remain elusive, the primary intent of this article is to shed light on this observed behavior of neural networks. Our research endeavors to prioritize comprehensive insight and to outline an initial response to this phenomenon. In line with this, we introduce Saliency Guided Dropout (SGDrop), a pioneering regularization approach tailored to address this specific issue. SGDrop utilizes attribution methods on the feature map to identify and then reduce the influence of the most salient features during training. This process encourages the network to diversify its attention and not focus solely on specific standout areas. Our experiments across several visual classification benchmarks validate SGDrop's role in enhancing generalization. Significantly, models incorporating SGDrop display more expansive attributions and neural activity, offering a more comprehensive view of input images in contrast to their traditionally trained counterparts.</li>

</ul>

<h3>Title: Optical Lens Attack on Deep Learning Based Monocular Depth Estimation</h3>

<ul>

<li><strong>Authors: </strong>Ce Zhou (1), Qiben Yan (1), Daniel Kent (1), Guangjing Wang (1), Ziqi Zhang (2), Hayder Radha (1) ((1) Michigan State University, (2) Peking University)</a></li>

<li><strong>Subjects: </strong>cs.CR, cs.CV</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17376">https://arxiv.org/abs/2409.17376</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17376">https://arxiv.org/pdf/2409.17376</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17376]] Optical Lens Attack on Deep Learning Based Monocular Depth Estimation(https://arxiv.org/abs/2409.17376)</code><input type="text"></li>

<li><strong>Keywords: </strong>security, attack</a></li>

<li><strong>Abstract: </strong>Monocular Depth Estimation (MDE) plays a crucial role in vision-based Autonomous Driving (AD) systems. It utilizes a single-camera image to determine the depth of objects, facilitating driving decisions such as braking a few meters in front of a detected obstacle or changing lanes to avoid collision. In this paper, we investigate the security risks associated with monocular vision-based depth estimation algorithms utilized by AD systems. By exploiting the vulnerabilities of MDE and the principles of optical lenses, we introduce LensAttack, a physical attack that involves strategically placing optical lenses on the camera of an autonomous vehicle to manipulate the perceived object depths. LensAttack encompasses two attack formats: concave lens attack and convex lens attack, each utilizing different optical lenses to induce false depth perception. We begin by constructing a mathematical model of our attack, incorporating various attack parameters. Subsequently, we simulate the attack and evaluate its real-world performance in driving scenarios to demonstrate its effect on state-of-the-art MDE models. The results highlight the significant impact of LensAttack on the accuracy of depth estimation in AD systems.</li>

</ul>

<h3>Title: Beyond Redundancy: Information-aware Unsupervised Multiplex Graph Structure Learning</h3>

<ul>

<li><strong>Authors: </strong>Zhixiang Shen, Shuo Wang, Zhao Kang</a></li>

<li><strong>Subjects: </strong>cs.LG, cs.AI, cs.SI</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17386">https://arxiv.org/abs/2409.17386</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17386">https://arxiv.org/pdf/2409.17386</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17386]] Beyond Redundancy: Information-aware Unsupervised Multiplex Graph Structure Learning(https://arxiv.org/abs/2409.17386)</code><input type="text"></li>

<li><strong>Keywords: </strong>robust</a></li>

<li><strong>Abstract: </strong>Unsupervised Multiplex Graph Learning (UMGL) aims to learn node representations on various edge types without manual labeling. However, existing research overlooks a key factor: the reliability of the graph structure. Real-world data often exhibit a complex nature and contain abundant task-irrelevant noise, severely compromising UMGL's performance. Moreover, existing methods primarily rely on contrastive learning to maximize mutual information across different graphs, limiting them to multiplex graph redundant scenarios and failing to capture view-unique task-relevant information. In this paper, we focus on a more realistic and challenging task: to unsupervisedly learn a fused graph from multiple graphs that preserve sufficient task-relevant information while removing task-irrelevant noise. Specifically, our proposed Information-aware Unsupervised Multiplex Graph Fusion framework (InfoMGF) uses graph structure refinement to eliminate irrelevant noise and simultaneously maximizes view-shared and view-unique task-relevant information, thereby tackling the frontier of non-redundant multiplex graph. Theoretical analyses further guarantee the effectiveness of InfoMGF. Comprehensive experiments against various baselines on different downstream tasks demonstrate its superior performance and robustness. Surprisingly, our unsupervised method even beats the sophisticated supervised approaches. The source code and datasets are available at this https URL.</li>

</ul>

<h3>Title: Scaling Behavior for Large Language Models regarding Numeral Systems: An Example using Pythia</h3>

<ul>

<li><strong>Authors: </strong>Zhejian Zhou, Jiayu Wang, Dahua Lin, Kai Chen</a></li>

<li><strong>Subjects: </strong>cs.CL</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17391">https://arxiv.org/abs/2409.17391</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17391">https://arxiv.org/pdf/2409.17391</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17391]] Scaling Behavior for Large Language Models regarding Numeral Systems: An Example using Pythia(https://arxiv.org/abs/2409.17391)</code><input type="text"></li>

<li><strong>Keywords: </strong>transformer, large language model</a></li>

<li><strong>Abstract: </strong>Though Large Language Models (LLMs) have shown remarkable abilities in mathematics reasoning, they are still struggling with performing numeric operations accurately, such as addition and multiplication. Numbers can be tokenized into tokens in various ways by different LLMs and affect the numeric operations performance. Currently, there are two representatives: 1) Tokenize into $1$-digit, and 2) Tokenize into $1\sim 3$ digit. The difference is roughly equivalent to using different numeral systems (namely base $10$ or base $10^{3}$). In light of this, we study the scaling behavior of different numeral systems in the context of transformer-based large language models. We empirically show that a base $10$ system is consistently more data-efficient than a base $10^{2}$ or $10^{3}$ system across training data scale, model sizes under from-scratch training settings, while different number systems have very similar fine-tuning performances. We attribute this to higher token frequencies of a base $10$ system. Additionally, we reveal extrapolation behavior patterns on addition and multiplication. We identify that base $100$ and base $1000$ systems struggle on token-level discernment and token-level operations. We also sheds light on the mechanism learnt by the models.</li>

</ul>

<h3>Title: Trading through Earnings Seasons using Self-Supervised Contrastive Representation Learning</h3>

<ul>

<li><strong>Authors: </strong>Zhengxin Joseph Ye, Bjoern Schuller</a></li>

<li><strong>Subjects: </strong>cs.LG, q-fin.TR</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17392">https://arxiv.org/abs/2409.17392</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17392">https://arxiv.org/pdf/2409.17392</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17392]] Trading through Earnings Seasons using Self-Supervised Contrastive Representation Learning(https://arxiv.org/abs/2409.17392)</code><input type="text"></li>

<li><strong>Keywords: </strong>transformer</a></li>

<li><strong>Abstract: </strong>Earnings release is a key economic event in the financial markets and crucial for predicting stock movements. Earnings data gives a glimpse into how a company is doing financially and can hint at where its stock might go next. However, the irregularity of its release cycle makes it a challenge to incorporate this data in a medium-frequency algorithmic trading model and the usefulness of this data fades fast after it is released, making it tough for models to stay accurate over time. Addressing this challenge, we introduce the Contrastive Earnings Transformer (CET) model, a self-supervised learning approach rooted in Contrastive Predictive Coding (CPC), aiming to optimise the utilisation of earnings data. To ascertain its effectiveness, we conduct a comparative study of CET against benchmark models across diverse sectors. Our research delves deep into the intricacies of stock data, evaluating how various models, and notably CET, handle the rapidly changing relevance of earnings data over time and over different sectors. The research outcomes shed light on CET's distinct advantage in extrapolating the inherent value of earnings data over time. Its foundation on CPC allows for a nuanced understanding, facilitating consistent stock predictions even as the earnings data ages. This finding about CET presents a fresh approach to better use earnings data in algorithmic trading for predicting stock price trends.</li>

</ul>

<h3>Title: Severity Prediction in Mental Health: LLM-based Creation, Analysis, Evaluation of a Novel Multilingual Dataset</h3>

<ul>

<li><strong>Authors: </strong>Konstantinos Skianis, John Pavlopoulos, A. Seza Doğruöz</a></li>

<li><strong>Subjects: </strong>cs.CL, cs.LG</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17397">https://arxiv.org/abs/2409.17397</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17397">https://arxiv.org/pdf/2409.17397</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17397]] Severity Prediction in Mental Health: LLM-based Creation, Analysis, Evaluation of a Novel Multilingual Dataset(https://arxiv.org/abs/2409.17397)</code><input type="text"></li>

<li><strong>Keywords: </strong>large language model</a></li>

<li><strong>Abstract: </strong>Large Language Models (LLMs) are increasingly integrated into various medical fields, including mental health support systems. However, there is a gap in research regarding the effectiveness of LLMs in non-English mental health support applications. To address this problem, we present a novel multilingual adaptation of widely-used mental health datasets, translated from English into six languages (Greek, Turkish, French, Portuguese, German, and Finnish). This dataset enables a comprehensive evaluation of LLM performance in detecting mental health conditions and assessing their severity across multiple languages. By experimenting with GPT and Llama, we observe considerable variability in performance across languages, despite being evaluated on the same translated dataset. This inconsistency underscores the complexities inherent in multilingual mental health support, where language-specific nuances and mental health data coverage can affect the accuracy of the models. Through comprehensive error analysis, we emphasize the risks of relying exclusively on large language models (LLMs) in medical settings (e.g., their potential to contribute to misdiagnoses). Moreover, our proposed approach offers significant cost savings for multilingual tasks, presenting a major advantage for broad-scale implementation.</li>

</ul>

<h3>Title: AgRegNet: A Deep Regression Network for Flower and Fruit Density Estimation, Localization, and Counting in Orchards</h3>

<ul>

<li><strong>Authors: </strong>Uddhav Bhattarai, Santosh Bhusal, Qin Zhang, Manoj Karkee</a></li>

<li><strong>Subjects: </strong>cs.CV, cs.AI</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17400">https://arxiv.org/abs/2409.17400</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17400">https://arxiv.org/pdf/2409.17400</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17400]] AgRegNet: A Deep Regression Network for Flower and Fruit Density Estimation, Localization, and Counting in Orchards(https://arxiv.org/abs/2409.17400)</code><input type="text"></li>

<li><strong>Keywords: </strong>segmentation</a></li>

<li><strong>Abstract: </strong>One of the major challenges for the agricultural industry today is the uncertainty in manual labor availability and the associated cost. Automated flower and fruit density estimation, localization, and counting could help streamline harvesting, yield estimation, and crop-load management strategies such as flower and fruitlet thinning. This article proposes a deep regression-based network, AgRegNet, to estimate density, count, and location of flower and fruit in tree fruit canopies without explicit object detection or polygon annotation. Inspired by popular U-Net architecture, AgRegNet is a U-shaped network with an encoder-to-decoder skip connection and modified ConvNeXt-T as an encoder feature extractor. AgRegNet can be trained based on information from point annotation and leverages segmentation information and attention modules (spatial and channel) to highlight relevant flower and fruit features while suppressing non-relevant background features. Experimental evaluation in apple flower and fruit canopy images under an unstructured orchard environment showed that AgRegNet achieved promising accuracy as measured by Structural Similarity Index (SSIM), percentage Mean Absolute Error (pMAE) and mean Average Precision (mAP) to estimate flower and fruit density, count, and centroid location, respectively. Specifically, the SSIM, pMAE, and mAP values for flower images were 0.938, 13.7%, and 0.81, respectively. For fruit images, the corresponding values were 0.910, 5.6%, and 0.93. Since the proposed approach relies on information from point annotation, it is suitable for sparsely and densely located objects. This simplified technique will be highly applicable for growers to accurately estimate yields and decide on optimal chemical and mechanical flower thinning practices.</li>

</ul>

<h3>Title: Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inference</h3>

<ul>

<li><strong>Authors: </strong>Qining Zhang, Lei Ying</a></li>

<li><strong>Subjects: </strong>cs.LG, stat.ML</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17401">https://arxiv.org/abs/2409.17401</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17401">https://arxiv.org/pdf/2409.17401</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17401]] Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inference(https://arxiv.org/abs/2409.17401)</code><input type="text"></li>

<li><strong>Keywords: </strong>large language model</a></li>

<li><strong>Abstract: </strong>Reward inference (learning a reward model from human preferences) is a critical intermediate step in Reinforcement Learning from Human Feedback (RLHF) for fine-tuning Large Language Models (LLMs) such as ChatGPT. In practice, reward inference faces several fundamental challenges, including double problem misspecification, reward model evaluation without ground truth, distribution shift, and overfitting in joint reward model and policy training. An alternative approach that avoids these pitfalls is direct policy optimization without reward inference, such as Direct Preference Optimization (DPO), which provides a much simpler pipeline and has shown empirical success in LLMs. However, DPO utilizes the closed-form expression between the optimal policy and the reward function, which only works under the bandit setting or deterministic MDPs. This paper develops two RLHF algorithms without reward inference, which work for general RL problems beyond bandits and deterministic MDPs, and general preference models beyond the Bradely-Terry model. The key idea is to estimate the local value function difference from human preferences and then approximate the policy gradient with a zeroth-order gradient approximator. For both algorithms, we establish rates of convergence in terms of the number of policy gradient iterations, as well as the number of trajectory samples and human preference queries per iteration. Our results show there exist provably efficient methods to solve general RLHF problems without reward inference.</li>

</ul>

<h3>Title: Transient Adversarial 3D Projection Attacks on Object Detection in Autonomous Driving</h3>

<ul>

<li><strong>Authors: </strong>Ce Zhou, Qiben Yan, Sijia Liu</a></li>

<li><strong>Subjects: </strong>cs.CR, cs.AI, cs.CV</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17403">https://arxiv.org/abs/2409.17403</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17403">https://arxiv.org/pdf/2409.17403</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17403]] Transient Adversarial 3D Projection Attacks on Object Detection in Autonomous Driving(https://arxiv.org/abs/2409.17403)</code><input type="text"></li>

<li><strong>Keywords: </strong>attack</a></li>

<li><strong>Abstract: </strong>Object detection is a crucial task in autonomous driving. While existing research has proposed various attacks on object detection, such as those using adversarial patches or stickers, the exploration of projection attacks on 3D surfaces remains largely unexplored. Compared to adversarial patches or stickers, which have fixed adversarial patterns, projection attacks allow for transient modifications to these patterns, enabling a more flexible attack. In this paper, we introduce an adversarial 3D projection attack specifically targeting object detection in autonomous driving scenarios. We frame the attack formulation as an optimization problem, utilizing a combination of color mapping and geometric transformation models. Our results demonstrate the effectiveness of the proposed attack in deceiving YOLOv3 and Mask R-CNN in physical settings. Evaluations conducted in an indoor environment show an attack success rate of up to 100% under low ambient light conditions, highlighting the potential damage of our attack in real-world driving scenarios.</li>

</ul>

<h3>Title: From Deception to Detection: The Dual Roles of Large Language Models in Fake News</h3>

<ul>

<li><strong>Authors: </strong>Dorsaf Sallami, Yuan-Chen Chang, Esma Aïmeur</a></li>

<li><strong>Subjects: </strong>cs.CL, cs.AI</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17416">https://arxiv.org/abs/2409.17416</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17416">https://arxiv.org/pdf/2409.17416</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17416]] From Deception to Detection: The Dual Roles of Large Language Models in Fake News(https://arxiv.org/abs/2409.17416)</code><input type="text"></li>

<li><strong>Keywords: </strong>robust, large language model</a></li>

<li><strong>Abstract: </strong>Fake news poses a significant threat to the integrity of information ecosystems and public trust. The advent of Large Language Models (LLMs) holds considerable promise for transforming the battle against fake news. Generally, LLMs represent a double-edged sword in this struggle. One major concern is that LLMs can be readily used to craft and disseminate misleading information on a large scale. This raises the pressing questions: Can LLMs easily generate biased fake news? Do all LLMs have this capability? Conversely, LLMs offer valuable prospects for countering fake news, thanks to their extensive knowledge of the world and robust reasoning capabilities. This leads to other critical inquiries: Can we use LLMs to detect fake news, and do they outperform typical detection models? In this paper, we aim to address these pivotal questions by exploring the performance of various LLMs. Our objective is to explore the capability of various LLMs in effectively combating fake news, marking this as the first investigation to analyze seven such models. Our results reveal that while some models adhere strictly to safety protocols, refusing to generate biased or misleading content, other models can readily produce fake news across a spectrum of biases. Additionally, our results show that larger models generally exhibit superior detection abilities and that LLM-generated fake news are less likely to be detected than human-written ones. Finally, our findings demonstrate that users can benefit from LLM-generated explanations in identifying fake news.</li>

</ul>

<h3>Title: Discovering the Gems in Early Layers: Accelerating Long-Context LLMs with 1000x Input Token Reduction</h3>

<ul>

<li><strong>Authors: </strong>Zhenmei Shi, Yifei Ming, Xuan-Phi Nguyen, Yingyu Liang, Shafiq Joty</a></li>

<li><strong>Subjects: </strong>cs.CL, cs.AI, cs.LG</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17422">https://arxiv.org/abs/2409.17422</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17422">https://arxiv.org/pdf/2409.17422</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17422]] Discovering the Gems in Early Layers: Accelerating Long-Context LLMs with 1000x Input Token Reduction(https://arxiv.org/abs/2409.17422)</code><input type="text"></li>

<li><strong>Keywords: </strong>interpretability, large language model</a></li>

<li><strong>Abstract: </strong>Large Language Models (LLMs) have demonstrated remarkable capabilities in handling long context inputs, but this comes at the cost of increased computational resources and latency. Our research introduces a novel approach for the long context bottleneck to accelerate LLM inference and reduce GPU memory consumption. Our research demonstrates that LLMs can identify relevant tokens in the early layers before generating answers to a query. Leveraging this insight, we propose an algorithm that uses early layers of an LLM as filters to select and compress input tokens, significantly reducing the context length for subsequent processing. Our method, GemFilter, demonstrates substantial improvements in both speed and memory efficiency compared to existing techniques, such as standard attention and SnapKV/H2O. Notably, it achieves a 2.4$\times$ speedup and 30\% reduction in GPU memory usage compared to SOTA methods. Evaluation on the Needle in a Haystack task shows that GemFilter significantly outperforms standard attention, SnapKV and demonstrates comparable performance on the LongBench challenge. GemFilter is simple, training-free, and broadly applicable across different LLMs. Crucially, it provides interpretability by allowing humans to inspect the selected input sequence. These findings not only offer practical benefits for LLM deployment, but also enhance our understanding of LLM internal mechanisms, paving the way for further optimizations in LLM design and inference. Our code is available at \url{this https URL}.</li>

</ul>

<h3>Title: HDFlow: Enhancing LLM Complex Problem-Solving with Hybrid Thinking and Dynamic Workflows</h3>

<ul>

<li><strong>Authors: </strong>Wenlin Yao, Haitao Mi, Dong Yu</a></li>

<li><strong>Subjects: </strong>cs.CL, cs.AI</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17433">https://arxiv.org/abs/2409.17433</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17433">https://arxiv.org/pdf/2409.17433</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17433]] HDFlow: Enhancing LLM Complex Problem-Solving with Hybrid Thinking and Dynamic Workflows(https://arxiv.org/abs/2409.17433)</code><input type="text"></li>

<li><strong>Keywords: </strong>large language model</a></li>

<li><strong>Abstract: </strong>Despite recent advancements in large language models (LLMs), their performance on complex reasoning problems requiring multi-step thinking and combining various skills is still limited. To address this, we propose a novel framework HDFlow for complex reasoning with LLMs that combines fast and slow thinking modes in an adaptive manner. Our approach consists of two key components: 1) a new approach for slow, deliberate reasoning called Dynamic Workflow, which automatically decomposes complex problems into more manageable sub-tasks and dynamically designs a workflow to assemble specialized LLM or symbolic reasoning tools to solve sub-tasks; 2) Hybrid Thinking, a general framework that dynamically combines fast and slow thinking based on problem complexity. Finally, we propose an easy-to-scale method for automatically synthesizing a large-scale dataset of 27K challenging reasoning problems for complex reasoning and a hybrid thinking tuning method that trains smaller LLMs on this dataset to internalize the fast/slow hybrid reasoning strategies. Experiments on four reasoning benchmark datasets demonstrate that our slow thinking with dynamic workflows significantly outperforms Chain-of-Thought, and hybrid thinking achieves the highest accuracy while providing an effective balance between computational efficiency and performance. Fine-tuning using our hybrid thinking approach also significantly boosts the complex reasoning capabilities of open-source language models. The results showcase the promise of slow thinking, dynamic workflows, and hybrid thinking in expanding the frontier of complex problem-solving with LLMs\footnote{Code and data will be released at \url{this https URL}.}.</li>

</ul>

<h3>Title: Rejection Sampling IMLE: Designing Priors for Better Few-Shot Image Synthesis</h3>

<ul>

<li><strong>Authors: </strong>Chirag Vashist, Shichong Peng, Ke Li</a></li>

<li><strong>Subjects: </strong>cs.CV, cs.LG</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17439">https://arxiv.org/abs/2409.17439</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17439">https://arxiv.org/pdf/2409.17439</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17439]] Rejection Sampling IMLE: Designing Priors for Better Few-Shot Image Synthesis(https://arxiv.org/abs/2409.17439)</code><input type="text"></li>

<li><strong>Keywords: </strong>diffusion, generative</a></li>

<li><strong>Abstract: </strong>An emerging area of research aims to learn deep generative models with limited training data. Prior generative models like GANs and diffusion models require a lot of data to perform well, and their performance degrades when they are trained on only a small amount of data. A recent technique called Implicit Maximum Likelihood Estimation (IMLE) has been adapted to the few-shot setting, achieving state-of-the-art performance. However, current IMLE-based approaches encounter challenges due to inadequate correspondence between the latent codes selected for training and those drawn during inference. This results in suboptimal test-time performance. We theoretically show a way to address this issue and propose RS-IMLE, a novel approach that changes the prior distribution used for training. This leads to substantially higher quality image generation compared to existing GAN and IMLE-based methods, as validated by comprehensive experiments conducted on nine few-shot image datasets.</li>

</ul>

<h3>Title: Enhancing Financial Sentiment Analysis with Expert-Designed Hint</h3>

<ul>

<li><strong>Authors: </strong>Chung-Chi Chen, Hiroya Takamura, Ichiro Kobayashi, Yusuke Miyao</a></li>

<li><strong>Subjects: </strong>cs.CL</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17448">https://arxiv.org/abs/2409.17448</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17448">https://arxiv.org/pdf/2409.17448</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17448]] Enhancing Financial Sentiment Analysis with Expert-Designed Hint(https://arxiv.org/abs/2409.17448)</code><input type="text"></li>

<li><strong>Keywords: </strong>large language model</a></li>

<li><strong>Abstract: </strong>This paper investigates the role of expert-designed hint in enhancing sentiment analysis on financial social media posts. We explore the capability of large language models (LLMs) to empathize with writer perspectives and analyze sentiments. Our findings reveal that expert-designed hint, i.e., pointing out the importance of numbers, significantly improve performances across various LLMs, particularly in cases requiring perspective-taking skills. Further analysis on tweets containing different types of numerical data demonstrates that the inclusion of expert-designed hint leads to notable improvements in sentiment analysis performance, especially for tweets with monetary-related numbers. Our findings contribute to the ongoing discussion on the applicability of Theory of Mind in NLP and open new avenues for improving sentiment analysis in financial domains through the strategic use of expert knowledge.</li>

</ul>

<h3>Title: AgMTR: Agent Mining Transformer for Few-shot Segmentation in Remote Sensing</h3>

<ul>

<li><strong>Authors: </strong>Hanbo Bi, Yingchao Feng, Yongqiang Mao, Jianning Pei, Wenhui Diao, Hongqi Wang, Xian Sun</a></li>

<li><strong>Subjects: </strong>cs.CV</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17453">https://arxiv.org/abs/2409.17453</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17453">https://arxiv.org/pdf/2409.17453</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17453]] AgMTR: Agent Mining Transformer for Few-shot Segmentation in Remote Sensing(https://arxiv.org/abs/2409.17453)</code><input type="text"></li>

<li><strong>Keywords: </strong>transformer, segmentation</a></li>

<li><strong>Abstract: </strong>Few-shot Segmentation (FSS) aims to segment the interested objects in the query image with just a handful of labeled samples (i.e., support images). Previous schemes would leverage the similarity between support-query pixel pairs to construct the pixel-level semantic correlation. However, in remote sensing scenarios with extreme intra-class variations and cluttered backgrounds, such pixel-level correlations may produce tremendous mismatches, resulting in semantic ambiguity between the query foreground (FG) and background (BG) pixels. To tackle this problem, we propose a novel Agent Mining Transformer (AgMTR), which adaptively mines a set of local-aware agents to construct agent-level semantic correlation. Compared with pixel-level semantics, the given agents are equipped with local-contextual information and possess a broader receptive field. At this point, different query pixels can selectively aggregate the fine-grained local semantics of different agents, thereby enhancing the semantic clarity between query FG and BG pixels. Concretely, the Agent Learning Encoder (ALE) is first proposed to erect the optimal transport plan that arranges different agents to aggregate support semantics under different local regions. Then, for further optimizing the agents, the Agent Aggregation Decoder (AAD) and the Semantic Alignment Decoder (SAD) are constructed to break through the limited support set for mining valuable class-specific semantics from unlabeled data sources and the query image itself, respectively. Extensive experiments on the remote sensing benchmark iSAID indicate that the proposed method achieves state-of-the-art performance. Surprisingly, our method remains quite competitive when extended to more common natural scenarios, i.e., PASCAL-5i and COCO-20i.</li>

</ul>

<h3>Title: Navigating the Shortcut Maze: A Comprehensive Analysis of Shortcut Learning in Text Classification by Language Models</h3>

<ul>

<li><strong>Authors: </strong>Yuqing Zhou, Ruixiang Tang, Ziyu Yao, Ziwei Zhu</a></li>

<li><strong>Subjects: </strong>cs.CL, cs.LG</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17455">https://arxiv.org/abs/2409.17455</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17455">https://arxiv.org/pdf/2409.17455</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17455]] Navigating the Shortcut Maze: A Comprehensive Analysis of Shortcut Learning in Text Classification by Language Models(https://arxiv.org/abs/2409.17455)</code><input type="text"></li>

<li><strong>Keywords: </strong>robust, large language model</a></li>

<li><strong>Abstract: </strong>Language models (LMs), despite their advances, often depend on spurious correlations, undermining their accuracy and generalizability. This study addresses the overlooked impact of subtler, more complex shortcuts that compromise model reliability beyond oversimplified shortcuts. We introduce a comprehensive benchmark that categorizes shortcuts into occurrence, style, and concept, aiming to explore the nuanced ways in which these shortcuts influence the performance of LMs. Through extensive experiments across traditional LMs, large language models, and state-of-the-art robust models, our research systematically investigates models' resilience and susceptibilities to sophisticated shortcuts. Our benchmark and code can be found at: this https URL.</li>

</ul>

<h3>Title: CadVLM: Bridging Language and Vision in the Generation of Parametric CAD Sketches</h3>

<ul>

<li><strong>Authors: </strong>Sifan Wu, Amir Khasahmadi, Mor Katz, Pradeep Kumar Jayaraman, Yewen Pu, Karl Willis, Bang Liu</a></li>

<li><strong>Subjects: </strong>cs.CV, cs.AI</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17457">https://arxiv.org/abs/2409.17457</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17457">https://arxiv.org/pdf/2409.17457</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17457]] CadVLM: Bridging Language and Vision in the Generation of Parametric CAD Sketches(https://arxiv.org/abs/2409.17457)</code><input type="text"></li>

<li><strong>Keywords: </strong>generative, large language model</a></li>

<li><strong>Abstract: </strong>Parametric Computer-Aided Design (CAD) is central to contemporary mechanical design. However, it encounters challenges in achieving precise parametric sketch modeling and lacks practical evaluation metrics suitable for mechanical design. We harness the capabilities of pre-trained foundation models, renowned for their successes in natural language processing and computer vision, to develop generative models specifically for CAD. These models are adept at understanding complex geometries and design reasoning, a crucial advancement in CAD technology. In this paper, we propose CadVLM, an end-to-end vision language model for CAD generation. Our approach involves adapting pre-trained foundation models to manipulate engineering sketches effectively, integrating both sketch primitive sequences and sketch images. Extensive experiments demonstrate superior performance on multiple CAD sketch generation tasks such as CAD autocompletion, CAD autoconstraint, and image conditional generation. To our knowledge, this is the first instance of a multimodal Large Language Model (LLM) being successfully applied to parametric CAD generation, representing a pioneering step in the field of computer-aided mechanical design.</li>

</ul>

<h3>Title: RED QUEEN: Safeguarding Large Language Models against Concealed Multi-Turn Jailbreaking</h3>

<ul>

<li><strong>Authors: </strong>Yifan Jiang, Kriti Aggarwal, Tanmay Laud, Kashif Munir, Jay Pujara, Subhabrata Mukherjee</a></li>

<li><strong>Subjects: </strong>cs.CR, cs.CL, cs.LG</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17458">https://arxiv.org/abs/2409.17458</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17458">https://arxiv.org/pdf/2409.17458</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17458]] RED QUEEN: Safeguarding Large Language Models against Concealed Multi-Turn Jailbreaking(https://arxiv.org/abs/2409.17458)</code><input type="text"></li>

<li><strong>Keywords: </strong>security, attack, large language model</a></li>

<li><strong>Abstract: </strong>The rapid progress of Large Language Models (LLMs) has opened up new opportunities across various domains and applications; yet it also presents challenges related to potential misuse. To mitigate such risks, red teaming has been employed as a proactive security measure to probe language models for harmful outputs via jailbreak attacks. However, current jailbreak attack approaches are single-turn with explicit malicious queries that do not fully capture the complexity of real-world interactions. In reality, users can engage in multi-turn interactions with LLM-based chat assistants, allowing them to conceal their true intentions in a more covert manner. To bridge this gap, we, first, propose a new jailbreak approach, RED QUEEN ATTACK. This method constructs a multi-turn scenario, concealing the malicious intent under the guise of preventing harm. We craft 40 scenarios that vary in turns and select 14 harmful categories to generate 56k multi-turn attack data points. We conduct comprehensive experiments on the RED QUEEN ATTACK with four representative LLM families of different sizes. Our experiments reveal that all LLMs are vulnerable to RED QUEEN ATTACK, reaching 87.62% attack success rate on GPT-4o and 75.4% on Llama3-70B. Further analysis reveals that larger models are more susceptible to the RED QUEEN ATTACK, with multi-turn structures and concealment strategies contributing to its success. To prioritize safety, we introduce a straightforward mitigation strategy called RED QUEEN GUARD, which aligns LLMs to effectively counter adversarial attacks. This approach reduces the attack success rate to below 1% while maintaining the model's performance across standard benchmarks. Full implementation and dataset are publicly accessible at this https URL.</li>

</ul>

<h3>Title: Autoregressive Multi-trait Essay Scoring via Reinforcement Learning with Scoring-aware Multiple Rewards</h3>

<ul>

<li><strong>Authors: </strong>Heejin Do, Sangwon Ryu, Gary Geunbae Lee</a></li>

<li><strong>Subjects: </strong>cs.CL, cs.AI</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17472">https://arxiv.org/abs/2409.17472</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17472">https://arxiv.org/pdf/2409.17472</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17472]] Autoregressive Multi-trait Essay Scoring via Reinforcement Learning with Scoring-aware Multiple Rewards(https://arxiv.org/abs/2409.17472)</code><input type="text"></li>

<li><strong>Keywords: </strong>robust</a></li>

<li><strong>Abstract: </strong>Recent advances in automated essay scoring (AES) have shifted towards evaluating multiple traits to provide enriched feedback. Like typical AES systems, multi-trait AES employs the quadratic weighted kappa (QWK) to measure agreement with human raters, aligning closely with the rating schema; however, its non-differentiable nature prevents its direct use in neural network training. In this paper, we propose Scoring-aware Multi-reward Reinforcement Learning (SaMRL), which integrates actual evaluation schemes into the training process by designing QWK-based rewards with a mean-squared error penalty for multi-trait AES. Existing reinforcement learning (RL) applications in AES are limited to classification models despite associated performance degradation, as RL requires probability distributions; instead, we adopt an autoregressive score generation framework to leverage token generation probabilities for robust multi-trait score predictions. Empirical analyses demonstrate that SaMRL facilitates model training, notably enhancing scoring of previously inferior prompts.</li>

</ul>

<h3>Title: Learning Quantized Adaptive Conditions for Diffusion Models</h3>

<ul>

<li><strong>Authors: </strong>Yuchen Liang, Yuchuan Tian, Lei Yu, Huao Tang, Jie Hu, Xiangzhong Fang, Hanting Chen</a></li>

<li><strong>Subjects: </strong>cs.CV</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17487">https://arxiv.org/abs/2409.17487</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17487">https://arxiv.org/pdf/2409.17487</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17487]] Learning Quantized Adaptive Conditions for Diffusion Models(https://arxiv.org/abs/2409.17487)</code><input type="text"></li>

<li><strong>Keywords: </strong>diffusion</a></li>

<li><strong>Abstract: </strong>The curvature of ODE trajectories in diffusion models hinders their ability to generate high-quality images in a few number of function evaluations (NFE). In this paper, we propose a novel and effective approach to reduce trajectory curvature by utilizing adaptive conditions. By employing a extremely light-weight quantized encoder, our method incurs only an additional 1% of training parameters, eliminates the need for extra regularization terms, yet achieves significantly better sample quality. Our approach accelerates ODE sampling while preserving the downstream task image editing capabilities of SDE techniques. Extensive experiments verify that our method can generate high quality results under extremely limited sampling costs. With only 6 NFE, we achieve 5.14 FID on CIFAR-10, 6.91 FID on FFHQ 64x64 and 3.10 FID on AFHQv2.</li>

</ul>

<h3>Title: Does Worst-Performing Agent Lead the Pack? Analyzing Agent Dynamics in Unified Distributed SGD</h3>

<ul>

<li><strong>Authors: </strong>Jie Hu, Yi-Ting Ma, Do Young Eun</a></li>

<li><strong>Subjects: </strong>cs.LG, math.OC, stat.ML</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17499">https://arxiv.org/abs/2409.17499</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17499">https://arxiv.org/pdf/2409.17499</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17499]] Does Worst-Performing Agent Lead the Pack? Analyzing Agent Dynamics in Unified Distributed SGD(https://arxiv.org/abs/2409.17499)</code><input type="text"></li>

<li><strong>Keywords: </strong>privacy, federate</a></li>

<li><strong>Abstract: </strong>Distributed learning is essential to train machine learning algorithms across heterogeneous agents while maintaining data privacy. We conduct an asymptotic analysis of Unified Distributed SGD (UD-SGD), exploring a variety of communication patterns, including decentralized SGD and local SGD within Federated Learning (FL), as well as the increasing communication interval in the FL setting. In this study, we assess how different sampling strategies, such as i.i.d. sampling, shuffling, and Markovian sampling, affect the convergence speed of UD-SGD by considering the impact of agent dynamics on the limiting covariance matrix as described in the Central Limit Theorem (CLT). Our findings not only support existing theories on linear speedup and asymptotic network independence, but also theoretically and empirically show how efficient sampling strategies employed by individual agents contribute to overall convergence in UD-SGD. Simulations reveal that a few agents using highly efficient sampling can achieve or surpass the performance of the majority employing moderately improved strategies, providing new insights beyond traditional analyses focusing on the worst-performing agent.</li>

</ul>

<h3>Title: HaloScope: Harnessing Unlabeled LLM Generations for Hallucination Detection</h3>

<ul>

<li><strong>Authors: </strong>Xuefeng Du, Chaowei Xiao, Yixuan Li</a></li>

<li><strong>Subjects: </strong>cs.LG, cs.CL</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17504">https://arxiv.org/abs/2409.17504</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17504">https://arxiv.org/pdf/2409.17504</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17504]] HaloScope: Harnessing Unlabeled LLM Generations for Hallucination Detection(https://arxiv.org/abs/2409.17504)</code><input type="text"></li>

<li><strong>Keywords: </strong>large language model</a></li>

<li><strong>Abstract: </strong>The surge in applications of large language models (LLMs) has prompted concerns about the generation of misleading or fabricated information, known as hallucinations. Therefore, detecting hallucinations has become critical to maintaining trust in LLM-generated content. A primary challenge in learning a truthfulness classifier is the lack of a large amount of labeled truthful and hallucinated data. To address the challenge, we introduce HaloScope, a novel learning framework that leverages the unlabeled LLM generations in the wild for hallucination detection. Such unlabeled data arises freely upon deploying LLMs in the open world, and consists of both truthful and hallucinated information. To harness the unlabeled data, we present an automated membership estimation score for distinguishing between truthful and untruthful generations within unlabeled mixture data, thereby enabling the training of a binary truthfulness classifier on top. Importantly, our framework does not require extra data collection and human annotations, offering strong flexibility and practicality for real-world applications. Extensive experiments show that HaloScope can achieve superior hallucination detection performance, outperforming the competitive rivals by a significant margin. Code is available at this https URL.</li>

</ul>

<h3>Title: Uni-Med: A Unified Medical Generalist Foundation Model For Multi-Task Learning Via Connector-MoE</h3>

<ul>

<li><strong>Authors: </strong>Xun Zhu, Ying Hu, Fanbin Mo, Miao Li, Ji Wu</a></li>

<li><strong>Subjects: </strong>cs.CV, cs.AI, cs.LG</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17508">https://arxiv.org/abs/2409.17508</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17508">https://arxiv.org/pdf/2409.17508</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17508]] Uni-Med: A Unified Medical Generalist Foundation Model For Multi-Task Learning Via Connector-MoE(https://arxiv.org/abs/2409.17508)</code><input type="text"></li>

<li><strong>Keywords: </strong>extraction, large language model</a></li>

<li><strong>Abstract: </strong>Multi-modal large language models (MLLMs) have shown impressive capabilities as a general-purpose interface for various visual and linguistic tasks. However, building a unified MLLM for multi-task learning in the medical field remains a thorny challenge. To mitigate the tug-of-war problem of multi-modal multi-task optimization, recent advances primarily focus on improving the LLM components, while neglecting the connector that bridges the gap between modalities. In this paper, we introduce Uni-Med, a novel medical generalist foundation model which consists of a universal visual feature extraction module, a connector mixture-of-experts (CMoE) module, and an LLM. Benefiting from the proposed CMoE that leverages a well-designed router with a mixture of projection experts at the connector, Uni-Med achieves efficient solution to the tug-of-war problem and can perform six different medical tasks including question answering, visual question answering, report generation, referring expression comprehension, referring expression generation and image classification. To the best of our knowledge, Uni-Med is the first effort to tackle multi-task interference at the connector. Extensive ablation experiments validate the effectiveness of introducing CMoE under any configuration, with up to an average 8% performance gains. We further provide interpretation analysis of the tug-of-war problem from the perspective of gradient optimization and parameter statistics. Compared to previous state-of-the-art medical MLLMs, Uni-Med achieves competitive or superior evaluation metrics on diverse tasks. Code, data and model will be soon available at GitHub.</li>

</ul>

<h3>Title: BioZero: An Efficient and Privacy-Preserving Decentralized Biometric Authentication Protocol on Open Blockchain</h3>

<ul>

<li><strong>Authors: </strong>Junhao Lai, Taotao Wang, Shengli Zhang, Qing Yang, Soung Chang Liew</a></li>

<li><strong>Subjects: </strong>cs.CR</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17509">https://arxiv.org/abs/2409.17509</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17509">https://arxiv.org/pdf/2409.17509</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17509]] BioZero: An Efficient and Privacy-Preserving Decentralized Biometric Authentication Protocol on Open Blockchain(https://arxiv.org/abs/2409.17509)</code><input type="text"></li>

<li><strong>Keywords: </strong>secure, security, privacy, protect, attack, biometric</a></li>

<li><strong>Abstract: </strong>Digital identity plays a vital role in enabling secure access to resources and services in the digital world. Traditional identity authentication methods, such as password-based and biometric authentications, have limitations in terms of security, privacy, and scalability. Decentralized authentication approaches leveraging blockchain technology have emerged as a promising solution. However, existing decentralized authentication methods often rely on indirect identity verification (e.g. using passwords or digital signatures as authentication credentials) and face challenges such as Sybil attacks. In this paper, we propose BioZero, an efficient and privacy-preserving decentralized biometric authentication protocol that can be implemented on open blockchain. BioZero leverages Pedersen commitment and homomorphic computation to protect user biometric privacy while enabling efficient verification. We enhance the protocol with non-interactive homomorphic computation and employ zero-knowledge proofs for secure on-chain verification. The unique aspect of BioZero is that it is fully decentralized and can be executed by blockchain smart contracts in a very efficient way. We analyze the security of BioZero and validate its performance through a prototype implementation. The results demonstrate the effectiveness, efficiency, and security of BioZero in decentralized authentication scenarios. Our work contributes to the advancement of decentralized identity authentication using biometrics.</li>

</ul>

<h3>Title: Comparing Unidirectional, Bidirectional, and Word2vec Models for Discovering Vulnerabilities in Compiled Lifted Code</h3>

<ul>

<li><strong>Authors: </strong>Gary A. McCully, John D. Hastings, Shengjie Xu, Adam Fortier</a></li>

<li><strong>Subjects: </strong>cs.CR, cs.CL, cs.LG, cs.SE</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17513">https://arxiv.org/abs/2409.17513</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17513">https://arxiv.org/pdf/2409.17513</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17513]] Comparing Unidirectional, Bidirectional, and Word2vec Models for Discovering Vulnerabilities in Compiled Lifted Code(https://arxiv.org/abs/2409.17513)</code><input type="text"></li>

<li><strong>Keywords: </strong>security, defense, transformer</a></li>

<li><strong>Abstract: </strong>Ransomware and other forms of malware cause significant financial and operational damage to organizations by exploiting long-standing and often difficult-to-detect software vulnerabilities. To detect vulnerabilities such as buffer overflows in compiled code, this research investigates the application of unidirectional transformer-based embeddings, specifically GPT-2. Using a dataset of LLVM functions, we trained a GPT-2 model to generate embeddings, which were subsequently used to build LSTM neural networks to differentiate between vulnerable and non-vulnerable code. Our study reveals that embeddings from the GPT-2 model significantly outperform those from bidirectional models of BERT and RoBERTa, achieving an accuracy of 92.5% and an F1-score of 89.7%. LSTM neural networks were developed with both frozen and unfrozen embedding model layers. The model with the highest performance was achieved when the embedding layers were unfrozen. Further, the research finds that, in exploring the impact of different optimizers within this domain, the SGD optimizer demonstrates superior performance over Adam. Overall, these findings reveal important insights into the potential of unidirectional transformer-based approaches in enhancing cybersecurity defenses.</li>

</ul>

<h3>Title: Dataset Distillation-based Hybrid Federated Learning on Non-IID Data</h3>

<ul>

<li><strong>Authors: </strong>Xiufang Shi, Wei Zhang, Mincheng Wu, Guangyi Liu, Zhenyu Wen, Shibo He, Tejal Shah, Rajiv Ranjan</a></li>

<li><strong>Subjects: </strong>cs.LG, cs.AI</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17517">https://arxiv.org/abs/2409.17517</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17517">https://arxiv.org/pdf/2409.17517</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17517]] Dataset Distillation-based Hybrid Federated Learning on Non-IID Data(https://arxiv.org/abs/2409.17517)</code><input type="text"></li>

<li><strong>Keywords: </strong>federate</a></li>

<li><strong>Abstract: </strong>In federated learning, the heterogeneity of client data has a great impact on the performance of model training. Many heterogeneity issues in this process are raised by non-independently and identically distributed (Non-IID) data. This study focuses on the issue of label distribution skew. To address it, we propose a hybrid federated learning framework called HFLDD, which integrates dataset distillation to generate approximately independent and equally distributed (IID) data, thereby improving the performance of model training. Particularly, we partition the clients into heterogeneous clusters, where the data labels among different clients within a cluster are unbalanced while the data labels among different clusters are balanced. The cluster headers collect distilled data from the corresponding cluster members, and conduct model training in collaboration with the server. This training process is like traditional federated learning on IID data, and hence effectively alleviates the impact of Non-IID data on model training. Furthermore, we compare our proposed method with typical baseline methods on public datasets. Experimental results demonstrate that when the data labels are severely imbalanced, the proposed HFLDD outperforms the baseline methods in terms of both test accuracy and communication cost.</li>

</ul>

<h3>Title: Multi-Designated Detector Watermarking for Language Models</h3>

<ul>

<li><strong>Authors: </strong>Zhengan Huang, Gongxian Zeng, Xin Mu, Yu Wang, Yue Yu</a></li>

<li><strong>Subjects: </strong>cs.CR, cs.AI</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17518">https://arxiv.org/abs/2409.17518</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17518">https://arxiv.org/pdf/2409.17518</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17518]] Multi-Designated Detector Watermarking for Language Models(https://arxiv.org/abs/2409.17518)</code><input type="text"></li>

<li><strong>Keywords: </strong>security, watermark, large language model</a></li>

<li><strong>Abstract: </strong>In this paper, we initiate the study of \emph{multi-designated detector watermarking (MDDW)} for large language models (LLMs). This technique allows model providers to generate watermarked outputs from LLMs with two key properties: (i) only specific, possibly multiple, designated detectors can identify the watermarks, and (ii) there is no perceptible degradation in the output quality for ordinary users. We formalize the security definitions for MDDW and present a framework for constructing MDDW for any LLM using multi-designated verifier signatures (MDVS). Recognizing the significant economic value of LLM outputs, we introduce claimability as an optional security feature for MDDW, enabling model providers to assert ownership of LLM outputs within designated-detector settings. To support claimable MDDW, we propose a generic transformation converting any MDVS to a claimable MDVS. Our implementation of the MDDW scheme highlights its advanced functionalities and flexibility over existing methods, with satisfactory performance metrics.</li>

</ul>

<h3>Title: EAGLE: Egocentric AGgregated Language-video Engine</h3>

<ul>

<li><strong>Authors: </strong>Jing Bi, Yunlong Tang, Luchuan Song, Ali Vosoughi, Nguyen Nguyen, Chenliang Xu</a></li>

<li><strong>Subjects: </strong>cs.CV, cs.AI</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17523">https://arxiv.org/abs/2409.17523</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17523">https://arxiv.org/pdf/2409.17523</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17523]] EAGLE: Egocentric AGgregated Language-video Engine(https://arxiv.org/abs/2409.17523)</code><input type="text"></li>

<li><strong>Keywords: </strong>large language model</a></li>

<li><strong>Abstract: </strong>The rapid evolution of egocentric video analysis brings new insights into understanding human activities and intentions from a first-person perspective. Despite this progress, the fragmentation in tasks like action recognition, procedure learning, and moment retrieval, \etc, coupled with inconsistent annotations and isolated model development, hinders a holistic interpretation of video content. In response, we introduce the EAGLE (Egocentric AGgregated Language-video Engine) model and the EAGLE-400K dataset to provide a unified framework that integrates various egocentric video understanding tasks. EAGLE-400K, the \textit{first} large-scale instruction-tuning dataset tailored for egocentric video, features 400K diverse samples to enhance a broad spectrum of tasks from activity recognition to procedure knowledge learning. Moreover, EAGLE, a strong video multimodal large language model (MLLM), is designed to effectively capture both spatial and temporal information. In addition, we propose a set of evaluation metrics designed to facilitate a thorough assessment of MLLM for egocentric video understanding. Our extensive experiments demonstrate EAGLE's superior performance over existing models, highlighting its ability to balance task-specific understanding with holistic video interpretation. With EAGLE, we aim to pave the way for research opportunities and practical applications in real-world scenarios.</li>

</ul>

<h3>Title: JoyType: A Robust Design for Multilingual Visual Text Creation</h3>

<ul>

<li><strong>Authors: </strong>Chao Li, Chen Jiang, Xiaolong Liu, Jun Zhao, Guoxin Wang</a></li>

<li><strong>Subjects: </strong>cs.CV</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17524">https://arxiv.org/abs/2409.17524</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17524">https://arxiv.org/pdf/2409.17524</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17524]] JoyType: A Robust Design for Multilingual Visual Text Creation(https://arxiv.org/abs/2409.17524)</code><input type="text"></li>

<li><strong>Keywords: </strong>robust, diffusion</a></li>

<li><strong>Abstract: </strong>Generating images with accurately represented text, especially in non-Latin languages, poses a significant challenge for diffusion models. Existing approaches, such as the integration of hint condition diagrams via auxiliary networks (e.g., ControlNet), have made strides towards addressing this issue. However, diffusion models often fall short in tasks requiring controlled text generation, such as specifying particular fonts or producing text in small fonts. In this paper, we introduce a novel approach for multilingual visual text creation, named JoyType, designed to maintain the font style of text during the image generation process. Our methodology begins with assembling a training dataset, JoyType-1M, comprising 1 million pairs of data. Each pair includes an image, its description, and glyph instructions corresponding to the font style within the image. We then developed a text control network, Font ControlNet, tasked with extracting font style information to steer the image generation. To further enhance our model's ability to maintain font style, notably in generating small-font text, we incorporated a multi-layer OCR-aware loss into the diffusion process. This enhancement allows JoyType to direct text rendering using low-level descriptors. Our evaluations, based on both visual and accuracy metrics, demonstrate that JoyType significantly outperforms existing state-of-the-art methods. Additionally, JoyType can function as a plugin, facilitating the creation of varied image styles in conjunction with other stable diffusion models on HuggingFace and CivitAI. Our project is open-sourced on this https URL.</li>

</ul>

<h3>Title: Drone Stereo Vision for Radiata Pine Branch Detection and Distance Measurement: Integrating SGBM and Segmentation Models</h3>

<ul>

<li><strong>Authors: </strong>Yida Lin, Bing Xue, Mengjie Zhang, Sam Schofield, Richard Green</a></li>

<li><strong>Subjects: </strong>cs.CV, cs.AI</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17526">https://arxiv.org/abs/2409.17526</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17526">https://arxiv.org/pdf/2409.17526</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17526]] Drone Stereo Vision for Radiata Pine Branch Detection and Distance Measurement: Integrating SGBM and Segmentation Models(https://arxiv.org/abs/2409.17526)</code><input type="text"></li>

<li><strong>Keywords: </strong>segmentation</a></li>

<li><strong>Abstract: </strong>Manual pruning of radiata pine trees presents significant safety risks due to their substantial height and the challenging terrains in which they thrive. To address these risks, this research proposes the development of a drone-based pruning system equipped with specialized pruning tools and a stereo vision camera, enabling precise detection and trimming of branches. Deep learning algorithms, including YOLO and Mask R-CNN, are employed to ensure accurate branch detection, while the Semi-Global Matching algorithm is integrated to provide reliable distance estimation. The synergy between these techniques facilitates the precise identification of branch locations and enables efficient, targeted pruning. Experimental results demonstrate that the combined implementation of YOLO and SGBM enables the drone to accurately detect branches and measure their distances from the drone. This research not only improves the safety and efficiency of pruning operations but also makes a significant contribution to the advancement of drone technology in the automation of agricultural and forestry practices, laying a foundational framework for further innovations in environmental management.</li>

</ul>

<h3>Title: Data Proportion Detection for Optimized Data Management for Large Language Models</h3>

<ul>

<li><strong>Authors: </strong>Hao Liang, Keshi Zhao, Yajie Yang, Bin Cui, Guosheng Dong, Zenan Zhou, Wentao Zhang</a></li>

<li><strong>Subjects: </strong>cs.CL</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17527">https://arxiv.org/abs/2409.17527</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17527">https://arxiv.org/pdf/2409.17527</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17527]] Data Proportion Detection for Optimized Data Management for Large Language Models(https://arxiv.org/abs/2409.17527)</code><input type="text"></li>

<li><strong>Keywords: </strong>large language model</a></li>

<li><strong>Abstract: </strong>Large language models (LLMs) have demonstrated exceptional performance across a wide range of tasks and domains, with data preparation playing a critical role in achieving these results. Pre-training data typically combines information from multiple domains. To maximize performance when integrating data from various domains, determining the optimal data proportion is essential. However, state-of-the-art (SOTA) LLMs rarely disclose details about their pre-training data, making it difficult for researchers to identify ideal data proportions. In this paper, we introduce a new topic, \textit{data proportion detection}, which enables the automatic estimation of pre-training data proportions by analyzing the generated outputs of LLMs. We provide rigorous theoretical proofs, practical algorithms, and preliminary experimental results for data proportion detection. Based on these findings, we offer valuable insights into the challenges and future directions for effective data proportion detection and data management.</li>

</ul>

<h3>Title: SimVG: A Simple Framework for Visual Grounding with Decoupled Multi-modal Fusion</h3>

<ul>

<li><strong>Authors: </strong>Ming Dai, Lingfeng Yang, Yihao Xu, Zhenhua Feng, Wankou Yang</a></li>

<li><strong>Subjects: </strong>cs.CV, cs.AI</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17531">https://arxiv.org/abs/2409.17531</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17531">https://arxiv.org/pdf/2409.17531</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17531]] SimVG: A Simple Framework for Visual Grounding with Decoupled Multi-modal Fusion(https://arxiv.org/abs/2409.17531)</code><input type="text"></li>

<li><strong>Keywords: </strong>robust, transformer</a></li>

<li><strong>Abstract: </strong>Visual grounding is a common vision task that involves grounding descriptive sentences to the corresponding regions of an image. Most existing methods use independent image-text encoding and apply complex hand-crafted modules or encoder-decoder architectures for modal interaction and query reasoning. However, their performance significantly drops when dealing with complex textual expressions. This is because the former paradigm only utilizes limited downstream data to fit the multi-modal feature fusion. Therefore, it is only effective when the textual expressions are relatively simple. In contrast, given the wide diversity of textual expressions and the uniqueness of downstream training data, the existing fusion module, which extracts multimodal content from a visual-linguistic context, has not been fully investigated. In this paper, we present a simple yet robust transformer-based framework, SimVG, for visual grounding. Specifically, we decouple visual-linguistic feature fusion from downstream tasks by leveraging existing multimodal pre-trained models and incorporating additional object tokens to facilitate deep integration of downstream and pre-training tasks. Furthermore, we design a dynamic weight-balance distillation method in the multi-branch synchronous learning process to enhance the representation capability of the simpler branch. This branch only consists of a lightweight MLP, which simplifies the structure and improves reasoning speed. Experiments on six widely used VG datasets, i.e., RefCOCO/+/g, ReferIt, Flickr30K, and GRefCOCO, demonstrate the superiority of SimVG. Finally, the proposed method not only achieves improvements in efficiency and convergence speed but also attains new state-of-the-art performance on these benchmarks. Codes and models will be available at \url{this https URL}.</li>

</ul>

<h3>Title: Privacy-Preserving Redaction of Diagnosis Data through Source Code Analysis</h3>

<ul>

<li><strong>Authors: </strong>Lixi Zhou, Lei Yu, Jia Zou, Hong Min</a></li>

<li><strong>Subjects: </strong>cs.CR</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17535">https://arxiv.org/abs/2409.17535</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17535">https://arxiv.org/pdf/2409.17535</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17535]] Privacy-Preserving Redaction of Diagnosis Data through Source Code Analysis(https://arxiv.org/abs/2409.17535)</code><input type="text"></li>

<li><strong>Keywords: </strong>privacy, protect</a></li>

<li><strong>Abstract: </strong>Protecting sensitive information in diagnostic data such as logs, is a critical concern in the industrial software diagnosis and debugging process. While there are many tools developed to automatically redact the logs for identifying and removing sensitive information, they have severe limitations which can cause either over redaction and loss of critical diagnostic information (false positives), or disclosure of sensitive information (false negatives), or both. To address the problem, in this paper, we argue for a source code analysis approach for log redaction. To identify a log message containing sensitive information, our method locates the corresponding log statement in the source code with logger code augmentation, and checks if the log statement outputs data from sensitive sources by using the data flow graph built from the source code. Appropriate redaction rules are further applied depending on the sensitiveness of the data sources to preserve the privacy information in the logs. We conducted experimental evaluation and comparison with other popular baselines. The results demonstrate that our approach can significantly improve the detection precision of the sensitive information and reduce both false positives and negatives.</li>

</ul>

<h3>Title: On the Implicit Relation Between Low-Rank Adaptation and Differential Privacy</h3>

<ul>

<li><strong>Authors: </strong>Saber Malekmohammadi, Golnoosh Farnadi</a></li>

<li><strong>Subjects: </strong>cs.LG, cs.AI, cs.CL</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17538">https://arxiv.org/abs/2409.17538</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17538">https://arxiv.org/pdf/2409.17538</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17538]] On the Implicit Relation Between Low-Rank Adaptation and Differential Privacy(https://arxiv.org/abs/2409.17538)</code><input type="text"></li>

<li><strong>Keywords: </strong>privacy, transformer</a></li>

<li><strong>Abstract: </strong>A significant approach in natural language processing involves large-scale pre-training on general domain data followed by adaptation to specific tasks or domains. As models grow in size, full fine-tuning all parameters becomes increasingly impractical. To address this, some methods for low-rank task adaptation of language models have been proposed, e.g. LoRA and FLoRA. These methods keep the pre-trained model weights fixed and incorporate trainable low-rank decomposition matrices into some layers of the transformer architecture, called adapters. This approach significantly reduces the number of trainable parameters required for downstream tasks compared to full fine-tuning all parameters. In this work, we look at low-rank adaptation from the lens of data privacy. We show theoretically that the low-rank adaptation used in LoRA and FLoRA is equivalent to injecting some random noise into the batch gradients w.r.t the adapter parameters coming from their full fine-tuning, and we quantify the variance of the injected noise. By establishing a Berry-Esseen type bound on the total variation distance between the noise distribution and a Gaussian distribution with the same variance, we show that the dynamics of LoRA and FLoRA are very close to differentially private full fine-tuning the adapters, which suggests that low-rank adaptation implicitly provides privacy w.r.t the fine-tuning data. Finally, using Johnson-Lindenstrauss lemma, we show that when augmented with gradient clipping, low-rank adaptation is almost equivalent to differentially private full fine-tuning adapters with a fixed noise scale.</li>

</ul>

<h3>Title: Logic-of-Thought: Injecting Logic into Contexts for Full Reasoning in Large Language Models</h3>

<ul>

<li><strong>Authors: </strong>Tongxuan Liu, Wenjiang Xu, Weizhe Huang, Xingyu Wang, Jiaxing Wang, Hailong Yang, Jing Li</a></li>

<li><strong>Subjects: </strong>cs.CL</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17539">https://arxiv.org/abs/2409.17539</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17539">https://arxiv.org/pdf/2409.17539</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17539]] Logic-of-Thought: Injecting Logic into Contexts for Full Reasoning in Large Language Models(https://arxiv.org/abs/2409.17539)</code><input type="text"></li>

<li><strong>Keywords: </strong>extraction, large language model</a></li>

<li><strong>Abstract: </strong>Large Language Models (LLMs) have demonstrated remarkable capabilities across various tasks but their performance in complex logical reasoning tasks remains unsatisfactory. Although some prompting methods, such as Chain-of-Thought, can improve the reasoning ability of LLMs to some extent, they suffer from an unfaithful issue where derived conclusions may not align with the generated reasoning chain. To address this issue, some studies employ the approach of propositional logic to further enhance logical reasoning abilities of LLMs. However, the potential omissions in the extraction of logical expressions in these methods can cause information loss in the logical reasoning process, thereby generating incorrect results. To this end, we propose Logic-of-Thought (LoT) prompting which employs propositional logic to generate expanded logical information from input context, and utilizes the generated logical information as an additional augmentation to the input prompts, thereby enhancing the capability of logical reasoning. The LoT is orthogonal to existing prompting methods and can be seamlessly integrated with them. Extensive experiments demonstrate that LoT boosts the performance of various prompting methods with a striking margin across five logical reasoning tasks. In particular, the LoT enhances Chain-of-Thought's performance on the ReClor dataset by +4.35%; moreover, it improves Chain-of-Thought with Self-Consistency's performance on LogiQA by +5%; additionally, it boosts performance of Tree-of-Thoughts on ProofWriter dataset by +8%.</li>

</ul>

<h3>Title: A Simple but Strong Baseline for Sounding Video Generation: Effective Adaptation of Audio and Video Diffusion Models for Joint Generation</h3>

<ul>

<li><strong>Authors: </strong>Masato Ishii, Akio Hayakawa, Takashi Shibuya, Yuki Mitsufuji</a></li>

<li><strong>Subjects: </strong>cs.LG, cs.MM, cs.SD, eess.AS</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17550">https://arxiv.org/abs/2409.17550</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17550">https://arxiv.org/pdf/2409.17550</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17550]] A Simple but Strong Baseline for Sounding Video Generation: Effective Adaptation of Audio and Video Diffusion Models for Joint Generation(https://arxiv.org/abs/2409.17550)</code><input type="text"></li>

<li><strong>Keywords: </strong>diffusion</a></li>

<li><strong>Abstract: </strong>In this work, we build a simple but strong baseline for sounding video generation. Given base diffusion models for audio and video, we integrate them with additional modules into a single model and train it to make the model jointly generate audio and video. To enhance alignment between audio-video pairs, we introduce two novel mechanisms in our model. The first one is timestep adjustment, which provides different timestep information to each base model. It is designed to align how samples are generated along with timesteps across modalities. The second one is a new design of the additional modules, termed Cross-Modal Conditioning as Positional Encoding (CMC-PE). In CMC-PE, cross-modal information is embedded as if it represents temporal position information, and the embeddings are fed into the model like positional encoding. Compared with the popular cross-attention mechanism, CMC-PE provides a better inductive bias for temporal alignment in the generated data. Experimental results validate the effectiveness of the two newly introduced mechanisms and also demonstrate that our method outperforms existing methods.</li>

</ul>

<h3>Title: Dynamic Subframe Splitting and Spatio-Temporal Motion Entangled Sparse Attention for RGB-E Tracking</h3>

<ul>

<li><strong>Authors: </strong>Pengcheng Shao, Tianyang Xu, Xuefeng Zhu, Xiaojun Wu, Josef Kittler</a></li>

<li><strong>Subjects: </strong>cs.CV</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17560">https://arxiv.org/abs/2409.17560</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17560">https://arxiv.org/pdf/2409.17560</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17560]] Dynamic Subframe Splitting and Spatio-Temporal Motion Entangled Sparse Attention for RGB-E Tracking(https://arxiv.org/abs/2409.17560)</code><input type="text"></li>

<li><strong>Keywords: </strong>transformer</a></li>

<li><strong>Abstract: </strong>Event-based bionic camera asynchronously captures dynamic scenes with high temporal resolution and high dynamic range, offering potential for the integration of events and RGB under conditions of illumination degradation and fast motion. Existing RGB-E tracking methods model event characteristics utilising attention mechanism of Transformer before integrating both modalities. Nevertheless, these methods involve aggregating the event stream into a single event frame, lacking the utilisation of the temporal information inherent in the event this http URL, the traditional attention mechanism is well-suited for dense semantic features, while the attention mechanism for sparse event features require revolution. In this paper, we propose a dynamic event subframe splitting strategy to split the event stream into more fine-grained event clusters, aiming to capture spatio-temporal features that contain motion cues. Based on this, we design an event-based sparse attention mechanism to enhance the interaction of event features in temporal and spatial dimensions. The experimental results indicate that our method outperforms existing state-of-the-art methods on the FE240 and COESOT datasets, providing an effective processing manner for the event data.</li>

</ul>

<h3>Title: General Compression Framework for Efficient Transformer Object Tracking</h3>

<ul>

<li><strong>Authors: </strong>Lingyi Hong, Jinglun Li, Xinyu Zhou, Shilin Yan, Pinxue Guo, Kaixun Jiang, Zhaoyu Chen, Shuyong Gao, Wei Zhang, Hong Lu, Wenqiang Zhang</a></li>

<li><strong>Subjects: </strong>cs.CV</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17564">https://arxiv.org/abs/2409.17564</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17564">https://arxiv.org/pdf/2409.17564</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17564]] General Compression Framework for Efficient Transformer Object Tracking(https://arxiv.org/abs/2409.17564)</code><input type="text"></li>

<li><strong>Keywords: </strong>transformer</a></li>

<li><strong>Abstract: </strong>Transformer-based trackers have established a dominant role in the field of visual object tracking. While these trackers exhibit promising performance, their deployment on resource-constrained devices remains challenging due to inefficiencies. To improve the inference efficiency and reduce the computation cost, prior approaches have aimed to either design lightweight trackers or distill knowledge from larger teacher models into more compact student trackers. However, these solutions often sacrifice accuracy for speed. Thus, we propose a general model compression framework for efficient transformer object tracking, named CompressTracker, to reduce the size of a pre-trained tracking model into a lightweight tracker with minimal performance degradation. Our approach features a novel stage division strategy that segments the transformer layers of the teacher model into distinct stages, enabling the student model to emulate each corresponding teacher stage more effectively. Additionally, we also design a unique replacement training technique that involves randomly substituting specific stages in the student model with those from the teacher model, as opposed to training the student model in isolation. Replacement training enhances the student model's ability to replicate the teacher model's behavior. To further forcing student model to emulate teacher model, we incorporate prediction guidance and stage-wise feature mimicking to provide additional supervision during the teacher model's compression process. Our framework CompressTracker is structurally agnostic, making it compatible with any transformer architecture. We conduct a series of experiment to verify the effectiveness and generalizability of CompressTracker. Our CompressTracker-4 with 4 transformer layers, which is compressed from OSTrack, retains about 96% performance on LaSOT (66.1% AUC) while achieves 2.17x speed up.</li>

</ul>

<h3>Title: Pixel-Space Post-Training of Latent Diffusion Models</h3>

<ul>

<li><strong>Authors: </strong>Christina Zhang, Simran Motwani, Matthew Yu, Ji Hou, Felix Juefei-Xu, Sam Tsai, Peter Vajda, Zijian He, Jialiang Wang</a></li>

<li><strong>Subjects: </strong>cs.CV, cs.AI, cs.LG</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17565">https://arxiv.org/abs/2409.17565</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17565">https://arxiv.org/pdf/2409.17565</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17565]] Pixel-Space Post-Training of Latent Diffusion Models(https://arxiv.org/abs/2409.17565)</code><input type="text"></li>

<li><strong>Keywords: </strong>diffusion, transformer</a></li>

<li><strong>Abstract: </strong>Latent diffusion models (LDMs) have made significant advancements in the field of image generation in recent years. One major advantage of LDMs is their ability to operate in a compressed latent space, allowing for more efficient training and deployment. However, despite these advantages, challenges with LDMs still remain. For example, it has been observed that LDMs often generate high-frequency details and complex compositions imperfectly. We hypothesize that one reason for these flaws is due to the fact that all pre- and post-training of LDMs are done in latent space, which is typically $8 \times 8$ lower spatial-resolution than the output images. To address this issue, we propose adding pixel-space supervision in the post-training process to better preserve high-frequency details. Experimentally, we show that adding a pixel-space objective significantly improves both supervised quality fine-tuning and preference-based post-training by a large margin on a state-of-the-art DiT transformer and U-Net diffusion models in both visual quality and visual flaw metrics, while maintaining the same text alignment quality.</li>

</ul>

<h3>Title: Flexiffusion: Segment-wise Neural Architecture Search for Flexible Denoising Schedule</h3>

<ul>

<li><strong>Authors: </strong>Hongtao Huang, Xiaojun Chang, Lina Yao</a></li>

<li><strong>Subjects: </strong>cs.CV</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17566">https://arxiv.org/abs/2409.17566</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17566">https://arxiv.org/pdf/2409.17566</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17566]] Flexiffusion: Segment-wise Neural Architecture Search for Flexible Denoising Schedule(https://arxiv.org/abs/2409.17566)</code><input type="text"></li>

<li><strong>Keywords: </strong>diffusion, generative</a></li>

<li><strong>Abstract: </strong>Diffusion models are cutting-edge generative models adept at producing diverse, high-quality images. Despite their effectiveness, these models often require significant computational resources owing to their numerous sequential denoising steps and the significant inference cost of each step. Recently, Neural Architecture Search (NAS) techniques have been employed to automatically search for faster generation processes. However, NAS for diffusion is inherently time-consuming as it requires estimating thousands of diffusion models to search for the optimal one. In this paper, we introduce Flexiffusion, a novel training-free NAS paradigm designed to accelerate diffusion models by concurrently optimizing generation steps and network structures. Specifically, we partition the generation process into isometric step segments, each sequentially composed of a full step, multiple partial steps, and several null steps. The full step computes all network blocks, while the partial step involves part of the blocks, and the null step entails no computation. Flexiffusion autonomously explores flexible step combinations for each segment, substantially reducing search costs and enabling greater acceleration compared to the state-of-the-art (SOTA) method for diffusion models. Our searched models reported speedup factors of $2.6\times$ and $1.5\times$ for the original LDM-4-G and the SOTA, respectively. The factors for Stable Diffusion V1.5 and the SOTA are $5.1\times$ and $2.0\times$. We also verified the performance of Flexiffusion on multiple datasets, and positive experiment results indicate that Flexiffusion can effectively reduce redundancy in diffusion models.</li>

</ul>

<h3>Title: ID$^3$: Identity-Preserving-yet-Diversified Diffusion Models for Synthetic Face Recognition</h3>

<ul>

<li><strong>Authors: </strong>Shen Li, Jianqing Xu, Jiaying Wu, Miao Xiong, Ailin Deng, Jiazhen Ji, Yuge Huang, Wenjie Feng, Shouhong Ding, Bryan Hooi</a></li>

<li><strong>Subjects: </strong>cs.CV</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17576">https://arxiv.org/abs/2409.17576</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17576">https://arxiv.org/pdf/2409.17576</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17576]] ID$^3$: Identity-Preserving-yet-Diversified Diffusion Models for Synthetic Face Recognition(https://arxiv.org/abs/2409.17576)</code><input type="text"></li>

<li><strong>Keywords: </strong>privacy, diffusion</a></li>

<li><strong>Abstract: </strong>Synthetic face recognition (SFR) aims to generate synthetic face datasets that mimic the distribution of real face data, which allows for training face recognition models in a privacy-preserving manner. Despite the remarkable potential of diffusion models in image generation, current diffusion-based SFR models struggle with generalization to real-world faces. To address this limitation, we outline three key objectives for SFR: (1) promoting diversity across identities (inter-class diversity), (2) ensuring diversity within each identity by injecting various facial attributes (intra-class diversity), and (3) maintaining identity consistency within each identity group (intra-class identity preservation). Inspired by these goals, we introduce a diffusion-fueled SFR model termed $\text{ID}^3$. $\text{ID}^3$ employs an ID-preserving loss to generate diverse yet identity-consistent facial appearances. Theoretically, we show that minimizing this loss is equivalent to maximizing the lower bound of an adjusted conditional log-likelihood over ID-preserving data. This equivalence motivates an ID-preserving sampling algorithm, which operates over an adjusted gradient vector field, enabling the generation of fake face recognition datasets that approximate the distribution of real-world faces. Extensive experiments across five challenging benchmarks validate the advantages of $\text{ID}^3$.</li>

</ul>

<h3>Title: Multimodal Banking Dataset: Understanding Client Needs through Event Sequences</h3>

<ul>

<li><strong>Authors: </strong>Mollaev Dzhambulat, Alexander Kostin, Postnova Maria, Ivan Karpukhin, Ivan A Kireev, Gleb Gusev, Andrey Savchenko</a></li>

<li><strong>Subjects: </strong>cs.LG, cs.AI</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17587">https://arxiv.org/abs/2409.17587</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17587">https://arxiv.org/pdf/2409.17587</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17587]] Multimodal Banking Dataset: Understanding Client Needs through Event Sequences(https://arxiv.org/abs/2409.17587)</code><input type="text"></li>

<li><strong>Keywords: </strong>privacy</a></li>

<li><strong>Abstract: </strong>Financial organizations collect a huge amount of data about clients that typically has a temporal (sequential) structure and is collected from various sources (modalities). Due to privacy issues, there are no large-scale open-source multimodal datasets of event sequences, which significantly limits the research in this area. In this paper, we present the industrial-scale publicly available multimodal banking dataset, MBD, that contains more than 1.5M corporate clients with several modalities: 950M bank transactions, 1B geo position events, 5M embeddings of dialogues with technical support and monthly aggregated purchases of four bank's products. All entries are properly anonymized from real proprietary bank data. Using this dataset, we introduce a novel benchmark with two business tasks: campaigning (purchase prediction in the next month) and matching of clients. We provide numerical results that demonstrate the superiority of our multi-modal baselines over single-modal techniques for each task. As a result, the proposed dataset can open new perspectives and facilitate the future development of practically important large-scale multimodal algorithms for event sequences. HuggingFace Link: this https URL Github Link: this https URL</li>

</ul>

<h3>Title: DualCoTs: Dual Chain-of-Thoughts Prompting for Sentiment Lexicon Expansion of Idioms</h3>

<ul>

<li><strong>Authors: </strong>Fuqiang Niu, Minghuan Tan, Bowen Zhang, Min Yang, Ruifeng Xu</a></li>

<li><strong>Subjects: </strong>cs.CL</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17588">https://arxiv.org/abs/2409.17588</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17588">https://arxiv.org/pdf/2409.17588</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17588]] DualCoTs: Dual Chain-of-Thoughts Prompting for Sentiment Lexicon Expansion of Idioms(https://arxiv.org/abs/2409.17588)</code><input type="text"></li>

<li><strong>Keywords: </strong>large language model</a></li>

<li><strong>Abstract: </strong>Idioms represent a ubiquitous vehicle for conveying sentiments in the realm of everyday discourse, rendering the nuanced analysis of idiom sentiment crucial for a comprehensive understanding of emotional expression within real-world texts. Nevertheless, the existing corpora dedicated to idiom sentiment analysis considerably limit research in text sentiment analysis. In this paper, we propose an innovative approach to automatically expand the sentiment lexicon for idioms, leveraging the capabilities of large language models through the application of Chain-of-Thought prompting. To demonstrate the effectiveness of this approach, we integrate multiple existing resources and construct an emotional idiom lexicon expansion dataset (called EmoIdiomE), which encompasses a comprehensive repository of Chinese and English idioms. Then we designed the Dual Chain-of-Thoughts (DualCoTs) method, which combines insights from linguistics and psycholinguistics, to demonstrate the effectiveness of using large models to automatically expand the sentiment lexicon for idioms. Experiments show that DualCoTs is effective in idioms sentiment lexicon expansion in both Chinese and English. For reproducibility, we will release the data and code upon acceptance.</li>

</ul>

<h3>Title: Improving Fast Adversarial Training via Self-Knowledge Guidance</h3>

<ul>

<li><strong>Authors: </strong>Chengze Jiang, Junkai Wang, Minjing Dong, Jie Gui, Xinli Shi, Yuan Cao, Yuan Yan Tang, James Tin-Yau Kwok</a></li>

<li><strong>Subjects: </strong>cs.CV, cs.AI</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17589">https://arxiv.org/abs/2409.17589</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17589">https://arxiv.org/pdf/2409.17589</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17589]] Improving Fast Adversarial Training via Self-Knowledge Guidance(https://arxiv.org/abs/2409.17589)</code><input type="text"></li>

<li><strong>Keywords: </strong>attack, robust</a></li>

<li><strong>Abstract: </strong>Adversarial training has achieved remarkable advancements in defending against adversarial attacks. Among them, fast adversarial training (FAT) is gaining attention for its ability to achieve competitive robustness with fewer computing resources. Existing FAT methods typically employ a uniform strategy that optimizes all training data equally without considering the influence of different examples, which leads to an imbalanced optimization. However, this imbalance remains unexplored in the field of FAT. In this paper, we conduct a comprehensive study of the imbalance issue in FAT and observe an obvious class disparity regarding their performances. This disparity could be embodied from a perspective of alignment between clean and robust accuracy. Based on the analysis, we mainly attribute the observed misalignment and disparity to the imbalanced optimization in FAT, which motivates us to optimize different training data adaptively to enhance robustness. Specifically, we take disparity and misalignment into consideration. First, we introduce self-knowledge guided regularization, which assigns differentiated regularization weights to each class based on its training state, alleviating class disparity. Additionally, we propose self-knowledge guided label relaxation, which adjusts label relaxation according to the training accuracy, alleviating the misalignment and improving robustness. By combining these methods, we formulate the Self-Knowledge Guided FAT (SKG-FAT), leveraging naturally generated knowledge during training to enhance the adversarial robustness without compromising training efficiency. Extensive experiments on four standard datasets demonstrate that the SKG-FAT improves the robustness and preserves competitive clean accuracy, outperforming the state-of-the-art methods.</li>

</ul>

<h3>Title: AsIf: Asset Interface Analysis of Industrial Automation Devices</h3>

<ul>

<li><strong>Authors: </strong>Thomas Rosenstatter, Christian Schäfer, Olaf Saßnick, Stefan Huber</a></li>

<li><strong>Subjects: </strong>cs.CR</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17593">https://arxiv.org/abs/2409.17593</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17593">https://arxiv.org/pdf/2409.17593</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17593]] AsIf: Asset Interface Analysis of Industrial Automation Devices(https://arxiv.org/abs/2409.17593)</code><input type="text"></li>

<li><strong>Keywords: </strong>security</a></li>

<li><strong>Abstract: </strong>As Industry 4.0 and the Industrial Internet of Things continue to advance, industrial control systems are increasingly adopting IT solutions, including communication standards and protocols. As these systems become more decentralized and interconnected, a critical need for enhanced security measures arises. Threat modeling is traditionally performed in structured brainstorming sessions involving domain and security experts. Such sessions, however, often fail to provide an exhaustive identification of assets and interfaces due to the lack of a systematic approach. This is a major issue, as it leads to poor threat modeling, resulting in insufficient mitigation strategies and, lastly, a flawed security architecture. We propose a method for the analysis of assets in industrial systems, with special focus on physical threats. Inspired by the ISO/OSI reference model, a systematic approach is introduced to help identify and classify asset interfaces. This results in an enriched system model of the asset, offering a comprehensive overview visually represented as an interface tree, thereby laying the foundation for subsequent threat modeling steps. To demonstrate the proposed method, the results of its application to a programmable logic controller (PLC) are presented. In support of this, a study involving a group of 12 security experts was conducted. Additionally, the study offers valuable insights into the experts' general perspectives and workflows on threat modeling.</li>

</ul>

<h3>Title: Unifying Dimensions: A Linear Adaptive Approach to Lightweight Image Super-Resolution</h3>

<ul>

<li><strong>Authors: </strong>Zhenyu Hu, Wanjie Sun</a></li>

<li><strong>Subjects: </strong>cs.CV</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17597">https://arxiv.org/abs/2409.17597</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17597">https://arxiv.org/pdf/2409.17597</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17597]] Unifying Dimensions: A Linear Adaptive Approach to Lightweight Image Super-Resolution(https://arxiv.org/abs/2409.17597)</code><input type="text"></li>

<li><strong>Keywords: </strong>transformer</a></li>

<li><strong>Abstract: </strong>Window-based transformers have demonstrated outstanding performance in super-resolution tasks due to their adaptive modeling capabilities through local self-attention (SA). However, they exhibit higher computational complexity and inference latency than convolutional neural networks. In this paper, we first identify that the adaptability of the Transformers is derived from their adaptive spatial aggregation and advanced structural design, while their high latency results from the computational costs and memory layout transformations associated with the local SA. To simulate this aggregation approach, we propose an effective convolution-based linear focal separable attention (FSA), allowing for long-range dynamic modeling with linear complexity. Additionally, we introduce an effective dual-branch structure combined with an ultra-lightweight information exchange module (IEM) to enhance the aggregation of information by the Token Mixer. Finally, with respect to the structure, we modify the existing spatial-gate-based feedforward neural networks by incorporating a self-gate mechanism to preserve high-dimensional channel information, enabling the modeling of more complex relationships. With these advancements, we construct a convolution-based Transformer framework named the linear adaptive mixer network (LAMNet). Extensive experiments demonstrate that LAMNet achieves better performance than existing SA-based Transformer methods while maintaining the computational efficiency of convolutional neural networks, which can achieve a \\(3\times\\) speedup of inference time. The code will be publicly available at: this https URL.</li>

</ul>

<h3>Title: TA-Cleaner: A Fine-grained Text Alignment Backdoor Defense Strategy for Multimodal Contrastive Learning</h3>

<ul>

<li><strong>Authors: </strong>Yuan Xun, Siyuan Liang, Xiaojun Jia, Xinwei Liu, Xiaochun Cao</a></li>

<li><strong>Subjects: </strong>cs.CV, cs.AI</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17601">https://arxiv.org/abs/2409.17601</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17601">https://arxiv.org/pdf/2409.17601</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17601]] TA-Cleaner: A Fine-grained Text Alignment Backdoor Defense Strategy for Multimodal Contrastive Learning(https://arxiv.org/abs/2409.17601)</code><input type="text"></li>

<li><strong>Keywords: </strong>defense, attack</a></li>

<li><strong>Abstract: </strong>Pre-trained large models for multimodal contrastive learning, such as CLIP, have been widely recognized in the industry as highly susceptible to data-poisoned backdoor attacks. This poses significant risks to downstream model training. In response to such potential threats, finetuning offers a simpler and more efficient defense choice compared to retraining large models with augmented data. In the supervised learning domain, fine-tuning defense strategies can achieve excellent defense performance. However, in the unsupervised and semi-supervised domain, we find that when CLIP faces some complex attack techniques, the existing fine-tuning defense strategy, CleanCLIP, has some limitations on defense performance. The synonym substitution of its text-augmentation is insufficient to enhance the text feature space. To compensate for this weakness, we improve it by proposing a fine-grained \textbf{T}ext \textbf{A}lignment \textbf{C}leaner (TA-Cleaner) to cut off feature connections of backdoor triggers. We randomly select a few samples for positive and negative subtext generation at each epoch of CleanCLIP, and align the subtexts to the images to strengthen the text self-supervision. We evaluate the effectiveness of our TA-Cleaner against six attack algorithms and conduct comprehensive zero-shot classification tests on ImageNet1K. Our experimental results demonstrate that TA-Cleaner achieves state-of-the-art defensiveness among finetuning-based defense techniques. Even when faced with the novel attack technique BadCLIP, our TA-Cleaner outperforms CleanCLIP by reducing the ASR of Top-1 and Top-10 by 52.02\% and 63.88\%, respectively.</li>

</ul>

<h3>Title: RmGPT: Rotating Machinery Generative Pretrained Model</h3>

<ul>

<li><strong>Authors: </strong>Yilin Wang, Yifei Yu, Kong Sun, Peixuan Lei, Yuxuan Zhang, Enrico Zio, Aiguo Xia, Yuanxiang Li</a></li>

<li><strong>Subjects: </strong>cs.LG</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17604">https://arxiv.org/abs/2409.17604</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17604">https://arxiv.org/pdf/2409.17604</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17604]] RmGPT: Rotating Machinery Generative Pretrained Model(https://arxiv.org/abs/2409.17604)</code><input type="text"></li>

<li><strong>Keywords: </strong>robust, extraction, generative</a></li>

<li><strong>Abstract: </strong>In industry, the reliability of rotating machinery is critical for production efficiency and safety. Current methods of Prognostics and Health Management (PHM) often rely on task-specific models, which face significant challenges in handling diverse datasets with varying signal characteristics, fault modes and operating conditions. Inspired by advancements in generative pretrained models, we propose RmGPT, a unified model for diagnosis and prognosis tasks. RmGPT introduces a novel token-based framework, incorporating Signal Tokens, Prompt Tokens, Time-Frequency Task Tokens and Fault Tokens to handle heterogeneous data within a unified model architecture. We leverage self-supervised learning for robust feature extraction and introduce a next signal token prediction pretraining strategy, alongside efficient prompt learning for task-specific adaptation. Extensive experiments demonstrate that RmGPT significantly outperforms state-of-the-art algorithms, achieving near-perfect accuracy in diagnosis tasks and exceptionally low errors in prognosis tasks. Notably, RmGPT excels in few-shot learning scenarios, achieving 92% accuracy in 16-class one-shot experiments, highlighting its adaptability and robustness. This work establishes RmGPT as a powerful PHM foundation model for rotating machinery, advancing the scalability and generalizability of PHM solutions.</li>

</ul>

<h3>Title: Good Data Is All Imitation Learning Needs</h3>

<ul>

<li><strong>Authors: </strong>Amir Samadi, Konstantinos Koufos, Kurt Debattista, Mehrdad Dianati</a></li>

<li><strong>Subjects: </strong>cs.CV, cs.LG</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17605">https://arxiv.org/abs/2409.17605</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17605">https://arxiv.org/pdf/2409.17605</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17605]] Good Data Is All Imitation Learning Needs(https://arxiv.org/abs/2409.17605)</code><input type="text"></li>

<li><strong>Keywords: </strong>robust</a></li>

<li><strong>Abstract: </strong>In this paper, we address the limitations of traditional teacher-student models, imitation learning, and behaviour cloning in the context of Autonomous/Automated Driving Systems (ADS), where these methods often struggle with incomplete coverage of real-world scenarios. To enhance the robustness of such models, we introduce the use of Counterfactual Explanations (CFEs) as a novel data augmentation technique for end-to-end ADS. CFEs, by generating training samples near decision boundaries through minimal input modifications, lead to a more comprehensive representation of expert driver strategies, particularly in safety-critical scenarios. This approach can therefore help improve the model's ability to handle rare and challenging driving events, such as anticipating darting out pedestrians, ultimately leading to safer and more trustworthy decision-making for ADS. Our experiments in the CARLA simulator demonstrate that CF-Driver outperforms the current state-of-the-art method, achieving a higher driving score and lower infraction rates. Specifically, CF-Driver attains a driving score of 84.2, surpassing the previous best model by 15.02 percentage points. These results highlight the effectiveness of incorporating CFEs in training end-to-end ADS. To foster further research, the CF-Driver code is made publicly available.</li>

</ul>

<h3>Title: ZALM3: Zero-Shot Enhancement of Vision-Language Alignment via In-Context Information in Multi-Turn Multimodal Medical Dialogue</h3>

<ul>

<li><strong>Authors: </strong>Zhangpu Li, Changhong Zou, Suxue Ma, Zhicheng Yang, Chen Du, Youbao Tang, Zhenjie Cao, Ning Zhang, Jui-Hsin Lai, Ruei-Sung Lin, Yuan Ni, Xingzhi Sun, Jing Xiao, Kai Zhang, Mei Han</a></li>

<li><strong>Subjects: </strong>cs.CL, cs.CV</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17610">https://arxiv.org/abs/2409.17610</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17610">https://arxiv.org/pdf/2409.17610</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17610]] ZALM3: Zero-Shot Enhancement of Vision-Language Alignment via In-Context Information in Multi-Turn Multimodal Medical Dialogue(https://arxiv.org/abs/2409.17610)</code><input type="text"></li>

<li><strong>Keywords: </strong>large language model</a></li>

<li><strong>Abstract: </strong>The rocketing prosperity of large language models (LLMs) in recent years has boosted the prevalence of vision-language models (VLMs) in the medical sector. In our online medical consultation scenario, a doctor responds to the texts and images provided by a patient in multiple rounds to diagnose her/his health condition, forming a multi-turn multimodal medical dialogue format. Unlike high-quality images captured by professional equipment in traditional medical visual question answering (Med-VQA), the images in our case are taken by patients' mobile phones. These images have poor quality control, with issues such as excessive background elements and the lesion area being significantly off-center, leading to degradation of vision-language alignment in the model training phase. In this paper, we propose ZALM3, a Zero-shot strategy to improve vision-language ALignment in Multi-turn Multimodal Medical dialogue. Since we observe that the preceding text conversations before an image can infer the regions of interest (RoIs) in the image, ZALM3 employs an LLM to summarize the keywords from the preceding context and a visual grounding model to extract the RoIs. The updated images eliminate unnecessary background noise and provide more effective vision-language alignment. To better evaluate our proposed method, we design a new subjective assessment metric for multi-turn unimodal/multimodal medical dialogue to provide a fine-grained performance comparison. Our experiments across three different clinical departments remarkably demonstrate the efficacy of ZALM3 with statistical significance.</li>

</ul>

<h3>Title: Benign or Not-Benign Overfitting in Token Selection of Attention Mechanism</h3>

<ul>

<li><strong>Authors: </strong>Keitaro Sakamoto, Issei Sato</a></li>

<li><strong>Subjects: </strong>cs.LG</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17625">https://arxiv.org/abs/2409.17625</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17625">https://arxiv.org/pdf/2409.17625</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17625]] Benign or Not-Benign Overfitting in Token Selection of Attention Mechanism(https://arxiv.org/abs/2409.17625)</code><input type="text"></li>

<li><strong>Keywords: </strong>transformer</a></li>

<li><strong>Abstract: </strong>Modern over-parameterized neural networks can be trained to fit the training data perfectly while still maintaining a high generalization performance. This "benign overfitting" phenomenon has been studied in a surge of recent theoretical work; however, most of these studies have been limited to linear models or two-layer neural networks. In this work, we analyze benign overfitting in the token selection mechanism of the attention architecture, which characterizes the success of transformer models. We first show the existence of a benign overfitting solution and explain its mechanism in the attention architecture. Next, we discuss whether the model converges to such a solution, raising the difficulties specific to the attention architecture. We then present benign overfitting cases and not-benign overfitting cases by conditioning different scenarios based on the behavior of attention probabilities during training. To the best of our knowledge, this is the first study to characterize benign overfitting for the attention mechanism.</li>

</ul>

<h3>Title: T3: A Novel Zero-shot Transfer Learning Framework Iteratively Training on an Assistant Task for a Target Task</h3>

<ul>

<li><strong>Authors: </strong>Xindi Tong, Yujin Zhu, Shijian Fan, Liang Xu</a></li>

<li><strong>Subjects: </strong>cs.CL, cs.AI</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17640">https://arxiv.org/abs/2409.17640</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17640">https://arxiv.org/pdf/2409.17640</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17640]] T3: A Novel Zero-shot Transfer Learning Framework Iteratively Training on an Assistant Task for a Target Task(https://arxiv.org/abs/2409.17640)</code><input type="text"></li>

<li><strong>Keywords: </strong>fair, large language model</a></li>

<li><strong>Abstract: </strong>Long text summarization, gradually being essential for efficiently processing large volumes of information, stays challenging for Large Language Models (LLMs) such as GPT and LLaMA families because of the insufficient open-sourced training datasets and the high requirement of contextual details dealing. To address the issue, we design a novel zero-shot transfer learning framework, abbreviated as T3, to iteratively training a baseline LLM on an assistant task for the target task, where the former should own richer data resources and share structural or semantic similarity with the latter. In practice, T3 is approached to deal with the long text summarization task by utilizing question answering as the assistant task, and further validated its effectiveness on the BBC summary, NarraSum, FairytaleQA, and NLQuAD datasets, with up to nearly 14% improvement in ROUGE, 35% improvement in BLEU, and 16% improvement in Factscore compared to three baseline LLMs, demonstrating its potential for more assistant-target task combinations.</li>

</ul>

<h3>Title: Efficient In-Domain Question Answering for Resource-Constrained Environments</h3>

<ul>

<li><strong>Authors: </strong>Isaac Chung, Phat Vo, Arman Kizilkale, Aaron Reite</a></li>

<li><strong>Subjects: </strong>cs.CL</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17648">https://arxiv.org/abs/2409.17648</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17648">https://arxiv.org/pdf/2409.17648</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17648]] Efficient In-Domain Question Answering for Resource-Constrained Environments(https://arxiv.org/abs/2409.17648)</code><input type="text"></li>

<li><strong>Keywords: </strong>robust, large language model</a></li>

<li><strong>Abstract: </strong>Retrieval Augmented Generation (RAG) is a common method for integrating external knowledge into pretrained Large Language Models (LLMs) to enhance accuracy and relevancy in question answering (QA) tasks. However, prompt engineering and resource efficiency remain significant bottlenecks in developing optimal and robust RAG solutions for real-world QA applications. Recent studies have shown success in using fine tuning to address these problems; in particular, Retrieval Augmented Fine Tuning (RAFT) applied to smaller 7B models has demonstrated superior performance compared to RAG setups with much larger models such as GPT-3.5. The combination of RAFT with parameter-efficient fine tuning (PEFT) techniques, such as Low-Rank Adaptation (LoRA), promises an even more efficient solution, yet remains an unexplored area. In this work, we combine RAFT with LoRA to reduce fine tuning and storage requirements and gain faster inference times while maintaining comparable RAG performance. This results in a more compute-efficient RAFT, or CRAFT, which is particularly useful for knowledge-intensive QA tasks in resource-constrained environments where internet access may be restricted and hardware resources limited.</li>

</ul>

<h3>Title: Provable Performance Guarantees of Copy Detection Patterns</h3>

<ul>

<li><strong>Authors: </strong>Joakim Tutt, Slava Voloshynovskiy</a></li>

<li><strong>Subjects: </strong>cs.CR, cs.CV</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17649">https://arxiv.org/abs/2409.17649</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17649">https://arxiv.org/pdf/2409.17649</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17649]] Provable Performance Guarantees of Copy Detection Patterns(https://arxiv.org/abs/2409.17649)</code><input type="text"></li>

<li><strong>Keywords: </strong>security</a></li>

<li><strong>Abstract: </strong>Copy Detection Patterns (CDPs) are crucial elements in modern security applications, playing a vital role in safeguarding industries such as food, pharmaceuticals, and cosmetics. Current performance evaluations of CDPs predominantly rely on empirical setups using simplistic metrics like Hamming distances or Pearson correlation. These methods are often inadequate due to their sensitivity to distortions, degradation, and their limitations to stationary statistics of printing and imaging. Additionally, machine learning-based approaches suffer from distribution biases and fail to generalize to unseen counterfeit samples. Given the critical importance of CDPs in preventing counterfeiting, including the counterfeit vaccines issue highlighted during the COVID-19 pandemic, there is an urgent need for provable performance guarantees across various criteria. This paper aims to establish a theoretical framework to derive optimal criteria for the analysis, optimization, and future development of CDP authentication technologies, ensuring their reliability and effectiveness in diverse security scenarios.</li>

</ul>

<h3>Title: A Comprehensive Review of TLSNotary Protocol</h3>

<ul>

<li><strong>Authors: </strong>Maciej Kalka, Marek Kirejczyk</a></li>

<li><strong>Subjects: </strong>cs.CR</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17670">https://arxiv.org/abs/2409.17670</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17670">https://arxiv.org/pdf/2409.17670</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17670]] A Comprehensive Review of TLSNotary Protocol(https://arxiv.org/abs/2409.17670)</code><input type="text"></li>

<li><strong>Keywords: </strong>secure, security</a></li>

<li><strong>Abstract: </strong>Transport Layer Security (TLS) protocol is a cryptographic protocol designed to secure communication over the internet. The TLS protocol has become a fundamental in secure communication, most commonly used for securing web browsing sessions. In this work, we investigate the TLSNotary protocol, which aim to enable the Client to obtain proof of provenance for data from TLS session, while getting as much as possible from the TLS security properties. To achieve such proofs without any Server-side adjustments or permissions, the power of secure multi-party computation (MPC) together with zero knowledge proofs is used to extend the standard TLS Protocol. To make the compliacted landscape of MPC as comprehensible as possible we first introduce the cryptographic primitives required to understand the TLSNotary protocol and go through standard TLS protocol. Finally, we look at the TLSNotary protocol in detail.</li>

</ul>

<h3>Title: Self-Supervised Learning of Deviation in Latent Representation for Co-speech Gesture Video Generation</h3>

<ul>

<li><strong>Authors: </strong>Huan Yang, Jiahui Chen, Chaofan Ding, Runhua Shi, Siyu Xiong, Qingqi Hong, Xiaoqi Mo, Xinhan Di</a></li>

<li><strong>Subjects: </strong>cs.CV</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17674">https://arxiv.org/abs/2409.17674</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17674">https://arxiv.org/pdf/2409.17674</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17674]] Self-Supervised Learning of Deviation in Latent Representation for Co-speech Gesture Video Generation(https://arxiv.org/abs/2409.17674)</code><input type="text"></li>

<li><strong>Keywords: </strong>diffusion</a></li>

<li><strong>Abstract: </strong>Gestures are pivotal in enhancing co-speech communication. While recent works have mostly focused on point-level motion transformation or fully supervised motion representations through data-driven approaches, we explore the representation of gestures in co-speech, with a focus on self-supervised representation and pixel-level motion deviation, utilizing a diffusion model which incorporates latent motion features. Our approach leverages self-supervised deviation in latent representation to facilitate hand gestures generation, which are crucial for generating realistic gesture videos. Results of our first experiment demonstrate that our method enhances the quality of generated videos, with an improvement from 2.7 to 4.5% for FGD, DIV, and FVD, and 8.1% for PSNR, 2.5% for SSIM over the current state-of-the-art methods.</li>

</ul>

<h3>Title: EM-Net: Efficient Channel and Frequency Learning with Mamba for 3D Medical Image Segmentation</h3>

<ul>

<li><strong>Authors: </strong>Ao Chang, Jiajun Zeng, Ruobing Huang, Dong Ni</a></li>

<li><strong>Subjects: </strong>cs.CV</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17675">https://arxiv.org/abs/2409.17675</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17675">https://arxiv.org/pdf/2409.17675</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17675]] EM-Net: Efficient Channel and Frequency Learning with Mamba for 3D Medical Image Segmentation(https://arxiv.org/abs/2409.17675)</code><input type="text"></li>

<li><strong>Keywords: </strong>transformer, segmentation</a></li>

<li><strong>Abstract: </strong>Convolutional neural networks have primarily led 3D medical image segmentation but may be limited by small receptive fields. Transformer models excel in capturing global relationships through self-attention but are challenged by high computational costs at high resolutions. Recently, Mamba, a state space model, has emerged as an effective approach for sequential modeling. Inspired by its success, we introduce a novel Mamba-based 3D medical image segmentation model called EM-Net. It not only efficiently captures attentive interaction between regions by integrating and selecting channels, but also effectively utilizes frequency domain to harmonize the learning of features across varying scales, while accelerating training speed. Comprehensive experiments on two challenging multi-organ datasets with other state-of-the-art (SOTA) algorithms show that our method exhibits better segmentation accuracy while requiring nearly half the parameter size of SOTA models and 2x faster training speed.</li>

</ul>

<h3>Title: Optimal Memorization Capacity of Transformers</h3>

<ul>

<li><strong>Authors: </strong>Tokio Kajitsuka, Issei Sato</a></li>

<li><strong>Subjects: </strong>cs.LG</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17677">https://arxiv.org/abs/2409.17677</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17677">https://arxiv.org/pdf/2409.17677</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17677]] Optimal Memorization Capacity of Transformers(https://arxiv.org/abs/2409.17677)</code><input type="text"></li>

<li><strong>Keywords: </strong>transformer</a></li>

<li><strong>Abstract: </strong>Recent research in the field of machine learning has increasingly focused on the memorization capacity of Transformers, but how efficient they are is not yet well understood. We demonstrate that Transformers can memorize labels with $\tilde{O}(\sqrt{N})$ parameters in a next-token prediction setting for $N$ input sequences of length $n$, which is proved to be optimal up to logarithmic factors. This indicates that Transformers can efficiently perform memorization with little influence from the input length $n$ owing to the benefit of parameter sharing. We also analyze the memorization capacity in the sequence-to-sequence setting, and find that $\tilde{O}(\sqrt{nN})$ parameters are not only sufficient, but also necessary at least for Transformers with hardmax. These results suggest that while self-attention mechanisms can efficiently identify input sequences, the feed-forward network becomes a bottleneck when associating a label to each token.</li>

</ul>

<h3>Title: Dark Miner: Defend against unsafe generation for text-to-image diffusion models</h3>

<ul>

<li><strong>Authors: </strong>Zheling Meng, Bo Peng, Xiaochuan Jin, Yue Jiang, Jing Dong, Wei Wang, Tieniu Tan</a></li>

<li><strong>Subjects: </strong>cs.CV</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17682">https://arxiv.org/abs/2409.17682</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17682">https://arxiv.org/pdf/2409.17682</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17682]] Dark Miner: Defend against unsafe generation for text-to-image diffusion models(https://arxiv.org/abs/2409.17682)</code><input type="text"></li>

<li><strong>Keywords: </strong>defense, attack, diffusion</a></li>

<li><strong>Abstract: </strong>Text-to-image diffusion models have been demonstrated with unsafe generation due to unfiltered large-scale training data, such as violent, sexual, and shocking images, necessitating the erasure of unsafe concepts. Most existing methods focus on modifying the generation probabilities conditioned on the texts containing unsafe descriptions. However, they fail to guarantee safe generation for unseen texts in the training phase, especially for the prompts from adversarial attacks. In this paper, we re-analyze the erasure task and point out that existing methods cannot guarantee the minimization of the total probabilities of unsafe generation. To tackle this problem, we propose Dark Miner. It entails a recurring three-stage process that comprises mining, verifying, and circumventing. It greedily mines embeddings with maximum generation probabilities of unsafe concepts and reduces unsafe generation more effectively. In the experiments, we evaluate its performance on two inappropriate concepts, two objects, and two styles. Compared with 6 previous state-of-the-art methods, our method achieves better erasure and defense results in most cases, especially under 4 state-of-the-art attacks, while preserving the model's native generation capability. Our code will be available on GitHub.</li>

</ul>

<h3>Title: Zero- and Few-shot Named Entity Recognition and Text Expansion in Medication Prescriptions using ChatGPT</h3>

<ul>

<li><strong>Authors: </strong>Natthanaphop Isaradech, Andrea Riedel, Wachiranun Sirikul, Markus Kreuzthaler, Stefan Schulz</a></li>

<li><strong>Subjects: </strong>cs.CL, cs.AI</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17683">https://arxiv.org/abs/2409.17683</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17683">https://arxiv.org/pdf/2409.17683</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17683]] Zero- and Few-shot Named Entity Recognition and Text Expansion in Medication Prescriptions using ChatGPT(https://arxiv.org/abs/2409.17683)</code><input type="text"></li>

<li><strong>Keywords: </strong>large language model</a></li>

<li><strong>Abstract: </strong>Introduction: Medication prescriptions are often in free text and include a mix of two languages, local brand names, and a wide range of idiosyncratic formats and abbreviations. Large language models (LLMs) have shown promising ability to generate text in response to input prompts. We use ChatGPT 3.5 to automatically structure and expand medication statements in discharge summaries and thus make them easier to interpret for people and machines. Methods: Named-entity Recognition (NER) and Text Expansion (EX) are used in a zero- and few-shot setting with different prompt strategies. 100 medication statements were manually annotated and curated. NER performance was measured by using strict and partial matching. For the task EX, two experts interpreted the results by assessing semantic equivalence between original and expanded statements. The model performance was measured by precision, recall, and F1 score. Results: For NER, the best-performing prompt reached an average F1 score of 0.94 in the test set. For EX, the few-shot prompt showed superior performance among other prompts, with an average F1 score of 0.87. Conclusion: Our study demonstrates good performance for NER and EX tasks in free-text medication statements using ChatGPT. Compared to a zero-shot baseline, a few-shot approach prevented the system from hallucinating, which would be unacceptable when processing safety-relevant medication data.</li>

</ul>

<h3>Title: Efficient Bias Mitigation Without Privileged Information</h3>

<ul>

<li><strong>Authors: </strong>Mateo Espinosa Zarlenga, Swami Sankaranarayanan, Jerone T. A. Andrews, Zohreh Shams, Mateja Jamnik, Alice Xiang</a></li>

<li><strong>Subjects: </strong>cs.LG, cs.AI</a></li>

<li><strong>Abstract URL: </strong><a href="https://arxiv.org/abs/2409.17691">https://arxiv.org/abs/2409.17691</a></li>

<li><strong>Pdf URL: </strong><a href="https://arxiv.org/pdf/2409.17691">https://arxiv.org/pdf/2409.17691</a></li>

<li><strong>Copy Paste: </strong><code><input type="checkbox">[[2409.17691]] Efficient Bias Mitigation Without Privileged Information(https://arxiv.org/abs/2409.17691)</code><input type="text"></li>

[View remainder of file in raw view](https://github.com/wwd29/arxiv-daily/raw/1a3074665786adbe2140d325f14a577e8297e535/html/user_0/2024-09-27.html)

## Footer

[ ](https://github.com "GitHub") (C) 2025 GitHub, Inc. 

### Footer navigation

  * [Terms](https://docs.github.com/site-policy/github-terms/github-terms-of-service)
  * [Privacy](https://docs.github.com/site-policy/privacy-policies/github-privacy-statement)
  * [Security](https://github.com/security)
  * [Status](https://www.githubstatus.com/)
  * [Docs](https://docs.github.com/)
  * [Contact](https://support.github.com?tags=dotcom-footer)
  * Manage cookies 
  * Do not share my personal information 



You can’t perform that action at this time. 


scrape_date: 2025-03-17T13:31:34.447775
content_type: markdown
hash: 14940c40c35bf6e23bb00b5bd0312243e0253bd1b0d9366bbac0513cf1d1d783
url:
https://github.com/placeboo/ArxivSummary/blob/901d2c613a773731c87cbb4962081419541aac2a/data/arxiv/2024-09-30/catalog.md

title:
ArxivSummary/data/arxiv/2024-09-30/catalog.md at 901d2c613a773731c87cbb4962081419541aac2a · placeboo/ArxivSummary · GitHub

content:
Skip to content

## Navigation Menu

Toggle navigation

[ ](/)

[ Sign in ](/login?return_to=https%3A%2F%2Fgithub.com%2Fplaceboo%2FArxivSummary%2Fblob%2F901d2c613a773731c87cbb4962081419541aac2a%2Fdata%2Farxiv%2F2024-09-30%2Fcatalog.md)

  * Product 

    * [ GitHub Copilot Write better code with AI  ](https://github.com/features/copilot)
    * [ Security Find and fix vulnerabilities  ](https://github.com/features/security)
    * [ Actions Automate any workflow  ](https://github.com/features/actions)
    * [ Codespaces Instant dev environments  ](https://github.com/features/codespaces)
    * [ Issues Plan and track work  ](https://github.com/features/issues)
    * [ Code Review Manage code changes  ](https://github.com/features/code-review)
    * [ Discussions Collaborate outside of code  ](https://github.com/features/discussions)
    * [ Code Search Find more, search less  ](https://github.com/features/code-search)

Explore
    * [ All features ](https://github.com/features)
    * [ Documentation  ](https://docs.github.com)
    * [ GitHub Skills  ](https://skills.github.com)
    * [ Blog  ](https://github.blog)

  * Solutions 

By company size
    * [ Enterprises ](https://github.com/enterprise)
    * [ Small and medium teams ](https://github.com/team)
    * [ Startups ](https://github.com/enterprise/startups)
    * [ Nonprofits ](/solutions/industry/nonprofits)

By use case
    * [ DevSecOps ](/solutions/use-case/devsecops)
    * [ DevOps ](/solutions/use-case/devops)
    * [ CI/CD ](/solutions/use-case/ci-cd)
    * [ View all use cases ](/solutions/use-case)

By industry
    * [ Healthcare ](/solutions/industry/healthcare)
    * [ Financial services ](/solutions/industry/financial-services)
    * [ Manufacturing ](/solutions/industry/manufacturing)
    * [ Government ](/solutions/industry/government)
    * [ View all industries ](/solutions/industry)

[ View all solutions  ](/solutions)

  * Resources 

Topics
    * [ AI ](/resources/articles/ai)
    * [ DevOps ](/resources/articles/devops)
    * [ Security ](/resources/articles/security)
    * [ Software Development ](/resources/articles/software-development)
    * [ View all ](/resources/articles)

Explore
    * [ Learning Pathways  ](https://resources.github.com/learn/pathways)
    * [ Events & Webinars  ](https://resources.github.com)
    * [ Ebooks & Whitepapers ](https://github.com/resources/whitepapers)
    * [ Customer Stories ](https://github.com/customer-stories)
    * [ Partners  ](https://partner.github.com)
    * [ Executive Insights ](https://github.com/solutions/executive-insights)

  * Open Source 

    * [ GitHub Sponsors Fund open source developers  ](/sponsors)

    * [ The ReadME Project GitHub community articles  ](https://github.com/readme)

Repositories
    * [ Topics ](https://github.com/topics)
    * [ Trending ](https://github.com/trending)
    * [ Collections ](https://github.com/collections)

  * Enterprise 

    * [ Enterprise platform AI-powered developer platform  ](/enterprise)

Available add-ons
    * [ Advanced Security Enterprise-grade security features  ](https://github.com/enterprise/advanced-security)
    * [ Copilot for business Enterprise-grade AI features  ](/features/copilot/copilot-business)
    * [ Premium Support Enterprise-grade 24/7 support  ](/premium-support)

  * [Pricing](https://github.com/pricing)



Search or jump to...

# Search code, repositories, users, issues, pull requests...

Search 

Clear




[Search syntax tips](https://docs.github.com/search-github/github-code-search/understanding-github-code-search-syntax)

#  Provide feedback 

We read every piece of feedback, and take your input very seriously.

Include my email address so I can be contacted

Cancel  Submit feedback 

#  Saved searches 

## Use saved searches to filter your results more quickly

Name

Query

To see all available qualifiers, see our [documentation](https://docs.github.com/search-github/github-code-search/understanding-github-code-search-syntax). 

Cancel  Create saved search 

[ Sign in ](/login?return_to=https%3A%2F%2Fgithub.com%2Fplaceboo%2FArxivSummary%2Fblob%2F901d2c613a773731c87cbb4962081419541aac2a%2Fdata%2Farxiv%2F2024-09-30%2Fcatalog.md)

[ Sign up ](/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fblob%2Fshow&source=header-repo&source_repo=placeboo%2FArxivSummary) Reseting focus

You signed in with another tab or window. [Reload]() to refresh your session. You signed out in another tab or window. [Reload]() to refresh your session. You switched accounts on another tab or window. [Reload]() to refresh your session. Dismiss alert

{{ message }}

[ placeboo ](/placeboo) / **[ArxivSummary](/placeboo/ArxivSummary) ** Public

  * [ Notifications ](/login?return_to=%2Fplaceboo%2FArxivSummary) You must be signed in to change notification settings
  * [ Fork 0 ](/login?return_to=%2Fplaceboo%2FArxivSummary)
  * [ Star  1 ](/login?return_to=%2Fplaceboo%2FArxivSummary)




  * [ Code ](/placeboo/ArxivSummary)
  * [ Issues 0 ](/placeboo/ArxivSummary/issues)
  * [ Pull requests 0 ](/placeboo/ArxivSummary/pulls)
  * [ Actions ](/placeboo/ArxivSummary/actions)
  * [ Projects 0 ](/placeboo/ArxivSummary/projects)
  * [ Security ](/placeboo/ArxivSummary/security)
  * [ Insights ](/placeboo/ArxivSummary/pulse)



Additional navigation options

  * [ Code  ](/placeboo/ArxivSummary)
  * [ Issues  ](/placeboo/ArxivSummary/issues)
  * [ Pull requests  ](/placeboo/ArxivSummary/pulls)
  * [ Actions  ](/placeboo/ArxivSummary/actions)
  * [ Projects  ](/placeboo/ArxivSummary/projects)
  * [ Security  ](/placeboo/ArxivSummary/security)
  * [ Insights  ](/placeboo/ArxivSummary/pulse)



## Files

901d2c6

## Breadcrumbs

  1. [ArxivSummary](/placeboo/ArxivSummary/tree/901d2c613a773731c87cbb4962081419541aac2a)
  2. /[data](/placeboo/ArxivSummary/tree/901d2c613a773731c87cbb4962081419541aac2a/data)
  3. /[arxiv](/placeboo/ArxivSummary/tree/901d2c613a773731c87cbb4962081419541aac2a/data/arxiv)
  4. /[2024-09-30](/placeboo/ArxivSummary/tree/901d2c613a773731c87cbb4962081419541aac2a/data/arxiv/2024-09-30)



/

# catalog.md

Copy path

Blame

Blame

## Latest commit

## History

[History](/placeboo/ArxivSummary/commits/901d2c613a773731c87cbb4962081419541aac2a/data/arxiv/2024-09-30/catalog.md)

[](/placeboo/ArxivSummary/commits/901d2c613a773731c87cbb4962081419541aac2a/data/arxiv/2024-09-30/catalog.md)

5204 lines (4204 loc) · 298 KB

901d2c6

## Breadcrumbs

  1. [ArxivSummary](/placeboo/ArxivSummary/tree/901d2c613a773731c87cbb4962081419541aac2a)
  2. /[data](/placeboo/ArxivSummary/tree/901d2c613a773731c87cbb4962081419541aac2a/data)
  3. /[arxiv](/placeboo/ArxivSummary/tree/901d2c613a773731c87cbb4962081419541aac2a/data/arxiv)
  4. /[2024-09-30](/placeboo/ArxivSummary/tree/901d2c613a773731c87cbb4962081419541aac2a/data/arxiv/2024-09-30)



/

# catalog.md

Top

## File metadata and controls

  * Preview

  * Code

  * Blame




5204 lines (4204 loc) · 298 KB

[Raw](https://github.com/placeboo/ArxivSummary/raw/901d2c613a773731c87cbb4962081419541aac2a/data/arxiv/2024-09-30/catalog.md)

## LML: Language Model Learning a Dataset for Data-Augmented Prediction

**Authors** : Praneeth Vadlapati

**Abstract** : This paper introduces a new approach to using Large Language Models (LLMs) for classification tasks, which are typically handled using Machine Learning (ML) models. Unlike ML models that rely heavily on data cleaning and feature engineering, this method streamlines the process using LLMs. This paper proposes a new concept called "Language Model Learning (LML)" powered by a new method called "Data-Augmented Prediction (DAP)". The classification is performed by LLMs using a method similar to humans manually exploring and understanding the data and deciding classifications using data as a reference. Training data is summarized and evaluated to determine the features that lead to the classification of each label the most. In the process of DAP, the system uses the data summary to automatically create a query, which is used to retrieve relevant rows from the dataset. A classification is generated by the LLM using data summary and relevant rows, ensuring satisfactory accuracy even with complex data. Usage of data summary and similar data in DAP ensures context-aware decision-making. The proposed method uses the words "Act as an Explainable Machine Learning Model" in the prompt to enhance the interpretability of the predictions by allowing users to review the logic behind each prediction. In some test cases, the system scored an accuracy above 90%, proving the effectiveness of the system and its potential to outperform conventional ML models in various scenarios. The code is available at <https://github.com/Pro-GenAI/LML-DAP>

**URL** : <http://arxiv.org/pdf/2409.18957v1>

**Published** : 2024-09-27

## Ruler: A Model-Agnostic Method to Control Generated Length for Large Language Models

**Authors** : Jiaming Li, Lei Zhang, Yunshui Li, Ziqiang Liu, yuelin bai, Run Luo, Longze Chen, Min Yang

**Abstract** : The instruction-following ability of large language models enables humans to interact with AI agents in a natural way. However, when required to generate responses of a specific length, large language models often struggle to meet users' needs due to their inherent difficulty in accurately perceiving numerical constraints. To explore the ability of large language models to control the length of generated responses, we propose the Target Length Generation Task (TLG) and design two metrics, Precise Match (PM) and Flexible Match (FM) to evaluate the model's performance in adhering to specified response lengths. Furthermore, we introduce a novel, model-agnostic approach called Ruler, which employs Meta Length Tokens (MLTs) to enhance the instruction-following ability of large language models under length-constrained instructions. Specifically, Ruler equips LLMs with the ability to generate responses of a specified length based on length constraints within the instructions. Moreover, Ruler can automatically generate appropriate MLT when length constraints are not explicitly provided, demonstrating excellent versatility and generalization. Comprehensive experiments show the effectiveness of Ruler across different LLMs on Target Length Generation Task, e.g., at All Level 27.97 average gain on PM, 29.57 average gain on FM. In addition, we conduct extensive ablation experiments to further substantiate the efficacy and generalization of Ruler. Our code and data is available at <https://github.com/Geaming2002/Ruler>.

**URL** : <http://arxiv.org/pdf/2409.18943v1>

**Published** : 2024-09-27

## From Seconds to Hours: Reviewing MultiModal Large Language Models on Comprehensive Long Video Understanding

**Authors** : Heqing Zou, Tianze Luo, Guiyang Xie, Victor, Zhang, Fengmao Lv, Guangcong Wang, Juanyang Chen, Zhuochen Wang, Hansheng Zhang, Huaijian Zhang

**Abstract** : The integration of Large Language Models (LLMs) with visual encoders has recently shown promising performance in visual understanding tasks, leveraging their inherent capability to comprehend and generate human-like text for visual reasoning. Given the diverse nature of visual data, MultiModal Large Language Models (MM-LLMs) exhibit variations in model designing and training for understanding images, short videos, and long videos. Our paper focuses on the substantial differences and unique challenges posed by long video understanding compared to static image and short video understanding. Unlike static images, short videos encompass sequential frames with both spatial and within-event temporal information, while long videos consist of multiple events with between-event and long-term temporal information. In this survey, we aim to trace and summarize the advancements of MM-LLMs from image understanding to long video understanding. We review the differences among various visual understanding tasks and highlight the challenges in long video understanding, including more fine-grained spatiotemporal details, dynamic events, and long-term dependencies. We then provide a detailed summary of the advancements in MM-LLMs in terms of model design and training methodologies for understanding long videos. Finally, we compare the performance of existing MM-LLMs on video understanding benchmarks of various lengths and discuss potential future directions for MM-LLMs in long video understanding.

**URL** : <http://arxiv.org/pdf/2409.18938v1>

**Published** : 2024-09-27

## AIPatient: Simulating Patients with EHRs and LLM Powered Agentic Workflow

**Authors** : Huizi Yu, Jiayan Zhou, Lingyao Li, Shan Chen, Jack Gallifant, Anye Shi, Xiang Li, Wenyue Hua, Mingyu Jin, Guang Chen, Yang Zhou, Zhao Li, Trisha Gupte, Ming-Li Chen, Zahra Azizi, Yongfeng Zhang, Themistocles L. Assimes, Xin Ma, Danielle S. Bitterman, Lin Lu, Lizhou Fan

**Abstract** : Simulated patient systems play a crucial role in modern medical education and research, providing safe, integrative learning environments and enabling clinical decision-making simulations. Large Language Models (LLM) could advance simulated patient systems by replicating medical conditions and patient-doctor interactions with high fidelity and low cost. However, ensuring the effectiveness and trustworthiness of these systems remains a challenge, as they require a large, diverse, and precise patient knowledgebase, along with a robust and stable knowledge diffusion to users. Here, we developed AIPatient, an advanced simulated patient system with AIPatient Knowledge Graph (AIPatient KG) as the input and the Reasoning Retrieval-Augmented Generation (Reasoning RAG) agentic workflow as the generation backbone. AIPatient KG samples data from Electronic Health Records (EHRs) in the Medical Information Mart for Intensive Care (MIMIC)-III database, producing a clinically diverse and relevant cohort of 1,495 patients with high knowledgebase validity (F1 0.89). Reasoning RAG leverages six LLM powered agents spanning tasks including retrieval, KG query generation, abstraction, checker, rewrite, and summarization. This agentic framework reaches an overall accuracy of 94.15% in EHR-based medical Question Answering (QA), outperforming benchmarks that use either no agent or only partial agent integration. Our system also presents high readability (median Flesch Reading Ease 77.23; median Flesch Kincaid Grade 5.6), robustness (ANOVA F-value 0.6126, p<0.1), and stability (ANOVA F-value 0.782, p<0.1). The promising performance of the AIPatient system highlights its potential to support a wide range of applications, including medical education, model evaluation, and system integration.

**URL** : <http://arxiv.org/pdf/2409.18924v1>

**Published** : 2024-09-27

## Soft Measures for Extracting Causal Collective Intelligence

**Authors** : Maryam Berijanian, Spencer Dork, Kuldeep Singh, Michael Riley Millikan, Ashlin Riggs, Aadarsh Swaminathan, Sarah L. Gibbs, Scott E. Friedman, Nathan Brugnone

**Abstract** : Understanding and modeling collective intelligence is essential for addressing complex social systems. Directed graphs called fuzzy cognitive maps (FCMs) offer a powerful tool for encoding causal mental models, but extracting high-integrity FCMs from text is challenging. This study presents an approach using large language models (LLMs) to automate FCM extraction. We introduce novel graph-based similarity measures and evaluate them by correlating their outputs with human judgments through the Elo rating system. Results show positive correlations with human evaluations, but even the best-performing measure exhibits limitations in capturing FCM nuances. Fine-tuning LLMs improves performance, but existing measures still fall short. This study highlights the need for soft similarity measures tailored to FCM extraction, advancing collective intelligence modeling with NLP.

**URL** : <http://arxiv.org/pdf/2409.18911v1>

**Published** : 2024-09-27

## IDGen: Item Discrimination Induced Prompt Generation for LLM Evaluation

**Authors** : Fan Lin, Shuyi Xie, Yong Dai, Wenlin Yao, Tianjiao Lang, Zishan Xu, Zhichao Hu, Xiao Xiao, Yuhong Liu, Yu Zhang

**Abstract** : As Large Language Models (LLMs) grow increasingly adept at managing complex tasks, the evaluation set must keep pace with these advancements to ensure it remains sufficiently discriminative. Item Discrimination (ID) theory, which is widely used in educational assessment, measures the ability of individual test items to differentiate between high and low performers. Inspired by this theory, we propose an ID-induced prompt synthesis framework for evaluating LLMs to ensure the evaluation set can continually update and refine according to model abilities. Our data synthesis framework prioritizes both breadth and specificity. It can generate prompts that comprehensively evaluate the capabilities of LLMs while revealing meaningful performance differences between models, allowing for effective discrimination of their relative strengths and weaknesses across various tasks and domains. To produce high-quality data, we incorporate a self-correct mechanism into our generalization framework, and develop two models to predict prompt discrimination and difficulty score to facilitate our data synthesis framework, contributing valuable tools to evaluation data synthesis research. We apply our generated data to evaluate five SOTA models. Our data achieves an average score of 51.92, accompanied by a variance of 10.06. By contrast, previous works (i.e., SELF-INSTRUCT and WizardLM) obtain an average score exceeding 67, with a variance below 3.2. The results demonstrate that the data generated by our framework is more challenging and discriminative compared to previous works. We will release a dataset of over 3,000 carefully crafted prompts to facilitate evaluation research of LLMs.

**URL** : <http://arxiv.org/pdf/2409.18892v1>

**Published** : 2024-09-27

## Mitigating Selection Bias with Node Pruning and Auxiliary Options

**Authors** : Hyeong Kyu Choi, Weijie Xu, Chi Xue, Stephanie Eckman, Chandan K. Reddy

**Abstract** : Large language models (LLMs) often show unwarranted preference for certain choice options when responding to multiple-choice questions, posing significant reliability concerns in LLM-automated systems. To mitigate this selection bias problem, previous solutions utilized debiasing methods to adjust the model's input and/or output. Our work, in contrast, investigates the model's internal representation of the selection bias. Specifically, we introduce a novel debiasing approach, Bias Node Pruning (BNP), which eliminates the linear layer parameters that contribute to the bias. Furthermore, we present Auxiliary Option Injection (AOI), a simple yet effective input modification technique for debiasing, which is compatible even with black-box LLMs. To provide a more systematic evaluation of selection bias, we review existing metrics and introduce Choice Kullback-Leibler Divergence (CKLD), which addresses the insensitivity of the commonly used metrics to label imbalance. Experiments show that our methods are robust and adaptable across various datasets when applied to three LLMs.

**URL** : <http://arxiv.org/pdf/2409.18857v1>

**Published** : 2024-09-27

## LLMs4Synthesis: Leveraging Large Language Models for Scientific Synthesis

**Authors** : Hamed Babaei Giglou, Jennifer D'Souza, Sören Auer

**Abstract** : In response to the growing complexity and volume of scientific literature, this paper introduces the LLMs4Synthesis framework, designed to enhance the capabilities of Large Language Models (LLMs) in generating high-quality scientific syntheses. This framework addresses the need for rapid, coherent, and contextually rich integration of scientific insights, leveraging both open-source and proprietary LLMs. It also examines the effectiveness of LLMs in evaluating the integrity and reliability of these syntheses, alleviating inadequacies in current quantitative metrics. Our study contributes to this field by developing a novel methodology for processing scientific papers, defining new synthesis types, and establishing nine detailed quality criteria for evaluating syntheses. The integration of LLMs with reinforcement learning and AI feedback is proposed to optimize synthesis quality, ensuring alignment with established criteria. The LLMs4Synthesis framework and its components are made available, promising to enhance both the generation and evaluation processes in scientific research synthesis.

**URL** : <http://arxiv.org/pdf/2409.18812v1>

**Published** : 2024-09-27

## Esports Debut as a Medal Event at 2023 Asian Games: Exploring Public Perceptions with BERTopic and GPT-4 Topic Fine-Tuning

**Authors** : Tyreal Yizhou Qian, Bo Yu, Weizhe Li, Chenglong Xu

**Abstract** : This study examined the public opinions of esports at the 2023 Asian Games and value co-creation during the event using an LLM-enhanced BERTopic modeling analysis. We identified five major themes representing public perceptions, as well as how major stakeholders co-created value within and beyond the esports ecosystem. Key findings highlighted the strategic use of social media marketing to influence public opinion and promote esports events and brands, emphasizing the importance of event logistics and infrastructure. Additionally, the study revealed the co-creation value contributed by stakeholders outside the traditional esports ecosystem, particularly in promoting national representation and performance. Our findings supported the ongoing efforts to legitimize esports as a sport, noting that mainstream recognition remains a challenge. The inclusion of esports as a medal event showcased broader acceptance and helped mitigate negative public perceptions. Moreover, contributions from non-traditional stakeholders underscored the value of cross-subcultural collaborations in esports.

**URL** : <http://arxiv.org/pdf/2409.18798v1>

**Published** : 2024-09-27

## A Survey on the Honesty of Large Language Models

**Authors** : Siheng Li, Cheng Yang, Taiqiang Wu, Chufan Shi, Yuji Zhang, Xinyu Zhu, Zesen Cheng, Deng Cai, Mo Yu, Lemao Liu, Jie Zhou, Yujiu Yang, Ngai Wong, Xixin Wu, Wai Lam

**Abstract** : Honesty is a fundamental principle for aligning large language models (LLMs) with human values, requiring these models to recognize what they know and don't know and be able to faithfully express their knowledge. Despite promising, current LLMs still exhibit significant dishonest behaviors, such as confidently presenting wrong answers or failing to express what they know. In addition, research on the honesty of LLMs also faces challenges, including varying definitions of honesty, difficulties in distinguishing between known and unknown knowledge, and a lack of comprehensive understanding of related research. To address these issues, we provide a survey on the honesty of LLMs, covering its clarification, evaluation approaches, and strategies for improvement. Moreover, we offer insights for future research, aiming to inspire further exploration in this important area.

**URL** : <http://arxiv.org/pdf/2409.18786v1>

**Published** : 2024-09-27

## Charting the Future: Using Chart Question-Answering for Scalable Evaluation of LLM-Driven Data Visualizations

**Authors** : James Ford, Xingmeng Zhao, Dan Schumacher, Anthony Rios

**Abstract** : We propose a novel framework that leverages Visual Question Answering (VQA) models to automate the evaluation of LLM-generated data visualizations. Traditional evaluation methods often rely on human judgment, which is costly and unscalable, or focus solely on data accuracy, neglecting the effectiveness of visual communication. By employing VQA models, we assess data representation quality and the general communicative clarity of charts. Experiments were conducted using two leading VQA benchmark datasets, ChartQA and PlotQA, with visualizations generated by OpenAI's GPT-3.5 Turbo and Meta's Llama 3.1 70B-Instruct models. Our results indicate that LLM-generated charts do not match the accuracy of the original non-LLM-generated charts based on VQA performance measures. Moreover, while our results demonstrate that few-shot prompting significantly boosts the accuracy of chart generation, considerable progress remains to be made before LLMs can fully match the precision of human-generated graphs. This underscores the importance of our work, which expedites the research process by enabling rapid iteration without the need for human annotation, thus accelerating advancements in this field.

**URL** : <http://arxiv.org/pdf/2409.18764v1>

**Published** : 2024-09-27

## OpenObject-NAV: Open-Vocabulary Object-Oriented Navigation Based on Dynamic Carrier-Relationship Scene Graph

**Authors** : Yujie Tang, Meiling Wang, Yinan Deng, Zibo Zheng, Jiagui Zhong, Yufeng Yue

**Abstract** : In everyday life, frequently used objects like cups often have unfixed positions and multiple instances within the same category, and their carriers frequently change as well. As a result, it becomes challenging for a robot to efficiently navigate to a specific instance. To tackle this challenge, the robot must capture and update scene changes and plans continuously. However, current object navigation approaches primarily focus on semantic-level and lack the ability to dynamically update scene representation. This paper captures the relationships between frequently used objects and their static carriers. It constructs an open-vocabulary Carrier-Relationship Scene Graph (CRSG) and updates the carrying status during robot navigation to reflect the dynamic changes of the scene. Based on the CRSG, we further propose an instance navigation strategy that models the navigation process as a Markov Decision Process. At each step, decisions are informed by Large Language Model's commonsense knowledge and visual-language feature similarity. We designed a series of long-sequence navigation tasks for frequently used everyday items in the Habitat simulator. The results demonstrate that by updating the CRSG, the robot can efficiently navigate to moved targets. Additionally, we deployed our algorithm on a real robot and validated its practical effectiveness.

**URL** : <http://arxiv.org/pdf/2409.18743v1>

**Published** : 2024-09-27

## Read Over the Lines: Attacking LLMs and Toxicity Detection Systems with ASCII Art to Mask Profanity

**Authors** : Sergey Berezin, Reza Farahbakhsh, Noel Crespi

**Abstract** : We introduce a novel family of adversarial attacks that exploit the inability of language models to interpret ASCII art. To evaluate these attacks, we propose the ToxASCII benchmark and develop two custom ASCII art fonts: one leveraging special tokens and another using text-filled letter shapes. Our attacks achieve a perfect 1.0 Attack Success Rate across ten models, including OpenAI's o1-preview and LLaMA 3.1. Warning: this paper contains examples of toxic language used for research purposes.

**URL** : <http://arxiv.org/pdf/2409.18708v1>

**Published** : 2024-09-27

## Beyond Single-Audio: Advancing Multi-Audio Processing in Audio Large Language Models

**Authors** : Yiming Chen, Xianghu Yue, Xiaoxue Gao, Chen Zhang, Luis Fernando D'Haro, Robby T. Tan, Haizhou Li

**Abstract** : Various audio-LLMs (ALLMs) have been explored recently for tackling different audio tasks simultaneously using a single, unified model. While existing evaluations of ALLMs primarily focus on single-audio tasks, real-world applications often involve processing multiple audio streams simultaneously. To bridge this gap, we propose the first multi-audio evaluation (MAE) benchmark that consists of 20 datasets from 11 multi-audio tasks encompassing both speech and sound scenarios. Comprehensive experiments on MAE demonstrate that the existing ALLMs, while being powerful in comprehending primary audio elements in individual audio inputs, struggling to handle multi-audio scenarios. To this end, we propose a novel multi-audio-LLM (MALLM) to capture audio context among multiple similar audios using discriminative learning on our proposed synthetic data. The results demonstrate that the proposed MALLM outperforms all baselines and achieves high data efficiency using synthetic data without requiring human annotations. The proposed MALLM opens the door for ALLMs towards multi-audio processing era and brings us closer to replicating human auditory capabilities in machines.

**URL** : <http://arxiv.org/pdf/2409.18680v1>

**Published** : 2024-09-27

## "Why" Has the Least Side Effect on Model Editing

**Authors** : Tsung-Hsuan Pan, Chung-Chi Chen, Hen-Hsen Huang, Hsin-Hsi Chen

**Abstract** : Training large language models (LLMs) from scratch is an expensive endeavor, particularly as world knowledge continually evolves. To maintain relevance and accuracy of LLMs, model editing has emerged as a pivotal research area. While these methods hold promise, they can also produce unintended side effects. Their underlying factors and causes remain largely unexplored. This paper delves into a critical factor-question type-by categorizing model editing questions. Our findings reveal that the extent of performance degradation varies significantly across different question types, providing new insights for experimental design in knowledge editing. Furthermore, we investigate whether insights from smaller models can be extrapolated to larger models. Our results indicate discrepancies in findings between models of different sizes, suggesting that insights from smaller models may not necessarily apply to larger models. Additionally, we examine the impact of batch size on side effects, discovering that increasing the batch size can mitigate performance drops.

**URL** : <http://arxiv.org/pdf/2409.18679v1>

**Published** : 2024-09-27

## Rehearsing Answers to Probable Questions with Perspective-Taking

**Authors** : Yung-Yu Shih, Ziwei Xu, Hiroya Takamura, Yun-Nung Chen, Chung-Chi Chen

**Abstract** : Question answering (QA) has been a long-standing focus in the NLP field, predominantly addressing reading comprehension and common sense QA. However, scenarios involving the preparation of answers to probable questions during professional oral presentations remain underexplored. In this paper, we pioneer the examination of this crucial yet overlooked topic by utilizing real-world QA conversation transcripts between company managers and professional analysts. We explore the proposed task using three causal knowledge graphs (KGs) and three large language models (LLMs). This work provides foundational insights into the application of LLMs in professional QA scenarios, highlighting the importance of causal KGs and perspective-taking in generating effective responses.

**URL** : <http://arxiv.org/pdf/2409.18678v1>

**Published** : 2024-09-27

## Not the Silver Bullet: LLM-enhanced Programming Error Messages are Ineffective in Practice

**Authors** : Eddie Antonio Santos, Brett A. Becker

**Abstract** : The sudden emergence of large language models (LLMs) such as ChatGPT has had a disruptive impact throughout the computing education community. LLMs have been shown to excel at producing correct code to CS1 and CS2 problems, and can even act as friendly assistants to students learning how to code. Recent work shows that LLMs demonstrate unequivocally superior results in being able to explain and resolve compiler error messages -- for decades, one of the most frustrating parts of learning how to code. However, LLM-generated error message explanations have only been assessed by expert programmers in artificial conditions. This work sought to understand how novice programmers resolve programming error messages (PEMs) in a more realistic scenario. We ran a within-subjects study with $n$ = 106 participants in which students were tasked to fix six buggy C programs. For each program, participants were randomly assigned to fix the problem using either a stock compiler error message, an expert-handwritten error message, or an error message explanation generated by GPT-4. Despite promising evidence on synthetic benchmarks, we found that GPT-4 generated error messages outperformed conventional compiler error messages in only 1 of the 6 tasks, measured by students' time-to-fix each problem. Handwritten explanations still outperform LLM and conventional error messages, both on objective and subjective measures.

**URL** : <http://arxiv.org/pdf/2409.18661v1>

**Published** : 2024-09-27

## When SAM2 Meets Video Camouflaged Object Segmentation: A Comprehensive Evaluation and Adaptation

**Authors** : Yuli Zhou, Guolei Sun, Yawei Li, Luca Benini, Ender Konukoglu

**Abstract** : This study investigates the application and performance of the Segment Anything Model 2 (SAM2) in the challenging task of video camouflaged object segmentation (VCOS). VCOS involves detecting objects that blend seamlessly in the surroundings for videos, due to similar colors and textures, poor light conditions, etc. Compared to the objects in normal scenes, camouflaged objects are much more difficult to detect. SAM2, a video foundation model, has shown potential in various tasks. But its effectiveness in dynamic camouflaged scenarios remains under-explored. This study presents a comprehensive study on SAM2's ability in VCOS. First, we assess SAM2's performance on camouflaged video datasets using different models and prompts (click, box, and mask). Second, we explore the integration of SAM2 with existing multimodal large language models (MLLMs) and VCOS methods. Third, we specifically adapt SAM2 by fine-tuning it on the video camouflaged dataset. Our comprehensive experiments demonstrate that SAM2 has excellent zero-shot ability of detecting camouflaged objects in videos. We also show that this ability could be further improved by specifically adjusting SAM2's parameters for VCOS. The code will be available at <https://github.com/zhoustan/SAM2-VCOS>

**URL** : <http://arxiv.org/pdf/2409.18653v1>

**Published** : 2024-09-27

## Model-based Preference Optimization in Abstractive Summarization without Human Feedback

**Authors** : Jaepill Choi, Kyubyung Chae, Jiwoo Song, Yohan Jo, Taesup Kim

**Abstract** : In abstractive summarization, the challenge of producing concise and accurate summaries arises from the vast amount of information contained in the source document. Consequently, although Large Language Models (LLMs) can generate fluent text, they often introduce inaccuracies by hallucinating content not found in the original source. While supervised fine-tuning methods that maximize likelihood contribute to this issue, they do not consistently enhance the faithfulness of the summaries. Preference-based optimization methods, such as Direct Preference Optimization (DPO), can further refine the model to align with human preferences. However, these methods still heavily depend on costly human feedback. In this work, we introduce a novel and straightforward approach called Model-based Preference Optimization (MPO) to fine-tune LLMs for improved summarization abilities without any human feedback. By leveraging the model's inherent summarization capabilities, we create a preference dataset that is fully generated by the model using different decoding strategies. Our experiments on standard summarization datasets and various metrics demonstrate that our proposed MPO significantly enhances the quality of generated summaries without relying on human feedback.

**URL** : <http://arxiv.org/pdf/2409.18618v1>

**Published** : 2024-09-27

## Do LLMs suffer from Multi-Party Hangover? A Diagnostic Approach to Addressee Recognition and Response Selection in Conversations

**Authors** : Nicolò Penzo, Maryam Sajedinia, Bruno Lepri, Sara Tonelli, Marco Guerini

**Abstract** : Assessing the performance of systems to classify Multi-Party Conversations (MPC) is challenging due to the interconnection between linguistic and structural characteristics of conversations. Conventional evaluation methods often overlook variances in model behavior across different levels of structural complexity on interaction graphs. In this work, we propose a methodological pipeline to investigate model performance across specific structural attributes of conversations. As a proof of concept we focus on Response Selection and Addressee Recognition tasks, to diagnose model weaknesses. To this end, we extract representative diagnostic subdatasets with a fixed number of users and a good structural variety from a large and open corpus of online MPCs. We further frame our work in terms of data minimization, avoiding the use of original usernames to preserve privacy, and propose alternatives to using original text messages. Results show that response selection relies more on the textual content of conversations, while addressee recognition requires capturing their structural dimension. Using an LLM in a zero-shot setting, we further highlight how sensitivity to prompt variations is task-dependent.

**URL** : <http://arxiv.org/pdf/2409.18602v1>

**Published** : 2024-09-27

## ASAG2024: A Combined Benchmark for Short Answer Grading

**Authors** : Gérôme Meyer, Philip Breuer, Jonathan Fürst

**Abstract** : Open-ended questions test a more thorough understanding than closed-ended questions and are often a preferred assessment method. However, open-ended questions are tedious to grade and subject to personal bias. Therefore, there have been efforts to speed up the grading process through automation. Short Answer Grading (SAG) systems aim to automatically score students' answers. Despite growth in SAG methods and capabilities, there exists no comprehensive short-answer grading benchmark across different subjects, grading scales, and distributions. Thus, it is hard to assess the capabilities of current automated grading methods in terms of their generalizability. In this preliminary work, we introduce the combined ASAG2024 benchmark to facilitate the comparison of automated grading systems. Combining seven commonly used short-answer grading datasets in a common structure and grading scale. For our benchmark, we evaluate a set of recent SAG methods, revealing that while LLM-based approaches reach new high scores, they still are far from reaching human performance. This opens up avenues for future research on human-machine SAG systems.

**URL** : <http://arxiv.org/pdf/2409.18596v1>

**Published** : 2024-09-27

## "Oh LLM, I'm Asking Thee, Please Give Me a Decision Tree": Zero-Shot Decision Tree Induction and Embedding with Large Language Models

**Authors** : Ricardo Knauer, Mario Koddenbrock, Raphael Wallsberger, Nicholas M. Brisson, Georg N. Duda, Deborah Falla, David W. Evans, Erik Rodner

**Abstract** : Large language models (LLMs) provide powerful means to leverage prior knowledge for predictive modeling when data is limited. In this work, we demonstrate how LLMs can use their compressed world knowledge to generate intrinsically interpretable machine learning models, i.e., decision trees, without any training data. We find that these zero-shot decision trees can surpass data-driven trees on some small-sized tabular datasets and that embeddings derived from these trees perform on par with data-driven tree-based embeddings on average. Our knowledge-driven decision tree induction and embedding approaches therefore serve as strong new baselines for data-driven machine learning methods in the low-data regime.

**URL** : <http://arxiv.org/pdf/2409.18594v1>

**Published** : 2024-09-27

## Hit the Sweet Spot! Span-Level Ensemble for Large Language Models

**Authors** : Yangyifan Xu, Jianghao Chen, Junhong Wu, Jiajun Zhang

**Abstract** : Ensembling various LLMs to unlock their complementary potential and leverage their individual strengths is highly valuable. Previous studies typically focus on two main paradigms: sample-level and token-level ensembles. Sample-level ensemble methods either select or blend fully generated outputs, which hinders dynamic correction and enhancement of outputs during the generation process. On the other hand, token-level ensemble methods enable real-time correction through fine-grained ensemble at each generation step. However, the information carried by an individual token is quite limited, leading to suboptimal decisions at each step. To address these issues, we propose SweetSpan, a span-level ensemble method that effectively balances the need for real-time adjustments and the information required for accurate ensemble decisions. Our approach involves two key steps: First, we have each candidate model independently generate candidate spans based on the shared prefix. Second, we calculate perplexity scores to facilitate mutual evaluation among the candidate models and achieve robust span selection by filtering out unfaithful scores. To comprehensively evaluate ensemble methods, we propose a new challenging setting (ensemble models with significant performance gaps) in addition to the standard setting (ensemble the best-performing models) to assess the performance of model ensembles in more realistic scenarios. Experimental results in both standard and challenging settings across various language generation tasks demonstrate the effectiveness, robustness, and versatility of our approach compared with previous ensemble methods.

**URL** : <http://arxiv.org/pdf/2409.18583v1>

**Published** : 2024-09-27

## Experimental Evaluation of Machine Learning Models for Goal-oriented Customer Service Chatbot with Pipeline Architecture

**Authors** : Nurul Ain Nabilah Mohd Isa, Siti Nuraishah Agos Jawaddi, Azlan Ismail

**Abstract** : Integrating machine learning (ML) into customer service chatbots enhances their ability to understand and respond to user queries, ultimately improving service performance. However, they may appear artificial to some users and affecting customer experience. Hence, meticulous evaluation of ML models for each pipeline component is crucial for optimizing performance, though differences in functionalities can lead to unfair comparisons. In this paper, we present a tailored experimental evaluation approach for goal-oriented customer service chatbots with pipeline architecture, focusing on three key components: Natural Language Understanding (NLU), dialogue management (DM), and Natural Language Generation (NLG). Our methodology emphasizes individual assessment to determine optimal ML models. Specifically, we focus on optimizing hyperparameters and evaluating candidate models for NLU (utilizing BERT and LSTM), DM (employing DQN and DDQN), and NLG (leveraging GPT-2 and DialoGPT). The results show that for the NLU component, BERT excelled in intent detection whereas LSTM was superior for slot filling. For the DM component, the DDQN model outperformed DQN by achieving fewer turns, higher rewards, as well as greater success rates. For NLG, the large language model GPT-2 surpassed DialoGPT in BLEU, METEOR, and ROUGE metrics. These findings aim to provide a benchmark for future research in developing and optimizing customer service chatbots, offering valuable insights into model performance and optimal hyperparameters.

**URL** : <http://arxiv.org/pdf/2409.18568v1>

**Published** : 2024-09-27

## Research on Predicting Public Opinion Event Heat Levels Based on Large Language Models

**Authors** : Yi Ren, Tianyi Zhang, Weibin Li, DuoMu Zhou, Chenhao Qin, FangCheng Dong

**Abstract** : In recent years, with the rapid development of large language models, serval models such as GPT-4o have demonstrated extraordinary capabilities, surpassing human performance in various language tasks. As a result, many researchers have begun exploring their potential applications in the field of public opinion analysis. This study proposes a novel large-language-models-based method for public opinion event heat level prediction. First, we preprocessed and classified 62,836 Chinese hot event data collected between July 2022 and December 2023. Then, based on each event's online dissemination heat index, we used the MiniBatchKMeans algorithm to automatically cluster the events and categorize them into four heat levels (ranging from low heat to very high heat). Next, we randomly selected 250 events from each heat level, totalling 1,000 events, to build the evaluation dataset. During the evaluation process, we employed various large language models to assess their accuracy in predicting event heat levels in two scenarios: without reference cases and with similar case references. The results showed that GPT-4o and DeepseekV2 performed the best in the latter case, achieving prediction accuracies of 41.4% and 41.5%, respectively. Although the overall prediction accuracy remains relatively low, it is worth noting that for low-heat (Level 1) events, the prediction accuracies of these two models reached 73.6% and 70.4%, respectively. Additionally, the prediction accuracy showed a downward trend from Level 1 to Level 4, which correlates with the uneven distribution of data across the heat levels in the actual dataset. This suggests that with the more robust dataset, public opinion event heat level prediction based on large language models will have significant research potential for the future.

**URL** : <http://arxiv.org/pdf/2409.18548v1>

**Published** : 2024-09-27

## Align$^2$LLaVA: Cascaded Human and Large Language Model Preference Alignment for Multi-modal Instruction Curation

**Authors** : Hongzhe Huang, Zhewen Yu, Jiang Liu, Li Cai, Dian Jiao, Wenqiao Zhang, Siliang Tang, Juncheng Li, Hao Jiang, Haoyuan Li, Yueting Zhuang

**Abstract** : Recent advances in Multi-modal Large Language Models (MLLMs), such as LLaVA-series models, are driven by massive machine-generated instruction-following data tuning. Such automatic instruction collection pipelines, however, inadvertently introduce significant variability in data quality. This paper introduces a novel instruction curation algorithm, derived from two unique perspectives, human and LLM preference alignment, to compress this vast corpus of machine-generated multimodal instructions to a compact and high-quality form: (i) For human preference alignment, we have collected a machine-generated multimodal instruction dataset and established a comprehensive set of both subjective and objective criteria to guide the data quality assessment critically from human experts. By doing so, a reward model was trained on the annotated dataset to internalize the nuanced human understanding of instruction alignment. (ii) For LLM preference alignment, given the instruction selected by the reward model, we propose leveraging the inner LLM used in MLLM to align the writing style of visual instructions with that of the inner LLM itself, resulting in LLM-aligned instruction improvement. Extensive experiments demonstrate that we can maintain or even improve model performance by compressing synthetic multimodal instructions by up to 90%. Impressively, by aggressively reducing the total training sample size from 158k to 14k (9$\times$ smaller), our model consistently outperforms its full-size dataset counterpart across various MLLM benchmarks. Our project is available at <https://github.com/DCDmllm/Align2LLaVA>.

**URL** : <http://arxiv.org/pdf/2409.18541v1>

**Published** : 2024-09-27

## A Survey on Complex Tasks for Goal-Directed Interactive Agents

**Authors** : Mareike Hartmann, Alexander Koller

**Abstract** : Goal-directed interactive agents, which autonomously complete tasks through interactions with their environment, can assist humans in various domains of their daily lives. Recent advances in large language models (LLMs) led to a surge of new, more and more challenging tasks to evaluate such agents. To properly contextualize performance across these tasks, it is imperative to understand the different challenges they pose to agents. To this end, this survey compiles relevant tasks and environments for evaluating goal-directed interactive agents, structuring them along dimensions relevant for understanding current obstacles. An up-to-date compilation of relevant resources can be found on our project website: <https://coli-saar.github.io/interactive-agents>.

**URL** : <http://arxiv.org/pdf/2409.18538v1>

**Published** : 2024-09-27

## Do We Need Domain-Specific Embedding Models? An Empirical Investigation

**Authors** : Yixuan Tang, Yi Yang

**Abstract** : Embedding models play a crucial role in representing and retrieving information across various NLP applications. Recent advancements in Large Language Models (LLMs) have further enhanced the performance of embedding models, which are trained on massive amounts of text covering almost every domain. These models are often benchmarked on general-purpose datasets like Massive Text Embedding Benchmark (MTEB), where they demonstrate superior performance. However, a critical question arises: Is the development of domain-specific embedding models necessary when general-purpose models are trained on vast corpora that already include specialized domain texts? In this paper, we empirically investigate this question, choosing the finance domain as an example. We introduce the Finance Massive Text Embedding Benchmark (FinMTEB), a counterpart to MTEB that consists of financial domain-specific text datasets. We evaluate the performance of seven state-of-the-art embedding models on FinMTEB and observe a significant performance drop compared to their performance on MTEB. To account for the possibility that this drop is driven by FinMTEB's higher complexity, we propose four measures to quantify dataset complexity and control for this factor in our analysis. Our analysis provides compelling evidence that state-of-the-art embedding models struggle to capture domain-specific linguistic and semantic patterns, even when trained on large general-purpose corpora. This study sheds light on the necessity of developing domain-specific embedding models in the LLM era, offering valuable insights for researchers and practitioners.

**URL** : <http://arxiv.org/pdf/2409.18511v1>

**Published** : 2024-09-27

## Evaluation of OpenAI o1: Opportunities and Challenges of AGI

**Authors** : Tianyang Zhong, Zhengliang Liu, Yi Pan, Yutong Zhang, Yifan Zhou, Shizhe Liang, Zihao Wu, Yanjun Lyu, Peng Shu, Xiaowei Yu, Chao Cao, Hanqi Jiang, Hanxu Chen, Yiwei Li, Junhao Chen, Huawen Hu, Yihen Liu, Huaqin Zhao, Shaochen Xu, Haixing Dai, Lin Zhao, Ruidong Zhang, Wei Zhao, Zhenyuan Yang, Jingyuan Chen, Peilong Wang, Wei Ruan, Hui Wang, Huan Zhao, Jing Zhang, Yiming Ren, Shihuan Qin, Tong Chen, Jiaxi Li, Arif Hassan Zidan, Afrar Jahin, Minheng Chen, Sichen Xia, Jason Holmes, Yan Zhuang, Jiaqi Wang, Bochen Xu, Weiran Xia, Jichao Yu, Kaibo Tang, Yaxuan Yang, Bolun Sun, Tao Yang, Guoyu Lu, Xianqiao Wang, Lilong Chai, He Li, Jin Lu, Lichao Sun, Xin Zhang, Bao Ge, Xintao Hu, Lian Zhang, Hua Zhou, Lu Zhang, Shu Zhang, Ninghao Liu, Bei Jiang, Linglong Kong, Zhen Xiang, Yudan Ren, Jun Liu, Xi Jiang, Yu Bao, Wei Zhang, Xiang Li, Gang Li, Wei Liu, Dinggang Shen, Andrea Sikora, Xiaoming Zhai, Dajiang Zhu, Tianming Liu

**Abstract** : This comprehensive study evaluates the performance of OpenAI's o1-preview large language model across a diverse array of complex reasoning tasks, spanning multiple domains, including computer science, mathematics, natural sciences, medicine, linguistics, and social sciences. Through rigorous testing, o1-preview demonstrated remarkable capabilities, often achieving human-level or superior performance in areas ranging from coding challenges to scientific reasoning and from language processing to creative problem-solving. Key findings include: -83.3% success rate in solving complex competitive programming problems, surpassing many human experts. -Superior ability in generating coherent and accurate radiology reports, outperforming other evaluated models. -100% accuracy in high school-level mathematical reasoning tasks, providing detailed step-by-step solutions. -Advanced natural language inference capabilities across general and specialized domains like medicine. -Impressive performance in chip design tasks, outperforming specialized models in areas such as EDA script generation and bug analysis. -Remarkable proficiency in anthropology and geology, demonstrating deep understanding and reasoning in these specialized fields. -Strong capabilities in quantitative investing. O1 has comprehensive financial knowledge and statistical modeling skills. -Effective performance in social media analysis, including sentiment analysis and emotion recognition. The model excelled particularly in tasks requiring intricate reasoning and knowledge integration across various fields. While some limitations were observed, including occasional errors on simpler problems and challenges with certain highly specialized concepts, the overall results indicate significant progress towards artificial general intelligence.

**URL** : <http://arxiv.org/pdf/2409.18486v1>

**Published** : 2024-09-27

## Leveraging Long-Context Large Language Models for Multi-Document Understanding and Summarization in Enterprise Applications

**Authors** : Aditi Godbole, Jabin Geevarghese George, Smita Shandilya

**Abstract** : The rapid increase in unstructured data across various fields has made multi-document comprehension and summarization a critical task. Traditional approaches often fail to capture relevant context, maintain logical consistency, and extract essential information from lengthy documents. This paper explores the use of Long-context Large Language Models (LLMs) for multi-document summarization, demonstrating their exceptional capacity to grasp extensive connections, provide cohesive summaries, and adapt to various industry domains and integration with enterprise applications/systems. The paper discusses the workflow of multi-document summarization for effectively deploying long-context LLMs, supported by case studies in legal applications, enterprise functions such as HR, finance, and sourcing, as well as in the medical and news domains. These case studies show notable enhancements in both efficiency and accuracy. Technical obstacles, such as dataset diversity, model scalability, and ethical considerations like bias mitigation and factual accuracy, are carefully analyzed. Prospective research avenues are suggested to augment the functionalities and applications of long-context LLMs, establishing them as pivotal tools for transforming information processing across diverse sectors and enterprise applications.

**URL** : <http://arxiv.org/pdf/2409.18454v1>

**Published** : 2024-09-27

## Exploring Language Model Generalization in Low-Resource Extractive QA

**Authors** : Saptarshi Sengupta, Wenpeng Yin, Preslav Nakov, Shreya Ghosh, Suhang Wang

**Abstract** : In this paper, we investigate Extractive Question Answering (EQA) with Large Language Models (LLMs) under domain drift, i.e., can LLMs generalize well to closed-domains that require specific knowledge such as medicine and law in a zero-shot fashion without additional in-domain training? To this end, we devise a series of experiments to empirically explain the performance gap. Our findings suggest that: a) LLMs struggle with dataset demands of closed-domains such as retrieving long answer-spans; b) Certain LLMs, despite showing strong overall performance, display weaknesses in meeting basic requirements as discriminating between domain-specific senses of words which we link to pre-processing decisions; c) Scaling model parameters is not always effective for cross-domain generalization; and d) Closed-domain datasets are quantitatively much different than open-domain EQA datasets and current LLMs struggle to deal with them. Our findings point out important directions for improving existing LLMs.

**URL** : <http://arxiv.org/pdf/2409.18446v1>

**Published** : 2024-09-27

## Easy2Hard-Bench: Standardized Difficulty Labels for Profiling LLM Performance and Generalization

**Authors** : Mucong Ding, Chenghao Deng, Jocelyn Choo, Zichu Wu, Aakriti Agrawal, Avi Schwarzschild, Tianyi Zhou, Tom Goldstein, John Langford, Anima Anandkumar, Furong Huang

**Abstract** : While generalization over tasks from easy to hard is crucial to profile language models (LLMs), the datasets with fine-grained difficulty annotations for each problem across a broad range of complexity are still blank. Aiming to address this limitation, we present Easy2Hard-Bench, a consistently formatted collection of 6 benchmark datasets spanning various domains, such as mathematics and programming problems, chess puzzles, and reasoning questions. Each problem within these datasets is annotated with numerical difficulty scores. To systematically estimate problem difficulties, we collect abundant performance data on attempts to each problem by humans in the real world or LLMs on the prominent leaderboard. Leveraging the rich performance data, we apply well-established difficulty ranking systems, such as Item Response Theory (IRT) and Glicko-2 models, to uniformly assign numerical difficulty scores to problems. Moreover, datasets in Easy2Hard-Bench distinguish themselves from previous collections by a higher proportion of challenging problems. Through extensive experiments with six state-of-the-art LLMs, we provide a comprehensive analysis of their performance and generalization capabilities across varying levels of difficulty, with the aim of inspiring future research in LLM generalization. The datasets are available at <https://huggingface.co/datasets/furonghuang-lab/Easy2Hard-Bench>.

**URL** : <http://arxiv.org/pdf/2409.18433v1>

**Published** : 2024-09-27

## VickreyFeedback: Cost-efficient Data Construction for Reinforcement Learning from Human Feedback

**Authors** : Guoxi Zhang, Jiuding Duan

**Abstract** : This paper addresses the cost-efficiency aspect of Reinforcement Learning from Human Feedback (RLHF). RLHF leverages datasets of human preferences over outputs of large language models (LLM) to instill human expectations into LLMs. While preference annotation comes with a monetized cost, the economic utility of a preference dataset has not been considered by far. What exacerbates this situation is that given complex intransitive or cyclic relationships in preference datasets, existing algorithms for fine-tuning LLMs are still far from capturing comprehensive preferences. This raises severe cost-efficiency concerns in production environments, where preference data accumulate over time. In this paper, we see the fine-tuning of LLMs as a monetized economy and introduce an auction mechanism to improve the efficiency of the preference data collection in dollar terms. We show that introducing an auction mechanism can play an essential role in enhancing the cost-efficiency of RLHF while maintaining satisfactory model performance. Experimental results demonstrate that our proposed auction-based protocol is cost-efficient for fine-tuning LLMs by concentrating on high-quality feedback.

**URL** : <http://arxiv.org/pdf/2409.18417v1>

**Published** : 2024-09-27

## SciDFM: A Large Language Model with Mixture-of-Experts for Science

**Authors** : Liangtai Sun, Danyu Luo, Da Ma, Zihan Zhao, Baocai Chen, Zhennan Shen, Su Zhu, Lu Chen, Xin Chen, Kai Yu

**Abstract** : Recently, there has been a significant upsurge of interest in leveraging large language models (LLMs) to assist scientific discovery. However, most LLMs only focus on general science, while they lack domain-specific knowledge, such as chemical molecules and amino acid sequences. To bridge these gaps, we introduce SciDFM, a mixture-of-experts LLM, which is trained from scratch and is able to conduct college-level scientific reasoning and understand molecules and amino acid sequences. We collect a large-scale training corpus containing numerous scientific papers and books from different disciplines as well as data from domain-specific databases. We further fine-tune the pre-trained model on lots of instruction data to improve performances on downstream benchmarks. From experiment results, we show that SciDFM achieves strong performance on general scientific benchmarks such as SciEval and SciQ, and it reaches a SOTA performance on domain-specific benchmarks among models of similar size. We further analyze the expert layers and show that the results of expert selection vary with data from different disciplines. To benefit the broader research community, we open-source SciDFM at <https://huggingface.co/OpenDFM/SciDFM-MoE-A5.6B-v1.0>.

**URL** : <http://arxiv.org/pdf/2409.18412v1>

**Published** : 2024-09-27

## Code Vulnerability Repair with Large Language Model using Context-Aware Prompt Tuning

**Authors** : Arshiya Khan, Guannan Liu, Xing Gao

**Abstract** : Large Language Models (LLMs) have shown significant challenges in detecting and repairing vulnerable code, particularly when dealing with vulnerabilities involving multiple aspects, such as variables, code flows, and code structures. In this study, we utilize GitHub Copilot as the LLM and focus on buffer overflow vulnerabilities. Our experiments reveal a notable gap in Copilot's abilities when dealing with buffer overflow vulnerabilities, with a 76% vulnerability detection rate but only a 15% vulnerability repair rate. To address this issue, we propose context-aware prompt tuning techniques designed to enhance LLM performance in repairing buffer overflow. By injecting a sequence of domain knowledge about the vulnerability, including various security and code contexts, we demonstrate that Copilot's successful repair rate increases to 63%, representing more than four times the improvement compared to repairs without domain knowledge.

**URL** : <http://arxiv.org/pdf/2409.18395v1>

**Published** : 2024-09-27

## MultiClimate: Multimodal Stance Detection on Climate Change Videos

**Authors** : Jiawen Wang, Longfei Zuo, Siyao Peng, Barbara Plank

**Abstract** : Climate change (CC) has attracted increasing attention in NLP in recent years. However, detecting the stance on CC in multimodal data is understudied and remains challenging due to a lack of reliable datasets. To improve the understanding of public opinions and communication strategies, this paper presents MultiClimate, the first open-source manually-annotated stance detection dataset with $100$ CC-related YouTube videos and $4,209$ frame-transcript pairs. We deploy state-of-the-art vision and language models, as well as multimodal models for MultiClimate stance detection. Results show that text-only BERT significantly outperforms image-only ResNet50 and ViT. Combining both modalities achieves state-of-the-art, $0.747$/$0.749$ in accuracy/F1. Our 100M-sized fusion models also beat CLIP and BLIP, as well as the much larger 9B-sized multimodal IDEFICS and text-only Llama3 and Gemma2, indicating that multimodal stance detection remains challenging for large language models. Our code, dataset, as well as supplementary materials, are available at <https://github.com/werywjw/MultiClimate>.

**URL** : <http://arxiv.org/pdf/2409.18346v1>

**Published** : 2024-09-26

## A Generalized LLM-Augmented BIM Framework: Application to a Speech-to-BIM system

**Authors** : Ghang Lee, Suhyung Jang, Seokho Hyun

**Abstract** : Performing building information modeling (BIM) tasks is a complex process that imposes a steep learning curve and a heavy cognitive load due to the necessity of remembering sequences of numerous commands. With the rapid advancement of large language models (LLMs), it is foreseeable that BIM tasks, including querying and managing BIM data, 4D and 5D BIM, design compliance checking, or authoring a design, using written or spoken natural language (i.e., text-to-BIM or speech-to-BIM), will soon supplant traditional graphical user interfaces. This paper proposes a generalized LLM-augmented BIM framework to expedite the development of LLM-enhanced BIM applications by providing a step-by-step development process. The proposed framework consists of six steps: interpret-fill-match-structure-execute-check. The paper demonstrates the applicability of the proposed framework through implementing a speech-to-BIM application, NADIA-S (Natural-language-based Architectural Detailing through Interaction with Artificial Intelligence via Speech), using exterior wall detailing as an example.

**URL** : <http://arxiv.org/pdf/2409.18345v1>

**Published** : 2024-09-26

## AER-LLM: Ambiguity-aware Emotion Recognition Leveraging Large Language Models

**Authors** : Xin Hong, Yuan Gong, Vidhyasaharan Sethu, Ting Dang

**Abstract** : Recent advancements in Large Language Models (LLMs) have demonstrated great success in many Natural Language Processing (NLP) tasks. In addition to their cognitive intelligence, exploring their capabilities in emotional intelligence is also crucial, as it enables more natural and empathetic conversational AI. Recent studies have shown LLMs' capability in recognizing emotions, but they often focus on single emotion labels and overlook the complex and ambiguous nature of human emotions. This study is the first to address this gap by exploring the potential of LLMs in recognizing ambiguous emotions, leveraging their strong generalization capabilities and in-context learning. We design zero-shot and few-shot prompting and incorporate past dialogue as context information for ambiguous emotion recognition. Experiments conducted using three datasets indicate significant potential for LLMs in recognizing ambiguous emotions, and highlight the substantial benefits of including context information. Furthermore, our findings indicate that LLMs demonstrate a high degree of effectiveness in recognizing less ambiguous emotions and exhibit potential for identifying more ambiguous emotions, paralleling human perceptual capabilities.

**URL** : <http://arxiv.org/pdf/2409.18339v1>

**Published** : 2024-09-26

## Cross-Institutional Structured Radiology Reporting for Lung Cancer Screening Using a Dynamic Template-Constrained Large Language Model

**Authors** : Chuang Niu, Parisa Kaviani, Qing Lyu, Mannudeep K. Kalra, Christopher T. Whitlow, Ge Wang

**Abstract** : Structured radiology reporting is advantageous for optimizing clinical workflows and patient outcomes. Current LLMs in creating structured reports face the challenges of formatting errors, content hallucinations, and privacy leakage concerns when uploaded to external servers. We aim to develop an enhanced open-source LLM for creating structured and standardized LCS reports from free-text descriptions. After institutional IRB approvals, 5,442 de-identified LCS reports from two institutions were retrospectively analyzed. 500 reports were randomly selected from the two institutions evenly and then manually labeled for evaluation. Two radiologists from the two institutions developed a standardized template including 29 features for lung nodule reporting. We proposed template-constrained decoding to enhance state-of-the-art open-source LLMs, including LLAMA, Qwen, and Mistral. The LLM performance was extensively evaluated in terms of F1 score, confidence interval, McNemar test, and z-test. Based on the structured reports created from the large-scale dataset, a nodule-level retrieval system was prototyped and an automatic statistical analysis was performed. Our software, vLLM-structure, is publicly available for local deployment with enhanced LLMs. Our template-constrained decoding approach consistently enhanced the LLM performance on multi-institutional datasets, with neither formatting errors nor content hallucinations. Our method improved the best open-source LLAMA-3.1 405B by up to 10.42%, and outperformed GPT-4o by 17.19%. A novel nodule retrieval system was successfully prototyped and demonstrated on a large-scale multimodal database using our enhanced LLM technologies. The automatically derived statistical distributions were closely consistent with the prior findings in terms of nodule type, location, size, status, and Lung-RADS.

**URL** : <http://arxiv.org/pdf/2409.18319v1>

**Published** : 2024-09-26

## Retrospective Comparative Analysis of Prostate Cancer In-Basket Messages: Responses from Closed-Domain LLM vs. Clinical Teams

**Authors** : Yuexing Hao, Jason M. Holmes, Jared Hobson, Alexandra Bennett, Daniel K. Ebner, David M. Routman, Satomi Shiraishi, Samir H. Patel, Nathan Y. Yu, Chris L. Hallemeier, Brooke E. Ball, Mark R. Waddle, Wei Liu

**Abstract** : In-basket message interactions play a crucial role in physician-patient communication, occurring during all phases (pre-, during, and post) of a patient's care journey. However, responding to these patients' inquiries has become a significant burden on healthcare workflows, consuming considerable time for clinical care teams. To address this, we introduce RadOnc-GPT, a specialized Large Language Model (LLM) powered by GPT-4 that has been designed with a focus on radiotherapeutic treatment of prostate cancer with advanced prompt engineering, and specifically designed to assist in generating responses. We integrated RadOnc-GPT with patient electronic health records (EHR) from both the hospital-wide EHR database and an internal, radiation-oncology-specific database. RadOnc-GPT was evaluated on 158 previously recorded in-basket message interactions. Quantitative natural language processing (NLP) analysis and two grading studies with clinicians and nurses were used to assess RadOnc-GPT's responses. Our findings indicate that RadOnc-GPT slightly outperformed the clinical care team in "Clarity" and "Empathy," while achieving comparable scores in "Completeness" and "Correctness." RadOnc-GPT is estimated to save 5.2 minutes per message for nurses and 2.4 minutes for clinicians, from reading the inquiry to sending the response. Employing RadOnc-GPT for in-basket message draft generation has the potential to alleviate the workload of clinical care teams and reduce healthcare costs by producing high-quality, timely responses.

**URL** : <http://arxiv.org/pdf/2409.18290v1>

**Published** : 2024-09-26

## Advancing Object Detection in Transportation with Multimodal Large Language Models (MLLMs): A Comprehensive Review and Empirical Testing

**Authors** : Huthaifa I. Ashqar, Ahmed Jaber, Taqwa I. Alhadidi, Mohammed Elhenawy

**Abstract** : This study aims to comprehensively review and empirically evaluate the application of multimodal large language models (MLLMs) and Large Vision Models (VLMs) in object detection for transportation systems. In the first fold, we provide a background about the potential benefits of MLLMs in transportation applications and conduct a comprehensive review of current MLLM technologies in previous studies. We highlight their effectiveness and limitations in object detection within various transportation scenarios. The second fold involves providing an overview of the taxonomy of end-to-end object detection in transportation applications and future directions. Building on this, we proposed empirical analysis for testing MLLMs on three real-world transportation problems that include object detection tasks namely, road safety attributes extraction, safety-critical event detection, and visual reasoning of thermal images. Our findings provide a detailed assessment of MLLM performance, uncovering both strengths and areas for improvement. Finally, we discuss practical limitations and challenges of MLLMs in enhancing object detection in transportation, thereby offering a roadmap for future research and development in this critical area.

**URL** : <http://arxiv.org/pdf/2409.18286v1>

**Published** : 2024-09-26

## Trustworthy AI: Securing Sensitive Data in Large Language Models

**Authors** : Georgios Feretzakis, Vassilios S. Verykios

**Abstract** : Large Language Models (LLMs) have transformed natural language processing (NLP) by enabling robust text generation and understanding. However, their deployment in sensitive domains like healthcare, finance, and legal services raises critical concerns about privacy and data security. This paper proposes a comprehensive framework for embedding trust mechanisms into LLMs to dynamically control the disclosure of sensitive information. The framework integrates three core components: User Trust Profiling, Information Sensitivity Detection, and Adaptive Output Control. By leveraging techniques such as Role-Based Access Control (RBAC), Attribute-Based Access Control (ABAC), Named Entity Recognition (NER), contextual analysis, and privacy-preserving methods like differential privacy, the system ensures that sensitive information is disclosed appropriately based on the user's trust level. By focusing on balancing data utility and privacy, the proposed solution offers a novel approach to securely deploying LLMs in high-risk environments. Future work will focus on testing this framework across various domains to evaluate its effectiveness in managing sensitive data while maintaining system efficiency.

**URL** : <http://arxiv.org/pdf/2409.18222v1>

**Published** : 2024-09-26

## MMMT-IF: A Challenging Multimodal Multi-Turn Instruction Following Benchmark

**Authors** : Elliot L. Epstein, Kaisheng Yao, Jing Li, Xinyi Bai, Hamid Palangi

**Abstract** : Evaluating instruction following capabilities for multimodal, multi-turn dialogue is challenging. With potentially multiple instructions in the input model context, the task is time-consuming for human raters and we show LLM based judges are biased towards answers from the same model. We propose MMMT-IF, an image based multi-turn Q$&$A evaluation set with added global instructions between questions, constraining the answer format. This challenges models to retrieve instructions dispersed across long dialogues and reason under instruction constraints. All instructions are objectively verifiable through code execution. We introduce the Programmatic Instruction Following ($\operatorname{PIF}$) metric to measure the fraction of the instructions that are correctly followed while performing a reasoning task. The $\operatorname{PIF-N-K}$ set of metrics further evaluates robustness by measuring the fraction of samples in a corpus where, for each sample, at least K out of N generated model responses achieve a $\operatorname{PIF}$ score of one. The $\operatorname{PIF}$ metric aligns with human instruction following ratings, showing 60 percent correlation. Experiments show Gemini 1.5 Pro, GPT-4o, and Claude 3.5 Sonnet, have a $\operatorname{PIF}$ metric that drops from 0.81 on average at turn 1 across the models, to 0.64 at turn 20. Across all turns, when each response is repeated 4 times ($\operatorname{PIF-4-4}$), GPT-4o and Gemini successfully follow all instructions only $11%$ of the time. When all the instructions are also appended to the end of the model input context, the $\operatorname{PIF}$ metric improves by 22.3 points on average, showing that the challenge with the task lies not only in following the instructions, but also in retrieving the instructions spread out in the model context. We plan to open source the MMMT-IF dataset and metric computation code.

**URL** : <http://arxiv.org/pdf/2409.18216v1>

**Published** : 2024-09-26

## AI Policy Projector: Grounding LLM Policy Design in Iterative Mapmaking

**Authors** : Michelle S. Lam, Fred Hohman, Dominik Moritz, Jeffrey P. Bigham, Kenneth Holstein, Mary Beth Kery

**Abstract** : Whether a large language model policy is an explicit constitution or an implicit reward model, it is challenging to assess coverage over the unbounded set of real-world situations that a policy must contend with. We introduce an AI policy design process inspired by mapmaking, which has developed tactics for visualizing and iterating on maps even when full coverage is not possible. With Policy Projector, policy designers can survey the landscape of model input-output pairs, define custom regions (e.g., "violence"), and navigate these regions with rules that can be applied to LLM outputs (e.g., if output contains "violence" and "graphic details," then rewrite without "graphic details"). Policy Projector supports interactive policy authoring using LLM classification and steering and a map visualization reflecting the policy designer's work. In an evaluation with 12 AI safety experts, our system helps policy designers to address problematic model behaviors extending beyond an existing, comprehensive harm taxonomy.

**URL** : <http://arxiv.org/pdf/2409.18203v1>

**Published** : 2024-09-26

## LowREm: A Repository of Word Embeddings for 87 Low-Resource Languages Enhanced with Multilingual Graph Knowledge

**Authors** : Daniil Gurgurov, Rishu Kumar, Simon Ostermann

**Abstract** : Contextualized embeddings based on large language models (LLMs) are available for various languages, but their coverage is often limited for lower resourced languages. Training LLMs for such languages is often difficult due to insufficient data and high computational cost. Especially for very low resource languages, static word embeddings thus still offer a viable alternative. There is, however, a notable lack of comprehensive repositories with such embeddings for diverse languages. To address this, we present LowREm, a centralized repository of static embeddings for 87 low-resource languages. We also propose a novel method to enhance GloVe-based embeddings by integrating multilingual graph knowledge, utilizing another source of knowledge. We demonstrate the superior performance of our enhanced embeddings as compared to contextualized embeddings extracted from XLM-R on sentiment analysis. Our code and data are publicly available under <https://huggingface.co/DFKI>.

**URL** : <http://arxiv.org/pdf/2409.18193v1>

**Published** : 2024-09-26

## Evaluation of Large Language Models for Summarization Tasks in the Medical Domain: A Narrative Review

**Authors** : Emma Croxford, Yanjun Gao, Nicholas Pellegrino, Karen K. Wong, Graham Wills, Elliot First, Frank J. Liao, Cherodeep Goswami, Brian Patterson, Majid Afshar

**Abstract** : Large Language Models have advanced clinical Natural Language Generation, creating opportunities to manage the volume of medical text. However, the high-stakes nature of medicine requires reliable evaluation, which remains a challenge. In this narrative review, we assess the current evaluation state for clinical summarization tasks and propose future directions to address the resource constraints of expert human evaluation.

**URL** : <http://arxiv.org/pdf/2409.18170v1>

**Published** : 2024-09-26

## Multi-View and Multi-Scale Alignment for Contrastive Language-Image Pre-training in Mammography

**Authors** : Yuexi Du, John Onofrey, Nicha C. Dvornek

**Abstract** : Contrastive Language-Image Pre-training (CLIP) shows promise in medical image analysis but requires substantial data and computational resources. Due to these restrictions, existing CLIP applications in medical imaging focus mainly on modalities like chest X-rays that have abundant image-report data available, leaving many other important modalities under-explored. Here, we propose the first adaptation of the full CLIP model to mammography, which presents significant challenges due to labeled data scarcity, high-resolution images with small regions of interest, and data imbalance. We first develop a specialized supervision framework for mammography that leverages its multi-view nature. Furthermore, we design a symmetric local alignment module to better focus on detailed features in high-resolution images. Lastly, we incorporate a parameter-efficient fine-tuning approach for large language models pre-trained with medical knowledge to address data limitations. Our multi-view and multi-scale alignment (MaMA) method outperforms state-of-the-art baselines for three different tasks on two large real-world mammography datasets, EMBED and RSNA-Mammo, with only 52% model size compared with the largest baseline.

**URL** : <http://arxiv.org/pdf/2409.18119v1>

**Published** : 2024-09-26

## Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Survey

**Authors** : Tiansheng Huang, Sihao Hu, Fatih Ilhan, Selim Furkan Tekin, Ling Liu

**Abstract** : Recent research demonstrates that the nascent fine-tuning-as-a-service business model exposes serious safety concerns -- fine-tuning over a few harmful data uploaded by the users can compromise the safety alignment of the model. The attack, known as harmful fine-tuning, has raised a broad research interest among the community. However, as the attack is still new, \textbf{we observe from our miserable submission experience that there are general misunderstandings within the research community.} We in this paper aim to clear some common concerns for the attack setting, and formally establish the research problem. Specifically, we first present the threat model of the problem, and introduce the harmful fine-tuning attack and its variants. Then we systematically survey the existing literature on attacks/defenses/mechanical analysis of the problem. Finally, we outline future research directions that might contribute to the development of the field. Additionally, we present a list of questions of interest, which might be useful to refer to when reviewers in the peer review process question the realism of the experiment/attack/defense setting. A curated list of relevant papers is maintained and made accessible at: \url{[https://github.com/git-disl/awesome_LLM-harmful-fine-tuning-papers.}](https://github.com/git-disl/awesome_LLM-harmful-fine-tuning-papers.%7D)

**URL** : <http://arxiv.org/pdf/2409.18169v1>

**Published** : 2024-09-26

## Data-Prep-Kit: getting your data ready for LLM application development

**Authors** : David Wood, Boris Lublinsky, Alexy Roytman, Shivdeep Singh, Abdulhamid Adebayo, Revital Eres, Mohammad Nassar, Hima Patel, Yousaf Shah, Constantin Adam, Petros Zerfos, Nirmit Desai, Daiki Tsuzuku, Takuya Goto, Michele Dolfi, Saptha Surendran, Paramesvaran Selvam, Sungeun An, Yuan Chi Chang, Dhiraj Joshi, Hajar Emami-Gohari, Xuan-Hong Dang, Yan Koyfman, Shahrokh Daijavad

**Abstract** : Data preparation is the first and a very important step towards any Large Language Model (LLM) development. This paper introduces an easy-to-use, extensible, and scale-flexible open-source data preparation toolkit called Data Prep Kit (DPK). DPK is architected and designed to enable users to scale their data preparation to their needs. With DPK they can prepare data on a local machine or effortlessly scale to run on a cluster with thousands of CPU Cores. DPK comes with a highly scalable, yet extensible set of modules that transform natural language and code data. If the user needs additional transforms, they can be easily developed using extensive DPK support for transform creation. These modules can be used independently or pipelined to perform a series of operations. In this paper, we describe DPK architecture and show its performance from a small scale to a very large number of CPUs. The modules from DPK have been used for the preparation of Granite Models [1] [2]. We believe DPK is a valuable contribution to the AI community to easily prepare data to enhance the performance of their LLM models or to fine-tune models with Retrieval-Augmented Generation (RAG).

**URL** : <http://arxiv.org/pdf/2409.18164v1>

**Published** : 2024-09-26

## The Nexus of AR/VR, Large Language Models, UI/UX, and Robotics Technologies in Enhancing Learning and Social Interaction for Children: A Systematic Review

**Authors** : Biplov Paneru, Bishwash Paneru

**Abstract** : The combination of large language models (LLMs), augmented reality (AR), and user interface/user experience (UI/UX) design in therapies for children, especially with disorders like autism spectrum disorder (ASD), is examined in this review study. 150 publications were found by a thorough literature search throughout PubMed, ACM, IEEE Xplore, Elsevier, and Google Scholar; 42 of them were chosen for in-depth study due to their methodological rigor and relevance. Three primary areas are covered in this review: how AR can improve social and learning results; how LLMs can help with communication; and how UI/UX design affects how effective these technologies are. Results reveal that while LLMs can provide individualized learning and communication support, AR has demonstrated promise in enhancing social skills, motivation, and attention. For children with ASD, accessible and interesting interventions depend heavily on effective UI/UX design. To optimize the benefits of these technologies in ASD therapies, the study emphasizes the need for additional research to address difficulties related to customization, accessibility, and integration.

**URL** : <http://arxiv.org/pdf/2409.18162v1>

**Published** : 2024-09-26

## DualAD: Dual-Layer Planning for Reasoning in Autonomous Driving

**Authors** : Dingrui Wang, Marc Kaufeld, Johannes Betz

**Abstract** : We present a novel autonomous driving framework, DualAD, designed to imitate human reasoning during driving. DualAD comprises two layers: a rule-based motion planner at the bottom layer that handles routine driving tasks requiring minimal reasoning, and an upper layer featuring a rule-based text encoder that converts driving scenarios from absolute states into text description. This text is then processed by a large language model (LLM) to make driving decisions. The upper layer intervenes in the bottom layer's decisions when potential danger is detected, mimicking human reasoning in critical situations. Closed-loop experiments demonstrate that DualAD, using a zero-shot pre-trained model, significantly outperforms rule-based motion planners that lack reasoning abilities. Our experiments also highlight the effectiveness of the text encoder, which considerably enhances the model's scenario understanding. Additionally, the integrated DualAD model improves with stronger LLMs, indicating the framework's potential for further enhancement. We make code and benchmarks publicly available.

**URL** : <http://arxiv.org/pdf/2409.18053v1>

**Published** : 2024-09-26

## EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotions

**Authors** : Kai Chen, Yunhao Gou, Runhui Huang, Zhili Liu, Daxin Tan, Jing Xu, Chunwei Wang, Yi Zhu, Yihan Zeng, Kuo Yang, Dingdong Wang, Kun Xiang, Haoyuan Li, Haoli Bai, Jianhua Han, Xiaohui Li, Weike Jin, Nian Xie, Yu Zhang, James T. Kwok, Hengshuang Zhao, Xiaodan Liang, Dit-Yan Yeung, Xiao Chen, Zhenguo Li, Wei Zhang, Qun Liu, Lanqing Hong, Lu Hou, Hang Xu

**Abstract** : GPT-4o, an omni-modal model that enables vocal conversations with diverse emotions and tones, marks a milestone for omni-modal foundation models. However, empowering Large Language Models to perceive and generate images, texts, and speeches end-to-end with publicly available data remains challenging in the open-source community. Existing vision-language models rely on external tools for the speech processing, while speech-language models still suffer from limited or even without vision-understanding abilities. To address this gap, we propose EMOVA (EMotionally Omni-present Voice Assistant), to enable Large Language Models with end-to-end speech capabilities while maintaining the leading vision-language performance. With a semantic-acoustic disentangled speech tokenizer, we notice surprisingly that omni-modal alignment can further enhance vision-language and speech abilities compared with the corresponding bi-modal aligned counterparts. Moreover, a lightweight style module is proposed for flexible speech style controls (e.g., emotions and pitches). For the first time, EMOVA achieves state-of-the-art performance on both the vision-language and speech benchmarks, and meanwhile, supporting omni-modal spoken dialogue with vivid emotions.

**URL** : <http://arxiv.org/pdf/2409.18042v1>

**Published** : 2024-09-26

## Compositional Hardness of Code in Large Language Models -- A Probabilistic Perspective

**Authors** : Yotam Wolf, Binyamin Rothberg, Dorin Shteyman, Amnon Shashua

**Abstract** : A common practice in large language model (LLM) usage for complex analytical tasks such as code generation, is to sample a solution for the entire task within the model's context window. Previous works have shown that subtask decomposition within the model's context (chain of thought), is beneficial for solving such tasks. In this work, we point a limitation of LLMs' ability to perform several sub-tasks within the same context window - an in-context hardness of composition, pointing to an advantage for distributing a decomposed problem in a multi-agent system of LLMs. The hardness of composition is quantified by a generation complexity metric, i.e., the number of LLM generations required to sample at least one correct solution. We find a gap between the generation complexity of solving a compositional problem within the same context relative to distributing it among multiple agents, that increases exponentially with the solution's length. We prove our results theoretically and demonstrate them empirically.

**URL** : <http://arxiv.org/pdf/2409.18028v1>

**Published** : 2024-09-26

## An Adversarial Perspective on Machine Unlearning for AI Safety

**Authors** : Jakub Łucki, Boyi Wei, Yangsibo Huang, Peter Henderson, Florian Tramèr, Javier Rando

**Abstract** : Large language models are finetuned to refuse questions about hazardous knowledge, but these protections can often be bypassed. Unlearning methods aim at completely removing hazardous capabilities from models and make them inaccessible to adversaries. This work challenges the fundamental differences between unlearning and traditional safety post-training from an adversarial perspective. We demonstrate that existing jailbreak methods, previously reported as ineffective against unlearning, can be successful when applied carefully. Furthermore, we develop a variety of adaptive methods that recover most supposedly unlearned capabilities. For instance, we show that finetuning on 10 unrelated examples or removing specific directions in the activation space can recover most hazardous capabilities for models edited with RMU, a state-of-the-art unlearning method. Our findings challenge the robustness of current unlearning approaches and question their advantages over safety training.

**URL** : <http://arxiv.org/pdf/2409.18025v1>

**Published** : 2024-09-26

## DARE: Diverse Visual Question Answering with Robustness Evaluation

**Authors** : Hannah Sterz, Jonas Pfeiffer, Ivan Vulić

**Abstract** : Vision Language Models (VLMs) extend remarkable capabilities of text-only large language models and vision-only models, and are able to learn from and process multi-modal vision-text input. While modern VLMs perform well on a number of standard image classification and image-text matching tasks, they still struggle with a number of crucial vision-language (VL) reasoning abilities such as counting and spatial reasoning. Moreover, while they might be very brittle to small variations in instructions and/or evaluation protocols, existing benchmarks fail to evaluate their robustness (or rather the lack of it). In order to couple challenging VL scenarios with comprehensive robustness evaluation, we introduce DARE, Diverse Visual Question Answering with Robustness Evaluation, a carefully created and curated multiple-choice VQA benchmark. DARE evaluates VLM performance on five diverse categories and includes four robustness-oriented evaluations based on the variations of: prompts, the subsets of answer options, the output format and the number of correct answers. Among a spectrum of other findings, we report that state-of-the-art VLMs still struggle with questions in most categories and are unable to consistently deliver their peak performance across the tested robustness evaluations. The worst case performance across the subsets of options is up to 34% below the performance in the standard case. The robustness of the open-source VLMs such as LLaVA 1.6 and Idefics2 cannot match the closed-source models such as GPT-4 and Gemini, but even the latter remain very brittle to different variations.

**URL** : <http://arxiv.org/pdf/2409.18023v1>

**Published** : 2024-09-26

## Role-RL: Online Long-Context Processing with Role Reinforcement Learning for Distinct LLMs in Their Optimal Roles

**Authors** : Lewei He, Tianyu Shi, Pengran Huang, Bingzhi Chen, Qianglong Chen, Jiahui Pan

**Abstract** : Large language models (LLMs) with long-context processing are still challenging because of their implementation complexity, training efficiency and data sparsity. To address this issue, a new paradigm named Online Long-context Processing (OLP) is proposed when we process a document of unlimited length, which typically occurs in the information reception and organization of diverse streaming media such as automated news reporting, live e-commerce, and viral short videos. Moreover, a dilemma was often encountered when we tried to select the most suitable LLM from a large number of LLMs amidst explosive growth aiming for outstanding performance, affordable prices, and short response delays. In view of this, we also develop Role Reinforcement Learning (Role-RL) to automatically deploy different LLMs in their respective roles within the OLP pipeline according to their actual performance. Extensive experiments are conducted on our OLP-MINI dataset and it is found that OLP with Role-RL framework achieves OLP benchmark with an average recall rate of 93.2% and the LLM cost saved by 79.4%. The code and dataset are publicly available at: <https://anonymous.4open.science/r/Role-RL>.

**URL** : <http://arxiv.org/pdf/2409.18014v1>

**Published** : 2024-09-26

## Control Industrial Automation System with Large Language Models

**Authors** : Yuchen Xia, Nasser Jazdi, Jize Zhang, Chaitanya Shah, Michael Weyrich

**Abstract** : Traditional industrial automation systems require specialized expertise to operate and complex reprogramming to adapt to new processes. Large language models offer the intelligence to make them more flexible and easier to use. However, LLMs' application in industrial settings is underexplored. This paper introduces a framework for integrating LLMs to achieve end-to-end control of industrial automation systems. At the core of the framework are an agent system designed for industrial tasks, a structured prompting method, and an event-driven information modeling mechanism that provides real-time data for LLM inference. The framework supplies LLMs with real-time events on different context semantic levels, allowing them to interpret the information, generate production plans, and control operations on the automation system. It also supports structured dataset creation for fine-tuning on this downstream application of LLMs. Our contribution includes a formal system design, proof-of-concept implementation, and a method for generating task-specific datasets for LLM fine-tuning and testing. This approach enables a more adaptive automation system that can respond to spontaneous events, while allowing easier operation and configuration through natural language for more intuitive human-machine interaction. We provide demo videos and detailed data on GitHub: <https://github.com/YuchenXia/LLM4IAS>

**URL** : <http://arxiv.org/pdf/2409.18009v1>

**Published** : 2024-09-26

## Multilingual Evaluation of Long Context Retrieval and Reasoning

**Authors** : Ameeta Agrawal, Andy Dang, Sina Bagheri Nezhad, Rhitabrat Pokharel, Russell Scheinberg

**Abstract** : Recent large language models (LLMs) demonstrate impressive capabilities in handling long contexts, some exhibiting near-perfect recall on synthetic retrieval tasks. However, these evaluations have mainly focused on English text and involved a single target sentence within lengthy contexts. Our work investigates how LLM performance generalizes to multilingual settings with multiple hidden target sentences. We comprehensively evaluate several long-context LLMs on retrieval and reasoning tasks across five languages: English, Vietnamese, Indonesian, Swahili, and Somali. These languages share the Latin script but belong to distinct language families and resource levels. Our analysis reveals a significant performance gap between languages. The best-performing models such as Gemini-1.5 and GPT-4o, achieve around 96% accuracy in English to around 36% in Somali with a single target sentence. However, this accuracy drops to 40% in English and 0% in Somali when dealing with three target sentences. Our findings highlight the challenges long-context LLMs face when processing longer contexts, an increase in the number of target sentences, or languages of lower resource levels.

**URL** : <http://arxiv.org/pdf/2409.18006v1>

**Published** : 2024-09-26

## Extracting Affect Aggregates from Longitudinal Social Media Data with Temporal Adapters for Large Language Models

**Authors** : Georg Ahnert, Max Pellert, David Garcia, Markus Strohmaier

**Abstract** : This paper proposes temporally aligned Large Language Models (LLMs) as a tool for longitudinal analysis of social media data. We fine-tune Temporal Adapters for Llama 3 8B on full timelines from a panel of British Twitter users, and extract longitudinal aggregates of emotions and attitudes with established questionnaires. We validate our estimates against representative British survey data and find strong positive, significant correlations for several collective emotions. The obtained estimates are robust across multiple training seeds and prompt formulations, and in line with collective emotions extracted using a traditional classification model trained on labeled data. To the best of our knowledge, this is the first work to extend the analysis of affect in LLMs to a longitudinal setting through Temporal Adapters. Our work enables new approaches towards the longitudinal analysis of social media data.

**URL** : <http://arxiv.org/pdf/2409.17990v1>

**Published** : 2024-09-26

## BEATS: Optimizing LLM Mathematical Capabilities with BackVerify and Adaptive Disambiguate based Efficient Tree Search

**Authors** : Linzhuang Sun, Hao Liang, Wentao Zhang

**Abstract** : Large Language Models (LLMs) have exhibited exceptional performance across a broad range of tasks and domains. However, they still encounter difficulties in solving mathematical problems due to the rigorous and logical nature of mathematics. Previous studies have employed techniques such as supervised fine-tuning (SFT), prompt engineering, and search-based methods to improve the mathematical problem-solving abilities of LLMs. Despite these efforts, their performance remains suboptimal and demands substantial computational resources. To address this issue, we propose a novel approach, BEATS, to enhance mathematical problem-solving abilities. Our method leverages newly designed prompts that guide the model to iteratively rewrite, advance by one step, and generate answers based on previous steps. Additionally, we introduce a new back-verification technique that uses LLMs to validate the correctness of the generated answers. Furthermore, we employ a pruning tree search to optimize search time while achieving strong performance. Notably, our method improves Qwen2-7b-Instruct's score from 36.94 to 61.52, outperforming GPT4's 42.5 on the MATH benchmark.

**URL** : <http://arxiv.org/pdf/2409.17972v1>

**Published** : 2024-09-26

## Weak-To-Strong Backdoor Attacks for LLMs with Contrastive Knowledge Distillation

**Authors** : Shuai Zhao, Leilei Gan, Zhongliang Guo, Xiaobao Wu, Luwei Xiao, Xiaoyu Xu, Cong-Duy Nguyen, Luu Anh Tuan

**Abstract** : Despite being widely applied due to their exceptional capabilities, Large Language Models (LLMs) have been proven to be vulnerable to backdoor attacks. These attacks introduce targeted vulnerabilities into LLMs by poisoning training samples and full-parameter fine-tuning. However, this kind of backdoor attack is limited since they require significant computational resources, especially as the size of LLMs increases. Besides, parameter-efficient fine-tuning (PEFT) offers an alternative but the restricted parameter updating may impede the alignment of triggers with target labels. In this study, we first verify that backdoor attacks with PEFT may encounter challenges in achieving feasible performance. To address these issues and improve the effectiveness of backdoor attacks with PEFT, we propose a novel backdoor attack algorithm from weak to strong based on contrastive knowledge distillation (W2SAttack). Specifically, we poison small-scale language models through full-parameter fine-tuning to serve as the teacher model. The teacher model then covertly transfers the backdoor to the large-scale student model through contrastive knowledge distillation, which employs PEFT. Theoretical analysis reveals that W2SAttack has the potential to augment the effectiveness of backdoor attacks. We demonstrate the superior performance of W2SAttack on classification tasks across four language models, four backdoor attack algorithms, and two different architectures of teacher models. Experimental results indicate success rates close to 100% for backdoor attacks targeting PEFT.

**URL** : <http://arxiv.org/pdf/2409.17946v1>

**Published** : 2024-09-26

## Atlas-Chat: Adapting Large Language Models for Low-Resource Moroccan Arabic Dialect

**Authors** : Guokan Shang, Hadi Abdine, Yousef Khoubrane, Amr Mohamed, Yassine Abbahaddou, Sofiane Ennadir, Imane Momayiz, Xuguang Ren, Eric Moulines, Preslav Nakov, Michalis Vazirgiannis, Eric Xing

**Abstract** : We introduce Atlas-Chat, the first-ever collection of large language models specifically developed for dialectal Arabic. Focusing on Moroccan Arabic, also known as Darija, we construct our instruction dataset by consolidating existing Darija language resources, creating novel datasets both manually and synthetically, and translating English instructions with stringent quality control. Atlas-Chat-9B and 2B models, fine-tuned on the dataset, exhibit superior ability in following Darija instructions and performing standard NLP tasks. Notably, our models outperform both state-of-the-art and Arabic-specialized LLMs like LLaMa, Jais, and AceGPT, e.g., achieving a 13% performance boost over a larger 13B model on DarijaMMLU, in our newly introduced evaluation suite for Darija covering both discriminative and generative tasks. Furthermore, we perform an experimental analysis of various fine-tuning strategies and base model choices to determine optimal configurations. All our resources are publicly accessible, and we believe our work offers comprehensive design methodologies of instruction-tuning for low-resource language variants, which are often neglected in favor of data-rich languages by contemporary LLMs.

**URL** : <http://arxiv.org/pdf/2409.17912v1>

**Published** : 2024-09-26

## Learning to Love Edge Cases in Formative Math Assessment: Using the AMMORE Dataset and Chain-of-Thought Prompting to Improve Grading Accuracy

**Authors** : Owen Henkel, Hannah Horne-Robinson, Maria Dyshel, Nabil Ch, Baptiste Moreau-Pernet, Ralph Abood

**Abstract** : This paper introduces AMMORE, a new dataset of 53,000 math open-response question-answer pairs from Rori, a learning platform used by students in several African countries and conducts two experiments to evaluate the use of large language models (LLM) for grading particularly challenging student answers. The AMMORE dataset enables various potential analyses and provides an important resource for researching student math acquisition in understudied, real-world, educational contexts. In experiment 1 we use a variety of LLM-driven approaches, including zero-shot, few-shot, and chain-of-thought prompting, to grade the 1% of student answers that a rule-based classifier fails to grade accurately. We find that the best-performing approach -- chain-of-thought prompting -- accurately scored 92% of these edge cases, effectively boosting the overall accuracy of the grading from 98.7% to 99.9%. In experiment 2, we aim to better understand the consequential validity of the improved grading accuracy, by passing grades generated by the best-performing LLM-based approach to a Bayesian Knowledge Tracing (BKT) model, which estimated student mastery of specific lessons. We find that relatively modest improvements in model accuracy at the individual question level can lead to significant changes in the estimation of student mastery. Where the rules-based classifier currently used to grade student, answers misclassified the mastery status of 6.9% of students across their completed lessons, using the LLM chain-of-thought approach this misclassification rate was reduced to 2.6% of students. Taken together, these findings suggest that LLMs could be a valuable tool for grading open-response questions in K-12 mathematics education, potentially enabling encouraging wider adoption of open-ended questions in formative assessment.

**URL** : <http://arxiv.org/pdf/2409.17904v1>

**Published** : 2024-09-26

## EMMA-500: Enhancing Massively Multilingual Adaptation of Large Language Models

**Authors** : Shaoxiong Ji, Zihao Li, Indraneil Paul, Jaakko Paavola, Peiqin Lin, Pinzhen Chen, Dayyán O'Brien, Hengyu Luo, Hinrich Schütze, Jörg Tiedemann, Barry Haddow

**Abstract** : In this work, we introduce EMMA-500, a large-scale multilingual language model continue-trained on texts across 546 languages designed for enhanced multilingual performance, focusing on improving language coverage for low-resource languages. To facilitate continual pre-training, we compile the MaLA corpus, a comprehensive multilingual dataset enriched with curated datasets across diverse domains. Leveraging this corpus, we conduct extensive continual pre-training of the Llama 2 7B model, resulting in EMMA-500, which demonstrates robust performance across a wide collection of benchmarks, including a comprehensive set of multilingual tasks and PolyWrite, an open-ended generation benchmark developed in this study. Our results highlight the effectiveness of continual pre-training in expanding large language models' language capacity, particularly for underrepresented languages, demonstrating significant gains in cross-lingual transfer, task generalization, and language adaptability.

**URL** : <http://arxiv.org/pdf/2409.17892v1>

**Published** : 2024-09-26

## Efficient Arbitrary Precision Acceleration for Large Language Models on GPU Tensor Cores

**Authors** : Shaobo Ma, Chao Fang, Haikuo Shao, Zhongfeng Wang

**Abstract** : Large language models (LLMs) have been widely applied but face challenges in efficient inference. While quantization methods reduce computational demands, ultra-low bit quantization with arbitrary precision is hindered by limited GPU Tensor Core support and inefficient memory management, leading to suboptimal acceleration. To address these challenges, we propose a comprehensive acceleration scheme for arbitrary precision LLMs. At its core, we introduce a novel bipolar-INT data format that facilitates parallel computing and supports symmetric quantization, effectively reducing data redundancy. Building on this, we implement an arbitrary precision matrix multiplication scheme that decomposes and recovers matrices at the bit level, enabling flexible precision while maximizing GPU Tensor Core utilization. Furthermore, we develop an efficient matrix preprocessing method that optimizes data layout for subsequent computations. Finally, we design a data recovery-oriented memory management system that strategically utilizes fast shared memory, significantly enhancing kernel execution speed and minimizing memory access latency. Experimental results demonstrate our approach's effectiveness, with up to 13\times speedup in matrix multiplication compared to NVIDIA's CUTLASS. When integrated into LLMs, we achieve up to 6.7\times inference acceleration. These improvements significantly enhance LLM inference efficiency, enabling broader and more responsive applications of LLMs.

**URL** : <http://arxiv.org/pdf/2409.17870v1>

**Published** : 2024-09-26

## Language Models as Zero-shot Lossless Gradient Compressors: Towards General Neural Parameter Prior Models

**Authors** : Hui-Po Wang, Mario Fritz

**Abstract** : Despite the widespread use of statistical prior models in various fields, such models for neural network gradients have long been overlooked. The inherent challenge stems from their high-dimensional structures and complex interdependencies, which complicate effective modeling. In this work, we demonstrate the potential of large language models (LLMs) to act as gradient priors in a zero-shot setting. We examine the property by considering lossless gradient compression -- a critical application in distributed learning -- that depends heavily on precise probability modeling. To achieve this, we introduce LM-GC, a novel method that integrates LLMs with arithmetic coding. Our technique converts plain gradients into text-like formats, enhancing token efficiency by up to 38 times compared to their plain representations. We ensure that this data conversion maintains a close alignment with the structure of plain gradients and the symbols commonly recognized by LLMs. Our experiments indicate that LM-GC surpasses existing state-of-the-art lossless compression methods, improving compression rates by 10% up to 17.2% across various datasets and architectures. Additionally, our approach shows promising compatibility with lossy compression techniques such as quantization and sparsification. These findings highlight the significant potential of LLMs as a model for effectively handling gradients. We will release the source code upon publication.

**URL** : <http://arxiv.org/pdf/2409.17836v1>

**Published** : 2024-09-26

## PEDRO: Parameter-Efficient Fine-tuning with Prompt DEpenDent Representation MOdification

**Authors** : Tianfang Xie, Tianjing Li, Wei Zhu, Wei Han, Yi Zhao

**Abstract** : Due to their substantial sizes, large language models (LLMs) are typically deployed within a single-backbone multi-tenant framework. In this setup, a single instance of an LLM backbone must cater to multiple users or tasks through the application of various parameter-efficient fine-tuning (PEFT) models. Despite the availability of numerous effective PEFT techniques such as LoRA, there remains a need for a PEFT approach that achieves both high efficiency during inference and competitive performance on downstream tasks. In this research, we introduce a new and straightforward PEFT methodology named \underline{P}rompt D\underline{E}pen\underline{D}ent \underline{R}epresentation M\underline{O}dification (PEDRO). The proposed method involves integrating a lightweight vector generator into each Transformer layer, which generates vectors contingent upon the input prompts. These vectors then modify the hidden representations created by the LLM through a dot product operation, thereby influencing the semantic output and generated content of the model. Extensive experimentation across a variety of tasks indicates that: (a) PEDRO surpasses recent PEFT benchmarks when using a similar number of tunable parameters. (b) Under the single-backbone multi-tenant deployment model, PEDRO exhibits superior efficiency compared to LoRA, indicating significant industrial potential.

**URL** : <http://arxiv.org/pdf/2409.17834v1>

**Published** : 2024-09-26

## BeanCounter: A low-toxicity, large-scale, and open dataset of business-oriented text

**Authors** : Siyan Wang, Bradford Levy

**Abstract** : Many of the recent breakthroughs in language modeling have resulted from scaling effectively the same model architecture to larger datasets. In this vein, recent work has highlighted performance gains from increasing training dataset size and quality, suggesting a need for novel sources of large-scale datasets. In this work, we introduce BeanCounter, a public dataset consisting of more than 159B tokens extracted from businesses' disclosures. We show that this data is indeed novel: less than 0.1% of BeanCounter appears in Common Crawl-based datasets and it is an order of magnitude larger than datasets relying on similar sources. Given the data's provenance, we hypothesize that BeanCounter is comparatively more factual and less toxic than web-based datasets. Exploring this hypothesis, we find that many demographic identities occur with similar prevalence in BeanCounter but with significantly less toxic context relative to other datasets. To demonstrate the utility of BeanCounter, we evaluate and compare two LLMs continually pre-trained on BeanCounter with their base models. We find an 18-33% reduction in toxic generation and improved performance within the finance domain for the continually pretrained models. Collectively, our work suggests that BeanCounter is a novel source of low-toxicity and high-quality domain-specific data with sufficient scale to train multi-billion parameter LLMs.

**URL** : <http://arxiv.org/pdf/2409.17827v2>

**Published** : 2024-09-26

## Inference-Time Language Model Alignment via Integrated Value Guidance

**Authors** : Zhixuan Liu, Zhanhui Zhou, Yuanfu Wang, Chao Yang, Yu Qiao

**Abstract** : Large language models are typically fine-tuned to align with human preferences, but tuning large models is computationally intensive and complex. In this work, we introduce $\textit{Integrated Value Guidance}$ (IVG), a method that uses implicit and explicit value functions to guide language model decoding at token and chunk-level respectively, efficiently aligning large language models purely at inference time. This approach circumvents the complexities of direct fine-tuning and outperforms traditional methods. Empirically, we demonstrate the versatility of IVG across various tasks. In controlled sentiment generation and summarization tasks, our method significantly improves the alignment of large models using inference-time guidance from $\texttt{gpt2}$-based value functions. Moreover, in a more challenging instruction-following benchmark AlpacaEval 2.0, we show that both specifically tuned and off-the-shelf value functions greatly improve the length-controlled win rates of large models against $\texttt{gpt-4-turbo}$ (e.g., $19.51% \rightarrow 26.51%$ for $\texttt{Mistral-7B-Instruct-v0.2}$ and $25.58% \rightarrow 33.75%$ for $\texttt{Mixtral-8x7B-Instruct-v0.1}$ with Tulu guidance).

**URL** : <http://arxiv.org/pdf/2409.17819v1>

**Published** : 2024-09-26

## Self-supervised Preference Optimization: Enhance Your Language Model with Preference Degree Awareness

**Authors** : Jian Li, Haojing Huang, Yujia Zhang, Pengfei Xu, Xi Chen, Rui Song, Lida Shi, Jingwen Wang, Hao Xu

**Abstract** : Recently, there has been significant interest in replacing the reward model in Reinforcement Learning with Human Feedback (RLHF) methods for Large Language Models (LLMs), such as Direct Preference Optimization (DPO) and its variants. These approaches commonly use a binary cross-entropy mechanism on pairwise samples, i.e., minimizing and maximizing the loss based on preferred or dis-preferred responses, respectively. However, while this training strategy omits the reward model, it also overlooks the varying preference degrees within different responses. We hypothesize that this is a key factor hindering LLMs from sufficiently understanding human preferences. To address this problem, we propose a novel Self-supervised Preference Optimization (SPO) framework, which constructs a self-supervised preference degree loss combined with the alignment loss, thereby helping LLMs improve their ability to understand the degree of preference. Extensive experiments are conducted on two widely used datasets of different tasks. The results demonstrate that SPO can be seamlessly integrated with existing preference optimization methods and significantly boost their performance to achieve state-of-the-art performance. We also conduct detailed analyses to offer comprehensive insights into SPO, which verifies its effectiveness. The code is available at <https://github.com/lijian16/SPO>.

**URL** : <http://arxiv.org/pdf/2409.17791v1>

**Published** : 2024-09-26

## Few-shot Pairwise Rank Prompting: An Effective Non-Parametric Retrieval Model

**Authors** : Nilanjan Sinhababu, Andrew Parry, Debasis Ganguly, Debasis Samanta, Pabitra Mitra

**Abstract** : A supervised ranking model, despite its advantage of being effective, usually involves complex processing - typically multiple stages of task-specific pre-training and fine-tuning. This has motivated researchers to explore simpler pipelines leveraging large language models (LLMs) that are capable of working in a zero-shot manner. However, since zero-shot inference does not make use of a training set of pairs of queries and their relevant documents, its performance is mostly worse than that of supervised models, which are trained on such example pairs. Motivated by the existing findings that training examples generally improve zero-shot performance, in our work, we explore if this also applies to ranking models. More specifically, given a query and a pair of documents, the preference prediction task is improved by augmenting examples of preferences for similar queries from a training set. Our proposed pairwise few-shot ranker demonstrates consistent improvements over the zero-shot baseline on both in-domain (TREC DL) and out-domain (BEIR subset) retrieval benchmarks. Our method also achieves a close performance to that of a supervised model without requiring any complex training pipeline.

**URL** : <http://arxiv.org/pdf/2409.17745v2>

**Published** : 2024-09-26

## Episodic Memory Verbalization using Hierarchical Representations of Life-Long Robot Experience

**Authors** : Leonard Bärmann, Chad DeChant, Joana Plewnia, Fabian Peller-Konrad, Daniel Bauer, Tamim Asfour, Alex Waibel

**Abstract** : Verbalization of robot experience, i.e., summarization of and question answering about a robot's past, is a crucial ability for improving human-robot interaction. Previous works applied rule-based systems or fine-tuned deep models to verbalize short (several-minute-long) streams of episodic data, limiting generalization and transferability. In our work, we apply large pretrained models to tackle this task with zero or few examples, and specifically focus on verbalizing life-long experiences. For this, we derive a tree-like data structure from episodic memory (EM), with lower levels representing raw perception and proprioception data, and higher levels abstracting events to natural language concepts. Given such a hierarchical representation built from the experience stream, we apply a large language model as an agent to interactively search the EM given a user's query, dynamically expanding (initially collapsed) tree nodes to find the relevant information. The approach keeps computational costs low even when scaling to months of robot experience data. We evaluate our method on simulated household robot data, human egocentric videos, and real-world robot recordings, demonstrating its flexibility and scalability.

**URL** : <http://arxiv.org/pdf/2409.17702v1>

**Published** : 2024-09-26

## MoJE: Mixture of Jailbreak Experts, Naive Tabular Classifiers as Guard for Prompt Attacks

**Authors** : Giandomenico Cornacchia, Giulio Zizzo, Kieran Fraser, Muhammad Zaid Hamed, Ambrish Rawat, Mark Purcell

**Abstract** : The proliferation of Large Language Models (LLMs) in diverse applications underscores the pressing need for robust security measures to thwart potential jailbreak attacks. These attacks exploit vulnerabilities within LLMs, endanger data integrity and user privacy. Guardrails serve as crucial protective mechanisms against such threats, but existing models often fall short in terms of both detection accuracy, and computational efficiency. This paper advocates for the significance of jailbreak attack prevention on LLMs, and emphasises the role of input guardrails in safeguarding these models. We introduce MoJE (Mixture of Jailbreak Expert), a novel guardrail architecture designed to surpass current limitations in existing state-of-the-art guardrails. By employing simple linguistic statistical techniques, MoJE excels in detecting jailbreak attacks while maintaining minimal computational overhead during model inference. Through rigorous experimentation, MoJE demonstrates superior performance capable of detecting 90% of the attacks without compromising benign prompts, enhancing LLMs security against jailbreak attacks.

**URL** : <http://arxiv.org/pdf/2409.17699v2>

**Published** : 2024-09-26

## MIO: A Foundation Model on Multimodal Tokens

**Authors** : Zekun Wang, King Zhu, Chunpu Xu, Wangchunshu Zhou, Jiaheng Liu, Yibo Zhang, Jiashuo Wang, Ning Shi, Siyu Li, Yizhi Li, Haoran Que, Zhaoxiang Zhang, Yuanxing Zhang, Ge Zhang, Ke Xu, Jie Fu, Wenhao Huang

**Abstract** : In this paper, we introduce MIO, a novel foundation model built on multimodal tokens, capable of understanding and generating speech, text, images, and videos in an end-to-end, autoregressive manner. While the emergence of large language models (LLMs) and multimodal large language models (MM-LLMs) propels advancements in artificial general intelligence through their versatile capabilities, they still lack true any-to-any understanding and generation. Recently, the release of GPT-4o has showcased the remarkable potential of any-to-any LLMs for complex real-world tasks, enabling omnidirectional input and output across images, speech, and text. However, it is closed-source and does not support the generation of multimodal interleaved sequences. To address this gap, we present MIO, which is trained on a mixture of discrete tokens across four modalities using causal multimodal modeling. MIO undergoes a four-stage training process: (1) alignment pre-training, (2) interleaved pre-training, (3) speech-enhanced pre-training, and (4) comprehensive supervised fine-tuning on diverse textual, visual, and speech tasks. Our experimental results indicate that MIO exhibits competitive, and in some cases superior, performance compared to previous dual-modal baselines, any-to-any model baselines, and even modality-specific baselines. Moreover, MIO demonstrates advanced capabilities inherent to its any-to-any feature, such as interleaved video-text generation, chain-of-visual-thought reasoning, visual guideline generation, instructional image editing, etc.

**URL** : <http://arxiv.org/pdf/2409.17692v1>

**Published** : 2024-09-26

## Zero- and Few-shot Named Entity Recognition and Text Expansion in Medication Prescriptions using ChatGPT

**Authors** : Natthanaphop Isaradech, Andrea Riedel, Wachiranun Sirikul, Markus Kreuzthaler, Stefan Schulz

**Abstract** : Introduction: Medication prescriptions are often in free text and include a mix of two languages, local brand names, and a wide range of idiosyncratic formats and abbreviations. Large language models (LLMs) have shown promising ability to generate text in response to input prompts. We use ChatGPT 3.5 to automatically structure and expand medication statements in discharge summaries and thus make them easier to interpret for people and machines. Methods: Named-entity Recognition (NER) and Text Expansion (EX) are used in a zero- and few-shot setting with different prompt strategies. 100 medication statements were manually annotated and curated. NER performance was measured by using strict and partial matching. For the task EX, two experts interpreted the results by assessing semantic equivalence between original and expanded statements. The model performance was measured by precision, recall, and F1 score. Results: For NER, the best-performing prompt reached an average F1 score of 0.94 in the test set. For EX, the few-shot prompt showed superior performance among other prompts, with an average F1 score of 0.87. Conclusion: Our study demonstrates good performance for NER and EX tasks in free-text medication statements using ChatGPT. Compared to a zero-shot baseline, a few-shot approach prevented the system from hallucinating, which would be unacceptable when processing safety-relevant medication data.

**URL** : <http://arxiv.org/pdf/2409.17683v1>

**Published** : 2024-09-26

## AssistantX: An LLM-Powered Proactive Assistant in Collaborative Human-Populated Environment

**Authors** : Nan Sun, Bo Mao, Yongchang Li, Lumeng Ma, Di Guo, Huaping Liu

**Abstract** : The increasing demand for intelligent assistants in human-populated environments has motivated significant research in autonomous robotic systems. Traditional service robots and virtual assistants, however, struggle with real-world task execution due to their limited capacity for dynamic reasoning and interaction, particularly when human collaboration is required. Recent developments in Large Language Models have opened new avenues for improving these systems, enabling more sophisticated reasoning and natural interaction capabilities. In this paper, we introduce AssistantX, an LLM-powered proactive assistant designed to operate autonomously in a physical office environment. Unlike conventional service robots, AssistantX leverages a novel multi-agent architecture, PPDR4X, which provides advanced inference capabilities and comprehensive collaboration awareness. By effectively bridging the gap between virtual operations and physical interactions, AssistantX demonstrates robust performance in managing complex real-world scenarios. Our evaluation highlights the architecture's effectiveness, showing that AssistantX can respond to clear instructions, actively retrieve supplementary information from memory, and proactively seek collaboration from team members to ensure successful task completion. More details and videos can be found at <https://assistantx-agent.github.io/AssistantX/>.

**URL** : <http://arxiv.org/pdf/2409.17655v1>

**Published** : 2024-09-26

## Digital Twin Ecosystem for Oncology Clinical Operations

**Authors** : Himanshu Pandey, Akhil Amod, Shivang, Kshitij Jaggi, Ruchi Garg, Abheet Jain, Vinayak Tantia

**Abstract** : Artificial Intelligence (AI) and Large Language Models (LLMs) hold significant promise in revolutionizing healthcare, especially in clinical applications. Simultaneously, Digital Twin technology, which models and simulates complex systems, has gained traction in enhancing patient care. However, despite the advances in experimental clinical settings, the potential of AI and digital twins to streamline clinical operations remains largely untapped. This paper introduces a novel digital twin framework specifically designed to enhance oncology clinical operations. We propose the integration of multiple specialized digital twins, such as the Medical Necessity Twin, Care Navigator Twin, and Clinical History Twin, to enhance workflow efficiency and personalize care for each patient based on their unique data. Furthermore, by synthesizing multiple data sources and aligning them with the National Comprehensive Cancer Network (NCCN) guidelines, we create a dynamic Cancer Care Path, a continuously evolving knowledge base that enables these digital twins to provide precise, tailored clinical recommendations.

**URL** : <http://arxiv.org/pdf/2409.17650v1>

**Published** : 2024-09-26

## Efficient In-Domain Question Answering for Resource-Constrained Environments

**Authors** : Isaac Chung, Phat Vo, Arman Kizilkale, Aaron Reite

**Abstract** : Retrieval Augmented Generation (RAG) is a common method for integrating external knowledge into pretrained Large Language Models (LLMs) to enhance accuracy and relevancy in question answering (QA) tasks. However, prompt engineering and resource efficiency remain significant bottlenecks in developing optimal and robust RAG solutions for real-world QA applications. Recent studies have shown success in using fine tuning to address these problems; in particular, Retrieval Augmented Fine Tuning (RAFT) applied to smaller 7B models has demonstrated superior performance compared to RAG setups with much larger models such as GPT-3.5. The combination of RAFT with parameter-efficient fine tuning (PEFT) techniques, such as Low-Rank Adaptation (LoRA), promises an even more efficient solution, yet remains an unexplored area. In this work, we combine RAFT with LoRA to reduce fine tuning and storage requirements and gain faster inference times while maintaining comparable RAG performance. This results in a more compute-efficient RAFT, or CRAFT, which is particularly useful for knowledge-intensive QA tasks in resource-constrained environments where internet access may be restricted and hardware resources limited.

**URL** : <http://arxiv.org/pdf/2409.17648v1>

**Published** : 2024-09-26

## AI Delegates with a Dual Focus: Ensuring Privacy and Strategic Self-Disclosure

**Authors** : Xi Chen, Zhiyang Zhang, Fangkai Yang, Xiaoting Qin, Chao Du, Xi Cheng, Hangxin Liu, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, Qi Zhang

**Abstract** : Large language model (LLM)-based AI delegates are increasingly utilized to act on behalf of users, assisting them with a wide range of tasks through conversational interfaces. Despite their advantages, concerns arise regarding the potential risk of privacy leaks, particularly in scenarios involving social interactions. While existing research has focused on protecting privacy by limiting the access of AI delegates to sensitive user information, many social scenarios require disclosing private details to achieve desired outcomes, necessitating a balance between privacy protection and disclosure. To address this challenge, we conduct a pilot study to investigate user preferences for AI delegates across various social relations and task scenarios, and then propose a novel AI delegate system that enables privacy-conscious self-disclosure. Our user study demonstrates that the proposed AI delegate strategically protects privacy, pioneering its use in diverse and dynamic social interactions.

**URL** : <http://arxiv.org/pdf/2409.17642v1>

**Published** : 2024-09-26

## T3: A Novel Zero-shot Transfer Learning Framework Iteratively Training on an Assistant Task for a Target Task

**Authors** : Xindi Tong, Yujin Zhu, Shijian Fan, Liang Xu

**Abstract** : Long text summarization, gradually being essential for efficiently processing large volumes of information, stays challenging for Large Language Models (LLMs) such as GPT and LLaMA families because of the insufficient open-sourced training datasets and the high requirement of contextual details dealing. To address the issue, we design a novel zero-shot transfer learning framework, abbreviated as T3, to iteratively training a baseline LLM on an assistant task for the target task, where the former should own richer data resources and share structural or semantic similarity with the latter. In practice, T3 is approached to deal with the long text summarization task by utilizing question answering as the assistant task, and further validated its effectiveness on the BBC summary, NarraSum, FairytaleQA, and NLQuAD datasets, with up to nearly 14% improvement in ROUGE, 35% improvement in BLEU, and 16% improvement in Factscore compared to three baseline LLMs, demonstrating its potential for more assistant-target task combinations.

**URL** : <http://arxiv.org/pdf/2409.17640v1>

**Published** : 2024-09-26

## ZALM3: Zero-Shot Enhancement of Vision-Language Alignment via In-Context Information in Multi-Turn Multimodal Medical Dialogue

**Authors** : Zhangpu Li, Changhong Zou, Suxue Ma, Zhicheng Yang, Chen Du, Youbao Tang, Zhenjie Cao, Ning Zhang, Jui-Hsin Lai, Ruei-Sung Lin, Yuan Ni, Xingzhi Sun, Jing Xiao, Kai Zhang, Mei Han

**Abstract** : The rocketing prosperity of large language models (LLMs) in recent years has boosted the prevalence of vision-language models (VLMs) in the medical sector. In our online medical consultation scenario, a doctor responds to the texts and images provided by a patient in multiple rounds to diagnose her/his health condition, forming a multi-turn multimodal medical dialogue format. Unlike high-quality images captured by professional equipment in traditional medical visual question answering (Med-VQA), the images in our case are taken by patients' mobile phones. These images have poor quality control, with issues such as excessive background elements and the lesion area being significantly off-center, leading to degradation of vision-language alignment in the model training phase. In this paper, we propose ZALM3, a Zero-shot strategy to improve vision-language ALignment in Multi-turn Multimodal Medical dialogue. Since we observe that the preceding text conversations before an image can infer the regions of interest (RoIs) in the image, ZALM3 employs an LLM to summarize the keywords from the preceding context and a visual grounding model to extract the RoIs. The updated images eliminate unnecessary background noise and provide more effective vision-language alignment. To better evaluate our proposed method, we design a new subjective assessment metric for multi-turn unimodal/multimodal medical dialogue to provide a fine-grained performance comparison. Our experiments across three different clinical departments remarkably demonstrate the efficacy of ZALM3 with statistical significance.

**URL** : <http://arxiv.org/pdf/2409.17610v1>

**Published** : 2024-09-26

## DualCoTs: Dual Chain-of-Thoughts Prompting for Sentiment Lexicon Expansion of Idioms

**Authors** : Fuqiang Niu, Minghuan Tan, Bowen Zhang, Min Yang, Ruifeng Xu

**Abstract** : Idioms represent a ubiquitous vehicle for conveying sentiments in the realm of everyday discourse, rendering the nuanced analysis of idiom sentiment crucial for a comprehensive understanding of emotional expression within real-world texts. Nevertheless, the existing corpora dedicated to idiom sentiment analysis considerably limit research in text sentiment analysis. In this paper, we propose an innovative approach to automatically expand the sentiment lexicon for idioms, leveraging the capabilities of large language models through the application of Chain-of-Thought prompting. To demonstrate the effectiveness of this approach, we integrate multiple existing resources and construct an emotional idiom lexicon expansion dataset (called EmoIdiomE), which encompasses a comprehensive repository of Chinese and English idioms. Then we designed the Dual Chain-of-Thoughts (DualCoTs) method, which combines insights from linguistics and psycholinguistics, to demonstrate the effectiveness of using large models to automatically expand the sentiment lexicon for idioms. Experiments show that DualCoTs is effective in idioms sentiment lexicon expansion in both Chinese and English. For reproducibility, we will release the data and code upon acceptance.

**URL** : <http://arxiv.org/pdf/2409.17588v1>

**Published** : 2024-09-26

## A Scalable Data-Driven Framework for Systematic Analysis of SEC 10-K Filings Using Large Language Models

**Authors** : Syed Affan Daimi, Asma Iqbal

**Abstract** : The number of companies listed on the NYSE has been growing exponentially, creating a significant challenge for market analysts, traders, and stockholders who must monitor and assess the performance and strategic shifts of a large number of companies regularly. There is an increasing need for a fast, cost-effective, and comprehensive method to evaluate the performance and detect and compare many companies' strategy changes efficiently. We propose a novel data-driven approach that leverages large language models (LLMs) to systematically analyze and rate the performance of companies based on their SEC 10-K filings. These filings, which provide detailed annual reports on a company's financial performance and strategic direction, serve as a rich source of data for evaluating various aspects of corporate health, including confidence, environmental sustainability, innovation, and workforce management. We also introduce an automated system for extracting and preprocessing 10-K filings. This system accurately identifies and segments the required sections as outlined by the SEC, while also isolating key textual content that contains critical information about the company. This curated data is then fed into Cohere's Command-R+ LLM to generate quantitative ratings across various performance metrics. These ratings are subsequently processed and visualized to provide actionable insights. The proposed scheme is then implemented on an interactive GUI as a no-code solution for running the data pipeline and creating the visualizations. The application showcases the rating results and provides year-on-year comparisons of company performance.

**URL** : <http://arxiv.org/pdf/2409.17581v1>

**Published** : 2024-09-26

## Dr. GPT in Campus Counseling: Understanding Higher Education Students' Opinions on LLM-assisted Mental Health Services

**Authors** : Owen Xingjian Zhang, Shuyao Zhou, Jiayi Geng, Yuhan Liu, Sunny Xun Liu

**Abstract** : In response to the increasing mental health challenges faced by college students, we sought to understand their perspectives on how AI applications, particularly Large Language Models (LLMs), can be leveraged to enhance their mental well-being. Through pilot interviews with ten diverse students, we explored their opinions on the use of LLMs across five fictional scenarios: General Information Inquiry, Initial Screening, Reshaping Patient-Expert Dynamics, Long-term Care, and Follow-up Care. Our findings revealed that students' acceptance of LLMs varied by scenario, with participants highlighting both potential benefits, such as proactive engagement and personalized follow-up care, and concerns, including limitations in training data and emotional support. These insights inform how AI technology should be designed and implemented to effectively support and enhance students' mental well-being, particularly in scenarios where LLMs can complement traditional methods, while maintaining empathy and respecting individual preferences.

**URL** : <http://arxiv.org/pdf/2409.17572v1>

**Published** : 2024-09-26

## Logic-of-Thought: Injecting Logic into Contexts for Full Reasoning in Large Language Models

**Authors** : Tongxuan Liu, Wenjiang Xu, Weizhe Huang, Xingyu Wang, Jiaxing Wang, Hailong Yang, Jing Li

**Abstract** : Large Language Models (LLMs) have demonstrated remarkable capabilities across various tasks but their performance in complex logical reasoning tasks remains unsatisfactory. Although some prompting methods, such as Chain-of-Thought, can improve the reasoning ability of LLMs to some extent, they suffer from an unfaithful issue where derived conclusions may not align with the generated reasoning chain. To address this issue, some studies employ the approach of propositional logic to further enhance logical reasoning abilities of LLMs. However, the potential omissions in the extraction of logical expressions in these methods can cause information loss in the logical reasoning process, thereby generating incorrect results. To this end, we propose Logic-of-Thought (LoT) prompting which employs propositional logic to generate expanded logical information from input context, and utilizes the generated logical information as an additional augmentation to the input prompts, thereby enhancing the capability of logical reasoning. The LoT is orthogonal to existing prompting methods and can be seamlessly integrated with them. Extensive experiments demonstrate that LoT boosts the performance of various prompting methods with a striking margin across five logical reasoning tasks. In particular, the LoT enhances Chain-of-Thought's performance on the ReClor dataset by +4.35%; moreover, it improves Chain-of-Thought with Self-Consistency's performance on LogiQA by +5%; additionally, it boosts performance of Tree-of-Thoughts on ProofWriter dataset by +8%.

**URL** : <http://arxiv.org/pdf/2409.17539v1>

**Published** : 2024-09-26

## Data Proportion Detection for Optimized Data Management for Large Language Models

**Authors** : Hao Liang, Keshi Zhao, Yajie Yang, Bin Cui, Guosheng Dong, Zenan Zhou, Wentao Zhang

**Abstract** : Large language models (LLMs) have demonstrated exceptional performance across a wide range of tasks and domains, with data preparation playing a critical role in achieving these results. Pre-training data typically combines information from multiple domains. To maximize performance when integrating data from various domains, determining the optimal data proportion is essential. However, state-of-the-art (SOTA) LLMs rarely disclose details about their pre-training data, making it difficult for researchers to identify ideal data proportions. In this paper, we introduce a new topic, \textit{data proportion detection}, which enables the automatic estimation of pre-training data proportions by analyzing the generated outputs of LLMs. We provide rigorous theoretical proofs, practical algorithms, and preliminary experimental results for data proportion detection. Based on these findings, we offer valuable insights into the challenges and future directions for effective data proportion detection and data management.

**URL** : <http://arxiv.org/pdf/2409.17527v1>

**Published** : 2024-09-26

## EAGLE: Egocentric AGgregated Language-video Engine

**Authors** : Jing Bi, Yunlong Tang, Luchuan Song, Ali Vosoughi, Nguyen Nguyen, Chenliang Xu

**Abstract** : The rapid evolution of egocentric video analysis brings new insights into understanding human activities and intentions from a first-person perspective. Despite this progress, the fragmentation in tasks like action recognition, procedure learning, and moment retrieval, \etc, coupled with inconsistent annotations and isolated model development, hinders a holistic interpretation of video content. In response, we introduce the EAGLE (Egocentric AGgregated Language-video Engine) model and the EAGLE-400K dataset to provide a unified framework that integrates various egocentric video understanding tasks. EAGLE-400K, the \textit{first} large-scale instruction-tuning dataset tailored for egocentric video, features 400K diverse samples to enhance a broad spectrum of tasks from activity recognition to procedure knowledge learning. Moreover, EAGLE, a strong video multimodal large language model (MLLM), is designed to effectively capture both spatial and temporal information. In addition, we propose a set of evaluation metrics designed to facilitate a thorough assessment of MLLM for egocentric video understanding. Our extensive experiments demonstrate EAGLE's superior performance over existing models, highlighting its ability to balance task-specific understanding with holistic video interpretation. With EAGLE, we aim to pave the way for research opportunities and practical applications in real-world scenarios.

**URL** : <http://arxiv.org/pdf/2409.17523v1>

**Published** : 2024-09-26

## Multi-Designated Detector Watermarking for Language Models

**Authors** : Zhengan Huang, Gongxian Zeng, Xin Mu, Yu Wang, Yue Yu

**Abstract** : In this paper, we initiate the study of \emph{multi-designated detector watermarking (MDDW)} for large language models (LLMs). This technique allows model providers to generate watermarked outputs from LLMs with two key properties: (i) only specific, possibly multiple, designated detectors can identify the watermarks, and (ii) there is no perceptible degradation in the output quality for ordinary users. We formalize the security definitions for MDDW and present a framework for constructing MDDW for any LLM using multi-designated verifier signatures (MDVS). Recognizing the significant economic value of LLM outputs, we introduce claimability as an optional security feature for MDDW, enabling model providers to assert ownership of LLM outputs within designated-detector settings. To support claimable MDDW, we propose a generic transformation converting any MDVS to a claimable MDVS. Our implementation of the MDDW scheme highlights its advanced functionalities and flexibility over existing methods, with satisfactory performance metrics.

**URL** : <http://arxiv.org/pdf/2409.17518v1>

**Published** : 2024-09-26

## From News to Forecast: Integrating Event Analysis in LLM-Based Time Series Forecasting with Reflection

**Authors** : Xinlei Wang, Maike Feng, Jing Qiu, Jinjin Gu, Junhua Zhao

**Abstract** : This paper introduces a novel approach to enhance time series forecasting using Large Language Models (LLMs) and Generative Agents. With language as a medium, our method adaptively integrates various social events into forecasting models, aligning news content with time series fluctuations for enriched insights. Specifically, we utilize LLM-based agents to iteratively filter out irrelevant news and employ human-like reasoning and reflection to evaluate predictions. This enables our model to analyze complex events, such as unexpected incidents and shifts in social behavior, and continuously refine the selection logic of news and the robustness of the agent's output. By compiling selected news with time series data, we fine-tune the LLaMa2 pre-trained model. The results demonstrate significant improvements in forecasting accuracy and suggest a potential paradigm shift in time series forecasting by effectively harnessing unstructured news data.

**URL** : <http://arxiv.org/pdf/2409.17515v1>

**Published** : 2024-09-26

## Uni-Med: A Unified Medical Generalist Foundation Model For Multi-Task Learning Via Connector-MoE

**Authors** : Xun Zhu, Ying Hu, Fanbin Mo, Miao Li, Ji Wu

**Abstract** : Multi-modal large language models (MLLMs) have shown impressive capabilities as a general-purpose interface for various visual and linguistic tasks. However, building a unified MLLM for multi-task learning in the medical field remains a thorny challenge. To mitigate the tug-of-war problem of multi-modal multi-task optimization, recent advances primarily focus on improving the LLM components, while neglecting the connector that bridges the gap between modalities. In this paper, we introduce Uni-Med, a novel medical generalist foundation model which consists of a universal visual feature extraction module, a connector mixture-of-experts (CMoE) module, and an LLM. Benefiting from the proposed CMoE that leverages a well-designed router with a mixture of projection experts at the connector, Uni-Med achieves efficient solution to the tug-of-war problem and can perform six different medical tasks including question answering, visual question answering, report generation, referring expression comprehension, referring expression generation and image classification. To the best of our knowledge, Uni-Med is the first effort to tackle multi-task interference at the connector. Extensive ablation experiments validate the effectiveness of introducing CMoE under any configuration, with up to an average 8% performance gains. We further provide interpretation analysis of the tug-of-war problem from the perspective of gradient optimization and parameter statistics. Compared to previous state-of-the-art medical MLLMs, Uni-Med achieves competitive or superior evaluation metrics on diverse tasks. Code, data and model will be soon available at GitHub.

**URL** : <http://arxiv.org/pdf/2409.17508v1>

**Published** : 2024-09-26

## HaloScope: Harnessing Unlabeled LLM Generations for Hallucination Detection

**Authors** : Xuefeng Du, Chaowei Xiao, Yixuan Li

**Abstract** : The surge in applications of large language models (LLMs) has prompted concerns about the generation of misleading or fabricated information, known as hallucinations. Therefore, detecting hallucinations has become critical to maintaining trust in LLM-generated content. A primary challenge in learning a truthfulness classifier is the lack of a large amount of labeled truthful and hallucinated data. To address the challenge, we introduce HaloScope, a novel learning framework that leverages the unlabeled LLM generations in the wild for hallucination detection. Such unlabeled data arises freely upon deploying LLMs in the open world, and consists of both truthful and hallucinated information. To harness the unlabeled data, we present an automated membership estimation score for distinguishing between truthful and untruthful generations within unlabeled mixture data, thereby enabling the training of a binary truthfulness classifier on top. Importantly, our framework does not require extra data collection and human annotations, offering strong flexibility and practicality for real-world applications. Extensive experiments show that HaloScope can achieve superior hallucination detection performance, outperforming the competitive rivals by a significant margin. Code is available at <https://github.com/deeplearningwisc/haloscope>.

**URL** : <http://arxiv.org/pdf/2409.17504v1>

**Published** : 2024-09-26

## Human Mobility Modeling with Limited Information via Large Language Models

**Authors** : Yifan Liu, Xishun Liao, Haoxuan Ma, Brian Yueshuai He, Chris Stanford, Jiaqi Ma

**Abstract** : Understanding human mobility patterns has traditionally been a complex challenge in transportation modeling. Due to the difficulties in obtaining high-quality training datasets across diverse locations, conventional activity-based models and learning-based human mobility modeling algorithms are particularly limited by the availability and quality of datasets. Furthermore, current research mainly focuses on the spatial-temporal travel pattern but lacks an understanding of the semantic information between activities, which is crucial for modeling the interdependence between activities. In this paper, we propose an innovative Large Language Model (LLM) empowered human mobility modeling framework. Our proposed approach significantly reduces the reliance on detailed human mobility statistical data, utilizing basic socio-demographic information of individuals to generate their daily mobility patterns. We have validated our results using the NHTS and SCAG-ABM datasets, demonstrating the effective modeling of mobility patterns and the strong adaptability of our framework across various geographic locations.

**URL** : <http://arxiv.org/pdf/2409.17495v1>

**Published** : 2024-09-26

## MaskLLM: Learnable Semi-Structured Sparsity for Large Language Models

**Authors** : Gongfan Fang, Hongxu Yin, Saurav Muralidharan, Greg Heinrich, Jeff Pool, Jan Kautz, Pavlo Molchanov, Xinchao Wang

**Abstract** : Large Language Models (LLMs) are distinguished by their massive parameter counts, which typically result in significant redundancy. This work introduces MaskLLM, a learnable pruning method that establishes Semi-structured (or ``N:M'') Sparsity in LLMs, aimed at reducing computational overhead during inference. Instead of developing a new importance criterion, MaskLLM explicitly models N:M patterns as a learnable distribution through Gumbel Softmax sampling. This approach facilitates end-to-end training on large-scale datasets and offers two notable advantages: 1) High-quality Masks - our method effectively scales to large datasets and learns accurate masks; 2) Transferability - the probabilistic modeling of mask distribution enables the transfer learning of sparsity across domains or tasks. We assessed MaskLLM using 2:4 sparsity on various LLMs, including LLaMA-2, Nemotron-4, and GPT-3, with sizes ranging from 843M to 15B parameters, and our empirical results show substantial improvements over state-of-the-art methods. For instance, leading approaches achieve a perplexity (PPL) of 10 or greater on Wikitext compared to the dense model's 5.12 PPL, but MaskLLM achieves a significantly lower 6.72 PPL solely by learning the masks with frozen weights. Furthermore, MaskLLM's learnable nature allows customized masks for lossless application of 2:4 sparsity to downstream tasks or domains. Code is available at \url{[https://github.com/NVlabs/MaskLLM}](https://github.com/NVlabs/MaskLLM%7D).

**URL** : <http://arxiv.org/pdf/2409.17481v1>

**Published** : 2024-09-26

## RED QUEEN: Safeguarding Large Language Models against Concealed Multi-Turn Jailbreaking

**Authors** : Yifan Jiang, Kriti Aggarwal, Tanmay Laud, Kashif Munir, Jay Pujara, Subhabrata Mukherjee

**Abstract** : The rapid progress of Large Language Models (LLMs) has opened up new opportunities across various domains and applications; yet it also presents challenges related to potential misuse. To mitigate such risks, red teaming has been employed as a proactive security measure to probe language models for harmful outputs via jailbreak attacks. However, current jailbreak attack approaches are single-turn with explicit malicious queries that do not fully capture the complexity of real-world interactions. In reality, users can engage in multi-turn interactions with LLM-based chat assistants, allowing them to conceal their true intentions in a more covert manner. To bridge this gap, we, first, propose a new jailbreak approach, RED QUEEN ATTACK. This method constructs a multi-turn scenario, concealing the malicious intent under the guise of preventing harm. We craft 40 scenarios that vary in turns and select 14 harmful categories to generate 56k multi-turn attack data points. We conduct comprehensive experiments on the RED QUEEN ATTACK with four representative LLM families of different sizes. Our experiments reveal that all LLMs are vulnerable to RED QUEEN ATTACK, reaching 87.62% attack success rate on GPT-4o and 75.4% on Llama3-70B. Further analysis reveals that larger models are more susceptible to the RED QUEEN ATTACK, with multi-turn structures and concealment strategies contributing to its success. To prioritize safety, we introduce a straightforward mitigation strategy called RED QUEEN GUARD, which aligns LLMs to effectively counter adversarial attacks. This approach reduces the attack success rate to below 1% while maintaining the model's performance across standard benchmarks. Full implementation and dataset are publicly accessible at <https://github.com/kriti-hippo/red_queen>.

**URL** : <http://arxiv.org/pdf/2409.17458v1>

**Published** : 2024-09-26

## CadVLM: Bridging Language and Vision in the Generation of Parametric CAD Sketches

**Authors** : Sifan Wu, Amir Khasahmadi, Mor Katz, Pradeep Kumar Jayaraman, Yewen Pu, Karl Willis, Bang Liu

**Abstract** : Parametric Computer-Aided Design (CAD) is central to contemporary mechanical design. However, it encounters challenges in achieving precise parametric sketch modeling and lacks practical evaluation metrics suitable for mechanical design. We harness the capabilities of pre-trained foundation models, renowned for their successes in natural language processing and computer vision, to develop generative models specifically for CAD. These models are adept at understanding complex geometries and design reasoning, a crucial advancement in CAD technology. In this paper, we propose CadVLM, an end-to-end vision language model for CAD generation. Our approach involves adapting pre-trained foundation models to manipulate engineering sketches effectively, integrating both sketch primitive sequences and sketch images. Extensive experiments demonstrate superior performance on multiple CAD sketch generation tasks such as CAD autocompletion, CAD autoconstraint, and image conditional generation. To our knowledge, this is the first instance of a multimodal Large Language Model (LLM) being successfully applied to parametric CAD generation, representing a pioneering step in the field of computer-aided mechanical design.

**URL** : <http://arxiv.org/pdf/2409.17457v1>

**Published** : 2024-09-26

## Navigating the Shortcut Maze: A Comprehensive Analysis of Shortcut Learning in Text Classification by Language Models

**Authors** : Yuqing Zhou, Ruixiang Tang, Ziyu Yao, Ziwei Zhu

**Abstract** : Language models (LMs), despite their advances, often depend on spurious correlations, undermining their accuracy and generalizability. This study addresses the overlooked impact of subtler, more complex shortcuts that compromise model reliability beyond oversimplified shortcuts. We introduce a comprehensive benchmark that categorizes shortcuts into occurrence, style, and concept, aiming to explore the nuanced ways in which these shortcuts influence the performance of LMs. Through extensive experiments across traditional LMs, large language models, and state-of-the-art robust models, our research systematically investigates models' resilience and susceptibilities to sophisticated shortcuts. Our benchmark and code can be found at: <https://github.com/yuqing-zhou/shortcut-learning-in-text-classification>.

**URL** : <http://arxiv.org/pdf/2409.17455v1>

**Published** : 2024-09-26

## Enhancing Financial Sentiment Analysis with Expert-Designed Hint

**Authors** : Chung-Chi Chen, Hiroya Takamura, Ichiro Kobayashi, Yusuke Miyao

**Abstract** : This paper investigates the role of expert-designed hint in enhancing sentiment analysis on financial social media posts. We explore the capability of large language models (LLMs) to empathize with writer perspectives and analyze sentiments. Our findings reveal that expert-designed hint, i.e., pointing out the importance of numbers, significantly improve performances across various LLMs, particularly in cases requiring perspective-taking skills. Further analysis on tweets containing different types of numerical data demonstrates that the inclusion of expert-designed hint leads to notable improvements in sentiment analysis performance, especially for tweets with monetary-related numbers. Our findings contribute to the ongoing discussion on the applicability of Theory of Mind in NLP and open new avenues for improving sentiment analysis in financial domains through the strategic use of expert knowledge.

**URL** : <http://arxiv.org/pdf/2409.17448v1>

**Published** : 2024-09-26

## HDFlow: Enhancing LLM Complex Problem-Solving with Hybrid Thinking and Dynamic Workflows

**Authors** : Wenlin Yao, Haitao Mi, Dong Yu

**Abstract** : Despite recent advancements in large language models (LLMs), their performance on complex reasoning problems requiring multi-step thinking and combining various skills is still limited. To address this, we propose a novel framework HDFlow for complex reasoning with LLMs that combines fast and slow thinking modes in an adaptive manner. Our approach consists of two key components: 1) a new approach for slow, deliberate reasoning called Dynamic Workflow, which automatically decomposes complex problems into more manageable sub-tasks and dynamically designs a workflow to assemble specialized LLM or symbolic reasoning tools to solve sub-tasks; 2) Hybrid Thinking, a general framework that dynamically combines fast and slow thinking based on problem complexity. Finally, we propose an easy-to-scale method for automatically synthesizing a large-scale dataset of 27K challenging reasoning problems for complex reasoning and a hybrid thinking tuning method that trains smaller LLMs on this dataset to internalize the fast/slow hybrid reasoning strategies. Experiments on four reasoning benchmark datasets demonstrate that our slow thinking with dynamic workflows significantly outperforms Chain-of-Thought, and hybrid thinking achieves the highest accuracy while providing an effective balance between computational efficiency and performance. Fine-tuning using our hybrid thinking approach also significantly boosts the complex reasoning capabilities of open-source language models. The results showcase the promise of slow thinking, dynamic workflows, and hybrid thinking in expanding the frontier of complex problem-solving with LLMs\footnote{Code and data will be released at \url{[https://github.com/wenlinyao/HDFlow}.}](https://github.com/wenlinyao/HDFlow%7D.%7D).

**URL** : <http://arxiv.org/pdf/2409.17433v1>

**Published** : 2024-09-25

## Discovering the Gems in Early Layers: Accelerating Long-Context LLMs with 1000x Input Token Reduction

**Authors** : Zhenmei Shi, Yifei Ming, Xuan-Phi Nguyen, Yingyu Liang, Shafiq Joty

**Abstract** : Large Language Models (LLMs) have demonstrated remarkable capabilities in handling long context inputs, but this comes at the cost of increased computational resources and latency. Our research introduces a novel approach for the long context bottleneck to accelerate LLM inference and reduce GPU memory consumption. Our research demonstrates that LLMs can identify relevant tokens in the early layers before generating answers to a query. Leveraging this insight, we propose an algorithm that uses early layers of an LLM as filters to select and compress input tokens, significantly reducing the context length for subsequent processing. Our method, GemFilter, demonstrates substantial improvements in both speed and memory efficiency compared to existing techniques, such as standard attention and SnapKV/H2O. Notably, it achieves a 2.4$\times$ speedup and 30% reduction in GPU memory usage compared to SOTA methods. Evaluation on the Needle in a Haystack task shows that GemFilter significantly outperforms standard attention, SnapKV and demonstrates comparable performance on the LongBench challenge. GemFilter is simple, training-free, and broadly applicable across different LLMs. Crucially, it provides interpretability by allowing humans to inspect the selected input sequence. These findings not only offer practical benefits for LLM deployment, but also enhance our understanding of LLM internal mechanisms, paving the way for further optimizations in LLM design and inference. Our code is available at \url{[https://github.com/SalesforceAIResearch/GemFilter}](https://github.com/SalesforceAIResearch/GemFilter%7D).

**URL** : <http://arxiv.org/pdf/2409.17422v1>

**Published** : 2024-09-25

## From Deception to Detection: The Dual Roles of Large Language Models in Fake News

**Authors** : Dorsaf Sallami, Yuan-Chen Chang, Esma Aïmeur

**Abstract** : Fake news poses a significant threat to the integrity of information ecosystems and public trust. The advent of Large Language Models (LLMs) holds considerable promise for transforming the battle against fake news. Generally, LLMs represent a double-edged sword in this struggle. One major concern is that LLMs can be readily used to craft and disseminate misleading information on a large scale. This raises the pressing questions: Can LLMs easily generate biased fake news? Do all LLMs have this capability? Conversely, LLMs offer valuable prospects for countering fake news, thanks to their extensive knowledge of the world and robust reasoning capabilities. This leads to other critical inquiries: Can we use LLMs to detect fake news, and do they outperform typical detection models? In this paper, we aim to address these pivotal questions by exploring the performance of various LLMs. Our objective is to explore the capability of various LLMs in effectively combating fake news, marking this as the first investigation to analyze seven such models. Our results reveal that while some models adhere strictly to safety protocols, refusing to generate biased or misleading content, other models can readily produce fake news across a spectrum of biases. Additionally, our results show that larger models generally exhibit superior detection abilities and that LLM-generated fake news are less likely to be detected than human-written ones. Finally, our findings demonstrate that users can benefit from LLM-generated explanations in identifying fake news.

**URL** : <http://arxiv.org/pdf/2409.17416v1>

**Published** : 2024-09-25

## Sociotechnical Approach to Enterprise Generative Artificial Intelligence (E-GenAI)

**Authors** : Leoncio Jimenez, Francisco Venegas

**Abstract** : In this theoretical article, a sociotechnical approach is proposed to characterize. First, the business ecosystem, focusing on the relationships among Providers, Enterprise, and Customers through SCM, ERP, and CRM platforms to align: (1) Business Intelligence (BI), Fuzzy Logic (FL), and TRIZ (Theory of Inventive Problem Solving), through the OID model, and (2) Knowledge Management (KM) and Imperfect Knowledge Management (IKM), through the OIDK model. Second, the article explores the E-GenAI business ecosystem, which integrates GenAI-based platforms for SCM, ERP, and CRM with GenAI-based platforms for BI, FL, TRIZ, KM, and IKM, to align Large Language Models (LLMs) through the E-GenAI (OID) model. Finally, to understand the dynamics of LLMs, we utilize finite automata to model the relationships between Followers and Followees. This facilitates the construction of LLMs that can identify specific characteristics of users on a social media platform.

**URL** : <http://arxiv.org/pdf/2409.17408v1>

**Published** : 2024-09-25

## Post-hoc Reward Calibration: A Case Study on Length Bias

**Authors** : Zeyu Huang, Zihan Qiu, Zili Wang, Edoardo M. Ponti, Ivan Titov

**Abstract** : Reinforcement Learning from Human Feedback aligns the outputs of Large Language Models with human values and preferences. Central to this process is the reward model (RM), which translates human feedback into training signals for optimising LLM behaviour. However, RMs can develop biases by exploiting spurious correlations in their training data, such as favouring outputs based on length or style rather than true quality. These biases can lead to incorrect output rankings, sub-optimal model evaluations, and the amplification of undesirable behaviours in LLMs alignment. This paper addresses the challenge of correcting such biases without additional data and training, introducing the concept of Post-hoc Reward Calibration. We first propose an intuitive approach to estimate the bias term and, thus, remove it to approximate the underlying true reward. We then extend the approach to a more general and robust form with the Locally Weighted Regression. Focusing on the prevalent length bias, we validate our proposed approaches across three experimental settings, demonstrating consistent improvements: (1) a 3.11 average performance gain across 33 reward models on the RewardBench dataset; (2) enhanced alignment of RM rankings with GPT-4 evaluations and human preferences based on the AlpacaEval benchmark; and (3) improved Length-Controlled win rate of the RLHF process in multiple LLM--RM combinations. Our method is computationally efficient and generalisable to other types of bias and RMs, offering a scalable and robust solution for mitigating biases in LLM alignment. Our code and results are available at <https://github.com/ZeroYuHuang/Reward-Calibration>.

**URL** : <http://arxiv.org/pdf/2409.17407v1>

**Published** : 2024-09-25

## Severity Prediction in Mental Health: LLM-based Creation, Analysis, Evaluation of a Novel Multilingual Dataset

**Authors** : Konstantinos Skianis, John Pavlopoulos, A. Seza Doğruöz

**Abstract** : Large Language Models (LLMs) are increasingly integrated into various medical fields, including mental health support systems. However, there is a gap in research regarding the effectiveness of LLMs in non-English mental health support applications. To address this problem, we present a novel multilingual adaptation of widely-used mental health datasets, translated from English into six languages (Greek, Turkish, French, Portuguese, German, and Finnish). This dataset enables a comprehensive evaluation of LLM performance in detecting mental health conditions and assessing their severity across multiple languages. By experimenting with GPT and Llama, we observe considerable variability in performance across languages, despite being evaluated on the same translated dataset. This inconsistency underscores the complexities inherent in multilingual mental health support, where language-specific nuances and mental health data coverage can affect the accuracy of the models. Through comprehensive error analysis, we emphasize the risks of relying exclusively on large language models (LLMs) in medical settings (e.g., their potential to contribute to misdiagnoses). Moreover, our proposed approach offers significant cost savings for multilingual tasks, presenting a major advantage for broad-scale implementation.

**URL** : <http://arxiv.org/pdf/2409.17397v1>

**Published** : 2024-09-25

## Scaling Behavior for Large Language Models regarding Numeral Systems: An Example using Pythia

**Authors** : Zhejian Zhou, Jiayu Wang, Dahua Lin, Kai Chen

**Abstract** : Though Large Language Models (LLMs) have shown remarkable abilities in mathematics reasoning, they are still struggling with performing numeric operations accurately, such as addition and multiplication. Numbers can be tokenized into tokens in various ways by different LLMs and affect the numeric operations performance. Currently, there are two representatives: 1) Tokenize into $1$-digit, and 2) Tokenize into $1\sim 3$ digit. The difference is roughly equivalent to using different numeral systems (namely base $10$ or base $10^{3}$). In light of this, we study the scaling behavior of different numeral systems in the context of transformer-based large language models. We empirically show that a base $10$ system is consistently more data-efficient than a base $10^{2}$ or $10^{3}$ system across training data scale, model sizes under from-scratch training settings, while different number systems have very similar fine-tuning performances. We attribute this to higher token frequencies of a base $10$ system. Additionally, we reveal extrapolation behavior patterns on addition and multiplication. We identify that base $100$ and base $1000$ systems struggle on token-level discernment and token-level operations. We also sheds light on the mechanism learnt by the models.

**URL** : <http://arxiv.org/pdf/2409.17391v2>

**Published** : 2024-09-25

## Search for Efficient Large Language Models

**Authors** : Xuan Shen, Pu Zhao, Yifan Gong, Zhenglun Kong, Zheng Zhan, Yushu Wu, Ming Lin, Chao Wu, Xue Lin, Yanzhi Wang

**Abstract** : Large Language Models (LLMs) have long held sway in the realms of artificial intelligence research. Numerous efficient techniques, including weight pruning, quantization, and distillation, have been embraced to compress LLMs, targeting memory reduction and inference acceleration, which underscore the redundancy in LLMs. However, most model compression techniques concentrate on weight optimization, overlooking the exploration of optimal architectures. Besides, traditional architecture search methods, limited by the elevated complexity with extensive parameters, struggle to demonstrate their effectiveness on LLMs. In this paper, we propose a training-free architecture search framework to identify optimal subnets that preserve the fundamental strengths of the original LLMs while achieving inference acceleration. Furthermore, after generating subnets that inherit specific weights from the original LLMs, we introduce a reformation algorithm that utilizes the omitted weights to rectify the inherited weights with a small amount of calibration data. Compared with SOTA training-free structured pruning works that can generate smaller networks, our method demonstrates superior performance across standard benchmarks. Furthermore, our generated subnets can directly reduce the usage of GPU memory and achieve inference acceleration.

**URL** : <http://arxiv.org/pdf/2409.17372v1>

**Published** : 2024-09-25

## Internalizing ASR with Implicit Chain of Thought for Efficient Speech-to-Speech Conversational LLM

**Authors** : Robin Shing-Hei Yuen, Timothy Tin-Long Tse, Jian Zhu

**Abstract** : Current speech-based LLMs are predominantly trained on extensive ASR and TTS datasets, excelling in tasks related to these domains. However, their ability to handle direct speech-to-speech conversations remains notably constrained. These models often rely on an ASR-to-TTS chain-of-thought pipeline, converting speech into text for processing before generating audio responses, which introduces latency and loses audio features. We propose a method that implicitly internalizes ASR chain of thought into a speech LLM, enhancing its native speech understanding capabilities. Our approach reduces latency and improves the model's native understanding of speech, paving the way for more efficient and natural real-time audio interactions. We also release a large-scale synthetic conversational dataset to facilitate further research.

**URL** : <http://arxiv.org/pdf/2409.17353v2>

**Published** : 2024-09-25

## Navigating the Nuances: A Fine-grained Evaluation of Vision-Language Navigation

**Authors** : Zehao Wang, Minye Wu, Yixin Cao, Yubo Ma, Meiqi Chen, Tinne Tuytelaars

**Abstract** : This study presents a novel evaluation framework for the Vision-Language Navigation (VLN) task. It aims to diagnose current models for various instruction categories at a finer-grained level. The framework is structured around the context-free grammar (CFG) of the task. The CFG serves as the basis for the problem decomposition and the core premise of the instruction categories design. We propose a semi-automatic method for CFG construction with the help of Large-Language Models (LLMs). Then, we induct and generate data spanning five principal instruction categories (i.e. direction change, landmark recognition, region recognition, vertical movement, and numerical comprehension). Our analysis of different models reveals notable performance discrepancies and recurrent issues. The stagnation of numerical comprehension, heavy selective biases over directional concepts, and other interesting findings contribute to the development of future language-guided navigation systems.

**URL** : <http://arxiv.org/pdf/2409.17313v1>

**Published** : 2024-09-25

## Proof of Thought : Neurosymbolic Program Synthesis allows Robust and Interpretable Reasoning

**Authors** : Debargha Ganguly, Srinivasan Iyengar, Vipin Chaudhary, Shivkumar Kalyanaraman

**Abstract** : Large Language Models (LLMs) have revolutionized natural language processing, yet they struggle with inconsistent reasoning, particularly in novel domains and complex logical sequences. This research introduces Proof of Thought, a framework that enhances the reliability and transparency of LLM outputs. Our approach bridges LLM-generated ideas with formal logic verification, employing a custom interpreter to convert LLM outputs into First Order Logic constructs for theorem prover scrutiny. Central to our method is an intermediary JSON-based Domain-Specific Language, which by design balances precise logical structures with intuitive human concepts. This hybrid representation enables both rigorous validation and accessible human comprehension of LLM reasoning processes. Key contributions include a robust type system with sort management for enhanced logical integrity, explicit representation of rules for clear distinction between factual and inferential knowledge, and a flexible architecture that allows for easy extension to various domain-specific applications. We demonstrate Proof of Thought's effectiveness through benchmarking on StrategyQA and a novel multimodal reasoning task, showing improved performance in open-ended scenarios. By providing verifiable and interpretable results, our technique addresses critical needs for AI system accountability and sets a foundation for human-in-the-loop oversight in high-stakes domains.

**URL** : <http://arxiv.org/pdf/2409.17270v1>

**Published** : 2024-09-25

## AAPM: Large Language Model Agent-based Asset Pricing Models

**Authors** : Junyan Cheng, Peter Chin

**Abstract** : In this study, we propose a novel asset pricing approach, LLM Agent-based Asset Pricing Models (AAPM), which fuses qualitative discretionary investment analysis from LLM agents and quantitative manual financial economic factors to predict excess asset returns. The experimental results show that our approach outperforms machine learning-based asset pricing baselines in portfolio optimization and asset pricing errors. Specifically, the Sharpe ratio and average $|\alpha|$ for anomaly portfolios improved significantly by 9.6% and 10.8% respectively. In addition, we conducted extensive ablation studies on our model and analysis of the data to reveal further insights into the proposed method.

**URL** : <http://arxiv.org/pdf/2409.17266v1>

**Published** : 2024-09-25

## Attention Prompting on Image for Large Vision-Language Models

**Authors** : Runpeng Yu, Weihao Yu, Xinchao Wang

**Abstract** : Compared with Large Language Models (LLMs), Large Vision-Language Models (LVLMs) can also accept images as input, thus showcasing more interesting emergent capabilities and demonstrating impressive performance on various vision-language tasks. Motivated by text prompting in LLMs, visual prompting has been explored to enhance LVLMs' capabilities of perceiving visual information. However, previous visual prompting techniques solely process visual inputs without considering text queries, limiting the models' ability to follow text instructions to complete tasks. To fill this gap, in this work, we propose a new prompting technique named Attention Prompting on Image, which just simply overlays a text-query-guided attention heatmap on the original input image and effectively enhances LVLM on various tasks. Specifically, we generate an attention heatmap for the input image dependent on the text query with an auxiliary model like CLIP. Then the heatmap simply multiplies the pixel values of the original image to obtain the actual input image for the LVLM. Extensive experiments on various vison-language benchmarks verify the effectiveness of our technique. For example, Attention Prompting on Image improves LLaVA-1.5 by 3.8% and 2.9% on MM-Vet and LLaVA-Wild benchmarks, respectively.

**URL** : <http://arxiv.org/pdf/2409.17143v1>

**Published** : 2024-09-25

## FineZip : Pushing the Limits of Large Language Models for Practical Lossless Text Compression

**Authors** : Fazal Mittu, Yihuan Bu, Akshat Gupta, Ashok Devireddy, Alp Eren Ozdarendeli, Anant Singh, Gopala Anumanchipalli

**Abstract** : While the language modeling objective has been shown to be deeply connected with compression, it is surprising that modern LLMs are not employed in practical text compression systems. In this paper, we provide an in-depth analysis of neural network and transformer-based compression techniques to answer this question. We compare traditional text compression systems with neural network and LLM-based text compression methods. Although LLM-based systems significantly outperform conventional compression methods, they are highly impractical. Specifically, LLMZip, a recent text compression system using Llama3-8B requires 9.5 days to compress just 10 MB of text, although with huge improvements in compression ratios. To overcome this, we present FineZip - a novel LLM-based text compression system that combines ideas of online memorization and dynamic context to reduce the compression time immensely. FineZip can compress the above corpus in approximately 4 hours compared to 9.5 days, a 54 times improvement over LLMZip and comparable performance. FineZip outperforms traditional algorithmic compression methods with a large margin, improving compression ratios by approximately 50%. With this work, we take the first step towards making lossless text compression with LLMs a reality. While FineZip presents a significant step in that direction, LLMs are still not a viable solution for large-scale text compression. We hope our work paves the way for future research and innovation to solve this problem.

**URL** : <http://arxiv.org/pdf/2409.17141v1>

**Published** : 2024-09-25

## Turn Every Application into an Agent: Towards Efficient Human-Agent-Computer Interaction with API-First LLM-Based Agents

**Authors** : Junting Lu, Zhiyang Zhang, Fangkai Yang, Jue Zhang, Lu Wang, Chao Du, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, Qi Zhang

**Abstract** : Multimodal large language models (MLLMs) have enabled LLM-based agents to directly interact with application user interfaces (UIs), enhancing agents' performance in complex tasks. However, these agents often suffer from high latency and low reliability due to the extensive sequential UI interactions. To address this issue, we propose AXIS, a novel LLM-based agents framework prioritize actions through application programming interfaces (APIs) over UI actions. This framework also facilitates the creation and expansion of APIs through automated exploration of applications. Our experiments on Office Word demonstrate that AXIS reduces task completion time by 65%-70% and cognitive workload by 38%-53%, while maintaining accuracy of 97%-98% compare to humans. Our work contributes to a new human-agent-computer interaction (HACI) framework and a fresh UI design principle for application providers in the era of LLMs. It also explores the possibility of turning every applications into agents, paving the way towards an agent-centric operating system (Agent OS).

**URL** : <http://arxiv.org/pdf/2409.17140v1>

**Published** : 2024-09-25

## Plurals: A System for Guiding LLMs Via Simulated Social Ensembles

**Authors** : Joshua Ashkinaze, Emily Fry, Narendra Edara, Eric Gilbert, Ceren Budak

**Abstract** : Recent debates raised concerns that language models may favor certain viewpoints. But what if the solution is not to aim for a 'view from nowhere' but rather to leverage different viewpoints? We introduce Plurals, a system and Python library for pluralistic AI deliberation. Plurals consists of Agents (LLMs, optionally with personas) which deliberate within customizable Structures, with Moderators overseeing deliberation. Plurals is a generator of simulated social ensembles. Plurals integrates with government datasets to create nationally representative personas, includes deliberation templates inspired by democratic deliberation theory, and allows users to customize both information-sharing structures and deliberation behavior within Structures. Six case studies demonstrate fidelity to theoretical constructs and efficacy. Three randomized experiments show simulated focus groups produced output resonant with an online sample of the relevant audiences (chosen over zero-shot generation in 75% of trials). Plurals is both a paradigm and a concrete system for pluralistic AI. The Plurals library is available at <https://github.com/josh-ashkinaze/plurals> and will be continually updated.

**URL** : <http://arxiv.org/pdf/2409.17213v2>

**Published** : 2024-09-25

## Programming Every Example: Lifting Pre-training Data Quality like Experts at Scale

**Authors** : Fan Zhou, Zengzhi Wang, Qian Liu, Junlong Li, Pengfei Liu

**Abstract** : Large language model pre-training has traditionally relied on human experts to craft heuristics for improving the corpora quality, resulting in numerous rules developed to date. However, these rules lack the flexibility to address the unique characteristics of individual example effectively. Meanwhile, applying tailored rules to every example is impractical for human experts. In this paper, we demonstrate that even small language models, with as few as 0.3B parameters, can exhibit substantial data refining capabilities comparable to those of human experts. We introduce Programming Every Example (ProX), a novel framework that treats data refinement as a programming task, enabling models to refine corpora by generating and executing fine-grained operations, such as string normalization, for each individual example at scale. Experimental results show that models pre-trained on ProX-curated data outperform either original data or data filtered by other selection methods by more than 2% across various downstream benchmarks. Its effectiveness spans various model sizes and pre-training corpora, including C4, RedPajama-V2, and FineWeb. Furthermore, ProX exhibits significant potential in domain-specific continual pre-training: without domain specific design, models trained on OpenWebMath refined by ProX outperform human-crafted rule-based methods, improving average accuracy by 7.6% over Mistral-7B, with 14.6% for Llama-2-7B and 20.3% for CodeLlama-7B, all within 10B tokens to be comparable to models like Llemma-7B trained on 200B tokens. Further analysis highlights that ProX significantly saves training FLOPs, offering a promising path for efficient LLM pre-training.We are open-sourcing ProX with >100B corpus, models, and sharing all training and implementation details for reproducible research and future innovation. Code: <https://github.com/GAIR-NLP/ProX>

**URL** : <http://arxiv.org/pdf/2409.17115v1>

**Published** : 2024-09-25

## Accumulator-Aware Post-Training Quantization

**Authors** : Ian Colbert, Fabian Grob, Giuseppe Franco, Jinjie Zhang, Rayan Saab

**Abstract** : Several recent studies have investigated low-precision accumulation, reporting improvements in throughput, power, and area across various platforms. However, the accompanying proposals have only considered the quantization-aware training (QAT) paradigm, in which models are fine-tuned or trained from scratch with quantization in the loop. As models continue to grow in size, QAT techniques become increasingly more expensive, which has motivated the recent surge in post-training quantization (PTQ) research. To the best of our knowledge, ours marks the first formal study of accumulator-aware quantization in the PTQ setting. To bridge this gap, we introduce AXE, a practical framework of accumulator-aware extensions designed to endow overflow avoidance guarantees to existing layer-wise PTQ algorithms. We theoretically motivate AXE and demonstrate its flexibility by implementing it on top of two state-of-the-art PTQ algorithms: GPFQ and OPTQ. We further generalize AXE to support multi-stage accumulation for the first time, opening the door for full datapath optimization and scaling to large language models (LLMs). We evaluate AXE across image classification and language generation models, and observe significant improvements in the trade-off between accumulator bit width and model accuracy over baseline methods.

**URL** : <http://arxiv.org/pdf/2409.17092v1>

**Published** : 2024-09-25

## Enhancing Post-Hoc Attributions in Long Document Comprehension via Coarse Grained Answer Decomposition

**Authors** : Pritika Ramu, Koustava Goswami, Apoorv Saxena, Balaji Vasan Srinivavsan

**Abstract** : Accurately attributing answer text to its source document is crucial for developing a reliable question-answering system. However, attribution for long documents remains largely unexplored. Post-hoc attribution systems are designed to map answer text back to the source document, yet the granularity of this mapping has not been addressed. Furthermore, a critical question arises: What exactly should be attributed? This involves identifying the specific information units within an answer that require grounding. In this paper, we propose and investigate a novel approach to the factual decomposition of generated answers for attribution, employing template-based in-context learning. To accomplish this, we utilize the question and integrate negative sampling during few-shot in-context learning for decomposition. This approach enhances the semantic understanding of both abstractive and extractive answers. We examine the impact of answer decomposition by providing a thorough examination of various attribution approaches, ranging from retrieval-based techniques to LLM-based attributors.

**URL** : <http://arxiv.org/pdf/2409.17073v2>

**Published** : 2024-09-25

## VPTQ: Extreme Low-bit Vector Post-Training Quantization for Large Language Models

**Authors** : Yifei Liu, Jicheng Wen, Yang Wang, Shengyu Ye, Li Lyna Zhang, Ting Cao, Cheng Li, Mao Yang

**Abstract** : Scaling model size significantly challenges the deployment and inference of Large Language Models (LLMs). Due to the redundancy in LLM weights, recent research has focused on pushing weight-only quantization to extremely low-bit (even down to 2 bits). It reduces memory requirements, optimizes storage costs, and decreases memory bandwidth needs during inference. However, due to numerical representation limitations, traditional scalar-based weight quantization struggles to achieve such extreme low-bit. Recent research on Vector Quantization (VQ) for LLMs has demonstrated the potential for extremely low-bit model quantization by compressing vectors into indices using lookup tables. In this paper, we introduce Vector Post-Training Quantization (VPTQ) for extremely low-bit quantization of LLMs. We use Second-Order Optimization to formulate the LLM VQ problem and guide our quantization algorithm design by solving the optimization. We further refine the weights using Channel-Independent Second-Order Optimization for a granular VQ. In addition, by decomposing the optimization problem, we propose a brief and effective codebook initialization algorithm. We also extend VPTQ to support residual and outlier quantization, which enhances model accuracy and further compresses the model. Our experimental results show that VPTQ reduces model quantization perplexity by $0.01$-$0.34$ on LLaMA-2, $0.38$-$0.68$ on Mistral-7B, $4.41$-$7.34$ on LLaMA-3 over SOTA at 2-bit, with an average accuracy improvement of $0.79$-$1.5%$ on LLaMA-2, $1%$ on Mistral-7B, $11$-$22%$ on LLaMA-3 on QA tasks on average. We only utilize $10.4$-$18.6%$ of the quantization algorithm execution time, resulting in a $1.6$-$1.8\times$ increase in inference throughput compared to SOTA.

**URL** : <http://arxiv.org/pdf/2409.17066v1>

**Published** : 2024-09-25

## Using LLM for Real-Time Transcription and Summarization of Doctor-Patient Interactions into ePuskesmas in Indonesia

**Authors** : Azmul Asmar Irfan, Nur Ahmad Khatim, Mansur M. Arief

**Abstract** : One of the key issues contributing to inefficiency in Puskesmas is the time-consuming nature of doctor-patient interactions. Doctors need to conduct thorough consultations, which include diagnosing the patient's condition, providing treatment advice, and transcribing detailed notes into medical records. In regions with diverse linguistic backgrounds, doctors often have to ask clarifying questions, further prolonging the process. While diagnosing is essential, transcription and summarization can often be automated using AI to improve time efficiency and help doctors enhance care quality and enable early diagnosis and intervention. This paper proposes a solution using a localized large language model (LLM) to transcribe, translate, and summarize doctor-patient conversations. We utilize the Whisper model for transcription and GPT-3 to summarize them into the ePuskemas medical records format. This system is implemented as an add-on to an existing web browser extension, allowing doctors to fill out patient forms while talking. By leveraging this solution for real-time transcription, translation, and summarization, doctors can improve the turnaround time for patient care while enhancing the quality of records, which become more detailed and insightful for future visits. This innovation addresses challenges like overcrowded facilities and the administrative burden on healthcare providers in Indonesia. We believe this solution will help doctors save time, provide better care, and produce more accurate medical records, representing a significant step toward modernizing healthcare and ensuring patients receive timely, high-quality care, even in resource-constrained settings.

**URL** : <http://arxiv.org/pdf/2409.17054v1>

**Published** : 2024-09-25

## How to Connect Speech Foundation Models and Large Language Models? What Matters and What Does Not

**Authors** : Francesco Verdini, Pierfrancesco Melucci, Stefano Perna, Francesco Cariaggi, Marco Gaido, Sara Papi, Szymon Mazurek, Marek Kasztelnik, Luisa Bentivogli, Sébastien Bratières, Paolo Merialdo, Simone Scardapane

**Abstract** : The remarkable performance achieved by Large Language Models (LLM) has driven research efforts to leverage them for a wide range of tasks and input modalities. In speech-to-text (S2T) tasks, the emerging solution consists of projecting the output of the encoder of a Speech Foundational Model (SFM) into the LLM embedding space through an adapter module. However, no work has yet investigated how much the downstream-task performance depends on each component (SFM, adapter, LLM) nor whether the best design of the adapter depends on the chosen SFM and LLM. To fill this gap, we evaluate the combination of 5 adapter modules, 2 LLMs (Mistral and Llama), and 2 SFMs (Whisper and SeamlessM4T) on two widespread S2T tasks, namely Automatic Speech Recognition and Speech Translation. Our results demonstrate that the SFM plays a pivotal role in downstream performance, while the adapter choice has moderate impact and depends on the SFM and LLM.

**URL** : <http://arxiv.org/pdf/2409.17044v1>

**Published** : 2024-09-25

## Counterfactual Token Generation in Large Language Models

**Authors** : Ivi Chatzi, Nina Corvelo Benz, Eleni Straitouri, Stratis Tsirtsis, Manuel Gomez-Rodriguez

**Abstract** : "Sure, I am happy to generate a story for you: Captain Lyra stood at the helm of her trusty ship, the Maelstrom's Fury, gazing out at the endless sea. [...] Lyra's eyes welled up with tears as she realized the bitter truth - she had sacrificed everything for fleeting riches, and lost the love of her crew, her family, and herself." Although this story, generated by a large language model, is captivating, one may wonder -- how would the story have unfolded if the model had chosen "Captain Maeve" as the protagonist instead? We cannot know. State-of-the-art large language models are stateless -- they maintain no internal memory or state. Given a prompt, they generate a sequence of tokens as an output using an autoregressive process. As a consequence, they cannot reason about counterfactual alternatives to tokens they have generated in the past. In this work, our goal is to enhance them with this functionality. To this end, we develop a causal model of token generation that builds upon the Gumbel-Max structural causal model. Our model allows any large language model to perform counterfactual token generation at almost no cost in comparison with vanilla token generation, it is embarrassingly simple to implement, and it does not require any fine-tuning nor prompt engineering. We implement our model on Llama 3 8B-instruct and conduct both qualitative and quantitative analyses of counterfactually generated text. We conclude with a demonstrative application of counterfactual token generation for bias detection, unveiling interesting insights about the model of the world constructed by large language models.

**URL** : <http://arxiv.org/pdf/2409.17027v1>

**Published** : 2024-09-25

## LLM-CARD: Towards a Description and Landscape of Large Language Models

**Authors** : Shengwei Tian, Lifeng Han, Erick Mendez Guzman, Goran Nenadic

**Abstract** : With the rapid growth of the Natural Language Processing (NLP) field, a vast variety of Large Language Models (LLMs) continue to emerge for diverse NLP tasks. As an increasing number of papers are presented, researchers and developers face the challenge of information overload. Thus, it is particularly important to develop a system that can automatically extract and organise key information about LLMs from academic papers (\textbf{LLM model card}). This work is to develop such a pioneer system by using Named Entity Recognition (\textbf{NER}) and Relation Extraction (\textbf{RE}) methods that automatically extract key information about large language models from the papers, helping researchers to efficiently access information about LLMs. These features include model \textit{licence}, model \textit{name}, and model \textit{application}. With these features, we can form a model card for each paper. \textbf{Data-contribution} wise, 106 academic papers were processed by defining three dictionaries - LLMs name, licence, and application. 11,051 sentences were extracted through dictionary lookup, and the dataset was constructed through manual review of the final selection of 129 sentences that have a link between the name and the licence, and 106 sentences that have a link between the model name and the application.

**URL** : <http://arxiv.org/pdf/2409.17011v1>

**Published** : 2024-09-25

## INT-FlashAttention: Enabling Flash Attention for INT8 Quantization

**Authors** : Shimao Chen, Zirui Liu, Zhiying Wu, Ce Zheng, Peizhuang Cong, Zihan Jiang, Yuhan Wu, Lei Su, Tong Yang

**Abstract** : As the foundation of large language models (LLMs), self-attention module faces the challenge of quadratic time and memory complexity with respect to sequence length. FlashAttention accelerates attention computation and reduces its memory usage by leveraging the GPU memory hierarchy. A promising research direction is to integrate FlashAttention with quantization methods. This paper introduces INT-FlashAttention, the first INT8 quantization architecture compatible with the forward workflow of FlashAttention, which significantly improves the inference speed of FlashAttention on Ampere GPUs. We implement our INT-FlashAttention prototype with fully INT8 activations and general matrix-multiplication (GEMM) kernels, making it the first attention operator with fully INT8 input. As a general token-level post-training quantization framework, INT-FlashAttention is also compatible with other data formats like INT4, etc. Experimental results show INT-FlashAttention achieves 72% faster inference speed and 82% smaller quantization error compared to standard FlashAttention with FP16 and FP8 data format.

**URL** : <http://arxiv.org/pdf/2409.16997v2>

**Published** : 2024-09-25

## Harnessing Diversity for Important Data Selection in Pretraining Large Language Models

**Authors** : Chi Zhang, Huaping Zhong, Kuan Zhang, Chengliang Chai, Rui Wang, Xinlin Zhuang, Tianyi Bai, Jiantao Qiu, Lei Cao, Ye Yuan, Guoren Wang, Conghui He

**Abstract** : Data selection is of great significance in pre-training large language models, given the variation in quality within the large-scale available training corpora. To achieve this, researchers are currently investigating the use of data influence to measure the importance of data instances, $i.e.,$ a high influence score indicates that incorporating this instance to the training set is likely to enhance the model performance. Consequently, they select the top-$k$ instances with the highest scores. However, this approach has several limitations. (1) Computing the influence of all available data is time-consuming. (2) The selected data instances are not diverse enough, which may hinder the pre-trained model's ability to generalize effectively to various downstream tasks. In this paper, we introduce \texttt{Quad}, a data selection approach that considers both quality and diversity by using data influence to achieve state-of-the-art pre-training results. In particular, noting that attention layers capture extensive semantic details, we have adapted the accelerated $iHVP$ computation methods for attention layers, enhancing our ability to evaluate the influence of data, $i.e.,$ its quality. For the diversity, \texttt{Quad} clusters the dataset into similar data instances within each cluster and diverse instances across different clusters. For each cluster, if we opt to select data from it, we take some samples to evaluate the influence to prevent processing all instances. To determine which clusters to select, we utilize the classic Multi-Armed Bandit method, treating each cluster as an arm. This approach favors clusters with highly influential instances (ensuring high quality) or clusters that have been selected less frequently (ensuring diversity), thereby well balancing between quality and diversity.

**URL** : <http://arxiv.org/pdf/2409.16986v1>

**Published** : 2024-09-25

## AXCEL: Automated eXplainable Consistency Evaluation using LLMs

**Authors** : P Aditya Sreekar, Sahil Verma, Suransh Chopra, Sarik Ghazarian, Abhishek Persad, Narayanan Sadagopan

**Abstract** : Large Language Models (LLMs) are widely used in both industry and academia for various tasks, yet evaluating the consistency of generated text responses continues to be a challenge. Traditional metrics like ROUGE and BLEU show a weak correlation with human judgment. More sophisticated metrics using Natural Language Inference (NLI) have shown improved correlations but are complex to implement, require domain-specific training due to poor cross-domain generalization, and lack explainability. More recently, prompt-based metrics using LLMs as evaluators have emerged; while they are easier to implement, they still lack explainability and depend on task-specific prompts, which limits their generalizability. This work introduces Automated eXplainable Consistency Evaluation using LLMs (AXCEL), a prompt-based consistency metric which offers explanations for the consistency scores by providing detailed reasoning and pinpointing inconsistent text spans. AXCEL is also a generalizable metric which can be adopted to multiple tasks without changing the prompt. AXCEL outperforms both non-prompt and prompt-based state-of-the-art (SOTA) metrics in detecting inconsistencies across summarization by 8.7%, free text generation by 6.2%, and data-to-text conversion tasks by 29.4%. We also evaluate the influence of underlying LLMs on prompt based metric performance and recalibrate the SOTA prompt-based metrics with the latest LLMs for fair comparison. Further, we show that AXCEL demonstrates strong performance using open source LLMs.

**URL** : <http://arxiv.org/pdf/2409.16984v1>

**Published** : 2024-09-25

## Decoding Large-Language Models: A Systematic Overview of Socio-Technical Impacts, Constraints, and Emerging Questions

**Authors** : Zeyneb N. Kaya, Souvick Ghosh

**Abstract** : There have been rapid advancements in the capabilities of large language models (LLMs) in recent years, greatly revolutionizing the field of natural language processing (NLP) and artificial intelligence (AI) to understand and interact with human language. Therefore, in this work, we conduct a systematic investigation of the literature to identify the prominent themes and directions of LLM developments, impacts, and limitations. Our findings illustrate the aims, methodologies, limitations, and future directions of LLM research. It includes responsible development considerations, algorithmic improvements, ethical challenges, and societal implications of LLM development. Overall, this paper provides a rigorous and comprehensive overview of current research in LLM and identifies potential directions for future development. The article highlights the application areas that could have a positive impact on society along with the ethical considerations.

**URL** : <http://arxiv.org/pdf/2409.16974v1>

**Published** : 2024-09-25

## Adaptive Self-Supervised Learning Strategies for Dynamic On-Device LLM Personalization

**Authors** : Rafael Mendoza, Isabella Cruz, Richard Liu, Aarav Deshmukh, David Williams, Jesscia Peng, Rohan Iyer

**Abstract** : Large language models (LLMs) have revolutionized how we interact with technology, but their personalization to individual user preferences remains a significant challenge, particularly in on-device applications. Traditional methods often depend heavily on labeled datasets and can be resource-intensive. To address these issues, we present Adaptive Self-Supervised Learning Strategies (ASLS), which utilizes self-supervised learning techniques to personalize LLMs dynamically. The framework comprises a user profiling layer for collecting interaction data and a neural adaptation layer for real-time model fine-tuning. This innovative approach enables continuous learning from user feedback, allowing the model to generate responses that align closely with user-specific contexts. The adaptive mechanisms of ASLS minimize computational demands and enhance personalization efficiency. Experimental results across various user scenarios illustrate the superior performance of ASLS in boosting user engagement and satisfaction, highlighting its potential to redefine LLMs as highly responsive and context-aware systems on-device.

**URL** : <http://arxiv.org/pdf/2409.16973v1>

**Published** : 2024-09-25

## Semi-Supervised Cognitive State Classification from Speech with Multi-View Pseudo-Labeling

**Authors** : Yuanchao Li, Zixing Zhang, Jing Han, Peter Bell, Catherine Lai

**Abstract** : The lack of labeled data is a common challenge in speech classification tasks, particularly those requiring extensive subjective assessment, such as cognitive state classification. In this work, we propose a Semi-Supervised Learning (SSL) framework, introducing a novel multi-view pseudo-labeling method that leverages both acoustic and linguistic characteristics to select the most confident data for training the classification model. Acoustically, unlabeled data are compared to labeled data using the Frechet audio distance, calculated from embeddings generated by multiple audio encoders. Linguistically, large language models are prompted to revise automatic speech recognition transcriptions and predict labels based on our proposed task-specific knowledge. High-confidence data are identified when pseudo-labels from both sources align, while mismatches are treated as low-confidence data. A bimodal classifier is then trained to iteratively label the low-confidence data until a predefined criterion is met. We evaluate our SSL framework on emotion recognition and dementia detection tasks. Experimental results demonstrate that our method achieves competitive performance compared to fully supervised learning using only 30% of the labeled data and significantly outperforms two selected baselines.

**URL** : <http://arxiv.org/pdf/2409.16937v2>

**Published** : 2024-09-25

## Investigating OCR-Sensitive Neurons to Improve Entity Recognition in Historical Documents

**Authors** : Emanuela Boros, Maud Ehrmann

**Abstract** : This paper investigates the presence of OCR-sensitive neurons within the Transformer architecture and their influence on named entity recognition (NER) performance on historical documents. By analysing neuron activation patterns in response to clean and noisy text inputs, we identify and then neutralise OCR-sensitive neurons to improve model performance. Based on two open access large language models (Llama2 and Mistral), experiments demonstrate the existence of OCR-sensitive regions and show improvements in NER performance on historical newspapers and classical commentaries, highlighting the potential of targeted neuron modulation to improve models' performance on noisy text.

**URL** : <http://arxiv.org/pdf/2409.16934v2>

**Published** : 2024-09-25

## Zero-Shot Detection of LLM-Generated Text using Token Cohesiveness

**Authors** : Shixuan Ma, Quan Wang

**Abstract** : The increasing capability and widespread usage of large language models (LLMs) highlight the desirability of automatic detection of LLM-generated text. Zero-shot detectors, due to their training-free nature, have received considerable attention and notable success. In this paper, we identify a new feature, token cohesiveness, that is useful for zero-shot detection, and we demonstrate that LLM-generated text tends to exhibit higher token cohesiveness than human-written text. Based on this observation, we devise TOCSIN, a generic dual-channel detection paradigm that uses token cohesiveness as a plug-and-play module to improve existing zero-shot detectors. To calculate token cohesiveness, TOCSIN only requires a few rounds of random token deletion and semantic difference measurement, making it particularly suitable for a practical black-box setting where the source model used for generation is not accessible. Extensive experiments with four state-of-the-art base detectors on various datasets, source models, and evaluation settings demonstrate the effectiveness and generality of the proposed approach. Code available at: \url{[https://github.com/Shixuan-Ma/TOCSIN}](https://github.com/Shixuan-Ma/TOCSIN%7D).

**URL** : <http://arxiv.org/pdf/2409.16914v1>

**Published** : 2024-09-25

## Pruning Multilingual Large Language Models for Multilingual Inference

**Authors** : Hwichan Kim, Jun Suzuki, Tosho Hirasawa, Mamoru Komachi

**Abstract** : Multilingual large language models (MLLMs), trained on multilingual balanced data, demonstrate better zero-shot learning performance in non-English languages compared to large language models trained on English-dominant data. However, the disparity in performance between English and non-English languages remains a challenge yet to be fully addressed. A distinctive characteristic of MLLMs is their high-quality translation capabilities, indicating an acquired proficiency in aligning between languages. This study explores how to enhance the zero-shot performance of MLLMs in non-English languages by leveraging their alignment capability between English and non-English languages. To achieve this, we first analyze the behavior of MLLMs when performing translation and reveal that there are large magnitude features that play a critical role in the translation process. Inspired by these findings, we retain the weights associated with operations involving the large magnitude features and prune other weights to force MLLMs to rely on these features for tasks beyond translation. We empirically demonstrate that this pruning strategy can enhance the MLLMs' performance in non-English language.

**URL** : <http://arxiv.org/pdf/2409.16911v1>

**Published** : 2024-09-25

## Enhancing Temporal Sensitivity and Reasoning for Time-Sensitive Question Answering

**Authors** : Wanqi Yang, Yanda Li, Meng Fang, Ling Chen

**Abstract** : Time-Sensitive Question Answering (TSQA) demands the effective utilization of specific temporal contexts, encompassing multiple time-evolving facts, to address time-sensitive questions. This necessitates not only the parsing of temporal information within questions but also the identification and understanding of time-evolving facts to generate accurate answers. However, current large language models still have limited sensitivity to temporal information and their inadequate temporal reasoning capabilities.In this paper, we propose a novel framework that enhances temporal awareness and reasoning through Temporal Information-Aware Embedding and Granular Contrastive Reinforcement Learning. Experimental results on four TSQA datasets demonstrate that our framework significantly outperforms existing LLMs in TSQA tasks, marking a step forward in bridging the performance gap between machine and human temporal understanding and reasoning.

**URL** : <http://arxiv.org/pdf/2409.16909v1>

**Published** : 2024-09-25

## A Roadmap for Embodied and Social Grounding in LLMs

**Authors** : Sara Incao, Carlo Mazzola, Giulia Belgiovine, Alessandra Sciutti

**Abstract** : The fusion of Large Language Models (LLMs) and robotic systems has led to a transformative paradigm in the robotic field, offering unparalleled capabilities not only in the communication domain but also in skills like multimodal input handling, high-level reasoning, and plan generation. The grounding of LLMs knowledge into the empirical world has been considered a crucial pathway to exploit the efficiency of LLMs in robotics. Nevertheless, connecting LLMs' representations to the external world with multimodal approaches or with robots' bodies is not enough to let them understand the meaning of the language they are manipulating. Taking inspiration from humans, this work draws attention to three necessary elements for an agent to grasp and experience the world. The roadmap for LLMs grounding is envisaged in an active bodily system as the reference point for experiencing the environment, a temporally structured experience for a coherent, self-related interaction with the external world, and social skills to acquire a common-grounded shared experience.

**URL** : <http://arxiv.org/pdf/2409.16900v1>

**Published** : 2024-09-25

## Ethical and Scalable Automation: A Governance and Compliance Framework for Business Applications

**Authors** : Haocheng Lin

**Abstract** : The popularisation of applying AI in businesses poses significant challenges relating to ethical principles, governance, and legal compliance. Although businesses have embedded AI into their day-to-day processes, they lack a unified approach for mitigating its potential risks. This paper introduces a framework ensuring that AI must be ethical, controllable, viable, and desirable. Balancing these factors ensures the design of a framework that addresses its trade-offs, such as balancing performance against explainability. A successful framework provides practical advice for businesses to meet regulatory requirements in sectors such as finance and healthcare, where it is critical to comply with standards like GPDR and the EU AI Act. Different case studies validate this framework by integrating AI in both academic and practical environments. For instance, large language models are cost-effective alternatives for generating synthetic opinions that emulate attitudes to environmental issues. These case studies demonstrate how having a structured framework could enhance transparency and maintain performance levels as shown from the alignment between synthetic and expected distributions. This alignment is quantified using metrics like Chi-test scores, normalized mutual information, and Jaccard indexes. Future research should explore the framework's empirical validation in diverse industrial settings further, ensuring the model's scalability and adaptability.

**URL** : <http://arxiv.org/pdf/2409.16872v1>

**Published** : 2024-09-25

## Multi-objective Evolution of Heuristic Using Large Language Model

**Authors** : Shunyu Yao, Fei Liu, Xi Lin, Zhichao Lu, Zhenkun Wang, Qingfu Zhang

**Abstract** : Heuristics are commonly used to tackle diverse search and optimization problems. Design heuristics usually require tedious manual crafting with domain knowledge. Recent works have incorporated large language models (LLMs) into automatic heuristic search leveraging their powerful language and coding capacity. However, existing research focuses on the optimal performance on the target problem as the sole objective, neglecting other criteria such as efficiency and scalability, which are vital in practice. To tackle this challenge, we propose to model heuristic search as a multi-objective optimization problem and consider introducing other practical criteria beyond optimal performance. Due to the complexity of the search space, conventional multi-objective optimization methods struggle to effectively handle multi-objective heuristic search. We propose the first LLM-based multi-objective heuristic search framework, Multi-objective Evolution of Heuristic (MEoH), which integrates LLMs in a zero-shot manner to generate a non-dominated set of heuristics to meet multiple design criteria. We design a new dominance-dissimilarity mechanism for effective population management and selection, which incorporates both code dissimilarity in the search space and dominance in the objective space. MEoH is demonstrated in two well-known combinatorial optimization problems: the online Bin Packing Problem (BPP) and the Traveling Salesman Problem (TSP). Results indicate that a variety of elite heuristics are automatically generated in a single run, offering more trade-off options than existing methods. It successfully achieves competitive or superior performance while improving efficiency up to 10 times. Moreover, we also observe that the multi-objective search introduces novel insights into heuristic design and leads to the discovery of diverse heuristics.

**URL** : <http://arxiv.org/pdf/2409.16867v1>

**Published** : 2024-09-25

## The Role of Language Models in Modern Healthcare: A Comprehensive Review

**Authors** : Amna Khalid, Ayma Khalid, Umar Khalid

**Abstract** : The application of large language models (LLMs) in healthcare has gained significant attention due to their ability to process complex medical data and provide insights for clinical decision-making. These models have demonstrated substantial capabilities in understanding and generating natural language, which is crucial for medical documentation, diagnostics, and patient interaction. This review examines the trajectory of language models from their early stages to the current state-of-the-art LLMs, highlighting their strengths in healthcare applications and discussing challenges such as data privacy, bias, and ethical considerations. The potential of LLMs to enhance healthcare delivery is explored, alongside the necessary steps to ensure their ethical and effective integration into medical practice.

**URL** : <http://arxiv.org/pdf/2409.16860v1>

**Published** : 2024-09-25

## PeerArg: Argumentative Peer Review with LLMs

**Authors** : Purin Sukpanichnant, Anna Rapberger, Francesca Toni

**Abstract** : Peer review is an essential process to determine the quality of papers submitted to scientific conferences or journals. However, it is subjective and prone to biases. Several studies have been conducted to apply techniques from NLP to support peer review, but they are based on black-box techniques and their outputs are difficult to interpret and trust. In this paper, we propose a novel pipeline to support and understand the reviewing and decision-making processes of peer review: the PeerArg system combining LLMs with methods from knowledge representation. PeerArg takes in input a set of reviews for a paper and outputs the paper acceptance prediction. We evaluate the performance of the PeerArg pipeline on three different datasets, in comparison with a novel end-2-end LLM that uses few-shot learning to predict paper acceptance given reviews. The results indicate that the end-2-end LLM is capable of predicting paper acceptance from reviews, but a variant of the PeerArg pipeline outperforms this LLM.

**URL** : <http://arxiv.org/pdf/2409.16813v1>

**Published** : 2024-09-25

## A Few Hypocrites: Few-Shot Learning and Subtype Definitions for Detecting Hypocrisy Accusations in Online Climate Change Debates

**Authors** : Paulina Garcia Corral, Avishai Green, Hendrik Meyer, Anke Stoll, Xiaoyue Yan, Myrthe Reuver

**Abstract** : The climate crisis is a salient issue in online discussions, and hypocrisy accusations are a central rhetorical element in these debates. However, for large-scale text analysis, hypocrisy accusation detection is an understudied tool, most often defined as a smaller subtask of fallacious argument detection. In this paper, we define hypocrisy accusation detection as an independent task in NLP, and identify different relevant subtypes of hypocrisy accusations. Our Climate Hypocrisy Accusation Corpus (CHAC) consists of 420 Reddit climate debate comments, expert-annotated into two different types of hypocrisy accusations: personal versus political hypocrisy. We evaluate few-shot in-context learning with 6 shots and 3 instruction-tuned Large Language Models (LLMs) for detecting hypocrisy accusations in this dataset. Results indicate that the GPT-4o and Llama-3 models in particular show promise in detecting hypocrisy accusations (F1 reaching 0.68, while previous work shows F1 of 0.44). However, context matters for a complex semantic concept such as hypocrisy accusations, and we find models struggle especially at identifying political hypocrisy accusations compared to personal moral hypocrisy. Our study contributes new insights in hypocrisy detection and climate change discourse, and is a stepping stone for large-scale analysis of hypocrisy accusation in online climate debates.

**URL** : <http://arxiv.org/pdf/2409.16807v1>

**Published** : 2024-09-25

## Large Language Model Predicts Above Normal All India Summer Monsoon Rainfall in 2024

**Authors** : Ujjawal Sharma, Madhav Biyani, Akhil Dev Suresh, Debi Prasad Bhuyan, Saroj Kanta Mishra, Tanmoy Chakraborty

**Abstract** : Reliable prediction of the All India Summer Monsoon Rainfall (AISMR) is pivotal for informed policymaking for the country, impacting the lives of billions of people. However, accurate simulation of AISMR has been a persistent challenge due to the complex interplay of various muti-scale factors and the inherent variability of the monsoon system. This research focuses on adapting and fine-tuning the latest LLM model, PatchTST, to accurately predict AISMR with a lead time of three months. The fine-tuned PatchTST model, trained with historical AISMR data, the Ni~no3.4 index, and categorical Indian Ocean Dipole values, outperforms several popular neural network models and statistical models. This fine-tuned LLM model exhibits an exceptionally low RMSE percentage of 0.07% and a Spearman correlation of 0.976. This is particularly impressive, since it is nearly 80% more accurate than the best-performing NN models. The model predicts an above-normal monsoon for the year 2024, with an accumulated rainfall of 921.6 mm in the month of June-September for the entire country.

**URL** : <http://arxiv.org/pdf/2409.16799v1>

**Published** : 2024-09-25

## Mitigating the Bias of Large Language Model Evaluation

**Authors** : Hongli Zhou, Hui Huang, Yunfei Long, Bing Xu, Conghui Zhu, Hailong Cao, Muyun Yang, Tiejun Zhao

**Abstract** : Recently, there has been a trend of evaluating the Large Language Model (LLM) quality in the flavor of LLM-as-a-Judge, namely leveraging another LLM to evaluate the current output quality. However, existing judges are proven to be biased, namely they would favor answers which present better superficial quality (such as verbosity, fluency) while ignoring the instruction following ability. In this work, we propose systematic research about the bias of LLM-as-a-Judge. Specifically, for closed-source judge models, we apply calibration to mitigate the significance of superficial quality, both on probability level and prompt level. For open-source judge models, we propose to mitigate the bias by contrastive training, with curated negative samples that deviate from instruction but present better superficial quality. We apply our methods on the bias evaluation benchmark, and experiment results show our methods mitigate the bias by a large margin while maintaining a satisfactory evaluation accuracy.

**URL** : <http://arxiv.org/pdf/2409.16788v1>

**Published** : 2024-09-25

## Holistic Automated Red Teaming for Large Language Models through Top-Down Test Case Generation and Multi-turn Interaction

**Authors** : Jinchuan Zhang, Yan Zhou, Yaxin Liu, Ziming Li, Songlin Hu

**Abstract** : Automated red teaming is an effective method for identifying misaligned behaviors in large language models (LLMs). Existing approaches, however, often focus primarily on improving attack success rates while overlooking the need for comprehensive test case coverage. Additionally, most of these methods are limited to single-turn red teaming, failing to capture the multi-turn dynamics of real-world human-machine interactions. To overcome these limitations, we propose HARM (Holistic Automated Red teaMing), which scales up the diversity of test cases using a top-down approach based on an extensible, fine-grained risk taxonomy. Our method also leverages a novel fine-tuning strategy and reinforcement learning techniques to facilitate multi-turn adversarial probing in a human-like manner. Experimental results demonstrate that our framework enables a more systematic understanding of model vulnerabilities and offers more targeted guidance for the alignment process.

**URL** : <http://arxiv.org/pdf/2409.16783v1>

**Published** : 2024-09-25

## LLaMa-SciQ: An Educational Chatbot for Answering Science MCQ

**Authors** : Marc-Antoine Allard, Matin Ansaripour, Maria Yuffa, Paul Teiletche

**Abstract** : Large Language Models (LLMs) often struggle with tasks requiring mathematical reasoning, particularly multiple-choice questions (MCQs). To address this issue, we developed LLaMa-SciQ, an educational chatbot designed to assist college students in solving and understanding MCQs in STEM fields. We begin by fine-tuning and aligning the models to human preferences. After comparing the performance of Mistral-7B and LLaMa-8B, we selected the latter as the base model due to its higher evaluation accuracy. To further enhance accuracy, we implement Retrieval-Augmented Generation (RAG) and apply quantization to compress the model, reducing inference time and increasing accessibility for students. For mathematical reasoning, LLaMa-SciQ achieved 74.5% accuracy on the GSM8k dataset and 30% on the MATH dataset. However, RAG does not improve performance and even reduces it, likely due to retriever issues or the model's unfamiliarity with context. Despite this, the quantized model shows only a 5% loss in performance, demonstrating significant efficiency improvements.

**URL** : <http://arxiv.org/pdf/2409.16779v1>

**Published** : 2024-09-25

## E-SQL: Direct Schema Linking via Question Enrichment in Text-to-SQL

**Authors** : Hasan Alp Caferoğlu, Özgür Ulusoy

**Abstract** : Translating Natural Language Queries into Structured Query Language (Text-to-SQL or NLQ-to-SQL) is a critical task extensively studied by both the natural language processing and database communities, aimed at providing a natural language interface to databases (NLIDB) and lowering the barrier for non-experts. Despite recent advancements made through the use of Large Language Models (LLMs), significant challenges remain. These include handling complex database schemas, resolving ambiguity in user queries, and generating SQL queries with intricate structures that accurately reflect the user's intent. In this work, we introduce E-SQL, a novel pipeline specifically designed to address these challenges through direct schema linking and candidate predicate augmentation. E-SQL enhances the natural language query by incorporating relevant database items (i.e., tables, columns, and values) and conditions directly into the question, bridging the gap between the query and the database structure. The pipeline leverages candidate predicate augmentation to mitigate erroneous or incomplete predicates in generated SQLs. We further investigate the impact of schema filtering, a technique widely explored in previous work, and demonstrate its diminishing returns when applied alongside advanced large language models. Comprehensive evaluations on the BIRD benchmark illustrate that E-SQL achieves competitive performance, particularly excelling in complex queries with a 66.29% execution accuracy on the test set. All code required to reproduce the reported results is publicly available on our GitHub repository.

**URL** : <http://arxiv.org/pdf/2409.16751v1>

**Published** : 2024-09-25

## RoleBreak: Character Hallucination as a Jailbreak Attack in Role-Playing Systems

**Authors** : Yihong Tang, Bo Wang, Xu Wang, Dongming Zhao, Jing Liu, Jijun Zhang, Ruifang He, Yuexian Hou

**Abstract** : Role-playing systems powered by large language models (LLMs) have become increasingly influential in emotional communication applications. However, these systems are susceptible to character hallucinations, where the model deviates from predefined character roles and generates responses that are inconsistent with the intended persona. This paper presents the first systematic analysis of character hallucination from an attack perspective, introducing the RoleBreak framework. Our framework identifies two core mechanisms-query sparsity and role-query conflict-as key factors driving character hallucination. Leveraging these insights, we construct a novel dataset, RoleBreakEval, to evaluate existing hallucination mitigation techniques. Our experiments reveal that even enhanced models trained to minimize hallucination remain vulnerable to attacks. To address these vulnerabilities, we propose a novel defence strategy, the Narrator Mode, which generates supplemental context through narration to mitigate role-query conflicts and improve query generalization. Experimental results demonstrate that Narrator Mode significantly outperforms traditional refusal-based strategies by reducing hallucinations, enhancing fidelity to character roles and queries, and improving overall narrative coherence.

**URL** : <http://arxiv.org/pdf/2409.16727v1>

**Published** : 2024-09-25

## PMSS: Pretrained Matrices Skeleton Selection for LLM Fine-tuning

**Authors** : Qibin Wang, Xiaolin Hu, Weikai Xu, Wei Liu, Jian Luan, Bin Wang

**Abstract** : Low-rank adaptation (LoRA) and its variants have recently gained much interest due to their ability to avoid excessive inference costs. However, LoRA still encounters the following challenges: (1) Limitation of low-rank assumption; and (2) Its initialization method may be suboptimal. To this end, we propose PMSS(Pre-trained Matrices Skeleton Selection), which enables high-rank updates with low costs while leveraging semantic and linguistic information inherent in pre-trained weight. It achieves this by selecting skeletons from the pre-trained weight matrix and only learning a small matrix instead. Experiments demonstrate that PMSS outperforms LoRA and other fine-tuning methods across tasks with much less trainable parameters. We demonstrate its effectiveness, especially in handling complex tasks such as DROP benchmark(+3.4%/+5.9% on LLaMA2-7B/13B) and math reasoning(+12.89%/+5.61%/+3.11% on LLaMA2-7B, Mistral-7B and Gemma-7B of GSM8K). The code and model will be released soon.

**URL** : <http://arxiv.org/pdf/2409.16722v1>

**Published** : 2024-09-25

## Beyond Turing Test: Can GPT-4 Sway Experts' Decisions?

**Authors** : Takehiro Takayanagi, Hiroya Takamura, Kiyoshi Izumi, Chung-Chi Chen

**Abstract** : In the post-Turing era, evaluating large language models (LLMs) involves assessing generated text based on readers' reactions rather than merely its indistinguishability from human-produced content. This paper explores how LLM-generated text impacts readers' decisions, focusing on both amateur and expert audiences. Our findings indicate that GPT-4 can generate persuasive analyses affecting the decisions of both amateurs and professionals. Furthermore, we evaluate the generated text from the aspects of grammar, convincingness, logical coherence, and usefulness. The results highlight a high correlation between real-world evaluation through audience reactions and the current multi-dimensional evaluators commonly used for generative models. Overall, this paper shows the potential and risk of using generated text to sway human decisions and also points out a new direction for evaluating generated text, i.e., leveraging the reactions and decisions of readers. We release our dataset to assist future research.

**URL** : <http://arxiv.org/pdf/2409.16710v1>

**Published** : 2024-09-25

## A Survey of Low-bit Large Language Models: Basics, Systems, and Algorithms

**Authors** : Ruihao Gong, Yifu Ding, Zining Wang, Chengtao Lv, Xingyu Zheng, Jinyang Du, Haotong Qin, Jinyang Guo, Michele Magno, Xianglong Liu

**Abstract** : Large language models (LLMs) have achieved remarkable advancements in natural language processing, showcasing exceptional performance across various tasks. However, the expensive memory and computational requirements present significant challenges for their practical deployment. Low-bit quantization has emerged as a critical approach to mitigate these challenges by reducing the bit-width of model parameters, activations, and gradients, thus decreasing memory usage and computational demands. This paper presents a comprehensive survey of low-bit quantization methods tailored for LLMs, covering the fundamental principles, system implementations, and algorithmic strategies. An overview of basic concepts and new data formats specific to low-bit LLMs is first introduced, followed by a review of frameworks and systems that facilitate low-bit LLMs across various hardware platforms. Then, we categorize and analyze techniques and toolkits for efficient low-bit training and inference of LLMs. Finally, we conclude with a discussion of future trends and potential advancements of low-bit LLMs. Our systematic overview from basic, system, and algorithm perspectives can offer valuable insights and guidelines for future works to enhance the efficiency and applicability of LLMs through low-bit quantization.

**URL** : <http://arxiv.org/pdf/2409.16694v1>

**Published** : 2024-09-25

## MSI-Agent: Incorporating Multi-Scale Insight into Embodied Agents for Superior Planning and Decision-Making

**Authors** : Dayuan Fu, Biqing Qi, Yihuai Gao, Che Jiang, Guanting Dong, Bowen Zhou

**Abstract** : Long-term memory is significant for agents, in which insights play a crucial role. However, the emergence of irrelevant insight and the lack of general insight can greatly undermine the effectiveness of insight. To solve this problem, in this paper, we introduce Multi-Scale Insight Agent (MSI-Agent), an embodied agent designed to improve LLMs' planning and decision-making ability by summarizing and utilizing insight effectively across different scales. MSI achieves this through the experience selector, insight generator, and insight selector. Leveraging a three-part pipeline, MSI can generate task-specific and high-level insight, store it in a database, and then use relevant insight from it to aid in decision-making. Our experiments show that MSI outperforms another insight strategy when planning by GPT3.5. Moreover, We delve into the strategies for selecting seed experience and insight, aiming to provide LLM with more useful and relevant insight for better decision-making. Our observations also indicate that MSI exhibits better robustness when facing domain-shifting scenarios.

**URL** : <http://arxiv.org/pdf/2409.16686v1>

**Published** : 2024-09-25

## SynTQA: Synergistic Table-based Question Answering via Mixture of Text-to-SQL and E2E TQA

**Authors** : Siyue Zhang, Anh Tuan Luu, Chen Zhao

**Abstract** : Text-to-SQL parsing and end-to-end question answering (E2E TQA) are two main approaches for Table-based Question Answering task. Despite success on multiple benchmarks, they have yet to be compared and their synergy remains unexplored. In this paper, we identify different strengths and weaknesses through evaluating state-of-the-art models on benchmark datasets: Text-to-SQL demonstrates superiority in handling questions involving arithmetic operations and long tables; E2E TQA excels in addressing ambiguous questions, non-standard table schema, and complex table contents. To combine both strengths, we propose a Synergistic Table-based Question Answering approach that integrate different models via answer selection, which is agnostic to any model types. Further experiments validate that ensembling models by either feature-based or LLM-based answer selector significantly improves the performance over individual models.

**URL** : <http://arxiv.org/pdf/2409.16682v1>

**Published** : 2024-09-25

## GraphLoRA: Structure-Aware Contrastive Low-Rank Adaptation for Cross-Graph Transfer Learning

**Authors** : Zhe-Rui Yang, Jindong Han, Chang-Dong Wang, Hao Liu

**Abstract** : Graph Neural Networks (GNNs) have demonstrated remarkable proficiency in handling a range of graph analytical tasks across various domains, such as e-commerce and social networks. Despite their versatility, GNNs face significant challenges in transferability, limiting their utility in real-world applications. Existing research in GNN transfer learning overlooks discrepancies in distribution among various graph datasets, facing challenges when transferring across different distributions. How to effectively adopt a well-trained GNN to new graphs with varying feature and structural distributions remains an under-explored problem. Taking inspiration from the success of Low-Rank Adaptation (LoRA) in adapting large language models to various domains, we propose GraphLoRA, an effective and parameter-efficient method for transferring well-trained GNNs to diverse graph domains. Specifically, we first propose a Structure-aware Maximum Mean Discrepancy (SMMD) to align divergent node feature distributions across source and target graphs. Moreover, we introduce low-rank adaptation by injecting a small trainable GNN alongside the pre-trained one, effectively bridging structural distribution gaps while mitigating the catastrophic forgetting. Additionally, a structure-aware regularization objective is proposed to enhance the adaptability of the pre-trained GNN to target graph with scarce supervision labels. Extensive experiments on six real-world datasets demonstrate the effectiveness of GraphLoRA against eleven baselines by tuning only 20% of parameters, even across disparate graph domains. The code is available at <https://anonymous.4open.science/r/GraphLoRA>.

**URL** : <http://arxiv.org/pdf/2409.16670v1>

**Published** : 2024-09-25

## A Character-Centric Creative Story Generation via Imagination

**Authors** : Kyeongman Park, Minbeom Kim, Kyomin Jung

**Abstract** : Creative story generation with diverse and detailed story elements is a long-standing goal for large language models. While existing methodologies generate long and coherent stories, they fall significantly short of human capabilities in terms of diversity and character detail. To address this, we introduce a novel story generation framework called CCI (Character-centric Creative story generation via Imagination). CCI features two innovative modules for creative story generation: IG (Image-Guided Imagination) and MW (Multi-Writer model). In the IG module, we utilize DALL-E 3 to create visual representations of key story elements. The IG generates more novel and concrete characters, backgrounds, and main plots than text-only methods. The MW module uses these story elements created by IG to generate multiple description candidates for the protagonist and select the best one. This method incorporates vivid and rich character descriptions into the story. We compared the stories generated by CCI and baseline models through human evaluation and statistical analysis. The results showed significant improvements in the creativity. Furthermore, by enabling interactive multi-modal story generation with users, we have opened up possibilities for human-LLM integration in cultural development.

**URL** : <http://arxiv.org/pdf/2409.16667v1>

**Published** : 2024-09-25

## Speech Recognition Rescoring with Large Speech-Text Foundation Models

**Authors** : Prashanth Gurunath Shivakumar, Jari Kolehmainen, Aditya Gourav, Yi Gu, Ankur Gandhe, Ariya Rastrow, Ivan Bulyko

**Abstract** : Large language models (LLM) have demonstrated the ability to understand human language by leveraging large amount of text data. Automatic speech recognition (ASR) systems are often limited by available transcribed speech data and benefit from a second pass rescoring using LLM. Recently multi-modal large language models, particularly speech and text foundational models have demonstrated strong spoken language understanding. Speech-Text foundational models leverage large amounts of unlabelled and labelled data both in speech and text modalities to model human language. In this work, we propose novel techniques to use multi-modal LLM for ASR rescoring. We also explore discriminative training to further improve the foundational model rescoring performance. We demonstrate cross-modal knowledge transfer in speech-text LLM can benefit rescoring. Our experiments demonstrate up-to 20% relative improvements over Whisper large ASR and up-to 15% relative improvements over text-only LLM.

**URL** : <http://arxiv.org/pdf/2409.16654v1>

**Published** : 2024-09-25

## Enabling Auditory Large Language Models for Automatic Speech Quality Evaluation

**Authors** : Siyin Wang, Wenyi Yu, Yudong Yang, Changli Tang, Yixuan Li, Jimin Zhuang, Xianzhao Chen, Xiaohai Tian, Jun Zhang, Guangzhi Sun, Lu Lu, Chao Zhang

**Abstract** : Speech quality assessment typically requires evaluating audio from multiple aspects, such as mean opinion score (MOS) and speaker similarity (SIM) etc., which can be challenging to cover using one small model designed for a single task. In this paper, we propose leveraging recently introduced auditory large language models (LLMs) for automatic speech quality assessment. By employing task-specific prompts, auditory LLMs are finetuned to predict MOS, SIM and A/B testing results, which are commonly used for evaluating text-to-speech systems. Additionally, the finetuned auditory LLM is able to generate natural language descriptions assessing aspects like noisiness, distortion, discontinuity, and overall quality, providing more interpretable outputs. Extensive experiments have been performed on the NISQA, BVCC, SOMOS and VoxSim speech quality datasets, using open-source auditory LLMs such as SALMONN, Qwen-Audio, and Qwen2-Audio. For the natural language descriptions task, a commercial model Google Gemini 1.5 Pro is also evaluated. The results demonstrate that auditory LLMs achieve competitive performance compared to state-of-the-art task-specific small models in predicting MOS and SIM, while also delivering promising results in A/B testing and natural language descriptions. Our data processing scripts and finetuned model checkpoints will be released upon acceptance.

**URL** : <http://arxiv.org/pdf/2409.16644v1>

**Published** : 2024-09-25

## Judgment of Thoughts: Courtroom of the Binary Logical Reasoning in Large Language Models

**Authors** : Sungjune Park, Daeseon Choi

**Abstract** : This paper proposes a novel prompt engineering technique called Judgment of Thought (JoT) that is specifically tailored for binary logical reasoning tasks. JoT employs three roles$\unicode{x2014}$lawyer, prosecutor, and judge$\unicode{x2014}$to facilitate more reliable and accurate reasoning by the model. In this framework, the judge utilizes a high$\unicode{x2010}$level model, while the lawyer and prosecutor utilize low$\unicode{x2010}$level models. This structure helps the judge better understand the responses from both the lawyer and prosecutor, enabling a more accurate judgment. Experimental results on large language model (LLM) benchmark datasets, such as BigBenchHard and Winogrande, demonstrate that JoT outperforms existing methods, including Chain of Thought (CoT) and Self$\unicode{x2010}$Consistency (SC), in binary logical reasoning tasks. Additionally, in real$\unicode{x2010}$world tasks, such as Fake News Detection and SMS Spam Detection, JoT shows comparable or improved performance compared to existing techniques. JoT significantly enhances the accuracy and reliability of models in binary reasoning tasks and show potential for practical applicability across various domains. Future research should aim to further broaden the applicability of JoT and optimize its implementation for real$\unicode{x2010}$world problem$\unicode{x2010}$solving.

**URL** : <http://arxiv.org/pdf/2409.16635v1>

**Published** : 2024-09-25

## Ascend HiFloat8 Format for Deep Learning

**Authors** : Yuanyong Luo, Zhongxing Zhang, Richard Wu, Hu Liu, Ying Jin, Kai Zheng, Minmin Wang, Zhanying He, Guipeng Hu, Luyao Chen, Tianchi Hu, Junsong Wang, Minqi Chen, Mikhaylov Dmitry, Korviakov Vladimir, Bobrin Maxim, Yuhao Hu, Guanfu Chen, Zeyi Huang

**Abstract** : This preliminary white paper proposes a novel 8-bit floating-point data format HiFloat8 (abbreviated as HiF8) for deep learning. HiF8 features tapered precision. For normal value encoding, it provides 7 exponent values with 3-bit mantissa, 8 exponent values with 2-bit mantissa, and 16 exponent values with 1-bit mantissa. For denormal value encoding, it extends the dynamic range by 7 extra powers of 2, from 31 to 38 binades (notice that FP16 covers 40 binades). Meanwhile, HiF8 encodes all the special values except that positive zero and negative zero are represented by only one bit-pattern. Thanks to the better balance between precision and dynamic range, HiF8 can be simultaneously used in both forward and backward passes of AI training. In this paper, we will describe the definition and rounding methods of HiF8, as well as the tentative training and inference solutions. To demonstrate the efficacy of HiF8, massive simulation results on various neural networks, including traditional neural networks and large language models (LLMs), will also be presented.

**URL** : <http://arxiv.org/pdf/2409.16626v2>

**Published** : 2024-09-25

## Entailment-Driven Privacy Policy Classification with LLMs

**Authors** : Bhanuka Silva, Dishanika Denipitiyage, Suranga Seneviratne, Anirban Mahanti, Aruna Seneviratne

**Abstract** : While many online services provide privacy policies for end users to read and understand what personal data are being collected, these documents are often lengthy and complicated. As a result, the vast majority of users do not read them at all, leading to data collection under uninformed consent. Several attempts have been made to make privacy policies more user friendly by summarising them, providing automatic annotations or labels for key sections, or by offering chat interfaces to ask specific questions. With recent advances in Large Language Models (LLMs), there is an opportunity to develop more effective tools to parse privacy policies and help users make informed decisions. In this paper, we propose an entailment-driven LLM based framework to classify paragraphs of privacy policies into meaningful labels that are easily understood by users. The results demonstrate that our framework outperforms traditional LLM methods, improving the F1 score in average by 11.2%. Additionally, our framework provides inherently explainable and meaningful predictions.

**URL** : <http://arxiv.org/pdf/2409.16621v1>

**Published** : 2024-09-25

## Claim-Guided Textual Backdoor Attack for Practical Applications

**Authors** : Minkyoo Song, Hanna Kim, Jaehan Kim, Youngjin Jin, Seungwon Shin

**Abstract** : Recent advances in natural language processing and the increased use of large language models have exposed new security vulnerabilities, such as backdoor attacks. Previous backdoor attacks require input manipulation after model distribution to activate the backdoor, posing limitations in real-world applicability. Addressing this gap, we introduce a novel Claim-Guided Backdoor Attack (CGBA), which eliminates the need for such manipulations by utilizing inherent textual claims as triggers. CGBA leverages claim extraction, clustering, and targeted training to trick models to misbehave on targeted claims without affecting their performance on clean data. CGBA demonstrates its effectiveness and stealthiness across various datasets and models, significantly enhancing the feasibility of practical backdoor attacks. Our code and data will be available at <https://github.com/PaperCGBA/CGBA>.

**URL** : <http://arxiv.org/pdf/2409.16618v1>

**Published** : 2024-09-25

## Evaluating and Enhancing Large Language Models for Novelty Assessment in Scholarly Publications

**Authors** : Ethan Lin, Zhiyuan Peng, Yi Fang

**Abstract** : Recent studies have evaluated the creativity/novelty of large language models (LLMs) primarily from a semantic perspective, using benchmarks from cognitive science. However, accessing the novelty in scholarly publications is a largely unexplored area in evaluating LLMs. In this paper, we introduce a scholarly novelty benchmark (SchNovel) to evaluate LLMs' ability to assess novelty in scholarly papers. SchNovel consists of 15000 pairs of papers across six fields sampled from the arXiv dataset with publication dates spanning 2 to 10 years apart. In each pair, the more recently published paper is assumed to be more novel. Additionally, we propose RAG-Novelty, which simulates the review process taken by human reviewers by leveraging the retrieval of similar papers to assess novelty. Extensive experiments provide insights into the capabilities of different LLMs to assess novelty and demonstrate that RAG-Novelty outperforms recent baseline models.

**URL** : <http://arxiv.org/pdf/2409.16605v1>

**Published** : 2024-09-25

## Disentangling Questions from Query Generation for Task-Adaptive Retrieval

**Authors** : Yoonsang Lee, Minsoo Kim, Seung-won Hwang

**Abstract** : This paper studies the problem of information retrieval, to adapt to unseen tasks. Existing work generates synthetic queries from domain-specific documents to jointly train the retriever. However, the conventional query generator assumes the query as a question, thus failing to accommodate general search intents. A more lenient approach incorporates task-adaptive elements, such as few-shot learning with an 137B LLM. In this paper, we challenge a trend equating query and question, and instead conceptualize query generation task as a "compilation" of high-level intent into task-adaptive query. Specifically, we propose EGG, a query generator that better adapts to wide search intents expressed in the BeIR benchmark. Our method outperforms baselines and existing models on four tasks with underexplored intents, while utilizing a query generator 47 times smaller than the previous state-of-the-art. Our findings reveal that instructing the LM with explicit search intent is a key aspect of modeling an effective query generator.

**URL** : <http://arxiv.org/pdf/2409.16570v1>

**Published** : 2024-09-25

## Enhancing disease detection in radiology reports through fine-tuning lightweight LLM on weak labels

**Authors** : Yishu Wei, Xindi Wang, Hanley Ong, Yiliang Zhou, Adam Flanders, George Shih, Yifan Peng

**Abstract** : Despite significant progress in applying large language models (LLMs) to the medical domain, several limitations still prevent them from practical applications. Among these are the constraints on model size and the lack of cohort-specific labeled datasets. In this work, we investigated the potential of improving a lightweight LLM, such as Llama 3.1-8B, through fine-tuning with datasets using synthetic labels. Two tasks are jointly trained by combining their respective instruction datasets. When the quality of the task-specific synthetic labels is relatively high (e.g., generated by GPT4- o), Llama 3.1-8B achieves satisfactory performance on the open-ended disease detection task, with a micro F1 score of 0.91. Conversely, when the quality of the task-relevant synthetic labels is relatively low (e.g., from the MIMIC-CXR dataset), fine-tuned Llama 3.1-8B is able to surpass its noisy teacher labels (micro F1 score of 0.67 v.s. 0.63) when calibrated against curated labels, indicating the strong inherent underlying capability of the model. These findings demonstrate the potential of fine-tuning LLMs with synthetic labels, offering a promising direction for future research on LLM specialization in the medical domain.

**URL** : <http://arxiv.org/pdf/2409.16563v1>

**Published** : 2024-09-25

## Dynamic-Width Speculative Beam Decoding for Efficient LLM Inference

**Authors** : Zongyue Qin, Zifan He, Neha Prakriya, Jason Cong, Yizhou Sun

**Abstract** : Large language models (LLMs) have shown outstanding performance across numerous real-world tasks. However, the autoregressive nature of these models makes the inference process slow and costly. Speculative decoding has emerged as a promising solution, leveraging a smaller auxiliary model to draft future tokens, which are then validated simultaneously by the larger model, achieving a speed-up of 1-2x. Although speculative decoding matches the same distribution as multinomial sampling, multinomial sampling itself is prone to suboptimal outputs, whereas beam sampling is widely recognized for producing higher-quality results by maintaining multiple candidate sequences at each step. This paper explores the novel integration of speculative decoding with beam sampling. However, there are four key challenges: (1) how to generate multiple sequences from the larger model's distribution given drafts sequences from the small model; (2) how to dynamically optimize the number of beams to balance efficiency and accuracy; (3) how to efficiently verify the multiple drafts in parallel; and (4) how to address the extra memory costs inherent in beam sampling. To address these challenges, we propose dynamic-width speculative beam decoding (DSBD). Specifically, we first introduce a novel draft and verification scheme that generates multiple sequences following the large model's distribution based on beam sampling trajectories from the small model. Then, we introduce an adaptive mechanism to dynamically tune the number of beams based on the context, optimizing efficiency and effectiveness. Besides, we extend tree-based parallel verification to handle multiple trees simultaneously, accelerating the verification process. Finally, we illustrate a simple modification to our algorithm to mitigate the memory overhead of beam sampling...

**URL** : <http://arxiv.org/pdf/2409.16560v1>

**Published** : 2024-09-25

## Demystifying Issues, Causes and Solutions in LLM Open-Source Projects

**Authors** : Yangxiao Cai, Peng Liang, Yifei Wang, Zengyang Li, Mojtaba Shahin

**Abstract** : With the advancements of Large Language Models (LLMs), an increasing number of open-source software projects are using LLMs as their core functional component. Although research and practice on LLMs are capturing considerable interest, no dedicated studies explored the challenges faced by practitioners of LLM open-source projects, the causes of these challenges, and potential solutions. To fill this research gap, we conducted an empirical study to understand the issues that practitioners encounter when developing and using LLM open-source software, the possible causes of these issues, and potential solutions.We collected all closed issues from 15 LLM open-source projects and labelled issues that met our requirements. We then randomly selected 994 issues from the labelled issues as the sample for data extraction and analysis to understand the prevalent issues, their underlying causes, and potential solutions. Our study results show that (1) Model Issue is the most common issue faced by practitioners, (2) Model Problem, Configuration and Connection Problem, and Feature and Method Problem are identified as the most frequent causes of the issues, and (3) Optimize Model is the predominant solution to the issues. Based on the study results, we provide implications for practitioners and researchers of LLM open-source projects.

**URL** : <http://arxiv.org/pdf/2409.16559v1>

**Published** : 2024-09-25

## Understanding the Cognitive Complexity in Language Elicited by Product Images

**Authors** : Yan-Ying Chen, Shabnam Hakimi, Monica Van, Francine Chen, Matthew Hong, Matt Klenk, Charlene Wu

**Abstract** : Product images (e.g., a phone) can be used to elicit a diverse set of consumer-reported features expressed through language, including surface-level perceptual attributes (e.g., "white") and more complex ones, like perceived utility (e.g., "battery"). The cognitive complexity of elicited language reveals the nature of cognitive processes and the context required to understand them; cognitive complexity also predicts consumers' subsequent choices. This work offers an approach for measuring and validating the cognitive complexity of human language elicited by product images, providing a tool for understanding the cognitive processes of human as well as virtual respondents simulated by Large Language Models (LLMs). We also introduce a large dataset that includes diverse descriptive labels for product images, including human-rated complexity. We demonstrate that human-rated cognitive complexity can be approximated using a set of natural language models that, combined, roughly capture the complexity construct. Moreover, this approach is minimally supervised and scalable, even in use cases with limited human assessment of complexity.

**URL** : <http://arxiv.org/pdf/2409.16521v1>

**Published** : 2024-09-25

## SynChart: Synthesizing Charts from Language Models

**Authors** : Mengchen Liu, Qixiu Li, Dongdong Chen, Dong Chen, Jianmin Bao, Yunsheng Li

**Abstract** : With the release of GPT-4V(O), its use in generating pseudo labels for multi-modality tasks has gained significant popularity. However, it is still a secret how to build such advanced models from its base large language models (LLMs). This work explores the potential of using LLMs alone for data generation and develop competitive multi-modality models focusing on chart understanding. We construct a large-scale chart dataset, SynChart, which contains approximately 4 million diverse chart images with over 75 million dense annotations, including data tables, code, descriptions, and question-answer sets. We trained a 4.2B chart-expert model using this dataset and achieve near-GPT-4O performance on the ChartQA task, surpassing GPT-4V.

**URL** : <http://arxiv.org/pdf/2409.16517v1>

**Published** : 2024-09-25

## Unsupervised Text Representation Learning via Instruction-Tuning for Zero-Shot Dense Retrieval

**Authors** : Qiuhai Zeng, Zimeng Qiu, Dae Yon Hwang, Xin He, William M. Campbell

**Abstract** : Dense retrieval systems are commonly used for information retrieval (IR). They rely on learning text representations through an encoder and usually require supervised modeling via labelled data which can be costly to obtain or simply unavailable. In this study, we introduce a novel unsupervised text representation learning technique via instruction-tuning the pre-trained encoder-decoder large language models (LLM) under the dual-encoder retrieval framework. We demonstrate the corpus representation can be augmented by the representations of relevant synthetic queries generated by the instruct-tuned LLM founded on the Rao-Blackwell theorem. Furthermore, we effectively align the query and corpus text representation with self-instructed-tuning. Specifically, we first prompt an open-box pre-trained LLM to follow defined instructions (i.e. question generation and keyword summarization) to generate synthetic queries. Next, we fine-tune the pre-trained LLM with defined instructions and the generated queries that passed quality check. Finally, we generate synthetic queries with the instruction-tuned LLM for each corpora and represent each corpora by weighted averaging the synthetic queries and original corpora embeddings. We evaluate our proposed method under low-resource settings on three English and one German retrieval datasets measuring NDCG@10, MRR@100, Recall@100. We significantly improve the average zero-shot retrieval performance on all metrics, increasing open-box FLAN-T5 model variations by [3.34%, 3.50%] in absolute and exceeding three competitive dense retrievers (i.e. mDPR, T-Systems, mBART-Large), with model of size at least 38% smaller, by 1.96%, 4.62%, 9.52% absolute on NDCG@10.

**URL** : <http://arxiv.org/pdf/2409.16497v1>

**Published** : 2024-09-24

## Exploring Knowledge Tracing in Tutor-Student Dialogues

**Authors** : Alexander Scarlatos, Andrew Lan

**Abstract** : Recent advances in large language models (LLMs) have led to the development of artificial intelligence (AI)-powered tutoring chatbots, showing promise in providing broad access to high-quality personalized education. Existing works have primarily studied how to make LLMs follow tutoring principles but not how to model student behavior in dialogues. However, analyzing student dialogue turns can serve as a formative assessment, since open-ended student discourse may indicate their knowledge levels and reveal specific misconceptions. In this work, we present a first attempt at performing knowledge tracing (KT) in tutor-student dialogues. We propose LLM prompting methods to identify the knowledge components/skills involved in each dialogue turn and diagnose whether the student responds correctly to the tutor, and verify the LLM's effectiveness via an expert human evaluation. We then apply a range of KT methods on the resulting labeled data to track student knowledge levels over an entire dialogue. We conduct experiments on two tutoring dialogue datasets, and show that a novel yet simple LLM-based method, LLMKT, significantly outperforms existing KT methods in predicting student response correctness in dialogues. We perform extensive qualitative analyses to highlight the challenges in dialogue KT and outline multiple avenues for future work.

**URL** : <http://arxiv.org/pdf/2409.16490v1>

**Published** : 2024-09-24

## Strategies for Improving NL-to-FOL Translation with LLMs: Data Generation, Incremental Fine-Tuning, and Verification

**Authors** : Ramya Keerthy Thatikonda, Jiuzhou Han, Wray Buntine, Ehsan Shareghi

**Abstract** : Logical reasoning is a fundamental task in natural language processing that presents significant challenges to Large Language Models (LLMs). The inherent characteristics of logical reasoning makes it well-suited for symbolic representations such as first-order logic (FOL). Research in symbolic logical reasoning explored FOL generation using state-of-the-art LLMs (i.e., GPT-4) to produce FOL translations of natural language (NL) statements, but errors in translation are usually not the focus. We address this by categorizing the translation errors in FOL statements generated by LLMs. To make progress towards improving the quality of FOL translations for smaller language models such as LLaMA-2 13B and Mistral 7B, we create ProofFOL, a high-quality FOL-annotated subset of ProofWriter dataset using GPT-4o. The models fine-tuned on this silver standard data achieve a significant gain in performance when compared to larger language models such as LLaMA-2 70B. In addition to improving the model using large data, we also tackle the issue of data scarcity and introduce an incremental framework encompassing of data augmentation and verification steps. In the augmentation process, a single pair of (premises, conclusion) is split into multiple new instances based on the predicates and FOLs. This data is used for fine-tuning, and the inference on this model generates FOLs with fewer errors over the model trained on the original data. Our investigation on the translation errors leads to generation of a perturbation dataset, which is used to train a verifier that corrects potential syntactic and semantic FOL translation errors. We demonstrate an efficient method for making the most of a limited existing human-annotated dataset. Our results show state-of-the-art performance for ProofWriter and ProntoQA datasets using ProofFOL on LLaMA-2 and Mistral models.

**URL** : <http://arxiv.org/pdf/2409.16461v1>

**Published** : 2024-09-24

## FMDLlama: Financial Misinformation Detection based on Large Language Models

**Authors** : Zhiwei Liu, Xin Zhang, Kailai Yang, Qianqian Xie, Jimin Huang, Sophia Ananiadou

**Abstract** : The emergence of social media has made the spread of misinformation easier. In the financial domain, the accuracy of information is crucial for various aspects of financial market, which has made financial misinformation detection (FMD) an urgent problem that needs to be addressed. Large language models (LLMs) have demonstrated outstanding performance in various fields. However, current studies mostly rely on traditional methods and have not explored the application of LLMs in the field of FMD. The main reason is the lack of FMD instruction tuning datasets and evaluation benchmarks. In this paper, we propose FMDLlama, the first open-sourced instruction-following LLMs for FMD task based on fine-tuning Llama3.1 with instruction data, the first multi-task FMD instruction dataset (FMDID) to support LLM instruction tuning, and a comprehensive FMD evaluation benchmark (FMD-B) with classification and explanation generation tasks to test the FMD ability of LLMs. We compare our models with a variety of LLMs on FMD-B, where our model outperforms all other open-sourced LLMs as well as ChatGPT.

**URL** : <http://arxiv.org/pdf/2409.16452v1>

**Published** : 2024-09-24

## A Comprehensive Survey of Bias in LLMs: Current Landscape and Future Directions

**Authors** : Rajesh Ranjan, Shailja Gupta, Surya Narayan Singh

**Abstract** : Large Language Models(LLMs) have revolutionized various applications in natural language processing (NLP) by providing unprecedented text generation, translation, and comprehension capabilities. However, their widespread deployment has brought to light significant concerns regarding biases embedded within these models. This paper presents a comprehensive survey of biases in LLMs, aiming to provide an extensive review of the types, sources, impacts, and mitigation strategies related to these biases. We systematically categorize biases into several dimensions. Our survey synthesizes current research findings and discusses the implications of biases in real-world applications. Additionally, we critically assess existing bias mitigation techniques and propose future research directions to enhance fairness and equity in LLMs. This survey serves as a foundational resource for researchers, practitioners, and policymakers concerned with addressing and understanding biases in LLMs.

**URL** : <http://arxiv.org/pdf/2409.16430v1>

**Published** : 2024-09-24

## HAICOSYSTEM: An Ecosystem for Sandboxing Safety Risks in Human-AI Interactions

**Authors** : Xuhui Zhou, Hyunwoo Kim, Faeze Brahman, Liwei Jiang, Hao Zhu, Ximing Lu, Frank Xu, Bill Yuchen Lin, Yejin Choi, Niloofar Mireshghallah, Ronan Le Bras, Maarten Sap

**Abstract** : AI agents are increasingly autonomous in their interactions with human users and tools, leading to increased interactional safety risks. We present HAICOSYSTEM, a framework examining AI agent safety within diverse and complex social interactions. HAICOSYSTEM features a modular sandbox environment that simulates multi-turn interactions between human users and AI agents, where the AI agents are equipped with a variety of tools (e.g., patient management platforms) to navigate diverse scenarios (e.g., a user attempting to access other patients' profiles). To examine the safety of AI agents in these interactions, we develop a comprehensive multi-dimensional evaluation framework that uses metrics covering operational, content-related, societal, and legal risks. Through running 1840 simulations based on 92 scenarios across seven domains (e.g., healthcare, finance, education), we demonstrate that HAICOSYSTEM can emulate realistic user-AI interactions and complex tool use by AI agents. Our experiments show that state-of-the-art LLMs, both proprietary and open-sourced, exhibit safety risks in over 50% cases, with models generally showing higher risks when interacting with simulated malicious users. Our findings highlight the ongoing challenge of building agents that can safely navigate complex interactions, particularly when faced with malicious users. To foster the AI agent safety ecosystem, we release a code platform that allows practitioners to create custom scenarios, simulate interactions, and evaluate the safety and performance of their agents.

**URL** : <http://arxiv.org/pdf/2409.16427v2>

**Published** : 2024-09-24

## Task-oriented Prompt Enhancement via Script Generation

**Authors** : Chung-Yu Wang, Alireza DaghighFarsoodeh, Hung Viet Pham

**Abstract** : Large Language Models (LLMs) have demonstrated remarkable abilities across various tasks, leveraging advanced reasoning. Yet, they struggle with task-oriented prompts due to a lack of specific prior knowledge of the task answers. The current state-of-the-art approach, PAL, utilizes code generation to address this issue. However, PAL depends on manually crafted prompt templates and examples while still producing inaccurate results. In this work, we present TITAN-a novel strategy designed to enhance LLMs' performance on task-oriented prompts. TITAN achieves this by generating scripts using a universal approach and zero-shot learning. Unlike existing methods, TITAN eliminates the need for detailed task-specific instructions and extensive manual efforts. TITAN enhances LLMs' performance on various tasks by utilizing their analytical and code-generation capabilities in a streamlined process. TITAN employs two key techniques: (1) step-back prompting to extract the task's input specifications and (2) chain-of-thought prompting to identify required procedural steps. This information is used to improve the LLMs' code-generation process. TITAN further refines the generated script through post-processing and the script is executed to retrieve the final answer. Our comprehensive evaluation demonstrates TITAN's effectiveness in a diverse set of tasks. On average, TITAN outperforms the state-of-the-art zero-shot approach by 7.6% and 3.9% when paired with GPT-3.5 and GPT-4. Overall, without human annotation, TITAN achieves state-of-the-art performance in 8 out of 11 cases while only marginally losing to few-shot approaches (which needed human intervention) on three occasions by small margins. This work represents a significant advancement in addressing task-oriented prompts, offering a novel solution for effectively utilizing LLMs in everyday life tasks.

**URL** : <http://arxiv.org/pdf/2409.16418v1>

**Published** : 2024-09-24

## Selection of Prompt Engineering Techniques for Code Generation through Predicting Code Complexity

**Authors** : Chung-Yu Wang, Alireza DaghighFarsoodeh, Hung Viet Pham

**Abstract** : Large Language Models (LLMs) have demonstrated impressive performance in software engineering tasks. However, improving their accuracy in generating correct and reliable code remains challenging. Numerous prompt engineering techniques (PETs) have been developed to address this, but no single approach is universally optimal. Selecting the right PET for each query is difficult for two primary reasons: (1) interactive prompting techniques may not consistently deliver the expected benefits, especially for simpler queries, and (2) current automated prompt engineering methods lack adaptability and fail to fully utilize multi-stage responses. To overcome these challenges, we propose PET-Select, a PET-agnostic selection model that uses code complexity as a proxy to classify queries and select the most appropriate PET. By incorporating contrastive learning, PET-Select effectively distinguishes between simple and complex problems, allowing it to choose PETs that are best suited for each query's complexity level. Our evaluations on the MBPP and HumanEval benchmarks using GPT-3.5 Turbo and GPT-4o show up to a 1.9% improvement in pass@1 accuracy, along with a 74.8% reduction in token usage. Additionally, we provide both quantitative and qualitative results to demonstrate how PET-Select effectively selects the most appropriate techniques for each code generation query, further showcasing its efficiency in optimizing PET selection.

**URL** : <http://arxiv.org/pdf/2409.16416v1>

**Published** : 2024-09-24

## Design and Evaluation of a CDSS for Drug Allergy Management Using LLMs and Pharmaceutical Data Integration

**Authors** : Gabriele De Vito, Filomena Ferrucci, Athanasios Angelakis

**Abstract** : Medication errors significantly threaten patient safety, leading to adverse drug events and substantial economic burdens on healthcare systems. Clinical Decision Support Systems (CDSSs) aimed at mitigating these errors often face limitations, including reliance on static databases and rule-based algorithms, which can result in high false alert rates and alert fatigue among clinicians. This paper introduces HELIOT, an innovative CDSS for drug allergy management, integrating Large Language Models (LLMs) with a comprehensive pharmaceutical data repository. HELIOT leverages advanced natural language processing capabilities to interpret complex medical texts and synthesize unstructured data, overcoming the limitations of traditional CDSSs. An empirical evaluation using a synthetic patient dataset and expert-verified ground truth demonstrates HELIOT's high accuracy, precision, recall, and F1 score, uniformly reaching 100% across multiple experimental runs. The results underscore HELIOT's potential to enhance decision support in clinical settings, offering a scalable, efficient, and reliable solution for managing drug allergies.

**URL** : <http://arxiv.org/pdf/2409.16395v1>

**Published** : 2024-09-24

## RISCORE: Enhancing In-Context Riddle Solving in Language Models through Context-Reconstructed Example Augmentation

**Authors** : Ioannis Panagiotopoulos, Giorgos Filandrianos, Maria Lymperaiou, Giorgos Stamou

**Abstract** : Riddle-solving requires advanced reasoning skills, pushing LLMs to engage in abstract thinking and creative problem-solving, often revealing limitations in their cognitive abilities. In this paper, we examine the riddle-solving capabilities of LLMs using a multiple-choice format, exploring how different prompting techniques impact performance on riddles that demand diverse reasoning skills. To enhance results, we introduce RISCORE (RIddle Solving with COntext REcontruciton) a novel fully automated prompting method that generates and utilizes contextually reconstructed sentence-based puzzles in conjunction with the original examples to create few-shot exemplars. Our experiments demonstrate that RISCORE significantly improves the performance of language models in both vertical and lateral thinking tasks, surpassing traditional exemplar selection strategies across a variety of few-shot settings.

**URL** : <http://arxiv.org/pdf/2409.16383v2>

**Published** : 2024-09-24

## Beyond Text-to-Text: An Overview of Multimodal and Generative Artificial Intelligence for Education Using Topic Modeling

**Authors** : Ville Heilala, Roberto Araya, Raija Hämäläinen

**Abstract** : Generative artificial intelligence (GenAI) can reshape education and learning. While large language models (LLMs) like ChatGPT dominate current educational research, multimodal capabilities, such as text-to-speech and text-to-image, are less explored. This study uses topic modeling to map the research landscape of multimodal and generative AI in education. An extensive literature search using Dimensions.ai yielded 4175 articles. Employing a topic modeling approach, latent topics were extracted, resulting in 38 interpretable topics organized into 14 thematic areas. Findings indicate a predominant focus on text-to-text models in educational contexts, with other modalities underexplored, overlooking the broader potential of multimodal approaches. The results suggest a research gap, stressing the importance of more balanced attention across different AI modalities and educational levels. In summary, this research provides an overview of current trends in generative AI for education, underlining opportunities for future exploration of multimodal technologies to fully realize the transformative potential of artificial intelligence in education.

**URL** : <http://arxiv.org/pdf/2409.16376v1>

**Published** : 2024-09-24

## Do the Right Thing, Just Debias! Multi-Category Bias Mitigation Using LLMs

**Authors** : Amartya Roy, Danush Khanna, Devanshu Mahapatra, Vasanthakumar, Avirup Das, Kripabandhu Ghosh

**Abstract** : This paper tackles the challenge of building robust and generalizable bias mitigation models for language. Recognizing the limitations of existing datasets, we introduce ANUBIS, a novel dataset with 1507 carefully curated sentence pairs encompassing nine social bias categories. We evaluate state-of-the-art models like T5, utilizing Supervised Fine-Tuning (SFT), Reinforcement Learning (PPO, DPO), and In-Context Learning (ICL) for effective bias mitigation. Our analysis focuses on multi-class social bias reduction, cross-dataset generalizability, and environmental impact of the trained models. ANUBIS and our findings offer valuable resources for building more equitable AI systems and contribute to the development of responsible and unbiased technologies with broad societal impact.

**URL** : <http://arxiv.org/pdf/2409.16371v1>

**Published** : 2024-09-24

## Quality Matters: Evaluating Synthetic Data for Tool-Using LLMs

**Authors** : Shadi Iskander, Nachshon Cohen, Zohar Karnin, Ori Shapira, Sofia Tolmach

**Abstract** : Training large language models (LLMs) for external tool usage is a rapidly expanding field, with recent research focusing on generating synthetic data to address the shortage of available data. However, the absence of systematic data quality checks poses complications for properly training and testing models. To that end, we propose two approaches for assessing the reliability of data for training LLMs to use external tools. The first approach uses intuitive, human-defined correctness criteria. The second approach uses a model-driven assessment with in-context evaluation. We conduct a thorough evaluation of data quality on two popular benchmarks, followed by an extrinsic evaluation that showcases the impact of data quality on model performance. Our results demonstrate that models trained on high-quality data outperform those trained on unvalidated data, even when trained with a smaller quantity of data. These findings empirically support the significance of assessing and ensuring the reliability of training data for tool-using LLMs.

**URL** : <http://arxiv.org/pdf/2409.16341v2>

**Published** : 2024-09-24

## LLM Echo Chamber: personalized and automated disinformation

**Authors** : Tony Ma

**Abstract** : Recent advancements have showcased the capabilities of Large Language Models like GPT4 and Llama2 in tasks such as summarization, translation, and content review. However, their widespread use raises concerns, particularly around the potential for LLMs to spread persuasive, humanlike misinformation at scale, which could significantly influence public opinion. This study examines these risks, focusing on LLMs ability to propagate misinformation as factual. To investigate this, we built the LLM Echo Chamber, a controlled digital environment simulating social media chatrooms, where misinformation often spreads. Echo chambers, where individuals only interact with like minded people, further entrench beliefs. By studying malicious bots spreading misinformation in this environment, we can better understand this phenomenon. We reviewed current LLMs, explored misinformation risks, and applied sota finetuning techniques. Using Microsoft phi2 model, finetuned with our custom dataset, we generated harmful content to create the Echo Chamber. This setup, evaluated by GPT4 for persuasiveness and harmfulness, sheds light on the ethical concerns surrounding LLMs and emphasizes the need for stronger safeguards against misinformation.

**URL** : <http://arxiv.org/pdf/2409.16241v1>

**Published** : 2024-09-24

## EuroLLM: Multilingual Language Models for Europe

**Authors** : Pedro Henrique Martins, Patrick Fernandes, João Alves, Nuno M. Guerreiro, Ricardo Rei, Duarte M. Alves, José Pombal, Amin Farajian, Manuel Faysse, Mateusz Klimaszewski, Pierre Colombo, Barry Haddow, José G. C. de Souza, Alexandra Birch, André F. T. Martins

**Abstract** : The quality of open-weight LLMs has seen significant improvement, yet they remain predominantly focused on English. In this paper, we introduce the EuroLLM project, aimed at developing a suite of open-weight multilingual LLMs capable of understanding and generating text in all official European Union languages, as well as several additional relevant languages. We outline the progress made to date, detailing our data collection and filtering process, the development of scaling laws, the creation of our multilingual tokenizer, and the data mix and modeling configurations. Additionally, we release our initial models: EuroLLM-1.7B and EuroLLM-1.7B-Instruct and report their performance on multilingual general benchmarks and machine translation.

**URL** : <http://arxiv.org/pdf/2409.16235v1>

**Published** : 2024-09-24

## Towards Enhancing Linked Data Retrieval in Conversational UIs using Large Language Models

**Authors** : Omar Mussa, Omer Rana, Benoît Goossens, Pablo Orozco-Terwengel, Charith Perera

**Abstract** : Despite the recent broad adoption of Large Language Models (LLMs) across various domains, their potential for enriching information systems in extracting and exploring Linked Data (LD) and Resource Description Framework (RDF) triplestores has not been extensively explored. This paper examines the integration of LLMs within existing systems, emphasising the enhancement of conversational user interfaces (UIs) and their capabilities for data extraction by producing more accurate SPARQL queries without the requirement for model retraining. Typically, conversational UI models necessitate retraining with the introduction of new datasets or updates, limiting their functionality as general-purpose extraction tools. Our approach addresses this limitation by incorporating LLMs into the conversational UI workflow, significantly enhancing their ability to comprehend and process user queries effectively. By leveraging the advanced natural language understanding capabilities of LLMs, our method improves RDF entity extraction within web systems employing conventional chatbots. This integration facilitates a more nuanced and context-aware interaction model, critical for handling the complex query patterns often encountered in RDF datasets and Linked Open Data (LOD) endpoints. The evaluation of this methodology shows a marked enhancement in system expressivity and the accuracy of responses to user queries, indicating a promising direction for future research in this area. This investigation not only underscores the versatility of LLMs in enhancing existing information systems but also sets the stage for further explorations into their potential applications within more specialised domains of web information systems.

**URL** : <http://arxiv.org/pdf/2409.16220v1>

**Published** : 2024-09-24

## CJEval: A Benchmark for Assessing Large Language Models Using Chinese Junior High School Exam Data

**Authors** : Qian-Wen Zhang, Haochen Wang, Fang Li, Siyu An, Lingfeng Qiao, Liangcai Gao, Di Yin, Xing Sun

**Abstract** : Online education platforms have significantly transformed the dissemination of educational resources by providing a dynamic and digital infrastructure. With the further enhancement of this transformation, the advent of Large Language Models (LLMs) has elevated the intelligence levels of these platforms. However, current academic benchmarks provide limited guidance for real-world industry scenarios. This limitation arises because educational applications require more than mere test question responses. To bridge this gap, we introduce CJEval, a benchmark based on Chinese Junior High School Exam Evaluations. CJEval consists of 26,136 samples across four application-level educational tasks covering ten subjects. These samples include not only questions and answers but also detailed annotations such as question types, difficulty levels, knowledge concepts, and answer explanations. By utilizing this benchmark, we assessed LLMs' potential applications and conducted a comprehensive analysis of their performance by fine-tuning on various educational tasks. Extensive experiments and discussions have highlighted the opportunities and challenges of applying LLMs in the field of education.

**URL** : <http://arxiv.org/pdf/2409.16202v2>

**Published** : 2024-09-24

## HelloBench: Evaluating Long Text Generation Capabilities of Large Language Models

**Authors** : Haoran Que, Feiyu Duan, Liqun He, Yutao Mou, Wangchunshu Zhou, Jiaheng Liu, Wenge Rong, Zekun Moore Wang, Jian Yang, Ge Zhang, Junran Peng, Zhaoxiang Zhang, Songyang Zhang, Kai Chen

**Abstract** : In recent years, Large Language Models (LLMs) have demonstrated remarkable capabilities in various tasks (e.g., long-context understanding), and many benchmarks have been proposed. However, we observe that long text generation capabilities are not well investigated. Therefore, we introduce the Hierarchical Long Text Generation Benchmark (HelloBench), a comprehensive, in-the-wild, and open-ended benchmark to evaluate LLMs' performance in generating long text. Based on Bloom's Taxonomy, HelloBench categorizes long text generation tasks into five subtasks: open-ended QA, summarization, chat, text completion, and heuristic text generation. Besides, we propose Hierarchical Long Text Evaluation (HelloEval), a human-aligned evaluation method that significantly reduces the time and effort required for human evaluation while maintaining a high correlation with human evaluation. We have conducted extensive experiments across around 30 mainstream LLMs and observed that the current LLMs lack long text generation capabilities. Specifically, first, regardless of whether the instructions include explicit or implicit length constraints, we observe that most LLMs cannot generate text that is longer than 4000 words. Second, we observe that while some LLMs can generate longer text, many issues exist (e.g., severe repetition and quality degradation). Third, to demonstrate the effectiveness of HelloEval, we compare HelloEval with traditional metrics (e.g., ROUGE, BLEU, etc.) and LLM-as-a-Judge methods, which show that HelloEval has the highest correlation with human evaluation. We release our code in <https://github.com/Quehry/HelloBench>.

**URL** : <http://arxiv.org/pdf/2409.16191v1>

**Published** : 2024-09-24

## Cyber Knowledge Completion Using Large Language Models

**Authors** : Braden K Webb, Sumit Purohit, Rounak Meyur

**Abstract** : The integration of the Internet of Things (IoT) into Cyber-Physical Systems (CPSs) has expanded their cyber-attack surface, introducing new and sophisticated threats with potential to exploit emerging vulnerabilities. Assessing the risks of CPSs is increasingly difficult due to incomplete and outdated cybersecurity knowledge. This highlights the urgent need for better-informed risk assessments and mitigation strategies. While previous efforts have relied on rule-based natural language processing (NLP) tools to map vulnerabilities, weaknesses, and attack patterns, recent advancements in Large Language Models (LLMs) present a unique opportunity to enhance cyber-attack knowledge completion through improved reasoning, inference, and summarization capabilities. We apply embedding models to encapsulate information on attack patterns and adversarial techniques, generating mappings between them using vector embeddings. Additionally, we propose a Retrieval-Augmented Generation (RAG)-based approach that leverages pre-trained models to create structured mappings between different taxonomies of threat patterns. Further, we use a small hand-labeled dataset to compare the proposed RAG-based approach to a baseline standard binary classification model. Thus, the proposed approach provides a comprehensive framework to address the challenge of cyber-attack knowledge graph completion.

**URL** : <http://arxiv.org/pdf/2409.16176v1>

**Published** : 2024-09-24

## Merging LoRAs like Playing LEGO: Pushing the Modularity of LoRA to Extremes Through Rank-Wise Clustering

**Authors** : Ziyu Zhao, Tao Shen, Didi Zhu, Zexi Li, Jing Su, Xuwu Wang, Kun Kuang, Fei Wu

**Abstract** : Low-Rank Adaptation (LoRA) has emerged as a popular technique for fine-tuning large language models (LLMs) to various domains due to its modular design and widespread availability on platforms like Huggingface. This modularity has sparked interest in combining multiple LoRAs to enhance LLM capabilities. However, existing methods for LoRA composition primarily focus on task-specific adaptations that require additional training, and current model merging techniques often fail to fully leverage LoRA's modular nature, leading to parameter interference and performance degradation. In this paper, we investigate the feasibility of disassembling and reassembling multiple LoRAs at a finer granularity, analogous to assembling LEGO blocks. We introduce the concept of Minimal Semantic Units (MSUs), where the parameters corresponding to each rank in LoRA function as independent units. These MSUs demonstrate permutation invariance and concatenation-summation equivalence properties, enabling flexible combinations to create new LoRAs. Building on these insights, we propose the LoRA-LEGO framework. This framework conducts rank-wise parameter clustering by grouping MSUs from different LoRAs into $k$ clusters. The centroid of each cluster serves as a representative MSU, enabling the assembly of a merged LoRA with an adjusted rank of $k$. Additionally, we apply a dual reweighting strategy to optimize the scale of the merged LoRA. Experiments across various benchmarks demonstrate that our method outperforms existing approaches in LoRA merging.

**URL** : <http://arxiv.org/pdf/2409.16167v1>

**Published** : 2024-09-24

## Controlling Risk of Retrieval-augmented Generation: A Counterfactual Prompting Framework

**Authors** : Lu Chen, Ruqing Zhang, Jiafeng Guo, Yixing Fan, Xueqi Cheng

**Abstract** : Retrieval-augmented generation (RAG) has emerged as a popular solution to mitigate the hallucination issues of large language models. However, existing studies on RAG seldom address the issue of predictive uncertainty, i.e., how likely it is that a RAG model's prediction is incorrect, resulting in uncontrollable risks in real-world applications. In this work, we emphasize the importance of risk control, ensuring that RAG models proactively refuse to answer questions with low confidence. Our research identifies two critical latent factors affecting RAG's confidence in its predictions: the quality of the retrieved results and the manner in which these results are utilized. To guide RAG models in assessing their own confidence based on these two latent factors, we develop a counterfactual prompting framework that induces the models to alter these factors and analyzes the effect on their answers. We also introduce a benchmarking procedure to collect answers with the option to abstain, facilitating a series of experiments. For evaluation, we introduce several risk-related metrics and the experimental results demonstrate the effectiveness of our approach.

**URL** : <http://arxiv.org/pdf/2409.16146v1>

**Published** : 2024-09-24

## HA-FGOVD: Highlighting Fine-grained Attributes via Explicit Linear Composition for Open-Vocabulary Object Detection

**Authors** : Yuqi Ma, Mengyin Liu, Chao Zhu, Xu-Cheng Yin

**Abstract** : Open-vocabulary object detection (OVD) models are considered to be Large Multi-modal Models (LMM), due to their extensive training data and a large number of parameters. Mainstream OVD models prioritize object coarse-grained category rather than focus on their fine-grained attributes, e.g., colors or materials, thus failed to identify objects specified with certain attributes. However, OVD models are pretrained on large-scale image-text pairs with rich attribute words, whose latent feature space can represent the global text feature as a linear composition of fine-grained attribute tokens without highlighting them. Therefore, we propose in this paper a universal and explicit approach for frozen mainstream OVD models that boosts their attribute-level detection capabilities by highlighting fine-grained attributes in explicit linear space. Firstly, a LLM is leveraged to highlight attribute words within the input text as a zero-shot prompted task. Secondly, by strategically adjusting the token masks, the text encoders of OVD models extract both global text and attribute-specific features, which are then explicitly composited as two vectors in linear space to form the new attribute-highlighted feature for detection tasks, where corresponding scalars are hand-crafted or learned to reweight both two vectors. Notably, these scalars can be seamlessly transferred among different OVD models, which proves that such an explicit linear composition is universal. Empirical evaluation on the FG-OVD dataset demonstrates that our proposed method uniformly improves fine-grained attribute-level OVD of various mainstream models and achieves new state-of-the-art performance.

**URL** : <http://arxiv.org/pdf/2409.16136v1>

**Published** : 2024-09-24

## MOSS: Enabling Code-Driven Evolution and Context Management for AI Agents

**Authors** : Ming Zhu, Yi Zhou

**Abstract** : Developing AI agents powered by large language models (LLMs) faces significant challenges in achieving true Turing completeness and adaptive, code-driven evolution. Current approaches often generate code independently of its runtime context, relying heavily on the LLM's memory, which results in inefficiencies and limits adaptability. Manual protocol development in sandbox environments further constrains the agent's autonomous adaptability. Crucially, achieving consistency in code and context across multi-turn interactions and ensuring isolation of local variables within each interaction remains an unsolved problem. We introduce MOSS (llM-oriented Operating System Simulation), a novel framework that addresses these challenges by integrating code generation with a dynamic context management system. MOSS ensures consistency and adaptability by using a mechanism that maintains the Python context across interactions, including isolation of local variables and preservation of runtime integrity. At its core, the framework employs an Inversion of Control (IoC) container in conjunction with decorators to enforce the least knowledge principle, allowing agents to focus on abstract interfaces rather than concrete implementations. This facilitates seamless integration of new tools and libraries, enables runtime instance replacement, and reduces prompt complexity, providing a "what you see is what you get" environment for the agent. Through a series of case studies, we show how this framework can enhance the efficiency and capabilities of agent development and highlight its advantages in moving towards Turing-complete agents capable of evolving through code.

**URL** : <http://arxiv.org/pdf/2409.16120v1>

**Published** : 2024-09-24

## LLM With Tools: A Survey

**Authors** : Zhuocheng Shen

**Abstract** : The integration of tools in augmenting large language models presents a novel approach toward enhancing the efficiency and accuracy of these models in handling specific, complex tasks. This paper delves into the methodology,challenges, and developments in the realm of teaching LLMs to use external tools, thereby pushing the boundaries of their capabilities beyond pre-existing knowledge bases. We introduce a standardized paradigm for tool integration guided by a series of functions that map user instructions to actionable plans and their execution, emphasizing the significance of understanding user intent, tool selection, and dynamic plan adjustment. Our exploration reveals the various challenges encountered, such as tool invocation timing, selection accuracy, and the need for robust reasoning processes. In addressing these challenges, we investigate techniques within the context of fine-tuning and incontext learning paradigms, highlighting innovative approaches to ensure diversity, augment datasets, and improve generalization.Furthermore, we investigate a perspective on enabling LLMs to not only utilize but also autonomously create tools, which may redefine their role from mere tool users to tool creators. Finally,we reproduced Chameleon's results on ScienceQA and analyzed the code structure.

**URL** : <http://arxiv.org/pdf/2409.18807v1>

**Published** : 2024-09-24

## Exploring Hint Generation Approaches in Open-Domain Question Answering

**Authors** : Jamshid Mozafari, Abdelrahman Abdallah, Bhawna Piryani, Adam Jatowt

**Abstract** : Automatic Question Answering (QA) systems rely on contextual information to provide accurate answers. Commonly, contexts are prepared through either retrieval-based or generation-based methods. The former involves retrieving relevant documents from a corpus like Wikipedia, whereas the latter uses generative models such as Large Language Models (LLMs) to generate the context. In this paper, we introduce a novel context preparation approach called HINTQA, which employs Automatic Hint Generation (HG) techniques. Unlike traditional methods, HINTQA prompts LLMs to produce hints about potential answers for the question rather than generating relevant context. We evaluate our approach across three QA datasets including TriviaQA, NaturalQuestions, and Web Questions, examining how the number and order of hints impact performance. Our findings show that the HINTQA surpasses both retrieval-based and generation-based approaches. We demonstrate that hints enhance the accuracy of answers more than retrieved and generated contexts.

**URL** : <http://arxiv.org/pdf/2409.16096v1>

**Published** : 2024-09-24

## Unlocking Markets: A Multilingual Benchmark to Cross-Market Question Answering

**Authors** : Yifei Yuan, Yang Deng, Anders Søgaard, Mohammad Aliannejadi

**Abstract** : Users post numerous product-related questions on e-commerce platforms, affecting their purchase decisions. Product-related question answering (PQA) entails utilizing product-related resources to provide precise responses to users. We propose a novel task of Multilingual Cross-market Product-based Question Answering (MCPQA) and define the task as providing answers to product-related questions in a main marketplace by utilizing information from another resource-rich auxiliary marketplace in a multilingual context. We introduce a large-scale dataset comprising over 7 million questions from 17 marketplaces across 11 languages. We then perform automatic translation on the Electronics category of our dataset, naming it as McMarket. We focus on two subtasks: review-based answer generation and product-related question ranking. For each subtask, we label a subset of McMarket using an LLM and further evaluate the quality of the annotations via human assessment. We then conduct experiments to benchmark our dataset, using models ranging from traditional lexical models to LLMs in both single-market and cross-market scenarios across McMarket and the corresponding LLM subset. Results show that incorporating cross-market information significantly enhances performance in both tasks.

**URL** : <http://arxiv.org/pdf/2409.16025v1>

**Published** : 2024-09-24

## AI Can Be Cognitively Biased: An Exploratory Study on Threshold Priming in LLM-Based Batch Relevance Assessment

**Authors** : Nuo Chen, Jiqun Liu, Xiaoyu Dong, Qijiong Liu, Tetsuya Sakai, Xiao-Ming Wu

**Abstract** : Cognitive biases are systematic deviations in thinking that lead to irrational judgments and problematic decision-making, extensively studied across various fields. Recently, large language models (LLMs) have shown advanced understanding capabilities but may inherit human biases from their training data. While social biases in LLMs have been well-studied, cognitive biases have received less attention, with existing research focusing on specific scenarios. The broader impact of cognitive biases on LLMs in various decision-making contexts remains underexplored. We investigated whether LLMs are influenced by the threshold priming effect in relevance judgments, a core task and widely-discussed research topic in the Information Retrieval (IR) coummunity. The priming effect occurs when exposure to certain stimuli unconsciously affects subsequent behavior and decisions. Our experiment employed 10 topics from the TREC 2019 Deep Learning passage track collection, and tested AI judgments under different document relevance scores, batch lengths, and LLM models, including GPT-3.5, GPT-4, LLaMa2-13B and LLaMa2-70B. Results showed that LLMs tend to give lower scores to later documents if earlier ones have high relevance, and vice versa, regardless of the combination and model used. Our finding demonstrates that LLM%u2019s judgments, similar to human judgments, are also influenced by threshold priming biases, and suggests that researchers and system engineers should take into account potential human-like cognitive biases in designing, evaluating, and auditing LLMs in IR tasks and beyond.

**URL** : <http://arxiv.org/pdf/2409.16022v1>

**Published** : 2024-09-24

## Bridging Speech and Text: Enhancing ASR with Pinyin-to-Character Pre-training in LLMs

**Authors** : Yang Yuhang, Peng Yizhou, Eng Siong Chng, Xionghu Zhong

**Abstract** : The integration of large language models (LLMs) with pre-trained speech models has opened up new avenues in automatic speech recognition (ASR). While LLMs excel in multimodal understanding tasks, effectively leveraging their capabilities for ASR remains a significant challenge. This paper presents a novel training approach to enhance LLM performance in ASR tasks. We propose pre-training LLMs on Pinyin embedding sequences, which represent pronunciation features, to generate corresponding Chinese characters. This step enables the LLM to adapt to generating text from pronunciation features before encountering real speech data. Furthermore, we fine-tune the LoRA parameters to enhance the LLM's understanding of speech modality information. In AISHELL-1 corpus, our approach yields a 9.5% relative improvement in ASR tasks compared to the baseline without Pinyi-to-Character pre-training. Additionally, incorporating auxiliary text data for Pinyi-to-Character pre-training further boosts performance, achieving a 19.0% relative improvement.

**URL** : <http://arxiv.org/pdf/2409.16005v1>

**Published** : 2024-09-24

## DataGpt-SQL-7B: An Open-Source Language Model for Text-to-SQL

**Authors** : Lixia Wu, Peng Li, Junhong Lou, Lei Fu

**Abstract** : In addressing the pivotal role of translating natural language queries into SQL commands, we propose a suite of compact, fine-tuned models and self-refine mechanisms to democratize data access and analysis for non-expert users, mitigating risks associated with closed-source Large Language Models. Specifically, we constructed a dataset of over 20K sample for Text-to-SQL as well as the preference dateset, to improve the efficiency in the domain of SQL generation. To further ensure code validity, a code corrector was integrated into the model. Our system, DataGpt-sql, achieved 87.2% accuracy on the spider-dev, respectively, showcasing the effectiveness of our solution in text-to-SQL conversion tasks. Our code, data, and models are available at \url{[https://github.com/CainiaoTechAi/datagpt-sql-7b}](https://github.com/CainiaoTechAi/datagpt-sql-7b%7D)

**URL** : <http://arxiv.org/pdf/2409.15985v1>

**Published** : 2024-09-24

## Finetuning LLMs for Comparative Assessment Tasks

**Authors** : Vatsal Raina, Adian Liusie, Mark Gales

**Abstract** : Automated assessment in natural language generation is a challenging task. Instruction-tuned large language models (LLMs) have shown promise in reference-free evaluation, particularly through comparative assessment. However, the quadratic computational complexity of pairwise comparisons limits its scalability. To address this, efficient comparative assessment has been explored by applying comparative strategies on zero-shot LLM probabilities. We propose a framework for finetuning LLMs for comparative assessment to align the model's output with the target distribution of comparative probabilities. By training on soft probabilities, our approach improves state-of-the-art performance while maintaining high performance with an efficient subset of comparisons.

**URL** : <http://arxiv.org/pdf/2409.15979v1>

**Published** : 2024-09-24

## Automated test generation to evaluate tool-augmented LLMs as conversational AI agents

**Authors** : Samuel Arcadinho, David Aparicio, Mariana Almeida

**Abstract** : Tool-augmented LLMs are a promising approach to create AI agents that can have realistic conversations, follow procedures, and call appropriate functions. However, evaluating them is challenging due to the diversity of possible conversations, and existing datasets focus only on single interactions and function-calling. We present a test generation pipeline to evaluate LLMs as conversational AI agents. Our framework uses LLMs to generate diverse tests grounded on user-defined procedures. For that, we use intermediate graphs to limit the LLM test generator's tendency to hallucinate content that is not grounded on input procedures, and enforces high coverage of the possible conversations. Additionally, we put forward ALMITA, a manually curated dataset for evaluating AI agents in customer support, and use it to evaluate existing LLMs. Our results show that while tool-augmented LLMs perform well in single interactions, they often struggle to handle complete conversations. While our focus is on customer support, our method is general and capable of AI agents for different domains.

**URL** : <http://arxiv.org/pdf/2409.15934v1>

**Published** : 2024-09-24

## SLIMER-IT: Zero-Shot NER on Italian Language

**Authors** : Andrew Zamai, Leonardo Rigutini, Marco Maggini, Andrea Zugarini

**Abstract** : Traditional approaches to Named Entity Recognition (NER) frame the task into a BIO sequence labeling problem. Although these systems often excel in the downstream task at hand, they require extensive annotated data and struggle to generalize to out-of-distribution input domains and unseen entity types. On the contrary, Large Language Models (LLMs) have demonstrated strong zero-shot capabilities. While several works address Zero-Shot NER in English, little has been done in other languages. In this paper, we define an evaluation framework for Zero-Shot NER, applying it to the Italian language. Furthermore, we introduce SLIMER-IT, the Italian version of SLIMER, an instruction-tuning approach for zero-shot NER leveraging prompts enriched with definition and guidelines. Comparisons with other state-of-the-art models, demonstrate the superiority of SLIMER-IT on never-seen-before entity tags.

**URL** : <http://arxiv.org/pdf/2409.15933v1>

**Published** : 2024-09-24

## Planning in the Dark: LLM-Symbolic Planning Pipeline without Experts

**Authors** : Sukai Huang, Nir Lipovetzky, Trevor Cohn

**Abstract** : Large Language Models (LLMs) have shown promise in solving natural language-described planning tasks, but their direct use often leads to inconsistent reasoning and hallucination. While hybrid LLM-symbolic planning pipelines have emerged as a more robust alternative, they typically require extensive expert intervention to refine and validate generated action schemas. It not only limits scalability but also introduces a potential for biased interpretation, as a single expert's interpretation of ambiguous natural language descriptions might not align with the user's actual intent. To address this, we propose a novel approach that constructs an action schema library to generate multiple candidates, accounting for the diverse possible interpretations of natural language descriptions. We further introduce a semantic validation and ranking module that automatically filter and rank the generated schemas and plans without expert-in-the-loop. The experiments showed our pipeline maintains superiority in planning over the direct LLM planning approach. These findings demonstrate the feasibility of a fully automated end-to-end LLM-symbolic planner that requires no expert intervention, opening up the possibility for a broader audience to engage with AI planning with less prerequisite of domain expertise.

**URL** : <http://arxiv.org/pdf/2409.15915v1>

**Published** : 2024-09-24

## Enhancing IoT based Plant Health Monitoring through Advanced Human Plant Interaction using Large Language Models and Mobile Applications

**Authors** : Kriti Agarwal, Samhruth Ananthanarayanan, Srinitish Srinivasan, Abirami S

**Abstract** : This paper presents the development of a novel plant communication application that allows plants to "talk" to humans using real-time sensor data and AI-powered language models. Utilizing soil sensors that track moisture, temperature, and nutrient levels, the system feeds this data into the Gemini API, where it is processed and transformed into natural language insights about the plant's health and "mood." Developed using Flutter, Firebase, and ThingSpeak, the app offers a seamless user experience with real-time interaction capabilities. By fostering human-plant connectivity, this system enhances plant care practices, promotes sustainability, and introduces innovative applications for AI and IoT technologies in both personal and agricultural contexts. The paper explores the technical architecture, system integration, and broader implications of AI-driven plant communication.

**URL** : <http://arxiv.org/pdf/2409.15910v1>

**Published** : 2024-09-24

## Enhancing Text-to-SQL Capabilities of Large Language Models via Domain Database Knowledge Injection

**Authors** : Xingyu Ma, Xin Tian, Lingxiang Wu, Xuepeng Wang, Xueming Tang, Jinqiao Wang

**Abstract** : Text-to-SQL is a subtask in semantic parsing that has seen rapid progress with the evolution of Large Language Models (LLMs). However, LLMs face challenges due to hallucination issues and a lack of domain-specific database knowledge(such as table schema and cell values). As a result, they can make errors in generating table names, columns, and matching values to the correct columns in SQL statements. This paper introduces a method of knowledge injection to enhance LLMs' ability to understand schema contents by incorporating prior knowledge. This approach improves their performance in Text-to-SQL tasks. Experimental results show that pre-training LLMs on domain-specific database knowledge and fine-tuning them on downstream Text-to-SQL tasks significantly improves the Execution Match (EX) and Exact Match (EM) metrics across various models. This effectively reduces errors in generating column names and matching values to the columns. Furthermore, the knowledge-injected models can be applied to many downstream Text-to-SQL tasks, demonstrating the generalizability of the approach presented in this paper.

**URL** : <http://arxiv.org/pdf/2409.15907v1>

**Published** : 2024-09-24

## Boosting Code-Switching ASR with Mixture of Experts Enhanced Speech-Conditioned LLM

**Authors** : Fengrun Zhang, Wang Geng, Hukai Huang, Cheng Yi, He Qu

**Abstract** : In this paper, we introduce a speech-conditioned Large Language Model (LLM) integrated with a Mixture of Experts (MoE) based connector to address the challenge of Code-Switching (CS) in Automatic Speech Recognition (ASR). Specifically, we propose an Insertion and Deletion of Interruption Token (IDIT) mechanism for better transfer text generation ability of LLM to speech recognition task. We also present a connecter with MoE architecture that manages multiple languages efficiently. To further enhance the collaboration of multiple experts and leverage the understanding capabilities of LLM, we propose a two-stage progressive training strategy: 1) The connector is unfrozen and trained with language-specialized experts to map speech representations to the text space. 2) The connector and LLM LoRA adaptor are trained with the proposed IDIT mechanism and all experts are activated to learn general representations. Experimental results demonstrate that our method significantly outperforms state-of-the-art models, including end-to-end and large-scale audio-language models.

**URL** : <http://arxiv.org/pdf/2409.15905v1>

**Published** : 2024-09-24

## Konstruktor: A Strong Baseline for Simple Knowledge Graph Question Answering

**Authors** : Maria Lysyuk, Mikhail Salnikov, Pavel Braslavski, Alexander Panchenko

**Abstract** : While being one of the most popular question types, simple questions such as "Who is the author of Cinderella?", are still not completely solved. Surprisingly, even the most powerful modern Large Language Models are prone to errors when dealing with such questions, especially when dealing with rare entities. At the same time, as an answer may be one hop away from the question entity, one can try to develop a method that uses structured knowledge graphs (KGs) to answer such questions. In this paper, we introduce Konstruktor - an efficient and robust approach that breaks down the problem into three steps: (i) entity extraction and entity linking, (ii) relation prediction, and (iii) querying the knowledge graph. Our approach integrates language models and knowledge graphs, exploiting the power of the former and the interpretability of the latter. We experiment with two named entity recognition and entity linking methods and several relation detection techniques. We show that for relation detection, the most challenging step of the workflow, a combination of relation classification/generation and ranking outperforms other methods. We report Konstruktor's strong results on four datasets.

**URL** : <http://arxiv.org/pdf/2409.15902v1>

**Published** : 2024-09-24

## Footer

[ ](https://github.com "GitHub") (C) 2025 GitHub, Inc. 

### Footer navigation

  * [Terms](https://docs.github.com/site-policy/github-terms/github-terms-of-service)
  * [Privacy](https://docs.github.com/site-policy/privacy-policies/github-privacy-statement)
  * [Security](https://github.com/security)
  * [Status](https://www.githubstatus.com/)
  * [Docs](https://docs.github.com/)
  * [Contact](https://support.github.com?tags=dotcom-footer)
  * Manage cookies 
  * Do not share my personal information 



You can’t perform that action at this time. 


