# documents

scrape_date: 2025-03-17T13:26:50.821397
content_type: markdown
hash: 630e6d86fcaf31932f79d79b4f6fd3aa62a49d8c760b72b11cc4725cad8add8f
url:
https://github.com/Luvata/arxive/blob/9341b486ad647204168aa013996db66c9df8196a/pages/2025-02-26-cs-cl.html

title:
arxive/pages/2025-02-26-cs-cl.html at 9341b486ad647204168aa013996db66c9df8196a · Luvata/arxive · GitHub

content:
Skip to content

## Navigation Menu

Toggle navigation

[ ](/)

[ Sign in ](/login?return_to=https%3A%2F%2Fgithub.com%2FLuvata%2Farxive%2Fblob%2F9341b486ad647204168aa013996db66c9df8196a%2Fpages%2F2025-02-26-cs-cl.html)

  * Product 

    * [ GitHub Copilot Write better code with AI  ](https://github.com/features/copilot)
    * [ Security Find and fix vulnerabilities  ](https://github.com/features/security)
    * [ Actions Automate any workflow  ](https://github.com/features/actions)
    * [ Codespaces Instant dev environments  ](https://github.com/features/codespaces)
    * [ Issues Plan and track work  ](https://github.com/features/issues)
    * [ Code Review Manage code changes  ](https://github.com/features/code-review)
    * [ Discussions Collaborate outside of code  ](https://github.com/features/discussions)
    * [ Code Search Find more, search less  ](https://github.com/features/code-search)

Explore
    * [ All features ](https://github.com/features)
    * [ Documentation  ](https://docs.github.com)
    * [ GitHub Skills  ](https://skills.github.com)
    * [ Blog  ](https://github.blog)

  * Solutions 

By company size
    * [ Enterprises ](https://github.com/enterprise)
    * [ Small and medium teams ](https://github.com/team)
    * [ Startups ](https://github.com/enterprise/startups)
    * [ Nonprofits ](/solutions/industry/nonprofits)

By use case
    * [ DevSecOps ](/solutions/use-case/devsecops)
    * [ DevOps ](/solutions/use-case/devops)
    * [ CI/CD ](/solutions/use-case/ci-cd)
    * [ View all use cases ](/solutions/use-case)

By industry
    * [ Healthcare ](/solutions/industry/healthcare)
    * [ Financial services ](/solutions/industry/financial-services)
    * [ Manufacturing ](/solutions/industry/manufacturing)
    * [ Government ](/solutions/industry/government)
    * [ View all industries ](/solutions/industry)

[ View all solutions  ](/solutions)

  * Resources 

Topics
    * [ AI ](/resources/articles/ai)
    * [ DevOps ](/resources/articles/devops)
    * [ Security ](/resources/articles/security)
    * [ Software Development ](/resources/articles/software-development)
    * [ View all ](/resources/articles)

Explore
    * [ Learning Pathways  ](https://resources.github.com/learn/pathways)
    * [ Events & Webinars  ](https://resources.github.com)
    * [ Ebooks & Whitepapers ](https://github.com/resources/whitepapers)
    * [ Customer Stories ](https://github.com/customer-stories)
    * [ Partners  ](https://partner.github.com)
    * [ Executive Insights ](https://github.com/solutions/executive-insights)

  * Open Source 

    * [ GitHub Sponsors Fund open source developers  ](/sponsors)

    * [ The ReadME Project GitHub community articles  ](https://github.com/readme)

Repositories
    * [ Topics ](https://github.com/topics)
    * [ Trending ](https://github.com/trending)
    * [ Collections ](https://github.com/collections)

  * Enterprise 

    * [ Enterprise platform AI-powered developer platform  ](/enterprise)

Available add-ons
    * [ Advanced Security Enterprise-grade security features  ](https://github.com/enterprise/advanced-security)
    * [ Copilot for business Enterprise-grade AI features  ](/features/copilot/copilot-business)
    * [ Premium Support Enterprise-grade 24/7 support  ](/premium-support)

  * [Pricing](https://github.com/pricing)



Search or jump to...

# Search code, repositories, users, issues, pull requests...

Search 

Clear




[Search syntax tips](https://docs.github.com/search-github/github-code-search/understanding-github-code-search-syntax)

#  Provide feedback 

We read every piece of feedback, and take your input very seriously.

Include my email address so I can be contacted

Cancel  Submit feedback 

#  Saved searches 

## Use saved searches to filter your results more quickly

Name

Query

To see all available qualifiers, see our [documentation](https://docs.github.com/search-github/github-code-search/understanding-github-code-search-syntax). 

Cancel  Create saved search 

[ Sign in ](/login?return_to=https%3A%2F%2Fgithub.com%2FLuvata%2Farxive%2Fblob%2F9341b486ad647204168aa013996db66c9df8196a%2Fpages%2F2025-02-26-cs-cl.html)

[ Sign up ](/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fblob%2Fshow&source=header-repo&source_repo=Luvata%2Farxive) Reseting focus

You signed in with another tab or window. [Reload]() to refresh your session. You signed out in another tab or window. [Reload]() to refresh your session. You switched accounts on another tab or window. [Reload]() to refresh your session. Dismiss alert

{{ message }}

[ Luvata ](/Luvata) / **[arxive](/Luvata/arxive) ** Public

  * [ Notifications ](/login?return_to=%2FLuvata%2Farxive) You must be signed in to change notification settings
  * [ Fork 1 ](/login?return_to=%2FLuvata%2Farxive)
  * [ Star  11 ](/login?return_to=%2FLuvata%2Farxive)




  * [ Code ](/Luvata/arxive)
  * [ Issues 0 ](/Luvata/arxive/issues)
  * [ Pull requests 0 ](/Luvata/arxive/pulls)
  * [ Actions ](/Luvata/arxive/actions)
  * [ Projects 0 ](/Luvata/arxive/projects)
  * [ Security ](/Luvata/arxive/security)
  * [ Insights ](/Luvata/arxive/pulse)



Additional navigation options

  * [ Code  ](/Luvata/arxive)
  * [ Issues  ](/Luvata/arxive/issues)
  * [ Pull requests  ](/Luvata/arxive/pulls)
  * [ Actions  ](/Luvata/arxive/actions)
  * [ Projects  ](/Luvata/arxive/projects)
  * [ Security  ](/Luvata/arxive/security)
  * [ Insights  ](/Luvata/arxive/pulse)



## Files

9341b48

## Breadcrumbs

  1. [arxive](/Luvata/arxive/tree/9341b486ad647204168aa013996db66c9df8196a)
  2. /[pages](/Luvata/arxive/tree/9341b486ad647204168aa013996db66c9df8196a/pages)



/

# 2025-02-26-cs-cl.html

Copy path

Blame

Blame

## Latest commit

## History

[History](/Luvata/arxive/commits/9341b486ad647204168aa013996db66c9df8196a/pages/2025-02-26-cs-cl.html)

[](/Luvata/arxive/commits/9341b486ad647204168aa013996db66c9df8196a/pages/2025-02-26-cs-cl.html)

1575 lines (1571 loc) · 348 KB

9341b48

## Breadcrumbs

  1. [arxive](/Luvata/arxive/tree/9341b486ad647204168aa013996db66c9df8196a)
  2. /[pages](/Luvata/arxive/tree/9341b486ad647204168aa013996db66c9df8196a/pages)



/

# 2025-02-26-cs-cl.html

Top

## File metadata and controls

  * Code

  * Blame




1575 lines (1571 loc) · 348 KB

[Raw](https://github.com/Luvata/arxive/raw/9341b486ad647204168aa013996db66c9df8196a/pages/2025-02-26-cs-cl.html)

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465

466

467

468

469

470

471

472

473

474

475

476

477

478

479

480

481

482

483

484

485

486

487

488

489

490

491

492

493

494

495

496

497

498

499

500

501

502

503

504

505

506

507

508

509

510

511

512

513

514

515

516

517

518

519

520

521

522

523

524

525

526

527

528

529

530

531

532

533

534

535

536

537

538

539

540

541

542

543

544

545

546

547

548

549

550

551

552

553

554

555

556

557

558

559

560

561

562

563

564

565

566

567

568

569

570

571

572

573

574

575

576

577

578

579

580

581

582

583

584

585

586

587

588

589

590

591

592

593

594

595

596

597

598

599

600

601

602

603

604

605

606

607

608

609

610

611

612

613

614

615

616

617

618

619

620

621

622

623

624

625

626

627

628

629

630

631

632

633

634

635

636

637

638

639

640

641

642

643

644

645

646

647

648

649

650

651

652

653

654

655

656

657

658

659

660

661

662

663

664

665

666

667

668

669

670

671

672

673

674

675

676

677

678

679

680

681

682

683

684

685

686

687

688

689

690

691

692

693

694

695

696

697

698

699

700

701

702

703

704

705

706

707

708

709

710

711

712

713

714

715

716

717

718

719

720

721

722

723

724

725

726

727

728

729

730

731

732

733

734

735

736

737

738

739

740

741

742

743

744

745

746

747

748

749

750

751

752

753

754

755

756

757

758

759

760

761

762

763

764

765

766

767

768

769

770

771

772

773

774

775

776

777

778

779

780

781

782

783

784

785

786

787

788

789

790

791

792

793

794

795

796

797

798

799

800

801

802

803

804

805

806

807

808

809

810

811

812

813

814

815

816

817

818

819

820

821

822

823

824

825

826

827

828

829

830

831

832

833

834

835

836

837

838

839

840

841

842

843

844

845

846

847

848

849

850

851

852

853

854

855

856

857

858

859

860

861

862

863

864

865

866

867

868

869

870

871

872

873

874

875

876

877

878

879

880

881

882

883

884

885

886

887

888

889

890

891

892

893

894

895

896

897

898

899

900

901

902

903

904

905

906

907

908

909

910

911

912

913

914

915

916

917

918

919

920

921

922

923

924

925

926

927

928

929

930

931

932

933

934

935

936

937

938

939

940

941

942

943

944

945

946

947

948

949

950

951

952

953

954

955

956

957

958

959

960

961

962

963

964

965

966

967

968

969

970

971

972

973

974

975

976

977

978

979

980

981

982

983

984

985

986

987

988

989

990

991

992

993

994

995

996

997

998

999

1000

<!DOCTYPE html>

<html>

<head>

<title>2025-02-26-cs-cl</title>

<style>

body {

font-family: Arial, sans-serif;

margin: 0;

padding: 0;

display: flex;

justify-content: center;

background-color: #f2f2f2;

}

.container {

width: 60%;

max-width: 900px;

min-width: 300px;

background-color: #fff;

padding: 20px;

margin: 20px;

border-radius: 10px;

box-shadow: 0px 0px 10px rgba(0, 0, 0, 0.05);

font-size: 0.9em;

}

@media (max-width: 768px) {

.container {

width: 90%;

font-size: 0.8em;

}

h1 {

font-size: 1.4em;

}

}

h1 {

color: #333;

font-size: 1.6em;

}

.article {

border-bottom: 1px solid #ddd;

padding: 10px 0;

}

.article a {

color: #007BFF;

text-decoration: none;

}

.article a:hover {

color: #0056b3;

}

.copy-button {

background-color: #007BFF;

color: #fff;

border: none;

padding: 5px 10px;

cursor: pointer;

margin-top: 10px;

}

.copy-button:active {

background-color: #0056b3;

}

.copy-message {

display: none;

background-color: #4CAF50;

color: #fff;

padding: 5px;

margin-top: 10px;

border-radius: 3px;

}

</style>

<script>

function copyToClipboard(text, buttonId) {

const el = document.createElement('textarea');

el.value = text;

document.body.appendChild(el);

el.select();

document.execCommand('copy');

document.body.removeChild(el);

const messageElement = document.getElementById('copy-message-' + buttonId);

messageElement.innerHTML = 'Copied "' + text + '" to clipboard';

messageElement.style.display = 'block';

setTimeout(function() {

messageElement.style.display = 'none';

}, 3000);

}

</script>

</head>

<body>

<div class="container">

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.17538">Policy Learning with a Natural Language Action Space: A Causal Approach</a></h1>

<p><b>Authors:</b> Bohan Zhang, Yixin Wang, Paramveer S. Dhillon</p>

<p>Abstract: This paper introduces a novel causal framework for multi-stage decision-making in natural language action spaces where outcomes are only observed after a sequence of actions. While recent approaches like Proximal Policy Optimization (PPO) can handle such delayed-reward settings in high-dimensional action spaces, they typically require multiple models (policy, value, and reward) and substantial training data. Our approach employs Q-learning to estimate Dynamic Treatment Regimes (DTR) through a single model, enabling data-efficient policy learning via gradient ascent on language embeddings. A key technical contribution of our approach is a decoding strategy that translates optimized embeddings back into coherent natural language. We evaluate our approach on mental health intervention, hate speech countering, and sentiment transfer tasks, demonstrating significant improvements over competitive baselines across multiple metrics. Notably, our method achieves superior transfer strength while maintaining content preservation and fluency, as validated through human evaluation. Our work provides a practical foundation for learning optimal policies in complex language tasks where training data is limited.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.17538', 0)">Copy Link</button>

<div id="copy-message-0" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.17571">Towards Conditioning Clinical Text Generation for User Control</a></h1>

<p><b>Authors:</b> Osman Alperen Kora\c{s}, Rabi Bahnan, Jens Kleesiek, Amin Dada</p>

<p>Abstract: Deploying natural language generation systems in clinical settings remains challenging despite advances in Large Language Models (LLMs), which continue to exhibit hallucinations and factual inconsistencies, necessitating human oversight. This paper explores automated dataset augmentation using LLMs as human proxies to condition LLMs for clinician control without increasing cognitive workload. On the BioNLP ACL'24 Discharge Me! Shared Task, we achieve new state-of-the-art results with simpler methods than prior submissions through more efficient training, yielding a 9\% relative improvement without augmented training and up to 34\% with dataset augmentation. Preliminary human evaluation further supports the effectiveness of our approach, highlighting the potential of augmenting clinical text generation for control to enhance relevance, accuracy, and factual consistency.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.17571', 1)">Copy Link</button>

<div id="copy-message-1" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.17589">End-to-End Chart Summarization via Visual Chain-of-Thought in Vision-Language Models</a></h1>

<p><b>Authors:</b> Raymond Choi, Frank Burns, Chase Lawrence</p>

<p>Abstract: Automated chart summarization is crucial for enhancing data accessibility and enabling efficient information extraction from visual data. While recent advances in visual-language models (VLMs) have demonstrated promise, existing methods often suffer from limitations in matching the generated summary to the chart data and in reasoning about complex chart patterns. This paper introduces End-to-End Visual Chain-of-Thought (V-CoT) for chart summarization, a novel approach optimized for Large Vision-Language Models (LVLMs). Our method directly trains an LVLM to process chart images and generate textual summaries in an end-to-end fashion, eliminating the need for explicit chart parsing modules. We incorporate a visual Chain-of-Thought mechanism through instruction fine-tuning, implicitly guiding the LVLM to perform visual reasoning steps during summary generation. Evaluated on the large-scale Chart-Sum-QA dataset, our V-CoT method significantly outperforms state-of-the-art baselines across a range of automatic metrics, including BLEU, BLEURT, CIDEr, and CS, and demonstrates superior matching degree and reasoning correctness in human evaluations. Ablation studies and detailed analyses further validate the effectiveness and robustness of our proposed approach, establishing a new benchmark for end-to-end chart summarization.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.17589', 2)">Copy Link</button>

<div id="copy-message-2" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.17591">Proactive Privacy Amnesia for Large Language Models: Safeguarding PII with Negligible Impact on Model Utility</a></h1>

<p><b>Authors:</b> Martin Kuo, Jingyang Zhang, Jianyi Zhang, Minxue Tang, Louis DiValentin, Aolin Ding, Jingwei Sun, William Chen, Amin Hass, Tianlong Chen, Yiran Chen, Hai Li</p>

<p>Abstract: With the rise of large language models (LLMs), increasing research has recognized their risk of leaking personally identifiable information (PII) under malicious attacks. Although efforts have been made to protect PII in LLMs, existing methods struggle to balance privacy protection with maintaining model utility. In this paper, inspired by studies of amnesia in cognitive science, we propose a novel approach, Proactive Privacy Amnesia (PPA), to safeguard PII in LLMs while preserving their utility. This mechanism works by actively identifying and forgetting key memories most closely associated with PII in sequences, followed by a memory implanting using suitable substitute memories to maintain the LLM's functionality. We conduct evaluations across multiple models to protect common PII, such as phone numbers and physical addresses, against prevalent PII-targeted attacks, demonstrating the superiority of our method compared with other existing defensive techniques. The results show that our PPA method completely eliminates the risk of phone number exposure by 100% and significantly reduces the risk of physical address exposure by 9.8% - 87.6%, all while maintaining comparable model utility performance.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.17591', 3)">Copy Link</button>

<div id="copy-message-3" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.17599">MEDA: Dynamic KV Cache Allocation for Efficient Multimodal Long-Context Inference</a></h1>

<p><b>Authors:</b> Zhongwei Wan, Hui Shen, Xin Wang, Che Liu, Zheda Mai, Mi Zhang</p>

<p>Abstract: Long-context Multimodal Large Language Models (MLLMs) that incorporate long text-image and text-video modalities, demand substantial resources as their multimodal Key-Value (KV) caches grow with increasing input lengths, challenging inference efficiency. Existing methods for KV cache compression, in both text-only and multimodal LLMs, have neglected attention density variations across layers, thus often adopting uniform or progressive reduction strategies for layer-wise cache allocation. In this work, we propose MEDA, a dynamic layer-wise KV cache allocation method for efficient multimodal long-context inference. As its core, MEDA utilizes cross-modal attention entropy to determine the KV cache size at each MLLMs layer. Given the dynamically allocated KV cache size at each layer, MEDA also employs a KV pair selection scheme to identify which KV pairs to select and a KV pair merging strategy that merges the selected and non-selected ones to preserve information from the entire context. MEDA achieves up to 72% KV cache memory reduction and 2.82 times faster decoding speed, while maintaining or enhancing performance on various multimodal tasks in long-context settings, including multi-images and long-video scenarios. Our code is released at https://github.com/AIoT-MLSys-Lab/MEDA.</p>

<p>URLs: <a href="https://github.com/AIoT-MLSys-Lab/MEDA.">https://github.com/AIoT-MLSys-Lab/MEDA.</a></p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.17599, https://github.com/AIoT-MLSys-Lab/MEDA.', 4)">Copy Link</button>

<div id="copy-message-4" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.17605">PICASO: Permutation-Invariant Context Composition with State Space Models</a></h1>

<p><b>Authors:</b> Tian Yu Liu, Alessandro Achille, Matthew Trager, Aditya Golatkar, Luca Zancato, Stefano Soatto</p>

<p>Abstract: Providing Large Language Models with relevant contextual knowledge at inference time has been shown to greatly improve the quality of their generations. This is often achieved by prepending informative passages of text, or 'contexts', retrieved from external knowledge bases to their input. However, processing additional contexts online incurs significant computation costs that scale with their length. State Space Models (SSMs) offer a promising solution by allowing a database of contexts to be mapped onto fixed-dimensional states from which to start the generation. A key challenge arises when attempting to leverage information present across multiple contexts, since there is no straightforward way to condition generation on multiple independent states in existing SSMs. To address this, we leverage a simple mathematical relation derived from SSM dynamics to compose multiple states into one that efficiently approximates the effect of concatenating textual contexts. Since the temporal ordering of contexts can often be uninformative, we enforce permutation-invariance by efficiently averaging states obtained via our composition algorithm across all possible context orderings. We evaluate our resulting method on WikiText and MSMARCO in both zero-shot and fine-tuned settings, and show that we can match the strongest performing baseline while enjoying on average 5.4x speedup.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.17605', 5)">Copy Link</button>

<div id="copy-message-5" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.17611">Evaluating the Effect of Retrieval Augmentation on Social Biases</a></h1>

<p><b>Authors:</b> Tianhui Zhang, Yi Zhou, Danushka Bollegala</p>

<p>Abstract: Retrieval Augmented Generation (RAG) has gained popularity as a method for conveniently incorporating novel facts that were not seen during the pre-training stage in Large Language Model (LLM)-based Natural Language Generation (NLG) systems. However, LLMs are known to encode significant levels of unfair social biases. The modulation of these biases by RAG in NLG systems is not well understood. In this paper, we systematically study the relationship between the different components of a RAG system and the social biases presented in the text generated across three languages (i.e. English, Japanese and Chinese) and four social bias types (i.e. gender, race, age and religion). Specifically, using the Bias Question Answering (BBQ) benchmark datasets, we evaluate the social biases in RAG responses from document collections with varying levels of stereotypical biases, employing multiple LLMs used as generators. We find that the biases in document collections are often amplified in the generated responses, even when the generating LLM exhibits a low-level of bias. Our findings raise concerns about the use of RAG as a technique for injecting novel facts into NLG systems and call for careful evaluation of potential social biases in RAG applications before their real-world deployment.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.17611', 6)">Copy Link</button>

<div id="copy-message-6" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.17664">Towards Typologically Aware Rescoring to Mitigate Unfaithfulness in Lower-Resource Languages</a></h1>

<p><b>Authors:</b> Tsan Tsai Chan, Xin Tong, Thi Thu Uyen Hoang, Barbare Tepnadze, Wojciech Stempniak</p>

<p>Abstract: Multilingual large language models (LLMs) are known to more frequently generate non-faithful output in resource-constrained languages (Guerreiro et al., 2023 - arXiv:2303.16104), potentially because these typologically diverse languages are underrepresented in their training data. To mitigate unfaithfulness in such settings, we propose using computationally light auxiliary models to rescore the outputs of larger architectures. As proof of the feasibility of such an approach, we show that monolingual 4-layer BERT models pretrained from scratch on less than 700 MB of data without fine-tuning are able to identify faithful summaries with a mean accuracy of 88.33% in three genetically unrelated languages that differ in their morphological complexity - Vietnamese, Polish and Georgian. The same hyperparameter combination moreover generalises well to three other tasks, suggesting applications for rescoring beyond improving faithfulness. In order to inform typologically aware model selection, we also investigate how morphological complexity interacts with regularisation, model depth and training objectives, ultimately demonstrating that morphologically complex languages are more likely to benefit from dropout, while across languages downstream performance is enhanced most by shallow architectures as well as training using the standard BERT objectives.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.17664', 7)">Copy Link</button>

<div id="copy-message-7" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.17669">Towards Human Cognition: Visual Context Guides Syntactic Priming in Fusion-Encoded Models</a></h1>

<p><b>Authors:</b> Bushi Xiao, Michael Bennie, Jayetri Bardhan, Daisy Zhe Wang</p>

<p>Abstract: We introduced PRISMATIC, the first multimodal structural priming dataset, and proposed a reference-free evaluation metric that assesses priming effects without predefined target sentences. Using this metric, we constructed and tested models with different multimodal encoding architectures (dual encoder and fusion encoder) to investigate their structural preservation capabilities. Our findings show that models with both encoding methods demonstrate comparable syntactic priming effects. However, only fusion-encoded models exhibit robust positive correlations between priming effects and visual similarity, suggesting a cognitive process more aligned with human psycholinguistic patterns. This work provides new insights into evaluating and understanding how syntactic information is processed in multimodal language models.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.17669', 8)">Copy Link</button>

<div id="copy-message-8" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.17670">Semantics drives analogical change in Germanic strong verb paradigms: a phylogenetic study</a></h1>

<p><b>Authors:</b> Alexandru Craevschi, Sarah Babinski, Chundra Cathcart</p>

<p>Abstract: A large body of research on morphological paradigms makes the prediction that irregular morphological patterns of allomorphy are more likely to emerge and persist when they serve to mark important functional distinctions. More specifically, it has been observed that in some Germanic languages in which narrative past tense is expressed by the past participle, there is a greater affinity for stem allomorphy shared by preterite forms and past participles to the exclusion of present forms (the so-called ABB pattern), as it serves to enhance marking of the binary semantic opposition between present and past. Using data from 107 cognate verbs attested across 14 archaic and contemporary Germanic languages and a novel hierarchical phylogenetic model, we show that there is a greater long-term preference for this alternation pattern in situations where narrative past tense has been extended to the past participle, confirming this hypothesis. We further elucidate the mechanisms underlying this association, demonstrating that this association holds because verbs with the ABB pattern are more likely to preserve it in situations where it marks an important binary semantic opposition; however, there is less evidence that the ABB pattern is extended to verbs with different patterns under the same circumstances. These results bear on debate as to whether the distribution of irregularity we observe cross-linguistically is due primarily to (1) the preservation of irregular patterns or (2) an active drive toward irregularization in certain contexts, and are more in line with the first hypothesis.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.17670', 9)">Copy Link</button>

<div id="copy-message-9" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.17715">Bridging Information Gaps with Comprehensive Answers: Improving the Diversity and Informativeness of Follow-Up Questions</a></h1>

<p><b>Authors:</b> Zhe Liu, Taekyu Kang, Haoyu Wang, Seyed Hossein Alavi, Vered Shwartz</p>

<p>Abstract: Effective conversational systems are expected to dynamically generate contextual follow-up questions to elicit new information while maintaining the conversation flow. While humans excel at asking diverse and informative questions by intuitively assessing both obtained and missing information, existing models often fall short of human performance on this task. To mitigate this, we propose a method that generates diverse and informative questions based on targeting unanswered information using a hypothetical LLM-generated "comprehensive answer". Our method is applied to augment an existing follow-up questions dataset. The experimental results demonstrate that language models fine-tuned on the augmented datasets produce follow-up questions of significantly higher quality and diversity. This promising approach could be effectively adopted to future work to augment information-seeking dialogues for reducing ambiguities and improving the accuracy of LLM answers.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.17715', 10)">Copy Link</button>

<div id="copy-message-10" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.17717">Knowledge Distillation with Training Wheels</a></h1>

<p><b>Authors:</b> Guanlin Liu, Anand Ramachandran, Tanmay Gangwani, Yan Fu, Abhinav Sethy</p>

<p>Abstract: Knowledge distillation is used, in generative language modeling, to train a smaller student model using the help of a larger teacher model, resulting in improved capabilities for the student model. In this paper, we formulate a more general framework for knowledge distillation where the student learns from the teacher during training, and also learns to ask for the teacher's help at test-time following rules specifying test-time restrictions. Towards this, we first formulate knowledge distillation as an entropy-regularized value optimization problem. Adopting Path Consistency Learning to solve this, leads to a new knowledge distillation algorithm using on-policy and off-policy demonstrations. We extend this using constrained reinforcement learning to a framework that incorporates the use of the teacher model as a test-time reference, within constraints. In this situation, akin to a human learner, the model needs to learn not only the learning material, but also the relative difficulty of different sections to prioritize for seeking teacher help. We examine the efficacy of our method through experiments in translation and summarization tasks, observing trends in accuracy and teacher use, noting that our approach unlocks operating points not available to the popular Speculative Decoding approach.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.17717', 11)">Copy Link</button>

<div id="copy-message-11" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.17720">Spontaneous Giving and Calculated Greed in Language Models</a></h1>

<p><b>Authors:</b> Yuxuan Li, Hirokazu Shirado</p>

<p>Abstract: Large language models, when trained with reinforcement learning, demonstrate advanced problem-solving capabilities through reasoning techniques like chain of thoughts and reflection. However, it is unclear how these reasoning capabilities extend to social intelligence. In this study, we investigate how reasoning influences model outcomes in social dilemmas. First, we examine the effects of chain-of-thought and reflection techniques in a public goods game. We then extend our analysis to six economic games on cooperation and punishment, comparing off-the-shelf non-reasoning and reasoning models. We find that reasoning models reduce cooperation and norm enforcement, prioritizing individual rationality. Consequently, groups with more reasoning models exhibit less cooperation and lower gains through repeated interactions. These behaviors parallel human tendencies of "spontaneous giving and calculated greed." Our results suggest the need for AI architectures that incorporate social intelligence alongside reasoning capabilities to ensure that AI supports, rather than disrupts, human cooperative intuition.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.17720', 12)">Copy Link</button>

<div id="copy-message-12" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.17728">LLM Inference Acceleration via Efficient Operation Fusion</a></h1>

<p><b>Authors:</b> Mahsa Salmani, Ilya Soloveychik</p>

<p>Abstract: The rapid development of the Transformer-based Large Language Models (LLMs) in recent years has been closely linked to their ever-growing and already enormous sizes. Many LLMs contain hundreds of billions of parameters and require dedicated hardware resources for training and inference. One of the key challenges inherent to the Transformer architecture is the requirement to support numerous non-linear transformations that involves normalization. For instance, each decoder block typically contains at least one Softmax operation and two Layernorms. The computation of the corresponding normalization scaling factors becomes a major bottleneck as it requires spatial collective operations. In other words, when it comes to the computation of denominators for Softmax and Layernorm, all vector elements must be aggregated into a single location, requiring significant communication. These collective operations slow down inference on Transformers by approximately 20%, defeating the whole purpose of distributed in-memory compute. In this work, we propose an extremely efficient technique that can completely hide the overhead caused by such collective operations. Note that each Softmax and Layernorm operation is typically followed by a linear layer. Since non-linear and linear operations are performed on different hardware engines, they can be easily parallelized once the algebra allows such commutation. By leveraging the inherent properties of linear operations, we can defer the normalization of the preceding Softmax and Layernorm until after the linear layer is computed. Now we can compute the collective scaling factors concurrently with the matrix multiplication and completely hide the latency of the former behind the latter. Such parallelization preserves the numerical accuracy while significantly improving the hardware utilization and reducing the overall latency.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.17728', 13)">Copy Link</button>

<div id="copy-message-13" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.17775">FoREST: Frame of Reference Evaluation in Spatial Reasoning Tasks</a></h1>

<p><b>Authors:</b> Tanawan Premsri, Parisa Kordjamshidi</p>

<p>Abstract: Spatial reasoning is a fundamental aspect of human intelligence. One key concept in spatial cognition is the Frame of Reference (FoR), which identifies the perspective of spatial expressions. Despite its significance, FoR has received limited attention in AI models that need spatial intelligence. There is a lack of dedicated benchmarks and in-depth evaluation of large language models (LLMs) in this area. To address this issue, we introduce the Frame of Reference Evaluation in Spatial Reasoning Tasks (FoREST) benchmark, designed to assess FoR comprehension in LLMs. We evaluate LLMs on answering questions that require FoR comprehension and layout generation in text-to-image models using FoREST. Our results reveal a notable performance gap across different FoR classes in various LLMs, affecting their ability to generate accurate layouts for text-to-image generation. This highlights critical shortcomings in FoR comprehension. To improve FoR understanding, we propose Spatial-Guided prompting, which improves LLMs ability to extract essential spatial concepts. Our proposed method improves overall performance across spatial reasoning tasks.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.17775', 14)">Copy Link</button>

<div id="copy-message-14" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.17785">Exploring the Potential of Large Language Models for Estimating the Reading Comprehension Question Difficulty</a></h1>

<p><b>Authors:</b> Yoshee Jain, John Hollander, Amber He, Sunny Tang, Liang Zhang, John Sabatini</p>

<p>Abstract: Reading comprehension is a key for individual success, yet the assessment of question difficulty remains challenging due to the extensive human annotation and large-scale testing required by traditional methods such as linguistic analysis and Item Response Theory (IRT). While these robust approaches provide valuable insights, their scalability is limited. There is potential for Large Language Models (LLMs) to automate question difficulty estimation; however, this area remains underexplored. Our study investigates the effectiveness of LLMs, specifically OpenAI's GPT-4o and o1, in estimating the difficulty of reading comprehension questions using the Study Aid and Reading Assessment (SARA) dataset. We evaluated both the accuracy of the models in answering comprehension questions and their ability to classify difficulty levels as defined by IRT. The results indicate that, while the models yield difficulty estimates that align meaningfully with derived IRT parameters, there are notable differences in their sensitivity to extreme item characteristics. These findings suggest that LLMs can serve as the scalable method for automated difficulty assessment, particularly in dynamic interactions between learners and Adaptive Instructional Systems (AIS), bridging the gap between traditional psychometric techniques and modern AIS for reading comprehension and paving the way for more adaptive and personalized educational assessments.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.17785', 15)">Copy Link</button>

<div id="copy-message-15" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.17787">AIR: Complex Instruction Generation via Automatic Iterative Refinement</a></h1>

<p><b>Authors:</b> Wei Liu, Yancheng He, Hui Huang, Chengwei Hu, Jiaheng Liu, Shilong Li, Wenbo Su, Bo Zheng</p>

<p>Abstract: With the development of large language models, their ability to follow simple instructions has significantly improved. However, adhering to complex instructions remains a major challenge. Current approaches to generating complex instructions are often irrelevant to the current instruction requirements or suffer from limited scalability and diversity. Moreover, methods such as back-translation, while effective for simple instruction generation, fail to leverage the rich contents and structures in large web corpora. In this paper, we propose a novel automatic iterative refinement framework to generate complex instructions with constraints, which not only better reflects the requirements of real scenarios but also significantly enhances LLMs' ability to follow complex instructions. The AIR framework consists of two stages: (1)Generate an initial instruction from a document; (2)Iteratively refine instructions with LLM-as-judge guidance by comparing the model's output with the document to incorporate valuable constraints. Finally, we construct the AIR-10K dataset with 10K complex instructions and demonstrate that instructions generated with our approach significantly improve the model's ability to follow complex instructions, outperforming existing methods for instruction generation.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.17787', 16)">Copy Link</button>

<div id="copy-message-16" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.17797">Enhancing Human Evaluation in Machine Translation with Comparative Judgment</a></h1>

<p><b>Authors:</b> Yixiao Song, Parker Riley, Daniel Deutsch, Markus Freitag</p>

<p>Abstract: Human evaluation is crucial for assessing rapidly evolving language models but is influenced by annotator proficiency and task design. This study explores the integration of comparative judgment into human annotation for machine translation (MT) and evaluates three annotation setups-point-wise Multidimensional Quality Metrics (MQM), side-by-side (SxS) MQM, and its simplified version SxS relative ranking (RR). In MQM, annotators mark error spans with categories and severity levels. SxS MQM extends MQM to pairwise error annotation for two translations of the same input, while SxS RR focuses on selecting the better output without labeling errors.

Key findings are: (1) the SxS settings achieve higher inter-annotator agreement than MQM; (2) SxS MQM enhances inter-translation error marking consistency compared to MQM by, on average, 38.5% for explicitly compared MT systems and 19.5% for others; (3) all annotation settings return stable system rankings, with SxS RR offering a more efficient alternative to (SxS) MQM; (4) the SxS settings highlight subtle errors overlooked in MQM without altering absolute system evaluations.

To spur further research, we will release the triply annotated datasets comprising 377 ZhEn and 104 EnDe annotation examples.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.17797', 17)">Copy Link</button>

<div id="copy-message-17" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.17800">Your Language Model May Think Too Rigidly: Achieving Reasoning Consistency with Symmetry-Enhanced Training</a></h1>

<p><b>Authors:</b> Yihang Yao, Zhepeng Cen, Miao Li, William Han, Yuyou Zhang, Emerson Liu, Zuxin Liu, Chuang Gan, Ding Zhao</p>

<p>Abstract: Large Language Models (LLMs) have demonstrated strong reasoning capabilities across various tasks. However, even minor variations in query phrasing, despite preserving the underlying semantic meaning, can significantly affect their performance. To address this, we focus on enhancing LLMs' awareness of symmetry in query variations and propose syMmetry-ENhanceD (MEND) Data Augmentation, a data-centric approach that improves the model's ability to extract useful information from context. Unlike existing methods that emphasize reasoning chain augmentation, our approach improves model robustness at the knowledge extraction stage through query augmentations, enabling more data-efficient training and stronger generalization to Out-of-Distribution (OOD) settings. Extensive experiments on both logical and arithmetic reasoning tasks show that MEND enhances reasoning performance across diverse query variations, providing new insight into improving LLM robustness through structured dataset curation.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.17800', 18)">Copy Link</button>

<div id="copy-message-18" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.17810">URO-Bench: A Comprehensive Benchmark for End-to-End Spoken Dialogue Models</a></h1>

<p><b>Authors:</b> Ruiqi Yan, Xiquan Li, Wenxi Chen, Zhikang Niu, Chen Yang, Ziyang Ma, Kai Yu, Xie Chen</p>

<p>Abstract: In recent years, with advances in large language models (LLMs), end-to-end spoken dialogue models (SDMs) have made significant strides. Compared to text-based LLMs, the evaluation of SDMs needs to take speech-related aspects into account, such as paralinguistic information and speech quality. However, there is still a lack of comprehensive evaluations for SDMs in speech-to-speech (S2S) scenarios. To address this gap, we propose URO-Bench, an extensive benchmark for SDMs. Notably, URO-Bench is the first S2S benchmark that covers evaluations about multilingualism, multi-round dialogues, and paralinguistics. Our benchmark is divided into two difficulty levels: basic track and pro track, consisting of 16 and 20 datasets respectively, evaluating the model's abilities in Understanding, Reasoning, and Oral conversation. Evaluations on our proposed benchmark reveal that current open-source SDMs perform rather well in daily QA tasks, but lag behind their backbone LLMs in terms of instruction-following ability and also suffer from catastrophic forgetting. Their performance in advanced evaluations of paralinguistic information and audio understanding remains subpar, highlighting the need for further research in this direction. We hope that URO-Bench can effectively facilitate the development of spoken dialogue models by providing a multifaceted evaluation of existing models and helping to track progress in this area.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.17810', 19)">Copy Link</button>

<div id="copy-message-19" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.17812">Can Multimodal LLMs Perform Time Series Anomaly Detection?</a></h1>

<p><b>Authors:</b> Xiongxiao Xu, Haoran Wang, Yueqing Liang, Philip S. Yu, Yue Zhao, Kai Shu</p>

<p>Abstract: Large language models (LLMs) have been increasingly used in time series analysis. However, the potential of multimodal LLMs (MLLMs), particularly vision-language models, for time series remains largely under-explored. One natural way for humans to detect time series anomalies is through visualization and textual description. Motivated by this, we raise a critical and practical research question: Can multimodal LLMs perform time series anomaly detection? To answer this, we propose VisualTimeAnomaly benchmark to evaluate MLLMs in time series anomaly detection (TSAD). Our approach transforms time series numerical data into the image format and feed these images into various MLLMs, including proprietary models (GPT-4o and Gemini-1.5) and open-source models (LLaVA-NeXT and Qwen2-VL), each with one larger and one smaller variant. In total, VisualTimeAnomaly contains 12.4k time series images spanning 3 scenarios and 3 anomaly granularities with 9 anomaly types across 8 MLLMs. Starting with the univariate case (point- and range-wise anomalies), we extend our evaluation to more practical scenarios, including multivariate and irregular time series scenarios, and variate-wise anomalies. Our study reveals several key insights:

1) MLLMs detect range- and variate-wise anomalies more effectively than point-wise anomalies.

2) MLLMs are highly robust to irregular time series, even with 25% of the data missing.

3) Open-source MLLMs perform comparably to proprietary models in TSAD. While open-source MLLMs excel on univariate time series, proprietary MLLMs demonstrate superior effectiveness on multivariate time series.

To the best of our knowledge, this is the first work to comprehensively investigate MLLMs for TSAD, particularly for multivariate and irregular time series scenarios. We release our dataset and code at https://github.com/mllm-ts/VisualTimeAnomaly to support future research.</p>

<p>URLs: <a href="https://github.com/mllm-ts/VisualTimeAnomaly">https://github.com/mllm-ts/VisualTimeAnomaly</a></p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.17812, https://github.com/mllm-ts/VisualTimeAnomaly', 20)">Copy Link</button>

<div id="copy-message-20" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.17817">Predicting Through Generation: Why Generation Is Better for Prediction</a></h1>

<p><b>Authors:</b> Md Kowsher, Nusrat Jahan Prottasha, Prakash Bhat, Chun-Nam Yu, Mojtaba Soltanalian, Ivan Garibay, Ozlem Garibay, Chen Chen, Niloofar Yousefi</p>

<p>Abstract: This paper argues that generating output tokens is more effective than using pooled representations for prediction tasks because token-level generation retains more mutual information. Since LLMs are trained on massive text corpora using next-token prediction, generation aligns naturally with their learned behavior. Using the Data Processing Inequality (DPI), we provide both theoretical and empirical evidence supporting this claim. However, autoregressive models face two key challenges when used for prediction: (1) exposure bias, where the model sees ground truth tokens during training but relies on its own predictions during inference, leading to errors, and (2) format mismatch, where discrete tokens do not always align with the tasks required output structure. To address these challenges, we introduce PredGen(Predicting Through Generating), an end to end framework that (i) uses scheduled sampling to reduce exposure bias, and (ii) introduces a task adapter to convert the generated tokens into structured outputs. Additionally, we introduce Writer-Director Alignment Loss (WDAL), which ensures consistency between token generation and final task predictions, improving both text coherence and numerical accuracy. We evaluate PredGen on multiple classification and regression benchmarks. Our results show that PredGen consistently outperforms standard baselines, demonstrating its effectiveness in structured prediction tasks.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.17817', 21)">Copy Link</button>

<div id="copy-message-21" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.17839">Say Less, Mean More: Leveraging Pragmatics in Retrieval-Augmented Generation</a></h1>

<p><b>Authors:</b> Haris Riaz, Ellen Riloff, Mihai Surdeanu</p>

<p>Abstract: We propose a simple, unsupervised method that injects pragmatic principles in retrieval-augmented generation (RAG) frameworks such as Dense Passage Retrieval~\cite{karpukhin2020densepassageretrievalopendomain} to enhance the utility of retrieved contexts. Our approach first identifies which sentences in a pool of documents retrieved by RAG are most relevant to the question at hand, cover all the topics addressed in the input question and no more, and then highlights these sentences within their context, before they are provided to the LLM, without truncating or altering the context in any other way. We show that this simple idea brings consistent improvements in experiments on three question answering tasks (ARC-Challenge, PubHealth and PopQA) using five different LLMs. It notably enhances relative accuracy by up to 19.7\% on PubHealth and 10\% on ARC-Challenge compared to a conventional RAG system.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.17839', 22)">Copy Link</button>

<div id="copy-message-22" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.17848">LR${}^{2}$Bench: Evaluating Long-chain Reflective Reasoning Capabilities of Large Language Models via Constraint Satisfaction Problems</a></h1>

<p><b>Authors:</b> Jianghao Chen, Zhenlin Wei, Zhenjiang Ren, Ziyong Li, Jiajun Zhang</p>

<p>Abstract: Recent progress in o1-like models has significantly enhanced the reasoning abilities of Large Language Models (LLMs), empowering them to tackle increasingly complex tasks through reflection capabilities, such as making assumptions, backtracking, and self-refinement. However, effectively evaluating such reflection capabilities remains challenging due to the lack of appropriate benchmarks. To bridge this gap, we introduce LR${}^{2}$Bench, a novel benchmark designed to evaluate the Long-chain Reflective Reasoning capabilities of LLMs. LR${}^{2}$Bench comprises 850 samples across six Constraint Satisfaction Problems (CSPs) where reflective reasoning is crucial for deriving solutions that meet all given constraints. Each type of task focuses on distinct constraint patterns, such as knowledge-based, logical, and spatial constraints, providing a comprehensive evaluation of diverse problem-solving scenarios. We conduct extensive evaluation on both conventional models and o1-like models. Our experimental results reveal that even the most advanced reasoning-specific models, such as DeepSeek-R1 and OpenAI o1-preview, struggle with tasks in LR${}^{2}$Bench, achieving an average Exact Match score of only 20.0% and 23.6%, respectively. These findings underscore the significant room for improvement in the reflective reasoning capabilities of current LLMs. The leaderboard of our benchmark is available at https://huggingface.co/spaces/UltraRonin/LR2Bench</p>

<p>URLs: <a href="https://huggingface.co/spaces/UltraRonin/LR2Bench">https://huggingface.co/spaces/UltraRonin/LR2Bench</a></p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.17848, https://huggingface.co/spaces/UltraRonin/LR2Bench', 23)">Copy Link</button>

<div id="copy-message-23" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.17857">SYNTHEMPATHY: A Scalable Empathy Corpus Generated Using LLMs Without Any Crowdsourcing</a></h1>

<p><b>Authors:</b> Run Chen, Jun Shin, Julia Hirschberg</p>

<p>Abstract: Previous research has shown that humans are more receptive towards language models that that exhibit empathetic behavior. While empathy is essential for developing helpful dialogue agents, very few large corpora containing empathetic dialogues are available for fine-tune LLMs. The few existing corpora have largely relied on crowdsourcing to simulate empathetic conversations, a process that is expensive, time-consuming, and not scalable to larger datasets. We propose a data generation framework for developing SYNTHEMPATHY, a large corpus containing 105k empathetic responses to real-life situations compiled through LLM generation. A base Mistral 7B model fine-tuned on our SYNTHEMPATHY corpus exhibits an increase in the average empathy score.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.17857', 24)">Copy Link</button>

<div id="copy-message-24" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.17878">Towards Enhanced Immersion and Agency for LLM-based Interactive Drama</a></h1>

<p><b>Authors:</b> Hongqiu Wu, Weiqi Wu, Tianyang Xu, Jiameng Zhang, Hai Zhao</p>

<p>Abstract: LLM-based Interactive Drama is a novel AI-based dialogue scenario, where the user (i.e. the player) plays the role of a character in the story, has conversations with characters played by LLM agents, and experiences an unfolding story. This paper begins with understanding interactive drama from two aspects: Immersion, the player's feeling of being present in the story, and Agency, the player's ability to influence the story world. Both are crucial to creating an enjoyable interactive experience, while they have been underexplored in previous work. To enhance these two aspects, we first propose Playwriting-guided Generation, a novel method that helps LLMs craft dramatic stories with substantially improved structures and narrative quality. Additionally, we introduce Plot-based Reflection for LLM agents to refine their reactions to align with the player's intentions. Our evaluation relies on human judgment to assess the gains of our methods in terms of immersion and agency.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.17878', 25)">Copy Link</button>

<div id="copy-message-25" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.17888">RankCoT: Refining Knowledge for Retrieval-Augmented Generation through Ranking Chain-of-Thoughts</a></h1>

<p><b>Authors:</b> Mingyan Wu, Zhenghao Liu, Yukun Yan, Xinze Li, Shi Yu, Zheni Zeng, Yu Gu, Ge Yu</p>

<p>Abstract: Retrieval-Augmented Generation (RAG) enhances the performance of Large Language Models (LLMs) by incorporating external knowledge. However, LLMs still encounter challenges in effectively utilizing the knowledge from retrieved documents, often being misled by irrelevant or noisy information. To address this issue, we introduce RankCoT, a knowledge refinement method that incorporates reranking signals in generating CoT-based summarization for knowledge refinement based on given query and all retrieval documents. During training, RankCoT prompts the LLM to generate Chain-of-Thought (CoT) candidates based on the query and individual documents. It then fine-tunes the LLM to directly reproduce the best CoT from these candidate outputs based on all retrieved documents, which requires LLM to filter out irrelevant documents during generating CoT-style summarization. Additionally, RankCoT incorporates a self-reflection mechanism that further refines the CoT outputs, resulting in higher-quality training data. Our experiments demonstrate the effectiveness of RankCoT, showing its superior performance over other knowledge refinement models. Further analysis reveals that RankCoT can provide shorter but effective refinement results, enabling the generator to produce more accurate answers. All code and data are available at https://github.com/NEUIR/RankCoT.</p>

<p>URLs: <a href="https://github.com/NEUIR/RankCoT.">https://github.com/NEUIR/RankCoT.</a></p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.17888, https://github.com/NEUIR/RankCoT.', 26)">Copy Link</button>

<div id="copy-message-26" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.17899">Can Large Language Models Identify Implicit Suicidal Ideation? An Empirical Evaluation</a></h1>

<p><b>Authors:</b> Tong Li, Shu Yang, Junchao Wu, Jiyao Wei, Lijie Hu, Mengdi Li, Derek F. Wong, Joshua R. Oltmanns, Di Wang</p>

<p>Abstract: We present a comprehensive evaluation framework for assessing Large Language Models' (LLMs) capabilities in suicide prevention, focusing on two critical aspects: the Identification of Implicit Suicidal ideation (IIS) and the Provision of Appropriate Supportive responses (PAS). We introduce \ourdata, a novel dataset of 1,308 test cases built upon psychological frameworks including D/S-IAT and Negative Automatic Thinking, alongside real-world scenarios. Through extensive experiments with 8 widely used LLMs under different contextual settings, we find that current models struggle significantly with detecting implicit suicidal ideation and providing appropriate support, highlighting crucial limitations in applying LLMs to mental health contexts. Our findings underscore the need for more sophisticated approaches in developing and evaluating LLMs for sensitive psychological applications.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.17899', 27)">Copy Link</button>

<div id="copy-message-27" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.17910">Scaling LLM Pre-training with Vocabulary Curriculum</a></h1>

<p><b>Authors:</b> Fangyuan Yu</p>

<p>Abstract: Modern language models rely on static vocabularies, fixed before pretraining, in contrast to the adaptive vocabulary acquisition observed in human language learning. To bridge this gap, we introduce vocabulary curriculum learning, an approach that improves pretraining efficiency with log-linear scaling gains relative to vocabulary size. Our method alternates between entropy-guided vocabulary expansion and model optimization, enabling models to learn transferable representations across diverse tokenization granularities. This approach naturally gives rise to an optimal computation allocation pattern: longer tokens capture predictable content, while shorter tokens focus on more complex, harder-to-predict contexts. Experiments on small-scale GPT models demonstrate improved scaling efficiency, reinforcing the effectiveness of dynamic tokenization. We release our code to support further research and plan to extend our experiments to larger models and diverse domains.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.17910', 28)">Copy Link</button>

<div id="copy-message-28" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.17924">FACT-AUDIT: An Adaptive Multi-Agent Framework for Dynamic Fact-Checking Evaluation of Large Language Models</a></h1>

<p><b>Authors:</b> Hongzhan Lin, Yang Deng, Yuxuan Gu, Wenxuan Zhang, Jing Ma, See-Kiong Ng, Tat-Seng Chua</p>

<p>Abstract: Large Language Models (LLMs) have significantly advanced the fact-checking studies. However, existing automated fact-checking evaluation methods rely on static datasets and classification metrics, which fail to automatically evaluate the justification production and uncover the nuanced limitations of LLMs in fact-checking. In this work, we introduce FACT-AUDIT, an agent-driven framework that adaptively and dynamically assesses LLMs' fact-checking capabilities. Leveraging importance sampling principles and multi-agent collaboration, FACT-AUDIT generates adaptive and scalable datasets, performs iterative model-centric evaluations, and updates assessments based on model-specific responses. By incorporating justification production alongside verdict prediction, this framework provides a comprehensive and evolving audit of LLMs' factual reasoning capabilities, to investigate their trustworthiness. Extensive experiments demonstrate that FACT-AUDIT effectively differentiates among state-of-the-art LLMs, providing valuable insights into model strengths and limitations in model-centric fact-checking analysis.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.17924', 29)">Copy Link</button>

<div id="copy-message-29" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.17927">Advantage-Guided Distillation for Preference Alignment in Small Language Models</a></h1>

<p><b>Authors:</b> Shiping Gao, Fanqi Wan, Jiajian Guo, Xiaojun Quan, Qifan Wang</p>

<p>Abstract: Alignment techniques enable Large Language Models (LLMs) to generate outputs that align with human preferences and play a crucial role in their effectiveness. However, their impact often diminishes when applied to Small Language Models (SLMs), likely due to the limited capacity of these models. Instead of directly applying existing alignment techniques to SLMs, we propose to utilize a well-aligned teacher LLM to guide the alignment process for these models, thereby facilitating the transfer of the teacher's knowledge of human preferences to the student model. To achieve this, we first explore a straightforward approach, Dual-Constrained Knowledge Distillation (DCKD), that employs knowledge distillation with two KL-divergence constraints from the aligned teacher to the unaligned student. To further enhance the student's ability to distinguish between preferred and dispreferred responses, we then propose Advantage-Guided Distillation for Preference Alignment (ADPA), which leverages an advantage function from the aligned teacher to deliver more nuanced, distribution-level reward signals for the student's alignment. Our experimental results show that these two approaches appreciably improve the alignment of SLMs and narrow the performance gap with larger counterparts. Among them, ADPA demonstrates superior performance and achieves even greater effectiveness when integrated with DCKD. Our code is available at https://github.com/SLIT-AI/ADPA.</p>

<p>URLs: <a href="https://github.com/SLIT-AI/ADPA.">https://github.com/SLIT-AI/ADPA.</a></p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.17927, https://github.com/SLIT-AI/ADPA.', 30)">Copy Link</button>

<div id="copy-message-30" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.17943">CaseGen: A Benchmark for Multi-Stage Legal Case Documents Generation</a></h1>

<p><b>Authors:</b> Haitao Li, Jiaying Ye, Yiran Hu, Jia Chen, Qingyao Ai, Yueyue Wu, Junjie Chen, Yifan Chen, Cheng Luo, Quan Zhou, Yiqun Liu</p>

<p>Abstract: Legal case documents play a critical role in judicial proceedings. As the number of cases continues to rise, the reliance on manual drafting of legal case documents is facing increasing pressure and challenges. The development of large language models (LLMs) offers a promising solution for automating document generation. However, existing benchmarks fail to fully capture the complexities involved in drafting legal case documents in real-world scenarios. To address this gap, we introduce CaseGen, the benchmark for multi-stage legal case documents generation in the Chinese legal domain. CaseGen is based on 500 real case samples annotated by legal experts and covers seven essential case sections. It supports four key tasks: drafting defense statements, writing trial facts, composing legal reasoning, and generating judgment results. To the best of our knowledge, CaseGen is the first benchmark designed to evaluate LLMs in the context of legal case document generation. To ensure an accurate and comprehensive evaluation, we design the LLM-as-a-judge evaluation framework and validate its effectiveness through human annotations. We evaluate several widely used general-domain LLMs and legal-specific LLMs, highlighting their limitations in case document generation and pinpointing areas for potential improvement. This work marks a step toward a more effective framework for automating legal case documents drafting, paving the way for the reliable application of AI in the legal field. The dataset and code are publicly available at https://github.com/CSHaitao/CaseGen.</p>

<p>URLs: <a href="https://github.com/CSHaitao/CaseGen.">https://github.com/CSHaitao/CaseGen.</a></p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.17943, https://github.com/CSHaitao/CaseGen.', 31)">Copy Link</button>

<div id="copy-message-31" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.17945">Assessing Large Language Models in Agentic Multilingual National Bias</a></h1>

<p><b>Authors:</b> Qianying Liu, Katrina Qiyao Wang, Fei Cheng, Sadao Kurohashi</p>

<p>Abstract: Large Language Models have garnered significant attention for their capabilities in multilingual natural language processing, while studies on risks associated with cross biases are limited to immediate context preferences. Cross-language disparities in reasoning-based recommendations remain largely unexplored, with a lack of even descriptive analysis. This study is the first to address this gap. We test LLM's applicability and capability in providing personalized advice across three key scenarios: university applications, travel, and relocation. We investigate multilingual bias in state-of-the-art LLMs by analyzing their responses to decision-making tasks across multiple languages. We quantify bias in model-generated scores and assess the impact of demographic factors and reasoning strategies (e.g., Chain-of-Thought prompting) on bias patterns. Our findings reveal that local language bias is prevalent across different tasks, with GPT-4 and Sonnet reducing bias for English-speaking countries compared to GPT-3.5 but failing to achieve robust multilingual alignment, highlighting broader implications for multilingual AI agents and applications such as education.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.17945', 32)">Copy Link</button>

<div id="copy-message-32" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.17947">DeepSeek-R1 Outperforms Gemini 2.0 Pro, OpenAI o1, and o3-mini in Bilingual Complex Ophthalmology Reasoning</a></h1>

<p><b>Authors:</b> Pusheng Xu, Yue Wu, Kai Jin, Xiaolan Chen, Mingguang He, Danli Shi</p>

<p>Abstract: Purpose: To evaluate the accuracy and reasoning ability of DeepSeek-R1 and three other recently released large language models (LLMs) in bilingual complex ophthalmology cases. Methods: A total of 130 multiple-choice questions (MCQs) related to diagnosis (n = 39) and management (n = 91) were collected from the Chinese ophthalmology senior professional title examination and categorized into six topics. These MCQs were translated into English using DeepSeek-R1. The responses of DeepSeek-R1, Gemini 2.0 Pro, OpenAI o1 and o3-mini were generated under default configurations between February 15 and February 20, 2025. Accuracy was calculated as the proportion of correctly answered questions, with omissions and extra answers considered incorrect. Reasoning ability was evaluated through analyzing reasoning logic and the causes of reasoning error. Results: DeepSeek-R1 demonstrated the highest overall accuracy, achieving 0.862 in Chinese MCQs and 0.808 in English MCQs. Gemini 2.0 Pro, OpenAI o1, and OpenAI o3-mini attained accuracies of 0.715, 0.685, and 0.692 in Chinese MCQs (all P<0.001 compared with DeepSeek-R1), and 0.746 (P=0.115), 0.723 (P=0.027), and 0.577 (P<0.001) in English MCQs, respectively. DeepSeek-R1 achieved the highest accuracy across five topics in both Chinese and English MCQs. It also excelled in management questions conducted in Chinese (all P<0.05). Reasoning ability analysis showed that the four LLMs shared similar reasoning logic. Ignoring key positive history, ignoring key positive signs, misinterpretation medical data, and too aggressive were the most common causes of reasoning errors. Conclusion: DeepSeek-R1 demonstrated superior performance in bilingual complex ophthalmology reasoning tasks than three other state-of-the-art LLMs. While its clinical applicability remains challenging, it shows promise for supporting diagnosis and clinical decision-making.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.17947', 33)">Copy Link</button>

<div id="copy-message-33" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.17955">Language Models&#x27; Factuality Depends on the Language of Inquiry</a></h1>

<p><b>Authors:</b> Tushar Aggarwal, Kumar Tanmay, Ayush Agrawal, Kumar Ayush, Hamid Palangi, Paul Pu Liang</p>

<p>Abstract: Multilingual language models (LMs) are expected to recall factual knowledge consistently across languages, yet they often fail to transfer knowledge between languages even when they possess the correct information in one of the languages. For example, we find that an LM may correctly identify Rashed Al Shashai as being from Saudi Arabia when asked in Arabic, but consistently fails to do so when asked in English or Swahili. To systematically investigate this limitation, we introduce a benchmark of 10,000 country-related facts across 13 languages and propose three novel metrics: Factual Recall Score, Knowledge Transferability Score, and Cross-Lingual Factual Knowledge Transferability Score-to quantify factual recall and knowledge transferability in LMs across different languages. Our results reveal fundamental weaknesses in today's state-of-the-art LMs, particularly in cross-lingual generalization where models fail to transfer knowledge effectively across different languages, leading to inconsistent performance sensitive to the language used. Our findings emphasize the need for LMs to recognize language-specific factual reliability and leverage the most trustworthy information across languages. We release our benchmark and evaluation framework to drive future research in multilingual knowledge transfer.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.17955', 34)">Copy Link</button>

<div id="copy-message-34" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.17956">Towards Better Understanding of Program-of-Thought Reasoning in Cross-Lingual and Multilingual Environments</a></h1>

<p><b>Authors:</b> Patomporn Payoungkhamdee, Pume Tuchinda, Jinheon Baek, Samuel Cahyawijaya, Can Udomcharoenchaikit, Potsawee Manakul, Peerat Limkonchotiwat, Ekapol Chuangsuwanich, Sarana Nutanong</p>

<p>Abstract: Multi-step reasoning is essential for large language models (LLMs), yet multilingual performance remains challenging. While Chain-of-Thought (CoT) prompting improves reasoning, it struggles with non-English languages due to the entanglement of reasoning and execution. Program-of-Thought (PoT) prompting separates reasoning from execution, offering a promising alternative but shifting the challenge to generating programs from non-English questions. We propose a framework to evaluate PoT by separating multilingual reasoning from code execution to examine (i) the impact of fine-tuning on question-reasoning alignment and (ii) how reasoning quality affects answer correctness. Our findings demonstrate that PoT fine-tuning substantially enhances multilingual reasoning, outperforming CoT fine-tuned models. We further demonstrate a strong correlation between reasoning quality (measured through code quality) and answer accuracy, highlighting its potential as a test-time performance improvement heuristic.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.17956', 35)">Copy Link</button>

<div id="copy-message-35" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.17957">On Synthetic Data Strategies for Domain-Specific Generative Retrieval</a></h1>

<p><b>Authors:</b> Haoyang Wen, Jiang Guo, Yi Zhang, Jiarong Jiang, Zhiguo Wang</p>

<p>Abstract: This paper investigates synthetic data generation strategies in developing generative retrieval models for domain-specific corpora, thereby addressing the scalability challenges inherent in manually annotating in-domain queries. We study the data strategies for a two-stage training framework: in the first stage, which focuses on learning to decode document identifiers from queries, we investigate LLM-generated queries across multiple granularity (e.g. chunks, sentences) and domain-relevant search constraints that can better capture nuanced relevancy signals. In the second stage, which aims to refine document ranking through preference learning, we explore the strategies for mining hard negatives based on the initial model's predictions. Experiments on public datasets over diverse domains demonstrate the effectiveness of our synthetic data generation and hard negative sampling approach.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.17957', 36)">Copy Link</button>

<div id="copy-message-36" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.17987">MAGE: Multi-Head Attention Guided Embeddings for Low Resource Sentiment Classification</a></h1>

<p><b>Authors:</b> Varun Vashisht, Samar Singh, Mihir Konduskar, Jaskaran Singh Walia, Vukosi Marivate</p>

<p>Abstract: Due to the lack of quality data for low-resource Bantu languages, significant challenges are presented in text classification and other practical implementations. In this paper, we introduce an advanced model combining Language-Independent Data Augmentation (LiDA) with Multi-Head Attention based weighted embeddings to selectively enhance critical data points and improve text classification performance. This integration allows us to create robust data augmentation strategies that are effective across various linguistic contexts, ensuring that our model can handle the unique syntactic and semantic features of Bantu languages. This approach not only addresses the data scarcity issue but also sets a foundation for future research in low-resource language processing and classification tasks.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.17987', 37)">Copy Link</button>

<div id="copy-message-37" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.18001">Unveiling the Key Factors for Distilling Chain-of-Thought Reasoning</a></h1>

<p><b>Authors:</b> Xinghao Chen, Zhijing Sun, Wenjin Guo, Miaoran Zhang, Yanjun Chen, Yirong Sun, Hui Su, Yijie Pan, Dietrich Klakow, Wenjie Li, Xiaoyu Shen</p>

<p>Abstract: Large Language Models (LLMs) excel in reasoning tasks through Chain-of-Thought (CoT) prompting. However, CoT prompting greatly increases computational demands, which has prompted growing interest in distilling CoT capabilities into Small Language Models (SLMs). This study systematically examines the factors influencing CoT distillation, including the choice of granularity, format and teacher model. Through experiments involving four teacher models and seven student models across seven mathematical and commonsense reasoning datasets, we uncover three key findings: (1) Unlike LLMs, SLMs exhibit a non-monotonic relationship with granularity, with stronger models benefiting from finer-grained reasoning and weaker models performing better with simpler CoT supervision; (2) CoT format significantly impacts LLMs but has minimal effect on SLMs, likely due to their reliance on supervised fine-tuning rather than pretraining preferences; (3) Stronger teacher models do NOT always produce better student models, as diversity and complexity in CoT supervision can outweigh accuracy alone. These findings emphasize the need to tailor CoT strategies to specific student model, offering actionable insights for optimizing CoT distillation in SLMs. The code and datasets are available at https://github.com/EIT-NLP/Distilling-CoT-Reasoning.</p>

<p>URLs: <a href="https://github.com/EIT-NLP/Distilling-CoT-Reasoning.">https://github.com/EIT-NLP/Distilling-CoT-Reasoning.</a></p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.18001, https://github.com/EIT-NLP/Distilling-CoT-Reasoning.', 38)">Copy Link</button>

<div id="copy-message-38" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.18018">Verdict: A Library for Scaling Judge-Time Compute</a></h1>

<p><b>Authors:</b> Nimit Kalra, Leonard Tang</p>

<p>Abstract: The use of LLMs as automated judges ("LLM-as-a-judge") is now widespread, yet standard judges suffer from a multitude of reliability issues. To address these challenges, we introduce Verdict, an open-source library for scaling judge-time compute to enhance the accuracy, reliability, and interpretability of automated evaluators. Verdict leverages the composition of modular reasoning units -- such as verification, debate, and aggregation -- and increased inference-time compute to improve LLM judge quality. Across a variety of challenging tasks such as content moderation, fact-checking, and hallucination detection, Verdict judges achieve state-of-the-art (SOTA) or near-SOTA performance, surpassing orders-of-magnitude larger fine-tuned judges, prompted judges, and reasoning models. Ultimately, we hope Verdict serves as a useful framework for researchers and practitioners building scalable, interpretable, and reliable LLM-based evaluators.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.18018', 39)">Copy Link</button>

<div id="copy-message-39" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.18020">AfroXLMR-Comet: Multilingual Knowledge Distillation with Attention Matching for Low-Resource languages</a></h1>

<p><b>Authors:</b> Joshua Sakthivel Raju, Sanjay S, Jaskaran Singh Walia, Srinivas Raghav, Vukosi Marivate</p>

<p>Abstract: Language model compression through knowledge distillation has emerged as a promising approach for deploying large language models in resource-constrained environments. However, existing methods often struggle to maintain performance when distilling multilingual models, especially for low-resource languages. In this paper, we present a novel hybrid distillation approach that combines traditional knowledge distillation with a simplified attention matching mechanism, specifically designed for multilingual contexts. Our method introduces an extremely compact student model architecture, significantly smaller than conventional multilingual models. We evaluate our approach on five African languages: Kinyarwanda, Swahili, Hausa, Igbo, and Yoruba. The distilled student model; AfroXLMR-Comet successfully captures both the output distribution and internal attention patterns of a larger teacher model (AfroXLMR-Large) while reducing the model size by over 85%. Experimental results demonstrate that our hybrid approach achieves competitive performance compared to the teacher model, maintaining an accuracy within 85% of the original model's performance while requiring substantially fewer computational resources. Our work provides a practical framework for deploying efficient multilingual models in resource-constrained environments, particularly benefiting applications involving African languages.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.18020', 40)">Copy Link</button>

<div id="copy-message-40" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.18023">Detecting Knowledge Boundary of Vision Large Language Models by Sampling-Based Inference</a></h1>

<p><b>Authors:</b> Zhuo Chen, Xinyu Wang, Yong Jiang, Zhen Zhang, Xinyu Geng, Pengjun Xie, Fei Huang, Kewei Tu</p>

<p>Abstract: Despite the advancements made in Visual Large Language Models (VLLMs), like text Large Language Models (LLMs), they have limitations in addressing questions that require real-time information or are knowledge-intensive. Indiscriminately adopting Retrieval Augmented Generation (RAG) techniques is an effective yet expensive way to enable models to answer queries beyond their knowledge scopes. To mitigate the dependence on retrieval and simultaneously maintain, or even improve, the performance benefits provided by retrieval, we propose a method to detect the knowledge boundary of VLLMs, allowing for more efficient use of techniques like RAG. Specifically, we propose a method with two variants that fine-tunes a VLLM on an automatically constructed dataset for boundary identification. Experimental results on various types of Visual Question Answering datasets show that our method successfully depicts a VLLM's knowledge boundary based on which we are able to reduce indiscriminate retrieval while maintaining or improving the performance. In addition, we show that the knowledge boundary identified by our method for one VLLM can be used as a surrogate boundary for other VLLMs. Code will be released at https://github.com/Chord-Chen-30/VLLM-KnowledgeBoundary</p>

<p>URLs: <a href="https://github.com/Chord-Chen-30/VLLM-KnowledgeBoundary">https://github.com/Chord-Chen-30/VLLM-KnowledgeBoundary</a></p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.18023, https://github.com/Chord-Chen-30/VLLM-KnowledgeBoundary', 41)">Copy Link</button>

<div id="copy-message-41" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.18036">Harnessing Multiple Large Language Models: A Survey on LLM Ensemble</a></h1>

<p><b>Authors:</b> Zhijun Chen, Jingzheng Li, Pengpeng Chen, Zhuoran Li, Kai Sun, Yuankai Luo, Qianren Mao, Dingqi Yang, Hailong Sun, Philip S. Yu</p>

<p>Abstract: LLM Ensemble -- which involves the comprehensive use of multiple large language models (LLMs), each aimed at handling user queries during downstream inference, to benefit from their individual strengths -- has gained substantial attention recently. The widespread availability of LLMs, coupled with their varying strengths and out-of-the-box usability, has profoundly advanced the field of LLM Ensemble. This paper presents the first systematic review of recent developments in LLM Ensemble. First, we introduce our taxonomy of LLM Ensemble and discuss several related research problems. Then, we provide a more in-depth classification of the methods under the broad categories of "ensemble-before-inference, ensemble-during-inference, ensemble-after-inference", and review all relevant methods. Finally, we introduce related benchmarks and applications, summarize existing studies, and suggest several future research directions. A curated list of papers on LLM Ensemble is available at https://github.com/junchenzhi/Awesome-LLM-Ensemble.</p>

<p>URLs: <a href="https://github.com/junchenzhi/Awesome-LLM-Ensemble.">https://github.com/junchenzhi/Awesome-LLM-Ensemble.</a></p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.18036, https://github.com/junchenzhi/Awesome-LLM-Ensemble.', 42)">Copy Link</button>

<div id="copy-message-42" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.18050">Uncertainty-aware abstention in medical diagnosis based on medical texts</a></h1>

<p><b>Authors:</b> Artem Vazhentsev, Ivan Sviridov, Alvard Barseghyan, Gleb Kuzmin, Alexander Panchenko, Aleksandr Nesterov, Artem Shelmanov, Maxim Panov</p>

<p>Abstract: This study addresses the critical issue of reliability for AI-assisted medical diagnosis. We focus on the selection prediction approach that allows the diagnosis system to abstain from providing the decision if it is not confident in the diagnosis. Such selective prediction (or abstention) approaches are usually based on the modeling predictive uncertainty of machine learning models involved.

This study explores uncertainty quantification in machine learning models for medical text analysis, addressing diverse tasks across multiple datasets. We focus on binary mortality prediction from textual data in MIMIC-III, multi-label medical code prediction using ICD-10 codes from MIMIC-IV, and multi-class classification with a private outpatient visits dataset. Additionally, we analyze mental health datasets targeting depression and anxiety detection, utilizing various text-based sources, such as essays, social media posts, and clinical descriptions.

In addition to comparing uncertainty methods, we introduce HUQ-2, a new state-of-the-art method for enhancing reliability in selective prediction tasks. Our results provide a detailed comparison of uncertainty quantification methods. They demonstrate the effectiveness of HUQ-2 in capturing and evaluating uncertainty, paving the way for more reliable and interpretable applications in medical text analysis.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.18050', 43)">Copy Link</button>

<div id="copy-message-43" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.18080">Towards Thinking-Optimal Scaling of Test-Time Compute for LLM Reasoning</a></h1>

<p><b>Authors:</b> Wenkai Yang, Shuming Ma, Yankai Lin, Furu Wei</p>

<p>Abstract: Recent studies have shown that making a model spend more time thinking through longer Chain of Thoughts (CoTs) enables it to gain significant improvements in complex reasoning tasks. While current researches continue to explore the benefits of increasing test-time compute by extending the CoT lengths of Large Language Models (LLMs), we are concerned about a potential issue hidden behind the current pursuit of test-time scaling: Would excessively scaling the CoT length actually bring adverse effects to a model's reasoning performance? Our explorations on mathematical reasoning tasks reveal an unexpected finding that scaling with longer CoTs can indeed impair the reasoning performance of LLMs in certain domains. Moreover, we discover that there exists an optimal scaled length distribution that differs across different domains. Based on these insights, we propose a Thinking-Optimal Scaling strategy. Our method first uses a small set of seed data with varying response length distributions to teach the model to adopt different reasoning efforts for deep thinking. Then, the model selects its shortest correct response under different reasoning efforts on additional problems for self-improvement. Our self-improved models built upon Qwen2.5-32B-Instruct outperform other distillation-based 32B o1-like models across various math benchmarks, and achieve performance on par with QwQ-32B-Preview.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.18080', 44)">Copy Link</button>

<div id="copy-message-44" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.18108">Uncertainty Quantification in Retrieval Augmented Question Answering</a></h1>

<p><b>Authors:</b> Laura Perez-Beltrachini, Mirella Lapata</p>

<p>Abstract: Retrieval augmented Question Answering (QA) helps QA models overcome knowledge gaps by incorporating retrieved evidence, typically a set of passages, alongside the question at test time. Previous studies show that this approach improves QA performance and reduces hallucinations, without, however, assessing whether the retrieved passages are indeed useful at answering correctly. In this work, we propose to quantify the uncertainty of a QA model via estimating the utility of the passages it is provided with. We train a lightweight neural model to predict passage utility for a target QA model and show that while simple information theoretic metrics can predict answer correctness up to a certain extent, our approach efficiently approximates or outperforms more expensive sampling-based methods. Code and data are available at https://github.com/lauhaide/ragu.</p>

<p>URLs: <a href="https://github.com/lauhaide/ragu.">https://github.com/lauhaide/ragu.</a></p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.18108, https://github.com/lauhaide/ragu.', 45)">Copy Link</button>

<div id="copy-message-45" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.18139">LevelRAG: Enhancing Retrieval-Augmented Generation with Multi-hop Logic Planning over Rewriting Augmented Searchers</a></h1>

<p><b>Authors:</b> Zhuocheng Zhang, Yang Feng, Min Zhang</p>

<p>Abstract: Retrieval-Augmented Generation (RAG) is a crucial method for mitigating hallucinations in Large Language Models (LLMs) and integrating external knowledge into their responses. Existing RAG methods typically employ query rewriting to clarify the user intent and manage multi-hop logic, while using hybrid retrieval to expand search scope. However, the tight coupling of query rewriting to the dense retriever limits its compatibility with hybrid retrieval, impeding further RAG performance improvements. To address this challenge, we introduce a high-level searcher that decomposes complex queries into atomic queries, independent of any retriever-specific optimizations. Additionally, to harness the strengths of sparse retrievers for precise keyword retrieval, we have developed a new sparse searcher that employs Lucene syntax to enhance retrieval accuracy.Alongside web and dense searchers, these components seamlessly collaborate within our proposed method, \textbf{LevelRAG}. In LevelRAG, the high-level searcher orchestrates the retrieval logic, while the low-level searchers (sparse, web, and dense) refine the queries for optimal retrieval. This approach enhances both the completeness and accuracy of the retrieval process, overcoming challenges associated with current query rewriting techniques in hybrid retrieval scenarios. Empirical experiments conducted on five datasets, encompassing both single-hop and multi-hop question answering tasks, demonstrate the superior performance of LevelRAG compared to existing RAG methods. Notably, LevelRAG outperforms the state-of-the-art proprietary model, GPT4o, underscoring its effectiveness and potential impact on the RAG field.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.18139', 46)">Copy Link</button>

<div id="copy-message-46" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.18148">NusaAksara: A Multimodal and Multilingual Benchmark for Preserving Indonesian Indigenous Scripts</a></h1>

<p><b>Authors:</b> Muhammad Farid Adilazuarda, Musa Izzanardi Wijanarko, Lucky Susanto, Khumaisa Nur'aini, Derry Wijaya, Alham Fikri Aji</p>

<p>Abstract: Indonesia is rich in languages and scripts. However, most NLP progress has been made using romanized text. In this paper, we present NusaAksara, a novel public benchmark for Indonesian languages that includes their original scripts. Our benchmark covers both text and image modalities and encompasses diverse tasks such as image segmentation, OCR, transliteration, translation, and language identification. Our data is constructed by human experts through rigorous steps. NusaAksara covers 8 scripts across 7 languages, including low-resource languages not commonly seen in NLP benchmarks. Although unsupported by Unicode, the Lampung script is included in this dataset. We benchmark our data across several models, from LLMs and VLMs such as GPT-4o, Llama 3.2, and Aya 23 to task-specific systems such as PP-OCR and LangID, and show that most NLP technologies cannot handle Indonesia's local scripts, with many achieving near-zero performance.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.18148', 47)">Copy Link</button>

<div id="copy-message-47" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.18156">Can LLMs Explain Themselves Counterfactually?</a></h1>

<p><b>Authors:</b> Zahra Dehghanighobadi, Asja Fischer, Muhammad Bilal Zafar</p>

<p>Abstract: Explanations are an important tool for gaining insights into the behavior of ML models, calibrating user trust and ensuring regulatory compliance. Past few years have seen a flurry of post-hoc methods for generating model explanations, many of which involve computing model gradients or solving specially designed optimization problems. However, owing to the remarkable reasoning abilities of Large Language Model (LLMs), self-explanation, that is, prompting the model to explain its outputs has recently emerged as a new paradigm. In this work, we study a specific type of self-explanations, self-generated counterfactual explanations (SCEs). We design tests for measuring the efficacy of LLMs in generating SCEs. Analysis over various LLM families, model sizes, temperature settings, and datasets reveals that LLMs sometimes struggle to generate SCEs. Even when they do, their prediction often does not agree with their own counterfactual reasoning.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.18156', 48)">Copy Link</button>

<div id="copy-message-48" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.18168">SECURA: Sigmoid-Enhanced CUR Decomposition with Uninterrupted Retention and Low-Rank Adaptation in Large Language Models</a></h1>

<p><b>Authors:</b> Zhang Yuxuan, Li Ruizhe</p>

<p>Abstract: With the rapid development of large language models (LLMs), fully fine-tuning (FT) these models has become increasingly impractical due to the high computational demands. Additionally, FT can lead to catastrophic forgetting. As an alternative, Low-Rank Adaptation (LoRA) has been proposed, which fine-tunes only a small subset of parameters, achieving similar performance to FT while significantly reducing resource requirements. However, since LoRA inherits FT's design, the issue of catastrophic forgetting remains.

To address these challenges, we propose SECURA: Sigmoid-Enhanced CUR Decomposition LoRA, a novel parameter-efficient fine-tuning (PEFT) variant that mitigates catastrophic forgetting while improving fine-tuning performance. Our method introduces a new normalization technique, SigNorm, to enhance parameter retention and overall performance.

SECURA has been evaluated on a variety of tasks, including mathematical problem-solving (GSM8K), challenging question-answering (CNNDM), translation (NewsDE), and complex multiple-choice reasoning (LogiQA). Experimental results show that SECURA achieves an average fine-tuning improvement of 3.59% across four multiple-choice question (MCQ) tasks and a 2.51% improvement across five question-answering (QA) tasks on models such as Gemma2 2b, Qwen2 1.5b, Qwen 2 7b, Llama3 8b, and Llama3.1 8b, compared to DoRA. Moreover, SECURA demonstrates superior knowledge retention capabilities, maintaining more than 70% accuracy on basic LLM knowledge across 16 continual learning tests, outperforming Experience Replay (ER), Sequential Learning (SEQ), EWC, I-LoRA, and CUR-LoRA.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.18168', 49)">Copy Link</button>

<div id="copy-message-49" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.18179">Problem Solved? Information Extraction Design Space for Layout-Rich Documents using LLMs</a></h1>

<p><b>Authors:</b> Gaye Colakoglu, G\"urkan Solmaz, Jonathan F\"urst</p>

<p>Abstract: This paper defines and explores the design space for information extraction (IE) from layout-rich documents using large language models (LLMs). The three core challenges of layout-aware IE with LLMs are 1) data structuring, 2) model engagement, and 3) output refinement. Our study delves into the sub-problems within these core challenges, such as input representation, chunking, prompting, and selection of LLMs and multimodal models. It examines the outcomes of different design choices through a new layout-aware IE test suite, benchmarking against the state-of-art (SoA) model LayoutLMv3. The results show that the configuration from one-factor-at-a-time (OFAT) trial achieves near-optimal results with 14.1 points F1-score gain from the baseline model, while full factorial exploration yields only a slightly higher 15.1 points gain at around 36x greater token usage. We demonstrate that well-configured general-purpose LLMs can match the performance of specialized models, providing a cost-effective alternative. Our test-suite is freely available at https://github.com/gayecolakoglu/LayIE-LLM.</p>

<p>URLs: <a href="https://github.com/gayecolakoglu/LayIE-LLM.">https://github.com/gayecolakoglu/LayIE-LLM.</a></p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.18179, https://github.com/gayecolakoglu/LayIE-LLM.', 50)">Copy Link</button>

<div id="copy-message-50" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.18205">Grandes modelos de lenguaje: de la predicci\&#x27;on de palabras a la comprensi\&#x27;on?</a></h1>

<p><b>Authors:</b> Carlos G\'omez-Rodr\'iguez</p>

<p>Abstract: Large language models, such as the well-known ChatGPT, have brought about an unexpected revolution in the field of artificial intelligence. On the one hand, they have numerous practical applications and enormous potential still to be explored. On the other hand, they are also the subject of debate from scientific, philosophical, and social perspectives: there are doubts about the exact mechanisms of their functioning and their actual capacity for language comprehension, and their applications raise ethical dilemmas. In this chapter, we describe how this technology has been developed and the fundamentals of its operation, allowing us to better understand its capabilities and limitations and to introduce some of the main debates surrounding its development and use.

\--

Los grandes modelos de lenguaje, como el conocido ChatGPT, han supuesto una inesperada revoluci\'on en el \'ambito de la inteligencia artificial. Por un lado, cuentan con multitud de aplicaciones pr\'acticas y un enorme potencial todav\'ia por explorar. Por otro lado, son tambi\'en objeto de debate, tanto desde el punto de vista cient\'ifico y filos\'ofico como social: hay dudas sobre los mecanismos exactos de su funcionamiento y su capacidad real de comprensi\'on del lenguaje, y sus aplicaciones plantean dilemas \'eticos. En este cap\'itulo describimos c\'omo se ha llegado a esta tecnolog\'ia y los fundamentos de su funcionamiento, permiti\'endonos as\'i comprender mejor sus capacidades y limitaciones e introducir algunos de los principales debates que rodean su desarrollo y uso.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.18205', 51)">Copy Link</button>

<div id="copy-message-51" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.18209">LAG: LLM agents for Leaderboard Auto Generation on Demanding</a></h1>

<p><b>Authors:</b> Jian Wu, Jiayu Zhang, Dongyuan Li, Linyi Yang, Aoxiao Zhong, Renhe Jiang, Qingsong Wen, Yue Zhang</p>

<p>Abstract: This paper introduces Leaderboard Auto Generation (LAG), a novel and well-organized framework for automatic generation of leaderboards on a given research topic in rapidly evolving fields like Artificial Intelligence (AI). Faced with a large number of AI papers updated daily, it becomes difficult for researchers to track every paper's proposed methods, experimental results, and settings, prompting the need for efficient automatic leaderboard construction. While large language models (LLMs) offer promise in automating this process, challenges such as multi-document summarization, leaderboard generation, and experiment fair comparison still remain under exploration. LAG solves these challenges through a systematic approach that involves the paper collection, experiment results extraction and integration, leaderboard generation, and quality evaluation. Our contributions include a comprehensive solution to the leaderboard construction problem, a reliable evaluation method, and experimental results showing the high quality of leaderboards.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.18209', 52)">Copy Link</button>

<div id="copy-message-52" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.18215">Connecting Voices: LoReSpeech as a Low-Resource Speech Parallel Corpus</a></h1>

<p><b>Authors:</b> Samy Ouzerrout</p>

<p>Abstract: Aligned audio corpora are fundamental to NLP technologies such as ASR and speech translation, yet they remain scarce for underrepresented languages, hindering their technological integration. This paper introduces a methodology for constructing LoReSpeech, a low-resource speech-to-speech translation corpus. Our approach begins with LoReASR, a sub-corpus of short audios aligned with their transcriptions, created through a collaborative platform. Building on LoReASR, long-form audio recordings, such as biblical texts, are aligned using tools like the MFA. LoReSpeech delivers both intra- and inter-language alignments, enabling advancements in multilingual ASR systems, direct speech-to-speech translation models, and linguistic preservation efforts, while fostering digital inclusivity. This work is conducted within Tutlayt AI project (https://tutlayt.fr).</p>

<p>URLs: <a href="https://tutlayt.fr).">https://tutlayt.fr).</a></p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.18215, https://tutlayt.fr).', 53)">Copy Link</button>

<div id="copy-message-53" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.18228">Debt Collection Negotiations with Large Language Models: An Evaluation System and Optimizing Decision Making with Multi-Agent</a></h1>

<p><b>Authors:</b> Xiaofeng Wang, Zhixin Zhang, Jinguang Zheng, Yiming Ai, Rui Wang</p>

<p>Abstract: Debt collection negotiations (DCN) are vital for managing non-performing loans (NPLs) and reducing creditor losses. Traditional methods are labor-intensive, while large language models (LLMs) offer promising automation potential. However, prior systems lacked dynamic negotiation and real-time decision-making capabilities. This paper explores LLMs in automating DCN and proposes a novel evaluation framework with 13 metrics across 4 aspects. Our experiments reveal that LLMs tend to over-concede compared to human negotiators. To address this, we propose the Multi-Agent Debt Negotiation (MADeN) framework, incorporating planning and judging modules to improve decision rationality. We also apply post-training techniques, including DPO with rejection sampling, to optimize performance. Our studies provide valuable insights for practitioners and researchers seeking to enhance efficiency and outcomes in this domain.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.18228', 54)">Copy Link</button>

<div id="copy-message-54" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.18273">Beyond In-Distribution Success: Scaling Curves of CoT Granularity for Language Model Generalization</a></h1>

<p><b>Authors:</b> Ru Wang, Wei Huang, Selena Song, Haoyu Zhang, Yusuke Iwasawa, Yutaka Matsuo, Jiaxian Guo</p>

<p>Abstract: Generalization to novel compound tasks under distribution shift is important for deploying transformer-based language models (LMs). This work investigates Chain-of-Thought (CoT) reasoning as a means to enhance OOD generalization. Through controlled experiments across several compound tasks, we reveal three key insights: (1) While QA-trained models achieve near-perfect in-distribution accuracy, their OOD performance degrades catastrophically, even with 10000k+ training examples; (2) the granularity of CoT data strongly correlates with generalization performance; finer-grained CoT data leads to better generalization; (3) CoT exhibits remarkable sample efficiency, matching QA performance with much less (even 80%) data.

Theoretically, we demonstrate that compound tasks inherently permit shortcuts in Q-A data that misalign with true reasoning principles, while CoT forces internalization of valid dependency structures, and thus can achieve better generalization. Further, we show that transformer positional embeddings can amplify generalization by emphasizing subtask condition recurrence in long CoT sequences. Our combined theoretical and empirical analysis provides compelling evidence for CoT reasoning as a crucial training paradigm for enabling LM generalization under real-world distributional shifts for compound tasks.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.18273', 55)">Copy Link</button>

<div id="copy-message-55" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.18277">Self-Adjust Softmax</a></h1>

<p><b>Authors:</b> Chuanyang Zheng, Yihang Gao, Guoxuan Chen, Han Shi, Jing Xiong, Xiaozhe Ren, Chao Huang, Xin Jiang, Zhenguo Li, Yu Li</p>

<p>Abstract: The softmax function is crucial in Transformer attention, which normalizes each row of the attention scores with summation to one, achieving superior performances over other alternative functions. However, the softmax function can face a gradient vanishing issue when some elements of the attention scores approach extreme values, such as probabilities close to one or zero. In this paper, we propose Self-Adjust Softmax (SA-Softmax) to address this issue by modifying $softmax(x)$ to $x \cdot softmax(x)$ and its normalized variant $\frac{(x - min(x_{\min},0))}{max(0,x_{max})-min(x_{min},0)} \cdot softmax(x)$. We theoretically show that SA-Softmax provides enhanced gradient properties compared to the vanilla softmax function. Moreover, SA-Softmax Attention can be seamlessly integrated into existing Transformer models to their attention mechanisms with minor adjustments. We conducted experiments to evaluate the empirical performance of Transformer models using SA-Softmax compared to the vanilla softmax function. These experiments, involving models with up to 2.7 billion parameters, are conducted across diverse datasets, language tasks, and positional encoding methods.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.18277', 56)">Copy Link</button>

<div id="copy-message-56" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.18282">Better Aligned with Survey Respondents or Training Data? Unveiling Political Leanings of LLMs on U.S. Supreme Court Cases</a></h1>

<p><b>Authors:</b> Shanshan Xu, T. Y. S. S Santosh, Yanai Elazar, Quirin Vogel, Barbara Plank, Matthias Grabmair</p>

<p>Abstract: The increased adoption of Large Language Models (LLMs) and their potential to shape public opinion have sparked interest in assessing these models' political leanings. Building on previous research that compared LLMs and human opinions and observed political bias in system responses, we take a step further to investigate the underlying causes of such biases by empirically examining how the values and biases embedded in training corpora shape model outputs. Specifically, we propose a method to quantitatively evaluate political leanings embedded in the large pretraining corpora. Subsequently we investigate to whom are the LLMs' political leanings more aligned with, their pretrainig corpora or the surveyed human opinions. As a case study, we focus on probing the political leanings of LLMs in 32 U.S. Supreme Court cases, addressing contentious topics such as abortion and voting rights. Our findings reveal that LLMs strongly reflect the political leanings in their training data, and no strong correlation is observed with their alignment to human opinions as expressed in surveys. These results underscore the importance of responsible curation of training data and the need for robust evaluation metrics to ensure LLMs' alignment with human-centered values.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.18282', 57)">Copy Link</button>

<div id="copy-message-57" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.18285">Uncertainty Modeling in Multimodal Speech Analysis Across the Psychosis Spectrum</a></h1>

<p><b>Authors:</b> Morteza Rohanian, Roya M. H\"uppi, Farhad Nooralahzadeh, Noemi Dannecker, Yves Pauli, Werner Surbeck, Iris Sommer, Wolfram Hinzen, Nicolas Langer, Michael Krauthammer, Philipp Homan</p>

<p>Abstract: Capturing subtle speech disruptions across the psychosis spectrum is challenging because of the inherent variability in speech patterns. This variability reflects individual differences and the fluctuating nature of symptoms in both clinical and non-clinical populations. Accounting for uncertainty in speech data is essential for predicting symptom severity and improving diagnostic precision. Speech disruptions characteristic of psychosis appear across the spectrum, including in non-clinical individuals. We develop an uncertainty-aware model integrating acoustic and linguistic features to predict symptom severity and psychosis-related traits. Quantifying uncertainty in specific modalities allows the model to address speech variability, improving prediction accuracy. We analyzed speech data from 114 participants, including 32 individuals with early psychosis and 82 with low or high schizotypy, collected through structured interviews, semi-structured autobiographical tasks, and narrative-driven interactions in German. The model improved prediction accuracy, reducing RMSE and achieving an F1-score of 83% with ECE = 4.5e-2, showing robust performance across different interaction contexts. Uncertainty estimation improved model interpretability by identifying reliability differences in speech markers such as pitch variability, fluency disruptions, and spectral instability. The model dynamically adjusted to task structures, weighting acoustic features more in structured settings and linguistic features in unstructured contexts. This approach strengthens early detection, personalized assessment, and clinical decision-making in psychosis-spectrum research.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.18285', 58)">Copy Link</button>

<div id="copy-message-58" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.18292">How Vital is the Jurisprudential Relevance: Law Article Intervened Legal Case Retrieval and Matching</a></h1>

<p><b>Authors:</b> Nuo Xu, Pinghui Wang, Zi Liang, Junzhou Zhao, Xiaohong Guan</p>

<p>Abstract: Legal case retrieval (LCR) aims to automatically scour for comparable legal cases based on a given query, which is crucial for offering relevant precedents to support the judgment in intelligent legal systems. Due to similar goals, it is often associated with a similar case matching (LCM) task. To address them, a daunting challenge is assessing the uniquely defined legal-rational similarity within the judicial domain, which distinctly deviates from the semantic similarities in general text retrieval. Past works either tagged domain-specific factors or incorporated reference laws to capture legal-rational information. However, their heavy reliance on expert or unrealistic assumptions restricts their practical applicability in real-world scenarios. In this paper, we propose an end-to-end model named LCM-LAI to solve the above challenges. Through meticulous theoretical analysis, LCM-LAI employs a dependent multi-task learning framework to capture legal-rational information within legal cases by a law article prediction (LAP) sub-task, without any additional assumptions in inference. Besides, LCM-LAI proposes an article-aware attention mechanism to evaluate the legal-rational similarity between across-case sentences based on law distribution, which is more effective than conventional semantic similarity. Weperform a series of exhaustive experiments including two different tasks involving four real-world datasets. Results demonstrate that LCM-LAI achieves state-of-the-art performance.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.18292', 59)">Copy Link</button>

<div id="copy-message-59" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.18308">RefuteBench 2.0 -- Agentic Benchmark for Dynamic Evaluation of LLM Responses to Refutation Instruction</a></h1>

<p><b>Authors:</b> Jianhao Yan, Yun Luo, Yue Zhang</p>

<p>Abstract: In the multi-turn interaction schema, large language models (LLMs) can leverage user feedback to enhance the quality and relevance of their responses. However, evaluating an LLM's ability to incorporate user refutation feedback is crucial yet challenging. In this study, we introduce RefuteBench 2.0, which significantly extends the original RefuteBench by incorporating LLM agents as refuters and evaluators, which allows for flexible and comprehensive assessment.

We design both transient and persistent refutation instructions with different validity periods. Meta-evaluation shows that the LLM-based refuter could generate more human-like refutations and the evaluators could assign scores with high correlation with humans. Experimental results of various LLMs show that current models could effectively satisfy the refutation but fail to memorize the refutation information. Interestingly, we also observe that the performance of the initial task decreases as the refutations increase. Analysis of the attention scores further shows a potential weakness of current LLMs: they struggle to retain and correctly use previous information during long context dialogues. https://github.com/ElliottYan/RefuteBench-2.0</p>

<p>URLs: <a href="https://github.com/ElliottYan/RefuteBench-2.0">https://github.com/ElliottYan/RefuteBench-2.0</a></p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.18308, https://github.com/ElliottYan/RefuteBench-2.0', 60)">Copy Link</button>

<div id="copy-message-60" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.18313">Looking forward: Linguistic theory and methods</a></h1>

<p><b>Authors:</b> John Mansfield, Ethan Gotlieb Wilcox</p>

<p>Abstract: This chapter examines current developments in linguistic theory and methods, focusing on the increasing integration of computational, cognitive, and evolutionary perspectives. We highlight four major themes shaping contemporary linguistics: (1) the explicit testing of hypotheses about symbolic representation, such as efficiency, locality, and conceptual semantic grounding; (2) the impact of artificial neural networks on theoretical debates and linguistic analysis; (3) the importance of intersubjectivity in linguistic theory; and (4) the growth of evolutionary linguistics. By connecting linguistics with computer science, psychology, neuroscience, and biology, we provide a forward-looking perspective on the changing landscape of linguistic research.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.18313', 61)">Copy Link</button>

<div id="copy-message-61" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.18316">WiCkeD: A Simple Method to Make Multiple Choice Benchmarks More Challenging</a></h1>

<p><b>Authors:</b> Ahmed Elhady, Eneko Agirre, Mikel Artetxe</p>

<p>Abstract: We introduce WiCkeD, a simple method to increase the complexity of existing multiple-choice benchmarks by randomly replacing a choice with "None of the above", a method often used in educational tests. We show that WiCkeD can be automatically applied to any existing benchmark, making it more challenging. We apply WiCkeD to 6 popular benchmarks and use it to evaluate 18 open-weight LLMs. The performance of the models drops 12.1 points on average with respect to the original versions of the datasets. When using chain-of-thought on 3 MMLU datasets, the performance drop for the WiCkeD variant is similar to the one observed when using the LLMs directly, showing that WiCkeD is also challenging for models with enhanced reasoning abilities. WiCkeD also uncovers that some models are more sensitive to the extra reasoning required, providing additional information with respect to the original benchmarks. We relase our code and data at https://github.com/ahmedselhady/wicked-benchmarks.</p>

<p>URLs: <a href="https://github.com/ahmedselhady/wicked-benchmarks.">https://github.com/ahmedselhady/wicked-benchmarks.</a></p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.18316, https://github.com/ahmedselhady/wicked-benchmarks.', 62)">Copy Link</button>

<div id="copy-message-62" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.18318">Mapping of Subjective Accounts into Interpreted Clusters (MOSAIC): Topic Modelling and LLM applied to Stroboscopic Phenomenology</a></h1>

<p><b>Authors:</b> Romy Beaut\'e, David J. Schwartzman, Guillaume Dumas, Jennifer Crook, Fiona Macpherson, Adam B. Barrett, Anil K. Seth</p>

<p>Abstract: Stroboscopic light stimulation (SLS) on closed eyes typically induces simple visual hallucinations (VHs), characterised by vivid, geometric and colourful patterns. A dataset of 862 sentences, extracted from 422 open subjective reports, was recently compiled as part of the Dreamachine programme (Collective Act, 2022), an immersive multisensory experience that combines SLS and spatial sound in a collective setting. Although open reports extend the range of reportable phenomenology, their analysis presents significant challenges, particularly in systematically identifying patterns. To address this challenge, we implemented a data-driven approach leveraging Large Language Models and Topic Modelling to uncover and interpret latent experiential topics directly from the Dreamachine's text-based reports. Our analysis confirmed the presence of simple VHs typically documented in scientific studies of SLS, while also revealing experiences of altered states of consciousness and complex hallucinations. Building on these findings, our computational approach expands the systematic study of subjective experience by enabling data-driven analyses of open-ended phenomenological reports, capturing experiences not readily identified through standard questionnaires. By revealing rich and multifaceted aspects of experiences, our study broadens our understanding of stroboscopically-induced phenomena while highlighting the potential of Natural Language Processing and Large Language Models in the emerging field of computational (neuro)phenomenology. More generally, this approach provides a practically applicable methodology for uncovering subtle hidden patterns of subjective experience across diverse research domains.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.18318', 63)">Copy Link</button>

<div id="copy-message-63" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.18331">BottleHumor: Self-Informed Humor Explanation using the Information Bottleneck Principle</a></h1>

<p><b>Authors:</b> EunJeong Hwang, Peter West, Vered Shwartz</p>

<p>Abstract: Humor is prevalent in online communications and it often relies on more than one modality (e.g., cartoons and memes). Interpreting humor in multimodal settings requires drawing on diverse types of knowledge, including metaphorical, sociocultural, and commonsense knowledge. However, identifying the most useful knowledge remains an open question. We introduce \method{}, a method inspired by the information bottleneck principle that elicits relevant world knowledge from vision and language models which is iteratively refined for generating an explanation of the humor in an unsupervised manner. Our experiments on three datasets confirm the advantage of our method over a range of baselines. Our method can further be adapted in the future for additional tasks that can benefit from eliciting and conditioning on relevant world knowledge and open new research avenues in this direction.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.18331', 64)">Copy Link</button>

<div id="copy-message-64" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.18339">Correlating and Predicting Human Evaluations of Language Models from Natural Language Processing Benchmarks</a></h1>

<p><b>Authors:</b> Rylan Schaeffer, Punit Singh Koura, Binh Tang, Ranjan Subramanian, Aaditya K Singh, Todor Mihaylov, Prajjwal Bhargava, Lovish Madaan, Niladri S. Chatterji, Vedanuj Goswami, Sergey Edunov, Dieuwke Hupkes, Sanmi Koyejo, Sharan Narang</p>

<p>Abstract: The explosion of high-performing conversational language models (LMs) has spurred a shift from classic natural language processing (NLP) benchmarks to expensive, time-consuming and noisy human evaluations - yet the relationship between these two evaluation strategies remains hazy. In this paper, we conduct a large-scale study of four Chat Llama 2 models, comparing their performance on 160 standard NLP benchmarks (e.g., MMLU, ARC, BIG-Bench Hard) against extensive human preferences on more than 11k single-turn and 2k multi-turn dialogues from over 2k human annotators. Our findings are striking: most NLP benchmarks strongly correlate with human evaluations, suggesting that cheaper, automated metrics can serve as surprisingly reliable predictors of human preferences. Three human evaluations, such as adversarial dishonesty and safety, are anticorrelated with NLP benchmarks, while two are uncorrelated. Moreover, through overparameterized linear regressions, we show that NLP scores can accurately predict human evaluations across different model scales, offering a path to reduce costly human annotation without sacrificing rigor. Overall, our results affirm the continued value of classic benchmarks and illuminate how to harness them to anticipate real-world user satisfaction - pointing to how NLP benchmarks can be leveraged to meet evaluation needs of our new era of conversational AI.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.18339', 65)">Copy Link</button>

<div id="copy-message-65" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.18341">Moderation Matters:Measuring Conversational Moderation Impact in English as a Second Language Group Discussion</a></h1>

<p><b>Authors:</b> Rena Gao, Ming-Bin Chen, Lea Frermann, Jey Han Lau</p>

<p>Abstract: English as a Second Language (ESL) speakers often struggle to engage in group discussions due to language barriers. While moderators can facilitate participation, few studies assess conversational engagement and evaluate moderation effectiveness. To address this gap, we develop a dataset comprising 17 sessions from an online ESL conversation club, which includes both moderated and non-moderated discussions. We then introduce an approach that integrates automatic ESL dialogue assessment and a framework that categorizes moderation strategies. Our findings indicate that moderators help improve the flow of topics and start/end a conversation. Interestingly, we find active acknowledgement and encouragement to be the most effective moderation strategy, while excessive information and opinion sharing by moderators has a negative impact. Ultimately, our study paves the way for analyzing ESL group discussions and the role of moderators in non-native conversation settings.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.18341', 66)">Copy Link</button>

<div id="copy-message-66" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.18342">BRIDO: Bringing Democratic Order to Abstractive Summarization</a></h1>

<p><b>Authors:</b> Junhyun Lee, Harshith Goka, Hyeonmok Ko</p>

<p>Abstract: Hallucination refers to the inaccurate, irrelevant, and inconsistent text generated from large language models (LLMs). While the LLMs have shown great promise in a variety of tasks, the issue of hallucination still remains a major challenge for many practical uses. In this paper, we tackle the issue of hallucination in abstract text summarization by mitigating exposure bias. Existing models targeted for exposure bias mitigation, namely BRIO, aim for better summarization quality in the ROUGE score. We propose a model that uses a similar exposure bias mitigation strategy but with a goal that is aligned with less hallucination. We conjecture that among a group of candidate outputs, ones with hallucinations will comprise the minority of the whole group. That is, candidates with less similarity with others will have a higher chance of containing hallucinated content. Our method uses this aspect and utilizes contrastive learning, incentivizing candidates with high inter-candidate ROUGE scores. We performed experiments on the XSum and CNN/DM summarization datasets, and our method showed 6.25% and 3.82% improvement, respectively, on the consistency G-Eval score over BRIO.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.18342', 67)">Copy Link</button>

<div id="copy-message-67" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.18353">DBR: Divergence-Based Regularization for Debiasing Natural Language Understanding Models</a></h1>

<p><b>Authors:</b> Zihao Li, Ruixiang Tang, Lu Cheng, Shuaiqiang Wang, Dawei Yin, Mengnan Du</p>

<p>Abstract: Pre-trained language models (PLMs) have achieved impressive results on various natural language processing tasks. However, recent research has revealed that these models often rely on superficial features and shortcuts instead of developing a genuine understanding of language, especially for natural language understanding (NLU) tasks. Consequently, the models struggle to generalize to out-of-domain data. In this work, we propose Divergence Based Regularization (DBR) to mitigate this shortcut learning behavior. Our method measures the divergence between the output distributions for original examples and examples where shortcut tokens have been masked. This process prevents the model's predictions from being overly influenced by shortcut features or biases. We evaluate our model on three NLU tasks and find that it improves out-of-domain performance with little loss of in-domain accuracy. Our results demonstrate that reducing the reliance on shortcuts and superficial features can enhance the generalization ability of large pre-trained language models.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.18353', 68)">Copy Link</button>

<div id="copy-message-68" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.18389">Monte Carlo Temperature: a robust sampling strategy for LLM&#x27;s uncertainty quantification methods</a></h1>

<p><b>Authors:</b> Nicola Cecere, Andrea Bacciu, Ignacio Fern\'andez Tob\'ias, Amin Mantrach</p>

<p>Abstract: Uncertainty quantification (UQ) in Large Language Models (LLMs) is essential for their safe and reliable deployment, particularly in critical applications where incorrect outputs can have serious consequences. Current UQ methods typically rely on querying the model multiple times using non-zero temperature sampling to generate diverse outputs for uncertainty estimation. However, the impact of selecting a given temperature parameter is understudied, and our analysis reveals that temperature plays a fundamental role in the quality of uncertainty estimates. The conventional approach of identifying optimal temperature values requires expensive hyperparameter optimization (HPO) that must be repeated for each new model-dataset combination. We propose Monte Carlo Temperature (MCT), a robust sampling strategy that eliminates the need for temperature calibration. Our analysis reveals that: 1) MCT provides more robust uncertainty estimates across a wide range of temperatures, 2) MCT improves the performance of UQ methods by replacing fixed-temperature strategies that do not rely on HPO, and 3) MCT achieves statistical parity with oracle temperatures, which represent the ideal outcome of a well-tuned but computationally expensive HPO process. These findings demonstrate that effective UQ can be achieved without the computational burden of temperature parameter calibration.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.18389', 69)">Copy Link</button>

<div id="copy-message-69" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.18397">KiRAG: Knowledge-Driven Iterative Retriever for Enhancing Retrieval-Augmented Generation</a></h1>

<p><b>Authors:</b> Jinyuan Fang, Zaiqiao Meng, Craig Macdonald</p>

<p>Abstract: Iterative retrieval-augmented generation (iRAG) models offer an effective approach for multi-hop question answering (QA). However, their retrieval process faces two key challenges: (1) it can be disrupted by irrelevant documents or factually inaccurate chain-of-thoughts; (2) their retrievers are not designed to dynamically adapt to the evolving information needs in multi-step reasoning, making it difficult to identify and retrieve the missing information required at each iterative step. Therefore, we propose KiRAG, which uses a knowledge-driven iterative retriever model to enhance the retrieval process of iRAG. Specifically, KiRAG decomposes documents into knowledge triples and performs iterative retrieval with these triples to enable a factually reliable retrieval process. Moreover, KiRAG integrates reasoning into the retrieval process to dynamically identify and retrieve knowledge that bridges information gaps, effectively adapting to the evolving information needs. Empirical results show that KiRAG significantly outperforms existing iRAG models, with an average improvement of 9.40% in R@3 and 5.14% in F1 on multi-hop QA.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.18397', 70)">Copy Link</button>

<div id="copy-message-70" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.18407">AgentRM: Enhancing Agent Generalization with Reward Modeling</a></h1>

<p><b>Authors:</b> Yu Xia, Jingru Fan, Weize Chen, Siyu Yan, Xin Cong, Zhong Zhang, Yaxi Lu, Yankai Lin, Zhiyuan Liu, Maosong Sun</p>

<p>Abstract: Existing LLM-based agents have achieved strong performance on held-in tasks, but their generalizability to unseen tasks remains poor. Hence, some recent work focus on fine-tuning the policy model with more diverse tasks to improve the generalizability. In this work, we find that finetuning a reward model to guide the policy model is more robust than directly finetuning the policy model. Based on this finding, we propose AgentRM, a generalizable reward model, to guide the policy model for effective test-time search. We comprehensively investigate three approaches to construct the reward model, including explicit reward modeling, implicit reward modeling and LLM-as-a-judge. We then use AgentRM to guide the answer generation with Best-of-N sampling and step-level beam search. On four types of nine agent tasks, AgentRM enhances the base policy model by $8.8$ points on average, surpassing the top general agent by $4.0$. Moreover, it demonstrates weak-to-strong generalization, yielding greater improvement of $12.6$ on LLaMA-3-70B policy model. As for the specializability, AgentRM can also boost a finetuned policy model and outperform the top specialized agent by $11.4$ on three held-in tasks. Further analysis verifies its effectiveness in test-time scaling. Codes will be released to facilitate the research in this area.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.18407', 71)">Copy Link</button>

<div id="copy-message-71" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.18414">GLEAN: Generalized Category Discovery with Diverse and Quality-Enhanced LLM Feedback</a></h1>

<p><b>Authors:</b> Henry Peng Zou, Siffi Singh, Yi Nian, Jianfeng He, Jason Cai, Saab Mansour, Hang Su</p>

<p>Abstract: Generalized Category Discovery (GCD) is a practical and challenging open-world task that aims to recognize both known and novel categories in unlabeled data using limited labeled data from known categories. Due to the lack of supervision, previous GCD methods face significant challenges, such as difficulty in rectifying errors for confusing instances, and inability to effectively uncover and leverage the semantic meanings of discovered clusters. Therefore, additional annotations are usually required for real-world applicability. However, human annotation is extremely costly and inefficient. To address these issues, we propose GLEAN, a unified framework for generalized category discovery that actively learns from diverse and quality-enhanced LLM feedback. Our approach leverages three different types of LLM feedback to: (1) improve instance-level contrastive features, (2) generate category descriptions, and (3) align uncertain instances with LLM-selected category descriptions. Extensive experiments demonstrate the superior performance of \MethodName over state-of-the-art models across diverse datasets, metrics, and supervision settings. Our code is available at https://github.com/amazon-science/Glean.</p>

<p>URLs: <a href="https://github.com/amazon-science/Glean.">https://github.com/amazon-science/Glean.</a></p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.18414, https://github.com/amazon-science/Glean.', 72)">Copy Link</button>

<div id="copy-message-72" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.18424">Compressing Language Models for Specialized Domains</a></h1>

<p><b>Authors:</b> Miles Williams, George Chrysostomou, Vitor Jeronymo, Nikolaos Aletras</p>

<p>Abstract: Compression techniques such as pruning and quantization offer a solution for more efficient deployment of language models (LMs), albeit with small performance drops in benchmark performance. However, general-purpose LM compression methods can negatively affect performance in specialized domains (e.g. biomedical or legal). Recent work has sought to address this, yet requires computationally expensive full-parameter fine-tuning. To this end, we propose cross-calibration, a novel training-free approach for improving the domain performance of compressed LMs. Our approach effectively leverages Hessian-based sensitivity to identify weights that are influential for both in-domain and general performance. Through extensive experimentation, we demonstrate that cross-calibration substantially outperforms existing approaches on domain-specific tasks, without compromising general performance. Notably, these gains come without additional computational overhead, displaying remarkable potential towards extracting domain-specialized compressed models from general-purpose LMs.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.18424', 73)">Copy Link</button>

<div id="copy-message-73" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.18431">TextGames: Learning to Self-Play Text-Based Puzzle Games via Language Model Reasoning</a></h1>

<p><b>Authors:</b> Frederikus Hudi, Genta Indra Winata, Ruochen Zhang, Alham Fikri Aji</p>

<p>Abstract: Reasoning is a fundamental capability of large language models (LLMs), enabling them to comprehend, analyze, and solve complex problems. In this paper, we introduce TextGames, an innovative benchmark specifically crafted to assess LLMs through demanding text-based games that require advanced skills in pattern recognition, spatial awareness, arithmetic, and logical reasoning. Our analysis probes LLMs' performance in both single-turn and multi-turn reasoning, and their abilities in leveraging feedback to correct subsequent answers through self-reflection. Our findings reveal that, although LLMs exhibit proficiency in addressing most easy and medium-level problems, they face significant challenges with more difficult tasks. In contrast, humans are capable of solving all tasks when given sufficient time. Moreover, we observe that LLMs show improved performance in multi-turn predictions through self-reflection, yet they still struggle with sequencing, counting, and following complex rules consistently. Additionally, models optimized for reasoning outperform pre-trained LLMs that prioritize instruction following, highlighting the crucial role of reasoning skills in addressing highly complex problems.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.18431', 74)">Copy Link</button>

<div id="copy-message-74" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.18434">Exploring Gender Disparities in Automatic Speech Recognition Technology</a></h1>

<p><b>Authors:</b> Hend ElGhazaly, Bahman Mirheidari, Nafise Sadat Moosavi, Heidi Christensen</p>

<p>Abstract: This study investigates factors influencing Automatic Speech Recognition (ASR) systems' fairness and performance across genders, beyond the conventional examination of demographics. Using the LibriSpeech dataset and the Whisper small model, we analyze how performance varies across different gender representations in training data. Our findings suggest a complex interplay between the gender ratio in training data and ASR performance. Optimal fairness occurs at specific gender distributions rather than a simple 50-50 split. Furthermore, our findings suggest that factors like pitch variability can significantly affect ASR accuracy. This research contributes to a deeper understanding of biases in ASR systems, highlighting the importance of carefully curated training data in mitigating gender bias.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.18434', 75)">Copy Link</button>

<div id="copy-message-75" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.18435">Reversal Blessing: Thinking Backward May Outpace Thinking Forward in Multi-choice Questions</a></h1>

<p><b>Authors:</b> Yizhe Zhang, Richard Bai, Zijin Gu, Ruixiang Zhang, Jiatao Gu, Emmanuel Abbe, Samy Bengio, Navdeep Jaitly</p>

<p>Abstract: Language models usually use left-to-right (L2R) autoregressive factorization. However, L2R factorization may not always be the best inductive bias. Therefore, we investigate whether alternative factorizations of the text distribution could be beneficial in some tasks. We investigate right-to-left (R2L) training as a compelling alternative, focusing on multiple-choice questions (MCQs) as a test bed for knowledge extraction and reasoning. Through extensive experiments across various model sizes (2B-8B parameters) and training datasets, we find that R2L models can significantly outperform L2R models on several MCQ benchmarks, including logical reasoning, commonsense understanding, and truthfulness assessment tasks. Our analysis reveals that this performance difference may be fundamentally linked to multiple factors including calibration, computability and directional conditional entropy. We ablate the impact of these factors through controlled simulation studies using arithmetic tasks, where the impacting factors can be better disentangled. Our work demonstrates that exploring alternative factorizations of the text distribution can lead to improvements in LLM capabilities and provides theoretical insights into optimal factorization towards approximating human language distribution, and when each reasoning order might be more advantageous.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.18435', 76)">Copy Link</button>

<div id="copy-message-76" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.18443">olmOCR: Unlocking Trillions of Tokens in PDFs with Vision Language Models</a></h1>

<p><b>Authors:</b> Jake Poznanski, Jon Borchardt, Jason Dunkelberger, Regan Huff, Daniel Lin, Aman Rangapur, Christopher Wilhelm, Kyle Lo, Luca Soldaini</p>

<p>Abstract: PDF documents have the potential to provide trillions of novel, high-quality tokens for training language models. However, these documents come in a diversity of types with differing formats and visual layouts that pose a challenge when attempting to extract and faithfully represent the underlying content for language model use. We present olmOCR, an open-source Python toolkit for processing PDFs into clean, linearized plain text in natural reading order while preserving structured content like sections, tables, lists, equations, and more. Our toolkit runs a fine-tuned 7B vision language model (VLM) trained on a sample of 260,000 pages from over 100,000 crawled PDFs with diverse properties, including graphics, handwritten text and poor quality scans. olmOCR is optimized for large-scale batch processing, able to scale flexibly to different hardware setups and convert a million PDF pages for only $190 USD. We release all components of olmOCR including VLM weights, data and training code, as well as inference code built on serving frameworks including vLLM and SGLang.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.18443', 77)">Copy Link</button>

<div id="copy-message-77" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.18448">Disambiguate First Parse Later: Generating Interpretations for Ambiguity Resolution in Semantic Parsing</a></h1>

<p><b>Authors:</b> Irina Saparina, Mirella Lapata</p>

<p>Abstract: Handling ambiguity and underspecification is an important challenge in natural language interfaces, particularly for tasks like text-to-SQL semantic parsing. We propose a modular approach that resolves ambiguity using natural language interpretations before mapping these to logical forms (e.g., SQL queries). Although LLMs excel at parsing unambiguous utterances, they show strong biases for ambiguous ones, typically predicting only preferred interpretations. We constructively exploit this bias to generate an initial set of preferred disambiguations and then apply a specialized infilling model to identify and generate missing interpretations. To train the infilling model, we introduce an annotation method that uses SQL execution to validate different meanings. Our approach improves interpretation coverage and generalizes across datasets with different annotation styles, database structures, and ambiguity types.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.18448', 78)">Copy Link</button>

<div id="copy-message-78" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.18452">FRIDA to the Rescue! Analyzing Synthetic Data Effectiveness in Object-Based Common Sense Reasoning for Disaster Response</a></h1>

<p><b>Authors:</b> Mollie Shichman, Claire Bonial, Austin Blodgett, Taylor Hudson, Francis Ferraro, Rachel Rudinger</p>

<p>Abstract: Large Language Models (LLMs) have the potential for substantial common sense reasoning. However, these capabilities are often emergent in larger models. This means smaller models that can be run locally are less helpful and capable with respect to certain reasoning tasks. To meet our problem space requirements, we fine-tune smaller LLMs to disaster domains, as these domains involve complex and low-frequency physical common sense knowledge. We introduce a pipeline to create Field Ready Instruction Decoding Agent (FRIDA) models, where domain experts and linguists combine their knowledge to make high-quality seed data that is used to generate synthetic data for fine-tuning. We create a set of 130 seed instructions for synthetic generation, a synthetic dataset of 25000 instructions, and 119 evaluation instructions relating to both general and earthquake-specific object affordances. We fine-tune several LLaMa and Mistral instruction-tuned models and find that FRIDA models outperform their base models at a variety of sizes. We then run an ablation study to understand which kinds of synthetic data most affect performance and find that training physical state and object function common sense knowledge alone improves over FRIDA models trained on all data. We conclude that the FRIDA pipeline is capable of instilling general common sense, but needs to be augmented with information retrieval for specific domain knowledge.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.18452', 79)">Copy Link</button>

<div id="copy-message-79" class="copy-message"></div>

</div>

<div class="article">

<h1> new <a href="https://arxiv.org/abs/2502.18460">DRAMA: Diverse Augmentation from Large Language Models to Smaller Dense Retrievers</a></h1>

<p><b>Authors:</b> Xueguang Ma, Xi Victoria Lin, Barlas Oguz, Jimmy Lin, Wen-tau Yih, Xilun Chen</p>

<p>Abstract: Large language models (LLMs) have demonstrated strong effectiveness and robustness while fine-tuned as dense retrievers. However, their large parameter size brings significant inference time computational challenges, including high encoding costs for large-scale corpora and increased query latency, limiting their practical deployment. While smaller retrievers offer better efficiency, they often fail to generalize effectively with limited supervised fine-tuning data. In this work, we introduce DRAMA, a training framework that leverages LLMs to train smaller generalizable dense retrievers. In particular, we adopt pruned LLMs as the backbone and train on diverse LLM-augmented data in a single-stage contrastive learning setup. Experiments show that DRAMA offers better multilingual and long-context capabilities than traditional encoder-based retrievers, and achieves strong performance across multiple tasks and languages. These highlight the potential of connecting the training of smaller retrievers with the growing advancements in LLMs, bridging the gap between efficiency and generalization.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.18460', 80)">Copy Link</button>

<div id="copy-message-80" class="copy-message"></div>

</div>

<div class="article">

<h1> cross <a href="https://arxiv.org/abs/2311.16831">Polarized Online Discourse on Abortion: Frames and Hostile Expressions among Liberals and Conservatives</a></h1>

<p><b>Authors:</b> Ashwin Rao, Rong-Ching Chang, Qiankun Zhong, Kristina Lerman, Magdalena Wojcieszak</p>

<p>Abstract: Abortion has been one of the most divisive issues in the United States. Yet, missing is comprehensive longitudinal evidence on how political divides on abortion are reflected in public discourse over time, on a national scale, and in response to key events before and after the overturn of Roe v Wade. We analyze a corpus of over 3.5M tweets related to abortion over the span of one year (January 2022 to January 2023) from over 1.1M users. We estimate users' ideology and rely on state-of-the-art transformer-based classifiers to identify expressions of hostility and extract five prominent frames surrounding abortion. We use those data to examine (a) how prevalent were expressions of hostility (i.e., anger, toxic speech, insults, obscenities, and hate speech), (b) what frames liberals and conservatives used to articulate their positions on abortion, and (c) the prevalence of hostile expressions in liberals and conservative discussions of these frames. We show that liberals and conservatives largely mirrored each other's use of hostile expressions: as liberals used more hostile rhetoric, so did conservatives, especially in response to key events. In addition, the two groups used distinct frames and discussed them in vastly distinct contexts, suggesting that liberals and conservatives have differing perspectives on abortion. Lastly, frames favored by one side provoked hostile reactions from the other: liberals use more hostile expressions when addressing religion, fetal personhood, and exceptions to abortion bans, whereas conservatives use more hostile language when addressing bodily autonomy and women's health. This signals disrespect and derogation, which may further preclude understanding and exacerbate polarization.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2311.16831', 81)">Copy Link</button>

<div id="copy-message-81" class="copy-message"></div>

</div>

<div class="article">

<h1> cross <a href="https://arxiv.org/abs/2502.17465">Bridging Brain Signals and Language: A Deep Learning Approach to EEG-to-Text Decoding</a></h1>

<p><b>Authors:</b> Mostafa El Gedawy, Omnia Nabil, Omar Mamdouh, Mahmoud Nady, Nour Alhuda Adel, Ahmed Fares</p>

<p>Abstract: Brain activity translation into human language delivers the capability to revolutionize machine-human interaction while providing communication support to people with speech disability. Electronic decoding reaches a certain level of achievement yet current EEG-to-text decoding methods fail to reach open vocabularies and depth of meaning and individual brain-specific variables. We introduce a special framework which changes conventional closed-vocabulary EEG-to-text decoding approaches by integrating subject-specific learning models with natural language processing methods to resolve detection obstacles. This method applies a deep representation learning approach to extract important EEG features which allow training of neural networks to create elaborate sentences that extend beyond original data content. The ZuCo dataset analysis demonstrates that research findings achieve higher BLEU, ROUGE and BERTScore performance when compared to current methods. The research proves how this framework functions as an effective approach to generate meaningful and correct texts while understanding individual brain variations. The proposed research aims to create a connection between open-vocabulary Text generation systems and human brain signal interpretation for developing efficacious brain-to-text systems. The research produces interdisciplinary effects through innovative assistive technology development and personalized communication systems which extend possibilities for human-computer interaction in various settings.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.17465', 82)">Copy Link</button>

<div id="copy-message-82" class="copy-message"></div>

</div>

<div class="article">

<h1> cross <a href="https://arxiv.org/abs/2502.17475">ECG-Expert-QA: A Benchmark for Evaluating Medical Large Language Models in Heart Disease Diagnosis</a></h1>

<p><b>Authors:</b> Xu Wang, Jiaju Kang, Puyu Han</p>

<p>Abstract: We present ECG-Expert-QA, a comprehensive multimodal dataset designed for evaluating diagnostic capabilities in ECG interpretation, integrating real clinical data with systematically generated synthetic cases. The dataset encompasses six fundamental diagnostic tasks, comprising 47,211 meticulously curated question-answer pairs that span a spectrum of clinical scenarios, from basic rhythm analysis to complex case interpretation. By simulating challenging clinical cases through a rigorous medical knowledge-guided process, ECG-Expert-QA not only enhances the availability of annotated diagnostic data but also significantly increases the complexity and diversity of clinical presentations, including rare cardiac conditions and temporal progression patterns. This design enables comprehensive evaluation of medical language models across multiple dimensions, including diagnostic accuracy, clinical reasoning, and knowledge integration. To facilitate global research collaboration, ECG-Expert-QA is available in both Chinese and English versions, with rigorous quality control ensuring linguistic and clinical consistency. The dataset's challenging diagnostic tasks, which include interpretation of complex arrhythmias, identification of subtle ischemic changes, and integration of clinical context, establish it as an effective benchmark for advancing AI-assisted ECG interpretation and pushing the boundaries of current diagnostic models. Our dataset is open-source and available at https://github.com/Zaozzz/ECG-Expert-QA.</p>

<p>URLs: <a href="https://github.com/Zaozzz/ECG-Expert-QA.">https://github.com/Zaozzz/ECG-Expert-QA.</a></p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.17475, https://github.com/Zaozzz/ECG-Expert-QA.', 83)">Copy Link</button>

<div id="copy-message-83" class="copy-message"></div>

</div>

<div class="article">

<h1> cross <a href="https://arxiv.org/abs/2502.17480">Brain-to-Text Decoding: A Non-invasive Approach via Typing</a></h1>

<p><b>Authors:</b> Jarod L\'evy, Mingfang Zhang, Svetlana Pinet, J\'er\'emy Rapin, Hubert Banville, St\'ephane d'Ascoli, Jean-R\'emi King</p>

<p>Abstract: Modern neuroprostheses can now restore communication in patients who have lost the ability to speak or move. However, these invasive devices entail risks inherent to neurosurgery. Here, we introduce a non-invasive method to decode the production of sentences from brain activity and demonstrate its efficacy in a cohort of 35 healthy volunteers. For this, we present Brain2Qwerty, a new deep learning architecture trained to decode sentences from either electro- (EEG) or magneto-encephalography (MEG), while participants typed briefly memorized sentences on a QWERTY keyboard. With MEG, Brain2Qwerty reaches, on average, a character-error-rate (CER) of 32% and substantially outperforms EEG (CER: 67%). For the best participants, the model achieves a CER of 19%, and can perfectly decode a variety of sentences outside of the training set. While error analyses suggest that decoding depends on motor processes, the analysis of typographical errors suggests that it also involves higher-level cognitive factors. Overall, these results narrow the gap between invasive and non-invasive methods and thus open the path for developing safe brain-computer interfaces for non-communicating patients.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.17480', 84)">Copy Link</button>

<div id="copy-message-84" class="copy-message"></div>

</div>

<div class="article">

<h1> cross <a href="https://arxiv.org/abs/2502.17498">Improving Value-based Process Verifier via Structural Prior Injection</a></h1>

<p><b>Authors:</b> Zetian Sun, Dongfang Li, Baotian Hu, Jun Yu, Min Zhang</p>

<p>Abstract: In the Large Language Model(LLM) reasoning scenario, people often estimate state value via Monte Carlo sampling. Though Monte Carlo estimation is an elegant method with less inductive bias, noise and errors are inevitably introduced due to the limited sampling. To handle the problem, we inject the structural prior into the value representation and transfer the scalar value into the expectation of a pre-defined categorical distribution, representing the noise and errors from a distribution perspective. Specifically, by treating the result of Monte Carlo sampling as a single sample from the prior ground-truth Binomial distribution, we quantify the sampling error as the mismatch between posterior estimated distribution and ground-truth distribution, which is thus optimized via distribution selection optimization. We test the performance of value-based process verifiers on Best-of-N task and Beam search task. Compared with the scalar value representation, we show that reasonable structural prior injection induced by different objective functions or optimization methods can improve the performance of value-based process verifiers for about 1$\sim$2 points at little-to-no cost. We also show that under different structural prior, the verifiers' performances vary greatly despite having the same optimal solution, indicating the importance of reasonable structural prior injection.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.17498', 85)">Copy Link</button>

<div id="copy-message-85" class="copy-message"></div>

</div>

<div class="article">

<h1> cross <a href="https://arxiv.org/abs/2502.17504">Protein Large Language Models: A Comprehensive Survey</a></h1>

<p><b>Authors:</b> Yijia Xiao, Wanjia Zhao, Junkai Zhang, Yiqiao Jin, Han Zhang, Zhicheng Ren, Renliang Sun, Haixin Wang, Guancheng Wan, Pan Lu, Xiao Luo, Yu Zhang, James Zou, Yizhou Sun, Wei Wang</p>

<p>Abstract: Protein-specific large language models (Protein LLMs) are revolutionizing protein science by enabling more efficient protein structure prediction, function annotation, and design. While existing surveys focus on specific aspects or applications, this work provides the first comprehensive overview of Protein LLMs, covering their architectures, training datasets, evaluation metrics, and diverse applications. Through a systematic analysis of over 100 articles, we propose a structured taxonomy of state-of-the-art Protein LLMs, analyze how they leverage large-scale protein sequence data for improved accuracy, and explore their potential in advancing protein engineering and biomedical research. Additionally, we discuss key challenges and future directions, positioning Protein LLMs as essential tools for scientific discovery in protein science. Resources are maintained at https://github.com/Yijia-Xiao/Protein-LLM-Survey.</p>

<p>URLs: <a href="https://github.com/Yijia-Xiao/Protein-LLM-Survey.">https://github.com/Yijia-Xiao/Protein-LLM-Survey.</a></p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.17504, https://github.com/Yijia-Xiao/Protein-LLM-Survey.', 86)">Copy Link</button>

<div id="copy-message-86" class="copy-message"></div>

</div>

<div class="article">

<h1> cross <a href="https://arxiv.org/abs/2502.17510">Recurrent Knowledge Identification and Fusion for Language Model Continual Learning</a></h1>

<p><b>Authors:</b> Yujie Feng, Xujia Wang, Zexin Lu, Shenghong Fu, Guangyuan Shi, Yongxin Xu, Yasha Wang, Philip S. Yu, Xu Chu, Xiao-Ming Wu</p>

<p>Abstract: Continual learning (CL) is crucial for deploying large language models (LLMs) in dynamic real-world environments without costly retraining. While recent model ensemble and model merging methods guided by parameter importance have gained popularity, they often struggle to balance knowledge transfer and forgetting, mainly due to the reliance on static importance estimates during sequential training. In this paper, we present Recurrent-KIF, a novel CL framework for Recurrent Knowledge Identification and Fusion, which enables dynamic estimation of parameter importance distributions to enhance knowledge transfer. Inspired by human continual learning, Recurrent-KIF employs an inner loop that rapidly adapts to new tasks while identifying important parameters, coupled with an outer loop that globally manages the fusion of new and historical knowledge through redundant knowledge pruning and key knowledge merging. These inner-outer loops iteratively perform multiple rounds of fusion, allowing Recurrent-KIF to leverage intermediate training information and adaptively adjust fusion strategies based on evolving importance distributions. Extensive experiments on two CL benchmarks with various model sizes (from 770M to 13B) demonstrate that Recurrent-KIF effectively mitigates catastrophic forgetting and enhances knowledge transfer.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.17510', 87)">Copy Link</button>

<div id="copy-message-87" class="copy-message"></div>

</div>

<div class="article">

<h1> cross <a href="https://arxiv.org/abs/2502.17514">SAE-V: Interpreting Multimodal Models for Enhanced Alignment</a></h1>

<p><b>Authors:</b> Hantao Lou, Changye Li, Jiaming Ji, Yaodong Yang</p>

<p>Abstract: With the integration of image modality, the semantic space of multimodal large language models (MLLMs) is more complex than text-only models, making their interpretability more challenging and their alignment less stable, particularly susceptible to low-quality data, which can lead to inconsistencies between modalities, hallucinations, and biased outputs. As a result, developing interpretability methods for MLLMs is crucial for improving alignment quality and efficiency. In text-only LLMs, Sparse Autoencoders (SAEs) have gained attention for their ability to interpret latent representations. However, extending SAEs to multimodal settings presents new challenges due to modality fusion and the difficulty of isolating cross-modal representations. To address these challenges, we introduce SAE-V, a mechanistic interpretability framework that extends the SAE paradigm to MLLMs. By identifying and analyzing interpretable features along with their corresponding data, SAE-V enables fine-grained interpretation of both model behavior and data quality, facilitating a deeper understanding of cross-modal interactions and alignment dynamics. Moreover, by utilizing cross-modal feature weighting, SAE-V provides an intrinsic data filtering mechanism to enhance model alignment without requiring additional models. Specifically, when applied to the alignment process of MLLMs, SAE-V-based data filtering methods could achieve more than 110% performance with less than 50% data. Our results highlight SAE-V's ability to enhance interpretability and alignment in MLLMs, providing insights into their internal mechanisms.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.17514', 88)">Copy Link</button>

<div id="copy-message-88" class="copy-message"></div>

</div>

<div class="article">

<h1> cross <a href="https://arxiv.org/abs/2502.17521">Recent Advances in Large Langauge Model Benchmarks against Data Contamination: From Static to Dynamic Evaluation</a></h1>

<p><b>Authors:</b> Simin Chen, Yiming Chen, Zexin Li, Yifan Jiang, Zhongwei Wan, Yixin He, Dezhi Ran, Tianle Gu, Haizhou Li, Tao Xie, Baishakhi Ray</p>

<p>Abstract: Data contamination has received increasing attention in the era of large language models (LLMs) due to their reliance on vast Internet-derived training corpora. To mitigate the risk of potential data contamination, LLM benchmarking has undergone a transformation from static to dynamic benchmarking. In this work, we conduct an in-depth analysis of existing static to dynamic benchmarking methods aimed at reducing data contamination risks. We first examine methods that enhance static benchmarks and identify their inherent limitations. We then highlight a critical gap-the lack of standardized criteria for evaluating dynamic benchmarks. Based on this observation, we propose a series of optimal design principles for dynamic benchmarking and analyze the limitations of existing dynamic benchmarks. This survey provides a concise yet comprehensive overview of recent advancements in data contamination research, offering valuable insights and a clear guide for future research efforts. We maintain a GitHub repository to continuously collect both static and dynamic benchmarking methods for LLMs. The repository can be found at this link.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.17521', 89)">Copy Link</button>

<div id="copy-message-89" class="copy-message"></div>

</div>

<div class="article">

<h1> cross <a href="https://arxiv.org/abs/2502.17533">From Euler to AI: Unifying Formulas for Mathematical Constants</a></h1>

<p><b>Authors:</b> Tomer Raz, Michael Shalyt, Elyasheev Leibtag, Rotem Kalisch, Yaron Hadad, Ido Kaminer</p>

<p>Abstract: The constant $\pi$ has fascinated scholars for centuries, inspiring the derivation of countless formulas rooted in profound mathematical insight. This abundance of formulas raises a question: Are they interconnected, and can a unifying structure explain their relationships?

We propose a systematic methodology for discovering and proving formula equivalences, leveraging modern large language models, large-scale data processing, and novel mathematical algorithms. Analyzing 457,145 arXiv papers, over a third of the validated formulas for $\pi$ were proven to be derivable from a single mathematical object - including formulas by Euler, Gauss, Lord Brouncker, and newer ones from algorithmic discoveries by the Ramanujan Machine.

Our approach extends to other constants, such as $e$, $\zeta(3)$, and Catalan's constant, proving its broad applicability. This work represents a step toward the automatic unification of mathematical knowledge, laying a foundation for AI-driven discoveries of connections across scientific domains.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.17533', 90)">Copy Link</button>

<div id="copy-message-90" class="copy-message"></div>

</div>

<div class="article">

<h1> cross <a href="https://arxiv.org/abs/2502.17535">The Lottery LLM Hypothesis, Rethinking What Abilities Should LLM Compression Preserve?</a></h1>

<p><b>Authors:</b> Zhenheng Tang, Xiang Liu, Qian Wang, Peijie Dong, Bingsheng He, Xiaowen Chu, Bo Li</p>

<p>Abstract: Motivated by reducing the computational and storage costs of LLMs, model compression and KV cache compression have attracted much attention from researchers. However, current methods predominantly emphasize maintaining the performance of compressed LLMs, as measured by perplexity or simple accuracy on tasks of common sense knowledge QA and basic arithmetic reasoning. In this blog, we present a brief review of recent advancements in LLMs related to retrieval-augmented generation, multi-step reasoning, external tools, and computational expressivity, all of which substantially enhance LLM performance. Then, we propose a lottery LLM hypothesis suggesting that for a given LLM and task, there exists a smaller lottery LLM capable of producing the same performance as the original LLM with the assistance of multi-step reasoning and external tools. Based on the review of current progress in LLMs, we discuss and summarize the essential capabilities that the lottery LLM and KV cache compression must possess, which are currently overlooked in existing methods.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.17535', 91)">Copy Link</button>

<div id="copy-message-91" class="copy-message"></div>

</div>

<div class="article">

<h1> cross <a href="https://arxiv.org/abs/2502.17540">PosterSum: A Multimodal Benchmark for Scientific Poster Summarization</a></h1>

<p><b>Authors:</b> Rohit Saxena, Pasquale Minervini, Frank Keller</p>

<p>Abstract: Generating accurate and concise textual summaries from multimodal documents is challenging, especially when dealing with visually complex content like scientific posters. We introduce PosterSum, a novel benchmark to advance the development of vision-language models that can understand and summarize scientific posters into research paper abstracts. Our dataset contains 16,305 conference posters paired with their corresponding abstracts as summaries. Each poster is provided in image format and presents diverse visual understanding challenges, such as complex layouts, dense text regions, tables, and figures. We benchmark state-of-the-art Multimodal Large Language Models (MLLMs) on PosterSum and demonstrate that they struggle to accurately interpret and summarize scientific posters. We propose Segment & Summarize, a hierarchical method that outperforms current MLLMs on automated metrics, achieving a 3.14% gain in ROUGE-L. This will serve as a starting point for future research on poster summarization.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.17540', 92)">Copy Link</button>

<div id="copy-message-92" class="copy-message"></div>

</div>

<div class="article">

<h1> cross <a href="https://arxiv.org/abs/2502.17541">Dataset Featurization: Uncovering Natural Language Features through Unsupervised Data Reconstruction</a></h1>

<p><b>Authors:</b> Michal Bravansky, Vaclav Kubon, Suhas Hariharan, Robert Kirk</p>

<p>Abstract: Interpreting data is central to modern research. Large language models (LLMs) show promise in providing such natural language interpretations of data, yet simple feature extraction methods such as prompting often fail to produce accurate and versatile descriptions for diverse datasets and lack control over granularity and scale. To address these limitations, we propose a domain-agnostic method for dataset featurization that provides precise control over the number of features extracted while maintaining compact and descriptive representations comparable to human expert labeling. Our method optimizes the selection of informative binary features by evaluating the ability of an LLM to reconstruct the original data using those features. We demonstrate its effectiveness in dataset modeling tasks and through two case studies: (1) Constructing a feature representation of jailbreak tactics that compactly captures both the effectiveness and diversity of a larger set of human-crafted attacks; and (2) automating the discovery of features that align with human preferences, achieving accuracy and robustness comparable to expert-crafted features. Moreover, we show that the pipeline scales effectively, improving as additional features are sampled, making it suitable for large and diverse datasets.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.17541', 93)">Copy Link</button>

<div id="copy-message-93" class="copy-message"></div>

</div>

<div class="article">

<h1> cross <a href="https://arxiv.org/abs/2502.17543">Training a Generally Curious Agent</a></h1>

<p><b>Authors:</b> Fahim Tajwar, Yiding Jiang, Abitha Thankaraj, Sumaita Sadia Rahman, J Zico Kolter, Jeff Schneider, Ruslan Salakhutdinov</p>

<p>Abstract: Efficient exploration is essential for intelligent systems interacting with their environment, but existing language models often fall short in scenarios that require strategic information gathering. In this paper, we present PAPRIKA, a fine-tuning approach that enables language models to develop general decision-making capabilities that are not confined to particular environments. By training on synthetic interaction data from different tasks that require diverse strategies, PAPRIKA teaches models to explore and adapt their behavior on a new task based on environment feedback in-context without more gradient updates. Experimental results show that models fine-tuned with PAPRIKA can effectively transfer their learned decision-making capabilities to entirely unseen tasks without additional training. Unlike traditional training, our approach's primary bottleneck lies in sampling useful interaction data instead of model updates. To improve sample efficiency, we propose a curriculum learning strategy that prioritizes sampling trajectories from tasks with high learning potential. These results suggest a promising path towards AI systems that can autonomously solve novel sequential decision-making problems that require interactions with the external world.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.17543', 94)">Copy Link</button>

<div id="copy-message-94" class="copy-message"></div>

</div>

<div class="article">

<h1> cross <a href="https://arxiv.org/abs/2502.17598">Hallucination Detection in LLMs Using Spectral Features of Attention Maps</a></h1>

<p><b>Authors:</b> Jakub Binkowski, Denis Janiak, Albert Sawczyn, Bogdan Gabrys, Tomasz Kajdanowicz</p>

<p>Abstract: Large Language Models (LLMs) have demonstrated remarkable performance across various tasks but remain prone to hallucinations. Detecting hallucinations is essential for safety-critical applications, and recent methods leverage attention map properties to this end, though their effectiveness remains limited. In this work, we investigate the spectral features of attention maps by interpreting them as adjacency matrices of graph structures. We propose the $\text{LapEigvals}$ method, which utilises the top-$k$ eigenvalues of the Laplacian matrix derived from the attention maps as an input to hallucination detection probes. Empirical evaluations demonstrate that our approach achieves state-of-the-art hallucination detection performance among attention-based methods. Extensive ablation studies further highlight the robustness and generalisation of $\text{LapEigvals}$, paving the way for future advancements in the hallucination detection domain.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.17598', 95)">Copy Link</button>

<div id="copy-message-95" class="copy-message"></div>

</div>

<div class="article">

<h1> cross <a href="https://arxiv.org/abs/2502.17607">Synthetic Text Generation for Training Large Language Models via Gradient Matching</a></h1>

<p><b>Authors:</b> Dang Nguyen, Zeman Li, Mohammadhossein Bateni, Vahab Mirrokni, Meisam Razaviyayn, Baharan Mirzasoleiman</p>

<p>Abstract: Synthetic data has the potential to improve the performance, training efficiency, and privacy of real training examples. Nevertheless, existing approaches for synthetic text generation are mostly heuristics and cannot generate human-readable text without compromising the privacy of real data or provide performance guarantees for training Large Language Models (LLMs). In this work, we propose the first theoretically rigorous approach for generating synthetic human-readable text that guarantees the convergence and performance of LLMs during fine-tuning on a target task. To do so, we leverage Alternating Direction Method of Multipliers (ADMM) that iteratively optimizes the embeddings of synthetic examples to match the gradient of the target training or validation data, and maps them to a sequence of text tokens with low perplexity. In doing so, the generated synthetic text can guarantee convergence of the model to a close neighborhood of the solution obtained by fine-tuning on real data. Experiments on various classification tasks confirm the effectiveness of our proposed approach.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.17607', 96)">Copy Link</button>

<div id="copy-message-96" class="copy-message"></div>

</div>

<div class="article">

<h1> cross <a href="https://arxiv.org/abs/2502.17651">METAL: A Multi-Agent Framework for Chart Generation with Test-Time Scaling</a></h1>

<p><b>Authors:</b> Bingxuan Li, Yiwei Wang, Jiuxiang Gu, Kai-Wei Chang, Nanyun Peng</p>

<p>Abstract: Chart generation aims to generate code to produce charts satisfying the desired visual properties, e.g., texts, layout, color, and type. It has great potential to empower the automatic professional report generation in financial analysis, research presentation, education, and healthcare. In this work, we build a vision-language model (VLM) based multi-agent framework for effective automatic chart generation. Generating high-quality charts requires both strong visual design skills and precise coding capabilities that embed the desired visual properties into code. Such a complex multi-modal reasoning process is difficult for direct prompting of VLMs. To resolve these challenges, we propose METAL, a multi-agent framework that decomposes the task of chart generation into the iterative collaboration among specialized agents. METAL achieves 5.2% improvement in accuracy over the current best result in the chart generation task. The METAL framework exhibits the phenomenon of test-time scaling: its performance increases monotonically as the logarithmic computational budget grows from 512 to 8192 tokens. In addition, we find that separating different modalities during the critique process of METAL boosts the self-correction capability of VLMs in the multimodal context.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.17651', 97)">Copy Link</button>

<div id="copy-message-97" class="copy-message"></div>

</div>

<div class="article">

<h1> cross <a href="https://arxiv.org/abs/2502.17701">From Perceptions to Decisions: Wildfire Evacuation Decision Prediction with Behavioral Theory-informed LLMs</a></h1>

<p><b>Authors:</b> Ruxiao Chen, Chenguang Wang, Yuran Sun, Xilei Zhao, Susu Xu</p>

<p>Abstract: Evacuation decision prediction is critical for efficient and effective wildfire response by helping emergency management anticipate traffic congestion and bottlenecks, allocate resources, and minimize negative impacts. Traditional statistical methods for evacuation decision prediction fail to capture the complex and diverse behavioral logic of different individuals. In this work, for the first time, we introduce FLARE, short for facilitating LLM for advanced reasoning on wildfire evacuation decision prediction, a Large Language Model (LLM)-based framework that integrates behavioral theories and models to streamline the Chain-of-Thought (CoT) reasoning and subsequently integrate with memory-based Reinforcement Learning (RL) module to provide accurate evacuation decision prediction and understanding. Our proposed method addresses the limitations of using existing LLMs for evacuation behavioral predictions, such as limited survey data, mismatching with behavioral theory, conflicting individual preferences, implicit and complex mental states, and intractable mental state-behavior mapping. Experiments on three post-wildfire survey datasets show an average of 20.47% performance improvement over traditional theory-informed behavioral models, with strong cross-event generalizability. Our complete code is publicly available at https://github.com/SusuXu-s-Lab/FLARE</p>

<p>URLs: <a href="https://github.com/SusuXu-s-Lab/FLARE">https://github.com/SusuXu-s-Lab/FLARE</a></p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.17701, https://github.com/SusuXu-s-Lab/FLARE', 98)">Copy Link</button>

<div id="copy-message-98" class="copy-message"></div>

</div>

<div class="article">

<h1> cross <a href="https://arxiv.org/abs/2502.17709">Contrastive Visual Data Augmentation</a></h1>

<p><b>Authors:</b> Yu Zhou, Bingxuan Li, Mohan Tang, Xiaomeng Jin, Te-Lin Wu, Kuan-Hao Huang, Heng Ji, Kai-Wei Chang, Nanyun Peng</p>

<p>Abstract: Large multimodal models (LMMs) often struggle to recognize novel concepts, as they rely on pre-trained knowledge and have limited ability to capture subtle visual details. Domain-specific knowledge gaps in training also make them prone to confusing visually similar, commonly misrepresented, or low-resource concepts. To help LMMs better align nuanced visual features with language, improving their ability to recognize and reason about novel or rare concepts, we propose a Contrastive visual Data Augmentation (CoDA) strategy. CoDA extracts key contrastive textual and visual features of target concepts against the known concepts they are misrecognized as, and then uses multimodal generative models to produce targeted synthetic data. Automatic filtering of extracted features and augmented images is implemented to guarantee their quality, as verified by human annotators. We show the effectiveness and efficiency of CoDA on low-resource concept and diverse scene recognition datasets including INaturalist and SUN. We additionally collect NovelSpecies, a benchmark dataset consisting of newly discovered animal species that are guaranteed to be unseen by LMMs. LLaVA-1.6 1-shot updating results on these three datasets show CoDA significantly improves SOTA visual data augmentation strategies by 12.3% (NovelSpecies), 5.1% (SUN), and 6.0% (iNat) absolute gains in accuracy.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.17709', 99)">Copy Link</button>

<div id="copy-message-99" class="copy-message"></div>

</div>

<div class="article">

<h1> cross <a href="https://arxiv.org/abs/2502.17710">Mind the Gesture: Evaluating AI Sensitivity to Culturally Offensive Non-Verbal Gestures</a></h1>

<p><b>Authors:</b> Akhila Yerukola, Saadia Gabriel, Nanyun Peng, Maarten Sap</p>

<p>Abstract: Gestures are an integral part of non-verbal communication, with meanings that vary across cultures, and misinterpretations that can have serious social and diplomatic consequences. As AI systems become more integrated into global applications, ensuring they do not inadvertently perpetuate cultural offenses is critical. To this end, we introduce Multi-Cultural Set of Inappropriate Gestures and Nonverbal Signs (MC-SIGNS), a dataset of 288 gesture-country pairs annotated for offensiveness, cultural significance, and contextual factors across 25 gestures and 85 countries. Through systematic evaluation using MC-SIGNS, we uncover critical limitations: text-to-image (T2I) systems exhibit strong US-centric biases, performing better at detecting offensive gestures in US contexts than in non-US ones; large language models (LLMs) tend to over-flag gestures as offensive; and vision-language models (VLMs) default to US-based interpretations when responding to universal concepts like wishing someone luck, frequently suggesting culturally inappropriate gestures. These findings highlight the urgent need for culturally-aware AI safety mechanisms to ensure equitable global deployment of AI technologies.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.17710', 100)">Copy Link</button>

<div id="copy-message-100" class="copy-message"></div>

</div>

<div class="article">

<h1> cross <a href="https://arxiv.org/abs/2502.17776">Tip of the Tongue Query Elicitation for Simulated Evaluation</a></h1>

<p><b>Authors:</b> Yifan He, To Eun Kim, Fernando Diaz, Jaime Arguello, Bhaskar Mitra</p>

<p>Abstract: Tip-of-the-tongue (TOT) search occurs when a user struggles to recall a specific identifier, such as a document title. While common, existing search systems often fail to effectively support TOT scenarios. Research on TOT retrieval is further constrained by the challenge of collecting queries, as current approaches rely heavily on community question-answering (CQA) websites, leading to labor-intensive evaluation and domain bias. To overcome these limitations, we introduce two methods for eliciting TOT queries - leveraging large language models (LLMs) and human participants - to facilitate simulated evaluations of TOT retrieval systems. Our LLM-based TOT user simulator generates synthetic TOT queries at scale, achieving high correlations with how CQA-based TOT queries rank TOT retrieval systems when tested in the Movie domain. Additionally, these synthetic queries exhibit high linguistic similarity to CQA-derived queries. For human-elicited queries, we developed an interface that uses visual stimuli to place participants in a TOT state, enabling the collection of natural queries. In the Movie domain, system rank correlation and linguistic similarity analyses confirm that human-elicited queries are both effective and closely resemble CQA-based queries. These approaches reduce reliance on CQA-based data collection while expanding coverage to underrepresented domains, such as Landmark and Person. LLM-elicited queries for the Movie, Landmark, and Person domains have been released as test queries in the TREC 2024 TOT track, with human-elicited queries scheduled for inclusion in the TREC 2025 TOT track. Additionally, we provide source code for synthetic query generation and the human query collection interface, along with curated visual stimuli used for eliciting TOT queries.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.17776', 101)">Copy Link</button>

<div id="copy-message-101" class="copy-message"></div>

</div>

<div class="article">

<h1> cross <a href="https://arxiv.org/abs/2502.17784">MuCoS: Efficient Drug-Target Prediction through Multi-Context-Aware Sampling</a></h1>

<p><b>Authors:</b> Haji Gul, Abdul Gani Haji Naim, Ajaz A. Bhat</p>

<p>Abstract: Drug-target interactions are critical for understanding biological processes and advancing drug discovery. However, traditional methods such as ComplEx-SE, TransE, and DistMult struggle with unseen relationships and negative triplets, which limits their effectiveness in drug-target prediction. To address these challenges, we propose Multi-Context-Aware Sampling (MuCoS), an efficient and positively accurate method for drug-target prediction. MuCoS reduces computational complexity by prioritizing neighbors of higher density to capture informative structural patterns. These optimized neighborhood representations are integrated with BERT, enabling contextualized embeddings for accurate prediction of missing relationships or tail entities. MuCoS avoids the need for negative triplet sampling, reducing computation while improving performance over unseen entities and relations. Experiments on the KEGG50k biomedical dataset show that MuCoS improved over existing models by 13\% on MRR, 7\% on Hits@1, 4\% on Hits@3, and 18\% on Hits@10 for the general relationship, and by 6\% on MRR, 1\% on Hits@1, 3\% on Hits@3, and 12\% on Hits@10 for prediction of drug-target relationship.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.17784', 102)">Copy Link</button>

<div id="copy-message-102" class="copy-message"></div>

</div>

<div class="article">

<h1> cross <a href="https://arxiv.org/abs/2502.17814">An Overview of Large Language Models for Statisticians</a></h1>

<p><b>Authors:</b> Wenlong Ji, Weizhe Yuan, Emily Getzen, Kyunghyun Cho, Michael I. Jordan, Song Mei, Jason E Weston, Weijie J. Su, Jing Xu, Linjun Zhang</p>

<p>Abstract: Large Language Models (LLMs) have emerged as transformative tools in artificial intelligence (AI), exhibiting remarkable capabilities across diverse tasks such as text generation, reasoning, and decision-making. While their success has primarily been driven by advances in computational power and deep learning architectures, emerging problems -- in areas such as uncertainty quantification, decision-making, causal inference, and distribution shift -- require a deeper engagement with the field of statistics. This paper explores potential areas where statisticians can make important contributions to the development of LLMs, particularly those that aim to engender trustworthiness and transparency for human users. Thus, we focus on issues such as uncertainty quantification, interpretability, fairness, privacy, watermarking and model adaptation. We also consider possible roles for LLMs in statistical analysis. By bridging AI and statistics, we aim to foster a deeper collaboration that advances both the theoretical foundations and practical applications of LLMs, ultimately shaping their role in addressing complex societal challenges.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.17814', 103)">Copy Link</button>

<div id="copy-message-103" class="copy-message"></div>

</div>

<div class="article">

<h1> cross <a href="https://arxiv.org/abs/2502.17823">A General Framework to Enhance Fine-tuning-based LLM Unlearning</a></h1>

<p><b>Authors:</b> Jie Ren, Zhenwei Dai, Xianfeng Tang, Hui Liu, Jingying Zeng, Zhen Li, Rahul Goutam, Suhang Wang, Yue Xing, Qi He, Hui Liu</p>

<p>Abstract: Unlearning has been proposed to remove copyrighted and privacy-sensitive data from Large Language Models (LLMs). Existing approaches primarily rely on fine-tuning-based methods, which can be categorized into gradient ascent-based (GA-based) and suppression-based methods. However, they often degrade model utility (the ability to respond to normal prompts). In this work, we aim to develop a general framework that enhances the utility of fine-tuning-based unlearning methods. To achieve this goal, we first investigate the common property between GA-based and suppression-based methods. We unveil that GA-based methods unlearn by distinguishing the target data (i.e., the data to be removed) and suppressing related generations, which is essentially the same strategy employed by suppression-based methods. Inspired by this finding, we introduce Gated Representation UNlearning (GRUN) which has two components: a soft gate function for distinguishing target data and a suppression module using Representation Fine-tuning (ReFT) to adjust representations rather than model parameters. Experiments show that GRUN significantly improves the unlearning and utility. Meanwhile, it is general for fine-tuning-based methods, efficient and promising for sequential unlearning.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.17823', 104)">Copy Link</button>

<div id="copy-message-104" class="copy-message"></div>

</div>

<div class="article">

<h1> cross <a href="https://arxiv.org/abs/2502.17882">Science Across Languages: Assessing LLM Multilingual Translation of Scientific Papers</a></h1>

<p><b>Authors:</b> Hannah Calzi Kleidermacher, James Zou</p>

<p>Abstract: Scientific research is inherently global. However, the vast majority of academic journals are published exclusively in English, creating barriers for non-native-English-speaking researchers. In this study, we leverage large language models (LLMs) to translate published scientific articles while preserving their native JATS XML formatting, thereby developing a practical, automated approach for implementation by academic journals. Using our approach, we translate articles across multiple scientific disciplines into 28 languages. To evaluate translation accuracy, we introduce a novel question-and-answer (QA) benchmarking method, in which an LLM generates comprehension-based questions from the original text and then answers them based on the translated text. Our benchmark results show an average performance of 95.9%, showing that the key scientific details are accurately conveyed. In a user study, we translate the scientific papers of 15 researchers into their native languages, finding that the authors consistently found the translations to accurately capture the original information in their articles. Interestingly, a third of the authors found many technical terms "overtranslated," expressing a preference to keep terminology more familiar in English untranslated. Finally, we demonstrate how in-context learning techniques can be used to align translations with domain-specific preferences such as mitigating overtranslation, highlighting the adaptability and utility of LLM-driven scientific translation. The code and translated articles are available at https://hankleid.github.io/ProjectMundo.</p>

<p>URLs: <a href="https://hankleid.github.io/ProjectMundo.">https://hankleid.github.io/ProjectMundo.</a></p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.17882, https://hankleid.github.io/ProjectMundo.', 105)">Copy Link</button>

<div id="copy-message-105" class="copy-message"></div>

</div>

<div class="article">

<h1> cross <a href="https://arxiv.org/abs/2502.17967">LLM Knows Geometry Better than Algebra: Numerical Understanding of LLM-Based Agents in A Trading Arena</a></h1>

<p><b>Authors:</b> Tianmi Ma, Jiawei Du, Wenxin Huang, Wenjie Wang, Liang Xie, Xian Zhong, Joey Tianyi Zhou</p>

<p>Abstract: Recent advancements in large language models (LLMs) have significantly improved performance in natural language processing tasks. However, their ability to generalize to dynamic, unseen tasks, particularly in numerical reasoning, remains a challenge. Existing benchmarks mainly evaluate LLMs on problems with predefined optimal solutions, which may not align with real-world scenarios where clear answers are absent. To bridge this gap, we design the Agent Trading Arena, a virtual numerical game simulating complex economic systems through zero-sum games, where agents invest in stock portfolios. Our experiments reveal that LLMs, including GPT-4o, struggle with algebraic reasoning when dealing with plain-text stock data, often focusing on local details rather than global trends. In contrast, LLMs perform significantly better with geometric reasoning when presented with visual data, such as scatter plots or K-line charts, suggesting that visual representations enhance numerical reasoning. This capability is further improved by incorporating the reflection module, which aids in the analysis and interpretation of complex data. We validate our findings on NASDAQ Stock dataset, where LLMs demonstrate stronger reasoning with visual data compared to text. Our code and data are publicly available at https://github.com/wekjsdvnm/Agent-Trading-Arena.git.</p>

<p>URLs: <a href="https://github.com/wekjsdvnm/Agent-Trading-Arena.git.">https://github.com/wekjsdvnm/Agent-Trading-Arena.git.</a></p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.17967, https://github.com/wekjsdvnm/Agent-Trading-Arena.git.', 106)">Copy Link</button>

<div id="copy-message-106" class="copy-message"></div>

</div>

<div class="article">

<h1> cross <a href="https://arxiv.org/abs/2502.18017">ViDoRAG: Visual Document Retrieval-Augmented Generation via Dynamic Iterative Reasoning Agents</a></h1>

<p><b>Authors:</b> Qiuchen Wang, Ruixue Ding, Zehui Chen, Weiqi Wu, Shihang Wang, Pengjun Xie, Feng Zhao</p>

<p>Abstract: Understanding information from visually rich documents remains a significant challenge for traditional Retrieval-Augmented Generation (RAG) methods. Existing benchmarks predominantly focus on image-based question answering (QA), overlooking the fundamental challenges of efficient retrieval, comprehension, and reasoning within dense visual documents. To bridge this gap, we introduce ViDoSeek, a novel dataset designed to evaluate RAG performance on visually rich documents requiring complex reasoning. Based on it, we identify key limitations in current RAG approaches: (i) purely visual retrieval methods struggle to effectively integrate both textual and visual features, and (ii) previous approaches often allocate insufficient reasoning tokens, limiting their effectiveness. To address these challenges, we propose ViDoRAG, a novel multi-agent RAG framework tailored for complex reasoning across visual documents. ViDoRAG employs a Gaussian Mixture Model (GMM)-based hybrid strategy to effectively handle multi-modal retrieval. To further elicit the model's reasoning capabilities, we introduce an iterative agent workflow incorporating exploration, summarization, and reflection, providing a framework for investigating test-time scaling in RAG domains. Extensive experiments on ViDoSeek validate the effectiveness and generalization of our approach. Notably, ViDoRAG outperforms existing methods by over 10% on the competitive ViDoSeek benchmark.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.18017', 107)">Copy Link</button>

<div id="copy-message-107" class="copy-message"></div>

</div>

<div class="article">

<h1> cross <a href="https://arxiv.org/abs/2502.18060">Defining bias in AI-systems: Biased models are fair models</a></h1>

<p><b>Authors:</b> Chiara Lindloff, Ingo Siegert</p>

<p>Abstract: The debate around bias in AI systems is central to discussions on algorithmic fairness. However, the term bias often lacks a clear definition, despite frequently being contrasted with fairness, implying that an unbiased model is inherently fair. In this paper, we challenge this assumption and argue that a precise conceptualization of bias is necessary to effectively address fairness concerns. Rather than viewing bias as inherently negative or unfair, we highlight the importance of distinguishing between bias and discrimination. We further explore how this shift in focus can foster a more constructive discourse within academic debates on fairness in AI systems.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.18060', 108)">Copy Link</button>

<div id="copy-message-108" class="copy-message"></div>

</div>

<div class="article">

<h1> cross <a href="https://arxiv.org/abs/2502.18101">Detecting Offensive Memes with Social Biases in Singapore Context Using Multimodal Large Language Models</a></h1>

<p><b>Authors:</b> Cao Yuxuan, Wu Jiayang, Alistair Cheong Liang Chuen, Bryan Shan Guanrong, Theodore Lee Chong Jen, Sherman Chann Zhi Shen</p>

<p>Abstract: Traditional online content moderation systems struggle to classify modern multimodal means of communication, such as memes, a highly nuanced and information-dense medium. This task is especially hard in a culturally diverse society like Singapore, where low-resource languages are used and extensive knowledge on local context is needed to interpret online content. We curate a large collection of 112K memes labeled by GPT-4V for fine-tuning a VLM to classify offensive memes in Singapore context. We show the effectiveness of fine-tuned VLMs on our dataset, and propose a pipeline containing OCR, translation and a 7-billion parameter-class VLM. Our solutions reach 80.62% accuracy and 0.8192 AUROC on a held-out test set, and can greatly aid human in moderating online contents. The dataset, code, and model weights will be open-sourced at https://github.com/aliencaocao/vlm-for-memes-aisg.</p>

<p>URLs: <a href="https://github.com/aliencaocao/vlm-for-memes-aisg.">https://github.com/aliencaocao/vlm-for-memes-aisg.</a></p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.18101, https://github.com/aliencaocao/vlm-for-memes-aisg.', 109)">Copy Link</button>

<div id="copy-message-109" class="copy-message"></div>

</div>

<div class="article">

<h1> cross <a href="https://arxiv.org/abs/2502.18116">Bayesian Optimization for Controlled Image Editing via LLMs</a></h1>

<p><b>Authors:</b> Chengkun Cai, Haoliang Liu, Xu Zhao, Zhongyu Jiang, Tianfang Zhang, Zongkai Wu, Jenq-Neng Hwang, Serge Belongie, Lei Li</p>

<p>Abstract: In the rapidly evolving field of image generation, achieving precise control over generated content and maintaining semantic consistency remain significant limitations, particularly concerning grounding techniques and the necessity for model fine-tuning. To address these challenges, we propose BayesGenie, an off-the-shelf approach that integrates Large Language Models (LLMs) with Bayesian Optimization to facilitate precise and user-friendly image editing. Our method enables users to modify images through natural language descriptions without manual area marking, while preserving the original image's semantic integrity. Unlike existing techniques that require extensive pre-training or fine-tuning, our approach demonstrates remarkable adaptability across various LLMs through its model-agnostic design. BayesGenie employs an adapted Bayesian optimization strategy to automatically refine the inference process parameters, achieving high-precision image editing with minimal user intervention. Through extensive experiments across diverse scenarios, we demonstrate that our framework significantly outperforms existing methods in both editing accuracy and semantic preservation, as validated using different LLMs including Claude3 and GPT-4.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.18116', 110)">Copy Link</button>

<div id="copy-message-110" class="copy-message"></div>

</div>

<div class="article">

<h1> cross <a href="https://arxiv.org/abs/2502.18125">HyperG: Hypergraph-Enhanced LLMs for Structured Knowledge</a></h1>

<p><b>Authors:</b> Sirui Huang, Hanqian Li, Yanggan Gu, Xuming Hu, Qing Li, Guandong Xu</p>

<p>Abstract: Given that substantial amounts of domain-specific knowledge are stored in structured formats, such as web data organized through HTML, Large Language Models (LLMs) are expected to fully comprehend this structured information to broaden their applications in various real-world downstream tasks. Current approaches for applying LLMs to structured data fall into two main categories: serialization-based and operation-based methods. Both approaches, whether relying on serialization or using SQL-like operations as an intermediary, encounter difficulties in fully capturing structural relationships and effectively handling sparse data. To address these unique characteristics of structured data, we propose HyperG, a hypergraph-based generation framework aimed at enhancing LLMs' ability to process structured knowledge. Specifically, HyperG first augment sparse data with contextual information, leveraging the generative power of LLMs, and incorporate a prompt-attentive hypergraph learning (PHL) network to encode both the augmented information and the intricate structural relationships within the data. To validate the effectiveness and generalization of HyperG, we conduct extensive experiments across two different downstream tasks requiring structured knowledge.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.18125', 111)">Copy Link</button>

<div id="copy-message-111" class="copy-message"></div>

</div>

<div class="article">

<h1> cross <a href="https://arxiv.org/abs/2502.18147">Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activations</a></h1>

<p><b>Authors:</b> Lucy Farnik, Tim Lawson, Conor Houghton, Laurence Aitchison</p>

<p>Abstract: Sparse autoencoders (SAEs) have been successfully used to discover sparse and human-interpretable representations of the latent activations of LLMs. However, we would ultimately like to understand the computations performed by LLMs and not just their representations. The extent to which SAEs can help us understand computations is unclear because they are not designed to "sparsify" computations in any sense, only latent activations. To solve this, we propose Jacobian SAEs (JSAEs), which yield not only sparsity in the input and output activations of a given model component but also sparsity in the computation (formally, the Jacobian) connecting them. With a na\"ive implementation, the Jacobians in LLMs would be computationally intractable due to their size. One key technical contribution is thus finding an efficient way of computing Jacobians in this setup. We find that JSAEs extract a relatively large degree of computational sparsity while preserving downstream LLM performance approximately as well as traditional SAEs. We also show that Jacobians are a reasonable proxy for computational sparsity because MLPs are approximately linear when rewritten in the JSAE basis. Lastly, we show that JSAEs achieve a greater degree of computational sparsity on pre-trained LLMs than on the equivalent randomized LLM. This shows that the sparsity of the computational graph appears to be a property that LLMs learn through training, and suggests that JSAEs might be more suitable for understanding learned transformer computations than standard SAEs.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.18147', 112)">Copy Link</button>

<div id="copy-message-112" class="copy-message"></div>

</div>

<div class="article">

<h1> cross <a href="https://arxiv.org/abs/2502.18186">Steering Language Model to Stable Speech Emotion Recognition via Contextual Perception and Chain of Thought</a></h1>

<p><b>Authors:</b> Zhixian Zhao, Xinfa Zhu, Xinsheng Wang, Shuiyuan Wang, Xuelong Geng, Wenjie Tian, Lei Xie</p>

<p>Abstract: Large-scale audio language models (ALMs), such as Qwen2-Audio, are capable of comprehending diverse audio signal, performing audio analysis and generating textual responses. However, in speech emotion recognition (SER), ALMs often suffer from hallucinations, resulting in misclassifications or irrelevant outputs. To address these challenges, we propose C$^2$SER, a novel ALM designed to enhance the stability and accuracy of SER through Contextual perception and Chain of Thought (CoT). C$^2$SER integrates the Whisper encoder for semantic perception and Emotion2Vec-S for acoustic perception, where Emotion2Vec-S extends Emotion2Vec with semi-supervised learning to enhance emotional discrimination. Additionally, C$^2$SER employs a CoT approach, processing SER in a step-by-step manner while leveraging speech content and speaking styles to improve recognition. To further enhance stability, C$^2$SER introduces self-distillation from explicit CoT to implicit CoT, mitigating error accumulation and boosting recognition accuracy. Extensive experiments show that C$^2$SER outperforms existing popular ALMs, such as Qwen2-Audio and SECap, delivering more stable and precise emotion recognition. We release the training code, checkpoints, and test sets to facilitate further research.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.18186', 113)">Copy Link</button>

<div id="copy-message-113" class="copy-message"></div>

</div>

<div class="article">

<h1> cross <a href="https://arxiv.org/abs/2502.18249">Iterative Counterfactual Data Augmentation</a></h1>

<p><b>Authors:</b> Mitchell Plyler, Min Chi</p>

<p>Abstract: Counterfactual data augmentation (CDA) is a method for controlling information or biases in training datasets by generating a complementary dataset with typically opposing biases. Prior work often either relies on hand-crafted rules or algorithmic CDA methods which can leave unwanted information in the augmented dataset. In this work, we show iterative CDA (ICDA) with initial, high-noise interventions can converge to a state with significantly lower noise. Our ICDA procedure produces a dataset where one target signal in the training dataset maintains high mutual information with a corresponding label and the information of spurious signals are reduced. We show training on the augmented datasets produces rationales on documents that better align with human annotation. Our experiments include six human produced datasets and two large-language model generated datasets.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.18249', 114)">Copy Link</button>

<div id="copy-message-114" class="copy-message"></div>

</div>

<div class="article">

<h1> cross <a href="https://arxiv.org/abs/2502.18274">Citrus: Leveraging Expert Cognitive Pathways in a Medical Language Model for Advanced Medical Decision Support</a></h1>

<p><b>Authors:</b> Guoxin Wang, Minyu Gao, Shuai Yang, Ya Zhang, Lizhi He, Liang Huang, Hanlin Xiao, Yexuan Zhang, Wanyue Li, Lu Chen, Jintao Fei, Xin Li</p>

<p>Abstract: Large language models (LLMs), particularly those with reasoning capabilities, have rapidly advanced in recent years, demonstrating significant potential across a wide range of applications. However, their deployment in healthcare, especially in disease reasoning tasks, is hindered by the challenge of acquiring expert-level cognitive data. In this paper, we introduce Citrus, a medical language model that bridges the gap between clinical expertise and AI reasoning by emulating the cognitive processes of medical experts. The model is trained on a large corpus of simulated expert disease reasoning data, synthesized using a novel approach that accurately captures the decision-making pathways of clinicians. This approach enables Citrus to better simulate the complex reasoning processes involved in diagnosing and treating medical conditions.To further address the lack of publicly available datasets for medical reasoning tasks, we release the last-stage training data, including a custom-built medical diagnostic dialogue dataset. This open-source contribution aims to support further research and development in the field. Evaluations using authoritative benchmarks such as MedQA, covering tasks in medical reasoning and language understanding, show that Citrus achieves superior performance compared to other models of similar size. These results highlight Citrus potential to significantly enhance medical decision support systems, providing a more accurate and efficient tool for clinical decision-making.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.18274', 115)">Copy Link</button>

<div id="copy-message-115" class="copy-message"></div>

</div>

<div class="article">

<h1> cross <a href="https://arxiv.org/abs/2502.18293">AMPO: Active Multi-Preference Optimization</a></h1>

<p><b>Authors:</b> Taneesh Gupta, Rahul Madhavan, Xuchao Zhang, Chetan Bansal, Saravan Rajmohan</p>

<p>Abstract: Multi-preference optimization enriches language-model alignment beyond pairwise preferences by contrasting entire sets of helpful and undesired responses, thereby enabling richer training signals for large language models. During self-play alignment, these models often produce numerous candidate answers per query, rendering it computationally infeasible to include all responses in the training objective. In this work, we propose $\textit{Active Multi-Preference Optimization}$ (AMPO), a novel approach that combines on-policy generation, a multi-preference group-contrastive loss, and active subset selection. Specifically, we score and embed large candidate pools of responses and then select a small, yet informative, subset that covers reward extremes and distinct semantic clusters for preference optimization. Our contrastive training scheme is capable of identifying not only the best and worst answers but also subtle, underexplored modes that are crucial for robust alignment. Theoretically, we provide guarantees for expected reward maximization using our active selection method, and empirically, AMPO achieves state-of-the-art results on $\textit{AlpacaEval}$ using Llama 8B.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.18293', 116)">Copy Link</button>

<div id="copy-message-116" class="copy-message"></div>

</div>

<div class="article">

<h1> cross <a href="https://arxiv.org/abs/2502.18418">Rank1: Test-Time Compute for Reranking in Information Retrieval</a></h1>

<p><b>Authors:</b> Orion Weller, Kathryn Ricci, Eugene Yang, Andrew Yates, Dawn Lawrie, Benjamin Van Durme</p>

<p>Abstract: We introduce Rank1, the first reranking model trained to take advantage of test-time compute. Rank1 demonstrates the applicability within retrieval of using a reasoning language model (i.e. OpenAI's o1, Deepseek's R1, etc.) for distillation in order to rapidly improve the performance of a smaller model. We gather and open-source a dataset of more than 600,000 examples of R1 reasoning traces from queries and passages in MS MARCO. Models trained on this dataset show: (1) state-of-the-art performance on advanced reasoning and instruction following datasets; (2) work remarkably well out of distribution due to the ability to respond to user-input prompts; and (3) have explainable reasoning chains that can be given to users or RAG-based systems. Further, we demonstrate that quantized versions of these models retain strong performance while using less compute/memory. Overall, Rank1 shows that test-time compute allows for a fundamentally new type of explainable and performant reranker model for search.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.18418', 117)">Copy Link</button>

<div id="copy-message-117" class="copy-message"></div>

</div>

<div class="article">

<h1> cross <a href="https://arxiv.org/abs/2502.18449">SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution</a></h1>

<p><b>Authors:</b> Yuxiang Wei, Olivier Duchenne, Jade Copet, Quentin Carbonneaux, Lingming Zhang, Daniel Fried, Gabriel Synnaeve, Rishabh Singh, Sida I. Wang</p>

<p>Abstract: The recent DeepSeek-R1 release has demonstrated the immense potential of reinforcement learning (RL) in enhancing the general reasoning capabilities of large language models (LLMs). While DeepSeek-R1 and other follow-up work primarily focus on applying RL to competitive coding and math problems, this paper introduces SWE-RL, the first approach to scale RL-based LLM reasoning for real-world software engineering. Leveraging a lightweight rule-based reward (e.g., the similarity score between ground-truth and LLM-generated solutions), SWE-RL enables LLMs to autonomously recover a developer's reasoning processes and solutions by learning from extensive open-source software evolution data -- the record of a software's entire lifecycle, including its code snapshots, code changes, and events such as issues and pull requests. Trained on top of Llama 3, our resulting reasoning model, Llama3-SWE-RL-70B, achieves a 41.0% solve rate on SWE-bench Verified -- a human-verified collection of real-world GitHub issues. To our knowledge, this is the best performance reported for medium-sized (<100B) LLMs to date, even comparable to leading proprietary LLMs like GPT-4o. Surprisingly, despite performing RL solely on software evolution data, Llama3-SWE-RL has even emerged with generalized reasoning skills. For example, it shows improved results on five out-of-domain tasks, namely, function coding, library use, code reasoning, mathematics, and general language understanding, whereas a supervised-finetuning baseline even leads to performance degradation on average. Overall, SWE-RL opens up a new direction to improve the reasoning capabilities of LLMs through reinforcement learning on massive software engineering data.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2502.18449', 118)">Copy Link</button>

<div id="copy-message-118" class="copy-message"></div>

</div>

<div class="article">

<h1> replace <a href="https://arxiv.org/abs/2204.06970">Can Visual Dialogue Models Do Scorekeeping? Exploring How Dialogue Representations Incrementally Encode Shared Knowledge</a></h1>

<p><b>Authors:</b> Brielen Madureira, David Schlangen</p>

<p>Abstract: Cognitively plausible visual dialogue models should keep a mental scoreboard of shared established facts in the dialogue context. We propose a theory-based evaluation method for investigating to what degree models pretrained on the VisDial dataset incrementally build representations that appropriately do scorekeeping. Our conclusion is that the ability to make the distinction between shared and privately known statements along the dialogue is moderately present in the analysed models, but not always incrementally consistent, which may partially be due to the limited need for grounding interactions in the original task.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2204.06970', 119)">Copy Link</button>

<div id="copy-message-119" class="copy-message"></div>

</div>

<div class="article">

<h1> replace <a href="https://arxiv.org/abs/2308.14321">Leveraging Medical Knowledge Graphs Into Large Language Models for Diagnosis Prediction: Design and Application Study</a></h1>

<p><b>Authors:</b> Yanjun Gao, Ruizhe Li, Emma Croxford, John Caskey, Brian W Patterson, Matthew Churpek, Timothy Miller, Dmitriy Dligach, Majid Afshar</p>

<p>Abstract: Electronic Health Records (EHRs) and routine documentation practices play a vital role in patients' daily care, providing a holistic record of health, diagnoses, and treatment. However, complex and verbose EHR narratives overload healthcare providers, risking diagnostic inaccuracies. While Large Language Models (LLMs) have showcased their potential in diverse language tasks, their application in the healthcare arena needs to ensure the minimization of diagnostic errors and the prevention of patient harm. In this paper, we outline an innovative approach for augmenting the proficiency of LLMs in the realm of automated diagnosis generation, achieved through the incorporation of a medical knowledge graph (KG) and a novel graph model: Dr.Knows, inspired by the clinical diagnostic reasoning process. We derive the KG from the National Library of Medicine's Unified Medical Language System (UMLS), a robust repository of biomedical knowledge. Our method negates the need for pre-training and instead leverages the KG as an auxiliary instrument aiding in the interpretation and summarization of complex medical concepts. Using real-world hospital datasets, our experimental results demonstrate that the proposed approach of combining LLMs with KG has the potential to improve the accuracy of automated diagnosis generation. More importantly, our approach offers an explainable diagnostic pathway, edging us closer to the realization of AI-augmented diagnostic decision support systems.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2308.14321', 120)">Copy Link</button>

<div id="copy-message-120" class="copy-message"></div>

</div>

<div class="article">

<h1> replace <a href="https://arxiv.org/abs/2311.05769">Are Chatbots Reliable Text Annotators? Sometimes</a></h1>

<p><b>Authors:</b> Ross Deans Kristensen-McLachlan, Miceal Canavan, M\'arton Kardos, Mia Jacobsen, Lene Aar{\o}e</p>

<p>Abstract: Recent research highlights the significant potential of ChatGPT for text annotation in social science research. However, ChatGPT is a closed-source product which has major drawbacks with regards to transparency, reproducibility, cost, and data protection. Recent advances in open-source (OS) large language models (LLMs) offer an alternative without these drawbacks. Thus, it is important to evaluate the performance of OS LLMs relative to ChatGPT and standard approaches to supervised machine learning classification. We conduct a systematic comparative evaluation of the performance of a range of OS LLMs alongside ChatGPT, using both zero- and few-shot learning as well as generic and custom prompts, with results compared to supervised classification models. Using a new dataset of tweets from US news media, and focusing on simple binary text annotation tasks, we find significant variation in the performance of ChatGPT and OS models across the tasks, and that the supervised classifier using DistilBERT generally outperforms both. Given the unreliable performance of ChatGPT and the significant challenges it poses to Open Science we advise caution when using ChatGPT for substantive text annotation tasks.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2311.05769', 121)">Copy Link</button>

<div id="copy-message-121" class="copy-message"></div>

</div>

<div class="article">

<h1> replace <a href="https://arxiv.org/abs/2311.13708">Dynamic Fault Analysis in Substations Based on Knowledge Graphs</a></h1>

<p><b>Authors:</b> Weiwei Li, Xing Liu, Wei Wang, Lu Chen, Sizhe Li, Hui Fan</p>

<p>Abstract: To address the challenge of identifying hidden danger in substations from unstructured text, a novel dynamic analysis method is proposed. We first extract relevant information from the unstructured text, and then leverages a flexible distributed search engine built on Elastic-Search to handle the data. Following this, the hidden Markov model is employed to train the data within the engine. The Viterbi algorithm is integrated to decipher the hidden state sequences, facilitating the segmentation and labeling of entities related to hidden dangers. The final step involves using the Neo4j graph database to dynamically create a knowledge graph that visualizes hidden dangers in the substation. The effectiveness of the proposed method is demonstrated through a case analysis from a specific substation with hidden dangers revealed in the text records.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2311.13708', 122)">Copy Link</button>

<div id="copy-message-122" class="copy-message"></div>

</div>

<div class="article">

<h1> replace <a href="https://arxiv.org/abs/2402.11000">ASGEA: Exploiting Logic Rules from Align-Subgraphs for Entity Alignment</a></h1>

<p><b>Authors:</b> Yangyifei Luo, Zhuo Chen, Lingbing Guo, Qian Li, Wenxuan Zeng, Zhixin Cai, Jianxin Li</p>

<p>Abstract: Entity alignment (EA) aims to identify entities across different knowledge graphs that represent the same real-world objects. Recent embedding-based EA methods have achieved state-of-the-art performance in EA yet faced interpretability challenges as they purely rely on the embedding distance and neglect the logic rules behind a pair of aligned entities. In this paper, we propose the Align-Subgraph Entity Alignment (ASGEA) framework to exploit logic rules from Align-Subgraphs. ASGEA uses anchor links as bridges to construct Align-Subgraphs and spreads along the paths across KGs, which distinguishes it from the embedding-based methods. Furthermore, we design an interpretable Path-based Graph Neural Network, ASGNN, to effectively identify and integrate the logic rules across KGs. We also introduce a node-level multi-modal attention mechanism coupled with multi-modal enriched anchors to augment the Align-Subgraph. Our experimental results demonstrate the superior performance of ASGEA over the existing embedding-based methods in both EA and Multi-Modal EA (MMEA) tasks.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2402.11000', 123)">Copy Link</button>

<div id="copy-message-123" class="copy-message"></div>

</div>

<div class="article">

<h1> replace <a href="https://arxiv.org/abs/2405.11874">xFinder: Large Language Models as Automated Evaluators for Reliable Evaluation</a></h1>

<p><b>Authors:</b> Qingchen Yu, Zifan Zheng, Shichao Song, Zhiyu Li, Feiyu Xiong, Bo Tang, Ding Chen</p>

<p>Abstract: The continuous advancement of large language models (LLMs) has brought increasing attention to the critical issue of developing fair and reliable methods for evaluating their performance. Particularly, the emergence of cheating phenomena, such as test set leakage and prompt format overfitting, poses significant challenges to the reliable evaluation of LLMs. As evaluation frameworks commonly use Regular Expression (RegEx) for answer extraction, models may adjust their responses to fit formats easily handled by RegEx. Nevertheless, the key answer extraction module based on RegEx frequently suffers from extraction errors. Furthermore, recent studies proposing fine-tuned LLMs as judge models for automated evaluation face challenges in terms of generalization ability and fairness. This paper comprehensively analyzes the entire LLM evaluation chain and demonstrates that optimizing the key answer extraction module improves extraction accuracy and enhances evaluation reliability. Our findings suggest that improving the key answer extraction module can lead to higher judgment accuracy and improved evaluation efficiency compared to the judge models. To address these issues, we propose xFinder, a novel evaluator for answer extraction and matching in LLM evaluation. As part of this process, we create a specialized dataset, the \textbf{K}ey \textbf{A}nswer \textbf{F}inder (KAF) dataset, to ensure effective model training and evaluation. Generalization tests and real-world evaluations show that the smallest xFinder model, with only 500 million parameters, achieves an average extraction accuracy of 93.42\%. In contrast, RegEx accuracy in the best evaluation framework is 74.38\%. The final judgment accuracy of xFinder reaches 97.61\%, outperforming existing evaluation frameworks and judge models.</p>

<button class="copy-button" onclick="copyToClipboard('https://arxiv.org/abs/2405.11874', 124)">Copy Link</button>

<div id="copy-message-124" class="copy-message"></div>

[View remainder of file in raw view](https://github.com/Luvata/arxive/raw/9341b486ad647204168aa013996db66c9df8196a/pages/2025-02-26-cs-cl.html)

## Footer

[ ](https://github.com "GitHub") (C) 2025 GitHub, Inc. 

### Footer navigation

  * [Terms](https://docs.github.com/site-policy/github-terms/github-terms-of-service)
  * [Privacy](https://docs.github.com/site-policy/privacy-policies/github-privacy-statement)
  * [Security](https://github.com/security)
  * [Status](https://www.githubstatus.com/)
  * [Docs](https://docs.github.com/)
  * [Contact](https://support.github.com?tags=dotcom-footer)
  * Manage cookies 
  * Do not share my personal information 



You can’t perform that action at this time. 


scrape_date: 2025-03-17T13:26:50.821269
content_type: markdown
hash: 8c429e69085bcf152c06ae62d2fe24c726fc1ad9ec896664e724f2aaef5ffe18
url:
https://github.com/Luvata/arxive/blob/9341b486ad647204168aa013996db66c9df8196a/pages/2023-11-20-cs-cl.html

title:
arxive/pages/2023-11-20-cs-cl.html at 9341b486ad647204168aa013996db66c9df8196a · Luvata/arxive · GitHub

content:
Skip to content

## Navigation Menu

Toggle navigation

[ ](/)

[ Sign in ](/login?return_to=https%3A%2F%2Fgithub.com%2FLuvata%2Farxive%2Fblob%2F9341b486ad647204168aa013996db66c9df8196a%2Fpages%2F2023-11-20-cs-cl.html)

  * Product 

    * [ GitHub Copilot Write better code with AI  ](https://github.com/features/copilot)
    * [ Security Find and fix vulnerabilities  ](https://github.com/features/security)
    * [ Actions Automate any workflow  ](https://github.com/features/actions)
    * [ Codespaces Instant dev environments  ](https://github.com/features/codespaces)
    * [ Issues Plan and track work  ](https://github.com/features/issues)
    * [ Code Review Manage code changes  ](https://github.com/features/code-review)
    * [ Discussions Collaborate outside of code  ](https://github.com/features/discussions)
    * [ Code Search Find more, search less  ](https://github.com/features/code-search)

Explore
    * [ All features ](https://github.com/features)
    * [ Documentation  ](https://docs.github.com)
    * [ GitHub Skills  ](https://skills.github.com)
    * [ Blog  ](https://github.blog)

  * Solutions 

By company size
    * [ Enterprises ](https://github.com/enterprise)
    * [ Small and medium teams ](https://github.com/team)
    * [ Startups ](https://github.com/enterprise/startups)
    * [ Nonprofits ](/solutions/industry/nonprofits)

By use case
    * [ DevSecOps ](/solutions/use-case/devsecops)
    * [ DevOps ](/solutions/use-case/devops)
    * [ CI/CD ](/solutions/use-case/ci-cd)
    * [ View all use cases ](/solutions/use-case)

By industry
    * [ Healthcare ](/solutions/industry/healthcare)
    * [ Financial services ](/solutions/industry/financial-services)
    * [ Manufacturing ](/solutions/industry/manufacturing)
    * [ Government ](/solutions/industry/government)
    * [ View all industries ](/solutions/industry)

[ View all solutions  ](/solutions)

  * Resources 

Topics
    * [ AI ](/resources/articles/ai)
    * [ DevOps ](/resources/articles/devops)
    * [ Security ](/resources/articles/security)
    * [ Software Development ](/resources/articles/software-development)
    * [ View all ](/resources/articles)

Explore
    * [ Learning Pathways  ](https://resources.github.com/learn/pathways)
    * [ Events & Webinars  ](https://resources.github.com)
    * [ Ebooks & Whitepapers ](https://github.com/resources/whitepapers)
    * [ Customer Stories ](https://github.com/customer-stories)
    * [ Partners  ](https://partner.github.com)
    * [ Executive Insights ](https://github.com/solutions/executive-insights)

  * Open Source 

    * [ GitHub Sponsors Fund open source developers  ](/sponsors)

    * [ The ReadME Project GitHub community articles  ](https://github.com/readme)

Repositories
    * [ Topics ](https://github.com/topics)
    * [ Trending ](https://github.com/trending)
    * [ Collections ](https://github.com/collections)

  * Enterprise 

    * [ Enterprise platform AI-powered developer platform  ](/enterprise)

Available add-ons
    * [ Advanced Security Enterprise-grade security features  ](https://github.com/enterprise/advanced-security)
    * [ Copilot for business Enterprise-grade AI features  ](/features/copilot/copilot-business)
    * [ Premium Support Enterprise-grade 24/7 support  ](/premium-support)

  * [Pricing](https://github.com/pricing)



Search or jump to...

# Search code, repositories, users, issues, pull requests...

Search 

Clear




[Search syntax tips](https://docs.github.com/search-github/github-code-search/understanding-github-code-search-syntax)

#  Provide feedback 

We read every piece of feedback, and take your input very seriously.

Include my email address so I can be contacted

Cancel  Submit feedback 

#  Saved searches 

## Use saved searches to filter your results more quickly

Name

Query

To see all available qualifiers, see our [documentation](https://docs.github.com/search-github/github-code-search/understanding-github-code-search-syntax). 

Cancel  Create saved search 

[ Sign in ](/login?return_to=https%3A%2F%2Fgithub.com%2FLuvata%2Farxive%2Fblob%2F9341b486ad647204168aa013996db66c9df8196a%2Fpages%2F2023-11-20-cs-cl.html)

[ Sign up ](/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fblob%2Fshow&source=header-repo&source_repo=Luvata%2Farxive) Reseting focus

You signed in with another tab or window. [Reload]() to refresh your session. You signed out in another tab or window. [Reload]() to refresh your session. You switched accounts on another tab or window. [Reload]() to refresh your session. Dismiss alert

{{ message }}

[ Luvata ](/Luvata) / **[arxive](/Luvata/arxive) ** Public

  * [ Notifications ](/login?return_to=%2FLuvata%2Farxive) You must be signed in to change notification settings
  * [ Fork 1 ](/login?return_to=%2FLuvata%2Farxive)
  * [ Star  11 ](/login?return_to=%2FLuvata%2Farxive)




  * [ Code ](/Luvata/arxive)
  * [ Issues 0 ](/Luvata/arxive/issues)
  * [ Pull requests 0 ](/Luvata/arxive/pulls)
  * [ Actions ](/Luvata/arxive/actions)
  * [ Projects 0 ](/Luvata/arxive/projects)
  * [ Security ](/Luvata/arxive/security)
  * [ Insights ](/Luvata/arxive/pulse)



Additional navigation options

  * [ Code  ](/Luvata/arxive)
  * [ Issues  ](/Luvata/arxive/issues)
  * [ Pull requests  ](/Luvata/arxive/pulls)
  * [ Actions  ](/Luvata/arxive/actions)
  * [ Projects  ](/Luvata/arxive/projects)
  * [ Security  ](/Luvata/arxive/security)
  * [ Insights  ](/Luvata/arxive/pulse)



## Files

9341b48

## Breadcrumbs

  1. [arxive](/Luvata/arxive/tree/9341b486ad647204168aa013996db66c9df8196a)
  2. /[pages](/Luvata/arxive/tree/9341b486ad647204168aa013996db66c9df8196a/pages)



/

# 2023-11-20-cs-cl.html

Copy path

Blame

Blame

## Latest commit

## History

[History](/Luvata/arxive/commits/9341b486ad647204168aa013996db66c9df8196a/pages/2023-11-20-cs-cl.html)

[](/Luvata/arxive/commits/9341b486ad647204168aa013996db66c9df8196a/pages/2023-11-20-cs-cl.html)

4132 lines (4129 loc) · 337 KB

9341b48

## Breadcrumbs

  1. [arxive](/Luvata/arxive/tree/9341b486ad647204168aa013996db66c9df8196a)
  2. /[pages](/Luvata/arxive/tree/9341b486ad647204168aa013996db66c9df8196a/pages)



/

# 2023-11-20-cs-cl.html

Top

## File metadata and controls

  * Code

  * Blame




4132 lines (4129 loc) · 337 KB

[Raw](https://github.com/Luvata/arxive/raw/9341b486ad647204168aa013996db66c9df8196a/pages/2023-11-20-cs-cl.html)

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465

466

467

468

469

470

471

472

473

474

475

476

477

478

479

480

481

482

483

484

485

486

487

488

489

490

491

492

493

494

495

496

497

498

499

500

501

502

503

504

505

506

507

508

509

510

511

512

513

514

515

516

517

518

519

520

521

522

523

524

525

526

527

528

529

530

531

532

533

534

535

536

537

538

539

540

541

542

543

544

545

546

547

548

549

550

551

552

553

554

555

556

557

558

559

560

561

562

563

564

565

566

567

568

569

570

571

572

573

574

575

576

577

578

579

580

581

582

583

584

585

586

587

588

589

590

591

592

593

594

595

596

597

598

599

600

601

602

603

604

605

606

607

608

609

610

611

612

613

614

615

616

617

618

619

620

621

622

623

624

625

626

627

628

629

630

631

632

633

634

635

636

637

638

639

640

641

642

643

644

645

646

647

648

649

650

651

652

653

654

655

656

657

658

659

660

661

662

663

664

665

666

667

668

669

670

671

672

673

674

675

676

677

678

679

680

681

682

683

684

685

686

687

688

689

690

691

692

693

694

695

696

697

698

699

700

701

702

703

704

705

706

707

708

709

710

711

712

713

714

715

716

717

718

719

720

721

722

723

724

725

726

727

728

729

730

731

732

733

734

735

736

737

738

739

740

741

742

743

744

745

746

747

748

749

750

751

752

753

754

755

756

757

758

759

760

761

762

763

764

765

766

767

768

769

770

771

772

773

774

775

776

777

778

779

780

781

782

783

784

785

786

787

788

789

790

791

792

793

794

795

796

797

798

799

800

801

802

803

804

805

806

807

808

809

810

811

812

813

814

815

816

817

818

819

820

821

822

823

824

825

826

827

828

829

830

831

832

833

834

835

836

837

838

839

840

841

842

843

844

845

846

847

848

849

850

851

852

853

854

855

856

857

858

859

860

861

862

863

864

865

866

867

868

869

870

871

872

873

874

875

876

877

878

879

880

881

882

883

884

885

886

887

888

889

890

891

892

893

894

895

896

897

898

899

900

901

902

903

904

905

906

907

908

909

910

911

912

913

914

915

916

917

918

919

920

921

922

923

924

925

926

927

928

929

930

931

932

933

934

935

936

937

938

939

940

941

942

943

944

945

946

947

948

949

950

951

952

953

954

955

956

957

958

959

960

961

962

963

964

965

966

967

968

969

970

971

972

973

974

975

976

977

978

979

980

981

982

983

984

985

986

987

988

989

990

991

992

993

994

995

996

997

998

999

1000

<!DOCTYPE html>

<html>

<head>

<title>2023-11-20-cs-cl</title>

<style>

body {

font-family: Arial, sans-serif;

margin: 0;

padding: 0;

display: flex;

justify-content: center;

background-color: #f2f2f2;

}

.container {

width: 60%;

max-width: 900px;

min-width: 300px;

background-color: #fff;

padding: 20px;

margin: 20px;

border-radius: 10px;

box-shadow: 0px 0px 10px rgba(0, 0, 0, 0.05);

font-size: 0.9em;

}

h1 {

color: #333;

font-size: 1.6em;

}

.article {

border-bottom: 1px solid #ddd;

padding: 10px 0;

}

.article a {

color: #007BFF;

text-decoration: none;

}

.article a:hover {

color: #0056b3;

}

@media (max-width: 768px) {

.container {

width: 90%;

font-size: 0.8em;

}

h1 {

font-size: 1.4em;

}

}

</style>

</head>

<body>

<div class="container">

<div class="article">

<h1><a href="http://arxiv.org/abs/2311.09224">The Cybersecurity Crisis of Artificial Intelligence: Unrestrained Adoption and Natural Language-Based Attacks. (arXiv:2311.09224v1 [cs.CY])</a></h1>

<p><b>Authors:</b> <a href="http://arxiv.org/find/cs/1/au:+Tsamados_A/0/1/0/all/0/1">Andreas Tsamados</a>, <a href="http://arxiv.org/find/cs/1/au:+Floridi_L/0/1/0/all/0/1">Luciano Floridi</a>, <a href="http://arxiv.org/find/cs/1/au:+Taddeo_M/0/1/0/all/0/1">Mariarosaria Taddeo</a></p>

<p>The widespread integration of autoregressive-large language models (AR-LLMs),

such as ChatGPT, across established applications, like search engines, has

introduced critical vulnerabilities with uniquely scalable characteristics. In

this commentary, we analyse these vulnerabilities, their dependence on natural

language as a vector of attack, and their challenges to cybersecurity best

practices. We offer recommendations designed to mitigate these challenges.

</p>

</p>

</div>

<div class="article">

<h1><a href="http://arxiv.org/abs/2311.09252">In the Red(dit): Social Media and Stock Prices. (arXiv:2311.09252v1 [cs.SI])</a></h1>

<p><b>Authors:</b> <a href="http://arxiv.org/find/cs/1/au:+Baker_J/0/1/0/all/0/1">James Baker</a></p>

<p>Spearheaded by retail traders on the website reddit, the GameStop short

squeeze of early 2021 shows that social media embeds information that

correlates with market movements. This paper seeks to examine this relationship

by using daily frequencies of classified comments and buzzwords as additional

factors in a Fama-French three factor model. Comments are classified using an

unsupervised clustering method, while past studies have used pretrained models

that are not specific to the domains being studied.

</p>

</p>

</div>

<div class="article">

<h1><a href="http://arxiv.org/abs/2311.09263">Auto-ICL: In-Context Learning without Human Supervision. (arXiv:2311.09263v1 [cs.LG])</a></h1>

<p><b>Authors:</b> <a href="http://arxiv.org/find/cs/1/au:+Yang_J/0/1/0/all/0/1">Jinghan Yang</a>, <a href="http://arxiv.org/find/cs/1/au:+Ma_S/0/1/0/all/0/1">Shuming Ma</a>, <a href="http://arxiv.org/find/cs/1/au:+Wei_F/0/1/0/all/0/1">Furu Wei</a></p>

<p>In the era of Large Language Models (LLMs), human-computer interaction has

evolved towards natural language, offering unprecedented flexibility. Despite

this, LLMs are heavily reliant on well-structured prompts to function

efficiently within the realm of In-Context Learning. Vanilla In-Context

Learning relies on human-provided contexts, such as labeled examples, explicit

instructions, or other guiding mechanisms that shape the model's outputs. To

address this challenge, our study presents a universal framework named

Automatic In-Context Learning. Upon receiving a user's request, we ask the

model to independently generate examples, including labels, instructions, or

reasoning pathways. The model then leverages this self-produced context to

tackle the given problem. Our approach is universally adaptable and can be

implemented in any setting where vanilla In-Context Learning is applicable. We

demonstrate that our method yields strong performance across a range of tasks,

standing up well when compared to existing methods.

</p>

</p>

</div>

<div class="article">

<h1><a href="http://arxiv.org/abs/2311.09277">Contrastive Chain-of-Thought Prompting. (arXiv:2311.09277v1 [cs.CL])</a></h1>

<p><b>Authors:</b> <a href="http://arxiv.org/find/cs/1/au:+Chia_Y/0/1/0/all/0/1">Yew Ken Chia</a>, <a href="http://arxiv.org/find/cs/1/au:+Chen_G/0/1/0/all/0/1">Guizhen Chen</a>, <a href="http://arxiv.org/find/cs/1/au:+Tuan_L/0/1/0/all/0/1">Luu Anh Tuan</a>, <a href="http://arxiv.org/find/cs/1/au:+Poria_S/0/1/0/all/0/1">Soujanya Poria</a>, <a href="http://arxiv.org/find/cs/1/au:+Bing_L/0/1/0/all/0/1">Lidong Bing</a></p>

<p>Despite the success of chain of thought in enhancing language model

reasoning, the underlying process remains less well understood. Although

logically sound reasoning appears inherently crucial for chain of thought,

prior studies surprisingly reveal minimal impact when using invalid

demonstrations instead. Furthermore, the conventional chain of thought does not

inform language models on what mistakes to avoid, which potentially leads to

more errors. Hence, inspired by how humans can learn from both positive and

negative examples, we propose contrastive chain of thought to enhance language

model reasoning. Compared to the conventional chain of thought, our approach

provides both valid and invalid reasoning demonstrations, to guide the model to

reason step-by-step while reducing reasoning mistakes. To improve

generalization, we introduce an automatic method to construct contrastive

demonstrations. Our experiments on reasoning benchmarks demonstrate that

contrastive chain of thought can serve as a general enhancement of

chain-of-thought prompting.

</p>

</p>

</div>

<div class="article">

<h1><a href="http://arxiv.org/abs/2311.09278">Symbol-LLM: Towards Foundational Symbol-centric Interface For Large Language Models. (arXiv:2311.09278v1 [cs.CL])</a></h1>

<p><b>Authors:</b> <a href="http://arxiv.org/find/cs/1/au:+Xu_F/0/1/0/all/0/1">Fangzhi Xu</a>, <a href="http://arxiv.org/find/cs/1/au:+Wu_Z/0/1/0/all/0/1">Zhiyong Wu</a>, <a href="http://arxiv.org/find/cs/1/au:+Sun_Q/0/1/0/all/0/1">Qiushi Sun</a>, <a href="http://arxiv.org/find/cs/1/au:+Ren_S/0/1/0/all/0/1">Siyu Ren</a>, <a href="http://arxiv.org/find/cs/1/au:+Yuan_F/0/1/0/all/0/1">Fei Yuan</a>, <a href="http://arxiv.org/find/cs/1/au:+Yuan_S/0/1/0/all/0/1">Shuai Yuan</a>, <a href="http://arxiv.org/find/cs/1/au:+Lin_Q/0/1/0/all/0/1">Qika Lin</a>, <a href="http://arxiv.org/find/cs/1/au:+Qiao_Y/0/1/0/all/0/1">Yu Qiao</a>, <a href="http://arxiv.org/find/cs/1/au:+Liu_J/0/1/0/all/0/1">Jun Liu</a></p>

<p>Large Language Models (LLMs) have greatly propelled the progress in natural

language(NL)-centric tasks based on NL interface. However, the NL form is not

enough for world knowledge. Current works focus on this question by injecting

specific symbolic knowledge into LLM, which ignore two critical challenges: the

interrelations between various symbols and the balance between symbolic-centric

and NL-centric capabilities. In this work, we tackle these challenges from both

a data and framework perspective and introduce Symbol-LLM series models. First,

we collect 34 symbolic tasks, covering ~20 different forms, which are unified

to capture symbol interrelations. Then, a two-stage tuning framework succeeds

in injecting symbolic knowledge without loss of the generality ability.

Extensive experiments on both symbol- and NL-centric tasks demonstrate the

balanced and superior performances of Symbol-LLM series models.

</p>

</p>

</div>

<div class="article">

<h1><a href="http://arxiv.org/abs/2311.09308">Divergences between Language Models and Human Brains. (arXiv:2311.09308v1 [cs.CL])</a></h1>

<p><b>Authors:</b> <a href="http://arxiv.org/find/cs/1/au:+Zhou_Y/0/1/0/all/0/1">Yuchen Zhou</a>, <a href="http://arxiv.org/find/cs/1/au:+Liu_E/0/1/0/all/0/1">Emmy Liu</a>, <a href="http://arxiv.org/find/cs/1/au:+Neubig_G/0/1/0/all/0/1">Graham Neubig</a>, <a href="http://arxiv.org/find/cs/1/au:+Wehbe_L/0/1/0/all/0/1">Leila Wehbe</a></p>

<p>Do machines and humans process language in similar ways? A recent line of

research has hinted in the affirmative, demonstrating that human brain signals

can be effectively predicted using the internal representations of language

models (LMs). This is thought to reflect shared computational principles

between LMs and human language processing. However, there are also clear

differences in how LMs and humans acquire and use language, even if the final

task they are performing is the same. Despite this, there is little work

exploring systematic differences between human and machine language processing

using brain data. To address this question, we examine the differences between

LM representations and the human brain's responses to language, specifically by

examining a dataset of Magnetoencephalography (MEG) responses to a written

narrative. In doing so we identify three phenomena that, in prior work, LMs

have been found to not capture well: emotional understanding, figurative

language processing, and physical commonsense. By fine-tuning LMs on datasets

related to these phenomena, we observe that fine-tuned LMs show improved

alignment with human brain responses across these tasks. Our study implies that

the observed divergences between LMs and human brains may stem from LMs'

inadequate representation of these specific types of knowledge.

</p>

</p>

</div>

<div class="article">

<h1><a href="http://arxiv.org/abs/2311.09319">Spoken Word2Vec: A Perspective And Some Techniques. (arXiv:2311.09319v1 [cs.CL])</a></h1>

<p><b>Authors:</b> <a href="http://arxiv.org/find/cs/1/au:+Sayeed_M/0/1/0/all/0/1">Mohammad Amaan Sayeed</a>, <a href="http://arxiv.org/find/cs/1/au:+Aldarmaki_H/0/1/0/all/0/1">Hanan Aldarmaki</a></p>

<p>Text word embeddings that encode distributional semantic features work by

modeling contextual similarities of frequently occurring words. Acoustic word

embeddings, on the other hand, typically encode low-level phonetic

similarities. Semantic embeddings for spoken words have been previously

explored using similar algorithms to Word2Vec, but the resulting vectors still

mainly encoded phonetic rather than semantic features. In this paper, we

examine the assumptions and architectures used in previous works and show

experimentally how Word2Vec algorithms fail to encode distributional semantics

when the input units are acoustically correlated. In addition, previous works

relied on the simplifying assumptions of perfect word segmentation and

clustering by word type. Given these conditions, a trivial solution identical

to text-based embeddings has been overlooked. We follow this simpler path using

automatic word type clustering and examine the effects on the resulting

embeddings, highlighting the true challenges in this task.

</p>

</p>

</div>

<div class="article">

<h1><a href="http://arxiv.org/abs/2311.09325">Improving fit to human reading times via temperature-scaled surprisal. (arXiv:2311.09325v1 [cs.CL])</a></h1>

<p><b>Authors:</b> <a href="http://arxiv.org/find/cs/1/au:+Liu_T/0/1/0/all/0/1">Tong Liu</a>, <a href="http://arxiv.org/find/cs/1/au:+Skrjanec_I/0/1/0/all/0/1">Iza &#x160;krjanec</a>, <a href="http://arxiv.org/find/cs/1/au:+Demberg_V/0/1/0/all/0/1">Vera Demberg</a></p>

<p>Past studies have provided broad support for that words with lower

predictability (i.e., higher surprisal) require more time for comprehension by

using large language models (LLMs) to simulate humans' cognitive load. In

general, these studies have implicitly assumed that the probability scores from

LLMs are accurate, ignoring the discrepancies between human cognition and LLMs

from this standpoint. Inspired by the concept of probability calibration, we

are the first work to focus on the probability distribution for human reading

simulation. We propose to use temperature-scaled surprisal, a surprisal

calculated by shaped probability, to be the predictor of human reading times.

Our results across three corpora consistently revealed that such a surprisal

can drastically improve the prediction of reading times. Setting the

temperature to be approximately 2.5 across all models and datasets can yield up

to an 89% of increase in delta log-likelihood in our setting. We also propose a

calibration metric to quantify the possible human-likeness bias. Further

analysis was done and provided insights into this phenomenon.

</p>

</p>

</div>

<div class="article">

<h1><a href="http://arxiv.org/abs/2311.09335">Lighter, yet More Faithful: Investigating Hallucinations in Pruned Large Language Models for Abstractive Summarization. (arXiv:2311.09335v1 [cs.CL])</a></h1>

<p><b>Authors:</b> <a href="http://arxiv.org/find/cs/1/au:+Chrysostomou_G/0/1/0/all/0/1">George Chrysostomou</a>, <a href="http://arxiv.org/find/cs/1/au:+Zhao_Z/0/1/0/all/0/1">Zhixue Zhao</a>, <a href="http://arxiv.org/find/cs/1/au:+Williams_M/0/1/0/all/0/1">Miles Williams</a>, <a href="http://arxiv.org/find/cs/1/au:+Aletras_N/0/1/0/all/0/1">Nikolaos Aletras</a></p>

<p>Despite their remarkable performance on abstractive summarization, large

language models (LLMs) face two significant challenges: their considerable size

and tendency to hallucinate. Hallucinations are concerning because they erode

the reliability of LLMs and raise safety issues. Pruning is a technique that

reduces model size by removing redundant weights to create sparse models that

enable more efficient inference. Pruned models yield comparable performance to

their counterpart full-sized models, making them ideal alternatives when

operating on a limited budget. However, the effect that pruning has upon

hallucinations in abstractive summarization with LLMs has yet to be explored.

In this paper, we provide an extensive empirical study on the hallucinations

produced by pruned models across three standard summarization tasks, two

pruning approaches, three instruction-tuned LLMs, and three hallucination

evaluation metrics. Surprisingly, we find that pruned LLMs hallucinate less

compared to their full-sized counterparts. Our follow-up analysis suggests that

pruned models tend to depend more on the source input and less on their

parametric knowledge from pre-training for generation. This greater dependency

on the source input leads to a higher lexical overlap between generated content

and the source input, which can be a reason for the reduction in

hallucinations.

</p>

</p>

</div>

<div class="article">

<h1><a href="http://arxiv.org/abs/2311.09336">Pinpoint, Not Criticize: Refining Large Language Models via Fine-Grained Actionable Feedback. (arXiv:2311.09336v1 [cs.CL])</a></h1>

<p><b>Authors:</b> <a href="http://arxiv.org/find/cs/1/au:+Xu_W/0/1/0/all/0/1">Wenda Xu</a>, <a href="http://arxiv.org/find/cs/1/au:+Deutsch_D/0/1/0/all/0/1">Daniel Deutsch</a>, <a href="http://arxiv.org/find/cs/1/au:+Finkelstein_M/0/1/0/all/0/1">Mara Finkelstein</a>, <a href="http://arxiv.org/find/cs/1/au:+Juraska_J/0/1/0/all/0/1">Juraj Juraska</a>, <a href="http://arxiv.org/find/cs/1/au:+Zhang_B/0/1/0/all/0/1">Biao Zhang</a>, <a href="http://arxiv.org/find/cs/1/au:+Liu_Z/0/1/0/all/0/1">Zhongtao Liu</a>, <a href="http://arxiv.org/find/cs/1/au:+Wang_W/0/1/0/all/0/1">William Yang Wang</a>, <a href="http://arxiv.org/find/cs/1/au:+Li_L/0/1/0/all/0/1">Lei Li</a>, <a href="http://arxiv.org/find/cs/1/au:+Freitag_M/0/1/0/all/0/1">Markus Freitag</a></p>

<p>Recent improvements in text generation have leveraged human feedback to

improve the quality of the generated output. However, human feedback is not

always available, especially during inference. In this work, we propose an

inference time optimization method FITO to use fine-grained actionable feedback

in the form of error type, error location and severity level that are predicted

by a learned error pinpoint model for iterative refinement. FITO starts with an

initial output, then iteratively incorporates the feedback via a refinement

model that generates an improved output conditioned on the feedback. Given the

uncertainty of consistent refined samples at iterative steps, we formulate

iterative refinement into a local search problem and develop a simulated

annealing based algorithm that balances exploration of the search space and

optimization for output quality. We conduct experiments on three text

generation tasks, including machine translation, long-form question answering

(QA) and topical summarization. We observe 0.8 and 0.7 MetricX gain on

Chinese-English and English-German translation, 4.5 and 1.8 ROUGE-L gain at

long form QA and topic summarization respectively, with a single iteration of

refinement. With our simulated annealing algorithm, we see further quality

improvements, including up to 1.7 MetricX improvements over the baseline

approach.

</p>

</p>

</div>

<div class="article">

<h1><a href="http://arxiv.org/abs/2311.09344">Language and Task Arithmetic with Parameter-Efficient Layers for Zero-Shot Summarization. (arXiv:2311.09344v1 [cs.CL])</a></h1>

<p><b>Authors:</b> <a href="http://arxiv.org/find/cs/1/au:+Chronopoulou_A/0/1/0/all/0/1">Alexandra Chronopoulou</a>, <a href="http://arxiv.org/find/cs/1/au:+Pfeiffer_J/0/1/0/all/0/1">Jonas Pfeiffer</a>, <a href="http://arxiv.org/find/cs/1/au:+Maynez_J/0/1/0/all/0/1">Joshua Maynez</a>, <a href="http://arxiv.org/find/cs/1/au:+Wang_X/0/1/0/all/0/1">Xinyi Wang</a>, <a href="http://arxiv.org/find/cs/1/au:+Ruder_S/0/1/0/all/0/1">Sebastian Ruder</a>, <a href="http://arxiv.org/find/cs/1/au:+Agrawal_P/0/1/0/all/0/1">Priyanka Agrawal</a></p>

<p>Parameter-efficient fine-tuning (PEFT) using labeled task data can

significantly improve the performance of large language models (LLMs) on the

downstream task. However, there are 7000 languages in the world and many of

these languages lack labeled data for real-world language generation tasks. In

this paper, we propose to improve zero-shot cross-lingual transfer by composing

language or task specialized parameters. Our method composes language and task

PEFT modules via element-wise arithmetic operations to leverage unlabeled data

and English labeled data. We extend our approach to cases where labeled data

from more languages is available and propose to arithmetically compose PEFT

modules trained on languages related to the target. Empirical results on

summarization demonstrate that our method is an effective strategy that obtains

consistent gains using minimal training of PEFT modules.

</p>

</p>

</div>

<div class="article">

<h1><a href="http://arxiv.org/abs/2311.09356">LePaRD: A Large-Scale Dataset of Judges Citing Precedents. (arXiv:2311.09356v1 [cs.CL])</a></h1>

<p><b>Authors:</b> <a href="http://arxiv.org/find/cs/1/au:+Mahari_R/0/1/0/all/0/1">Robert Mahari</a>, <a href="http://arxiv.org/find/cs/1/au:+Stammbach_D/0/1/0/all/0/1">Dominik Stammbach</a>, <a href="http://arxiv.org/find/cs/1/au:+Ash_E/0/1/0/all/0/1">Elliott Ash</a>, <a href="http://arxiv.org/find/cs/1/au:+Pentland_A/0/1/0/all/0/1">Alex `Sandy&#x27; Pentland</a></p>

<p>We present the Legal Passage Retrieval Dataset LePaRD. LePaRD is a massive

collection of U.S. federal judicial citations to precedent in context. The

dataset aims to facilitate work on legal passage prediction, a challenging

practice-oriented legal retrieval and reasoning task. Legal passage prediction

seeks to predict relevant passages from precedential court decisions given the

context of a legal argument. We extensively evaluate various retrieval

approaches on LePaRD, and find that classification appears to work best.

However, we note that legal precedent prediction is a difficult task, and there

remains significant room for improvement. We hope that by publishing LePaRD, we

will encourage others to engage with a legal NLP task that promises to help

expand access to justice by reducing the burden associated with legal research.

A subset of the LePaRD dataset is freely available and the whole dataset will

be released upon publication.

</p>

</p>

</div>

<div class="article">

<h1><a href="http://arxiv.org/abs/2311.09358">Empirical evaluation of Uncertainty Quantification in Retrieval-Augmented Language Models for Science. (arXiv:2311.09358v1 [cs.CL])</a></h1>

<p><b>Authors:</b> <a href="http://arxiv.org/find/cs/1/au:+Wagle_S/0/1/0/all/0/1">Sridevi Wagle</a>, <a href="http://arxiv.org/find/cs/1/au:+Munikoti_S/0/1/0/all/0/1">Sai Munikoti</a>, <a href="http://arxiv.org/find/cs/1/au:+Acharya_A/0/1/0/all/0/1">Anurag Acharya</a>, <a href="http://arxiv.org/find/cs/1/au:+Smith_S/0/1/0/all/0/1">Sara Smith</a>, <a href="http://arxiv.org/find/cs/1/au:+Horawalavithana_S/0/1/0/all/0/1">Sameera Horawalavithana</a></p>

<p>Large language models (LLMs) have shown remarkable achievements in natural

language processing tasks, producing high-quality outputs. However, LLMs still

exhibit limitations, including the generation of factually incorrect

information. In safety-critical applications, it is important to assess the

confidence of LLM-generated content to make informed decisions. Retrieval

Augmented Language Models (RALMs) is relatively a new area of research in NLP.

RALMs offer potential benefits for scientific NLP tasks, as retrieved

documents, can serve as evidence to support model-generated content. This

inclusion of evidence enhances trustworthiness, as users can verify and explore

the retrieved documents to validate model outputs. Quantifying uncertainty in

RALM generations further improves trustworthiness, with retrieved text and

confidence scores contributing to a comprehensive and reliable model for

scientific applications. However, there is limited to no research on UQ for

RALMs, particularly in scientific contexts. This study aims to address this gap

by conducting a comprehensive evaluation of UQ in RALMs, focusing on scientific

tasks. This research investigates how uncertainty scores vary when scientific

knowledge is incorporated as pretraining and retrieval data and explores the

relationship between uncertainty scores and the accuracy of model-generated

outputs. We observe that an existing RALM finetuned with scientific knowledge

as the retrieval data tends to be more confident in generating predictions

compared to the model pretrained only with scientific knowledge. We also found

that RALMs are overconfident in their predictions, making inaccurate

predictions more confidently than accurate ones. Scientific knowledge provided

either as pretraining or retrieval corpus does not help alleviate this issue.

We released our code, data and dashboards at https://github.com/pnnl/EXPERT2.

</p>

</p>

</div>

<div class="article">

<h1><a href="http://arxiv.org/abs/2311.09363">Investigating the Emergent Audio Classification Ability of ASR Foundation Models. (arXiv:2311.09363v1 [cs.CL])</a></h1>

<p><b>Authors:</b> <a href="http://arxiv.org/find/cs/1/au:+Ma_R/0/1/0/all/0/1">Rao Ma</a>, <a href="http://arxiv.org/find/cs/1/au:+Liusie_A/0/1/0/all/0/1">Adian Liusie</a>, <a href="http://arxiv.org/find/cs/1/au:+Gales_M/0/1/0/all/0/1">Mark J. F. Gales</a>, <a href="http://arxiv.org/find/cs/1/au:+Knill_K/0/1/0/all/0/1">Kate M. Knill</a></p>

<p>Text and vision foundation models can perform many tasks in a zero-shot

setting, a desirable property that enables these systems to be applied in

general and low-resource settings. However, there has been significantly less

work on the zero-shot abilities of ASR foundation models, with these systems

typically fine-tuned to specific tasks or constrained to applications that

match their training criterion and data annotation. In this work we investigate

the ability of Whisper and MMS, ASR foundation models trained primarily for

speech recognition, to perform zero-shot audio classification. We use simple

template-based text prompts at the decoder and use the resulting decoding

probabilities to generate zero-shot predictions. Without training the model on

extra data or adding any new parameters, we demonstrate that Whisper shows

promising zero-shot classification performance on a range of 8

audio-classification datasets, outperforming existing state-of-the-art

zero-shot baseline's accuracy by an average of 9%. One important step to unlock

the emergent ability is debiasing, where a simple unsupervised reweighting

method of the class probabilities yields consistent significant performance

gains. We further show that performance increases with model size, implying

that as ASR foundation models scale up, they may exhibit improved zero-shot

performance.

</p>

</p>

</div>

<div class="article">

<h1><a href="http://arxiv.org/abs/2311.09366">LOKE: Linked Open Knowledge Extraction for Automated Knowledge Graph Construction. (arXiv:2311.09366v1 [cs.CL])</a></h1>

<p><b>Authors:</b> <a href="http://arxiv.org/find/cs/1/au:+McCusker_J/0/1/0/all/0/1">Jamie McCusker</a></p>

<p>While the potential of Open Information Extraction (Open IE) for Knowledge

Graph Construction (KGC) may seem promising, we find that the alignment of Open

IE extraction results with existing knowledge graphs to be inadequate. The

advent of Large Language Models (LLMs), especially the commercially available

OpenAI models, have reset expectations for what is possible with deep learning

models and have created a new field called prompt engineering. We investigate

the use of GPT models and prompt engineering for knowledge graph construction

with the Wikidata knowledge graph to address a similar problem to Open IE,

which we call Open Knowledge Extraction (OKE) using an approach we call the

Linked Open Knowledge Extractor (LOKE, pronounced like "Loki"). We consider the

entity linking task essential to construction of real world knowledge graphs.

We merge the CaRB benchmark scoring approach with data from the TekGen dataset

for the LOKE task. We then show that a well engineered prompt, paired with a

naive entity linking approach (which we call LOKE-GPT), outperforms AllenAI's

OpenIE 4 implementation on the OKE task, although it over-generates triples

compared to the reference set due to overall triple scarcity in the TekGen set.

Through an analysis of entity linkability in the CaRB dataset, as well as

outputs from OpenIE 4 and LOKE-GPT, we see that LOKE-GPT and the "silver"

TekGen triples show that the task is significantly different in content from

OIE, if not structure. Through this analysis and a qualitative analysis of

sentence extractions via all methods, we found that LOKE-GPT extractions are of

high utility for the KGC task and suitable for use in semi-automated extraction

settings.

</p>

</p>

</div>

<div class="article">

<h1><a href="http://arxiv.org/abs/2311.09367">A Survey on Online User Aggression: Content Detection and Behavioural Analysis on Social Media Platforms. (arXiv:2311.09367v1 [cs.CL])</a></h1>

<p><b>Authors:</b> <a href="http://arxiv.org/find/cs/1/au:+Mane_S/0/1/0/all/0/1">Swapnil Mane</a>, <a href="http://arxiv.org/find/cs/1/au:+Kundu_S/0/1/0/all/0/1">Suman Kundu</a>, <a href="http://arxiv.org/find/cs/1/au:+Sharma_R/0/1/0/all/0/1">Rajesh Sharma</a></p>

<p>The rise of social media platforms has led to an increase in cyber-aggressive

behavior, encompassing a broad spectrum of hostile behavior, including

cyberbullying, online harassment, and the dissemination of offensive and hate

speech. These behaviors have been associated with significant societal

consequences, ranging from online anonymity to real-world outcomes such as

depression, suicidal tendencies, and, in some instances, offline violence.

Recognizing the societal risks associated with unchecked aggressive content,

this paper delves into the field of Aggression Content Detection and Behavioral

Analysis of Aggressive Users, aiming to bridge the gap between disparate

studies. In this paper, we analyzed the diversity of definitions and proposed a

unified cyber-aggression definition. We examine the comprehensive process of

Aggression Content Detection, spanning from dataset creation, feature selection

and extraction, and detection algorithm development. Further, we review studies

on Behavioral Analysis of Aggression that explore the influencing factors,

consequences, and patterns associated with cyber-aggressive behavior. This

systematic literature review is a cross-examination of content detection and

behavioral analysis in the realm of cyber-aggression. The integrated

investigation reveals the effectiveness of incorporating sociological insights

into computational techniques for preventing cyber-aggressive behavior.

Finally, the paper concludes by identifying research gaps and encouraging

further progress in the unified domain of socio-computational aggressive

behavior analysis.

</p>

</p>

</div>

<div class="article">

<h1><a href="http://arxiv.org/abs/2311.09383">Long-form Question Answering: An Iterative Planning-Retrieval-Generation Approach. (arXiv:2311.09383v1 [cs.CL])</a></h1>

<p><b>Authors:</b> <a href="http://arxiv.org/find/cs/1/au:+Akash_P/0/1/0/all/0/1">Pritom Saha Akash</a>, <a href="http://arxiv.org/find/cs/1/au:+Roy_K/0/1/0/all/0/1">Kashob Kumar Roy</a>, <a href="http://arxiv.org/find/cs/1/au:+Popa_L/0/1/0/all/0/1">Lucian Popa</a>, <a href="http://arxiv.org/find/cs/1/au:+Chang_K/0/1/0/all/0/1">Kevin Chen-Chuan Chang</a></p>

<p>Long-form question answering (LFQA) poses a challenge as it involves

generating detailed answers in the form of paragraphs, which go beyond simple

yes/no responses or short factual answers. While existing QA models excel in

questions with concise answers, LFQA requires handling multiple topics and

their intricate relationships, demanding comprehensive explanations. Previous

attempts at LFQA focused on generating long-form answers by utilizing relevant

contexts from a corpus, relying solely on the question itself. However, they

overlooked the possibility that the question alone might not provide sufficient

information to identify the relevant contexts. Additionally, generating

detailed long-form answers often entails aggregating knowledge from diverse

sources. To address these limitations, we propose an LFQA model with iterative

Planning, Retrieval, and Generation. This iterative process continues until a

complete answer is generated for the given question. From an extensive

experiment on both an open domain and a technical domain QA dataset, we find

that our model outperforms the state-of-the-art models on various textual and

factual metrics for the LFQA task.

</p>

</p>

</div>

<div class="article">

<h1><a href="http://arxiv.org/abs/2311.09387">Banach-Tarski Embeddings and Transformers. (arXiv:2311.09387v1 [cs.LG])</a></h1>

<p><b>Authors:</b> <a href="http://arxiv.org/find/cs/1/au:+Maher_J/0/1/0/all/0/1">Joshua Maher</a></p>

<p>We introduce a new construction of embeddings of arbitrary recursive data

structures into high dimensional vectors. These embeddings provide an

interpretable model for the latent state vectors of transformers. We

demonstrate that these embeddings can be decoded to the original data structure

when the embedding dimension is sufficiently large. This decoding algorithm has

a natural implementation as a transformer. We also show that these embedding

vectors can be manipulated directly to perform computations on the underlying

data without decoding. As an example we present an algorithm that constructs

the embedded parse tree of an embedded token sequence using only vector

operations in embedding space.

</p>

</p>

</div>

<div class="article">

<h1><a href="http://arxiv.org/abs/2311.09389">Neural machine translation for automated feedback on children&#x27;s early-stage writing. (arXiv:2311.09389v1 [cs.CL])</a></h1>

<p><b>Authors:</b> <a href="http://arxiv.org/find/cs/1/au:+Jensen_J/0/1/0/all/0/1">Jonas Vestergaard Jensen</a>, <a href="http://arxiv.org/find/cs/1/au:+Jordahn_M/0/1/0/all/0/1">Mikkel Jordahn</a>, <a href="http://arxiv.org/find/cs/1/au:+Andersen_M/0/1/0/all/0/1">Michael Riis Andersen</a></p>

<p>In this work, we address the problem of assessing and constructing feedback

for early-stage writing automatically using machine learning. Early-stage

writing is typically vastly different from conventional writing due to phonetic

spelling and lack of proper grammar, punctuation, spacing etc. Consequently,

early-stage writing is highly non-trivial to analyze using common linguistic

metrics. We propose to use sequence-to-sequence models for "translating"

early-stage writing by students into "conventional" writing, which allows the

translated text to be analyzed using linguistic metrics. Furthermore, we

propose a novel robust likelihood to mitigate the effect of noise in the

dataset. We investigate the proposed methods using a set of numerical

experiments and demonstrate that the conventional text can be predicted with

high accuracy.

</p>

</p>

</div>

<div class="article">

<h1><a href="http://arxiv.org/abs/2311.09390">LEEETs-Dial: Linguistic Entrainment in End-to-End Task-oriented Dialogue systems. (arXiv:2311.09390v1 [cs.CL])</a></h1>

<p><b>Authors:</b> <a href="http://arxiv.org/find/cs/1/au:+Kumar_N/0/1/0/all/0/1">Nalin Kumar</a>, <a href="http://arxiv.org/find/cs/1/au:+Dusek_O/0/1/0/all/0/1">Ond&#x159;ej Du&#x161;ek</a></p>

<p>Linguistic entrainment, or alignment, represents a phenomenon where

linguistic patterns employed by conversational participants converge to one

another. While alignment has been shown to produce a more natural user

experience, most dialogue systems do not have any provisions for it. In this

work, we introduce methods for achieving dialogue alignment in a GPT-2-based

end-to-end dialogue system through the utilization of shared vocabulary. We

experiment with training instance weighting, alignment-specific loss, and

additional conditioning to generate responses that align with the user. By

comparing different entrainment techniques on the MultiWOZ dataset, we

demonstrate that all three approaches produce significantly better-aligned

results than the baseline, as confirmed by both automated and manual evaluation

metrics.

</p>

</p>

</div>

<div class="article">

<h1><a href="http://arxiv.org/abs/2311.09404">To Translate or Not to Translate: A Systematic Investigation of Translation-Based Cross-Lingual Transfer to Low-Resource Languages. (arXiv:2311.09404v1 [cs.CL])</a></h1>

<p><b>Authors:</b> <a href="http://arxiv.org/find/cs/1/au:+Ebing_B/0/1/0/all/0/1">Benedikt Ebing</a>, <a href="http://arxiv.org/find/cs/1/au:+Glavas_G/0/1/0/all/0/1">Goran Glava&#x161;</a></p>

<p>Perfect machine translation (MT) would render cross-lingual transfer (XLT) by

means of multilingual language models (LMs) superfluous. Given, on one hand,

the large body of work on improving XLT with multilingual LMs and, on the other

hand, recent advances in massively multilingual MT, in this work, we

systematically evaluate existing and propose new translation-based XLT

approaches for transfer to low-resource languages. We show that all

translation-based approaches dramatically outperform zero-shot XLT with

multilingual LMs, rendering the approach that combines the round-trip

translation of the source-language training data with the translation of the

target-language test instances the most effective. We next show that one can

obtain further empirical gains by adding reliable translations to other

high-resource languages to the training data. Moreover, we propose an effective

translation-based XLT strategy even for languages not supported by the MT

system. Finally, we show that model selection for XLT based on target-language

validation data obtained with MT outperforms model selection based on the

source-language data. We hope that our findings encourage adoption of more

robust translation-based baselines in XLT research.

</p>

</p>

</div>

<div class="article">

<h1><a href="http://arxiv.org/abs/2311.09406">Alternatives to the Scaled Dot Product for Attention in the Transformer Neural Network Architecture. (arXiv:2311.09406v1 [cs.LG])</a></h1>

<p><b>Authors:</b> <a href="http://arxiv.org/find/cs/1/au:+Bernhard_J/0/1/0/all/0/1">James Bernhard</a></p>

<p>The transformer neural network architecture uses a form of attention in which

the dot product of query and key is divided by the square root of the key

dimension before applying softmax. This scaling of the dot product is designed

to avoid the absolute value of the dot products becoming so large that applying

softmax leads to vanishing gradients. In this paper, we propose some

alternative scalings, including dividing the dot product instead by the sum of

the key lengths before applying softmax. We use simulated keys and queries to

show that in many situations this appears to be more effective at avoiding

regions where applying softmax leads to vanishing gradients.

</p>

</p>

</div>

<div class="article">

<h1><a href="http://arxiv.org/abs/2311.09410">When Large Language Models contradict humans? Large Language Models&#x27; Sycophantic Behaviour. (arXiv:2311.09410v1 [cs.CL])</a></h1>

<p><b>Authors:</b> <a href="http://arxiv.org/find/cs/1/au:+Ranaldi_L/0/1/0/all/0/1">Leonardo Ranaldi</a>, <a href="http://arxiv.org/find/cs/1/au:+Pucci_G/0/1/0/all/0/1">Giulia Pucci</a></p>

<p>Large Language Models (LLMs) have been demonstrating the ability to solve

complex tasks by delivering answers that are positively evaluated by humans due

in part to the intensive use of human feedback that refines responses. However,

the suggestibility transmitted through human feedback increases the inclination

to produce responses that correspond to the user's beliefs or misleading

prompts as opposed to true facts, a behaviour known as sycophancy. This

phenomenon decreases the bias, robustness, and, consequently, their

reliability.

</p>

<p>In this paper, we shed light on the suggestibility of LLMs to sycophantic

behaviour, demonstrating these tendencies via human-influenced prompts over

different tasks. Our investigation reveals that LLMs show sycophantic

tendencies when responding to queries involving subjective opinions and

statements that should elicit a contrary response based on facts, demonstrating

a lack of robustness.

</p>

</p>

</div>

<div class="article">

<h1><a href="http://arxiv.org/abs/2311.09422">Predicting generalization performance with correctness discriminators. (arXiv:2311.09422v1 [cs.CL])</a></h1>

<p><b>Authors:</b> <a href="http://arxiv.org/find/cs/1/au:+Yao_Y/0/1/0/all/0/1">Yuekun Yao</a>, <a href="http://arxiv.org/find/cs/1/au:+Koller_A/0/1/0/all/0/1">Alexander Koller</a></p>

<p>The ability to predict an NLP model's accuracy on unseen, potentially

out-of-distribution data is a prerequisite for trustworthiness. We present a

novel model that establishes upper and lower bounds on the accuracy, without

requiring gold labels for the unseen data. We achieve this by training a

discriminator which predicts whether the output of a given sequence-to-sequence

model is correct or not. We show across a variety of tagging, parsing, and

semantic parsing tasks that the gold accuracy is reliably between the predicted

upper and lower bounds, and that these bounds are remarkably close together.

</p>

</p>

</div>

<div class="article">

<h1><a href="http://arxiv.org/abs/2311.09428">Beyond Detection: Unveiling Fairness Vulnerabilities in Abusive Language Models. (arXiv:2311.09428v1 [cs.CL])</a></h1>

<p><b>Authors:</b> <a href="http://arxiv.org/find/cs/1/au:+Liang_Y/0/1/0/all/0/1">Yueqing Liang</a>, <a href="http://arxiv.org/find/cs/1/au:+Cheng_L/0/1/0/all/0/1">Lu Cheng</a>, <a href="http://arxiv.org/find/cs/1/au:+Payani_A/0/1/0/all/0/1">Ali Payani</a>, <a href="http://arxiv.org/find/cs/1/au:+Shu_K/0/1/0/all/0/1">Kai Shu</a></p>

<p>This work investigates the potential of undermining both fairness and

detection performance in abusive language detection. In a dynamic and complex

digital world, it is crucial to investigate the vulnerabilities of these

detection models to adversarial fairness attacks to improve their fairness

robustness. We propose a simple yet effective framework FABLE that leverages

backdoor attacks as they allow targeted control over the fairness and detection

performance. FABLE explores three types of trigger designs (i.e., rare,

artificial, and natural triggers) and novel sampling strategies. Specifically,

the adversary can inject triggers into samples in the minority group with the

favored outcome (i.e., ``non-abusive'') and flip their labels to the unfavored

outcome, i.e., ``abusive''. Experiments on benchmark datasets demonstrate the

effectiveness of FABLE attacking fairness and utility in abusive language

detection.

</p>

</p>

</div>

<div class="article">

<h1><a href="http://arxiv.org/abs/2311.09431">Striped Attention: Faster Ring Attention for Causal Transformers. (arXiv:2311.09431v1 [cs.LG])</a></h1>

<p><b>Authors:</b> <a href="http://arxiv.org/find/cs/1/au:+Brandon_W/0/1/0/all/0/1">William Brandon</a>, <a href="http://arxiv.org/find/cs/1/au:+Nrusimha_A/0/1/0/all/0/1">Aniruddha Nrusimha</a>, <a href="http://arxiv.org/find/cs/1/au:+Qian_K/0/1/0/all/0/1">Kevin Qian</a>, <a href="http://arxiv.org/find/cs/1/au:+Ankner_Z/0/1/0/all/0/1">Zachary Ankner</a>, <a href="http://arxiv.org/find/cs/1/au:+Jin_T/0/1/0/all/0/1">Tian Jin</a>, <a href="http://arxiv.org/find/cs/1/au:+Song_Z/0/1/0/all/0/1">Zhiye Song</a>, <a href="http://arxiv.org/find/cs/1/au:+Ragan_Kelley_J/0/1/0/all/0/1">Jonathan Ragan-Kelley</a></p>

<p>To help address the growing demand for ever-longer sequence lengths in

transformer models, Liu et al. recently proposed Ring Attention, an exact

attention algorithm capable of overcoming per-device memory bottle- necks by

distributing self-attention across multiple devices. In this paper, we study

the performance characteristics of Ring Attention in the important special case

of causal transformer models, and identify a key workload imbal- ance due to

triangular structure of causal attention computations. We propose a simple

extension to Ring Attention, which we call Striped Attention to fix this

imbalance. Instead of devices having contiguous subsequences, each device has a

subset of tokens distributed uniformly throughout the sequence, which we

demonstrate leads to more even workloads. In experiments running Striped

Attention on A100 GPUs and TPUv4s, we are able to achieve up to 1.45x

end-to-end throughput improvements over the original Ring Attention algorithm

on causal transformer training at a sequence length of 256k. Furthermore, on 16

TPUv4 chips, we were able to achieve 1.65x speedups at sequence lengths of

786k. We release the code for our experiments as open source

</p>

</p>

</div>

<div class="article">

<h1><a href="http://arxiv.org/abs/2311.09433">Backdoor Activation Attack: Attack Large Language Models using Activation Steering for Safety-Alignment. (arXiv:2311.09433v1 [cs.CR])</a></h1>

<p><b>Authors:</b> <a href="http://arxiv.org/find/cs/1/au:+Wang_H/0/1/0/all/0/1">Haoran Wang</a>, <a href="http://arxiv.org/find/cs/1/au:+Shu_K/0/1/0/all/0/1">Kai Shu</a></p>

<p>To ensure AI safety, instruction-tuned Large Language Models (LLMs) are

specifically trained to ensure alignment, which refers to making models behave

in accordance with human intentions. While these models have demonstrated

commendable results on various safety benchmarks, the vulnerability of their

safety alignment has not been extensively studied. This is particularly

troubling given the potential harm that LLMs can inflict. Existing attack

methods on LLMs often rely on poisoned training data or the injection of

malicious prompts. These approaches compromise the stealthiness and

generalizability of the attacks, making them susceptible to detection.

Additionally, these models often demand substantial computational resources for

implementation, making them less practical for real-world applications. In this

work, we introduce a novel attack framework, called Backdoor Activation Attack,

which injects trojan steering vectors into the activation layers of LLMs. These

malicious steering vectors can be triggered at inference time to steer the

models toward attacker-desired behaviors by manipulating their activations. In

particular, the steering vectors are generated by taking the difference between

benign and malicious activations. Then, the most effective steering vector is

selected and added to the forward passes of the LLMs. Our experiment results on

four primary alignment tasks show that our proposed method is highly effective

and adds little or no overhead to attack efficiency. Additionally, we discuss

potential countermeasures against such activation attacks. Our code and data

are available at https://email-haoran-for-link. Warning: this paper contains

content that can be offensive or upsetting.

</p>

</p>

</div>

<div class="article">

<h1><a href="http://arxiv.org/abs/2311.09438">Labeled Interactive Topic Models. (arXiv:2311.09438v1 [cs.LG])</a></h1>

<p><b>Authors:</b> <a href="http://arxiv.org/find/cs/1/au:+Seelman_K/0/1/0/all/0/1">Kyle Seelman</a>, <a href="http://arxiv.org/find/cs/1/au:+Zhang_M/0/1/0/all/0/1">Mozhi Zhang</a>, <a href="http://arxiv.org/find/cs/1/au:+Boyd_Graber_J/0/1/0/all/0/1">Jordan Boyd-Graber</a></p>

<p>Topic models help users understand large document collections; however, topic

models do not always find the ``right'' topics. While classical probabilistic

and anchor-based topic models have interactive variants to guide models toward

better topics, such interactions are not available for neural topic models such

as the embedded topic model (\abr{etm}). We correct this lacuna by adding an

intuitive interaction to neural topic models: users can label a topic with a

word, and topics are updated so that the topic words are close to the label.

This allows a user to refine topics based on their information need. While,

interactivity is intuitive for \abr{etm}, we extend this framework to work with

other neural topic models as well. We develop an interactive interface which

allows users to interact and relabel topic models as they see fit. We evaluate

our method through a human study, where users can relabel topics to find

relevant documents. Using our method, user labeling improves document rank

scores, helping to find more relevant documents to a given query when compared

to no user labeling.

</p>

</p>

</div>

<div class="article">

<h1><a href="http://arxiv.org/abs/2311.09443">Subtle Misogyny Detection and Mitigation: An Expert-Annotated Dataset. (arXiv:2311.09443v1 [cs.CL])</a></h1>

<p><b>Authors:</b> <a href="http://arxiv.org/find/cs/1/au:+Sheppard_B/0/1/0/all/0/1">Brooklyn Sheppard</a>, <a href="http://arxiv.org/find/cs/1/au:+Richter_A/0/1/0/all/0/1">Anna Richter</a>, <a href="http://arxiv.org/find/cs/1/au:+Cohen_A/0/1/0/all/0/1">Allison Cohen</a>, <a href="http://arxiv.org/find/cs/1/au:+Smith_E/0/1/0/all/0/1">Elizabeth Allyn Smith</a>, <a href="http://arxiv.org/find/cs/1/au:+Kneese_T/0/1/0/all/0/1">Tamara Kneese</a>, <a href="http://arxiv.org/find/cs/1/au:+Pelletier_C/0/1/0/all/0/1">Carolyne Pelletier</a>, <a href="http://arxiv.org/find/cs/1/au:+Baldini_I/0/1/0/all/0/1">Ioana Baldini</a>, <a href="http://arxiv.org/find/cs/1/au:+Dong_Y/0/1/0/all/0/1">Yue Dong</a></p>

<p>Using novel approaches to dataset development, the Biasly dataset captures

the nuance and subtlety of misogyny in ways that are unique within the

literature. Built in collaboration with multi-disciplinary experts and

annotators themselves, the dataset contains annotations of movie subtitles,

capturing colloquial expressions of misogyny in North American film. The

dataset can be used for a range of NLP tasks, including classification,

severity score regression, and text generation for rewrites. In this paper, we

discuss the methodology used, analyze the annotations obtained, and provide

baselines using common NLP algorithms in the context of misogyny detection and

mitigation. We hope this work will promote AI for social good in NLP for bias

detection, explanation, and removal.

</p>

</p>

</div>

<div class="article">

<h1><a href="http://arxiv.org/abs/2311.09447">How Trustworthy are Open-Source LLMs? An Assessment under Malicious Demonstrations Shows their Vulnerabilities. (arXiv:2311.09447v1 [cs.CL])</a></h1>

<p><b>Authors:</b> <a href="http://arxiv.org/find/cs/1/au:+Mo_L/0/1/0/all/0/1">Lingbo Mo</a>, <a href="http://arxiv.org/find/cs/1/au:+Wang_B/0/1/0/all/0/1">Boshi Wang</a>, <a href="http://arxiv.org/find/cs/1/au:+Chen_M/0/1/0/all/0/1">Muhao Chen</a>, <a href="http://arxiv.org/find/cs/1/au:+Sun_H/0/1/0/all/0/1">Huan Sun</a></p>

<p>The rapid progress in open-source Large Language Models (LLMs) is

significantly driving AI development forward. However, there is still a limited

understanding of their trustworthiness. Deploying these models at scale without

sufficient trustworthiness can pose significant risks, highlighting the need to

uncover these issues promptly. In this work, we conduct an assessment of

open-source LLMs on trustworthiness, scrutinizing them across eight different

aspects including toxicity, stereotypes, ethics, hallucination, fairness,

sycophancy, privacy, and robustness against adversarial demonstrations. We

propose an enhanced Chain of Utterances-based (CoU) prompting strategy by

incorporating meticulously crafted malicious demonstrations for trustworthiness

attack. Our extensive experiments encompass recent and representative series of

open-source LLMs, including Vicuna, MPT, Falcon, Mistral, and Llama 2. The

empirical outcomes underscore the efficacy of our attack strategy across

diverse aspects. More interestingly, our result analysis reveals that models

with superior performance in general NLP tasks do not always have greater

trustworthiness; in fact, larger models can be more vulnerable to attacks.

Additionally, models that have undergone instruction tuning, focusing on

instruction following, tend to be more susceptible, although fine-tuning LLMs

for safety alignment proves effective in mitigating adversarial trustworthiness

attacks.

</p>

</p>

</div>

<div class="article">

<h1><a href="http://arxiv.org/abs/2311.09458">Lexical Repetitions Lead to Rote Learning: Unveiling the Impact of Lexical Overlap in Train and Test Reference Summaries. (arXiv:2311.09458v1 [cs.CL])</a></h1>

<p><b>Authors:</b> <a href="http://arxiv.org/find/cs/1/au:+Choubey_P/0/1/0/all/0/1">Prafulla Kumar Choubey</a>, <a href="http://arxiv.org/find/cs/1/au:+Fabbri_A/0/1/0/all/0/1">Alexander R. Fabbri</a>, <a href="http://arxiv.org/find/cs/1/au:+Xiong_C/0/1/0/all/0/1">Caiming Xiong</a>, <a href="http://arxiv.org/find/cs/1/au:+Wu_C/0/1/0/all/0/1">Chien-Sheng Wu</a></p>

<p>Ideal summarization models should generalize to novel summary-worthy content

without remembering reference training summaries by rote. However, a single

average performance score on the entire test set is inadequate in determining

such model competencies. We propose a fine-grained evaluation protocol by

partitioning a test set based on the lexical similarity of reference test

summaries with training summaries. We observe up to a 5x (1.2x) difference in

ROUGE-2 (entity recall) scores between the subsets with the lowest and highest

similarity. Next, we show that such training repetitions also make a model

vulnerable to rote learning, reproducing data artifacts such as factual errors,

especially when reference test summaries are lexically close to training

summaries. Consequently, we propose to limit lexical repetitions in training

summaries during both supervised fine-tuning and likelihood calibration stages

to improve the performance on novel test cases while retaining average

performance. Our automatic and human evaluations on novel test subsets and

recent news articles show that limiting lexical repetitions in training

summaries can prevent rote learning and improve generalization.

</p>

</p>

</div>

<div class="article">

<h1><a href="http://arxiv.org/abs/2311.09467">Think While You Write: Hypothesis Verification Promotes Faithful Knowledge-to-Text Generation. (arXiv:2311.09467v1 [cs.CL])</a></h1>

<p><b>Authors:</b> <a href="http://arxiv.org/find/cs/1/au:+Qiu_Y/0/1/0/all/0/1">Yifu Qiu</a>, <a href="http://arxiv.org/find/cs/1/au:+Embar_V/0/1/0/all/0/1">Varun Embar</a>, <a href="http://arxiv.org/find/cs/1/au:+Cohen_S/0/1/0/all/0/1">Shay B. Cohen</a>, <a href="http://arxiv.org/find/cs/1/au:+Han_B/0/1/0/all/0/1">Benjamin Han</a></p>

<p>Neural knowledge-to-text generation models often struggle to faithfully

generate descriptions for the input facts: they may produce hallucinations that

contradict the given facts, or describe facts not present in the input. To

reduce hallucinations, we propose a novel decoding method, TWEAK (Think While

Effectively Articulating Knowledge). TWEAK treats the generated sequences at

each decoding step and its future sequences as hypotheses, and ranks each

generation candidate based on how well their corresponding hypotheses support

the input facts using a Hypothesis Verification Model (HVM). We first

demonstrate the effectiveness of TWEAK by using a Natural Language Inference

(NLI) model as the HVM and report improved faithfulness with minimal impact on

the quality. We then replace the NLI model with our task-specific HVM trained

with a first-of-a-kind dataset, FATE (Fact-Aligned Textual Entailment), which

pairs input facts with their faithful and hallucinated descriptions with the

hallucinated spans marked. The new HVM improves the faithfulness and the

quality further and runs faster. Overall the best TWEAK variants improve on

average 2.22/7.17 points on faithfulness measured by FactKB over WebNLG and

TekGen/GenWiki, respectively, with only 0.14/0.32 points degradation on quality

measured by BERTScore over the same datasets. Since TWEAK is a decoding-only

approach, it can be integrated with any neural generative model without

retraining.

</p>

</p>

</div>

<div class="article">

<h1><a href="http://arxiv.org/abs/2311.09469">Clarify When Necessary: Resolving Ambiguity Through Interaction with LMs. (arXiv:2311.09469v1 [cs.CL])</a></h1>

<p><b>Authors:</b> <a href="http://arxiv.org/find/cs/1/au:+Zhang_M/0/1/0/all/0/1">Michael J.Q. Zhang</a>, <a href="http://arxiv.org/find/cs/1/au:+Choi_E/0/1/0/all/0/1">Eunsol Choi</a></p>

<p>Resolving ambiguities through interaction is a hallmark of natural language,

and modeling this behavior is a core challenge in crafting AI assistants. In

this work, we study such behavior in LMs by proposing a task-agnostic framework

for resolving ambiguity by asking users clarifying questions. Our framework

breaks down this objective into three subtasks: (1) determining when

clarification is needed, (2) determining what clarifying question to ask, and

(3) responding accurately with the new information gathered through

clarification. We evaluate systems across three NLP applications: question

answering, machine translation and natural language inference. For the first

subtask, we present a novel uncertainty estimation approach, intent-sim, that

determines the utility of querying for clarification by estimating the entropy

over user intents. Our method consistently outperforms existing uncertainty

estimation approaches at identifying predictions that will benefit from

clarification. When only allowed to ask for clarification on 10% of examples,

our system is able to double the performance gains over randomly selecting

examples to clarify. Furthermore, we find that intent-sim is robust,

demonstrating improvements across a wide range of NLP tasks and LMs. Together,

our work lays foundation for studying clarifying interactions with LMs.

</p>

</p>

</div>

<div class="article">

<h1><a href="http://arxiv.org/abs/2311.09473">JAB: Joint Adversarial Prompting and Belief Augmentation. (arXiv:2311.09473v1 [cs.AI])</a></h1>

<p><b>Authors:</b> <a href="http://arxiv.org/find/cs/1/au:+Mehrabi_N/0/1/0/all/0/1">Ninareh Mehrabi</a>, <a href="http://arxiv.org/find/cs/1/au:+Goyal_P/0/1/0/all/0/1">Palash Goyal</a>, <a href="http://arxiv.org/find/cs/1/au:+Ramakrishna_A/0/1/0/all/0/1">Anil Ramakrishna</a>, <a href="http://arxiv.org/find/cs/1/au:+Dhamala_J/0/1/0/all/0/1">Jwala Dhamala</a>, <a href="http://arxiv.org/find/cs/1/au:+Ghosh_S/0/1/0/all/0/1">Shalini Ghosh</a>, <a href="http://arxiv.org/find/cs/1/au:+Zemel_R/0/1/0/all/0/1">Richard Zemel</a>, <a href="http://arxiv.org/find/cs/1/au:+Chang_K/0/1/0/all/0/1">Kai-Wei Chang</a>, <a href="http://arxiv.org/find/cs/1/au:+Galstyan_A/0/1/0/all/0/1">Aram Galstyan</a>, <a href="http://arxiv.org/find/cs/1/au:+Gupta_R/0/1/0/all/0/1">Rahul Gupta</a></p>

<p>With the recent surge of language models in different applications, attention

to safety and robustness of these models has gained significant importance.

Here we introduce a joint framework in which we simultaneously probe and

improve the robustness of a black-box target model via adversarial prompting

and belief augmentation using iterative feedback loops. This framework utilizes

an automated red teaming approach to probe the target model, along with a

belief augmenter to generate instructions for the target model to improve its

robustness to those adversarial probes. Importantly, the adversarial model and

the belief generator leverage the feedback from past interactions to improve

the effectiveness of the adversarial prompts and beliefs, respectively. In our

experiments, we demonstrate that such a framework can reduce toxic content

generation both in dynamic cases where an adversary directly interacts with a

target model and static cases where we use a static benchmark dataset to

evaluate our model.

</p>

</p>

</div>

<div class="article">

<h1><a href="http://arxiv.org/abs/2311.09476">ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems. (arXiv:2311.09476v1 [cs.CL])</a></h1>

<p><b>Authors:</b> <a href="http://arxiv.org/find/cs/1/au:+Saad_Falcon_J/0/1/0/all/0/1">Jon Saad-Falcon</a>, <a href="http://arxiv.org/find/cs/1/au:+Khattab_O/0/1/0/all/0/1">Omar Khattab</a>, <a href="http://arxiv.org/find/cs/1/au:+Potts_C/0/1/0/all/0/1">Christopher Potts</a>, <a href="http://arxiv.org/find/cs/1/au:+Zaharia_M/0/1/0/all/0/1">Matei Zaharia</a></p>

<p>Evaluating retrieval-augmented generation (RAG) systems traditionally relies

on hand annotations for input queries, passages to retrieve, and responses to

generate. We introduce ARES, an Automated RAG Evaluation System, for evaluating

RAG systems along the dimensions of context relevance, answer faithfulness, and

answer relevance. Using synthetic training data, ARES finetunes lightweight LM

judges to assess the quality of individual RAG components. To mitigate

potential prediction errors, ARES utilizes a small set of human-annotated

datapoints for prediction-powered inference (PPI). Across six different

knowledge-intensive tasks in KILT and SuperGLUE, ARES accurately evaluates RAG

systems while using a few hundred human annotations during evaluation.

Furthermore, ARES judges remain effective across domain shifts, proving

accurate even after changing the type of queries and/or documents used in the

evaluated RAG systems. We make our datasets and code for replication and

deployment available at https://github.com/stanford-futuredata/ARES.

</p>

</p>

</div>

<div class="article">

<h1><a href="http://arxiv.org/abs/2311.09480">Show Your Work with Confidence: Confidence Bands for Tuning Curves. (arXiv:2311.09480v1 [cs.CL])</a></h1>

<p><b>Authors:</b> <a href="http://arxiv.org/find/cs/1/au:+Lourie_N/0/1/0/all/0/1">Nicholas Lourie</a>, <a href="http://arxiv.org/find/cs/1/au:+Cho_K/0/1/0/all/0/1">Kyunghyun Cho</a>, <a href="http://arxiv.org/find/cs/1/au:+He_H/0/1/0/all/0/1">He He</a></p>

<p>The choice of hyperparameters greatly impacts performance in natural language

processing. Often, it is hard to tell if a method is better than another or

just better tuned. Tuning curves fix this ambiguity by accounting for tuning

effort. Specifically, they plot validation performance as a function of the

number of hyperparameter choices tried so far. While several estimators exist

for these curves, it is common to use point estimates, which we show fail

silently and give contradictory results when given too little data.

</p>

<p>Beyond point estimates, confidence bands are necessary to rigorously

establish the relationship between different approaches. We present the first

method to construct valid confidence bands for tuning curves. The bands are

exact, simultaneous, and distribution-free, thus they provide a robust basis

for comparing methods.

</p>

<p>Empirical analysis shows that while bootstrap confidence bands, which serve

as a baseline, fail to approximate their target confidence, ours achieve it

exactly. We validate our design with ablations, analyze the effect of sample

size, and provide guidance on comparing models with our method. To promote

confident comparisons in future work, we release a library implementing the

method at https://github.com/nalourie/opda .

</p>

</p>

</div>

<div class="article">

<h1><a href="http://arxiv.org/abs/2311.09481">Personalized Jargon Identification for Enhanced Interdisciplinary Communication. (arXiv:2311.09481v1 [cs.CL])</a></h1>

<p><b>Authors:</b> <a href="http://arxiv.org/find/cs/1/au:+Guo_Y/0/1/0/all/0/1">Yue Guo</a>, <a href="http://arxiv.org/find/cs/1/au:+Chang_J/0/1/0/all/0/1">Joseph Chee Chang</a>, <a href="http://arxiv.org/find/cs/1/au:+Antoniak_M/0/1/0/all/0/1">Maria Antoniak</a>, <a href="http://arxiv.org/find/cs/1/au:+Bransom_E/0/1/0/all/0/1">Erin Bransom</a>, <a href="http://arxiv.org/find/cs/1/au:+Cohen_T/0/1/0/all/0/1">Trevor Cohen</a>, <a href="http://arxiv.org/find/cs/1/au:+Wang_L/0/1/0/all/0/1">Lucy Lu Wang</a>, <a href="http://arxiv.org/find/cs/1/au:+August_T/0/1/0/all/0/1">Tal August</a></p>

<p>Scientific jargon can impede researchers when they read materials from other

domains. Current methods of jargon identification mainly use corpus-level

familiarity indicators (e.g., Simple Wikipedia represents plain language).

However, researchers' familiarity of a term can vary greatly based on their own

background. We collect a dataset of over 10K term familiarity annotations from

11 computer science researchers for terms drawn from 100 paper abstracts.

Analysis of this data reveals that jargon familiarity and information needs

vary widely across annotators, even within the same sub-domain (e.g., NLP). We

investigate features representing individual, sub-domain, and domain knowledge

to predict individual jargon familiarity. We compare supervised and

prompt-based approaches, finding that prompt-based methods including personal

publications yields the highest accuracy, though zero-shot prompting provides a

strong baseline. This research offers insight into features and methods to

integrate personal data into scientific jargon identification.

</p>

</p>

</div>

<div class="article">

<h1><a href="http://arxiv.org/abs/2311.09502">SQATIN: Supervised Instruction Tuning Meets Question Answering for Improved Dialogue NLU. (arXiv:2311.09502v1 [cs.CL])</a></h1>

<p><b>Authors:</b> <a href="http://arxiv.org/find/cs/1/au:+Razumovskaia_E/0/1/0/all/0/1">Evgeniia Razumovskaia</a>, <a href="http://arxiv.org/find/cs/1/au:+Glavas_G/0/1/0/all/0/1">Goran Glava&#x161;</a>, <a href="http://arxiv.org/find/cs/1/au:+Korhonen_A/0/1/0/all/0/1">Anna Korhonen</a>, <a href="http://arxiv.org/find/cs/1/au:+Vulic_I/0/1/0/all/0/1">Ivan Vuli&#x107;</a></p>

<p>Task-oriented dialogue (ToD) systems help users execute well-defined tasks

across a variety of domains (e.g., $\textit{flight booking}$ or $\textit{food

ordering}$), with their Natural Language Understanding (NLU) components being

dedicated to the analysis of user utterances, predicting users' intents

($\textit{Intent Detection}$, ID) and extracting values for informational slots

($\textit{Value Extraction}$, VE). In most domains, labelled NLU data is

scarce, making sample-efficient learning -- enabled with effective transfer

paradigms -- paramount. In this work, we introduce SQATIN, a new framework for

dialog NLU based on (i) instruction tuning and (ii) question-answering-based

formulation of ID and VE tasks. According to the evaluation on established NLU

benchmarks, SQATIN sets the new state of the art in dialogue NLU, substantially

surpassing the performance of current models based on standard fine-tuning

objectives in both in-domain training and cross-domain transfer. SQATIN yields

particularly large performance gains in cross-domain transfer, owing to the

fact that our QA-based instruction tuning leverages similarities between

natural language descriptions of classes (i.e., slots and intents) across

domains.

</p>

</p>

</div>

<div class="article">

<h1><a href="http://arxiv.org/abs/2311.09505">SegMix: A Simple Structure-Aware Data Augmentation Method. (arXiv:2311.09505v1 [cs.CL])</a></h1>

<p><b>Authors:</b> <a href="http://arxiv.org/find/cs/1/au:+Pei_Y/0/1/0/all/0/1">Yuxin Pei</a>, <a href="http://arxiv.org/find/cs/1/au:+Bhuse_P/0/1/0/all/0/1">Pushkar Bhuse</a>, <a href="http://arxiv.org/find/cs/1/au:+Liu_Z/0/1/0/all/0/1">Zhengzhong Liu</a>, <a href="http://arxiv.org/find/cs/1/au:+Xing_E/0/1/0/all/0/1">Eric Xing</a></p>

<p>Interpolation-based Data Augmentation (DA) methods (Mixup) linearly

interpolate the inputs and labels of two or more training examples. Mixup has

more recently been adapted to the field of Natural Language Processing (NLP),

mainly for sequence labeling tasks. However, such a simple adoption yields

mixed or unstable improvements over the baseline models. We argue that the

direct-adoption methods do not account for structures in NLP tasks. To this

end, we propose SegMix, a collection of interpolation-based DA algorithms that

can adapt to task-specific structures. SegMix poses fewer constraints on data

structures, is robust to various hyperparameter settings, applies to more task

settings, and adds little computational overhead. In the algorithm's core, we

apply interpolation methods on task-specific meaningful segments, in contrast

to applying them on sequences as in prior work. We find SegMix to be a flexible

framework that combines rule-based DA methods with interpolation-based methods,

creating interesting mixtures of DA techniques. We show that SegMix

consistently improves performance over strong baseline models in Named Entity

Recognition (NER) and Relation Extraction (RE) tasks, especially under

data-scarce settings. Furthermore, this method is easy to implement and adds

negligible training overhead.

</p>

</p>

</div>

<div class="article">

<h1><a href="http://arxiv.org/abs/2311.09510">One Size Does Not Fit All: Customizing Open-Domain Procedures. (arXiv:2311.09510v1 [cs.CL])</a></h1>

<p><b>Authors:</b> <a href="http://arxiv.org/find/cs/1/au:+Lal_Y/0/1/0/all/0/1">Yash Kumar Lal</a>, <a href="http://arxiv.org/find/cs/1/au:+Zhang_L/0/1/0/all/0/1">Li Zhang</a>, <a href="http://arxiv.org/find/cs/1/au:+Brahman_F/0/1/0/all/0/1">Faeze Brahman</a>, <a href="http://arxiv.org/find/cs/1/au:+Majumder_B/0/1/0/all/0/1">Bodhisattwa Prasad Majumder</a>, <a href="http://arxiv.org/find/cs/1/au:+Clark_P/0/1/0/all/0/1">Peter Clark</a>, <a href="http://arxiv.org/find/cs/1/au:+Tandon_N/0/1/0/all/0/1">Niket Tandon</a></p>

<p>How-to procedures, such as how to plant a garden, are ubiquitous. But one

size does not fit all - humans often need to customize these procedural plans

according to their specific needs, e.g., planting a garden without pesticides.

While LLMs can fluently generate generic procedures, we present the first study

on how well LLMs can customize open-domain procedures. We introduce

CustomPlans, a probe dataset of customization hints that encodes diverse user

needs for open-domain How-to procedures. Using LLMs as CustomizationAgent and

ExecutionAgent in different settings, we establish their abilities to perform

open-domain procedure customization. Human evaluation shows that using these

agents in a Sequential setting is the best, but they are good enough only ~51%

of the time. Error analysis shows that LLMs do not sufficiently address user

customization needs in their generated procedures.

</p>

</p>

</div>

<div class="article">

<h1><a href="http://arxiv.org/abs/2311.09513">Sequencing Matters: A Generate-Retrieve-Generate Model for Building Conversational Agents. (arXiv:2311.09513v1 [cs.CL])</a></h1>

<p><b>Authors:</b> <a href="http://arxiv.org/find/cs/1/au:+Patwardhan_Q/0/1/0/all/0/1">Quinn Patwardhan</a>, <a href="http://arxiv.org/find/cs/1/au:+Yang_G/0/1/0/all/0/1">Grace Hui Yang</a></p>

<p>This paper contains what the Georgetown InfoSense group has done in regard to

solving the challenges presented by TREC iKAT 2023. Our submitted runs

outperform the median runs by a significant margin, exhibiting superior

performance in nDCG across various cut numbers and in overall success rate. Our

approach uses a Generate-Retrieve-Generate method, which we've found to greatly

outpace Retrieve-Then-Generate approaches for the purposes of iKAT. Our

solution involves the use of Large Language Models (LLMs) for initial answers,

answer grounding by BM25, passage quality filtering by logistic regression, and

answer generation by LLMs again. We leverage several purpose-built Language

Models, including BERT, Chat-based, and text-to-transfer-based models, for text

understanding, classification, generation, and summarization. The official

results of the TREC evaluation contradict our initial self-evaluation, which

may suggest that a decrease in the reliance on our retrieval and classification

methods is better. Nonetheless, our findings suggest that the sequence of

involving these different components matters, where we see an essentiality of

using LLMs before using search engines.

</p>

</p>

</div>

<div class="article">

<h1><a href="http://arxiv.org/abs/2311.09517">GEE! Grammar Error Explanation with Large Language Models. (arXiv:2311.09517v1 [cs.CL])</a></h1>

<p><b>Authors:</b> <a href="http://arxiv.org/find/cs/1/au:+Song_Y/0/1/0/all/0/1">Yixiao Song</a>, <a href="http://arxiv.org/find/cs/1/au:+Krishna_K/0/1/0/all/0/1">Kalpesh Krishna</a>, <a href="http://arxiv.org/find/cs/1/au:+Bhatt_R/0/1/0/all/0/1">Rajesh Bhatt</a>, <a href="http://arxiv.org/find/cs/1/au:+Gimpel_K/0/1/0/all/0/1">Kevin Gimpel</a>, <a href="http://arxiv.org/find/cs/1/au:+Iyyer_M/0/1/0/all/0/1">Mohit Iyyer</a></p>

<p>Grammatical error correction tools are effective at correcting grammatical

errors in users' input sentences but do not provide users with \textit{natural

language} explanations about their errors. Such explanations are essential for

helping users learn the language by gaining a deeper understanding of its

grammatical rules (DeKeyser, 2003; Ellis et al., 2006). To address this gap, we

propose the task of grammar error explanation, where a system needs to provide

one-sentence explanations for each grammatical error in a pair of erroneous and

corrected sentences. We analyze the capability of GPT-4 in grammar error

explanation, and find that it only produces explanations for 60.2% of the

errors using one-shot prompting. To improve upon this performance, we develop a

two-step pipeline that leverages fine-tuned and prompted large language models

to perform structured atomic token edit extraction, followed by prompting GPT-4

to generate explanations. We evaluate our pipeline on German and Chinese

grammar error correction data sampled from language learners with a wide range

of proficiency levels. Human evaluation reveals that our pipeline produces

93.9% and 98.0% correct explanations for German and Chinese data, respectively.

To encourage further research in this area, we will open-source our data and

code.

</p>

</p>

</div>

<div class="article">

<h1><a href="http://arxiv.org/abs/2311.09519">Leveraging Code to Improve In-context Learning for Semantic Parsing. (arXiv:2311.09519v1 [cs.CL])</a></h1>

<p><b>Authors:</b> <a href="http://arxiv.org/find/cs/1/au:+Bogin_B/0/1/0/all/0/1">Ben Bogin</a>, <a href="http://arxiv.org/find/cs/1/au:+Gupta_S/0/1/0/all/0/1">Shivanshu Gupta</a>, <a href="http://arxiv.org/find/cs/1/au:+Clark_P/0/1/0/all/0/1">Peter Clark</a>, <a href="http://arxiv.org/find/cs/1/au:+Sabharwal_A/0/1/0/all/0/1">Ashish Sabharwal</a></p>

<p>In-context learning (ICL) is an appealing approach for semantic parsing due

to its few-shot nature and improved generalization. However, learning to parse

to rare domain-specific languages (DSLs) from just a few demonstrations is

challenging, limiting the performance of even the most capable LLMs. In this

work, we improve the effectiveness of ICL for semantic parsing by (1) using

general-purpose programming languages such as Python instead of DSLs, and (2)

augmenting prompts with a structured domain description that includes, e.g.,

the available classes and functions. We show that both these changes

significantly improve accuracy across three popular datasets. Combined, they

lead to dramatic improvements (e.g. 7.9% to 66.5% on SMCalFlow compositional

split), nearly closing the performance gap between easier i.i.d.\ and harder

compositional splits when used with a strong model, and reducing the need for a

large number of demonstrations. We find that the resemblance of the target

parse language to general-purpose code is a more important factor than the

language's popularity in pre-training corpora. Our findings provide an improved

methodology for building semantic parsers in the modern context of ICL with

LLMs.

</p>

</p>

</div>

<div class="article">

<h1><a href="http://arxiv.org/abs/2311.09521">AMRFact: Enhancing Summarization Factuality Evaluation with AMR-driven Training Data Generation. (arXiv:2311.09521v1 [cs.CL])</a></h1>

<p><b>Authors:</b> <a href="http://arxiv.org/find/cs/1/au:+Qiu_H/0/1/0/all/0/1">Haoyi Qiu</a>, <a href="http://arxiv.org/find/cs/1/au:+Huang_K/0/1/0/all/0/1">Kung-Hsiang Huang</a>, <a href="http://arxiv.org/find/cs/1/au:+Qu_J/0/1/0/all/0/1">Jingnong Qu</a>, <a href="http://arxiv.org/find/cs/1/au:+Peng_N/0/1/0/all/0/1">Nanyun Peng</a></p>

<p>Ensuring factual consistency is crucial in various natural language

processing tasks, particularly in abstractive summarization, where preserving

the integrity of information is paramount. Prior entailment-based approaches

often generate factually inconsistent summaries and then train a classifier on

the generated data. However, summaries produced by these approaches are either

of low coherence or lack error-type coverage. To address these issues, we

propose AMRFact, a novel framework that generates factually inconsistent

summaries using Abstract Meaning Representation (AMR). Our approach parses

factually correct summaries into AMR graphs and injects controlled factual

inconsistencies to create negative examples, allowing for coherent factually

inconsistent summaries to be generated with high error-type coverage.

Additionally, we present a data selection module NegFilter based on natural

language inference and BARTScore to ensure the quality of the generated

negative samples. Experimental results demonstrate that our approach

significantly outperforms previous systems on the AggreFact-SOTA dataset,

showcasing its efficacy in assessing factuality in abstractive summarization.

</p>

</p>

</div>

<div class="article">

<h1><a href="http://arxiv.org/abs/2311.09528">HelpSteer: Multi-attribute Helpfulness Dataset for SteerLM. (arXiv:2311.09528v1 [cs.CL])</a></h1>

<p><b>Authors:</b> <a href="http://arxiv.org/find/cs/1/au:+Wang_Z/0/1/0/all/0/1">Zhilin Wang</a>, <a href="http://arxiv.org/find/cs/1/au:+Dong_Y/0/1/0/all/0/1">Yi Dong</a>, <a href="http://arxiv.org/find/cs/1/au:+Zeng_J/0/1/0/all/0/1">Jiaqi Zeng</a>, <a href="http://arxiv.org/find/cs/1/au:+Adams_V/0/1/0/all/0/1">Virginia Adams</a>, <a href="http://arxiv.org/find/cs/1/au:+Sreedhar_M/0/1/0/all/0/1">Makesh Narsimhan Sreedhar</a>, <a href="http://arxiv.org/find/cs/1/au:+Egert_D/0/1/0/all/0/1">Daniel Egert</a>, <a href="http://arxiv.org/find/cs/1/au:+Delalleau_O/0/1/0/all/0/1">Olivier Delalleau</a>, <a href="http://arxiv.org/find/cs/1/au:+Scowcroft_J/0/1/0/all/0/1">Jane Polak Scowcroft</a>, <a href="http://arxiv.org/find/cs/1/au:+Kant_N/0/1/0/all/0/1">Neel Kant</a>, <a href="http://arxiv.org/find/cs/1/au:+Swope_A/0/1/0/all/0/1">Aidan Swope</a>, <a href="http://arxiv.org/find/cs/1/au:+Kuchaiev_O/0/1/0/all/0/1">Oleksii Kuchaiev</a></p>

[View remainder of file in raw view](https://github.com/Luvata/arxive/raw/9341b486ad647204168aa013996db66c9df8196a/pages/2023-11-20-cs-cl.html)

## Footer

[ ](https://github.com "GitHub") (C) 2025 GitHub, Inc. 

### Footer navigation

  * [Terms](https://docs.github.com/site-policy/github-terms/github-terms-of-service)
  * [Privacy](https://docs.github.com/site-policy/privacy-policies/github-privacy-statement)
  * [Security](https://github.com/security)
  * [Status](https://www.githubstatus.com/)
  * [Docs](https://docs.github.com/)
  * [Contact](https://support.github.com?tags=dotcom-footer)
  * Manage cookies 
  * Do not share my personal information 



You can’t perform that action at this time. 


title: Mastering Prompt Engineering Strategies and Tactics
scrape_date: 2025-03-17T13:26:50.821568
content_type: markdown
hash: 143dfc8819bebee50ebbc27a71c154c777bf769fb3870d68f7ca163998291f6d
url:
https://www.linkedin.com/pulse/mastering-prompt-engineering-strategies-tactics-krishna-srikanth-k-f8enc

content:
Agree & Join LinkedIn 

By clicking Continue to join or sign in, you agree to LinkedIn’s [User Agreement](/legal/user-agreement?trk=linkedin-tc_auth-button_user-agreement), [Privacy Policy](/legal/privacy-policy?trk=linkedin-tc_auth-button_privacy-policy), and [Cookie Policy](/legal/cookie-policy?trk=linkedin-tc_auth-button_cookie-policy). 

`` `` `` `` `` ``

##  Sign in to view more content 

Create your free account or sign in to continue your search 

Sign in 

##  Welcome back 

`` `` `` `` `` `` `` `` `` ``

Email or phone 

Password 

Show

[Forgot password?](https://www.linkedin.com/uas/request-password-reset?trk=pulse-article_contextual-sign-in-modal_sign-in-modal_forgot_password) Sign in 

or 

By clicking Continue to join or sign in, you agree to LinkedIn’s [User Agreement](/legal/user-agreement?trk=pulse-article_contextual-sign-in-modal_sign-in-modal_auth-button_user-agreement), [Privacy Policy](/legal/privacy-policy?trk=pulse-article_contextual-sign-in-modal_sign-in-modal_auth-button_privacy-policy), and [Cookie Policy](/legal/cookie-policy?trk=pulse-article_contextual-sign-in-modal_sign-in-modal_auth-button_cookie-policy). 

New to LinkedIn? [Join now](https://www.linkedin.com/signup/cold-join?session_redirect=%2Fpulse%2Fmastering-prompt-engineering-strategies-tactics-krishna-srikanth-k-f8enc&trk=pulse-article_contextual-sign-in-modal_sign-in-modal_join-link)

or 

New to LinkedIn? [Join now](https://www.linkedin.com/signup/cold-join?session_redirect=%2Fpulse%2Fmastering-prompt-engineering-strategies-tactics-krishna-srikanth-k-f8enc&trk=pulse-article_contextual-sign-in-modal_join-link)

By clicking Continue to join or sign in, you agree to LinkedIn’s [User Agreement](/legal/user-agreement?trk=linkedin-tc_auth-button_user-agreement), [Privacy Policy](/legal/privacy-policy?trk=linkedin-tc_auth-button_privacy-policy), and [Cookie Policy](/legal/cookie-policy?trk=linkedin-tc_auth-button_cookie-policy). 

`` `` `` `` `` `` `` Skip to main content  [ LinkedIn ](/?trk=article-ssr-frontend-pulse_nav-header-logo)

  * [ Articles  ](https://www.linkedin.com/pulse/topics/home/?trk=article-ssr-frontend-pulse_guest_nav_menu_articles)
  * [ People  ](https://www.linkedin.com/pub/dir/+/+?trk=article-ssr-frontend-pulse_guest_nav_menu_people)
  * [ Learning  ](https://www.linkedin.com/learning/search?trk=article-ssr-frontend-pulse_guest_nav_menu_learning)
  * [ Jobs  ](https://www.linkedin.com/jobs/search?trk=article-ssr-frontend-pulse_guest_nav_menu_jobs)
  * [ Games  ](https://www.linkedin.com/games?trk=article-ssr-frontend-pulse_guest_nav_menu_games)



[ Join now ](https://www.linkedin.com/signup/cold-join?session_redirect=%2Fpulse%2Fmastering-prompt-engineering-strategies-tactics-krishna-srikanth-k-f8enc&trk=article-ssr-frontend-pulse_nav-header-join) [ Sign in ](https://www.linkedin.com/uas/login?session_redirect=%2Fpulse%2Fmastering-prompt-engineering-strategies-tactics-krishna-srikanth-k-f8enc&fromSignIn=true&trk=article-ssr-frontend-pulse_nav-header-signin) [ ](https://www.linkedin.com/uas/login?session_redirect=%2Fpulse%2Fmastering-prompt-engineering-strategies-tactics-krishna-srikanth-k-f8enc&fromSignIn=true&trk=article-ssr-frontend-pulse_nav-header-signin)

`` `` `` ``

![Mastering Prompt Engineering Strategies and Tactics](https://media.licdn.com/dms/image/v2/D5612AQH544BVxTq70A/article-cover_image-shrink_720_1280/article-cover_image-shrink_720_1280/0/1710225165100?e=2147483647&v=beta&t=w97k0Rntcrjo4J-P7dRgfJh9N82ZQJv2vH_hYkk2yU0)

# Mastering Prompt Engineering Strategies and Tactics

  * [ Report this article ](/uas/login?session_redirect=https%3A%2F%2Fwww.linkedin.com%2Fpulse%2Fmastering-prompt-engineering-strategies-tactics-krishna-srikanth-k-f8enc&trk=article-ssr-frontend-pulse_ellipsis-menu-semaphore-sign-in-redirect&guestReportContentType=PONCHO_ARTICLE&_f=guest-reporting)



[ Krishna Srikanth K  ](https://sg.linkedin.com/in/k-sk)

###  Krishna Srikanth K 

####  Senior Solutions Architect 

Published Mar 12, 2024 

[ \+ Follow ](https://www.linkedin.com/signup/cold-join?session_redirect=%2Fpulse%2Fmastering-prompt-engineering-strategies-tactics-krishna-srikanth-k-f8enc&trk=article-ssr-frontend-pulse_publisher-author-card)

##  I. Introduction

### i. What is Prompt Engineering

Prompt engineering refers to the intentional design and crafting of prompts or inputs provided to language models, aiming to influence their behavior and generate desired outputs. It is a crucial aspect of training and utilizing natural language processing (NLP) models effectively. By carefully constructing prompts, users can guide language models to produce responses that align with specific criteria, styles, or domains.

Prompt engineering encompasses various strategies and techniques to achieve specific goals, such as controlling the tone, style, or content of generated text. This process is particularly important in fine-tuning models for specific applications, adapting them to various domains, or ensuring they generate content that meets certain criteria.

### ii. What are Large Language Models (LLMs)

LLMs typically refer to Large Language Models. Large Language Models are advanced natural language processing models that are trained on massive amounts of text data to understand and generate human-like language. These models are part of the broader field of artificial intelligence and machine learning.

Large Language Models have become instrumental in numerous applications, including text generation, translation, summarization, question-answering, and more. Their pre-training on vast amounts of data allows them to capture intricate language patterns and generate contextually relevant outputs across a wide range of tasks. These models are often fine-tuned for specific applications or industries to optimize their performance in real-world scenarios.

### iii. Key Building Blocks (Prompts and LLMs)

The integration of large language models into applications involves several steps to ensure smooth and effective incorporation. Here's an overview of the process:

1\. Select the Appropriate Model: Choose a large language model that aligns with the requirements of your application. Consider factors such as model capabilities, size, and the nature of the tasks it excels in.

2\. Access the Model: Depending on the chosen language model, you may need to access it through an API (Application Programming Interface) provided by the model's developers or deploy it on your infrastructure if the model is open source.

3\. Set Up the Development Environment: Prepare your development environment by installing the necessary libraries, frameworks, or software development kits (SDKs) required to interact with and deploy the chosen language model.

4\. Define Integration Points: Identify specific integration points within your application where the language model will be utilized. This could include areas such as text generation, summarization, sentiment analysis, or any other natural language processing task.

5\. Data Preprocessing: If your application involves processing user-generated data, ensure that the input data is pre-processed appropriately to match the format and requirements of the language model. This may include tokenization, encoding, or normalization of text data.

6\. Integration with API: If the language model is provided through an API, integrate the API into your application. This typically involves sending requests to the API with input data and receiving the model's predictions or responses.

7\. Model Deployment (if applicable): If you are using a deployable version of the language model, set up the model on your servers or cloud infrastructure. Ensure that the model is accessible and can handle the expected load from your application.

8\. Error Handling and Monitoring: Implement robust error-handling mechanisms to manage potential issues, such as network failures or API errors. Set up monitoring tools to track the performance of the integrated language model, allowing for real-time insights and troubleshooting.

9\. User Interface (UI) Integration: If applicable, integrate the language model seamlessly into the user interface of your application. Ensure that user interactions with the language model are intuitive and enhance the overall user experience.

10\. Security Considerations: Implement security measures to protect both user data and the language model itself. Use secure communication protocols, encryption, and access controls to safeguard sensitive information.

11\. Testing and Quality Assurance: Conduct thorough testing of the integrated language model across various scenarios to identify and address any issues. Ensure that the model's outputs meet the desired quality standards.

12\. Continuous Improvement: Regularly monitor the performance of the integrated language model and seek opportunities for improvement. This may involve updating the model, refining prompts, or incorporating user feedback to enhance the overall functionality of your application.

By following these steps, you can successfully integrate large language models into your applications, providing users with powerful natural language processing capabilities while ensuring a seamless and reliable user experience.

### v. Industry-Specific Usecases

Prompt engineering can have a significant impact on various industries by shaping the behavior and outputs of language models to meet specific requirements and objectives. Prompt engineering, when tailored to the specific needs of each industry, plays a crucial role in ensuring that language models generate outputs that are accurate, relevant, and aligned with industry-specific standards and requirements.

Some examples of specific industries impacted include-

1\. Customer Support and Service: In industries providing customer support and service, prompt engineering helps in crafting effective prompts for chatbots or virtual assistants. This ensures accurate, helpful, and contextually relevant responses, enhancing the overall customer experience and reducing the workload on human support agents.

2\. Education and E-Learning: In the education sector, prompt engineering aids in developing instructional materials, generating quiz questions, or assisting students in understanding complex concepts. It allows for the creation of prompts that align with educational objectives, providing tailored and informative responses.

3\. Healthcare: In healthcare, prompt engineering can be applied to guide language models in generating accurate and precise medical information. This is particularly beneficial for creating content for patient education, medical documentation, or assisting healthcare professionals with information retrieval.

4\. Content Creation and Marketing: In content creation and marketing, prompt engineering influences the generation of creative and engaging content. Marketers can use specific prompts to guide language models in producing content that aligns with brand messaging, targets specific audiences, and adheres to desired styles.

5\. Legal and Compliance: In legal and compliance industries, prompt engineering is valuable for generating accurate and contextually appropriate legal documents. It helps in crafting prompts that ensure language models produce content adhering to specific legal terminologies, compliance standards, and regulatory requirements.

6\. Technology and Software Development: In technology and software development, prompt engineering can be employed for generating code snippets, documentation, or assisting with problem-solving. It enables developers to craft prompts that align with coding conventions, ensuring language models provide relevant and functional code solutions.

7\. Market Research and Analysis: In market research, prompt engineering assists in generating reports, analyses, and insights. Researchers can use specific prompts to guide language models in producing content that addresses research questions, ensuring the extraction of valuable information.

8\. Entertainment and Gaming: In the entertainment and gaming industry, prompt engineering is crucial for creating interactive narratives, dialogues, and content. It helps in crafting prompts that elicit engaging and contextually relevant responses, enhancing the overall user experience in games and interactive storytelling.

9\. Finance and Banking: In the finance sector, prompt engineering assists in generating accurate and compliant financial reports, documents, and customer communications. It ensures language models produce content that aligns with financial regulations and provides precise information for decision-making.

10\. Human Resources: In human resources, prompt engineering is beneficial for generating HR documentation, employee communications, and training materials. It helps in crafting prompts that ensure language models produce content aligned with HR policies, procedures, and compliance requirements.

12\. Manufacturing and Supply Chain: In manufacturing and supply chain management, prompt engineering can assist in generating documentation, reports, or inventory-related content. It ensures that language models produce outputs that adhere to industry-specific terminology and standards.

13\. Real Estate: In real estate, prompt engineering helps generate property listings, descriptions, and contracts. It ensures that language models provide accurate and compelling content for real estate professionals and potential buyers.

14\. Energy and Utilities: In the energy and utilities sector, prompt engineering can assist in generating documentation, safety guidelines, or reports related to energy production and distribution. It ensures language models produce content that aligns with industry-specific terminology and safety standards.

15\. Telecommunications: In the telecommunications industry, prompt engineering aids in generating customer communications, technical documentation, or responses to user queries. It ensures that language models provide accurate and contextually relevant information in a rapidly evolving field.

By tailoring prompts to suit the unique requirements of each industry, prompt engineering ensures that language models generate outputs that are accurate, contextually appropriate, and aligned with specific goals and standards. This versatility makes prompt engineering a valuable tool across a wide range of applications and industries.

##  II. Strategies and Tactics Used

Let us understand the key aspects of this strategy and the tactics with an example used for each tactic in enabling the strategy.

### i. Specify Domain or Style

The "Specify Domain or Style" strategy in prompt engineering involves explicitly instructing a language model to generate responses that adhere to a particular domain or style of writing. This strategy is employed to tailor the model's outputs to specific contexts, industries, or writing styles, ensuring relevance and coherence. This strategic approach proves invaluable when there is a need to align language model outputs with targeted domains or exhibit predetermined styles.

For instance, when fine-tuning models for industry-specific language, this strategy ensures the incorporation of specialized terminology and jargon. In scenarios requiring a particular level of formality or informality, such as educational content creation or conversational dialogues, explicit instructions guide the language model's output to meet communication standards.

Key aspects of this strategy include-:

· Domain Definition

· Style Guidance

· Vocabulary Emphasis

· Industry Jargon Inclusion

· Task Alignment with Domain

· Consistency in Content

· Adaptation to Different Domains

· Iterative Refinement

· User Feedback Integration

· Combine with Other Strategies

Some of the tactics that can be used with the above strategy-:

1\. Domain Tags: Explicitly mention the domain in your prompt using specific tags.

Example: "[MEDICAL] Explain the process of blood clot formation."

2\. Contextual Phrasing: Incorporate contextual phrasing to set the tone for the desired domain or style.

Example: "In a legal context, discuss the implications of intellectual property rights on innovation."

3\. Task Introduction: Begin the prompt with a clear introduction that specifies the domain or style.

Example: "In the context of finance, analyze the impact of interest rate changes on the stock market."

4\. Style Indicators: Use style indicators to guide the model's response.

Example: "[FORMAL] Draft a formal letter requesting additional information about the project."

5\. Domain-Specific Terms: Include domain-specific terms or jargon to signal the intended context.

Example: "Using geological terminology, explain the process of sedimentary rock formation."

6\. Example Scenarios: Present example scenarios within the specified domain or style.

Example: "Imagine you are a historian. Describe the historical events leading up to the Industrial Revolution."

7\. Parallel Construction: Use parallel construction to mirror the language associated with a particular domain or style.

Example: "From a marketing perspective, outline the key features of the new product."

8\. Contextual Instructions: Include contextual instructions that reinforce the desired domain or style.

Example: "[SCIENTIFIC] Provide a scientific analysis of the experiment results."

9\. Custom Tokens: Create custom tokens that represent specific domains or styles.

Example: "[LAW] Analyze the legal implications of the given case."

10\. Prompt Examples: Offer prompt examples that exemplify the intended domain or style.

Example: "As a technology blogger, write a review of the latest smartphone release."

### ii. Frame as a Conversation

The "Frame as a Conversation" strategy in prompt engineering involves structuring prompts to simulate a conversational context, guiding the language model to generate responses as if it were engaged in a dialogue with the user. This strategy leverages the natural flow of conversation to prompt more contextually relevant and coherent responses.

The Frame as a Conversation prompt engineering strategy involves crafting prompts in a conversational style to achieve more natural and contextually relevant language model outputs. Particularly useful for chatbots, virtual assistants, and interactive applications, this strategy enhances user interactions, making experiences more seamless and engaging.

By adopting tactics like open-ended questions, it facilitates a more dynamic language model output. The strategy finds utility in customer support, language tutoring, and scenarios where simulating human-like interactions is crucial for an immersive user experience.

Key aspects of this strategy include-

· Conversational Tone

· Dialogue Structure

· User Engagement

· Follow-up Questions

· Contextual Awareness

· User Role Clarification

· Imitate Interaction Patterns

· Experiment with Phrasing

· Balance Formality

· Iterative Refinement

Some of the tactics that can be used with the above strategy-

1\. Dialogue Openers: Begin the prompt with a dialogue opener to set the conversational tone.

Example: "User: Can you explain the concept of renewable energy sources?"

2\. Role-Playing: Frame the prompt as a role-play scenario, assigning specific roles to the model and the user.

Example: "You are an expert chef. Share your insights on creating a gourmet meal using simple ingredients."

3\. Question-Response Structure: Structure the prompt in a question-response format, mimicking a natural conversation.

Example: "User: What are the benefits of regular exercise? Model: Well, regular exercise has several benefits, including..."

4\. Back-and-Forth Interaction: Create a back-and-forth interaction by building on the model's previous responses.

Example: "User: Tell me about famous inventors. Model: Sure, let me start with Thomas Edison. User: That's interesting, who else?"

5\. Follow-up Queries: Include follow-up queries within the prompt to simulate a continued conversation.

Example: "After explaining the process of photosynthesis, the user asks, 'How does this process vary in different plant species?'"

6\. Contextual Reference: Make contextual references to previous parts of the conversation to maintain coherence.

Example: "As we discussed earlier, let's dive deeper into the impact of climate change on ecosystems."

7\. Friendly Tone: Use a friendly and approachable tone to create a conversational atmosphere.

Example: "Hey there! I'm curious to know your thoughts on the role of artificial intelligence in healthcare."

8\. Clarifying Questions: Incorporate clarifying questions as if seeking further information in a conversation.

Example: "User: Can you explain quantum computing? Model: Sure, are you familiar with classical computing concepts?"

9\. Storytelling Approach: Frame the prompt as part of a storytelling or narrative context.

Example: "Imagine you are a historian recounting the events that led to the fall of the Roman Empire."

10\. Acknowledgment and Response: Include acknowledgments or responses to simulate the model's awareness of the ongoing conversation.

Example: "User: What's the weather like today? Model: I'll check for you. Oh, it looks like a sunny day!"

### iii. Guide the Model’s Thinking

The "Guide the Model's Thinking" strategy in prompt engineering involves providing explicit instructions to direct and shape the thought process of a language model during the generation of responses. This strategy aims to influence the model's reasoning, decision-making, and approach to problem-solving. It involves instructing the model to follow a structured approach or consider various factors, ensuring coherent and intentional outputs. Utilizing tactics like step-by-step instructions, this strategy facilitates intentional language model outputs in diverse natural language processing tasks.

Key aspects of this strategy include-

· Specify Logical Steps

· Clarity in Instructions

· Provide Context

· Highlight Relevant Information

· Encourage Consideration of Alternatives

· Specify Criteria for Evaluation

· Incorporate Constraints

· Encourage Critical Thinking

· Interactive Clarifications

· Iterative Refinement

· Experiment with Language

· Feedback Integration

Some of the tactics that can be used with the above strategy-

1\. Step-by-Step Instructions: Break down complex tasks into step-by-step instructions to guide the model's thought process.

Example: "Step 1: Identify the main components. Step 2: Analyze the relationships between these components."

2\. Thinking Aloud: Frame the prompt as if the model is thinking aloud while solving a problem or providing an explanation.

Example: "Imagine you are explaining this concept to a friend. How would you break it down for them?"

3\. Decision-Making Scenarios: Present scenarios that require decision-making, guiding the model through a logical thought process.

Example: "You are a project manager facing a tight deadline. What steps would you take to prioritize tasks and meet the deadline?"

4\. Provide Context and Constraints: Set the context and constraints to guide the model's thinking within specific boundaries.

Example: "Considering the limitations of current technology, propose a solution for enhancing energy efficiency in urban areas."

5\. Define Variables or Factors: Explicitly define variables or factors to focus the model's attention on specific elements of the task.

Example: "Given factors A, B, and C, discuss how changes in each could impact the overall outcome."

6\. Comparisons and Contrasts: Frame prompts that require the model to compare different aspects, guiding its analytical thinking.

Example: "Compare the advantages and disadvantages of two competing theories in the field."

7\. Hypothesis Generation: Encourage the model to generate hypotheses or predictions by explicitly stating that it should consider potential outcomes.

Example: "What are some possible consequences of implementing the proposed policy changes?"

8\. Explore Alternative Perspectives: Prompt the model to consider alternative perspectives or viewpoints on a given topic.

Example: "Think about how individuals with differing opinions might view this issue. Provide insights from multiple perspectives."

9\. Interpretation Challenges: Pose challenges that require the model to interpret ambiguous information, guiding its reasoning.

Example: "Given the conflicting data, explain how you would interpret and reconcile the information to draw meaningful conclusions."

10\. Clarify Assumptions: Ask the model to explicitly state its assumptions, guiding it to be aware of and consider underlying premises.

Example: "List the assumptions you are making in your analysis and discuss how they may impact the results."

### iv. Control Tokens or Codes

The "Control Tokens" or "Control Codes" strategy in prompt engineering involves incorporating specific tokens or codes within a prompt to guide and influence the behavior of a language model. These tokens act as instructions that provide explicit directives to the model, shaping its responses according to predefined criteria.

This precise approach is valuable for maintaining consistency in style, tone, and formality, emphasizing or de-emphasizing content aspects, adapting to specific domains, mitigating biases, customizing for user preferences, adhering to brand guidelines, and controlling language elements.

Key aspects of this strategy include-

· Token Insertion

· Task Specification

· Format and Structure Guidance

· Content Focus

· Bias Mitigation

· Sentiment Adjustment

· Conditional Generation

· Customization for Domains

· Temperature and Sampling Adjustment

· Iterative Experimentation

· User Interaction Guidance

· Documentation of Control Tokens

Some of the tactics that can be used with the above strategy-

1\. Explicit Instruction: Use a specific control token to provide explicit instructions.

Example: "[INSTRUCT] Provide a detailed explanation of..."

2\. Task Segmentation: Divide tasks into segments using control tokens.

Example: "[STEP1] Identify the key components. [STEP2] Analyze the implications."

3\. Conditional Statements: Introduce conditions using control tokens.

Example: "[IF] The weather is sunny, [THEN] describe outdoor activities. [ELSE] focus on indoor options."

4\. Formatting Tokens: Use tokens for formatting instructions.

Example: "[CODE] Write a function that calculates the average. [OUTPUT] Display the result."

5\. Special Keywords: Define special keywords for specific behaviors.

Example: "[SUMMARIZE] Provide a concise summary of the following paragraph."

6\. Attention Mechanism Control: Direct attention using control tokens.

Example: "[ATTENTION] Focus on the details related to customer feedback."

7\. Prompt Reinforcement: Reinforce instructions using control tokens.

Example: "[REPEAT] Emphasize the importance of safety throughout the response."

8\. Temperature and Top-P: Adjust temperature and top-p values for output control.

Example: "[TEMP_LOW] Provide a focused and specific response."

9\. Experimentation and Iteration: Experiment with different control tokens and iterate based on results.

Example: "[EXP1] Test different control tokens for optimizing code generation."

10\. Proxy Labels: Use proxy labels or codes for concept guidance.

Example: "[CATEGORY_A] Generate content related to category A."

### v. Clarify Ambiguities

The "Clarify Ambiguities" strategy in prompt engineering involves addressing and minimizing any potential ambiguities within prompts to ensure clearer and more precise understanding by the language model. This strategy aims to reduce uncertainty and improve the accuracy of the model's responses.

By adding details, specifying focus, or offering examples, the strategy ensures the language model's accurate and contextually relevant responses. It mitigates the risk of misinterpretation, enhancing precision in output, especially in professional or technical contexts. The strategy reduces errors and guides the language model to produce nuanced and accurate responses across various applications.

Key aspects of this strategy include-

· Identify Ambiguous Elements

· Provide Explicit Definitions

· Contextual Clarifications

· Specify Constraints

· Use Examples

· Avoid Ambiguous Phrasing

· Interactive Clarifications

· Address multi-meaning words

· Iterative Refinement

· Feedback Integration

· User Assistance Instructions

Some of the tactics that can be used with the above strategy-

1\. Define Terms: Clearly define any terms that could be interpreted in multiple ways.

Example: "By 'efficiency,' I mean the ability to achieve a specific task with minimal resource utilization. Please elaborate on the efficiency of the proposed solution."

2\. Specify Context: Provide additional context to narrow down the interpretation of the prompt.

Example: "In the context of environmental impact, discuss the consequences of industrial pollution on local ecosystems."

3\. Offer Examples: Provide specific examples to illustrate the intended meaning of ambiguous terms.

Example: "For instance, when I mention 'innovation,' I'm referring to the introduction of new ideas, products, or processes. Share your thoughts on the role of innovation in economic growth."

4\. Conditional Instructions: Use conditional instructions to guide the model's understanding of potential ambiguities.

Example: "If the data is inconclusive, discuss possible interpretations and their implications."

5\. Ask for Clarification: Explicitly ask the model to seek clarification if certain details are unclear.

Example: "If any part of the prompt is unclear, please ask for clarification before proceeding with your response."

6\. Limit Scope: Narrow down the scope of the prompt to reduce potential misunderstandings.

Example: "Focus specifically on the social impact of the policy changes rather than the overall implications."

7\. Provide Constraints: Set clear constraints to guide the model's interpretation of ambiguous elements.

Example: "Considering a realistic budget, propose cost-effective solutions to address the identified challenges."

8\. Scenario-Based Clarification: Pose hypothetical scenarios to help the model understand and address potential ambiguities.

Example: "Imagine you are a journalist covering a controversial topic. How would you navigate the ethical challenges associated with reporting on sensitive issues?"

9\. Highlight Key Criteria: Emphasize key criteria or parameters that should be considered in the response.

Example: "When discussing 'efficiency,' take into account both time and resource utilization as critical factors."

10\. Feedback Loop: Establish a feedback loop by asking the model to confirm its understanding before proceeding.

Example: "Confirm your understanding of the key terms and criteria before providing a detailed analysis."

### vi. Explore Different Phrasings

The "Explore Different Phrasings" strategy is employed to experiment with various linguistic formulations of prompts. This approach proves valuable when anticipating diverse user inputs, enhancing language model robustness, adapting to regional language differences, improving responsiveness to varied queries, and reducing biases in responses.

Tactics involve testing variations in sentence structure, using synonyms, and altering word order to expose the model to a range of linguistic expressions. By applying this strategy, language models become more adaptable, responsive, and less prone to biases in their outputs across a variety of natural language processing tasks.

Key aspects of this strategy include-

· Variety in Sentence Structure

· Synonyms and Alternate Phrases

· Adjustment of Tone and Style

· Phrasing for Specific Contexts

· Precision vs. Open-Endedness

· Use of Keywords

· Implicit vs. Explicit Instruction

· Consideration of User Intent

· Iterative Refinement

· User Feedback Integration

· Domain-Specific Nuances

Some of the tactics that can be used with the above strategy-

1\. Synonyms and Variations: Use synonyms and variations in your prompts to gauge how the model responds to different expressions of the same idea.

Example: "Explain the concept of artificial intelligence" vs. "Describe what artificial intelligence means."

2\. Rephrase with Contextual Changes: Change the context or perspective while rephrasing to assess the model's adaptability.

Example: "As a scientist, discuss climate change" vs. "In a political context, analyze the impacts of climate change."

3\. Negation and Affirmation: Experiment with negations and affirmations to observe how the model handles positive and negative framing.

Example: "Explain the benefits of renewable energy" vs. "Discuss the drawbacks of renewable energy."

4\. Switching Active and Passive Voice: Switch between active and passive voice to see how the model responds to changes in sentence structure.

Example: "The team implemented the solution" vs. "The solution was implemented by the team."

5\. Question vs. Statement: Convert statements into questions or vice versa to explore how the model interprets and responds.

Example: "Provide an analysis of market trends" vs. "Can you analyze current market trends?"

6\. Changing Tense and Timeframe: Alter the tense or timeframe to assess the model's ability to handle past, present, or future scenarios.

Example: "Examine historical events" vs. "Examine potential future developments."

7\. Varying Degrees of Formality: Experiment with different levels of formality to observe how the model adapts to changes in tone.

Example: "Conduct a formal analysis" vs. "Share your thoughts on the topic."

8\. Adding or Removing Details: Add or remove specific details to see how the model's responses are affected by changes in granularity.

Example: "Provide a detailed breakdown of the process" vs. "Summarize the main steps."

9\. Use of Descriptive Language: Introduce more or less descriptive language to see how the model responds to variations in detail.

Example: "Explain the concept concisely" vs. "Elaborate on the intricacies of the concept."

10\. Challenge the Model: Pose challenging or abstract phrasings to assess the model's ability to handle complex instructions.

Example: "Delve into the philosophical implications of technology" vs. "Discuss the broader ethical considerations of technological advancements."

### vii. Monitor amp; Evaluate Responses

The "Monitor & Evaluate Responses" prompt engineering strategy is crucial for assessing and analyzing language model outputs during training and deployment. This approach is applied to ensure the quality, accuracy, and appropriateness of generated responses across diverse natural language processing tasks.

Regular monitoring aids in quality assurance, enabling the identification and correction of inconsistencies or errors in language model outputs. It is instrumental in detecting and mitigating biases, adapting the model to evolving language patterns, and optimizing overall performance.

Additionally, this strategy incorporates user feedback validation, where real-world interactions are used to refine the model based on user input. By establishing evaluation metrics, conducting regular reviews, and iterating based on feedback, the "Monitor & Evaluate Responses" strategy ensures continuous improvement and adaptability in language models for diverse linguistic applications.

Key aspects of this strategy include-

· Establish Evaluation Criteria

· Regular Monitoring

· Automated Evaluation Tools

· Human Evaluation

· Feedback Collection

· Benchmarking

· Iterative Refinement

· Addressing Biases and Ethical Considerations

· Contextual Analysis

· User Experience Considerations

· Documentation of Findings

· Adaptation to User Feedback

Some of the tactics that can be used with the above strategy-

1\. Define Success Criteria: Clearly define the criteria for a successful response based on your specific goals.

Example: If the prompt is asking for a summary, success might be determined by the accuracy and conciseness of the summary.

2\. Compare Against Reference Answers: Have reference answers or ideal responses ready to compare against the model's outputs.

Example: If you're asking for a translation, compare the model's output against a professionally translated version.

3\. Evaluate Consistency: Check for consistency in responses by making similar requests multiple times.

Example: Repeat a prompt and analyze if the model provides consistent answers across different runs.

4\. Assess Relevance to Prompt: Evaluate the relevance of the model's response to the specific instructions in the prompt.

Example: If the prompt asks for advantages and disadvantages, assess whether the model addresses both aspects.

5\. Check for Bias: Be vigilant for biases in the model's responses and address them by adjusting the prompts.

Example: If bias is detected in political analysis, refine prompts to encourage a more balanced perspective.

6\. Analyze Coherence: Examine the coherence and logical flow of the model's responses.

Example: For complex topics, ensure that the model's explanations follow a logical sequence.

7\. Evaluate Fluency: Assess the fluency and naturalness of the language used in the model's responses.

Example: Check if the model's outputs read smoothly without awkward phrasing or grammatical errors.

8\. Consider Divergent Inputs: Test the model with inputs that explore the boundaries of its capabilities.

Example: Provide extreme cases or ambiguous scenarios to observe how well the model handles such inputs.

9\. Iterative Refinement: Continuously refine prompts based on the analysis of previous responses.

Example: If the model consistently provides inaccurate information, adjust prompts to provide more context or clarification.

10\. Solicit User Feedback: Gather feedback from end-users or domain experts to validate the quality of the model's responses.

Example: Use user feedback to identify areas where the model may need improvement or clarification.

11\. Track Model Confidence: Monitor the confidence scores associated with model responses to gauge reliability.

Example: If the model expresses low confidence, it may indicate areas where the prompt needs clarification.

12\. Quantitative Metrics: Implement quantitative metrics, such as accuracy or precision, to measure the performance of the model systematically.

Example: Calculate the percentage of correct answers or the similarity score between model-generated and reference answers.

### viii. Combine Instructions

The "Combine Instructions" strategy is essential for tasks requiring the language model to follow multiple instructions or constraints comprehensively. Particularly useful for complex tasks, multi-step processes, and balancing criteria, this strategy streamlines communication and guides the model to address each element cohesively.

It enhances specificity in outputs by providing detailed and nuanced guidelines within a single prompt. Employing clear and concise language, this strategy ensures that combined instructions complement each other, avoiding ambiguity.

Key aspects of this strategy include-

· Integration of Directives

· Clarity and Conciseness

· Logical Organization

· Hierarchy of Instructions

· Consideration of Constraints

· Balancing Specificity and Openness

· User-Centric Language

· Alignment with Task Goals

· Evaluation Criteria Inclusion

· Iterative Refinement

· Feedback Integration

· Documentation of Combined Instructions

Some of the tactics that can be used with the above strategy-

1\. Clear Separation: Clearly separate different instructions within the prompt to avoid ambiguity.

Example: "Provide an overview of renewable energy sources and discuss their environmental impact."

2\. Sequential Ordering: Arrange instructions in a sequential order to guide the model through a step-by-step process.

Example: "First, explain the theory behind quantum computing. Then, discuss its potential applications."

3\. Conditional Statements: Use conditional statements to create dependencies between instructions.

Example: "If the data is inconclusive, propose alternative approaches to the problem. Otherwise, analyze the existing data."

4\. Hierarchy of Importance: Assign different levels of importance to instructions, making it clear which aspects should be prioritized.

Example: "Primary focus: Evaluate the economic impact of the policy changes. Secondary: Discuss social implications."

5\. Task Integration: Integrate related tasks seamlessly to encourage a holistic approach.

Example: "Analyze the current market trends and, based on your analysis, propose strategies for market expansion."

6\. Interrelated Elements: Emphasize the interrelated nature of instructions to encourage a comprehensive response.

Example: "Consider the historical context when analyzing the causes of the economic recession and its long-term effects."

7\. Bridge Phrases: Use bridge phrases to connect different instructions and ensure a smooth transition between tasks.

Example: "After discussing the technological advancements, transition to exploring their societal impact."

8\. Parallel Construction: Use parallel construction to present instructions in a similar grammatical structure.

Example: "Examine the cultural influences on art. Explore the historical factors shaping artistic movements."

9\. Enumerate Sub-Tasks: Enumerate sub-tasks or sub-questions within the prompt to guide the model through detailed components.

Example: "List the key features of the new product. Explain how each feature contributes to its overall functionality."

10\. Interactive Dialogue: Frame instructions as an interactive dialogue, simulating a conversation with the model.

Example: "User: Explain the concept of machine learning. Model: Certainly! Let's start with the basic principles..."

11\. Synthesis of Information: Encourage the synthesis of information by combining instructions that require merging multiple perspectives or data sources.

Example: "Integrate insights from both marketing and customer feedback to propose a product improvement strategy."

12\. Iterative Refinement: Iteratively refine combined instructions based on the model's responses, adjusting the level of complexity as needed.

Example: If the model struggles with a highly complex prompt, simplify or clarify the instructions in subsequent attempts.

ix. Formatting Instructions

The "Formatting Instructions" strategy in prompt engineering involves providing explicit guidelines or commands within the prompt to instruct a language model on how to format the generated response. This strategy is particularly useful when you want to control the structure, organization, or presentation of the information in the output.

We use this strategy to ensure consistency, clarity, and professionalism in various types of content creation. This approach provides explicit guidelines on how information should be structured, presented, and formatted. By specifying the layout, style, and organization of documents, messages, or materials, we enhance readability, maintain a cohesive visual identity, and meet industry or organizational standards.

Formatting instructions are particularly valuable for complex documents, such as legal agreements, technical manuals, and academic papers, where clear presentation is essential. Additionally, this strategy aids in effective communication, helping convey information in a visually appealing and easily understandable manner across different stakeholders and industries.

Key aspects of this strategy include-

· Structural Guidance

· Heading and Subheading Usage

· Bullet Points or Numbered Lists

· Paragraph Formatting

· Table or Chart Creation

· Emphasis and Styling

· Font Size and Style

· Alignment Instructions

· Consistency Across Sections

· Experiment with Formatting Variations

· Consider User Experience

Some of the tactics that can be used with the above strategy-

1\. Specify Formatting Style: Clearly define the desired formatting style.

Example: "Compose a research paper abstract in APA format summarizing the key findings of the study."

2\. Include Structure Guidelines: Provide guidelines for the structure of the response.

Example: "Craft a response with an introduction, three main points, and a conclusion, each consisting of 2-3 sentences."

3\. Use Bullet Points or Numbering: Specify the use of bullet points or numbering for a structured response.

Example: "List three advantages of renewable energy sources using bullet points."

4\. Mention Word or Character Limits: Clearly state any word or character limits for brevity.

Example: "Summarize the article in 150 words, highlighting the main arguments."

5\. Indicate Heading Usage: Instruct the use of headings for organization.

Example: "Provide a structured response with headings for each section: Introduction, Methodology, Results, Conclusion."

6\. Specify Font and Font Size: Mention the preferred font and font size.

Example: "Compose the response in Times New Roman font, size 12."

7\. Include Citation Requirements: Specify any citation or referencing requirements.

Example: "Include APA citations for any external sources mentioned in your response."

8\. Incorporate Line Spacing Instructions: Instruct the desired line spacing for the response.

Example: "Maintain double-spacing throughout the response for readability."

9\. Clarify Image or Graph Requirements: If applicable, specify any requirements for including images or graphs.

Example: "If discussing statistical data, include a bar graph with proper labels and source citation."

10\. Provide Sample Formatting: Offer a sample or template to illustrate the expected formatting.

Example: "Format your response similar to the following: [Sample Format]."

11.Specify File Format: Instruct the preferred file format for submission.

Example: "Submit your response as a PDF document."

### x. User Feedback Loop

The "User Feedback Strategy" in prompt engineering involves actively seeking and incorporating feedback from end users or stakeholders to improve the performance and behavior of language models. This strategy recognizes the importance of user input in refining prompts, addressing issues, and enhancing the overall user experience.

The User Feedback Loop strategy is crucial for continuous language model improvement by incorporating real-world user feedback. Applied across various tasks, such as chatbots, content generation, and customer support, this strategy optimizes responses, addresses shortcomings, and ensures alignment with user preferences.

Key aspects of this strategy include-

· Solicit User Input

· Feedback on Model Outputs

· Prompt Improvement Suggestions

· Bias Identification

· User Experience Insights

· Evaluate User Preferences

· Iterative Refinement

· Incorporate Positive Feedback

· Feedback Analysis

· Communication and Transparency

· Educate Users

Some of the tactics that can be used with the above strategy-

1\. Clear Feedback Channels: Establish clear and accessible channels for users to provide feedback. Include options such as in-app forms, surveys, and contact points to encourage user input.

Example: Include a "Feedback" button in a mobile app that allows users to submit comments and suggestions directly.

2\. Prompt and Encourage Feedback: Actively prompt users to provide feedback at strategic points during their interaction with a product or service. Encourage users to share their thoughts on specific features or experiences.

Example: Trigger a feedback prompt after a user completes a transaction or utilizes a new feature within an application.

3\. Incentivize Feedback: Provide incentives for users to submit feedback, such as discounts, exclusive access, or participation in giveaways. Incentives can motivate users to share their opinions.

Example: Offer a discount code to users who take the time to complete a feedback survey about their shopping experience.

4\. Real-Time Feedback Mechanisms: Implement real-time feedback mechanisms that allow users to express their opinions while the experience is fresh in their minds. This can lead to more accurate and detailed feedback.

Example: Integrate in-app rating prompts that appear immediately after a user completes a specific action or task.

5\. Segmented Feedback Requests: Segment feedback requests based on user behavior, preferences, or demographics. Tailor feedback requests to specific user segments to gather targeted insights.

Example: Request feedback on a new feature from users who have actively engaged with that feature recently.

6\. Multi-Channel Feedback Collection: Collect feedback through multiple channels, including social media, email, and online forums. Diversifying feedback sources ensures a broader perspective.

Example: Monitor social media platforms for mentions and comments related to a product, and use those insights to inform improvements.

7\. User-Friendly Feedback Forms: Design user-friendly feedback forms that are easy to complete and provide options for both quantitative and qualitative feedback. Streamlined forms increase the likelihood of user participation.

Example: Create a concise survey with a mix of rating scales and open-ended questions to gather detailed feedback.

8\. Anonymous Feedback Options: Offer users the option to provide feedback anonymously. This can encourage users to share honest opinions without concerns about repercussions.

Example: Include an anonymous feedback option in employee satisfaction surveys to gather candid insights.

9\. Feedback Response and Acknowledgment: Acknowledge and respond to user feedback promptly. Users are more likely to provide input if they see that their opinions are valued and considered.

Example: Send automated acknowledgment emails when users submit feedback and follow up with personalized responses as needed.

10\. Feedback Analysis Tools: Utilize feedback analysis tools or sentiment analysis to process large volumes of user feedback efficiently. Automated tools can help identify patterns and trends.

Example: Use natural language processing tools to analyze customer reviews and extract common themes or issues.

11\. Regular Feedback Surveys: Implement regular feedback surveys at intervals to gauge ongoing user satisfaction and track changes over time. Scheduled surveys can provide valuable longitudinal insights.

Example: Conduct quarterly surveys to gather feedback on software updates and assess user satisfaction with new features.

12\. Integration with Product Development: Integrate user feedback directly into the product development process. Use feedback to inform feature prioritization, bug fixes, and overall product roadmaps.

Example: Maintain a transparent feedback board or system where users can see the status of their suggestions and track their implementation.

13\. Continuous Improvement Loop: Establish a continuous improvement loop where feedback is collected, analyzed, and used to iteratively enhance products or services. Demonstrate a commitment to ongoing improvement.

Example: Host regular team meetings to review user feedback, identify actionable items, and plan for future updates or optimizations.

### xi. Task Description

The "Task Description" strategy in prompt engineering involves providing a clear and detailed description of the task or question within the prompt to guide a language model in generating a response. This strategy aims to explicitly communicate the user's intent, providing specific instructions and context for the model to understand and address the task accurately.

Task descriptions are essential tools for clear communication and understanding of specific assignments, responsibilities, or objectives. Whether in project management, workplaces, education, freelance or contract work, performance reviews, task delegation, event planning, volunteer organizations, personal task management, or software development, clear task descriptions foster understanding, accountability, and effective task management.

Key aspects of this strategy include-

· Clear Task Definition

· Specific Context

· Format and Structure Guidelines

· Include Constraints

· System-Level Instructions

· Role Definition (if applicable)

· Feedback Iteration

· Combine with Example-Based Instructions

· Evaluate and Adjust

Some of the tactics that can be used with the above strategy-

1\. Clearly Define the Task: Clearly articulate the specific task or goal the model should accomplish.

Example: "Compose a detailed analysis of the impact of social media on mental health."

2\. Break Down Complex Tasks: Break down complex tasks into smaller, manageable components.

Example: "In separate paragraphs, discuss the positive and negative effects of artificial intelligence on job markets."

3\. Specify Required Elements: Clearly outline the essential elements or components expected in the response.

Example: "Craft a product review including details on features, performance, and overall user experience."

4\. Set Boundaries or Constraints: Establish any specific boundaries or constraints for the task.

Example: "Write a short story with a maximum word count of 500 words."

5\. Include Contextual Details: Provide relevant contextual details to guide the model's understanding.

Example: "Imagine you are a travel blogger. Write a blog post about your recent trip, emphasizing cultural experiences."

6\. Specify Audience or Purpose: Clearly state the intended audience or purpose of the task.

Example: "Compose a persuasive speech aimed at convincing a skeptical audience about the importance of renewable energy."

7\. Encourage Critical Thinking: Encourage critical thinking by posing questions within the task.

Example: "Analyze the given data and formulate recommendations for improving customer engagement based on the findings."

8\. Define Scope and Depth: Clearly define the scope and depth expected in the response.

Example: "Summarize the research paper, focusing on methodology, results, and implications, within a concise paragraph."

9\. Provide Relevant Background Information: Offer pertinent background information to set the stage for the task.

Example: "Considering recent advancements, write an article discussing the future implications of artificial intelligence in healthcare."

## Recommended by LinkedIn

[ A beginner’s introduction to Natural Language… CloudMoyo  2 months ago  ](https://www.linkedin.com/pulse/beginners-introduction-natural-language-processing-nlp-cloudmoyo-ctpjf)

[ What is a Large Language Model? ESP Softtech PVT LTD  7 months ago  ](https://www.linkedin.com/pulse/what-large-language-model-espsofttech-w3gaf)

[ What is Natural Language Processing?-ONE Technology… One Technology Services  2 years ago  ](https://www.linkedin.com/pulse/what-natural-language-processing-one-technology-)

10\. Reference Task-Specific Examples: Provide task-specific examples to guide the model.

Example: "Craft a dialogue between two characters discussing the pros and cons of incorporating blockchain technology in a business setting."

11\. Encourage Creativity within Task Constraints: Allow for creativity while maintaining task constraints.

Example: "Design a visually appealing infographic summarizing key statistics on climate change, ensuring clarity and accuracy."

### xii. Length Experimentation

The Length Experimentation prompt engineering strategy is implemented to fine-tune and optimize the length of generated output from language models. This strategy becomes relevant when there is a necessity to control or experiment with the length of the generated content to align with specific requirements or preferences.

The strategy involves adjusting prompts, experimenting with different length constraints, and evaluating the output to determine the most suitable length for the given task or application. In essence, the Length Experimentation strategy provides a versatile and adaptable approach to tailor the language model's output length according to specific needs and preferences in diverse natural language processing contexts.

Key aspects of this strategy include-

· Max Tokens Parameter

· Controlled Experimentation

· Output Review

· Brevity vs. Detail

· Task-Specific Considerations

· Format and Readability

· Combination with Other Strategies

· User Experience

· Iterative Adjustment

· Balance with Creativity

Some of the tactics that can be used with the above strategy-

1\. Define Content Objectives: Clearly define the objectives of the content in terms of length. Consider factors such as readability, engagement, or conveying comprehensive information.

Example: In blog writing, experiment with different post lengths to determine the optimal length for audience engagement.

2\. Short Form vs. Long Form Testing: Conduct experiments comparing short-form and long-form content. Evaluate audience response and engagement metrics to determine the ideal length for the given context.

Example: Test the effectiveness of short and concise product descriptions against longer, more detailed descriptions for e-commerce listings.

3\. A/B Testing: Implement A/B testing with variations in content length. Compare user interactions, such as click-through rates or time spent on page, to identify the length that yields the best results.

Example: A/B test email campaign content with different lengths to optimize open rates and click-through rates.

4\. Audience Segmentation: Segment the audience and tailor content length based on audience preferences. Consider different demographics and user behaviors in the length experimentation process.

Example: Customize social media posts for diverse audience segments with varied content lengths to maximize engagement.

5\. Storytelling Experiments: Experiment with storytelling techniques at different lengths. Assess how the narrative structure impacts audience engagement and comprehension.

Example: Test the effectiveness of a short, impactful story versus a longer, more detailed narrative in conveying a brand's message.

6\. Information Density Testing: Experiment with the density of information in content. Compare responses to concise, information-packed content versus content with more space and elaboration.

Example: Test the reception of educational content with varying levels of detail to find the balance between depth and accessibility.

7\. SEO Impact Analysis: Evaluate the impact of content length on search engine optimization (SEO) performance. Monitor how different content lengths affect search rankings and organic traffic.

Example: Experiment with blog post lengths to understand how they influence search engine visibility and organic reach.

8\. Time-Sensitive Content Testing: Test content length in time-sensitive contexts, such as social media updates or breaking news. Assess how quickly and effectively different lengths convey the necessary information.

Example: Experiment with tweet lengths during live events to gauge audience responsiveness and interaction.

9\. Multichannel Experiments: Extend length experimentation across multiple communication channels. Analyze how content length affects engagement on platforms such as social media, email, and websites.

Example: Test the performance of video content of different lengths on YouTube, Instagram, and other platforms to optimize viewer retention.

10\. Iterative Refinement: Use feedback and performance metrics to iteratively refine content length strategies. Continuously adjust content length based on insights gained from previous experiments.

Example: After analyzing the response to different podcast episode lengths, refine future episodes to align with listener preferences and engagement patterns.

### xiii. Incorporate Constraints

The "Incorporate Constraints" strategy in prompt engineering involves explicitly specifying limitations, guidelines, or requirements within the prompt to guide a language model in generating responses that adhere to certain conditions. The Incorporate Constraints prompt engineering strategy is employed to guide and constrain the language model's output based on predefined rules or limitations. This strategy is particularly valuable when generating content that must comply with regulatory or legal constraints, ensuring alignment with specific guidelines.

Additionally, it is used in tasks such as fine-tuning language models for tone, style, or formality, incorporating constraints to meet communication standards. Ensuring privacy and sensitivity, avoiding offensive language in public-facing content, customizing output for platforms with character limits, maintaining consistency in vocabulary or formatting, and adhering to user preferences or brand guidelines are also instances where this strategy is beneficial.

Key aspects of this strategy include-

· Explicit Definition

· Format Specifications

· Length Restrictions

· Content Inclusion/Exclusion

· System-Level Instructions

· Combining Constraints

· Iteration and Refinement

· Balancing Creativity and Constraints

· User Experience Consideration

· Task-Specific Constraints

Some of the tactics that can be used with the above strategy-

1\. Define Clear Constraints: Clearly articulate and define the constraints that need to be incorporated into the process. This provides a foundation for decision-making and creativity within set boundaries.

Example: In product design, specify constraints related to materials, dimensions, and cost.

2\. Prioritize Constraints: Prioritize constraints based on their impact and importance. Identify key constraints that significantly influence the outcome.

Example: In project management, prioritize time constraints when planning and scheduling tasks.

3\. Iterative Constraint Adjustment: Allow for iterative adjustments to constraints based on ongoing evaluation and feedback. Flexibility within the constraints ensures adaptability.

Example: In software development, iterate on project timelines and features while considering budget constraints.

4\. Incorporate User Input: Involve end-users or stakeholders in defining constraints. Incorporate their perspectives to ensure that constraints align with practical needs.

Example: In web design, gather user feedback to determine constraints related to user interface preferences and accessibility.

5\. Set Realistic Boundaries: Establish constraints that are realistic and achievable. Unrealistic constraints can hinder progress and lead to frustration.

Example: When setting performance goals for a marketing campaign, ensure budget constraints are realistic for the desired outcomes.

6\. Constraints as Creative Catalysts: Encourage creative thinking within the defined constraints. Use constraints as catalysts for innovative solutions and problem-solving.

Example: In art and design, use limited color palettes or materials to stimulate creativity within defined artistic constraints.

7\. Periodic Review and Adjustment: Periodically review and adjust constraints based on evolving project needs or external factors. Ensure that constraints remain aligned with the project's goals.

Example: In urban planning, periodically review zoning constraints to adapt to changing community needs.

8\. Cross-Functional Collaboration: Collaborate with individuals from different functional areas to identify and integrate relevant constraints. Cross-functional input ensures a comprehensive understanding of project requirements.

Example: In event planning, collaborate with logistics, marketing, and finance teams to incorporate constraints related to venue capacity, budget, and promotional strategies.

9\. Experiment within Constraints: Encourage experimentation and innovation within the established constraints. Explore different approaches to optimize outcomes.

Example: In content creation, experiment with various storytelling techniques while adhering to word count constraints.

10\. Risk Mitigation within Constraints: Identify potential risks associated with constraints and develop mitigation strategies. Proactively address challenges to minimize the impact of constraints on project success.

Example: In construction projects, anticipate weather-related constraints and implement contingency plans to avoid delays.

### xiv. Addressing Biases

The "Addressing Biases" strategy in prompt engineering involves implementing measures to identify, mitigate, or correct biases in the responses generated by language models. Biases in language models can emerge due to the biases present in the training data they were exposed to. To address biases, prompt engineering strategies aim to guide the model in producing more balanced, fair, and unbiased outputs.

The Addressing Biases prompt engineering strategy is implemented to mitigate and rectify biases present in language models, ensuring fair and unbiased responses. This strategy comes into play when biases are identified in the model's output, and the objective is to minimize or eliminate them. 

The strategy involves adjusting prompts and training the model to produce less biased output, promoting fairness and impartiality in diverse applications. Addressing biases is an ongoing process that requires continuous evaluation and refinement to enhance the overall fairness of language models in their responses and interactions.

Key aspects of this strategy include-

· Awareness of Biases

· Explicit Bias Instruction

· Counteract Stereotypes

· Diversity in Examples

· Bias Evaluation Metrics

· Adjust Prompt Language

· Incorporate Counterfactuals

· Continuous Iteration

· User Feedback Integration

· Ethical Considerations.

Some of the tactics that can be used with the above strategy-

1\. Diverse Dataset Collection: Ensure that training datasets are diverse and representative of the target population. Include varied demographics, backgrounds, and perspectives to reduce bias.

Example: When training a facial recognition model, use a dataset that includes a diverse range of skin tones, ages, and gender identities.

2\. Bias Detection and Analysis: Implement tools and processes for detecting biases in both input data and model outputs. Regularly analyze and evaluate the model's performance for potential biases.

Example: Use bias detection algorithms to identify and analyze disparities in credit scoring models across different demographic groups.

3\. Bias-Aware Algorithm Design: Design algorithms with awareness of potential biases. Consider the implications of features and parameters on different groups to minimize disparate impact.

Example: Develop recommendation algorithms that avoid reinforcing stereotypes by considering a user's individual preferences rather than generalizations.

4\. Transparency and Explainability: Prioritize transparency and explainability in model development. Make model decision-making processes understandable to users and stakeholders to address concerns related to bias.

Example: Implement model interpretability techniques, such as SHAP values, to provide insights into how the model makes decisions.

5\. Regular Model Audits: Conduct regular audits of models to assess and address potential biases. Ensure that models are re-evaluated as new data becomes available.

Example: Periodically review and audit a hiring algorithm to identify and rectify any biases in the recruitment process.

6\. Ethical AI Guidelines: Establish and adhere to ethical guidelines for AI development. Clearly define principles that prioritize fairness, equity, and the avoidance of biased outcomes.

Example: Adopt ethical AI frameworks such as the Fairness, Accountability, and Transparency in Machine Learning (FAT/ML) guidelines.

7\. User Feedback Integration: Actively seek and incorporate user feedback to identify biases and address concerns. Include diverse user perspectives in the ongoing refinement of AI systems.

Example: In a chatbot application, gather feedback from users to understand potential biases in responses and adjust the system accordingly.

8\. Bias Mitigation Techniques: Employ specific techniques to mitigate bias, such as re-weighting of training samples, adversarial training, or regularization methods.

Example: Use adversarial training to reduce bias in a sentiment analysis model by training it to resist perturbations that might introduce biased predictions.

9\. Diverse Development Teams: Build diverse development and research teams to bring a variety of perspectives and experiences to the table. A diverse team is more likely to identify and address potential biases.

Example: In a healthcare AI project, involve medical professionals, ethicists, and community representatives to provide diverse viewpoints.

10\. Bias Impact Assessments: Conduct impact assessments to understand the potential consequences of biases on different groups. Use these assessments to guide adjustments and improvements.

Example: Evaluate the impact of a language translation model on different linguistic and cultural groups to identify and rectify bias.

### xv. Critical Evaluation

The "Critical Evaluation" strategy in prompt engineering involves a thoughtful and analytical approach to assessing the outputs generated by language models. It emphasizes the need to critically evaluate the model's responses in terms of accuracy, coherence, relevance, and other criteria.

This strategy aims to iteratively refine prompts based on the insights gained from evaluating the initial model outputs.The Critical Evaluation prompt engineering strategy is applied to scrutinize and enhance the reliability, accuracy, and appropriateness of language model outputs.

This approach becomes crucial when assessing the model's responses in scenarios where precision and reliability are paramount. Its utility is evident in various natural language processing tasks, such as fact-checking, generating medical or legal information, professional writing, educational content creation, news and journalism, technical documentation, customer support responses, and drafting legal documents.

Key aspects of this strategy include-

· Establish Evaluation Criteria

· Iterative Assessment

· User Feedback Integration

· Benchmarking and Baselines

· Identify and Address Biases

· Consider Task-Specific Metrics

· Balance Creativity and Accuracy

· Contextual Considerations

· Documentation of Insights

· Iterate Based on Results

Some of the tactics that can be used with the above strategy-

1\. Clearly Define Evaluation Criteria: Establish clear and specific criteria for evaluation. Define the key metrics and standards against which the performance or outcomes will be measured.

Example: When evaluating a software application, criteria may include usability, performance, security, and user satisfaction.

2\. Quantitative and Qualitative Analysis: Combine quantitative and qualitative analysis to gain a comprehensive understanding. Use numerical data and qualitative insights to form a well-rounded assessment.

Example: Evaluate the success of a marketing campaign by analyzing both the conversion rates (quantitative) and customer feedback (qualitative).

3\. Benchmarking Against Standards: Benchmark the performance or outcomes against industry standards or best practices. This provides a reference point for understanding the relative success or areas for improvement.

Example: Benchmark the energy efficiency of a manufacturing process against industry standards to identify potential optimizations.

4\. Stakeholder Involvement: Involve relevant stakeholders in the evaluation process. Gather input from end-users, clients, and other stakeholders to ensure a holistic perspective.

Example: In the assessment of a new software feature, seek feedback from both internal teams (developers) and external users to gauge usability and functionality.

5\. Iterative Evaluation Cycles: Implement iterative evaluation cycles to continuously assess and refine processes. Regular evaluation allows for agile adjustments and improvements.

Example: In software development, conduct regular sprint reviews and retrospectives to evaluate the team's performance and identify areas for enhancement.

6\. Root Cause Analysis: Perform root cause analysis to identify the underlying factors contributing to successes or failures. Addressing root causes leads to more effective solutions.

Example: When evaluating a project delay, conduct a root cause analysis to determine if it was due to inadequate planning, resource constraints, or unforeseen challenges.

7\. Peer Review and Collaboration: Facilitate peer reviews and collaboration among team members. Encourage constructive feedback to enhance the quality and reliability of evaluations.

Example: In academic research, peer review is a common tactic to ensure the rigor and validity of study findings.

8\. Cross-Functional Assessment: Engage individuals from different functional areas to provide diverse perspectives during evaluations. Cross-functional teams can bring varied expertise to the assessment process.

Example: Evaluate the performance of a customer service department by involving representatives from marketing, operations, and IT.

9\. Performance Metrics Alignment: Align performance metrics with organizational goals and objectives. Ensure that the evaluation criteria directly contribute to overarching strategic aims.

Example: Align the performance evaluation of a sales team with revenue targets and customer satisfaction goals.

10\. Scenario Testing: Conduct scenario testing to evaluate how well a system or process performs under different conditions. Assess the robustness and adaptability of the subject under evaluation.

Example: In software development, perform scenario testing to evaluate how an application handles varying user loads or unexpected inputs.

11\. Continuous Learning Culture: Foster a culture of continuous learning and improvement. Encourage teams to embrace feedback, learn from evaluations, and proactively seek opportunities for growth.

### xvi. Use of System Prompts

The use of system prompts strategy involves incorporating specific instructions or context-setting prompts that guide the behavior of a language model during interactions. System prompts serve to provide explicit instructions to the model, influencing the nature and style of the generated responses. This strategy is often employed in fine-tuning or customizing language models to better align with user preferences or specific use cases.

By incorporating carefully crafted system-generated prompts, this approach is employed to ensure consistency in the model's responses across different queries or user interactions. It proves beneficial in fine-tuning models for specialized language domains, adapting to user preferences, maintaining brand consistency in content generation, and emphasizing specific traits in the language model's output, such as creativity or formality.

Key aspects of this strategy include-

· Guiding Model Behavior

· Customization for Specific Tasks

· Controlling Tone and Style

· Task-Specific Adaptation

· Mitigating Unintended Biases

· Fine-Tuning for User Preferences

· Scenario-Specific Guidance

· Iterative Refinement

· Enhancing Model Versatility

· Task-Specific Output Requirements

Some of the tactics that can be used with the above strategy-

1\. Clearly Define System Objectives: Clearly define the objectives and desired outcomes of the system prompts. Establish specific goals to guide the model's responses in a targeted manner.

Example: When using system prompts for a language translation model, define the objective as accurate and contextually appropriate translations.

2\. Tailor Prompts to User Intent: Craft prompts that align with the user's intent or the specific task at hand. Ensure that prompts are phrased in a way that encourages the model to generate relevant responses.

Example: In a chatbot for customer support, design prompts that prompt the model to provide helpful and accurate solutions to user queries.

3\. Experiment with Prompt Variations: Experiment with variations of prompts to understand how different phrasings impact model outputs. Iterate on prompts based on performance and user feedback.

Example: Test variations of prompts for a text generation model to identify the most effective way to elicit desired content.

4\. Incorporate Contextual Cues: Include contextual cues in prompts to provide additional information or context to the model. This helps guide the model's understanding and generate more contextually relevant responses.

Example: When instructing a summarization model, include contextual cues such as the intended audience or the key points to focus on.

5\. Balance Specificity and Generality: Strike a balance between providing specific instructions and allowing for generalization. Fine-tune prompts to be specific enough to guide the model, but not overly restrictive.

Example: In a content generation model, balance specificity by providing a general theme while allowing the model creative freedom.

6\. Iterative Refinement of Prompts: Engage in iterative refinement of prompts based on model performance and user interactions. Continuously assess and adjust prompts for optimal outcomes.

Example: Regularly review and refine prompts for a recommendation system based on user feedback and evolving preferences.

7\. Adapt to User Feedback: Actively seek and adapt to user feedback on the effectiveness of system prompts. Use feedback to enhance prompt formulations and improve the user experience.

Example: For a voice-activated assistant, adjust prompts based on user feedback regarding the clarity and effectiveness of voice commands.

8\. Dynamic Prompt Adjustments: Implement dynamic adjustments to prompts based on contextual changes or evolving user needs. Allow the system to adapt prompts dynamically for real-time responsiveness.

Example: In a weather forecasting application, dynamically adjust prompts based on the user's location and recent weather patterns.

9\. User-Centric Prompt Design: Design prompts with the end-user in mind, considering their preferences, language, and communication style. Ensure that prompts align with user expectations.

Example: Create prompts for a virtual assistant that resonate with the user's language and communication preferences to enhance user engagement.

10\. Benchmark Prompt Effectiveness: Benchmark the effectiveness of different prompts against predefined criteria. Evaluate how well prompts lead to desired outcomes and adjust strategies accordingly.

Example: Compare the performance of prompts for a recommendation system by analyzing user satisfaction and the relevance of suggested content.

### xvii. Temperature and Top-p Adjustment

The Temperature and Top-p Adjustment strategy involves fine-tuning hyperparameters in language models like GPT (Generative Pre-trained Transformer) to influence the randomness and diversity of generated outputs. These adjustments are particularly useful for controlling the creativity and specificity of responses.

The Temperature and Top-p Adjustment strategy is crucial for fine-tuning language models in natural language generation. It allows control over creativity, coherence, and length in diverse applications such as content creation, chatbots, summarization, translation, auto-completion, creative writing, and code generation. This strategy offers a customizable approach to tailor language output in different natural language processing tasks.

Key aspects of this strategy include-

1\. Temperature: Temperature is a hyperparameter used during sampling in language models. It controls the diversity of the generated text. A higher temperature results in more diverse and random outputs, while a lower temperature leads to more conservative and deterministic outputs.

With a higher temperature, the model is more likely to explore a broader range of possibilities, potentially resulting in more creative but less coherent responses.

Conversely, a lower temperature encourages the model to stick to the most probable tokens, leading to more conservative and focused responses.

2\. Top-p (Nucleus) Sampling: Top-p sampling, also known as nucleus sampling, is an alternative to traditional random sampling. Instead of considering all possible tokens, it selects from the top-p probability distribution of tokens.

The parameter p determines the cumulative probability threshold. Only tokens with cumulative probabilities that exceed this threshold are considered for sampling.

By adjusting the value of p, users can control the diversity of generated outputs. A higher p allows for more tokens to be considered, potentially resulting in more diverse outputs, while a lower p limits the selection to a narrower range of tokens, resulting in more focused outputs.

Some of the tactics that can be used with the above strategy-

1\. Temperature Adjustment: Adjusting the temperature parameter influences the randomness of the generated output. Higher values (e.g., 0.8) increase diversity, introducing more randomness, while lower values (e.g., 0.2) make the output more focused and deterministic.

Example: When generating creative writing prompts, use a higher temperature to encourage more imaginative and varied suggestions.

2\. Top-p (Nucleus) Adjustment: Top-p adjustment determines the nucleus of possible next words. Lower values (e.g., 0.8) restrict the choices to a smaller set, while higher values (e.g., 0.2) allow a broader range of words. This influences the precision and diversity of generated responses.

Example: In a chatbot scenario, adjusting Top-p can control the model's response, ensuring it selects from a narrower or broader range of possible replies based on the context.

3\. Balancing Temperature and Top-p: Experiment with a combination of temperature and Top-p adjustments to strike a balance between creativity and coherence. Fine-tune these parameters based on the desired outcome for a specific task or application.

Example: In content creation, finding the right balance can lead to varied and engaging outputs without sacrificing relevance.

4\. Task-Specific Tuning: Tailor temperature and Top-p settings based on the nature of the task. For creative writing, you might lean towards higher temperature values, while for factual information, lower values could enhance accuracy.

Example: In a summarization task, using a lower temperature with Top-p adjustment can yield concise and focused summaries.

5\. User Feedback Integration: Incorporate user feedback to iteratively refine temperature and Top-p parameters. Adjustments can be made based on the user's preferences and the desired output quality.

Example: In a conversational AI application, gather feedback on generated responses to continuously improve the user experience by adjusting the parameters accordingly.

6\. Dynamic Parameter Adjustment: Implement dynamic adjustments based on the specific context or phase of interaction. For example, start with higher temperature for creative brainstorming and then transition to lower values for refining and narrowing down options.

Example: In a brainstorming tool, dynamically adjust parameters to encourage diverse ideas initially and then converge towards more specific suggestions.

### xviii. Iterative Refinement

The Iterative Refinement strategy in prompt engineering involves an iterative process of refining and adjusting prompts based on the initial outputs from a language model. It is a method of progressively interacting with the model, analyzing its responses, and making modifications to the prompt to guide the model towards more accurate and desirable results.

This strategy comes into play when the initial model output falls short of desired criteria, typically in natural language processing tasks. It is applied in scenarios such as fine-tuning models for specific domains, mitigating biases, boosting creativity, refining language aspects, customizing output for user preferences, optimizing for specific metrics, and adapting to evolving language patterns or user expectations.

Key aspects of this strategy include-

· Start with a Simple Prompt

· Review Initial Output

· Identify Areas for Improvement

· Refine and Modify Prompt

· Submit Refined Prompt

· Experiment with Phrasing and Structure

· Fine-Tune Parameters

· Evaluate and Adjust

· Document Progress and Insights

· Iterate Based on Feedback

Some of the tactics that can be used with the above strategy-

1\. Define Clear Objectives: Clearly define the goals and objectives of the task or project at the outset. This provides a clear direction for iterative refinement.

Example: In software development, clearly define the features and functionalities expected in each iteration.

2\. Incremental Changes: Introduce changes in small increments rather than attempting large-scale modifications. This allows for easier evaluation and adjustment.

Example: In UI/UX design, iterate on small design elements before making broader interface changes.

3\. Continuous Evaluation: Regularly assess the current state of the project or task to identify areas that need improvement. Frequent evaluations enable timely adjustments.

Example: In a marketing campaign, continuously monitor performance metrics to identify and refine strategies based on real-time data.

4\. Gather Feedback: Actively seek feedback from stakeholders or end-users. Feedback provides valuable insights for refinement and ensures that adjustments align with user expectations.

Example: In content creation, gather feedback on drafts to refine the writing style, tone, and overall impact.

5\. Prioritize Issues: Prioritize identified issues or areas for improvement based on their impact and importance. Address critical issues first before moving on to less significant ones.

Example: In project management, prioritize tasks based on their impact on project timelines and goals.

6\. Data-Driven Decision Making: Base refinement decisions on data and analytics. Analyze relevant data to inform adjustments and ensure that changes are data-driven.

Example: In e-commerce, analyze user behavior data to refine the product recommendation algorithm for better personalization.

7\. Maintain Flexibility: Stay flexible and adaptable throughout the iterative process. Be open to adjusting the refinement strategy based on evolving requirements or unforeseen challenges.

Example: In software development, adapt the project plan based on changing customer needs or emerging technologies.

8\. Document Changes: Keep detailed documentation of each iteration, including changes made, lessons learned, and outcomes. Documentation facilitates knowledge transfer and informs future iterations.

Example: In research projects, document changes made to experimental setups and methodologies in each iteration.

9\. Involve Cross-Functional Teams: Include members with diverse expertise in the refinement process. Cross-functional collaboration ensures a comprehensive approach to problem-solving.

Example: In product development, involve engineers, designers, and marketers in the iterative refinement process to address various aspects of the product.

10\. Celebrate Achievements: Acknowledge and celebrate achievements and milestones reached in each iteration. Positive reinforcement boosts morale and motivation for the refinement process.

Example: In a content creation team, celebrate successful campaigns or impactful pieces to motivate the team for further improvements.

### xix. Explicit Instruction for Creativity

The Explicit Instruction for Creativity prompt engineering strategy is utilized to explicitly guide and enhance creative elements in language model outputs. This strategic approach is particularly beneficial in various natural language processing tasks where creativity is a key requirement. In applications like creative writing, poetry generation, and storytelling, this strategy involves providing clear instructions to encourage the language model to produce imaginative and inventive content.

It is also applied in content creation for marketing, aiming to generate engaging and creative language for advertisements or promotional materials. Additionally, when seeking innovative ideas, brainstorming concepts, or creating entertainment content such as jokes or memes, explicit instructions stimulate creative thinking in the language model outputs.

Key aspects of this strategy include-

· Encouraging Creative Thinking

· Use of Open-Ended Questions

· Task Framing for Creativity

· Incorporate Creative Elements

· Specify Desired Tone or Style

· Balance Constraints and Freedom

· Iterative Refinement

· Encourage Divergent Thinking

· Explore Hypothetical Scenarios

· Feedback Incorporation

· Experiment with Language

Some of the tactics that can be used with the above strategy-

1\. Encouraging Creative Storytelling: Specify the task as creative storytelling, encouraging the model to generate imaginative and inventive narratives.

Example: "Generate a captivating and imaginative short story set in a futuristic world where humans coexist with advanced AI companions."

2\. Poetry Generation: Instruct the model explicitly to generate poetic expressions, fostering creativity in language use.

Example: "Compose a short poem that evokes a sense of wonder and beauty, exploring the theme of nature and technology harmoniously intertwining."

3\. Interactive Dialogue with Characters: Frame the prompt as an interactive dialogue with fictional characters, prompting the model to generate creative responses.

Example: "Engage in a dialogue between two fictional characters discussing the possibilities of time travel, injecting humor and unexpected twists into their conversation."

4\. Generating Creative Marketing Slogans: Task the model with creating catchy and creative marketing slogans to promote a fictional product or concept.

Example: "Devise compelling and imaginative slogans for a new line of eco-friendly, space-themed energy drinks aimed at inspiring adventure."

5\. Fantasy World Building: Instruct the model to build a fantastical world with vivid details, fostering creativity in describing unique landscapes, creatures, and cultures.

Example: "Create an imaginative and detailed description of a magical realm inhabited by sentient floating islands and mythical creatures, emphasizing a sense of awe and enchantment."

6\. Humorous Content Creation: Task the language model with generating humorous content, encouraging the inclusion of wit and cleverness.

Example: "Craft a series of amusing one-liners or jokes related to the challenges faced by robots attempting to understand human emotions."

### xx. Contextual Description

The Contextual Description strategy is versatile, enhancing communication by providing additional background or details for better comprehension. It aids in various contexts, including e-commerce, education, technical documentation, project management, business reports, customer support, marketing campaigns, legal documents, and communication plans. Whether clarifying product features, guiding students in complex subjects, or informing stakeholders about business trends, this strategy ensures a comprehensive understanding in diverse scenarios.

Key aspects of this strategy include-

· Rich Contextual Information

· Situational Details

· User Role or Perspective

· Temporal or Spatial Context

· Task-Specific Constraints

· Iterative Refinement

· Combine with Other Strategies

Some of the tactics that can be used with the above strategy-

1\. Establish Scenario Background: Clearly outline the background or scenario to provide context for the task.

Example: "Imagine you are a tech journalist reporting on the latest developments in artificial intelligence. Write an article summarizing the key breakthroughs."

2\. Specify Relevant Details: Specify relevant details that shape the context for the model.

Example: "As a historian, craft a summary of the cultural changes in the United States during the 1960s, highlighting the impact on societal norms and values."

3\. Set Historical or Temporal Context: Establish a historical or temporal context for the task.

Example: "Transport yourself to the Renaissance era. Write a letter as if you were Leonardo da Vinci discussing the importance of creativity in art and science."

4\. Provide Character Roles: Assign roles or characters to create a contextual framework.

Example: "You are a sustainability consultant. Draft a proposal outlining eco-friendly practices for a local business seeking to reduce its environmental impact."

5\. Incorporate Setting Details: Include setting details to immerse the model in a specific environment.

Example: "Picture yourself in a bustling urban market. Write a descriptive piece capturing the sights, sounds, and aromas that characterize the scene."

6\. Specify Social or Cultural Context: Specify the social or cultural context to influence the model's perspective.

Example: "Considering contemporary social dynamics, write a blog post discussing the evolving role of women in technology."

7\. Use Imaginary Scenarios: Create imaginary scenarios to set the stage for the task.

Example: "In a futuristic society, draft a policy proposal addressing the ethical considerations of advanced artificial intelligence integration into everyday life."

8\. Encourage Personal Reflection: Encourage the model to adopt a personal or reflective stance within a given context.

Example: "Reflect on a meaningful personal experience and write a narrative essay discussing the lessons learned and their impact on your perspective."

9\. Specify Tone or Mood: Set the tone or mood for the context to guide the model's writing style.

Example: "In a light-hearted tone, compose a humorous script for a short skit centered around everyday challenges in a futuristic world."

10\. Reference Relevant Events: Reference specific events to anchor the task in a relevant context.

Example: "Considering recent environmental protests, write an op-ed discussing the role of activism in shaping climate policies."

### xxi. Example Based Instructions

The "Example-Based Instructions" strategy involves providing specific examples within a prompt to guide a language model in generating responses that align with those examples. Instead of relying solely on abstract instructions, this strategy leverages concrete instances to illustrate the desired output.

By including examples, users can offer explicit demonstrations of the format, content, or structure they expect in the generated response. The model then learns from these examples to produce outputs that closely match the provided instances. This strategy aims to enhance clarity and specificity in the generated responses by grounding instructions in tangible examples.

The example-based instructions strategy is employed across various instructional contexts, from education and technical documentation to programming and problem-solving scenarios. It enhances comprehension by providing practical illustrations that bridge theoretical concepts with real-world applications.

Key aspects of this strategy include-

· Concrete Demonstrations

· Clarity Enhancement

· Structural Guidance

· Content Specificity

· Variety of Examples

· Iterative Refinement

· Alignment with User Expectations

· Example Combination

· Feedback Incorporation

· Specify Context

· Balancing Constraints and Creativity

Some of the tactics that can be used with the above strategy-

1\. Provide Clear Examples: Include concrete examples to illustrate the desired output.

Example: "Craft a paragraph summarizing the impact of climate change using specific examples such as extreme weather events and rising sea levels."

2\. Use Case Scenarios: Present real-world scenarios to guide the model's understanding.

Example: "Imagine you are a technology journalist. Write a news article discussing the implications of the latest advancements in artificial intelligence, citing examples from recent breakthroughs."

3\. Reference Previous Successful Outputs: Refer to successful past outputs as benchmarks.

Example: "Generate a creative piece inspired by the style and tone of the short story 'The Garden of Forking Paths' by Jorge Luis Borges."

4\. Include Diverse Examples: Incorporate a variety of examples to ensure a comprehensive understanding.

Example: "Compose a dialogue between two characters discussing the benefits and drawbacks of renewable energy, considering examples from solar and wind power."

5\. Specify Tone and Style with Examples: Define the desired tone and style through examples.

Example: "Write a persuasive article advocating for environmental conservation, emulating the passionate and informative tone found in articles like 'Silent Spring' by Rachel Carson."

6\. Highlight Relevant Keywords: Emphasize keywords that should be incorporated into the response.

Example: "Construct a product description for a new smartphone, ensuring inclusion of keywords such as 'innovative design,' 'advanced features,' and 'user-friendly interface.'"

7\. Mention Successful Patterns: Highlight successful patterns in previous responses for emulation.

Example: "Generate a poem evoking emotions similar to those found in renowned works such as 'The Raven' by Edgar Allan Poe."

8\. Encourage Creativity with Examples: Encourage creativity by providing open-ended examples.

Example: "Create a fictional short story with unexpected plot twists, drawing inspiration from authors like O. Henry."

9\. Specify Use of Literary Devices: Instruct the use of specific literary devices through examples.

Example: "Craft a persuasive speech utilizing rhetorical devices like metaphors, similes, and parallelism to strengthen your argument."

10\. Request Multiple Examples: Ask for multiple examples to ensure diversity and depth.

Example: "Provide three distinct examples illustrating the concept of 'artificial intelligence ethics' in different societal contexts."

### xxii. Task Decomposition

The "Task Decomposition" strategy in prompt engineering involves breaking down a complex or multifaceted task into simpler, more manageable sub-tasks or components. By decomposing the task, the goal is to guide the language model more effectively, providing clearer instructions and improving the chances of obtaining accurate and coherent responses.

The Task Description prompt engineering strategy plays a pivotal role in guiding language models by providing explicit and specific instructions for generating content tailored to tasks or objectives. This strategy is crucial when there is a need to ensure that language models produce output aligned with specific task requirements in various natural language processing applications.

Key aspects of this strategy include- 

· Identify Sub-Tasks

· Create Separate Prompts for Sub-Tasks

· Specify Dependencies

· Sequential or Parallel Execution

· Aggregate Responses

· Iterative Refinement

· Evaluate Model Understanding

· Balance Complexity

· Address Ambiguity

· User Guidance

Some of the tactics that can be used with the above strategy-

1\. Define Clear Objectives: Clearly define the overall objective of the task before decomposing it. Ensure a shared understanding of the end goal among team members.

Example: Before decomposing a software development project, clearly define the desired features and functionalities.

2\. Identify Key Components: Identify the key components or sub-tasks that contribute to the overall task. Break down the task into smaller units based on its inherent structure.

Example: For a marketing campaign, identify sub-tasks such as content creation, social media promotion, and performance analysis.

3\. Hierarchical Decomposition: Organize sub-tasks in a hierarchical structure. Break down the task into major components and further decompose each component into smaller, more specific tasks.

Example: Decompose a project management task into phases, then break down each phase into specific tasks and activities.

4\. Dependency Analysis: Analyze dependencies between sub-tasks to understand the sequence and relationships. Identify tasks that can be worked on concurrently and those that depend on others.

Example: In software development, identify tasks that require the completion of certain modules before others can begin.

5\. Parallel Execution: Exploit opportunities for parallel execution by decomposing the task in a way that allows team members to work on different aspects simultaneously.

Example: Decompose a research project so that data collection and analysis can occur concurrently.

6\. Iterative Decomposition: Adopt an iterative approach to decomposition. Refine the decomposition based on feedback, changing requirements, and the evolving understanding of the task.

Example: In project management, regularly review and adjust task decomposition as project requirements evolve.

7\. Assign Responsibility: Clearly assign responsibility for each sub-task to specific team members. Ensure that individuals or teams are accountable for the successful completion of their assigned components.

Example: Assign different team members to handle content creation, design, and development in a website development project.

8\. Modular Development: Approach task decomposition with a modular mindset. Design sub-tasks as modular components that can be developed, tested, and integrated independently.

Example: Decompose a software application into modular features that can be developed and tested independently before integration.

9\. Continuous Monitoring: Continuously monitor the progress of each sub-task. Regularly assess whether sub-tasks are on track and address any challenges or roadblocks promptly.

Example: Use project management tools to monitor the progress of individual tasks and identify any delays or issues.

10\. Feedback Integration: Integrate feedback loops into the task decomposition process. Gather feedback on completed sub-tasks to inform adjustments and improvements for subsequent phases.

Example: Collect feedback from users after the completion of specific features in a software application to guide further development.

11\. Use of Templates: Develop templates or guidelines for task decomposition to maintain consistency across projects. Provide a structured framework for breaking down tasks.

Example: Create a template for decomposing marketing campaigns, including sections for target audience analysis, content creation, and promotion strategies.

### xxiii. Clarity amp; Specificity

The "Clarity & Specificity" strategy in prompt engineering involves formulating prompts clearly, unambiguous, and specific. This strategy is designed to reduce potential confusion and guide the language model to generate responses that align closely with the user's intentions.

This strategy minimizes ambiguity, reduces the risk of misinterpretation, and facilitates more effective communication between parties. Specificity further refines communication by providing explicit details, instructions, or requirements. It leaves little room for confusion and ambiguity, enabling better alignment of expectations and outcomes.

Whether in written instructions, project briefs, or any form of communication, the clarity and specificity strategy promote precision, reduces misunderstandings, and ultimately contributes to successful and efficient interactions.

Key aspects of this strategy include-

· Clear Articulation

· Specific Instruction

· Explicit Details

· Avoid Ambiguity

· Contextual Clarifications

· Specify Constraints Clearly

· Iterative Refinement

· Task Definition

· System-Level Instructions

· Precision in Phrasing

Some of the tactics that can be used with the above strategy-

1\. Specify the Task Clearly: Clearly define the task or instruction to leave no ambiguity.

Vague: "Discuss." Refined: "Discuss the impact of renewable energy on climate change."

2\. Provide Details and Context: Include additional details to guide the model.

Vague: "Explain." Refined: "Explain the concept of blockchain technology and its applications in the financial industry."

3\. Use Examples to Illustrate: Include examples to illustrate the desired output.

Vague: "Describe." Refined: "Describe the characteristics and habitat of the endangered Bengal tiger."

4\. Set Clear Scope: Clearly set the scope of the task.

Vague: "Analyze." Refined: "Analyze the economic factors contributing to the decline of the global tourism industry post-pandemic."

5\. Include Relevant Keywords: Use specific keywords to guide the model.

Vague: "Explore." Refined: "Explore the technological advancements in artificial intelligence within the last five years."

6\. Specify Parameters or Conditions: Clearly define parameters or conditions if applicable.

Vague: "Compare." Refined: "Compare and contrast the cultural influences in two different Shakespearean plays."

7\. Address a Specific Audience: Specify the target audience or context.

Vague: "Examine." Refined: "Examine the psychological effects of social media on teenagers aged 13-18."

8\. Reference Specific Content: Reference specific content or topics.

Vague: "Summarize." Refined: "Summarize the key findings of the research paper titled 'The Impact of Climate Change on Coastal Ecosystems.'"

9\. Specify Timeframe or Historical Context: Include a timeframe or historical context.

Vague: "Discuss." Refined: "Discuss the socio-economic changes in the United States during the Great Depression."

10\. Indicate Required Format: Specify the desired format of the response.

Vague: "Review." Refined: "Write a critical review of the latest science fiction novel, focusing on character development and plot structure."

### xxiv. Formatting Instructions

The "Formatting Instructions" strategy in prompt engineering involves providing explicit guidelines or commands within the prompt to instruct a language model on how to format the generated response. This strategy is particularly useful when you want to control the structure, organization, or presentation of the information in the output. By including formatting instructions, you guide the model in producing responses that adhere to specific formatting requirements. This can be beneficial for tasks where the clarity and visual presentation of the information are crucial.

We use this strategy to ensure consistency, clarity, and professionalism in various types of content creation. This approach provides explicit guidelines on how information should be structured, presented, and formatted. By specifying the layout, style, and organization of documents, messages, or materials, we enhance readability, maintain a cohesive visual identity, and meet industry or organizational standards.

Formatting instructions are particularly valuable for complex documents, such as legal agreements, technical manuals, and academic papers, where clear presentation is essential. Additionally, this strategy aids in effective communication, helping convey information in a visually appealing and easily understandable manner across different stakeholders and industries.

Key aspects of this strategy include-

· Structural Guidance

· Heading and Subheading Usage

· Bullet Points or Numbered Lists

· Paragraph Formatting

· Table or Chart Creation

· Emphasis and Styling

· Font Size and Style

· Alignment Instructions

· Consistency Across Sections

· Experiment with Formatting Variations

· Consider User Experience

Some of the tactics that can be used with the above strategy-

1\. Specify Formatting Style: Clearly define the desired formatting style.

Example: "Compose a research paper abstract in APA format summarizing the key findings of the study."

2\. Include Structure Guidelines: Provide guidelines for the structure of the response.

Example: "Craft a response with an introduction, three main points, and a conclusion, each consisting of 2-3 sentences."

3\. Use Bullet Points or Numbering: Specify the use of bullet points or numbering for a structured response.

Example: "List three advantages of renewable energy sources using bullet points."

4\. Mention Word or Character Limits: Clearly state any word or character limits for brevity.

Example: "Summarize the article in 150 words, highlighting the main arguments."

5\. Indicate Heading Usage: Instruct the use of headings for organization.

Example: "Provide a structured response with headings for each section: Introduction, Methodology, Results, Conclusion."

6\. Specify Font and Font Size: Mention the preferred font and font size.

Example: "Compose the response in Times New Roman font, size 12."

7\. Include Citation Requirements: Specify any citation or referencing requirements.

Example: "Include APA citations for any external sources mentioned in your response."

8\. Incorporate Line Spacing Instructions: Instruct the desired line spacing for the response.

Example: "Maintain double-spacing throughout the response for readability."

9\. Clarify Image or Graph Requirements: If applicable, specify any requirements for including images or graphs.

Example: "If discussing statistical data, include a bar graph with proper labels and source citation."

10\. Provide Sample Formatting: Offer a sample or template to illustrate the expected formatting.

Example: "Format your response similar to the following: [Sample Format]."

11\. Specify File Format: Instruct the preferred file format for submission.

Example: "Submit your response as a PDF document."

##  III. Conclusion

In conclusion, prompt engineering stands as a critical facet in unleashing the full potential of large language models (LLMs). It involves a nuanced approach, necessitating clarity, precision, and strategic thinking to elicit desired responses. However, this practice is not without its challenges.

Ambiguity, bias, ethical considerations, and the complexity of contextual understanding require continuous attention. Crafting prompts that strike the right balance between specificity and generality is an ongoing challenge. Despite these hurdles, prompt engineering remains a powerful tool for tailoring LLMs to diverse tasks, from content creation to problem-solving.

The iterative nature of experimentation and refinement, coupled with the incorporation of human feedback, ensures the adaptability of prompt engineering strategies. As we advance, addressing these challenges responsibly is paramount to harnessing the potential of LLMs effectively and ethically across a spectrum of applications.

  


`` `` `` `` ``

``

[ Like ](https://www.linkedin.com/signup/cold-join?session_redirect=%2Fpulse%2Fmastering-prompt-engineering-strategies-tactics-krishna-srikanth-k-f8enc&trk=article-ssr-frontend-pulse_x-social-details_like-toggle_like-cta)

Like

Celebrate

Support

Love

Insightful

Funny

[ Comment  ](https://www.linkedin.com/signup/cold-join?session_redirect=%2Fpulse%2Fmastering-prompt-engineering-strategies-tactics-krishna-srikanth-k-f8enc&trk=article-ssr-frontend-pulse_comment-cta)

`` ``

  * Copy
  * LinkedIn
  * Facebook
  * Twitter

Share 

`` ``

[ 9  ](https://www.linkedin.com/signup/cold-join?session_redirect=%2Fpulse%2Fmastering-prompt-engineering-strategies-tactics-krishna-srikanth-k-f8enc&trk=article-ssr-frontend-pulse_x-social-details_likes-count_social-actions-reactions) `` `` `` `` `` `` ``

To view or add a comment, [sign in](https://www.linkedin.com/signup/cold-join?session_redirect=%2Fpulse%2Fmastering-prompt-engineering-strategies-tactics-krishna-srikanth-k-f8enc&trk=article-ssr-frontend-pulse_x-social-details_feed-cta-banner-cta)

##  More articles by Krishna Srikanth K 

  * [ Building Scalable Data Pipelines: The Role of Medallion, Lambda, and Kappa Architectures  ](https://www.linkedin.com/pulse/building-scalable-data-pipelines-role-medallion-lambda-k-9npbc)

Mar 6, 2025

###  Building Scalable Data Pipelines: The Role of Medallion, Lambda, and Kappa Architectures 

Medallion Architecture is a modern data architecture approach designed to improve the efficiency, scalability, and… 

`` ``

3 

`` `` `` `` `` `` ``

  * [ Mastering Azure Landing Zones: Design Considerations and Implementation Best Practices  ](https://www.linkedin.com/pulse/mastering-azure-landing-zones-design-considerations-implementation-k-wcbpc)

Feb 22, 2025

###  Mastering Azure Landing Zones: Design Considerations and Implementation Best Practices 

1\. Introduction to Azure Landing Zones Overview of Azure Landing Zones Landing Zones serve as a foundation for building… 

`` ``

11 

`` `` `` `` `` `` ``

  * [ How Generative AI Models Optimize Code  ](https://www.linkedin.com/pulse/how-generative-ai-models-optimize-code-krishna-srikanth-k-wovtc)

Nov 7, 2024

###  How Generative AI Models Optimize Code 

The article "How Generative AI Models Optimize Code" is highly timely as the role of AI in software development is… 

`` ``

4 

`` `` `` `` `` `` ``

  * [ Data Excellence: Leveraging a Target-Driven Approach and Ralph Kimball's Principles for Effective KPIs and KDEs  ](https://www.linkedin.com/pulse/data-excellence-leveraging-target-driven-approach-ralph-k-mh6jc)

Sep 29, 2024

###  Data Excellence: Leveraging a Target-Driven Approach and Ralph Kimball's Principles for Effective KPIs and KDEs 

Introduction The growing importance of data in shaping business strategy has transformed how organizations operate and… 

`` ``

11 

`` `` `` `` `` `` ``

  * [ Revolutionizing AI Landscapes: Leveraging Azure OpenAI Models for Diverse Functions and Fine-Tuned Solutions  ](https://www.linkedin.com/pulse/revolutionizing-ai-landscapes-leveraging-azure-openai-k-4kgpf)

May 7, 2024

###  Revolutionizing AI Landscapes: Leveraging Azure OpenAI Models for Diverse Functions and Fine-Tuned Solutions 

I. Introduction Influence of Azure OpenAI Models on the AI landscape. 

`` ``

7 

`` `` `` `` `` `` ``

  * [ Harnessing the Power of Tagging: Strategies for Effective Cloud Resource Management  ](https://www.linkedin.com/pulse/harnessing-power-tagging-strategies-effective-cloud-resource-k-r7sqc)

Apr 18, 2024

###  Harnessing the Power of Tagging: Strategies for Effective Cloud Resource Management 

Prologue In the evolving landscape of cloud computing, effective resource management is essential for organizations to… 

`` ``

4 

`` `` `` `` `` `` ``

  * [ Understanding the Data Vault Model: ABC to Advanced Strategies and Best Practices for Data Vault Modeling  ](https://www.linkedin.com/pulse/understanding-data-vault-model-abc-advanced-strategies-k-qpfxc)

Apr 9, 2024

###  Understanding the Data Vault Model: ABC to Advanced Strategies and Best Practices for Data Vault Modeling 

I. Introduction to Data Modeling A. 

`` ``

12 

`` `` `` `` `` `` ``

  * [ Maximizing Value: Cost Management and Optimization Strategies in Azure Kubernetes Service (AKS)  ](https://www.linkedin.com/pulse/maximizing-value-cost-management-optimization-strategies-k-lmgwc)

Apr 4, 2024

###  Maximizing Value: Cost Management and Optimization Strategies in Azure Kubernetes Service (AKS) 

Prologue In today's rapidly evolving digital landscape, businesses are increasingly turning to cloud-native… 

`` ``

5 

`` `` `` `` `` `` ``

  * [ Choosing the Right Programming Language for Cloud-Native Applications  ](https://www.linkedin.com/pulse/choosing-right-programming-language-cloud-native-applications-k-2o6qc)

Apr 2, 2024

###  Choosing the Right Programming Language for Cloud-Native Applications 

Introduction A. Definition of cloud-native applications Cloud-native applications are software applications… 

`` ``

14 

`` `` `` `` `` `` ``

1 Comment 

  * [ Demystifying Azure App Configuration: Centralized Configuration Management for Modern Applications  ](https://www.linkedin.com/pulse/demystifying-azure-app-configuration-centralized-management-k-wlkfc)

Mar 28, 2024

###  Demystifying Azure App Configuration: Centralized Configuration Management for Modern Applications 

1\. Introduction i. 

`` ``

10 

`` `` `` `` `` `` ``


Show more 

[ See all articles ](https://sg.linkedin.com/in/k-sk/recent-activity/articles/)

## Sign in

Stay updated on your professional world

[ Sign in ](https://www.linkedin.com/uas/login?session_redirect=%2Fpulse%2Fmastering-prompt-engineering-strategies-tactics-krishna-srikanth-k-f8enc&trk=article-ssr-frontend-pulse_xandr-ad-fallback_signin)

By clicking Continue to join or sign in, you agree to LinkedIn’s [User Agreement](/legal/user-agreement?trk=article-ssr-frontend-pulse_auth-button_user-agreement), [Privacy Policy](/legal/privacy-policy?trk=article-ssr-frontend-pulse_auth-button_privacy-policy), and [Cookie Policy](/legal/cookie-policy?trk=article-ssr-frontend-pulse_auth-button_cookie-policy). 

New to LinkedIn? [Join now](https://www.linkedin.com/signup/cold-join?session_redirect=%2Fpulse%2Fmastering-prompt-engineering-strategies-tactics-krishna-srikanth-k-f8enc&trk=article-ssr-frontend-pulse_xandr-ad-fallback_join-link)

##  Insights from the community 

  * [ Data Structures  What are some common applications and challenges of string matching in natural language processing?  ](https://www.linkedin.com/advice/1/what-some-common-applications-challenges-string)
  * [ Artificial Intelligence  How can you use natural language to interact with computers?  ](https://www.linkedin.com/advice/1/how-can-you-use-natural-language-interact)
  * [ Data Science  How do you use linear algebra and calculus to implement natural language processing techniques?  ](https://www.linkedin.com/advice/1/how-do-you-use-linear-algebra-calculus-implement-natural)
  * [ Computer Science  How can you make your website accessible to everyone with machine learning?  ](https://www.linkedin.com/advice/3/how-can-you-make-your-website-accessible-everyone-mmpee)
  * [ Artificial Intelligence  How can you use transformers to generate text in multiple languages?  ](https://www.linkedin.com/advice/0/how-can-you-use-transformers-generate-text-u3zqf)
  * [ Operations Research  What do you do if you want to enhance your career in Operations Research with natural language processing?  ](https://www.linkedin.com/advice/3/what-do-you-want-enhance-your-career-operations-fsjie)
  * [ Search  What are the most effective methods for resolving ambiguity and anaphora in natural language search?  ](https://www.linkedin.com/advice/0/what-most-effective-methods-resolving-ambiguity-anaphora)
  * [ Natural Language Processing  What are the main challenges of named entity recognition for low-resource languages?  ](https://www.linkedin.com/advice/0/what-main-challenges-named-entity-recognition)
  * [ Predictive Modeling  What are some emerging trends and challenges in feature engineering for natural language processing?  ](https://www.linkedin.com/advice/0/what-some-emerging-trends-challenges-feature)
  * [ Natural Language Processing  What are some best practices for using word embeddings to compare texts?  ](https://www.linkedin.com/advice/3/what-some-best-practices-using-word)

Show more  Show less 

##  Others also viewed 

  * ### [ Getting started with Dialogflow  TMedia Business Solution Pvt. Ltd.  2y  ](https://www.linkedin.com/pulse/getting-started-dialogflow-tmedia-business-solution)
  * ### [ The Next Evolution of AI: Trading Tokens for Concepts - Large Concept Models  Ganesh Raju  2mo  ](https://www.linkedin.com/pulse/next-evolution-ai-trading-tokens-concepts-large-concept-ganesh-raju-cdgwc)
  * ### [ Byte-Pair Encoding, WordPiece, and Unigram Tokenization  Jyoti Dabass, Ph.D  3mo  ](https://www.linkedin.com/pulse/byte-pair-encoding-wordpiece-unigram-tokenization-jyoti-dabass-ph-d-mjdre)
  * ### [ Retrieval Augmented Generation (RAG)  Giovanni MASI  7mo  ](https://www.linkedin.com/pulse/retrieval-augmented-generation-rag-giovanni-masi-vulbf)
  * ### [ LLMs and False Promise of Creativity; LLMs as Optimizers; Running Thousands of LLMs on One GPU; 10 GPTs You Should Know; and More  Danny Butvinik  1y  ](https://www.linkedin.com/pulse/llms-false-promise-creativity-optimizers-running-one-gpu-butvinik-fqnif)
  * ### [ Exploring Text Classification with Large Language Models: A Deeper Dive  Sanjay Kumar MBA,MS,PhD  1y  ](https://www.linkedin.com/pulse/exploring-text-classification-large-language-models-dive-mba-ms-phd-lxr4c)
  * ### [ Exploring Popular LLM Orchestration Frameworks  Dr. Rabi Prasad Padhy  11mo  ](https://www.linkedin.com/pulse/exploring-popular-llm-orchestration-frameworks-dr-rabi-prasad-hjxkc)
  * ### [ Q&A with Chat-GPT about Prompt Engineering  Trisna Widia  2y  ](https://www.linkedin.com/pulse/qa-chat-gpt-prompt-engineering-trisna-widia)
  * ### [ Phi-2: A Small Language Model That Packs a Big Punch  ChandraKumar R Pillai  1y  ](https://www.linkedin.com/pulse/phi-2-small-language-model-packs-big-punch-chandrakumar-r-pillai-jzave)
  * ### [ Understanding Tokenization in Natural Language Processing: The Foundation of Text Analysis  Nasr Ullah  4mo  ](https://www.linkedin.com/pulse/understanding-tokenization-natural-language-processing-nasr-ullah-7rk4f)

Show more  Show less 

##  Explore topics 

  * [ Sales ](https://www.linkedin.com/pulse/topics/sales-s5/)
  * [ Marketing ](https://www.linkedin.com/pulse/topics/marketing-s2461/)
  * [ IT Services ](https://www.linkedin.com/pulse/topics/it-services-s57547/)
  * [ Business Administration ](https://www.linkedin.com/pulse/topics/business-administration-s50111/)
  * [ HR Management ](https://www.linkedin.com/pulse/topics/hr-management-s50359/)
  * [ Engineering ](https://www.linkedin.com/pulse/topics/engineering-s166/)
  * [ Soft Skills ](https://www.linkedin.com/pulse/topics/soft-skills-s2976/)
  * [ See All ](https://www.linkedin.com/pulse/topics/home/)



  * LinkedIn (C) 2025
  * [ About ](https://about.linkedin.com?trk=d_flagship2_pulse_read_footer-about)
  * [ Accessibility ](https://www.linkedin.com/accessibility?trk=d_flagship2_pulse_read_footer-accessibility)
  * [ User Agreement ](https://www.linkedin.com/legal/user-agreement?trk=d_flagship2_pulse_read_footer-user-agreement)
  * [ Privacy Policy ](https://www.linkedin.com/legal/privacy-policy?trk=d_flagship2_pulse_read_footer-privacy-policy)
  * [ Your California Privacy Choices ](https://www.linkedin.com/legal/california-privacy-disclosure?trk=d_flagship2_pulse_read_footer-california-privacy-rights-act)
  * [ Cookie Policy ](https://www.linkedin.com/legal/cookie-policy?trk=d_flagship2_pulse_read_footer-cookie-policy)
  * [ Copyright Policy ](https://www.linkedin.com/legal/copyright-policy?trk=d_flagship2_pulse_read_footer-copyright-policy)
  * [ Brand Policy ](https://brand.linkedin.com/policies?trk=d_flagship2_pulse_read_footer-brand-policy)
  * [ Guest Controls ](https://www.linkedin.com/psettings/guest-controls?trk=d_flagship2_pulse_read_footer-guest-controls)
  * [ Community Guidelines ](https://www.linkedin.com/legal/professional-community-policies?trk=d_flagship2_pulse_read_footer-community-guide)
  *     * العربية (Arabic) 
    * বাংলা (Bangla) 
    * Čeština (Czech) 
    * Dansk (Danish) 
    * Deutsch (German) 
    * Ελληνικά (Greek) 
    * **English (English)**
    * Español (Spanish) 
    * فارسی (Persian) 
    * Suomi (Finnish) 
    * Français (French) 
    * हिंदी (Hindi) 
    * Magyar (Hungarian) 
    * Bahasa Indonesia (Indonesian) 
    * Italiano (Italian) 
    * עברית (Hebrew) 
    * 日本語 (Japanese) 
    * 한국어 (Korean) 
    * मराठी (Marathi) 
    * Bahasa Malaysia (Malay) 
    * Nederlands (Dutch) 
    * Norsk (Norwegian) 
    * ਪੰਜਾਬੀ (Punjabi) 
    * Polski (Polish) 
    * Português (Portuguese) 
    * Română (Romanian) 
    * Русский (Russian) 
    * Svenska (Swedish) 
    * తెలుగు (Telugu) 
    * ภาษาไทย (Thai) 
    * Tagalog (Tagalog) 
    * Türkçe (Turkish) 
    * Українська (Ukrainian) 
    * Tiếng Việt (Vietnamese) 
    * 简体中文 (Chinese (Simplified)) 
    * 正體中文 (Chinese (Traditional)) 
Language 


