paper-system/llm_processor

LLM Processor Module

A Go module for processing papers through language models with configurable criteria.

Installation

go get llm_processor

Command Line Usage

The module provides a CLI interface for processing papers:

go run main.go -input papers.json -output results.json -criteria criteria.txt

API Usage

Import the module in your Go code:

import "llm_processor"

Example: Processing Papers

package main

import (
	"fmt"
	"llm_processor/models"
	"llm_processor/processor"
	"llm_processor/storage"
)

func main() {
	// Load papers
	papers, err := storage.LoadPapers("papers.json")
	if err != nil {
		panic(err)
	}

	// Initialize processor
	proc := processor.NewProcessor("gpt-4", 32)

	// Process papers
	results := proc.ProcessPapers(papers, "criteria.txt")

	// Save results
	err = storage.SaveResults("results.json", results)
	if err != nil {
		panic(err)
	}

	fmt.Println("Processing complete!")
}

Configuration

The processor supports the following configuration:

• Model selection (gpt-3.5-turbo, gpt-4, etc.)
• Batch size
• Custom criteria files
• Input/output file paths

Dependencies

• Go 1.21+
• OpenAI API key (set as environment variable OPENAI_API_KEY)

Contributing

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.

License

MIT