83 lines
1.4 KiB
Markdown
83 lines
1.4 KiB
Markdown
|
# LLM Processor Module
|
||
|
|
|
||
|
|
A Go module for processing papers through language models with configurable criteria.
|
||
|
|
|
||
|
|
## Installation
|
||
|
|
|
||
|
|
```bash
|
||
|
|
go get llm_processor
|
||
|
|
```
|
||
|
|
|
||
|
|
## Command Line Usage
|
||
|
|
|
||
|
|
The module provides a CLI interface for processing papers:
|
||
|
|
|
||
|
|
```bash
|
||
|
|
go run main.go -input papers.json -output results.json -criteria criteria.txt
|
||
|
|
```
|
||
|
|
|
||
|
|
## API Usage
|
||
|
|
|
||
|
|
Import the module in your Go code:
|
||
|
|
|
||
|
|
```go
|
||
|
|
import "llm_processor"
|
||
|
|
```
|
||
|
|
|
||
|
|
### Example: Processing Papers
|
||
|
|
|
||
|
|
```go
|
||
|
|
package main
|
||
|
|
|
||
|
|
import (
|
||
|
|
"fmt"
|
||
|
|
"llm_processor/models"
|
||
|
|
"llm_processor/processor"
|
||
|
|
"llm_processor/storage"
|
||
|
|
)
|
||
|
|
|
||
|
|
func main() {
|
||
|
|
// Load papers
|
||
|
|
papers, err := storage.LoadPapers("papers.json")
|
||
|
|
if err != nil {
|
||
|
|
panic(err)
|
||
|
|
}
|
||
|
|
|
||
|
|
// Initialize processor
|
||
|
|
proc := processor.NewProcessor("gpt-4", 32)
|
||
|
|
|
||
|
|
// Process papers
|
||
|
|
results := proc.ProcessPapers(papers, "criteria.txt")
|
||
|
|
|
||
|
|
// Save results
|
||
|
|
err = storage.SaveResults("results.json", results)
|
||
|
|
if err != nil {
|
||
|
|
panic(err)
|
||
|
|
}
|
||
|
|
|
||
|
|
fmt.Println("Processing complete!")
|
||
|
|
}
|
||
|
|
```
|
||
|
|
|
||
|
|
## Configuration
|
||
|
|
|
||
|
|
The processor supports the following configuration:
|
||
|
|
|
||
|
|
- Model selection (gpt-3.5-turbo, gpt-4, etc.)
|
||
|
|
- Batch size
|
||
|
|
- Custom criteria files
|
||
|
|
- Input/output file paths
|
||
|
|
|
||
|
|
## Dependencies
|
||
|
|
|
||
|
|
- Go 1.21+
|
||
|
|
- OpenAI API key (set as environment variable OPENAI_API_KEY)
|
||
|
|
|
||
|
|
## Contributing
|
||
|
|
|
||
|
|
Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.
|
||
|
|
|
||
|
|
## License
|
||
|
|
|
||
|
|
[MIT](https://choosealicense.com/licenses/mit/)
|