ira/.note/decision_log.md

# Decision Log

## 2025-02-27: Initial Project Setup

### Decision: Use Jina AI APIs for Semantic Search
- **Context**: Need for semantic search capabilities that understand context beyond keywords
- **Options Considered**:
  1. Build custom embedding solution
  2. Use open-source models locally
  3. Use Jina AI's APIs
- **Decision**: Use Jina AI's APIs for embedding generation and similarity computation
- **Rationale**:
  - High-quality embeddings with state-of-the-art models
  - No need to manage model deployment and infrastructure
  - Simple API integration with reasonable pricing
  - Support for long texts through segmentation

### Decision: Separate Markdown Segmentation from Similarity Computation
- **Context**: Need to handle potentially long markdown documents
- **Options Considered**:
  1. Integrate segmentation directly into the similarity module
  2. Create a separate module for segmentation
- **Decision**: Create a separate module (markdown_segmenter.py) for document segmentation
- **Rationale**:
  - Better separation of concerns
  - More modular design allows for independent use of components
  - Easier to maintain and extend each component separately

### Decision: Use Environment Variables for API Keys
- **Context**: Need to securely manage API credentials
- **Options Considered**:
  1. Configuration files
  2. Environment variables
  3. Secret management service
- **Decision**: Use environment variables (JINA_API_KEY)
- **Rationale**:
  - Simple to implement
  - Standard practice for managing secrets
  - Works well across different environments
  - Prevents accidental commit of credentials to version control

### Decision: Use Cosine Similarity with Normalized Vectors
- **Context**: Need a metric for comparing semantic similarity between text embeddings
- **Options Considered**:
  1. Euclidean distance
  2. Cosine similarity
  3. Dot product
- **Decision**: Use cosine similarity with normalized vectors
- **Rationale**:
  - Standard approach for semantic similarity
  - Normalized vectors simplify computation (dot product equals cosine similarity)
  - Less sensitive to embedding magnitude, focusing on direction (meaning)

## 2025-02-27: Research System Architecture

### Decision: Implement a Multi-Stage Research Pipeline
- **Context**: Need to define the overall architecture for the intelligent research system
- **Options Considered**:
  1. Monolithic application with tightly coupled components
  2. Microservices architecture with independent services
  3. Pipeline architecture with distinct processing stages
- **Decision**: Implement an 8-stage pipeline architecture
- **Rationale**:
  - Clear separation of concerns with each stage having a specific responsibility
  - Easier to develop and test individual components
  - Flexibility to swap or enhance specific stages without affecting others
  - Natural flow of data through the system matches the research process

### Decision: Use Multiple Search Sources
- **Context**: Need to gather comprehensive information from various sources
- **Options Considered**:
  1. Use a single search API for simplicity
  2. Implement custom web scraping for all sources
  3. Use multiple specialized search APIs
- **Decision**: Integrate multiple search sources (Google, Serper, Jina Search, Google Scholar, arXiv)
- **Rationale**:
  - Different sources provide different types of information (academic, general, etc.)
  - Increases the breadth and diversity of search results
  - Specialized APIs like arXiv provide domain-specific information
  - Redundancy ensures more comprehensive coverage

### Decision: Use Jina AI for Semantic Processing
- **Context**: Need for advanced semantic understanding in document processing
- **Options Considered**:
  1. Use simple keyword matching
  2. Implement custom embedding models
  3. Use Jina AI's suite of APIs
- **Decision**: Use Jina AI's APIs for embedding generation, similarity computation, and reranking
- **Rationale**:
  - High-quality embeddings with state-of-the-art models
  - Comprehensive API suite covering multiple needs (embeddings, segmentation, reranking)
  - Simple integration with reasonable pricing
  - Consistent approach across different semantic processing tasks

## 2025-02-27: Search Execution Architecture

### Decision: Search Execution Architecture
- **Context**: We needed to implement a search execution module that could execute search queries across multiple search engines and process the results in a standardized way.

- **Decision**:
  1. Create a modular search execution architecture:
    - Implement a base handler interface (`BaseSearchHandler`) for all search API handlers
    - Create specific handlers for each search engine (Google, Serper, Scholar, arXiv)
    - Develop a central `SearchExecutor` class to manage execution across multiple engines
    - Implement a `ResultCollector` class for processing and organizing results

  2. Use parallel execution for search queries:
    - Implement thread-based parallelism using `concurrent.futures`
    - Add support for both synchronous and asynchronous execution
    - Include timeout management and error handling

  3. Standardize search results:
    - Define a common result format across all search engines
    - Include metadata specific to each search engine in a standardized way
    - Implement deduplication and scoring for result ranking

- **Rationale**:
  - A modular architecture allows for easy addition of new search engines
  - Parallel execution significantly improves search performance
  - Standardized result format simplifies downstream processing
  - Separation of concerns between execution and result processing

- **Alternatives Considered**:
  1. Sequential execution of search queries:
    - Simpler implementation
    - Much slower performance
    - Would not scale well with additional search engines

  2. Separate modules for each search engine:
    - Would lead to code duplication
    - More difficult to maintain
    - Less consistent result format

  3. Using a third-party search aggregation service:
    - Would introduce additional dependencies
    - Less control over the search process
    - Potential cost implications

- **Impact**:
  - Efficient execution of search queries across multiple engines
  - Consistent result format for downstream processing
  - Flexible architecture that can be extended with new search engines
  - Clear separation of concerns between different components

## 2025-02-27: Search Execution Module Refinements

### Decision: Remove Google Search Handler
- **Context**: Both Google and Serper handlers were implemented, but Serper is essentially a front-end for Google search
- **Options Considered**:
  1. Keep both handlers for redundancy
  2. Remove the Google handler and only use Serper
- **Decision**: Remove the Google search handler
- **Rationale**:
  - Redundant functionality as Serper provides the same results
  - Simplifies the codebase and reduces maintenance
  - Reduces API costs by avoiding duplicate searches
  - Serper provides a more reliable and consistent API for Google search

### Decision: Modify LLM Query Enhancement Prompt
- **Context**: The LLM was returning enhanced queries with explanations, which caused issues with search APIs
- **Options Considered**:
  1. Post-process the LLM output to extract just the query
  2. Modify the prompt to request only the enhanced query
- **Decision**: Modify the LLM prompt to request only the enhanced query without explanations
- **Rationale**:
  - More reliable than post-processing, which could be error-prone
  - Cleaner implementation that addresses the root cause
  - Ensures consistent output format for downstream processing
  - Reduces the risk of exceeding API character limits

### Decision: Implement Query Truncation
- **Context**: Enhanced queries could exceed the Serper API's 2048 character limit
- **Options Considered**:
  1. Limit the LLM's output length
  2. Truncate queries before sending to the API
  3. Split long queries into multiple searches
- **Decision**: Implement query truncation in the search executor
- **Rationale**:
  - Simple and effective solution
  - Preserves as much of the enhanced query as possible
  - Ensures API requests don't fail due to length constraints
  - Can be easily adjusted if API limits change

## 2025-02-27: Testing Strategy for Query Processor

### Context
After integrating Groq and OpenRouter as additional LLM providers, we needed to verify that the query processor module functions correctly with these new providers.

### Decision
1. Create dedicated test scripts to validate the query processor functionality:
   - A basic test script for the core processing pipeline
   - A comprehensive test script for detailed component testing
   
2. Use monkey patching to ensure tests consistently use the Groq model:
   - Create a global LLM interface with the Groq model
   - Override the `get_llm_interface` function to always return this interface
   - This approach allows testing without modifying the core code

3. Test all key functionality of the query processor:
   - Query enhancement
   - Query classification
   - Search query generation
   - End-to-end processing pipeline

### Rationale
- Dedicated test scripts provide a repeatable way to verify functionality
- Monkey patching allows testing with specific models without changing the core code
- Comprehensive testing ensures all components work correctly with the new providers
- Saving test results to a JSON file provides a reference for future development

### Alternatives Considered
1. Modifying the query processor to accept a model parameter:
   - Would require changing the core code
   - Could introduce bugs in the production code
   
2. Using environment variables to control model selection:
   - Less precise control over which model is used
   - Could interfere with other tests or production use

### Impact
- Verified that the query processor works correctly with Groq models
- Established a testing approach that can be used for other modules
- Created reusable test scripts for future development