chatterbox-ui/.note/detailed_migration_plan.md

Chatterbox TTS: Gradio to FastAPI & Vanilla JS Migration Plan

This plan outlines the steps to re-implement the dialog generation features of the Chatterbox TTS application, moving from the current Gradio-based implementation to a FastAPI backend and a vanilla JavaScript frontend. It incorporates findings from gradio_app.py and aligns with the existing high-level strategy (MEMORY[c20c2cce-46d4-453f-9bc3-c18e05dbc66f]).

1. Backend (FastAPI) Development

Objective: Create a robust API to handle TTS generation, speaker management, and file delivery.

Key Modules/Components:

• API Endpoints:
  • POST /api/dialog/generate:
    • Input: Structured list: [{type: "speech", speaker_id: "str", text: "str"}, {type: "silence", duration: float}], output_base_name: str.
    • Output: JSON with log: str, concatenated_audio_url: str, zip_archive_url: str.
  • GET /api/speakers: Returns list of available speakers ([{id: "str", name: "str", sample_path: "str"}]).
  • POST /api/speakers: Adds a new speaker. Input: name: str, audio_sample_file: UploadFile. Output: {id: "str", name: "str", message: "str"}.
  • DELETE /api/speakers/{speaker_id}: Removes a speaker.
• Core Logic & Services:
  • TTSService:
    • Manages ChatterboxTTS model instance(s) (loading, inference, memory cleanup).
    • Handles ChatterboxTTS.generate() calls, incorporating parameters like exaggeration, cfg_weight, temperature (decision needed on exposure vs. defaults).
    • Implements rigorous memory management (inspired by generate_audio and process_dialog's reinit_each_line concept).
  • DialogProcessorService:
    • Orchestrates dialog generation using TTSService.
    • Implements split_text_at_sentence_boundaries logic for long text inputs.
    • Manages generation of individual audio segments.
  • AudioManipulationService:
    • Concatenates audio segments using torch and torchaudio, inserting specified silences.
    • Creates ZIP archives of all generated audio files using zipfile.
  • SpeakerManagementService:
    • Manages speakers.yaml (or alternative storage) for speaker metadata.
    • Handles storage and retrieval of speaker audio samples (e.g., in speaker_samples/).
• File Handling:
  • Strategy for storing and serving generated .wav and .zip files (e.g., FastAPI StaticFiles, temporary directories, or cloud storage).

Implementation Steps (Phase 1):

1. Project Setup: Initialize FastAPI project, define dependencies (fastapi, uvicorn, python-multipart, pyyaml, torch, torchaudio, chatterbox-tts).
2. Speaker Management: Implement SpeakerManagementService and the /api/speakers endpoints.
3. TTS Core: Develop TTSService, focusing on model loading, inference, and critical memory management.
4. Dialog Processing: Implement DialogProcessorService including text splitting.
5. Audio Utilities: Create AudioManipulationService for concatenation and zipping.
6. Main Endpoint: Implement POST /api/dialog/generate orchestrating the services.
7. Configuration: Manage paths (speakers.yaml, sample storage, output directories) and TTS settings.
8. Testing: Thoroughly test all API endpoints using tools like Postman or curl.

2. Frontend (Vanilla JavaScript) Development

Objective: Create an intuitive UI for dialog construction, speaker management, and interaction with the backend.

Key Modules/Components:

• HTML (index.html): Structure for dialog editor, speaker controls, results display.
• CSS (style.css): Styling for a clean and usable interface.
• **JavaScript (app.js, api.js, ui.js):
  • api.js: Functions for all backend API communications (fetch).
  • ui.js: DOM manipulation for dynamic dialog lines, speaker lists, and results rendering.
  • app.js: Main application logic, event handling, state management (for dialog lines, speaker data).

Implementation Steps (Phase 2):

1. Basic Layout: Create index.html and style.css.
2. API Client: Develop api.js to interface with all backend endpoints.
3. Speaker UI:

• Fetch and display speakers using ui.js and api.js.
• Implement forms and logic for adding (with file upload) and removing speakers.

4. Dialog Editor UI:

• Dynamically add/remove/reorder dialog lines (speech/silence).
• Inputs for speaker selection (populated from API), text, and silence duration.
• Input for output_base_name.

5. Interaction & Results:

• "Generate Dialog" button to submit data via api.js.
• Display generation log, audio player for concatenated output, and download link for ZIP file.

3. Integration & Testing (Phase 3)

1. Full System Connection: Ensure seamless frontend-backend communication.
2. End-to-End Testing: Test various dialog scenarios, speaker configurations, and error conditions.
3. Performance & Memory: Profile backend memory usage during generation; refine TTSService memory strategies if needed.
4. UX Refinement: Iterate on UI/UX based on testing feedback.

4. Advanced Features & Deployment (Phase 4)

• (As per MEMORY[c20c2cce-46d4-453f-9bc3-c18e05dbc66f])
• Real-time Updates: Consider WebSockets for live progress during generation.
• Deployment Strategy: Plan for deploying the FastAPI application and serving the static frontend assets.

Key Considerations from gradio_app.py Analysis:

• Memory Management for TTS Model: This is critical. The reinit_each_line option and explicit cleanup in generate_audio highlight this. The FastAPI backend must handle this robustly.
• Text Chunking: The split_text_at_sentence_boundaries (max 300 chars) logic is essential and must be replicated.
• Dialog Parsing: The Speaker: "Text" and Silence: duration format should be the basis for the frontend data structure sent to the backend.
• TTS Parameters: Decide whether to expose advanced TTS parameters (exaggeration, cfg_weight, temperature) for dialog lines in the new API.
• File Output: The backend needs to replicate the generation of individual segment files, a concatenated file, and a ZIP archive.