Only "Whisper-Based Translation" Works

[technohippies]

Firstly, thanks for such an awesome plugin.

I cannot get translations to work, unless I use Whisper-Based Translation, in which case the original subtitles in English don't show either (as you described in your YT video).

• I'm using the newest version 0.3.7
• Whisper Tiny works great for English transcription
• I've tried both the M2M-100 418 model, and the NLLB 200 600M model, with many different languages
• It's not a font related problem: 1. Spanish doesn't work, and 2. If I manually paste something into the Text field, it shows
• I'm running a Linux Ubuntu distro with an Nvidia GPU

My main solution could be to run 2 instances of the Audio Filter, with another running Whisper-Based Translation, but I'm guessing this would further cook my GPU.

Not sure what I'm doing wrong. Any help appreciated!

[technohippies]

Sorry for duplicate, Internet dropped

[royshil]

@technohippies can you share the OBS logs?

[technohippies]

@technohippies can you share the OBS logs?

it looks like this is the error:
[obs-localvocal] Error: No SGEMM backend on CPU

[technohippies]

[ctranslate2] [thread 275234] [info] - Use Intel MKL: false
[ctranslate2] [thread 275234] [info] - SGEMM backend: none (packed: false)

added some logs, trying to figure this out...

[technohippies]

I ended up using ollama to translate the output textfile as I couldn't get this working. This technique might be better anyways given the accuracy of these tiny new LLMs

[Snugglyninja]

I'm having this same error, can you explain more about how you utilized ollama to accomplish the same thing?

[technohippies]

I'm having this same error, can you explain more about how you utilized ollama to accomplish the same thing?

1. Write to text file as he describes on YT
2. I created a Python script to listen to that file for new lines
3. Python script sends new line to Ollama API
4. Ollama writes to text file
5. Display via another LocalVocal instance

[technohippies]

import time
import requests
import json
import logging
from watchdog.observers import Observer
from watchdog.events import FileSystemEventHandler

# Set up logging
logging.basicConfig(level=logging.DEBUG, format='%(asctime)s - %(levelname)s - %(message)s')

# File paths
input_file = 'stream_output.txt'
output_file = 'translated_output.txt'
obs_file = 'latest_translation.txt'

# Ollama API endpoint
ollama_url = 'http://localhost:11434/api/generate'

class FileChangeHandler(FileSystemEventHandler):
    def __init__(self):
        self.last_position = 0

    def on_modified(self, event):
        if event.src_path == input_file:
            logging.info(f"Detected change in {input_file}")
            self.translate_new_content()

    def translate_new_content(self):
        try:
            with open(input_file, 'r') as file:
                file.seek(self.last_position)
                new_content = file.read().strip()
                self.last_position = file.tell()

            if new_content:
                logging.info(f"New content detected: {new_content}")
                translated_text = self.translate_to_chinese(new_content)
                self.append_to_output(new_content, translated_text)
                self.update_obs_file(new_content, translated_text)
            else:
                logging.info("No new content detected")
        except Exception as e:
            logging.error(f"Error in translate_new_content: {str(e)}")

    def translate_to_chinese(self, text):
        logging.info("Sending translation request to Ollama")
        prompt = f"""Translate the following English text to Mandarin Chinese, respond with nothing but the translation (no commentary, notes, or explanations):

{text}

Chinese translation:"""
        
        data = {
            "model": "qwen2.5:3b",
            "prompt": prompt,
            "stream": False
        }

        try:
            response = requests.post(ollama_url, json=data)
            if response.status_code == 200:
                result = json.loads(response.text)
                translated = result['response'].strip()
                logging.info(f"Translation received: {translated}")
                return translated
            else:
                logging.error(f"Ollama API error: {response.status_code}")
                return "Translation error"
        except Exception as e:
            logging.error(f"Error in translate_to_chinese: {str(e)}")
            return "Translation error"

    def append_to_output(self, original, translated):
        try:
            with open(output_file, 'a') as file:
                file.write(f"Original: {original}\n")
                file.write(f"Translated: {translated}\n\n")
            logging.info(f"Appended translation to {output_file}")
        except Exception as e:
            logging.error(f"Error appending to output file: {str(e)}")

    def update_obs_file(self, original, translated):
        try:
            with open(obs_file, 'a', encoding='utf-8') as file:
                file.write(f"{translated}\n")
            logging.info(f"Appended translation to OBS file: {translated}")
        except Exception as e:
            logging.error(f"Error updating OBS file: {str(e)}")

if __name__ == "__main__":
    event_handler = FileChangeHandler()
    observer = Observer()
    observer.schedule(event_handler, path=input_file, recursive=False)
    observer.start()

    logging.info(f"Monitoring {input_file} for changes.")
    logging.info(f"Translations will be appended to {output_file}")
    logging.info(f"Latest translation for OBS will be updated in {obs_file}")
    try:
        while True:
            time.sleep(1)
    except KeyboardInterrupt:
        observer.stop()
    observer.join()