Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

improve task creation and cancellation #1085

Open
wants to merge 17 commits into
base: main
Choose a base branch
from

Conversation

aconchillo
Copy link
Contributor

Please describe the changes in your PR. If it is addressing an issue, please reference that as well.

If a FrameProcessor needs to create a task it should use FrameProcessor.create_task() and FrameProcessor.cancel_task(). This gives Pipecat more control over all the tasks that are created in Pipecat.

Both functions internally use the utils module: utils.create_task() and utils.cancel_task() which should also be used outside of FrameProcessors. That is, unless strictly necessary, we should avoid using asyncio.create_task().

@markbackman markbackman changed the title imporve task creation and cancellation improve task creation and cancellation Jan 24, 2025
@aconchillo aconchillo force-pushed the aleix/task-creation-and-cancellation branch 2 times, most recently from 3c0ef99 to 9017c22 Compare January 24, 2025 23:40
If a FrameProcessor needs to create a task it should use
FrameProcessor.create_task() and FrameProcessor.cancel_task(). This gives
Pipecat more control over all the tasks that are created in Pipecat.

Both functions internally use the utils module: utils.create_task() and
utils.cancel_task() which should also be used outside of FrameProcessors. That
is, unless strictly necessary, we should avoid using asyncio.create_task().
@aconchillo aconchillo force-pushed the aleix/task-creation-and-cancellation branch from 9017c22 to 2e15253 Compare January 25, 2025 02:13
@aconchillo aconchillo force-pushed the aleix/task-creation-and-cancellation branch from 32d40ce to 54cade1 Compare January 25, 2025 08:38
@markbackman
Copy link
Contributor

This is really great! I've found two issues so far:

  1. This is a regression which causes overlapping speech. To repro, run example 14-function-calling.py, where you set the start_fetch_weather function to:
async def start_fetch_weather(function_name, llm, context):
    """Push a frame to the LLM; this is handy when the LLM response might take a while."""
    await llm.push_frame(
        TextFrame(
            "Let me check on that. One moment while I pull up the weather for you. Ok, got it."
        )
    )
    logger.debug(f"Starting fetch_weather_from_api with function_name: {function_name}")

This will result in the TTS speech overlapping—both the TextFrame and LLM text will be spoken at the same time. This isn’t happening on main.

  1. I think this revealed an issue with the GeminiMultimodalLiveLLMService. I ran the simple-chatbot demo using the bot-gemini.py bot file and a user connecting via a javascript client. Upon the user leaving, I got this warning:
2025-01-25 08:59:48.521 | WARNING  | pipecat.pipeline.runner:_print_dangling_tasks:63 - Dangling tasks detected: ['GeminiMultimodalLiveLLMService#0::_handle_transcribe_user_audio', 'GeminiMultimodalLiveLLMService#0::_handle_transcribe_user_audio', 'GeminiMultimodalLiveLLMService#0::_handle_transcribe_model_audio', 'GeminiMultimodalLiveLLMService#0::_handle_transcribe_user_audio', 'GeminiMultimodalLiveLLMService#0::_handle_transcribe_model_audio', 'GeminiMultimodalLiveLLMService#0::_handle_transcribe_user_audio', 'GeminiMultimodalLiveLLMService#0::_handle_transcribe_model_audio', 'GeminiMultimodalLiveLLMService#0::_handle_transcribe_model_audio', 'GeminiMultimodalLiveLLMService#0::_handle_transcribe_user_audio', 'GeminiMultimodalLiveLLMService#0::_handle_transcribe_model_audio', 'GeminiMultimodalLiveLLMService#0::_handle_transcribe_model_audio', 'GeminiMultimodalLiveLLMService#0::_handle_transcribe_user_audio']

@markbackman
Copy link
Contributor

Also, in running the twilio-chatbot example, I get some version of this frequently:

2025-01-25 09:32:57.674 | WARNING  | pipecat.pipeline.runner:_print_dangling_tasks:63 - Dangling tasks detected: ['FastAPIWebsocketInputTransport#0::_receive_messages']

Should we be concerned about dangling tasks?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants