Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[TensorRT] Support Multiple EP Context #23294

Open
wants to merge 6 commits into
base: main
Choose a base branch
from
Open

Conversation

jingyanwangms
Copy link
Contributor

Description

  • Use CreateEpContextModel from graph_partitioner.cc to save model with context ep. Now multi ep context in a model is supported
  • Updated merging ep context related options from session option and tensorrt option
  • Updated and adding unit test

Supported scenarios:

  • Save/run static single ep context node using engine cache
  • Save/run static single ep context node with embedded ep context info
  • Save/run static multiple ep context node using engine cache
  • Save/run static multiple ep context node with embedded ep context info
  • Save/run dynamic multiple ep context node using engine cache
  • Save/run dynamic multiple ep context node with embedded ep context info

Unsupported scenarios:

  • Subsequent runs with dynamic ep context node where dynamic input dimension changed
    Supporting this will require a call up from execution provider to CreateEpContextModel in graph_partitioner.cc during run time. It will require significant changes in the existing infrastructure.

Motivation and Context

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant