Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cell magics break Pylance in Jupyter notebooks #6889

Closed
DePasqualeOrg opened this issue Jan 24, 2025 · 27 comments
Closed

Cell magics break Pylance in Jupyter notebooks #6889

DePasqualeOrg opened this issue Jan 24, 2025 · 27 comments
Assignees
Labels
needs repro Issue has not been reproduced yet user responded Was "waiting for user response" and they responded

Comments

@DePasqualeOrg
Copy link

Environment data

  • Pylance version: 2024.12.1
  • OS and version: macOS 15.2
  • Python version: 3.12.8

Repro Steps

  1. Open any of these notebooks (or any notebook with imports) with VS Code in a workspace: https://github.com/DePasqualeOrg/mlx-intro/tree/main/notebooks

Expected behavior

Intellisense works on mouseover

Actual behavior

All imports have this error on mouseover: "some_import" is not defined. Pylance doesn't work at all.

@github-actions github-actions bot added the needs repro Issue has not been reproduced yet label Jan 24, 2025
@rchiodo
Copy link
Contributor

rchiodo commented Jan 24, 2025

Can you include the logs as described here:
https://github.com/microsoft/pylance-release/blob/main/TROUBLESHOOTING.md#filing-an-issue

They should list out why we can't find an import. It should be looking in the same import path as the kernel selected in the notebook.

@DePasqualeOrg
Copy link
Author

DePasqualeOrg commented Jan 24, 2025

I started logging, and when I run my notebook, the output panel is blank. I'm getting all these "not defined" errors on imports.

@rchiodo
Copy link
Contributor

rchiodo commented Jan 24, 2025

That would mean you don't have Pylance installed or it failed to start for some reason.

Is pylance listed in the extensions tab?

If it is, can you look in the Help | Developer Tools -> Console and see if there are any errors there?

@rchiodo
Copy link
Contributor

rchiodo commented Jan 24, 2025

Additionally, the Python output tab may show some errors for whether or not Pylance started. The Python extension starts the pylance extension.

@DePasqualeOrg
Copy link
Author

Pylance is definitely installed, because I see errors like this when mousing over imports from the virtual environment:

Image

There's nothing at all in the output tab.

Developer tools console:

Image

@rchiodo
Copy link
Contributor

rchiodo commented Jan 24, 2025

There's nothing in this output tab?

Image

If you set this:

Image

That should generate the exact paths we searched for mlx_lm in that output tab.

@DePasqualeOrg
Copy link
Author

Okay, now I have the log output. I can only share a short excerpt here:

2025-01-24 22:49:57.304 [info] [Info  - 10:49:57 PM] (4385)     file:///Users/anthony/files/projects/mlx-intro/.venv/lib/python3.12/site-packages
2025-01-24 22:49:57.304 [info] [Info  - 10:49:57 PM] (4385)   Looking in python search path 'file:///opt/homebrew/Cellar/python%403.12/3.12.8/Frameworks/Python.framework/Versions/3.12/lib/python3.12'
2025-01-24 22:49:57.304 [info] [Info  - 10:49:57 PM] (4385)   Attempting to resolve stub package using root path 'file:///opt/homebrew/Cellar/python%403.12/3.12.8/Frameworks/Python.framework/Versions/3.12/lib/python3.12'
2025-01-24 22:49:57.304 [info] [Info  - 10:49:57 PM] (4385)   Attempting to resolve using root path 'file:///opt/homebrew/Cellar/python%403.12/3.12.8/Frameworks/Python.framework/Versions/3.12/lib/python3.12'
2025-01-24 22:49:57.304 [info] [Info  - 10:49:57 PM] (4385)   Looking in python search path 'file:///opt/homebrew/Cellar/python%403.12/3.12.8/Frameworks/Python.framework/Versions/3.12/lib/python3.12/lib-dynload'
2025-01-24 22:49:57.304 [info] [Info  - 10:49:57 PM] (4385)   Attempting to resolve stub package using root path 'file:///opt/homebrew/Cellar/python%403.12/3.12.8/Frameworks/Python.framework/Versions/3.12/lib/python3.12/lib-dynload'
2025-01-24 22:49:57.304 [info] [Info  - 10:49:57 PM] (4385)   Attempting to resolve using root path 'file:///opt/homebrew/Cellar/python%403.12/3.12.8/Frameworks/Python.framework/Versions/3.12/lib/python3.12/lib-dynload'
2025-01-24 22:49:57.304 [info] [Info  - 10:49:57 PM] (4385)   Looking in python search path 'file:///Users/anthony/files/projects/mlx-intro/.venv/lib/python3.12/site-packages'
2025-01-24 22:49:57.304 [info] [Info  - 10:49:57 PM] (4385)   Attempting to resolve stub package using root path 'file:///Users/anthony/files/projects/mlx-intro/.venv/lib/python3.12/site-packages'
2025-01-24 22:49:57.304 [info] [Info  - 10:49:57 PM] (4385)   Attempting to resolve using root path 'file:///Users/anthony/files/projects/mlx-intro/.venv/lib/python3.12/site-packages'
2025-01-24 22:49:57.304 [info] [Info  - 10:49:57 PM] (4385)   Looking in bundled stubs path 'file:///Users/anthony/.vscode/extensions/ms-python.vscode-pylance-2024.12.1/dist/bundled/stubs'
2025-01-24 22:49:57.304 [info] [Info  - 10:49:57 PM] (4385)   Attempting to resolve stub package using root path 'file:///Users/anthony/.vscode/extensions/ms-python.vscode-pylance-2024.12.1/dist/bundled/stubs'
2025-01-24 22:49:57.304 [info] [Info  - 10:49:57 PM] (4385)   Attempting to resolve using root path 'file:///Users/anthony/.vscode/extensions/ms-python.vscode-pylance-2024.12.1/dist/bundled/stubs'
2025-01-24 22:49:57.304 [info] [Info  - 10:49:57 PM] (4385)   Looking for typeshed third-party path
2025-01-24 22:49:57.304 [info] [Info  - 10:49:57 PM] (4385)   Looking for typeshed stubs path
2025-01-24 22:49:57.304 [info] [Info  - 10:49:57 PM] (4385)   Typeshed path not found
2025-01-24 22:49:57.304 [info] (4385) [BG(1)]     binding: file:///Users/anthony/files/projects/mlx-intro/.venv/lib/python3.12/site-packages/transformers/utils/import_utils.py (7ms)
2025-01-24 22:49:57.305 [info] (4385) [BG(1)]     parsing: file:///Users/anthony/.vscode/extensions/ms-python.vscode-pylance-2024.12.1/dist/typeshed-fallback/stdlib/logging/__init__.pyi [fs read 1ms] (5ms)
2025-01-24 22:49:57.305 [info] (4385) [BG(1)]     binding: file:///Users/anthony/.vscode/extensions/ms-python.vscode-pylance-2024.12.1/dist/typeshed-fallback/stdlib/logging/__init__.pyi (3ms)
2025-01-24 22:49:57.305 [info] (4385) [BG(1)]     parsing: file:///Users/anthony/files/projects/mlx-intro/.venv/lib/python3.12/site-packages/huggingface_hub/utils/__init__.py [fs read 0ms] (2ms)
2025-01-24 22:49:57.305 [info] (4385) [BG(1)]     binding: file:///Users/anthony/files/projects/mlx-intro/.venv/lib/python3.12/site-packages/huggingface_hub/utils/__init__.py (0ms)
2025-01-24 22:49:57.305 [info] (4385) [BG(1)]     parsing: file:///Users/anthony/.vscode/extensions/ms-python.vscode-pylance-2024.12.1/dist/typeshed-fallback/stdlib/threading.pyi [fs read 0ms] (3ms)
2025-01-24 22:49:57.305 [info] (4385) [BG(1)]     binding: file:///Users/anthony/.vscode/extensions/ms-python.vscode-pylance-2024.12.1/dist/typeshed-fallback/stdlib/threading.pyi (1ms)
2025-01-24 22:49:57.305 [info] (4385) [BG(1)]     parsing: file:///Users/anthony/files/projects/mlx-intro/.venv/lib/python3.12/site-packages/huggingface_hub/utils/tqdm.py [fs read 0ms] (12ms)
2025-01-24 22:49:57.305 [info] (4385) [BG(1)]     binding: file:///Users/anthony/files/projects/mlx-intro/.venv/lib/python3.12/site-packages/huggingface_hub/utils/tqdm.py (1ms)
2025-01-24 22:49:57.306 [info] (4385) [BG(1)]     parsing: file:///Users/anthony/.vscode/extensions/ms-python.vscode-pylance-2024.12.1/dist/typeshed-fallback/stdlib/enum.pyi [fs read 0ms] (3ms)
2025-01-24 22:49:57.306 [info] (4385) [BG(1)]     binding: file:///Users/anthony/.vscode/extensions/ms-python.vscode-pylance-2024.12.1/dist/typeshed-fallback/stdlib/enum.pyi (1ms)
2025-01-24 22:49:57.307 [info] (4385) [BG(1)]     parsing: file:///Users/anthony/files/projects/mlx-intro/.venv/lib/python3.12/site-packages/transformers/utils/doc.py [fs read 1ms] (7ms)
2025-01-24 22:49:57.307 [info] (4385) [BG(1)]     binding: file:///Users/anthony/files/projects/mlx-intro/.venv/lib/python3.12/site-packages/transformers/utils/doc.py (1ms)
2025-01-24 22:49:57.307 [info] (4385) [BG(1)]     parsing: file:///Users/anthony/files/projects/mlx-intro/.venv/lib/python3.12/site-packages/transformers/tokenization_utils_fast.py [fs read 1ms] (7ms)
2025-01-24 22:49:57.307 [info] (4385) [BG(1)]     binding: file:///Users/anthony/files/projects/mlx-intro/.venv/lib/python3.12/site-packages/transformers/tokenization_utils_fast.py (3ms)
2025-01-24 22:49:57.307 [info] (4385) [BG(1)]     parsing: file:///Users/anthony/files/projects/mlx-intro/.venv/lib/python3.12/site-packages/mlx_vlm/prompt_utils.py [fs read 0ms] (1ms)
2025-01-24 22:49:57.308 [info] (4385) [BG(1)]     binding: file:///Users/anthony/files/projects/mlx-intro/.venv/lib/python3.12/site-packages/mlx_vlm/prompt_utils.py (1ms)
2025-01-24 22:49:57.308 [info] (4385) [BG(1)]     parsing: file:///Users/anthony/.vscode/extensions/ms-python.vscode-pylance-2024.12.1/dist/typeshed-fallback/stdlib/pathlib.pyi [fs read 0ms] (1ms)
2025-01-24 22:49:57.309 [info] (4385) [BG(1)]     binding: file:///Users/anthony/.vscode/extensions/ms-python.vscode-pylance-2024.12.1/dist/typeshed-fallback/stdlib/pathlib.pyi (1ms)
2025-01-24 22:49:57.318 [info] (4385) [BG(1)]     parsing: file:///Users/anthony/.vscode/extensions/ms-python.vscode-pylance-2024.12.1/dist/typeshed-fallback/stdlib/os/__init__.pyi [fs read 1ms] (9ms)
2025-01-24 22:49:57.322 [info] (4385) [BG(1)]     binding: file:///Users/anthony/.vscode/extensions/ms-python.vscode-pylance-2024.12.1/dist/typeshed-fallback/stdlib/os/__init__.pyi (3ms)
2025-01-24 22:49:57.346 [info] (4385) [BG(1)]   checking: file:///Users/anthony/files/projects/mlx-intro/ignore/test.py (564ms)
2025-01-24 22:49:57.346 [info] (4385) [BG(1)] analyzing: file:///Users/anthony/files/projects/mlx-intro/ignore/test.py (565ms)

@rchiodo
Copy link
Contributor

rchiodo commented Jan 24, 2025

You cut it off right in the part where it's saying why it can't find mlx_lm. That is listing the paths where it looked for it.

2025-01-24 22:49:57.304 [info] [Info  - 10:49:57 PM] (4385)     file:///Users/anthony/files/projects/mlx-intro/.venv/lib/python3.12/site-packages
2025-01-24 22:49:57.304 [info] [Info  - 10:49:57 PM] (4385)   Looking in python search path 'file:///opt/homebrew/Cellar/python%403.12/3.12.8/Frameworks/Python.framework/Versions/3.12/lib/python3.12'
2025-01-24 22:49:57.304 [info] [Info  - 10:49:57 PM] (4385)   Attempting to resolve stub package using root path 'file:///opt/homebrew/Cellar/python%403.12/3.12.8/Frameworks/Python.framework/Versions/3.12/lib/python3.12'
2025-01-24 22:49:57.304 [info] [Info  - 10:49:57 PM] (4385)   Attempting to resolve using root path 'file:///opt/homebrew/Cellar/python%403.12/3.12.8/Frameworks/Python.framework/Versions/3.12/lib/python3.12'
2025-01-24 22:49:57.304 [info] [Info  - 10:49:57 PM] (4385)   Looking in python search path 'file:///opt/homebrew/Cellar/python%403.12/3.12.8/Frameworks/Python.framework/Versions/3.12/lib/python3.12/lib-dynload'
2025-01-24 22:49:57.304 [info] [Info  - 10:49:57 PM] (4385)   Attempting to resolve stub package using root path 'file:///opt/homebrew/Cellar/python%403.12/3.12.8/Frameworks/Python.framework/Versions/3.12/lib/python3.12/lib-dynload'
2025-01-24 22:49:57.304 [info] [Info  - 10:49:57 PM] (4385)   Attempting to resolve using root path 'file:///opt/homebrew/Cellar/python%403.12/3.12.8/Frameworks/Python.framework/Versions/3.12/lib/python3.12/lib-dynload'
2025-01-24 22:49:57.304 [info] [Info  - 10:49:57 PM] (4385)   Looking in python search path 'file:///Users/anthony/files/projects/mlx-intro/.venv/lib/python3.12/site-packages'
2025-01-24 22:49:57.304 [info] [Info  - 10:49:57 PM] (4385)   Attempting to resolve stub package using root path 'file:///Users/anthony/files/projects/mlx-intro/.venv/lib/python3.12/site-packages'
2025-01-24 22:49:57.304 [info] [Info  - 10:49:57 PM] (4385)   Attempting to resolve using root path 'file:///Users/anthony/files/projects/mlx-intro/.venv/lib/python3.12/site-packages'
2025-01-24 22:49:57.304 [info] [Info  - 10:49:57 PM] (4385)   Looking in bundled stubs path 'file:///Users/anthony/.vscode/extensions/ms-python.vscode-pylance-2024.12.1/dist/bundled/stubs'
2025-01-24 22:49:57.304 [info] [Info  - 10:49:57 PM] (4385)   Attempting to resolve stub package using root path 'file:///Users/anthony/.vscode/extensions/ms-python.vscode-pylance-2024.12.1/dist/bundled/stubs'
2025-01-24 22:49:57.304 [info] [Info  - 10:49:57 PM] (4385)   Attempting to resolve using root path 'file:///Users/anthony/.vscode/extensions/ms-python.vscode-pylance-2024.12.1/dist/bundled/stubs'
2025-01-24 22:49:57.304 [info] [Info  - 10:49:57 PM] (4385)   Looking for typeshed third-party path
2025-01-24 22:49:57.304 [info] [Info  - 10:49:57 PM] (4385)   Looking for typeshed stubs path
2025-01-24 22:49:57.304 [info] [Info  - 10:49:57 PM] (4385)   Typeshed path not found

The first line though is where I'd expect it to be:

file:///Users/anthony/files/projects/mlx-intro/.venv/lib/python3.12/site-packages

Is that where you installed it?

@DePasqualeOrg
Copy link
Author

It should be there. The notebook runs and can import the library, but Pylance apparently can't find it. I'm using a virtual environment inside this directory.

@rchiodo
Copy link
Contributor

rchiodo commented Jan 24, 2025

Pylance would be looking for the directory 'mlx_lm' in the site-packages folder. Does it exist? Perhaps 'mlx_lm' is a strange install.

@DePasqualeOrg
Copy link
Author

It exists. Everything that should be installed runs fine in the notebook, but with no intellisense because Pylance can't find anything.

@rchiodo
Copy link
Contributor

rchiodo commented Jan 24, 2025

Can you include the full log? It should list out what exactly it was looking for. I'm having trouble myself installing 'mlx-lm'.

@rchiodo
Copy link
Contributor

rchiodo commented Jan 24, 2025

'mlx-lm' worked for me on WSL (not Windows):

Image

The full log should list why we didn't find it.

@rchiodo rchiodo added the waiting for user response Requires more information from user label Jan 24, 2025
@DePasqualeOrg
Copy link
Author

@github-actions github-actions bot added user responded Was "waiting for user response" and they responded and removed waiting for user response Requires more information from user labels Jan 24, 2025
@rchiodo
Copy link
Contributor

rchiodo commented Jan 24, 2025

I don't see the 'mlx-lm' import listed there but a bunch of others.

They're all looking here:

 Received 3 paths from interpreter
2025-01-24 23:56:45.581 [info] [Info  - 11:56:45 PM] (4790)     file:///opt/homebrew/Cellar/python%403.12/3.12.8/Frameworks/Python.framework/Versions/3.12/lib/python3.12
2025-01-24 23:56:45.581 [info] [Info  - 11:56:45 PM] (4790)     file:///opt/homebrew/Cellar/python%403.12/3.12.8/Frameworks/Python.framework/Versions/3.12/lib/python3.12/lib-dynload
2025-01-24 23:56:45.581 [info] [Info  - 11:56:45 PM] (4790)     file:///Users/anthony/files/projects/mlx-intro/.venv/lib/python3.12/site-packages

If you run this code in a cell, what does it say:

import tensorflow
tensorflow.__file__

@DePasqualeOrg
Copy link
Author

DePasqualeOrg commented Jan 24, 2025

Tensorflow is not imported by the code running in the notebook. I think that's an artifact of Transformers. It's not installed and not needed.

Perhaps mlx_lm is a better example: '/Users/anthony/files/projects/mlx-intro/.venv/lib/python3.12/site-packages/mlx_lm/__init__.py'

@rchiodo
Copy link
Contributor

rchiodo commented Jan 24, 2025

So this log is from something that imports 'mlx_lm' and gives the error you mentioned? I don't see that in the log. It should show an import error if it's giving the import unresolved error in the IDE.

@rchiodo
Copy link
Contributor

rchiodo commented Jan 24, 2025

I mean maybe the bug here is that the errors aren't being updated and it is finding 'mlx_lm'

@rchiodo
Copy link
Contributor

rchiodo commented Jan 24, 2025

What happens if you run 'Python: Restart Language Server'?

@DePasqualeOrg
Copy link
Author

Yes, this is the log from the notebook that's importing mlx_lm. Restarting the language server doesn't change anything.

@DePasqualeOrg
Copy link
Author

By the way, you can see several other people complaining about this same problem in the ratings of Pylance on the VS Code extension store.

@rchiodo
Copy link
Contributor

rchiodo commented Jan 24, 2025

Is this the notebook?
https://github.com/DePasqualeOrg/mlx-intro/blob/main/notebooks/text-generation.ipynb

It's this:

%%capture
import mlx_lm
from IPython.display import display, clear_output, Markdown

%%capture makes us ignore the entire cell. We don't read this cell so it doesn't know about the import. It's a function of cell magics not being valid python.

@rchiodo
Copy link
Contributor

rchiodo commented Jan 24, 2025

Which is an enhancement request:
#4969

@DePasqualeOrg
Copy link
Author

Wow, well that's really not obvious. But now I get errors like this in code that works:

Object of type "NaiveStreamingDetokenizer" is not callable
  Attribute "__call__" is unknownPylance[reportCallIssue](https://github.com/microsoft/pyright/blob/main/docs/configuration.md#reportCallIssue)
Object of type "set[Unknown]" is not callable
  Attribute "__call__" is unknownPylance[reportCallIssue](https://github.com/microsoft/pyright/blob/main/docs/configuration.md#reportCallIssue)

@DePasqualeOrg
Copy link
Author

Also this, even though Markdown works.

"Markdown" is unknown import symbolPylance[reportAttributeAccessIssue](https://github.com/microsoft/pyright/blob/main/docs/configuration.md#reportAttributeAccessIssue)

@rchiodo rchiodo changed the title Pylance doesn't work in Jupyter notebooks in VS Code Cell magics break Pylance in Jupyter notebooks Jan 24, 2025
@rchiodo
Copy link
Contributor

rchiodo commented Jan 24, 2025

The Markdown error is the same issue. If you're importing it in a cell with %%capture in it, we'll never see it.

The other problem is likely to do with the typing of the NaiveStreamingDetokenizer. It may be incorrectly typed or we can't infer types for some reason.

@DePasqualeOrg
Copy link
Author

Okay, thanks for walking me through this. Unfortunately I have always had so much trouble getting tooling set up properly for Python. The whole experience is so complicated. There are so many steps you need to go through with VS Code. I don't know how beginners manage it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
needs repro Issue has not been reproduced yet user responded Was "waiting for user response" and they responded
Projects
None yet
Development

No branches or pull requests

3 participants