django.db.utils.InterfaceError: connection already closed when running pytest

[k0286]

environment:

Postgresql 14.7.0
Python 3.10
Django==4.2.8
djangorestframework==3.14.0
pytest-django==4.7.0
pytest==7.4.3
pytest-celery==0.0.0
pytest-django==4.7.0
pytest-elasticsearch==4.0.2
pytest-factoryboy==2.6.0
pytest-mock==3.12.0
pytest-xdist==3.5.0

the error log

self = <django.contrib.sessions.backends.db.SessionStore object at 0x7f28520365f0>
no_load = False

    def _get_session(self, no_load=False):
        """
        Lazily load session from storage (unless "no_load" is True, when only
        an empty dict is stored) and store it in the current instance.
        """
        self.accessed = True
        try:
>           return self._session_cache
E           AttributeError: 'SessionStore' object has no attribute '_session_cache'

/usr/local/lib/python3.10/site-packages/django/contrib/sessions/backends/base.py:187: AttributeError

During handling of the above exception, another exception occurred:

self = <DatabaseWrapper vendor='postgresql' alias='default'>, name = None

    def _cursor(self, name=None):
        self.close_if_health_check_failed()
        self.ensure_connection()
        with self.wrap_database_errors:
>           return self._prepare_cursor(self.create_cursor(name))

/usr/local/lib/python3.10/site-packages/django/db/backends/base/base.py:308: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/local/lib/python3.10/site-packages/django/utils/asyncio.py:26: in inner
    return func(*args, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <DatabaseWrapper vendor='postgresql' alias='default'>, name = None

    @async_unsafe
    def create_cursor(self, name=None):
        if name:
            # In autocommit mode, the cursor will be used outside of a
            # transaction, hence use a holdable cursor.
            cursor = self.connection.cursor(
                name, scrollable=False, withhold=self.connection.autocommit
            )
        else:
>           cursor = self.connection.cursor()
E           psycopg2.InterfaceError: connection already closed

/usr/local/lib/python3.10/site-packages/django/db/backends/postgresql/base.py:330: InterfaceError

The above exception was the direct cause of the following exception:

client = <django.test.client.Client object at 0x7f28b0a61c30>
django_user_model = <class 'django.contrib.auth.models.User'>

    @pytest.mark.django_db
    def test_auth_view(client, django_user_model):
        user = django_user_model.objects.create_user("test", "test1234", is_staff=True, is_superuser=True)
        client.force_login(user)
        url = reverse('admin:auth_user_add')
>       resp = client.post(url, data={"username": "test2", "password1": "1qaz@WSX3edc", "password2": "1qaz@WSX3edc"}, follow=True)

notification/tests/test_admin.py:63: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/local/lib/python3.10/site-packages/django/test/client.py:957: in post
    response = self._handle_redirects(
/usr/local/lib/python3.10/site-packages/django/test/client.py:1173: in _handle_redirects
    response = request_method(
/usr/local/lib/python3.10/site-packages/django/test/client.py:927: in get
    response = super().get(path, data=data, secure=secure, headers=headers, **extra)
/usr/local/lib/python3.10/site-packages/django/test/client.py:457: in get
    return self.generic(
/usr/local/lib/python3.10/site-packages/django/test/client.py:609: in generic
    return self.request(**r)
/usr/local/lib/python3.10/site-packages/django/test/client.py:891: in request
    self.check_exception(response)
/usr/local/lib/python3.10/site-packages/django/test/client.py:738: in check_exception
    raise exc_value
/usr/local/lib/python3.10/site-packages/django/core/handlers/exception.py:55: in inner
    response = get_response(request)
/usr/local/lib/python3.10/site-packages/django/utils/deprecation.py:136: in __call__
    response = self.process_response(request, response)
activity/middleware.py:44: in process_response
    user_id = request.user.id
/usr/local/lib/python3.10/site-packages/django/utils/functional.py:266: in inner
    self._setup()
/usr/local/lib/python3.10/site-packages/django/utils/functional.py:419: in _setup
    self._wrapped = self._setupfunc()
/usr/local/lib/python3.10/site-packages/django/contrib/auth/middleware.py:25: in <lambda>
    request.user = SimpleLazyObject(lambda: get_user(request))
/usr/local/lib/python3.10/site-packages/django/contrib/auth/middleware.py:11: in get_user
    request._cached_user = auth.get_user(request)
/usr/local/lib/python3.10/site-packages/django/contrib/auth/__init__.py:191: in get_user
    user_id = _get_user_session_key(request)
/usr/local/lib/python3.10/site-packages/django/contrib/auth/__init__.py:60: in _get_user_session_key
    return get_user_model()._meta.pk.to_python(request.session[SESSION_KEY])
/usr/local/lib/python3.10/site-packages/django/contrib/sessions/backends/base.py:53: in __getitem__
    return self._session[key]
/usr/local/lib/python3.10/site-packages/django/contrib/sessions/backends/base.py:192: in _get_session
    self._session_cache = self.load()
/usr/local/lib/python3.10/site-packages/django/contrib/sessions/backends/db.py:42: in load
    s = self._get_session_from_db()
/usr/local/lib/python3.10/site-packages/django/contrib/sessions/backends/db.py:32: in _get_session_from_db
    return self.model.objects.get(
/usr/local/lib/python3.10/site-packages/django/db/models/manager.py:87: in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
/usr/local/lib/python3.10/site-packages/django/db/models/query.py:633: in get
    num = len(clone)
/usr/local/lib/python3.10/site-packages/django/db/models/query.py:380: in __len__
    self._fetch_all()
/usr/local/lib/python3.10/site-packages/django/db/models/query.py:1881: in _fetch_all
    self._result_cache = list(self._iterable_class(self))
/usr/local/lib/python3.10/site-packages/django/db/models/query.py:91: in __iter__
    results = compiler.execute_sql(
/usr/local/lib/python3.10/site-packages/django/db/models/sql/compiler.py:1560: in execute_sql
    cursor = self.connection.cursor()
/usr/local/lib/python3.10/site-packages/django/utils/asyncio.py:26: in inner
    return func(*args, **kwargs)
/usr/local/lib/python3.10/site-packages/django/db/backends/base/base.py:330: in cursor
    return self._cursor()
/usr/local/lib/python3.10/site-packages/django/db/backends/base/base.py:307: in _cursor
    with self.wrap_database_errors:
/usr/local/lib/python3.10/site-packages/django/db/utils.py:91: in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
/usr/local/lib/python3.10/site-packages/django/db/backends/base/base.py:308: in _cursor
    return self._prepare_cursor(self.create_cursor(name))
/usr/local/lib/python3.10/site-packages/django/utils/asyncio.py:26: in inner
    return func(*args, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <DatabaseWrapper vendor='postgresql' alias='default'>, name = None

    @async_unsafe
    def create_cursor(self, name=None):
        if name:
            # In autocommit mode, the cursor will be used outside of a
            # transaction, hence use a holdable cursor.
            cursor = self.connection.cursor(
                name, scrollable=False, withhold=self.connection.autocommit
            )
        else:
>           cursor = self.connection.cursor()
E           django.db.utils.InterfaceError: connection already closed

/usr/local/lib/python3.10/site-packages/django/db/backends/postgresql/base.py:330: InterfaceError

the test case

@pytest.mark.django_db
def test_auth_view(client, django_user_model):
    user = django_user_model.objects.create_user(
        "test", "test1234", is_staff=True, is_superuser=True
    )
    client.force_login(user)
    url = reverse("admin:auth_user_add")
    resp = client.post(
        url,
        data={"username": "test2", "password1": "1qaz@WSX3edc", "password2": "1qaz@WSX3edc"},
        follow=True,
    )
    assert resp.status_code == 200
    assert User.objects.filter(username="test2").exists()

---

It seems like the connection is close by some one, probably client, before the function is over.
the test case can pass if I use the @pytest.mark.django_db(transaction=True) marker. But I don't know the reason.

[hvieira]

I was having a similar issue, but with (I assume) a significant difference of not using @async_unsafe.

In my case, I was using a fixture to create a requests client - https://www.django-rest-framework.org/api-guide/testing/#requestsclient. After some experiments, I figured out that using this client instead of the usual django client resulted in connection already closed errors - perhaps the testing django client is more closely tied to the test flow?

environment info:
python 3.10.13
pytest-django - 4.9.0
pytest - 8.3.3
django - 5.1.1
psycopg2 - 2.9.9

[delta1513]

Just wanted to say that adding @pytest.mark.django_db(transaction=True) made these tests pass for my own project.

I experienced a similar issue when using graphene-python's test utils. This issue would only occur in CI and I could not replicate it locally (the test would pass).

I did a bit of digging and see that within the graphQL view implementation, the call to graphql-core to execute the query leads to some async functionality. I suspect this would have something to do with it, but it's well beyond my knowledge at the moment 😕