Attestation generation failure: ... already has a publish attestation

[caleb531]

I publish my Python packages via GitHub Actions (GHA) with a Trusted Published setup, which has been working successfully for some time. However, I recently tried publishing since the PyPI attestation functionality was rolled out, and now I am getting the following error from production PyPI (Test PyPI seems to publish fine):

Checking dist/imessage_conversation_analyzer-2.5.0rc1-py3-none-any.whl: PASSED
Checking dist/imessage_conversation_analyzer-2.5.0rc1.tar.gz: PASSED
Notice: Generating and uploading digital attestations
Fulcio client using URL: https://fulcio.sigstore.dev/
TUF metadata: /root/.local/share/sigstore-python/tuf/https%3A%2F%2Ftuf-repo-cdn.sigstore.dev
TUF targets cache: /root/.cache/sigstore-python/tuf/https%3A%2F%2Ftuf-repo-cdn.sigstore.dev
Found and verified trusted root
Generating ephemeral keys...
Requesting ephemeral certificate...
Retrieving signed certificate...
Found <Name(O=sigstore.dev,CN=sigstore-intermediate)> as issuer, verifying if it is a ca
attempting to verify SCT with key ID [not sure if the key represents a secret, so redacting]
Successfully verified SCT...
Error: Attestation generation failure: /github/workspace/dist/imessage_conversation_analyzer-2.5.0rc1.tar.gz already has a publish attestation: /github/workspace/dist/imessage_conversation_analyzer-2.5.0rc1.tar.gz.publish.attestation

I believe my GHA job for publishing is correct:

name: publish

on:
  push:
    tags: ["*"]

jobs:
  publish:
    runs-on: ubuntu-latest
    environment: release
    permissions:
      id-token: write

    steps:
    - uses: actions/checkout@v3

    - name: Install Python 3
      uses: actions/setup-python@v3
      with:
        python-version: "3.11"

    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
        python -m pip install -r requirements.txt

    - name: Install pypa/build
      run: python -m pip install build --user

    - name: Build a binary wheel and a source tarball
      run: python -m build --sdist --wheel --outdir dist/ .

    - name: Publish distribution to Test PyPI
      uses: pypa/gh-action-pypi-publish@release/v1
      with:
        repository-url: https://test.pypi.org/legacy/
        verbose: true

    - name: Publish distribution to PyPI
      uses: pypa/gh-action-pypi-publish@release/v1
      with:
        repository-url: https://upload.pypi.org/legacy/
        verbose: true

(link to workflow source code)

My Trusted Publisher setup is the same as it has been—which has been working successfully before this—so I know it is correct.

I have done much searching on Google and through this repository, but cannot find a clear reason for why this is happening or what I might be doing incorrectly. As you might imagine, passing attestations: false to the pypa/gh-action-pypi-publish action publishes successfully, but I'm assuming this is a terrible workaround because it relaxes the verification requirements that attestations are meant to offer.

Is this a bug? Or what can I do to fix this? Thank you in advance for any help you can provide.

Link to package on PyPI

[webknjaz]

@caleb531 could you read https://github.com/marketplace/actions/pypi-publish#Non-goals and let me know if I can make it clearer somehow? What would've helped you discover it on your own?

[caleb531]

@webknjaz If it's outside the scope of what you can do for X technical reasons (such as troubleshooting challenges), that's all fine. I suppose my issue comes down to a lack of understanding about how attestations are supposed to work. And while the Non-Goals section you linked is clear, I still have the following questions:

1. Are attestations something automatically generated by the action, or am I generating them myself at build time?
2. If the feature is still experimental / not yet stable, why has it been enabled by default?
3. I've already read through Producing attestations fails when attempting publishing from unsupported reusable workflows (when enabled) #283 about a similar attestation error for reusable workflows. But based on the definition from GitHub, I don't believe my workflow counts as a reusable workflow. Or am I misunderstanding?

Thank you for looking at this!

[facutuesca]

hey @caleb531! The issue:

Error: Attestation generation failure: /github/workspace/dist/imessage_conversation_analyzer-2.5.0rc1.tar.gz already has a publish attestation: /github/workspace/dist/imessage_conversation_analyzer-2.5.0rc1.tar.gz.publish.attestation

is because the gh-action-pypi-publish action is running twice in the same job (once for TestPyPI, and once for PyPI). That means the first time it runs it generates the attestations and uploads everything to TestPyPI, and the second time it runs it tries to generate the attestations again, but fails because they are already there.

The section of the docs @webknjaz linked addresses this near the end, and suggests a solution:

And finally, invoking pypi-publish more than once in the same job is not considered supported. It may work in a limited number of scenarios but please, don't do this. If you want to publish to several indexes, build the dists in one job and add several publishing jobs, one per upload.

[caleb531]

@facutuesca I see. Do you have a recommendation, then, for publishing to TestPyPI as a prerequisite to production PyPI publish? Because ideally, I want to publish to production PyPI if and only if Test PyPI publish has succeeded. Hence why I run both in succession as opposed to running TestPyPI publish in a separate workflow/job.

EDIT: I think I found what I'm looking for: https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/using-jobs-in-a-workflow#defining-prerequisite-jobs — closing this issue now that the problem has been identified, along with a solution.

[webknjaz]

@caleb531 another (or complimentary) solution would be setting up required reviewers in the environment you assigned to the job (usually called pypi), in the repository settings — this would make the job with the pypi environment pause until you allow it to proceed, while the one using the testpypi environment would start right away. It's more of a soft job sequencing, and you can decide to start the PyPI one sooner in this config.

P.S. The reason I was asking to check that README was mainly my attempt to identify how difficult it is for somebody reading it for the first time to find the information. I think the docs should be improved, but I don't fully understand what's the best way to do that is so far.