Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Empty session.answer for exhaustive search without an answer #723

Open
mskarlin opened this issue Nov 26, 2024 · 1 comment
Open

Empty session.answer for exhaustive search without an answer #723

mskarlin opened this issue Nov 26, 2024 · 1 comment
Labels
bug Something isn't working

Comments

@mskarlin
Copy link
Collaborator

The new complete tool allows the agent to call it quits after trying many search iterations. This is great but if it finds no relevant evidence, the GenerateAnswer tool won't be called either, so the session has nothing populated for the answer. We should have some kind of sentinel auto-injected here that can be displayed as output rather than just an empty string for this case.

@dosubot dosubot bot added the bug Something isn't working label Nov 26, 2024
@jamesbraza
Copy link
Collaborator

Yeah to add an example:

import asyncio

from paperqa.litqa import LitQAEvaluation

qa_prompt2, eval_fn2 = LitQAEvaluation.from_question(
    ideal="circular dichroism",
    distractors=["cryo EM", "x-ray crystallography", "NMR"],
    question=(
        "What method was used to demonstrate that the enzyme PafA is stable after"
        " incubation with 4M urea for 14 days?"
    ),
)
print(asyncio.run(eval_fn2("")))  # correct

I am actually working on this right now

The issue is without an answer, the LLM begins to pull on its innate knowledge

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants