Add new Acorn-esque filtered HNSW search heuristic

[benwtrent]

This is a continuation and completion of the work started by @benchaplin in #14085

The algorithm is fairly simple:

• Only score and then explore vectors that actually match the filtering criteria
• Since this will make the graph even sparser, the search spread is increased to also include the candidate's neighbor neighbors (e.g. generally maxConn * maxConn exploration)
• Additionally, even more scored candidates for a given NSW are considered to combat the increased sparsity

Some of the changes to the baseline Acorn algorithm are:

• There is some general threshold of filtering that bypasses this algorithm altogether. Early benchmarking seems to indicate that this might be around 50%, but honestly, its not fully convincing...
• The number of additional neighbors explored is predicated on the percentage of the immediate neighborhood that is filtered out
• Only look at the extended neighbors if less than 90% of the current neighborhood matches the filter.

Here are some numbers for 1M vectors, float32 and then int4 quantized.

https://docs.google.com/spreadsheets/d/1GqD7Jw42IIqimr2nB78fzEfOohrcBlJzOlpt0NuUVDQ/edit?gid=163290867#gid=163290867

Something I am unsure about:

• How to expose this setting to the users? While I am not a fan of more configuration at query time, the behavior seems different enough to justify it.

TODO:

• More manual testing over more datasets
• Add some unit and functional tests.

closes: #13940

[benchaplin]

FYI missed a javadoc param for filterSize

[benchaplin]

Thoughts on abstracting this since it's identical (minus source of graph) to HnswGraphSearcher::findBestEntryPoint? I suppose we might want to investigate multiple entry points in the future so maybe the duplicate code will be gone soon.

[benwtrent]

I don't mind having two pieces of code vs. the wrong abstraction. We can refactor if we wish in a separate PR later. But if only two things are copy-pasting code, its probably ok.

[benwtrent]

@msokolov I wonder your opinion here?

Do you think the behavior change/result change is worth waiting for a major? I do think folks should be able to use this now, but be able to opt out.

Another option I could think of is injecting a parameter or something directly into SPI loading for the hnsw vector readers. But I am not 100% sure how to do that. It does seem like it should be something that is a "global" configuration for a given Lucene instance instead of one that is provided at query time.

[benchaplin]

I was expecting else { toExplore.add(friendOfAFriendOrd) } here for >2-hop exploration. Did you drop that idea after performance testing?

[benwtrent]

@benchaplin yeah, going further than 2 hops didn't seem to improve much. We can adjust it later, but it didn't improve anything.

[benchaplin]

What's the purpose of explorationVisited?

[benwtrent]

We need to separate out the explored for 2-hop candidates vs immediate neighborhoods. If we did an expanded search for a candidate, we don't want to search it more during expanded search.

However, if that candidate matches, we DON'T want to skip it (e.g. during regular search).

Maybe we can collapse the two visited sets together, but two of them seemed OK to me.

[benchaplin]

Ah, yes that makes sense. So for immediate neighborhoods wouldn't we just want to do:

if (visited.setAndGet(friendOrd)) {
  continue;
}

?