Published Apr 2024

spice add spiceai/xlangai-brightspice connect spiceai/xlangai-brightdependencies:
- spiceai/xlangai-brightBRIGHT biology corpus. One row per passage of a web page.
BRIGHT biology questions, with the ids of their relevant (gold_ids) and excluded (excluded_ids) passages.
LLM-as-a-judge rerankers (chat models). rerank() sends them all candidates in one listwise prompt by default.
version: v1
kind: Spicepod
name: xlangai_bright
# User-defined functions are disabled by default; the search pipelines below are `functions:`.
runtime:
functions:
enabled: true
# BRIGHT (https://huggingface.co/datasets/xlangai/BRIGHT), biology domain: StackExchange
# questions whose relevant documents need reasoning to find, not keyword overlap.
datasets:
- from: hf://datasets/xlangai/BRIGHT/documents/biology-00000-of-00001.parquet
name: docs
description: BRIGHT biology corpus. One row per passage of a web page.
params:
# Authenticated requests get higher Hugging Face rate limits.
hf_token: ${secrets:HF_TOKEN}
acceleration:
enabled: true
columns:
- name: content
full_text_search:
enabled: true
row_id:
- id
- from: hf://datasets/xlangai/BRIGHT/examples/biology-00000-of-00001.parquet
name: queries
description: BRIGHT biology questions, with the ids of their relevant (gold_ids) and excluded (excluded_ids) passages.
params:
# Authenticated requests get higher Hugging Face rate limits.
hf_token: ${secrets:HF_TOKEN}
acceleration:
enabled: true
# One table function per setup. Each takes a question `q`, reranks the BM25 top 50, and
# returns (id, score); order by score descending for the ranking:
#
# SELECT id FROM bright_rerank_jev('Why are insects attracted to light?') ORDER BY score DESC LIMIT 10;
#
# Inside text_search, `q` is rewritten to a literal before planning. Anywhere else, read the
# argument from the one-row `args` table (see jev_ask). rerank() takes its query from the
# search, so it has no `query =>` argument. `id` is LargeUtf8 in the dataset; arrow_cast
# makes it the declared utf8.
functions:
- name: bm25
from: sql
kind: table
description: BM25 full-text search only, top 50.
signature:
args: [{ name: q, type: utf8 }]
returns:
- { name: id, type: utf8 }
- { name: score, type: float64 }
body: |
SELECT arrow_cast(id, 'Utf8') AS id, CAST(_score AS DOUBLE) AS score
FROM text_search(docs, q, content, limit => 50)
- name: rerank_jev
from: sql
kind: table
description: BM25 top 50 reranked by jev with rerank(), pointwise.
signature:
args: [{ name: q, type: utf8 }]
returns:
- { name: id, type: utf8 }
- { name: score, type: float64 }
body: |
SELECT arrow_cast(id, 'Utf8') AS id, CAST(rerank_score AS DOUBLE) AS score
FROM rerank(text_search(docs, q, content, limit => 50), model => 'jev', document => 'content')
# rerank() asks a fixed relevance question. This asks jev BRIGHT's own definition of
# relevance instead, with a description of the collection. Also pointwise.
- name: jev_ask
from: sql
kind: table
description: BM25 top 50 scored by jev with ai_probability() and BRIGHT's definition of relevance.
signature:
args: [{ name: q, type: utf8 }]
returns:
- { name: id, type: utf8 }
- { name: score, type: float64 }
body: |
SELECT arrow_cast(t.id, 'Utf8') AS id, ai_probability(
named_struct(
'collection', 'Passages from web pages (mostly Wikipedia and science articles) cited in answers on Biology StackExchange. Each question is a user''s post.',
'question', args.q,
'document', t.content),
'Does `document` contain knowledge that helps answer `question`, such as a concept, mechanism, or fact that a good answer would rely on, even if it does not use the question''s words?',
model => 'jev') AS score
FROM text_search(docs, q, content, limit => 50) AS t
CROSS JOIN args
- name: rerank_luna
from: sql
kind: table
description: BM25 top 50 reranked by gpt-6-luna as an LLM judge, listwise.
signature:
args: [{ name: q, type: utf8 }]
returns:
- { name: id, type: utf8 }
- { name: score, type: float64 }
body: |
SELECT arrow_cast(id, 'Utf8') AS id, CAST(rerank_score AS DOUBLE) AS score
FROM rerank(text_search(docs, q, content, limit => 50), model => 'luna', document => 'content')
- name: rerank_opus
from: sql
kind: table
description: BM25 top 50 reranked by claude-opus-5-5 as an LLM judge, listwise.
signature:
args: [{ name: q, type: utf8 }]
returns:
- { name: id, type: utf8 }
- { name: score, type: float64 }
body: |
SELECT arrow_cast(id, 'Utf8') AS id, CAST(rerank_score AS DOUBLE) AS score
FROM rerank(text_search(docs, q, content, limit => 50), model => 'opus', document => 'content')
models:
# Decision model: scores each candidate with a calibrated P(relevant).
- from: typesafe:jev
name: jev
params:
typesafe_api_key: ${secrets:JEV_API_KEY}
- from: openai:gpt-6-luna
name: luna
params:
openai_api_key: ${secrets:OPENAI_API_KEY}
- from: anthropic:claude-opus-5-5
name: opus
description: >-
LLM-as-a-judge rerankers (chat models).
rerank() sends them all candidates in one listwise prompt by default.
params:
anthropic_api_key: ${secrets:ANTHROPIC_API_KEY}
BRIGHT (opens in a new tab) is a text retrieval benchmark designed for reasoning beyond mere keyword overlap. Text search models alone will not score highly.
In this spice app, full-text-search (FTS) is combined with a variety of rerank models to improve scores. Search pipelines are defined in spicepod.yaml using custom user-defined SQL functions. For example
functions:
- name: luna_rerank
from: sql
kind: table
description: BM25 top 100 reranked by gpt-6-luna as an LLM judge, listwise.
signature:
args: [{ name: q, type: utf8 }]
returns:
- { name: id, type: utf8 }
- { name: score, type: float64 }
body: |
SELECT id, rerank_score as _score
FROM rerank(
text_search(docs, q, content, limit => 100),
model => 'luna',
document => 'content'
)
Can be invoked in SQL
select id, _score from luna_rerank("Do animals exhibit handedness (paw-ness?) preference?") limit 5
Published Apr 2024
Published Sep 2024
Published Sep 2024
Published Sep 2024