You cannot select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
40 lines
1.1 KiB
Markdown
40 lines
1.1 KiB
Markdown
# NV2 - Embedding persistence and job orchestration
|
|
|
|
This patch continues the new parallel `at.procon.dip.embedding.*` subsystem introduced in NV1.
|
|
|
|
## Scope
|
|
|
|
NV2 adds:
|
|
|
|
- representation-driven selection policy
|
|
- `DOC.doc_embedding_job` queue table
|
|
- job lifecycle service with retry scheduling
|
|
- model-catalog sync into `DOC.doc_embedding_model`
|
|
- persistence of vectors into `DOC.doc_embedding`
|
|
- orchestrator for enqueueing and processing jobs
|
|
- unit tests for the new orchestration layer
|
|
|
|
## Still intentionally missing
|
|
|
|
- no cutover of the old vectorization route
|
|
- no scheduler / background polling by default
|
|
- no semantic search engine yet
|
|
- no migration / backfill yet
|
|
|
|
## Intended usage
|
|
|
|
New code can now do:
|
|
|
|
1. enqueue a document or representation for embedding with a configured model key
|
|
2. process the pending jobs through the new provider-based subsystem
|
|
3. store the resulting vectors in the generic DOC embedding tables
|
|
|
|
## Next step after NV2
|
|
|
|
NV3 should add:
|
|
|
|
- `PgVectorSemanticSearchEngine`
|
|
- semantic repository
|
|
- query embedding integration into the generic search engine
|
|
- hybrid lexical + semantic fusion
|