TriTih
/
DIP
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
master
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from '19a02cdcf7'
${ noResults }
DIP/docs/embedding/NV2_IMPLEMENTATION_NOTES.md

1.1 KiB
Raw Blame History

NV2 - Embedding persistence and job orchestration

This patch continues the new parallel at.procon.dip.embedding.* subsystem introduced in NV1.

Scope

NV2 adds:

  • representation-driven selection policy
  • DOC.doc_embedding_job queue table
  • job lifecycle service with retry scheduling
  • model-catalog sync into DOC.doc_embedding_model
  • persistence of vectors into DOC.doc_embedding
  • orchestrator for enqueueing and processing jobs
  • unit tests for the new orchestration layer

Still intentionally missing

  • no cutover of the old vectorization route
  • no scheduler / background polling by default
  • no semantic search engine yet
  • no migration / backfill yet

Intended usage

New code can now do:

  1. enqueue a document or representation for embedding with a configured model key
  2. process the pending jobs through the new provider-based subsystem
  3. store the resulting vectors in the generic DOC embedding tables

Next step after NV2

NV3 should add:

  • PgVectorSemanticSearchEngine
  • semantic repository
  • query embedding integration into the generic search engine
  • hybrid lexical + semantic fusion