# Parallel embedding subsystem plan (NV1–NV3)

This plan assumes the old vectorization subsystem remains in place temporarily, while a new generic embedding subsystem is built in parallel.

## Principles

- Build the new subsystem under `at.procon.dip.embedding.*`.
- Do not shape it around old TED-specific or legacy vectorization services.
- Operate on `DocumentTextRepresentation` and `DocumentEmbedding` as the core abstraction.
- Keep the new subsystem configurable and provider-based.
- Migrate and cut over later.

## NV1 — provider/model/query foundation

### Goal
Create a standalone embedding foundation that can:
- resolve configured providers
- resolve configured models
- embed arbitrary text lists
- embed search queries
- support deterministic testing

### Deliverables
- `EmbeddingProperties`
- `EmbeddingUseCase`
- `EmbeddingRequest`
- `EmbeddingProviderResult`
- `EmbeddingModelDescriptor`
- `ResolvedEmbeddingProviderConfig`
- `EmbeddingProvider`
- `ExternalHttpEmbeddingProvider`
- `MockEmbeddingProvider`
- `EmbeddingProviderRegistry`
- `EmbeddingModelRegistry`
- `EmbeddingProviderConfigResolver`
- `EmbeddingExecutionService`
- `QueryEmbeddingService`
- startup validation of provider/model wiring

### Notes
- No cutover to the old vectorization path yet.
- No persistence/job orchestration yet.
- New subsystem should be safe to include in the app while disabled by default.

## NV2 — persistence and job orchestration

### Goal
Make the new subsystem able to create and process embedding jobs against `DocumentTextRepresentation`.

### Deliverables
- `EmbeddingJob` entity/repository/service
- retry / backoff policy
- default `EmbeddingSelectionPolicy`
- representation-level embedding execution
- `DocumentEmbedding` persistence updates through the new subsystem

## NV3 — generic semantic search engine

### Goal
Add semantic search into the generic search platform using only the new subsystem.

### Deliverables
- `PgVectorSemanticSearchEngine`
- `DocumentSemanticSearchRepository`
- query embedding through `QueryEmbeddingService`
- chunk-aware retrieval and collapse
- fusion with lexical search

## Migration philosophy

Because the app is still in development, prefer:

1. migrate documents and text representations first
2. re-embed through the new subsystem
3. only preserve old raw vector data if there is a strong operational reason

## Recommended implementation order

1. NV1 foundation
2. NV1 tests with mock provider
3. NV2 jobs and selection policy
4. NV3 semantic search
5. migration/backfill
6. cutover