Compare commits

...

7 Commits

Author SHA1 Message Date
trifonovt 14a6f8d42e Enrich operating periods with raw activity details 2026-05-06 10:25:42 +02:00
trifonovt 94767bc161 Add Esper operating-period evaluation PoC 2026-05-06 10:24:53 +02:00
trifonovt 818009555a Add configurable Esper activity pipeline and driver identity model 2026-05-06 09:02:38 +02:00
trifonovt 094007d817 Fix YellowFox fallback executor wiring 2026-05-05 15:33:18 +02:00
trifonovt e4e9137af5 Add Esper tachograph activity PoC 2026-05-05 15:29:10 +02:00
trifonovt 17e2bbedf4 Use import time points in tachograph extraction 2026-05-05 12:56:30 +02:00
trifonovt 900bfa5918 Add YellowFox D8 ingestion pipeline 2026-05-05 12:54:49 +02:00
90 changed files with 7038 additions and 342 deletions

View File

@ -542,23 +542,29 @@ order by p.received_at desc;
## Check acquired events
`eventhub.acquired_event` was replaced by the normalized `eventhub.event` and `eventhub.event_detail` tables.
```sql
select occurred_at,
driver_source_entity_id,
driver_card_nation,
driver_card_number,
vehicle_source_entity_id,
vehicle_vin,
vehicle_registration_nation,
vehicle_registration_number,
event_domain,
event_type,
lifecycle,
event_signature_hash,
event_details,
payload
from eventhub.acquired_event
order by occurred_at desc;
select e.occurred_at,
e.driver_source_entity_id,
e.driver_card_nation,
e.driver_card_number,
e.vehicle_source_entity_id,
e.vehicle_vin,
e.vehicle_registration_nation,
e.vehicle_registration_number,
e.event_domain,
e.event_type,
e.lifecycle,
e.event_signature_hash,
d.detail_type,
d.attributes as event_details,
e.payload
from eventhub.event e
left join eventhub.event_detail d
on d.event_occurred_at = e.occurred_at
and d.event_id = e.id
order by e.occurred_at desc;
```
## Next implementation steps
@ -639,7 +645,7 @@ For tachograph JDBC extraction, `sourcePackageId` is the original `FileLog.ID`.
N planned data_package rows, one per extraction definition and time chunk
```
The SQL extraction routes are intentionally separated from run planning. They should pick planned extraction packages, execute the corresponding SQL, map rows to `EventHubEventDto`, set `sourcePackageRef` when known, and send them to `direct:eventhub-normalized-input`.
When `executeImmediately=true` or a configured plan is started with `triggerMode=EXECUTE`, the concrete extractor executes the planned packages, maps rows to `EventHubEventDto`, sets `sourcePackageRef` when known, persists synchronous JDBC batches through `EventHubIngestionService`, and advances the import cursor only after successful persistence.
### Initial import
@ -705,15 +711,18 @@ This is preferred because newly imported original driver-card/VU packages can co
### Extraction route contract
A future concrete SQL extraction route should do this:
The concrete JDBC extraction route now does this:
```text
planned data_package
-> execute SQL for extraction_code and chunk/import scope
-> apply source-package watermark parameters for incremental imports
-> map source rows to EventHubEventDto
-> populate sourcePackageRef if source package metadata is available
-> send to direct:eventhub-normalized-input
-> only advance eventhub.import_cursor after successful import
-> populate sourcePackageRef when source package metadata is available
-> hand off mapped events according to eventhub.tachograph.jdbc-extraction-ingest-mode
- SYNC_DIRECT: persist controlled DB_EXTRACT packages through EventHubIngestionService
- CAMEL_ROUTE: persist the same controlled batches through direct:eventhub-batch-persist-input
-> advance eventhub.import_cursor after the configured extraction handoff completed
```
## Configurable scheduled tachograph imports
@ -739,6 +748,7 @@ eventhub:
scheduler-enabled: false
scheduler-poll-interval-ms: 60000
scheduler-trigger-mode: PLAN_ONLY
jdbc-extraction-ingest-mode: SYNC_DIRECT
import-plans:
- plan-key: kralowetz-tachograph-org-147
enabled: false
@ -790,13 +800,13 @@ POST /api/eventhub/acquisition/tachograph/imports/configured-plans/kralowetz-tac
## Concrete extraction extension point
The scheduler and import-run service are now implemented, but the generated skeleton still does not know the real tachograph DB SQL. The extension point is:
The scheduler, import-run service and first JDBC SQL extractor are implemented. The extension point remains:
```java
TachographExtractionBatchExecutor
```
Replace `NoopTachographExtractionBatchExecutor` with an implementation that:
`NoopTachographExtractionBatchExecutor` is used only when `eventhub.tachograph.datasource.jdbc-url` is empty. A custom executor can still replace it and should:
```text
1. receives importRunId, packageId, TachographImportRequest, planItem and time chunk
@ -805,15 +815,27 @@ Replace `NoopTachographExtractionBatchExecutor` with an implementation that:
4. applies source-package watermark or source-row watermark for incremental updates
5. maps rows to EventHubEventDto
6. sets sourcePackageRef when the row can be traced to an original card/VU package
7. sends events to direct:eventhub-normalized-input or EventHubIngestionService
8. returns TachographExtractionBatchResultDto with cursor watermarks
7. persist events through `EventHubIngestionService` or a route with explicit completion tracking
8. return `TachographExtractionBatchResultDto` with cursor watermarks
```
The import cursor is advanced only when the executor reports `executed=true`. The default no-op executor returns `executed=false`, so it does not move cursors accidentally.
## JDBC tachograph extraction
The first concrete extractor is `JdbcTachographExtractionBatchExecutor`. It is enabled only when `eventhub.tachograph.datasource.jdbc-url` is configured. Without that datasource, the application keeps using `NoopTachographExtractionBatchExecutor`.
The first concrete extractor is `JdbcTachographExtractionBatchExecutor`. It is enabled only when `eventhub.tachograph.datasource.jdbc-url` is configured. The default `application.yml` maps this to `${TACHOGRAPH_DB_JDBC_URL:}`, so an empty environment does not create the tachograph datasource and the application keeps using `NoopTachographExtractionBatchExecutor`.
The JDBC extractor now has a configurable handoff mode via `eventhub.tachograph.jdbc-extraction-ingest-mode`:
```yaml
eventhub:
tachograph:
jdbc-extraction-ingest-mode: SYNC_DIRECT # or CAMEL_ROUTE
```
`SYNC_DIRECT` is the default and persists controlled `DB_EXTRACT` packages directly through `EventHubIngestionService`. This is the recommended mode for cursor-sensitive scheduled imports, because each extraction batch is written before the package is marked imported and before the cursor is advanced.
`CAMEL_ROUTE` keeps a Camel-based alternative without returning to the unsafe timeout-driven aggregation handoff: the extractor still builds controlled DB extraction batches, but sends each `EventHubEventBatchDto` through `direct:eventhub-batch-persist-input`. This route invokes the same `EventHubIngestionService` synchronously, so cursor advancement remains deterministic while the persistence handoff still goes through Camel.
Currently implemented extraction definitions:

View File

@ -0,0 +1,94 @@
# Esper PoC: Tachograph Driver-Card Activity Evaluation
This PoC intentionally uses only existing imported EventHub source events:
- provider: `TACHOGRAPH`
- source kind: `DRIVER_CARD`
- extraction code: `CARD_ACTIVITY`
- event domain: `DRIVER_ACTIVITY`
- event types: `DRIVE`, `WORK`, `AVAILABILITY`, `BREAK_REST`
- lifecycles: `START`, `END`
It does not introduce canonical-event tables yet. The goal is to prove that Esper can replay one driver and one selected period and produce activity-level and operating-period results.
## Endpoint
```http
GET /api/eventhub/esper-poc/tachograph/driver-card-activities
```
Query parameters:
| Parameter | Required | Example | Meaning |
|---|---:|---|---|
| `tenantKey` | yes | `default` | EventHub tenant key. |
| `driverEntityId` | yes | UUID | Existing `eventhub.event.driver_entity_id`. |
| `occurredFrom` | yes | `2026-04-01T00:00:00Z` | Requested period start. |
| `occurredTo` | yes | `2026-05-01T00:00:00Z` | Requested period end. |
| `guardHours` | no | `24` | Extra loading window before/after requested period. Needed for activities crossing midnight/month boundaries and long rests crossing period boundaries. |
| `significantDrivingMinutes` | no | `3` | DRIVE intervals longer than this threshold count as significant driving periods. |
| `mergeGapSeconds` | no | `60` | Consecutive identical activities are merged if the gap is at most this value. |
| `operatingPeriodSplitRestHours` | no | `7` | A `BREAK_REST` activity longer than this threshold splits operating time periods. |
## Produced levels
### Level 1: Raw
`raw` contains the original point events from `eventhub.event`.
### Level 2: Activities
Esper consumes the raw point events and produces intervals by pairing:
```text
START + END with same sourceRowId, activity type, driver and card slot
```
The service merges consecutive identical activities in the full guard window first, then clips the merged activities to the requested period. This is important because a long `BREAK_REST` crossing the requested boundary must keep its full guard-window duration for operating-period splitting.
## Operating time periods
`operatingTimePeriods` are derived from the merged activity timeline.
A `BREAK_REST` interval splits operating periods when:
```text
activityType = BREAK_REST
and duration > operatingPeriodSplitRestHours
```
The default is 7 hours. With the default, a `BREAK_REST` of exactly 7 hours does not split; it must be longer than 7 hours.
Each operating period contains:
- `sequenceNumber`
- `startedAt`
- `endedAt`
- `durationSeconds`
- `activities`
- `workingOperationTimes`
- `drivingTimeInterruptionEvaluation`
- optional references to the long rest before/after the period
Departure and arrival are evaluated per operating period:
- departure = first significant `DRIVE` interval inside that operating period
- arrival = end of the last significant `DRIVE` interval inside that operating period
- middle/interruption = gaps between significant `DRIVE` intervals inside the same operating period
## Result semantics
- `workResultPerDriver` and `workingOperationTimesPerEmployee` currently use the same PoC summary for the whole requested period.
- `workingSeconds = DRIVE + WORK`.
- `operationSeconds = DRIVE + WORK + AVAILABILITY`.
- `breakRestSeconds` is reported separately.
- Top-level `drivingTimeInterruptionEvaluation` evaluates the whole requested period.
- Each item in `operatingTimePeriods` has its own `drivingTimeInterruptionEvaluation`.
## Current limitations
- Uses the existing source-level `driver_entity_id`, not a canonical employee table.
- Reads only tachograph driver-card activity events.
- Does not merge VU/card duplication.
- Does not persist results; the endpoint returns a PoC calculation response.
- Esper is used for interval creation. Summary, clipping, operating-period split, and merged activity report calculation are implemented in Java for auditability and easier future migration to canonical events.

17
pom.xml
View File

@ -20,6 +20,7 @@
<properties>
<java.version>21</java.version>
<camel.version>4.18.2</camel.version>
<esper.version>9.0.0</esper.version>
</properties>
<dependencyManagement>
@ -88,6 +89,22 @@
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>com.espertech</groupId>
<artifactId>esper-common</artifactId>
<version>${esper.version}</version>
</dependency>
<dependency>
<groupId>com.espertech</groupId>
<artifactId>esper-compiler</artifactId>
<version>${esper.version}</version>
</dependency>
<dependency>
<groupId>com.espertech</groupId>
<artifactId>esper-runtime</artifactId>
<version>${esper.version}</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>

View File

@ -0,0 +1,150 @@
{
"info": {
"name": "EventHub Esper PoC",
"schema": "https://schema.getpostman.com/json/collection/v2.1.0/collection.json"
},
"item": [
{
"name": "Evaluate tachograph driver-card activities",
"request": {
"method": "GET",
"header": [],
"url": {
"raw": "{{baseUrl}}/api/eventhub/esper-poc/tachograph/driver-card-activities?tenantKey={{tenantKey}}&driverEntityId={{driverEntityId}}&occurredFrom={{occurredFrom}}&occurredTo={{occurredTo}}&guardHours=24&significantDrivingMinutes=3&mergeGapSeconds=60&operatingPeriodSplitRestHours=7",
"host": [
"{{baseUrl}}"
],
"path": [
"api",
"eventhub",
"esper-poc",
"tachograph",
"driver-card-activities"
],
"query": [
{
"key": "tenantKey",
"value": "{{tenantKey}}"
},
{
"key": "driverEntityId",
"value": "{{driverEntityId}}"
},
{
"key": "occurredFrom",
"value": "{{occurredFrom}}"
},
{
"key": "occurredTo",
"value": "{{occurredTo}}"
},
{
"key": "guardHours",
"value": "24"
},
{
"key": "significantDrivingMinutes",
"value": "3"
},
{
"key": "mergeGapSeconds",
"value": "60"
},
{
"key": "operatingPeriodSplitRestHours",
"value": "7"
}
]
}
}
},
{
"name": "Evaluate tachograph operating periods",
"request": {
"method": "GET",
"header": [],
"url": {
"raw": "{{baseUrl}}/api/eventhub/esper-poc/tachograph/operating-period-evaluation?tenantKey={{tenantKey}}&driverId={{driverId}}&occurredFrom={{occurredFrom}}&occurredTo={{occurredTo}}&guardHours=24&operatingSplitIdleHours=7&significantDrivingMinutes=3&mergeGapSeconds=0&gapDetectionToleranceSeconds=0&unknownTreatmentMode=AS_BREAK_REST",
"host": [
"{{baseUrl}}"
],
"path": [
"api",
"eventhub",
"esper-poc",
"tachograph",
"operating-period-evaluation"
],
"query": [
{
"key": "tenantKey",
"value": "{{tenantKey}}"
},
{
"key": "driverId",
"value": "{{driverId}}"
},
{
"key": "occurredFrom",
"value": "{{occurredFrom}}"
},
{
"key": "occurredTo",
"value": "{{occurredTo}}"
},
{
"key": "guardHours",
"value": "24"
},
{
"key": "operatingSplitIdleHours",
"value": "7"
},
{
"key": "significantDrivingMinutes",
"value": "3"
},
{
"key": "mergeGapSeconds",
"value": "0"
},
{
"key": "gapDetectionToleranceSeconds",
"value": "0"
},
{
"key": "unknownTreatmentMode",
"value": "AS_BREAK_REST"
}
]
}
}
}
],
"variable": [
{
"key": "baseUrl",
"value": "http://localhost:8080"
},
{
"key": "tenantKey",
"value": "default"
},
{
"key": "driverEntityId",
"value": "00000000-0000-0000-0000-000000000000"
},
{
"key": "driverId",
"value": "00000000-0000-0000-0000-000000000000"
},
{
"key": "occurredFrom",
"value": "2026-04-01T00:00:00Z"
},
{
"key": "occurredTo",
"value": "2026-05-01T00:00:00Z"
}
]
}

View File

@ -69,6 +69,10 @@ public class EventHubCommonIngestionRoute extends RouteBuilder {
.forceCompletionOnStop()
.process(batchBuildProcessor)
.bean(ingestionService, "ingest");
from("direct:eventhub-batch-persist-input")
.routeId("eventhub-direct-batch-persist-route")
.bean(ingestionService, "ingest");
}
private String batchInputUri() {

View File

@ -1,5 +1,8 @@
package at.procon.eventhub.config;
import at.procon.eventhub.esperpoc.dto.EsperActivityMergeMode;
import at.procon.eventhub.esperpoc.dto.EsperUnknownTreatmentMode;
import at.procon.eventhub.esperpoc.dto.EsperShiftResolutionMode;
import java.time.Duration;
import java.time.OffsetDateTime;
import java.util.ArrayList;
@ -26,6 +29,7 @@ public class EventHubProperties {
private final Batch batch = new Batch();
private final Tachograph tachograph = new Tachograph();
private final EsperPoc esperPoc = new EsperPoc();
private final YellowFox yellowFox = new YellowFox();
public Batch getBatch() {
@ -36,10 +40,90 @@ public class EventHubProperties {
return tachograph;
}
public EsperPoc getEsperPoc() {
return esperPoc;
}
public YellowFox getYellowFox() {
return yellowFox;
}
public static class EsperPoc {
private EsperActivityMergeMode activityMergeMode = EsperActivityMergeMode.JAVA;
private EsperShiftResolutionMode shiftResolutionMode = EsperShiftResolutionMode.JAVA;
private final OperatingPeriodEvaluation operatingPeriodEvaluation = new OperatingPeriodEvaluation();
public EsperActivityMergeMode getActivityMergeMode() {
return activityMergeMode;
}
public void setActivityMergeMode(EsperActivityMergeMode activityMergeMode) {
this.activityMergeMode = activityMergeMode == null ? EsperActivityMergeMode.JAVA : activityMergeMode;
}
public EsperShiftResolutionMode getShiftResolutionMode() {
return shiftResolutionMode;
}
public void setShiftResolutionMode(EsperShiftResolutionMode shiftResolutionMode) {
this.shiftResolutionMode = shiftResolutionMode == null ? EsperShiftResolutionMode.JAVA : shiftResolutionMode;
}
public OperatingPeriodEvaluation getOperatingPeriodEvaluation() {
return operatingPeriodEvaluation;
}
}
public static class OperatingPeriodEvaluation {
private int operatingSplitIdleHours = 7;
private int significantDrivingMinutes = 3;
private int mergeGapSeconds = 0;
private int gapDetectionToleranceSeconds = 0;
private EsperUnknownTreatmentMode unknownTreatmentMode = EsperUnknownTreatmentMode.AS_BREAK_REST;
public int getOperatingSplitIdleHours() {
return operatingSplitIdleHours;
}
public void setOperatingSplitIdleHours(int operatingSplitIdleHours) {
this.operatingSplitIdleHours = Math.max(1, operatingSplitIdleHours);
}
public int getSignificantDrivingMinutes() {
return significantDrivingMinutes;
}
public void setSignificantDrivingMinutes(int significantDrivingMinutes) {
this.significantDrivingMinutes = Math.max(1, significantDrivingMinutes);
}
public int getMergeGapSeconds() {
return mergeGapSeconds;
}
public void setMergeGapSeconds(int mergeGapSeconds) {
this.mergeGapSeconds = Math.max(0, mergeGapSeconds);
}
public int getGapDetectionToleranceSeconds() {
return gapDetectionToleranceSeconds;
}
public void setGapDetectionToleranceSeconds(int gapDetectionToleranceSeconds) {
this.gapDetectionToleranceSeconds = Math.max(0, gapDetectionToleranceSeconds);
}
public EsperUnknownTreatmentMode getUnknownTreatmentMode() {
return unknownTreatmentMode;
}
public void setUnknownTreatmentMode(EsperUnknownTreatmentMode unknownTreatmentMode) {
this.unknownTreatmentMode = unknownTreatmentMode == null
? EsperUnknownTreatmentMode.AS_BREAK_REST
: unknownTreatmentMode;
}
}
public static class Batch {
/** Number of events collected before a package is persisted. */
private int completionSize = 5000;

View File

@ -0,0 +1,95 @@
package at.procon.eventhub.esperpoc.api;
import at.procon.eventhub.esperpoc.dto.EsperActivityMergeMode;
import at.procon.eventhub.esperpoc.dto.EsperOperatingPeriodRequest;
import at.procon.eventhub.esperpoc.dto.EsperOperatingPeriodResultDto;
import at.procon.eventhub.esperpoc.dto.EsperPocRequest;
import at.procon.eventhub.esperpoc.dto.EsperPocResultDto;
import at.procon.eventhub.esperpoc.dto.EsperShiftResolutionMode;
import at.procon.eventhub.esperpoc.dto.EsperUnknownTreatmentMode;
import at.procon.eventhub.esperpoc.service.EsperOperatingPeriodEvaluationService;
import at.procon.eventhub.esperpoc.service.EsperPocDriverCardActivityService;
import java.time.OffsetDateTime;
import java.util.UUID;
import org.springframework.format.annotation.DateTimeFormat;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;
@RestController
@RequestMapping("/api/eventhub/esper-poc")
public class EsperPocController {
private final EsperPocDriverCardActivityService service;
private final EsperOperatingPeriodEvaluationService operatingPeriodEvaluationService;
public EsperPocController(
EsperPocDriverCardActivityService service,
EsperOperatingPeriodEvaluationService operatingPeriodEvaluationService
) {
this.service = service;
this.operatingPeriodEvaluationService = operatingPeriodEvaluationService;
}
@GetMapping("/tachograph/driver-card-activities")
public ResponseEntity<EsperPocResultDto> evaluateDriverCardActivities(
@RequestParam String tenantKey,
@RequestParam UUID driverEntityId,
@RequestParam @DateTimeFormat(iso = DateTimeFormat.ISO.DATE_TIME) OffsetDateTime occurredFrom,
@RequestParam @DateTimeFormat(iso = DateTimeFormat.ISO.DATE_TIME) OffsetDateTime occurredTo,
@RequestParam(defaultValue = "24") Integer guardHours,
@RequestParam(defaultValue = "3") Integer significantDrivingMinutes,
@RequestParam(defaultValue = "60") Integer mergeGapSeconds,
@RequestParam(defaultValue = "7") Integer operatingPeriodSplitRestHours,
@RequestParam(defaultValue = "420") Integer shiftEndMarkPeriodMinutes,
@RequestParam(defaultValue = "5") Integer absenceBeginEndMinActivityMinutes,
@RequestParam(required = false) EsperActivityMergeMode activityMergeMode,
@RequestParam(required = false) EsperShiftResolutionMode shiftResolutionMode
) {
EsperPocRequest request = new EsperPocRequest(
tenantKey,
driverEntityId,
occurredFrom,
occurredTo,
guardHours,
significantDrivingMinutes,
mergeGapSeconds,
operatingPeriodSplitRestHours,
shiftEndMarkPeriodMinutes,
absenceBeginEndMinActivityMinutes,
activityMergeMode,
shiftResolutionMode
);
return ResponseEntity.ok(service.evaluate(request));
}
@GetMapping("/tachograph/operating-period-evaluation")
public ResponseEntity<EsperOperatingPeriodResultDto> evaluateOperatingPeriods(
@RequestParam String tenantKey,
@RequestParam UUID driverId,
@RequestParam @DateTimeFormat(iso = DateTimeFormat.ISO.DATE_TIME) OffsetDateTime occurredFrom,
@RequestParam @DateTimeFormat(iso = DateTimeFormat.ISO.DATE_TIME) OffsetDateTime occurredTo,
@RequestParam(defaultValue = "24") Integer guardHours,
@RequestParam(required = false) Integer operatingSplitIdleHours,
@RequestParam(required = false) Integer significantDrivingMinutes,
@RequestParam(required = false) Integer mergeGapSeconds,
@RequestParam(required = false) Integer gapDetectionToleranceSeconds,
@RequestParam(required = false) EsperUnknownTreatmentMode unknownTreatmentMode
) {
EsperOperatingPeriodRequest request = new EsperOperatingPeriodRequest(
tenantKey,
driverId,
occurredFrom,
occurredTo,
guardHours,
operatingSplitIdleHours,
significantDrivingMinutes,
mergeGapSeconds,
gapDetectionToleranceSeconds,
unknownTreatmentMode
);
return ResponseEntity.ok(operatingPeriodEvaluationService.evaluate(request));
}
}

View File

@ -0,0 +1,96 @@
package at.procon.eventhub.esperpoc.dto;
import java.time.Duration;
import java.time.OffsetDateTime;
import java.util.List;
import java.util.UUID;
public record ActivityIntervalDto(
UUID driverEntityId,
UUID vehicleId,
UUID vehicleRegistrationId,
String activityType,
String cardSlot,
String cardStatus,
String drivingStatus,
String sourceKind,
OffsetDateTime startedAt,
OffsetDateTime endedAt,
long durationSeconds,
String sourceRowId,
List<String> sourceRowIds,
boolean clippedToRequestedPeriod,
String level
) {
public static ActivityIntervalDto raw(
UUID driverEntityId,
UUID vehicleId,
UUID vehicleRegistrationId,
String activityType,
String cardSlot,
String cardStatus,
String drivingStatus,
String sourceKind,
OffsetDateTime startedAt,
OffsetDateTime endedAt,
String sourceRowId
) {
return new ActivityIntervalDto(
driverEntityId,
vehicleId,
vehicleRegistrationId,
activityType,
cardSlot,
cardStatus,
drivingStatus,
sourceKind,
startedAt,
endedAt,
Duration.between(startedAt, endedAt).getSeconds(),
sourceRowId,
sourceRowId == null ? List.of() : List.of(sourceRowId),
false,
"RAW_INTERVAL"
);
}
public ActivityIntervalDto withTime(OffsetDateTime newStartedAt, OffsetDateTime newEndedAt, boolean clipped) {
return new ActivityIntervalDto(
driverEntityId,
vehicleId,
vehicleRegistrationId,
activityType,
cardSlot,
cardStatus,
drivingStatus,
sourceKind,
newStartedAt,
newEndedAt,
Duration.between(newStartedAt, newEndedAt).getSeconds(),
sourceRowId,
sourceRowIds,
clipped,
level
);
}
public ActivityIntervalDto asMerged(List<String> mergedSourceRowIds) {
return new ActivityIntervalDto(
driverEntityId,
vehicleId,
vehicleRegistrationId,
activityType,
cardSlot,
cardStatus,
drivingStatus,
sourceKind,
startedAt,
endedAt,
durationSeconds,
sourceRowId,
mergedSourceRowIds,
clippedToRequestedPeriod,
"MERGED_ACTIVITY"
);
}
}

View File

@ -0,0 +1,19 @@
package at.procon.eventhub.esperpoc.dto;
import java.time.OffsetDateTime;
import java.util.Map;
import java.util.UUID;
public record DriverWorkSummaryDto(
UUID driverEntityId,
OffsetDateTime periodFrom,
OffsetDateTime periodTo,
long drivingSeconds,
long workSeconds,
long availabilitySeconds,
long breakRestSeconds,
long workingSeconds,
long operationSeconds,
Map<String, Long> secondsByActivity
) {
}

View File

@ -0,0 +1,12 @@
package at.procon.eventhub.esperpoc.dto;
import java.time.OffsetDateTime;
public record DrivingInterruptionDto(
OffsetDateTime from,
OffsetDateTime to,
long durationSeconds,
String previousDrivingSourceRowId,
String nextDrivingSourceRowId
) {
}

View File

@ -0,0 +1,6 @@
package at.procon.eventhub.esperpoc.dto;
public enum EsperActivityMergeMode {
JAVA,
ESPER
}

View File

@ -0,0 +1,30 @@
package at.procon.eventhub.esperpoc.dto;
import jakarta.validation.constraints.NotBlank;
import jakarta.validation.constraints.NotNull;
import java.time.OffsetDateTime;
import java.util.UUID;
public record EsperOperatingPeriodRequest(
@NotBlank String tenantKey,
@NotNull UUID driverId,
@NotNull OffsetDateTime occurredFrom,
@NotNull OffsetDateTime occurredTo,
Integer guardHours,
Integer operatingSplitIdleHours,
Integer significantDrivingMinutes,
Integer mergeGapSeconds,
Integer gapDetectionToleranceSeconds,
EsperUnknownTreatmentMode unknownTreatmentMode
) {
public EsperOperatingPeriodRequest {
if (occurredFrom != null && occurredTo != null && !occurredFrom.isBefore(occurredTo)) {
throw new IllegalArgumentException("occurredFrom must be before occurredTo");
}
guardHours = guardHours == null ? 24 : Math.max(0, guardHours);
operatingSplitIdleHours = operatingSplitIdleHours == null ? null : Math.max(1, operatingSplitIdleHours);
significantDrivingMinutes = significantDrivingMinutes == null ? null : Math.max(1, significantDrivingMinutes);
mergeGapSeconds = mergeGapSeconds == null ? null : Math.max(0, mergeGapSeconds);
gapDetectionToleranceSeconds = gapDetectionToleranceSeconds == null ? null : Math.max(0, gapDetectionToleranceSeconds);
}
}

View File

@ -0,0 +1,39 @@
package at.procon.eventhub.esperpoc.dto;
import java.time.OffsetDateTime;
import java.util.List;
import java.util.UUID;
public record EsperOperatingPeriodResultDto(
String tenantKey,
UUID driverId,
OffsetDateTime requestedFrom,
OffsetDateTime requestedTo,
OffsetDateTime loadedFrom,
OffsetDateTime loadedTo,
int rawEventCount,
int driverCardRawEventCount,
int vehicleUnitRawEventCount,
int driverCardIntervalCount,
int vehicleUnitIntervalCount,
int resolvedKnownIntervalCount,
int evaluationIntervalCount,
int periodizedIntervalCount,
int mergedIntervalCount,
int nonDrivingIntervalCount,
int operatingPeriodCount,
int operatingSplitIdleHours,
int significantDrivingMinutes,
int mergeGapSeconds,
int gapDetectionToleranceSeconds,
EsperUnknownTreatmentMode unknownTreatmentMode,
List<RawActivityEventDto> rawEvents,
List<ActivityIntervalDto> resolvedKnownIntervals,
List<ActivityIntervalDto> evaluationIntervals,
List<OperatingPeriodActivityIntervalDto> periodizedIntervals,
List<OperatingPeriodActivityIntervalDto> mergedIntervals,
List<NonDrivingIntervalDto> nonDrivingIntervals,
List<OperatingPeriodDto> operatingPeriods,
List<String> notes
) {
}

View File

@ -0,0 +1,33 @@
package at.procon.eventhub.esperpoc.dto;
import jakarta.validation.constraints.NotBlank;
import jakarta.validation.constraints.NotNull;
import java.time.OffsetDateTime;
import java.util.UUID;
public record EsperPocRequest(
@NotBlank String tenantKey,
@NotNull UUID driverEntityId,
@NotNull OffsetDateTime occurredFrom,
@NotNull OffsetDateTime occurredTo,
Integer guardHours,
Integer significantDrivingMinutes,
Integer mergeGapSeconds,
Integer operatingPeriodSplitRestHours,
Integer shiftEndMarkPeriodMinutes,
Integer absenceBeginEndMinActivityMinutes,
EsperActivityMergeMode activityMergeMode,
EsperShiftResolutionMode shiftResolutionMode
) {
public EsperPocRequest {
if (occurredFrom != null && occurredTo != null && !occurredFrom.isBefore(occurredTo)) {
throw new IllegalArgumentException("occurredFrom must be before occurredTo");
}
guardHours = guardHours == null ? 24 : Math.max(0, guardHours);
significantDrivingMinutes = significantDrivingMinutes == null ? 3 : Math.max(1, significantDrivingMinutes);
mergeGapSeconds = mergeGapSeconds == null ? 60 : Math.max(0, mergeGapSeconds);
operatingPeriodSplitRestHours = operatingPeriodSplitRestHours == null ? 7 : Math.max(1, operatingPeriodSplitRestHours);
shiftEndMarkPeriodMinutes = shiftEndMarkPeriodMinutes == null ? 420 : Math.max(1, shiftEndMarkPeriodMinutes);
absenceBeginEndMinActivityMinutes = absenceBeginEndMinActivityMinutes == null ? 5 : Math.max(1, absenceBeginEndMinActivityMinutes);
}
}

View File

@ -0,0 +1,38 @@
package at.procon.eventhub.esperpoc.dto;
import java.time.OffsetDateTime;
import java.util.List;
import java.util.UUID;
public record EsperPocResultDto(
String tenantKey,
UUID driverEntityId,
OffsetDateTime requestedFrom,
OffsetDateTime requestedTo,
OffsetDateTime loadedFrom,
OffsetDateTime loadedTo,
int rawEventCount,
int driverCardRawEventCount,
int vehicleUnitRawEventCount,
int driverCardRawIntervalCount,
int vehicleUnitRawIntervalCount,
int rawIntervalCount,
int mergedActivityCount,
int operatingTimePeriodCount,
int resolvedWorkShiftCount,
int operatingPeriodSplitRestHours,
int shiftEndMarkPeriodMinutes,
int absenceBeginEndMinActivityMinutes,
EsperActivityMergeMode activityMergeMode,
EsperShiftResolutionMode shiftResolutionMode,
List<RawActivityEventDto> raw,
List<ActivityIntervalDto> rawIntervals,
List<ActivityIntervalDto> activities,
List<OperatingTimePeriodDto> operatingTimePeriods,
List<ResolvedWorkShiftDto> workingShifts,
DriverWorkSummaryDto workResultPerDriver,
DriverWorkSummaryDto workingOperationTimesPerEmployee,
ShiftDrivingEvaluationDto drivingTimeInterruptionEvaluation,
List<String> notes
) {
}

View File

@ -0,0 +1,6 @@
package at.procon.eventhub.esperpoc.dto;
public enum EsperShiftResolutionMode {
JAVA,
ESPER
}

View File

@ -0,0 +1,6 @@
package at.procon.eventhub.esperpoc.dto;
public enum EsperUnknownTreatmentMode {
AS_BREAK_REST,
AS_UNKNOWN_NON_BREAK
}

View File

@ -0,0 +1,21 @@
package at.procon.eventhub.esperpoc.dto;
import java.time.OffsetDateTime;
import java.util.UUID;
public record NonDrivingIntervalDto(
UUID driverId,
UUID vehicleId,
UUID vehicleRegistrationId,
String cardSlot,
OffsetDateTime startedAt,
OffsetDateTime endedAt,
long durationSeconds,
long operatingPeriodNo,
OffsetDateTime operatingPeriodStartedAt,
String closedBy,
boolean containsUnknown,
long unknownSeconds,
boolean clippedToRequestedPeriod
) {
}

View File

@ -0,0 +1,110 @@
package at.procon.eventhub.esperpoc.dto;
import java.time.Duration;
import java.time.OffsetDateTime;
import java.util.List;
import java.util.UUID;
public record OperatingPeriodActivityIntervalDto(
UUID driverId,
UUID vehicleId,
UUID vehicleRegistrationId,
String activityType,
String cardSlot,
String cardStatus,
String drivingStatus,
String sourceKind,
OffsetDateTime startedAt,
OffsetDateTime endedAt,
long durationSeconds,
String sourceRowId,
List<String> sourceRowIds,
boolean clippedToRequestedPeriod,
String level,
long operatingPeriodNo,
OffsetDateTime operatingPeriodStartedAt,
boolean newOperatingPeriod,
Long gapSincePreviousActivitySeconds,
boolean synthetic
) {
public static OperatingPeriodActivityIntervalDto periodized(
ActivityIntervalDto interval,
long operatingPeriodNo,
OffsetDateTime operatingPeriodStartedAt,
boolean newOperatingPeriod,
Long gapSincePreviousActivitySeconds
) {
return new OperatingPeriodActivityIntervalDto(
interval.driverEntityId(),
interval.vehicleId(),
interval.vehicleRegistrationId(),
interval.activityType(),
interval.cardSlot(),
interval.cardStatus(),
interval.drivingStatus(),
interval.sourceKind(),
interval.startedAt(),
interval.endedAt(),
interval.durationSeconds(),
interval.sourceRowId(),
interval.sourceRowIds(),
interval.clippedToRequestedPeriod(),
"PERIODIZED_ACTIVITY",
operatingPeriodNo,
operatingPeriodStartedAt,
newOperatingPeriod,
gapSincePreviousActivitySeconds,
"UNKNOWN_GAP".equals(interval.level())
);
}
public OperatingPeriodActivityIntervalDto withTime(OffsetDateTime newStartedAt, OffsetDateTime newEndedAt, boolean clipped) {
return new OperatingPeriodActivityIntervalDto(
driverId,
vehicleId,
vehicleRegistrationId,
activityType,
cardSlot,
cardStatus,
drivingStatus,
sourceKind,
newStartedAt,
newEndedAt,
Duration.between(newStartedAt, newEndedAt).getSeconds(),
sourceRowId,
sourceRowIds,
clipped,
level,
operatingPeriodNo,
operatingPeriodStartedAt,
newOperatingPeriod,
gapSincePreviousActivitySeconds,
synthetic
);
}
public OperatingPeriodActivityIntervalDto asMerged(OffsetDateTime newEndedAt, List<String> mergedSourceRowIds) {
return new OperatingPeriodActivityIntervalDto(
driverId,
vehicleId,
vehicleRegistrationId,
activityType,
cardSlot,
cardStatus,
drivingStatus,
sourceKind,
startedAt,
newEndedAt,
Duration.between(startedAt, newEndedAt).getSeconds(),
sourceRowId,
mergedSourceRowIds,
clippedToRequestedPeriod,
"MERGED_ACTIVITY",
operatingPeriodNo,
operatingPeriodStartedAt,
newOperatingPeriod,
gapSincePreviousActivitySeconds,
synthetic
);
}
}

View File

@ -0,0 +1,22 @@
package at.procon.eventhub.esperpoc.dto;
import java.time.OffsetDateTime;
import java.util.List;
public record OperatingPeriodDto(
long operatingPeriodNo,
OffsetDateTime startedAt,
OffsetDateTime endedAt,
long durationSeconds,
String closedBy,
List<ActivityIntervalDto> rawActivities,
long breakRestSeconds,
long drivingSeconds,
long workSeconds,
long availabilitySeconds,
long unknownSeconds,
int intervalCount,
ShiftDrivingEvaluationDto drivingTimeInterruptionEvaluation,
boolean clippedToRequestedPeriod
) {
}

View File

@ -0,0 +1,17 @@
package at.procon.eventhub.esperpoc.dto;
import java.time.OffsetDateTime;
import java.util.List;
public record OperatingTimePeriodDto(
int sequenceNumber,
OffsetDateTime startedAt,
OffsetDateTime endedAt,
long durationSeconds,
ActivityIntervalDto splitStartedAfterLongRest,
ActivityIntervalDto splitEndedByLongRest,
List<ActivityIntervalDto> activities,
DriverWorkSummaryDto workingOperationTimes,
ShiftDrivingEvaluationDto drivingTimeInterruptionEvaluation
) {
}

View File

@ -0,0 +1,23 @@
package at.procon.eventhub.esperpoc.dto;
import java.time.OffsetDateTime;
import java.util.UUID;
public record RawActivityEventDto(
UUID eventId,
OffsetDateTime occurredAt,
String sourceRowId,
String externalSourceEventId,
String sourceKind,
String extractionCode,
UUID driverEntityId,
UUID vehicleId,
UUID vehicleRegistrationId,
String activityType,
String lifecycle,
String cardSlot,
String cardStatus,
String drivingStatus,
String sourcePackageId
) {
}

View File

@ -0,0 +1,40 @@
package at.procon.eventhub.esperpoc.dto;
import java.time.OffsetDateTime;
import java.util.List;
import java.util.UUID;
public record ResolvedWorkShiftDto(
int sequenceNumber,
OffsetDateTime startedAt,
OffsetDateTime endedAt,
OffsetDateTime nextShiftStartedAt,
long durationSeconds,
long dailyRestingTimeSeconds,
OffsetDateTime drivingFrom,
OffsetDateTime drivingTo,
OffsetDateTime absenceFrom,
OffsetDateTime absenceTo,
List<UUID> vehicleIds,
List<UUID> vehicleRegistrationIds,
String usedDataKind,
String shiftKind,
boolean beginTimeDeviationInRange,
boolean endTimeDeviationInRange,
boolean noDrivingActivity,
boolean dailyRestingTimeBeforeShiftOver24Hours,
boolean durationOver24Hours,
boolean dailyRestingTimeBetween9And11Hours,
boolean dailyRestingTimeBetween7And9Hours,
int activitiesCount,
int drivingCount,
int availabilityCount,
int workCount,
int breakRestCount,
int unknownCount,
int workingTimeCount,
DriverWorkSummaryDto workingOperationTimes,
ShiftDrivingEvaluationDto drivingTimeInterruptionEvaluation,
List<ActivityIntervalDto> activities
) {
}

View File

@ -0,0 +1,14 @@
package at.procon.eventhub.esperpoc.dto;
import java.time.OffsetDateTime;
import java.util.List;
public record ShiftDrivingEvaluationDto(
int significantDrivingMinutes,
OffsetDateTime departureAt,
OffsetDateTime arrivalAt,
ActivityIntervalDto firstSignificantDrivingPeriod,
ActivityIntervalDto lastSignificantDrivingPeriod,
List<DrivingInterruptionDto> interruptionsBetweenSignificantDrivingPeriods
) {
}

View File

@ -0,0 +1,95 @@
package at.procon.eventhub.esperpoc.persistence;
import at.procon.eventhub.esperpoc.dto.RawActivityEventDto;
import java.time.OffsetDateTime;
import java.util.List;
import java.util.UUID;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.stereotype.Repository;
@Repository
public class EsperPocActivityRepository {
private final JdbcTemplate jdbcTemplate;
public EsperPocActivityRepository(JdbcTemplate jdbcTemplate) {
this.jdbcTemplate = jdbcTemplate;
}
public List<RawActivityEventDto> findDriverActivityEvents(
String tenantKey,
UUID driverEntityId,
OffsetDateTime occurredFrom,
OffsetDateTime occurredTo
) {
return jdbcTemplate.query(
"""
select
event.id,
event.occurred_at,
coalesce(
event.payload #>> '{raw,sourceRowId}',
regexp_replace(event.external_source_event_id, ':(START|END)$', '')
) as source_row_id,
event.external_source_event_id,
source.source_kind,
coalesce(pkg.extraction_code,
case
when source.source_kind = 'VEHICLE_UNIT' then 'VU_ACTIVITY'
else 'CARD_ACTIVITY'
end
) as extraction_code,
event.driver_id,
event.vehicle_id,
event.vehicle_registration_id,
event.event_type,
event.lifecycle,
detail.attributes ->> 'cardSlot' as card_slot,
detail.attributes ->> 'cardStatus' as card_status,
detail.attributes ->> 'drivingStatus' as driving_status,
event.source_package_id
from eventhub.event event
join eventhub.event_source source on source.id = event.event_source_id
join eventhub.data_package pkg on pkg.id = event.data_package_id
left join eventhub.event_detail detail on detail.event_occurred_at = event.occurred_at
and detail.event_id = event.id
and detail.detail_type = 'DRIVER_ACTIVITY'
where pkg.tenant_key = ?
and source.provider_key = 'TACHOGRAPH'
and (
(source.source_kind = 'DRIVER_CARD' and coalesce(pkg.extraction_code, 'CARD_ACTIVITY') = 'CARD_ACTIVITY')
or
(source.source_kind = 'VEHICLE_UNIT' and coalesce(pkg.extraction_code, 'VU_ACTIVITY') = 'VU_ACTIVITY')
)
and event.driver_id = ?
and event.occurred_at >= ?
and event.occurred_at < ?
and event.event_domain = 'DRIVER_ACTIVITY'
and event.event_type in ('DRIVE', 'WORK', 'AVAILABILITY', 'BREAK_REST')
and event.lifecycle in ('START', 'END')
order by event.occurred_at, event.lifecycle, event.event_type, event.id
""",
(rs, rowNum) -> new RawActivityEventDto(
(UUID) rs.getObject("id"),
rs.getObject("occurred_at", OffsetDateTime.class),
rs.getString("source_row_id"),
rs.getString("external_source_event_id"),
rs.getString("source_kind"),
rs.getString("extraction_code"),
(UUID) rs.getObject("driver_id"),
(UUID) rs.getObject("vehicle_id"),
(UUID) rs.getObject("vehicle_registration_id"),
rs.getString("event_type"),
rs.getString("lifecycle"),
rs.getString("card_slot"),
rs.getString("card_status"),
rs.getString("driving_status"),
rs.getString("source_package_id")
),
tenantKey,
driverEntityId,
occurredFrom,
occurredTo
);
}
}

View File

@ -0,0 +1,24 @@
package at.procon.eventhub.esperpoc.service;
import java.time.OffsetDateTime;
import java.util.List;
import java.util.UUID;
public record EsperActivityIntervalEvent(
UUID driverEntityId,
UUID vehicleId,
UUID vehicleRegistrationId,
String activityType,
String cardSlot,
String cardStatus,
String drivingStatus,
String sourceKind,
OffsetDateTime startedAt,
OffsetDateTime endedAt,
long durationSeconds,
String sourceRowId,
List<String> sourceRowIds,
boolean clippedToRequestedPeriod,
String level
) {
}

View File

@ -0,0 +1,54 @@
package at.procon.eventhub.esperpoc.service;
import at.procon.eventhub.esperpoc.dto.ActivityIntervalDto;
import java.time.Duration;
import java.time.LocalDate;
import java.util.Objects;
import java.util.Set;
final class EsperActivitySemantics {
private static final Set<String> ACTIVE_ACTIVITY_TYPES = Set.of("DRIVE", "WORK", "AVAILABILITY");
private EsperActivitySemantics() {
}
static boolean canMerge(ActivityIntervalDto left, ActivityIntervalDto right, Duration tolerance) {
boolean sameDriver = Objects.equals(left.driverEntityId(), right.driverEntityId());
boolean sameActivity = Objects.equals(left.activityType(), right.activityType());
boolean sameSlot = Objects.equals(left.cardSlot(), right.cardSlot());
boolean sameCardStatus = Objects.equals(left.cardStatus(), right.cardStatus());
boolean sameDrivingStatus = Objects.equals(left.drivingStatus(), right.drivingStatus());
boolean sameSourceKind = Objects.equals(left.sourceKind(), right.sourceKind());
long gapSeconds = Duration.between(left.endedAt(), right.startedAt()).getSeconds();
boolean adjacentOrOverlapping = gapSeconds <= tolerance.getSeconds();
return sameDriver && sameActivity && sameSlot && sameCardStatus && sameDrivingStatus && sameSourceKind && adjacentOrOverlapping;
}
static boolean isShiftActivity(ActivityIntervalDto interval) {
return isKnownActivity(interval) && ACTIVE_ACTIVITY_TYPES.contains(interval.activityType());
}
static boolean isRestOrUnknown(ActivityIntervalDto interval) {
return ("BREAK_REST".equals(interval.activityType()) && isKnownActivity(interval))
|| isUnknownActivity(interval);
}
static boolean isKnownActivity(ActivityIntervalDto interval) {
return interval.cardStatus() == null
|| !"NOT_INSERTED".equals(interval.cardStatus())
|| "KNOWN".equals(interval.drivingStatus());
}
static boolean isUnknownActivity(ActivityIntervalDto interval) {
if ("UNKNOWN_ACTIVITY".equals(interval.activityType())) {
return true;
}
return "UNKNOWN".equals(interval.drivingStatus())
|| ("NOT_INSERTED".equals(interval.cardStatus()) && !"KNOWN".equals(interval.drivingStatus()));
}
static LocalDate recordDate(ActivityIntervalDto interval) {
return interval.startedAt().toLocalDate();
}
}

View File

@ -0,0 +1,12 @@
package at.procon.eventhub.esperpoc.service;
import java.time.OffsetDateTime;
record EsperClosedOperatingPeriod(
long operatingPeriodNo,
OffsetDateTime startedAt,
OffsetDateTime endedAt,
long durationSeconds,
String closedBy
) {
}

View File

@ -0,0 +1,405 @@
package at.procon.eventhub.esperpoc.service;
import at.procon.eventhub.esperpoc.dto.ActivityIntervalDto;
import at.procon.eventhub.esperpoc.dto.RawActivityEventDto;
import com.espertech.esper.common.client.EPCompiled;
import com.espertech.esper.common.client.EventBean;
import com.espertech.esper.common.client.configuration.Configuration;
import com.espertech.esper.compiler.client.CompilerArguments;
import com.espertech.esper.compiler.client.EPCompileException;
import com.espertech.esper.compiler.client.EPCompilerProvider;
import com.espertech.esper.runtime.client.EPDeployException;
import com.espertech.esper.runtime.client.EPDeployment;
import com.espertech.esper.runtime.client.EPRuntime;
import com.espertech.esper.runtime.client.EPRuntimeProvider;
import java.time.Duration;
import java.time.OffsetDateTime;
import java.util.ArrayList;
import java.util.Comparator;
import java.util.List;
import java.util.UUID;
import java.util.concurrent.atomic.AtomicLong;
import java.util.function.Consumer;
import org.springframework.stereotype.Component;
@Component
public class EsperDriverActivityEngine {
private static final AtomicLong RUNTIME_COUNTER = new AtomicLong();
private static final String INTERVAL_EPL = """
@name('driverCardActivityIntervals')
select
s.driverEntityId as driverEntityId,
s.vehicleId as vehicleId,
s.vehicleRegistrationId as vehicleRegistrationId,
s.eventType as activityType,
s.cardSlot as cardSlot,
s.cardStatus as cardStatus,
s.drivingStatus as drivingStatus,
s.sourceKind as sourceKind,
s.occurredAt as startedAt,
e.occurredAt as endedAt,
s.sourceRowId as sourceRowId
from pattern [
every s = RawDriverActivityPoint(lifecycle = 'START') ->
e = RawDriverActivityPoint(
lifecycle = 'END',
sourceRowId = s.sourceRowId,
eventType = s.eventType,
driverEntityId = s.driverEntityId,
cardSlot = s.cardSlot
)
]
""";
private static final String INTERVAL_STREAM_EPL = """
@name('driverActivityIntervalStream')
select * from DriverActivityInterval
""";
public List<ActivityIntervalDto> buildIntervals(List<RawActivityEventDto> rawEvents) {
if (rawEvents == null || rawEvents.isEmpty()) {
return List.of();
}
List<ActivityIntervalDto> intervals = new ArrayList<>();
executeWithRuntime(
configuration -> configuration.getCommon().addEventType("RawDriverActivityPoint", EsperRawDriverActivityPoint.class),
INTERVAL_EPL,
"driverCardActivityIntervals",
newData -> collectIntervals(newData, intervals),
runtime -> {
List<EsperRawDriverActivityPoint> points = rawEvents.stream()
.sorted(Comparator.comparing(RawActivityEventDto::occurredAt).thenComparing(RawActivityEventDto::lifecycle))
.map(this::toEsperPoint)
.toList();
for (EsperRawDriverActivityPoint point : points) {
runtime.getEventService().sendEventBean(point, "RawDriverActivityPoint");
}
}
);
return intervals.stream()
.filter(interval -> interval.endedAt().isAfter(interval.startedAt()))
.sorted(Comparator.comparing(ActivityIntervalDto::startedAt).thenComparing(ActivityIntervalDto::endedAt))
.toList();
}
public List<ActivityIntervalDto> mergeConsecutiveIdenticalActivities(
List<ActivityIntervalDto> intervals,
Duration mergeGapTolerance
) {
List<ActivityIntervalDto> sorted = sortedPositiveIntervals(intervals);
if (sorted.isEmpty()) {
return List.of();
}
MergedActivityCollector collector = new MergedActivityCollector(mergeGapTolerance);
executeIntervalStream(sorted, collector::accept);
return collector.finish();
}
public List<EsperResolvedShiftSpan> resolveShiftSpans(
List<ActivityIntervalDto> intervals,
Duration shiftEndThreshold,
Duration contiguousGapTolerance
) {
List<ActivityIntervalDto> sorted = sortedPositiveIntervals(intervals);
if (sorted.isEmpty()) {
return List.of();
}
ShiftSpanCollector collector = new ShiftSpanCollector(shiftEndThreshold, contiguousGapTolerance);
executeIntervalStream(sorted, collector::accept);
return collector.finish();
}
private void executeIntervalStream(List<ActivityIntervalDto> intervals, Consumer<ActivityIntervalDto> consumer) {
executeWithRuntime(
configuration -> configuration.getCommon().addEventType("DriverActivityInterval", EsperActivityIntervalEvent.class),
INTERVAL_STREAM_EPL,
"driverActivityIntervalStream",
newData -> collectInputIntervals(newData, consumer),
runtime -> {
for (ActivityIntervalDto interval : intervals) {
runtime.getEventService().sendEventBean(toEsperInterval(interval), "DriverActivityInterval");
}
}
);
}
private void executeWithRuntime(
Consumer<Configuration> configurationSetup,
String epl,
String statementName,
Consumer<EventBean[]> listener,
Consumer<EPRuntime> sender
) {
EPRuntime runtime = null;
try {
Configuration configuration = new Configuration();
configurationSetup.accept(configuration);
String runtimeUri = "eventhub-esper-poc-" + RUNTIME_COUNTER.incrementAndGet();
runtime = EPRuntimeProvider.getRuntime(runtimeUri, configuration);
CompilerArguments arguments = new CompilerArguments(configuration);
EPCompiled compiled = EPCompilerProvider.getCompiler().compile(epl, arguments);
EPDeployment deployment = runtime.getDeploymentService().deploy(compiled);
runtime.getDeploymentService()
.getStatement(deployment.getDeploymentId(), statementName)
.addListener((newData, oldData, statement, rt) -> listener.accept(newData));
sender.accept(runtime);
} catch (EPCompileException | EPDeployException e) {
throw new IllegalStateException("Cannot compile/deploy Esper PoC EPL", e);
} finally {
if (runtime != null) {
runtime.destroy();
}
}
}
private void collectIntervals(EventBean[] newData, List<ActivityIntervalDto> intervals) {
if (newData == null) {
return;
}
for (EventBean event : newData) {
OffsetDateTime startedAt = (OffsetDateTime) event.get("startedAt");
OffsetDateTime endedAt = (OffsetDateTime) event.get("endedAt");
intervals.add(ActivityIntervalDto.raw(
(UUID) event.get("driverEntityId"),
(UUID) event.get("vehicleId"),
(UUID) event.get("vehicleRegistrationId"),
(String) event.get("activityType"),
(String) event.get("cardSlot"),
(String) event.get("cardStatus"),
(String) event.get("drivingStatus"),
(String) event.get("sourceKind"),
startedAt,
endedAt,
(String) event.get("sourceRowId")
));
}
}
private void collectInputIntervals(EventBean[] newData, Consumer<ActivityIntervalDto> consumer) {
if (newData == null) {
return;
}
for (EventBean event : newData) {
EsperActivityIntervalEvent interval = (EsperActivityIntervalEvent) event.getUnderlying();
consumer.accept(new ActivityIntervalDto(
interval.driverEntityId(),
interval.vehicleId(),
interval.vehicleRegistrationId(),
interval.activityType(),
interval.cardSlot(),
interval.cardStatus(),
interval.drivingStatus(),
interval.sourceKind(),
interval.startedAt(),
interval.endedAt(),
interval.durationSeconds(),
interval.sourceRowId(),
interval.sourceRowIds(),
interval.clippedToRequestedPeriod(),
interval.level()
));
}
}
private EsperRawDriverActivityPoint toEsperPoint(RawActivityEventDto event) {
return new EsperRawDriverActivityPoint(
event.eventId(),
event.occurredAt(),
event.sourceRowId(),
event.externalSourceEventId(),
event.driverEntityId(),
event.vehicleId(),
event.vehicleRegistrationId(),
event.activityType(),
event.lifecycle(),
event.cardSlot(),
event.cardStatus(),
event.drivingStatus(),
event.sourceKind()
);
}
private EsperActivityIntervalEvent toEsperInterval(ActivityIntervalDto interval) {
return new EsperActivityIntervalEvent(
interval.driverEntityId(),
interval.vehicleId(),
interval.vehicleRegistrationId(),
interval.activityType(),
interval.cardSlot(),
interval.cardStatus(),
interval.drivingStatus(),
interval.sourceKind(),
interval.startedAt(),
interval.endedAt(),
interval.durationSeconds(),
interval.sourceRowId(),
interval.sourceRowIds(),
interval.clippedToRequestedPeriod(),
interval.level()
);
}
private List<ActivityIntervalDto> sortedPositiveIntervals(List<ActivityIntervalDto> intervals) {
if (intervals == null || intervals.isEmpty()) {
return List.of();
}
return intervals.stream()
.filter(interval -> interval.endedAt().isAfter(interval.startedAt()))
.sorted(Comparator.comparing(ActivityIntervalDto::startedAt)
.thenComparing(ActivityIntervalDto::endedAt)
.thenComparing(ActivityIntervalDto::activityType, Comparator.nullsLast(String::compareTo)))
.toList();
}
private static final class MergedActivityCollector {
private final Duration mergeGapTolerance;
private final List<ActivityIntervalDto> merged = new ArrayList<>();
private ActivityIntervalDto current;
private List<String> currentSources = List.of();
private MergedActivityCollector(Duration mergeGapTolerance) {
this.mergeGapTolerance = mergeGapTolerance;
}
private void accept(ActivityIntervalDto next) {
if (current == null) {
current = next;
currentSources = new ArrayList<>(next.sourceRowIds());
return;
}
if (EsperActivitySemantics.canMerge(current, next, mergeGapTolerance)) {
currentSources.addAll(next.sourceRowIds());
OffsetDateTime newEndedAt = current.endedAt().isAfter(next.endedAt()) ? current.endedAt() : next.endedAt();
current = new ActivityIntervalDto(
current.driverEntityId(),
current.vehicleId() == null ? next.vehicleId() : current.vehicleId(),
current.vehicleRegistrationId() == null ? next.vehicleRegistrationId() : current.vehicleRegistrationId(),
current.activityType(),
current.cardSlot(),
current.cardStatus(),
current.drivingStatus(),
current.sourceKind(),
current.startedAt(),
newEndedAt,
Duration.between(current.startedAt(), newEndedAt).getSeconds(),
current.sourceRowId(),
List.copyOf(currentSources),
current.clippedToRequestedPeriod() || next.clippedToRequestedPeriod(),
"MERGED_ACTIVITY"
);
return;
}
merged.add(current.asMerged(List.copyOf(currentSources)));
current = next;
currentSources = new ArrayList<>(next.sourceRowIds());
}
private List<ActivityIntervalDto> finish() {
if (current != null) {
merged.add(current.asMerged(List.copyOf(currentSources)));
}
return List.copyOf(merged);
}
}
private static final class ShiftSpanCollector {
private final Duration shiftEndThreshold;
private final Duration contiguousGapTolerance;
private final List<ActivityIntervalDto> seen = new ArrayList<>();
private final List<EsperResolvedShiftSpan> spans = new ArrayList<>();
private OffsetDateTime shiftBegin;
private long accumulatedRestSeconds;
private OffsetDateTime prevActivityEnd;
private OffsetDateTime pendingShiftEndTriggerAt;
private ShiftSpanCollector(Duration shiftEndThreshold, Duration contiguousGapTolerance) {
this.shiftEndThreshold = shiftEndThreshold;
this.contiguousGapTolerance = contiguousGapTolerance;
}
private void accept(ActivityIntervalDto interval) {
seen.add(interval);
if (shiftBegin == null) {
if (EsperActivitySemantics.isShiftActivity(interval)) {
shiftBegin = interval.startedAt();
}
prevActivityEnd = interval.endedAt();
return;
}
if (!shiftBegin.isBefore(interval.startedAt())) {
prevActivityEnd = interval.endedAt();
return;
}
if (pendingShiftEndTriggerAt != null && EsperActivitySemantics.isShiftActivity(interval)) {
appendShiftSpan(shiftBegin, interval.startedAt());
shiftBegin = interval.startedAt();
pendingShiftEndTriggerAt = null;
accumulatedRestSeconds = 0;
}
if (EsperActivitySemantics.isRestOrUnknown(interval)) {
long gapSeconds = prevActivityEnd == null ? 0 : Math.max(0, Duration.between(prevActivityEnd, interval.startedAt()).getSeconds());
boolean contiguous = prevActivityEnd != null
&& Math.abs(Duration.between(prevActivityEnd, interval.startedAt()).getSeconds()) <= contiguousGapTolerance.getSeconds();
boolean crossedRecordDate = prevActivityEnd != null
&& !interval.startedAt().toLocalDate().equals(prevActivityEnd.toLocalDate());
if (contiguous) {
accumulatedRestSeconds += interval.durationSeconds();
} else if (crossedRecordDate) {
accumulatedRestSeconds = interval.durationSeconds() + gapSeconds;
} else {
accumulatedRestSeconds = interval.durationSeconds();
}
if (accumulatedRestSeconds >= shiftEndThreshold.getSeconds() && pendingShiftEndTriggerAt == null) {
pendingShiftEndTriggerAt = interval.startedAt();
}
} else if (EsperActivitySemantics.isShiftActivity(interval)) {
long gapSeconds = prevActivityEnd == null ? 0 : Math.max(0, Duration.between(prevActivityEnd, interval.startedAt()).getSeconds());
boolean crossedRecordDate = prevActivityEnd != null
&& !interval.startedAt().toLocalDate().equals(prevActivityEnd.toLocalDate());
if (crossedRecordDate && accumulatedRestSeconds + gapSeconds >= shiftEndThreshold.getSeconds()) {
appendShiftSpan(shiftBegin, interval.startedAt());
shiftBegin = interval.startedAt();
}
accumulatedRestSeconds = 0;
pendingShiftEndTriggerAt = null;
}
prevActivityEnd = interval.endedAt();
}
private List<EsperResolvedShiftSpan> finish() {
if (shiftBegin != null) {
appendShiftSpan(shiftBegin, null);
}
return List.copyOf(spans);
}
private void appendShiftSpan(OffsetDateTime currentShiftBegin, OffsetDateTime nextShiftBegin) {
if (currentShiftBegin == null) {
return;
}
OffsetDateTime shiftEnd = seen.stream()
.filter(interval -> !interval.startedAt().isBefore(currentShiftBegin))
.filter(interval -> nextShiftBegin == null || interval.startedAt().isBefore(nextShiftBegin))
.filter(EsperActivitySemantics::isShiftActivity)
.map(ActivityIntervalDto::endedAt)
.max(OffsetDateTime::compareTo)
.orElse(null);
if (shiftEnd == null || !shiftEnd.isAfter(currentShiftBegin)) {
return;
}
long restSeconds = nextShiftBegin == null ? 0 : Math.max(0, Duration.between(shiftEnd, nextShiftBegin).getSeconds());
spans.add(new EsperResolvedShiftSpan(currentShiftBegin, shiftEnd, nextShiftBegin, restSeconds));
}
}
}

View File

@ -0,0 +1,257 @@
package at.procon.eventhub.esperpoc.service;
import at.procon.eventhub.esperpoc.dto.ActivityIntervalDto;
import at.procon.eventhub.esperpoc.dto.OperatingPeriodActivityIntervalDto;
import com.espertech.esper.common.client.EPCompiled;
import com.espertech.esper.common.client.EventBean;
import com.espertech.esper.common.client.configuration.Configuration;
import com.espertech.esper.compiler.client.CompilerArguments;
import com.espertech.esper.compiler.client.EPCompileException;
import com.espertech.esper.compiler.client.EPCompilerProvider;
import com.espertech.esper.runtime.client.EPDeployException;
import com.espertech.esper.runtime.client.EPDeployment;
import com.espertech.esper.runtime.client.EPRuntime;
import com.espertech.esper.runtime.client.EPRuntimeProvider;
import java.time.Duration;
import java.time.OffsetDateTime;
import java.util.ArrayList;
import java.util.Comparator;
import java.util.List;
import java.util.concurrent.atomic.AtomicLong;
import org.springframework.stereotype.Component;
@Component
public class EsperOperatingPeriodEngine {
private static final AtomicLong RUNTIME_COUNTER = new AtomicLong();
private static final String INPUT_STREAM_EPL = """
@name('operatingPeriodIntervalStream')
select * from OperatingPeriodIntervalInputEvent
""";
public EsperOperatingPeriodEvaluation evaluate(
List<ActivityIntervalDto> intervals,
Duration operatingSplitIdleThreshold
) {
List<ActivityIntervalDto> sorted = sortedPositiveIntervals(intervals);
if (sorted.isEmpty()) {
return new EsperOperatingPeriodEvaluation(List.of(), List.of());
}
PeriodizationCollector collector = new PeriodizationCollector(operatingSplitIdleThreshold);
executeWithRuntime(
configuration -> configuration.getCommon().addEventType("OperatingPeriodIntervalInputEvent", EsperOperatingPeriodIntervalInputEvent.class),
INPUT_STREAM_EPL,
"operatingPeriodIntervalStream",
newData -> collectInputIntervals(newData, collector),
runtime -> {
for (ActivityIntervalDto interval : sorted) {
runtime.getEventService().sendEventBean(toInputEvent(interval), "OperatingPeriodIntervalInputEvent");
}
}
);
return collector.finish();
}
private void executeWithRuntime(
java.util.function.Consumer<Configuration> configurationSetup,
String epl,
String statementName,
java.util.function.Consumer<EventBean[]> listener,
java.util.function.Consumer<EPRuntime> sender
) {
EPRuntime runtime = null;
try {
Configuration configuration = new Configuration();
configurationSetup.accept(configuration);
String runtimeUri = "eventhub-esper-operating-period-" + RUNTIME_COUNTER.incrementAndGet();
runtime = EPRuntimeProvider.getRuntime(runtimeUri, configuration);
CompilerArguments arguments = new CompilerArguments(configuration);
EPCompiled compiled = EPCompilerProvider.getCompiler().compile(epl, arguments);
EPDeployment deployment = runtime.getDeploymentService().deploy(compiled);
runtime.getDeploymentService()
.getStatement(deployment.getDeploymentId(), statementName)
.addListener((newData, oldData, statement, rt) -> listener.accept(newData));
sender.accept(runtime);
} catch (EPCompileException | EPDeployException e) {
throw new IllegalStateException("Cannot compile/deploy Esper operating-period EPL", e);
} finally {
if (runtime != null) {
runtime.destroy();
}
}
}
private void collectInputIntervals(EventBean[] newData, PeriodizationCollector collector) {
if (newData == null) {
return;
}
for (EventBean event : newData) {
collector.accept((EsperOperatingPeriodIntervalInputEvent) event.getUnderlying());
}
}
private EsperOperatingPeriodIntervalInputEvent toInputEvent(ActivityIntervalDto interval) {
return new EsperOperatingPeriodIntervalInputEvent(
interval.driverEntityId(),
interval.vehicleId(),
interval.vehicleRegistrationId(),
interval.activityType(),
interval.cardSlot(),
interval.cardStatus(),
interval.drivingStatus(),
interval.sourceKind(),
interval.startedAt().toInstant().toEpochMilli(),
interval.endedAt().toInstant().toEpochMilli(),
interval.durationSeconds() * 1000L,
interval.sourceRowId(),
interval.sourceRowIds(),
interval.clippedToRequestedPeriod(),
"UNKNOWN_GAP".equals(interval.level())
);
}
private List<ActivityIntervalDto> sortedPositiveIntervals(List<ActivityIntervalDto> intervals) {
if (intervals == null || intervals.isEmpty()) {
return List.of();
}
return intervals.stream()
.filter(interval -> interval.endedAt().isAfter(interval.startedAt()))
.sorted(Comparator.comparing(ActivityIntervalDto::startedAt)
.thenComparing(ActivityIntervalDto::endedAt)
.thenComparing(ActivityIntervalDto::activityType, Comparator.nullsLast(String::compareTo)))
.toList();
}
public record EsperOperatingPeriodEvaluation(
List<OperatingPeriodActivityIntervalDto> periodizedIntervals,
List<EsperClosedOperatingPeriod> closedPeriods
) {
}
private static final class PeriodizationCollector {
private final Duration operatingSplitIdleThreshold;
private final List<OperatingPeriodActivityIntervalDto> periodizedIntervals = new ArrayList<>();
private final List<EsperClosedOperatingPeriod> closedPeriods = new ArrayList<>();
private boolean hasOpenPeriod;
private long operatingPeriodNo;
private OffsetDateTime operatingPeriodStartedAt;
private OffsetDateTime lastKnownActivityEndAt;
private PeriodizationCollector(Duration operatingSplitIdleThreshold) {
this.operatingSplitIdleThreshold = operatingSplitIdleThreshold;
}
private void accept(EsperOperatingPeriodIntervalInputEvent interval) {
ActivityIntervalDto dto = new ActivityIntervalDto(
interval.driverId(),
interval.vehicleId(),
interval.vehicleRegistrationId(),
interval.activityType(),
interval.cardSlot(),
interval.cardStatus(),
interval.drivingStatus(),
interval.sourceKind(),
OffsetDateTime.ofInstant(java.time.Instant.ofEpochMilli(interval.startTs()), java.time.ZoneOffset.UTC),
OffsetDateTime.ofInstant(java.time.Instant.ofEpochMilli(interval.endTs()), java.time.ZoneOffset.UTC),
interval.durationMs() / 1000L,
interval.sourceRowId(),
interval.sourceRowIds(),
interval.clippedToRequestedPeriod(),
interval.synthetic() ? "UNKNOWN_GAP" : "RAW_INTERVAL"
);
if ("UNKNOWN".equals(dto.activityType())) {
if (!hasOpenPeriod) {
return;
}
if (dto.durationSeconds() >= operatingSplitIdleThreshold.getSeconds()) {
closeCurrent("UNKNOWN_GAP");
return;
}
periodizedIntervals.add(OperatingPeriodActivityIntervalDto.periodized(
dto,
operatingPeriodNo,
operatingPeriodStartedAt,
false,
Math.max(0, Duration.between(lastKnownActivityEndAt, dto.startedAt()).getSeconds())
));
return;
}
if (!hasOpenPeriod) {
operatingPeriodNo = operatingPeriodNo < 1 ? 1 : operatingPeriodNo + 1;
hasOpenPeriod = true;
operatingPeriodStartedAt = dto.startedAt();
lastKnownActivityEndAt = dto.endedAt();
periodizedIntervals.add(OperatingPeriodActivityIntervalDto.periodized(
dto,
operatingPeriodNo,
operatingPeriodStartedAt,
true,
null
));
return;
}
long gapSeconds = Math.max(0, Duration.between(lastKnownActivityEndAt, dto.startedAt()).getSeconds());
if (gapSeconds >= operatingSplitIdleThreshold.getSeconds()) {
closeCurrent("IDLE_GAP");
operatingPeriodNo++;
hasOpenPeriod = true;
operatingPeriodStartedAt = dto.startedAt();
lastKnownActivityEndAt = dto.endedAt();
periodizedIntervals.add(OperatingPeriodActivityIntervalDto.periodized(
dto,
operatingPeriodNo,
operatingPeriodStartedAt,
true,
gapSeconds
));
return;
}
periodizedIntervals.add(OperatingPeriodActivityIntervalDto.periodized(
dto,
operatingPeriodNo,
operatingPeriodStartedAt,
false,
gapSeconds
));
if (dto.endedAt().isAfter(lastKnownActivityEndAt)) {
lastKnownActivityEndAt = dto.endedAt();
}
}
private EsperOperatingPeriodEvaluation finish() {
if (hasOpenPeriod) {
closeCurrent("FLUSH");
}
return new EsperOperatingPeriodEvaluation(
periodizedIntervals.stream()
.sorted(Comparator.comparing(OperatingPeriodActivityIntervalDto::startedAt)
.thenComparing(OperatingPeriodActivityIntervalDto::endedAt)
.thenComparing(OperatingPeriodActivityIntervalDto::activityType, Comparator.nullsLast(String::compareTo)))
.toList(),
closedPeriods.stream()
.sorted(Comparator.comparing(EsperClosedOperatingPeriod::startedAt))
.toList()
);
}
private void closeCurrent(String closedBy) {
if (!hasOpenPeriod || operatingPeriodStartedAt == null || lastKnownActivityEndAt == null) {
hasOpenPeriod = false;
return;
}
closedPeriods.add(new EsperClosedOperatingPeriod(
operatingPeriodNo,
operatingPeriodStartedAt,
lastKnownActivityEndAt,
Duration.between(operatingPeriodStartedAt, lastKnownActivityEndAt).getSeconds(),
closedBy
));
hasOpenPeriod = false;
}
}
}

View File

@ -0,0 +1,807 @@
package at.procon.eventhub.esperpoc.service;
import at.procon.eventhub.config.EventHubProperties;
import at.procon.eventhub.esperpoc.dto.ActivityIntervalDto;
import at.procon.eventhub.esperpoc.dto.DrivingInterruptionDto;
import at.procon.eventhub.esperpoc.dto.EsperOperatingPeriodRequest;
import at.procon.eventhub.esperpoc.dto.EsperOperatingPeriodResultDto;
import at.procon.eventhub.esperpoc.dto.EsperUnknownTreatmentMode;
import at.procon.eventhub.esperpoc.dto.NonDrivingIntervalDto;
import at.procon.eventhub.esperpoc.dto.OperatingPeriodActivityIntervalDto;
import at.procon.eventhub.esperpoc.dto.OperatingPeriodDto;
import at.procon.eventhub.esperpoc.dto.RawActivityEventDto;
import at.procon.eventhub.esperpoc.dto.ShiftDrivingEvaluationDto;
import at.procon.eventhub.esperpoc.persistence.EsperPocActivityRepository;
import java.time.Duration;
import java.time.OffsetDateTime;
import java.util.ArrayList;
import java.util.Comparator;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.UUID;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
@Service
public class EsperOperatingPeriodEvaluationService {
private static final Logger log = LoggerFactory.getLogger(EsperOperatingPeriodEvaluationService.class);
private final EsperPocActivityRepository activityRepository;
private final EsperDriverActivityEngine activityEngine;
private final EsperOperatingPeriodEngine operatingPeriodEngine;
private final EventHubProperties properties;
public EsperOperatingPeriodEvaluationService(
EsperPocActivityRepository activityRepository,
EsperDriverActivityEngine activityEngine,
EsperOperatingPeriodEngine operatingPeriodEngine
) {
this(activityRepository, activityEngine, operatingPeriodEngine, null);
}
@Autowired
public EsperOperatingPeriodEvaluationService(
EsperPocActivityRepository activityRepository,
EsperDriverActivityEngine activityEngine,
EsperOperatingPeriodEngine operatingPeriodEngine,
EventHubProperties properties
) {
this.activityRepository = activityRepository;
this.activityEngine = activityEngine;
this.operatingPeriodEngine = operatingPeriodEngine;
this.properties = properties;
}
public EsperOperatingPeriodResultDto evaluate(EsperOperatingPeriodRequest request) {
long startedNanos = System.nanoTime();
OffsetDateTime requestedFrom = utc(request.occurredFrom());
OffsetDateTime requestedTo = utc(request.occurredTo());
OffsetDateTime loadedFrom = requestedFrom.minusHours(request.guardHours());
OffsetDateTime loadedTo = requestedTo.plusHours(request.guardHours());
Duration splitIdleThreshold = Duration.ofHours(resolveOperatingSplitIdleHours(request));
Duration significantDrivingThreshold = Duration.ofMinutes(resolveSignificantDrivingMinutes(request));
Duration mergeGapTolerance = Duration.ofSeconds(resolveMergeGapSeconds(request));
Duration gapDetectionTolerance = Duration.ofSeconds(resolveGapDetectionToleranceSeconds(request));
EsperUnknownTreatmentMode unknownTreatmentMode = resolveUnknownTreatmentMode(request);
long dbStartedNanos = System.nanoTime();
List<RawActivityEventDto> rawEvents = activityRepository.findDriverActivityEvents(
request.tenantKey(),
request.driverId(),
loadedFrom,
loadedTo
);
long dbElapsedMs = elapsedMillis(dbStartedNanos);
List<RawActivityEventDto> driverCardRawEvents = rawEvents.stream()
.filter(event -> "DRIVER_CARD".equals(event.sourceKind()))
.toList();
List<RawActivityEventDto> vehicleUnitRawEvents = rawEvents.stream()
.filter(event -> "VEHICLE_UNIT".equals(event.sourceKind()))
.toList();
long cardIntervalsStartedNanos = System.nanoTime();
List<ActivityIntervalDto> driverCardRawIntervals = activityEngine.buildIntervals(driverCardRawEvents);
long cardIntervalsElapsedMs = elapsedMillis(cardIntervalsStartedNanos);
long vuIntervalsStartedNanos = System.nanoTime();
List<ActivityIntervalDto> vehicleUnitRawIntervals = activityEngine.buildIntervals(vehicleUnitRawEvents);
long vuIntervalsElapsedMs = elapsedMillis(vuIntervalsStartedNanos);
long vuGapFillStartedNanos = System.nanoTime();
List<ActivityIntervalDto> resolvedKnownLoadedIntervals = resolveVuFillGaps(driverCardRawIntervals, vehicleUnitRawIntervals);
long vuGapFillElapsedMs = elapsedMillis(vuGapFillStartedNanos);
long unknownGapStartedNanos = System.nanoTime();
List<ActivityIntervalDto> evaluationLoadedIntervals = synthesizeUnknownGaps(
resolvedKnownLoadedIntervals,
gapDetectionTolerance
);
long unknownGapElapsedMs = elapsedMillis(unknownGapStartedNanos);
long periodizeStartedNanos = System.nanoTime();
EsperOperatingPeriodEngine.EsperOperatingPeriodEvaluation evaluation = operatingPeriodEngine.evaluate(
evaluationLoadedIntervals,
splitIdleThreshold
);
long periodizeElapsedMs = elapsedMillis(periodizeStartedNanos);
long mergeStartedNanos = System.nanoTime();
List<OperatingPeriodActivityIntervalDto> mergedLoadedIntervals = mergeConsecutiveActivities(
evaluation.periodizedIntervals(),
mergeGapTolerance
);
long mergeElapsedMs = elapsedMillis(mergeStartedNanos);
long nonDrivingStartedNanos = System.nanoTime();
List<NonDrivingIntervalDto> nonDrivingLoadedIntervals = buildNonDrivingIntervals(
mergedLoadedIntervals,
significantDrivingThreshold,
unknownTreatmentMode
);
long nonDrivingElapsedMs = elapsedMillis(nonDrivingStartedNanos);
List<OperatingPeriodActivityIntervalDto> periodizedIntervals = clipOperatingIntervals(
evaluation.periodizedIntervals(),
requestedFrom,
requestedTo
);
List<OperatingPeriodActivityIntervalDto> mergedIntervals = clipOperatingIntervals(
mergedLoadedIntervals,
requestedFrom,
requestedTo
);
List<NonDrivingIntervalDto> nonDrivingIntervals = clipNonDrivingIntervals(
nonDrivingLoadedIntervals,
requestedFrom,
requestedTo
);
List<OperatingPeriodDto> operatingPeriods = buildOperatingPeriods(
evaluation.closedPeriods(),
periodizedIntervals,
clipKnownActivities(resolvedKnownLoadedIntervals, requestedFrom, requestedTo),
requestedFrom,
requestedTo,
mergeGapTolerance,
significantDrivingThreshold
);
long totalElapsedMs = elapsedMillis(startedNanos);
log.info("Esper operating-period evaluation tenant={} driverId={} requestedFrom={} requestedTo={} loadedFrom={} loadedTo={} unknownMode={} rawEvents={} cardRawEvents={} vuRawEvents={} cardIntervals={} vuIntervals={} resolvedKnownIntervals={} evaluationIntervals={} periodizedIntervals={} mergedIntervals={} nonDrivingIntervals={} operatingPeriods={} timingsMs={{dbRetrieve={}, cardIntervalEsper={}, vuIntervalEsper={}, vuGapFill={}, synthUnknown={}, periodizeEsper={}, merge={}, nonDriving={}, total={}}}",
request.tenantKey(),
request.driverId(),
requestedFrom,
requestedTo,
loadedFrom,
loadedTo,
unknownTreatmentMode,
rawEvents.size(),
driverCardRawEvents.size(),
vehicleUnitRawEvents.size(),
driverCardRawIntervals.size(),
vehicleUnitRawIntervals.size(),
resolvedKnownLoadedIntervals.size(),
evaluationLoadedIntervals.size(),
periodizedIntervals.size(),
mergedIntervals.size(),
nonDrivingIntervals.size(),
operatingPeriods.size(),
dbElapsedMs,
cardIntervalsElapsedMs,
vuIntervalsElapsedMs,
vuGapFillElapsedMs,
unknownGapElapsedMs,
periodizeElapsedMs,
mergeElapsedMs,
nonDrivingElapsedMs,
totalElapsedMs);
return new EsperOperatingPeriodResultDto(
request.tenantKey(),
request.driverId(),
requestedFrom,
requestedTo,
loadedFrom,
loadedTo,
rawEvents.size(),
driverCardRawEvents.size(),
vehicleUnitRawEvents.size(),
driverCardRawIntervals.size(),
vehicleUnitRawIntervals.size(),
resolvedKnownLoadedIntervals.size(),
evaluationLoadedIntervals.size(),
periodizedIntervals.size(),
mergedIntervals.size(),
nonDrivingIntervals.size(),
operatingPeriods.size(),
resolveOperatingSplitIdleHours(request),
resolveSignificantDrivingMinutes(request),
resolveMergeGapSeconds(request),
resolveGapDetectionToleranceSeconds(request),
unknownTreatmentMode,
rawEvents,
resolvedKnownLoadedIntervals,
evaluationLoadedIntervals,
periodizedIntervals,
mergedIntervals,
nonDrivingIntervals,
operatingPeriods,
notes(
unknownTreatmentMode,
resolveOperatingSplitIdleHours(request),
resolveSignificantDrivingMinutes(request),
resolveGapDetectionToleranceSeconds(request)
)
);
}
List<ActivityIntervalDto> synthesizeUnknownGaps(
List<ActivityIntervalDto> resolvedKnownLoadedIntervals,
Duration gapDetectionTolerance
) {
List<ActivityIntervalDto> allKnown = sortedPositiveIntervals(resolvedKnownLoadedIntervals);
List<ActivityIntervalDto> nonRestActivities = allKnown.stream()
.filter(interval -> !"BREAK_REST".equals(interval.activityType()))
.toList();
if (nonRestActivities.isEmpty()) {
return List.of();
}
List<ActivityIntervalDto> result = new ArrayList<>();
for (int index = 0; index < nonRestActivities.size(); index++) {
ActivityIntervalDto current = nonRestActivities.get(index);
result.add(current);
if (index + 1 >= nonRestActivities.size()) {
continue;
}
ActivityIntervalDto next = nonRestActivities.get(index + 1);
if (!next.startedAt().isAfter(current.endedAt())) {
continue;
}
OffsetDateTime gapStart = current.endedAt();
OffsetDateTime gapEnd = next.startedAt();
List<ActivityIntervalDto> uncoveredGapSegments = subtractCoverage(
unknownGapTemplate(current, next, gapStart, gapEnd),
allKnown
);
for (ActivityIntervalDto gap : uncoveredGapSegments) {
if (gap.durationSeconds() > gapDetectionTolerance.getSeconds()) {
result.add(gap);
}
}
}
return result.stream()
.sorted(Comparator.comparing(ActivityIntervalDto::startedAt)
.thenComparing(ActivityIntervalDto::endedAt)
.thenComparing(ActivityIntervalDto::activityType, Comparator.nullsLast(String::compareTo)))
.toList();
}
List<OperatingPeriodActivityIntervalDto> mergeConsecutiveActivities(
List<OperatingPeriodActivityIntervalDto> intervals,
Duration mergeGapTolerance
) {
if (intervals == null || intervals.isEmpty()) {
return List.of();
}
List<OperatingPeriodActivityIntervalDto> sorted = intervals.stream()
.filter(interval -> interval.endedAt().isAfter(interval.startedAt()))
.sorted(Comparator.comparing(OperatingPeriodActivityIntervalDto::startedAt)
.thenComparing(OperatingPeriodActivityIntervalDto::endedAt)
.thenComparing(OperatingPeriodActivityIntervalDto::activityType, Comparator.nullsLast(String::compareTo)))
.toList();
List<OperatingPeriodActivityIntervalDto> result = new ArrayList<>();
OperatingPeriodActivityIntervalDto current = null;
List<String> currentSources = new ArrayList<>();
for (OperatingPeriodActivityIntervalDto next : sorted) {
if (current == null) {
current = next;
currentSources = new ArrayList<>(next.sourceRowIds());
continue;
}
if (canMerge(current, next, mergeGapTolerance)) {
currentSources.addAll(next.sourceRowIds());
current = current.asMerged(max(current.endedAt(), next.endedAt()), List.copyOf(currentSources));
} else {
result.add(current.asMerged(current.endedAt(), List.copyOf(currentSources)));
current = next;
currentSources = new ArrayList<>(next.sourceRowIds());
}
}
if (current != null) {
result.add(current.asMerged(current.endedAt(), List.copyOf(currentSources)));
}
return result;
}
List<NonDrivingIntervalDto> buildNonDrivingIntervals(
List<OperatingPeriodActivityIntervalDto> mergedIntervals,
Duration significantDrivingThreshold,
EsperUnknownTreatmentMode unknownTreatmentMode
) {
if (mergedIntervals == null || mergedIntervals.isEmpty()) {
return List.of();
}
List<NonDrivingIntervalDto> result = new ArrayList<>();
NonDrivingAccumulator open = null;
for (OperatingPeriodActivityIntervalDto interval : mergedIntervals) {
if (open != null && open.operatingPeriodNo != interval.operatingPeriodNo()) {
result.add(open.close("NEW_OPERATING_PERIOD"));
open = null;
}
if (startsOrExtendsNonDriving(interval, unknownTreatmentMode)) {
open = open == null ? NonDrivingAccumulator.open(interval) : open.extend(interval);
continue;
}
if ("UNKNOWN".equals(interval.activityType()) && unknownTreatmentMode == EsperUnknownTreatmentMode.AS_UNKNOWN_NON_BREAK) {
if (open != null) {
result.add(open.close("UNKNOWN_GAP"));
open = null;
}
continue;
}
if ("DRIVE".equals(interval.activityType())) {
if (interval.durationSeconds() >= significantDrivingThreshold.getSeconds()) {
if (open != null) {
result.add(open.close("SIGNIFICANT_DRIVING"));
open = null;
}
} else if (open != null) {
open = open.extend(interval);
}
}
}
if (open != null) {
result.add(open.close("FLUSH"));
}
return result;
}
private boolean startsOrExtendsNonDriving(
OperatingPeriodActivityIntervalDto interval,
EsperUnknownTreatmentMode unknownTreatmentMode
) {
if ("WORK".equals(interval.activityType()) || "AVAILABILITY".equals(interval.activityType())) {
return true;
}
return "UNKNOWN".equals(interval.activityType())
&& unknownTreatmentMode == EsperUnknownTreatmentMode.AS_BREAK_REST;
}
private boolean canMerge(
OperatingPeriodActivityIntervalDto left,
OperatingPeriodActivityIntervalDto right,
Duration mergeGapTolerance
) {
long gapSeconds = Duration.between(left.endedAt(), right.startedAt()).getSeconds();
return Objects.equals(left.driverId(), right.driverId())
&& left.operatingPeriodNo() == right.operatingPeriodNo()
&& Objects.equals(left.activityType(), right.activityType())
&& Objects.equals(left.cardSlot(), right.cardSlot())
&& Objects.equals(left.cardStatus(), right.cardStatus())
&& Objects.equals(left.drivingStatus(), right.drivingStatus())
&& Objects.equals(left.sourceKind(), right.sourceKind())
&& left.synthetic() == right.synthetic()
&& gapSeconds <= mergeGapTolerance.getSeconds();
}
private List<OperatingPeriodDto> buildOperatingPeriods(
List<EsperClosedOperatingPeriod> closedPeriods,
List<OperatingPeriodActivityIntervalDto> clippedPeriodizedIntervals,
List<ActivityIntervalDto> clippedKnownActivities,
OffsetDateTime requestedFrom,
OffsetDateTime requestedTo,
Duration mergeGapTolerance,
Duration significantDrivingThreshold
) {
Map<Long, List<OperatingPeriodActivityIntervalDto>> intervalsByPeriod = new LinkedHashMap<>();
for (OperatingPeriodActivityIntervalDto interval : clippedPeriodizedIntervals) {
intervalsByPeriod.computeIfAbsent(interval.operatingPeriodNo(), ignored -> new ArrayList<>()).add(interval);
}
List<OperatingPeriodDto> result = new ArrayList<>();
for (EsperClosedOperatingPeriod closedPeriod : closedPeriods) {
if (!closedPeriod.endedAt().isAfter(requestedFrom) || !closedPeriod.startedAt().isBefore(requestedTo)) {
continue;
}
OffsetDateTime start = max(closedPeriod.startedAt(), requestedFrom);
OffsetDateTime end = min(closedPeriod.endedAt(), requestedTo);
if (!end.isAfter(start)) {
continue;
}
List<OperatingPeriodActivityIntervalDto> intervals = intervalsByPeriod.getOrDefault(closedPeriod.operatingPeriodNo(), List.of());
List<ActivityIntervalDto> rawActivities = clipKnownActivitiesToPeriod(clippedKnownActivities, start, end);
long breakRestSeconds = rawActivities.stream()
.filter(activity -> "BREAK_REST".equals(activity.activityType()))
.mapToLong(ActivityIntervalDto::durationSeconds)
.sum();
ShiftDrivingEvaluationDto drivingEvaluation = evaluateSignificantDrivingWithEsper(
rawActivities,
mergeGapTolerance,
significantDrivingThreshold
);
long drivingSeconds = sumActivitySeconds(intervals, "DRIVE");
long workSeconds = sumActivitySeconds(intervals, "WORK");
long availabilitySeconds = sumActivitySeconds(intervals, "AVAILABILITY");
long unknownSeconds = sumActivitySeconds(intervals, "UNKNOWN");
result.add(new OperatingPeriodDto(
closedPeriod.operatingPeriodNo(),
start,
end,
Duration.between(start, end).getSeconds(),
closedPeriod.closedBy(),
rawActivities,
breakRestSeconds,
drivingSeconds,
workSeconds,
availabilitySeconds,
unknownSeconds,
intervals.size(),
drivingEvaluation,
!start.equals(closedPeriod.startedAt()) || !end.equals(closedPeriod.endedAt())
));
}
return result;
}
private long sumActivitySeconds(List<OperatingPeriodActivityIntervalDto> intervals, String activityType) {
return intervals.stream()
.filter(interval -> activityType.equals(interval.activityType()))
.mapToLong(OperatingPeriodActivityIntervalDto::durationSeconds)
.sum();
}
private List<OperatingPeriodActivityIntervalDto> clipOperatingIntervals(
List<OperatingPeriodActivityIntervalDto> intervals,
OffsetDateTime requestedFrom,
OffsetDateTime requestedTo
) {
return intervals.stream()
.map(interval -> {
OffsetDateTime start = max(interval.startedAt(), requestedFrom);
OffsetDateTime end = min(interval.endedAt(), requestedTo);
if (!end.isAfter(start)) {
return null;
}
boolean clipped = interval.clippedToRequestedPeriod()
|| !start.equals(interval.startedAt())
|| !end.equals(interval.endedAt());
return interval.withTime(start, end, clipped);
})
.filter(Objects::nonNull)
.toList();
}
private List<ActivityIntervalDto> clipKnownActivities(
List<ActivityIntervalDto> activities,
OffsetDateTime requestedFrom,
OffsetDateTime requestedTo
) {
return clipKnownActivitiesToPeriod(activities, requestedFrom, requestedTo);
}
private List<ActivityIntervalDto> clipKnownActivitiesToPeriod(
List<ActivityIntervalDto> activities,
OffsetDateTime periodFrom,
OffsetDateTime periodTo
) {
return activities.stream()
.map(activity -> {
OffsetDateTime start = max(activity.startedAt(), periodFrom);
OffsetDateTime end = min(activity.endedAt(), periodTo);
if (!end.isAfter(start)) {
return null;
}
boolean clipped = activity.clippedToRequestedPeriod()
|| !start.equals(activity.startedAt())
|| !end.equals(activity.endedAt());
return activity.withTime(start, end, clipped);
})
.filter(Objects::nonNull)
.sorted(Comparator.comparing(ActivityIntervalDto::startedAt)
.thenComparing(ActivityIntervalDto::endedAt)
.thenComparing(ActivityIntervalDto::activityType, Comparator.nullsLast(String::compareTo)))
.toList();
}
private List<NonDrivingIntervalDto> clipNonDrivingIntervals(
List<NonDrivingIntervalDto> intervals,
OffsetDateTime requestedFrom,
OffsetDateTime requestedTo
) {
return intervals.stream()
.map(interval -> {
OffsetDateTime start = max(interval.startedAt(), requestedFrom);
OffsetDateTime end = min(interval.endedAt(), requestedTo);
if (!end.isAfter(start)) {
return null;
}
return new NonDrivingIntervalDto(
interval.driverId(),
interval.vehicleId(),
interval.vehicleRegistrationId(),
interval.cardSlot(),
start,
end,
Duration.between(start, end).getSeconds(),
interval.operatingPeriodNo(),
interval.operatingPeriodStartedAt(),
interval.closedBy(),
interval.containsUnknown(),
interval.unknownSeconds(),
!start.equals(interval.startedAt()) || !end.equals(interval.endedAt())
);
})
.filter(Objects::nonNull)
.toList();
}
private ActivityIntervalDto unknownGapTemplate(
ActivityIntervalDto previous,
ActivityIntervalDto next,
OffsetDateTime gapStart,
OffsetDateTime gapEnd
) {
return new ActivityIntervalDto(
previous.driverEntityId(),
Objects.equals(previous.vehicleId(), next.vehicleId()) ? previous.vehicleId() : null,
Objects.equals(previous.vehicleRegistrationId(), next.vehicleRegistrationId()) ? previous.vehicleRegistrationId() : null,
"UNKNOWN",
Objects.equals(previous.cardSlot(), next.cardSlot()) ? previous.cardSlot() : null,
previous.cardStatus(),
"UNKNOWN",
"SYNTHETIC_GAP",
gapStart,
gapEnd,
Duration.between(gapStart, gapEnd).getSeconds(),
null,
List.of(),
false,
"UNKNOWN_GAP"
);
}
private List<ActivityIntervalDto> resolveVuFillGaps(
List<ActivityIntervalDto> driverCardRawIntervals,
List<ActivityIntervalDto> vehicleUnitRawIntervals
) {
List<ActivityIntervalDto> driverCard = sortedPositiveIntervals(driverCardRawIntervals);
List<ActivityIntervalDto> vehicleUnit = sortedPositiveIntervals(vehicleUnitRawIntervals);
if (driverCard.isEmpty()) {
return vehicleUnit;
}
if (vehicleUnit.isEmpty()) {
return driverCard;
}
List<ActivityIntervalDto> resolved = new ArrayList<>(driverCard);
for (ActivityIntervalDto vuInterval : vehicleUnit) {
resolved.addAll(subtractCoverage(vuInterval, driverCard));
}
return resolved.stream()
.sorted(Comparator.comparing(ActivityIntervalDto::startedAt)
.thenComparing(ActivityIntervalDto::endedAt)
.thenComparing(ActivityIntervalDto::sourceKind, Comparator.nullsLast(String::compareTo)))
.toList();
}
private List<ActivityIntervalDto> subtractCoverage(
ActivityIntervalDto candidate,
List<ActivityIntervalDto> coverage
) {
List<ActivityIntervalDto> result = new ArrayList<>();
OffsetDateTime cursor = candidate.startedAt();
for (ActivityIntervalDto covered : coverage) {
if (!covered.endedAt().isAfter(cursor)) {
continue;
}
if (!covered.startedAt().isBefore(candidate.endedAt())) {
break;
}
OffsetDateTime overlapStart = max(cursor, covered.startedAt());
if (overlapStart.isAfter(cursor)) {
result.add(candidate.withTime(cursor, overlapStart, candidate.clippedToRequestedPeriod()));
}
if (covered.endedAt().isAfter(cursor)) {
cursor = max(cursor, covered.endedAt());
}
if (!candidate.endedAt().isAfter(cursor)) {
break;
}
}
if (candidate.endedAt().isAfter(cursor)) {
result.add(candidate.withTime(cursor, candidate.endedAt(), candidate.clippedToRequestedPeriod()));
}
return result.stream()
.filter(interval -> interval.endedAt().isAfter(interval.startedAt()))
.toList();
}
private List<ActivityIntervalDto> sortedPositiveIntervals(List<ActivityIntervalDto> intervals) {
if (intervals == null || intervals.isEmpty()) {
return List.of();
}
return intervals.stream()
.filter(interval -> interval.endedAt().isAfter(interval.startedAt()))
.sorted(Comparator.comparing(ActivityIntervalDto::startedAt)
.thenComparing(ActivityIntervalDto::endedAt)
.thenComparing(ActivityIntervalDto::activityType, Comparator.nullsLast(String::compareTo)))
.toList();
}
private ShiftDrivingEvaluationDto evaluateSignificantDrivingWithEsper(
List<ActivityIntervalDto> rawActivities,
Duration mergeGapTolerance,
Duration significantDrivingThreshold
) {
if (rawActivities.isEmpty()) {
return new ShiftDrivingEvaluationDto(
(int) significantDrivingThreshold.toMinutes(),
null,
null,
null,
null,
List.of()
);
}
List<ActivityIntervalDto> mergedActivities = activityEngine.mergeConsecutiveIdenticalActivities(
rawActivities,
mergeGapTolerance
);
List<ActivityIntervalDto> significantDrivingPeriods = mergedActivities.stream()
.filter(activity -> "DRIVE".equals(activity.activityType()))
.filter(activity -> activity.durationSeconds() > significantDrivingThreshold.getSeconds())
.sorted(Comparator.comparing(ActivityIntervalDto::startedAt))
.toList();
if (significantDrivingPeriods.isEmpty()) {
return new ShiftDrivingEvaluationDto(
(int) significantDrivingThreshold.toMinutes(),
null,
null,
null,
null,
List.of()
);
}
List<DrivingInterruptionDto> interruptions = new ArrayList<>();
for (int index = 1; index < significantDrivingPeriods.size(); index++) {
ActivityIntervalDto previous = significantDrivingPeriods.get(index - 1);
ActivityIntervalDto next = significantDrivingPeriods.get(index);
if (next.startedAt().isAfter(previous.endedAt())) {
interruptions.add(new DrivingInterruptionDto(
previous.endedAt(),
next.startedAt(),
Duration.between(previous.endedAt(), next.startedAt()).getSeconds(),
previous.sourceRowId(),
next.sourceRowId()
));
}
}
ActivityIntervalDto first = significantDrivingPeriods.get(0);
ActivityIntervalDto last = significantDrivingPeriods.get(significantDrivingPeriods.size() - 1);
return new ShiftDrivingEvaluationDto(
(int) significantDrivingThreshold.toMinutes(),
first.startedAt(),
last.endedAt(),
first,
last,
interruptions
);
}
private int resolveOperatingSplitIdleHours(EsperOperatingPeriodRequest request) {
if (request.operatingSplitIdleHours() != null) {
return request.operatingSplitIdleHours();
}
return properties == null ? 7 : properties.getEsperPoc().getOperatingPeriodEvaluation().getOperatingSplitIdleHours();
}
private int resolveSignificantDrivingMinutes(EsperOperatingPeriodRequest request) {
if (request.significantDrivingMinutes() != null) {
return request.significantDrivingMinutes();
}
return properties == null ? 3 : properties.getEsperPoc().getOperatingPeriodEvaluation().getSignificantDrivingMinutes();
}
private int resolveMergeGapSeconds(EsperOperatingPeriodRequest request) {
if (request.mergeGapSeconds() != null) {
return request.mergeGapSeconds();
}
return properties == null ? 0 : properties.getEsperPoc().getOperatingPeriodEvaluation().getMergeGapSeconds();
}
private int resolveGapDetectionToleranceSeconds(EsperOperatingPeriodRequest request) {
if (request.gapDetectionToleranceSeconds() != null) {
return request.gapDetectionToleranceSeconds();
}
return properties == null ? 0 : properties.getEsperPoc().getOperatingPeriodEvaluation().getGapDetectionToleranceSeconds();
}
private EsperUnknownTreatmentMode resolveUnknownTreatmentMode(EsperOperatingPeriodRequest request) {
if (request.unknownTreatmentMode() != null) {
return request.unknownTreatmentMode();
}
return properties == null
? EsperUnknownTreatmentMode.AS_BREAK_REST
: properties.getEsperPoc().getOperatingPeriodEvaluation().getUnknownTreatmentMode();
}
private List<String> notes(
EsperUnknownTreatmentMode unknownTreatmentMode,
int operatingSplitIdleHours,
int significantDrivingMinutes,
int gapDetectionToleranceSeconds
) {
return List.of(
"This endpoint runs in parallel to the existing working-shift PoC and does not change its semantics.",
"BREAK_REST events are ignored for activity evaluation but still prevent synthetic UNKNOWN intervals from being created over covered rest spans.",
"Synthetic UNKNOWN intervals are created only for uncovered gaps between non-rest activities.",
"UNKNOWN treatment mode is " + unknownTreatmentMode + ".",
"Operating periods split after " + operatingSplitIdleHours + " hours of no non-rest activity; significant driving closes non-driving intervals from " + significantDrivingMinutes + " minutes onward.",
"Synthetic UNKNOWN gaps are only emitted when uncovered time exceeds " + gapDetectionToleranceSeconds + " seconds."
);
}
private OffsetDateTime utc(OffsetDateTime value) {
return value == null ? null : value.withOffsetSameInstant(java.time.ZoneOffset.UTC);
}
private OffsetDateTime max(OffsetDateTime left, OffsetDateTime right) {
return left.isAfter(right) ? left : right;
}
private OffsetDateTime min(OffsetDateTime left, OffsetDateTime right) {
return left.isBefore(right) ? left : right;
}
private long elapsedMillis(long startedNanos) {
return Math.round((System.nanoTime() - startedNanos) / 1_000_000.0d);
}
private static final class NonDrivingAccumulator {
private final UUID driverId;
private UUID vehicleId;
private UUID vehicleRegistrationId;
private final String cardSlot;
private final OffsetDateTime startedAt;
private OffsetDateTime endedAt;
private final long operatingPeriodNo;
private final OffsetDateTime operatingPeriodStartedAt;
private boolean containsUnknown;
private long unknownSeconds;
private NonDrivingAccumulator(OperatingPeriodActivityIntervalDto interval) {
this.driverId = interval.driverId();
this.vehicleId = interval.vehicleId();
this.vehicleRegistrationId = interval.vehicleRegistrationId();
this.cardSlot = interval.cardSlot();
this.startedAt = interval.startedAt();
this.endedAt = interval.endedAt();
this.operatingPeriodNo = interval.operatingPeriodNo();
this.operatingPeriodStartedAt = interval.operatingPeriodStartedAt();
this.containsUnknown = "UNKNOWN".equals(interval.activityType());
this.unknownSeconds = "UNKNOWN".equals(interval.activityType()) ? interval.durationSeconds() : 0L;
}
private static NonDrivingAccumulator open(OperatingPeriodActivityIntervalDto interval) {
return new NonDrivingAccumulator(interval);
}
private NonDrivingAccumulator extend(OperatingPeriodActivityIntervalDto interval) {
if (interval.vehicleId() != null) {
this.vehicleId = interval.vehicleId();
}
if (interval.vehicleRegistrationId() != null) {
this.vehicleRegistrationId = interval.vehicleRegistrationId();
}
if (interval.endedAt().isAfter(this.endedAt)) {
this.endedAt = interval.endedAt();
}
if ("UNKNOWN".equals(interval.activityType())) {
this.containsUnknown = true;
this.unknownSeconds += interval.durationSeconds();
}
return this;
}
private NonDrivingIntervalDto close(String closedBy) {
return new NonDrivingIntervalDto(
driverId,
vehicleId,
vehicleRegistrationId,
cardSlot,
startedAt,
endedAt,
Duration.between(startedAt, endedAt).getSeconds(),
operatingPeriodNo,
operatingPeriodStartedAt,
closedBy,
containsUnknown,
unknownSeconds,
false
);
}
}
}

View File

@ -0,0 +1,23 @@
package at.procon.eventhub.esperpoc.service;
import java.util.List;
import java.util.UUID;
public record EsperOperatingPeriodIntervalInputEvent(
UUID driverId,
UUID vehicleId,
UUID vehicleRegistrationId,
String activityType,
String cardSlot,
String cardStatus,
String drivingStatus,
String sourceKind,
long startTs,
long endTs,
long durationMs,
String sourceRowId,
List<String> sourceRowIds,
boolean clippedToRequestedPeriod,
boolean synthetic
) {
}

View File

@ -0,0 +1,102 @@
package at.procon.eventhub.esperpoc.service;
import java.time.OffsetDateTime;
import java.util.UUID;
public final class EsperRawDriverActivityPoint {
private final UUID eventId;
private final OffsetDateTime occurredAt;
private final String sourceRowId;
private final String externalSourceEventId;
private final UUID driverEntityId;
private final UUID vehicleId;
private final UUID vehicleRegistrationId;
private final String eventType;
private final String lifecycle;
private final String cardSlot;
private final String cardStatus;
private final String drivingStatus;
private final String sourceKind;
public EsperRawDriverActivityPoint(
UUID eventId,
OffsetDateTime occurredAt,
String sourceRowId,
String externalSourceEventId,
UUID driverEntityId,
UUID vehicleId,
UUID vehicleRegistrationId,
String eventType,
String lifecycle,
String cardSlot,
String cardStatus,
String drivingStatus,
String sourceKind
) {
this.eventId = eventId;
this.occurredAt = occurredAt;
this.sourceRowId = sourceRowId;
this.externalSourceEventId = externalSourceEventId;
this.driverEntityId = driverEntityId;
this.vehicleId = vehicleId;
this.vehicleRegistrationId = vehicleRegistrationId;
this.eventType = eventType;
this.lifecycle = lifecycle;
this.cardSlot = cardSlot;
this.cardStatus = cardStatus;
this.drivingStatus = drivingStatus;
this.sourceKind = sourceKind;
}
public UUID getEventId() {
return eventId;
}
public OffsetDateTime getOccurredAt() {
return occurredAt;
}
public String getSourceRowId() {
return sourceRowId;
}
public String getExternalSourceEventId() {
return externalSourceEventId;
}
public UUID getDriverEntityId() {
return driverEntityId;
}
public UUID getVehicleId() {
return vehicleId;
}
public UUID getVehicleRegistrationId() {
return vehicleRegistrationId;
}
public String getEventType() {
return eventType;
}
public String getLifecycle() {
return lifecycle;
}
public String getCardSlot() {
return cardSlot;
}
public String getCardStatus() {
return cardStatus;
}
public String getDrivingStatus() {
return drivingStatus;
}
public String getSourceKind() {
return sourceKind;
}
}

View File

@ -0,0 +1,15 @@
package at.procon.eventhub.esperpoc.service;
import java.time.Duration;
import java.time.OffsetDateTime;
record EsperResolvedShiftSpan(
OffsetDateTime startedAt,
OffsetDateTime endedAt,
OffsetDateTime nextShiftStartedAt,
long dailyRestingTimeSeconds
) {
long durationSeconds() {
return Duration.between(startedAt, endedAt).getSeconds();
}
}

View File

@ -1,16 +1,12 @@
package at.procon.eventhub.importing;
import at.procon.eventhub.config.EventHubProperties;
import at.procon.eventhub.dto.EventHubPackageRequest;
import at.procon.eventhub.dto.EventSourceDto;
import at.procon.eventhub.dto.ImportRunStatus;
import at.procon.eventhub.importing.persistence.ImportCursorRepository;
import at.procon.eventhub.importing.persistence.ImportRunRepository;
import at.procon.eventhub.persistence.DataPackageRepository.CamelBatchGroupStatus;
import at.procon.eventhub.persistence.DataPackageRepository;
import at.procon.eventhub.persistence.EventSourceRepository;
import java.time.Duration;
import java.time.Instant;
import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.List;
@ -26,32 +22,23 @@ import org.slf4j.LoggerFactory;
*/
public abstract class AbstractImportExecutionService<R extends ImportRunRequest, B extends ExtractionBatchResult> {
private static final Duration ASYNC_INGEST_AWAIT_TIMEOUT = Duration.ofHours(6);
private static final Duration ASYNC_INGEST_POLL_INTERVAL = Duration.ofSeconds(2);
private static final Duration ASYNC_INGEST_FAILURE_GRACE_PERIOD = Duration.ofSeconds(90);
private static final Duration ASYNC_INGEST_STALL_GRACE_PERIOD = Duration.ofSeconds(90);
private static final Duration ASYNC_INGEST_PROGRESS_LOG_INTERVAL = Duration.ofSeconds(30);
private final Logger log = LoggerFactory.getLogger(getClass());
private final EventSourceRepository eventSourceRepository;
private final ImportRunRepository importRunRepository;
private final DataPackageRepository dataPackageRepository;
private final ImportCursorRepository importCursorRepository;
private final EventHubProperties eventHubProperties;
protected AbstractImportExecutionService(
EventSourceRepository eventSourceRepository,
ImportRunRepository importRunRepository,
DataPackageRepository dataPackageRepository,
ImportCursorRepository importCursorRepository,
EventHubProperties eventHubProperties
ImportCursorRepository importCursorRepository
) {
this.eventSourceRepository = eventSourceRepository;
this.importRunRepository = importRunRepository;
this.dataPackageRepository = dataPackageRepository;
this.importCursorRepository = importCursorRepository;
this.eventHubProperties = eventHubProperties;
}
protected ImportRunResultDto createImportRun(R request, boolean executeImmediately) {
@ -193,7 +180,6 @@ public abstract class AbstractImportExecutionService<R extends ImportRunRequest,
"extractedEventTypeCounts", result.eventTypeCounts()
));
}
awaitAsyncIngestCompletion(importRunId, request, plannedPackage, result);
results.add(result);
dataPackageRepository.markImported(plannedPackage.packageId(), result.eventsInserted());
if (result.executed()) {
@ -221,107 +207,6 @@ public abstract class AbstractImportExecutionService<R extends ImportRunRequest,
results.stream().filter(ExtractionBatchResult::executed).count());
}
private void awaitAsyncIngestCompletion(UUID importRunId, R request, PlannedPackage plannedPackage, B result) {
int expectedCamelBatches = expectedCamelBatchCount(result.eventsInserted());
if (expectedCamelBatches <= 0) {
return;
}
EventHubPackageRequest packageInfo = extractionAggregatePackageInfo(
importRunId,
request,
eventSourceForItem(request.eventSource(), plannedPackage.planItem()),
plannedPackage.planItem(),
plannedPackage.chunk()
);
String aggregatePackageKey = aggregatePackageKey(packageInfo);
Instant deadline = Instant.now().plus(ASYNC_INGEST_AWAIT_TIMEOUT);
Instant nextProgressLogAt = Instant.now().plus(ASYNC_INGEST_PROGRESS_LOG_INTERVAL);
Instant failedStateObservedAt = null;
Instant lastStateChangeAt = Instant.now();
CamelBatchGroupStatus previousState = null;
while (Instant.now().isBefore(deadline)) {
CamelBatchGroupStatus state = dataPackageRepository.findCamelBatchGroupStatus(
plannedPackage.eventSourceId(),
request.tenantKey(),
aggregatePackageKey
);
if (state.totalCount() >= expectedCamelBatches && state.successCount() >= expectedCamelBatches) {
return;
}
boolean stateChanged = previousState == null || !previousState.equals(state);
if (stateChanged) {
lastStateChangeAt = Instant.now();
}
if (state.failedCount() > 0 && stateChanged) {
failedStateObservedAt = Instant.now();
} else if (state.failedCount() == 0) {
failedStateObservedAt = null;
}
if (state.failedCount() > 0
&& failedStateObservedAt != null
&& Instant.now().isAfter(failedStateObservedAt.plus(ASYNC_INGEST_FAILURE_GRACE_PERIOD))) {
throw new IllegalStateException(
"Async EventHub ingest failed for importRunId=" + importRunId
+ " aggregatePackageKey=" + aggregatePackageKey
+ " expectedCamelBatches=" + expectedCamelBatches
+ " observedCamelBatches=" + state.totalCount()
+ " failedCamelBatches=" + state.failedCount()
+ " failedMessage=" + state.failedMessage()
);
}
if (state.totalCount() < expectedCamelBatches
&& state.importingCount() == 0
&& state.failedCount() == 0
&& Instant.now().isAfter(lastStateChangeAt.plus(ASYNC_INGEST_STALL_GRACE_PERIOD))) {
throw new IllegalStateException(
"Async EventHub ingest stalled for importRunId=" + importRunId
+ " aggregatePackageKey=" + aggregatePackageKey
+ " expectedCamelBatches=" + expectedCamelBatches
+ " observedCamelBatches=" + state.totalCount()
+ " successfulCamelBatches=" + state.successCount()
+ " importingCamelBatches=" + state.importingCount()
+ " failedCamelBatches=" + state.failedCount()
);
}
if (Instant.now().isAfter(nextProgressLogAt)) {
log.info("Waiting for async EventHub ingest provider={} importRunId={} extractionPackageId={} aggregatePackageKey={} expectedCamelBatches={} observedCamelBatches={} successfulCamelBatches={} failedCamelBatches={} importingCamelBatches={}",
providerPackagePrefix(),
importRunId,
plannedPackage.packageId(),
aggregatePackageKey,
expectedCamelBatches,
state.totalCount(),
state.successCount(),
state.failedCount(),
state.importingCount());
nextProgressLogAt = Instant.now().plus(ASYNC_INGEST_PROGRESS_LOG_INTERVAL);
}
previousState = state;
sleepQuietly(ASYNC_INGEST_POLL_INTERVAL);
}
CamelBatchGroupStatus finalState = dataPackageRepository.findCamelBatchGroupStatus(
plannedPackage.eventSourceId(),
request.tenantKey(),
aggregatePackageKey
);
throw new IllegalStateException(
"Timed out waiting for async EventHub ingest for importRunId=" + importRunId
+ " aggregatePackageKey=" + aggregatePackageKey
+ " expectedCamelBatches=" + expectedCamelBatches
+ " observedCamelBatches=" + finalState.totalCount()
+ " successfulCamelBatches=" + finalState.successCount()
+ " failedCamelBatches=" + finalState.failedCount()
);
}
private EventHubPackageRequest packageRequestFor(
R request,
EventSourceDto itemEventSource,
@ -339,43 +224,6 @@ public abstract class AbstractImportExecutionService<R extends ImportRunRequest,
);
}
private EventHubPackageRequest extractionAggregatePackageInfo(
UUID importRunId,
R request,
EventSourceDto itemEventSource,
ImportPlanItemDto item,
ImportTimeChunkDto chunk
) {
return new EventHubPackageRequest(
request.tenantKey(),
itemEventSource,
request.sourceGroup(),
chunkScope(request.importScope(), chunk),
item.eventFamily().name(),
chunk.occurredFrom() == null ? null : chunk.occurredFrom().toLocalDate(),
providerPackagePrefix() + ":" + item.sourceKind() + ":" + item.extractionCode()
+ ":RUN-" + importRunId + ":CHUNK-" + chunk.sequence()
);
}
private int expectedCamelBatchCount(int extractedEventCount) {
if (extractedEventCount <= 0) {
return 0;
}
int completionSize = Math.max(1, eventHubProperties.getBatch().getCompletionSize());
return ((extractedEventCount - 1) / completionSize) + 1;
}
private String aggregatePackageKey(EventHubPackageRequest packageInfo) {
return packageInfo.tenantKey()
+ ":" + packageInfo.eventSource().stableKey()
+ ":" + (packageInfo.sourceGroup() == null ? "NO_GROUP" : packageInfo.sourceGroup().stableKey())
+ ":" + (packageInfo.importScope() == null ? "NO_SCOPE" : packageInfo.importScope().stableKey())
+ ":" + packageInfo.eventFamily()
+ ":" + (packageInfo.businessDate() == null ? "NO_DATE" : packageInfo.businessDate())
+ ":" + packageInfo.externalPackageId();
}
private at.procon.eventhub.dto.ImportScopeDto chunkScope(at.procon.eventhub.dto.ImportScopeDto scope, ImportTimeChunkDto chunk) {
if (scope == null) {
return at.procon.eventhub.dto.ImportScopeDto.tenantAll(chunk.occurredFrom(), chunk.occurredTo());
@ -389,15 +237,6 @@ public abstract class AbstractImportExecutionService<R extends ImportRunRequest,
);
}
private void sleepQuietly(Duration duration) {
try {
Thread.sleep(duration.toMillis());
} catch (InterruptedException ex) {
Thread.currentThread().interrupt();
throw new IllegalStateException("Interrupted while waiting for async EventHub ingest", ex);
}
}
private record PlannedPackage(UUID packageId, int eventSourceId, ImportPlanItemDto planItem, ImportTimeChunkDto chunk) {
}
}

View File

@ -1,7 +1,12 @@
package at.procon.eventhub.importing.extraction;
import at.procon.eventhub.config.EventHubProperties;
import at.procon.eventhub.dto.AcquisitionStrategy;
import at.procon.eventhub.dto.DataPackageType;
import at.procon.eventhub.dto.EventHubEventBatchDto;
import at.procon.eventhub.dto.EventHubEventDto;
import at.procon.eventhub.dto.EventHubPackageRequest;
import at.procon.eventhub.dto.EventHubPackageResult;
import at.procon.eventhub.dto.EventSourceDto;
import at.procon.eventhub.dto.ImportCursorStateDto;
import at.procon.eventhub.dto.ImportScopeDto;
@ -10,11 +15,14 @@ import at.procon.eventhub.importing.ImportPlanItemDto;
import at.procon.eventhub.importing.ImportRunRequest;
import at.procon.eventhub.importing.ImportTimeChunkDto;
import at.procon.eventhub.importing.persistence.ImportCursorRepository;
import at.procon.eventhub.service.EventHubIngestionService;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.time.OffsetDateTime;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Optional;
import java.util.UUID;
@ -33,18 +41,24 @@ public abstract class AbstractJdbcExtractionBatchExecutor<R extends ImportRunReq
private static final int EVENT_EXTRACTION_PROGRESS_LOG_INTERVAL = 5000;
private final NamedParameterJdbcTemplate jdbcTemplate;
private final EventHubIngestionService ingestionService;
private final ProducerTemplate producerTemplate;
private final EventHubProperties eventHubProperties;
private final ResourceLoader resourceLoader;
private final ImportCursorRepository importCursorRepository;
protected AbstractJdbcExtractionBatchExecutor(
NamedParameterJdbcTemplate jdbcTemplate,
EventHubIngestionService ingestionService,
ProducerTemplate producerTemplate,
EventHubProperties eventHubProperties,
ResourceLoader resourceLoader,
ImportCursorRepository importCursorRepository
) {
this.jdbcTemplate = jdbcTemplate;
this.ingestionService = ingestionService;
this.producerTemplate = producerTemplate;
this.eventHubProperties = eventHubProperties;
this.resourceLoader = resourceLoader;
this.importCursorRepository = importCursorRepository;
}
@ -87,17 +101,22 @@ public abstract class AbstractJdbcExtractionBatchExecutor<R extends ImportRunReq
Map<String, Object> params = parameters(request, chunkScope, cursor);
String sql = loadSql(definition.sqlResource());
ExtractedEventStats stats = new ExtractedEventStats();
log.info("Reading EventHub events provider={} tenant={} importRunId={} packageId={} extractionCode={} sourceKind={} chunk={} occurredFrom={} occurredTo={}",
List<EventHubEventDto> pendingEvents = new ArrayList<>(jdbcPersistBatchSize());
log.info("Reading EventHub events provider={} tenant={} importRunId={} packageId={} extractionCode={} sourceKind={} chunk={} occurredFrom={} occurredTo={} ingestMode={}",
providerPackagePrefix(), request.tenantKey(), importRunId, packageId, planItem.extractionCode(), planItem.sourceKind(),
chunk.sequence(), chunk.occurredFrom(), chunk.occurredTo());
chunk.sequence(), chunk.occurredFrom(), chunk.occurredTo(), jdbcExtractionIngestMode());
jdbcTemplate.query(sql, params, rs -> {
EventHubEventDto event = definition.rowMapper().map(rs, stats.eventsMapped(), context);
producerTemplate.sendBody(normalizedInputUri(), event);
pendingEvents.add(event);
stats.accept(event);
if (pendingEvents.size() >= jdbcPersistBatchSize()) {
flushPersistBatch(request, importRunId, packageId, planItem, chunk, packageInfo, pendingEvents, stats);
}
if (stats.eventsMapped() % EVENT_EXTRACTION_PROGRESS_LOG_INTERVAL == 0) {
logEventExtractionProgress(request, importRunId, packageId, planItem, chunk, stats);
}
});
flushPersistBatch(request, importRunId, packageId, planItem, chunk, packageInfo, pendingEvents, stats);
logEventExtractionFinished(request, importRunId, packageId, planItem, chunk, stats);
return resultFor(packageId, planItem, chunk, cursor, stats);
@ -119,8 +138,111 @@ public abstract class AbstractJdbcExtractionBatchExecutor<R extends ImportRunReq
return "SOURCE";
}
protected String normalizedInputUri() {
return "direct:eventhub-normalized-input";
protected int jdbcPersistBatchSize() {
return Math.max(1, eventHubProperties.getBatch().getCompletionSize());
}
protected EventHubProperties.JdbcExtractionIngestMode jdbcExtractionIngestMode() {
return eventHubProperties.getTachograph().getJdbcExtractionIngestMode();
}
protected int eventsInsertedOrSubmitted(ExtractedEventStats stats) {
return stats.eventsInserted();
}
protected String camelBatchPersistUri() {
return "direct:eventhub-batch-persist-input";
}
private void flushPersistBatch(
R request,
UUID importRunId,
UUID extractionPackageId,
ImportPlanItemDto planItem,
ImportTimeChunkDto chunk,
EventHubPackageRequest packageInfo,
List<EventHubEventDto> pendingEvents,
ExtractedEventStats stats
) {
if (pendingEvents.isEmpty()) {
return;
}
int batchNo = stats.nextPersistBatchNo();
List<EventHubEventDto> eventsToPersist = List.copyOf(pendingEvents);
pendingEvents.clear();
EventHubEventBatchDto batch = new EventHubEventBatchDto(
packageInfo.externalPackageId() + ":JDBC-" + batchNo,
packageInfo,
DataPackageType.DB_EXTRACT,
occurredFrom(eventsToPersist, chunk),
occurredTo(eventsToPersist, chunk),
eventsToPersist,
persistBatchMetadata(request, importRunId, extractionPackageId, planItem, chunk, batchNo, eventsToPersist)
);
EventHubPackageResult result = persistBatch(batch);
stats.acceptPersistResult(result);
log.info("Persisted EventHub extraction batch provider={} tenant={} importRunId={} extractionPackageId={} extractionCode={} sourceKind={} chunk={} batchNo={} received={} inserted={}",
providerPackagePrefix(), request.tenantKey(), importRunId, extractionPackageId, planItem.extractionCode(), planItem.sourceKind(),
chunk.sequence(), batchNo, result.receivedCount(), result.insertedCount());
}
private EventHubPackageResult persistBatch(EventHubEventBatchDto batch) {
if (jdbcExtractionIngestMode() == EventHubProperties.JdbcExtractionIngestMode.CAMEL_ROUTE) {
return producerTemplate.requestBody(camelBatchPersistUri(), batch, EventHubPackageResult.class);
}
return ingestionService.ingest(batch);
}
private Map<String, Object> persistBatchMetadata(
R request,
UUID importRunId,
UUID extractionPackageId,
ImportPlanItemDto planItem,
ImportTimeChunkDto chunk,
int batchNo,
List<EventHubEventDto> events
) {
Map<String, Object> metadata = new LinkedHashMap<>();
metadata.put("ingestMode", jdbcExtractionIngestMode().name());
metadata.put("importRunId", importRunId.toString());
metadata.put("extractionPackageId", extractionPackageId.toString());
metadata.put("tenantKey", request.tenantKey());
metadata.put("mode", request.mode().name());
metadata.put("acquisitionStrategy", request.acquisitionStrategy().name());
metadata.put("eventFamily", planItem.eventFamily().name());
metadata.put("sourceKind", planItem.sourceKind());
metadata.put("extractionCode", planItem.extractionCode());
metadata.put("entityAxis", planItem.entityAxis());
metadata.put("chunkSequence", chunk.sequence());
metadata.put("chunkOccurredFrom", chunk.occurredFrom() == null ? null : chunk.occurredFrom().toString());
metadata.put("chunkOccurredTo", chunk.occurredTo() == null ? null : chunk.occurredTo().toString());
metadata.put("batchNo", batchNo);
metadata.put("receivedEventCount", events.size());
metadata.put("sourcePackageRefPolicy", "Original source package is preserved per acquired event.");
return metadata;
}
private OffsetDateTime occurredFrom(List<EventHubEventDto> events, ImportTimeChunkDto chunk) {
if (chunk.occurredFrom() != null) {
return chunk.occurredFrom();
}
return events.stream()
.map(EventHubEventDto::occurredAt)
.filter(value -> value != null)
.min(OffsetDateTime::compareTo)
.orElse(null);
}
private OffsetDateTime occurredTo(List<EventHubEventDto> events, ImportTimeChunkDto chunk) {
if (chunk.occurredTo() != null) {
return chunk.occurredTo();
}
return events.stream()
.map(EventHubEventDto::occurredAt)
.filter(value -> value != null)
.max(OffsetDateTime::compareTo)
.orElse(null);
}
private void logEventExtractionProgress(
@ -159,6 +281,7 @@ public abstract class AbstractJdbcExtractionBatchExecutor<R extends ImportRunReq
params.put("occurredTo", scope == null ? null : scope.occurredTo());
params.put("organisationId", organisationId);
params.put("includeChildren", scope != null && scope.includeChildren());
params.put("sourcePackageWatermarkEnabled", request.acquisitionStrategy() == AcquisitionStrategy.SOURCE_PACKAGE_WATERMARK ? 1 : 0);
params.put("lastSourcePackageImportedAt", cursor == null ? null : cursor.lastSourcePackageImportedAt());
params.put("lastSourcePackageId", cursor == null ? null : cursor.lastSourcePackageId());
params.put("lastSourcePackageIdNumeric", parseLong(cursor == null ? null : cursor.lastSourcePackageId()));
@ -277,10 +400,28 @@ public abstract class AbstractJdbcExtractionBatchExecutor<R extends ImportRunReq
return maxSourcePackageId;
}
private int eventsInserted;
private int persistBatchNo;
public int eventsInserted() {
return eventsInserted;
}
public Map<String, Integer> eventTypeCounts() {
return eventTypeCounts;
}
private int nextPersistBatchNo() {
persistBatchNo++;
return persistBatchNo;
}
private void acceptPersistResult(EventHubPackageResult result) {
if (result != null) {
eventsInserted += result.insertedCount();
}
}
private void accept(EventHubEventDto event) {
eventsMapped++;
eventTypeCounts.merge(eventTypeKey(event), 1, Integer::sum);
@ -290,10 +431,17 @@ public abstract class AbstractJdbcExtractionBatchExecutor<R extends ImportRunReq
OffsetDateTime importedAt = event.sourcePackageRef().importedIntoSourceAt();
String sourcePackageId = event.sourcePackageRef().sourcePackageId();
if (importedAt != null
&& (lastSourcePackageImportedAt == null || importedAt.compareTo(lastSourcePackageImportedAt) > 0)) {
lastSourcePackageImportedAt = importedAt;
lastSourcePackageIdByImportedAt = sourcePackageId;
if (importedAt != null) {
if (lastSourcePackageImportedAt == null || importedAt.compareTo(lastSourcePackageImportedAt) > 0) {
lastSourcePackageImportedAt = importedAt;
lastSourcePackageIdByImportedAt = sourcePackageId;
} else if (importedAt.compareTo(lastSourcePackageImportedAt) == 0
&& sourcePackageId != null
&& !sourcePackageId.isBlank()
&& (lastSourcePackageIdByImportedAt == null
|| compareSourcePackageId(sourcePackageId, lastSourcePackageIdByImportedAt) > 0)) {
lastSourcePackageIdByImportedAt = sourcePackageId;
}
}
if (sourcePackageId != null
&& !sourcePackageId.isBlank()

View File

@ -48,6 +48,13 @@ public class DataPackageRepository {
SourceGroupRefDto sourceGroup = packageInfo == null ? null : packageInfo.sourceGroup();
ImportScopeDto importScope = packageInfo == null ? null : packageInfo.importScope();
SourceGroupRefDto rootOrg = importScope == null ? null : importScope.rootSourceOrganisation();
UUID importRunId = metadataUuid(metadata, "importRunId");
String extractionCode = metadataString(metadata, "extractionCode");
String extractionSourceKind = metadataString(metadata, "sourceKind");
String entityAxis = metadataString(metadata, "entityAxis");
Integer batchNo = metadataInteger(metadata, "batchNo");
OffsetDateTime chunkFrom = metadataOffsetDateTime(metadata, "chunkOccurredFrom");
OffsetDateTime chunkTo = metadataOffsetDateTime(metadata, "chunkOccurredTo");
return jdbcTemplate.query(
con -> {
@ -64,9 +71,16 @@ public class DataPackageRepository {
received_at, event_count, metadata
) values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, now(), ?, ?::jsonb)
on conflict (tenant_key, event_source_id, package_key) do update
set status = excluded.status,
set import_run_id = excluded.import_run_id,
status = excluded.status,
occurred_from = excluded.occurred_from,
occurred_to = excluded.occurred_to,
extraction_code = excluded.extraction_code,
extraction_source_kind = excluded.extraction_source_kind,
entity_axis = excluded.entity_axis,
batch_no = excluded.batch_no,
chunk_from = excluded.chunk_from,
chunk_to = excluded.chunk_to,
event_count = excluded.event_count,
metadata = excluded.metadata,
error_message = null,
@ -75,7 +89,7 @@ public class DataPackageRepository {
""");
ps.setObject(1, id);
ps.setInt(2, eventSourceId);
ps.setObject(3, null);
ps.setObject(3, importRunId);
ps.setString(4, packageInfo == null ? "default" : packageInfo.tenantKey());
ps.setString(5, packageKey);
ps.setString(6, packageType.name());
@ -94,12 +108,12 @@ public class DataPackageRepository {
ps.setString(19, packageInfo == null ? null : packageInfo.eventFamily());
ps.setObject(20, packageInfo == null ? null : packageInfo.businessDate());
ps.setString(21, packageInfo == null ? packageKey : packageInfo.externalPackageId());
ps.setString(22, null);
ps.setString(23, null);
ps.setString(24, null);
ps.setObject(25, null);
ps.setObject(26, null);
ps.setObject(27, null);
ps.setString(22, extractionCode);
ps.setString(23, extractionSourceKind);
ps.setString(24, entityAxis);
ps.setObject(25, batchNo);
ps.setObject(26, chunkFrom);
ps.setObject(27, chunkTo);
ps.setString(28, null);
ps.setString(29, null);
ps.setString(30, null);
@ -343,6 +357,61 @@ public class DataPackageRepository {
);
}
private UUID metadataUuid(Map<String, Object> metadata, String key) {
String value = metadataString(metadata, key);
if (value == null) {
return null;
}
try {
return UUID.fromString(value);
} catch (IllegalArgumentException ignored) {
return null;
}
}
private String metadataString(Map<String, Object> metadata, String key) {
if (metadata == null) {
return null;
}
Object value = metadata.get(key);
if (value == null) {
return null;
}
String text = value.toString().trim();
return text.isEmpty() ? null : text;
}
private Integer metadataInteger(Map<String, Object> metadata, String key) {
String value = metadataString(metadata, key);
if (value == null) {
return null;
}
try {
return Integer.parseInt(value);
} catch (NumberFormatException ignored) {
return null;
}
}
private OffsetDateTime metadataOffsetDateTime(Map<String, Object> metadata, String key) {
Object value = metadata == null ? null : metadata.get(key);
if (value == null) {
return null;
}
if (value instanceof OffsetDateTime offsetDateTime) {
return offsetDateTime;
}
String text = value.toString().trim();
if (text.isEmpty()) {
return null;
}
try {
return OffsetDateTime.parse(text);
} catch (RuntimeException ignored) {
return null;
}
}
private String toJson(Map<String, Object> value) {
try {
return objectMapper.writeValueAsString(normalizeMetadataMap(value));

View File

@ -0,0 +1,372 @@
package at.procon.eventhub.persistence;
import at.procon.eventhub.dto.DriverCardRefDto;
import at.procon.eventhub.dto.DriverRefDto;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.ObjectMapper;
import java.time.OffsetDateTime;
import java.util.LinkedHashMap;
import java.util.Map;
import java.util.UUID;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.stereotype.Repository;
import org.springframework.transaction.annotation.Transactional;
@Repository
public class DriverIdentityRepository {
private final JdbcTemplate jdbcTemplate;
private final ObjectMapper objectMapper;
public DriverIdentityRepository(JdbcTemplate jdbcTemplate, ObjectMapper objectMapper) {
this.jdbcTemplate = jdbcTemplate;
this.objectMapper = objectMapper;
}
public UUID resolveOrCreateDriverId(
String tenantKey,
int eventSourceId,
DriverRefDto driverRef
) {
if (driverRef == null || !driverRef.hasAnyReference()) {
return null;
}
String normalizedTenantKey = normalizeRequired(tenantKey, "tenantKey");
String sourceDriverEntityId = normalizeNullable(driverRef.sourceEntityId());
DriverCardRefDto driverCard = driverRef.driverCard();
String cardNation = driverCard == null ? null : normalizeNullable(driverCard.nation());
String cardNumber = driverCard == null ? null : normalizeNullable(driverCard.number());
UUID driverId = resolveDriverId(normalizedTenantKey, eventSourceId, sourceDriverEntityId, cardNation, cardNumber);
if (driverId == null) {
Map<String, Object> payload = new LinkedHashMap<>();
put(payload, "source", "event");
put(payload, "source_driver_entity_id", sourceDriverEntityId);
put(payload, "card_nation", cardNation);
put(payload, "card_number", cardNumber);
driverId = createDriver(
normalizedTenantKey,
eventSourceId,
sourceDriverEntityId,
cardNation,
cardNumber,
null,
null,
null,
payload
);
}
touchDriver(driverId, sourceDriverEntityId, cardNation, cardNumber);
return driverId;
}
@Transactional
public int reconcileFromMasterData(String tenantKey, int eventSourceId) {
String normalizedTenantKey = normalizeRequired(tenantKey, "tenantKey");
int updates = reconcileDriversFromMasterData(normalizedTenantKey, eventSourceId);
updates += projectDriverCardsFromMasterData(normalizedTenantKey, eventSourceId);
return updates;
}
private int reconcileDriversFromMasterData(String tenantKey, int eventSourceId) {
Long count = jdbcTemplate.queryForObject(
compatibleSourcesCte() + """
, master_drivers as (
select distinct on (nullif(trim(source_entity_id), ''))
event_source_id,
nullif(trim(source_entity_id), '') as source_driver_entity_id,
nullif(trim(payload ->> 'first_names'), '') as first_names,
coalesce(
nullif(trim(payload ->> 'last_name'), ''),
nullif(trim(payload ->> 'surname'), '')
) as last_name,
cast(nullif(trim(payload ->> 'birth_date'), '') as date) as birth_date,
source_updated_at,
payload
from eventhub.source_master_entity
where tenant_key = ?
and event_source_id in (select id from compatible_sources)
and entity_type = 'DRIVER'
and nullif(trim(source_entity_id), '') is not null
and source_entity_id not like 'DRIVER_CARD:%'
order by nullif(trim(source_entity_id), ''), updated_at desc
),
updated_by_source as (
update eventhub.driver driver
set first_names = coalesce(master.first_names, driver.first_names),
last_name = coalesce(master.last_name, driver.last_name),
birth_date = coalesce(master.birth_date, driver.birth_date),
source_updated_at = master.source_updated_at,
payload = driver.payload || master.payload,
updated_at = now()
from master_drivers master
where driver.tenant_key = ?
and driver.event_source_id in (select id from compatible_sources)
and driver.source_driver_entity_id = master.source_driver_entity_id
returning driver.id
),
inserted as (
insert into eventhub.driver(
id, tenant_key, event_source_id, source_driver_entity_id,
first_names, last_name, birth_date, source_updated_at, payload, updated_at
)
select gen_random_uuid(),
?,
master.event_source_id,
master.source_driver_entity_id,
master.first_names,
master.last_name,
master.birth_date,
master.source_updated_at,
master.payload,
now()
from master_drivers master
where not exists (
select 1
from eventhub.driver existing
where existing.tenant_key = ?
and existing.event_source_id in (select id from compatible_sources)
and existing.source_driver_entity_id = master.source_driver_entity_id
)
returning id
)
select (select count(*) from updated_by_source)
+ (select count(*) from inserted)
""",
Long.class,
eventSourceId,
tenantKey,
tenantKey,
tenantKey,
tenantKey
);
return count == null ? 0 : Math.toIntExact(count);
}
private int projectDriverCardsFromMasterData(String tenantKey, int eventSourceId) {
Long count = jdbcTemplate.queryForObject(
compatibleSourcesCte() + """
, driver_card_projection as (
select distinct on (rel.to_source_entity_id)
rel.to_source_entity_id as source_driver_entity_id,
nullif(trim(card.payload ->> 'card_nation'), '') as card_nation,
nullif(trim(card.payload ->> 'card_number'), '') as card_number,
rel.source_updated_at
from eventhub.source_master_relation rel
join eventhub.source_master_entity card
on card.tenant_key = rel.tenant_key
and card.event_source_id = rel.event_source_id
and card.entity_type = 'DRIVER_CARD'
and card.source_entity_id = rel.from_source_entity_id
where rel.tenant_key = ?
and rel.event_source_id in (select id from compatible_sources)
and rel.relation_type = 'DRIVER_CARD_DRIVER'
and rel.from_entity_type = 'DRIVER_CARD'
and rel.to_entity_type = 'DRIVER'
order by rel.to_source_entity_id,
rel.valid_to desc nulls last,
rel.valid_from desc nulls last,
rel.updated_at desc
),
updated_by_source as (
update eventhub.driver driver
set card_nation = coalesce(driver.card_nation, projection.card_nation),
card_number = coalesce(driver.card_number, projection.card_number),
source_updated_at = coalesce(projection.source_updated_at, driver.source_updated_at),
updated_at = now()
from driver_card_projection projection
where driver.tenant_key = ?
and driver.event_source_id in (select id from compatible_sources)
and driver.source_driver_entity_id = projection.source_driver_entity_id
and (
(driver.card_nation is null and projection.card_nation is not null)
or (driver.card_number is null and projection.card_number is not null)
)
returning driver.id
)
select count(*)
from updated_by_source
""",
Long.class,
eventSourceId,
tenantKey,
tenantKey
);
return count == null ? 0 : Math.toIntExact(count);
}
private UUID resolveDriverId(
String tenantKey,
int eventSourceId,
String sourceDriverEntityId,
String cardNation,
String cardNumber
) {
UUID driverId = findBySourceDriverEntityId(tenantKey, eventSourceId, sourceDriverEntityId);
if (driverId == null) {
driverId = findByCard(tenantKey, eventSourceId, cardNation, cardNumber);
}
return driverId;
}
private UUID findBySourceDriverEntityId(String tenantKey, int eventSourceId, String sourceDriverEntityId) {
if (sourceDriverEntityId == null) {
return null;
}
return jdbcTemplate.query(
compatibleSourcesCte() + """
select d.id
from eventhub.driver d
where d.tenant_key = ?
and d.event_source_id in (select id from compatible_sources)
and d.source_driver_entity_id = ?
order by d.updated_at desc
limit 1
""",
rs -> rs.next() ? (UUID) rs.getObject("id") : null,
eventSourceId,
tenantKey,
sourceDriverEntityId
);
}
private UUID findByCard(String tenantKey, int eventSourceId, String cardNation, String cardNumber) {
if (cardNation == null || cardNumber == null) {
return null;
}
return jdbcTemplate.query(
compatibleSourcesCte() + """
select d.id
from eventhub.driver d
where d.tenant_key = ?
and d.event_source_id in (select id from compatible_sources)
and d.card_nation = ?
and d.card_number = ?
order by d.updated_at desc
limit 1
""",
rs -> rs.next() ? (UUID) rs.getObject("id") : null,
eventSourceId,
tenantKey,
cardNation,
cardNumber
);
}
private UUID createDriver(
String tenantKey,
int eventSourceId,
String sourceDriverEntityId,
String cardNation,
String cardNumber,
String firstNames,
String lastName,
OffsetDateTime sourceUpdatedAt,
Map<String, Object> payload
) {
UUID driverId = UUID.randomUUID();
jdbcTemplate.update(
"""
insert into eventhub.driver(
id, tenant_key, event_source_id, source_driver_entity_id,
card_nation, card_number, first_names, last_name,
source_updated_at, payload, updated_at
) values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?::jsonb, now())
""",
driverId,
tenantKey,
eventSourceId,
sourceDriverEntityId,
cardNation,
cardNumber,
firstNames,
lastName,
sourceUpdatedAt,
toJson(payload)
);
return driverId;
}
private void touchDriver(
UUID driverId,
String sourceDriverEntityId,
String cardNation,
String cardNumber
) {
if (sourceDriverEntityId == null && cardNation == null && cardNumber == null) {
return;
}
jdbcTemplate.update(
"""
update eventhub.driver
set source_driver_entity_id = coalesce(source_driver_entity_id, cast(? as text)),
card_nation = coalesce(card_nation, cast(? as text)),
card_number = coalesce(card_number, cast(? as text)),
updated_at = now()
where id = ?
and (
(source_driver_entity_id is null and cast(? as text) is not null)
or (card_nation is null and cast(? as text) is not null)
or (card_number is null and cast(? as text) is not null)
)
""",
sourceDriverEntityId,
cardNation,
cardNumber,
driverId,
sourceDriverEntityId,
cardNation,
cardNumber
);
}
private String compatibleSourcesCte() {
return """
with source_context as (
select tenant_key, provider_key, source_instance_key, coalesce(tenant_provider_setting_key, '') as tenant_provider_setting_key
from eventhub.event_source
where id = ?
),
compatible_sources as (
select es.id
from eventhub.event_source es
join source_context ctx on ctx.tenant_key = es.tenant_key
and ctx.provider_key = es.provider_key
and ctx.source_instance_key = es.source_instance_key
and ctx.tenant_provider_setting_key = coalesce(es.tenant_provider_setting_key, '')
)
""";
}
private String toJson(Map<String, Object> value) {
try {
return objectMapper.writeValueAsString(value == null ? Map.of() : value);
} catch (JsonProcessingException e) {
throw new IllegalArgumentException("Cannot serialize driver identity payload", e);
}
}
private void put(Map<String, Object> payload, String key, Object value) {
if (value != null) {
payload.put(key, value);
}
}
private String normalizeRequired(String value, String fieldName) {
String normalized = normalizeNullable(value);
if (normalized == null) {
throw new IllegalArgumentException(fieldName + " must not be blank");
}
return normalized;
}
private String normalizeNullable(String value) {
if (value == null) {
return null;
}
String trimmed = value.trim();
return trimmed.isEmpty() ? null : trimmed;
}
}

View File

@ -32,6 +32,7 @@ public class EventRepository {
private final ObjectMapper objectMapper;
private final EventAcquisitionRecordKeyService recordKeyService;
private final SourceMasterDataRepository sourceMasterDataRepository;
private final DriverIdentityRepository driverIdentityRepository;
private final VehicleIdentityRepository vehicleIdentityRepository;
public EventRepository(
@ -39,12 +40,14 @@ public class EventRepository {
ObjectMapper objectMapper,
EventAcquisitionRecordKeyService recordKeyService,
SourceMasterDataRepository sourceMasterDataRepository,
DriverIdentityRepository driverIdentityRepository,
VehicleIdentityRepository vehicleIdentityRepository
) {
this.jdbcTemplate = jdbcTemplate;
this.objectMapper = objectMapper;
this.recordKeyService = recordKeyService;
this.sourceMasterDataRepository = sourceMasterDataRepository;
this.driverIdentityRepository = driverIdentityRepository;
this.vehicleIdentityRepository = vehicleIdentityRepository;
}
@ -89,7 +92,7 @@ public class EventRepository {
packageId,
eventSourceId,
event.externalSourceEventId(),
refs.driverEntityId(),
refs.driverId(),
refs.vehicleId(),
refs.vehicleRegistrationId(),
sourcePackageId,
@ -121,7 +124,7 @@ public class EventRepository {
data_package_id uuid not null,
event_source_id integer not null,
external_source_event_id text not null,
driver_entity_id uuid,
driver_id uuid,
vehicle_id uuid,
vehicle_registration_id uuid,
source_package_id text,
@ -149,7 +152,7 @@ public class EventRepository {
"""
insert into eventhub_event_import_stage(
row_no, source_record_key_hash, requested_event_id, data_package_id, event_source_id,
external_source_event_id, driver_entity_id, vehicle_id, vehicle_registration_id,
external_source_event_id, driver_id, vehicle_id, vehicle_registration_id,
source_package_id, source_package_entity_id, occurred_at, received_partner_at, received_hub_at,
event_domain, event_type, lifecycle, odometer_m, longitude, latitude,
payload, manual_entry, event_signature_hash
@ -165,7 +168,7 @@ public class EventRepository {
ps.setObject(4, row.packageId());
ps.setInt(5, row.eventSourceId());
ps.setString(6, row.externalSourceEventId());
ps.setObject(7, row.driverEntityId());
ps.setObject(7, row.driverId());
ps.setObject(8, row.vehicleId());
ps.setObject(9, row.vehicleRegistrationId());
ps.setString(10, row.sourcePackageId());
@ -233,7 +236,7 @@ public class EventRepository {
insert into eventhub.event(
id, event_source_id, data_package_id,
external_source_event_id,
driver_entity_id, vehicle_id, vehicle_registration_id,
driver_id, vehicle_id, vehicle_registration_id,
source_package_id, source_package_entity_id,
occurred_at, received_partner_at, received_hub_at,
event_domain, event_type, lifecycle,
@ -244,7 +247,7 @@ public class EventRepository {
select
source_record.event_id, stage.event_source_id, stage.data_package_id,
stage.external_source_event_id,
stage.driver_entity_id, stage.vehicle_id, stage.vehicle_registration_id,
stage.driver_id, stage.vehicle_id, stage.vehicle_registration_id,
stage.source_package_id, stage.source_package_entity_id,
source_record.event_occurred_at, stage.received_partner_at, stage.received_hub_at,
stage.event_domain, stage.event_type, stage.lifecycle,
@ -357,13 +360,13 @@ public class EventRepository {
Map<String, UUID> entityIdCache,
Map<String, List<VehicleRefCacheEntry>> vehicleRefCache
) {
UUID driverEntityId = resolveDriverEntityId(tenantKey, eventSourceId, event, entityIdCache);
UUID driverId = resolveDriverId(tenantKey, eventSourceId, event, entityIdCache);
ResolvedVehicleReference vehicleRef = resolveVehicleReference(tenantKey, eventSourceId, event, vehicleRefCache);
UUID sourcePackageEntityId = resolveSourcePackageEntityId(tenantKey, eventSourceId, event, entityIdCache);
return new ResolvedEntityRefs(driverEntityId, vehicleRef.vehicleId(), vehicleRef.vehicleRegistrationId(), sourcePackageEntityId);
return new ResolvedEntityRefs(driverId, vehicleRef.vehicleId(), vehicleRef.vehicleRegistrationId(), sourcePackageEntityId);
}
private UUID resolveDriverEntityId(
private UUID resolveDriverId(
String tenantKey,
int eventSourceId,
EventHubEventDto event,
@ -373,32 +376,16 @@ public class EventRepository {
if (driverRef == null || !driverRef.hasAnyReference()) {
return null;
}
DriverCardRefDto card = driverRef.driverCard();
String cardKey = card == null || !card.hasValue() ? null : card.stableKey();
String sourceEntityId = normalizeNullable(driverRef.sourceEntityId());
if (sourceEntityId == null && cardKey != null) {
sourceEntityId = "DRIVER_CARD:" + cardKey;
String cacheKey = "DRIVER|" + driverRef.stableKey();
UUID cached = entityIdCache.get(cacheKey);
if (cached != null) {
return cached;
}
if (sourceEntityId == null) {
return null;
UUID resolved = driverIdentityRepository.resolveOrCreateDriverId(tenantKey, eventSourceId, driverRef);
if (resolved != null) {
entityIdCache.put(cacheKey, resolved);
}
Map<String, Object> payload = new LinkedHashMap<>();
put(payload, "source_entity_id", driverRef.sourceEntityId());
put(payload, "driver_card_nation", card == null ? null : card.nation());
put(payload, "driver_card_number", card == null ? null : card.number());
return resolveEntityId(
tenantKey,
eventSourceId,
"DRIVER",
sourceEntityId,
cardKey,
sourceEntityId,
null,
payload,
entityIdCache
);
return resolved;
}
private ResolvedVehicleReference resolveVehicleReference(
@ -599,7 +586,7 @@ public class EventRepository {
}
private record ResolvedEntityRefs(
UUID driverEntityId,
UUID driverId,
UUID vehicleId,
UUID vehicleRegistrationId,
UUID sourcePackageEntityId
@ -618,7 +605,7 @@ public class EventRepository {
UUID packageId,
int eventSourceId,
String externalSourceEventId,
UUID driverEntityId,
UUID driverId,
UUID vehicleId,
UUID vehicleRegistrationId,
String sourcePackageId,

View File

@ -10,7 +10,7 @@ import org.springframework.jdbc.core.namedparam.NamedParameterJdbcTemplate;
import org.springframework.jdbc.datasource.DriverManagerDataSource;
@Configuration
@ConditionalOnExpression("'${eventhub.tachograph.datasource.jdbc-url:}' != ''")
@ConditionalOnExpression("T(org.springframework.util.StringUtils).hasText('${eventhub.tachograph.datasource.jdbc-url:}')")
public class TachographDataSourceConfig {
private static final String SQL_SERVER_DRIVER_CLASS = "com.microsoft.sqlserver.jdbc.SQLServerDriver";

View File

@ -1,5 +1,6 @@
package at.procon.eventhub.tachograph.service;
import at.procon.eventhub.config.EventHubProperties;
import at.procon.eventhub.dto.EventSourceDto;
import at.procon.eventhub.dto.ImportCursorStateDto;
import at.procon.eventhub.dto.ImportScopeDto;
@ -8,6 +9,7 @@ import at.procon.eventhub.importing.ImportTimeChunkDto;
import at.procon.eventhub.importing.extraction.AbstractJdbcExtractionBatchExecutor;
import at.procon.eventhub.importing.extraction.ExtractionDefinition;
import at.procon.eventhub.importing.persistence.ImportCursorRepository;
import at.procon.eventhub.service.EventHubIngestionService;
import at.procon.eventhub.tachograph.dto.TachographExtractionBatchResultDto;
import at.procon.eventhub.tachograph.dto.TachographImportRequest;
import java.time.LocalDateTime;
@ -26,7 +28,7 @@ import org.springframework.stereotype.Service;
@Service
@ConditionalOnBean(name = "tachographNamedParameterJdbcTemplate")
@ConditionalOnExpression("'${eventhub.tachograph.datasource.jdbc-url:}' != ''")
@ConditionalOnExpression("T(org.springframework.util.StringUtils).hasText('${eventhub.tachograph.datasource.jdbc-url:}')")
public class JdbcTachographExtractionBatchExecutor
extends AbstractJdbcExtractionBatchExecutor<TachographImportRequest, TachographExtractionBatchResultDto>
implements TachographExtractionBatchExecutor {
@ -35,12 +37,14 @@ public class JdbcTachographExtractionBatchExecutor
public JdbcTachographExtractionBatchExecutor(
@Qualifier("tachographNamedParameterJdbcTemplate") NamedParameterJdbcTemplate tachographJdbcTemplate,
EventHubIngestionService ingestionService,
ProducerTemplate producerTemplate,
EventHubProperties eventHubProperties,
ResourceLoader resourceLoader,
TachographExtractionDefinitionRegistry definitionRegistry,
ImportCursorRepository importCursorRepository
) {
super(tachographJdbcTemplate, producerTemplate, resourceLoader, importCursorRepository);
super(tachographJdbcTemplate, ingestionService, producerTemplate, eventHubProperties, resourceLoader, importCursorRepository);
this.definitionRegistry = definitionRegistry;
}
@ -78,7 +82,7 @@ public class JdbcTachographExtractionBatchExecutor
planItem.sourceKind(),
stats.eventsMapped(),
stats.eventsMapped(),
stats.eventsMapped(),
eventsInsertedOrSubmitted(stats),
0,
true,
lastSourcePackageImportedAt(stats, cursor),

View File

@ -19,7 +19,7 @@ import org.springframework.stereotype.Service;
*/
@Service
@ConditionalOnMissingBean(TachographExtractionBatchExecutor.class)
@ConditionalOnExpression("'${eventhub.tachograph.datasource.jdbc-url:}' == ''")
@ConditionalOnExpression("!T(org.springframework.util.StringUtils).hasText('${eventhub.tachograph.datasource.jdbc-url:}')")
public class NoopTachographExtractionBatchExecutor
extends AbstractNoopExtractionBatchExecutor<TachographImportRequest, TachographExtractionBatchResultDto>
implements TachographExtractionBatchExecutor {

View File

@ -1,6 +1,5 @@
package at.procon.eventhub.tachograph.service;
import at.procon.eventhub.config.EventHubProperties;
import at.procon.eventhub.dto.EventSourceDto;
import at.procon.eventhub.importing.AbstractImportExecutionService;
import at.procon.eventhub.importing.ImportPlanDto;
@ -33,11 +32,10 @@ public class TachographImportExecutionService
ImportRunRepository importRunRepository,
DataPackageRepository dataPackageRepository,
ImportCursorRepository importCursorRepository,
EventHubProperties eventHubProperties,
TachographMasterDataRefreshService masterDataRefreshService,
TachographExtractionBatchExecutor extractionBatchExecutor
) {
super(eventSourceRepository, importRunRepository, dataPackageRepository, importCursorRepository, eventHubProperties);
super(eventSourceRepository, importRunRepository, dataPackageRepository, importCursorRepository);
this.planService = planService;
this.masterDataRefreshService = masterDataRefreshService;
this.extractionBatchExecutor = extractionBatchExecutor;

View File

@ -6,6 +6,7 @@ import at.procon.eventhub.importing.masterdata.SourceMasterRelationUpsert;
import at.procon.eventhub.dto.EventSourceDto;
import at.procon.eventhub.persistence.EventSourceRepository;
import at.procon.eventhub.persistence.SourceMasterDataRepository;
import at.procon.eventhub.persistence.DriverIdentityRepository;
import at.procon.eventhub.persistence.VehicleIdentityRepository;
import at.procon.eventhub.tachograph.dto.TachographImportRequest;
import java.io.IOException;
@ -52,6 +53,7 @@ public class TachographMasterDataRefreshService {
private final ObjectProvider<NamedParameterJdbcTemplate> tachographJdbcTemplateProvider;
private final SourceMasterDataRepository sourceMasterDataRepository;
private final EventSourceRepository eventSourceRepository;
private final DriverIdentityRepository driverIdentityRepository;
private final VehicleIdentityRepository vehicleIdentityRepository;
private final ResourceLoader resourceLoader;
@ -59,12 +61,14 @@ public class TachographMasterDataRefreshService {
@Qualifier("tachographNamedParameterJdbcTemplate") ObjectProvider<NamedParameterJdbcTemplate> tachographJdbcTemplateProvider,
SourceMasterDataRepository sourceMasterDataRepository,
EventSourceRepository eventSourceRepository,
DriverIdentityRepository driverIdentityRepository,
VehicleIdentityRepository vehicleIdentityRepository,
ResourceLoader resourceLoader
) {
this.tachographJdbcTemplateProvider = tachographJdbcTemplateProvider;
this.sourceMasterDataRepository = sourceMasterDataRepository;
this.eventSourceRepository = eventSourceRepository;
this.driverIdentityRepository = driverIdentityRepository;
this.vehicleIdentityRepository = vehicleIdentityRepository;
this.resourceLoader = resourceLoader;
}
@ -98,13 +102,17 @@ public class TachographMasterDataRefreshService {
int relationCount = streamRelations(tachographJdbcTemplate, tenantKey, eventSourceId, RELATIONS_SQL_RESOURCE, loadSql(RELATIONS_SQL_RESOURCE));
log.info("Reconciling tachograph driver identities from source master data tenant={} source={}",
tenantKey, masterDataSource.stableKey());
int reconciledDrivers = driverIdentityRepository.reconcileFromMasterData(tenantKey, eventSourceId);
log.info("Reconciling tachograph vehicle identities from source master data tenant={} source={}",
tenantKey, masterDataSource.stableKey());
int reconciledVehicles = vehicleIdentityRepository.reconcileFromMasterData(tenantKey, eventSourceId);
MasterDataRefreshResult result = new MasterDataRefreshResult(entities, relationCount);
log.info("Refreshed tachograph source master data tenant={} source={} entities={} relations={} reconciledVehicles={}",
tenantKey, masterDataSource.stableKey(), result.entitiesUpserted(), result.relationsUpserted(), reconciledVehicles);
log.info("Refreshed tachograph source master data tenant={} source={} entities={} relations={} reconciledDrivers={} reconciledVehicles={}",
tenantKey, masterDataSource.stableKey(), result.entitiesUpserted(), result.relationsUpserted(), reconciledDrivers, reconciledVehicles);
return result;
}

View File

@ -0,0 +1,93 @@
package at.procon.eventhub.yellowfox.api;
import at.procon.eventhub.dto.AcquisitionStrategy;
import at.procon.eventhub.dto.ImportMode;
import at.procon.eventhub.dto.SchedulerTriggerMode;
import at.procon.eventhub.yellowfox.dto.ConfiguredYellowFoxD8ImportPlanDto;
import at.procon.eventhub.yellowfox.dto.YellowFoxD8ImportRequest;
import at.procon.eventhub.yellowfox.dto.YellowFoxD8ImportRunResultDto;
import at.procon.eventhub.yellowfox.dto.YellowFoxD8ImportTriggerResultDto;
import at.procon.eventhub.yellowfox.service.YellowFoxD8ConfiguredImportPlanService;
import at.procon.eventhub.yellowfox.service.YellowFoxD8ImportExecutionService;
import at.procon.eventhub.yellowfox.service.YellowFoxD8ImportPlanService;
import jakarta.validation.Valid;
import java.time.OffsetDateTime;
import java.util.List;
import org.apache.camel.ProducerTemplate;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;
@RestController
@RequestMapping("/api/eventhub/acquisition/yellowfox/d8")
public class YellowFoxD8ImportController {
private final ProducerTemplate producerTemplate;
private final YellowFoxD8ImportPlanService importPlanService;
private final YellowFoxD8ConfiguredImportPlanService configuredImportPlanService;
private final YellowFoxD8ImportExecutionService importExecutionService;
public YellowFoxD8ImportController(
ProducerTemplate producerTemplate,
YellowFoxD8ImportPlanService importPlanService,
YellowFoxD8ConfiguredImportPlanService configuredImportPlanService,
YellowFoxD8ImportExecutionService importExecutionService
) {
this.producerTemplate = producerTemplate;
this.importPlanService = importPlanService;
this.configuredImportPlanService = configuredImportPlanService;
this.importExecutionService = importExecutionService;
}
@PostMapping("/imports/plan")
public ResponseEntity<?> planYellowFoxD8Import(@Valid @RequestBody YellowFoxD8ImportRequest request) {
return ResponseEntity.ok(importPlanService.createPlan(request));
}
@PostMapping("/imports/start")
public ResponseEntity<YellowFoxD8ImportRunResultDto> startYellowFoxD8Import(
@Valid @RequestBody YellowFoxD8ImportRequest request,
@RequestParam(defaultValue = "false") boolean execute
) {
YellowFoxD8ImportRunResultDto result = execute
? importExecutionService.startAndExecuteImport(request)
: producerTemplate.requestBody("direct:yellowfox-d8-import-start", request, YellowFoxD8ImportRunResultDto.class);
return ResponseEntity.accepted().body(result);
}
@GetMapping("/imports/configured-plans")
public ResponseEntity<List<ConfiguredYellowFoxD8ImportPlanDto>> listConfiguredYellowFoxPlans() {
return ResponseEntity.ok(configuredImportPlanService.listPlans());
}
@GetMapping("/imports/configured-plans/{planKey}")
public ResponseEntity<ConfiguredYellowFoxD8ImportPlanDto> getConfiguredYellowFoxPlan(@PathVariable String planKey) {
return ResponseEntity.ok(configuredImportPlanService.getPlan(planKey));
}
@PostMapping("/imports/configured-plans/{planKey}/start")
public ResponseEntity<YellowFoxD8ImportTriggerResultDto> startConfiguredYellowFoxPlan(
@PathVariable String planKey,
@RequestParam(required = false) ImportMode mode,
@RequestParam(required = false) AcquisitionStrategy strategy,
@RequestParam(defaultValue = "PLAN_ONLY") SchedulerTriggerMode triggerMode
) {
YellowFoxD8ImportRequest request = configuredImportPlanService.createRequest(planKey, mode, strategy);
YellowFoxD8ImportRunResultDto result = triggerMode == SchedulerTriggerMode.EXECUTE
? importExecutionService.startAndExecuteImport(request)
: importExecutionService.startImport(request);
return ResponseEntity.accepted().body(new YellowFoxD8ImportTriggerResultDto(
planKey,
request.mode(),
request.acquisitionStrategy(),
triggerMode,
OffsetDateTime.now(),
result
));
}
}

View File

@ -1,6 +1,11 @@
package at.procon.eventhub.yellowfox.camel;
import at.procon.eventhub.dto.EventHubEventDto;
import at.procon.eventhub.yellowfox.dto.YellowFoxD8BookingDto;
import at.procon.eventhub.yellowfox.service.YellowFoxD8BookingEventMapper;
import at.procon.eventhub.yellowfox.service.YellowFoxD8IgnitionTransitionDetector;
import java.util.ArrayList;
import java.util.List;
import org.apache.camel.builder.RouteBuilder;
import org.springframework.stereotype.Component;
@ -8,17 +13,34 @@ import org.springframework.stereotype.Component;
public class YellowFoxD8BookingInputRoute extends RouteBuilder {
private final YellowFoxD8BookingEventMapper mapper;
private final YellowFoxD8IgnitionTransitionDetector ignitionTransitionDetector;
public YellowFoxD8BookingInputRoute(YellowFoxD8BookingEventMapper mapper) {
public YellowFoxD8BookingInputRoute(
YellowFoxD8BookingEventMapper mapper,
YellowFoxD8IgnitionTransitionDetector ignitionTransitionDetector
) {
this.mapper = mapper;
this.ignitionTransitionDetector = ignitionTransitionDetector;
}
@Override
public void configure() {
from("direct:yellowfox-d8-booking-input")
.routeId("yellowfox-d8-booking-input-route")
.process(exchange -> {
List<YellowFoxD8BookingDto> bookings = exchange.getMessage().getBody(List.class);
YellowFoxD8IgnitionTransitionDetector.Session ignitionSession = ignitionTransitionDetector.newSession(false);
List<EventHubEventDto> events = new ArrayList<>();
for (YellowFoxD8BookingDto booking : bookings) {
events.add(mapper.map(booking));
EventHubEventDto ignitionEvent = ignitionSession.detect(booking);
if (ignitionEvent != null) {
events.add(ignitionEvent);
}
}
exchange.getMessage().setBody(events);
})
.split(body())
.bean(mapper, "map")
.to("direct:eventhub-normalized-input")
.end();
}

View File

@ -0,0 +1,33 @@
package at.procon.eventhub.yellowfox.camel;
import at.procon.eventhub.yellowfox.dto.YellowFoxD8ImportRequest;
import at.procon.eventhub.yellowfox.service.YellowFoxD8ImportExecutionService;
import org.apache.camel.builder.RouteBuilder;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.stereotype.Component;
@Component
public class YellowFoxD8ImportRequestRoute extends RouteBuilder {
private static final Logger log = LoggerFactory.getLogger(YellowFoxD8ImportRequestRoute.class);
private final YellowFoxD8ImportExecutionService executionService;
public YellowFoxD8ImportRequestRoute(YellowFoxD8ImportExecutionService executionService) {
this.executionService = executionService;
}
@Override
public void configure() {
from("direct:yellowfox-d8-import-start")
.routeId("yellowfox-d8-import-start-route")
.process(exchange -> {
YellowFoxD8ImportRequest request = exchange.getMessage().getBody(YellowFoxD8ImportRequest.class);
var result = executionService.startImport(request);
log.info("Prepared YellowFox D8 import run importRunId={} plannedPackages={} status={}",
result.importRunId(), result.plannedPackageCount(), result.status());
exchange.getMessage().setBody(result);
});
}
}

View File

@ -0,0 +1,96 @@
package at.procon.eventhub.yellowfox.config;
import javax.sql.DataSource;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.boot.autoconfigure.condition.ConditionalOnExpression;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.jdbc.core.namedparam.NamedParameterJdbcTemplate;
import org.springframework.jdbc.datasource.DriverManagerDataSource;
@Configuration
@ConditionalOnExpression("T(org.springframework.util.StringUtils).hasText('${eventhub.yellow-fox.datasource.jdbc-url:}')")
public class YellowFoxDataSourceConfig {
@Bean
@ConfigurationProperties(prefix = "eventhub.yellow-fox.datasource")
public YellowFoxDataSourceProperties yellowFoxDataSourceProperties() {
return new YellowFoxDataSourceProperties();
}
@Bean(defaultCandidate = false)
public DataSource yellowFoxDataSource(YellowFoxDataSourceProperties config) {
DriverManagerDataSource dataSource = new DriverManagerDataSource();
dataSource.setUrl(validateJdbcUrl(config));
dataSource.setUsername(config.getUsername());
dataSource.setPassword(config.getPassword());
String driverClassName = trimToNull(config.getDriverClassName());
if (driverClassName != null) {
dataSource.setDriverClassName(driverClassName);
}
return dataSource;
}
@Bean
public NamedParameterJdbcTemplate yellowFoxNamedParameterJdbcTemplate(
@Qualifier("yellowFoxDataSource") DataSource yellowFoxDataSource
) {
return new NamedParameterJdbcTemplate(yellowFoxDataSource);
}
private String validateJdbcUrl(YellowFoxDataSourceProperties config) {
String jdbcUrl = trimToNull(config.getJdbcUrl());
if (jdbcUrl == null) {
throw new IllegalStateException("eventhub.yellow-fox.datasource.jdbc-url must not be empty");
}
return jdbcUrl;
}
private String trimToNull(String value) {
if (value == null) {
return null;
}
String trimmedValue = value.trim();
return trimmedValue.isEmpty() ? null : trimmedValue;
}
public static class YellowFoxDataSourceProperties {
private String jdbcUrl;
private String username;
private String password;
private String driverClassName;
public String getJdbcUrl() {
return jdbcUrl;
}
public void setJdbcUrl(String jdbcUrl) {
this.jdbcUrl = jdbcUrl;
}
public String getUsername() {
return username;
}
public void setUsername(String username) {
this.username = username;
}
public String getPassword() {
return password;
}
public void setPassword(String password) {
this.password = password;
}
public String getDriverClassName() {
return driverClassName;
}
public void setDriverClassName(String driverClassName) {
this.driverClassName = driverClassName;
}
}
}

View File

@ -0,0 +1,30 @@
package at.procon.eventhub.yellowfox.dto;
import at.procon.eventhub.dto.AcquisitionStrategy;
import at.procon.eventhub.dto.EventFamily;
import at.procon.eventhub.dto.EventSourceDto;
import at.procon.eventhub.dto.ImportMode;
import at.procon.eventhub.dto.ImportScopeDto;
import at.procon.eventhub.dto.SourceGroupRefDto;
import java.time.OffsetDateTime;
import java.util.Set;
public record ConfiguredYellowFoxD8ImportPlanDto(
String planKey,
boolean enabled,
String cron,
String tenantKey,
EventSourceDto eventSource,
SourceGroupRefDto sourceGroup,
ImportScopeDto importScope,
Set<EventFamily> eventFamilies,
ImportMode initialMode,
ImportMode scheduledMode,
AcquisitionStrategy initialStrategy,
AcquisitionStrategy scheduledStrategy,
boolean refreshMasterDataFirst,
OffsetDateTime initialOccurredFrom,
OffsetDateTime initialOccurredTo,
boolean runInitialOnStartup
) {
}

View File

@ -11,8 +11,10 @@ public record YellowFoxD8BookingDto(
String sourceInstanceKey,
String tenantProviderSettingKey,
String externalFleetKey,
String externalFleetName,
String eventId,
String key,
Integer ignition,
Integer eventType,
Integer state,
OffsetDateTime occurredAt,

View File

@ -0,0 +1,27 @@
package at.procon.eventhub.yellowfox.dto;
import at.procon.eventhub.importing.ExtractionBatchResult;
import java.time.OffsetDateTime;
import java.util.Map;
import java.util.UUID;
public record YellowFoxD8ExtractionBatchResultDto(
UUID packageId,
String extractionCode,
String sourceKind,
int sourceRowsRead,
int eventsMapped,
int eventsInserted,
int alreadyImported,
boolean executed,
OffsetDateTime lastSourcePackageImportedAt,
String lastSourcePackageId,
OffsetDateTime lastSourceRowUpdatedAt,
OffsetDateTime lastOccurredTo,
Map<String, Integer> eventTypeCounts
) implements ExtractionBatchResult {
public YellowFoxD8ExtractionBatchResultDto {
eventTypeCounts = eventTypeCounts == null ? Map.of() : Map.copyOf(eventTypeCounts);
}
}

View File

@ -0,0 +1,48 @@
package at.procon.eventhub.yellowfox.dto;
import at.procon.eventhub.dto.AcquisitionStrategy;
import at.procon.eventhub.dto.EventFamily;
import at.procon.eventhub.dto.EventSourceDto;
import at.procon.eventhub.dto.ImportMode;
import at.procon.eventhub.dto.ImportScopeDto;
import at.procon.eventhub.dto.SourceGroupRefDto;
import at.procon.eventhub.importing.ImportRunRequest;
import jakarta.validation.Valid;
import jakarta.validation.constraints.NotBlank;
import jakarta.validation.constraints.NotNull;
import java.util.EnumSet;
import java.util.Set;
public record YellowFoxD8ImportRequest(
@NotBlank String tenantKey,
@Valid @NotNull EventSourceDto eventSource,
@Valid SourceGroupRefDto sourceGroup,
@Valid @NotNull ImportScopeDto importScope,
Set<EventFamily> eventFamilies,
ImportMode mode,
boolean refreshMasterDataFirst,
AcquisitionStrategy acquisitionStrategy
) implements ImportRunRequest {
public YellowFoxD8ImportRequest {
tenantKey = tenantKey == null ? null : tenantKey.trim();
if (eventSource == null) {
eventSource = new EventSourceDto("YELLOWFOX", "TELEMATICS_PLATFORM", "YELLOWFOX_D8", null, null, null);
}
if (importScope == null) {
importScope = ImportScopeDto.tenantAll(null, null);
}
if (eventFamilies == null || eventFamilies.isEmpty()) {
eventFamilies = EnumSet.of(EventFamily.DRIVER_ACTIVITY, EventFamily.DRIVER_CARD);
} else {
eventFamilies = EnumSet.copyOf(eventFamilies);
}
if (mode == null) {
mode = ImportMode.INITIAL_BACKFILL;
}
if (acquisitionStrategy == null || acquisitionStrategy == AcquisitionStrategy.SOURCE_PACKAGE_WATERMARK) {
acquisitionStrategy = mode == ImportMode.INCREMENTAL_UPDATE
? AcquisitionStrategy.SOURCE_ROW_WATERMARK
: AcquisitionStrategy.OCCURRED_AT_WINDOW_WITH_OVERLAP;
}
}
}

View File

@ -0,0 +1,15 @@
package at.procon.eventhub.yellowfox.dto;
import at.procon.eventhub.dto.ImportRunStatus;
import at.procon.eventhub.importing.ImportPlanDto;
import java.util.List;
import java.util.UUID;
public record YellowFoxD8ImportRunResultDto(
UUID importRunId,
ImportRunStatus status,
int plannedPackageCount,
ImportPlanDto plan,
List<UUID> plannedPackageIds
) {
}

View File

@ -0,0 +1,16 @@
package at.procon.eventhub.yellowfox.dto;
import at.procon.eventhub.dto.AcquisitionStrategy;
import at.procon.eventhub.dto.ImportMode;
import at.procon.eventhub.dto.SchedulerTriggerMode;
import java.time.OffsetDateTime;
public record YellowFoxD8ImportTriggerResultDto(
String planKey,
ImportMode mode,
AcquisitionStrategy acquisitionStrategy,
SchedulerTriggerMode triggerMode,
OffsetDateTime triggeredAt,
YellowFoxD8ImportRunResultDto result
) {
}

View File

@ -0,0 +1,238 @@
package at.procon.eventhub.yellowfox.service;
import at.procon.eventhub.config.EventHubProperties;
import at.procon.eventhub.dto.AcquisitionStrategy;
import at.procon.eventhub.dto.EventHubEventDto;
import at.procon.eventhub.dto.EventType;
import at.procon.eventhub.dto.ImportCursorStateDto;
import at.procon.eventhub.dto.ImportScopeDto;
import at.procon.eventhub.dto.SourceGroupType;
import at.procon.eventhub.importing.ImportPlanItemDto;
import at.procon.eventhub.importing.ImportTimeChunkDto;
import at.procon.eventhub.importing.persistence.ImportCursorRepository;
import at.procon.eventhub.yellowfox.dto.YellowFoxD8BookingDto;
import at.procon.eventhub.yellowfox.dto.YellowFoxD8ExtractionBatchResultDto;
import at.procon.eventhub.yellowfox.dto.YellowFoxD8ImportRequest;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.time.OffsetDateTime;
import java.util.HashMap;
import java.util.LinkedHashMap;
import java.util.Map;
import java.util.UUID;
import org.apache.camel.ProducerTemplate;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.boot.autoconfigure.condition.ConditionalOnExpression;
import org.springframework.core.io.Resource;
import org.springframework.core.io.ResourceLoader;
import org.springframework.jdbc.core.namedparam.NamedParameterJdbcTemplate;
import org.springframework.stereotype.Service;
import org.springframework.util.StreamUtils;
@Service
@ConditionalOnExpression("T(org.springframework.util.StringUtils).hasText('${eventhub.yellow-fox.datasource.jdbc-url:}')")
public class JdbcYellowFoxD8BookingExtractionBatchExecutor implements YellowFoxD8ExtractionBatchExecutor {
private static final Logger log = LoggerFactory.getLogger(JdbcYellowFoxD8BookingExtractionBatchExecutor.class);
private static final String EXTRACTION_CODE = "YELLOWFOX_D8_BOOKING";
private static final int PROGRESS_LOG_INTERVAL = 5000;
private final NamedParameterJdbcTemplate jdbcTemplate;
private final ProducerTemplate producerTemplate;
private final ResourceLoader resourceLoader;
private final ImportCursorRepository importCursorRepository;
private final EventHubProperties properties;
private final YellowFoxD8BookingRowMapper rowMapper;
private final YellowFoxD8BookingEventMapper eventMapper;
private final YellowFoxD8IgnitionTransitionDetector ignitionTransitionDetector;
public JdbcYellowFoxD8BookingExtractionBatchExecutor(
@Qualifier("yellowFoxNamedParameterJdbcTemplate") NamedParameterJdbcTemplate jdbcTemplate,
ProducerTemplate producerTemplate,
ResourceLoader resourceLoader,
ImportCursorRepository importCursorRepository,
EventHubProperties properties,
YellowFoxD8BookingRowMapper rowMapper,
YellowFoxD8BookingEventMapper eventMapper,
YellowFoxD8IgnitionTransitionDetector ignitionTransitionDetector
) {
this.jdbcTemplate = jdbcTemplate;
this.producerTemplate = producerTemplate;
this.resourceLoader = resourceLoader;
this.importCursorRepository = importCursorRepository;
this.properties = properties;
this.rowMapper = rowMapper;
this.eventMapper = eventMapper;
this.ignitionTransitionDetector = ignitionTransitionDetector;
}
@Override
public YellowFoxD8ExtractionBatchResultDto execute(
UUID importRunId,
UUID packageId,
int eventSourceId,
YellowFoxD8ImportRequest request,
ImportPlanItemDto planItem,
ImportTimeChunkDto chunk
) {
if (!EXTRACTION_CODE.equals(planItem.extractionCode())) {
throw new IllegalArgumentException("Unsupported YellowFox extraction code " + planItem.extractionCode());
}
ImportScopeDto chunkScope = chunkScope(request.importScope(), chunk);
ImportCursorStateDto cursor = findCursor(eventSourceId, request, planItem);
Map<String, Object> params = parameters(request, chunkScope, cursor);
Stats stats = new Stats();
YellowFoxD8IgnitionTransitionDetector.Session ignitionSession = ignitionTransitionDetector
.newSession(properties.getYellowFox().isEmitInitialIgnitionSnapshot());
log.info("Reading YellowFox D8 bookings tenant={} importRunId={} packageId={} chunk={} occurredFrom={} occurredTo={} fleetId={} strategy={}",
request.tenantKey(), importRunId, packageId, chunk.sequence(), chunk.occurredFrom(), chunk.occurredTo(), params.get("fleetId"), request.acquisitionStrategy());
jdbcTemplate.query(loadSql(), params, rs -> {
stats.sourceRowsRead++;
YellowFoxD8BookingDto booking = rowMapper.map(
rs,
request.tenantKey(),
request.eventSource().sourceInstanceKey(),
request.eventSource().tenantProviderSettingKey()
);
stats.acceptSourceRow(booking);
if (!hasEventReference(booking)) {
stats.skippedRows++;
return;
}
EventHubEventDto primaryEvent = eventMapper.map(booking);
send(primaryEvent, stats);
EventHubEventDto ignitionEvent = ignitionSession.detect(booking);
if (ignitionEvent != null) {
send(ignitionEvent, stats);
}
if (stats.sourceRowsRead % PROGRESS_LOG_INTERVAL == 0) {
log.info("YellowFox D8 extraction progress tenant={} importRunId={} packageId={} rows={} events={} byType={}",
request.tenantKey(), importRunId, packageId, stats.sourceRowsRead, stats.eventsSent, stats.eventTypeCounts);
}
});
log.info("Finished YellowFox D8 extraction tenant={} importRunId={} packageId={} rows={} events={} skippedRows={} byType={}",
request.tenantKey(), importRunId, packageId, stats.sourceRowsRead, stats.eventsSent, stats.skippedRows, stats.eventTypeCounts);
return new YellowFoxD8ExtractionBatchResultDto(
packageId,
planItem.extractionCode(),
planItem.sourceKind(),
stats.sourceRowsRead,
stats.eventsSent,
stats.eventsSent,
stats.skippedRows,
true,
null,
stats.lastEventId,
stats.lastOccurredAt,
stats.lastOccurredAt,
stats.eventTypeCounts
);
}
private ImportCursorStateDto findCursor(int eventSourceId, YellowFoxD8ImportRequest request, ImportPlanItemDto planItem) {
String scopeHash = request.importScope() == null ? "NO_SCOPE" : request.importScope().stableKey();
return importCursorRepository.findCursor(
request.tenantKey(),
eventSourceId,
scopeHash,
planItem.eventFamily(),
planItem.sourceKind(),
request.acquisitionStrategy()
);
}
private Map<String, Object> parameters(YellowFoxD8ImportRequest request, ImportScopeDto scope, ImportCursorStateDto cursor) {
Map<String, Object> params = new HashMap<>();
params.put("occurredFrom", scope == null ? null : scope.occurredFrom());
params.put("occurredTo", scope == null ? null : scope.occurredTo());
params.put("fleetId", fleetId(request));
if (request.acquisitionStrategy() == AcquisitionStrategy.SOURCE_ROW_WATERMARK && cursor != null) {
OffsetDateTime lastOccurredTo = cursor.lastOccurredTo();
String lastSourceRowId = cursor.lastSourcePackageId();
if (lastOccurredTo != null && properties.getYellowFox().getOccurredAtOverlap() != null
&& !properties.getYellowFox().getOccurredAtOverlap().isZero()) {
lastOccurredTo = lastOccurredTo.minus(properties.getYellowFox().getOccurredAtOverlap());
lastSourceRowId = null;
}
params.put("lastOccurredTo", lastOccurredTo);
params.put("lastSourceRowId", lastSourceRowId);
} else {
params.put("lastOccurredTo", null);
params.put("lastSourceRowId", null);
}
return params;
}
private Integer fleetId(YellowFoxD8ImportRequest request) {
if (request.sourceGroup() == null || request.sourceGroup().type() != SourceGroupType.FLEET) {
return null;
}
String value = request.sourceGroup().sourceEntityId() == null
? request.sourceGroup().code()
: request.sourceGroup().sourceEntityId();
if (value == null || value.isBlank()) {
return null;
}
try {
return Integer.parseInt(value.trim());
} catch (NumberFormatException ex) {
throw new IllegalArgumentException("YellowFox D8 fleet sourceEntityId/code must be numeric for JDBC import: " + value, ex);
}
}
private ImportScopeDto chunkScope(ImportScopeDto scope, ImportTimeChunkDto chunk) {
if (scope == null) {
return ImportScopeDto.tenantAll(chunk.occurredFrom(), chunk.occurredTo());
}
return new ImportScopeDto(scope.type(), scope.rootSourceOrganisation(), scope.includeChildren(), chunk.occurredFrom(), chunk.occurredTo());
}
private boolean hasEventReference(YellowFoxD8BookingDto booking) {
return (booking.driverRef() != null && booking.driverRef().hasAnyReference())
|| (booking.vehicleRef() != null && booking.vehicleRef().hasAnyReference());
}
private void send(EventHubEventDto event, Stats stats) {
producerTemplate.sendBody("direct:eventhub-normalized-input", event);
stats.acceptEvent(event);
}
private String loadSql() {
Resource resource = resourceLoader.getResource("classpath:sql/yellowfox/d8-bookings.sql");
try (var in = resource.getInputStream()) {
return StreamUtils.copyToString(in, StandardCharsets.UTF_8);
} catch (IOException ex) {
throw new IllegalStateException("Failed to load YellowFox D8 extraction SQL", ex);
}
}
private static class Stats {
private int sourceRowsRead;
private int eventsSent;
private int skippedRows;
private OffsetDateTime lastOccurredAt;
private String lastEventId;
private final Map<String, Integer> eventTypeCounts = new LinkedHashMap<>();
private void acceptSourceRow(YellowFoxD8BookingDto booking) {
if (booking.occurredAt() != null) {
lastOccurredAt = booking.occurredAt();
lastEventId = booking.eventId();
}
}
private void acceptEvent(EventHubEventDto event) {
eventsSent++;
EventType type = event.eventType();
String key = event.eventDomain().name() + "/" + (type == null ? "UNKNOWN" : type.name());
eventTypeCounts.merge(key, 1, Integer::sum);
}
}
}

View File

@ -0,0 +1,45 @@
package at.procon.eventhub.yellowfox.service;
import at.procon.eventhub.importing.ImportPlanItemDto;
import at.procon.eventhub.importing.ImportTimeChunkDto;
import at.procon.eventhub.importing.extraction.AbstractNoopExtractionBatchExecutor;
import at.procon.eventhub.yellowfox.dto.YellowFoxD8ExtractionBatchResultDto;
import at.procon.eventhub.yellowfox.dto.YellowFoxD8ImportRequest;
import java.util.UUID;
import org.springframework.boot.autoconfigure.condition.ConditionalOnExpression;
import org.springframework.stereotype.Service;
@Service
@ConditionalOnExpression("!T(org.springframework.util.StringUtils).hasText('${eventhub.yellow-fox.datasource.jdbc-url:}')")
public class NoopYellowFoxD8ExtractionBatchExecutor
extends AbstractNoopExtractionBatchExecutor<YellowFoxD8ImportRequest, YellowFoxD8ExtractionBatchResultDto>
implements YellowFoxD8ExtractionBatchExecutor {
@Override
protected YellowFoxD8ExtractionBatchResultDto emptyResult(
UUID packageId,
ImportPlanItemDto planItem,
ImportTimeChunkDto chunk
) {
return new YellowFoxD8ExtractionBatchResultDto(
packageId,
planItem.extractionCode(),
planItem.sourceKind(),
0,
0,
0,
0,
false,
null,
null,
null,
null,
java.util.Map.of()
);
}
@Override
protected String providerName() {
return "yellowfox-d8";
}
}

View File

@ -2,7 +2,8 @@ package at.procon.eventhub.yellowfox.service;
import at.procon.eventhub.dto.CardSlot;
import at.procon.eventhub.dto.CardStatus;
import at.procon.eventhub.dto.DrivingStatus;
import at.procon.eventhub.dto.DriverRefDto;
import at.procon.eventhub.dto.EventDetailsDto;
import at.procon.eventhub.dto.EventDomain;
import at.procon.eventhub.dto.EventHubEventDto;
import at.procon.eventhub.dto.EventHubPackageRequest;
@ -13,6 +14,7 @@ import at.procon.eventhub.dto.GeoPointDto;
import at.procon.eventhub.dto.ImportScopeDto;
import at.procon.eventhub.dto.SourceGroupRefDto;
import at.procon.eventhub.dto.SourceGroupType;
import at.procon.eventhub.dto.VehicleRefDto;
import at.procon.eventhub.service.EventDetailsFactory;
import at.procon.eventhub.yellowfox.dto.YellowFoxD8BookingDto;
import java.time.LocalDate;
@ -25,6 +27,8 @@ import org.springframework.stereotype.Component;
@Component
public class YellowFoxD8BookingEventMapper {
public static final String DETAIL_SOURCE = "YELLOWFOX_D8";
private final EventDetailsFactory detailsFactory;
public YellowFoxD8BookingEventMapper(EventDetailsFactory detailsFactory) {
@ -33,30 +37,42 @@ public class YellowFoxD8BookingEventMapper {
public EventHubEventDto map(YellowFoxD8BookingDto source) {
NormalizedEvent normalized = normalize(source.eventType(), source.state());
Map<String, Object> payload = new LinkedHashMap<>();
if (source.payload() != null) {
payload.putAll(source.payload());
}
payload.put("provider", "YELLOWFOX");
payload.put("sourceKind", "TELEMATICS_PLATFORM");
payload.put("yellowFoxEventType", source.eventType());
payload.put("yellowFoxState", source.state());
payload.put("yellowFoxKey", source.key());
return event(source, sourceEventId(source, "D8_BOOKING"), normalized, detailsFor(source, normalized), false);
}
EventSourceDto eventSource = new EventSourceDto(
"YELLOWFOX",
"TELEMATICS_PLATFORM",
"YELLOWFOX_D8",
source.sourceInstanceKey(),
source.tenantProviderSettingKey(),
source.externalFleetKey()
public EventHubEventDto mapIgnitionTransition(YellowFoxD8BookingDto source, Integer previousIgnitionState) {
if (source.ignition() == null) {
return null;
}
boolean ignitionOn = source.ignition() == 1;
NormalizedEvent normalized = new NormalizedEvent(
EventDomain.IGNITION,
ignitionOn ? EventType.IGNITION_ON : EventType.IGNITION_OFF,
ignitionOn ? EventLifecycle.ON : EventLifecycle.OFF,
null,
null
);
Map<String, Object> attributes = baseAttributes(source, normalized);
attributes.put("previousIgnitionState", previousIgnitionState);
attributes.put("previousIgnitionMeaning", ignitionMeaning(previousIgnitionState));
attributes.put("derivedFrom", "YELLOWFOX_D8_BOOKING");
attributes.put("sourceEventId", sourceEventId(source, "D8_BOOKING"));
EventDetailsDto details = new EventDetailsDto("IGNITION", detailsFactory.payloadFromMap(attributes));
return event(source, sourceEventId(source, "D8_IGNITION") + ":" + (ignitionOn ? "ON" : "OFF"), normalized, details, true);
}
private EventHubEventDto event(
YellowFoxD8BookingDto source,
String externalSourceEventId,
NormalizedEvent normalized,
EventDetailsDto details,
boolean derived
) {
EventSourceDto eventSource = eventSource(source);
LocalDate businessDate = source.occurredAt().toLocalDate();
var occurredFrom = businessDate.atStartOfDay(source.occurredAt().getOffset()).toOffsetDateTime();
var occurredTo = businessDate.plusDays(1).atStartOfDay(source.occurredAt().getOffset()).toOffsetDateTime();
SourceGroupRefDto sourceGroup = source.externalFleetKey() == null || source.externalFleetKey().isBlank()
? null
: new SourceGroupRefDto(SourceGroupType.FLEET, source.externalFleetKey(), source.externalFleetKey(), null);
SourceGroupRefDto sourceGroup = sourceGroup(source);
EventHubPackageRequest packageInfo = new EventHubPackageRequest(
tenantOrDefault(source.tenantKey()),
eventSource,
@ -69,8 +85,8 @@ public class YellowFoxD8BookingEventMapper {
return new EventHubEventDto(
UUID.randomUUID(),
source.eventId(),
source.driverRef(),
externalSourceEventId,
derived && normalized.domain() == EventDomain.IGNITION ? null : source.driverRef(),
source.vehicleRef(),
source.occurredAt(),
source.receivedPartnerAt(),
@ -79,60 +95,170 @@ public class YellowFoxD8BookingEventMapper {
normalized.type(),
normalized.lifecycle(),
source.odometerM(),
new GeoPointDto(source.latitude(), source.longitude()),
detailsFor(normalized),
point(source),
details,
null,
detailsFactory.payloadFromMap(payload),
detailsFactory.payloadFromMap(payload(source, derived)),
false,
packageInfo
);
}
public EventSourceDto eventSource(YellowFoxD8BookingDto source) {
return new EventSourceDto(
"YELLOWFOX",
"TELEMATICS_PLATFORM",
"YELLOWFOX_D8",
source.sourceInstanceKey(),
source.tenantProviderSettingKey(),
source.externalFleetKey()
);
}
private SourceGroupRefDto sourceGroup(YellowFoxD8BookingDto source) {
if (source.externalFleetKey() == null || source.externalFleetKey().isBlank()) {
return null;
}
return new SourceGroupRefDto(
SourceGroupType.FLEET,
source.externalFleetKey(),
source.externalFleetKey(),
source.externalFleetName() == null ? source.externalFleetKey() : source.externalFleetName()
);
}
private GeoPointDto point(YellowFoxD8BookingDto source) {
if (source.latitude() == null && source.longitude() == null) {
return null;
}
return new GeoPointDto(source.latitude(), source.longitude());
}
private String tenantOrDefault(String value) {
return value == null || value.isBlank() ? "default" : value.trim();
}
private at.procon.eventhub.dto.EventDetailsDto detailsFor(NormalizedEvent normalized) {
if (normalized.domain() == EventDomain.DRIVER_ACTIVITY) {
return detailsFactory.driverActivity(CardSlot.DRIVER, CardStatus.INSERTED, DrivingStatus.UNKNOWN);
private String sourceEventId(YellowFoxD8BookingDto source, String prefix) {
String eventId = source.eventId() == null || source.eventId().isBlank() ? "UNKNOWN_EVENT" : source.eventId().trim();
return prefix + ":" + eventId + ":" + source.occurredAt();
}
private Map<String, Object> payload(YellowFoxD8BookingDto source, boolean derived) {
Map<String, Object> payload = new LinkedHashMap<>();
if (source.payload() != null) {
payload.putAll(source.payload());
}
if (normalized.domain() == EventDomain.DRIVER_CARD) {
CardStatus status = normalized.lifecycle() == EventLifecycle.INSERT ? CardStatus.INSERTED : CardStatus.NOT_INSERTED;
return detailsFactory.driverCard(CardSlot.DRIVER, status);
}
return null;
payload.put("provider", "YELLOWFOX");
payload.put("sourceKind", "TELEMATICS_PLATFORM");
payload.put("sourceKey", "YELLOWFOX_D8");
payload.put("derived", derived);
payload.put("yellowFoxEventType", source.eventType());
payload.put("yellowFoxState", source.state());
payload.put("yellowFoxIgnition", source.ignition());
payload.put("yellowFoxKey", source.key());
return payload;
}
private EventDetailsDto detailsFor(YellowFoxD8BookingDto source, NormalizedEvent normalized) {
String type = switch (normalized.domain()) {
case DRIVER_ACTIVITY -> "DRIVER_ACTIVITY";
case DRIVER_CARD -> "DRIVER_CARD";
default -> "YELLOWFOX_D8_BOOKING";
};
return new EventDetailsDto(type, detailsFactory.payloadFromMap(baseAttributes(source, normalized)));
}
private Map<String, Object> baseAttributes(YellowFoxD8BookingDto source, NormalizedEvent normalized) {
Map<String, Object> attributes = new LinkedHashMap<>();
attributes.put("source", DETAIL_SOURCE);
attributes.put("yellowFoxEventType", source.eventType());
attributes.put("yellowFoxState", source.state());
attributes.put("yellowFoxStateMeaning", stateMeaning(source.eventType(), source.state()));
attributes.put("yellowFoxKey", source.key());
attributes.put("ignitionState", source.ignition());
attributes.put("ignitionMeaning", ignitionMeaning(source.ignition()));
attributes.put("odometer", source.odometerM());
attributes.put("slot", normalized.slot() == null ? null : normalized.slot().name());
attributes.put("slotNumber", normalized.slotNumber());
attributes.put("cardSlot", normalized.slot() == null ? null : normalized.slot().name());
attributes.put("eventDomain", normalized.domain().name());
attributes.put("eventType", normalized.type().name());
attributes.put("lifecycle", normalized.lifecycle().name());
removeNullValues(attributes);
return attributes;
}
private void removeNullValues(Map<String, Object> attributes) {
attributes.entrySet().removeIf(entry -> entry.getValue() == null);
}
private NormalizedEvent normalize(Integer eventType, Integer state) {
if (eventType == null || state == null) {
return new NormalizedEvent(EventDomain.TELEMATICS_DATA, EventType.UNKNOWN_EVENT, EventLifecycle.SNAPSHOT);
return new NormalizedEvent(EventDomain.TELEMATICS_DATA, EventType.UNKNOWN_EVENT, EventLifecycle.SNAPSHOT, null, null);
}
return switch (eventType) {
case 0, 1 -> normalizeCardActivity(state);
case 2, 3 -> normalizeDriverActivity(state);
default -> new NormalizedEvent(EventDomain.TELEMATICS_DATA, EventType.UNKNOWN_EVENT, EventLifecycle.SNAPSHOT);
case 0 -> normalizeCardActivity(CardSlot.DRIVER, 1, state);
case 1 -> normalizeCardActivity(CardSlot.CO_DRIVER, 2, state);
case 2 -> normalizeDriverActivity(CardSlot.DRIVER, 1, state);
case 3 -> normalizeDriverActivity(CardSlot.CO_DRIVER, 2, state);
default -> new NormalizedEvent(EventDomain.TELEMATICS_DATA, EventType.UNKNOWN_EVENT, EventLifecycle.SNAPSHOT, null, null);
};
}
private NormalizedEvent normalizeCardActivity(Integer state) {
private NormalizedEvent normalizeCardActivity(CardSlot slot, int slotNumber, Integer state) {
return switch (state) {
case 0 -> new NormalizedEvent(EventDomain.DRIVER_CARD, EventType.CARD_WITHDRAWN, EventLifecycle.WITHDRAW);
case 1 -> new NormalizedEvent(EventDomain.DRIVER_CARD, EventType.CARD_INSERTED, EventLifecycle.INSERT);
default -> new NormalizedEvent(EventDomain.DRIVER_CARD, EventType.UNKNOWN_EVENT, EventLifecycle.SNAPSHOT);
case 0 -> new NormalizedEvent(EventDomain.DRIVER_CARD, EventType.CARD_WITHDRAWN, EventLifecycle.WITHDRAW, slot, slotNumber);
case 1 -> new NormalizedEvent(EventDomain.DRIVER_CARD, EventType.CARD_INSERTED, EventLifecycle.INSERT, slot, slotNumber);
default -> new NormalizedEvent(EventDomain.DRIVER_CARD, EventType.UNKNOWN_EVENT, EventLifecycle.SNAPSHOT, slot, slotNumber);
};
}
private NormalizedEvent normalizeDriverActivity(Integer state) {
private NormalizedEvent normalizeDriverActivity(CardSlot slot, int slotNumber, Integer state) {
return switch (state) {
case 0 -> new NormalizedEvent(EventDomain.DRIVER_ACTIVITY, EventType.BREAK_REST, EventLifecycle.SNAPSHOT);
case 1 -> new NormalizedEvent(EventDomain.DRIVER_ACTIVITY, EventType.AVAILABILITY, EventLifecycle.SNAPSHOT);
case 2 -> new NormalizedEvent(EventDomain.DRIVER_ACTIVITY, EventType.WORK, EventLifecycle.SNAPSHOT);
case 3 -> new NormalizedEvent(EventDomain.DRIVER_ACTIVITY, EventType.DRIVE, EventLifecycle.SNAPSHOT);
default -> new NormalizedEvent(EventDomain.DRIVER_ACTIVITY, EventType.UNKNOWN_ACTIVITY, EventLifecycle.SNAPSHOT);
case 0 -> new NormalizedEvent(EventDomain.DRIVER_ACTIVITY, EventType.BREAK_REST, EventLifecycle.START, slot, slotNumber);
case 1 -> new NormalizedEvent(EventDomain.DRIVER_ACTIVITY, EventType.AVAILABILITY, EventLifecycle.START, slot, slotNumber);
case 2 -> new NormalizedEvent(EventDomain.DRIVER_ACTIVITY, EventType.WORK, EventLifecycle.START, slot, slotNumber);
case 3 -> new NormalizedEvent(EventDomain.DRIVER_ACTIVITY, EventType.DRIVE, EventLifecycle.START, slot, slotNumber);
default -> new NormalizedEvent(EventDomain.DRIVER_ACTIVITY, EventType.UNKNOWN_ACTIVITY, EventLifecycle.SNAPSHOT, slot, slotNumber);
};
}
private record NormalizedEvent(EventDomain domain, EventType type, EventLifecycle lifecycle) {
private String stateMeaning(Integer eventType, Integer state) {
if (eventType == null || state == null) {
return null;
}
if (eventType == 0 || eventType == 1) {
return switch (state) {
case 0 -> "CARD_REMOVED";
case 1 -> "CARD_INSERTED";
default -> "UNKNOWN_CARD_STATE";
};
}
if (eventType == 2 || eventType == 3) {
return switch (state) {
case 0 -> "PAUSE_OR_REST";
case 1 -> "STANDBY_TIME";
case 2 -> "WORK_TIME";
case 3 -> "STEERING_TIME";
default -> "UNKNOWN_ACTIVITY_STATE";
};
}
return "UNKNOWN_EVENT_TYPE";
}
private String ignitionMeaning(Integer ignition) {
if (ignition == null) {
return null;
}
return ignition == 1 ? "ON" : ignition == 0 ? "OFF" : "UNKNOWN";
}
private record NormalizedEvent(
EventDomain domain,
EventType type,
EventLifecycle lifecycle,
CardSlot slot,
Integer slotNumber
) {
}
}

View File

@ -0,0 +1,117 @@
package at.procon.eventhub.yellowfox.service;
import at.procon.eventhub.dto.DriverCardRefDto;
import at.procon.eventhub.dto.DriverRefDto;
import at.procon.eventhub.dto.VehicleRefDto;
import at.procon.eventhub.dto.VehicleRegistrationRefDto;
import at.procon.eventhub.yellowfox.dto.YellowFoxD8BookingDto;
import com.fasterxml.jackson.core.type.TypeReference;
import com.fasterxml.jackson.databind.ObjectMapper;
import java.math.BigDecimal;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.time.OffsetDateTime;
import java.util.LinkedHashMap;
import java.util.Map;
import org.springframework.stereotype.Component;
@Component
public class YellowFoxD8BookingRowMapper {
private static final TypeReference<Map<String, Object>> MAP_TYPE = new TypeReference<>() {
};
private final ObjectMapper objectMapper;
public YellowFoxD8BookingRowMapper(ObjectMapper objectMapper) {
this.objectMapper = objectMapper;
}
public YellowFoxD8BookingDto map(ResultSet rs, String tenantKey, String sourceInstanceKey, String tenantProviderSettingKey) throws SQLException {
String eventId = rs.getString("eventid");
OffsetDateTime occurredAt = rs.getObject("utc", OffsetDateTime.class);
Integer vehicleId = getInteger(rs, "vehicle_id");
Integer driverId = getInteger(rs, "driver_id");
String vehicleVrn = rs.getString("vehicle_vrn");
String vehicleVin = rs.getString("vehicle_vin");
String driverCard = rs.getString("driver_card_number");
Integer fleetId = getInteger(rs, "fleet_id");
String fleetName = rs.getString("fleet_name");
Integer odometer = getInteger(rs, "odometer");
Map<String, Object> payload = payload(rs.getString("payload"));
put(payload, "yellowFoxEventId", eventId);
put(payload, "yellowFoxOdometerRaw", odometer);
put(payload, "vehicleVrn", vehicleVrn);
put(payload, "vehicleVin", vehicleVin);
put(payload, "driverCardNumber", driverCard);
put(payload, "driverFirstName", rs.getString("driver_firstname"));
put(payload, "driverLastName", rs.getString("driver_lastname"));
put(payload, "fleetId", fleetId);
put(payload, "fleetName", fleetName);
put(payload, "telematicProviderId", getInteger(rs, "telematic_provider_id"));
put(payload, "telematicProviderName", rs.getString("telematic_provider_name"));
DriverRefDto driverRef = driverId == null && isBlank(driverCard)
? null
: new DriverRefDto(driverId == null ? null : driverId.toString(), new DriverCardRefDto(null, driverCard));
VehicleRefDto vehicleRef = vehicleId == null && isBlank(vehicleVin) && isBlank(vehicleVrn)
? null
: new VehicleRefDto(
vehicleId == null ? null : vehicleId.toString(),
vehicleVin,
vehicleId == null ? null : vehicleId.toString(),
new VehicleRegistrationRefDto(null, vehicleVrn)
);
return new YellowFoxD8BookingDto(
tenantKey,
sourceInstanceKey,
tenantProviderSettingKey,
fleetId == null ? null : fleetId.toString(),
fleetName,
eventId,
rs.getString("key"),
getInteger(rs, "ignition"),
getInteger(rs, "eventtype"),
getInteger(rs, "state"),
occurredAt,
null,
driverRef,
vehicleRef,
odometer == null ? null : odometer.longValue() * 1000L,
rs.getBigDecimal("latitude"),
rs.getBigDecimal("longitude"),
payload
);
}
private Integer getInteger(ResultSet rs, String column) throws SQLException {
int value = rs.getInt(column);
return rs.wasNull() ? null : value;
}
private Map<String, Object> payload(String json) {
if (json == null || json.isBlank()) {
return new LinkedHashMap<>();
}
try {
return new LinkedHashMap<>(objectMapper.readValue(json, MAP_TYPE));
} catch (Exception ex) {
Map<String, Object> fallback = new LinkedHashMap<>();
fallback.put("rawPayload", json);
fallback.put("payloadParseError", ex.getMessage());
return fallback;
}
}
private void put(Map<String, Object> target, String key, Object value) {
if (value != null) {
target.put(key, value instanceof BigDecimal bd ? bd.stripTrailingZeros().toPlainString() : value);
}
}
private boolean isBlank(String value) {
return value == null || value.isBlank();
}
}

View File

@ -0,0 +1,62 @@
package at.procon.eventhub.yellowfox.service;
import at.procon.eventhub.config.EventHubProperties;
import at.procon.eventhub.dto.AcquisitionStrategy;
import at.procon.eventhub.dto.ImportMode;
import at.procon.eventhub.dto.ImportScopeDto;
import at.procon.eventhub.importing.AbstractConfiguredImportPlanService;
import at.procon.eventhub.importing.ConfiguredImportPlanDto;
import at.procon.eventhub.yellowfox.dto.ConfiguredYellowFoxD8ImportPlanDto;
import at.procon.eventhub.yellowfox.dto.YellowFoxD8ImportRequest;
import org.springframework.stereotype.Service;
@Service
public class YellowFoxD8ConfiguredImportPlanService
extends AbstractConfiguredImportPlanService<YellowFoxD8ImportRequest, ConfiguredYellowFoxD8ImportPlanDto> {
public YellowFoxD8ConfiguredImportPlanService(EventHubProperties properties) {
super(() -> properties.getYellowFox().getImportPlans(), "yellowfox-d8");
}
@Override
protected YellowFoxD8ImportRequest buildRequest(
EventHubProperties.ConfiguredImportPlan plan,
ImportMode mode,
AcquisitionStrategy strategy,
ImportScopeDto scope
) {
return new YellowFoxD8ImportRequest(
plan.getTenantKey(),
plan.getEventSource(),
plan.getSourceGroup(),
scope,
plan.getEventFamilies(),
mode,
false,
strategy
);
}
@Override
protected ConfiguredYellowFoxD8ImportPlanDto toDto(EventHubProperties.ConfiguredImportPlan plan) {
ConfiguredImportPlanDto dto = genericDto(plan);
return new ConfiguredYellowFoxD8ImportPlanDto(
dto.planKey(),
dto.enabled(),
dto.cron(),
dto.tenantKey(),
dto.eventSource(),
dto.sourceGroup(),
dto.importScope(),
dto.eventFamilies(),
dto.initialMode(),
dto.scheduledMode(),
dto.initialStrategy(),
dto.scheduledStrategy(),
false,
dto.initialOccurredFrom(),
dto.initialOccurredTo(),
dto.runInitialOnStartup()
);
}
}

View File

@ -0,0 +1,22 @@
package at.procon.eventhub.yellowfox.service;
import at.procon.eventhub.importing.ImportPlanItemDto;
import at.procon.eventhub.importing.ImportTimeChunkDto;
import at.procon.eventhub.importing.extraction.ExtractionBatchExecutor;
import at.procon.eventhub.yellowfox.dto.YellowFoxD8ExtractionBatchResultDto;
import at.procon.eventhub.yellowfox.dto.YellowFoxD8ImportRequest;
import java.util.UUID;
public interface YellowFoxD8ExtractionBatchExecutor
extends ExtractionBatchExecutor<YellowFoxD8ImportRequest, YellowFoxD8ExtractionBatchResultDto> {
@Override
YellowFoxD8ExtractionBatchResultDto execute(
UUID importRunId,
UUID packageId,
int eventSourceId,
YellowFoxD8ImportRequest request,
ImportPlanItemDto planItem,
ImportTimeChunkDto chunk
);
}

View File

@ -0,0 +1,47 @@
package at.procon.eventhub.yellowfox.service;
import at.procon.eventhub.dto.EventHubEventDto;
import at.procon.eventhub.yellowfox.dto.YellowFoxD8BookingDto;
import java.util.HashMap;
import java.util.Map;
import org.springframework.stereotype.Component;
@Component
public class YellowFoxD8IgnitionTransitionDetector {
private final YellowFoxD8BookingEventMapper mapper;
public YellowFoxD8IgnitionTransitionDetector(YellowFoxD8BookingEventMapper mapper) {
this.mapper = mapper;
}
public Session newSession(boolean emitInitialSnapshot) {
return new Session(mapper, emitInitialSnapshot);
}
public static class Session {
private final YellowFoxD8BookingEventMapper mapper;
private final boolean emitInitialSnapshot;
private final Map<String, Integer> lastIgnitionByVehicle = new HashMap<>();
private Session(YellowFoxD8BookingEventMapper mapper, boolean emitInitialSnapshot) {
this.mapper = mapper;
this.emitInitialSnapshot = emitInitialSnapshot;
}
public EventHubEventDto detect(YellowFoxD8BookingDto booking) {
if (booking == null || booking.ignition() == null || booking.vehicleRef() == null || !booking.vehicleRef().hasAnyReference()) {
return null;
}
String vehicleKey = booking.vehicleRef().stableKey();
Integer previous = lastIgnitionByVehicle.put(vehicleKey, booking.ignition());
if (previous == null) {
return emitInitialSnapshot ? mapper.mapIgnitionTransition(booking, null) : null;
}
if (!previous.equals(booking.ignition())) {
return mapper.mapIgnitionTransition(booking, previous);
}
return null;
}
}
}

View File

@ -0,0 +1,52 @@
package at.procon.eventhub.yellowfox.service;
import at.procon.eventhub.config.EventHubProperties;
import at.procon.eventhub.dto.AcquisitionStrategy;
import at.procon.eventhub.dto.EventFamily;
import at.procon.eventhub.importing.ImportChunkPlanner;
import at.procon.eventhub.importing.ImportPlanDto;
import at.procon.eventhub.importing.ImportPlanItemDto;
import at.procon.eventhub.yellowfox.dto.YellowFoxD8ImportRequest;
import java.util.List;
import org.springframework.stereotype.Service;
@Service
public class YellowFoxD8ImportPlanService {
public static final String SOURCE_KIND = "TELEMATICS_PLATFORM";
public static final String EXTRACTION_CODE = "YELLOWFOX_D8_BOOKING";
private final EventHubProperties properties;
private final ImportChunkPlanner chunkPlanner;
public YellowFoxD8ImportPlanService(EventHubProperties properties, ImportChunkPlanner chunkPlanner) {
this.properties = properties;
this.chunkPlanner = chunkPlanner;
}
public ImportPlanDto createPlan(YellowFoxD8ImportRequest request) {
return new ImportPlanDto(
request.tenantKey(),
request.mode(),
request.acquisitionStrategy(),
request.refreshMasterDataFirst(),
request.importScope(),
request.sourceGroup(),
request.eventSource(),
chunkPlanner.chunksFor(request, properties.getYellowFox().getDefaultChunkDays()),
List.of(item(request.acquisitionStrategy()))
);
}
private ImportPlanItemDto item(AcquisitionStrategy strategy) {
return new ImportPlanItemDto(
EventFamily.DRIVER_ACTIVITY,
SOURCE_KIND,
EXTRACTION_CODE,
List.of("data.d8_booking", "data.vehicle", "data.driver", "data.fleet", "data.telematic_provider"),
"VEHICLE",
"YellowFox D8 booking card/activity events with derived ignition transitions",
strategy
);
}
}

View File

@ -0,0 +1,79 @@
package at.procon.eventhub.yellowfox.service;
import at.procon.eventhub.config.EventHubProperties;
import at.procon.eventhub.dto.SchedulerTriggerMode;
import at.procon.eventhub.importing.AbstractImportScheduler;
import at.procon.eventhub.yellowfox.dto.YellowFoxD8ImportRequest;
import java.util.List;
import org.springframework.boot.context.event.ApplicationReadyEvent;
import org.springframework.context.event.EventListener;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Service;
@Service
public class YellowFoxD8ImportScheduler extends AbstractImportScheduler<YellowFoxD8ImportRequest> {
private final EventHubProperties properties;
private final YellowFoxD8ConfiguredImportPlanService configuredPlanService;
private final YellowFoxD8ImportExecutionService executionService;
public YellowFoxD8ImportScheduler(
EventHubProperties properties,
YellowFoxD8ConfiguredImportPlanService configuredPlanService,
YellowFoxD8ImportExecutionService executionService
) {
this.properties = properties;
this.configuredPlanService = configuredPlanService;
this.executionService = executionService;
}
@EventListener(ApplicationReadyEvent.class)
public void triggerInitialPlansOnStartup() {
super.triggerInitialPlansOnStartup();
}
@Scheduled(fixedDelayString = "${eventhub.yellow-fox.scheduler-poll-interval-ms:60000}")
public void pollConfiguredPlans() {
super.pollConfiguredPlans();
}
@Override
protected boolean schedulerEnabled() {
return properties.getYellowFox().isSchedulerEnabled();
}
@Override
protected List<EventHubProperties.ConfiguredImportPlan> configuredPlans() {
return properties.getYellowFox().getImportPlans();
}
@Override
protected SchedulerTriggerMode schedulerTriggerMode() {
return properties.getYellowFox().getSchedulerTriggerMode();
}
@Override
protected YellowFoxD8ImportRequest createInitialRequest(EventHubProperties.ConfiguredImportPlan plan) {
return configuredPlanService.createInitialRequest(plan);
}
@Override
protected YellowFoxD8ImportRequest createScheduledRequest(EventHubProperties.ConfiguredImportPlan plan) {
return configuredPlanService.createScheduledRequest(plan);
}
@Override
protected void startImport(YellowFoxD8ImportRequest request) {
executionService.startImport(request);
}
@Override
protected void startAndExecuteImport(YellowFoxD8ImportRequest request) {
executionService.startAndExecuteImport(request);
}
@Override
protected String providerName() {
return "yellowfox-d8";
}
}

View File

@ -2,17 +2,17 @@ spring:
application:
name: eventhub-ingestion-service
datasource:
url: jdbc:postgresql://localhost:5432/eventhub
username: postgres
password: P54!pcd#Wi
url: jdbc:postgresql://${DB_HOST:localhost}:${DB_PORT:5432}/${DB_NAME:eventhub}
username: ${DB_USER:}
password: ${DB_PASSWORD:}
hikari:
maximum-pool-size: 16
minimum-idle: 4
connection-timeout: 30000
validation-timeout: 5000
idle-timeout: 60000
keepalive-time: 30000
max-lifetime: 90000
idle-timeout: 300000
keepalive-time: 120000
max-lifetime: 540000
flyway:
enabled: true
default-schema: eventhub
@ -54,27 +54,33 @@ eventhub:
default-chunk-days: 1
occurred-at-overlap: 7d
# Configure this block to enable JdbcTachographExtractionBatchExecutor.
# Set TACHOGRAPH_DB_JDBC_URL to enable JdbcTachographExtractionBatchExecutor.
datasource:
jdbc-url: jdbc:sqlserver://db.bytebar.eu:22996;databaseName=ByteBarDriverSettlement;trustServerCertificate=true
username: ReadOnly
password: p2=race!
jdbc-url: ${TACHOGRAPH_DB_JDBC_URL:}
username: ${TACHOGRAPH_DB_USER:}
password: ${TACHOGRAPH_DB_PASSWORD:}
driver-class-name: com.microsoft.sqlserver.jdbc.SQLServerDriver
# Enables the scheduler that regularly triggers configured tachograph import plans.
# Default is safe: no scheduled import starts unless explicitly enabled.
scheduler-enabled: true
scheduler-poll-interval-ms: 60000
scheduler-poll-interval-ms: 3600000
# PLAN_ONLY creates import_run + planned extraction packages.
# EXECUTE also invokes the configured TachographExtractionBatchExecutor.
scheduler-trigger-mode: EXECUTE
# JDBC extraction handoff mode:
# SYNC_DIRECT = persist controlled JDBC batches directly, cursor-safe default.
# CAMEL_ROUTE = persist the same controlled batches through direct:eventhub-batch-persist-input.
jdbc-extraction-ingest-mode: ${TACHOGRAPH_JDBC_EXTRACTION_INGEST_MODE:SYNC_DIRECT}
# Example plan. Keep disabled until the tachograph datasource/extractor is wired.
import-plans:
- plan-key: kralowetz-tachograph-org-147
- plan-key: tachograph-org-14708
enabled: true
cron: "0 15 * * * *" # hourly at minute 15
tenant-key: kralowetz
tenant-key: Procon
event-source:
provider-key: TACHOGRAPH
source-kind: MIXED
@ -83,16 +89,16 @@ eventhub:
tenant-provider-setting-key: ByteBar-DriverSettlement
source-group:
type: ORGANISATION
source-entity-id: "147"
code: "147"
name: Kralowetz root organisation
source-entity-id: "14708"
code: "14708"
name: Zeller root organisation
import-scope:
type: SOURCE_ORGANISATION_SUBTREE
root-source-organisation:
type: ORGANISATION
source-entity-id: "147"
code: "147"
name: Kralowetz root organisation
source-entity-id: "14708"
code: "14708"
name: Zeller root organisation
include-children: true
occurred-from: null
occurred-to: null
@ -109,7 +115,56 @@ eventhub:
scheduled-mode: INCREMENTAL_UPDATE
initial-strategy: OCCURRED_AT_WINDOW_WITH_OVERLAP
scheduled-strategy: SOURCE_PACKAGE_WATERMARK
refresh-master-data-first: false
refresh-master-data-first: true
initial-occurred-from: "2026-01-21T00:00:00+01:00"
initial-occurred-to: "2026-01-25T00:00:00+01:00"
initial-occurred-to:
run-initial-on-startup: true
esper-poc:
activity-merge-mode: JAVA
shift-resolution-mode: JAVA
operating-period-evaluation:
operating-split-idle-hours: 7
significant-driving-minutes: 3
merge-gap-seconds: 0
gap-detection-tolerance-seconds: 0
unknown-treatment-mode: AS_BREAK_REST
yellow-fox:
default-chunk-days: 1
occurred-at-overlap: 2h
emit-initial-ignition-snapshot: false
datasource:
jdbc-url: ${YELLOWFOX_DB_JDBC_URL:}
username: ${YELLOWFOX_DB_USERNAME:}
password: ${YELLOWFOX_DB_PASSWORD:}
driver-class-name: org.postgresql.Driver
scheduler-enabled: false
scheduler-poll-interval-ms: 60000
scheduler-trigger-mode: PLAN_ONLY
import-plans:
- plan-key: yellowfox-d8-default
enabled: false
cron: "0 */5 * * * *"
tenant-key: default
event-source:
provider-key: YELLOWFOX
source-kind: TELEMATICS_PLATFORM
source-key: YELLOWFOX_D8
source-instance-key: logistics-db-prod
tenant-provider-setting-key: yellowfox-main
import-scope:
type: TENANT_ALL
include-children: false
event-families:
- DRIVER_ACTIVITY
- DRIVER_CARD
initial-mode: INITIAL_BACKFILL
scheduled-mode: INCREMENTAL_UPDATE
initial-strategy: OCCURRED_AT_WINDOW_WITH_OVERLAP
scheduled-strategy: SOURCE_ROW_WATERMARK
refresh-master-data-first: false
run-initial-on-startup: false

View File

@ -0,0 +1,15 @@
create index if not exists idx_event_detail_yellowfox_slot
on eventhub.event_detail(detail_type, (attributes ->> 'slot'), event_occurred_at)
where detail_type in ('DRIVER_ACTIVITY', 'DRIVER_CARD');
create index if not exists idx_event_detail_yellowfox_eventtype_state
on eventhub.event_detail(
(attributes ->> 'yellowFoxEventType'),
(attributes ->> 'yellowFoxState'),
event_occurred_at
)
where attributes ? 'yellowFoxEventType';
create index if not exists idx_event_detail_yellowfox_ignition
on eventhub.event_detail(detail_type, (attributes ->> 'ignitionState'), event_occurred_at)
where attributes ? 'ignitionState';

View File

@ -0,0 +1,137 @@
create table if not exists eventhub.driver (
id uuid primary key,
tenant_key text not null,
event_source_id integer not null references eventhub.event_source(id),
source_driver_entity_id text,
card_nation text,
card_number text,
first_names text,
last_name text,
birth_date date,
source_updated_at timestamptz,
payload jsonb not null default '{}'::jsonb,
created_at timestamptz not null default now(),
updated_at timestamptz not null default now(),
constraint chk_driver_card_nation_when_number
check (card_number is null or card_nation is not null)
);
insert into eventhub.driver(
id, tenant_key, event_source_id, source_driver_entity_id,
card_nation, card_number, first_names, last_name, birth_date,
source_updated_at, payload, created_at, updated_at
)
select gen_random_uuid(),
sme.tenant_key,
sme.event_source_id,
case
when sme.source_entity_id like 'DRIVER_CARD:%' then null
else sme.source_entity_id
end as source_driver_entity_id,
coalesce(
nullif(trim(sme.payload ->> 'card_nation'), ''),
nullif(trim(sme.payload ->> 'driver_card_nation'), ''),
nullif(split_part(case when sme.source_entity_id like 'DRIVER_CARD:%' then substring(sme.source_entity_id from length('DRIVER_CARD:') + 1) else null end, ':', 1), '')
) as card_nation,
coalesce(
nullif(trim(sme.payload ->> 'card_number'), ''),
nullif(trim(sme.payload ->> 'driver_card_number'), ''),
nullif(substring(case when sme.source_entity_id like 'DRIVER_CARD:%' then substring(sme.source_entity_id from length('DRIVER_CARD:') + 1) else null end from position(':' in case when sme.source_entity_id like 'DRIVER_CARD:%' then substring(sme.source_entity_id from length('DRIVER_CARD:') + 1) else '' end) + 1), '')
) as card_number,
coalesce(
nullif(trim(sme.payload ->> 'first_names'), ''),
nullif(trim(sme.payload ->> 'firstnames'), '')
) as first_names,
coalesce(
nullif(trim(sme.payload ->> 'last_name'), ''),
nullif(trim(sme.payload ->> 'surname'), '')
) as last_name,
cast(nullif(trim(sme.payload ->> 'birth_date'), '') as date) as birth_date,
sme.source_updated_at,
sme.payload,
sme.created_at,
sme.updated_at
from eventhub.source_master_entity sme
where sme.entity_type = 'DRIVER'
and not exists (
select 1
from eventhub.driver existing
where existing.tenant_key = sme.tenant_key
and existing.event_source_id = sme.event_source_id
and existing.source_driver_entity_id is not distinct from case
when sme.source_entity_id like 'DRIVER_CARD:%' then null
else sme.source_entity_id
end
and existing.card_nation is not distinct from coalesce(
nullif(trim(sme.payload ->> 'card_nation'), ''),
nullif(trim(sme.payload ->> 'driver_card_nation'), ''),
nullif(split_part(case when sme.source_entity_id like 'DRIVER_CARD:%' then substring(sme.source_entity_id from length('DRIVER_CARD:') + 1) else null end, ':', 1), '')
)
and existing.card_number is not distinct from coalesce(
nullif(trim(sme.payload ->> 'card_number'), ''),
nullif(trim(sme.payload ->> 'driver_card_number'), ''),
nullif(substring(case when sme.source_entity_id like 'DRIVER_CARD:%' then substring(sme.source_entity_id from length('DRIVER_CARD:') + 1) else null end from position(':' in case when sme.source_entity_id like 'DRIVER_CARD:%' then substring(sme.source_entity_id from length('DRIVER_CARD:') + 1) else '' end) + 1), '')
)
);
alter table eventhub.event
add column if not exists driver_id uuid references eventhub.driver(id);
with source_driver_map as (
select distinct on (sme.id)
sme.id as source_master_entity_id,
driver.id as driver_id
from eventhub.source_master_entity sme
join eventhub.driver driver
on driver.tenant_key = sme.tenant_key
and driver.event_source_id = sme.event_source_id
and (
(
sme.source_entity_id not like 'DRIVER_CARD:%'
and driver.source_driver_entity_id = sme.source_entity_id
) or (
sme.source_entity_id like 'DRIVER_CARD:%'
and driver.card_nation is not distinct from coalesce(
nullif(trim(sme.payload ->> 'card_nation'), ''),
nullif(trim(sme.payload ->> 'driver_card_nation'), ''),
nullif(split_part(substring(sme.source_entity_id from length('DRIVER_CARD:') + 1), ':', 1), '')
)
and driver.card_number is not distinct from coalesce(
nullif(trim(sme.payload ->> 'card_number'), ''),
nullif(trim(sme.payload ->> 'driver_card_number'), ''),
nullif(substring(substring(sme.source_entity_id from length('DRIVER_CARD:') + 1) from position(':' in substring(sme.source_entity_id from length('DRIVER_CARD:') + 1)) + 1), '')
)
)
)
where sme.entity_type = 'DRIVER'
order by sme.id, driver.updated_at desc
)
update eventhub.event event
set driver_id = map.driver_id
from source_driver_map map
where event.driver_entity_id = map.source_master_entity_id
and event.driver_id is null;
create index if not exists idx_driver_source_entity
on eventhub.driver(tenant_key, event_source_id, source_driver_entity_id)
where source_driver_entity_id is not null;
create index if not exists idx_driver_card
on eventhub.driver(tenant_key, event_source_id, card_nation, card_number)
where card_number is not null;
create index if not exists idx_event_driver_time
on eventhub.event(driver_id, occurred_at desc)
where driver_id is not null;
alter table eventhub.event
drop constraint if exists chk_event_driver_or_vehicle_ref;
alter table eventhub.event
add constraint chk_event_driver_or_vehicle_ref
check (
driver_id is not null
or driver_entity_id is not null
or vehicle_id is not null
or vehicle_registration_id is not null
);

View File

@ -0,0 +1,239 @@
/*
* Repairs and normalizes tachograph driver aggregates after introducing eventhub.driver.
*
* What it does:
* 1. Ensures tachograph DRIVER master-data payload carries last_name while keeping source_master_entity.display_name unchanged.
* 2. Upserts eventhub.driver rows from MASTER_DATA DRIVER entities.
* 3. Projects card nation/number onto eventhub.driver from DRIVER_CARD_DRIVER relations.
* 4. Remaps event.driver_id from provisional card-only drivers to proper source-driver aggregates when possible.
* 5. Deletes now-unreferenced provisional tachograph driver rows with no source_driver_entity_id.
*
* Assumptions:
* - Tachograph master-data source is provider_key=TACHOGRAPH, source_kind=MASTER_DATA, source_key=TACHOGRAPH_MASTER_DATA.
* - eventhub.driver and event.driver_id already exist.
*/
-- 1) Keep display_name, but ensure DRIVER payload has last_name.
with master_sources as (
select es.id, es.tenant_key
from eventhub.event_source es
where es.provider_key = 'TACHOGRAPH'
and es.source_kind = 'MASTER_DATA'
and es.source_key = 'TACHOGRAPH_MASTER_DATA'
),
updated_master_payload as (
update eventhub.source_master_entity sme
set payload = jsonb_strip_nulls(
sme.payload
|| jsonb_build_object(
'first_names', coalesce(sme.payload ->> 'first_names', sme.payload ->> 'firstnames'),
'last_name', coalesce(sme.payload ->> 'last_name', sme.payload ->> 'surname')
)
),
updated_at = now()
from master_sources ms
where sme.tenant_key = ms.tenant_key
and sme.event_source_id = ms.id
and sme.entity_type = 'DRIVER'
returning sme.id
)
select count(*) as updated_master_payload
from updated_master_payload;
-- 2) Upsert driver aggregates from tachograph master data.
with master_sources as (
select es.id,
es.tenant_key,
es.source_instance_key,
coalesce(es.tenant_provider_setting_key, '') as tenant_provider_setting_key
from eventhub.event_source es
where es.provider_key = 'TACHOGRAPH'
and es.source_kind = 'MASTER_DATA'
and es.source_key = 'TACHOGRAPH_MASTER_DATA'
),
master_drivers as (
select ms.id as master_event_source_id,
ms.tenant_key,
ms.source_instance_key,
ms.tenant_provider_setting_key,
d.source_entity_id as source_driver_entity_id,
coalesce(nullif(trim(d.payload ->> 'first_names'), ''), nullif(trim(d.payload ->> 'firstnames'), '')) as first_names,
coalesce(nullif(trim(d.payload ->> 'last_name'), ''), nullif(trim(d.payload ->> 'surname'), '')) as last_name,
cast(nullif(trim(d.payload ->> 'birth_date'), '') as date) as birth_date,
d.source_updated_at,
d.payload
from master_sources ms
join eventhub.source_master_entity d
on d.tenant_key = ms.tenant_key
and d.event_source_id = ms.id
and d.entity_type = 'DRIVER'
and d.source_entity_id not like 'DRIVER_CARD:%'
),
compatible_targets as (
select md.*,
es.id as target_event_source_id
from master_drivers md
join eventhub.event_source es
on es.tenant_key = md.tenant_key
and es.provider_key = 'TACHOGRAPH'
and es.source_instance_key = md.source_instance_key
and coalesce(es.tenant_provider_setting_key, '') = md.tenant_provider_setting_key
),
updated_drivers as (
update eventhub.driver driver
set first_names = coalesce(ct.first_names, driver.first_names),
last_name = coalesce(ct.last_name, driver.last_name),
birth_date = coalesce(ct.birth_date, driver.birth_date),
source_updated_at = ct.source_updated_at,
payload = driver.payload || ct.payload,
updated_at = now()
from compatible_targets ct
where driver.tenant_key = ct.tenant_key
and driver.event_source_id = ct.target_event_source_id
and driver.source_driver_entity_id = ct.source_driver_entity_id
returning driver.id
),
inserted_drivers as (
insert into eventhub.driver(
id, tenant_key, event_source_id, source_driver_entity_id,
first_names, last_name, birth_date, source_updated_at, payload, updated_at
)
select gen_random_uuid(),
ct.tenant_key,
ct.target_event_source_id,
ct.source_driver_entity_id,
ct.first_names,
ct.last_name,
ct.birth_date,
ct.source_updated_at,
ct.payload,
now()
from compatible_targets ct
where not exists (
select 1
from eventhub.driver existing
where existing.tenant_key = ct.tenant_key
and existing.event_source_id = ct.target_event_source_id
and existing.source_driver_entity_id = ct.source_driver_entity_id
)
returning id
)
select (select count(*) from updated_drivers) as updated_drivers,
(select count(*) from inserted_drivers) as inserted_drivers;
-- 3) Project driver-card identifiers from master-data relations.
with master_sources as (
select es.id,
es.tenant_key,
es.source_instance_key,
coalesce(es.tenant_provider_setting_key, '') as tenant_provider_setting_key
from eventhub.event_source es
where es.provider_key = 'TACHOGRAPH'
and es.source_kind = 'MASTER_DATA'
and es.source_key = 'TACHOGRAPH_MASTER_DATA'
),
card_projection as (
select distinct on (ms.tenant_key, ms.source_instance_key, ms.tenant_provider_setting_key, rel.to_source_entity_id)
ms.tenant_key,
ms.source_instance_key,
ms.tenant_provider_setting_key,
rel.to_source_entity_id as source_driver_entity_id,
nullif(trim(card.payload ->> 'card_nation'), '') as card_nation,
nullif(trim(card.payload ->> 'card_number'), '') as card_number,
rel.source_updated_at
from master_sources ms
join eventhub.source_master_relation rel
on rel.tenant_key = ms.tenant_key
and rel.event_source_id = ms.id
and rel.relation_type = 'DRIVER_CARD_DRIVER'
and rel.from_entity_type = 'DRIVER_CARD'
and rel.to_entity_type = 'DRIVER'
join eventhub.source_master_entity card
on card.tenant_key = ms.tenant_key
and card.event_source_id = ms.id
and card.entity_type = 'DRIVER_CARD'
and card.source_entity_id = rel.from_source_entity_id
order by ms.tenant_key,
ms.source_instance_key,
ms.tenant_provider_setting_key,
rel.to_source_entity_id,
rel.valid_to desc nulls last,
rel.valid_from desc nulls last,
rel.updated_at desc
),
updated_driver_cards as (
update eventhub.driver driver
set card_nation = coalesce(driver.card_nation, projection.card_nation),
card_number = coalesce(driver.card_number, projection.card_number),
source_updated_at = coalesce(projection.source_updated_at, driver.source_updated_at),
updated_at = now()
from card_projection projection
join eventhub.event_source es
on es.id = driver.event_source_id
where driver.tenant_key = projection.tenant_key
and es.provider_key = 'TACHOGRAPH'
and es.source_instance_key = projection.source_instance_key
and coalesce(es.tenant_provider_setting_key, '') = projection.tenant_provider_setting_key
and driver.source_driver_entity_id = projection.source_driver_entity_id
and (
(driver.card_nation is null and projection.card_nation is not null)
or (driver.card_number is null and projection.card_number is not null)
)
returning driver.id
)
select count(*) as updated_driver_cards
from updated_driver_cards;
-- 4) Remap events from provisional card-only drivers to proper source-driver aggregates.
with provisional_to_real as (
select provisional.id as provisional_driver_id,
real.id as real_driver_id
from eventhub.driver provisional
join eventhub.event_source provisional_source
on provisional_source.id = provisional.event_source_id
and provisional_source.provider_key = 'TACHOGRAPH'
join eventhub.driver real
on real.tenant_key = provisional.tenant_key
and real.source_driver_entity_id is not null
and real.card_nation = provisional.card_nation
and real.card_number = provisional.card_number
join eventhub.event_source real_source
on real_source.id = real.event_source_id
and real_source.provider_key = provisional_source.provider_key
and real_source.tenant_key = provisional_source.tenant_key
and real_source.source_instance_key = provisional_source.source_instance_key
and coalesce(real_source.tenant_provider_setting_key, '') = coalesce(provisional_source.tenant_provider_setting_key, '')
where provisional.source_driver_entity_id is null
and provisional.card_nation is not null
and provisional.card_number is not null
and provisional.id <> real.id
),
updated_events as (
update eventhub.event e
set driver_id = map.real_driver_id
from provisional_to_real map
where e.driver_id = map.provisional_driver_id
and e.driver_id <> map.real_driver_id
returning e.id
)
select count(*) as remapped_events
from updated_events;
-- 5) Delete now-unreferenced provisional tachograph driver rows.
with deleted_drivers as (
delete from eventhub.driver driver
using eventhub.event_source es
where es.id = driver.event_source_id
and es.provider_key = 'TACHOGRAPH'
and driver.source_driver_entity_id is null
and driver.card_nation is not null
and driver.card_number is not null
and not exists (
select 1
from eventhub.event e
where e.driver_id = driver.id
)
returning driver.id
)
select count(*) as deleted_provisional_drivers
from deleted_drivers;

View File

@ -98,6 +98,8 @@ Base as (
left join dbo.Vehicle v on v.ID = cvu.ID_Vehicle
left join dbo.Nation vn on vn.ID = v.ID_Nation
)
,
Extracted as (
select
cast(base.ID as varchar(128)) as source_row_id,
concat('TACHOGRAPH:CARD_ACTIVITY:', base.ID, ':', evt.lifecycle) as external_source_event_id,
@ -134,6 +136,7 @@ select
'DRIVER_CARD' as source_package_kind,
cast(base.source_package_id_raw as varchar(128)) as source_package_id,
base.source_package_id_raw as source_package_sort_id,
cast(base.source_package_entity_id_raw as varchar(128)) as source_package_entity_id,
base.source_package_period_from,
base.source_package_period_to,
@ -148,3 +151,20 @@ where (:occurredFrom is null or evt.occurred_at >= :occurredFrom)
/*
* Organisation filter: driver membership in GetOrganisationTree(null, :organisationId, 0, null).
*/
)
select *
from Extracted extracted
where (
:sourcePackageWatermarkEnabled = 0
or :lastSourcePackageImportedAt is null
or extracted.source_package_imported_at is null
or extracted.source_package_imported_at > :lastSourcePackageImportedAt
or (
extracted.source_package_imported_at = :lastSourcePackageImportedAt
and (
:lastSourcePackageIdNumeric is null
or extracted.source_package_sort_id is null
or extracted.source_package_sort_id > :lastSourcePackageIdNumeric
)
)
)

View File

@ -65,6 +65,8 @@ Base as (
)
)
)
,
Extracted as (
select
cast(base.ID as varchar(128)) as source_row_id,
concat('TACHOGRAPH:CARD_BORDER_CROSSING:', base.ID) as external_source_event_id,
@ -89,8 +91,26 @@ select
'DRIVER_CARD' as source_package_kind,
cast(base.source_package_id_raw as varchar(128)) as source_package_id,
base.source_package_id_raw as source_package_sort_id,
cast(base.source_package_entity_id_raw as varchar(128)) as source_package_entity_id,
base.source_package_period_from,
base.source_package_period_to,
base.source_package_imported_at
from Base base
)
select *
from Extracted extracted
where (
:sourcePackageWatermarkEnabled = 0
or :lastSourcePackageImportedAt is null
or extracted.source_package_imported_at is null
or extracted.source_package_imported_at > :lastSourcePackageImportedAt
or (
extracted.source_package_imported_at = :lastSourcePackageImportedAt
and (
:lastSourcePackageIdNumeric is null
or extracted.source_package_sort_id is null
or extracted.source_package_sort_id > :lastSourcePackageIdNumeric
)
)
)

View File

@ -62,6 +62,8 @@ Base as (
)
)
)
,
Extracted as (
select
cast(base.ID as varchar(128)) as source_row_id,
concat('TACHOGRAPH:CARD_LOAD_UNLOAD:', base.ID) as external_source_event_id,
@ -94,8 +96,26 @@ select
'DRIVER_CARD' as source_package_kind,
cast(base.source_package_id_raw as varchar(128)) as source_package_id,
base.source_package_id_raw as source_package_sort_id,
cast(base.source_package_entity_id_raw as varchar(128)) as source_package_entity_id,
base.source_package_period_from,
base.source_package_period_to,
base.source_package_imported_at
from Base base
)
select *
from Extracted extracted
where (
:sourcePackageWatermarkEnabled = 0
or :lastSourcePackageImportedAt is null
or extracted.source_package_imported_at is null
or extracted.source_package_imported_at > :lastSourcePackageImportedAt
or (
extracted.source_package_imported_at = :lastSourcePackageImportedAt
and (
:lastSourcePackageIdNumeric is null
or extracted.source_package_sort_id is null
or extracted.source_package_sort_id > :lastSourcePackageIdNumeric
)
)
)

View File

@ -66,6 +66,8 @@ Base as (
)
)
)
,
Extracted as (
select
cast(base.ID as varchar(128)) as source_row_id,
concat('TACHOGRAPH:CARD_PLACE:', base.ID) as external_source_event_id,
@ -93,8 +95,26 @@ select
'DRIVER_CARD' as source_package_kind,
cast(base.source_package_id_raw as varchar(128)) as source_package_id,
base.source_package_id_raw as source_package_sort_id,
cast(base.source_package_entity_id_raw as varchar(128)) as source_package_entity_id,
base.source_package_period_from,
base.source_package_period_to,
base.source_package_imported_at
from Base base
)
select *
from Extracted extracted
where (
:sourcePackageWatermarkEnabled = 0
or :lastSourcePackageImportedAt is null
or extracted.source_package_imported_at is null
or extracted.source_package_imported_at > :lastSourcePackageImportedAt
or (
extracted.source_package_imported_at = :lastSourcePackageImportedAt
and (
:lastSourcePackageIdNumeric is null
or extracted.source_package_sort_id is null
or extracted.source_package_sort_id > :lastSourcePackageIdNumeric
)
)
)

View File

@ -61,6 +61,8 @@ Base as (
)
)
)
,
Extracted as (
select
cast(base.ID as varchar(128)) as source_row_id,
concat('TACHOGRAPH:CARD_POSITION:', base.ID) as external_source_event_id,
@ -83,8 +85,26 @@ select
'DRIVER_CARD' as source_package_kind,
cast(base.source_package_id_raw as varchar(128)) as source_package_id,
base.source_package_id_raw as source_package_sort_id,
cast(base.source_package_entity_id_raw as varchar(128)) as source_package_entity_id,
base.source_package_period_from,
base.source_package_period_to,
base.source_package_imported_at
from Base base
)
select *
from Extracted extracted
where (
:sourcePackageWatermarkEnabled = 0
or :lastSourcePackageImportedAt is null
or extracted.source_package_imported_at is null
or extracted.source_package_imported_at > :lastSourcePackageImportedAt
or (
extracted.source_package_imported_at = :lastSourcePackageImportedAt
and (
:lastSourcePackageIdNumeric is null
or extracted.source_package_sort_id is null
or extracted.source_package_sort_id > :lastSourcePackageIdNumeric
)
)
)

View File

@ -59,6 +59,8 @@ Base as (
)
)
)
,
Extracted as (
select
cast(base.ID as varchar(128)) as source_row_id,
concat('TACHOGRAPH:CARD_SPECIFIC_CONDITION:', base.ID) as external_source_event_id,
@ -81,8 +83,26 @@ select
'DRIVER_CARD' as source_package_kind,
cast(base.source_package_id_raw as varchar(128)) as source_package_id,
base.source_package_id_raw as source_package_sort_id,
cast(base.source_package_entity_id_raw as varchar(128)) as source_package_entity_id,
base.source_package_period_from,
base.source_package_period_to,
base.source_package_imported_at
from Base base
)
select *
from Extracted extracted
where (
:sourcePackageWatermarkEnabled = 0
or :lastSourcePackageImportedAt is null
or extracted.source_package_imported_at is null
or extracted.source_package_imported_at > :lastSourcePackageImportedAt
or (
extracted.source_package_imported_at = :lastSourcePackageImportedAt
and (
:lastSourcePackageIdNumeric is null
or extracted.source_package_sort_id is null
or extracted.source_package_sort_id > :lastSourcePackageIdNumeric
)
)
)

View File

@ -80,6 +80,8 @@ Base as (
and (:occurredFrom is null or evt.occurred_at >= :occurredFrom)
and (:occurredTo is null or evt.occurred_at < :occurredTo)
)
,
Extracted as (
select
cast(base.ID as varchar(128)) as source_row_id,
concat('TACHOGRAPH:CARD_VEHICLES_USED:', base.ID, ':', base.lifecycle) as external_source_event_id,
@ -107,8 +109,26 @@ select
'DRIVER_CARD' as source_package_kind,
cast(base.source_package_id_raw as varchar(128)) as source_package_id,
base.source_package_id_raw as source_package_sort_id,
cast(base.source_package_entity_id_raw as varchar(128)) as source_package_entity_id,
base.source_package_period_from,
base.source_package_period_to,
base.source_package_imported_at
from Base base
)
select *
from Extracted extracted
where (
:sourcePackageWatermarkEnabled = 0
or :lastSourcePackageImportedAt is null
or extracted.source_package_imported_at is null
or extracted.source_package_imported_at > :lastSourcePackageImportedAt
or (
extracted.source_package_imported_at = :lastSourcePackageImportedAt
and (
:lastSourcePackageIdNumeric is null
or extracted.source_package_sort_id is null
or extracted.source_package_sort_id > :lastSourcePackageIdNumeric
)
)
)

View File

@ -71,6 +71,8 @@ Base as (
and (:occurredFrom is null or evt.occurred_at >= :occurredFrom)
and (:occurredTo is null or evt.occurred_at < :occurredTo)
)
,
Extracted as (
select
cast(base.ID as varchar(128)) as source_row_id,
concat('TACHOGRAPH:IW_CYCLE:', base.ID, ':', base.lifecycle) as external_source_event_id,
@ -98,6 +100,7 @@ select
'VEHICLE_UNIT' as source_package_kind,
cast(base.source_package_id_raw as varchar(128)) as source_package_id,
base.source_package_id_raw as source_package_sort_id,
cast(base.source_package_entity_id_raw as varchar(128)) as source_package_entity_id,
base.source_package_period_from,
base.source_package_period_to,
@ -126,3 +129,20 @@ where (
and rel.GILT_BIS is null
)
)
)
select *
from Extracted extracted
where (
:sourcePackageWatermarkEnabled = 0
or :lastSourcePackageImportedAt is null
or extracted.source_package_imported_at is null
or extracted.source_package_imported_at > :lastSourcePackageImportedAt
or (
extracted.source_package_imported_at = :lastSourcePackageImportedAt
and (
:lastSourcePackageIdNumeric is null
or extracted.source_package_sort_id is null
or extracted.source_package_sort_id > :lastSourcePackageIdNumeric
)
)
)

View File

@ -9,6 +9,7 @@ select
d.LastUpdate as source_updated_at,
d.ID as driver_id,
d.Surname as surname,
d.Surname as last_name,
d.Firstnames as first_names,
d.Birthdate as birth_date,
d.BirthPlace as birth_place,

View File

@ -64,6 +64,8 @@ Events as (
'END' as lifecycle
from Base base
)
,
Extracted as (
select
concat(cast(events.ID as varchar(128)), ':', events.lifecycle) as source_row_id,
concat('TACHOGRAPH:SPEEDING_EVENTS:', events.ID, ':', events.lifecycle) as external_source_event_id,
@ -86,6 +88,7 @@ select
'VEHICLE_UNIT' as source_package_kind,
cast(events.source_package_id_raw as varchar(128)) as source_package_id,
events.source_package_id_raw as source_package_sort_id,
cast(events.source_package_entity_id_raw as varchar(128)) as source_package_entity_id,
events.source_package_period_from,
events.source_package_period_to,
@ -116,3 +119,20 @@ where (:occurredFrom is null or events.occurred_at >= :occurredFrom)
and rel.GILT_BIS is null
)
)
)
select *
from Extracted extracted
where (
:sourcePackageWatermarkEnabled = 0
or :lastSourcePackageImportedAt is null
or extracted.source_package_imported_at is null
or extracted.source_package_imported_at > :lastSourcePackageImportedAt
or (
extracted.source_package_imported_at = :lastSourcePackageImportedAt
and (
:lastSourcePackageIdNumeric is null
or extracted.source_package_sort_id is null
or extracted.source_package_sort_id > :lastSourcePackageIdNumeric
)
)
)

View File

@ -124,6 +124,8 @@ Base as (
left join dbo.Card c on c.ID = iw.ID_Card
left join dbo.Nation cn on cn.ID = c.ID_Nation
)
,
Extracted as (
select
cast(base.ID as varchar(128)) as source_row_id,
concat('TACHOGRAPH:VU_ACTIVITY:', base.ID, ':', evt.lifecycle) as external_source_event_id,
@ -160,6 +162,7 @@ select
'VEHICLE_UNIT' as source_package_kind,
cast(base.source_package_id_raw as varchar(128)) as source_package_id,
base.source_package_id_raw as source_package_sort_id,
cast(base.source_package_entity_id_raw as varchar(128)) as source_package_entity_id,
base.source_package_period_from,
base.source_package_period_to,
@ -174,3 +177,20 @@ where (:occurredFrom is null or evt.occurred_at >= :occurredFrom)
/*
* Organisation filter: vehicle membership in GetOrganisationTree(null, :organisationId, 0, null).
*/
)
select *
from Extracted extracted
where (
:sourcePackageWatermarkEnabled = 0
or :lastSourcePackageImportedAt is null
or extracted.source_package_imported_at is null
or extracted.source_package_imported_at > :lastSourcePackageImportedAt
or (
extracted.source_package_imported_at = :lastSourcePackageImportedAt
and (
:lastSourcePackageIdNumeric is null
or extracted.source_package_sort_id is null
or extracted.source_package_sort_id > :lastSourcePackageIdNumeric
)
)
)

View File

@ -46,6 +46,8 @@ Base as (
where (:occurredFrom is null or border.Timestamp >= :occurredFrom)
and (:occurredTo is null or border.Timestamp < :occurredTo)
)
,
Extracted as (
select
cast(base.ID as varchar(128)) as source_row_id,
concat('TACHOGRAPH:VU_BORDER_CROSSING:', base.ID) as external_source_event_id,
@ -70,6 +72,7 @@ select
'VEHICLE_UNIT' as source_package_kind,
cast(base.source_package_id_raw as varchar(128)) as source_package_id,
base.source_package_id_raw as source_package_sort_id,
cast(base.source_package_entity_id_raw as varchar(128)) as source_package_entity_id,
base.source_package_period_from,
base.source_package_period_to,
@ -98,3 +101,20 @@ where (
and rel.GILT_BIS is null
)
)
)
select *
from Extracted extracted
where (
:sourcePackageWatermarkEnabled = 0
or :lastSourcePackageImportedAt is null
or extracted.source_package_imported_at is null
or extracted.source_package_imported_at > :lastSourcePackageImportedAt
or (
extracted.source_package_imported_at = :lastSourcePackageImportedAt
and (
:lastSourcePackageIdNumeric is null
or extracted.source_package_sort_id is null
or extracted.source_package_sort_id > :lastSourcePackageIdNumeric
)
)
)

View File

@ -43,6 +43,8 @@ Base as (
where (:occurredFrom is null or lu.Timestamp >= :occurredFrom)
and (:occurredTo is null or lu.Timestamp < :occurredTo)
)
,
Extracted as (
select
cast(base.ID as varchar(128)) as source_row_id,
concat('TACHOGRAPH:VU_LOAD_UNLOAD:', base.ID) as external_source_event_id,
@ -75,6 +77,7 @@ select
'VEHICLE_UNIT' as source_package_kind,
cast(base.source_package_id_raw as varchar(128)) as source_package_id,
base.source_package_id_raw as source_package_sort_id,
cast(base.source_package_entity_id_raw as varchar(128)) as source_package_entity_id,
base.source_package_period_from,
base.source_package_period_to,
@ -103,3 +106,20 @@ where (
and rel.GILT_BIS is null
)
)
)
select *
from Extracted extracted
where (
:sourcePackageWatermarkEnabled = 0
or :lastSourcePackageImportedAt is null
or extracted.source_package_imported_at is null
or extracted.source_package_imported_at > :lastSourcePackageImportedAt
or (
extracted.source_package_imported_at = :lastSourcePackageImportedAt
and (
:lastSourcePackageIdNumeric is null
or extracted.source_package_sort_id is null
or extracted.source_package_sort_id > :lastSourcePackageIdNumeric
)
)
)

View File

@ -44,6 +44,8 @@ Base as (
where (:occurredFrom is null or place.EntryTime >= :occurredFrom)
and (:occurredTo is null or place.EntryTime < :occurredTo)
)
,
Extracted as (
select
cast(base.ID as varchar(128)) as source_row_id,
concat('TACHOGRAPH:VU_PLACE:', base.ID) as external_source_event_id,
@ -71,6 +73,7 @@ select
'VEHICLE_UNIT' as source_package_kind,
cast(base.source_package_id_raw as varchar(128)) as source_package_id,
base.source_package_id_raw as source_package_sort_id,
cast(base.source_package_entity_id_raw as varchar(128)) as source_package_entity_id,
base.source_package_period_from,
base.source_package_period_to,
@ -99,3 +102,20 @@ where (
and rel.GILT_BIS is null
)
)
)
select *
from Extracted extracted
where (
:sourcePackageWatermarkEnabled = 0
or :lastSourcePackageImportedAt is null
or extracted.source_package_imported_at is null
or extracted.source_package_imported_at > :lastSourcePackageImportedAt
or (
extracted.source_package_imported_at = :lastSourcePackageImportedAt
and (
:lastSourcePackageIdNumeric is null
or extracted.source_package_sort_id is null
or extracted.source_package_sort_id > :lastSourcePackageIdNumeric
)
)
)

View File

@ -42,6 +42,8 @@ Base as (
where (:occurredFrom is null or pos.Timestamp >= :occurredFrom)
and (:occurredTo is null or pos.Timestamp < :occurredTo)
)
,
Extracted as (
select
cast(base.ID as varchar(128)) as source_row_id,
concat('TACHOGRAPH:VU_POSITION:', base.ID) as external_source_event_id,
@ -64,6 +66,7 @@ select
'VEHICLE_UNIT' as source_package_kind,
cast(base.source_package_id_raw as varchar(128)) as source_package_id,
base.source_package_id_raw as source_package_sort_id,
cast(base.source_package_entity_id_raw as varchar(128)) as source_package_entity_id,
base.source_package_period_from,
base.source_package_period_to,
@ -92,3 +95,20 @@ where (
and rel.GILT_BIS is null
)
)
)
select *
from Extracted extracted
where (
:sourcePackageWatermarkEnabled = 0
or :lastSourcePackageImportedAt is null
or extracted.source_package_imported_at is null
or extracted.source_package_imported_at > :lastSourcePackageImportedAt
or (
extracted.source_package_imported_at = :lastSourcePackageImportedAt
and (
:lastSourcePackageIdNumeric is null
or extracted.source_package_sort_id is null
or extracted.source_package_sort_id > :lastSourcePackageIdNumeric
)
)
)

View File

@ -32,6 +32,8 @@ Base as (
where (:occurredFrom is null or cond.EntryTime >= :occurredFrom)
and (:occurredTo is null or cond.EntryTime < :occurredTo)
)
,
Extracted as (
select
cast(base.ID as varchar(128)) as source_row_id,
concat('TACHOGRAPH:VU_SPECIFIC_CONDITION:', base.ID) as external_source_event_id,
@ -54,6 +56,7 @@ select
'VEHICLE_UNIT' as source_package_kind,
cast(base.source_package_id_raw as varchar(128)) as source_package_id,
base.source_package_id_raw as source_package_sort_id,
cast(base.source_package_entity_id_raw as varchar(128)) as source_package_entity_id,
base.source_package_period_from,
base.source_package_period_to,
@ -82,3 +85,20 @@ where (
and rel.GILT_BIS is null
)
)
)
select *
from Extracted extracted
where (
:sourcePackageWatermarkEnabled = 0
or :lastSourcePackageImportedAt is null
or extracted.source_package_imported_at is null
or extracted.source_package_imported_at > :lastSourcePackageImportedAt
or (
extracted.source_package_imported_at = :lastSourcePackageImportedAt
and (
:lastSourcePackageIdNumeric is null
or extracted.source_package_sort_id is null
or extracted.source_package_sort_id > :lastSourcePackageIdNumeric
)
)
)

View File

@ -0,0 +1,10 @@
# YellowFox D8 SQL
Recommended source-side index for incremental import:
```sql
create index if not exists d8_booking_utc_eventid_idx
on data.d8_booking (utc asc, eventid asc);
```
The import uses SOURCE_ROW_WATERMARK with `(utc, eventid)`.

View File

@ -0,0 +1,49 @@
select
b.eventid,
b.utc,
b.vehicle_id,
b.driver_id,
b.key,
b.ignition,
b.eventtype,
b.state,
b.odometer,
st_y(b.geom) as latitude,
st_x(b.geom) as longitude,
b.payload,
v.vrn as vehicle_vrn,
v.vin as vehicle_vin,
v.fleet_id as vehicle_fleet_id,
d.firstname as driver_firstname,
d.name as driver_lastname,
d.drivers_card as driver_card_number,
d.fleet_id as driver_fleet_id,
f.id as fleet_id,
f.name as fleet_name,
tp.id as telematic_provider_id,
tp.name as telematic_provider_name
from data.d8_booking b
left join data.vehicle v
on v.id = b.vehicle_id
left join data.driver d
on d.id = b.driver_id
left join data.fleet f
on f.id = coalesce(v.fleet_id, d.fleet_id)
left join data.telematic_provider tp
on tp.id = v.telematic_provider_id
where (:occurredFrom is null or b.utc >= :occurredFrom)
and (:occurredTo is null or b.utc < :occurredTo)
and (:fleetId is null or f.id = :fleetId)
and (
:lastOccurredTo is null
or b.utc > :lastOccurredTo
or (
b.utc = :lastOccurredTo
and (:lastSourceRowId is null or b.eventid > :lastSourceRowId)
)
)
order by b.utc asc, b.eventid asc;

View File

@ -1,5 +1,6 @@
package at.procon.eventhub;
import at.procon.eventhub.dto.CardSlot;
import at.procon.eventhub.dto.DriverCardRefDto;
import at.procon.eventhub.dto.DriverRefDto;
import at.procon.eventhub.dto.EventDomain;
@ -7,8 +8,8 @@ import at.procon.eventhub.dto.EventLifecycle;
import at.procon.eventhub.dto.EventType;
import at.procon.eventhub.dto.VehicleRefDto;
import at.procon.eventhub.dto.VehicleRegistrationRefDto;
import at.procon.eventhub.yellowfox.dto.YellowFoxD8BookingDto;
import at.procon.eventhub.service.EventDetailsFactory;
import at.procon.eventhub.yellowfox.dto.YellowFoxD8BookingDto;
import at.procon.eventhub.yellowfox.service.YellowFoxD8BookingEventMapper;
import com.fasterxml.jackson.databind.ObjectMapper;
import java.time.OffsetDateTime;
@ -23,16 +24,67 @@ class YellowFoxD8BookingEventMapperTest {
);
@Test
void mapsYellowFoxDrivingStateAsSnapshotWithEventSourceAndDetails() {
YellowFoxD8BookingDto source = new YellowFoxD8BookingDto(
void mapsYellowFoxDriverSlotActivityAsStartWithSlotDetails() {
var source = booking(2, 3, 1);
var result = mapper.map(source);
assertThat(result.eventDomain()).isEqualTo(EventDomain.DRIVER_ACTIVITY);
assertThat(result.eventType()).isEqualTo(EventType.DRIVE);
assertThat(result.lifecycle()).isEqualTo(EventLifecycle.START);
assertThat(result.externalSourceEventId()).startsWith("D8_BOOKING:event-1:");
assertThat(result.eventDetails().type()).isEqualTo("DRIVER_ACTIVITY");
assertThat(result.eventDetails().attributes().get("slot").asText()).isEqualTo(CardSlot.DRIVER.name());
assertThat(result.eventDetails().attributes().get("slotNumber").asInt()).isEqualTo(1);
assertThat(result.eventDetails().attributes().get("yellowFoxStateMeaning").asText()).isEqualTo("STEERING_TIME");
assertThat(result.eventDetails().attributes().get("ignitionState").asInt()).isEqualTo(1);
}
@Test
void mapsYellowFoxCoDriverSlotActivityAsStartWithSlotDetails() {
var result = mapper.map(booking(3, 2, 0));
assertThat(result.eventDomain()).isEqualTo(EventDomain.DRIVER_ACTIVITY);
assertThat(result.eventType()).isEqualTo(EventType.WORK);
assertThat(result.lifecycle()).isEqualTo(EventLifecycle.START);
assertThat(result.eventDetails().attributes().get("slot").asText()).isEqualTo(CardSlot.CO_DRIVER.name());
assertThat(result.eventDetails().attributes().get("slotNumber").asInt()).isEqualTo(2);
}
@Test
void mapsCardInsertedUsingSameTachographDomainTypeAndLifecycle() {
var result = mapper.map(booking(0, 1, 1));
assertThat(result.eventDomain()).isEqualTo(EventDomain.DRIVER_CARD);
assertThat(result.eventType()).isEqualTo(EventType.CARD_INSERTED);
assertThat(result.lifecycle()).isEqualTo(EventLifecycle.INSERT);
assertThat(result.eventDetails().attributes().get("slot").asText()).isEqualTo(CardSlot.DRIVER.name());
}
@Test
void mapsIgnitionTransitionOnlyWhenDetectorRequestsIt() {
var result = mapper.mapIgnitionTransition(booking(2, 3, 1), 0);
assertThat(result.eventDomain()).isEqualTo(EventDomain.IGNITION);
assertThat(result.eventType()).isEqualTo(EventType.IGNITION_ON);
assertThat(result.lifecycle()).isEqualTo(EventLifecycle.ON);
assertThat(result.externalSourceEventId()).contains("D8_IGNITION:event-1:");
assertThat(result.eventDetails().attributes().get("previousIgnitionState").asInt()).isEqualTo(0);
assertThat(result.eventDetails().attributes().get("ignitionMeaning").asText()).isEqualTo("ON");
}
private YellowFoxD8BookingDto booking(int eventType, int state, int ignition) {
return new YellowFoxD8BookingDto(
"tenant-1",
"yellowfox-tenant-7",
"tenant-yellowfox-7",
"7",
"Fleet 7",
"event-1",
"key-1",
2,
3,
ignition,
eventType,
state,
OffsetDateTime.parse("2026-04-29T08:15:00+02:00"),
null,
new DriverRefDto("driver-source-100", new DriverCardRefDto("AT", "D123456789")),
@ -42,20 +94,5 @@ class YellowFoxD8BookingEventMapperTest {
null,
null
);
var result = mapper.map(source);
assertThat(result.eventDomain()).isEqualTo(EventDomain.DRIVER_ACTIVITY);
assertThat(result.eventType()).isEqualTo(EventType.DRIVE);
assertThat(result.lifecycle()).isEqualTo(EventLifecycle.SNAPSHOT);
assertThat(result.externalSourceEventId()).isEqualTo("event-1");
assertThat(result.packageInfo().tenantKey()).isEqualTo("tenant-1");
assertThat(result.packageInfo().eventSource().providerKey()).isEqualTo("YELLOWFOX");
assertThat(result.packageInfo().eventSource().sourceKey()).isEqualTo("YELLOWFOX_D8");
assertThat(result.driverRef().driverCard().nation()).isEqualTo("AT");
assertThat(result.vehicleRef().vehicleRegistration().nation()).isEqualTo("AT");
assertThat(result.eventDetails().type()).isEqualTo("DRIVER_ACTIVITY");
assertThat(result.payload().get("yellowFoxEventType").asInt()).isEqualTo(2);
assertThat(result.payload().get("yellowFoxState").asInt()).isEqualTo(3);
}
}

View File

@ -0,0 +1,61 @@
package at.procon.eventhub.esperpoc.service;
import static org.assertj.core.api.Assertions.assertThat;
import at.procon.eventhub.esperpoc.dto.RawActivityEventDto;
import java.time.OffsetDateTime;
import java.util.List;
import java.util.UUID;
import org.junit.jupiter.api.Test;
class EsperDriverActivityEngineTest {
private final EsperDriverActivityEngine engine = new EsperDriverActivityEngine();
@Test
void buildsActivityIntervalsFromStartAndEndEvents() {
UUID driverId = UUID.randomUUID();
UUID vehicleId = UUID.randomUUID();
List<RawActivityEventDto> raw = List.of(
raw("100", "START", "DRIVE", "2026-04-30T23:55:00Z", driverId, vehicleId),
raw("100", "END", "DRIVE", "2026-05-01T00:10:00Z", driverId, vehicleId),
raw("101", "START", "WORK", "2026-05-01T00:10:00Z", driverId, vehicleId),
raw("101", "END", "WORK", "2026-05-01T00:40:00Z", driverId, vehicleId)
);
var intervals = engine.buildIntervals(raw);
assertThat(intervals).hasSize(2);
assertThat(intervals.get(0).activityType()).isEqualTo("DRIVE");
assertThat(intervals.get(0).durationSeconds()).isEqualTo(15 * 60L);
assertThat(intervals.get(1).activityType()).isEqualTo("WORK");
assertThat(intervals.get(1).durationSeconds()).isEqualTo(30 * 60L);
}
private RawActivityEventDto raw(
String rowId,
String lifecycle,
String activity,
String occurredAt,
UUID driverId,
UUID vehicleId
) {
return new RawActivityEventDto(
UUID.randomUUID(),
OffsetDateTime.parse(occurredAt),
rowId,
"TACHOGRAPH:CARD_ACTIVITY:" + rowId + ":" + lifecycle,
"DRIVER_CARD",
"CARD_ACTIVITY",
driverId,
vehicleId,
null,
activity,
lifecycle,
"DRIVER",
"INSERTED",
"SINGLE",
"package-1"
);
}
}

View File

@ -0,0 +1,154 @@
package at.procon.eventhub.esperpoc.service;
import static org.assertj.core.api.Assertions.assertThat;
import at.procon.eventhub.esperpoc.dto.ActivityIntervalDto;
import at.procon.eventhub.esperpoc.dto.EsperUnknownTreatmentMode;
import at.procon.eventhub.esperpoc.dto.NonDrivingIntervalDto;
import at.procon.eventhub.esperpoc.dto.OperatingPeriodActivityIntervalDto;
import java.time.Duration;
import java.time.OffsetDateTime;
import java.util.List;
import java.util.UUID;
import org.junit.jupiter.api.Test;
class EsperOperatingPeriodEvaluationServiceTest {
private final EsperOperatingPeriodEngine operatingPeriodEngine = new EsperOperatingPeriodEngine();
private final EsperOperatingPeriodEvaluationService service = new EsperOperatingPeriodEvaluationService(
null,
new EsperDriverActivityEngine(),
operatingPeriodEngine
);
@Test
void synthesizeUnknownGapsSkipsCoveredBreakRestAndCreatesOnlyUncoveredUnknown() {
UUID driverId = UUID.randomUUID();
List<ActivityIntervalDto> intervals = List.of(
activity(driverId, "WORK", "2026-04-01T08:00:00Z", "2026-04-01T09:00:00Z", "w1", "DRIVER_CARD"),
activity(driverId, "BREAK_REST", "2026-04-01T09:00:00Z", "2026-04-01T10:00:00Z", "r1", "DRIVER_CARD"),
activity(driverId, "AVAILABILITY", "2026-04-01T10:00:00Z", "2026-04-01T11:00:00Z", "a1", "DRIVER_CARD"),
activity(driverId, "WORK", "2026-04-01T12:00:00Z", "2026-04-01T13:00:00Z", "w2", "DRIVER_CARD")
);
List<ActivityIntervalDto> evaluationIntervals = service.synthesizeUnknownGaps(intervals, Duration.ZERO);
assertThat(evaluationIntervals).extracting(ActivityIntervalDto::activityType)
.containsExactly("WORK", "AVAILABILITY", "UNKNOWN", "WORK");
assertThat(evaluationIntervals.get(2).startedAt()).isEqualTo(OffsetDateTime.parse("2026-04-01T11:00:00Z"));
assertThat(evaluationIntervals.get(2).endedAt()).isEqualTo(OffsetDateTime.parse("2026-04-01T12:00:00Z"));
}
@Test
void periodizationSplitsAfterLongUnknownGapAndShortDrivingDoesNotCloseNonDriving() {
UUID driverId = UUID.randomUUID();
List<ActivityIntervalDto> evaluationIntervals = List.of(
activity(driverId, "WORK", "2026-04-01T08:00:00Z", "2026-04-01T09:00:00Z", "w1", "DRIVER_CARD"),
activity(driverId, "AVAILABILITY", "2026-04-01T10:00:00Z", "2026-04-01T11:00:00Z", "a1", "DRIVER_CARD"),
activity(driverId, "DRIVE", "2026-04-01T11:00:00Z", "2026-04-01T11:02:00Z", "d1", "DRIVER_CARD"),
activity(driverId, "WORK", "2026-04-01T11:02:00Z", "2026-04-01T12:00:00Z", "w2", "DRIVER_CARD"),
unknown(driverId, "2026-04-01T12:00:00Z", "2026-04-01T20:00:00Z"),
activity(driverId, "DRIVE", "2026-04-01T20:00:00Z", "2026-04-01T20:30:00Z", "d2", "DRIVER_CARD")
);
EsperOperatingPeriodEngine.EsperOperatingPeriodEvaluation evaluation = operatingPeriodEngine.evaluate(
evaluationIntervals,
Duration.ofHours(7)
);
assertThat(evaluation.periodizedIntervals()).extracting(OperatingPeriodActivityIntervalDto::activityType)
.containsExactly("WORK", "AVAILABILITY", "DRIVE", "WORK", "DRIVE");
assertThat(evaluation.closedPeriods()).hasSize(2);
assertThat(evaluation.closedPeriods().get(0).closedBy()).isEqualTo("UNKNOWN_GAP");
assertThat(evaluation.closedPeriods().get(1).closedBy()).isEqualTo("FLUSH");
List<OperatingPeriodActivityIntervalDto> mergedIntervals = service.mergeConsecutiveActivities(
evaluation.periodizedIntervals(),
Duration.ZERO
);
List<NonDrivingIntervalDto> nonDrivingIntervals = service.buildNonDrivingIntervals(
mergedIntervals,
Duration.ofMinutes(3),
EsperUnknownTreatmentMode.AS_BREAK_REST
);
assertThat(nonDrivingIntervals).hasSize(1);
assertThat(nonDrivingIntervals.get(0).startedAt()).isEqualTo(OffsetDateTime.parse("2026-04-01T08:00:00Z"));
assertThat(nonDrivingIntervals.get(0).endedAt()).isEqualTo(OffsetDateTime.parse("2026-04-01T12:00:00Z"));
assertThat(nonDrivingIntervals.get(0).closedBy()).isEqualTo("NEW_OPERATING_PERIOD");
}
@Test
void unknownCanCloseNonDrivingWhenConfiguredAsNonBreak() {
UUID driverId = UUID.randomUUID();
List<OperatingPeriodActivityIntervalDto> mergedIntervals = List.of(
periodized(activity(driverId, "WORK", "2026-04-01T08:00:00Z", "2026-04-01T09:00:00Z", "w1", "DRIVER_CARD"), 1),
periodized(unknown(driverId, "2026-04-01T09:00:00Z", "2026-04-01T09:30:00Z"), 1),
periodized(activity(driverId, "AVAILABILITY", "2026-04-01T09:30:00Z", "2026-04-01T10:00:00Z", "a1", "DRIVER_CARD"), 1)
);
List<NonDrivingIntervalDto> nonDrivingIntervals = service.buildNonDrivingIntervals(
mergedIntervals,
Duration.ofMinutes(3),
EsperUnknownTreatmentMode.AS_UNKNOWN_NON_BREAK
);
assertThat(nonDrivingIntervals).hasSize(2);
assertThat(nonDrivingIntervals.get(0).closedBy()).isEqualTo("UNKNOWN_GAP");
assertThat(nonDrivingIntervals.get(0).endedAt()).isEqualTo(OffsetDateTime.parse("2026-04-01T09:00:00Z"));
assertThat(nonDrivingIntervals.get(1).startedAt()).isEqualTo(OffsetDateTime.parse("2026-04-01T09:30:00Z"));
}
private ActivityIntervalDto activity(
UUID driverId,
String activity,
String from,
String to,
String sourceRowId,
String sourceKind
) {
return ActivityIntervalDto.raw(
driverId,
null,
null,
activity,
"DRIVER",
"INSERTED",
"KNOWN",
sourceKind,
OffsetDateTime.parse(from),
OffsetDateTime.parse(to),
sourceRowId
);
}
private ActivityIntervalDto unknown(UUID driverId, String from, String to) {
return new ActivityIntervalDto(
driverId,
null,
null,
"UNKNOWN",
"DRIVER",
null,
"UNKNOWN",
"SYNTHETIC_GAP",
OffsetDateTime.parse(from),
OffsetDateTime.parse(to),
Duration.between(OffsetDateTime.parse(from), OffsetDateTime.parse(to)).getSeconds(),
null,
List.of(),
false,
"UNKNOWN_GAP"
);
}
private OperatingPeriodActivityIntervalDto periodized(ActivityIntervalDto interval, long operatingPeriodNo) {
return OperatingPeriodActivityIntervalDto.periodized(
interval,
operatingPeriodNo,
interval.startedAt(),
false,
0L
);
}
}

View File

@ -0,0 +1,237 @@
package at.procon.eventhub.esperpoc.service;
import static org.assertj.core.api.Assertions.assertThat;
import at.procon.eventhub.esperpoc.dto.ActivityIntervalDto;
import at.procon.eventhub.esperpoc.dto.EsperActivityMergeMode;
import at.procon.eventhub.esperpoc.dto.EsperPocRequest;
import at.procon.eventhub.esperpoc.dto.EsperShiftResolutionMode;
import java.time.Duration;
import java.time.OffsetDateTime;
import java.util.List;
import java.util.UUID;
import org.junit.jupiter.api.Test;
class EsperPocDriverCardActivityServiceTest {
private final EsperDriverActivityEngine engine = new EsperDriverActivityEngine();
private final EsperPocDriverCardActivityService service = new EsperPocDriverCardActivityService(null, engine);
@Test
void mergesIdenticalActivitiesAcrossUtcMidnight() {
UUID driverId = UUID.randomUUID();
ActivityIntervalDto beforeMidnight = ActivityIntervalDto.raw(
driverId,
null,
null,
"BREAK_REST",
"DRIVER",
"INSERTED",
"SINGLE",
"DRIVER_CARD",
OffsetDateTime.parse("2026-04-30T23:50:00Z"),
OffsetDateTime.parse("2026-05-01T00:00:00Z"),
"1"
);
ActivityIntervalDto afterMidnight = ActivityIntervalDto.raw(
driverId,
null,
null,
"BREAK_REST",
"DRIVER",
"INSERTED",
"SINGLE",
"DRIVER_CARD",
OffsetDateTime.parse("2026-05-01T00:00:00Z"),
OffsetDateTime.parse("2026-05-01T00:20:00Z"),
"2"
);
var merged = service.mergeConsecutiveIdenticalActivities(
List.of(beforeMidnight, afterMidnight),
Duration.ZERO
);
assertThat(merged).hasSize(1);
assertThat(merged.get(0).startedAt()).isEqualTo(OffsetDateTime.parse("2026-04-30T23:50:00Z"));
assertThat(merged.get(0).endedAt()).isEqualTo(OffsetDateTime.parse("2026-05-01T00:20:00Z"));
assertThat(merged.get(0).durationSeconds()).isEqualTo(30 * 60L);
assertThat(merged.get(0).sourceRowIds()).containsExactly("1", "2");
}
@Test
void splitsOperatingPeriodsByBreakRestLongerThanConfiguredHoursAndEvaluatesDepartureArrivalPerPeriod() {
UUID driverId = UUID.randomUUID();
EsperPocRequest request = request(driverId, null, null);
List<ActivityIntervalDto> activities = List.of(
activity(driverId, "DRIVE", "2026-04-01T06:00:00Z", "2026-04-01T08:00:00Z", "d1", "DRIVER_CARD"),
activity(driverId, "WORK", "2026-04-01T08:00:00Z", "2026-04-01T09:00:00Z", "w1", "DRIVER_CARD"),
activity(driverId, "DRIVE", "2026-04-01T09:30:00Z", "2026-04-01T11:00:00Z", "d2", "DRIVER_CARD"),
activity(driverId, "BREAK_REST", "2026-04-01T20:00:00Z", "2026-04-02T04:30:00Z", "r1", "DRIVER_CARD"),
activity(driverId, "DRIVE", "2026-04-02T05:00:00Z", "2026-04-02T07:00:00Z", "d3", "DRIVER_CARD"),
activity(driverId, "BREAK_REST", "2026-04-02T10:00:00Z", "2026-04-02T10:30:00Z", "r2", "DRIVER_CARD"),
activity(driverId, "DRIVE", "2026-04-02T10:30:00Z", "2026-04-02T12:00:00Z", "d4", "DRIVER_CARD")
);
var periods = service.buildOperatingTimePeriods(
request,
request.occurredFrom(),
request.occurredTo(),
activities
);
assertThat(periods).hasSize(2);
assertThat(periods.get(0).startedAt()).isEqualTo(OffsetDateTime.parse("2026-04-01T06:00:00Z"));
assertThat(periods.get(0).endedAt()).isEqualTo(OffsetDateTime.parse("2026-04-01T11:00:00Z"));
assertThat(periods.get(0).drivingTimeInterruptionEvaluation().departureAt())
.isEqualTo(OffsetDateTime.parse("2026-04-01T06:00:00Z"));
assertThat(periods.get(0).drivingTimeInterruptionEvaluation().arrivalAt())
.isEqualTo(OffsetDateTime.parse("2026-04-01T11:00:00Z"));
assertThat(periods.get(0).drivingTimeInterruptionEvaluation().interruptionsBetweenSignificantDrivingPeriods())
.hasSize(1);
assertThat(periods.get(1).startedAt()).isEqualTo(OffsetDateTime.parse("2026-04-02T05:00:00Z"));
assertThat(periods.get(1).endedAt()).isEqualTo(OffsetDateTime.parse("2026-04-02T12:00:00Z"));
assertThat(periods.get(1).drivingTimeInterruptionEvaluation().departureAt())
.isEqualTo(OffsetDateTime.parse("2026-04-02T05:00:00Z"));
assertThat(periods.get(1).drivingTimeInterruptionEvaluation().arrivalAt())
.isEqualTo(OffsetDateTime.parse("2026-04-02T12:00:00Z"));
assertThat(periods.get(1).workingOperationTimes().breakRestSeconds()).isEqualTo(30 * 60L);
}
@Test
void usesVuIntervalsOnlyForUncoveredDriverCardGaps() {
UUID driverId = UUID.randomUUID();
List<ActivityIntervalDto> resolved = service.resolveVuFillGaps(
List.of(
activity(driverId, "DRIVE", "2026-04-01T08:00:00Z", "2026-04-01T10:00:00Z", "c1", "DRIVER_CARD")
),
List.of(
activity(driverId, "DRIVE", "2026-04-01T09:00:00Z", "2026-04-01T11:00:00Z", "v1", "VEHICLE_UNIT")
)
);
assertThat(resolved).hasSize(2);
assertThat(resolved.get(0).sourceKind()).isEqualTo("DRIVER_CARD");
assertThat(resolved.get(1).sourceKind()).isEqualTo("VEHICLE_UNIT");
assertThat(resolved.get(1).startedAt()).isEqualTo(OffsetDateTime.parse("2026-04-01T10:00:00Z"));
assertThat(resolved.get(1).endedAt()).isEqualTo(OffsetDateTime.parse("2026-04-01T11:00:00Z"));
}
@Test
void resolvesTwoWorkingShiftsAcrossLongRestGap() {
UUID driverId = UUID.randomUUID();
EsperPocRequest request = request(driverId, null, null);
var shifts = service.buildResolvedWorkShifts(
request,
request.occurredFrom(),
request.occurredTo(),
List.of(
activity(driverId, "DRIVE", "2026-04-01T06:00:00Z", "2026-04-01T10:00:00Z", "d1", "DRIVER_CARD"),
activity(driverId, "BREAK_REST", "2026-04-01T10:00:00Z", "2026-04-01T17:30:00Z", "r1", "DRIVER_CARD"),
activity(driverId, "WORK", "2026-04-01T17:30:00Z", "2026-04-01T19:00:00Z", "w1", "VEHICLE_UNIT")
)
);
assertThat(shifts).hasSize(2);
assertThat(shifts.get(0).startedAt()).isEqualTo(OffsetDateTime.parse("2026-04-01T06:00:00Z"));
assertThat(shifts.get(0).endedAt()).isEqualTo(OffsetDateTime.parse("2026-04-01T10:00:00Z"));
assertThat(shifts.get(0).dailyRestingTimeSeconds()).isEqualTo(7L * 60L * 60L + 30L * 60L);
assertThat(shifts.get(1).startedAt()).isEqualTo(OffsetDateTime.parse("2026-04-01T17:30:00Z"));
assertThat(shifts.get(1).usedDataKind()).isEqualTo("VEHICLE_UNIT");
}
@Test
void esperMergeModeMatchesJavaMergeMode() {
UUID driverId = UUID.randomUUID();
List<ActivityIntervalDto> activities = List.of(
activity(driverId, "BREAK_REST", "2026-04-01T08:00:00Z", "2026-04-01T09:00:00Z", "r1", "DRIVER_CARD"),
activity(driverId, "BREAK_REST", "2026-04-01T09:00:00Z", "2026-04-01T10:00:00Z", "r2", "DRIVER_CARD"),
activity(driverId, "WORK", "2026-04-01T10:05:00Z", "2026-04-01T11:00:00Z", "w1", "DRIVER_CARD")
);
List<ActivityIntervalDto> javaMerged = service.mergeConsecutiveIdenticalActivities(
activities,
Duration.ofSeconds(60)
);
List<ActivityIntervalDto> esperMerged = engine.mergeConsecutiveIdenticalActivities(
activities,
Duration.ofSeconds(60)
);
assertThat(esperMerged).usingRecursiveComparison().isEqualTo(javaMerged);
}
@Test
void esperShiftResolutionModeMatchesJavaShiftResolutionMode() {
UUID driverId = UUID.randomUUID();
EsperPocRequest javaRequest = request(driverId, EsperActivityMergeMode.JAVA, EsperShiftResolutionMode.JAVA);
EsperPocRequest esperRequest = request(driverId, EsperActivityMergeMode.JAVA, EsperShiftResolutionMode.ESPER);
List<ActivityIntervalDto> intervals = List.of(
activity(driverId, "DRIVE", "2026-04-01T06:00:00Z", "2026-04-01T10:00:00Z", "d1", "DRIVER_CARD"),
activity(driverId, "BREAK_REST", "2026-04-01T10:00:00Z", "2026-04-01T17:30:00Z", "r1", "DRIVER_CARD"),
activity(driverId, "WORK", "2026-04-01T17:30:00Z", "2026-04-01T19:00:00Z", "w1", "VEHICLE_UNIT")
);
var javaShifts = service.buildResolvedWorkShifts(
javaRequest,
javaRequest.occurredFrom(),
javaRequest.occurredTo(),
intervals
);
var esperShifts = service.buildResolvedWorkShifts(
esperRequest,
esperRequest.occurredFrom(),
esperRequest.occurredTo(),
intervals
);
assertThat(esperShifts).usingRecursiveComparison().isEqualTo(javaShifts);
}
private EsperPocRequest request(
UUID driverId,
EsperActivityMergeMode activityMergeMode,
EsperShiftResolutionMode shiftResolutionMode
) {
return new EsperPocRequest(
"default",
driverId,
OffsetDateTime.parse("2026-04-01T00:00:00Z"),
OffsetDateTime.parse("2026-04-03T00:00:00Z"),
24,
3,
60,
7,
420,
5,
activityMergeMode,
shiftResolutionMode
);
}
private ActivityIntervalDto activity(
UUID driverId,
String activity,
String from,
String to,
String sourceRowId,
String sourceKind
) {
return ActivityIntervalDto.raw(
driverId,
null,
null,
activity,
"DRIVER",
"INSERTED",
"KNOWN",
sourceKind,
OffsetDateTime.parse(from),
OffsetDateTime.parse(to),
sourceRowId
).asMerged(List.of(sourceRowId));
}
}