Sentinel Core

Manager Portal

M
AI Agents / ALPR License Plate Detective
AGT-MOT-006 Motor Vehicle AI-based

ALPR License Plate Detective

AGT-MOT-006 is an Automatic License Plate Recognition (ALPR) agent that extracts vehicle registration numbers from photographs, dashcam video frames, and CCTV images, then queries multiple authoritative databases in real-time to verify the vehicle's legal status. The agent uses YOLOv8 for high-speed license plate detection, Tesseract OCR with custom Vietnamese/ASEAN number plate models for character recognition, and OpenALPR for plate structure validation. Recognised plate numbers are immediately cross-referenced against the Traffic Police stolen vehicle database, the court-ordered seizure list, fraud ring vehicle watchlist, and the insured vehicle registry to confirm the plate number matches the vehicle described in the policy.

Tech Stack

Python 3.11 Runtime
OpenALPR License plate structure detection and region-specific OCR
YOLOv8 (Ultralytics) Real-time plate bounding box detection in images and video frames
Tesseract OCR 5.x Character recognition with custom Vietnamese plate font training data
OpenCV 4.x Image preprocessing, deskewing, contrast enhancement
aiohttp 3.x Async HTTP client for Traffic Police REST API queries
Redis 7.x Caching of database query results for rate-limit compliance
PostgreSQL 15 Local fraud ring vehicle watchlist database

Input

One or more vehicle images or dashcam video, plus the policy-declared vehicle registration number for cross-reference.

Accepted Formats

JPEG PNG MP4 AVI

Fields

Name Type Req Description
media_files array<binary> Yes Images or video frames containing the vehicle
declared_plate_number string Yes License plate number declared in the insurance policy
declared_vehicle_vin string No VIN number from policy (for secondary cross-check if plate is unreadable)
plate_region string No Plate region code for regional format validation (e.g. VN-HN, VN-HCM)

Output

Recognised plate numbers, database query results for each, and a verdict on vehicle legality and identity consistency.

Format:

JSON

Fields

Name Type Description
detected_plates array<object> All plates found: {plate_number, confidence, bounding_box, source_frame}
best_plate_match string Highest-confidence plate reading across all media
plate_match_policy boolean Whether best_plate_match equals declared_plate_number
stolen_vehicle_check object {queried: bool, is_stolen: bool, report_date: string|null, case_number: string|null}
seizure_check object {queried: bool, is_seized: bool, court_order: string|null}
fraud_watchlist_check object {is_on_watchlist: bool, associated_claims: array<string>}
flags array<string> FLAG_STOLEN_VEHICLE, FLAG_SEIZED_VEHICLE, FLAG_WATCHLIST_HIT, FLAG_PLATE_MISMATCH, FLAG_PLATE_UNREADABLE
risk_score float Normalised risk contribution 0.0–1.0
verdict string PASS | FLAG | INCONCLUSIVE

Example Response

{
  "detected_plates": [{"plate_number": "51F-29847", "confidence": 0.96, "source_frame": 1}],
  "best_plate_match": "51F-29847",
  "plate_match_policy": false,
  "stolen_vehicle_check": {"queried": true, "is_stolen": false, "report_date": null},
  "seizure_check": {"queried": true, "is_seized": true, "court_order": "TAND-2024/0088"},
  "fraud_watchlist_check": {"is_on_watchlist": false, "associated_claims": []},
  "flags": ["FLAG_PLATE_MISMATCH", "FLAG_SEIZED_VEHICLE"],
  "risk_score": 0.95,
  "verdict": "FLAG"
}

How It Works

Motor vehicle fraud frequently involves plate swapping: moving a registered insurance policy plate onto an uninsured or already-damaged vehicle to make it appear covered. It also involves staging accidents with stolen or seized vehicles that cannot legally be on the road. ALPR detection provides an objective, automated check against both patterns.

The agent's recognition pipeline handles real-world challenges: motion blur from dashcam footage, partial occlusion, night-time low-contrast conditions, mud-obscured characters, and deliberate plate obstruction. The dual-OCR approach (Tesseract + OpenALPR with character-level voting) is significantly more robust than either engine alone.

The database query stage is where the agent's value is highest. The Traffic Police stolen vehicle database is updated in real-time as theft reports are filed. A vehicle reported stolen three days before an 'accident' is extremely suspicious — the owner may have staged the crash to claim insurance on a vehicle they can no longer sell. The court seizure database catches vehicles that have been impounded in ongoing legal proceedings.

The fraud watchlist is the most powerful internal resource: it links plate numbers to known fraud ring participants. A plate appearing in multiple claims across different policyholders is a strong indicator of a coordinated ring where vehicles are passed between conspirators to file repeated claims.

All database results are returned with the specific record identifiers (case numbers, court order numbers) that enable adjudicators to request official verification documentation.

Thinking Steps

1

Media Ingestion & Frame Extraction

For image files, load directly. For video files, extract keyframes using OpenCV at 1-second intervals (or detect the sharpest frame per scene using blur score) to select the best frames for plate recognition. Assess image quality: minimum 480p resolution required for reliable OCR.

Dashcam footage often has motion blur; extracting the sharpest frame per second rather than every frame reduces OCR errors significantly.

2

License Plate Detection (YOLOv8)

Run YOLOv8 plate detection model on each frame to locate license plate bounding boxes. The model was fine-tuned on Vietnamese motorcycle and car plates and achieves 94.2% detection rate at IoU 0.5. Crop and deskew detected plate regions for OCR.

Vietnamese plates come in two formats: motorcycle (2 rows, smaller) and car (1 or 2 rows, larger). YOLOv8 handles both aspect ratios.

3

OCR Character Recognition

Apply a preprocessing pipeline to each plate crop: grayscale, adaptive thresholding, Otsu binarisation, morphological cleanup. Run both Tesseract (with custom Vietnamese plate font tessdata) and OpenALPR in parallel and merge results using a character-level confidence voting scheme.

The dual-OCR approach reduces single-character errors (e.g. confusing '0' and 'D', '1' and 'I') by using the result with higher character-level confidence for each position.

4

Plate Format Validation

Validate the recognised plate against Vietnamese and regional plate format rules: province prefix (2 digits), letter series (1–2 uppercase letters), sequence number (4–5 digits). Flag any plate that cannot be validated against known regional formats as potentially modified or obstructed.

Deliberate plate obstruction (mud, tape, bent metal) is common in staged accidents to prevent ALPR detection.

5

Policy Plate Cross-Reference

Compare the best-confidence recognised plate number against the declared_plate_number in the insurance policy. A mismatch indicates either a wrong-vehicle photo submission or a swap — the insured plate has been moved to an uninsured vehicle for the purposes of the claim.

Minor OCR errors (1-character difference) trigger INCONCLUSIVE rather than FLAG; only a clear format-valid mismatch triggers FLAG_PLATE_MISMATCH.

6

Multi-Database Query (Async Parallel)

Fire simultaneous async queries to: (1) Traffic Police stolen vehicle REST API, (2) Court seizure order database, (3) Internal fraud ring watchlist (PostgreSQL). Results are aggregated after all queries complete (with 10-second timeout fallback).

Rate limits on the Traffic Police API are managed via Redis-based token bucket — the agent queues requests rather than dropping them.

7

Risk Aggregation & Verdict

Assign risk weights: FLAG_STOLEN_VEHICLE=1.0, FLAG_SEIZED_VEHICLE=0.95, FLAG_WATCHLIST_HIT=0.80, FLAG_PLATE_MISMATCH=0.75, FLAG_PLATE_UNREADABLE=0.30. Cap total at 1.0. Verdict: FLAG if any critical flag is set, INCONCLUSIVE if only FLAG_PLATE_UNREADABLE.

A stolen vehicle flag alone warrants immediate escalation to the anti-fraud team regardless of other signals.

Thinking Tree

  • Root Question: Is this vehicle legally registered, non-stolen, and does the plate match the policy?
    • Can the license plate be read from media?
      • Yes, confidence ≥ 0.80 → proceed to validation
      • No, unreadable → FLAG_PLATE_UNREADABLE (INCONCLUSIVE)
    • Does plate match policy declaration?
      • Plates match → proceed to DB checks
      • Plates differ → FLAG_PLATE_MISMATCH
    • Traffic Police stolen vehicle DB
      • Vehicle not in stolen DB → continue
      • Vehicle reported stolen → FLAG_STOLEN_VEHICLE (critical)
    • Court seizure order DB
      • No active seizure order → continue
      • Active seizure order found → FLAG_SEIZED_VEHICLE
    • Internal fraud ring watchlist
      • Plate not on watchlist → PASS
      • Plate associated with known fraud ring → FLAG_WATCHLIST_HIT

Decision Tree

Is the license plate readable with confidence ≥ 0.80?

Yes → d2 No → flag_unreadable
d1

Does recognised plate match the policy-declared plate?

Yes → d3 No → flag_mismatch
d2

Is the vehicle in the stolen vehicle database?

Yes → flag_stolen No → d4
d3

Is the vehicle subject to a court seizure order?

Yes → flag_seized No → d5
d4

Is the plate on the internal fraud ring watchlist?

Yes → flag_watchlist No → pass
d5

INCONCLUSIVE — Plate unreadable; possible deliberate obstruction

flag_unreadable

FLAG — PLATE_MISMATCH: Photographed plate differs from policy-declared plate

flag_mismatch

FLAG — STOLEN_VEHICLE: Vehicle is listed as stolen in Traffic Police database

flag_stolen

FLAG — SEIZED_VEHICLE: Vehicle is subject to active court seizure order

flag_seized

FLAG — WATCHLIST_HIT: Plate associated with known insurance fraud ring

flag_watchlist

PASS — Vehicle is legal, not stolen, and plate matches policy declaration

pass

Technical Design

Architecture

AGT-MOT-006 is an async FastAPI microservice. ALPR and OCR processing runs on CPU using ONNX-quantised YOLOv8. Database queries are fully async (aiohttp + asyncpg for PostgreSQL). Redis caches Traffic Police API responses for 24 hours to reduce API costs and comply with rate limits. The service processes up to 50 media files per claim request.

Components

Component Role Technology
FrameExtractor Extracts optimal frames from video; loads images directly OpenCV VideoCapture + blur score
YOLOv8Detector Detects and crops license plate regions Ultralytics YOLOv8 ONNX
PlatePreprocessor Deskews, binarises, and enhances plate crops OpenCV adaptive threshold + morphology
TesseractOCR Primary character recognition with custom plate tessdata Tesseract 5.x + custom traineddata
OpenALPROCR Secondary OCR for confidence voting OpenALPR C++ library via Python bindings
PlateFormatValidator Validates plate against regional format rules Python regex + format DB
TrafficPoliceAPIClient Queries stolen vehicle and seizure databases aiohttp + Redis cache
FraudWatchlistChecker Queries internal watchlist for plate associations asyncpg + PostgreSQL

Architecture Diagram

┌──────────────────────────────────┐
│  POST /analyze                   │
│  (media[] + declared plate)      │
└──────────────┬───────────────────┘
               │
               ▼
┌──────────────────────────────────┐
│       FrameExtractor             │
│  (video → keyframes)             │
└──────────────┬───────────────────┘
               │
               ▼
┌──────────────────────────────────┐
│       YOLOv8Detector             │
│  (detect plate bboxes)           │
└──────────────┬───────────────────┘
               │
               ▼
┌──────────────────────────────────┐
│       PlatePreprocessor          │
│  (deskew + binarise)             │
└──────────┬───────────────────────┘
           │
    ┌──────┴──────┐
    ▼             ▼
┌──────────┐  ┌──────────┐
│Tesseract │  │OpenALPR  │
│  OCR     │  │  OCR     │
└──────┬───┘  └──────┬───┘
       └──────┬───────┘
              │ vote
              ▼
┌─────────────────────────┐
│   PlateFormatValidator  │
└──────────┬──────────────┘
           │
    ┌──────┴──────────────────┐
    ▼         ▼               ▼
┌──────────┐ ┌──────────┐ ┌──────────────┐
│ Traffic  │ │  Court   │ │   Fraud      │
│ Police   │ │ Seizure  │ │  Watchlist   │
│   API    │ │    DB    │ │  Checker     │
└──────┬───┘ └──────┬───┘ └──────┬───────┘
       └────────────┴─────────────┘
                    │
                    ▼
          JSON verdict

Data Flow

API Gateway FrameExtractor | Raw media binaries + declared plate
FrameExtractor YOLOv8Detector | JPEG frames
YOLOv8Detector PlatePreprocessor | Cropped plate region images
PlatePreprocessor TesseractOCR | Binarised plate image
PlatePreprocessor OpenALPROCR | Binarised plate image
TesseractOCR PlateFormatValidator | OCR string + character confidences
OpenALPROCR PlateFormatValidator | OCR string + character confidences
PlateFormatValidator TrafficPoliceAPIClient | Validated plate number
PlateFormatValidator FraudWatchlistChecker | Validated plate number
TrafficPoliceAPIClient API Gateway | Database query results + full verdict
Back to AI Agents AGT-MOT-006