# Random Corp

This repo is a reference end-to-end example implementation of Data Products using Markdown Data Definition Language (MD-DDL). This project implements data products for sourcing and the Financial Crime Domain.

The project leverages:

- https://git.semprini.me/paul/semprini-core to provide I&AM, Code Repos
- https://git.semprini.me/paul/semprini-data for data management, data product metadata, real-time data sourcing, and data transform platforms.
- https://git.semprini.me/paul/semprini-data-domain template for infrastructure to host foundational (domain aligned) and experience (consumer aligned) data products.

## Project Goals

Primary purpose is a feedback loop for the data domain template and markdown data definition language (MD-DDL) and become an exemplar of data domains. The domain template and md-ddl are included in this project as sub-modules only - not to be edited in this project - feedback and issues should be added into the projects git project issues.

### Phase 1:

1. A running, stable Financial Crime data domain instantiated from the domain template. This means:
  - Metadata defining the domain is all this project definies
  - The metadata together with the domain template creates a fully functional domain

3. Metadata driven integration into the rest of the ecosystem:
 - Core platform: Leverage of the core running systems for I&AM, certs, DNS etc. These systems should already be running and the design can be found in https://git.semprini.me/paul/semprini-core
 - Data management: Registration of the domain into the data management systems - data marketplace, data catalogue etc. These systems should already be running and the design can be found in https://git.semprini.me/paul/semprini-data

### Phase 2:

1. Synthetic data generators per defined source system - replicating realistic cadence of data change and system semantics in source systems. Source system definitions are found in the /sources folder. This will be achieved by:
 - A single source system Postgres DB server for all sources
 - Per source system database instances
 - Python (faker) synthetic data generator per source system
    - Ability to set change cadence
 - Use the MD-DDL source definitions to define schemas and synthetic generators as per md-ddl.

3. Per source system, source aligned data products. This will be defined once the synthetic data generators are running.
 - Deployed on the data bus stack. This should be running and the design can be found in https://git.semprini.me/paul/semprini-data

4. Creation of domain aligned data product templates. A template is purely driven by metadata (e.g. Open Data Product Specification/Schemas etc) Features of this class of data product are:
 - Kafka topics per aggregate root entity - enforcing canonical schema
 - GraphQL API endpoints for canonical schema
 - Kafka sync into PostgreSQL DB in real-time
 - pg_duckpipe ingest into ducklake
 - Analytics SQL interface to ducklake via pg_ducklake
 - Registration to data marketplace and catalog

5. Domain aligned data products. Driving the domain aligned data product template using metadata. Details to be defined later but the data products will be:
 - Party Core
 - Financial Transaction
 - Product Core

6. Consumer aligned data products. To be defined later.