paul
0aeb893c0f
- Added Makefile for orchestrating domain instantiation and management. - Created deploy README for guiding the setup process of the Financial Crime domain. - Introduced domain configuration file for mapping metadata to the deployment template. - Implemented pre-commit hook for validating metadata before commits. - Developed instantiation script to manage the lifecycle of the Financial Crime domain. - Added Salesforce CRM source with associated metadata and transformation files. - Added SAP Fraud Management source with associated metadata and transformation files. - Added Temenos Payment source with associated metadata and transformation files. - Removed obsolete payment event and parties transformation files.
Random Corp
This repo is a reference end-to-end example implementation of Data Products using Markdown Data Definition Language (MD-DDL). This project implements data products for sourcing and the Financial Crime Domain.
The project leverages:
- https://git.semprini.me/paul/semprini-core to provide I&AM, Code Repos
- https://git.semprini.me/paul/semprini-data for data management, data product metadata, real-time data sourcing, and data transform platforms.
- https://git.semprini.me/paul/semprini-data-domain template for infrastructure to host foundational (domain aligned) and experience (consumer aligned) data products.
Project Goals
Primary purpose is a feedback loop for the data domain template and markdown data definition language (MD-DDL) and become an exemplar of data domains. The domain template and md-ddl are included in this project as sub-modules only - not to be edited in this project - feedback and issues should be added into the projects git project issues.
Phase 1:
- A running, stable Financial Crime data domain instantiated from the domain template. This means:
- Metadata defining the domain is all this project definies
- The metadata together with the domain template creates a fully functional domain
- Metadata driven integration into the rest of the ecosystem:
- Core platform: Leverage of the core running systems for I&AM, certs, DNS etc. These systems should already be running and the design can be found in https://git.semprini.me/paul/semprini-core
- Data management: Registration of the domain into the data management systems - data marketplace, data catalogue etc. These systems should already be running and the design can be found in https://git.semprini.me/paul/semprini-data
Phase 2:
- Synthetic data generators per defined source system - replicating realistic cadence of data change and system semantics in source systems. Source system definitions are found in the /sources folder. This will be achieved by:
- A single source system Postgres DB server for all sources
- Per source system database instances
- Python (faker) synthetic data generator per source system
- Ability to set change cadence
- Use the MD-DDL source definitions to define schemas and synthetic generators as per md-ddl.
- Per source system, source aligned data products. This will be defined once the synthetic data generators are running.
- Deployed on the data bus stack. This should be running and the design can be found in https://git.semprini.me/paul/semprini-data
- Creation of domain aligned data product templates. A template is purely driven by metadata (e.g. Open Data Product Specification/Schemas etc) Features of this class of data product are:
- Kafka topics per aggregate root entity - enforcing canonical schema
- GraphQL API endpoints for canonical schema
- Kafka sync into PostgreSQL DB in real-time
- pg_duckpipe ingest into ducklake
- Analytics SQL interface to ducklake via pg_ducklake
- Registration to data marketplace and catalog
- Domain aligned data products. Driving the domain aligned data product template using metadata. Details to be defined later but the data products will be:
- Party Core
- Financial Transaction
- Product Core
- Consumer aligned data products. To be defined later.