What means Activation and Publishing?
In my context the activation of a data product (at least 1 outport) means 2 things to me:
- Inform the data mesh platform management about the availability of the data product
- Sign-Up for at least one event which will trigger the pipeline to build the data product
For the publishing of the data product I see the following thing:
- Upsert the meta data for each outport that’s included within the data product
Technical Workflow Overview
Here I describe the technical flow of developing and setting an azure synapse (jupiter notebook) based data product productive:
Activation of the data product
After looking for an event where we can attach the activation of a data product – we finaly came to the result, that this needs to be done by the developer manual. It’s actually a “one-time” action and needs to be decided by the publisher.
Publishing of the data product
For the publishing of the data product – upsert the meta-data – we decided to use a custom python library within the spark notebook. This library actually connects to the manament api of the platform and updates the catalog with the latest meta data of the outports.