For a datamesh implementation, we need to be able to use a devops project as backbone. Take a look at the following architectural concept:

Datamesh Devops Orchestration Architecture

The architecture shows that a Pipeline within the management project is setting permissions for the data domain repositories and pipelines. The following permissions for repositories can be set:

PermissionCLI Permission NameComment
Bypass policies when completing pull requestsPullRequestBypassPolicy
Bypass policies when pushingPolicyExempt
Contribute to pull requestsPullRequestContribute
Create branchCreateBranch
Create tagCreateTag
Delete or disable repositoryDeleteRepository
Edit policiesEditPolicies
Force push (rewrite history, delete branches and tags)ForcePush
Manage notesManageNote
Manage permissionsManagePermissions
Remove others’ locksRemoveOthersLocks
Rename repositoryRenameRepository
Azure Devops Repository Permissions CLI

In order to assign the permissions, within the mesh we create two groups of users:

  1. Data Domain Engineers
  2. Data Engineers

The data domain engineer actually takes care about the infrastructure within the domain and is able to adapt the minimal infrastructure and policies with custom infrastructure required by the data domain.

The data engineer is responsible for the delivery of data products and all ingestion, transformation and load processes.

On the 1st of November 2022 the only way to set rights and permissions in DevOps overlapping projects is done using the bash within the pipeline in the context of a user – using a PAT (Personal Access Token) of that specific user.

The code can be split in 4 logical steps:

  1. Fetch all information needed for the environment
  2. Create the groups while not existing
  3. Assign the user to the recently created groups
  4. Assign the permissions to the user groups
##### Fetch Environment ##### org="<yourorganisation>" data_engineer_security_group_name="DataEngineers" data_domain_engineer_security_group_name="DataDomainEngineers" echo ${PAT} | az devops login --organization $org ###### Permissions Group ###### az devops security group create --name $data_engineer_security_group_name --description 'All data engineers that use this data domain' --organization $org --project $project; az devops security group create --name $data_domain_engineer_security_group_name--description 'All data domain engineers that use this data domain' --organization $org --project $project; data_engineer_security_group_descriptor=$(az devops security group list --organization $org --project $project --query "graphGroups[?displayName=='$data_engineer_security_group_name'].descriptor" --output tsv) data_domain_engineer_security_group_descriptor=$(az devops security group list --organization $org --project $project --query "graphGroups[?displayName=='$data_domain_engineer_security_group_name'].descriptor" --output tsv) project_id=$(az devops project list --organization $org --query "value[?name == '$project'].id" --output tsv) ###### Repository ###### data_engineer_repository_id=$(az repos list --organization $org --project $project --query "[?name == '$data_engineer_repository'].id" --output tsv) data_domain_engineer_repository_id=$(az repos list --organization $org --project $project --query "[?name == '$data_domain_engineer_repository'].id" --output tsv) namespace_git_repositories=$(az devops security permission namespace list --org $org --query "[? == 'Git Repositories'].namespaceId | [0]" --output tsv) namespace_build=$(az devops security permission namespace list --org $org --query "[? == 'Build'].namespaceId | [0]" --output tsv) ##### Roles an data engineer needs to access in the repository ##### declare -a arr=("PullRequestContribute" "CreateTag" "CreateBranch" "ForcePush" "GenericContribute" "GenericRead") for role in "${arr[@]}" do bit=$(az devops security permission namespace show --namespace-id $namespace_git_repositories --org $org --query "[0].actions[?name == '$role'].bit" --output tsv) az devops security permission update --allow-bit $bit --namespace-id $namespace_git_repositories --organization $org --subject $data_engineer_security_group_descriptor --token "repoV2/$project_id/$data_engineer_repository_id" az devops security permission update --allow-bit $bit --namespace-id $namespace_git_repositories --organization $org --subject $data_domain_engineer_security_group_descriptor --token "repoV2/$project_id/$data_domain_engineer_repository_id" done
Code-Sprache: HTML, XML (xml)

This code needs to embedded into a bash step:

stages: - stage: ${{ parameters.stage_name }} jobs: - job: permission_group displayName: Create permissions group and configure them with the proper access in repositories and pipelines steps: - bash: | <code from above> env: PAT: $(PAT) project: ${{ parameters.project }} data_engineer_repository: ${{ parameters.data_engineer_repository }} data_domain_engineer_repository: ${{ parameters.data_domain_engineer_repository }} pipeline: ${{ parameters.pipeline }}
Code-Sprache: YAML (yaml)