DevOps set Project, Repository and Pipeline Security via Pipeline

For a datamesh implementation, we need to be able to use a devops project as backbone. Take a look at the following architectural concept:

The architecture shows that a Pipeline within the management project is setting permissions for the data domain repositories and pipelines. The following permissions for repositories can be set:

Permission	CLI Permission Name	Comment
Bypass policies when completing pull requests	PullRequestBypassPolicy
Bypass policies when pushing	PolicyExempt
Contribute	GenericContribute
Contribute to pull requests	PullRequestContribute
Create branch	CreateBranch
Create tag	CreateTag
Delete or disable repository	DeleteRepository
Edit policies	EditPolicies
Force push (rewrite history, delete branches and tags)	ForcePush
Manage notes	ManageNote
Manage permissions	ManagePermissions
Read	GenericRead
Remove others’ locks	RemoveOthersLocks
Rename repository	RenameRepository

In order to assign the permissions, within the mesh we create two groups of users:

1. Data Domain Engineers
2. Data Engineers

The data domain engineer actually takes care about the infrastructure within the domain and is able to adapt the minimal infrastructure and policies with custom infrastructure required by the data domain.

The data engineer is responsible for the delivery of data products and all ingestion, transformation and load processes.

On the 1st of November 2022 the only way to set rights and permissions in DevOps overlapping projects is done using the bash within the pipeline in the context of a user – using a PAT (Personal Access Token) of that specific user.

The code can be split in 4 logical steps:

1. Fetch all information needed for the environment
2. Create the groups while not existing
3. Assign the user to the recently created groups
4. Assign the permissions to the user groups

##### Fetch Environment #####
org="https://dev.azure.com/<yourorganisation>"
data_engineer_security_group_name="DataEngineers"
data_domain_engineer_security_group_name="DataDomainEngineers"

echo ${PAT} | az devops login --organization $org
          
###### Permissions Group ######
az devops security group create --name $data_engineer_security_group_name --description 'All data engineers that use this data domain' --organization $org --project $project;
az devops security group create --name $data_domain_engineer_security_group_name--description 'All data domain engineers that use this data domain' --organization $org --project $project;
data_engineer_security_group_descriptor=$(az devops security group list  --organization $org --project $project --query "graphGroups[?displayName=='$data_engineer_security_group_name'].descriptor" --output tsv)
data_domain_engineer_security_group_descriptor=$(az devops security group list  --organization $org --project $project --query "graphGroups[?displayName=='$data_domain_engineer_security_group_name'].descriptor" --output tsv)

project_id=$(az devops project list --organization $org --query "value[?name == '$project'].id" --output tsv)

###### Repository ######

data_engineer_repository_id=$(az repos list --organization $org --project $project --query "[?name == '$data_engineer_repository'].id" --output tsv)
data_domain_engineer_repository_id=$(az repos list --organization $org --project $project --query "[?name == '$data_domain_engineer_repository'].id" --output tsv)
namespace_git_repositories=$(az devops security permission namespace list --org $org --query "[?@.name == 'Git Repositories'].namespaceId | [0]" --output tsv)
namespace_build=$(az devops security permission namespace list --org $org --query "[?@.name == 'Build'].namespaceId | [0]" --output tsv)

##### Roles an data engineer needs to access in the repository #####
declare -a arr=("PullRequestContribute" "CreateTag" "CreateBranch" "ForcePush" "GenericContribute" "GenericRead")
for role in "${arr[@]}"
do
     bit=$(az devops security permission namespace show --namespace-id $namespace_git_repositories --org $org --query "[0].actions[?name == '$role'].bit" --output tsv)
     az devops security permission update --allow-bit $bit --namespace-id $namespace_git_repositories --organization $org --subject $data_engineer_security_group_descriptor --token "repoV2/$project_id/$data_engineer_repository_id"
     az devops security permission update --allow-bit $bit --namespace-id $namespace_git_repositories --organization $org --subject $data_domain_engineer_security_group_descriptor --token "repoV2/$project_id/$data_domain_engineer_repository_id"
done


Code-Sprache: HTML, XML (xml)

This code needs to embedded into a bash step:

stages:
- stage: ${{ parameters.stage_name }}
  jobs:
  - job: permission_group
    displayName: Create permissions group and configure them with the proper access in repositories and pipelines
    steps:
    - bash: |
        <code from above>
      env:
        PAT: $(PAT)
        project: ${{ parameters.project }}
        data_engineer_repository: ${{ parameters.data_engineer_repository }}
        data_domain_engineer_repository: ${{ parameters.data_domain_engineer_repository }}
        pipeline: ${{ parameters.pipeline }} 
Code-Sprache: YAML (yaml)