Credentials for Azure aks agic identity refresh

Problem

I am deploying an aks cluster with agic (application gateway ingress controller) with bicep

This is declared in aks.bicep.

resource aksCluster 'Microsoft.ContainerService/managedClusters@2021-03-01' = {
  name: 'aks-core-${env}'
  location: resourceGroup().location
  identity: {
    type: 'SystemAssigned'
  }
  properties: {
    ...
    addonProfiles: {
      ...
      ingressApplicationGateway: {
        enabled: true
        config: {
          applicationGatewayId: applicationGatewayId
          effectiveApplicationGatewayId: applicationGatewayId
        }
      }

    }
  }
}

However, the identity produced by aks appears to require a role in the resource group created for the node pool, as evidenced by the error “requires contributor role,” so I added the following:

resource contributorRoleDefinition 'Microsoft.Authorization/roleDefinitions@2020-08-01-preview' existing = {
  scope: subscription()
  name: 'xxx'
}

resource aksfix 'Microsoft.Authorization/roleAssignments@2020-04-01-preview' = {
  name: guid(resourceGroup().id,'aksfix','Contributor')
  scope: resourceGroup()
  properties: {
    description: 'fixes aks cross resource group principal permissions for agic'
    principalId: aksCluster.properties.addonProfiles.ingressApplicationGateway.identity.objectId
    principalType: 'ServicePrincipal'
    roleDefinitionId: contributorRoleDefinition.id
  }
}

Now to be clear this WORKS, but not always. Even if the principal has the appropriate permissions, the same error keeps popping up for 30 min to 1-2 hours and then by magic it works!

I had a similar error when trying to use internal load balancers that require network contributor to the kubelet principal, and the same behavior appears, it takes an inordinate amount of time to reflect the role changes, now what seems interesting is a message on the agic pod that says “if your permissions changed, try refreshing your credentials,” is there a way to force a cluster to refresh its credentials (without creating a new service principal)?

Asked by Diego Alejandro Llanos Gareca

Solution #1

Both the aks magnaged cluster identity on the resource group where the application gateway resources are present and the ingress application gateway identity on the resource group should be assigned Contributor roles.

As a result, it will be preferable to utilise a user-assigned identity and provide the contributor position in AKS identification, as shown below:

resource aksClusterUserDefinedManagedIdentity 'Microsoft.ManagedIdentity/userAssignedIdentities@2018-11-30' = {
  name: 'aksClusterUserDefinedManagedIdentityName'
  location: resourceGroup().location
}
resource akscontributorroleassignement 'Microsoft.Authorization/roleAssignments@2020-10-01-preview' = {
  name: guid(concat(resourceGroup().id, aksClusterUserDefinedManagedIdentity.name,aksclustername))
  scope: resourceGroup()
  properties: {
    description: 'Contributor role to the AKS identity to access the AGIC reosurce'
    principalId: aksClusterUserDefinedManagedIdentity.properties.principalId
    principalType: 'ServicePrincipal'
    roleDefinitionId: subscriptionResourceId('Microsoft.Authorization/roleDefinitions', 'b24988ac-6180-42a0-ab88-20f7382dd24c')
  }
}
resource aksCluster 'Microsoft.ContainerService/managedClusters@2021-03-01' = {
  name: 'aks-core-${env}'
  dependsOn:akscontributorroleassignement
  location: resourceGroup().location
  identity: {
    type: 'UserAssigned',
    userAssignedIdentities: {aksClusterUserDefinedManagedIdentity.id}
  }
  properties: {
    ...
    addonProfiles: {
      ...
      ingressApplicationGateway: {
        enabled: true
        config: {
          applicationGatewayId: applicationGatewayId
          effectiveApplicationGatewayId: applicationGatewayId
        }
      }

    }
  }
}


resource contributorRoleDefinition 'Microsoft.Authorization/roleDefinitions@2020-08-01-preview' existing = {
  scope: subscription()
  name: 'xxx'
}

resource aksfix 'Microsoft.Authorization/roleAssignments@2020-04-01-preview' = {
  name: guid(resourceGroup().id,'aksfix','Contributor')
  scope: resourceGroup()
  properties: {
    description: 'fixes aks cross resource group principal permissions for agic'
    principalId: aksCluster.properties.addonProfiles.ingressApplicationGateway.identity.objectId
    principalType: 'ServicePrincipal'
    roleDefinitionId: contributorRoleDefinition.id
  }
}

Answered by AnsumanBal-MT

Post is based on https://stackoverflow.com/questions/70688225/azure-aks-agic-identity-refresh-credentials