Automating DevOps to Create Standard Tasks

On an Integration project there is a significant amount of time and effort managing the delivery of work. In almost all cases you will be delivering integration solutions which requrire the involvement of many teams such as:

• The business analyst team who will be dealing with requirements
• The application teams who look after the applications you will be integrating with and who may need to develop changes to support the integration scenarios
• Support teams who will help look after the interfaces you build
• Security teams who will be providing governance and protection of the systems and data you will integrate
• Enterprise architecture who will ensure the solution is in adherence with the standards and patterns for the organisation

One of the things that makes integration projects hard to manage is it can feel like each interface is almost its own new project.

One of the things we have done recently that works pretty well is use Power Automate to automate the generation of standard tasks in Azure DevOps so we can get everyone used to a consistent way of working and expectation on what needs to be done for each interface.

We know each interface can be different and some will need new things and others will not need some of the standard tasks but having a consistent approach can give you a great place to start to help make everyones life easier.

In devops you might be using 1 of a number of different templates for work item management. We are using CMMI at the moment because it best fits a big multi-project IT program, but you can also use this approach with scrum and agile and just modify the tasks you want to suit.

Process

The process we have involves a new requirement coming to the team. Often in the rel world these requirements can be wooly and inclear. You might range from getting a requirements document to getting a few emails with some bullet points. You might also get things like a recording of some meetings.

One of the most painful parts of the requirements process in integration projects is working out what we need to do.

The process we follow involves a series of steps where at each point we use Power Automate to apply a set of work items as children to your requirement or feature work item to help you list all of the work thats needed to complete that step. The steps we use are:

• Step 1 - Create Requirement or Feature work item
• Step 2 - Run our flow to add the requirements capture work items
• Step 3 - Run our flow to add the high level design and elaboration work items
• Step 4 - Run our flow to add the design, development, testing and deployment work items

Step 1 - Create Requirement or Feature work item

At the start we will create a feature or requirement work item in our backlog. This depends on the devops process you are following. From a portfolio planning perspective we group our work items under epics so we have a tree structure representing the programs and projects that the team is working on and each integration requirement will live under that.

We will set a few attributes about the requirement which ill talk about in a future post.

Step 2 - Run our flow to add the requirements capture work items

When we have the requirement/feature the next step is to use our Power Automate Flow and we will trigger it with the flow button so its easy to run it from your mobile or in the browser and we need 2 bits of information.

• Work Item ID for the requirement/feature
• The name of the manifest file to use

The manifest file is a json file which describes the work items we want to apply to this feature/requirement.

At this step we will use a flow (which ill show later) which will read the manifest file from SharePoint, it will get the work item you have supplied when triggering the flow and the flow will look over the json file which contains a list of items which will be used to create product back log work items that will be children and then for each product backlog item it will create tasks under it that are needed to complete that PBI.

The flow will also take some properties from the parent feature/requirement and copy them to the child work items and some data from the file to populate properties.

At the end of the flow running we will have something like below.

You can customize the names of items and list of tasks depending on what works for your project.

At this point you now have a clear set of things that need to be done to complete the collection of requirements to get all of the things your team will usually need. Obviously you can manually add more tasks to devops if needed once the standard ones are setup.

Step 3 - Run our flow to add the high level design and elaboration work items

Once all of the requirement gathering is complete or progressed far enough that we have enough information to layout the next bit of work we need to do we will go back to power automate and add the next set of tasks.

We will use the same Flow and work item id but this time we will supply the name for a different json manifest file in SharePoint and it will add all of our tasks that are needed to get a high level design done, reviewed and work planned out across the teams needed to deliver the interface.

The work items we setup look like the below:

This gives us a clear path of stuff we need completed, reviewed and planned to get ready for development.

Step 4 - Run our flow to add the design, development, testing and deployment work items

When all of the design and planning is ready we can then use the design to help us work out which work is required to complete the interface requirement.

Sometimes its just a small change that just needs the integration team, sometimes its a bit bigger and needs some support from another team and sometimes it can be much bigger and involves people from multiple teams being involved.

We use the same approach but we have a different manifest file for the work required in each scenario and we just run our flow again with the parent work item id and pass the file name for the file we want to use to setup work items. It will setup the tasks depending on how we thing the work needs to be performed.

Below is an example of the additional PBI for our full interface build which covers the more complex requirements.

Note I have collapsed the work items in the tree in the above picture because it makes it easier to see whats going on but they all have tasks under them to let us see what needs to be done.

Automation of Work Item Creation with Power Automate

If we now take a look at the flow that does this for us. You can see below it starts off using the flow button trigger to capture the input data ive mentioned a few times. It then gets the parent work item from Azure DevOps using the out of the box get work item details task and also goes and gets a file from SharePoint with the manifest file in it.

Using Get File Content using path I am able to have a single folder in sharepoint with the manifest files in them and just change the name depending on the input parameter.

I then parse the json returned from sharepoint using the json parse action. My schema is below:

{
    "type": "object",
    "properties": {
        "TeamDeliverables": {
            "type": "array",
            "items": {
                "type": "object",
                "properties": {
                    "Name": {
                        "type": "string"
                    },
                    "SolutionArea": {
                        "type": "string"
                    },
                    "OwnerTeam": {
                        "type": "string"
                    },
                    "Activity": {
                        "type": "string"
                    },
                    "Tasks": {
                        "type": "array",
                        "items": {
                            "type": "string"
                        }
                    }
                },
                "required": [
                    "Name",
                    "SolutionArea",
                    "OwnerTeam",
                    "Activity",
                    "Tasks"
                ]
            }
        }
    }
}

I then iterate over the collection in the file and create a PBI for each child which is linked to the parent requirement/feature and then for each item in the file I iterate over its child tasks in the json file and create work items for those too as shown below.

A couple of interesting points to note if your new to using the Azure DevOps connector are:

• You can set custom properties like shown in the diagram below with the bottom arrow. Im taking these from the json file but you can also copy them from the parent work item
• You can set the parent relationship using the link type Heirachy-Reverse with the url to the parent work item. You set the child work items parent like this.

Thats pretty much it for the Power Automate. Its not a complicated flow but combining it with a process like this it helps save us a load of time in setting up tasks in Azure DevOps because we can automate most of the creation and then we also benefit from consistency and standardization by default across projects.

# # What might we do next?
This is pretty cool so far, one of the things we might do next would be to help our estimation. Im thinking that we could use a t-shirt sizing approach for the requirement/feature and then when we generate work items at each stage we could then extend the json manifest to calculate an effort estimate and maybe remaining hours for PBI and maybe even tasks. We could obviously then modify them as required as we elaborate and plan work but from this initial T-Shirt size we could begin to get some ideas on sizings and work durations which could help automate some of our planning.

Summary

I hope this idea might give others some ideas on how devops automation might help you manage work effectively on an integration project and also help to reduce the admin time needed to use DevOps effectively.

Id love to hear if others do anything similar.

Some additional resources which might help are below.

Manifest File Examples

Below are some example manifest files ive used.

Requirements Collection

{
    "TeamDeliverables" : [
        {
            "Name":"Requirements - Collection",
            "SolutionArea" : "Program",
            "OwnerTeam" : "Business Analyst",
            "Activity" : "Requirements",
            "Tasks" : [                
                    "DevOps requirement work item is described",
                    "Source and destination systems added to one note",
                    "Feature definition in one note",
                    "Business context in one note",
                    "User journey documented in one note",
                    "Green Path Scenario documented in one note",
                    "Exception cases documented in one note",
                    "Non functional requirements documented in one note",
                    "Assumptions documented in one note",
                    "Data mappings documented in one note",                    
                    "Requirements questions in one note are answered",
                    "Review by Source Application Team",
                    "Review by Destination Application Team",
                    "Review by Integration Team",
                    "Review by Test Team"
                ]
        }            
    ]
}

High Level Design and Plan

{
    "TeamDeliverables" : [               
        {
            "Name":"Design - High Level Architecture",
            "SolutionArea" : "Integration",
            "OwnerTeam" : "Integration Team",
            "Activity" : "Design",
            "Tasks" : [                
                    "Review Requirements",
                    "High Level Design Proposal Created",
                    "Stakeholders Identified and added to One Note",
                    "Requirements linked to interface catalog",
                    "Add Stakeholder teams to later work items"                    
                ]
        },
        {
            "Name":"Design - High Level Design - Planning",
            "SolutionArea" : "Program",
            "OwnerTeam" : "Project Manager",
            "Activity" : "Planning",
            "Tasks" : [                
                    "Identify Resources to review design",
                    "Schedule when designs will be reviewed",                    
                    "Chase down review completion",
                    "Dependency projects and interfaces are planned"                
                ]
        },
        {
            "Name":"Design - High Level Design - Reviews",
            "SolutionArea" : "Program",
            "OwnerTeam" : "Project Manager",
            "Activity" : "Planning",
            "Tasks" : [                
                    "Business Analyst Review",
                    "Source Application Team Review",
                    "Destination Application Team Review",
                    "Security Team Review",
                    "Integration Team Review",
                    "Testing Team Review"
                ]
        },
        {
            "Name":"Requirements - Additional Requirements for DBT",
            "SolutionArea" : "Program",
            "OwnerTeam" : "Testing",
            "Activity" : "Design",
            "Tasks" : [                
                    "Answer requirement questions",
                    "Get additional level of data mappings"
                ]
        }            
    ]
}

Full Build / Test

{
    "TeamDeliverables" : [               
        {
            "Name":"Design - Test Case Design",
            "SolutionArea" : "Program",
            "OwnerTeam" : "Testing",
            "Activity" : "Design",
            "Tasks" : [                
                    "Write System Test Cases",
                    "Write UAT Test Cases"
                ]
        },
        {
            "Name":"Design - Detailed - Source Application Team",
            "SolutionArea" : "",
            "OwnerTeam" : "",
            "Activity" : "Design",
            "Tasks" : [                
                    "Add list of tasks to complete any detailed design",
                    "Complete Detailed Design",                    
                    "Add list of tasks to team build item"
                ]
        },
        {
            "Name":"Design - Detailed - Destination Application Team",
            "SolutionArea" : "",
            "OwnerTeam" : "",
            "Activity" : "Design",
            "Tasks" : [                
                    "Add list of tasks to complete any detailed design",
                    "Complete Detailed Design",                    
                    "Add list of tasks to team build item"
                ]
        },
        {
            "Name":"Design - Detailed - Integration Team",
            "SolutionArea" : "Integration",
            "OwnerTeam" : "Integration Team",
            "Activity" : "Design",
            "Tasks" : [                
                    "Review source application detailed design",
                    "Review destination application detailed design",
                    "Complete Integration Detailed Design",                    
                    "Add list of tasks to team build item",
                    "Add estimation for build tasks",
                    "Add dependencies to one note"
                ]
        },
        {
            "Name":"Build - Source Application Team",
            "SolutionArea" : "",
            "OwnerTeam" : "",
            "Activity" : "Development",
            "Tasks" : [                
                    "Complete build",
                    "Unit Test",
                    "Deploy to System Test",
                    "Deploy to UAT"
                ]
        },
        {
            "Name":"Build - Destination Application Team",
            "SolutionArea" : "",
            "OwnerTeam" : "",
            "Activity" : "Development",
            "Tasks" : [                
                    "Complete build",
                    "Unit Test",
                    "Deploy to System Test",
                    "Deploy to UAT"
                ]
        },
        {
            "Name":"Build - Integration Team",
            "SolutionArea" : "Integration",
            "OwnerTeam" : "Integration Team",
            "Activity" : "Development",
            "Tasks" : [                
                    "Complete build",
                    "Unit Test",
                    "Deploy to System Test",
                    "Deploy to UAT"
                ]
        },        
        {
            "Name":"Test - System Testing",
            "SolutionArea" : "Program",
            "OwnerTeam" : "Testing",
            "Activity" : "Testing",
            "Tasks" : [                
                    "Complete System Testing"
            ]
        },
        {
            "Name":"Test - UAT Testing",
            "SolutionArea" : "Program",
            "OwnerTeam" : "Testing",
            "Activity" : "Testing",
            "Tasks" : [                
                    "Complete UAT Testing",
                    "Complete UAT Sign Off Documentation",
                    "Add note to requirement with location of sign off"
            ]
        },
        {
            "Name":"Deploy - Release",
            "SolutionArea" : "Integration",
            "OwnerTeam" : "Integration Team",
            "Activity" : "Deployment",
            "Tasks" : [                
                    "Raise Change Control",
                    "Add CAB number to requirement work item",
                    "Add collect test evidence for CAB",
                    "Add implementation plan",
                    "Verify documentation"
            ]
        }
    ]
}

Small Integration Team Only Build / Test

{
    "TeamDeliverables" : [
        {
            "Name":"Requirements - Collection",
            "SolutionArea" : "Program",
            "OwnerTeam" : "Business Analyst",
            "Activity" : "Requirements",
            "Tasks" : [                
                    "DevOps requirement work item is described",                    
                    "Data mappings documented in one note",                    
                    "Requirements questions in one note are answered",                    
                    "Review by Integration Team"                    
                ]
        },        
        {
            "Name":"Design - High Level Architecture",
            "SolutionArea" : "Integration",
            "OwnerTeam" : "Integration Team",
            "Activity" : "Design",
            "Tasks" : [                                    
                    "Requirements linked to interface catalog",                  
                    "Documentation updates done"
                ]
        },
        {
            "Name":"Design - Test Case Design",
            "SolutionArea" : "Program",
            "OwnerTeam" : "Testing",
            "Activity" : "Design",
            "Tasks" : [                
                    "Write System Test Cases",
                    "Write UAT Test Cases"
                ]
        },
        {
            "Name":"Build - Integration Team",
            "SolutionArea" : "Integration",
            "OwnerTeam" : "Integration Team",
            "Activity" : "Development",
            "Tasks" : [                
                    "Complete build",
                    "Unit Test",
                    "Deploy to System Test",
                    "Deploy to UAT"
                ]
        },        
        {
            "Name":"Test - System Testing",
            "SolutionArea" : "Program",
            "OwnerTeam" : "Testing",
            "Activity" : "Testing",
            "Tasks" : [                
                    "Complete System Testing"
            ]
        },
        {
            "Name":"Test - UAT Testing",
            "SolutionArea" : "Program",
            "OwnerTeam" : "Testing",
            "Activity" : "Testing",
            "Tasks" : [                
                    "Complete UAT Testing",
                    "Complete UAT Sign Off Documentation",
                    "Add note to requirement with location of sign off"
            ]
        },
        {
            "Name":"Deploy - Release",
            "SolutionArea" : "Integration",
            "OwnerTeam" : "Integration Team",
            "Activity" : "Deployment",
            "Tasks" : [                
                    "Raise Change Control",
                    "Add CAB number to requirement work item",
                    "Add collect test evidence for CAB",
                    "Add implementation plan"
            ]
        }
    ]
}