If you are using Azure Data Factory, you have probably built or debugged a pipeline. If you find yourself constantly creating pipelines from scratch, consider using 'My Templates.'
'My Templates' enable you to create your own standardised pipeline templates, allowing you to get started quickly.
Introduction
Azure Data Factory (ADF) comes already with a handful of built-in templates to allow you to get started quickly. You can find the available built-in templates by navigating to the ADF UI and navigating to the template gallery:
Each template has a short description about its purpose. Of course, you can extend/modify the selected template option as per your needs. Examples of built-in templates:
These templates are meant to get you off the ground. However, if you are searching for something specific you may not find it in here. In this case, you often start with an empty pipeline and build it up with ADF activities - all from scratch. If you feel that the new pipeline can be reused for other projects/pipelines, you may wish to create a template to make it more universal. In thig blog post, we will look at the 'My Templates' option to create a standardised template.
Benefits of using 'My Template'
- Your secret weapon for kicking off pipeline projects at lightning speed, slashing developers' setup time to near zero.
- Rock-solid templates set the gold standard, fostering stellar practices across the team and organisation, while paving the way for a seamless pipeline creation.
-
Templates help establish best practices for data integration, minimise errors, and maintain consistency throughout projects.
- Templates can be customised to meet specific needs, making them versatile for different clients and data sources.
Creating 'My Template'
My Templates require the ADF instance to be linked to a repos. Once linked, any saved templates are created in a folder (/templates folder in repos). The template can be utilised by simply selecting in through the ADF Template Gallery. Any references, such as linked services, will need to be resolved. Therefore knowing the pre-requisites for each template is beneficial.
After linking the ADF instance to a repos, we can visually confirm this by looking at the ADF UI - you should see something similar to the below screenshot (top left red box). Depending on your repos branching strategy, you may need to create a new branch (e.g. feature branch). Now you can build your standardised pipeline. Try to keep any parameters or names standardised so it can be re-used with the same names/parameters across different projects/clients/environments. Once you are happy, you can save this pipeline as a template (see the screenshot below 'Save as template' - this option is only available if you have linked your ADF instance to a repos!).
In the above example, a single activity was chosen for demonstration purposes.
When clicking the 'Save as template' option, you will be prompted to provide the template with some metadata. It is recommended to fill in all the metadata configuration to allow yourself AND other users to identify the purpose of each template.
Using 'My Template'
Now when opening up the template gallery, we can see the new template:
When choosing your template, you may need to resolve any dependencies if they do not exist already, e.g. linked services.
Behind the Scenes
When navigating to the ADF repos, we can see a new folder and JSON files under the '/templates' path (for this demo I used Azure DevOps Repos):
When viewing the contents of the JSON files, we can see our pipeline described with all its configuration. You may find references to dependencies such as linked services. The expectation is when you utilise this template, any dependencies are resolved at that very moment. It is best to create documentation around any dependencies/prerequisites for a given template to aid other users when using your template. The user trying to utilise the template could ensure that the dependencies exist beforehand, e.g. Linked Services.
'My Templates' become powerful when you deploy ADF using a Infrastructure as a Code (IaC) approach, e.g. by using Terraform / Azure DevOps. You can deploy multiple standardised templates in different environments/customers/projects and select the most suitable template to get started with.
Conclusion
ADF Template Gallery provides you with a vast choice of built-in templates to allow you to get started quickly. You can choose a template and customise it as per your needs.
My Templates are custom templates, often built by a user, e.g. work colleague, allowing common pipelines to be reused quickly without the need to rebuild the pipeline from scratch - saving mundane tasks and more importantly saving time! My Templates require the ADF instance to be linked to a Repos. Once linked, any saved templates are created in a folder named '/templates' in repos. The template can be utilised by simply selecting it through the ADF Template Gallery. Any references, such as linked services, will need to be resolved to at that very moment therefore knowing the pre-requisites for each template is beneficial.
If you’re looking for assistance in how you can best approach your data platform implementation, Advancing Analytics can help! Contact us to talk to our Engineering team today.
*Disclaimer: The blog post image was generated by AI and does not depict any real person, place, or event.
References
