By: Ron L'Esteve | Updated: 2022-10-05 | Comments | Related: > Cloud Strategy
Problem
DevOps is a hot topic that is increasing in demand within the information technology (IT) industry. Since DevOps is more about process and methodology than technology, customers often lack clarity around how to design and implement DevOps within their organizations. Furthermore, buzz words such as machine learning operations (MLOps), continuous integration and continuous deployment (CI/CD), Infrastructure as code (IaC), Agile software development and more complicate people's comprehension of what it truly means to be a 'DevOps' driven team and organization. Specifically, as customers begin their cloud adoption journey, they seek to gain a deeper understanding of exactly what DevOps is, along with what are some common DevOps practices and tools. As customers plan and design DevOps solutions within their organization, they want to justify the benefits of DevOps within their Enterprise Cloud Data Lakehouse Platforms and realize how best to implement DevOps to enhance their cloud transformation journey and deliver business value to their stakeholders.
Solution
The term 'DevOps' has traditionally been defined as a collaborative approach that unifies the development and operations teams within an organization with the intent to change how the two teams go about performing their tasks. The broad term 'DevOps' can be thought of as a culture, methodology, and philosophy which changes how different teams in an organization work together to achieve their business goals with the support of tools and processes. Much like the Agile software development methodologies and frameworks, DevOps is a set of practices that combines software development and IT operations with the goal of minimizing the complexities of the software development lifecycle (SDLC) through continuous integration (CI) and continuous delivery (CD) of high-quality software.
This article aims to cover the key roles DevOps plays within the cloud transformation and innovation space as organizations embark on their cloud adoption journey. As customers seek to mature their SDLC processes through automated continuous integration (CI) and continuous deployment (CD) of their cloud data and advanced analytics platform's technology components, we will dive into the details of how they can truly deliver an accelerated time to market and collaborative experience for all the stakeholders that are part of this process.
Top 5 Benefits of DevOps
There are several benefits to infusing DevOps tools and processes within your organization. The following captures the top five benefits of DevOps.
Collaboration
DevOps promotes an environment where different teams can work together closely to achieve common goals by facilitating collaboration by breaking down visible silos across several teams, including development, operations, quality assurance, and others. The collaborative approach of DevOps encourages these teams to work together towards a common goal of creating more organizational value by providing them with the tools and recommended processes to do so efficiently.
Accelerate Time to Market
DevOps accelerates the pace for delivering quality software to customers through automated continuous integration and delivery models, which can account for automated testing and near real-time reviews through automated approval gates, tightening the feedback loops and leading to faster bug fixes and production-ready solutions delivered directly to consumers.
High Performing Teams
Rapid delivery of software products leads developers to shift their focus on more value-added and innovative work efforts that enhance the productivity of the collective teams who benefit from DevOps tools and iterative processes.
Improved Software Quality and Performance
Since testing is a critical component of DevOps, it ensures high-quality software is constantly delivered through rigorous automated testing procedures across multiple environments and approval gates until the software meets the highest quality and performance standards to obtain full approval for production-grade deployments.
Frequent Iterative Software Deployments
With automated integration and deployment process and pipelines driving continuous software deployments, DevOps-infused software leads to faster and more frequent delivery of iterative updates and features, which both delight customers and support the growth of organizational initiatives to remain highly competitive in the ever-evolving marketplace.
Agile and DevOps
Since Agile and DevOps both aim toward delivering quality software in a timely manner, it is worth briefly discussing their similarities and unique attributes. The difference between Agile and DevOps is that Agile focuses on optimizing the development life-cycle, while DevOps unites development and operations in a CI/CD environment. DevOps and Agile are not mutually exclusive since DevOps is an extension of Agile built around practices that are not in Agile's focus. Both practices improve software development and lead to better products when used together. The chart below compares features similar to both Agile with DevOps.
DevOps Practices and Tools
There are quite a few DevOps practices and tools that developers and organizations use frequently. Many of these tools are available on major cloud platforms. This section will detail many of these DevOps tools and processes. The figure below illustrates the DevOps infinity loop, which describes the continuous integration and development process at the heart of DevOps.
Planning
During the planning stage, multiple teams, including development, IT operations, and business stakeholders, come together to determine desired features for the various stages of a defined project and the overall development plan. Project leaders, including Project Managers and Scrum Masters, lead these project plans in an Agile fashion with the help of tools that serve as centralized repositories, such as Jira, Confluence, Azure DevOps Boards, and others. All these tools support the management of work within these software projects through Scrum framework capabilities, customizable dashboards, and integrated reporting. Once these project plans and artifacts have been centralized, access can be granted to various project stakeholders.
Code
Once the plans have been outlined and documented, developers can begin their coding work in an Interactive Development Environment (IDE) such as Visual Studio to write scalable code that can be checked into source control to prepare for the build, continuous integration, and deployment phases. Version control, also known as source control, is the practice of tracking and managing changes to software code. Version control systems such as GitHub, GitLab, Git, and others are software tools that help software teams manage changes to source code over time. Several cloud providers also provide a set of version control tools that can be used to manage code and its different versions by the development team. It helps track the changes made in the code over the period with different versions. When changes are made to the code, the DevOps tool takes snapshots of the files being checked in so that these changes can be tracked. GitHub enhances collaboration through robust workflows and source control repositories to secure code and link it with Cloud-based DevOps pipelines for continuous automated testing, integration, and deployment from development to higher environments. Developers often struggle to choose the right source control tool for their needs. The figure below is one among many that illustrate a comparison between one tool and another, which in this case is Git vs. GitHub.
The topic of Infrastructure as Code (IaC) is also very important within DevOps. The purpose of IaC is to enable developers or operations teams to automatically manage, monitor, and provision resources, rather than manually configure discrete hardware devices and operating systems. The following chart illustrates the comparisons between some of the popular languages for writing IaC. For cloud agnostic IaC, Terraform is used quite often. In addition, the industry is also moving more towards Bicep since it offers an easy-to-use, declarative syntax for writing and deploying cloud resources.
Build
After the code has been checked into the centralized code repository, the build phase will retrieve this code through an automated tool, such as Chef or Puppet. The tool will compile the software code into binary artifacts, execute functional tests, and then publish to artifact into a shared repo.
Continuous Integration
In this phase, code changes are checked into a central repository, which triggers an automated build process to test and perform QA on the code. Major cloud platforms offer Build Pipelines to achieve this, which supports the implementation of CI/CD pipelines to build, test, and deploy code and software continuously.
Testing
In this phase, DevOps processes can be implemented to automate certain aspects of continuous testing with tools such as Selenium or Junit to test multiple codebases in parallel. This automated testing is not meant to fully replace human testers but merely to supplement them to ensure there are no flaws in the code's functionality. Many of these automated testing tools can produce detailed test report results, which can help developers and other relevant stakeholders iteratively improve the quality and accuracy of their code.
Continuous Deployment
Within this phase, automated testing will validate if changes to a codebase are correct and stable for immediate autonomous deployment to a production environment via a release pipeline. These pipelines can be scheduled as often as needed, or they can also be triggered as the code is checked into the source control repository.
Operations
In this phase, the IT administrators manage the software during production with tools such as Ansible, PowerShell, Chef, Puppet, Salt, Azure Automation, and other configuration management tools. Configuration Management refers to the process by which all environments hosting software are configured and maintained. This is critical in DevOps infrastructures since it facilitates speed, accuracy, and efficiency. Using these tools allows desired states across your entire infrastructure estate to be enforced, configuration updates can be easily rolled out, and unexpected changes and issues can be resolved through automation. Many configuration management services allow you to write, manage, and compile PowerShell Desired State Configuration (DSC) configurations for nodes in any cloud or on-premises datacenter. Yet another operationalization service is Kubernetes which allows you to ship containerized apps faster and operate them more easily using a fully managed Kubernetes service.
Continuous Monitoring
The process for continuously measuring the performance of the environment and the application's health can support gaining robust insights into applications, infrastructure, and networks. It also supports collaboration and communication among teams to resolve production issues. Some of these monitoring tools include Datadog, Azure Monitor, Grafana, and Splunk.
DevOps Success Stories in Practice
While discussing principles, frameworks, methodologies, and recommended best practices is always great, presenting the success stories of real-work DevOps implementations for organizations is always much more impactful. In this section, we will go through a few of these stories which relate to true cloud transformation and innovation initiatives for integrating DevOps with SQL Databases, ELT Pipelines, and Machine Learning Models.
SQL Database
Customers are always asking for the capabilities of promoting code changes within their SQL databases from development to quality assurance to a production environment in real-time, along with automated testing and manual approval gates along the way. This is known as DataOps, which is a set of practices, processes, and technologies that combines an integrated and process-oriented perspective on data with automation and methods from Agile software engineering to improve quality, speed, and collaboration and promote a culture of continuous improvement in the area of data analytics.
While several major cloud providers support this capability, the following figure illustrates this architectural pattern from an Azure-centric point of view. A developer would begin the process by developing SQL code in their IDE of choice, in this case, Visual Studio. They would then check their local code into a linked source repository extension, which in this case is a GitHub extension for Visual Studio. After the code is checked in, their linked and pre-built Azure DevOps Pipelines containing a continuous integration (Build) and continuous deployment (Release) pipeline would test and promote the code to the pre-defined higher environments. Even though this is an Azure-centric example, the technologies can easily be swapped, and the principles, frameworks, and methodologies will remain constant. Customers who have implemented this DevOps pattern for SQL Database have reaped all of the benefits detailed in the previous sections.
Extract-Load-Transform (ELT) Pipelines
Like the DevOps CI and CI pipelines for a SQL Database, customers have been implementing DevOps pipelines for other cloud technologies, including modern cloud-based ELT technologies. The figure below illustrates how Azure DevOps can be used in conjunction with a GitHub source repository to promote an Azure Data Factory ELT environment from development to quality assurance to a production environment in real-time by using Azure pipelines that have automated testing and manual approval gates along the way. This successful DevOps pattern allows customers to quickly test, review, collaborate, and promote their ELT pipelines into production.
Machine Learning (ML) Models
As a final example, the following figure shows a more advanced DevOps CI and CD pipeline promotion process for promoting Azure Machine Learning (AML) models to multiple higher environments. This DevOps process, also known as MLOps, is a more recent rising pattern of success. It allows Data Science teams to partake in the benefits of DevOps to promote their models into production. While AML has been used in this pattern, Databricks, another cloud-based technology that supports a Machine Learning workspace, can also be used as a replacement to AML in this pattern to promote ML Models into production.
The Future of DevOps
The future of DevOps is bright, with multiple practices embarking on their journey, which may lead to more automated decisions around strategy, policies, alerts configuration, tuning, coding, and planning. Gartner predicts, "By 2023, 40% of DevOps teams will augment application and infrastructure monitoring with artificial intelligence for IT Operations capabilities." In addition to DataOps and MLOps, the "Ops Party" is also beginning to branch out to other disciplines such as DevSecOps, which involves trying to figure out how to incorporate and automate security earlier in the SDLC, right from the time of coding, prevents all sorts of hazards, substantially reduces costs and speeds things up. These highly automated processes may also minimize human interaction, ultimately making operational teams obsolete.
Other burning DevOps trends in the cloud space are Microservices and Serverless architectures. Microservices is essentially cutting applications into small chunks, independent services, units, or entities that are flexible and scalable. Kubernetes has often been used in microservices architectures. It is easy to start a local Kubernetes cluster on a developer's machine. This allows developers to utilize and rely on Kubernetes features early in the development cycle, promoting futuristic DevOps practices. The Serverless Architecture market is projected to grow from USD 7.6 billion in 2020 to USD 21.1 billion by 2025 due to "the rising need of shifting from CAPEX to OPEX by removing the need to manage servers, thereby reducing the infrastructure cost." The flexibility, speed, and cost efficiency of paying for what you use make it a force to be reckoned with, relieving developers from most maintenance issues. This shift can be seen as the next generation of Microservices.
Finally, IaC, which we discussed in a previous section, is seen as another burning industry DevOps trend. By turning all servers, networks, databases, and storage into lines of code, you can simplify and automate. In addition, it allows for easy rollback and recovery via version control and simple monitoring and analytics (KPIs). Coupled with a great configuration management strategy, this is also, without a doubt, a futuristic DevOps practice.
Summary
In this article, you learned more about applying DevOps's benefits for Cloud transformation and innovation. We started by gaining a deeper understanding of the top five benefits of DevOps: collaboration, time-to-market acceleration, high-performing teams, improved software quality and performance, and frequent iterative software deployments. We then discussed the similarities and differences between Agile and DevOps. Remember that Agile focuses on iterative project development and testing while DevOps focuses on the collaboration and productivity of multiple teams and stakeholders. You also learned about various DevOps practices and tools for planning, coding, building, integrating, testing, deploying, operationalizing, and monitoring your software code and solutions. Towards the end, you learned about a few DevOps success stories for SQL Database, ELT Pipelines, and ML Model promotions to production. Finally, you learned about some of the burning DevOps industry trends and where the future of DevOps is headed from the perspective of cloud transformation and innovation.
Next Steps
- For more information on learning to apply DevOps for Data and AI Innovation, please see my webinar recording and corresponding downloadable PowerPoint slides: Applying DevOps for Data and AI Innovation
- Read my previous article on Azure DevOps CI CD using GitHub Repo and Visual Studio Azure SQL Database Project
- Read my previous articles to learn how to Deploy CI/CD Changes for Azure Data Factory with Azure DevOps and for Using Azure DevOps CI CD to Deploy Azure Data Factory Environments
- Read my article on Sensitive Data Secured in Azure DevOps
- Read my previous article on Azure DevOps' Azure Boards for Project Management
About the author
Ron L'Esteve is a trusted information technology thought leader and professional Author residing in Illinois. He brings over 20 years of IT experience and is well-known for his impactful books and article publications on Data & AI Architecture, Engineering, and Cloud Leadership. Ron completed his Master’s in Business Administration and Finance from Loyola University in Chicago. Ron brings deep tecThis author pledges the content of this article is based on professional experience and not AI generated.
View all my tips
Article Last Updated: 2022-10-05
