LesserGeek

While working for different organizations, reading articles, discussions at meetups and with fellow architects, engineers and consultants I have come across some key challenges faced in the Devops environments. In this article I am presenting my thoughts.

1. Building a strong and motivated team

This is probably the biggest challenge the IT industry faces today. Till a few years ago when cloud and Devops were in their infancy it was easier to find resources for to manage infrastructure and developers to write applications. With cloud adoptions came a huge requirement for engineers who could understand and manage infrastructure and at the same time develop code to deploy and manage the infrastructure as a code. We still have a huge gap which needs to be bridged to ensure the infrastructure engineers can learn and adopt development principles and developers can understand the infrastructure principles.

There is a need for the organizations to understand that it takes years of experience to gain skills required for devops. Candidates should be hired based on their talent and capabilities and then time and effort needs to be invested to bridge the gap. Organizations that have a good ratio of experience and talent are bound to succeed. Smart hiring involves understanding the candidate and their experience. Interviewers often end up losing great candidates as they expect candidate to be able to ace the interview, designing something on the fly, that they themselves have taken hours to read and research for. Once hired the candidates need to be motivated. Constant training and upskilling opportunities keep the workforce ready for new challenges in the ever-evolving IT landscape while keeping them motivated.

Ample opportunities need to be provided to employees to contribute and ideas should be welcome. At one of the organization I worked, we had a “ThinkWeek” where every employee was given the opportunity to present their ideas and if the idea was good it could be reviewed and accepted by the top leadership.

2. Devops is much more than Infrastructure as a Code

Organizations spend a lot of time and effort to create architecture, study compliance requirements and develop security requirements. Once the architecture is created and security guidelines are listed it is expected that the Devops engineers would start developing code straight away.

Many organizations additionally develop the much required Devops strategy and architecture including the tools they would want to use, how the code needs to be maintained and delivered. Coding best practices that need to be followed, static code analysis tools that need to be used, a clear and well-defined strategy for not only Infrastructure as a Code but Security and Testing as a Code needs to be developed. Organizations that understand that “one shoe fits all approach will not succeed” invest time to decide coding languages and tools for specific delivery. Companies using multi-cloud, multi-vendor environments dive deeper to understand the strengths and weaknesses of the products, coding languages and tools that are available.

3. A DevSecOps mindset needs to be built

Don’t forget security during your Devops journey. While the organizations switch from traditional on-premise infrastructure to infrastructure in the cloud, they often forget that the security for both the environments is not the same. The biggest difference is the physical security. While the on-prem environments provide some inherent security of physical walls, locks, security and guards, the cloud environments come with both benefits, specific security considerations and compliance requirements.

Guard-rails, security policies, conditional access, multi-factor authentication, privileged access and management, personal access tokens, managed identities, role-based access & controls logging, monitoring security policies, test strategies all need to be designed and planned specific to the cloud environment being used.

4. Security is everybody’s responsibility

Pressure to deliver based on deadlines often end up pushing teams to fast track the development. Almost every Devops engineer at some time would have hard coded secrets or keys in the code. Even if it is for testing purpose this poses a huge risk of the secrets or keys ending up in logs or source history. There have been instances when the code with secrets ended up in the public domain leading to huge financial losses and penalties.

Security is everyone’s responsibility and this mindset needs to be strengthened at every level. Code reviewers and approvers need to pay special attention to details and avoid hard coded secrets getting into code. There are plenty of options available to securely store secrets, keys and certificates in a resource such as keyvaults/HSM/key management Service and then using the secrets and keys by reference. CI/CD pipelines can use managed identities, service principles and bearer tokens to keep the secrets obfuscated and fetch them only when needed. There is absolutely no need to hard code passwords, keys, secrets in code.

5. Source Control is important

Everyone likes to drive on planned, well-paved and well-maintained roads. Similarly, a well-planned, structured, well-maintained code is necessary for a successful and smooth running Devops environment. Organizations must adopt a strong source control strategy which governs how the source code will be maintained. Version control, branching strategy, modular development, testing and release planning are some of the initiatives that could help maintain code health.

Source control (or version control) is the practice of tracking and managing changes to code. Source control management (SCM) systems provide a running history of code development and help to resolve conflicts when merging contribution from multiple developers. Version control protects source code from both catastrophe and the casual degradation of human error and unintended consequences. 

The ultimate result of feature branches, distributed development, pull requests, and a stable community is a faster release cycle. These capabilities facilitate an agile workflow.

Keeping the code clean and healthy at all-times means that the code that is no longer needed is removed from the code repositories. Instead of storing large files in the code repositories, artificatories, S3 buckets, storage accounts and other similar options can be used. Large files ending up in a repository can bloat the repository and even after the files are deleted the repository size may not reduce due to the way source history is maintained.

6. Static Code Analysis tools are a must

Even after all the manual efforts, planning, code reviews, it is highly likely that the code will end up having security issues, bugs and vulnerabilities that can inevitably impact the environment. As the Devsecops culture matures there are likely to be fewer issues over a period of time, however since new team members join, new technologies and tools get adopted the organization will always be in an ever-evolving state. No amount of manual review, unt testing and manual test can guarantee 100% DeSecOps compliance .

To ensure stability, less vulnerabilities to end up in the code, it is highly recommended that the organizations implement static code analysis tools. Static code analysis tools go beyond catching indentations, formatting and other common coding malpractices. There are tools available today that can hunt for common patterns in the code and even recommend fixes. They can help intercept common coding errors, bugs and security vulnerabilities early during the development.

Code quality and modularity – Every coding language I used has one or more ways to maintain code libraries or modules. It is a way to keep related code grouped together in a structured manner. Due to various reasons engineers working in a Devsecops environment end up creating separate script files with random names and unrelated folder structures. This leads not only to code duplication, code bloating and eventually a poorly managed code base. It becomes a administrative nightmare to manage such code repositories. In another article I would explain a few ways to keep the code modular.

Code modularity also helps avoid having monolithic bloated CI/CD pipelines and repositories. Code modules can be independently versioned, updated and managed.

7. Testing, Monitoring and Auditing

The test and the production environments need to be isolated and should have as little difference as possible to avoid unexpected behaviors during production. Most organizations have a Preview/Test/Lab environment, a Non-production Environment the Production environment. It is important that the code is tested in the lower environments thoroughly before deploying the code in production. The organizations should have a strategy to create development branches and feature branches for code through which the lower environments are tested and only after sufficient artifacts/evidence is gathered, code is reviewed, approved and merged into the Master branch. Master branch is expected to be the cleanest and most stable branch. Deployments to the production environment must always happen from the Master branch.

Monitoring and auditing all activities, asset & data classification, encryption, logging & reporting, key management, configuration hardening, logical segmentation, boundary enforcement, conditional access, authorization & authentication, access control and user behavior analysis is crucial for success of DevSecOps.

8. Tech Debts can be expensive and interests could be high

In the effort of achieving delivery goals and targets the tech debts are often forgotten or parked in backlog. Each Tech debt needs to be categorized and handled before it leads to a bigger issue, security vulnerability or a financial impact. If technical debt is not repaid, it can accumulate ‘interest’, making it harder to implement changes. Unaddressed technical debt increases software entropy. Some experts claim that the “technical debt” metaphor tends to minimize the impact, which results in insufficient prioritization of the necessary work to correct it.

9. Teams must not be working in Silos

Teams working on common objective needs to be in a sync. Teams working in silos leads to gaps and long-term issues. At every stage of the Devsecops planning and implementation teams (squads and tribes) must remain in sync. Code as well as changes need to be reviewed together to avoid tech debts. Cross training amongst team members is necessary for success. I have been lucky to work for organizations where teams talk and keep each other updated. Leadership meetings and discussions flow down to every engineer that helps keep everyone aware about where we are heading together as a team. The Agile environments help teams meet on a daily bases and plan activities. In some organizations where I worked, you could see kanban boards with sticky notes, for teams, on walls across the floor. You could simply walk over and see who is working on what.

10. Compliance is non-negotiable

Almost all organization large or small have to adhere to some kind of regulatory compliance, internal security policies and best practices. Non-compliance can be detrimental to project and individual success. It can also come with costly consequences – heavy penalties, fines, risks such as loss of brand value and stakeholder trust. Most organizations take compliance very seriously and build processes and designs around the compliance and ensure sufficient logging and monitoring processes are built. With the advancement of technology there are several tools available today that run analytics around the logs and go beyond just the compliance and non-compliance status of resources and environment. Trend analysis, predictive analysis and early detection are helping organizations plan ahead of time and prevent incidents. Gartner even coined a new term AIOps in 2019.

11. Devops as a branch of operations

Devops is really a branch of operations. It is important to understand that all the management and governance principles of change and incident management applies to devops. Any bugs or incidents leading to a discovery of an issue in the code at any stage that involves making a code change must follow the standard operations processes. Organizations working on Agile or other frameworks must be able to track code changes and the reasons behind the code change.

DevOps (a portmanteau of “development” and “operations”) is the combination of practices and tools designed to increase an organization’s ability to deliver applications and services faster than traditional delivery methods. It includes the automation of repetitive tasks and operations by building redeployable patterns. The speed and agility are expected to enable organizations to better serve their customers and compete more effectively in the market.

In simple terms, DevOps is about removing the barriers between traditionally siloed teams, development and operations. Under a DevOps model, development and operations teams work together across the entire development and delivery life cycle, from development and test through deployment to operations.

Benefits of DevOps

Speed and Agility: DevOps practices let you move at the speed you need to innovate faster, adapt to ever evolving IT landscape better, and become more efficient at driving business results.

Rapid Delivery: When you increase the pace of releases, you can improve your product faster and build competitive advantage. Rapid delivery makes room for innovation. Teams have more time to focus on POC’s, trainings, improving existing products rather than doing repetitive tasks with traditional ClickOps.

Reliability: DevOps practices like continuous integration and continuous delivery can ensure the quality of product updates and infrastructure changes so you can reliably deliver at a more rapid pace while maintaining an optimum experience for end users and customers.

Improved collaboration: Under a DevOps model, developers and operations teams collaborate closely, share responsibilities, and combine their workflows. This reduces inefficiencies and saves time.

Security: You can adopt a DevOps model without sacrificing security by using automated, integrated security testing tools. As a matter of fact, the repeatable patterns guarantee consistency of security across multiple environments and deployment of fully tested code into production.

Teams adopting a set of practices alone cannot further DevOps evolution; you have to make the corresponding structural changes to optimize the way teams work. The DevOps evolution model (see chart) shows that organizations do not progress to self-service and security integration until Stages 4 and 5, after individual people are given more autonomy to work without manual approval from outside the team (Stage 3). Stage 3 is a critical point of convergence — trust has been built up in Stages 1 and Z teams are granted more autonomy. and deployment is no longer a four-alarm fire. At this point, teams can expand their new ways of collaborating across more functional boundaries. beyond Dev and Ops. In Stages 3 to 5 we see a loosening of one-size-fits-all rules and processes, with an underlying focus on automation. At these stages, automation has expanded beyond solving local problems for a single individual or team to an explicit — and higher — focus on creating value for the business. This is what it means to scale DevOps practices: By empowering individuals and teams to rely on their knowledge and experience — and by automating — you are able to optimize at an organization-wide scale. You are now able to focus on eliminating waste across multiple delivery streams, and help the business achieve its goals.

12. A Special Message

Devops is not project or a product, Devops is a Framework, it is a mindset.

It must be adopted everywhere to build repeatable, accountable, traceable patterns that can be securely delivered. To convert experience and documentation into code which can be quickly enhanced. This will free up resources for innovation.

Every IT product today provide coding capabilities to build automation around the product.

Any organization not investing in Devops will most likely not survive the stiff competition in the ever changing IT landscape.

Conclusion

The journey towards a mature Devsecops environment can be fun. During this journey organizations can reap the benefits of agility, security, better governance, better growth, less vulnerabilities, minimal risks, regulatory compliance and better return on investments.

Feel free to send me your thoughts and if you want to contribute to this article.

gsjutla@lessergeek.com