Security and Cloud 24/7

A diagram of a common DevOps lifecycle:

The DevOps world meant to provide complementary solution for both quick development (such as Agile) and a solution for cloud environments, where IT personnel become integral part of the development process. In the DevOps world, managing large number of development environments manually is practically infeasible. Monitoring mixed environments become a complex solution and deploying large number of different builds is becoming extremely fast and sensitive to changes.

The idea behind any DevOps solution is to provide a solution for deploying an entire CI/CD process, which means supporting constant changes and immediate deployment of builds/versions.
For the security department, this kind of process is at first look a nightmare – dozen builds, partial tests, no human control for any change, etc.

For this reason, it is crucial for the security department to embrace DevOps attitude, which means, embedding security in any part of the development lifecycle, software deployment or environment change.

It is important to understand that there are no constant stages as we used to have in waterfall development lifecycle, and most of the stages are parallel – in the CI/CD world everything changes quickly, components can be part of different stages, and for this reason it is important to confer the processes, methods and tools in all developments and DevOps teams.

In-order to better understand how to embed security into the DevOps lifecycle, we need to review the different stages in the development lifecycle:

Planning phase

This stage in the development process is about gathering business requirements.

At this stage, it is important to embed the following aspects:

• Gather information security requirements (such as authentication, authorization, auditing, encryptions, etc.)
• Conduct threat modeling in-order to detect possible code weaknesses
• Training / awareness programs for developers and DevOps personnel about secure coding

Creation / Code writing phase

This stage in the development process is about the code writing itself.

At this stage, it is important to embed the following aspects:

• Connect the development environments (IDE) to a static code analysis products
• Review the solution architecture by a security expert or a security champion on his behalf
• Review open source components embedded inside the code

Verification / Testing phase

This stage in the development process is about testing, conducted mostly by QA personnel.

At this stage, it is important to embed the following aspects:

• Run SAST (Static application security tools) on the code itself (pre-compiled stage)
• Run DAST (Dynamic application security tools) on the binary code (post-compile stage)
• Run IAST (Interactive application security tools) against the application itself
• Run SCA (Software composition analysis) tools in-order to detect known vulnerabilities in open source components or 3^rd party components

Software packaging and pre-production phase

This stage in the development process is about software packaging of the developed code before deployment/distribution phase.

At this stage, it is important to embed the following aspects:

• Run IAST (Interactive application security tools) against the application itself
• Run fuzzing tools in-order to detect buffer overflow vulnerabilities – this can be done automatically as part of the build environment by embedding security tests for functional testing / negative testing
• Perform code signing to detect future changes (such as malwares)

Software packaging release phase

This stage is between the packaging and deployment stages.

At this stage, it is important to embed the following aspects:

• Compare code signature with the original signature from the software packaging stage
• Conduct integrity checks to the software package
• Deploy the software package to a development environment and conduct automate or stress tests
• Deploy the software package in a green/blue methodology for software quality and further security quality tests

Software deployment phase

At this stage, the software package (such as mobile application code, docker container, etc.) is moving to the deployment stage.

At this stage, it is important to embed the following aspects:

• Review permissions on destination folder (in case of code deployment for web servers)
• Review permissions for Docker registry
• Review permissions for further services in a cloud environment (such as storage, database, application, etc.) and fine-tune the service role for running the code

Configure / operate / Tune phase

At this stage, the development is in the production phase and passes modifications (according to business requirements) and on-going maintenance.

At this stage, it is important to embed the following aspects:

• Patch management processes or configuration management processes using tools such as Chef, Ansible, etc.
• Scanning process for detecting vulnerabilities using vulnerability assessment tools
• Deleting and re-deployment of vulnerable environments with an up-to-date environments (if possible)

On-going monitoring phase

At this stage, constant application monitoring is being conducted by the infrastructure or monitoring teams.

At this stage, it is important to embed the following aspects:

• Run RASP (Runtime application self-production) tools
• Implement defense at the application layer using WAF (Web application firewall) products
• Implement products for defending the application from Botnet attacks
• Implement products for defending the application from DoS / DDoS attacks
• Conduct penetration testing
• Implement monitoring solution using automated rules such as automated recovery of sensitive changes (tools such as GuardRails)

Security recommendations for developments based on CI/CD / DevOps process

• It is highly recommended to perform on-going training for the development and DevOps teams on security aspects and secure development
• It is highly recommended to nominate a security champion among the development and DevOps teams in-order to allow them to conduct threat modeling at early stages of the development lifecycle and in-order to embed security aspects as soon as possible in the development lifecycle
• Use automated tools for deploying environments in a simple and standard form.
  Tools such as Puppet require root privileges for folders it has access to. In-order to lower the risk, it is recommended to enable folder access auditing.
• Avoid storing passwords and access keys, hard-coded inside scripts and code.
• It is highly recommended to store credentials (SSH keys, privileged credentials, API keys, etc.) in a vault (Solutions such as HashiCorp vault or CyberArk).
• It is highly recommended to limit privilege access based on role (Role based access control) using least privileged.
• It is recommended to perform network separation between production environment and Dev/Test environments.
• Restrict all developer teams’ access to production environments, and allow only DevOps team’s access to production environments.
• Enable auditing and access control for all development environments and identify access attempts anomalies (such as developers access attempt to a production environment)
• Make sure sensitive data (such as customer data, credentials, etc.) doesn’t pass in clear text at transit. In-case there is a business requirement for passing sensitive data at transit, make sure the data is passed over encrypted protocols (such as SSH v2, TLS 1.2, etc.), while using strong cipher suites.
• It is recommended to follow OWASP organization recommendations (such as OWASP Top10, OWASP ASVS, etc.)
• When using Containers, it is recommended to use well-known and signed repositories.
• When using Containers, it is recommended not to rely on open source libraries inside the containers, and to conduct scanning to detect vulnerable versions (including dependencies) during the build creation process.
• When using Containers, it is recommended to perform hardening using guidelines such as CIS Docker Benchmark or CIS Kubernetes Benchmark.
• It is recommended to deploy automated tools for on-going tasks, starting from build deployments, code review for detecting vulnerabilities in the code and open source code, and patch management processes that will be embedded inside the development and build process.
• It is recommended to perform scanning to detect security weaknesses, using vulnerability management tools during the entire system lifetime.
• It is recommended to deploy configuration management tools, in-order to detect and automatically remediate configuration anomalies from the original configuration.

Additional reading sources:

• 20 Ways to Make Application Security Move at the Speed of DevOps
• The DevOps Security Checklist
• Making AppSec Testing Work in CI/CD
• Value driven threat modeling
• Automated Security Testing
• Security at the Speed of DevOps
• DevOps Security Best Practices
• The integration of DevOps and security
• When DevOps met Security - DevSecOps in a nutshell
• Grappling with DevOps Security
• Minimizing Risk and Improving Security in DevOps
• Security In A DevOps World
• Application Security in Devops
• Five Security Defenses Every Containerized Application Needs
• 5 ways to find and fix open source vulnerabilities

This article was written by Eyal Estrin, cloud security architect and Vitaly Unic, application security architect.