Our Thinking
Bringing Security into the Pipeline
05 June 2017 | Posted by Gary Tate

Let me introduce myself, my name is Gary and I’m a Senior Security Engineer with Kainos. Over the past few years I’ve taken on many security roles, from compliance, vulnerbility management, penetration testing and looking after a small security team. Most recently I’ve been getting more involved in the world of security testing and “DevSecOps”.

In this post, I’ll be going discussing bringing security into the pipeline to enable secure by design solutions. I feel it’ll be impossible to cover off everything I want to in a single post so will follow up with other posts including more technical hands on implementation and configuration of the solutions discussed throughout the post.

Security has been in the spotlight of late, everyday we’re hearing of more and more breaches. It is said that 2/3 organisations will be hit by a breach, and given the average cost is upward of £1m – a serious breach could result in the demise of many organisations.

People have different solutions to this: more pen tests, more security operations, building / expanding their security team – and the list goes on. One key thing we all need to accept is things have changed, technology has changed, our reliance has grown and most importantly, cyber crime is more profitable than ever.

Security is often an after thought; we build a solution, we than pad it out with security run a pen test, fix some issues and we’re done. Sound familiar? For many this has been the standard approach for years, sometimes they’ll run annual testing mainly to meet compliance and highlight any issues that propped up over the last 12 months.

The problem with this approach is its far too easy for issues to be missed, and even when we fix all security issues highlighted the secure status will only be true at the time of testing. Every day new exploits are found, new approaches are discovered and so on. Additionally in the world agile, development never really ends additional features will be added or tweaked on a regular basis.

With all this in mind, maybe its time to change our thinking towards security, rather than bring it in towards the end as an added feature how about we bring security into every step of the pipeline, before you commit code, before it reaches production what if security checks were built into the deployment? Building solutions that are secure by design. This is a new way of thinking and only a few organisations have adapted this approach, but given how technology has grown over the years maybe its time for a new approach.

Continuous integration and continuous delivery has become more and more common place, this mind set and CI/CD technologies can aid us in the process on implementing security into the pipeline.

Every pipeline will slightly vary from company to company and project to project. To make things simple I’ll be covering 4 stages that will exist in all CI/CD pipeline, as below:

Pre-commit

The pre-commit stage, before you commit any code, what should we be looking at? If you want to push the idea of secure by design we need to understand what the risks are and now to mitigate from them. To do this threat modelling is key, this is something that is often either overlooked completely or only done too meet compliance as a box ticking exercise. This shouldn’t be the case and the importance shouldn’t be under estimated, it will not only help identify areas where you’re at risk but will help create awareness and build in the security mind set though out the team. Key thing to remember, while threat modelling is very much a security exercise it isn’t the job of the “security people” it’s everyone’s job!

Additionally some light touch review of the code should be done, highlighting any obvious issues and to ensure best practices.

Continuous Integration

Once the pre-commit checks have been completed you’ll want to ensure you have some automated checks in place to initiate once you click that commit button. If issues are flagged ideally the code will be flagged for review before being committed. This is not a new concept for many. Most CI/CD pipeline line will include static analysis to ensure the code meeting best practices, highlight obvious bugs etc. The same can be done with security checks, most well known languages have tools allowing automated checks.

There are many tools that enable us to automatically review code, highlight dependencies and known vulnerabilities for those dependencies.

You’ll notice that throughout this post I’ll be recommending different tools and frameworks, unfortunately the above will be very dependent on the chosen language, the below link lists many of the most common static code analysis tools.

https://en.wikipedia.org/wiki/List_of_tools_for_static_code_analysis

Acceptance Stage

This is where the real fun starts, we have our code deployed to our development environment being prepped from production. At this stage we’re able to take the environment and run a wide range of tests, some automated, some manual. Again the level of testing can sometime depend on the results of the automated testing or can be refined for smaller changes.

For the most part, we want to ensure any low-hanging fruit is caught at this stage, preferably during the automated tested. Catching the silly things earlier on will not only help make our application more secure it’ll free up our pen testers to focus in on bigger issues and dig deeper.

I’m going to do my best to summarise some of the commonly used tools and what they have to offer, look forward to separate posts around implementing and configuring some of them.

Burp Suite and OWASP ZAP are commonly used tools within the security testing community, they can not only help us perform simple checks but enable us to bring our testing to a whole new level. So it would make sense to try and automate these to some degree, if you search you’ll find a few methods to integrate these into the pipeline – the below is methods I’ve had success with, but keep in mind the best solution will vary from project to project.

Burp Suite — there are a few options around this from using Docker containers, APIs and using various frameworks. The method I’ve had the most success with is running Burp via REST API. Simple to setup, and once done its just a matter of sending HTTP requests to trigger and get feedback. Another option for Burp Suite is MITTN by F-Secure. This solution is more complex to set up and configure, but provides extra functionality providing the ability to record and track previous vulnerabilities making it easier to stop false positives failing the tests.

ZAP — ZAP is basically a free alternative to Burp, while I tend to prefer Burp Suite its always good to make use of multiple vulnerability scanners. Again we have a few options with ZAP: we have Docker containers which are maintained by the OWASP community, REST API and a few attack frameworks.

In the past I’ve made use of a solution by Continuum Security, BDD-Security. This solution is basically a wrapper for various tools, mainly ZAP with Selenium Web Driver integration. BDD-Security enables us to bring ZAP into the CI/CD pipeline with ease.

Gauntlt — Another attack framework, comprising of commonly used tools including.

  • arachni — Offering a range of web based attacks (Gauntlt only support XSS out of the box but easy to configure additional attacks.)
  • dirb — Will looks for known directories (via wordlist)
  • Garmr — Inspects web responses for obvious issues
  • nmap — Port and service scan (ensuring only the correct ports are exposed)
  • sqlmap — SQL Injection checks for chosen parameters
  • sslyze — SSL/TLS configuration checks

Gauntlt is a real powerhouse when it comes to CI/CD security, implementing this alone will make a huge difference. Again, this post is only going to summarise the functionality, but in a later post I’ll go over the configuring and how to expand out the attacks.

Nessus — One of the best known vulnerability scanners and often one which is requested by our clients. Once again, bring this into the pipeline, and we’re faced with multiple options BDD-Security support Nessus integration so if you’re wanting to make use of ZAP and Nessus perhaps that’ll be the best solution. Another option and probably the easiest option for Nessus integration is making use of Tenable’s REST API thoroughly simple to setup and “Dockerise” if need be this is probably the fastest way to integrate Nessus within the pipeline. One negative is the lack of feedback, you can easily kick off a scan and have it show the progress, but little in the way of pass and fail. As a work around you can schedule mails to create incidents within you CI Tool to be reviewed.

Manual testing — Running automated tools is great but they’ll only get you so far sometimes we need human interaction whether that’s for manual verification or more comprehensive testing time. Bringing automated testing into the CI/CD pipeline will help capture the low-hanging fruit, any obvious or easy to spot issues, allowing for our pen testers to go deeper into the environment and focus on the more complex attack vectors. Manual testing is expensive, both in terms of time and money, so ensuring the easy parts have been covered off first will allow us to get the most out of our investment.

Push to Production

Using the above enables us to create more secure solutions. In truth, everything can be hacked, all it takes one missed bug or a zero day that isn’t mitigated in an suitable timeframe. It important we have the correct measures and processes in place to help protect our environments. The main things to look at include:

  • Scheduled Vulnerability scans — Ensuring we capture any zero days an mitigate in a swift manner
  • Real time defence, WAF, HIDS — If you’re putting an application out there, it will be attacked! It’s important to ensure we have the correct measures in place to detect and block any malicious activity, there are lots of options out there – which will be covered in a future post.
  • Monitoring — Ensuring you have the correct logs in place to monitor and capture any malicious activity.
  • Red / Blue Teaming — A great exercise that’s becoming increasingly popular will help to fill any security gaps missed in the previous stages, will also help spread security knowledge across the project teams.
  • Bug bounties — Again, this is becoming increasingly popular and can be very beneficial, offering rewards for external users to highlight vulnerabilities. Just ensure you have a scope and rules of engagement are in place.
  • Scheduled manual testing — More manual testing, on a regular basis.

The list can go on and on – these are the key things I believe you need to implement to enable a secure by design and maintain it.

Today having security as an afterthought just isn’t good enough, cyber crime has come too profitable and honestly too easy too pull off, bring security into the your pipeline will give us the much needed edge in the world of security.

1 comment
June 15, 2017
Darren Rea

Superb post from a colleague who came to Kainos as a placement infrastructure engineer and has matured into a role that he has made his own. Sharing this knowledge with the Service Desk team has made us all more security aware. The odd trip to Vegas is a nice perk for a security specialist too! Now, I’m off to update my anti-virus!

Leave a Comment