In last week’s post, we discussed the first four elements in a Secure Software Development Lifecycle (SSDLC) model. As indicated before, the content of the “activity boxes” generally depends on the development model and the type of project, and the definitions below can only serve as a starting point for a customized model:

5. Coding

Writing quality code does include several aspects, of which two educational aspects are particularly relevant for secure software development: using proven patterns, and following best practices that apply to the task at hand. The usage of proven patterns will help to create code that is easier to understand, easier to review, and easier to maintain – all of which will ultimately contribute to a more secure codebase. Following best practices prevents the developer from having to reinvent the wheel, and then probably miss some important spokes when doing so. A third aspect around secure coding includes static code analysis, which helps developers better understand their code before it is deployed. Depending on the static analysis tool used, the code to be analyzed need not even yet compile! Static code analysis can be a very helpful tool for the security architect running an SSDLC program, and there are several free and commercial tools available for this task.

6. Integration and Testing

Once the code is in a stage that compiles cleanly and produces executable artifacts, it is possible to begin runtime integration of the various components. This should start as early as possible, following the “release early, release often” paradigm, even if the “release” is just for the internal quality assurance (QA) teams. Besides the non-security related QA activities, the verification process should include security specific activities such as fuzz testing, manual black box security penetration testing, and automated black box security testing. There are several tools available for automated security testing, with a focus on automated web application security testing. A major motivating factor for this is the high cost of manual security testing. There are, however, strategies for QA that allow reducing costs while still providing in-depth manual pen testing services. A second important security activity during the integration phase is for the security architect to re-review the results from the requirements and design analysis with the team, and make sure that the requirements have been correctly implemented, and the attack surface meets the results from the threat analysis step.

7. Release and Deployment

Once the final security review has been completed and the product is being deployed, the product team should already have a solid plan about how to respond to security incidents, create fixes for security issues discovered in production, quickly deliver them to the affected customers, and communicate the solution. Depending on the development model of the specific product, the team may also need to create an asset archiving strategy to make sure the required artifacts are available for maintenance later on. The team should also lay the foundations for a deployment security strategy, which includes documentation and best practices that are crucial to securely configuring the product.

8. Operations and Maintenance

The team operating the solution is responsible for deployment security, building on the foundations led earlier during the final development stages. While the operations team, which can be an internal team or a customer’s team, is responsible for securely operating the solution, the development team still has maintenance responsibilities. There are some development models that delegate maintenance tasks to a dedicated team (e.g. a “Current Product Engineering” / CPE teams), while other models (in particular agile models) only have one team that is responsible for the entire lifecycle of a product which includes operations and maintenance. In either case, the team responsible for maintenance must execute the incident response plan created earlier, and make sure that any fix that addresses security vulnerabilities or privacy violations is ported to all supported versions of the product.

There are many resources available to help an organization build their own SSDLC process, and also support them in creating metrics on how successful their program actually is. Examples include BSIMM and OpenSAMM, which provide excellent metrics on process definition and execution. There are other approaches which include processes implemented by major software vendors (e.g. Microsoft’s SDL) and consulting companies, as well as processes that may be mandatory when working with specific customers (such as the currently under-discussion NIST process which will be helpful when working with US government agencies). While all of these processes differ slightly in how and when specific steps are performed, good processes should be customizable to individual organizations – and they should actually be customized! Trying to employ a heavy-weight process that has been designed for a very policy-driven organization on a small startup will most certainly fail, while a process that works for a small startup will usually not work without modifications for a big corporation. Yet despite any customizations, the SSDLC process chosen should remain easily translatable to address new market segments and also give customers the opportunity to compare the program with other models.

As discussed earlier [1, 2], the Secure Software Development Lifecycle (SSDLC) process that I commonly use has an inner core that is built around policies, standards, and best practices, and an outer shell of ongoing activities around security training and education.

The middle circle groups the activities that need to be performed for every release of the product. It does not matter whether the product team is using, for instance, a waterfall model or an agile model; the basic activities are always the same. It is obvious though that in an agile model, where the release cycles are much shorter, some of the activities take considerably less time. This allows the agile team to keep their short release cycle, and the respective SSDLC activities to benefit from early “customer” feedback, which is an integral part of the agile philosophy. Depending on the project, the “customer” can vary, sometimes even between cycles: this role can be filled by company internal customers, operations teams, integration teams, external customers, and many more.

There are eight activities in the SSDLC, and each of the activities can be its own more or less complex process. The content of the “activity boxes” generally depends on the development model and the type of project, but I found the following definitions to be pretty universal and a good starting point:

1. System Concept Development

This activity answers important questions on a comparatively high level for executives, but it is also a good elevator pitch. Questions that need to be answered here are things like: What should the system do? Does it integrate with existing solutions? What is the value add (both intrinsic and extrinsic)? Did anyone else already build this? Why should we build this? Is the solution worth funding? In particular the last question is very interesting for everyone involved. If there are specific security implications (e.g. from a system managing PII), this should have already come up in the discussion by this point.

2. Planning

This is basic project management 101. At this time, a core team has usually been appointed for the project, and roles have been assigned within the core team (remember, this holds both for waterfall and agile models, where roles may change once a Potentially Shippable Increment [PSI] has been completed). Questions answered at this stage include things like: What needs to be built? Do we have all the resources we need to complete this iteration? What are the timeframes? Are there dependencies on other groups, or are other groups depending on this project release? Toward the end of this stage, the epics implemented in this phase will be known with high certainty, which allows the security architect to start thinking about their security implications.

3. Requirements Analysis

The requirements analysis is somewhat intertwined with the planning activities. In particular in agile development models, it is not uncommon that teams jump back and forth between planning and requirements analysis, although this happens less frequently the further the agile project progresses. A specific part of the requirements analysis is the security requirements analysis. As in a regular requirements analysis, a lot of the work is driven by the product vision and system concept, as well as the relevant standards, policies, and industry best practices. Based on a security and privacy risk assessment, the team should establish a solid set of security and privacy requirements, as well as quality requirements that will later on help establish acceptance criteria for implemented features.

4. Design Analysis

Once the requirements analysis is complete, the team should have a pretty solid understanding of the “what” they want to build. The design analysis answers the questions around “how” things should be built. The first step in the design analysis requires the architects to create design specifications that include major system components, with information about the type of data these components are processing, the users that are accessing them, and the trust zones in which they are operated. Part of the general design analysis is the threat analysis, which will produce a set of design requirements based on an attack surface analysis. The threat modelling process is probably the most complex part in a Secure Software Development Lifecycle process, and while there are tools and methodologies available that help structuring this process and make it repeatable, it usually requires a skilled security architect.

The Secure Software Development Lifecycle (SSDLC) process discussed earlier is built around a custom security policy, security standards, and security best practices, and completed through extensive security training and education. While the security policy is an important factor, security standards, best practices, and education are crucial to make an SSDLC program successful.

The security standards and security best practices include security relevant government standards and regulations (eg. NIST, HIPAA, PII regulations, …), but also established industry best practices (e.g. OWASP best practices for web application development, PCI-DSS compliance requirements for credit card pyaments, etc). Some standards and best practices are fairly universal, while others may only be relevant for specific projects. As an example, a web application that is processing credit card information will have to follow PCI-DSS regulations, be compliant with the relevant privacy standards, and implement a good deal of the OWASP recommended best practices. An application for a smartphone without its own billing system and without any credit card payment processing on the other hand can skip the PCI specific requirements.

An ongoing activity in the SSDLC is continuous security training and security education. This is fundamental for implementing a successful SSDLC program. Training and education must include all project members: developers, QA, architects, legal, project management, etc. Everyone needs to get a tailored training to understand both how the SSDLC works, and about foundational concepts like secure design, threat modeling, secure coding, and security testing. Depending on the project, the training can include aspects on relevant standards and best practices.

Security training can come in many forms, such as instructor led trainings, recorded video trainings, books, and training on the job. Once everyone in the team has reached a minimum baseline, I found training on the job to be the most effective and efficient in particular for the technical staff. When someone in the team (good case) or in the public (customers, white hats, black hats – bad case) has found a security vulnerability, I recommend to get at least the entire technical team (architect, dev, QA, operations) together for a post-mortem analysis. The person who found the issue then explains the problem, and asks the team to create a fix and a regression test to prevent the problem from happening again in the future. Also, the team must come up with a mitigation that can be used by operations to protect deployments when the latest update with the security fix cannot yet be installed.

There are various secure development lifecycle models. Some of them (like the Microsoft SDL – http://www.microsoft.com/security/sdl/default.aspx) are representing the development process as a timeline, others are representing it as a circle. I personally like the circle representation better, because it symbolizes the life “cycle” very well, and makes it pretty clear that the work does not end after the code has been released, but usually goes into the next round for the next release.

The circle below shows a representation of the Secure Software Development Lifecycle (SSDLC) process similar to the one I created for various businesses in HP, and that I use when I am training teams on secure software development. It is rather generic, which allows it to tie in with existing business processes and development models.

The SSDLC process I usually use is (like many others) built around a custom security policy, security standards, and security best practices.

The security policy is commonly defined by the organization developing the product. The policy may include all kinds of assertions that the organization makes around software development. I have seen policies that are rather philosophical (“we will do our best to crate secure software and fix defects timely”), and policies that are very stringent and precise (“a medium security defect will be fixed in 5 days or less”).

From my experience, having a security policy is more important than the content of the security policy (at least while the content is somewhat useful), because it shows that an organization is committed to security. However, it is crucial that the security policy is known to everyone in the organization, and that everyone is following the policy on peril of losing their job or at least being removed from the project. The security policy must be absolutely binding for everyone, and “everyone” includes people managers, developers, architect, legal, marketing, and everyone else who is working on the project. Such a stringent policy helps to create awareness and gives the security person in charge for the project (for instance, the security architect) the instrument necessary to delay or even stop delivery of the project in case of major security flaws. This may sound very drastic, and it is by far better to rely on good arguments to convince the team members to address a major security issue instead of wielding the security policy club. However, experience shows that without such a powerful security policy, security goals may be quickly sacrificed to meet a release deadline. On a side note, HP has a very strong security policy, and everyone values it so highly that in all the years I have worked as a security architect for HP, I never had to cite it even once.

The first step in any project is creating a list of requirements. In enterprise software development, this step may actually one of the most time-consuming parts. It frequently requires coordinating with several departments and stakeholders, each of them providing information about what the product should do, how it should behave, and how it should tie in with existing solutions. This may include diverse groups such as legal, marketing, engineering, and others. The result of the process is a prioritized wish-list, and most likely some sketches that show how the new product may connect to existing products.

From here on, the next steps somewhat depend on the individual organization and their development model. Although there are a variety of models, a large percentage of projects are developed using a variant of the waterfall model or some flavor of agile development models. Interestingly enough, these two models are representing the two extremes with respect to release cycle lengths. There is a lot of fuzz around how different agile is from waterfall, and sometimes people can get quite agitated around which model works better. I will skip this discussion here, and just note that each of these models has similar phases that are relevant for a secure software development process. The main difference is how the phases are executed, and how the results are used in the development process.

As an example, each project shares the initial requirements gathering phase. This phase is so universal because we always have to find out what we want to build, whether the market needs this, and how the development will be funded. In this stage, it does not matter whether the product is commercial or open source or a mix, because even a developer working in their free time will ask the question of whether implementing e.g. yet another web content management system would be worth their time.

Once an investment decision has been made, the next steps differ to a certain extent. For example, a team using a waterfall style development model will now start a very detailed analysis of the high level requirements gathered earlier, and turn them into much more detailed requirements and specifications. A team following an agile approach will start with a more preliminary design, release code early and release often, and continuously refine the product until it fulfills all of the project sponsor’s requirements. Such a release is often called a “Potentially Shippable Increment”, or PSI. And although agile seems to be so much different from the waterfall model, the agile team will perform similar steps in each PSI as waterfall team does. The difference is in the planning horizon: while the waterfall team plans for e.g. two years, an agile team may only plan for a couple of weeks. Still, the agile process requires a longer term vision to keep the project on track.

The Secure Software Development Lifecycle Process, or SSDLC process for short, should tie in seamlessly into the existing development model that a specific team choses. There are several models of SSDLC processes, but none of the better processes requires the teams to actually change how they develop their product.

Software security still has an aura of secrecy and mystery, and some people still think that security experts are magicians that can secure a product with a tip of their wand. Some self-appointed “security gurus” actually play a big part in creating this mystery and keeping it alive, because they make it seem as if it there is no structured approach to software security and that a security guru is a crucial part in building secure software.

This is, however, not true. Building secure software can be a very well structured process, which makes it reliable and, as importantly, repeatable. Process repeatability and consistent metrics are as important for software security as for any other aspect of structured software development.

Software security is not security software. Software security is about building things properly. Applying magic fairy dust in the form of cryptography, security tools, or pen testing consultants does not automatically make software secure! An important person to have on your team however is a good Security Architect. Security Architects can cover a broad field, which includes supporting the team with application design, establishing and supervising execution of a secure development process, project and program management, engagements with key customers (pro-active and in case of a security breach), securing operations of cloud solutions, and fire-fighting in case something went wrong. Good Security Architects are hard to find, because they must not only be experts in security and master architects, but also have advanced people skills to master the daily tensions of conflicts arising from competing requirements and at the same time be presentable to key customers.

A product does often not primarily sell because it is secure, but it will often miserably fail if it has security issues. Consequently, the Security Architect needs to work closely with product management to balance new features that customers are pushing for against security requirements and defects that must be fixed, but for which customers are usually not paying for directly (and in the best case do not even know about). While Security Architects are often empowered to stop a product release if they have major security concerns, they should use this power very wisely and rather secure a product by creating superior designs that address both security concerns and provide new distinguishing features.

This kind of work often starts by gathering security requirements from customers, creating a design that covers both the functional and non-functional requirements (often in collaboration with other domain architects), and then coordinating the work of several R&D teams to implement the specs and turn them into a secure product.

I will create a series of posts to review one possible embodiment of a secure development process, covering everything from the early requirements analysis stage over the design analysis to implementation and secure operations.