Cybersecurity Competency Assessment Using Augmented Qualification Standards

Determining the capabilities of cybersecurity personnel is essential to support the Department of Defense (DoD) Cyber Strategy.

The cyber ability of the DoD is contingent upon the continued high standard of performance of cybersecurity and computer network defense (CND) personnel. These personnel are all members of the DoD, the parent organization, but are dispersed in a wide variety of component (subordinate) organizations. Methods to assure a certain minimum level of competency, such as industry certifications and service component schools, can certify and qualify individual ability but are likely unable to qualify cyber individuals on the specific operations of component organizations. This article describes a framework for developing individual-centric and organization-specific qualification standards to augment existing qualification standards to assess the required cybersecurity skills that are unique or specific to component organizations.

Augmented Qualification Standards

Department-wide qualification and certification standards are necessary to support the DoD Cyber Security Strategy, and to ensure a consistent and standardized baseline for individual and organizational cybersecurity ability exists throughout the DoD [2]. However, when considering the specific operations and activities of component organization, there is a unique challenge. If standards take a generalized approach, then the standard can be applied to all component organizations of the parent organization but cannot address the unique requirements and nuances of these component organizations. If standards take a specific approach, then the standards can incorporate all of the requirements of each component organization, but the standard becomes time-consuming to develop, contains large portions of content that are not applicable to component organizations, and places an unnecessary burden on individuals participating in the qualification process [4].

To overcome these limitations, organizations can deploy augmented qualification standards that support the existing qualification requirements of the DoD while addressing additional organization-specific and individual-centric qualifiers. In doing so, the component organization satisfies both the parent organization requirements for a general baseline ability, and the organization-level requirements for tailored operations and capabilities. The DoD is positioned to understand the strategic requirements of its component organizations, but these individual component organizations are best poised to understand their own operational requirements and should design their qualifications accordingly [3]. To accomplish this, while ensuring that qualification standards remain relevant, organizations should strive to rapidly develop and deploy qualification standards for their operational personnel. In line with the current tradition of qualifying individuals, these standards should be individual-centric. This is partially realized in initiatives to develop job-based requirements, such as the US Navy’s Job Qualification Requirements [1], but there is no defined emphasis on rapid development to maintain currency.

The Department of the Navy (DoN) implements a Personnel Qualification Standard (PQS) to certify a required minimum level of competency for individuals when performing certain job functions or tasks [5]. The structure of the Personnel Qualification Standard, as outlined in the Personnel Qualification Standard Unit Coordinators Guide, was used as the inspiration for the development of the Analyst Qualification Standard (AQS). To facilitate the rapid development and deployment of a qualification standard, the framework for the PQS was condensed to five sections encompassed the minimum necessary qualification tasks and knowledge.

Individual Qualification Standards Structure

Line Items and Qualifiers

Line items are specific pieces of knowledge or tasks that make up the required sections and content for a qualification standard. Line items in a qualification standard are identified and grouped into sections, and sections are further grouped into levels. When an individual demonstrates knowledge of a subject or the ability to perform a task, a qualified individual, known as the Qualifier, indicates completion of the line item with his signature. When all requisite line items have been completed and appropriately signed by a Qualifier, an individual has completed their qualification standard and obtained the necessary qualification [5].

Fundamentals Level (1000 Level)

Each qualification has fundamental and basic knowledge that is required to understand and perform certain duties. These pieces of knowledge are applied to other areas of the qualification, using the law of primacy, individuals first master the basics which are then applied and expanded upon throughout the qualification to ensure mastery of the material. The original framework from the DoN referred to these as the Fundamentals Section. The developed AQS framework embraced these as the Fundamentals (1000 Level) that contain the basic fundamentals of technical knowledge necessary to perform cybersecurity duties [5].

Systems Level (2000 Level)

In addition to fundamental knowledge, cybersecurity personnel require knowledge of the specific tools and systems used to perform and conduct cybersecurity activities [4]. To address these, the complex systems used in performance of duties are broken down into the most basic components, termed systems. This breakdown allows the content to be covered expediently with greater emphasis on the overall complex system. Ultimately, this knowledge is combined with fundamental knowledge, then synthesized and applied, to accomplish practical tasks duties. The original framework from the DoN referred to these as the Systems Section. The developed AQS framework embraced these as the Systems (2000 Level), which to contain tools, techniques, and methods necessary to perform cybersecurity duties [5].

Applications Level (3000 Level)

Individuals who are qualified to participate in component organization cybersecurity operations must be able to execute required practical tasks in accordance with DoD and component organization policies, procedures, and guidelines. This execution ability is necessary to demonstrate complete synthesis of fundamentals into the use of tools, techniques, and methods, and the application of this to perform real-word, practical tasks. As such, individuals must be able to perform required tasks in accordance with the requirements in the 1000 and 2000 Levels. The original framework from the DoN referred to these as the Watchstation Section. The developed AQS framework embraced these as the Applications (3000 Level), which contain the execution of key operational tasks of the component organization and the application of 1000 and 2000 Levels skills to address scenarios and solve complex problems [5].

Final Qualification

Qualified cybersecurity personnel must discharge their duty and participate in operations in a consistent and reliable fashion. Piecemeal assessment of fundamentals—tools, techniques, and methods—and practical application are ideal for obtaining knowledge, but assessment of actual ability is best determined in a simulated or practical environment [5]. The original framework from the DoN referred to this as a Final Qualification that, at the discretion of a superior authority, consisted of recommendations from qualifiers, observation of duties, a written examination, or an oral board examination. The developed AQS framework embraced this verbatim as a Final Qualification and selected an oral board examination to assess viable knowledge and tangible practical application with the least amount of administrative burden or time requirements. During the oral board examination, a panel of three qualified individuals assess both the theoretical and practical knowledge of a candidate on any topic or content of the AQS in a formal, closed book session.

Analyst Qualification Standards Development Framework

The AQS Development Framework uses a three-phase process. First, requirements for qualification knowledge and practice are identified using four key organizational inputs. Second, the requirements are analyzed and categorized, and then used to create required line items. Finally, a comprehensive review and revision process is used to develop a final AQS for immediate and rapid deployment and distribution.

Development Methodology

Requirements Identification. Requirements were identified using four key component organizational inputs: (1) parent organization required certifications, (2) component organization policies and standards, (3) component organizational operations and procedures, and (4) component organization practitioner experiences and histories. Figure 1 illustrates these inputs, the AQS Levels the enumerated requirements map to, and the application of these into a resultant qualification standard.

Using both parent organization and component organization mandatory certifications, requirements are enumerated from the certification objectives and common bodies of knowledge, using document analysis, to identify and generate requirements for the Fundamentals Level of the AQS. These represent specific knowledge needs, as identified by the parent organization, for individuals to perform job functions. Further document analysis on component organization policies and standards is used to enumerate component organization specific knowledge about operations, tools, techniques, and methods. This is also considered fundamental knowledge and is used to further populate requirements for the Fundamentals Level.

Thorough analysis of component organization operations, performance requirements for individuals can be enumerated using a combination of document analysis and active participant observation, and/or active participation. Document analysis of component organization operating procedures enumerates key tasks and steps required of individuals, while observation and/or active participation on events enumerates key activities that are performed. Each of these analyses creates performance requirements that populate the Applications Level of the AQS. Further, through the use of unstructured interviews with key component organization personnel, as identified by upper management, the resultant narrative can be subjected to a primitive coding process, using both priori and grounded coding, to conduct pattern analyses. This will identify necessary job functions, classified as performance requirements, which populate the Applications Level of the AQS.

Further analysis of the identified Fundamentals and Applications Level requirements is used to generate the Systems Level requirements. By treating each of these requirements lists as documents, document analysis is applied to decompose the Applications Level requirements into the individual tools, techniques, or methods required, resulting in a list of simple systems for the Systems Levels. Document analysis is then applied to the Fundamentals Levels requirements; both validate the Applications Level requirements decomposition by mapping fundamental knowledge to required tools, techniques, and methods, to create a comprehensive list of simple systems required for qualification. Gaps that emerge from the validation, such as fundamental knowledge that is not represented in a simple system, are used to compose additional requirement as a collation(s) of this fundamental knowledge.

Section and Line Item Creation. After requirements identification is complete, the content from the AQS manifests as the creation of specific sections, within the Fundamentals, Systems, and Applications Levels, to classify and contain requirements. Individual line items are then generated to represent each requirement for these Levels. The process is conducted independently for each level in the AQS. Figure 2 presents this process.

The requirements list for each Level is treated as a narrative, and thematic analysis1 is used to develop the sections in each Level. Using primitive coding2 on the requirements, the frequency and commonality of codes is used to group requirements into themes. The resultant themes are identified, named, and converted into sections within the respective Level. Line items are created from the requirements in each section, using the originating data from inputs as a guide, to identify the knowledge or tasks that must be performed to satisfy the identified requirements. The result of application of this process is a qualification standard in which the Fundamentals, Systems, and Applications Levels all consist of individual sections, with lines items populated in each section. The Final Qualification Standard is not considered to be a separate Level, but rather the final section in the Applications Levels, consisting of the signatures of all board members indicating satisfactory completion of the required oral examination board. Conduct of this oral examination board is at the discretion of the component organization.

Standard Finalization. Finalization of the AQS is the final step prior to component organization deployment and is necessary to ensure the resultant AQS document satisfies the needs and requirements of the organization. Figure 3 illustrates the validation and fielding process for an AQS.

After document analysis and practitioners’ inputs have been consolidated into a draft AQS, member checking is necessary to ensure validity and viability of the identified AQS. The member checking involves providing original practitioner participants with the draft AQS, soliciting all practitioner input, and consolidating feedback into revisions based upon requirements and dependencies present in the AQS. This process repeats iteratively and indefinitely until either no conclusive feedback is received or the component organization exhausts their available review time. At this time, the AQS draft is considered finalized AQS, and the qualification standard is distributed immediately and rapidly throughout the component organization using existing or established channels. At a minimum, the AQS and the instructions (including component organization requirements) for use and completion of the AQS should be distributed. Distribution of the first AQS will require these documents to be generated from scratch, but future AQS releases can repurpose existing documentation with minor revisions or changes. When the AQS is considered to be at or near obsolescence, the entire qualification standard is restarted.

Rapid Development and Deployment

Rapid development and subsequent deployment of the finalized AQS is necessary to preserve the relevance of the qualification standards. This strategic initiatives represent a manifestation of long-term planning which addresses organizational objectives and goals that may encompass the parent organization’s mission requirements to include component’s mission requirements; however, inherently it is not intended to address specific component’s operational requirements. This, coupled with the continuous evolution of information technology and cybersecurity knowledge, creates volatile operational requirements; it also mandates rapid development and deployment of qualifications standards to ensure relevance for the longest possible period. Additionally, this manifestation must adapt with industry and operational changes to address qualification of component’s operational requirements.

Framework Presentation

The resultant framework for the development methodology is illustrated in Figure 4.

This framework represents the combination of the three models into a single method for rapidly developing an AQS to satisfy component organization requirements. First, using the requirements identification model, component organizations enumerate performance requirements using a combination of document analysis and personnel interviews, ultimately developing a requirements document for each of the three AQS Levels. The resultant requirements documents are inputs to the AQS construction model, in which primitive coding and thematic analysis are applied to item themes for sections and these Levels, and then decomposition is used to generate specific line items. This results in a fully populated draft AQS that is used as an input for the AQS finalization model, in which an iterative review and revision process is used to develop a finalized AQS for rapid deployment.

Discussion and Limitations

The AQS Development Framework represents a methodology that component organizations can use to rapidly develop their own qualification standards to augment and support the existing qualification requirements of their parent organizations. Using the framework and recommended methodology, organizations can reasonably expect to deploy AQS products rapidly enough to establish currency and relevancy and meet rapidly evolving operational requirements. When developing an AQS product, it is imperative for component organizations to minimize the amount of overlap with existing qualification standards. While some level of overlap is to be expected, particularly in the Fundamentals Level, substantial overlap represents a suboptimal situation, as the AQS is not augmenting existing standards but instead duplicating them. To this end, component organizations should strive to develop AQS products that are differentiated from their parent organization qualification standard with predominantly organization-specific knowledge and applications items.

There are inherent limitations in the use of the AQS Development Framework. The framework addressed the rapid development of an AQS product but assumes that a component organization has the means to rapidly distribute this product. In instances where this is not the case, component organizations will need to develop rapid distribution channels for the greatest viability. Additionally, the framework does not address the development of the supporting documentation necessary for the successful use of an AQS product. It is necessary for component organizations to, at a minimum, develop and deploy instructions and guidance for use and completion of an AQS product. Further, the framework reduces the extensive experience and tacit knowledge of component organization individuals and operations into a simplified series of line items. While this is a viable method to capture qualification requirements, there are inherent experiences and tacit knowledge that cannot be expressed in such a manner, and will inevitably be excluded from capture with this method. Finally, developers of the AQS standard will need to have, or develop, the ability to execute the primitive coding, thematic analysis, and decomposition skills to populate the AQS content. This means there may be additional workload by component organizations to prepare their environment for AQS use.

Future Work

One major burden of the development of AQS using the AQS Development Framework is the decomposition and collation process necessary to populate line items for the Levels of the AQS. One possible method to overcome this would be future research that attempts to create prior codes, ideally realized through taxonomy development, that would provide developers with specific themes of knowledge areas to consider when developing the requisite line items. Additionally, as the use of the AQS Development Framework requires developers to execute primitive coding, thematic analysis, and decompositions—skills not always readily available in component organizations—further research into developing a simplified methodology of this process for practitioner or developer use could simplify the AQS development process.

References

1. Grenert, J. (2014). Personnel Qualification Standards Program (OPNAVINST 3500.34G).
2. Grimes, J. (2005). Information Assurance Workforce Improvement Program (DoD Manual 8570.01).
3. Mudrinich, E. M. (2012). Cyber 3.0: The Department of Defense strategy for operating in cyberspace and the attribution problem. AFL Review, 68, 167.
4. Paulsen, C., McDuffie, E., Newhouse, W., & Toth, P. (2012). NICE: Creating a cybersecurity workforce and aware public. IEEE Security & Privacy, 10(3), 76-79.
5. United States Navy. (2014). Personnel Qualification Standard Unit Coordinators Guide (NAVEDTRA 43100-1J).