Crowd-Driven Tactical Decision Game: Training Tactical Creativity

In this article we describe how we used the concepts of serious gaming and crowd-sourcing to design, develop and play-test a game to train tactical creativity for staff officers (level 5-6, battalion and brigade) in the RNLA. In the game, trainees solve complex tactical challenges, analyse and discuss them with peers in a structured manner and make adaptations based on the creative insights they gain. Evaluation of the proof-of-concept shows that the crowd-driven game is a promising way of social learning for tactical creativity.

Introduction

The (Dutch) Ministry of Defense has to operate in a fast-changing, dynamic and complex world. The armed forces need to always be able to conduct several sorts of missions and the Chief of Defense must at all times have units ready to contribute to (international) missions. But, like many other Ministries of Defense, the Dutch Ministry has been facing financial budget-cuts for many years. Ever since the economic crisis a severe decrease in defense spending has been ongoing. The budget-cuts have led to several consequences, one of which was the fact that less money is available for training and education of military units at all levels.

In order to retain a certain level of readiness other means to educate and train military personnel have been sought. One option is the use of simulation. Like the Dutch Ministry of Defense’s Strategic Knowledge & Innovation Agenda (SKIA) puts it: “simulation can be used for education and training purposes, but also for other purposes like mission preparation, doctrine development etc.” (SKIA 2011-2015). According to the SKIA, simulation, and serious gaming in particular, can play an important role in (partially) preventing, or at least minimizing, a decline in readiness in certain training areas. In order to do so the potential of serious gaming must be used to the utmost.

The goal of our research project was to establish how the use of serious gaming can enhance the education and training of the individual staff officer at level 5 (battalion) or 6 (brigade). The present article is the result of that 2-year research project. In order to gain insight into the education and training related challenges that staff officers at level 5 and 6 face, we conducted a series of 16 interviews with army officers and other subject matter experts, we visited exercises and we studied literature. This paper describes how we think that serious gaming can help the individual staff officer gain knowledge and experience in one particular topic, tactical creativity, by playing the “crowd-driven tactical decision game”. The central research question is “is it possible to (better) train tactical creativity by using a serious game?”

Tactical Creativity

In the exploration phase of the project the main goal was to determine what the biggest challenges are in the education & training branch of the Royal Netherlands Army (RNLA). In order to do so we conducted interviews, observations and did a literature study. All our interviewees (16) were somehow involved in the education & training branch of the RNLA and / or have affinity with serious gaming. Moreover we have observed during staff exercises and used literature to get an idea of the challenges in the education and training branch.

During the interviews, several topics came up. Topics ranging from “thinking out of the box” to “efficiency of large-scale exercises” were mentioned by various interviewees. There was however one topic that was mentioned as an important challenge by all 16 interviewees and that is the problem of tactical creativity. According to the interviewees, officers are not / no longer proficient in ‘The Art of War’¹, because there is too little time available to train them in this topic. The Art of war “requires the intuitive ability to grasp the essence of a unique military situation and the creative ability to devise a practical solution”. (US Marine Corps, 1997: 18). An important topic, however in the current set-up of the education & training cycle, staff officers gain only some experience with it.

Tactical creativity is an important aspect of the Art of War and is defined as thinking and reasoning fast and without knowing everything being able to come to a plan or a decision. This plan does not have to be perfect, since “there is no such thing as a perfect strategy or even plan; indeed, to seek such perfection is to forget or deny that the enemy is not inert and has a free creative part in the conflict which is directly opposed to one’s own. As such, one should be seeking to devise a strategy that is better than his in the circumstances” (Smith, 2005: 13). Even though the plan does not have to be perfect, it needs to be composed faster than your adversary’s plan. For this, tactical creativity is indispensable.

We found that within the education and training branch of the RNLA, attention is mostly focused at things that are somehow measurable. This is not surprising of course, because it is only human to want to determine what the result of an effort is. The effect however is that during exercises only an operations process is carried out with too little focus on the quality (i.e. suitability) of the plan.

But by ‘just’ following the steps to come to a plan, there is very little need for the staff officers to be creative during the process.

A critical article written by a captain of the RNLA identifies and summarizes the problem of tactical creativity as well. He puts that the desire to measure effects (in exercises) hinders the way tactical creativity ought to be trained. The exercises are not dynamic enough because education within the military focuses mainly at analytical skills and not on the actual execution of the plan. As Captain Soldaat² puts it: “tactics needs to be exiting, fun and dynamic again, because that is the sole essence of tactics. The best way to enable this is to use a game” (Soldaat).

Tactical creativity; a definition

In order to address the challenge of training tactical creativity, a clear definition of the concept is needed.

Creativity

Creativity is a broad concept that knows many different definitions. For our project we have used Sternberg and Lubart’s definition of creativity. They define it as follows: “it [creativity] is a production of ideas, insights or products that are both original and suitable” (Sternberg & Lubart, 1999). Ideas that are original but not a suitable solution to the problem are considered irrelevant.

Tactics

The Dutch dictionary describes tactics as a “well thought-out plan to reach your goal”. In order to develop such a (military) plan, battalions and brigades in the RNLA use a tactical decision making model.

Tactical creativity

What then is tactical creativity? Using both earlier descriptions we can describe tactical creativity as follows: “it is a way of thinking where original, yet useable insights, help to reach a military goal”.

Types of creativity

In her research on creativity, Margaret Boden (2004) identifies three types of creativity. Where conceptual creativity explores the possible actions to be taken within a set conceptual space, transformational creativity bends the rules of this conceptual space in order to extend the possibilities. The third type of creativity, novel combination creativity, is about making unfamiliar combinations of familiar ideas. This concept is of particular interest in this research as interviews with didactical experts and the Ministry of Defense’s competence dictionary indicate that exchanging ideas between peers can lead to new, more creative solutions to a problem.

Concluding, we will explore how the concept of novel combination creativity can be used to gain original, yet usable insights that help reach a military goal.

Gaming for tactical creativity

A game can help present the tactical situation in an engaging manner and let actions and solutions be programmed. These solutions can then be shared among trainees for creative reflection. To explore this potential, we first examine the state of the art of serious gaming for tactical training.

Serious Games for Tactical Creativity

Serious games and simulations are already being used for tactical training at several levels of command in NATO military forces. Especially at platoon and company level the use of serious games for tactical training is common. Simulations like Virtual BattleSpace and Steel Beasts are used to practice combat procedures. However, when concerned with battalion and brigade level, decision making becomes more complex, procedures become less strict and simulation less straight-forward.

In this chapter we explore what games are being used for tactical education and training at this level of command, and if these games support the training of tactical creativity. By determining the game elements successful for training tactical creativity, this serves as a starting point for developing a dedicated tactical creativity game concept.

A number of games have been reviewed varying from proven designs to experimental concepts and from commercial products to hobby projects. The results are based on expert sessions with the Simulation Centre of the RNLA, literature research and a visit to the 2014 CONNECTIONS wargaming conference in Quantico, USA.

The games are distinguished on four main characteristics:

• Interface: The way players interact with the game. By digital or paper interface?
• Adjudication: The means of judging actions and effects. By a human judge, rulebook or digital model?
• Perspective: The way the state of the game world is represented. Conceptual indicators, a tactical map or a virtual environment?
• Single/multiplayer: Whether the game involves one or more players.

Three typical game examples are discussed below.

TacOps4 – A single player, digitally adjudicated game

Interface	Digital
Adjudication	Digital model
Single/Multiplayer	Single player
Perspective	Tactical map

TacOps 4 is a game about modern tactical warfare, and is an officially issued standard training tool in the US military. Tactical maneuvers are simulated on battalion and brigade level. The collection of vehicles, units and weapons from BLUFOR and OPFOR forces is elaborate and these can be issued a range of maneuver related orders. However, the type of actions that the trainee can include in his course of action are predetermined and especially when one considers a more comprehensive approach, for example influencing local population, the limitations of the game are clear. This is no bad thing as the learning goal is tactical procedures, but for encouraging tactical creativity this game design is limiting. The design with predetermined actions and predetermined effects without a human adjudicator in the loop, makes a digital model responsible for responding to complex battalion or brigade level decision making. Additionally, more complex collaborative or adversary interaction is limited because the game is single player only. Human ingenuity still trumps digital models when it comes to out-of-the-box tactical solutions.

MASA Sword: A multiplayer, digitally adjudicated game

Interface	Digital
Adjudication	Digital model
Single/Multiplayer	Multi player
Perspective	Tactical map

 

MASA Sword is a wargame that simulates forces with a high level of artificial intelligence, used for tactical training and analysis. Compared to the previous category of game, it is more complex in that it supports a large number of human players as allies or opposing forces. Furthermore, it includes tools to model AI decision trees, making it possible to expand the digital adjudication model. Still, the decision trees pass judgement on player actions given a predetermined set of parameters, which not necessarily encompass the complexity of a players reasoning. Thus, while multiple players explore their conceptual space and apply novel combination creativity, they cannot apply transformational creativity.

While this simulation platform is great for training and analysis of tactical procedures, it will not likely result in unprecedented tactical solutions, such as the case of the historic battle for castle Itter, where German forces allied with US forces to defend against SS troops.

Diplomacy: A multiplayer, human adjudicated game

Interface	Board game or digital
Adjudication	Human / rulebook
Single/Multiplayer	Multi player
Perspective	Tactical map

Diplomacy is a game in which players represent one of seven world powers in Europe at the start of the 20th century. The goal is to take ownership of as many supply lines as possible. Originally, it was released as a board game but in 2005 a digital version was published. The game is a great example of a limited set of simple rules combined with the complexity of human strategizing and the freedom of partly human adjudication. The game master can take into account not just the player’s actions, but also the reasoning behind them, and respond accordingly. These characteristics support all three types of creativity, including transformational creativity, and therefore innovative, unpredictable solutions can be played out.

The three game examples above illustrate how multiplayer, human adjudicated games are best suited for training both conceptual, transformational as well as novel combination creativity. Therefore, teaching tactical creativity is most feasible in these kind of games. The human-in-the-loop design is essential for this learning goal. Current classes in the MMO course already include such training in the form of paper Combat Tactical Challenges (CTC), but the nature of the paper-based approach makes it harder to share creative solutions among trainees, anytime, anywhere. The idea was formed of creating a digital platform that leverages the concept of crowd sourcing to make tactical challenges more accessible and to gather and exchange volumes of creative insight.

Crowdsourcing

Examples of crowdsourcing are ubiquitous in the current day: Kickstarter projects ask crowds for micro-investments in innovative technology, the protein folding game Foldit uses the combined wits of thousands of volunteer players for scientific efforts and increasingly social media is used by commercial and governmental bodies to source opinions and judgements. Given that it sees such varied usage, a clear definition of the concept is desirable. According to an extensive literature study, crowdsourcing is “a participative online activity in which a non-commissioned, undefined public is called upon for the voluntary undertaking of a task” (Estellés-Arolas & González-Ladrón-de-Guevara, 2012).

In history this voluntary concept has been applied before, such as gathering meteor observations of the public for astronomy research, but the emergence of online platforms has enabled crowdsourcing to become much more practical. Online or digital platforms make it possible to reach out to a worldwide audience, with minimal delay. This way, creative input can be sourced or digital tasks can be distributed. Often, volunteers are intrinsically motivated to contribute, think for example of Wikipedia contributors or moderators. In other crowdsourcing applications, game elements motivate participants to engage in seemingly boring tasks. For example, the Foldit game uses 3D visuals and a points reward system to create incentive. The AstroDrone project challenges players to fly a drone through an augmented reality obstacle course; while it collects data on facial expressions.

The premise of crowdsourcing combined with gaming elements, namely intrinsic motivation to engage in meaningful tasks, could be applied to the process of learning. Where gaming elements can help the student engage with the learning material, the crowdsourcing element can support social learning by exchanging insights and lessons learned with peers. To see if this idea could potentially help educate tactical creativity, the concept of the Crowd-Driven Tactical Decision Game was developed.

The Crowd-Driven Game Concept

To determine whether crowdsourcing indeed has potential for training tactical creativity, a game concept was designed and implemented as proof-of-concept at a technology readiness level (TLR) of 3. The goal of this proof-of-concept is to give an impression of the gameplay and to determine whether it can be of value for education and training for the Defense organization. In an evaluation session, the game concept was used to evaluate learning outcomes in a mid-career course on tactical level land operations (MMO) in a classroom setting.

Game design process

To work towards an innovative concept, an iterative process of design was used. First, a paper prototype of the game was developed and evaluated with Defense and TNO experts in the field of simulation and gaming. Based on the feedback, the first digital version was implemented and play tested with instructors of the MMO class. This led to the second and final version of the game for evaluation in a pilot with MMO trainees.

Game concept

The game concept is a Combat Tactical Challenge (CTC) where trainees are challenged to solve tactical problems on a digital tactical map. However, the trainees are also asked to analyze and dissect each other’s solutions. This is facilitated by a digital interface that visualizes the differences between individual solutions in terms of military principles. A database stores all tactical solutions – a growing database of tactical approaches that can serve as inspiration for subsequent classes.

The game concept combines the proven use of CTC in the MMO class with the innovative aspects of crowdsourcing and peer analysis.

• Crowdsourcing: Trainees generate tactical solutions for a series of tactical scenarios. These solutions are stored in an ever-growing database. By collecting solutions through the years, a substantial database of tactical knowledge with different creative approaches arises. This database can then be used to inspire tactical decision makers within the military. Furthermore, the fact that one is contributing to a body of tactical knowledge can enhance intrinsic motivation for learning, as it is no longer one-way traffic.
• Peer analysis: By asking trainees to analyze each other’s solutions in a structured manner, they gain insight in the diversity of tactical approaches. The structured approach involves the use of 13 tactical characteristics for deliberation, which stimulate the trainee to think outside the box yet within suitable and realistic bounds. An important aspect is that trainees are not asked to judge solutions as right or wrong, but to characterize them by attributing a fixed number of points over the 13 characteristics. In this way, diversity is stimulated by not focusing on a singular optimal strategy. The alternating role between problem solver and analyst creates an opportunity to gather creative inspiration and directly apply this in an adaptation of your earlier solution. The face-to-face interaction between trainees in this process of alternating roles harnesses social learning.

Design of the digital environment

The proof-of-concept has to meet a number of practical demands:

• It has to be playable on standard-issue laptops in use in the classroom
• The tactical solutions have to be stored centrally on the network
• The freedom for tactical solutions should not be limited by digital interface interaction

Platform

A web-based platform based on ASP.Net, JQuery and SQL was chosen to quickly develop a prototype that can be accessed easily in a distributed manner. The use of JQuery allows for relatively easy adaptations in interaction-design. ASP.Net provides standard functionality for a Model-View-Controller framework and SQL takes care of the central database. These development frameworks have allowed us to make quick progress in an iterative manner.

Menu interface

The interface of the game is opened in a web-browser. The trainee logs in with his or her personal account, and chooses to either view solutions or to solve challenges. The trainee then chooses a scenario. In case of viewing solutions, the trainee can choose from all solutions that are available in the database for the given scenario. The main tactical map view of the game is first displayed.

Tactical map view

The tactical map is schematic with limited detail information on the terrain. There is also no possibility to zoom in to a more detailed level. This is chosen so trainees are not tempted to micro-manage their tactical approach: comparisons should be made at the same level of abstraction.

Database

To store tactical solutions, a data model was created that connects the identity of trainees, solutions, tactical scenarios and instantiations of standardized tactical symbols from the military APP-6C and comprehensive approach V1227 tactical symbol sets³. Furthermore, the analysis of tactical solutions are stored. This makes it possible to characterize solutions by averaging over a number of analyses.

Tactical symbols

An important design decision was to limit the usual APP-6C symbol set, because it is nigh impossible to work swiftly with more than a thousand symbol possibilities. Therefore, based on an analysis of tactical solutions from earlier classes, the symbols where limited to the task-verb and operation activity categories of APP-6C and furthermore the whole of the V1227 comprehensive approach symbol set. The total number of possible tactical symbols amounts to 45.

Figure 2: Task verbs breach, destroy, seize, secure

When placing symbols on the tactical map, the trainee is asked to add a descriptive text of the effect and furthermore the actor/unit the symbol relates to. More extensive functionality, such as adding comprehensive approach themes or drawing areas of effect for a symbol, has not been implemented.

As we do not want to limit the creative freedom of the trainee, he should not be limited to the symbol set only. Therefore, the trainee also has the option to use a free-drawing tool. This can be used to supplement symbols with areas of effect, movement directions, and so on. These illustrations are stored together with the symbols in the database.

Analysis interface

As the trainees switch role from problem solver to tactical analyst, they enter a different interface view. In this interface, the trainee controls a spider diagram with 13 dimensions, each representing an aspect of tactical characteristics:

• Versatility
• Surprise
• Sustainability
• Safety
• Legitimacy
• Economical use of resources
• Simplicity
• Offensive action
• Independence
• Development activities
• Hearts & Minds
• Diplomacy
• Collaboration

Each dimension can be attributed a maximum of 5 points, out of 40 points in total that have to be attributed. By dragging the axis of each dimension, the trainee divides the points according to what he thinks is the focus of the solution. In this way, a tactical blueprint is created that is descriptive of the solution, at least according to this analyst. These blueprints are aggregated and stored in the database.

Figure 3: The analysis diagram

Step-by-step game process

The game play cycle consists of three main steps. Some steps are done individually, others are done in couples.

Step 1: Tactical solution (individual)

The trainees are introduced to the problem scenario and get 60 minutes to enter their initial solution in the digital interface.

Figure 4: Trainees working on their initial plan during the pilot

Step 2: Analysis (in couples and individual)

In the second step, the trainees come together in couples and asked to explain their tactical solution to each other. They then separate and judge each other’s solution using the analysis criteria in the digital spider diagram.

Step 3: Adjustment (individual)

Based on insights in other solutions, and feedback from the judgement of other trainees, the trainee now gets the opportunity to adjust his tactical solution, or even create a completely new one.

The game session ends with a plenary classroom reflection and discussion on the initial solution, insights and adaptations based on the peer-analysis process.

Limitations of the proof-of-concept

The proof of concept was built at a TRL of 3 and therefore there are some limitations to its implementation, when compared to the design outlined above.

To share solutions, network file sharing was used instead of a fully functional database. This choice was made as our database implementation was not complete enough to guarantee an unrestricted creative process. This means the premise of an ever-growing database of tactical solutions still remains to be tested.

Discussion about the tactical solutions was done in person in the classroom, while the finalized concept would offer the possibility to view, add and discuss solutions in the web-based application.

To evaluate the premise of the crowd-driven tactical decision game we conducted a pilot with our primary target audience, battalion and brigade staff officers. The pilot was carried out during a mid-career course on tactical level land operations in the RNLA (MMO). The 14 participants, all MMO trainees, are ranked Captain or Major and their background is diverse; from infantry officers to medical doctors. The main goal of this particular course is to prepare the officers to function as staff officer at level 5 or 6 and therefore much attention is paid to the Tactical Decision Making process. The level and experience of the participants vary; some of them have already acted at level 5 or 6 where others have not. In our pilot group the division between experienced and non-experienced at this level was about 50/50.

A scenario was developed based on the general Trutta scenario that is often used for comprehensive approach training in the RNLA. The participants were challenged to act as the commander of the Dutch troops in Monkat, Trutta, contributing to an international operation carried out by a coalition of the willing.

The scenario provided the participants with several capacities (construction, logistical, a battlegroup and a staff). The participants were free to use other capacities (available via the next-higher level) if needed. The mission’s goal was to establish peace and stability in the region and to monitor the conflict between two rivaling clans. A central challenge in the scenario was the restoring of the control of the airport in the area, a central hub for food supply etc., which had been taken by insurgents from both (rivalling) clans. It is the commander’s task to make sure that the airport is freed from the sieging parties and made operational as fast as possible. It is to be suspected that the insurgents will not accept this without a fight. The participants were challenged to define their Course of Action within the hour.

Results and reactions

During the pilot we have observed the reactions of the participants, and asked them to fill out an evaluation form to get their feedback on the CDTDG concept. We asked their feedback on the design (user-friendliness), content of the game and on their motivation to play the game if it would be made available to them (future use). The first reactions to the concept and the user-friendliness of the game were positive. The participants were able to use the system without much explanation from our side and the debates on the plans were lively and triggered discussions amongst the participants.

Available time

In our set-up we gave the participants 60 minutes to develop their initial plan. However while they were busy working on their plan we noticed that 60 minutes was not enough and therefore extended this to 75 minutes.

Use of symbols

There was a great difference between participants who immediately used the symbols provided by us to draw their plan in the digital environment and the participants who first designed their plan on paper, without using the symbols. The instructors identified the participants using the symbols as the experienced staff officers at battalion or brigade level. The participants who found the use of the symbols difficult had –in general- no experience at this level. Eventually all participants were able to draw their plan in the digital environment.

The scenario

We did not use a ‘standard’ scenario that is normally used in exercises. We did this on purpose hoping that we could persuade the participants to think differently. The scenario we used was fully written out and little to none military abbreviations were used. Overall the participants could work with the scenario, but some told us they missed the Organization of Battle (ORBAT) and the commander’s intent.

Reflection and analysis

The digital environment was regarded as the essential part of the game. Discussing and analyzing each other’s plan was not directly seen as part of the game. During the pilot the atmosphere amongst the students during the reflection, feedback, discussion and analysis was rather casual. The plans were heavily discussed, but in the end not many of the participants adapted their original plan. The participants did tell us that this last step is useful. Why then not many of them adapted their plan (or only very limited) has not become clear to us. Perhaps the participants were (still) satisfied with their original plan.

Criteria for analysis

The criteria for analysis we provided to the participants were useful to analyze their peer’s plan, however all of the participants missed some criteria. In general the participants listed the fundamental principles for military performance as most important. If the game is further developed in the future, these principles will be used as analysis criteria. Other criteria that were suggested are, for instance: is the plan the right estimate of timings and space? Are the desired effects accomplished?

The digital environment

As mentioned before, all participants could easily work with, and in, the digital environment. Except for some small design mistakes the digital environment functioned as designed. All participants did however give us the feedback that they would rather see this game environment integrated into one of the systems that are already in use by the RNLA, for instance Steel Beasts. This is a conclusion that we had already drawn and for a possible further development of the game this is indispensable.

Feasibility for training tactical creativity

The participants indicated that playing the game and discussing their tactical solutions helped them reflect on their tactical options. When asked whether the game could improve their tactical creativity, most participants answered positively.

Future of the game

We asked the participants whether they would voluntarily play this game if this was available at the workplace. Interestingly enough their feedback was that they would like to play the game more often, but that they felt like they would not have the time. This is in line with our previous experience with informal learning; when people are busy in their daily life, these are the types of activities that people tend to skip first. We therefore have concluded that for the game to have to desired effect, it must either be easily accessible anytime, anywhere, or integrated in the education and training curriculum.

Conclusions and Recommendations

The main research question in this paper was “is it possible to (better) train tactical creativity by using a serious game?” This research presents an indication that the use of peer analysis and crowd-driven game mechanics can indeed help train tactical creativity. While the Crowd-Driven Tactical Decision Game was implemented as a proof of concept only and tested on a small scale, military officers offered positive feedback regarding the future usage for training tactical creativity. The results of the pilot lead us to believe that with more refinement and higher production values, this concept can be turned into an effective training instrument.

Future use

The participants were very positive about the idea of using crowd-sourcing within the Ministry of Defense. But on the question whether they would play this game when it is not part of an assignment or their daily job, we received less enthusiastic responses. Participants indicate that they are too busy with their daily tasks.

One solution may be to make the game more attractive and easily accessible anytime, anywhere. This is why a follow-up project was initiated that explores the use of tactical challenges in a mobile application.⁴

Another solution is to make the game part of the education & training curriculum and guarantee structural use through inclusion in classroom sessions.

The premise of crowd-sourcing

Once the game is structurally used in classes throughout the years, a growing body of creative tactical solutions can be recorded in the database along with their characteristics. This enables future trainees to make more novel combinations of tactical solutions. In this way, the learning material becomes fluid, able to include new solutions for a changing solution space: warfare in the battlefield of the future. The potential of crowd-sourcing for both social learning and adaptation to the changing circumstances of warfare, is enticing.

Modeling of effects

While participants indicated that they were learning from each other’s tactical solutions, they were hesitant to adapt their plans. A possible explanation is that because the game does not present any effects, the participant has no clear indication of which solution is better suited. In future, partial digital adjudication could give more incentive to adapt. However, it remains of utmost importance to refrain from judgement as right or wrong, in order to stimulate creativity.

Note

THIS PAPER WAS ORIGINALLY SUBMITTED FOR, AND PUBLISHED BY, THE 2016 MSG 143 SYMPOSIUM, NATO STO
(COPYRIGHT NATO STO) (PAPER STO-MP-MSG-143-05, ISBN 978-92-837-2060-7). IT IS REPRINTED WITH PROPER PERMISSIONS.

Endnotes

References

1. Estellés-Arolas, Enrique; González-Ladrón-de-Guevara, Fernando (2012), “Towards an Integrated Crowdsourcing Definition” (PDF), Journal of Information Science 38 (2): 189–200, doi:10.1177/0165551512437638
2. Boyle, Alan, “Gamers solve molecular puzzle that baffled scientists” (WebCite archive), MSNBC.com, September 18, 2011.
3. Smith, R (2005), “The utility of force. The Art of war in the modern world”, Penguin, London
4. Soldaat, P (unknown), “De commandant als leeghoofd”, unpublished
5. US Marine Corps (1997), “Warfighting”, Department of the Navy, Washington