Intelligence and Information Gathering Through Deliberative Crowdsourcing

The hollowing of the state has added new challenges for administrators attending to the competing values of the administration. This article examines how the wisdom of the crowds can be used in a deliberative manner to extract new knowledge through crowdsourcing. We will specifically examine cases of intelligence and information gathering through the analysis of a suspected nuclear reactor in Syria and the use of the crowd in mapping unknown or rapidly changing environments. Through case analysis, this article seeks to understand if crowdsourcing can offer a potential opportunity for public managers to reduce transactions costs while engaging the crowd in a form of deliberative governance to understand and potentially solve public problems. Our approach involves applying the seven lessons of deliberative governance (Scott, Adams, & Wechsler, 2004) to our cases in order to produce five administrative concepts for creating mini-publics for deliberative crowdsourcing.

"A thousand pairs of eyes will spot potential problems easier and a thousand heads will come up with more new ideas than just a few." -Oras Tynkkynen, Member of the Finish Parliament (cited in Aitamurto 2012, p.5) Tynkkynen's quote is the essence of why crowdsourcing for intelligence and information gathering offers such great potential.Crowdsourced intelligence is producing one of the most counterintuitive developments in the recent history of foreign intelligence.As hard intelligence is increasingly cloaked in secretive regimes, impenetrable cultures, and information overload, amateurs in plain sight are conducting intelligence collection and analysis.Embracing this free source of surprisingly high-quality analysis is quietly becoming a useful complement to the craft of intelligence, as connecting the dots exceeds the capabilities of traditional institutions.The capability to gather intelligence via the crowd has implications across the public sector, not just in the intelligence community.
While crowdsourcing in government is not necessarily new, it is still somewhat novel and misunderstood.In the United States, government crowdsourcing has been enshrined through the National Open Source Enterprise (NOSE) and the position of the assistant deputy director of National Intelligence for Open Source (ADDNI/OS).Externally, NATO, Interpol, and national military and security agencies have employed similar open-source programs, and a cluster of universities and private organizations have grown to address the push toward open-source intelligence.These approaches increasingly involve administrators working in conjunction with the public and, in some ways, "employing" the public as an extension of their organization.
When Milward and Provan (2000) talked about the hollowing of the state, they were focusing on the replacement of the public sector with a network of third-party providers of service.They spoke of how bureaucracy and its "command and control mechanisms" was "being replaced by a much more complicated" set of relationships for service delivery (Milward & Provan, 2000, p. 359).They also spoke about how "no one organization is able to produce all the services that individual clients need" (Milward & Provan, 2000, p. 359), a similar phenomenon is seen occurring in the area of public sector crowdsourcing.Crowdsourcing is the process by which the crowd, or public generally, is used as a source of labor, energies, resources, and ideas.
We are not proposing that crowdsourcing in its current incarnation is in anyway approaching the hollowing out of government described by Milward and Provan, but rather that a small set of activities are being supplemented by this new form of intelligence/information generation.The focus of this article is to evaluate several cases in which crowdsourcing can be harnessed to gather information and intelligence in an ever-evolving and technologically advanced world, while concurrently flushing out administrative challenges and benefits of this approach.And while we know that networks of providers that deliver traditional services to government are "less stable than firms or governments" (Milward & Provan, 2000, p. 359), they offer a flexibility that the rigid bureaucracies may not-and this becomes particularly evident when we expand the notion of networks to that of the crowd and crowdsourcing.
This article builds upon the framework of crowdsourcing in government (see figure 1) proposed by Clark, Zingale, Logan, and Brudney (2016).In figure 1, crowdsourced problems are arranged along two axes and range from simple to complex.We seek to better understand the more complex challenges posed and how crowdsourcing can "allow the best ideas to rise to the level of discourse while crowding out less helpful ones" (Clark et al., 2016, p. 63).This article focuses on only one type of crowdsourcing-the wisdom of the crowds (top right quadrant in the figure).The terminology of the wisdom of crowds is derived from a similarly named book by Surowiecki (2004), in which the author explores the wisdom of the crowds and finds that the crowd can generate high-quality products through collective intelligence in the absence of topical experts.This notion is reinforced by recent empirical studies of crowdsourcing information accuracy by Kittur and colleagues (Kittur, Chi, & Suh, 2008;Kittur, Smus, Khamkar, & Kraut, 2011).Noveck (2009) finds that, in crowdsourcing, "the greatest challenge is one of design" (p.41).By organizing or designing the way in which government interacts with the public via 57 crowdsourcing, we can create an "organizing processes that allow for not just sharing opinion but a broader participation has the potential to look more like the community, thus be more representative" (Clark et al., 2016, p. 61-62) and deliberative-though representation may not always be a goal.
In this article we are analyzing two cases of crowdsourced intelligence using the deliberative governance lessons from Scott, Adams, and Wechsler (2004) and Brabham's (2013a) best practices in crowdsourcing.Through this process we extract five administrative concepts helpful to engaging in crowdsourcing deliberatively to gather information.The article provides cases to demonstrate applications of crowdsourcing for the intelligence community that have applications in many non-intelligence settings.The first case is described as the "Box on the Euphrates."The Box, it is discovered by the crowd, is home to a Syrian nuclear reactor.The second case is what we will call "crowdmapping".In this example, we explore a number of instances in which crowdsourcing has developed highly detailed maps of a number of unknown or quickly changing places or environments.
With these cases of crowdsourcing, we will demonstrate how public organizations broadly, and not just the intelligence community, may be better able to engage in deliberative processes to gathering complex information (often under conditions of uncertainty) by utilizing technologies powered by the crowd in order to reach a decision.Crowdsourcing in the public context does not have the intention of getting rid of traditional citizen participation, "but rather augmenting more traditional participation routes such as elections and referendums" (Lehdonvirta & Bright, 2015, p. 263).We, like Dryzek (2010), view deliberation as a process of social inquiry in which gaining understanding, rather than winning the argument of the day, is sought.Deliberation is what occurs when varying discourses intersect (Dryzek, 2010).Deliberative processes involve mechanisms for driving and supporting interactions within and between governance networks consisting of, but not limited to, public agencies, non-governmental organizations, interest groups, corporations, research institutes, and universities (Dryzek, 2010).According to Dryzek (2010), these networks are often populated by society's elites; thus, there exists the potential for anti-democratic representation, which could narrow the context for deliberation.Consequently, one of the goals of deliberative governance should be to encourage the formation of mini-publics (a small group composed of the non-elite), which involve citizens discussing contentious and/or complex public issues that have the potential to enhance the quality and number of public deliberative spaces (Dryzek, 2010).It is within this context that our research on deliberative crowdsourcing resides.We are interested in the ways in which crowdsourcing can be used as a tool to create mini-publics engaged in deliberation.To date only a handful of studies have approached the topic of crowdsourcing from the deliberative governance perspective, and all of these articles have focused on policymaking rather than its use in the management of public programs (Aitamurto, 2012;Aitamurto & Landemore, 2015;Landemore, 2015).
In the national intelligence community, there has been widespread outsourcing of government intelligence gathering in recent years (Chesterman, 2008;Shorrock, 2008).Approximately 70% of the Office of the Director of National Intelligence has been spent on the outsourcing of intelligence gathering and analysis (Shorrock, 2008).The private sector intelligence community has inherent conflicts in its operations, as they are profit driven, have shareholders, and are at times outside the reach of the law (Chesterman, 2008;Johnson, 2010;Shorrock, 2008).Organizations beyond the intelligence community access intelligence or information from outside their organization to guide policy and decision making.The Federal Reserve System, for example, relies upon information compiled by Fed staff.These reports draw upon not just data and analysis done by Fed staff but also from privately sourced data on economic conditions, bond yields, and real estate markets, to name a few.The former president of the Federal Reserve Bank in Minneapolis, Kocherlakota (2010), has stated that "the Federal Reserve System is deliberately designed so that the residents of Main Street are able to have a voice in monetary policy." It is clear that information on a wide range of government functions are not solely generated within the government internally.And while "spying for hire," as Shorrock (2008) puts it, may have become the norm, spying or gathering intelligence for "pleasure" is rare, not driven by profit, is done in the open, and exploits publicly available information.The primary case examples we present in this article are from instances of intelligence/information gathering that occurred outside the context of direct government control-though we do present a number of smaller examples throughout the article that include government input or management.
Public sector managers, whether working for the National Security Agency (NSA) or the Federal Reserve Bank, draw upon information from a wide range of sources.The primary cases we present in this article are relevant to the investigation of public sector intelligence/information gathering because they demonstrate public deliberation, the gathering of a mini-public, and the production of new knowledge that could be key to making informed decisions that provide a public benefit.And while these primary cases had their genesis outside of the public sector, there is no reason to believe that these cases are unique to the context outside of public management.Government actors have the potential to produce similar results, given they encourage deliberations following the lessons we learn from our cases.There are limitations and reservations that should be considered before using crowdsourcing in the deliberative processes of intelligence gathering or other public management scenarios, and these are presented in our conclusion.
The remainder of this article is organized as follows: first we provide a brief summary of deliberative governance focusing on the primary lessons offered by Scott et al. (2004), as they relate to the best practices for using crowdsourcing in government as described by Brabham (2013a).These streams of literature were applied to a series of case studies in order to produce a concept of deliberative crowdsourcing.We then discuss the payoffs and limitations of crowdsourced information gathering.We conclude with the implications of crowdsourced information gathering for public organizations.

Deliberative Governance as a Lens to Understand Crowdsourcing
Deliberative governance provides a theoretical lens for understanding the applications of crowdsourcing discussed in the cases that follow.We will demonstrate in our two cases of crowdsourced intelligence/information gathering that discourse, a multi-way and multi-partner dialogue, provides a reasoned approach to engaging the vast and dispersed crowd to create public value.
As governmental entities seek to engage the crowd to provide this public value, the enhancing administrative legitimacy in these dialogues becomes important.A dialogue-focused approach, claims Dryzek (2010), rather than simple "head counts," will enhance administrative legitimacy.To Dryzek (2010), discourse is most effective as a feature of mini-publics, which, in turn, empower participants as a form of discursive representation to form a meta-consensus.A critical feature of mini-publics involves smaller groups of deliberators that act as almost instructive microcosms to a larger conversation.These smaller groups involve a more localized or task-specific forum purposed to reach consensus, discover new ways of considering something, or figuring out solutions while deepening an understanding of social problems.This conceptualization of a process of discourse demonstrates politics in action not merely for purposes of reaching an agreement, but instead to encourage civic competence through engagement and contestation (Spicer, 2010).The cases of crowdsourcing in this article represent mini-publics derived from civic engagement, which is driven by the crowd's competence and dedication to the task-though not all crowdsourcing is dialogue-driven; others include micro-tasking (small jobs completed, typically for compensation) and tournament crowdsourcing (Aitamurto & Landemore, 2015;Prpić, Taeihagh, & Melton 2015).This article is focused on investigating the ways in which deliberative governance can be applied, rather than just the theory that Dryzek and others have developed.We turn to the work of Scott et al. (2004), which designated seven preliminary lessons of deliberative governance by working on a case involving a rural agricultural-based community.They note that deliberative governance "is both descriptive of a growing set of processes that involve citizens in public issues, and a normative response to our currently depleted levels of social capital" (Scott et al., 2004, p. 17).The concern here resides in the ongoing polarization of our politics, which reduces the possibilities for intersecting discourses and creates an increasingly difficult balancing act for administrators needing to manage tensions within governance systems.Our interest is in exploring the role of crowdsourcing as a mini-public deliberative medium to open up possibilities for engagement and discourse.To further develop the research and theory in this area of practice, we will present a set of practical concepts that can be derived from deliberative governance, as described by Scott et al. (2004), in the discussion of our cases.Scott et al. (2004), building from Dryzek's work, sought to explore deliberative processes in action that could provide a promising strategy for renewing public trust and involvement in democratic institutions.Their research produced seven lessons to guide the way in which public administrators could apply deliberative governance to broaden and deepen their approach to citizen engagement and public issues.Scott et al. (2004) and the deliberative governance literature is theoretically based, with little evidence of associated empirical testing.As such, we will also draw from a set of empirical best practices from the government crowdsourcing literature (Brabham, 2013a).Brabham's (2013a) best practices focus on the question of "how": how we can learn about the motivations of the crowd; how to define a clear/interesting problem; etc. Brabham (2013a) describes 10 best practices across three crowdsourcing phases: planning, implementation, and post-implementation.These best practices were created to provide "a practical guide for any government organization hoping to extend their problem solving abilities by crowdsourcing the public participation process of governance" (Brabham, 2013a, p. 21).While the best practices contribute to an applied perspective for initiating a crowdsourcing activity, they are, at best, only superficially connected to the wealth of public administration literature associated with public participation, governance, and deliberation.Furthermore, the applied steps offer little to public managers tasked with a crowdsourcing initiative from a conceptual perspective.In other words, the best practices are useful when organizing primary tasks, but offer little when considering the challenges that arise when engaging in deliberative practices to organize and manage the process.Moreover, by combining Brabham's (2013a) best practices to Scott et al.'s (2004) deliberation lessons, we build upon the crowdsourcing literature by connecting theory to practice.We note that there remains slippage between the literature streams from which additional administrative conceptual footing necessary for implementing and managing a crowdsourced initiative can be explored.For example, an administrator can appreciate that deliberation is an ongoing process that requires facilitation skills (Scott et al., 2004).Or that crowdsourcing must begin by defining the problem, determining the level of commitment to the outcomes, and knowing the community (Brabham, 2013a), there are still "Asses the project from many angles" (p.28).
remaining conceptual questions, such as the role of administration, transaction costs, and relationship-building approaches, or when/if an initiative can be called off or shut down.
Furthermore, because deliberation, as described by Scott et al. (2004), and crowdsourcing, as described by Brabham (2013a), both depict a shared top-down and bottom-up approach in which the locus of control is between the organization and the (online) community, common administrative concepts such as those associated with the checks and balances of hierarchy, control, and structure formation designed to manage input may no longer hold.This means that administrators are on new footing when engaging in a crowdsourced initiatives and therefore need updated conceptual frameworks for managing, analyzing, and deciding upon the next steps.Therefore, our purpose is to create a connection between the experiences of public managers and explore the theory-practice intersection through our case analysis by applying the deliberation lessons from Scott et al. (2004) to the 10 best practices of Brabham (2013a) to flush out administrative concepts around the practice of crowdsourcing.
In light of this, we set out to explore what administrative concepts could be learned for crowdsourcing intelligence/information gathering.Table 1 summarizes the seven deliberative governance lessons from Scott et al. (2004) and the 10 best practices in government crowdsourcing (Brabham, 2013a(Brabham, , 2013b)).These two applied literature streams set the stage for our cases and subsequent analysis.With these in mind, we enter into an analysis of the cases.

Cases Studies
The following sections examine two cases of crowdsourced intelligence or information gathering.The first case demonstrates how a blog comments section turned up "intelligence to die for" in evaluating a mysterious Syrian site.While the second is a set of shorter, inter-related case examples, that demonstrate how the crowd can gather a wide range of information and map it jointly to enhance our knowledge of the unknown or rapidly changing environments-often called crowdmapping.

Case 1: The Box on the Euphrates
A boxy building rests on the bank of the Euphrates in al Kibar, Syria-the purpose and contents are unclear.What is known is that this "Box on the Euphrates" is not an original-it replaced an earlier building on the site, and plenty is known about that building.The Box's predecessor allegedly housed a nuclear reactor, a structure that had been in place for more than six years, and was likely built with North Korean technical assistance.Although the precise intent of building the reactor is unclear, one thing is known: in early September 2007, during the Six-Party Talks with North Korea, Israel executed air strikes on the Box.The next month, Syria scraped the site clean.This is the story not of the Box itself, but of how we came to know what we know about this site.Without security clearances, dedicated satellites, wiretapping, or travel budgets, a loose collective crowdsourced the collection and analysis of publicly available information sources to create estimates that complement and compete with the products of the intelligence community and its global counterparts.

The Public Deliberation of the Box on the Euphrates
On October 25, 2007, the blog ArmsControlWonk.com(Lewis, 2007a) published a brief post on the destruction of the "Box on the Euphrates," a structure near the river in Syria suspected to house a nuclear reactor.The post mentioned two New York Times articles describing the site, an Israeli air strike on it the previous month, and purported aid provided by North Korea to Syria in establishing its nuclear program.Almost immediately people began populating the blog comments section with analyses of both satellite imagery and the political dynamics involved.
Various commenters speculated that the site was "scraped" to build a new building, and identified gates, a railroad with a fenced-in stop and evidence of additional scraping on a nearby hill.On October 26 (Lewis, 2007b), the blog featured a wider-angle satellite photo with another building nearby and speculated on how the intelligence was being addressed from a political frame.The commenters again began providing spontaneous analysis using a surprising amount of publicly available information.One poster noted, "not sure about your assessment of that second image as a 'box.' Instead of another building with 150-square-foot footprint, I see two 40-foot-x-140-foot buildings spaced 50 feet apart; the space between them is pretty close to the same color as the nearby paving" (Lewis, 2007b).Another observed, "the scale/speed of the cleanup suggests that they did not have to remove a lot of concrete (if they did, that should have been visible on someone's remote sensing systems).Removal of large amounts of reinforced concrete is a laborious task unless they chose to dynamite it first.Demolition is likely to be seen and heard.Even if the physical destruction is not seen, the movement of earth movers, dump trucks can be seen" (Lewis, 2007b).
An October 29 post (Lewis, 2007c) confirmed that the intelligence community had known about the Box for quite some time, noting that intelligence about the building was restricted to a few senior officials (and that the intelligence community was kept largely in the dark).The 43 comments that followed exemplified the potentially high quality of crowdsourced analysis, including NASA photos of the site attesting to the age of the building, analysis photos from the Institute for Science and International Security, a cutaway illustration of what a reactor looks like, pictures of reactor pumps, and images of the Box's suspected cousin in Yongbyon, North Korea and a British reactor.One commenter provided a speculative timeline to explain the timing of Israel's attack and diplomatic silence.Others considered the Box's site and surrounding geography with possible motives for the decision to place the structure there.
The next post on the subject, on November 5 (Lewis, 2007d), featured an on-the-ground picture of the region and mentioned its proximity to well-known tourist sites.The post observed that portrayal of the site as remote indicated biases toward the conclusion that the site held a reactor.The comments section again came alive with coordinates, satellite photos, and insightful analyses of what was known and what current theories had emerged.A post on the blog the next day (Lewis 2007e) complemented analysis that led to construction dates, height of the building, and more reliable images taken by a Japanese tourist.
Discussion of the Box was then quiet for two months until January 12, 2008 (Lewis 2008a), when a new post observed that the Box appeared to be under re-construction and warned again of a bias toward concluding that the site was intended to house another reactor.Commenters began to evaluate the height of the box by establishing the time the satellite passed over, calculating the length of the shadow it cast, and then counting the pixels in the satellite images.
Characteristics of a team began to emerge from the commenters, with one person offering to perform calculations if someone else could provide a satellite over-flight time, and yet another doing the research to find these data.With the over-flight time, the former speculated, "if the sun's altitude was 26.850 degrees then the height of the north west corner of the roof is 0.235 x the width (i.e., the east-west dimension) of the building.Given the … estimate that the building is 60 metres by 60 metres, this puts the edge of the roof at 14.1 metres high.By a similar method, the raised centre of the roof is then 15.7 metres high" (Lewis, 2008a).After some discussion, the commenter revised the estimate: "If I correct my original estimates, I end up at 16.5 m for the corner and 18.4 m for the roof apex, which, in the grand scheme of things, is the same as Yale's values.After all, the pixels are a bit blurry, and at the angles we're talking about a change of (an assumed) building width from just 60 to 62 m will change the height calculation by 0.5 m." A January 24, 2008, post (Lewis, 2008b) noted that two prominent experts had an op-ed in the Washington Post on North Korea's compliance with its nuclear commitments.In the op-ed, Albright and Shire (2008) reference the newly built "Box in the Desert" and claim, "It is almost certainly not a reactor."After three months of relative silence, an April 24 post (Lewis, 2008c) highlighted articles in the New York Times (Sanger, 2008) and Washington Post (Wright, 2008a(Wright, 2008b) ) offering photographic evidence that the Box was, in fact, a reactor.Commenters discussed comparisons of the Box and Yongbyon and typical procedures preventing the introduction of cameras into such a facility.A comment near the end of the thread makes an observation that includes an analysis that allows for the estimation of the number of fuel rods and eventual output of the reactor, particular as it compares with the Yongbyon facility (Lewis, 2008b).Another commenter noted that the ArmsControlWonk.comvolunteers and the US Central Intelligence Agency (CIA) had encountered the same issues in explaining the absence of a hot water outlet pipe, and yet another located that pipe on Google Maps.
A follow-up post the next day (Lewis, 2008d) questioned the timing of the release of evidence about the Box, followed in the comments with a pragmatic discussion of the politics and espionage methods involved in the release, especially regarding the progress of the Six-Party Talks with North Korea.A subsequent post of a CIA briefing on the matter resulted in an extended parsing of the briefing's wording, inflections, and omissions.A frequent commenter observed that the arms control community tends to think in terms of national programs and suggested that modern nuclear programs may employ a division of labor, making use of different nation-states' strengths while keeping the collaboration below suspicion on the whole.An April 27 post (Lewis, 2008e) featured impressive graphic representations of the Box's facilities configuration and that of the Yongbyon reactor.The comments for this post included ground-level photographs of the Syrian reactor during construction and noted significant differences between the Box and Yongbyon.A commenter, obviously familiar with nuclear plants, observed: "I also am not convinced by the suggestion that the spent fuel pond would be within the building, as that puts it rather close to the reactor.I doubt having so much water so close to a reactor is good for safety reasons, and also the pond water should be as cool as possible to minimise Magnox fuel corrosion, and having it so close to a big heat source could make that tricky" (Lewis, 2008e).It followed with analysis that included highly specific details on the reactors and calculations of their size.Contributors discussed further differences in the infrastructure between the Box and Yongbyon, including animated images with grid overlays to aid in assessing dimension.This was followed by a fairly technical discussion among the commenters, including terminology and methods of size assessment.
Blog commenters offer more examples of the crowd-produced outcomes, but, for brevity's sake, it seems clear from these examples that the potential for complex analysis exists in this context.A prominent intelligence analyst, featured in a New York Times article, claimed that the work done by the ArmsControlWonk analysis was "intel to die for" (Broad, 2008).While not always predictable, it is clearer and clearer that the crowd is capable of real quality work.

Case 2: Crowdmapping
In the United States we take for granted that we have maps to find all of the places that we might need to find.Using a traditional map made from paper, we can follow street names on that map to find our way to where we need to go.Increasingly, GPS-enabled mapping, using dedicated GPS mapping devices or through a smartphone, has become the norm.However, in many places in the world there are no accurate or reliable maps.If you want to get from point A to point B in a place shrouded in secrecy like North Korea, how do you go about gathering that geographical information?This is where geospatial intelligence (GEOINT) comes into play.In fact, there is an agency within the US military infrastructure dedicated to gathering GEOINT, the National Geospatial-Intelligence Agency (NGA)."Anyone who sails a U.S. ship, flies a U.S. aircraft, makes national policy decisions, fights wars, locates targets, responds to natural disasters, or even navigates with a cellphone relies on NGA" (National Geospatial-Intelligence Agency, 2013).
Whether they are on the battlefield or on a humanitarian mission, those in the military need to know where to go and understand their surroundings.Using GEOINT gathered from a variety of sources, the NGA says that they help to "Know the Earth … Show the Way … Understand the World" (National Geospatial-Intelligence Agency, 2013).
The NGA and other parts of the national intelligence community gather information from many traditional sources of intelligence.This might include government and commercial satellite and aerial imagery (using visible and infrared images) and information from the personnel on the ground.These data sources can provide precise information about elevation, depths, topography, and other physical features.However, these images may not always provide a complete picture of the reality on the ground.There also may not be sufficient information from the ground to provide more detailed data on local conditions to provide a full picture of the environment.In a dynamic and changing crisis environment, gathering information may be challenging for the formal intelligence community, particularly if they do not yet have any boots on the ground.
With the advent of crowdsourcing, the NGA and other mechanisms within the broader intelligence community do not have to rely solely upon their own agents and analysts to collect and analyze potentially vital geographical information.They can now rely upon crowdmapping to fill in the gaps.For many applications of crowdmapping, the information could not be collected "without mass-participation, and would most likely be left undone without crowdsourcing" (Aitamurto, 2012, p. 10).These mapping projects, whether or not they have national intelligence applications, "are an efficient way to visually demonstrate the geographical spread of a phenomenon, whether that is violence, bribing, snow storms, or traffic jams" (Aitamurto, 2012, p. 10).

Crowdmapping Example 1: Google MapMaker
Few places in the world are as mysterious as North Korea; and few places in the world have also escaped Google Maps.However, thanks to crowdsourcing North Korea can now be added to the places we can search using Google Maps.In January 2013, Google released its North Korean map, which includes monuments, parks, streets, gulags, and train stops in its capital, Pyongyang.Prior to the January release, the map entry for North Korea was vacant aside from a place marker for Pyongyang and a single river.Google has credited a group of crowdmappers with the details of North Korea, stating that they started building it in 2009 "based on satellite images, public information and local knowledge" (Mysore, 2013).
The crowdmappers used Google MapMaker, a service that allows users to add data to areas with imprecise or inaccurate details.The program works much like Wikipedia, wherein users check the data submitted by other users, and information is constantly updated.Along with street names and other details, Google has linked photographs to their respective sites, allowing people from around the world a truer view of one of Earth's most unknown places.
Richard Hintz, a Google map contributor, explained that using the MapMaker program "you can trace over the imagery and define a road" (How Google finally mapped North Korea, 2013).The contributors, like Hintz, are not necessarily people with on-the-ground knowledge of North Korea but rather just had an interest in contributing to the mapping project.To ensure that "there is a continuum of credibility" for the maps, a check was established where each new feature that is added "is held for review" before being added to the final product (How Google finally mapped North Korea, 2013).This review process can help to avoid some common complaints of wikis that allow open access to editing.
The Google MapMaker platform is very much a bottom-up approach to mapmaking-even if Google directs it.Crowdsourcing needs direction but seems to work best with a broad range of contributors.Democratization in map making has the potential "to add to an old science by allowing anyone with access to a computer to upload their findings" (Valdes, 2013).De Leeuw et al. (2011) find that the local knowledge that is aggregated via crowdsourcing actually had greater level accuracy than professional surveyors in urban areas of western Kenya.This finding provides support to the power of the crowd and the importance of local knowledge in many map-making endeavors.
However, it is "also important that we acknowledge the pitfalls and limits of crowdmapping.In many parts of the world such citizen mapping has proven challenging, if not downright dangerous.In many places, little can be achieved without the approval of local and or national authorities-especially in North Korea" (Valdes, 2013).Crowdmapping has the benefit of being flexible and easy to modify in response to the changing environment and new (and local) knowledge.But that changing environment may be changing so fast and may be so volatile that the just-created map is already out of date or misleading.Without third-party verification or trust systems (rating users, for example), faith in these maps may be limited.

Crowdmapping Example 2: OpenStreetMap
OpenStreetMap bills itself as a project that "creates and distributes free geographic data for the world."The creators of the service say it was started due to most "free" maps "[having] legal or technical restrictions on their use, holding back people from using them in creative, productive, or unexpected ways" (OpenStreetMap, 2013).
The service is a true wiki, which allows users to add, modify, and delete data.In order to modify or create a map, a user must sign up and agree to share their contributions under OpenStreetMap's license, ensuring that data stays free for everyone while allowing information to be attributed to specific sources.OpenStreetMap specifically mentions that pseudonyms or nicknames are preferred, and that email addresses will never be sold or used for purposes other than validation and user messaging.
The OpenStreetMap platform is interesting in that it unites communities that are interested in many different features of maps.In its special-interest topics section, a user can find a map for the blind, humanitarian relief maps, infrastructure maps, and a wide variety of other specialized designations.These maps allow certain niche map-users a specialized and experienced community with which to collaborate and consult.
The humanitarian map section of OpenStreetMap shows the value of these services.Humanitarian maps were created for many crises around the world, including the recent coups and secession in Mali, post-election crisis in Iran, and continuing conflicts in the Palestinian Territories.

Crowdmapping Example 3: Ushahidi
Disaster relief organizations and those in need of disaster assistance have been using the website Ushahidi.com 1 as a gathering point on the Internet to identify where resources are and where they need to go.In the aftermath of the January 2010 earthquake in Haiti, people were able to send text messages (SMS), email messages, Twitter messages, or Internet postings to Ushahidi's website identifying their location and their needs.Ushahidi had volunteers in Boston who translated these messages from the original Creole/French into English.This information was then logged, mapped, and put onto their website (Nelson & Patel, 2011).With this information from the Ushahidi volunteers, aid agencies from around the world working in Haiti were better 1 Ushahidi means "testimony" in Swahili.

Case Analysis and Discussion
These cases offer a range of potential benefits to not just the intelligence community but to public organizations across all levels of government that are seeking alternative methods for gathering information on their communities.To analyze the cases, we look first to the theoretical deliberative governance lessons of Scott et al. (2004) and then draw upon the empirical best practices proposed by Brabham (2013a)-lists of both can be found in table 1.
From the cases we presented in this article and the lessons of deliberative governance and the practical best practices, we propose five administrative concepts that draw from theory and practice to guide managers in the utilization of crowdsourcing for intelligence/information gathering.An overview of these concepts are provided in table 2.

Concept 1: Administrators must determine what role they want to play.
Do you want to act as a process manager, as process participants, and/or as receivers of information?The crowd may gather with or without an agency's involvement, thus identifying the agency role is vital (though some roles may be outside of your hands).Being involved gives administrators a choice in their role because they will likely not be able to stop the discourse from happening.
In the cases presented in this article, the discourse was an ongoing, developmental process that should be understood as a series of connected experiences (Deliberative Governance Lesson 1 or DG 1).The commenters on ArmsControlWonk came back to the blog time and time again to provide further insight, provide corrections, and discuss what they thought might be going on on the shores of the Euphrates.The blog provided a venue for the creation of a mini-public-a space for nonelites to have open discourse on a complex public issue (Dryzek, 2010) that ultimately created high-quality intelligence.In the crowdmapping examples a variety of entities provided administrative roles: some were private sector (North Korea and Google), some are open-source driven (OpenStreetMaps), and some were a hybrid of public, private, and volunteer administration (Ushahidi).In these examples, the public sector stood mostly to the side and let the crowd create intelligence for them.This does not have to be the case-public organizations can be more of a driver of these activities, as NASA experienced when it engaged the crowd to find star clusters in newly release photos of galaxies (Wolford, 2014).
Individuals from around the world were permitted to share either their time (tracing roads visible from satellite images), providing their personal experiences and insight from visits to these places (Syria and North Korea, for example), or providing their technical expertise (nuclear physics, mathematics, satellite imagery).The participants in these projects were connected to the goal and understood it as an ongoing and developmental process (DG Lesson 1).Their contributions through these deliberative processes did not provide them with material benefits but rather a community of similarly oriented people.
When choosing the role a public organization wants to have in the crowdsourcing process, it ought to take heed of DG Lesson 2, which is that deliberation works best when it begins early so that changed minds can make a difference.If the project is administered (or steered) by a public organization, the organization should recognize that deliberation should not be an afterthought; rather, it needs to be there from the start, allowing participants to have a role in steering as well.
By allowing the mini-publics to form and have a role in the outcome, we can draw from the DG Lesson 4, which would indicate that, due to the uncertain outcomes of many of these projects, knowing the interests of participants will facilitate a more successful outcome.If people providing Creole to English translation are forced into action without consent or without knowledge of the goal, their motivations to contribute will be diminished.If we asked people to trace roads onto a map without sharing with those participants the outcome of the map, would they contribute?Furthermore, if we asked for the crowd to help create these maps, then did not allow them access to the cumulative project at the end of the project, the outcome does not provide something in the shared/mutual interest; rather, it just creates something for the private interest of those that hold the new knowledge.Regardless of what role the public sector plays in a crowdsourcing venture, if the deliberations generated by crowdsourcing are going to be able to change minds, expand knowledge, and provide for the public good, these connected and shared experiences of crowdsourcing in the public sector should be able to produce openly shared knowledge and administrators need to consider their role in this process.
As active participants in crowdsourcing, i.e., more than just passive recipients of information, governments need "to communicate to the online community exactly how much impact usersubmitted ideas and labor will have on the organization," which is part of what Brabham (2013a, p. 22) delineates when asking governments to determine their "level of commitment to the outcomes."In the cases we present in this article, the outcomes are informational or advisory, and participants are aware of this.The Obama Administration implemented an online petition process that can create new knowledge for the administration on the priorities of the public.However, the expectation the Administration set out from the start was not that they would implement a petition as is, but rather that it would provide an official response.While some petitions are humorous (see the request to build the Death Star (Shawcross, 2013), others are more serious (such as the request to resettle Syrian Refugees (We The People Team, 2015).
What is common regardless of the seriousness is that, once the set threshold for participation is surpassed, the White House follows through and creates a response-though the depth of the official response does vary substantially.These roles require resources from governments, which leads to our next concept.
Transactions costs can, however, be minimized when the crowd is a willing and engaged participant.This means engaging the crowd early and often by providing feedback on how the crowd's involvement and ideas matter.Crowdsourcing is reducing transactions costs for sharing information and deliberating (Clark et al., 2016).DG Lesson 3 indicates that deliberation requires facilitation skills, management, and a significant investment of public resources.While the crowd can produce "free" labor, the coordination and management of these projects is not free.If a public agency were to use the case examples provided in this article as templates for engaging the crowd to produce intelligence/information, it should be clear that consistent engagement, one that has pay-off or value for both sides, will be necessary-though the benefits may be less visible at times when building social capital (DG Lesson 5).
In the case of the Syrian box analysis, the group's ability to coordinate their energies to gather and analyze information was facilitated by a simple set of blog posts and the comments that followed each post.Comments in the blog were managed by ArmsControlWonk contributors, not the commenters themselves.Controlling or moderating the flow of information via this type of deliberation is one of the biggest transaction costs for crowdsourcing.For example, the magazine Popular Science recently shut down the comments section of it website because of what the editors describe as "a fractious minority" that is able to wield "enough power to skew a reader's perception of a story" (LaBarre, 2013).They further note that "commenters shape public opinion; public opinion shapes public policy; public policy shapes how and whether and what research gets funded-you start to see why we feel compelled to hit the 'off' switch" (LaBarre, 2013).Online trolls have been driving what Popular Science sees as a "politically motivated, decades-long war on expertise [which] has eroded the popular consensus on a wide variety of scientifically validated topics.Everything, from evolution to the origins of climate change, is mistakenly up for grabs again" (LaBarre, 2013).
In the case of the Box in Syria the Internet troll problem did not create a degradation of the deliberations because someone internally, a blog moderator, has the control of the off switch (i.e., not allowing a comment to be posted); thus, there are clear trade-offs.It takes resources to moderate comments, but the moderation removes the motivation of the trolls-instant gratification and attention to their flame throwing comments.Their "every sadism" cannot infect the crowd if the moderators are able to prevent them from entering the scene.Clearly, utilizing this tool in the public sector becomes more challenging than it does for a private entity like ArmsControlWonk, because of perceptions of First Amendment encroachment that moderation may create.This exemplifies another advantage of the distance that a private blog like the one used in our example provides.In the end, the ArmsControlWonk blog is not a democracy but more a benevolent dictatorship.Its governance is subjective, but it provides just enough structure to allow for deliberations to move the conversation forward.Though unwritten, the rules between and among the blog and its commenters are clear, and they are followed.This is one of the keys to the success of private sector crowdsourced solutions as well.
The tools used in the crowdmapping projects are a bit more technically complex than a web page comment box, yet still quite accessible and rapidly reduce the transactions costs associated with coordinating the deliberations of the crowd.The real tool in all of these mapping examples is the intelligence that goes into making them, while the technology itself is merely facilitative in reducing transactions costs.The successes of these mapping projects that are connecting the crowd appear to have some similarities to those that we saw in the Box example.Again, we have a situation where people voluntarily bring bits of intelligence to the table and by and large agree to be bound by some minimal rules.The Google MapMaker project would appear to have a lot more checks in place to assure reliability, as additions to the maps are held for review-similar to a comment moderation in our Box case.When they made this map of North Korea, it was not done to aid in a military action or provide humanitarian aid to the people in that country (though this information may be used in that way in the future).This map appears to have been done for intellectual curiosity or to serve as a challenge for those involved.Nonetheless, the cost of producing this map was not free and again required a substantial investment to moderate the flow and quality of information.It was not intended to be a democratic map making process, but rather one that is as accurate as possible.
As the name would imply, the OpenStreetMap project is more open to change than the Google mapping project but still offers a set of tools to users to assure reliability over the long run.Using these maps you can see what changes have been made and by whom (although the users are mostly anonyms).And Ushahidi offers some tools similar to those seen on OpenStreetMap through its integration with social media, email, and SMS-essentially allowing real-time information sharing that can be shared to a map.These two open platforms could be ripe for manipulation, but the bargain here is that the open nature allows for faster updates in times when information is needed quickly (humanitarian crises, for example).For the sake of certain time-sensitive situations, we might be able to excuse some of the noise that is introduced to our intelligence to avoid unnecessarily increasing the transactions cost when speed of information delivery may be the most important feature.Nonetheless, deliberative crowdsourcing requires an investment in time and resources to appropriately produce public value.
Looking at Concept 2 with the Brabham (2013a) best practice perspective, it becomes clear that other factors can drive the cost of crowdsourcing.He finds that it is the clarity of the tasks the crowd is being asked to complete that drive success (Brabham, 2013a).It is clear that the crowd can perform complex analysis (see the analysis of the Box as an example) by breaking down the problem into smaller tasks on their own.This made the process of analysis manageable and clear, rather than just saying "let's figure this out."New information was posted, deliberated, and conclusions drawn piecemeal, rather than from one vague request.A well-framed problem, with clear parameters on expected contributions will improve the experience (Brabham, 2013a) and has the real potential to reduce the transactions cost.The citizen map makers were not given a broad task, like: produce a map of the roads of North Korea.Rather they were given access to tools, like Google MapMaker, that allowed them to complete very discreet taskstracing individual roads at their own pace, for example.
An overly broad question can "generate thoughtful responses and may turn out to be quite a valuable exercise in public participation," but these questions "will also elicit from citizens a wide range of responses, many vague, few feasible, which present city planners with the problem of selecting the best ideas from a mixed bag of apples and oranges" (Brabham, 2013a, p. 21).Brabham (2013a) suggests that "only a specific question will effectively engage citizens in the cocreation of a useful information resource or new actionable idea" (p.21).Once the task, or ask, has been developed and is in the public sphere, managers need to be aware of the reactions that will be generated, which leads to our next concept.
Concept #3: Know when to hit the off switch.
Projects will fail and can create unnecessary risk.There is a difference between destructive trolling and constructive deliberation.The proposed Concept 2 and Concept 3 have overlap in the challenges with online trolls.Online trolls, the flame throwing, mostly anonymous commenters, diminish the value of the free flow of ideas found in the comment sections (Buckels, Trapnell, & Paulhus, 2014;CBC News, 2014;Findlay, 2014;LaBarre, 2013;West & McDonnell, 2013).Trolls spew their hatred, falsehoods, and negativity with the veil of anonymity provided online.Trolls and their comments are creating "a false sense that a topic is more controversial than it really is" and do so because "when it comes to online commenting, throwing bombs gets more attention than being nice, and makes readers double down on their preexisting beliefs" (West & McDonnell, 2013).Buckels et al. (2014) found in their research on Internet trolls that the trolls do their work because they enjoy the harm it causes, or more specifically: "trolling correlated positively with sadism, psychopathy, and Machiavellianism" and is described as an "[i]nternet manifestation of everyday sadism" (p. 1).
Knowing when to hit the off switch on the deliberation is also based in DG Lesson 4, in that the outcomes of deliberation are uncertain.You will not know the outcome at the start of the process, but by understanding and assisting stakeholders throughout, agencies can better understand when to stop the process.
When the cost of engaging and conducting discourse becomes too high, public officials need to disengage from the process.They need to make it clear that the behavior is counter to the public interest.This is, of course, a very difficult line to walk.In the aftermath of the bombing of the Boston Marathon, Reddit users misidentified suspects, and newspapers printed images of the individuals, potentially ruining these innocent people's lives (Sanchez, 2013).This is a clear example of when the discourse went wrong and created a destructive force that could lead to very real dangers for innocent individuals.This incident represents what can happen with no moderation in the discourse.The Reddit users that identified the wrong individuals in the Boston case were likely genuine in their interest to provide for the public good, but since there are little to no consequences to the crowd's failures, negative consequences in this quest are unsurprising.
If gone unchecked, un-monitored, or un-moderated, trolls (or misguided actors) could introduce risk into projects like those of OpenStreetMap or Ushahidi.DG Lesson 3 provides further insight into our proposed concept, by asserting that deliberation requires facilitation skills, management, and a significant investment of public resources.Brabham's (2013a) best practice 5 can provide further depth to this concept as well.The legal issues associated with these deliberations "cluster around issues of preserving free speech and navigating copyright and intellectual property issues" (Brabham, 2013a, p. 24).Managing crowdsourcing and knowing when to exit the process is not about smashing dissent, but rather it goes back to our second concept-this process is not free.
Understanding the types of resistance the public may express sheds light on what an agency might expect.Brabham (2013a) outlines four types of resistance: disruptive, destructive, cracking, and ignoring.Disruptive "crowdslapping" is a form of rational deliberation "that might normally appear in any face-to-face traditional public participation activity" (Brabham, 2013a, p. 24).These deliberations are useful, expected, and a welcome part of our democratic process.
Hitting the off-switch here would not be beneficial to the process.The destructive form of "crowdslapping" takes the form of the previously mentioned Internet trolls and is seen as being much "more aggressive" because it can take over deliberations "with repetitive or offensive content that discourages others from engaging in a productive dialogue" (Brabham, 2013a, p. 24).This is a situation when an agency might think of hitting the off switch and finding mechanisms to throttle back or redirect deliberations or risk efforts falling apart on their own.Cracking takes destructive crowdslapping several steps farther.It is "the term for malicious hacking" and will prevent "other citizens from participating" in the deliberations (Brabham, 2013a, p. 25).Cracking is "akin to…calling in a bomb threat" and should not be tolerated (Brabham, 2013a, p. 25).Brabham (2013a) describes ignoring as "the most powerful form of protest in crowdsourcing" (p.25).This was seen clearly in the ArmsControlWonk case with the April 30 post entitled "Wonk School" (Lewis, 2008e), which the crowd largely ignored.
If the kill-switch is initiated, agencies should do so in "content-neutral ways for the sake of public discourse," which can be accomplished similarly to "public forum in legal terms, which means government can control the time, place, and manner of speech" (Brabham, 2013a, p. 25).
Allowing citizens to govern themselves also alleviates some pressure of public officials to do so.Noveck (2009) and Brabham (2013a) both find that the crowd itself is effective in measuring contributor quality-often this comes as raking or "reputational icons attached to users" (Brabham, 2013a, p. 25).In a fair, open, and agreed-upon process by which people can participate, negative outcomes or their effects can be reduced.This leads to our next concept.
Concept #4: Relationships matter and have to be nurtured.
Relationships matter for public involvement and are important for sustaining despite pitfalls of initiatives.Things will often not go according to plan; thus administrators must develop the necessary soft skills for nurturing meaningful online community relationships that will allow for sustainability in the face of failure.
Failure may occur for a lot of reasons when public agencies are reliant upon the public for inputs.When crowdsourcing works, it can work very well.Unfortunately, it is not always going to work reliably (even without the problem of Internet trolls).In the case of the Box in Syria, there was no formal request to contribute to the analysis.In fact, there was no mention of performing any collection or analysis from the outset.Although a central actor may not be yet present or evident, there are major actors that dominate or guide discussions, such as the ones who initiate a conversation, in this case the blogger ArmsControlWonk Jeffery Lewis.Lewis and the other bloggers on the site come to the discussion with an interest, sophistication, and imbedded knowledge in the subject area-similar to what would be seen in a government agent moderating such a discussion.
Interestingly, an April 30 "assignment" post entitled "Wonk School" (Lewis, 2008e) generated very low response and high resistance.The post explicitly asked for readers to examine a satellite image, and several respondents noted that they would rather not waste time on the assignment.Many were the same people who contributed willingly to the Box analysis.It would appear difficult, if not impossible to order up crowdsourced intelligence without a better understanding of what types of questions and assignments will draw in the crowd.By nurturing relationships over the long-run, assuring a pay-off to the interested parties, showing a responsiveness by the government to the intelligence/information that is gathered, then, and perhaps only then, could a more on-demand style analysis be generated.DG Lesson 6 contends that the deliberation itself is only part of the story and should be placed in a broader context of public involvement.By looking more broadly at public involvement, we are able to see why people engage and why they contribute, so that we can better understand what motives people to contribute in one instance but not in another seemingly identical situation.Brabham (2013a) proposes that governments need to "be honest, transparent, and responsive" and that participants "should feel as though their voices will be heard and their ideas handled with care" (p.26).Thus, when local governments around the country are implementing 311 systems (nonemergency versions of 911), online reporting portals, and smartphone applications that all allow for the easy reporting of nonemergency quality of life issues, they need to be honest-not just about how long it will take to repair something but also that they are genuinely invested in the process.This may come in the form of a filled pothole or in the form of an acknowledgment and thanks to participants (Brabham, 2013a).Cities using the SeeClickFix platform for online and mobile non-emergency reporting acknowledge top users, giving ranks/titles ("Jane Jacobs" or "Digital Superhero," for example) to the users, and also gives Civic Points to these participants based on how many requests they have submitted (SeeClickFix, 2015).
When cities are responsive in how they deal with submitted requests via these platforms, citizens have been shown to be more satisfied with their local governments (Clark & Shurik, 2016).Conversely, when local governments do not seek to nurture these relationships and are generally unresponsive to citizen requests, citizens will disengage from the deliberative process (Clark & Shurik, 2016).And it is this inaction and unresponsive relationships that leads to the fifth and final concept.
Concept #5: Crowdsourcing does not operate in a vacuum.
The important concept of a mini-public as it relates to crowdsourcing is that the mini-publics act as a microcosm to the public at large, and facilitate a more "manageable" form of discourse that can include diverse opinions.However, the critical issue for a mini-public is the bringing together of differing discursive elements to enhance inquiry.The fact that mini-publics do not operate in a vacuum has less to do with "how many" people are involved but more about the overlapping discursive elements that have the potential to enrich or enhance the depth of understanding and learning.The deliberative governance lessons provide backing to our proposed concept.DG Lesson 7 asserts that deliberative governance will not develop in isolation; therefore, practitioners and scholars should take steps to build informal networks and learning communities in order to share experiences.Thus, by fostering the developing of networks, the potential of the mini-public is enhanced by bringing in the overlapping discursive elements (the diversity of the crowd) to improve the quality of the intelligence that is generated.Both of our cases bring together these diverse elements of the population to engage in discourse.The individuals participating in the Box case were not all nuclear nonproliferation experts or nuclear physicists.Rather, they were a motley collection of individuals that came together with their varying pieces of expertise that allowed the group (mini-public) to create the intelligence to die for.The mapping of North Korean roadways was not done by geographers or military intelligence officers.Rather, they drew from the intelligence and labor of people from around the world who simply had a desire to contribute.In both of these instances, intellectually diversity mini-publics were created and enriched how all of these different individuals might think about these particular problems.
Building and sustaining these mini-publics means that governments need to work to promote their efforts (Brabham, 2013a).When the city of Cleveland launched its 311 service, the city was in a soft-launch mode and did not tell anyone about it (D.have not yet actively used the new deliberative tools to engage the public, nor told the public they exist (Meyer, 2013).Without actively engaging and promoting the use of these tools, deliberations never begin, mini-publics never form, and no public or private value is created.

Conclusion
The aims of ArmsControlWonk and crowdmapping are not to replace the intelligence community.Nor is the aim of these projects just to blow off steam.For the Box case, the aim was likely a collective effort to solve a puzzle, and it worked.Similarly, when we look at the examples of crowdmapping, the goal is not to supplant traditional mapping endeavors in the private sector or in government.But what these crowdmapping projects do appear to promise, is an augmentation to these traditional sources by offering a product to a niche audience or client, frequently in real-time (or close to it compared with traditional paper maps).They do not promise the expertise or precision of the NGA or commercial entities, but rather, they offer a new layer of (potentially) rich intelligence in rapidly changing environments.
And while none of these cases were directed by a government entity, they did create products of value to government and the public at large.Governments at all levels have successfully utilized crowdsourcing projects to gather information or intelligence.The formal structures of modern governments, particularly the rules and regulations that guide their everyday operations, limit the ability to use internal resources to follow up on theories or ideas.Crowdsourcing activities like the Box were possible because a large group of intelligent people had some very good ideas and very few barriers to sharing them.This case demonstrates the potential to generate knowledge in ways not possible within organizations that are more strictly rule-based and mission-driven.There is little reason to believe that this sort of intelligence gathering and analysis is only possible on the ArmsControlWonk blog-thus replicability is likely.Finding mini-publics and sustaining them will not be free, but the concepts we presented in this article will help to guide public managers in thinking about how they might work for their organizations.
The North Korean map's creation was more structured in its development but is still driven by people's innate curiosity with a place we know little about.If a similar task were given to map a place that was not mysterious or newsworthy, we likely would have seen a different result.As with the case of the Box, the North Korean mapping project was successful in part because it had some ground rules to participation and some direction (by Google), but the participants were able to contribute as much or as little as they wanted, solely based on their intellectual curiosity.Both OpenStreetMaps and Ushahidi are more open platforms for crowdsourcing than the example of the North Korean map-though all three rely upon the crowd for their intelligence.
With any platform that is open to information in the way these platforms are, there is a need to balance the necessity of accuracy, with the speed at which information must be transmitted.When responding to humanitarian crises, where delays of minutes or hours may be vital to a person's survival, removing barriers to communication and facilitating information transfer will be the most important characteristic.In those instances the moderation, which can improve the quality of information, would need to be eased to ensure a rapid response when necessary.When, as was the case for the Box or the North Korean map, information flows are not contributing to life-or-death decision-making and a high level of accuracy is the key, then tighter control and moderation become critical for success.
A predicament inherent in the public sector use of crowdsourcing is the decision of what crowdsourcing should and should not be doing for government.Do we use it primarily as an information gathering tool, an analytic tool, public participation tool, or for something else altogether?Managers in the public sector have utilized the crowd to provide intelligence in some form or another for many years, but the forms that it is taking today were never possible because the transactions costs associated with coordinating such a large and diverse group were so high.Thus, the question becomes, how seriously should public sector managers in general and those in the intelligence community, more specifically, take this information?This question remains unanswered and will be a struggle as agencies find beneficial ways to incorporate it.Of course, when it comes to some glaringly obvious issues-the intelligence community should look at publicly available information as well as classified data-there is lingering suspicion that every hack with a John LeCarré novel under one arm and a Time magazine under the other thinks he will bring down al Qaeda or ISIS.Fortunately, the wisdom of the crowds is remarkably efficient, at weeding out such cavalier exceptions in favor of thoughtful, reasoned analysis of the available data.The question of how much analysis public managers can expect from the crowd will remain an issue, and certainly point to the need of continued professional staffing at all levels of government to make quality, informed decisions.
What many prior crowdsourcing efforts, such as the National Open Source Enterprise (NOSE) and a director for open source intelligence within the US government, missed was the novelty and velocity of mass collaboration.As NOSE and others try to build the structure and control familiar to established, cohesive organizations, open-source intelligence is moving far fasterand without waiting for permission.
Evidence revealed in the Edward Snowden case has demonstrated that governments (United States and others) are forging ahead with efforts to mine huge data sources (some authorized and some not) to find clues to the next terrorist attack.With computing power increasing exponentially and the piles of data growing even faster, it is clear that the need for thoughtful, reasoned collection and analysis of intelligence will continue to increase.The crowd can play a complementary role, particularly in cases where human intelligence continues to trump sophisticated algorithms and search protocols.

Table 1 .
Deliberative Governance Lessons and Crowdsourcing Best Practices

Table 2 .
Administrative Concepts for Crowdsourced Intelligence/Information Gathering Crowdmap service is a crowdsource mapping tool that provides a streamlined way in which maps can be created from reports sent in from various sources (primarily phones and computers), creating a near real-time picture of a situation.Crowdmap opens this service up to the public without the need of a server.Crowdmap allows anyone to build any map free of charge, creating various streams of information that are open to the public at large.
Crowdmap is incredibly simple and malleable.Any individual can create a subdomain to host his or her specific map.Because this site is hosted on Ushahidi servers, it does not require installation of any software, nor does it necessitate other components such as server space or domain and hosting services.Once the site is set up, crowd can input the data through various technological platforms; this data are aggregated in real-time or near real-time on the site.This data can be interactive, allowing users to upload pictures and videos.The Crowdmap and Ushadidi platforms can be linked to social media as well, including Twitter and Facebook, which allows it to be readily accessible in the contemporary internet era.Additionally, all input is logged and tracked over time, and Crowdmap includes several tools to analyze real-time data December 4, 2015; B. Sylvester, personal communication, December 4, 2015; S. Weber, personal  communication, October 21, 2015; J. Takacs, personal communication, October 21, 2015), what has separated Cleveland from other cities is that it stayed in soft-launch for more than five years, Brown, personal communication, May  22, 2014).While this practice is not uncommon to work out the kinks of new deliberative formats(A.Soverign, personal communication, December 9, 2015; J. Valle, personal  communication, November 25, 2015; J. Eaton, personal communication, November 25, 2015; R.  Dietz, personal communication, November 2, 2015; C. McEwen, personal communication,