A Comparative Analysis of Two Streams of Implementation Research

s. Yet, this concern for target groups is one of the primary intents of the multi-level framework: to link ultimate outcomes to the different levels within the governance system (Lynn et al., 2001; Heinrich, Hill, & Lynn, 2004; Heinrich & Lynn, 2001). While this lack of focus could be due in part to the limited list of target populations, the content analysis of full articles revealed a similar trend. The risk, of course, is that management or systems changes not aligned with the ultimate change desired in the target population may be misdirected (Moynihan & Soss, 2014; Sandfort & Moulton, 2015). Further, despite the potential of multi-level frameworks in theory, a very small proportion of empirical studies are conducting implementation research that crosses multiple levels of analysis (13 percent). This result provides empirical backing to the concern raised by Hupe (2014) that mainstream implementation studies typically do not focus on the multiple layers of the implementation system. Interestingly though, this analysis suggests that implementation scientists are incorporating multiple levels of analysis at relatively higher rates than public affairs investigators (19 percent in the implementation science core journals versus 11 percent in the public affairs journals). In addition to observing differences between streams, our analysis reveals differences based on the level of analysis of the findings. Program-level findings are more likely to result from quantitative analyses, whereas organizational, frontline, and policy findings are more likely to employ qualitative methods. Quantitative analysis may be more feasible, and desirable, for program evaluations, while an understanding of organizational and front-line factors may require a qualitative approach. Mixed methods research designs are likely needed to inform more generalizable findings across policy levels. Of course, there are some limitations of this study. The first limitation noted earlier is the limited ability to systematically explore full articles except with the purposively-selected subsample of 100 articles, though this review of the full articles revealed substantial concurrence between the abstracts and the full articles. Second is the use of relatively narrow search criteria in building the article database: only articles using the exact terms “policy implementation” and “program implementation” in the abstract, key words, or title were used, and our findings could reflect a bias stemming from the use of those terms. However, the purpose of this analysis is not to identify all possible relevant implementation articles, but rather to compare two streams of implementation research and identify general research trends. Additionally, limiting the scope

The purpose of this manuscript is to systematically compare implementation research taking place within the new scholarly stream of implementation science to the body of implementation literature within public affairs.We build from prior comparative studies of implementation scholarship that trace trends in the intellectual scope, focus, and/or methods of published implementation literature (Saetren, 2005;2014;O'Toole, 1986O'Toole, , 2000)), and assess the extent to which research taking place in both streams is situated across multiple levels of the implementation system.The study of implementation within public affairs has theoretically embraced a multi-level framework of governance, where the outcomes of public policies and programs are best understood as resulting from multiple levels within a larger system (Berman, 1981;Hill & Hupe, 2014;Lynn, Heinrich, & Hill, 2001;Robichau & Lynn, 2009;Van Meter & Van Horn, 1975;Winter, 2003).Similar multi-level frameworks have been proposed for the new study of implementation science, but there are worries about whether or not the predominant theories or methods support this more holistic approach (Nilsen et al., 2013).Little is known empirically about the extent to which research studies on implementation are situated across multiple levels of the governance system.Our study addresses this gap by examining the degree to which current policy and program implementation literature is situated at multiple levels of these systems and identifying the relationships between the level of analysis with the content, focus, and methods of published research.
To achieve these aims, we conduct a systematic review of research published in scholarly journals indexed by the Social Science Citation Index (SSI) from 2004 to 2013.From the pool of more than 15 million articles, 1,507 articles are extracted that include the terms "policy implementation" or "program implementation" in the title, abstract, or key words.We code the articles based on whether they are published in a public affairs journal, a core implementation science journal, or a journal outside of those research streams.We then conduct a content analysis of the abstracts to assess patterns in the policy areas, methods, and levels of analysis used in implementation research.To help validate our abstract coding and to contextualize differences in findings within and across multiple levels, we conduct a more in-depth review of a randomly selected subsample of 100 full articles.
The findings demonstrate interesting differences and similarities between the two streams of research.The amount of implementation research conducted in both streams appears to be accelerating over the decade under study, which is reflective of more general trends in implementation research outside these streams.In implementation science, the content areas examined are predominantly health and education, while there is more heterogeneity of topic areas in public affairs.Interestingly, neither stream consistently mentions particular policies as their analytical focus and, while there is diversity in research methods, the use of mixed methods, which is arguably the most important way of studying the complex dynamics of policy and program implementation, is not common in either stream.Further, over a third of the articles published in the last decade contribute findings relevant only to specific programs, the least generalizable level of analysis for those interested in understanding implementation occurring in a broader system.In contrast, only 15 percent of articles published in the last decade produce findings that cross multiple levels of the implementation system.Multi-level analyses are more common in implementation science articles than in public affairs articles, however they still comprise less than one-fifth of the total.These findings can help inform avenues for future research across both streams.

A "New" Study of Implementation Science
The history of implementation scholarship within the field of public affairs has been welldocumented in prior literature (Hill & Hupe, 2014;Saetren, 2005;2014;Sandfort & Moulton, 2015;Winter, 2003).One of the recurring themes of these reviews is the dissipation of research on implementation within the core discipline of public affairs, beginning in the 1990s with an increase in implementation literature published outside of the field (Meier, 1999;Saetren, 2005).In a review, Saetren documented 3,523 research articles published in the field over the 70-year period he studies and finds 70 percent of the research was published between 1985 and 2003.Yet much of this growth was happening outside of venues where public affairs research was published.Nilsen et al. (2013) provides a frame for understanding this development.In areas such as medicine, community psychology, early childhood development, and education, researchers have made considerable inroads in developing models and methods for studying program implementation.These investigations are mainly interested in the diffusion and replication of effective research-based interventions or programs and thus have a more modest scope than public affairs studies.A new section of the American Psychological Association, the new journal Implementation Science, and a biennial conference sponsored by the Global Implementation Initiative bespeak the growth of this research stream.
Like the study of policy implementation in the 1970s and 1980s, many models and theories have recently been developed to achieve this aim in the field of implementation science research (Tabak, Khoong, Chambers, & Brownson, 2012).Recent reviews are seeking to develop integrative frameworks or conceptual models (Durlak & DuPre, 2008;Fixsen et al., 2005;Greenhalgh et al., 2004;Meyers et al., 2012).Yet, in these efforts, it is recognized that scant attention is paid to the organizational or policy environment.In a recent review of implementation science research, only 13 percent of the models incorporated policy activities even though they are widely understood as important (Tabak et al., 2012).In the widely used 'Consolidated Framework for Implementation Research,' organizational and policy factors are merely referred to as the "outer setting" (Damschroder, Aron, Keith, Kirsch, Alexander, & Lowery, 2009).
In spite of this fairly limited conception, a number of federal agencies, including the U.S. Departments of Education, Health and Human Services, and Veterans Affairs, are investing in the implementation science approach.Similarly, governments and foundations in Western Europe are developing institutes of implementation science or contracting with research firms providing this type of implementation analysis.For policy makers interested in affecting outcomes, knowledge relevant to diffusing evidence-based interventions is essential.However, many of these policymakers seem unaware of prior literature on implementation within the field of public affairs.While the implementation science approach has a different starting point, health sciences rather than social sciences, there are similarities in the frameworks being employed to make sense of implementation dynamics.Perhaps most fundamentally, both streams of research acknowledge the complexity of implementation that occurs in multi-level, multi-actor systems.

Multi-Level Frameworks For Implementation Analysis
One of the predominant themes in public affairs research today is the multi-actor, multi-level nature of the system in which policy and management takes place.For example, the study of governance, which has been defined as the study of how government, nonprofit, and private actors systematically shape "policy-relevant" outcomes (O'Toole, 2000), implies a more comprehensive approach which includes both non-governmental actors and multiple factors that interact in a decentralized and oftentimes networked structure (Agranoff & McGuire, 2003;Feldman & Khademian, 2002;Frederickson, 2005;Lynn et al., 2001;Provan & Milward, 2001).This work, such as the multi-level logic of governance framework put forth by Lynn, Heinrich, and Hill (2000), provides a way to think about how policy decisions made in the federal government resonate down to lower levels of state agencies, service providers, and eventually target populations.
While governance includes much more than implementation, recent implementation scholars stress that the concepts of multi-level and multi-actor systems of action are foundational to understanding implementation processes and results (Goggin et al., 1990;Hill & Hupe, 2014;Robichau and Lynn, 2009;May & Winter, 2009;Shea, 2011).Though most implementation science articles do not address specific policies or larger governance issues, the research often introduces a multi-level approach by positioning frontline service delivery within organizational contexts (Damschroder et al., 2009;Durlak & DuPre, 2008;Fixsen et al., 2005;Greenhalgh et al., 2004;Tabak et al., 2012).
Despite the variety in research approaches and differences in terminology, implementation findings from prior scholarship can generally be parsed into three levels: frontlines, organization, and the policy field.First, following the lead of Lipsky (1980), some studies document the significance of implementation at the frontlines, where the policy system interacts with the target population, such as children, tax payers, or business owners.Many factors may be significant, such as a target group's composition and attitudes, staff background and experiences, operational tasks, or institutional values (Garrow & Grusky, 2013;Lipsky, 1980;May & Winter, 2009;Meyers & Nielsen, 2012;Sandfort, 2010;Watkins-Hayes, 2009).
Second, frontline conditions are directly shaped by other factors at the organizational level.Service organizations' resources, structures, cultures, and competing programmatic responsibilities often determine the most prudent way the agency responds to implementation pressures (Lin, 2000;Sandfort, Selden, & Sowa, 2008;Spillane, 1998).Empirical evidence is clear about the significance of organizational structure and climate in supporting the dissemination of evidence-based interventions (Aarons, Glisson, Green, Hoagwood, Kelleher, Landsverk, & Research Network on Youth Mental Health, 2012;Fixsen, 2005;Glisson, Hemmelgarn, Green, & Williams, 2013).
Finally, at the policy field level, other macro-level factors come into focus (Milward & Wamsley, 1985;Sandfort, 2010;Stone & Sandfort, 2009;Weible, Sabatier, & McQueen, 2009).1 Policy fields are bounded networks among organizations carrying out a substantive policy and program area, such as homeland security or healthcare, and include organizations responsible for delivering services as well as intermediary organizations providing support to the network (Shea, 2011).These networks are, in large part, structured by the rules developed at the agency and legislative levels (Hall & O'Toole, 2004) and, in practice, reflect the core elements of policy environments in a geographic area.
While the dynamics within each level of an implementation system are unique, the levels themselves are nested within each other, much like Russian dolls.In that the processes of one level are influenced by the terminology, decisions, and structures adopted by the other (Fligstein & McAdam, 2012;Hill & Hupe, 2014).While it is easy to conceptualize this theoretically, the degree to which this orientation is reflected in empirical investigations is much less clear.By differentiating among these levels and probing in more detail the unique aspects of implementation operating at each level, this study documents what published studies reflect about multi-level implementation systems and highlights new areas for research in this vein.

Data and Methods
For this analysis, a systematic review of literature (Boruch & Petrosino, 2004) is conducted, including implementation research published in scholarly journals in the ten-year period spanning 2004 to 2013.The sample is drawn from the more than 8,500 journals listed in the Expanded Social Science Citation Index.From a potential population of over 15 million articles published over these years, any articles which include the terms "policy implementation" or "program implementation" in the title, abstract, or key words are retained.The sample is further refined to exclude articles that employ the key terms but do not fit within either research stream, such as research on the implementation of a computer program, resulting in a final sample of 1,507 articles. 2o undertake this systematic review both bibliometric analysis and content analysis are employed.First, a bibliometric analysis is conducted of the journals in which the sample articles are published, identifying those that are published in the core journals within the two streams.It is expected that those articles employing the term "policy implementation" will be represented more strongly in public affairs journals, while those employing the term "program implementation" will be represented more strongly in core implementation science journals.Public affairs journals are coded using the Web of Knowledge (ISI), including the over 190 journals indexed as "public administration" or "political science".ISI does not have an implementation science category however, ten core implementation journals are identified that are most frequently cited as significant outlets for implementation science in literature reviews (Durlak & DuPre, 2008; Fixsen et al., 2005; Meyers et al., 2012). 3  Second, the two streams of implementation literature are compared through a content analysis of abstracts. 4Using Nvivo software, the content area, focus, and research methods of the articles are coded.The level of analysis within which the research is situated is also examined.While abstracts are often imperfect proxies for the full content of articles, the assumption is that an abstract will indicate the central focus of an article, including the central question(s), unit(s) of analysis and research methods.While we rely on abstracts for the primary analysis, we validate our abstract coding through a detailed review of a random subsample of 100 full articles.The reliability of the coding scheme was tested with an independent coder who coded the level of findings (e.g.frontlines, organizational, etc.) for this sub-sample of full articles; the inter-rater reliability was 96 percent.Generally speaking, the overall patterns found in the abstract coding were also found in the full article review.Though the validity of the abstract-coding approach is reinforced through this subsample analysis, it is plausible that some studies may not reference all relevant elements of their study in the abstract.This is a limitation of this approach.
To understand the specific types of policies and program examined by implementation scholars, abstracts are first coded based on content areas.Content areas are not mutually exclusive and include health, education, environment, social welfare, crime, agriculture, city and regional planning, energy, transportation, science and technology, food, international development, monetary policy, international relations, business-specific implementation, general implementation, and miscellaneous.While most of these areas are relatively clear, articles coded as "general implementation" are those that did not focus on a specific content area, but rather the topic of implementation itself.For example, a study focusing on how American federalism inhibits efficient policy implementation would be coded as "general implementation." In addition to content areas, abstracts are also coded for particular foci of the research that may vary across the two implementation streams.Specifically, we considered the extent to which articles reference a specific policy or piece of legislation and the extent to which articles reference specific target populations.As noted above, prior implementation scholarship viewed policies themselves as the central feature of implementation research and coding for policy references allows us to examine the degree to which policies remain a focus in implementation research.For specific policy references, abstracts are coded based on whether they make explicit reference to a particular public policy as being central to their analysis, and are also coded based on the scope of the policy in question (e.g.international, federal/national, or state/local).
The impact that target population conditions have on the policy system has also been of interest for policy researchers (e.g.Schneider & Ingram, 1993;Watkins-Hayes, 2009), and coding for target population references provides insight on the degree to which key populations are a focus in implementation research specifically.For target populations, abstracts are coded based on whether they focus on the following populations: children, the disabled, the elderly, the medically vulnerable, parents and families, and racial/ethnic groups.These categories were chosen as they roughly track with populations typically understood to be beneficiaries of public interventions.Not all articles in the sample are coded to a specific target population and some are coded in more than one of these categories.
The research methods in the articles are first coded as "conceptual" or "empirical."Conceptual studies are defined as those without any data or a specific case under study and empirical studies as those with a source of data or specific instance of policy or program implementation.Empirical studies are further subdivided as "quantitative," "qualitative," or "mixed."Qualitative studies reference the collection of generally unstructured data, including small-n surveys, interviews, and content analyses, where these data sources are analyzed through qualitative methods (e.g., comparative case studies, narrative analysis).Quantitative research refers to large-n analyses of structured data, traditionally positivist research designs including experiments and quasi-experiments, or statistical analyses (including meta-analyses summarizing quantitative outcomes).Mixed research strategies are those that employ both qualitative and quantitative methods.Abstracts that do not include any explicit reference to methods are coded as "ambiguous."Finally, articles are identified that do not employ any rigorous empirical or conceptual frame in their analysis and are generally only focused on describing a single case of implementation; these articles are coded as "descriptive case studies."Further analysis is conducted on a subsample of articles with an empirical orientation (n=1,128), which excludes these descriptive case studies.Specifying the subsample in this way allows for an enhanced focus on research employing an explicit research design, one of the hallmarks of the third generation implementation research paradigm in public affairs (Goggin et al, 1990).This specification also allows the research to build from earlier work by Saetren (2014), which explored how the use of research designs in implementation studies evolved over time.Within this subsample, abstracts are further coded based on the levels of analysis described in the preceding section, here designated as "frontline," "organizational," and "policy field."5Those studies contributing findings only at the program level are also identified."Program-specific" findings include findings that are directly related to the evaluation of a specific program or policy in terms of its outcomes or impacts, but not other levels of analysis.

Results
The comparative review begins with a bibliometric analysis of the 1,507 articles citing "policy implementation" or "program implementation" in their title, keywords, or abstracts.Overall, the analysis finds that the majority of recent policy or program implementation articles are being published in journals outside of the core public affairs journals (see table 1).While this approach to classifying journals as "core" differs from Saetren (2005), the finding that the majority of published implementation research takes place outside of public affairs venues is similar.This analysis further finds that five percent of implementation articles in the sample (80 articles in total) are being published in one of the top ten "core" implementation science journals, while eleven percent (169 articles) are being published in core public affairs journals.This result is of particular interest because, while public affairs journals publish twice as many implementation articles as the core implementation science journals, there are over 190 public affairs journals and only ten core implementation science journals.The implementation science core journals are, proportionally speaking, contributing to the overall implementation literature at a much higher rate than public affairs journals.Of the 1,507 total implementation articles, 905 articles (60 percent) use the term "policy implementation," while 610 articles (40 percent) employ the term "program implementation" (and a small number of articles employ both terms)."Policy implementation" is referred to in almost every public affairs article (96 percent) and only about a fifth of implementation science articles, while "program implementation" is largely the purview of implementation science articles, 81 percent of which focus on program implementation.By contrast, journals outside these streams are much more evenly focused on both policy and program implementation (58 percent of other journals referenced "policy implementation" while 43 percent referenced "program implementation).This finding is not particularly surprising but adds evidence to the claim that there are distinct streams of scholarship focused on distinct topics within the banner of "implementation." It is notable that implementation research appears to be expanding markedly over the relatively short timeframe examined here.Saetren (2005) found that implementation research expanded over the period from 1970 to 2000; for our sample, spanning 2004-2013, we find that twothirds of the articles in the sample were published in the last half of the timeframe, and a third were published in the first half.This indicates that interest in implementation research is continuing to grow.To illustrate this, figure 1 traces the percent of sampled implementation articles published in each year of the analysis.Though not shown in the graph, articles from both the public affairs and implementation science research streams follow similar growth trajectories to the full sample of articles.
Next, the content areas of policy and program implementation research are compared.Table 2 compares research published in public affairs journals (per the ISI classification), core implementation science journals (see footnote 3), and any other journals outside those classifications.The difference in content areas between the implementation science articles and the public affairs articles is stark.Almost all implementation science articles focus on health, education, or social welfare issues, with very little content outside of those topics.While it would be tempting to ascribe this result to the makeup of the journals in the implementation science core, the fact that this focus is largely replicated for journals outside of this category reinforces the idea that most of the work in implementation science is happening in a few discrete areas.
The public affairs articles are by contrast relatively more diverse, with around 15 percent of articles falling under environmental, health, and social welfare content areas, and over a fifth of articles falling into the "other" content area category.8Additionally, 30 percent of public affairs articles focus on implementation issues generally, rather than implementation within the context of any specific content area.
Next, the research methods employed within the two different research streams are compared.These results are also in table 2. The most prevalent research methods are quantitative.As expected, preference for quantitative methods is stronger within implementation science articles than public affairs implementation articles.Compared to implementation science articles, public affairs implementation articles are over twice as likely to include conceptual articles and almost three times as likely to employ descriptive case studies.The use of qualitative methods is roughly similar between the two streams, and implementation science articles are much more likely to use mixed methods than public affairs implementation articles.
Less reliance on quantitative methods and heavier use of conceptual articles and case studies is not necessarily endemic to public affairs research, but rather the study of implementation within that tradition (Goggin et al., 1990;Saetren, 2005Saetren, , 2014)); prior implementation scholars have noted that the context-rich nature of implementation tends to drive researchers towards case studies (Goggin et al., 1990;Saetren, 2005).However, it is noteworthy that implementation science articles do not appear to have this same tendency, perhaps given the heavy reliance on randomized control trial designs and quantitative methods in the health sciences.Aside from the general content area and research strategy, the policy focus of implementation research is also compared.Abstracts are coded based on whether or not they include mention of a specific public policy including legislation, executive orders, or agency mandates.The results are in table 3.Those abstracts mentioning specific policies are further identified by the level of government at which the policy operates.Of the articles in the sample, only a fifth of the abstracts mention a particular policy, and over half of these abstracts focus on policies enacted at the federal or national level.While it is not surprising that almost 90 percent of articles from the implementation science core journals lack focus on specific public policies, it is striking that only one-fifth of implementation articles in public affairs journals are focused on specific policies.This may be driven by the focus on general implementation theory-building or by the move toward more interest in implementation dynamics across specific policy initiatives.In terms of the level of the policy in question, articles from public affairs journals tend to place more emphasis on the federal or national level, while those from the implementation science journals tend to focus more on the state and local levels.
Because implementation is often about making a change in behaviors or conditions in specific populations, the extent to which articles reference particular target groups is also considered.While the possible target groups are diverse and numerous, this analysis limits its attention is to children, the disabled, the elderly, the medically vulnerable, parents and families, the poor, and racial or ethnic minority groups.The results are also found in table 3.Only 34 percent of the full sample of articles references one or more of these groups, with the most common being the medically vulnerable, followed by children and parents/family.The target population focus differs substantially between the research streams: Almost two-thirds of articles emerging from the implementation science core journals have an explicit target population focus, while almost none of those from the public affairs journals are oriented towards a specific population.
In addition to a comparative analysis of the policy and program implementation research streams, a second contribution of this study is to consider where current research is situated within a multi-level framework.As this research is specifically interested in the level of analysis employed in rigorous implementation research, the sample is limited to empirical articles that employ conventional social science methodology (n=1,128).
The results can be seen in table 4. Interestingly, the findings demonstrate that implementation science articles are about twice as likely as implementation articles from public affairs journals to focus on the frontlines or the organizational level and, unsurprisingly, are about four times as likely to focus on straightforward program evaluations.Public affairs implementation articles, by contrast, are almost twice as likely to focus on field level issues.Given that scholars in both streams have emphasized that implementation occurs at multiple levels, one might expect to see this represented empirically with studies taking place at multiple levels of analysis.Yet only 13 percent of the empirical studies present findings relevant to more than one level of the implementation system, with implementation science articles being almost twice as likely to focus on multiple levels as public affairs articles.
To further unpack the nature of research taking place at different levels of analysis, table 5 breaks down the content areas and research methods observed within each level. 9In terms of how content areas differ by the level of analysis explored, results are restricted to only focus on the four most prominent areas: education, environment, health, and social welfare.Within each level of analysis health studies are the most highly represented, which is a function of the general predominance of healthcare-related studies in the total sample, and are most concentrated in the program-specific level and least concentrated in the policy field level.Education, the second most prominent research area, has a comparatively high focus at the frontline and organizational levels.Environmental studies are less concentrated at the program, frontline, and organizational levels, but make up almost a fifth of the findings at the policy field level, showing that studies in this area generally focus on the broadest level of analysis.Social welfare studies, on the other hand, do not really have a strong emphasis on any one level of analysis.In terms of studies that cross multiple levels of analysis, the education, environmental, and health research areas all contribute to this category roughly in proportion to their prevalence in the sample as a whole, while social welfare articles contribute a disproportionately higher share to multi-level studies.
Among empirical articles, research strategies differ substantially depending on the level of analysis, as shown at the bottom of table 5.At the program-specific level, over three-fourths of studies are quantitative, with an additional eight percent being mixed quantitative/qualitative.There is much more emphasis on qualitative studies at the frontline, organizational, and policy field levels, and purely qualitative studies comprise almost half of studies operating at multiple levels of analysis, compared to just a fifth for quantitative studies.
With regard to the focus of the studies in this sample, those with an explicit policy mention in their abstracts are, unsurprisingly, the least prevalent at the program level and the most prevalent at the policy field level.At the program level there is relatively more emphasis on state and local policy implementation than there is at any other level, while at the frontline, organizational, and policy field levels the majority of policy-focused studies concentrate on the federal or national level.
Over half (53 percent) of studies at the program-specific level have a target population focus, a proportion that declines as the level of analysis broadens.Additionally, 40 percent of studies that focus on multiple levels of analysis have a target population focus.With respect to the specific populations, studies that focus on children comprise a larger proportion of the studies at the program, frontline, and organizational level (likely due to the prevalence of educationfocused studies at these levels), while studies about impoverished populations comprise a relatively high percentage of studies at the policy field level.Studies focusing on specific races or ethnicities are, by contrast, disproportionately focused on the frontlines.

Supplemental Analysis
One of the central limitations of this analysis is the reliance on abstracts as its primary data source.As a supplemental analysis, we conduct an in-depth analysis of a random sample of 100 full articles.We purposively sample articles across analytical levels, with 20 studies selected within each of the following categories: frontline, organizational, policy field, studies that report findings relevant to multiple levels, and studies that contribute findings that are only programspecific.This supplemental analysis is aimed at validating the abstract coding and adding context to the differences in the topics and types of findings reported at each level of analysis.
In terms of validation, the general characteristics of the articles in the subsample are similar to the characteristics of the articles in the full sample.For example, the majority of articles focus in the health (48 percent) or education (14 percent) content areas, particularly those regarding the implementation of particular program interventions.Within specific content areas, our in-depth analysis reveals that implementation studies focus on diverse topics.Topics such as Chinese monetary policy, charter schools, hospital quality improvement, sustainable energy development in Europe, and cardiac care are all substantive topics found in this sub-sample.In line with the findings reported in table 2, the in-depth coding reveals a similar pattern of diverse research strategies; these research strategies do not clearly sort by the level of analysis.
A second purpose of the full article analysis is to add context to differences in findings by level of analysis.Frontline studies typically provide more contextual information than studies focused on other levels.In reporting their findings, authors rely upon descriptive accounts and refer to "ground realities" and "ineffective policies" that are disconnected.There is more attention to both the particular occupational groups found at the frontlines, i.e. doctors, teachers, social workers, etc., and the characteristics of the target groups.As a result, the significance of relationships, perceptions, and negotiation between these actors, as well as their competing worldviews or perspectives, is often stressed in research findings.Rather than being focused on predicting program replications, these studies are more concerned with uncovering the mechanisms whereby staff and target groups shape these processes.
care agencies, explore how specific management characteristics, structures, or processes influence desired behaviors in either frontline staff or patients.Yet more broadly, these articles frequently noted resources, culture, leadership, and coordinating structure for their roles in shaping both the process and results of implementation.
The policy field studies were the most heterogeneous.Some focus at the national or transnational level, others at the state or local level.Some attempt to understand network or governance arrangements, while others investigate ideology's role in shaping coalitions.Throughout most, the roles of both public and private institutions, as well as their relationships and power, are recognized as significant in implementation.Yet, there is clearly no unifying conception of the significant questions or research approaches to be pursued in this type of analysis.
We also reviewed studies that only contribute findings relevant to a particular program.Our motivation to analyze these studies in relation to larger system roles comes from our interest in generating an empirical foundation to bolster the conception of multi-level implementation analysis.The content analysis of abstracts revealed some scholars are implicitly pursuing analysis that crosses levels.As a result, we included another stratum to reflect this in our sampling frame of in-depth articles.For example, a study of relationships between national HIV/AIDS organizations in China that affected frontline health educators' practice simultaneously explores the policy-field networks' influence on those same frontline practices.Interestingly, the articles in this category resembled those focused on the policy-field and organizational levels.Like the policy-field studies, the substantive significance of relationships and networks among diverse stakeholders is stressed as a theme in this subsample of articles.
Like the organizational-level studies, the essential role of capacity building and resources in guaranteeing success is emphasized.However, the analytical focus of these studies was inconsistent, making it difficult to draw any additional conclusions from our investigations of implementation studies reporting findings that cross levels.

Discussion
There are notable differences between the new stream of research in implementation science and implementation research within public affairs.In terms of focus area, implementation science is heavily dominated by health care, comprising over two-thirds of studies reviewed.By contrast, the public affairs implementation literature is more likely to engage regulatory topics like environmental policy or issues related to general implementation theory building, and generally deal with a more heterogeneous array of policy topic areas.This differentiation in focus is likely driven in part by the origins of implementation science in concerns about replication of randomized control trial results in health care and fidelity to the treatment in other settings.In contrast, public affairs scholars bring broader content area interests and more diverse questions about implementation processes and systems to the field.
Methods also vary between the two streams, with quantitative and experimental methods more prevalent in the implementation science research, and case studies and conceptual literature more common in public affairs journals.While this may change in the near future as public affairs pushes for increased use of quantitative and experimental methods (Margetts, 2011;Perry, 2012), the reality is that the narrower concerns of program evaluation and replication can be more tightly controlled than the concerns of the broad field-level implementation studies like those often found in public affairs research.
Perhaps one of the most vexing challenges is that across both of these streams of investigation there appears to be minimal focus on specific target populations or the ultimate results of the policy or program being implemented.Though the research emerging from the core implementation science journals has a relatively strong orientation towards target populations (around two-thirds have a target population focus), only a third of total articles and fewer than ten percent of public affairs implementation articles have target populations identified in their abstracts.Yet, this concern for target groups is one of the primary intents of the multi-level framework: to link ultimate outcomes to the different levels within the governance system (Lynn et al., 2001;Heinrich, Hill, & Lynn, 2004;Heinrich & Lynn, 2001).While this lack of focus could be due in part to the limited list of target populations, the content analysis of full articles revealed a similar trend.The risk, of course, is that management or systems changes not aligned with the ultimate change desired in the target population may be misdirected (Moynihan & Soss, 2014;Sandfort & Moulton, 2015).
Further, despite the potential of multi-level frameworks in theory, a very small proportion of empirical studies are conducting implementation research that crosses multiple levels of analysis (13 percent).This result provides empirical backing to the concern raised by Hupe (2014) that mainstream implementation studies typically do not focus on the multiple layers of the implementation system.Interestingly though, this analysis suggests that implementation scientists are incorporating multiple levels of analysis at relatively higher rates than public affairs investigators (19 percent in the implementation science core journals versus 11 percent in the public affairs journals).
In addition to observing differences between streams, our analysis reveals differences based on the level of analysis of the findings.Program-level findings are more likely to result from quantitative analyses, whereas organizational, frontline, and policy findings are more likely to employ qualitative methods.Quantitative analysis may be more feasible, and desirable, for program evaluations, while an understanding of organizational and front-line factors may require a qualitative approach.Mixed methods research designs are likely needed to inform more generalizable findings across policy levels.
Of course, there are some limitations of this study.The first limitation noted earlier is the limited ability to systematically explore full articles except with the purposively-selected subsample of 100 articles, though this review of the full articles revealed substantial concurrence between the abstracts and the full articles.Second is the use of relatively narrow search criteria in building the article database: only articles using the exact terms "policy implementation" and "program implementation" in the abstract, key words, or title were used, and our findings could reflect a bias stemming from the use of those terms.However, the purpose of this analysis is not to identify all possible relevant implementation articles, but rather to compare two streams of implementation research and identify general research trends.Additionally, limiting the scope of this analysis to only those articles published in peer-reviewed journals means research taking place elsewhere, specifically in scholarly books, dissertations, and policy reports, is necessarily excluded.While these research venues are valuable sources of information on implementation studies, the focus of this work is on research subject to peer-review, warranting their exclusion.

Conclusions
This analysis probes the paradox which Saetren (2005;2014) articulates: public administration scholars believe interest in policy implementation research has faded, and yet there is a thriving research stream that explores implementation questions relevant to, and being funded by, public managers and policy makers.Rather than a discrete body of literature within public administration or political science, research on implementation today is more heterogeneous and spread across a variety of fields.This new stream of implementation science literature has emerged relatively independently of traditional implementation research in public administration, and the evidence presented in this article demonstrates that interest in implementation issues continues to expand over time.
Both streams make important contributions to an understanding of implementation dynamics in the pursuit of addressing messy public problems.However, each exhibits certain tendencies that, taken in isolation, may make the research less relevant in practice.With an emphasis on formal policies and reliance on case study methodologies, literature in the public affairs research stream may be criticized for being too broad and without applicability for improving specific practices.On the other hand, literature in the implementation science stream often emphasizes programmatic elements without regard for the broader implementation system, with a preference for quantitative and experimental methods.The result of this is rather narrow contributions, focused so singularly on a particular program and specific elements subjected to randomization that the research is not generalizable outside of a particular context.Mixed methods that employ a multi-level research design may offer a strategy to bridge the artificial intellectual divide.

Table 2 .
Saetren's (2005)s the distribution of articles across 10 distinct content categories.6Health is the most common content area for the sampled implementation articles, with nearly half of the total sample coming from this arena.Education, the environment, and social welfare are the next most common focus areas, and a handful of papers concentrate on the other areas.A comparison toSaetren's (2005)assessment of 1933-2003 publications suggests the growth of implementation studies is focused in health care, which is consistent with the historical Implementation Articles by Content Area, and Research Strategy % of Total Ɨ Subcategories are not mutually exclusive as articles could focus on more than one content area.development of the implementation science research stream being fueled by interest in this policy domain.7

Table 3 .
Implementation Articles by Policy Mention and Target Population Focus Ɨ Subcategories are not mutually exclusive as articles could focus on more than one target population.

Table 4 .
Multi-Level Focus: Findings by Policy or Program FocusThis column adds up to less than 100 percent due to a number of articles that did not explicitly focus on any one level of analysis, but rather discussed implementation issues generally.
Ɨ Sub-categories are not mutually exclusive.†

Table 5 .
Content Areas and Research Strategy by Level of Findings Ɨ Subcategories are not mutually exclusive.
These program-specific studies focus on a policy or intervention, examining implementation processes and results in relation to time, control groups, or comparison settings.For example, one study looks at the adoption of a physical activity program, Animal Trackers, developed in primary schools to early childhood programs.Another looks at the replication of the Toward No Drug Abuse curriculum in high schools and documents the continued efficacy of the intervention.In each, investigators test various training and coaching combinations in order for facilitators or teachers to see which approach produces stronger fidelity to the initial model.As suggested by these examples, the key analytical question of program-specific implementation studies focuses on predicting successful replication in other settings and with other target groups.As such, this type of implementation study is closely related to the larger field of program evaluation.