Conceptualizing and Measuring the Promotion of Nonprofit Organizations’ Evidence Use by U.S. Social Service Funding Programs

Previous research on U.S. federal promotion of evidence-based programming has focused on evidence-based program registries and concludes their usefulness is undermined by prioritizing internal validity over external validity. This research explores how federal funding programs are actually promoting funded nonprofit organizations’ evidence use instead of what we might infer from registries alone. An inductively developed conceptual framework is applied to describe all 53 fiscal year (FY) 2019 social service funding programs that include nonprofit organizations among the eligible applicants, finding they promote multiple types of evidence use, with generally low coerciveness, and with applicants frequently co-determining what counts as evidence. These findings point to promotion of evidence use that balances evidencedriven prescriptiveness and enabling nonprofits’ innovation.

have also been developed by independent nonprofit organizations. These evidence-based program registries (EBPRs) provide program administrators lists of formal programs and general strategies that have been rigorously evaluated and found to meet standards for demonstrating effectiveness-that is, programs and strategies that have received an evidencebased 'stamp of approval.' Previous research on federal promotion of the use of evidence in programming has focused largely on these EBPRs. The tenor of research on the registries might be characterized as 'cautiously optimistic with major reservations.' Three primary areas of concern are: (1) an overreliance on evidence from experimental research and evaluation, (2) the exclusion of program decision-makers from the process of generating evidence, and (3) evidence use requirements that inhibit adaptation and innovation.
Carol Weiss and colleagues (2008) and Sexton and Kelley (2010) agreed that the registries are a worthwhile step toward strengthening program decision making, but that they fail to promote good fit between programs and local context, largely due to their near-exclusive focus on experimental-design evaluations that prioritize internal validity over external validity. External validity, though, is precisely what is needed to answer the registries' intended audiences' primary questions, "Will the program be effective here in my community, implemented by my organization, offered to my clients, run by my staff?" (Horne, 2017, p. 7), and registries' reports generally provide insufficient detail about program context to answer such questions (Buckley et al., 2020;Horne, 2017; see also, Cartwright & Hardy, 2012;Chen, 2010;Cronbach, 1982;Urban et al., 2014). Without such details, EBPRs can rarely provide context-specific program adaptation guidance for evidence-based programs, which is perceived to be a key barrier to their implementation by community-based nonprofit leaders and practitioners themselves (Kushner, 2015;Nelson et al., 2006;Ramanadhan et al., 2012).
The broader research and evaluation utilization literature suggests a second potential pitfall of EBPRs: the separation of the intended decision-makers from the process of generating the evidence. In the EBPR approach to promoting evidence use, program administrators are solely on the receiving end of evidence-evidence that has been generated by others, whether through earlier evaluations of other organizations' programs or research about the efficacy of general program strategies. Scholars of evaluation use, though, agree on the importance of involving decision makers in evaluation planning to pave the way for their eventual use of evaluation findings and ongoing communication between the producers and users of evidence (Chelimsky, 2015;Greene, 2015;Johnson et al., 2009;Leviton & Hughes, 1981;Patton, 1997). Strategies to promote evidence use that separate evidence producers and users may be selflimiting by design.
Third, restricting funding to applicants who propose to deliver a predefined portfolio of evidence-based programs and strategies could impede innovation, a purported benefit of service delivery via grants and contracts, especially when awarded to nonprofit organizations (Perri, 1993;Smith & Lipsky, 1993). Applying evolutionary concepts to program planning and evaluation, Urban et al. (2014) warned against the risk of evidence-based requirements leading to the development of program "monocultures," in which the portfolio of funded programs with the same goal converges on nearly identical program models: "With limited variability, there would be fewer programs from which to select, which impedes further evolution or adaptation, especially when circumstances or contexts change" (p. 131), and fewer innovative programs and program modifications would ever be developed and tested.
As criticisms of how the federal government promotes the use of evidence in its funding programs, these concerns all rest on a series of 'what ifs.' What if federal funding programs require applicants to choose programs from pre-approved lists? What if the only type of evidence recognized by federal funders is that from experimental research and evaluation? What if federal funders, alone, decide what does and does not count as evidence? To move beyond speculation, this research examines what federal funding programs are actually promoting in terms of evidence use instead of what we might be left to infer from EBPRs alone. The research presented here explores these related questions: What kinds of evidence use are promoted by federal social service funding programs? How often is evidence use promoted in these funding programs? Who decides what counts as evidence? To what extent is nonprofits' evidence use a required condition for funding?

Data and Methods
These research questions were explored through a two-stage research process spanning fiscal years (FYs) 2015 to 2020. During both stages, the authors systematically analyzed the text of federal Notices of Funding Availability (NOFAs) for grant and purchase-of-service contracting programs. In brief, the goal of the first phase was to conceptualize and operationalize dimensions of the promotion of evidence use in federal funding programs. This conceptual framework and operationalization were applied in the second stage to quantify the promotion of evidence use across all federal social service funding programs.

Stage 1 Methods: Conceptualization and Operationalization Based on All FY 2015 and FY 2016 Youth Development NOFAs
The conceptualization and operationalization stage of the project is based on systematic analysis of all federal funding announcements in the area of youth development from fiscal years 2015 and 2016: 35 funding programs from 6 agencies awarding funding to nonprofits for the direct provision of youth development programming (see Table A1 in the Appendix). The funding programs were identified by searching the Catalog of Federal Domestic Assistance (www.cfda.gov, now part of the General Service Administration's System for Award Management, beta.sam.gov), cross-checking those search results with funding opportunities listed at www.grants.gov, and then by searching the websites of individual funding agencies to be certain no relevant funding programs were overlooked. When funding programs were duplicated from FY 2015 to FY 2016, only the FY 2016 NOFAs were included.
Youth development programming has goals related to improving adolescents' socioemotional health and future-orientedness and to strengthening their resistance to negative peer pressure and risky behaviors like drug and alcohol abuse, sex, and violence. Youth development funding announcements were selected because this program area has extensive resources to support evidence-based programming, including EBPRs sponsored by the Department of Education, Department of Justice, Department of Health and Human Services, and several nonprofit organizations (Horne, 2017). Federal funding programs for youth development also rely very heavily on grants to nonprofit organizations for implementation. Youth development funding, then, is a 'critical case' for studying the incorporation of evidence use provisions-if such provisions were to be widely implemented and robustly developed in any field, it should be youth development. A conceptual framework for describing evidence use based on this critical case should be correspondingly robust.
Analysis of the funding announcements followed a general inductive approach (Thomas, 2006), informed by the grounded theory approach (Charmaz, 2006;Locke, 2001;Strauss & Corbin, 1990), to inductively develop a conceptual framework for describing the funding programs' promotion of evidence use. First, ten announcements were open coded by two authors independently. This open coding was conducted primarily inductively but guided by a set of sensitizing concepts and research goals: (1) Our research questions provided a "domain of relevance for conducting the analysis," but "not a set of expectations about specific findings" (Thomas, 2006, p. 239).
(2) Nonetheless, the authors sought to identify all segments of text deemed helpful for describing how evidence use is promoted. (3) The authors adopted a shared, broad definition of evidence: any knowledge based on systematic collection and analysis of data, whether primary to the applicant (such as findings from their original analysis of data describing their intended beneficiaries' needs) or secondary to the applicant (such as published evaluation and research reports). (4) The authors also adopted a shared, broad definition of use: the intended application of evidence to inform decision-making about programming. The authors were primarily interested in instrumental use (applying evidence directly to specific decisions about program operations) but remained alert to coding instances of conceptual use (adding evidence to the general stock of knowledge that helps make sense of programming) and even persuasive or symbolic use (using evidence to justify preexisting decisions), using the conventional types of use from the evaluation use literature (Leviton & Hughes, 1981).
Following open coding, the authors discussed the themes that emerged and developed an agreed-upon set of hierarchical codes that captured those themes (see Table 1 in the findings section below). This final set of codes was then used to code all 35 funding announcements to determine the reliability of the coding scheme. Each NOFA was coded independently by two authors; codes were applied identically in 147 (84%) of the 175 cells (when the youth development NOFA findings were organized as the social service NOFAs in Table A3).

Stage 2 Methods: Quantification Based on Analysis of All FY 2019 Social Service NOFAs
Having established the reliability of the hierarchical coding scheme, this set of codes served as the conceptual framework for describing the promotion of evidence use in the population of FY 2019 social service NOFAs. This second set of NOFAs was identified following the same process as in the first stage, but with the goal of identifying every social service funding program that included nonprofit organizations among eligible applicants. Salamon's (1992) definition of social services guided the determination of which NOFAs were in-scope: Forms of assistance, other than outright cash aid, that help individuals and families to function in the face of social, economic, or physical problems, or that provide assistance that families or neighbors once provided informally. Included is daycare services, adoption assistance, family counseling, residential care for individuals who cannot function on their own (e.g., the elderly or the physically mentally handicapped), vocational rehabilitation, disaster assistance, refugee assistance, emergency food assistance, substance abuse treatment, neighborhood improvement and more. (pp. 81-82) Funding programs primarily for medical care, classroom education, research, and professional development were excluded. 53 NOFAs administered by 13 federal agencies were identified as in-scope and included in the analysis (Table A2).
After reviewing the coding scheme, discussing examples, and practicing coding ten NOFAs together, each NOFA was coded independently by one author, but the authors discussed and reached consensus on coding decisions whenever less than completely certain. The lead author reviewed the coding of all 53 NOFAs and made minor changes after consultation with at least one co-author to ensure coding consistency. The authors also remained alert to the need to expand or amend the original coding scheme and had 'other' codes at the ready, but none of these were applied. The coding scheme developed in the first stage, then, does not appear to be overfitted to the youth development NOFAs, but suitable for describing evidence use in the primary field of interest, social services, as well.

Stage 1 Findings: Dimensions of NOFAs' Promotion of Evidence Use
All of the first stage's open coding coalesced around three dimensions that served as the conceptual framework for describing the funding programs' evidence use provisions: (1) the intended uses of different types of evidence, (2) the degree to which use is required, and (3) who-funder or fundee-determines 'what counts' as evidence. These dimensions are summarized in Table 1.

Types of Intended Use
The types of intended evidence use map onto different decision points in the program planning process: (1) use of a needs assessment to inform program design, (2) adoption of one of a range of formal programs that have already been deemed evidence-based, (3) use of existing evidence about general strategies to inform a proposed program design, (4) generating evidence in the course of program implementation and using it for ongoing program improvement, and (5) generating evidence about program effectiveness to contribute to the larger body of knowledge in the program area for future use by other service providers. No NOFA language was coded as intentionally promoting conceptual, persuasive, or symbolic use.
The first three types of evidence use bring evidence to bear on decisions about the proposed program design. The first type of intended use is the use of a needs assessment to inform program design. Applicants are asked to demonstrate their commitment to understanding the needs of their intended program beneficiaries and to using data about their needs to inform program design and implementation decisions, such as where to locate program services or what types of services to provide. In some funding programs, this needs assessment is expected to have been conducted prior to applying for the funding; in other programs, applicants are asked to describe how they would conduct a needs assessment during an initial planning period of the grant. The second type of intended evidence use is the selection of a program design from a list of existing formal programs that have already been evaluated and found to be effective, such as those listed in an EBPR. ('Formal programs' are those that are named and manualized, such as Life On Point and LifeSkills.) The third type of evidence use is the application of existing published research about general strategies, such as mentoring or on-the-job training, to program design.
The fourth type of intended evidence use is the collection and analysis of program performance data and use of those findings to guide ongoing program improvement. Funding programs often ask applicants to describe their human resource and technical capacity for collecting and analyzing data. Some of the Administration for Children and Families NOFAs summarize this type of evidence use well: 'Program performance evaluation that will contribute to continuous quality improvement'…'explain how the inputs, processes, and outcomes will be measured, and how the resulting information will be used to inform improvement of funded activities.' The final type of intended use of evidence enlists grant recipients in generating evidence to contribute to the larger body of knowledge about social service programming for others to use in future programming. The means of knowledge dissemination take on several forms. Knowledge is sometimes expected to be shared across funded sites through formal peer-topeer technical assistance programs and presentations at grant recipient meetings. In other funding programs, applicants submit detailed plans for disseminating evaluation reports, including their goals and objectives for dissemination, identification of target audiences, strategies, allocation of staff time and budget, and plans for evaluating whether the target audiences received the information as intended. Seeking publication of program-based research in peer-reviewed journals is expected of some grant recipients as well.

Degree to Which Use is Required
The funding programs vary in the extent to which recipients are required to pursue these different types of evidence use from being strictly required for funding, to being expressly preferred, to being only recommended to applicants. The strictest language in the funding announcements uses the terms 'must,' 'shall,' 'required' 'a condition of acceptance', and 'We only consider applications that meet this priority' to describe the evidence use required of grant recipients as a condition of funding.
Other funding programs stop short of requiring evidence use but formally express a preference for some type of evidence use. In some announcements, such preferences are found in statements of funding priorities that apply generally across all of the funding agencies' grant programs, such as general priority given to funding evidence-based program designs. Similar language is also used to describe funding priorities specific to a particular funding program. Preferences for evidence use is most formalized in funding programs that award points for different types of evidence use when scoring proposals and include these details in proposal scoring guides published with the funding announcement.
Plans for evidence use can also be merely recommended in funding announcements, connoting a deliberately light-handed approach to promoting the use of evidence. Such minimally restrictive language found in the announcements includes: 'applicants are encouraged to,' 'have the option to propose,' 'may propose,' 'are not required to,' 'Examples of possible deliverables include…Implementation of evidence-and practice-based approaches,' and, in describing available resources for evidence-based program design, 'We encourage you to review these resources.'

Arbiters of Evidence
The funding announcements' evidence use provisions also vary in whether 'what counts' as evidence is determined solely by the federal funding agency, by the applicant, or by either. The funding agency serves as the arbiter of evidence by providing lists of formal program designs and more general program strategies they have already assigned evidence-based status, such as those included in EBPRs and, for example, the Department of Health and Human Service's list of 35 program models designated as evidence-based for reducing teen pregnancy rates, sexually transmitted infections, and sexual risk behaviors. Funding agencies may also specify what data funded programs should collect for needs analyses, for program monitoring and improvement, and for generating evidence to share with others. In some funding programs, grant recipients automatically become participants in larger, federally designed and conducted multisite studies.
Applicants, too, can act as arbiters of evidence. While funding agencies do eventually assess proposals' evidence and plans for evidence use, some NOFAs defer to applicants in deciding what evidence to use. Some funding programs invite applicants to propose evaluation and research questions they will pursue alongside service delivery. Applicants sometimes are asked to develop their own evidence base independently by locating peer-reviewed research and applying its findings to their program designs. Some applicants are expected to develop their own program monitoring tools and plans for using such evidence for ongoing program improvement. Applicants are also sometimes given latitude in designing local evaluations, choosing whether they will conduct descriptive, process-oriented evaluations or causal, outcomes-oriented evaluations and proposing their own evaluation methodologies.
The determination of what counts as evidence may also be shared by the funding agency and grant recipient. Some funding programs identify goals for evaluation that apply across all funded program sites while also giving opportunities for individual program sites to develop their own evaluation goals and strategies. Likewise, some needs assessment data are prescribed while also expected to be supplemented by data identified by grant recipients. In designing programs, some funding announcements both offer resources for evidence-based program design and also invite applicants to identify and draw from resources they select themselves.

Stage 2 Findings: Quantifying the Promotion of Evidence Use in Social Service NOFAs
Promotion of evidence use is a very common feature of federal NOFAs that fund nonprofit social service delivery. All but one (98%) of the 53 NOFAs promote at least one type of evidence use, and 33 (62%) promote at least three of the five types of evidence use. (See Table A3 for case-level summaries of evidence use promotion.) The use of needs assessments is, by far, the most common type of evidence use promoted, with 96% of the NOFAs promoting the use of formal, data-based needs assessment to inform program design (Figure 1). A large majority-62%-promote incorporating evidence about general program strategies to inform program design and using performance monitoring data to inform ongoing program improvement. Fewer promote the use of evidence by having applicants select from existing formal evidencebased program designs (34%) or contribute to the larger body of knowledge about social service programming for future use by other service providers (32%).
The types of evidence use promoted in the funding announcements are best understood in light of the extent to which they are optional or required (Figure 2). The highest degree of coerciveness, requiring the applicant to use evidence as a condition of funding, is most consistently applied to the use of needs assessment (68% of all NOFAs) and program implementation monitoring data (47%). The funding programs are the least coercive in their promotion of evidence use by selecting an existing formal evidence-based program design (such as from an EBPR), which is only included as a recommendation in 2% of the NOFAs, as A large majority-89%-of the funding programs grant both the funding agency and the applicant some degree of authority to determine what counts as evidence. Across the different types of use, both the agency and the applicant most commonly have a role in identifying evidence, as opposed to that role being assigned to one or the other exclusively ( Figure 3). Even when promoting the use of existing formal evidence-based programs, applicants are typically given the option to use a government-identified resource, such as an EBPR, or to locate evidence-based programs on their own. The only exception to this is in the promotion of evidence use by generating evidence to contribute to a larger body of knowledge; fundees are often required to participate in multisite evaluations planned by the funding agency.

Discussion
The conceptual framework that emerged from the NOFAs reflects the broad range of types of evidence use they promote. Funding agencies promote evidence use to improve programming across the full range of the program life cycle, from needs assessment, to program design, to program implementation, to program evaluation and knowledge dissemination. This wide range of uses of evidence maps nicely onto advice given by proponents of evidence-based program planning (such as, Kettner et al., 2013;and, Sylvia & Sylvia, 2012) and suggests a maturation of the role of evidence in social service funding programs.
All of the 'what if' concerns deduced from previous research and commentary are allayed by the findings of this research: Fundees are never required to select from a list of pre-approved programs, and they are usually involved in co-determining, with the funding agencies, what evidence to consider in the development of their programming, including evidence from nonexperimental research. This is particularly surprising in light of the reservations many observers have about the evidence standards applied in the EBPRs. While these criticisms may be warranted-EBPRs do, indeed, rely nearly exclusively on evidence derived from experimental-design evaluations (Horne, 2017;Means et al., 2015); requirements to use EBPRs as the only route for nonprofits to receive federal social service funding do not exist.
Instead, applicants generally have the latitude to develop a portfolio of evidence generated by experimental methods, with their strength in establishing internal validity, as well as a complementary range of methods that better establish external validity as called for by methodologically pluralist EBPR critics (Horne, 2017;Sexton & Kelley, 2010;Urban et al., 2014;Weiss et al., 2008) and those who emphasize the importance of involving intended evidence users in identifying and generating evidence (Chelimsky, 2015;Greene, 2015;Johnson et al., 2009;Leviton & Hughes, 1981;Patton, 1997).
Given the ample opportunities for future public service professionals to be involved in identifying, generating, and using evidence, the conceptual framework may also be valuable to their educators as well. The types of use promoted by federal funding agencies could be used to deliberately design curriculum to prepare students to meet expectations for evidence-based practice. In this vein, EBPRs have already demonstrated value as a tool for teaching nonprofit management students to critically consume and apply evidence (Horne, 2020). More generally, students of public administration, nonprofit management, public policy, evaluation, and substantive service-oriented fields, such as social work, criminal justice, and workforce development, should be prepared to both generate and consume evidence that can inform each stage of a program's life cycle. Teaching these skills warrants educators' deliberate attention; previous research has well documented nonprofit managers' limited capacity for identifying and understanding evidence as a common obstacle to evidence use (Bach-Mortensen & Montgomery, 2018;Bryan et al., 2020;Carman & Fredericks, 2010;Carnochan et al., 2014;Despard, 2016).
The conceptual framework may also provide funding agencies a tool for being more deliberate in their choices about how to promote the use of evidence. Some agencies frequently use their own boilerplate language in their funding announcements. Examples of this are reflected in the similar patterns of codes assigned within the group of Substance Abuse and Mental Health Services Administration funding programs and-with relative scarcity-within the group of Office on Violence Against Women programs (Table A3). At times, the boilerplate language does not connect to the funding program-specific language; agencies may have stated a priority for funding programs that are evidence-based in one way or another but not carry that priority through to the provisions of the specific funding opportunity. Funding agencies may also use the framework to consider whether they have missed opportunities to incorporate the use of evidence in programming. The framework reveals, for instance, that Department of Agriculture, Administration for Children and Families, Housing and Urban Development, Office on Violence Against Women, and Department of Labor funding programs typically do not use funding announcements to set expectations for applicants' use of program implementation monitoring data to guide ongoing program improvement.

Directions for Future Research
This study advances the discussion about federal funders' expectations for evidence use beyond speculation to describe what they actually ask for in NOFAs, but it is limited to social service NOFAs that include nonprofit organizations among eligible applicants. Additional research is needed to know whether these findings describe the promotion of evidence in other domains, such as health care, classroom education, and social services delivered solely by government entities. These obstacles may even be exacerbated in the context of government-funded services, raising additional questions for future research to explore. Funded nonprofits may scale their efforts to meet funders' bare minimum requirements (Bryan et al., 2020;Carman, 2011;Mitchell & Berlan, 2016;Thomson, 2010); do the funding programs promote evidence use of sufficient rigor or merely box-checking? Nonprofits do commonly engage in symbolic use of evaluation (Carman & Fredericks, 2008;Lee, 2020); to what extent do funding programs inadvertently promote symbolic use of the broader range of types of evidence, with nonprofits cherry-picking evidence to support their favored, predetermined program design choices? How do government funding expectations affect organizations' cultural dispositions toward learning, an essential prerequisite to evidence use (Bach-Mortensen & Montgomery, 2018; Bryan et al., 2020;Despard, 2016;Lee, 2020;Mitchell & Berlan, 2016, 2018? Given the common problem of low levels of evaluation and performance measurement expertise in nonprofits, are applicants able to accurately budget for the evidence use requirements in their proposals, or do funded nonprofits incur cost overruns, and do evidence use expectations favor larger, more established, and more professionalized nonprofit organizations?
Finally, the funding programs' generally low levels of coerciveness, minimal prescriptiveness in how previous research may inform program design, and lack of requirements to adopt EBPR-approved programs would seem to leave room for nonprofits to exercise creativity. Future research should explore whether these evidence use provisions actually stimulate or inhibit program adaptation and innovation.

Conclusion
Federal funding agencies (as well as other funders) face competing goals. On the one hand, funders can play a positive role in promoting more and better use of evidence toward more effective and efficient programming. On the other hand, grantmaking is an opportunity to foster innovation and community-specific adaptation in the provision of services. Pursuing either goal exclusively could undermine the other. Program funding could be subject to tightly prescriptive requirements for evidence use, with programs converging on an ever-narrowing range of evidence-based designs (Urban et al., 2014). Or, funding programs could overprioritize innovation in program design, leading to innovation for innovation's sake and the underutilization of what has been learned from experience. In its current form, federal funding of nonprofit-delivered social services, in general, appears to be on a path toward balancing these goals. The breadth of types of evidence use promoted, the generally low degree of coerciveness, and the common role of grant applicants as co-arbiters of evidence indicatewhether intentionally or not-a balance between evidence-driven prescriptiveness in program design and encouraging innovation. This balance is a worthwhile goal, as is further institutionalizing it through thoughtful, intentional design of evidence use provisions in funding programs and in commensurate training of government and nonprofit administrators.

Disclosure Statement
The authors declare that there are no conflicts of interest that relate to the research, authorship, or publication of this article.