Detailed Information on the
EPA Pesticides and Toxics Research Assessment

Develop a formal response to the Board of Scientific Counselors (BOSC) independent expert review report, address action items, and make progress toward long-term and annual targets.

Reassess meaningfulness of current efficiency measure in light of recent National Academy of Sciences (NAS) report on efficiency measurement.

Develop a system to utilize quarterly performance measurement reporting to improve program performance rather than solely revising annual and long-term plans.

Develop a system to utilize quarterly performance measurement reporting to improve program performance rather than solely revising annual and long-term plans.

Measure: Utility of ORD's methods, models, and data for EPA's Office of Prevention, Pesticides, and Toxic Substances and other organizations to prioritize testing requirements; enhance interpretation of data to improve human health and ecological risk assessments; and inform decision-making regarding high priority pesticides and toxic substances.

Explanation:This measure captures the assessment by an independent expert review panel of the appropriateness, quality, and use of the program's research under Long-Term Goal 1. Using a well-defined, consistent methodology developed through an OMB/ORD/Board of Scientific Counselors (BOSC) workgroup, the BOSC provides a qualitative rating and summary narrative regarding the performance of each Long-Term Goal. Rating categories include: Exceptional, Exceeds Expectations, Satisfactory, and Not Satisfactory. Full ratings are expected approximatey every 4 years, though the BOSC will provide progress ratings at the mid-point between full program reviews.

Measure: Percentage of planned outputs delivered in support of the Office of Prevention, Pesticides and Toxic Substances' and other organizations' needs for methods, models, and data to prioritize testing requirements; enhance interpretation of data to improve human health and ecological risk assessments; and inform decision-making regarding high priority pesticides and toxic substances.

Explanation:Annual research outputs are included in the program's Multi-Year Plan. At the end of the fiscal year, the program reports on its success in meeting its planned annual outputs. The program strives to complete 100% of its planned outputs each year.

Measure: Utility of ORD's methods, models, and data for EPA's Office of Prevention, Pesticides, and Toxic Substances and other organizations to make probabilistic risk assessments to protect natural populations of birds, fish, other wildlife, and non-target plants.

Explanation:This measure captures the assessment by an independent expert review panel of the appropriateness, quality, and use of the program's research under Long-Term Goal 2. Using a well-defined, consistent methodology developed through an OMB/ORD/Board of Scientific Counselors (BOSC) workgroup, the BOSC provides a qualitative rating and summary narrative regarding the performance of each Long-Term Goal. Rating categories include: Exceptional, Exceeds Expectations, Satisfactory, and Not Satisfactory. Full ratings are expected approximately every 4 years, though the BOSC will provide progress ratings at the mid-point between full program reviews.

Measure: Percentage of planned outputs delivered in support of the Office of Prevention, Pesticides and Toxic Substances' and other organizations' needs for methods, models, and data for probabilistic risk assessments to protect natural populations of birds, fish, other wildlife, and non-target plants.

Explanation:Annual research outputs are included in the program's Multi-Year Plan. At the end of the fiscal year, the program reports on its success in meeting its planned annual outputs. The program strives to complete 100% of its planned outputs each year.

Measure: Utility of ORD's methods, models, and data for EPA's Office of Prevention, Pesticides, and Toxic Substances and other organizations to make decisions related to products of biotechnology.

Explanation:This measure captures the assessment by an independent expert review panel of the appropriateness, quality, and use of the program's research under Long-Term Goal 3. Using a well-defined, consistent methodology developed through an OMB/ORD/Board of Scientific Counselors (BOSC) workgroup, the BOSC provides a qualitative rating and summary narrative regarding the performance of each Long-Term Goal. Rating categories include: Exceptional, Exceeds Expectations, Satisfactory, and Not Satisfactory. Full ratings are expected approximately every 4 years, though the BOSC will provide progress ratings at the mid-point between full program reviews.

Measure: Percentage of planned outputs delivered in support of the Office of Prevention, Pesticides and Toxic Substances' and other organizations' needs for methods, models, and data to make decisions related to products of biotechnology.

Explanation:Annual research outputs are included in the program's Multi-Year Plan. At the end of the fiscal year, the program reports on its success in meeting its planned annual outputs. The program strives to complete 100% of its planned outputs each year.

Measure: Percentage of SP2 publications rated as highly cited publications.

Explanation:This metric provides a systematic way of quantifying research performance and impact by counting the number of times an article is cited within other publications. The "highly cited" data are based on the percentage of all program publications that are cited in the top 10% of their field, as determined by "Thomson's Essential Science Indicator" (ESI). Each analysis evaluates the publications from the last ten year period, and is timed to match the cycle for independent expert program reviews by the Board of Scientific Counselors.

Measure: Percentage of SP2 publications in "high impact" journals.

Explanation:This measure provides a systematic way of quantifying research quality and impact by counting those articles that are published in prestigious journals. The "high impact" data are based on the percentage of all program articles that are published in prestigious journals, as determined by "Thomson's Journal Citation Reports" (JCR). Each analysis evaluates the publications from the last ten year period, and is timed to match the cycle for independent expert program reviews by the Board of Scientific Counselors.

Measure: Percent variance from planned cost and schedule.

Explanation:This measure captures the ability of the program to increase cost effectiveness based on the extent to which it delivers annual research outputs relative to the amount of funds spent. Using an approach similar to Earned Value Management, the data are calculated by: 1) determining the difference between planned and actual performance and cost for each Long-Term Goal, 2) adding these data together to generate program totals, and 3) dividing the Earned Value of all work completed by the Actual Cost of all program activities.

Is the program purpose clear?

Explanation: The purpose of the Pesticides and Toxics Research Program, known as the Safe Pesticides/Safe Products (SP2) Research Program, is to provide EPA's Office of Prevention, Pesticides, and Toxic Substances (OPPTS) with the scientific information it needs to reduce or prevent unreasonable risks to humans, wildlife, and non-target plants from exposures to pesticides, toxic chemicals, and products of biotechnology. By specifically addressing OPPTS' high priority research needs, the SP2 Research program provides OPPTS with the research foundation for meeting the regulatory mandates of the Toxic Substances Control Act (TSCA), the Federal Insecticide, Fungicide, and Rodenticide Act (FIFRA), and the Food Quality Protection Act (FQPA).

Evidence: 1) EPA Strategic Plan for 2006-2011 (page 114 in pdf), http://www.epa.gov/ocfo/plan/2006/entire_report.pdf; 2) Multi-Year Plan for Safe Pesticides/Safe Products (pages 5-7 in pdf), http://www.epa.gov/osp/myp/SP2%20MYP%20120106final.pdf; 3) SP2 Program Design Model; 4) Toxic Substances Control Act (TSCA), http://www.access.gpo.gov/uscode/title15/chapter53_.html; 5) Federal Insecticide, Fungicide, and Rodenticide Act (FIFRA), http://www.access.gpo.gov/uscode/title7/chapter6_.html; 6) Food Quality Protection Act (FQPA), http://www.epa.gov/pesticides/regulating/laws/fqpa.

Does the program address a specific and existing problem, interest, or need?

Explanation: In its 2007 program review, EPA's Board of Scientific Counselors (BOSC) noted that the "SP2 program is fully relevant to the Agency's needs" and that "key science questions are well focused, on-target, and appear to meet EPA's current and near-term needs." Specifically regarding the program's work under long-term goal 3, the BOSC commented, "[T]his type of research is nowhere available off the shelf, and, therefore, it has to be performed to meet the 'decision making' needs of the ORD and OPPTS. Because this type of research is unique, it serves as a template for research elsewhere in the world." The SP2 Research Program provides OPPTS with the scientific research needed to meet the mandates of TSCA, FIFRA, FQPA, and the Pollution Prevention Act. For example, as authorized by TSCA, FIFRA, and FQPA, OPPTS needs, and requests, data from the SP2 Program in order to: 1) evaluate the potential effects of industrial chemicals, pesticides, and products of biotechnology on human health and the environment, 2) discern when additional data are needed, 3) set allowable levels of exposure or releases to the environment that are protective of human health and the environment, and 4) determine whether risk management approaches are needed, and if so, which ones. No other single research organization has such an extensive portfolio of ongoing research in providing these methods, models, and data for reducing scientific uncertainty regarding pesticides, toxic substances, and products of biotechnology.

Evidence: 1) Toxic Substances Control Act (TSCA) of 1976, http://www.access.gpo.gov/uscode/title15/chapter53_.html, 2) Federal Insecticide, Fungicide, and Rodenticide Act of 1996 (FIFRA), http://www.access.gpo.gov/uscode/title7/chapter6_.html; 3) Food Quality Protection Act (FQPA), http://www.epa.gov/pesticides/regulating/laws/fqpa; 4) Pollution Prevention Act of 1990, http://www.epa.gov/opptintr/p2home/p2policy/act1990.htm; 5) OPPTS Assistant Administrator's Program Priorities for FY06-07, www.epa.gov/oppts/pubs/programpriorities.htm; 6) OPPTS/ORD White Paper: A Sustainable Risk Assessment Paradigm to Support TSCA and FIFRA/Decisions: Closing the Scientific Gaps; 7) OPPTS/ORD Strategic Plan for Supporting TSCA and FIFRA/FQPA Decisions Under the ORD National Program Director Process; 8) EPA Strategic Plan for 2006-2011 (page 114 in pdf), http://www.epa.gov/ocfo/plan/2006/entire_report.pdf; 9) Office of Research and Development Strategic Plan (page 37 in pdf), http://www.epa.gov/osp/strtplan/documents/final.pdf; 10) Multi-Year Plan for Safe Pesticides/Safe Products, pages 7-9, 20-42, http://www.epa.gov/osp/myp/SP2%20MYP%20120106final.pdf; 11) SP2 Program Design Model; 12) Draft SP2 BOSC Subcommittee Report (pages 5, 6, 12,16).

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: With such a broad array of research activities for validation, modeling or test methods, it is likely that some duplicative efforts exist, however the SP2 Research Program is designed to provide specific tools and information that OPPTS needs to meet its regulatory mandates. In the development of the research program, ORD carefully considers other ongoing research efforts in the area. The SP2 Multi-Year Plan describes in greater detail the complementary activities ongoing in other research organizations. While some US Agencies conduct or support research in the area of test methods' development, their scope/approach differs from EPA's research under LTG 1. For example, the National Toxicology Program coordinates interagency toxicology testing, conducts an interagency validation process for alternative tests, and convenes panels to develop risk assessments on targeted chemicals. ORD scientists participate in and are kept abreast of these activities to ensure that research/science activities are not duplicative. NIEHS is supporting the development of alternative methods and models. Scientists from ORD's Computational Toxicology and SP2 Research Programs are collaborating with NIEHS on some of these efforts through an Interagency Agreement (IAG) and with the NIH's Chemical Genomics Center through another IAG. Various US Agencies (e.g., DOE , US Army, and NOAA) also conduct research to develop probability-based exposure models for ecological risk assessments, complementing ORD's research under LTG 2. Research on different aspects of this area is also being supported by other countries and the private industry (e.g., UK's Department for Environment, Food, and Rural Affairs, Germany, CANTOX Environmental, Bayer Crop Science). The scope of ORD's research?? developing specific tools for use by OPP to ensure that the mandates of the Endangered Species Act are considered in making decisions about pesticides use?? differs from the scope of these other research organizations. ORD also participates in several interagency activities relevant to SP2 research under LTG 3. ORD co-chairs the Committee on the Environment and Natural Resources and its Toxics and Risk Subcommittee under the auspices of OSTP. ORD is a member of the Biotechnology Steering Committee, under OSTP's Committee on Science, and its Working Groups on Biotechnology Research and Agricultural Biotechnology Risk Analysis (AGRA). In addition, ORD participates on the United States-European Union Biotechnology Task Force. Through these panels, research activities are shared across the agencies/countries to facilitate collaborations and avoid redundancy. Further at the international level, ORD participates on collaborative efforts with the OECD and the IPCS of the WHO on chemical testing protocols development, risk assessment approaches, and the application of toxicogenomics in chemical assessments. The chemical and pesticide industries develop the chemical-specific data that are submitted to OPPTS in accordance with regulatory requirements. In addition, the American Chemistry Council sponsors health and environmental with three focus areas: improved methods, susceptible populations, chemicals in the environment. Commonalities exist among these topics with the general objectives of the research in SP2, and other EPA research programs. ORD and ACC have issued a joint request for applications through an MOU. In the draft BOSC report, the reviewers noted about LTG 3 that "[T]his type of research is nowhere available off the shelf, and, therefore, it has to be performed to meet the 'decision making' needs of the ORD and OPPTS. Because this type of research is unique, it serves as a template for research elsewhere in the world." The report pointed out specific examples under LTG 2 were there was "clearly and ingeniously demonstrated a strong collaboration of EPA scientists with outside researchers including STAR recipients."

Evidence: 1) Multi-Year Plan for Safe Pesticides/Safe Products (pages 10-13); 2) National Toxicology Program; 3) Example website for National Toxicology Program's Assessments of Phthalates; 4) National Institute of Environmental Health Sciences; 5) IAG with NIEHS; 6) IAG with NCGC; 7) OSTP Committees; 8) United States-European Union Biotechnology Task Force; 9) Organization for Economic Cooperation and Development; 10) International Programme on Chemical Safety (IPCS) of the World Health Organization (WHO); 11) American Chemistry Council Long Range Initiative; 12) ORD and American Chemistry Council (ACC) Joint Request for Applications; 13) Draft SP2 BOSC Subcommittee Report (Page 20); 14) Chart of Similar Organizations.

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: The SP2 Research Program's design?? outlined in the Multi-Year Plan (MYP) and Program Design Model?? is free from flaws that would limit the program's effectiveness or efficiency. The program is designed to provide the scientific tools and underpinnings for OPPTS decisionmaking; accordingly, OPPTS, Regional Offices, and other stakeholders are involved in the development of the program's MYP. The program periodically updates its MYP with respect to scientific progress and clients' recommendations for changes in strategic directions and scientific needs. In its review of the program in February 2007, EPA's Board of Scientific Counselors (BOSC) was asked to comment on the program's structure. In its initial report out, the Subcommittee noted that "ORD has successfully organized their unique capabilities for multidisciplinary research in a manner to be most responsive to OPPTS." Further it noted that the approaches being used in the program have "greatly facilitated the development of opportunities and advancements and created an avenue to make the findings immediately available to the client base." The reviewers felt that the structure of the program is "logical and laid out in a reasonable and integrate manner" and that it "facilitates organization and planning, both in the context of research directions and opportunities, and through demonstration of program outcomes as they relate to the client base."

Evidence: 1) Multi-Year Plan for Safe Pesticides/Safe Products (pages 14-42), http://www.epa.gov/osp/myp/SP2%20MYP%20120106final.pdf; 2) SP2 Program Design Model; 3) ORD's Office of Science Policy Research Planning Process, http://www.epa.gov/osp/features/factsheets.htm; 4) Draft SP2 BOSC Subcommittee Report, (pages 2, 6, 10).

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: As illustrated in its Program Design Model, the SP2 Research Program is targeted to support client needs and priorities. The directions of the SP2 research program are developed jointly with OPPTS??the program's primary client?? ensuring that SP2 research products address key priorities. Similarly, requests for application for the extramural portion of the research program are developed jointly with OPPTS and other EPA client offices to ensure relevance. As described in the SP2 Multi-Year Plan (MYP), the program keeps abreast of research conducted by other organizations and coordinates to ensure that the program does not subsidize efforts that exist under other organizations' missions or scopes. Within ORD, the SP2 Research Program addresses OPPTS' high priority needs not supported by any of the other research programs (e.g., Endocrine Disruptors, Human Health, Drinking Water, Ecological Assessment). Research results are effectively disseminated to OPPTS and other program clients (such as EPA program and regional offices, other federal agencies, agencies in other countries, and the scientific community at large) through 1) specific briefings/seminars for Agency client offices and stakeholders, 2) presentations at scientific meetings/ symposia/ workshops, and 3) scientific journal articles. Beginning in 2000, ORD initiated a seminar program with OPPTS in which intramural and extramural principal investigators meet with program scientists to inform them of the progress and outputs of their research. Teleconference capability is used with these seminars to allow Regional Offices and other stakeholders to hear the presentations. Additionally, scientists from the SP2 research program frequently use national and internationally convened meetings to disseminate their results. For example, at the 2005, 2006, and 2007 Society of Toxicology Meetings, over 27, 37, and 30 presentations from SP2, respectively, were delivered either orally or by poster. Similarly, at the 2005 and 2006 Society for Environmental Toxicology and Chemistry meeting, over 56 and 48 oral and poster presentations were delivered, respectively. EPA also holds periodic internal workshops to bring ORD/Program Office/Regional Office scientists together to review the progress of the research and determine the future directions. Overall, the SP2 program has resulted in over 467 publications in peer reviewed journals. According to the SP2 BOSC Subcommittee's report, the strategy the program has developed "has generated a feedback loop from clients that helps to apprise EPA scientists concerning the utility of particular approaches in a somewhat real-time mode" and that the research has "had, and will likely continue to have, substantive impacts on EPA decision-making." The report pointed out that "[H]igh level managers of the Program Office ?? gave testimony to the great utility of the work to both their short-term and long-term needs." Furthermore, expert external peer panels participating in Laboratory Division-level reviews have frequently cited the responsiveness of specific research projects in addressing critical Agency needs.

Evidence: 1) SP2 Program Logic Design Model, 2) OPPTS Assistant Administrator's Program Priorities for FY06-07, www.epa.gov/oppts/pubs/programpriorities.htm; 3) Presentations by OPPTS Senior Managers at BOSC Review; 4) NCER Policy for Jointly Developing RFAs with Client Offices; 5) Multi-Year Plan for Safe Pesticides/Safe Products (figures 1; pages 5-6 and 42-44), http://www.epa.gov/osp/myp/SP2%20MYP%20120106final.pdf; 6) Chart of Similar Organizations; 7) Lists of Seminars to OPPTS and Other Clients; 8) Workshop Proceedings (Comp Tox), http://www.epa.gov/comptox/comptox_workshop/agenda.html; 9) Workshop Proceedings (Biotechnology); 10) Bibliography of Peer-Reviewed SP2 Journal Articles; 11) Draft SP2 BOSC Subcommittee Report (pages 6, 7); 12) Examples of Division-Level Reviews Citing SP2 Research Responsiveness to Targeted Agency Needs.

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The SP2 Research Program's purpose is to ensure that the Agency can make scientifically informed decisions to reduce or prevent unreasonable risks to humans and wildlife from exposures to pesticides, toxic substances, and products of biotechnology. To achieve these outcomes, the program's long-term goals (LTGs) address high priority needs identified by OPPTS and focus on developing improved methods, models, and data for use by OPPTS and/or other organizations as scientific foundation for: 1) prioritization of testing requirements; enhanced interpretation of data to improve human health and ecological risk assessments; and decisionmaking regarding specific individual or classes of pesticides and toxic substances that are of high priority; 2) probabilistic risk assessments to protect natural populations of birds, fish, other wildlife, and non-target plants; and 3) decisionmaking related to products of biotechnology. Progress toward these LTGs will be measured through independent expert review panel ratings.

Evidence: 1) Program Measures; 2) EPA Strategic Plan for 2006-2011(pages 85-92 and 114), http://www.epa.gov/ocfo/plan/2006/entire_report.pdf; 3) Multi-Year Plan for Safe Pesticides/Safe Products, (pages 14-42 and figures 3-5 on pages 49-51in pdf) , http://www.epa.gov/osp/myp/SP2%20MYP%20120106final.pdf; 4) Charge to SP2 BOSC Subcommittee; 5) Draft BOSC Handbook for Subcommittee Members.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: The program has proposed ambitious targets and timeframes for its long-term measures. In its first review of the research program, the SP2 BOSC Subcommittee rated LTG 1 as "exceeds expectations" and LTGs 2 and 3 as "satisfactory." In the next comprehensive BOSC review, the program aims to receive an ambitous rating of "exceptional" for LTG 1 and an ambitious rating of "exceeds expectations" for LTGs 2 and 3, based on the baseline ratings established in 2007. To achieve these ambitious targets, the program will create an action plan based on feedback from the BOSC. This action plan will help ensure that the program is on track to maintain and increase its high level of performance. Additionally, the program's Multi-Year Plan (MYP) includes a schedule with annual milestones to achieve the cumulative completion of the program's LTGs. The timeframes in the MYP ensure that relevant research results are transferred to OPPTS in a timely fashion for their decisionmaking processes - e.g., prioritizing testing requirements, updating testing requirements, evaluating industry submitted data, and conducting human health and ecological risk assessments on pesticides, toxic substances, and products of biotechnology.

Evidence: 1) Program Measures; 2) Multi-Year Plan for Safe Pesticides/Safe Products; 3) Draft BOSC Handbook for Subcommittee Members; 4) Charge to SP2 BOSC Subcommittee.

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: The program has a limited number of specific annual performance measures that demonstrate progress toward achieving the program's long-term goals. In its Multi-Year Plan (MYP), the SP2 Research Program sets out an annual schedule for completion of a set of research outputs. Based on the MYP, the program has three annual output performance measures that assess the percentage of these activities that are achieved each year. Each annual measure ties directly to one of the program's three research goals and directly supports the program's outcome measures. The SP2 Program Design Model illustrates how the annual performance measures (outputs) and the long term measures (short term outcomes) are linked to the Agency's desired intermediate and long term outcomes. Coupled with the annual output measures are annual bibliometric measures which serve as an indicator for the quality of EPA's research. These measures represent the frequency for which EPA's research is cited in the top 10% of research in the field, and the frequency to which EPA's research is published in the high impact journals in the field. EPA collects this data biennially. The BOSC is charged with using the data from annual measures to inform their decision on their rating for the long-term measures. By definition, the BOSC can not rate the program as 'Exceeds Expectations' or 'Exceptional' unless the program has met all annual goals.

Evidence: 1) Program Measures; 2) Multi-Year Plan for Safe Pesticides/Safe Products; 3) Methodology for Tracking Annual Measures; 4) SP2 Program Design Model; 5) Bibliometric Analysis Report; 6) Methodologies for Long-Term Measurement: Bibliometric Analysis; 7) ORD Standards for Bibliometric Analysis.

Does the program have baselines and ambitious targets for its annual measures?

Explanation: The program has trend data for each of its three annual measures, and strives to complete 100% of its planned outputs each year. Targets of 100% are ambitious because MYP planning assumes full utilization of available resources, and these conditions may change with time. Additionally, significant coordination across all organizational elements is required to meet all outputs on schedule. The 1% targeted increase every two years for the program's bibliometric measures is ambitious as there are questions within the field as to whether or not the program will be able to achieve these targets. EPA has set these targets based upon trend data from other EPA research programs that have a longer history of data.

Evidence: 1) Program Measures; 2) ORD Draft Accountability Handbook & Policy; 3) Methodology for Tracking Annual Measures; 4) Multi-Year Plan for Safe Pesticides/Safe Products 5) Bibliometric Analysis Report; 6) Methodologies for Long-Term Measurement: Bibliometric Analysis; 7) ORD Standards for Bibliometric Analysis.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: The program's partners, including grantees and cooperators, commit to and work toward the annual and/or long-term goals of the SP2 research program. EPA's program and regional offices commit to the annual and long-term goals of the Research Program through ORD's Research Planning Process, in which they participate as part of the SP2 Research Coordination Team. All research developed in the Research Planning Process is in accordance with the program's annual and long term goals. Additionally, extramural solicitations for grant proposals are developed jointly by ORD and the program/regional offices to ensure that the extramurally funded research is consistent with the LTGs and APGs of the research program and supportive of the long-term Agency goals. Research grants and cooperative agreements also undergo a rigorous competitive selection process, composed of two levels of review, and are selected for funding based on quality of the proposal and responsiveness to the solicitation, including commitment to address program priorities (relevancy). Annual reports from extramurally funded proposals are reviewed for consistency with stated goals to ensure funded research supports long term Agency goals. Periodic site visits are made to the awardees. Periodic program reviews, that bring together intramural and extramural scientists supported through the program, are held to monitor the progress toward the long term goals. The SP2 research program has issued four requests for application (RFAs) in the area of computational toxicology and made six awards in the form of grants and three awards in the form of cooperative agreements. The program has also issued one RFA in the area of biotechnology and has awarded four grants. A subcommittee of ORD's Board of Scientific Counselors reviewed the overall (intramural and extramural) research program earlier this year in terms of its progress, relevance, directions, and scientific leadership. The BOSC reviewers identified the joint collaboration between ORD scientists and an extramural STAR center through a cooperative agreement as a "notable example of excellence. The collaborative relationship fills an important synthetic area." In another instance, they noted that there was "clearly and ingeniously demonstrated a strong collaboration of EPA scientists with outside researchers including STAR recipients." These statements imply that the program's partners are effectively working toward program goals. In 2007, ORD implemented a system for more explicitly linking contractor work to goals and research objectives. Specifically, in each contract's Extramural Package Routing and Transmittal Checklist, ORD programs are required to identify the program goals the contractor work will support. Statements of Work also identify regular performance reporting requirements.

Evidence: 1) Introduction to Research at the Environmental Protection Agency (includes a description of SP2 Research Coordination Team Process); 2) the Office of Research and Development's Extramural Grants Program; 3) Standard Instructions for Submitting a STAR Application (pages 10-11), http://www.epa.gov/ncer/rfa/forms/standinstr_03-04a.pdf ; 4) Terms and Conditions for Grants/ Cooperative Agreements; 5) Computational Toxicology and Biotechnology Requests for Applications (RFAs) http://es.epa.gov/ncer/rfa/archive/grants/03/current/2003high_throughput.html, http://es.epa.gov/ncer/rfa/current/2003_comptox.html, http://es.epa.gov/ncer/rfa/2004/2004_comp_tox.html, http://es.epa.gov/ncer/rfa/2005/2005_star_biotech.html, http://es.epa.gov/ncer/rfa/2007/2007_comp_tox.html; 6) Agendas for Workshops with Grantees, http://www.epa.gov/comptox/comptox_workshop/agenda.html; 7) Site Visit Reports; 8) Draft SP2 BOSC Report (pages 3 and 20); 8) Draft ORD Policy for Extramural Awards.

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: The SP2 Subcommittee of ORD's Board of Scientific counselors, an independent expert review panel, conducted a program-wide evaluation of the SP2 research program in February 2007. The panel was charged with evaluating the program's relevance, structure, performance, quality, scientific leadership, communication and coordination, and outcomes. The SP2 Subcommittee of the BOSC is comprised of a distinguished body of scientists drawn from academia, industry, non-EPA government, and the environmental field. BOSC designated federal officers use FACA procedures to develop this subcommittee and to manage the review process. As special government employees, panelists are required to take ethics training and to submit financial information to maintain independence and minimize/prevent conflicts of interests. The BOSC will continue to conduct program-wide reviews every 4 to 5 years, with mid-cycle reviews planned mid-way between the comprehensive reviews. In addition, independent peer reviews have been conducted separately for virtually all components of the research program. The National Academy of Sciences (NAS) conducted an overall assessment of the STAR program; additionally, grants in the program undergo a rigorous external peer review before they are awarded. All of ORD's Laboratories have initiated division level reviews as a means of evaluating the quality of their major research areas. External reviewers are asked not only whether the program is doing the science right (quality), but whether the program is doing the right science (relevance) and whether it is making a difference (impact). Often the division-level research that is reviewed is conducted under multiple research programs (e.g., SP2, Human Health, Drinking Water). Each Laboratory division undergoes a review of its scientific program every 5 years by an ad hoc panel of external experts from academia, industry, other government agencies, and private organizations. As a way to track progress and hold divisions accountable to the commitments made following their review, a less formal one-day evaluation is held midway through the review cycle (called a mid-cycle review) by a subset of 2 or 3 reviewers from the prior review. Following the reviews the division develops an Action Plan in response to reviewer's recommendations. The Biotechnology Research Framework was reviewed by a subcommittee of the BOSC. The Computational Toxicology extramural research, which is a component of the SP2 research program's first long term goal, has been reviewed as part of an overall review of the Computational Toxicology program by both a Science Advisory Board (SAB) subcommittee and by another subcommittee of the BOSC. Furthermore, methods and models that ORD develops are also often subjected to an external peer review. Lastly, products of ORD's research that are used in Agency decision making are also usually reviewed by an ad hoc scientific panel or the Agency's Science Advisory Board or the FIFRA Scientific Advisory Panel. For all external reviews ORD follows the guidance issued in the Agency's Peer Review Handbook.

Evidence: Evidence: 1) Board of Scientific Counselors (BOSC), http://www.epa.gov/OSP/bosc; 2) BOSC Fact Sheet; 3) Draft SP2 BOSC Subcommittee Report; 4) Charge to SP2 BOSC Subcommittee; 5) Draft Schedule for BOSC Reviews; 6) Confidential Financial Disclosure Form for Special Government Employees Serving on EPA FACA Committees; 7) EPA's Science Advisory Board (SAB), http://www.epa.gov/sab/ ; 8) FIFRA Scientific Advisory Panel, http://www.epa.gov/scipoly/sap/pubs/about.htm ; 9) EPA Peer Review Handbook, http://www.epa.gov/PeerReview/pdfs/Peer%20Review%20HandbookMay06.pdf; 10) List of Peer Reviews of Laboratory Divisions, Research Plans, and Research Products.

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: The SP2 research program budgets at the long term goal level, clearly defining the relationship between long term performance targets and resources on an annual basis. Additionally, the detailed budget requests are aligned with the program priorities and outputs identified in the MYP. The program's budget presentation in the annual Congressional Justification contains information on how funding shifts or reductions affect the program's performance. The BOSC ratings indicating the extent to which the program has accomplished its long-term performance goals are independent of the budget process, although funding decisions may be based upon BOSC recommendations to close research gaps or eliminate duplicative research. In an effort to better integrate budget and performance, the program recently adopted an efficiency measure - percent deviation from planned cost and schedule - against which progress will be reported in future Congressional Justifications. Additionally, the program is working to provide more transparent discussions in budget documents regarding how shifts in resources are based on program performance and feedback from the BOSC, and the extent to which resources levels are expected to affect future performance.

Evidence: 1) FY 2007 Congressional Justification; 2) Draft ORD Policy for Extramural Awards.

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: The BOSC SP2 Subcommittee's comprehensive review of the program, as well as BOSC and other reviews of components of the program, are designed to identify strategic planning deficiencies. Panels are specifically charged with evaluating program design and structure. The SP2 BOSC Subcommittee reviewed the entirety of the program in February 2007; when the subcommittee finalizes its report, the program will consider recommendations and create an action plan to address identified deficiencies. In the meantime, the SP2 BOSC Subcommittee in its draft report found that "[T]he evolution of thinking and introductions of technologies to address the range of complex questions have led to refinements inherent in the program that strongly demonstrate that the program clearly has been responsive to recommendations they have received from knowledgeable outside advisory boards and stakeholders." Follow-up actions in the program's improvement plan will also be addressed by the SP2 RCT under the leadership of the National Program Director for Pesticides and Toxics. Assessment of how the program has addressed deficiencies will be evaluated at the mid-cycle BOSC review and next full BOSC review. As noted in Q. 2.6, ORD Laboratory Divisions hold periodic external peer reviews. One of the areas the reviewers are asked to comment on is the quality of the science and its strategic directions. In a recent Division level review the panelists noted that the division's "reorganization in some senses followed the recommendations of the previous review and is seen as a progress within itself. Overall, the present review panel felt that progress since the last review has been excellent. The three major recommendations made in the last executive summary ?? have all been followed." Additionally, the Office of Research and Development is preparing to issue new policies and procedures for extramural awards that will require award documents to be more transparent about the extent to which the program's partners are committed to the program goals.

Evidence: 1) Charge to SP2 BOSC Subcommittee; 2) Draft SP2 BOSC Subcommittee Report; 3) Documentation of Action to Address Deficiencies (Reproductive Toxicology Division Peer Review, page 6); 4) Draft ORD Policy for Extramural Awards.

If applicable, does the program assess and compare the potential benefits of efforts within the program and (if relevant) to other efforts in other programs that have similar goals?

Explanation: Given the primary purpose of the SP2 research program is to provide OPPTS with the tools it needs to carry out its regulatory mandates, there are no other programs that have similar goals, in terms of scope and mission. EPA's research is multi-disciplinary, including: 1) research across all aspects of the risk assessment/risk management paradigm, i.e., in effects, exposure, risk assessment, and risk management; and, 2 ) as related to humans, wildlife, and plants. No other single organization has such an extensive or diverse portfolio of ongoing research in providing methods, models, and data and reducing scientific uncertainty regarding pesticides, toxic substances, and products of biotechnology. Comparison of potential benefits is conducted from a scientific perspective through coordinating with other research programs, participation at national and international scientific fora, and keeping abreast of the state of the science. EPA's SP2 program includes areas that are of unique importance in helping OPPTS meet its legislative mandates, such as requiring industry to submit data on pesticides, toxic substances, and products of biotechnology, and includes research areas that serve to improve the basic scientific understanding regarding these agents that the Agency needs to evaluate data submissions, conduct risk assessments, and to make informed management decisions. As noted in Q 1.3, participation on interagency and international fora provide an opportunity to stay aware of research ongoing at other agencies/countries, help to ensure that work is not duplicated, and help to find potential collaborators.

Evidence: 1) Chart of Similar Organizations; 2) EPA Strategic Plan for 2006-2011 (page 114 in pdf), http://www.epa.gov/ocfo/plan/2006/entire_report.pdf; 3) Multi-Year Plan for Safe Pesticides/Safe Products (pages 10-13), http://www.epa.gov/osp/myp/SP2%20MYP%20120106final.pdf; 4) National Toxicology Program, http://ntp-server.niehs.nih.gov/index.cfm?objectid=720163C9-BDB7-CEBA-FE4B970B9E72BF54; 5) Example Websites for National Toxicology Program's Assessments of Phthalates, http://cerhr.niehs.nih.gov/chemicals/dehp/dehp-eval.html; http://cerhr.niehs.nih.gov/chemicals/phthalates/dbp/dbp-eval.html, 6) National Institute of Environmental Health Sciences Introduction, http://www.niehs.nih.gov/external/intro.htm; 7) National Institute of Environmental Health Sciences Research Initiatives, www.niehs.nih.gov/external/resinits/ri-11.htm; 8) National Institute of Environmental Health Sciences Factsheets, www.niehs.nig.gov/oc/factsheets/analt.htm; 7) IAG with NIEHS; 8) IAG with NCGC; 9) OSTP Committee on the Environment and Natural Resources; OSTP Toxics and Risk Subcommittee; OSTP Biotechnology Steering Committee; OSTP Biotechnology Research Working Group (BRWG); OSTP Agricultural Biotechnology Risk Analysis (AGRA) Task Group, http://www.ostp.gov/nstc/html/_committees.html; 10) United States-European Union Biotechnology Task Force, http://useu.usmission.gov/Article.asp?ID=4885564A-6146-4C06-B303-47E4B399C72A; 11) Organization for Economic Cooperation and Development, www.oecd.org; 12) International Programme on Chemical Safety (IPCS) of the World Health Organization (WHO), http://www.who.int/ipcs/en/; 13) American Chemistry Council Long Range Initiative, www.uslri.org; 14) ORD and American Chemistry Council (ACC) Joint Request for Applications, http://es.epa.gov/ncer/rfa/current/2003_enviro_stat.html, http://es.epa.gov/ncer/rfa/partners/acc/mou_signed.pdf.

Does the program use a prioritization process to guide budget requests and funding decisions?

Explanation: The SP2 MYP identifies how and when EPA will be addressing specific research areas over the next eight years. In developing the priorities for the MYP, the degree of emphasis for research areas and long term goals, respectively, are based on the following criteria: 1) an assessment of the importance of the research to OPPTS/regional offices; 2) the magnitude of the uncertainties in the knowledge base; 3) the sequence of research needed to obtain the final answer; 4) the possibility that the research would result in a significant product for hazard identification; 5) risk characterization or risk management; 6) the technical feasibility of conducting a successful project; and, 7) legislatively mandated time frames. The priorities are refined with each update of the MYP and take into consideration input from the Agency's program and regional offices, the research progress to date, and an awareness of research ongoing in other organizations. These priorities inform ORD's annual risk-based contingency planning process, in which the Program and Regional Office members of the multi-media Research Coordination Teams (RCTs) identify their problem-driven research needs, RCTs prioritize the needs, and ORD and Agency management base resource decisions including the development of the OMB Justification, Congressional Justification and the annual Operating Plan on the prioritized information.

Evidence: 1) Multi-Year Plan for Safe Pesticides/Safe Products (page 17), http://www.epa.gov/osp/myp/SP2%20MYP%20120106final.pdf; 2) EPA Strategic Plan for 2006-2011 (page 114), http://www.epa.gov/ocfo/plan/2006/entire_report.pdf; 3) FY08 SP2 Congressional Justification, (pages 129-133), http://www.epa.gov/ocfo/budget/sciencetech.pdf; 4) OPPTS/ORD White Paper: A Sustainable Risk Assessment Paradigm to Support TSCA and FIFRA/Decisions: Closing the Scientific Gaps; 5) OPPTS/ORD Strategic Plan for Supporting TSCA and FIFRA/FQPA Decisions Under the ORD National Program Director Process; 6) OPPTS Assistant Administrator's Program Priorities for FY06-07, www.epa.gov/oppts/pubs/programpriorities.htm; 7) ORD Research Priority Setting, 8) Websites for Reviews of Research Used in Agency Decisionmaking.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: The SP2 research program collects quarterly performance information on its output milestones. Output milestones are tracked in ORD's Integrated Resource Management System (IRMS), and the quarterly information is shared with the National Program Director and ORD's DAA for Management. It is clear the performance information is used to manage the program, namely inform the annual planning process, needed updates to the MYP, and long-term strategic planning to address additional research needs and priorities. However, it is not clear that the program uses the information to improve performance. Notably, ORD Lab and Center directors are asked to report quarterly on only those annual performance measures where work is not proceeding according to schedule and the milestones may be missed. The directors then supply projected milestones and dates for when the work may be accomplished. There is no evidence to suggest the program actively addresses missed milestones other than to revise its research plans accordingly.

Evidence: 1) SP2 IRMS Reports; 2) Quarter 1 Performance Tracking Memo; 3) Quarter 2 Performance Tracking Memo; 4) Quarter 3 Performance Tracking Memo; 5) Quarter 4 Performance Tracking Memo; 6) Multi-Year Plan for Safe Pesticides/Safe Products (pages 20, 22, 28-32, Tables 1-3 on pages 52-61), http://www.epa.gov/osp/myp/SP2%20MYP%20120106final.pdf; 7) SP2 Planning Process; 8) OPPTS White Paper: A Sustainable Risk Assessment Paradigm to Support TSCA and FIFRA/Decisions: Closing the Scientific Gaps; 9) OPPTS/ORD Strategic Plan for Supporting TSCA and FIFRA/FQPA Decisions Under the ORD National Program Director Process; 10) Perflourinated Compounds Website, http://www.epa.gov/opptintr/pfoa/pubs/pfoainfo.htm; 11) CCA Website; 12) Asbestos Website, http://www.epa.gov/asbestos/pubs/asbestosprojectplan.pdf; 13) List of Peer Reviews of SP2 Research Program (pages 1-2).

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: The SP2 research program, in accord with EPA policy, has adopted a five-tier performance appraisal system for all employees requiring managers to incorporate program performance measures into personnel performance evaluation criteria. In mid-year and end-of-year performance reviews, managers and employees are held accountable for specific performance standards relating to program goals, including progress toward achieving the targets and timelines described in the multi-year plan. Additionally the program develops detailed statements of work with partners defining the budget, schedule and work to be performed. The program annually reviews the grants at its two centers of excellence and reviews grantee expenditures in accord with Agency policy.

Evidence: 1) National Program Director Performance Standards; 2) Assistant Laboratory Director Performance Standards; 3) Division Director Standards; 4) SES Performance Standards; 5) Breakdown of Program Dollars by Funding Type; 6) Grants Progress Reports; 7) Cooperative Agreement Progress Reports; 8) NCER Post-Award Monitoring Plan; 9) Sample Statement of Work.

Are funds (Federal and partners') obligated in a timely manner, spent for the intended purpose and accurately reported?

Explanation: The SP2 Research Program develops a yearly operating plan and obligates its funds by object class. Resource tracking is also done at the long-term goal level. Both the program and its partners take resource needs into account when establishing schedules for obligations. For the past three years, the percentage of obligated funds was 102.8%, 100.4%, and 79.4%, with seven months remaining on the FY06 two-year appropriation. Obligations and expenditures are tracked in the Agency's Integrated Financial Management System (IFMS) against the Operating Plan. Fund transfers between program objectives in excess of Congressionally-established limits and/or program direction require Congressional notification and/or approval. The post-award monitoring of assistance agreements and grants includes monitoring the draw-down of funds against progress on workplan tasks and deliverables. This monitoring ensures that recipients are spending the funds designated to each program area for the intended purpose. EPA received a clean audit opinion and had no material weaknesses in its latest financial statement audits.

Evidence: 1) Budget Report Showing BOC and FTE Breakout; 2) Unobligated Balances for FY04-FY06; 3) SP2 IFMS Report; 4) Example Protocol for Monitoring Assistance Agreements; 5) [Program-Specific Evidence/ Example of Monitoring Funds Against Grant Workplan]; 6) EPA Policy on Compliance, Review, and Monitoring; 7) Information on EPA's Budget Automation System; 8) Information on EPA's Resources Management Directives; 9) OIG Audit Opinions and EPA Financial Statements FY04-FY06.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: The SP2 Research Program has an efficiency measure with baseline data and targets to track the percentage variance from the planned cost and schedule for delivering key program outputs that contribute toward achieving long-term outcomes. This measure incorporates elements of cost and performance for important annual decision points, and targets are set to achieve 100% efficiency by FY 2010. To promote efficiency, the SP2 Research Program holds annual meetings with ORD, Program, and Regional Office scientists to determine the highest priorities of the research program and to determine whether parts of the program can be combined or eliminated to achieve better cost effectiveness. In 2007, select research areas were combined into two other long-term goals to facilitate a better integrated and streamlined research program. The program continuously participates in organizational efforts to improve efficiency as part of EPA's Office of Research and Development's (ORD) IT consolidations and administrative efficiencies. Additionally, ORD's National Health and Environmental Effects Research Laboratory (NHEERL), through which most of the in-house SP2 Research Program is conducted, has taken significant measures to develop cross-divisional core support operations to provide more effective and efficient use of the laboratory capabilities and expertise. Instead of each Division/Branch purchasing equipment, the Laboratory as a whole, as well as other ORD Laboratories/Centers, would have one centralized place for these critical laboratory needs.

Evidence: 1) Program Measures; 2) Methodology for Tracking Efficiency Measure; 3) SP2 Research Planning Process; 4) ORD-OPPTS-Regional Seminar Series Titles; 5) Multi-Year Plan for Safe Pesticides/ Safe Products (pages 13-14); 6) ORD's Total Cost of Ownership Initiative; 7) Documentation on NHEERL's Core Facilities (two memos from the Lab Director and a proposal for NTC).

Does the program collaborate and coordinate effectively with related programs?

Explanation: Within the Office of Research and Development, research in the SP2 program is coordinated and leveraged with other research programs, where appropriate. For example, SP2 research on developing protocols to characterize pesticides in drinking water is leveraged with research in ORD's Drinking Water research program. SP2 research on developing prioritization and screening tools is coordinated with ORD's research programs on Computational Toxicology, Human Health and Endocrine Disruptors. The draft BOSC report points out that "[S]ystems for internal communication are strong (e.g., national program managers, science-based divisions)" which allow the program to coordinate with other ORD programs where there are "significant useful scientific overlaps." At the national level, ORD participates in several interagency activities relevant to SP2 research. ORD co-chairs the Committee on the Environment and Natural Resources and its Toxics and Risk Subcommittee under the auspices of the Office of Science and Technology Policy (OSTP). ORD is a member of the Biotechnology Steering Committee under OSTP's Committee on Science. ORD is also a member of the Biotechnology Research Working Group (BRWG) and the Agricultural Biotechnology Risk Analysis (AGRA) Task Group, both working groups under the Steering Committee. Through these panels, research activities are shared across the agencies to facilitate collaborations and avoid redundancy. At the international level, ORD participates on the United States-European Union Biotechnology Task Force. Their annual meeting provides an excellent forum for sharing research activities on common areas of interest and for identifying collaborators. Furthermore, ORD, in conjunction with OPPTS, participates on many of the Organization for Economic Cooperation and Development (OECD) working groups aimed at promoting the development and harmonization of chemical testing guidelines and risk assessment approaches. For example, ORD has taken the lead within the OECD in revising the testing guidelines on developmental neurotoxicity. While the OECD itself does not conduct research, it promotes scientific innovation and encourages cross-member collaboration on research leading to the development of new testing guidelines and paradigms; and improved risk assessment and risk management tools. For example, ORD is participating in collaborative efforts through OPPTS with the OECD and the International Programme on Chemical Safety of the World Health Organization on protocol development and the application of toxicogenomics in chemical assessments. Further details on national and international collaborations with other federal agencies, international organizations and governments of other countries, and with industry can be found in the answer to Q. 1.3.

Evidence: 1) Multi-Year Plan for Safe Pesticides/Safe Products (pages 10-12), http://www.epa.gov/osp/myp/SP2%20MYP%20120106final.pdf; 2) OSTP Committee on the Environment and Natural Resources; OSTP Toxics and Risk Subcommittee; OSTP Biotechnology Steering Committee; OSTP Biotechnology Research Working Group (BRWG); OSTP Agricultural Biotechnology Risk Analysis (AGRA) Task Group, http://www.ostp.gov/nstc/html/_committees.html; 3) United States-European Union Biotechnology Task Force, http://useu.usmission.gov/Article.asp?ID=4885564A-6146-4C06-B303-47E4B399C72A; 4) Organization for Economic Cooperation and Development, www.oecd.org; 5) International Programme on Chemical Safety (IPCS) of the World Health Organization (WHO), http://www.who.int/ipcs/en/; 6) Draft SP2 BOSC Subcommittee Report , (pages 5-6); 7) Chart of Similar Organizations.

Does the program use strong financial management practices?

Explanation: The SP2 research program follows EPA's financial management guidelines for committing, obligating, reprogramming, and reconciling appropriated funds. Agency officials have a system of controls and accountability (EPA's Resources Management Directives System), based on GAO, Treasury and OMB guidance as well as generally accepted accounting practices (GAAP), to minimize improper payments. The program is served by Funds Control Officers (FCOs) that have documented experience and/or training in EPA's budget execution and financial management systems. EPA, and consequently the program, has no material weaknesses as reported by the Office of the Inspector General and has procedures in place to minimize erroneous payments.

Evidence: 1) ORD Memorandum from December 16, 2004 on Policies and Procedures, which revised and updated ORD extramural training to rely on the EPA Office of Acquisition and Management requirement; 2) EPA's Resource Management Directives System; 3) EPA's Generally Accepted Accounting Practices (GAAP); 4) EPA Financial Statements, Including Audit Opinions (in Performance & Accountability Report); 5) L/C/O FMFIA compliance letters; 6) Records Schedule 299??Budget Automation System (BAS).

Has the program taken meaningful steps to address its management deficiencies?

Explanation: While the program currently has a system to identify management deficiencies, the program has little evidence to demonstrate that it has identified and addressed management deficiencies in the past. A Board of Scientific Counselors (BOSC) is asked to comment on the program's structure, relevance, progress, communication and coordination, and outcomes. The BOSC recently completed its first review of the program, and when the subcommittee finalizes its report, the program will consider recommendations and create an action plan to address identified deficiencies. In addition to the BOSC, on-going research is evaluated by an external peer review process either at the laboratory or project level. Findings and recommendations are used by senior management for research planning, resource allocation and strategic direction. For example, at one review the panelists found that the "leadership of [the division] requires strengthening and change of focus." In response, the Laboratory Director appointed new leadership "committed to be strategic, and to focus on quality science and on the most pressing agency problems." The SP2 research planning team also meets routinely to evaluate the status of on-going research; deficiencies noted by the National Program Director and planning team are remediated through the Assistant Laboratory/Center Directors and/or Directors of ORD Labs/Centers.

Evidence: 1) Charge to the SP2 BOSC Subcommittee; 2) Draft SP2 BOSC Subcommittee Report; 3) Example of How Management Deficiencies Have Been Addressed; 4) SP2 Planning Process.

Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?

Explanation: All of the SP2 research grants and cooperative agreements are awarded through ORD's competitive STAR extramural grants program. All STAR research grants and cooperative agreements are subject to a rigorous competitive process as articulated on EPA's web site. In 2006, 100% of SP2-related grants were distributed according to this competitive process. To attract new investigators, research solicitations are posted on the NCER website and on grants.gov for at least 90 days, emailed to institutions and individuals that have indicated an interest in receiving them, distributed at scientific conferences, and disseminated to researchers by other federal agencies. NCER does not accept renewal applications. All applications, whether from new or returning applicants, are subject to the same high standards of peer and programmatic review and are governed by the Agency's revised competition order. Peer reviewers are selected so that a diversity of backgrounds is incorporated into the Peer Review Panel. The panels typically consist of scientists from academia, federal, state, and local government, private industry including not-for-profits. Qualified peer reviewers are selected by: 1) evaluating the RFA and the applications received to ascertain expertise required, 2) evaluating recommendations from work associates, past panelists, 3) querying numerous databases. Of the 13 awards under the SP2 research program (2003-2006), three are in the form of cooperative agreements and the remaining as grants. All of the decisions regarding the type of award made was dependent on the degree of interaction EPA scientists and the awardees were interested in having. The soundness of NCER's approaches has been recognized by the Agency for best practices in award solicitations. All ORD programs, including the SP2 Research program, are required to follow the EPA Office of Acquisition Management's (OAM) procurement policies, as detailed in EPA's Contract Management Manual. OAM follows competitive procedures in accordance with the Federal Acquisition Regulations, and it is the practice of EPA to utilize full and open competition procedures to the greatest extent possible to procure services/ supplies in support of EPA's programs/mission. After receipt of proposals submitted in response to a solicitation, EPA personnel, including program office personnel as members of a technical evaluation panel, perform a merit review of the proposals. Typically awards are made using a best value approach where for different types of acquisitions the relative importance of cost/price versus technical merit will vary. In FY 2006, 82.5% of EPA's contractual actions were considered competitive.

Evidence: 1) EPA National Center for Environmental Research (NCER) Website, http://es.epa.gov/ncer/; 2) The Measure of STAR, (pages 20-63), http://www4.nationalacademies.org/news.nsf/isbn/0309089387?OpenDocument; 3) Documentation of Percent of Grants Awarded Competitively; 4) Grant Application Review Information; 5) Instructions to Peer Reviewers; 6) Evaluation Criteria for Peer Reviewers; 7) NCER citation for best practices in award solicitations, http://intranet.epa.gov/ogd/competition/compet/bestpra.htm; 8) EPA's Contract Management Manual; 9) Source OAM FY 2006 Acquisition Activity Report; 10) The SP2 Research program follows all Agency and Federal policy regarding contracts, including competing contracts on Fedbizops (http://www.fedbizopps.gov/) as required.

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: The research program designates NCER grant project officers to monitor grant??and, in following, cooperative agreement?? performance, including submission of annual progress reports and documented compliance with federal requirements. Prior to award, project officers closely review the projects' budgets and often request additional explanations or modifications. In addition to scientific progress reports, project officers receive an annual report on expenditures and funds remaining from grantees, and project officers have access to the EPA funding system which permits them to review the current and entire expense records on any EPA grant at any time. Awardees of grants and cooperative agreements provide a list of publications, presentations and other activities on an annual basis and at the end of their award period. In 2006, NCER project officers conducted formal site visits to monitor the progress of approximately 15.4% of the SP2 research program's active grants. EPA offices are required to conduct evaluative reviews on at least 10% of active recipient institutions each year. Project officers routinely communicate with the program's National Program Director via regularly scheduled conference calls, planning meetings, as well as other special events in order to keep them informed on grantee performance and research progress. The NPD is also invited and many times is directly involved in Progress Review workshops for grants and cooperative agreements. The Integrated Financial Management System (IFMS) assists EPA Project Officers on contracts (Contracting Officer Representatives) in tracking contractors expenditure of funds. Site visits to contractor locales are encouraged on an as needed basis. Office of Acquisition Management auditors perform periodic audits of contractors. Internal monitoring reviews are performed on extramural vehicles on a rotating schedule as part of ORD's Management Integrity program.

Evidence: 1) NCER Website for Annual Reports, Publications, and Activities, http://www.epa.gov/ncer; 2) EPA Policy on Compliance, Review, and Monitoring; 2) Evidence of Overseeing Grant Budget Status; 3) Agendas and Proceedings of Workshops and Meetings with Awardees of Grants and Cooperative Agreements; 4) Information on Integrated Grants Management System (IGMS); 5) Documentation of Site Visits; 6)

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: An annual progress report is submitted by each awardee of a grant or cooperative agreement and posted on the EPA National Center for Environmental Research website. Reports are also distributed to EPA staff to disseminate to interested parties. These reports include summaries of progress in relation to project objectives as well as publications of research results. For example, 23 publications have resulted already from the computational toxicology awards that were made beginning in 2003. Awardees of grants and cooperative agreements also present results at national and international scientific conferences held annually where the topic of their grants fits most appropriately. Grants and cooperative agreements have been used to train at least 33 predoctoral students and 23 postdoctoral researchers. Results from grants and cooperative agreements have also been used to achieve at least 15 other grants from EPA and/or other federal agencies and organizations totaling over $18.7 million. Additionally, Contracting Officer Representatives (CORs) monitor contractor performance via monthly progress reports.

Evidence: 1) NCER Website for Progress Reports and Publication Lists, http://cfpub.epa.gov/ncer_abstracts/index.cfm/fuseaction/recipients.display/rfa_id/354, http://cfpub.epa.gov/ncer_abstracts/index.cfm/fuseaction/recipients.display/rfa_id/424, http://cfpub.epa.gov/ncer_abstracts/index.cfm/fuseaction/recipients.display/rfa_id/334, http://cfpub1.epa.gov/ncer_abstracts/index.cfm/fuseaction/searchControlled.sort?RequestTimeout=180; 2) Evidence of Awardee Presentations at Conferences and to Program Clients (Agendas and Proceedings of Workshops and Meetings with Awardees of Grants and Cooperative Agreements); 3) Bibliography of Peer-Reviewed Journal Articles from Safe Pesticides/Safe Products Research Program (note those citations from the extramural STAR program are delineated as "NCER" followed by the award number); 4) Example of Cooperative Agreement and Annual Progress Reports; 5) Tables on Additional Funding and Trainees; 6) Contractor Annual Performance Reports.

For R&D programs other than competitive grants programs, does the program allocate funds and use management processes that maintain program quality?

Explanation: The SP2 Research Program funds internal research and an external, competitive (STAR) grants program. The internal program funds are not allocated via a competitive approach. Rather, internal funding is allocated to high priority project areas as determined by ORD's planning process (which incorporates feedback from key clients) as well as by internal programmatic reviews. Internal non-competitive funding is justified because of the unique research disciplines and facilities that are available in the labs and utilized by the program. EPA's SP2 research is unique in its multi-disciplinary nature; that is, it includes research: 1) across all aspects of the risk assessment/risk management paradigm, i.e., in effects, exposure, risk assessment, and risk management; and, 2) as related to humans, wildlife, and plants. In order to conduct this research in-house, ORD has: 1) significant multi-disciplinary, and hence unique, expertise in the areas of toxicology [National Health and Environmental Effects Research Laboratory (NHEERL), National Center for Environmental Assessment (NCEA)], model development [NHEERL, NCEA, National Exposure Research Laboratory (NERL)], engineering [National Risk Management Research Laboratory (NRMRL)], and environmental exposures (NERL, NRMRL), relating to both humans and ecological systems, and in providing solutions to environmental problems and 2) unique capability and capacity across its laboratory facilities for this research to be conducted. Data quality is maintained through a comprehensive quality assurance program. Division-level peer reviews that evaluate the quality as well as direction and progress of research take place every four to five years (see answer to Q. 2.6). Furthermore, in accordance with the Administration's Research and Development Investment Criteria, EPA has initiated a formalized mechanism for conducting independent external expert reviews of the quality, relevance, and performance of EPA research programs. These prospective and retrospective reviews, which also take place every four to five years, provide a qualitative assessment of the success of EPA research programs in advancing the state of the science and reducing uncertainty. The recent SP2 BOSC review provides the program with another measure to evaluate the quality of its research and identify high priority research areas. For example, in addressing the question of the quality of the program, the BOSC Subcommittee "commended ORD for providing an environment that fosters creativity, innovation, cooperation, and integration in project design, execution, and communication." ORD will use feedback from the review to improve the program design, measurement, and management of the SP2 research program.

Evidence: 1) Breakout of Program Funds; 2) SP2 Planning Process; 3) Overview of the EPA Quality System for Environmental Data and Technology (EPA/240/R-02/003), http://www.epa.gov/quality1/qs-docs/overview-final.pdf; 4) Multi-Year Plan for SP2 (pages 12-14); 5) Draft SP2 BOSC Subcommittee Report (pages 20-22).

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: In January 2007, the SP2 Subcommittee of the BOSC rated the performance of the research program based on the appropriateness, quality, and use of the program's research by OPPTS and other decision-makers to achieve results. According to the draft BOSC report, the ratings were based largely on the testimony of OPPTS in demonstrating the contribution of the program's research to key human health and environmental outcomes. Specifically, the BOSC established baseline ratings of 'exceeds expectations' for work under LTG 1, 'satisfactory' for work under LTG 2, and 'satisfactory' for work under LTG 3, according to the draft BOSC report. Although no prior ratings or targets are available to measure the extent to which the program has improved, these independent ratings clearly demonstrate a high level of performance. However, given the recent rating, there is no data to support whether the program is, or is not on a track to achieve the next set of rating increases as identified for its long-term goals. As a result, the program has earned partial but not a majority or full credit for demonstrating progress toward achieving its long-term goal, but it could increase the score on this question by meeting its annual targets and following the action plan which the program intends to develop. The BOSC will assess progress from the program baseline ratings every 4 to 5 years. In the next review, the program aims to achieve ratings of "exceptional" for LTG 1 and "exceeds expectations" for LTGs 2 and 3. To achieve these ambitious targets, the program will create an action plan based on feedback from the BOSC. This action plan will help ensure that the program is on track to maintain and increase its high level of performance.

Evidence: 1) Program Measures; 2) SP2 BOSC Subcommittee Report; 3) Information on the Baseline State of OPPTS' Programs, 4) Examples of Division-Level Reviews Citing SP2 Research Responsiveness to Targeted Agency Needs, 5) Examples of use of SP2 Research in Agency Decisions (pages 2-3 in List of Peer Reviews), 6) Example of Use of SP2 Research in Recent Decision by State to Lower Health Based Standard; 7) Rationale for State Decision on PFOs; 8) Rationale for State Decision on PFOA.

Does the program (including program partners) achieve its annual performance goals?

Explanation: Overall, the program has demonstrated improved completion of its outputs on time under LTG 1, but has yet to achieve it's 100% target. Under LTG 2, the program has met all of its annual targets. Under LTG 3, these data denote a positive trend, but this measure was only recently put in place. These data should be a more robust measure of performance going forward, due in part to new ORD policies and procedures that will ensure more meaningful APMs and greater accountability for data tracking and reporting. In addition, the program has established baselines for its bibliometric analysis- informed long-term measures. Although there are no prior data or targets to measure improvement toward these measures, these data similarly demonstrate a high level of performance. For example, the program's publications are 2.3 times more "highly cited" than others in the field. The program will collect data for these measures every two years.

Evidence: 1) Program Measures; 2) SP2 Multi-Year Plan; 3) IRMS APM Reports (Met/ Not Met APMs); 4) ORD Methodology for Tracking Annual Outputs; 5) Bibliometric Analysis Report; 6) Methodologies for Long-Term Measurement: Bibliometric Analysis; 7) ORD Standards for Bibliometric Analysis.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: The program has developed an efficiency measure and established baselines and targets for the measure, however there is little data to demonstrate improved efficiencies by virtue of the fact that it is a new measure. The limited data on cost and performance since FY 2004 indicate that the program achieved a higher level of efficiency in 2005. Additionally, as part of EPA's Office of Research and Development (ORD), the program continuously participates in larger organizational efforts to improve efficiency. For example, program savings achieved through information technology and administrative efficiencies are redirected toward mission-critical research and other needs to improve program performance. ORD's Information Technology Improvement Project achieved a savings of $2 million in FY 2007 by investing in a more powerful, shared platform for high performance computing and reducing storage costs. ORD's Total Cost of Ownership Initiative created a standard desktop platform, established a centralized Call Center, and consolidated aspects of ORD's core computer infrastructure and maintenance to achieve an annual savings of $2 million starting in FY 2005. These savings were reinvested in computational toxicology and human health risk assessment research, two high priority areas for ORD.

Evidence: 1) Program Measures; 2) Efficiency Measure Data. 3) ORD's Information Technology Improvement Project, http://intranet.epa.gov/ordintra/itproject/; 4) ORD's Total Cost of Ownership Initiative.

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: In response to Question 2.RD.1, the program noted that "there are no other programs that have similar goals, in terms of scope and mission." However, the extent to which a program's publications are cited in scientific literature is a measure of the program's influence in relation to research in the program's field, therefore it is possible to compare the quality of research, not to other programs, but to other research in the field. Given the relative measures of performance the program has earned partial credit for this question despite a lack of programs for comparison. In FY 2006, an independent bibliometric analysis of the 467 publications from the SP2 Research Program found that 23% of the program's publications are "highly cited" (within in the top 10% of highly cited journals) - 2.3 times the expected amount. Additionally, 35% of the program's publications are published in "high impact journals" (the top 10% of journals that have articles cited in a particular year)??3.5 times the expected amount. The Impact Factor helps evaluate a journal's relative importance, especially when compared to other journals in the same field. Therefore, the results of the analysis show that more than one-third of the SP2 papers are published in the highest quality journals. Additionally, the bibliometric analysis analyzed the program's ratio of actual citations to expected citations, finding that, overall, the SP2 Research Program papers are more highly cited than the average program in the field by 34%. The multi-disciplinary and unique nature of the SP2 research program, as a whole, is confirmed in the bibliometric analysis which identifies that publications appear in 16 of the 22 fields or disciplines (e.g., environment/ecology, chemistry, agricultural sciences, pharmacology/toxicology) for which citations are evaluated. Finally, 2.3% of the SP2 papers were identified as "hot papers," meaning that they were highly cited shortly after they were published (nearly 23 times higher than expected). Additionally, in its February 2007 program review, EPA's Board of Scientific Counselors (BOSC) stated in their draft report: "There are several areas in which the SP2 Program has played a leadership role at the national and sometimes international level" and cited a series of specific examples as evidence. They concluded: "Taken as a whole, the evidence speaks to a community of highly trained and energized researchers, many of whom are leaders in their field, and engaged in research that is providing leadership to the US and international scientific communities."

Evidence: 1) Bibliometric Analysis Report; 2) Draft SP2 BOSC Subcommittee Report (pages 22-25); 3) Chart of Similar Organizations.

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: In its February 2007 program review, EPA's Board of Scientific Counselors (BOSC) indicated that the program is effective and achieving results in most areas. Specifically, the Subcommittee rated the program's appropriateness, quality, and use under each long-term goal. The program received the following ratings: 'exceeds expectations' for work under LTG 1,'satisfactory' for work under LTG 2, and 'satisfactory' for work under LTG 3. An 'exceeds expectations' rating indicates that the program is meeting all of its goals, addressing the appropriate scientific questions to meet its goals, and that the science is competent or better. To achieve this rating, the program must also be exceeding expectations for either the high quality of the science or for the speed at which products are being produced and milestones met. A 'satisfactory' rating indicates that the program is meeting most of its goals, addressing the appropriate scientific questions to meet its goals, producing products and reaching milestones in a timely manner, and that the quality of the science is competent or better. Given that the program is overall achieving most of its goals, the program receives a rating of Large Extent for this question. In addition the draft BOSC report states that "[T]he scientific quality of the program is exceptional, which is clearly necessary for achieving and exceeding goals" and that the "research has important outcomes as well as outputs." The report also cited, as examples, that "ORD research has already resulted in improved test methods used by OPPTS and more knowledge for chemical-specific assessments" (regarding LTG 1); that "OPPTS and/or other organizations use the results of ORD's research as the scientific foundation for probabilistic risk assessments to protect natural populations of birds, fish, other wildlife, and non-target plants" (regarding LTG 2); and that "The research is scientifically advanced, both at ORD and by investigators working under the STAR program. The questions are of high relevance and urgency for the Agency, the scientific community and the general public. The scientists involved on these projects are internationally recognized and their findings and organized panels serve to establish regulatory guidance around the world. Their work has been published in high-impact peer-reviewed journals. The nature of this novel work has been the basis for "ground-breaking" research" (regarding LTG 3). Furthermore, independent peer reviews have been conducted for virtually all components of the research program.

Evidence: 1) Draft SP2 BOSC Subcommittee Report (pages 2-9); 2) List of Peer Reviews (of Laboratories, Study Designs/ Research Plans, and Research Products); 3) Examples of Division-Level Reviews Citing SP2 Research Responsiveness to Targeted Agency Needs.