Detailed Information on the
CDC: Occupational Safety and Health Assessment

NIOSH will advance its work with the National Academy of Sciences to develop a standard method of measuring the impact of their research on the occupational safety and health field. The Academy will also rate NIOSH activities on progress in reducing workplace illness and injuries.

NIOSH will use performance information from its research efforts to help improve program direction, allocate resources and develop annual budgets.

The program will begin tracking performance on the percent of firefighters and first responders with access to chemical, biological, radiological, and nuclear respirators, the percent reductions in respirable coal dust overexposure, and the percent reduction in fatalities and injuries in roadway construction.

Measure: Reduce occupational illness and injury as measured by: a) percent reductions in respirable coal dust overexposure; b) percent reduction in fatalities and injuries in roadway construction, and c) percent of firefighters and first responders with access to chemical, biological, radiological, and nuclear respirators

Explanation:In many areas of occupational safety and health, NIOSH is one contributor among many and national illness and injury do not provide an adequate measure of the program's contributions. This measure focuses on three high priority and high impact areas where NIOSH has a more direct impact on end stage improvements in health and safety. The target year for submeasure c) is 2010. Baseline for a) ranges from 3% to 14% by position, for b) is 154 fatalities from construction vehicles from 1992 to 1998; for c) is 3%-7%.

Measure: Progress in targeting new research to the areas of occupational safety and health most relevant to future improvements in workplace protection, as judged by independent panels of external customers, stakeholders and experts.

Explanation:The approach in this measure is to evaluate the impact and relevance of NIOSH research through (1) an analysis of how research results and recommendation are used, and (2) and evaluation of the impact that results will have in reducing risk factors in the workplace.

Measure: The percentage of companies employing those with NIOSH training that rank the value added to the organization as good or excellent and the percentage of professionals with academic or continuing education training.

Explanation:NIOSH provides opportunities for education for occupational safety and health professionals to voluntarily attend training through its Education Research Centers. When NIOSH was assessed in 2004, the long-term target was established as FY 2009 for this measure. This is based on a survey that is not conducted routinely and the FY 2009 long-term target is due to be reported on in December 2009; therefore, without trend data, and given the irregularity in data collection, setting a FY 2013 target is not possible.

Measure: The percentage of NIOSH programs that will have completed program-specific outcome measures and targets in conjuction with stakeholders and customers.

Explanation:The second long-term measure will require major new efforts in NIOSH to develop measures and targets for the impact of each program activity. This annual measure tracks the progress in the goals setting process.

Measure: The percentage of NIOSH trained professionals who enter the field of occupational safety and health after graduation.

Explanation:NIOSH currently funds training for between two and three percent of the occupational safety and health workforce. The measure captures the percentage of these professionals that enter work in the field. The program provides infrastructure support to help train up to 10-15 percent of the occupational safety and health workforce.

Measure: Percent of grant award/funding decisions made available to applicants within nine months of application receipt or deadline date, while maintaining a credible and efficient, two-level peer review system.

Explanation:CDC partners with the National Institutes of Health (NIH) Center for Scientific Review to process grant applications. In keeping with the effort to coordinate resources across HHS, CDC utilizes NIH's peer review and management system and computer program (IMPACC II) for receipt and referral of grant applications. By doing so, CDC streamlines services for the extramural community, ensures uniformity of responses to applicants and achieves cost efficiencies for the Institute.

Is the program purpose clear?

Explanation: As authorized, the program purpose is to conduct research and related activities in the field of occupational safety and health and make recommendations to identify and prevent work-related illness and injury. The National Institute for Occupational Safety and Health (NIOSH) is the lead federal agency for research on the occupational health of US workers. The program conducts and supports research, responds to requests for investigation into workplace injuries, supports training, and disseminates findings for use in implementing programs and issuing regulations. The program's mission statement and research portfolio are consistent with this authorization. The program was established in part to provide independent scientific leadership and research outside of the Department of Labor.

Evidence: The program was established by the Occupational Safety and Health Act of 1970. It is part of the Centers for Disease Control and Prevention within the US Department of Health and Human Services. NIOSH is part of the Centers for Disease Control and Prevention in the US Department of Health and Human Services. Authorizations include the Public Health Service Act, Occupational Safety and Health Act of 1970, Federal Mine Safety and Health Act of 1977, the Radiation Exposure Compensation Act, the Energy Employees Occupational Illness Compensation Program Act of 2000 (and Executive Order 13179) and the Floyd D. Spence National Defense Authorization Act. Health and safety functions of the former U.S. Bureau of Mines were transferred to NIOSH by law through the appropriations bill for the Department of Health and Human Services in September, 1996.

Does the program address a specific and existing problem, interest or need?

Explanation: Workplace injuries and deaths have declined substantially over the years, but hazards remain. According to the program, an average of 9,000 U.S. workers sustain disabling injuries on the job every day, 16 workers die from an injury suffered at work, and 137 workers die from work-related diseases. An estimated 1.7 million workers are exposed to respirable crystalline silica. Agriculture, construction, manufacturing, and transportation report injury rates above the average of 6.6 per 100 full-time workers. Transportation excluding commuting is a significant area. There are research gaps to address hazards in work methods and technology and in new industries and practices. There are high estimated costs associated with occupational illness and injuries.

Evidence: Evidence of the problem of workplace safety and health includes data from the program and CDC's Worker Health Chartbook, 2000. Evidence of specific research areas is taken from the program's National Occupational and Research Agenda (NORA). According to a 2002 Liberty Mutual Research Institute for Safety report, $40.1 billion in wage and medical payments were made to injured workers in 1999. The US Bureau of Labor Statistics 2002 Census of Fatal Injuries recorded 5,524 fatal injuries. Of these, fatal highway incidents account for a quarter of fatal injuries. The only major fatality event the BLS recorded an increase of in 2002 was from exposure to harmful substance environments, including heat stroke.

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: The program is the main federal research program focused on the health and safety of the workforce. The program is focused on identification of hazards through research, dissemination of research results, and some interventions in the form of education and training. Other federal, state and private partners are focused on implementing changes in workplace practices. The research is guided by the National Occupational and Research Agenda (NORA), which gathers input from external organizations. This process also helps reduce any duplication with other efforts. The program maintains Cooperative Research and Development Agreements with a wide range of partners from academia and industry.

Evidence: The Occupational Safety and Health Act of 1970 created both NIOSH and the Occupational Safety and Health Administration. OSHA developes and enforces workplace safety and health regulations. NIOSH supports research, information, education, and training in the field of occupational safety and health. Federal entities with shared interests that have a different program purpose include the National Institutes of Health, Mine Safety and and Health Administration, Occupational Safety and Health Administration, Bureau of Labor Statistics and the Environmental Protection Agency. A NIOSH survey of other occupational safety and health research reported federal agencies outside of NIOSH invest $51 million in this area of research in FY 2000. NIOSH invested $215 million the same year. The largest contributors outside of NIOSH are at the National Institutes of Health.

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: There is no strong evidence that another mechanism would be more effective or efficient to achieve the program purpose. NIOSH addresses the program purpose through a combination of intramural and extramural research, training and educational activities. The extramural program is modeled after that used by the National Institutes of Health. The program also jointly funds and conducts research with other entities.

Evidence: NIOSH supports $43 million in extramural research grants with a 20% success rate and $51 million in intramural research. The program supports roughly 170 research grants, 10 agricultural and prevention centers, 42 training grants, and 16 university-based education and research centers. NIOSH is headquartered in Washington, DC, and has laboratories and offices in Cincinnati, OH, Morgantown, WV, Pittsburgh, PA, Spokane, WA and Atlanta, GA. As of March 11, 2004, the program has 1,399 staff with backgrounds in epidemiology, medicine, industrial hygiene, safety, psychology, engineering, chemistry, and statistics.

Is the program effectively targeted, so that resources will reach intended beneficiaries and/or otherwise address the program's purpose directly?

Explanation: The program's National Occupational Research Agenda is a 10-year research agenda that began in 1996 with the input of multiple external groups. NORA covers 21 areas of research determined to be of highest priority. Examples of priority areas include traumatic injury, asthma and chronic obstructive pulmonary disease and hearing loss. The program targets research to these priorities and forms partnership teams on a given topic area. The program has targeted increasing amounts of intramural and extramural funding through NORA since FY 1996 from a base of $15 million up to an estimated $94 million in FY 2005.

Evidence: Evidence includes agency documentation on NORA, the OMB and Appropriations Committeee budget justification documents. In FY 2002, major areas of NORA research funding include mining ($16.9 million), construction ($13.5 million), agriculture ($11 million) and health care ($7.6 million). The program publishes annual updates of NORA and published a description in the American J of Public Health, 1998, 88. The program has published some publications in Spanish for agricultural workers. The program also conducts targeted workplace health hazard evaluations for specific employers. An example of a NORA team product is a publication of priorities for research methods in occupational cancer, Environmental Health Perspectives, 111, 1, 2003, which includes among the conclusions, for example, that less expensive ways of screening new substances for potential carcinogenicity must be developed and applied before or early in their commercial use.

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The program adopted new long-term output measures that taken together help capture the outcome of the program on occupational safety, illness and/or mortality. The first measure focuses on reducing occupational illness and injury as measured by: a) percent reductions in respirable coal dust overexposure; b) percent reduction in fatalities and injuries in roadway construction, and c) percent of firefighters and first responders access to chemical, biological, radiological, and nuclear respirators. This measure will be used in combination with relying on expert review to measure effectiveness. The National Academy of Sciences will rate NIOSH activities on a scale of 1 to 5 for progress in reducing workplace illness and injuries. A third measure percentagewill track the percentage of companies employing those with NIOSH training that rank the value added to the organization as good or excellent and the percentage of professionals with academic or continuing education training.

Evidence: The first measure focuses on three high priority and high impact areas where NIOSH has a more direct impact on end stage improvements in health and safety. The target year is 2014 with the exception of submeasure c) which is 2010. The approach in the second measure is to evaluate the impact of NIOSH research through an analysis of how research results and recommendation are used and an evaluation of the impact that results will have in reducing risk factors in the workplace. There is no set of metrics that realistically captures this information. Independent external review by stakeholders, customers, and experts will provide the most accurate mechanism to evaluate impact. An external review panel can evaluate what NIOSH is producing and determine whether it is credible to credit NIOSH research with changes in workplace practices, or whether the changes are the result of other factors unrelated to NIOSH. The third measure of NIOSH training will be supported by surveys of employers.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: The program adopted targets and timeframes for the long-term measures.

Evidence: The program's targets for the first measure are reduction in coal dust exposure of -50% by 2014, reduction in roadway fatalities and deaths of -40% by 2014 and 75% of firefighters and first responders have CBRN equipment by 2010. The target for the second measure is by FY 2009, >95% of NIOSH program activities will rate 4 or 5 on a scale of 1-5 with 5 being the highest for impact as judged by independent panels of external customers, stakeholders and experts. The target for the third measure is by FY 2009, 80% of companies employing those with NIOSH training rank the value added to the organization as good or excellent and a 15% increase in professionals with academic or continuing education training.

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: The program has discrete and quantifiable annual output measures that can demonstrate progress toward achieving the program's long-term outcomes. The measures are focused on the relevance, quality and usefulness of NIOSH research and the effectiveness of NIOSH training with respect to entry into the field of occupational safety and health. These measures are outputs, but by focusing on relevance, quality and usefulness and reasonably tied to the outcome of the program's efforts. An efficiency measure is under development.

Evidence: A first annual measure is the number of NIOSH research programs with program-specific outcome measures and targets. A second annual measure is the relevance metric score for NIOSH research for future improvements in workplace protection. A third annual measure is the percentage of graduates trained by the program that enter the field of occupational safety and health.

Does the program have baselines and ambitious targets for its annual measures?

Explanation: The program has targets for its new annual performance measures and has baselines. In some cases these baselines are estimates. An efficiency measure is under development and no targets or baselines are available.

Evidence: The program's target for the first annual measure is by FY 2005, one third of NIOSH programs will have completed program-specific outcome measures and targets in conjuction with stakeholders and customers. The program's target for the third measure is by FY 2005, to increase the percentage of graduates trained by the program that enter into occupational safety and health to 75%. The program does not yet have a target for the second annual measure of the relevance metric score for NIOSH research for future improvements in workplace protection.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: The program's grants correspond with priority areas in NORA and the program also targets funding announcements to specific NORA topics. Program review panels rate applications for program relevance to the goals of NORA. Grantees must then report progress to the program. For example, the grant announcement for education programs in occupational safety and health requires Education and Research Center Training Grant and Training Project Grant applicants to provide measures of effectiveness that are objective and quantitative and demonstrate impact. Applicants are also to consider NORA priorities.

Evidence: Evidence includes program announcements and extramural progress reports. For example, Federal Register notice April 8, 2004, 18580-18588. A February 2003 Gallup Organization survey of the Board of Scientific Counselors found the board had levels of satisfaction with the people, process and outcome.

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: To ensure independence of the review process for its newly established performance measures, the National Research Council at the National Academies of Science will conduct the reviews under contract. As designed, this process will meet the standards of this question for independence, quality and scope. Overall, the program has not previously undergone sufficient external evaluations and these and other additional efforts are warranted. The relevance of NIOSH research is evaluated by the Board of Scientific Counselors on average of three times per year and through targeted evaluations in specific areas. A NORA liaison committee meets semi-annually and provides feedback. The University of Cincinnati completed a survey of occupational safety and health officials in March 2004 that measured their views of NIOSH products. In 1996, the HHS Office of the Inspector General conducted a survey of the Educational Resource Centers. OIG is reviewing the program's oversight of an NAS study on musculosketal disorders. There are over 20 GAO reports that touch on NIOSH's work or cite NIOSH findings.

Evidence: For each program activity, the review panel will be provided with activity-specific goals, outcomes, outputs, and other relevant information or evidence of impact. The panel will rate the performance of the program activity for the impact of the program in the workplace and for the success of the program in achieving its goals. For cases where the impact is difficult to measure the panel may evaluate performance by using existing intermediate outcomes to estimate impact. The panels will also rate the relevance of ongoing or recently completed research for which the impact cannot be evaluated. The NRC will retain complete control over the review process, ensuring that the panels are unbiased, independent, and free from conflicts of interest. The charter of the Board of Scientific Counselors specifies the evaluation role. Subcommittee provide detailed feedback. Additional evidence includes the NORA liason committee minutes, HHS OIG report, Cincinnati survey, and GAO B226196, 1987. The BSC is a Federal advisory committee appointed by the Secretary of external scientists and representatives from labor and industry.

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: The program has made progress in this area but has not yet reached an integrated development of the program budget and performance information that meets the standards set out for this question. The program includes outputs and information on program accomplishments in budget documents. The draft 2006 OMB budget justification also incorporates measures into the budget document. The program is unable to quantify or estimate the impact of a given change in funding level on specific program outcomes and is unable to provide information on the added level of performance associated with incremental changes in funding in the budget request.

Evidence: Evidence includes the GPRA plans and reports, the 2005 Congressional Justification, and a draft 2006 OMB budget justification.

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: The program is drafting new long-term outcome measures and is working with NAS to develop standardized methods for measuring the impact of program research on the occupational safety and health field. The program has been taking some steps to tie budget requests to performance goals. The program has incorporated performance measures in the budget document for FY 2006. Changes in performance can not yet be estimated based on changes in funding, but the development of outcome performance goals can help facilitate this integration. The program is also working on a lessons learned report from the first 10 years of NORA to help improve program direction and will tie these findings to decision making on resource allocation and budget development. The program is also establishing a new means of reviewing the relevance of research through a contract. The program is compiling an inventory of projects under common desired outcomes to tie program goals to the project level. The program is working to translate outputs into outcomes through a new research to practice effort.

Evidence: Evidence includes agency planning documents, draft performance measures and back-up materials provided for the assessment. NAS reviews are also to be supported to serve as evaluations. The reviewers will have complete discretion over the direction and findings. The program's goal with Research to Practice is to translate research findings into effective prevention practices and products. The effort is included in the FY 2005 project planning form and project officers are to address the translation of research findings during initial project development. Projects can directly address the research to practice effort, but most research projects are to include a component that addresses translation. The Future's Initiative is helping direct the agency toward including meaningful outcome measures in agency programs and may also help support external evaluations of the performance and outcomes of agency programs. The research relevance contract will help assess the body of knowledge developed by the program and how well the program is having an impact on regulatory functions, best practices, and new technologies in industry.

If applicable, does the program assess and compare the potential benefits of efforts within the program to other efforts that have similar goals?

Explanation: The program is not an applied research and development activity pursuing multiple options toward achieving similar public benefits and according to the guidance this question is not applicable.

Evidence: Evidence includes the guidance document and program grant portfolio.

Does the program use a prioritization process to guide budget requests and funding decisions?

Explanation: As described in Section I, the program has a well established mechanism for setting priorities to guide budget requests and funding decisions through the National Occupational Research Agenda. The program also relies on the Board of Scientific Counselors and the Mine Safety and Health Advisory Committee to set research and program priorities.

Evidence: Evidence includes planning guidance, NORA documents and BSC documents.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: Outcomes from extramural partners are collected through annual progress reports and end of project reports. The program produces outcomes of program efforts with specific information on findings, uptake and other assessments of program impact. Performance information on intramural projects is limited to tracking of NIOSH publications. The program is advancing a Research to Practice agenda to better target certain research efforts to lead to improvements in occupational safety and health practice. The effort will focus on moving from characterizing risk to communicating risk and eliminating risk. The effort will focus on providing knowledge and technology in a form that is usable and adaptable in order to succeed in having an impact.

Evidence: Annual NORA update, grantee progress reports. An example of an announcement is the April 8, 2004 Federal Register notice for Education and Research Center training grants that requires applicants to provide quantitative measures of effectiveness and designate a qualified director to manage the program.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: Senior program managers are responsible for cost and schedule outcomes and performance results. Senior executive service managers have performance-based contracts. The program director's performance contract includes ongoing goals and intermediate performance measures with targets on occupational safety and health, such as to reduce the incidents of fatal fall injuries in construction by 5%. Non-SES program managers do not have performance-based contracts, but employee performance plans and evaluations tie to program goals. Program partners are held accountable through program deliverables and financial controls. The program has withheld funding, terminated awards and required assignment of new principal investigators. Extramural grantees are required to produce a final progress report, financial status report and a statement of whether or not an invention resulted from work under the grant. The program makes clear in grantee guidance that failure to do so may affect future funding. Non-competing grants and research career awards also provide detailed progress reports.

Evidence: Evidence includes the performance plans of senior managers, progress reports and program evaluation documents for grantees, and documentation of cancelled funding. Intramural projects undergo initial external reviews and a mid-year review before an internal review group. Progress reviews are used for intramural researchers annual performance evaluations. Performance contracts are held at the division director level. The grant system, IRIS, tracks project goals and plans with specific project objectives through project status and performance reports. The program is also phasing in performance goals with measurable outcomes for the intramural NORA awards.

Are funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: The program uses the Integrated Resource Information System to track obligations and obligates funding within 99% of the program's ceiling allocation in the year for which funds are appropriated. Intramural project plans provide information on work to be accomplished and spending plans. As stated previously, the program uses the extramural program review system established by NIH. Grantees provide fiscal management reports to program managers on an annual basis. The agency conducts a review of pre-commitments over $10,000 to validate completeness of documentation and appropriateness of the authority and obligation of funds.

Evidence: Evidence includes documentation provided by the program for the assessment, end of year reports and grantee guidance.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: The program has targeted management deficiencies through a leadership team that is charged with improving management effectiveness and efficiency across NIOSH. The program is commiting to increase supervisor to employee ratios by at least 0.5 over the current year. The program is hiring non-federal employees for activities that are not inherently governmental and is focusing federal hires on leadership and management. The program is bulk purchasing natural gas for the laboratories and contracting with local companies to obtain below market rates. NIOSH is also adopting NIH's grants management system. CDC has also taken steps in this area. The agency is consolidating budget execution, travel, training and graphics; has implemented a paperless contracting and purchasing system; consolidated IT; improved the supervisory ratio and reduced management reporting layers to no more than four levels. The agency administratively merged NCEH and ATSDR and dissolved OPPE.

Evidence: Evidence includes documentation provided by the program for the assessment. Roughly 30% of intramural researchers are fellows, which gives the program the ability to change with shifting needs and priorities. The program is contracting out library, printing and graphics functions and has contracted out activities in the field offices. The program's leadership team holds management meetings to more quickly resolve problems in program execution in areas such as business consolidation, pay for performance, peer review, ways to reduce maintenance costs. The leadership team consists of managers in the office of the director and division and program managers. The program revised document review and clearance to make the system less cumbersome. The program's supervisory ratio is 11.06. The agency has roughly 6,000 contractor staff to conduct commercially-oriented activities. The agency is working to reduce by 15% mission support positions.

Does the program collaborate and coordinate effectively with related programs?

Explanation: In addition to the NORA structured described previously that facilitates ongoing collaboration with a wide range of partners, the program collaborates regularly with other federal entities. The program collaborates with OSHA and MSHA through memorandums of understanding. NIOSH is engaged in an OSHA-NIOSH issues exchange group to encourage interaction and meets with OSHA on an as needed basis to help with rule making and support jointly sponsored endeavors. NIOSH also collaborates with NIH. The program's adoption of NIH's extramural research process has facilitated collaboration. The program tracks as a performance measure the amount of funding reported by other federal agencies for NORA-related research.

Evidence: Evidence includes copies of memorandums of understanding, NORA documentation and materials from the NIOSH-OSHA exchange group. NORA partnership teams are broadly representative and are lead by intramural scientists. The teams are organized around the 21 areas of NORA and author papers on research needs and hold workshops to generate requests for applications and help direct intramural and extramural research. The program currently has 14 active partnerships with NIH.

Does the program use strong financial management practices?

Explanation: An independent auditor's report in Section IV of the FY 2003 HHS Performance and Accountability Report concludes the CDC/ATSDR central financial system lacks the ability to generate financial statements, trian balance and financial statements need to be created offline, which is manually intensive, inefficient and increases the risk of error. The FY 2002 report noted reportable conditions relating to information systems; the internal controls over preparation, analysis and monitoring of financial information; reimbursable agreements; and grants accounting and oversight. None of the reportable conditions are considered material internal control weaknesses. GAO reported the agency's financial management capacity systems and procedures were insufficiently developed to address the agency's mission and budget growth. CDC automated reimbursable billings, enhanced year end closing transactions and implemented a new indirect cost methodology. CDC is also addressing staffing needs.

Evidence: Evidence includes the FY 2003 PAR, the FY 2002 Chief Financial Officers annual report, including summary of reportable conditions, summary documents on end of year balances, IRIS reports. Four areas of findings were also documented the prior year. CDC has received five consecutive unqualified opinions on the agency's financial statements. Additional data include that CDC issued 64 duplicate or erroneous payments in FY 2002, or 0.042% of all payments and has a 97% compliance rate for prompt payments. GAO-01-40, November 2000. CDC will implement UFMS in October 2004. The agency submitted first quarter financial statements to the Department ahead of schedule.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: CDC is continuing to make improvements to financial management processes, including restructuring its budget and financial accounting system to more accurately track expenditures and developing a more consistent and accurate system for charging overhead. A January 2004 GAO report notes that CDC established a Chief Operating Officer position with clear oversight authority in financial management, information technology and other areas and has made improvements in the ability to respond to public health emergencies, and that additional changes are needed to improve oversight of programs. The agency is also putting considerable effort into setting priorities and reorganizing operations through the Future's Initiative. The program's leadership team is also actively involved in identifying and resolving management issues. Examples include improving document tracking, peer review policy, electronic communication with stakeholders, and employee development planning. Future's will establish metrics for business systems, such as time it takes to hire, the cost of procurements and grants, and efficiency of grants officers.

Evidence: Management changes were documented in a GAO report (04-219). CDC initiated changes in core accounting competencies, professional staff recruitment, financial systems training, and customer service. CDC commissioned a business case for timelines, cost estimates and functional and technical solutions. CDC will be the first to pilot HHS' Unified Financial Management System and will automate the financial accounting processes. The FY 2003 PAR cites improvements in preparing financial statements. CDC launched a technical team and business transformation team and has tasked a data validation team to sample daily commitments for adherence to policy, procedure and purpose and reason for the expenditure. The agency is establishing the commitment accounting process and will review indirect expenses to reduce central management costs. CDC added reimbursable agreements as an automated system, implemented a risk management framework, completed risk assessments and security plans for 14 of 26 critical systems and is obtaining certification and accreditation for the financial systems that will feed into UFMS.

Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?

Explanation: All program grants and cooperative agreements are made competitively based on merit as determined by peer review and a secondary review for mission relevance. Extramural awards are made according to NIH practices. New applicants are solicited through workshops, meetings, a listserve and the Internet. The program operates an annual project planning process for intramural investigators, driven by a NORA established umbrella agenda. The program maintains an open announcement with opportunities to apply three times per year.

Evidence: Evidence includes grant announcements and awards, the program website, and lists of awarded grants and cooperative agreements. NIOSH projects are subject to external peer review at project inception and at least once every five years. The reviews consist of written reviews by two peer reviewers from outside CDC at a minimum and larger projects require higher levels of review. Peer reviews consider scientific merit and program importance.

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: The program has oversight practices that provide sufficient knowledge of grantee activities. The office of extramural programs reviews investigator progress reports, which include information on progress according to set goals, chantges in personnel and justication of carryover above 25 percent. Site visits are made to resolve significant discrepencies. The program produces informational letters following the visit that detail findings in fiscal management and programmatic and technical review. The program also uses special emphasis panels to review applications and proposals for research projects and grants. The disease, disability and injury prevention panel is selected by the Secretary and operates under the Federal Advisory Committee Act.

Evidence: Evidence includes grantee guidance, progress reports, site visit documentation, and documentation on cancelled funding. The split between intramural and extramural is 75/25 and between directed requests for applications and investigator initiated is 60/40. Extramural grants have grown 10% since 1996. The program supports roughly 500 intramural projects that are three to five years in duration.

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: The program collects performance data on an annual basis. The program uses a variety of outlets to distribute program findings. The program does not make all data public due to intellectual property issues.

Evidence: It publishes report abstracts and includes the information on the program website. The program also includes abstracts of new and ongoing grants in the CRISP database operated by NIH. Evidence includes the program website, report abstracts, and compendiums of research.

For R&D programs other than competitive grants programs, does the program allocate funds and use management processes that maintain program quality?

Explanation: The program allocates funds and uses management process that maintain program quality. The main instrument used is the peer review process for the initial awarding of research funds. NORA supported intramural research is reviewed externally. Publications also receive peer review.

Evidence: Evidence includes program policy documents on peer review of intramural and extramural projects, and the document peer review policy.

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: The program is assessed a small extent because it has shown progress on indicators for the long-term goal by helping reduce coal dust and silica dust exposure from 2001 to 2003 in longwall positions and continuous miner operators and partial progress for roof bolter operators and surface drill operators. Evidence of progress for an additional indicator for the long-term goal includes the percent of first responders and professional fire fighters with CBRN respirators has increased to between 3-7% in 2003 from a baseline of zero when the first respirator was certified in May 2002. The external reviews for the second long-term goal have not been conducted and progress data are not available. The program does not have baseline or trend data for the third goal.

Evidence: The process for the second measure will be supported through a contract and conducted by the National Academy of Sciences. The program has maintained a compendium of outcomes from research, alerts, standard setting, investigations and consultations that document specific actions and impacts of completed projects. Examples of outcomes include documented reductions in risk after program supported interventions and changes made in industry as a result of research findings. Taken together, these outcomes indicate progress toward meeting long-term objectives of the program and will be useful in tracking evidence of progress once the new measures are adopted.

Does the program (including program partners) achieve its annual performance goals?

Explanation: The program is assessed a small extent because it has adopted specific annual performance measures, but does not yet have sufficient data to indicate progress on the three new annual measures. For the first annual measure, the program is completing the process for one major program activity. For the second annual measure, the program has maintained a compendium of research projects that reflect advancements in discrete areas. The program does not have trend data for the third annual measure.

Evidence: The program has not yet evaluated the impact of NIOSH research through an analysis of how research results and recommendation are used and an evaluation of the impact that results will have in reducing risk factors in the workplace. The program has maintained a compendium of outcomes from research, alerts, standard setting, investigations and consultations that document specific actions and impacts of completed projects. Examples of outcomes include documented reductions in risk after program supported interventions and changes made in industry as a result of research findings. Taken together, these outcomes indicate progress toward meeting the long-term objectives of the program. In addition, the program has GPRA data on quality of research as measured by peer review.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: The program is assessed as a small extent because the center has achieved some administrative savings, but does not have sufficient data to quantify the impact. For example, the program has abolished administrative sections. The program has achieved savings through new methods of obtaining natural gas for its laboratories. Additional savings are anticipated in the future through new human resource management practices the program is testing. The agency has conducted or is conducting A-76 studies for library services, office automation, animal care, laboratory glassware and laundry services, printing, and material management services.

Evidence: Some data are available at the agency level where the agency has consolidated IT and projects a savings of $11.5 million (15%) in FY 2004 and redeployment of 39 FTE (16%) to other positions. In consolidating the administrative functions of NCEH and ATSDR, the agency will save 48 FTE (24%) by September 2004. By dissolving OPPE, the agency abolished additional FTE. Agency-wide 200 organizational sections were eliminated. The program eliminated 39 organizational sections.

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: According to the guidance, a not applicable is given because no comparable federal, state, local or private sector programs exist.

Evidence: The program conducts roughly 85% of all federal research in occupational safety and health. While other federal entities conduct research in this area, there are important differences in overall focus and purpose. State, local and private entities do not conduct significant levels of research in this area.

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: A small extent is given because the program lacks a recent, comprehensive evaluation, but has select findings showing positive program performance. The OIG report described in Section II found 82% of center graduates obtain work in occupational safety and health, 70% found training to be high quality and 94% report training prepared them adequately. The Cincinnati study found 70%-80% of occupational safety and health professionals use or refer to NIOSH products and had taken a course where NIOSH materials were used and 99% of these professionals agree or strongly agree NIOSH is an important resource for the field. A GAO report on indoor air quality at select facilities (RCED-98-149R) documented the program's activities, recommendations and contributions. The Gallup survey described in Section II found high levels of satisfication overall. Exceptions on recommendations being used effectively/responsive to agency's needs and availability of results. A GAO report on EEIOCPA noted the NIOSH-associated physician panel process has been a bottleneck to processing claims (04-298T).

Evidence: HHS Office of Inspector General, Centers for Disease Control and Prevention's Educational Resource Centers, OEI-04-92-00900, March 1996. GAO (RCED-99-254) noted NIOSH made up less than two percent of Federal funding of research on indoor pollution, ($14.6 million from FY87-FY99, and noted the program has developed standardized protocols for investigations of suspected problems and has done epidemiologic work in nonindustrial indoor workplaces to identify pollutant components that cause symptoms in over 30 percent of office workers. The review also noted the NORA process in recommending how Federal agencies should develop a consensus research agenda. The customer satisfaction survey was prepared by NIOSH and analyzed by the Institute for Policy Research, University of Cincinnati, March 2004. The BSC sub-committee for agricultural review found NIOSH made significant progress in developing a diverse agricultural research program responsive to Congressional intent to have a significant and measurable impact on the health of rural Americans.