Detailed Information on the
National Center for Health Statistics Assessment

Explicitly tie budget requests to the accomplishment of annual and long-term goals, and present resource needs in a complete and transparent manner.

Independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results.

Measure: The number of months for release of data as measured by the time from end of data collection to data release on internet.

Explanation:This measure will address the performance element of timeliness. Through this measure, NCHS will track it's improvement in the timeliness of data provided to the nation's health decision makers. The measure will adddress NCHS data in the aggregate. The unit of measurement is months.

Measure: Percentage of key data users and policy makers, including reimbursable collaborators, that are satisfied with data quality and relevance.

Explanation:This measure is under development. It addresses the performance element of quality and scope. NCHS will implement a systematic approach and tool for assesing the satisfaction of key data users and policy makers (e.g., reimbursable collaborators, ASPE, OMB, CRS and others) relative to data quality and scope. An independent group such as the NCHS Board of Scientific Counselors will be used to help identify the list of key data users and policy makers to be surveyed, along with those organizations that directly work with NCHS through interagency agreements. Performance results will be used by NCHS managers to drive program improvements. A baseline does not currently exist for this measure.

Measure: Number of improved user tools and technologies and web visits as a proxy for the use of NCHS data.

Explanation:A primary objective of NCHS is to maximize the use of data collected through investment of public funds. The greater the use of data, the more "bang for the buck" from the investment, and therefore, more efficient. One way to increase use is to make data available in more easily accessible forms. NCHS makes its data available in a variety of forms through the Internet and works to improve the speed and efficiency with which people access the data by: a) development of data input statements/programs that allow people quick access to our data files; b) development of masked variance files that allow researchers to more quickly access data; c) development of Fast Stats and Quick Stats to quickly access data files; and d) use of Beyond 20/20 software making it more likely that systems like the NCHS Data Warehouse on Trends in Health and Aging, Asthma, Healthy People 2010, and Healthy Women: State Trends in Health and Mortality, will be found and used, therefore getting more use of data already collected.

Measure: The number of new or revised charts and tables and major methodological changes in Health, United States, as a proxy for continuous improvement and innovation in the scope and detail of information.

Explanation:This measure addresses the performance element of scope. Health, United States is the most comprehensive publication producted by NCHS; it draws information from each data system, as well as data from other federal partners and collaborators. Improvements in the scope and detail of Health United States are a proxy for the scope of data produced and made available by NCHS. Improvement and innovation in Health, United States can be assessed through four components: 1) new charts in the Chartbook; 2) new trend tables; 3) tables substantially revised; and 4) major methodological changes. The published archived volumes can be inspected yearly and compared to their predessors to measure the continuous improvement and innovation.

Measure: The number of months for release of data as measured by the time from end of data collection to data release on internet.

Explanation:Results for this measure are determined by calculating the number of months from the end of data collection to release of data on the web. When calculations were prepared for the NCHS PART review, all survey/data collection systems used the first January following the data year as the end of data collection. When updating the data to report on FY 2005 results, it was determined that this methodology did not appropriately consider the process by which vital records are received the 57 registration areas. Using the original method, the end of data collection for vital records was the January following the year that the event occurred. However, records for the calendar year are not received by January of the following year. The revised method is now based on the interval between the date that all records have been received from all 57 registration areas, and the date of publication on the web. Using this revised methodology for FY2005 (reported in FY2008) the result is 8.7 months (proposed target (original method) was 13.5 months).

Is the program purpose clear?

Explanation: The purpose of the National Center for Health Statistics (NCHS) at the Centers for Disease Control and Prevention (CDC) within the Department of Health and Human Services (DHHS) is focused and well defined. The program purpose is to provide statistical information to guide actions and policies to improve the health of the American people. The purpose is consistent with authorizing legislation, mission statements, and program documents.

Evidence: Key evidence includes the program's legislative authorities (Section 306 of the Public Health Service Act, as amended, 42 U.S.C. 242k) and mission statements (www.cdc.gov/nchs/about/mission/mission.htm). The program's legislative history begins in 1956 with the National Health Survey Act.

Does the program address a specific and existing problem, interest, or need?

Explanation: The program addresses a specific need that can be clearly defined and currently exists. The program addresses the need for data on health, illness and disability, the effects of health hazards, management of medical conditions, health care costs and financing, family size and make-up and birth and death. The data are used by researchers, policymakers, industry, government and the public to monitor and study changes in the health of the US population, design and manage programs, and allocate resources. To fulfill their need for specific information from the program, public and private entities work with NCHS to develop specific questions or data elements for surveys and to finance the collection of data.

Evidence: An example of a product the program provides to the public to address the need for timely and reliable information on a wide range of topics includes the publication "Health, United States" and all of its backup materials. Examples of programs that use data collected by NCHS includes the Temporary Assistance for Needy Families program for data on out of wedlock births, the CDC to set and track agency goals, and the Health Resources and Services Administration on health care needs of children. One charge of the HHS Data Council is to ensure data collection by programs such as NCHS meets specific and existing problems, interests and needs.

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: The program is designed to fill a unique role and takes steps to ensure it does not duplicate or compete with other Federal or non-federal programs. The program supports nationally-representative surveys that cover a wide range of topics. Directly and through the HHS Data Council, the program coordinates with related efforts to avoid redundancy. NCHS works with partners to add data elements to their existing surveys rather than create new surveys. Other agencies within HHS support health related surveys such as CDC's Behavioral Risk Factor Surveillance System and the Agency for Health Care Quality and Research's (AHRQ) Medical Expenditure Panel Survey (MEPS). There is not significant redundancy between the program and these other surveys.

Evidence: An example of consultation with a related Federal survey includes the coordination of content and sample between NCHS' National Health Interview Survey and MEPS. MEPS uses a subsample of NCHS's more broad based National Health Interview Survey (NHIS) to conduct more focused study. NCHS also coordinates with other national survey efforts, such as the US Census, to avoid redundant efforts in designing and conducting their surveys. The US Department of Agriculture (USDA) and NCHS now collaborate to collect diet and nutrition data through the National Health and Nutrition Examination Survey (NHANES), which enabled USDA to terminate a separate study. States can adopt and use BRFSS in addition to reporting data to the CDC. NCHS' State and Local Area Integrated Telephone Survey (SLAITS) provides in-depth state and local data primarily for the Federal level to supplement national data collection efforts, including the National Health Interview Survey (NHIS).

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: The program does not have major design flaws that would limit its efficiency or effectiveness and there is no strong evidence that another approach, such as grants to states, would be more effective. The program primarily uses contracts for data collection and a mix of contracts and direct staffing for compilation, analysis and presentation. In order to exert control over the methods, content and quality of the data, the program uses contracts in lieu of grants or cooperative agreements. Design of the surveys themselves is reviewed by NCHS as a lead statistical agency, through evaluations supported by the Public Health Service Act evaluation funds, and through the OMB clearance process.

Evidence: The program collects vital statistics by contracting with states and conducts health care surveys by contracting with the US Census Bureau, but conducts the majority of data collection at the Federal level through contracts, contractors and Federal staff. NHANES provides an example of an innovative program design where the program collects survey and biometric data from participants across the country through specially outfitted tractor trailers. In FY 2004, out of a total budget of $107 million, the program spent $51 million on contractural services and less than $1 million on grants and cooperative agreements. The remainder of resources were for personnel and other operating expenses.

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: The program is designed so that resources directly support the program's purpose and that efforts reach the intended beneficiaries. For this program, program benefits are the data and analysis provided by the program and recipients are the agencies, researchers, policymakers and other consumers of the data. Many of these recipients provide direct input into the data collection effort and provide financial support for the data collection. The program distributes the information through publications and websites and by providing more limited access to data centers where public release cannot be supported. Access is controlled in some cases due to privacy considerations. The program also provides assistance with editing, quality control and interpretation of the data for program partners. While the program collaborates with other entities and uses information from other Federal entities such as the Census, supported activities would not occur without the program.

Evidence: More than 30 entities reimburse NCHS in exchange for collecting needed data. NCHS' authorizing legislation requires dissemination of data as widely as possible while not compromising the confidentiality of individuals. Examples of release of the data and associated technical assistance include "Health: United States", the NCHS website, the NCHS data users conference, CDC's Morbidity and Mortality Weekly Report, Healthy People 2010, the reimbursable work program and outreach training. The program is examining additional outreach, such as to health economists.

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The program adopted two long-term outcome measures. The first is focused on the amount of time it takes to release survey data publicly from the end of data collection. The measure is significant in that it gets to the timeliness and relevance of NCHS data and research. The second is focused on the satisfaction of key data users and policymakers on the program's data quality and relevance. The Interagency Council on Statistical Policy focused on product quality, namely relevance, accuracy and timeliness, and program performance, namely cost, dissemination and mission achievement. The NCHS performance measures are consistent with these focus areas.

Evidence: The first measure is the number of months for release of data (from the end of data collection to a significant release on the internet). The second measure is the percentage of key data users and policy makers, including reimbursable collaborators that are satisfied with data quality and relevance. The types of questions to be asked to the groups of data users for the second measure include: 1) Reimbursable collaborators: Did NCHS meet the purpose and requirement of the reimbursable agreement? 2) Policy makers: Is NCHS addressing the content areas needed, and is NCHS maximizing the use of its data systems to address priority policy areas? 3) Data users: Are the data produced of sufficient quality, detail, and accessibility to meet user needs? 4) Web users: Are users able to identify and access NCHS data resources?

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: The program has ambitious targets and timeframes for its long-term measures.

Evidence: The program's target for the first measure is a 25 percent reduction in the amount of time for data release by 2010 from a baseline of 14.5 months to a target of 10.9 months. Five years is an ambitious but appropriate time period to meet a quarter reduction in the amount of time it takes to release data. The target for the second measure is a satisfaction rate of 85 percent.

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: The program adopted annual measures that contribute to the long-term outcome measures.

Evidence: The first measure is the number of new or revised charts and tables and major methodological changes in Health, United States, as a proxy for continuous improvement and innovation in the scope and detail of information. The second measure is the number of improved user tools and technologies and web visits as a proxy for the use of NCHS data. the number of months for release of data (from the end of data collection to a significant release on the internet). The efficiency measure is the number of months for release of data as measured by the time from end of data collection to data release on internet.

Does the program have baselines and ambitious targets for its annual measures?

Explanation: The program has ambitious targets and timeframes for its annual measures.

Evidence: The program's first target is 15 new or revised charts and tables and major methodological changes in Health, United States. Health, United States is a proxy measure for change throughout the NCHS data systems. The proposed measure is a composite measure of different types of improvements such as new tables, sections and methods. The publication is mature and focuses on maintaining trend data. The changes in 2004 and 2005 of 21 and 36 were the unique result of a one-time top to bottom review of the publication. Fifteen annual improvements and innovations in the scope and detail of Health, United States is an ambitious and realistic target representing 10 percent of the publication. The program's second target is 5 improved user tools and technologies and an additional 850,000 web uses of the data. The program's third target is a 25 percent reduction in the amount of time for data release by 2007 from a baseline of 14.5 months to a target of 12.6 months.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: The program has mechanisms in place to help ensure program partners support the annual and long-term performance goals of the program. The program relies primarily on contracts that include specific deliverables to meet the program's objectives. Examples include private sector contracts for NHANES, National Nursing Home Survey (NNHS), SLAITS and National Survey of Family Growth (NSFG) and agreements with the US Census for NHIS and other collection activities. The program's vital statistics efforts are supported through contracts with state health departments that require commitment to timeliness, completeness, accuracy and other factors. The program's other agency partners commit to the goals of the program through direct financial contributions, technical consultation, and reviews.

Evidence: Evidence includes state contracts, meetings minutes with other Federal agency partners, Federal reimbursement agreements, and examples of specific collaborative projects such as a bioterrorism related project for the National Ambulatory Medical Care Survey.

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: Taken together, a variety of activities supported by the program constitute independent and comprehensive evaluations of NCHS activities. The program relies on the NCHS Board of Scientific Counselors (BSC) for regular review and feedback on program activities and direction. Members and the Chair are selected by the HHS Secretary and have experience in relevant areas of science. The BSC is beginning a multi-year review of each of the six major NCHS programs. The board is beginning with a review of NCHS vital statistics activities, pending the development of a structured process. The review would consider elements such as productivity, satisfaction, relevance, and program weaknesses and strengths. The program also obtains external views through the National Committee on Vital and Health Statistics (NCVHS). The program has supported more than 70 PHS Act evaluations on a regular basis and covered every program data system.

Evidence: Evidence includes schedules and minutes of BSC meetings and multiple PHS Act evaluations of program activities. The PHS Act evaluations cover such issues as methodology, quality, analysis, applicability, response, integration, and other factors for specific NCHS surveys and systems. The evaluations are conducted by individuals and organizations under contract by NCHS, often in the form of studies.

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: Budget requests for the program are not explicitly tied to accomplishment of annual and long-term goals of the program. At the total program level, the effects of proposed funding on projected results has not been clear. Internal resource allocation decisions have been guided by desired performance levels for individual surveys and the program is making progress in tying budget to performance goals. The program has begun to integrate the budget and performance presentation for annual requests. Budget requests are tied more generally to maintaining the scope of surveys and improving their timeliness.

Evidence: Evidence includes the annual budget submissions to OMB and the Congress. NCHS' FY 2005 budget justification showed progress in tying the request for resources to specific program activities. Additional work is needed to integrate the presentation of resources with the accomplishment of performance goals.

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: For deficiencies identified in this section, the program is taking steps to explicitly tie accomplishment of annual and long-term performance goals to budget requests.

Evidence: Evidence includes progress in tying the accomplishment of annual and long-term performance goals to budget requests as seen in changes to the annual budget submissions to OMB and the Congress and documentation from the agency level on a new process for ceiling letters and other initiatives to advance budget and performance integration.

If applicable, does the program assess and compare the potential benefits of efforts within the program and (if relevant) to other efforts in other programs that have similar goals?

Explanation: On an ongoing and unstructured basis, the program compares the benefits of its approaches and efforts with alternatives within the program and with other external statistical agencies. For example, the program measures its relative potential benefits by benchmarking methods and performance with other organizations through the Interagency Council on Federal Statistics, Interagency Forum on Child and Family Statistics and Interagency Forum on Aging-Related Statistics, and HHS Data Council. The program also compares its approaches to the Health Division of Statistics Canada through annual meetings and by conducting a joint US-Canada survey to compare and learn from methodological differences. The program also makes regular adjustments to its methodology and approaches through the development and consideration of new alternatives. NCHS reviews priority programs including NHANES, NHIS and others.

Evidence: Evidence includes descriptions of changes in methodology over time among a variety of alternatives, such as with NHANES, and information on the data councils and joint survey.

Does the program use a prioritization process to guide budget requests and funding decisions?

Explanation: The program has a process for identifying priorities that it uses in decision-making and has a current set of priorities that it uses in making resource allocation decisions. NCHS has a process where components submit funding needs to the NCHS Director to identify needs to meet specific performance objectives. Beginning in FY 2005, the requests are tied to new performance measures on timeliness, quality and content. NCHS requires program components to provide details on objectives, project plans, funding requirements, and the link to aggregate NCHS objectives. NCHS uses the information to identify funding needs and priorities and to tie spending to performance measures. Each component develops performance objectives for the budget year and highlights how those objectives contribute to NCHS performance measures.

Evidence: Evidence includes FY 2005 Project Proposal Submissions, FY 2005 proposed projects and priority designations, and NCHS budget process guidance for FY 2005.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: The program collects data on performance and the performance of its partners and uses the data to inform resource and program decisions. The information is focused most heavily on timeliness and quality of survey data. The information is used most commonly to improve procedures and policies for surveys. The program has examples in multiple areas for how performance information is used to improve surveys and make minor adjustments in survey funding.

Evidence: NHANES employs a continuous quality improvement effort to reduce release time frames, ensure survey response rates and make other changes. NHIS collects performance information from partners, including Census Bureau field operations and users of the data, to evaluate how well contractors are performing and the timliness and availability of the data. The 2004 National Nursing Home Survey identified and addressed a potential response rate problem by working with interviewers. During the National Survey of Family Growth, the program modified the protocol and increased incentives to improve response rates that were found to be too low during continous monitoring. Vital Statistics is reviewing factors that contribute to the delay between release of preliminary data and issuance of final data, including internal program factors and coding issues at the State level.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: CDC has performance contracts for all SES employees, including the NCHS Director, Deputy Director and Associate Director for Planning, Budget, and Legislation. The plans link individual performance to the attainment of agency-specific and Department-wide goals. NCHS specific goals include elements of timeliness, quality and scope. The CDC Director's performance plan also identifies measurable objectives that the Director is accountable for during the performance review period. CDC is working to cascade performance measures to other employees. The program uses contracts for most of its funding arrangements and keeps those partners accountable for cost, schedule and performance results. Some activities, such as Vital Statistics, rely more heavily on voluntary compliance from program partners, however, the program does negotiate funding and deliverables for the five-year state vital statistics contractors.

Evidence: Field work for NCHS' largest surveys is conducted by commercial survey firms under competitively bid contracts. The program recompetes the contracts periodically to improve efficiency and management. For example, recent competitions were performed on the National Nursing Home Study in FY 2002, NHANES in FY 2003, Slaits and NIS in FY 2004. The SES performance plan includes as a criteria for appraisal the implementation of actions to improve timeliness of data, which is one of the main areas of performance measurement, and the reduction of support positions to improve program efficiency. Evidence also includes the program's contract for NHANES, which includes specific deliverables and timeframes for the duration of the performance period.

Are funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: Funds for the program are obligated in a timely manner and spent for the intended purpose. At the agency level, CDC consolidated budget execution functions in 2004 into a central office that is now charged with quality assurance and data validation for program execution. The agency approves internal reallocations that vary from spend plans; regularly reviews unliquidated obligations; and established a standard operating procedure for spending plan execution to help ensure program funds are obligated consistently with the overall objective of the program and that allotted funds are fully executed in a timely manner. CDC conducted risk assessments to determine whether specific programs were susceptible to improper payments exceeding $10 million and a 2.5 percent error rate and will estimate improper payments. At the agency level, data validation of commitments is used to help identify whether funds are committed for the alloted purpose and done correctly to track with budget and accounting systems.

Evidence: Evidence includes operating procedures of the budget execution branch at CDC, sample data validation reports, the budget execution standard operating procedures, agency procedures for development and submission of annual spending plans, budget execution spending plans and obligation reports for NCHS, CDC's February 2005 submission for risk assessments under the Improper Payments Information Act. The spending plans are to be used to certify and monitor the status of funds at the program and agency level.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: The program has management procedures in place to be more efficient in program execution. The program has developed an efficiency measure that uses a timing target. The program regularly reviews progress on improving timeliness as an indicator of success.

Evidence: The program has de-layered its management structure by achieving higher supervisory ratios and eliminating organizational units. The program has participated in agency level competitive sourcing studies for the program's office automation and printing activities. The program's efficiency measure is to reduce the number of months for release of data, the time from end of data collection to the first significant data release on the internet, by 25 percent by 2010. CDC has an aggressive business services improvement agenda to consolidate services and make the agency more efficient. CDC is revamping IT across the agency and projects major savings in FTE and funding.

Does the program collaborate and coordinate effectively with related programs?

Explanation: A significant portion of NCHS' work is done for other agencies within HHS, non-Federal sources, and from other Federal agencies. Internally, the program is active in helping establish and track CDC's new strategic planning efforts across multiple program areas. NCHS coordinates with members of HHS' Data Council, the National Committee on Vital and Health Statistics, and the Interagency Council on Statistical Policy. Through the Reimbursable Work Program, NCHS works on the direct needs of other partners by providing information on their topics and issues using NCHS' data collection mechanisms. Partners include other areas of CDC, NIH, the Food and Drug Administration, the Health Resources and Services Administration, the Environmental Protection Agency, and the Departments of Housing and Urban Development, Agriculture and Energy. There may be areas of needed improvement. For example, NCVHS members have recommended the program consider partnering with other Federal agencies, such as the Department of Veterans Affairs.

Evidence: Evidence includes a collection of partnerships across NCHS with other CDC programs, other HHS agencies, and other Federal and non-Federal partners and copies of interagency agreements. The program has nearly 280 reimbursable agreements. The integration of the NHIS and the Agency for Health Care Quality and Research's Medical Expenditure Panel Survey is a good example of how NCHS enables partners to make use of their infrastructure while also extending their accomplishments.

Does the program use strong financial management practices?

Explanation: CDC recently underwent a major effort to bring on the new Unified Financial Management System that is intended to reconcile any remaining weaknesses in this area. The system and CDC's associated process changes are to provide more real time data, streamlined processes, absolute funds control, and improved monitoring. UFMS automated funds control has enabled tracking of commitments, including aged balances, to provide better management information; improved financial planning using commitment data; produced tracking of current data on obligations to date, commitments and balances to provide information on spending actions, trends, plans, and the resulting impact on remaining funds availability; and provided historical data on all commitments, undelivered orders, payables, and payment transactions. Continued success will require positive documentation from independent auditors that indicates the new system has resolved weaknesses. Prior Performance and Accountability Reports noted continued weaknesses with CDC's financial systems, including a material weakness.

Evidence: Evidence includes error reports and other preliminary financial controls data from CDC's initial experience with fully implementing the Unified Financial Management System. CDC also automated reimbursable billings, enhanced year end closing transactions, implemented a new indirect cost methodology, and addressed staff needs. A December 2003 report by the OIG noted the agency had not implemented a system to allocate indirect costs until FY 2003, but found the new system to be a significant improvement for equity and accuracy. The OIG recommends CDC periodically review indirect costing methods. CDC has received five consecutive unqualified opinions. CDC issued 64 duplicate or erroneous payments in FY 2002, or 0.042 percent of all payments and has a 97 percent compliance rate for prompt payments. In November 2000, GAO (GAO-01-40) reported the agency's financial management capacity systems and procedures were insufficiently developed to address the agency's mission and budget growth.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: The program has not scored no's in this section and the weighting for this question is adjusted to zero. In areas where further improvement is warranted, the agency is extending the incorporation of performance measures into employee evaluations and work contracts. The agency is also putting considerable effort into setting priorities and reorganizing operations through the Future's Initiative, including to improve CDC's business practices. The agency will continue to monitor the performance of a new financial management system and the impact the system has on resolving prior weaknesses. NCHS is also engaging its Board of Scientific Counselors to assist with ongoing evaluation of survey programs, which will provide valuable input into priority decisions on funding.

Evidence: CDC implemented UFMS in April of 2005. The system is intended to correct the agency's prior weaknesses in financial management. The FY 2003 PAR cites improvements in preparing financial statements. The agency submitted financial statements to the Department ahead of schedule. The Director, Deputy Director, Associate Director for Planning, Budget and Legislation, and the Associate Director for Management and Operations meet regularly to review the status of performance goals, program operations, and other issues to ensure deficiencies are identified and issues are addressed in a timely basis.

For R&D programs other than competitive grants programs, does the program allocate funds and use management processes that maintain program quality?

Explanation: The program allocates funding following review for scientific and technical merit. Most funding is not devoted to individual intramural or extramural research projects. The program's research effort is data collection and analysis through a series of surveys. Funding is provided primarily for field operations of data systems. The program uses methods for the distribution of resources, namely survey funding levels and questionnaire time, that mimic those of competitive research review as much as possible. For scientific and technical merit, the program uses peer-review of many research and analytic reports; review of methods and effectiveness by the independent NCHS Board of Scientific Counselors; review of NCHS budget and research priorities by the HHS Data Council; and reviews of proposals for inclusion in surveys from reimburseable agreements. The program can justify the unique capabilities of each project performer, such as the US Census, that is allocated funds.

Evidence: Funding is not allocated by Congressional earmarks. The program solicits and scores proposals for NHANES based on published criteria for inclusion in the survey. Proposals for use of NHANES specimen bank (surplus sera from previous NHANES studies) and for DNA specimens from previous NHANES cycles are solicited and reviewed using a process modeled on the NIH grant review processes. The program has conducted meetings in a similar format with external data users, collaborators, and content experts for the national healthcare surveys. NCHS also convened a panel of experts to evaluate and recommend updates to the HHS-national standard certificates to ensure the limited contents of the birth and death certificates were devoted to data items that met defined, targeted user needs. US Census has a unique capability in its expertise and reach.

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: The program developed a new long-term outcome measure that focuses on the amount of time it takes to release survey data to the public after collection. A second measure is the percentage of key data users and policy makers, including reimbursable collaborators, that are satisfied with data quality and relevance. A small extent reflects progress on the first of the two measures. The baseline for the second measure is under development.

Evidence: The program overall has reduced the time for data release from 14.5 months in 2002 to 13.8 months in 2004. NHANES has reduced the timeframe for data release from 3 1/2 years for NHANES II (1988-1994) to 16 months for the 2001-2002 data. Data are not yet available for tracking the second measure of satisfaction of key data users and policy makers.

Does the program (including program partners) achieve its annual performance goals?

Explanation: The program developed new annual performance measures, including an efficiency measure. A Yes reflects progress on all three annual measures.

Evidence: The program overall has reduced the time for data release from 14.5 months in 2002 to 13.8 months in 2004. The program produced 10, 21 and 36 new or revised charts and tables and major methodological changes in Health, United States in 2003-2005, respectively. The program introduced 7, 6, 7 and 5 improved user tools and technologies and in 2002-2005, respectively, and increased web access to the data from 3.4 million visits in 2002 to an estimated 5.6 million in 2005.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: The program is assessed as a large extent because the NCHS has an efficiency measure and has achieved administrative savings, such as a reduction in the number of administrative management and support, but does not have substantial data to quantify savings. The program has also abolished administrative sections and improved the supervisory ratio.

Evidence: The program abolished 24 organizational units by increasing the span of control and organizational structure of remaining branches. The program has obtained a supervisory ration of 1:13, up from 1:10. The program has reduced administrative management and support FTE by 24 percent. CDC consolidated budget execution functions into the financial management office and budget formulation across the agency. CDC is consolidating graphics/web design services and professional training to provide services centrally. NCHS consolidated the administrative officer function.

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: The program is assessed as a yes because the program has data that show NCHS activities compare favorably to similar data collection efforts in select areas such as survey response rates and multiple examples of where organizations adopt NCHS methods and approaches. The program is frequently cited in the Principles and Practices for a Federal Statistical Agency; the sampling frame of NHIS was viewed favorably and is now used for MEPS; surveys adopt questions developed for NCHS to benchmark results; multiple states have replicated the encounter form of the National Ambulatory Medical Care Survey; multiple organizations seek assistance from NCHS to develop questions or forms; NCHS' tools for coding and classification of causes of death are now routinely used in all States and in English-speaking countries; NCHS developed techniques for bridging between the old and 1997 OMB race-ethnicity classifications are widely used. There have been no comprehensive evaluations that allow a comparison of the program with similar programs, but the program has data that allow a comparison.

Evidence: NCHS is a Federal statistical agency and is highly regarded for its methodology and reliability. Specific evidence for this answer is limited to response rates for surveys, which are an indicator of survey quality. NCHS obtains 75 percent and 55 percent to 61 percent response rates on the National Immunization Survey and SLAITS telephone surveys, respectively. Similar federal surveys achieved response rates between 45 percent and 59 percent. The National Health Interview Survey response rate is 90 percent, compared to 59 percent for a related National Survey of America's Families. NCHS' Ambulatory Medical Care Survey achieved a response rate of 71 percent in 2002, compared to 50 percent for the Patient Care Physician Survey. Additional examples of the program being used as a leading resource include New York City and Statistics Canada have emulated protocols and procedures from NHANES; NYC and the National Children's Study have adopted NHANES data management and processing software; multiple other agencies model their confidentiality procedures on the NCHS Staff Manual on Confidentiality.

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: The program is assessed a small extent because, while multiple PHS Evaluation studies have been conducted, there is relatively little evidence from those efforts related to the ultimate effectiveness and impact of the program. Available findings indicate strengths in some areas and, as intended from the evaluation designs, areas of potential improvement. Evaluations from the BSC should be available at a later date.

Evidence: An evaluation of NCHS health care statistics surveys conducted through an NCHS contract found publications using these surveys used them primarily for descriptive statistics to tell a story about trends in visits and service utilization and do not use the data to test hypotheses or conduct policy analysis. A customer satisfication study conducted by a consulting firm found high levels of familiarity and use of NCHS health care statistics, with higher use levels among policymakers and analysts than healthcare providers, but some difficulty in obtaining reports. A study of drug data collection methods for the National Hospital Discharge Survey found problems in abstracting, but focused on considerations and next steps.