These Frequently Asked Questions are intended to help agency staff, OMB staff, and the public have a more thorough understanding of the Program Assessment Rating Tool (PART). The most complete source for PART information is the 2006 PART guidance available at:
Many of the FAQ answers refer to the relevant sections in that guidance. If you have additional questions, please send them to firstname.lastname@example.org and a member of OMB’s Performance Evaluation Team will respond to your inquiry.
Purpose of the PART
The PART was developed to assess and improve program performance so that the Federal government can achieve better results. A PART review helps identify a program’s strengths and weaknesses to inform funding and management decisions aimed at making the program more effective. The PART therefore looks at all factors that affect and reflect program performance including program purpose and design; performance measurement, evaluations, and strategic planning; program management; and program results. Because the PART includes a consistent series of analytical questions, it allows programs to show improvements over time, and allows comparisons between similar programs.
A program is an activity or set of activities intended to help achieve a particular outcome for the public. A program would therefore generally be recognized by the Executive Branch and the Congress when making budget or other decisions. When defining programs for the purpose of completing a PART, it may be particularly important to ensure that programs comprised of multiple activities share common performance goals and are managed as one entity. Because the nature of programs vary dramatically across the Federal government, agencies and OMB have a great deal of flexibility in defining what a program is for a PART review.
Programs in the federal government vary greatly in size. For example, Medicare (over $325 billion) and the Consumer Product Safety Commission ($60 million) are both federal programs with which the public is generally familiar. Congress authorizes and funds programs of varying sizes, depending on what they aim to achieve and the relative role of the federal government in achieving that goal.
Agencies and OMB work together to decide which programs will be reviewed using the PART for the coming year. Generally this is done as part of the preparation of the budget. For example, as the FY 2005 Budget was being finalized, OMB and agencies were determining which programs were to have a PART review in 2006. A number of factors may be taken into considering in deciding when to complete a PART analysis such as authorization cycles, efforts to develop common goals among similar programs, availability of data, and staff workload.
Agency and OMB staff work together to fill out the PART. Agency staff often complete the first draft of the PART, although sometimes OMB staff carry out that task. Within agencies, PARTs are usually completed by budget office and/or program office staff. In all cases, the drafts are based on a wide variety of information including input from program staff, financial management staff, evaluations, Inspector General reports, General Accounting Office reports, congressional inquiries, etc. Once a first draft is written, OMB and agency staff review the answers and resolve differences. OMB is responsible for editing the document into final form, and making it available to the public on the internet and in the Budget.
If you are new to the PART, it is essential first to review the PART guidance (www.whitehouse.gov/omb/performance/2006_part_guidance.pdf ) to become familiar with the intent of the questions and the required evidence. To complete the PART, you will first need to decide which of which of the seven versions of the PART you will use (see FAQ #7), whether all the PART questions relate to your program, and whether the question weights should be changed (see PART Guidance, p.2). Usually the areas of the PART that require attention first include ensuring that the program has strong performance measures, and that data are available to answer PART questions and assess performance. It’s a good strategy to review performance measures and availability of data even before a program is undergoing a PART review since it can take of few years to gather baseline and performance data. (See PART Guidance pp. 6-10 for more information on performance measures)
Seven different PART worksheets are available reflecting the fact that Federal programs deliver goods and services using different mechanisms (see Part Guidance, p. 12). The seven PART types are Direct Federal, Competitive Grant, Block/Formula Grant, Research and Development, Capital Assets and Acquisition, Credit, and Regulatory. Each type of PART includes the 25 basic questions that comprise the Direct Federal PART, plus selected questions tailored to that program type to help further flesh out the PART analysis. Most programs fall primarily into one of the seven types.
The PART types apply to both discretionary and mandatory programs. For example, both Medicare and many education programs are formula/block grants even though one is mandatory and the other discretionary.
Some programs use more than one mechanism to achieve their goals and therefore have components addressed by more than one type of PART. For example, a competitive grant program may also have a significant regulatory component or a research and development program may be implemented through competitive grants. In those cases, choose the PART type that reflects the bulk of what the program does, then add relevant questions from other types of PARTs to create a “mixed” PART. The OMB examiner should consult with a Performance Evaluation Team member if considering using a mixed PART. The agency and OMB must both agree to this approach and on which questions to add. The Research and Development PART has already addressed this issue by including questions from the Competitive Grants PART, to be used when relevant. (See FAQ #40 on adjusting the PART worksheet to create a mixed PART.)
The PART aims to improve the performance of federal programs. Administrative functions themselves usually do not constitute a program. To the extent that administrative activities support particular programs, their costs will eventually be allocated to those programs, if they haven’t already been, as part of full cost budgeting. Therefore, in most cases, administrative functions relevant to individual programs, such as financial management or program monitoring, will be addressed when doing a PART on that program. Centralized administrative functions are generally evaluated using the President’s Management Agenda scorecard. However, there may be cases when an agency chooses to assess administrative functions using the PART, in particular when a Federal program’s services are primarily administrative in nature; for example, separate PARTs have been done on some General Services Administration programs and loan programs.
The PART and GPRA
The Government Performance and Results Act (GPRA) is a law that guides how agencies prepare strategic plans, performance plans, and performance reports that set goals and report on achieving them. The PART is a method for assessing program performance and how the program achieves goals. The PART reinforces the ambitious outcome-oriented performance measurement framework developed under GPRA. Also, the PART builds on GPRA by encouraging agencies to integrate operational decisions with strategic and performance planning. The PART can play an important role in improving performance measurement when existing measures are not outcome-oriented or sufficiently ambitious. Performance measures in GPRA plans and reports and those developed or revised through the PART process must be consistent. (See PART Guidance p. 3)
An outcome refers to the events or conditions of direct importance to the public/beneficiary that are external to the program. An outcome answers the question “What is the program’s goal or purpose?” For example, the goal of a job training program is to give someone the skills to find a job, as opposed to giving out a grant. An outcome measure may be the number and percent of people employed within six months of completing the job training program.
An output refers to the internal activities of a program (e.g., the products or services delivered). The output answers the question “What does the program do to achieve its goal or purpose?” For example, a job training program may provide a class to teach someone the skills necessary to find a job. An output measure may be the number of people who complete a job training program.
Together, outcome, output, and efficiency measures should tell a comprehensive story of program performance. For more information, please see PART guidance pp. 6-10 and “Performance Measurement Challenges and Strategies” (http:///www.whitehouse.gov/omb/performance/challenges_strategies.html).
An acceptable efficiency measure captures a program’s ability to implement its activities and achieve results (an outcome or output), relative to resources (an input such as cost and/or time). The best kind of efficiency measure addresses the cost of achieving a unit of outcome. Efficiency measures must be useful, relevant to program purpose, and help improve program performance. (see PART Guidance, pp. 6-10 and for examples of efficiency measures, please see www.whitehouse.gov/omb/performance/performance_measure_examples.pdf)
Yes, every program should have an efficiency measure or be in the process of developing one. Programs are required to have an annual efficiency measure to receive credit on the PART (question 2.3). Programs can receive credit for efficiency measures that are under development and may also receive credit for efficiency measures that are longer-term in scope. Although not required, programs are encouraged to develop outcome-based efficiency measures. (See PART guidance, pp. 6-10 and PART guidance on PART questions 2.1 and 2.3, pp. 18-22.)
A performance measure is “meaningful” if it measures the right thing – usually the outcome the program is intended to achieve. In assessing performance measures it is helpful to start with the definition of the program’s purpose and the strategic goal(s) to which the program contributes. Performance measures should be relevant to the program, and therefore capture the most important aspects of a program’s mission and priorities. Meaningful measures will be useful for the program partners, stakeholders, and citizens. Although it is tempting to design measures around existing data, those are not always the most meaningful.
No. There is no standard threshold or percent increase that is required for a target to be considered “ambitious.” Every program will be different given the context in which it operates and its current baseline. An “ambitious” target is one that promotes continued improvement, not just increases. For example, sometimes data collection for a program will indicate an increase in desired results due to demographic or other changes outside of the program. An ambitious target should strive for improved performance above and beyond these shifts. It is sometimes helpful to benchmark a program’s targets against similar programs or against its own historical trends.
Performance targets should be set assuming current budget level, allowing for reasonable changes in the future (where they can be anticipated). Agencies should not set targets that are not achievable given current program characteristics, or that would require significant increases in funding. It is anticipated that having good performance information will assist agencies in developing and justifying their budget requests. Agencies must be able to adjust their targets based on actual, enacted budgets.
First, consult the PART Guidance, which is available at /omb/performance/2006_part_guidance.pdf. If the guidance does not adequately address the area of disagreement, agencies and OMB examiners should consult with members of the Performance Evaluation Team (PET) for additional assistance. Agencies and OMB examiners are encouraged to first consult with the PET member in their respective division. Additional guidance may be issued if the PET determines that the PART question results in significant confusion on the question’s intent, criteria for “yes/no”, standards for evidence, etc. Agencies and OMB examiners will be notified if such action is taken.
The process for resolving disagreements relating to PART answers parallels that used in resolving budget disputes. Every effort should be made to resolve issues at the staff level working through the examiner, Branch Chief, and Deputy Associate Director. If issues remain unresolved, no later than June 23rd, agencies should identify programs and particular questions where there is disagreement. They should provide a brief explanation as to why there is sufficient evidence to warrant a change on a PART answer and submit it to OMB. Final decisions will be made no later than June 30th.
The goal of the consistency check is to ensure that the standards in the PART guidance are applied consistently. A team will review a sample of completed PARTs to see if there are discrepancies in interpretation. The reviewers also look to see that the evidentiary standards for a “yes/no” answer are being met.
If discrepancies are identified, OMB will provide an explanation of how the problem should be solved to ensure consistency and/or issue clarifications of PART guidance, as appropriate. Based on the results of the consistency check, all completed PARTs will be reviewed and some may need to be revised to address the findings of the consistency check.
PARTs are completed in the spring so that agencies have the opportunity to use the PART to inform their budget process. In addition, as discussed in FAQ #19, the consistency check needs to be completed with enough time to make any necessary revisions to the PART. In November and early December the PART database will be open for limited updates agreed to between agencies and OMB so that the information included in the PART can be more timely when we release the budget. For example, agencies and OMB may agree to update the PART for new performance measures and performance data included in the FY 2006 budget.
PART assessments should be based on the most recent data available. For PARTs completed for the FY 2006 budget, the most recent data will likely be from FY 2003, although there may be exceptions to this. Agencies should consult with their OMB examiner to explain time lags in performance reporting.
In regard to program evaluations, the PART requires that evaluations be conducted on a regular basis, such as every two to five years, or whatever time schedule is reasonable based on the specific program and its mission and goals, in order to be timely.
In general, for an evaluation to be independent it should be conducted by a group which is separate from those who administer the program. (see PART Guidance, p. 23-26)
What if the agency or program has its own evaluation office?
If the agency has an evaluation office that is an administratively separate unit from the program, the office will likely be sufficiently "independent" to conduct an evaluation of the program. An evaluation conducted by the office administering the program would generally not provide the kind of independence needed to ensure that there is no conflict of interest.
What about programs administered by contractors?
If the contractor is administering the program on behalf of the program office, neither the program office overseeing the contractor nor the contractor itself would be considered "independent" for purposes of conducting an evaluation of the program's effectiveness.
What about an Inspector General?
If the inspector general is not responsible for the operation of the program, including the managerial decision-making that determines whether the program is effective or not, and does not report to the administrator of the program, the inspector general probably has sufficient "independence" to conduct an evaluation of the program.
The "randomized controlled trial" represents the best means to assess the impact of programs. However, this approach may not be feasible for all programs. Other potential evaluation methodologies are discussed in the supplementary guidance posted on www.omb.gov/part. Consult an external or internal program evaluation expert, as necessary, to help determine the best type of evaluation for your program. (see PART Guidance pp. 23-26.)
24. How do we determine what is the appropriate method for evaluating the program (PART Question 2.6)? If we cannot demonstrate the program’s “impact”, can we still receive credit for PART Question 4.5?
The PART Guidance includes a brief discussion of what constitutes "rigorous evidence" of a program's effectiveness and references several documents available over the internet that you may find helpful. Also, additional guidance has been developed and is posted on www.omb.gov/part.
The key question to answer is whether there is a form of evaluation that can provide strong evidence of a program's impact. If not, then consider the strongest evaluation possible that sheds light on the program's impact--even if only answering how or why a program is effective (or ineffective). The bottom line, though, is that the evaluation must be appropriate to the type of program. When there is doubt, agencies (and OMB) should consult program evaluation experts to help identify the most feasible and informative evaluation for a program.
If it is not possible to demonstrate impact, you can still receive credit for Question 4.5 if there are appropriate measures in place to show effectiveness. Credit can be given for evaluations that help determine how or why a program is effective (or ineffective). Again, consult an evaluation expert to help make this determination. (see PART Guidance pp. 23-26 and pp. 47-48)
Meaning and Use of PART Ratings
Not necessarily. The PART score in and of itself does not determine funding recommendations. The detailed PART findings, rather than the program’s rating by itself, inform budget recommendations. To the extent that the obstacles to better performance are budgetary, and the program can demonstrate the magnitude of results at different levels of funding, a low score could lead to an increase in funding. However, if the program’s problems are prohibitively difficult or costly to fix and/or better results could be achieved through a different program, then the low-scoring program’s funding may be reallocated.
Because a number of factors contribute to a program’s requested funding level, a good rating does not guarantee a specific level of funding. Programs may be effective but have completed their mission, be unnecessarily duplicative of other programs, or may not be the highest priority for the Administration. For example, despite the Department of Energy's Distributed Energy Resources Program's "Moderately Effective" rating, the Administration proposed a small reduction in funding for the program in the FY 2005 Budget. The program funds research for improved energy efficiency of and reduced emissions from on-site energy production. The 13 percent decrease in funding is attributable not to the program's rating, but to relative priorities among Department of Energy programs.
Of the 399 programs that have been assessed by PART, the distribution of scores is as follows:
Numerical scores are a measurement of the distance that a program has come or needs to go in performing to 100% of standards for all aspects that determine effectiveness. However, numerical scores are not so precise as to be able to reliably compare differences of a few points among different programs. That is why overall numerical scores are not published; they are rather used as a guide to determine qualitative ratings that are more generally comparable across programs.
Numerical scores for each section of a PART help to focus on where a program needs the most improvement, whether in design and purpose, planning, management, or results/accountability. For an individual program, an increase from 78% to 84% overall from one year to the next would generally indicate that the program has made some kind of improvement.
PART scores are translated into qualitative ratings as described below. While the Administration hopes that, over time, increasing numbers of programs will fall into the “Effective” category, we also stress that the PART recommendations are even more important than the PART ratings. The focus of the PART is more on the continuous improvement of program performance. Programs with scores that fall very close to the break points will receive close attention to ensure that their rating is truly representative of the program. The bands and associated ratings are as follows:
Regardless of overall score, a rating of Results Not Demonstrated is given when programs do not have agreed-upon performance measures or lack baselines and performance data. Please see FAQ # 33 for more details.
The detailed answers for every PART assessment as well as summary reports which include the major recommendations are available at: /omb/budget/fy2005/part.html.
The numerical scores for each section of the PART and the overall qualitative rating (Effective, Moderately Effective, etc.) for each program are included with this information.
Of course, the goal is for all programs to be moving in the direction of being effective and able to measure results. Although all programs are expected to follow-up on PART findings, programs rated Results Not Demonstrated (RND) will receive the greatest pressure to implement recommendations that will enable them to illustrate their level of effectiveness. Specifically, in order to get to yellow in status on the Budget and Performance Integration initiative scorecard, agencies must not have more than 50% of their programs rated RND more than two years in a row. To get to green in status, agencies must not have more than 10% of their programs rated RND more than two years in a row.
32. Why does the PART ask questions about a program’s purpose and design, which is determined by statute, not the agency that manages the program? Why should agencies get a low score for something that is out of our control?
The purpose of the PART is to improve program effectiveness, so the tool is designed to assess all elements that impact the effectiveness of the program, even if they are out of the direct control of the program manager. If a program manager’s ability to achieve results is hindered by an unclear purpose or suboptimal design, it should be useful to the manager for such obstacles to be highlighted. A systematic assessment that highlights these issues will help program managers communicate with those who more directly control those elements and who also care about the outcomes of the program. If statutory provisions impede effectiveness – for example, if resources are not targeted effectively or spread too thinly to have an impact – one result of a PART review may be recommendations for legislative changes.
Results Not Demonstrated
A rating of RND means that a program does not have sufficient performance measurement or performance information to show results, and therefore it is not possible to assess whether it has achieved its goals. RND is not on the same scale as the ratings “Ineffective”, “Adequate”, “Moderately Effective”, or “Effective”, each of which indicate there is evidence of a certain level of program performance. Because the PART employs an evidence-based analysis, the PART’s first goal is to help ensure that Federal programs have a basis on which to assess their effectiveness, to the extent that the collection of data and impact information are feasible and cost effective. Therefore, moving programs out of an RND rating is a high priority (also see FAQ #31).
34. How long do we have to be in Results Not Demonstrated (RND)? Our program received an RND rating in the 2005 budget, but we developed new performance measures and are developing baseline data this year. Are new measures and baseline data enough to warrant re-rating the program or do we have to wait until we achieve actual performance results?
A program that has not been able to establish long-term and short-term performance measures or does not have data to indicate how it has been performing under measures that have been established will receive a rating of Results Not Demonstrated. Programs that develop new measures may receive a “Yes” to 2.1 and 2.3, but will have to wait to receive a “Yes” on questions 4.1 and 4.2 until they have data and are able to demonstrate results.
However, a program that has adopted new measures, established baselines, set targets, and begun to collect this information from grantees could receive a rating higher than “No” for PART questions 4.1 and 4.2. Other considerations such as the involvement and commitment of program partners to these measures are important elements of questions 4.1 and 4.2.
In many ways, the most important aspect of the PART is the set of budget, management, policy, or legislative recommendations that grow out of a PART review. These recommendations are intended to link budgeting and performance and to create a cycle of continuous program improvement in order to help programs reach their goals more effectively. In the first two years of the PART, OMB developed recommendations based on completed PARTs, then worked with agencies to come to agreement on them. In the FY 2006 Budget cycle, PARTs will be completed much earlier so agencies will be able to include PART recommendations as a component of their Budget submissions to OMB. The agreed-upon recommendations will be included in the President’s 2006 Budget.
In general, agencies are responsible for implementing PART recommendations since they manage the programs. OMB provides oversight and assistance as appropriate. More specifically, a new feature of the PART identifies the key agency officials accountable for ensuring implementation of the recommendations. In addition, since the PART is an aspect of the President’s Budget and Performance Integration (BPI) initiative, the BPI scorecard holds agencies accountable for utilizing the PART to improve program performance. Agencies are developing systems to track the implementation of PART recommendations and they report on PART-related program improvements in their budget submissions to OMB.
Programs have the opportunity to be reassessed each year. There is no limit to the number of times a program can be reassessed. However, programs should only be reassessed when significant changes have been made to improve the rating of the program. For example, programs might be reassessed when new performance measures are agreed upon, PART recommendations have been implemented, new performance data have been compiled, or a program evaluation has been completed. To keep workloads reasonable, only the questions with significant new information should be re-opened.
While some changes have been made to improve the consistency and accuracy of the PART ratings, they are not significant enough to prevent comparisons between years or show changes over time. The guidance and a few PART questions changed from 2004 to 2005. For the 2006 PART, the questions remained the same as 2005, but changes were made to the PART guidance.
Congress is an important partner in the PART process. The Administration is inviting Members of Congress and other stakeholders to review the PART details and provide feedback to OMB on the conclusions OMB and agencies have made in answers to PART questions. In addition to meetings with individual Members, the Administration has sponsored numerous briefings for committee and subcommittee staff on the PART. We have also brought agency officials and Congressional staff together to discuss the particular challenges associated with integrating performance information into the budget process. The Administration will also provide the list of programs that will be PARTed in preparation for the FY 2006 Budget and continue to actively solicit input to the PART process from Members of Congress and their staffs. To further support the use of the PART in congressional deliberations, agencies will include relevant PART findings in congressional justifications and testimony. In general, because the PART is an evidence-based, objective analysis, the Administration believes it is a useful tool in budget and policy formulation for all branches of government.
PART Worksheets and Technical Issues
The type-specific questions are in hidden rows at the bottom of each section. Unhide the rows, re-hide those you don’t need (make sure they have no answer and no weight), then re-weight all of the remaining questions in the section.
If you paste the weight into a block of rows at once, the weight will be pasted into any hidden rows in that block as well. So you need to make sure the rows you want hidden have zero weight, then enter the weight individually into each row you want to display.
The most thorough and readily available resource is the PART guidance available at www.whitehouse.gov/omb/performance/2006_part_guidance.pdf. It provides an explanation of the purpose of each PART question, standards of evidence, and some background and suggested resources on performance measurement and program evaluation. At the same web address are also papers on performance measurement challenges and examples of good performance measures. Your OMB examiner will also be able to answer many of your PART-related questions, in particular those specific to your program. And finally, members of the Performance Evaluation Team (PET) are available to help interpret the PART guidance, resolve disagreements, and provide examples of programs that may be facing similar challenges or issues. The PET conducts training each year in the use of the PART. In 2004, they were held in March and over 1,000 people were trained.
It is likely that you can find programs that perform well and scored well on the PART within your agency. Ask your Budget and Performance Integration contact for the names of program managers whose programs scored well on the PART. There is a comprehensive list of completed PARTs and their summaries organized by agency on the OMB website. In addition, there is a PART database that includes program ratings, PART type, and funding levels. The web address for these documents is www.whitehouse.gov/omb/budget/fy2005/part.html. A descriptive list of exemplary performance measures is available at www.whitehouse.gov/omb/performance/performance_measure_examples.pdf.