1. What
is the purpose of the PART?
The PART was developed to assess and improve program performance
so that the Federal government can achieve better results. A PART review
helps identify a program’s strengths and weaknesses to inform funding
and management decisions aimed at making the program more effective. The
PART therefore looks at all factors that affect and reflect program performance
including program purpose and design; performance measurement, evaluations,
and strategic planning; program management; and program results. Because
the PART includes a consistent series of analytical questions, it allows
programs to show improvements over time, and allows comparisons between
similar programs.
Defining Programs
2. What
is a “program”?
A program is an activity or set of activities intended
to help achieve a particular outcome for the public. A program would therefore
generally be recognized by the Executive Branch and the Congress when
making budget or other decisions. When defining programs for the purpose
of completing a PART, it may be particularly important to ensure that
programs comprised of multiple activities share common performance goals
and are managed as one entity. Because the nature of programs vary dramatically
across the Federal government, agencies and OMB have a great deal of flexibility
in defining what a program is for a PART review.
3. Why
does the size of the programs assessed
with the PART vary so much?
Programs in the federal government vary greatly in size.
For example, Medicare (over $325 billion) and the Consumer Product Safety
Commission ($60 million) are both federal programs with which the public
is generally familiar. Congress authorizes and funds programs of varying
sizes, depending on what they aim to achieve and the relative role of
the federal government in achieving that goal.
Getting Started
4. Who
determines that a program should be assessed using the PART?
Agencies and OMB work together to decide which programs
will be reviewed using the PART for the coming year. Generally this is
done as part of the preparation of the budget. For example, as the FY
2005 Budget was being finalized, OMB and agencies were determining which
programs were to have a PART review in 2006. A number of factors may be
taken into considering in deciding when to complete a PART analysis such
as authorization cycles, efforts to develop common goals among similar
programs, availability of data, and staff workload.
5. Who
fills out the PART—agency staff or OMB?
Agency and OMB staff work together to fill out the PART.
Agency staff often complete the first draft of the PART, although sometimes
OMB staff carry out that task. Within agencies, PARTs are usually completed
by budget office and/or program office staff. In all cases, the drafts
are based on a wide variety of information including input from program
staff, financial management staff, evaluations, Inspector General reports,
General Accounting Office reports, congressional inquiries, etc. Once
a first draft is written, OMB and agency staff review the answers and
resolve differences. OMB is responsible for editing the document into
final form, and making it available to the public on the internet and
in the Budget.
6. Where
do I start in completing a PART?
If you are new to the PART, it is essential first to
review the PART guidance (www.whitehouse.gov/omb/performance/2006_part_guidance.pdf
) to become familiar with the intent of the questions and the required
evidence. To complete the PART, you will first need to decide which of
which of the seven versions of the PART you will use (see FAQ #7), whether
all the PART questions relate to your program, and whether the question
weights should be changed (see PART Guidance, p.2). Usually the areas
of the PART that require attention first include ensuring that the program
has strong performance measures, and that data are available to answer
PART questions and assess performance. It’s a good strategy to review
performance measures and availability of data even before a program is
undergoing a PART review since it can take of few years to gather baseline
and performance data. (See PART Guidance pp. 6-10 for more information
on performance measures)
7. There
are seven different types of PARTs. How do we know which PART form to
use to assess a given program?
Seven different PART worksheets are available reflecting
the fact that Federal programs deliver goods and services using different
mechanisms (see Part Guidance, p. 12). The seven PART types are Direct
Federal, Competitive Grant, Block/Formula Grant, Research and Development,
Capital Assets and Acquisition, Credit, and Regulatory. Each type of PART
includes the 25 basic questions that comprise the Direct Federal PART,
plus selected questions tailored to that program type to help further
flesh out the PART analysis. Most programs fall primarily into one of
the seven types.
The PART types apply to both discretionary and mandatory
programs. For example, both Medicare and many education programs are formula/block
grants even though one is mandatory and the other discretionary.
8. What
form should I use if a program can be classified in more than one program
area (e.g., competitive grant and research & development)?
Some programs use more than one mechanism to achieve
their goals and therefore have components addressed by more than one type
of PART. For example, a competitive grant program may also have a significant
regulatory component or a research and development program may be implemented
through competitive grants. In those cases, choose the PART type that
reflects the bulk of what the program does, then add relevant questions
from other types of PARTs to create a “mixed” PART. The OMB
examiner should consult with a Performance Evaluation Team member if considering
using a mixed PART. The agency and OMB must both agree to this approach
and on which questions to add. The Research and Development PART has already
addressed this issue by including questions from the Competitive Grants
PART, to be used when relevant. (See FAQ #40 on adjusting the PART worksheet
to create a mixed PART.)
9.
Should I do a PART on administrative functions?
The PART aims to improve the performance of federal programs.
Administrative functions themselves usually do not constitute a program.
To the extent that administrative activities support particular programs,
their costs will eventually be allocated to those programs, if they haven’t
already been, as part of full cost budgeting. Therefore, in most cases,
administrative functions relevant to individual programs, such as financial
management or program monitoring, will be addressed when doing a PART
on that program. Centralized administrative functions are generally evaluated
using the President’s Management Agenda scorecard. However, there
may be cases when an agency chooses to assess administrative functions
using the PART, in particular when a Federal program’s services
are primarily administrative in nature; for example, separate PARTs have
been done on some General Services Administration programs and loan programs.
The PART and GPRA
10. What
is the connection between the PART and GPRA?
The Government Performance and Results Act (GPRA) is
a law that guides how agencies prepare strategic plans, performance plans,
and performance reports that set goals and report on achieving them. The
PART is a method for assessing program performance and how the program
achieves goals. The PART reinforces the ambitious outcome-oriented performance
measurement framework developed under GPRA. Also, the PART builds on GPRA
by encouraging agencies to integrate operational decisions with strategic
and performance planning. The PART can play an important role in improving
performance measurement when existing measures are not outcome-oriented
or sufficiently ambitious. Performance measures in GPRA plans and reports
and those developed or revised through the PART process must be consistent.
(See PART Guidance p. 3)
Performance measurement
11. What
are outcomes and outputs?
An outcome refers to the events or conditions
of direct importance to the public/beneficiary that are external to the
program. An outcome answers the question “What is the program’s
goal or purpose?” For example, the goal of a job training program
is to give someone the skills to find a job, as opposed to giving out
a grant. An outcome measure may be the number and percent of people employed
within six months of completing the job training program.
An output refers to the internal
activities of a program (e.g., the products or services delivered). The
output answers the question “What does the program do to achieve
its goal or purpose?” For example, a job training program may provide
a class to teach someone the skills necessary to find a job. An output
measure may be the number of people who complete a job training program.
Together, outcome, output, and efficiency measures should
tell a comprehensive story of program performance. For more information,
please see PART guidance pp. 6-10 and “Performance Measurement Challenges
and Strategies” (http:///www.whitehouse.gov/omb/performance/challenges_strategies.html).
12. What
is an acceptable efficiency measure?
An acceptable efficiency measure captures a program’s
ability to implement its activities and achieve results (an outcome or
output), relative to resources (an input such as cost and/or time). The
best kind of efficiency measure addresses the cost of achieving a unit
of outcome. Efficiency measures must be useful, relevant to program purpose,
and help improve program performance. (see PART Guidance, pp. 6-10 and
for examples of efficiency measures, please see www.whitehouse.gov/omb/performance/performance_measure_examples.pdf)
13. Do
we have to have an efficiency measure for every program?
Yes, every program should have an efficiency measure
or be in the process of developing one. Programs are required to have
an annual efficiency measure to receive credit on the PART (question 2.3).
Programs can receive credit for efficiency measures that are under development
and may also receive credit for efficiency measures that are longer-term
in scope. Although not required, programs are encouraged to develop outcome-based
efficiency measures. (See PART guidance, pp. 6-10 and PART guidance on
PART questions 2.1 and 2.3, pp. 18-22.)
14.
The PART Guidance says that performance measures must be “meaningful.”
What does that mean?
A performance measure is “meaningful” if
it measures the right thing – usually the outcome the program is
intended to achieve. In assessing performance measures it is helpful to
start with the definition of the program’s purpose and the strategic
goal(s) to which the program contributes. Performance measures should
be relevant to the program, and therefore capture the most important aspects
of a program’s mission and priorities. Meaningful measures will
be useful for the program partners, stakeholders, and citizens. Although
it is tempting to design measures around existing data, those are not
always the most meaningful.
15. To
be considered “ambitious,” is there a certain threshold a
target must meet?
No. There is no standard threshold or percent increase
that is required for a target to be considered “ambitious.”
Every program will be different given the context in which it operates
and its current baseline. An “ambitious” target is one that
promotes continued improvement, not just increases. For example, sometimes
data collection for a program will indicate an increase in desired results
due to demographic or other changes outside of the program. An ambitious
target should strive for improved performance above and beyond these shifts.
It is sometimes helpful to benchmark a program’s targets against
similar programs or against its own historical trends.
16. How
can we set targets if we do not know what our future budgets will be?
Performance targets should be set assuming current budget
level, allowing for reasonable changes in the future (where they can be
anticipated). Agencies should not set targets that are not achievable
given current program characteristics, or that would require significant
increases in funding. It is anticipated that having good performance information
will assist agencies in developing and justifying their budget requests.
Agencies must be able to adjust their targets based on actual, enacted
budgets.
Completing PARTs
17. What
if OMB and agencies disagree about how to interpret a PART question?
First, consult the PART Guidance, which is available
at /omb/performance/2006_part_guidance.pdf.
If the guidance does not adequately address the area of disagreement,
agencies and OMB examiners should consult with members of the Performance
Evaluation Team (PET) for additional assistance. Agencies and OMB examiners
are encouraged to first consult with the PET member in their respective
division. Additional guidance may be issued if the PET determines that
the PART question results in significant confusion on the question’s
intent, criteria for “yes/no”, standards for evidence, etc.
Agencies and OMB examiners will be notified if such action is taken.
18. What
if OMB and agencies disagree on a PART answer?
The process for resolving disagreements relating to PART
answers parallels that used in resolving budget disputes. Every effort
should be made to resolve issues at the staff level working through the
examiner, Branch Chief, and Deputy Associate Director. If issues remain
unresolved, no later than June 23rd, agencies should
identify programs and particular questions where there is disagreement.
They should provide a brief explanation as to why there is sufficient
evidence to warrant a change on a PART answer and submit it to OMB. Final
decisions will be made no later than June 30th.
19. The
timeline says that PARTs must be completed by June 7 for a “consistency
check.” What is that and how does it work?
The goal of the consistency check is to ensure that the
standards in the PART guidance are applied consistently. A team will review
a sample of completed PARTs to see if there are discrepancies in interpretation.
The reviewers also look to see that the evidentiary standards for a “yes/no”
answer are being met.
If discrepancies are identified, OMB will provide an explanation
of how the problem should be solved to ensure consistency and/or issue
clarifications of PART guidance, as appropriate. Based on the results
of the consistency check, all completed PARTs will be reviewed and some
may need to be revised to address the findings of the consistency check.
20.
Why do we have to complete PARTs now when they won’t be released
until next February with the 2006 Budget?
PARTs are completed in the spring so that agencies have
the opportunity to use the PART to inform their budget process. In addition,
as discussed in FAQ #19, the consistency check needs to be completed with
enough time to make any necessary revisions to the PART. In November and
early December the PART database will be open for limited updates agreed
to between agencies and OMB so that the information included in the PART
can be more timely when we release the budget. For example, agencies and
OMB may agree to update the PART for new performance measures and performance
data included in the FY 2006 budget.
21. How
recent does the data have to be?
PART assessments should be based on the most recent data
available. For PARTs completed for the FY 2006 budget, the most recent
data will likely be from FY 2003, although there may be exceptions to
this. Agencies should consult with their OMB examiner to explain time
lags in performance reporting.
In regard to program evaluations, the PART requires that
evaluations be conducted on a regular basis, such as every two to five
years, or whatever time schedule is reasonable based on the specific program
and its mission and goals, in order to be timely.
Program Evaluation
22. What should we consider
when determining whether an office or official is sufficiently “independent”
to conduct a program evaluation (PART Question 2.6)?
In general, for an evaluation to be independent it should
be conducted by a group which is separate from those who administer the
program. (see PART Guidance, p. 23-26)
What if the agency or program has its own evaluation
office?
If the agency has an evaluation office that is an administratively
separate unit from the program, the office will likely be sufficiently
"independent" to conduct an evaluation of the program. An evaluation
conducted by the office administering the program would generally not
provide the kind of independence needed to ensure that there is no conflict
of interest.
What about programs administered by contractors?
If the contractor is administering the program on behalf
of the program office, neither the program office overseeing the contractor
nor the contractor itself would be considered "independent"
for purposes of conducting an evaluation of the program's effectiveness.
What about an Inspector General?
If the inspector general is not responsible for the operation
of the program, including the managerial decision-making that determines
whether the program is effective or not, and does not report to the administrator
of the program, the inspector general probably has sufficient "independence"
to conduct an evaluation of the program.
23. How
do I demonstrate impact through a program evaluation?
The "randomized controlled trial" represents
the best means to assess the impact of programs. However, this approach
may not be feasible for all programs. Other potential evaluation methodologies
are discussed in the supplementary guidance posted on www.omb.gov/part.
Consult an external or internal program evaluation expert, as necessary,
to help determine the best type of evaluation for your program. (see PART
Guidance pp. 23-26.)
24. How
do we determine what is the appropriate method for evaluating the program
(PART Question 2.6)? If we cannot demonstrate the program’s “impact”,
can we still receive credit for PART Question 4.5?
The PART Guidance includes a brief discussion of what
constitutes "rigorous evidence" of a program's effectiveness
and references several documents available over the internet that you
may find helpful. Also, additional guidance has been developed and is
posted on www.omb.gov/part.
The key question to answer is whether there is a form
of evaluation that can provide strong evidence of a program's impact.
If not, then consider the strongest evaluation possible that sheds light
on the program's impact--even if only answering how or why a program is
effective (or ineffective). The bottom line, though, is that the evaluation
must be appropriate to the type of program. When there is doubt, agencies
(and OMB) should consult program evaluation experts to help identify the
most feasible and informative evaluation for a program.
If it is not possible to demonstrate impact, you can still
receive credit for Question 4.5 if there are appropriate measures in place
to show effectiveness. Credit can be given for evaluations that help determine
how or why a program is effective (or ineffective).
Again, consult an evaluation expert to help make this determination. (see
PART Guidance pp. 23-26 and pp. 47-48)
Meaning and Use of PART Ratings
25. Will
a poor PART rating lead OMB to recommend that a program’s funding
be cut?
Not necessarily. The PART score in and of itself does
not determine funding recommendations. The detailed PART findings, rather
than the program’s rating by itself, inform budget recommendations.
To the extent that the obstacles to better performance are budgetary,
and the program can demonstrate the magnitude of results at different
levels of funding, a low score could lead to an increase in funding. However,
if the program’s problems are prohibitively difficult or costly
to fix and/or better results could be achieved through a different program,
then the low-scoring program’s funding may be reallocated.
26. Is
it possible for a program to receive a good rating and get a funding cut?
Because a number of factors contribute to a program’s
requested funding level, a good rating does not guarantee a specific level
of funding. Programs may be effective but have completed their mission,
be unnecessarily duplicative of other programs, or may not be the highest
priority for the Administration. For example, despite the Department of
Energy's Distributed Energy Resources Program's "Moderately Effective"
rating, the Administration proposed a small reduction in funding for the
program in the FY 2005 Budget. The program funds research for improved
energy efficiency of and reduced emissions from on-site energy production.
The 13 percent decrease in funding is attributable not to the program's
rating, but to relative priorities among Department of Energy programs.
27. What
is the distribution of PART ratings to date?
Of the 399 programs that have been assessed by PART,
the distribution of scores is as follows:
Regardless of overall score, a rating of Results Not Demonstrated
is given when programs do not have agreed-upon performance measures or
lack baselines and performance data. Please see FAQ # 33 for more details.
30.
Are PARTs and their recommendations made public?
The detailed answers for every PART assessment as well
as summary reports which include the major recommendations are available
at: /omb/budget/fy2005/part.html.
The numerical scores for each section of the PART and
the overall qualitative rating (Effective, Moderately Effective, etc.)
for each program are included with this information.
31. How
does OMB view different types of PART ratings? (Effective versus Ineffective
versus RND, for example.)
Of course, the goal is for all programs to be moving
in the direction of being effective and able to measure results. Although
all programs are expected to follow-up on PART findings, programs rated
Results Not Demonstrated (RND) will receive the greatest pressure to implement
recommendations that will enable them to illustrate their level of effectiveness.
Specifically, in order to get to yellow in status on the Budget and Performance
Integration initiative scorecard, agencies must not have more than 50%
of their programs rated RND more than two years in a row. To get to green
in status, agencies must not have more than 10% of their programs rated
RND more than two years in a row.
32. Why
does the PART ask questions about a program’s purpose and design,
which is determined by statute, not the agency that manages the program?
Why should agencies get a low score for something that is out of our control?
The purpose of the PART is to improve program effectiveness,
so the tool is designed to assess all elements that impact the effectiveness
of the program, even if they are out of the direct control of
the program manager. If a program manager’s ability to achieve results
is hindered by an unclear purpose or suboptimal design, it should be useful
to the manager for such obstacles to be highlighted. A systematic assessment
that highlights these issues will help program managers communicate with
those who more directly control those elements and who also care about
the outcomes of the program. If statutory provisions impede effectiveness
– for example, if resources are not targeted effectively or spread
too thinly to have an impact – one result of a PART review may be
recommendations for legislative changes.
Results Not Demonstrated
33. What
does “Results Not Demonstrated” (RND) Mean?
A rating of RND means that a program does not have sufficient
performance measurement or performance information to show results, and
therefore it is not possible to assess whether it has achieved its goals.
RND is not on the same scale as the ratings “Ineffective”,
“Adequate”, “Moderately Effective”, or “Effective”,
each of which indicate there is evidence of a certain level of program
performance. Because the PART employs an evidence-based analysis, the
PART’s first goal is to help ensure that Federal programs have a
basis on which to assess their effectiveness, to the extent that the collection
of data and impact information are feasible and cost effective. Therefore,
moving programs out of an RND rating is a high priority (also see FAQ
#31).
34.
How long do we have to be in Results Not Demonstrated (RND)? Our program
received an RND rating in the 2005 budget, but we developed new performance
measures and are developing baseline data this year. Are new measures
and baseline data enough to warrant re-rating the program or do we have
to wait until we achieve actual performance results?
A program that has not been able to establish long-term
and short-term performance measures or does not have data to
indicate how it has been performing under measures that have been established
will receive a rating of Results Not Demonstrated. Programs that develop
new measures may receive a “Yes” to 2.1 and 2.3, but will
have to wait to receive a “Yes” on questions 4.1 and 4.2 until
they have data and are able to demonstrate results.
However, a program that has adopted new measures, established
baselines, set targets, and begun to collect this information from grantees
could receive a rating higher than “No” for PART questions
4.1 and 4.2. Other considerations such as the involvement and commitment
of program partners to these measures are important elements of questions
4.1 and 4.2.
PART recommendations
35. Who
determines what the PART recommendations are?
In many ways, the most important aspect of the PART is
the set of budget, management, policy, or legislative recommendations
that grow out of a PART review. These recommendations are intended to
link budgeting and performance and to create a cycle of continuous program
improvement in order to help programs reach their goals more effectively.
In the first two years of the PART, OMB developed recommendations based
on completed PARTs, then worked with agencies to come to agreement on
them. In the FY 2006 Budget cycle, PARTs will be completed much earlier
so agencies will be able to include PART recommendations as a component
of their Budget submissions to OMB. The agreed-upon recommendations will
be included in the President’s 2006 Budget.
36. Who
is accountable for implementing PART recommendations after a program has
had a PART review?
In general, agencies are responsible for implementing
PART recommendations since they manage the programs. OMB provides oversight
and assistance as appropriate. More specifically, a new feature of the
PART identifies the key agency officials accountable for ensuring implementation
of the recommendations. In addition, since the PART is an aspect of the
President’s Budget and Performance Integration (BPI) initiative,
the BPI scorecard holds agencies accountable for utilizing the PART to
improve program performance. Agencies are developing systems to track
the implementation of PART recommendations and they report on PART-related
program improvements in their budget submissions to OMB.
Reassessments
37. How
often and under what circumstances are programs reassessed?
Programs have the opportunity to be reassessed each year.
There is no limit to the number of times a program can be reassessed.
However, programs should only be reassessed when significant changes have
been made to improve the rating of the program. For example, programs
might be reassessed when new performance measures are agreed upon, PART
recommendations have been implemented, new performance data have been
compiled, or a program evaluation has been completed. To keep workloads
reasonable, only the questions with significant new information should
be re-opened.
38. Given
the changes to the guidance and questions, can PART scores and ratings
be compared from one year to the next?
While some changes have been made to improve the consistency
and accuracy of the PART ratings, they are not significant enough to prevent
comparisons between years or show changes over time. The guidance and
a few PART questions changed from 2004 to 2005. For the 2006 PART, the
questions remained the same as 2005, but changes were made to the PART
guidance.
Congress
39.
What is the Administration doing to advance the use of the PART by Congress?
Congress is an important partner in the PART process.
The Administration is inviting Members of Congress and other stakeholders
to review the PART details and provide feedback to OMB on the conclusions
OMB and agencies have made in answers to PART questions. In addition to
meetings with individual Members, the Administration has sponsored numerous
briefings for committee and subcommittee staff on the PART. We have also
brought agency officials and Congressional staff together to discuss the
particular challenges associated with integrating performance information
into the budget process. The Administration will also provide the list
of programs that will be PARTed in preparation for the FY 2006 Budget
and continue to actively solicit input to the PART process from Members
of Congress and their staffs. To further support the use of the PART in
congressional deliberations, agencies will include relevant PART findings
in congressional justifications and testimony. In general, because the
PART is an evidence-based, objective analysis, the Administration believes
it is a useful tool in budget and policy formulation for all branches
of government.
PART Worksheets and Technical Issues
40. How
do I add questions for a mixed PART?
The type-specific questions are in hidden rows at the
bottom of each section. Unhide the rows, re-hide those you don’t
need (make sure they have no answer and no weight), then re-weight all
of the remaining questions in the section.
41. When
I add a type-specific question and change all the weights in the section,
I am getting an error that the section doesn’t add to 100%. What
should I do?
If you paste the weight into a block of rows at once,
the weight will be pasted into any hidden rows in that block as well.
So you need to make sure the rows you want hidden have zero weight, then
enter the weight individually into each row you want to display.
Examples/Resources
42. What
resources are available to help me with the PART?
The most thorough and readily available resource is the
PART guidance available at www.whitehouse.gov/omb/performance/2006_part_guidance.pdf.
It provides an explanation of the purpose of each PART question, standards
of evidence, and some background and suggested resources on performance
measurement and program evaluation. At the same web address are also papers
on performance measurement challenges and examples of good performance
measures. Your OMB examiner will also be able to answer many of your PART-related
questions, in particular those specific to your program. And finally,
members of the Performance Evaluation Team (PET) are available to help
interpret the PART guidance, resolve disagreements, and provide examples
of programs that may be facing similar challenges or issues. The PET conducts
training each year in the use of the PART. In 2004, they were held in
March and over 1,000 people were trained.
43.
Where can we find examples of good programs—programs that have scored
well on the PART?
It is likely that you can find programs that perform
well and scored well on the PART within your agency. Ask your Budget and
Performance Integration contact for the names of program managers whose
programs scored well on the PART. There is a comprehensive list of completed
PARTs and their summaries organized by agency on the OMB website. In addition,
there is a PART database that includes program ratings, PART type, and
funding levels. The web address for these documents is www.whitehouse.gov/omb/budget/fy2005/part.html.
A descriptive list of exemplary performance measures is available at www.whitehouse.gov/omb/performance/performance_measure_examples.pdf.