OFFICE OF MANAGEMENT AND BUDGET
PERFORMANCE MEASUREMENT ADVISORY COUNCIL
SUMMARY OF MEETING
Measurement Advisory Council (PMAC) was convened for its third and final
meeting at 1:00 P.M. on March 3, 2003, at the White House Conference Center,
Washington, D.C. Reid Cramer, acting as the Council’s Designated
Federal Officer, opened the meeting along with Mortimer Downey, the Council’s
with the provisions of Public Law governing Federal Advisory Committees,
the meeting was open to the public.
Harry P. Hatry
Donald F. Kettl
Joseph Wright, Jr.
Called to Order
Downey opened the meeting. He stated that the main focus of the meeting
would be to offer feedback on the presentation of the program assessments
in the 2004 Budget. He stated his view that this was an incremental process,
where improvements should be made along the way.
reported that input from the PMAC’s first two meetings was helpful
in developing both the PART assessment tool and the budget presentation.
The release of the President’s Budget provides the public with an
unprecedented amount of information revealing what the government achieves
with its public resources. At this point, OMB intends to use the PART
again as a central element of the 2005 budget process, rating an additional
20% of federal programs. OMB also anticipates communicating the findings
of these analyses in a similar presentation. Although wholesale changes
to the PART are not expected, OMB is interested in strengthening the process,
the instrument, and the budget presentation.
by Individual Members
asked to summarize their responses to the set of questions that had been
distributed to members prior to the meeting and were included on page
52 and 53 in the Budget volume.
questions were considered by the Council:
do you see as the strengths and weaknesses of the presentation of performance
information in the PART summaries?
- How can
the PART be improved or revised? Page 52 of the Budget described a series
of issues for the PART and invited public comments. These issues included
increasing consistency, defining “adequate” measures, minimizing
subjectivity, and measuring progress towards results.
that the PART provides an incentive for agencies to improve performance
information, how should the PART treat programs that have acceptable
measures but bad targets?
programs lack outcome measures either because they cannot pinpoint the
contribution of the program to achievement of the overall outcome or
because the nature of their work is process-oriented. Are there acceptable
alternative measures when there are no outcome goals?
- To get
credit for results, the PART required that a program have acceptable
measures, either long-term or annual. Should the PART continue to enforce
a linkage between strategic planning and results?
articulated his views that the results information should be the primary
basis for program assessment and he liked the separation of the results
score. He thought the distinction between long-term and annual goals is
confusing and not constructive.
Ingraham thought the presentation was clear and understandable, achieving
significant transparency. She was concerned that the objectives of a program
do not always match the measures. Contextual issues, such as changing
missions, should be taken into account.
stated he thought that the presentation cries out for web access to make
it accessible and available to be manipulated for analysis.
observed that the assessment volume and the worksheets offer a rich data
set, but comparisons are important. More crosscutting analysis should
be pursued in the future.
said that he thought the PART material was well integrated into the individual
chapters of the Budget, but attention should be paid to making it a web-based
document. He believes the common measures exercise is an important and
related exercise that needs to be highlighted further.
liked the format. He said the assessment volume should look like a budget
book and it does. He appreciated the graphics. He suggested OMB consider
talking to its oversight committees in Congress to ask that they require
it. This would facilitate institutionalization of the process. Many of
the assessments would benefit from have more long-range targets. He thought
that some attention should be given to establishing a sense of priorities
because currently everything is given the same weight. He believes that
the Budget is a policy document and the assessments should not shy away
from asking each program if there is a justifiable federal role.
recognized that the PART was valuable in helping to shed light on the
work of small programs, many of which were looedk at for the first time.
This was constructive.
stated that he would like to see much discussion of goals and purpose
because for many programs the purpose was unclear. The linkage between
the assessment findings and the budget recommendation should also be made
stronger. Consistency is a major issue that is related to OMB resource
commitments to this process.
Ingraham noted that as much training as possible would be helpful.
suggested that the first three sections of the PART be given a grade but
the weighting of each individual question could be dropped. Harry also
expressed concerns that the PART is compatible with GPRA. OMB should provide
clear guidance on the relationship between the two exercises. In the future
measures need to be improved, both in the selection and data collection
procedures. Intermediate outcomes and ultimate outcomes are both important.
In-depth program evaluations should be used when there are recent and
of good quality. Sometimes the findings of these reports are more important
than the program’s outcome measures. Also, OMB should look at ways
of capturing the distributional effects of a program, if they are important.
cited some examples of where he thought the PART was applied inconsistently.
A similar program with a better set of targets scored lower than another.
He suggested that the process could be improved by engaging with the community
of Inspector Generals to assist in the auditing of program assessments.
This could help address OMB’s workload issue as it expands the number
of programs assessed.
stated that OMB could do a better job of stating the purpose of the assessments
and also improving the connection between the budget recommendations and
the assessments. He thought the assessments need to describe the trajectory
of a program (whether it was improving over time). OMB should use the
PART to identify statutory changes and regulatory barriers but assess
program on what they should be doing. The selection of programs to be
assessed should be done according to themes so that it supports the common
measures effort. In terms of training, OMB should find ways of highlighting
PART best practices.
thought that transparency will help with subjectivity. The process should
continue to be as transparent as possible. OMB should systematically survey
participants over ways to improve the process.
expressed satisfaction that the PART represents a good effort to date
and put the “M” back in OMB. Still OMB needs to look for every
hook to make the process stick.
adjourned at 3:30 P.M.
certify that, to the best of my knowledge, the foregoing minutes are accurate
Designated Federal Officer for the March 3rd meeting.