NATIONAL
ADVISORY COUNCIL
OFFICE OF MANAGEMENT AND BUDGET
PERFORMANCE MEASUREMENT ADVISORY COUNCIL
SUMMARY OF MEETING
September
13, 2002
The Performance
Measurement Advisory Council (PMAC) was convened for its second meeting
at 9:00 A.M. on September 13, 2002, at the White House Conference Center,
Washington, D.C. Thomas Reilly, the Council’s Designated Federal
Officer, opened the meeting along with Mortimer Downey, the Council’s
Chair.
In accordance
with the provisions of Public Law governing Federal Advisory Committees,
the meeting was open to the public.
Council members present:
Mortimer
L. Downey
William Eggers
Harry P. Hatry
Patricia Ingraham
Donald F. Kettl
Joseph Wright, Jr.
Staff members
of the Office of Management and Budget (OMB) attending were:
Thomas
Reilly, Designated Federal Officer
Marcus Peacock
Reid Cramer
Other OMB
representatives present for all or portions of the meeting were:
Diana Espinosa
Mathew Schneider
Michael Hickey
Anthony Piccininno
Norris Cochran
Karen Lee
John Kelly
Mortimer
Downey opened the meeting. He stated that the main focus of the meeting
would be to offer advice and generate feedback on the program assessment
exercise, so that this information can be processed and included in the
2004 Budget.
•
Discussion of Ratings Exercise
Thomas Reilly
provided a summary update for the Council on the ratings exercise. After
considering advice from the Council and other stakeholders, such as the
President’s Management Council (PMC), the Budget Director issued
a memorandum to agency heads on July 16th that described the process.
Included in this memo was a list of agency programs that will be assessed
as a part of the 2004 Budget. OMB’s Program Effectiveness Team held
a series of training sessions for agency and OMB staff designed to provide
an overview of the exercise and ensure the consistent application of the
Program Assessment Ratings Tool (PART).
The Council
noted that the definition of programs will affect presentation issues
and the selection of relevant performance goals. Thomas Reilly responded
that included in OMB’s guidance was a set of criteria for defining
programs and selecting programs to be rated this year.
Thomas Reilly
described how OMB staff and agency program staff had been collaborating
on completing the PARTS and reported that there is anecdotal evidence
of excellent dialogue on performance measures. He also reviewed the proposed
schedule for completing the PARTs and settling agency appeals. OMB has
formed an Interagency Review Panel (IRP), which is comprised of agency
representatives and OMB staff, to audit a sample of completed PARTs, ensure
consistent application of guidance, and make recommendations on specific
agency appeals. The IRP is scheduled to meet at the end of September and
complete its work by October 7th.
Council members
inquired about the engagement of Congress. Joe Wright stressed the importance
of working with Congress to institutionalize the review of program performance.
He indicated the possibility of linking this effort to the existing Government
Performance and Results Act framework, and then proposing ways to connect
the GPRA format to the presentation of the Budget. This could be proposed
as a refinement to GPRA.
Harry Hatry
suggested that OMB might ask the General Accounting Office (GAO) to obtain
feedback on the assessment process. The Council asked Paul Posner of GAO
to comment. He said that he would like this process to lead to congressional
interest in performance evaluation and the current inventory of GAO analysis
could be used in the assessment process.
Council suggested
that a review of the entire exercise should be conducted to refine the
process for future years. This should include a survey of the agencies
to obtain their feedback as to how the process was supporting their analytical
and managerial needs.
•
Fall Review
Thomas Reilly
reported that OMB is currently considering how to reflect PART assessments
in the Director’s Fall Budget Review. As a part of this process,
OMB has developed a database which will facilitate the tracking of budget
decisions to the program assessments.
The Council
offered several suggestions for future efforts, including focusing on
measurements of cost effectiveness and equity in program results
•
Presentation Issues
Reid Cramer
reported that the Budget Director explained in his July 16th memo that
final decisions regarding the presentation of PART results in the FY 2004
Budget have not been made. Further, the Director stated that presentation
options would be discussed by the Performance Measurement Advisory Council
(PMAC) at its September meeting.
The Council
discussed a range of presentation options and considered their advantages
and disadvantages in the context of the Budget documents. Several issues
were highlighted.
Issue
1: Transparency. How should performance information be made available?
OMB staff
reported that they intended to recommend that at a minimum, the name of
each program assessed with the tool should be identified, along with section
scores, and an overall effectiveness rating should be included in the
Budget. The Council agreed with this recommendation. OMB staff reported
that they are proceeding with the assumption that the PART worksheets
for each program will be accessible to the public on an OMB website. The
Council agreed with this approach to transparency but several members
were concerned that the manner in which the summary assessments were presented
would greatly influence the public’s perception of the initiative.
The Council
recognizes that there are many places within the four-volume budget where
PART related information can be presented, such as in Analytical Perspectives
or in a summary chapter on the President’s Management Agenda. While
the Council recognized the time constraints for this preparing the 2004
Budget, they strongly recommended pursuing web-based presentation alternatives
for future years. OMB should explore ways to use technology to reduce
the size of the printed Budget and develop a flexible database that could
be useful to many different types of stakeholders
Issue
2: Format. How should the PART score be converted into program
assessments?
Effectively
communicating program assessments for such a large number of programs
places a premium on simplicity, distinguishing performance among programs,
and ability to measure improvements over time. The Council recommended
that the numeric scores of the PART and the section weights should be
disclosed somewhere in the Budget but this information was not necessary
to include in a summary table in each department chapter. Summary tables
could use verbal assessments, such as “effective,” “moderately
effective,” “minimally effective,” and “ineffective”.
Issue
3: Presentation. What information from the PART should be shown
in the Budget and where should it be shown?
The Council
recommended that a summary table be included in each departmental budget
chapter with the name of assessed programs, overall assessments, explanations
of the assessments, and descriptions of the budget recommendation or policy
proposal for each program. They recommended that this table be supported
by a summary report for each assessed program in another portion of the
Budget. This summary report would include additional information derived
from the assessment including key performance measures and results.
Several members
of the Council recommended that OMB consider additional alternatives in
the future.
Harry Hatry
asked that OMB continue to consider ways to highlight the results
score from the PART as distinct from the overall assessment. For example,
the overall effectiveness rating could be based on the results score and
be relabeled as an “Outcome Rating.” The first three sections
could be combined into one “program management” rating and
used as a part of the explanation of program results. Hatry noted that
if OMB does not take this suggestion, it should footnote on each
program rating sheet, that the overall rating is based on the weighted
average of all four sections and identify the weights used.
Joe Wright
recommended that program assessments include a more explicit focus on
cost effectiveness in addition to results and program purpose. Planning
and management will feed into results and therefore do not need to be
separated out.
Council members
were concerned with how the presentation of program assessments would
compare to the presentation of the scorecard for the President’s
Management Agenda. The stoplight format of the scorecard attracts attention
and even though these assessments are made at the department level, while
the PART assessments are performed at the program level, the Budget must
communicate how they are connected.
•
Appeal and Audit Process
Thomas Reilly
described the appeal and audit process that has been outlined in the Budget
Procedures Memo No. 855. Central to this process is the formation of an
8 member interagency Review Panel that will include OMB and agency staff.
•
Common Measures
Marcus Peacock
described the current state of the common measures exercise. Council members
discussed ways to expand and institutionalize the effort, such as developing
benchmarks from the private sector, focusing on efficiency measures, and
controlling for divergent quality in programs that have diverse constituencies.
•
Next Steps
The next
meeting of the PMAC is not scheduled. It may be held in February. Potential
agenda items include agency presentations on their experience with PART,
examination of assessment process, refinements for the 2005 Budget.
ADJOURNMENT
The meeting
adjourned at 1:45 P.M.
I hereby
certify that, to the best of my knowledge, the foregoing minutes are accurate
and complete.
Thomas Reilly,
Designated Federal Officer
The Council
at its next meeting will formally consider these minutes, and any corrections
or notations will be incorporated in the minutes.
|