Detailed Information on the
Projects for Assistance in Transition from Homelessness Assessment

Defining data elements for outreach to individuals who are homeless.

Conducting independent evaluation of program.

Evaluating the SOAR (SSI/SSDI Outreach, Access and Recovery) technical assistance initiative.

Revise the PATH annual data collection instrument and procedures and align them with data elements in the HUD Homeless Management Information System (HMIS).

Determining the feasibility of using the HUD Homeless MIS to assist in obtaining outcome data from PATH-funded efforts.

Tracking and improving program performance using newly developed long-term outcome and efficiency measures.

Measure: Percentage of enrolled homeless persons who receive community mental health services

Explanation:

Measure: Percentage of contacted homeless persons with serious mental illness who are enrolled in services

Explanation:

Measure: Average federal cost for enrolling a homeless person with serious mental illness into services

Explanation:This measure reflects unit costs for this program. It appears on OMB's list of exemplary efficiency measures. Data shown correct previously reported data.

Measure: Increase number of homeless persons contacted

Explanation:The number of individuals served is a key measure for all SAMHSA programs that fund services. For the PATH program, outreach to homeless individuals creates the opportunity for appropriate services.

Is the program purpose clear?

Explanation: The purpose of Projects for Assistance in Transition from Homelessness (PATH) is to make formula grants to states and territories to provide outreach, mental health and other supportive services to homeless individuals with serious mental illness. Federal funds are also designed to leverage state and local funds at the provider level. The purpose is stated clearly in the authorizing legislation and is commonly shared by interested parties.

Evidence: Authorized as part of the McKinney homeless legislation of 1990, PATH authorities are in sections 521-535 of the Public Health Services Act. The legislation specifies PATH is a formula grant to states to provide outreach, referrals and services to individuals with serious mental illness who are homeless or at imminent risk. The program is run by the Substance Abuse and Mental Health Services Administration (SAMHSA).

Does the program address a specific interest, problem or need?

Explanation: The program addresses a problem that can be clearly defined, though data on the problem are limited. PATH is designed to support assertive outreach to homeless individuals with serious mental illness who need assistance but are not pursuing mental health treatment and other services on their own. These individuals are widely considered among society's most vulnerable. The problem is specific, however, reliable data on the target population are not available.

Evidence: National data on the total number of homeless individuals are flawed. A 1996 national survey estimates 20% of 2-3 million homeless individuals have a serious mental illness. There are no valid estimates of people at risk of homelessness. The agency uses an estimate of 600,000 homeless overall on any given night.

Is the program designed to have a significant impact in addressing the interest, problem or need?

Explanation: PATH is a formula grant to all 50 states and requires a one to three grantee match. Overall, the program is designed to support outreach efforts for a hard to reach population in order to enroll them in mainstream services, including public housing programs, community mental health treatment systems, and entitlement programs such as Supplemental Security Income (SSI) and Medicaid. The final impact of the program can only be as significant as the foundation of services to which referrals and enrollments can be made. Setting aside the reliance on a much larger set of systems, the program is designed to provide outreach services to homeless individuals with serious mental illness in order to get them engaged in service systems. PATH funds represent a portion of the outreach effort for this population and local agencies blend with other state, local and Federal sources, which complicates efforts to delineate the program's impact in the context of all other factors.

Evidence: A 1992 report of the Task Force on Homelessness and Severe Mental Illness called for aggressive outreach services for this population beyond an existing SSA demonstration and VA outreach program. There is evidence that in the years prior to the establishment of the program, this population was generally considered out of the reach of treatment and other service systems and specialized outreach efforts were uncommon. PATH contacts over 100,000 homeless individuals with serious mental illness a year, and over half of states exceed the required match. An evaluation found that because PATH funds constitute a portion of the cost of intervening with the target population it is difficult to disaggregate the impact of PATH dollars.

Is the program designed to make a unique contribution in addressing the interest, problem or need (i.e., not needlessly redundant of any other Federal, state, local or private efforts)?

Explanation: State and local governments and private foundations invest in outreach efforts for this population and provide funds to the same entities funded by PATH. As referenced in question three, the program succeeds in leveraging funds and a significant reduction in outreach would result from the program's absence, however, there is nothing inherently unique in the Federal contribution. PATH is, however, the only Federal program designed to provide outreach to the general population of persons with serious mental illness who are homeless or at risk of being homeless. Other Federal programs provide services to homeless individuals with mental illness, including the Mental Health Block Grant and HUD's Supportive Housing Program. An interagency 5-year demonstration called ACCESS funded 18 sites in nine states to support outreach to engage homeless persons and also provided a comprehensive range of services, including mental health and substance abuse treatment, job placement, housing, and other services.

Evidence: A 1999 GAO report cites overlaps among Federal homeless programs, including mental health support, but did not find a Federal program that shares PATH's mission. In a 1996 evaluation of the health centers program, the IG found community health centers provide outreach to homeless individuals, but that clients are often unwilling to receive services. Data on total state and local spending on outreach for this population are not available.

Is the program optimally designed to address the interest, problem or need?

Explanation: The program distributes funds to states through a formula grant and is still relevant to current conditions. The formula is based on urban populations, which is a reasonable proxy for homeless populations and the matching requirement can help prevent supplantation.

Evidence: There is no evidence that providing support through a competitive grant or other mechanism would be more effective or efficient than PATH's design.

Does the program have a limited number of specific, ambitious long-term performance goals that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: PATH has adopted a limited number of long-term outcome goals related to its mission. The long-term goal most focused on the desired end results of the program is the percentage of homeless persons enrolled by the program who receive community mental health services. Because the focus of the program is to support outreach to get homeless people with serious mental illness ready to access mainstream services, this long-term goal looks not at the outcome of treatment, but the receipt of treatment as an outcome of outreach. In order to track the success of contacts, the long-term goals also measure enrollment and case management rates. The program believes enrollment to be a useful measure because it signals the opening of a case record and the initiation of screening for additional services. The program relies on case management as a measure of success because there is little chance that an individual homeless person assigned a case manager will drop out without continued contact. PATH seeks to contribute to the HHS and HUD broad objective to end chronic homelessness.

Evidence: These long-term goals were adopted as part of the PART review process. The goals will be referenced in the agency's future GPRA plan. The goals include to increase the percentage of enrolled homeless persons with serious mental illness who receive community mental health services; increase the percentage of contacted homeless persons with serious mental illness who are enrolled in services; and increase the percentage of contacted homeless persons with serious mental illness who receive case management services.

Does the program have a limited number of annual performance goals that demonstrate progress toward achieving the long-term goals?

Explanation: PATH has a limited number of valid annual goals that track progress toward achieving the long-term outcome goals of the program. The goals will be referenced in the agency's future GPRA plan.

Evidence: PATH's key annual goals include: increase the number of persons contacted through outreach; and maintain the percentage of people who are enrolled into services. A third goal is to increase the percentage of participating agencies that offer outreach services.

Do all partners (grantees, sub-grantees, contractors, etc.) support program planning efforts by committing to the annual and/or long-term goals of the program?

Explanation: Program partners support the overall goals of the program and measure and report on their performance as it relates to accomplishing those goals. PATH's direct grantees provide performance data on annual goals using a common software program. The agency also supports biennial meetings, workgroups and calls to discuss program and planning information with PATH grantees.

Evidence: Grantees input performance data into a database. Data is compiled to report progress on annual goals, identify poor performers, and design technical assistance. Aggregated data are provided in the agency performance reports.

Does the program collaborate and coordinate effectively with related programs that share similar goals and objectives?

Explanation: PATH is unique among Federal programs in that it targets homeless individuals with serious mental illness, however, it shares goals with many HUD programs and relies on Federal entitlement programs including Medicaid and Social Security. The program shows evidence of meaningful collaboration with Federal partners and of encouraging collaboration at the local level. The program is also working with HUD to improve grant coordination.

Evidence: The program collaborates with SSA to improve client enrollment in SSI and with the Centers for Medicare & Medicaid Services to identify impediments to use of Medicaid. HHS and HUD have also been meeting on issues around HHS taking on a greater involvement in support of services for homeless populations, such as through joint grant reviews for HUD continuum of care grants. The program has collaborated with HUD, CMS, other parts of SAMHSA, the Administration for Children and Families, and the Health Resources and Services Administration on policy academies to improve homeless services.

Are independent and quality evaluations of sufficient scope conducted on a regular basis or as needed to fill gaps in performance information to support program improvements and evaluate effectiveness?

Explanation: By statute, SAMHSA must evaluate PATH every three years to ensure expenditures are consistent with the authorization and to recommend changes in program design or operations. The evaluation is performed by contract and considers program results relative to its annual goals. The evaluation may be strengthened by adding additional client outcomes.

Evidence: Section 528 of the PATH authorization requires a regular evaluation. The most recent evaluation was conducted by Westat and supported by SAMHSA and HHS' Assistant Secretary for Planning and Evaluation.

Is the program budget aligned with the program goals in such a way that the impact of funding, policy, and legislative changes on performance is readily known?

Explanation: The program can estimate costs of contacts as an output goal, which is directly associated with the program's outcome goals. The program's annual budget display does not meet all standards of alignment. However, the program's ability to attribute cost to output and the connection between that output and the desired outcomes of the program meets the standards of this question. The program budget supports one major grant activity, easing the task of alignment. Budget planning is tied to strategic planning. The program has measured its impact and can also estimate the impact of funding changes on the number of homeless individuals with mental illness contacted by the program. Program management funds are budgeted elsewhere. Annual budget requests could be improved through an increased focus on what is needed to accomplish program goals. The program develops estimates on past experience, and can also make further progress in estimating actual cost.

Evidence: This assessment is based on the annual budget submission to OMB and the Congress.

Has the program taken meaningful steps to address its strategic planning deficiencies?

Explanation: The deficiency in this section had been program goals. Through this process, the program has adopted new long-term goals that capture intended outcomes of the program, such as the percentage of homeless individuals with serious mental illness contacted by the program who become enrolled in mental health treatment. The program is estimating the likely outcomes of the program based on past performance. Having these measures in place will also further enable the program to integrate budget planning and strategic planning and determine the level of financial resources needed to obtain long-term outcomes.

Evidence: The program has adopted new long-term goals. The agency is also drafting a blueprint to end homelessness. The agency also reports developing performance based budgeting to strengthen the links between performance and budget. The agency's restructuring plan consolidates budget formulation, planning and Government Performance and Results Act activities within one unit.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: PATH grantees submit annual data that are used to measure progress toward achieving annual goals. Data are also used to ensure compliance with program legislation and identify technical assistance needs. Performance is also monitored through regularly scheduled and rotating site visits.

Evidence: Evidence is from their annual reporting form, annual performance reports and evaluations. An example of an action taken in response to performance data is PATH putting in place a training manual for providers in response to low performance in SSI enrollment.

Are Federal managers and program partners (grantees, subgrantees, contractors, etc.) held accountable for cost, schedule and performance results?

Explanation: This question considers accountability for performance of program partners and at the Federal level. Federal staff are assigned grantees by region and track performance in meeting annual goals, but are not held accountable for performance results through employee evaluations or other mechanisms. While staff and managers performance is evaluated regularly on tasks and responsibilities associated with the position, the program agency has not identified the managers who are responsible for achieving key program results and established performance standards for those managers. While funds are distributed by formula, the program is highly engaged with grantees and does reserve the right to withhold funds for failing to show progress in objectives. At the local level, grantees do often use performance-based contracting.

Evidence: The assessment is based on discussions with the agency and grants management documents. Employee evaluations at the agency are independently handled by each of the agency's three centers. The agency reports additional efforts to enhance accountability of Federal managers for program performance.

Are all funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: The program obligates funds on schedule and monitors use for the intended purpose. States have one year from the beginning of the award period to obligate funds and two years to spend.

Evidence: The assessment is based on apportionments, PATH funding documents and financial status reports. The agency is also working on establishing waves of grant announcements to improve the distribution of obligations through the fiscal year.

Does the program have incentives and procedures (e.g., competitive sourcing/cost comparisons, IT improvements) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: The program can take additional steps to improve administrative efficiency, but does have some incentives and procedures in place. The program operates with a relatively limited number of Federal staff. The program's application and performance data are reported electronically. Federal staff review proposed budgets to identify excessive costs. The program relies on an HHS service clearinghouse known as the Program Support Center for many internal services. The agency is meeting FAIR Act targets and appears to be making progress toward outsourcing additional services. Outsourced activities include accounting, graphics, human resources, and property management.

Evidence: The assessment is based on discussions with the agency, FAIR Act reports, and the description of services directed to HHS' consolidated Program Support Center.

Does the agency estimate and budget for the full annual costs of operating the program (including all administrative costs and allocated overhead) so that program performance changes are identified with changes in funding levels?

Explanation: The program does not have a financial management system that fully allocates program costs and associates those costs with specific performance measures. The program is unable to cost out resources needed to achieve targets and results. The program does not capture all direct and indirect costs borne by the program agency, including applicable agency overhead, retirement, and other costs budgeted elsewhere. FTE and administrative expenses are not tied to annual program budgets and the program has not developed a procedure for splitting overhead and capital costs between outputs. The program does develop annual budget proposals that include associated FTE costs, or include informational displays in the budget that present the full cost of outputs.

Evidence: Assessment is based on annual program management budget requests to OMB and Congress.

Does the program use strong financial management practices?

Explanation: IG audits of the agency's financial management have identified no material internal control weaknesses. The agency's fiscal monitoring of grant awards is conducted through the SAMHSA Grants Information Management System, which tracks awards and obligations, carry over and submission of quarterly reports, application renewals and final reports. The system is used to flag grantee financial management issues for project officers and Federal managers.

Evidence: The assessment is based on conversations with the agency, audited statements and Office of the Inspector General reports.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: The main deficiencies include use of performance data to enhance accountability and the ability to identify changes in performance with changes in funding levels. Most significantly, at the agency level additional steps are underway to increase accountability for program performance at the Federal level.

Evidence: The agency has begun rolling out performance contracts as part of an overall management reform plan that will set specific, quantitative targets. These contracts are to include outcome elements focused on program goals. The agency's restructuring plan consolidated budget formulation, planning and Government Performance and Results Act activities within one unit.

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: The agency and its contractors conduct regularly scheduled site visits to visit every state every five years. Annual applications include detailed information by provider on services funded, clients served and client characteristics.

Evidence: Site visit protocol, site visit reports, grantee annual reports, guidance for applicants documents and instructions for annual data reporting.

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: PATH annual performance data are summarized in the performance report and made available on the agency web site. New measures will provide additional useful data. Periodic evaluations of the program are posted on the agency's web site. Fact sheets on state performance are also available.

Evidence: Agency web site (www.samhsa.gov) and state fact sheets.

Has the program demonstrated adequate progress in achieving its long-term outcome goal(s)?

Explanation: The program has adopted new long-term goals and has baseline data available that show progress toward meeting its long-term outcome goals. The focus of the program is to support outreach efforts for a hard to reach population in order to enroll them in mainstream services. The first measure tracks the program's success enrolling persons who are homeless and have serious mental illness after contact, or the first stage of intervention. Providers find temporary or longer-term shelter for persons contacted and arrange for mental health treatment, housing, case management and other services for enrolled clients. The second measure captures the portion of homeless individuals who receive mental health treatment, a key outcome of the program. The third measure is an efficiency measure of whether the program is able to maintain unit Federal cost of enrolling a homeless person with serious mental illness into services from a baseline of roughly $668 per enrollment. A Yes on this question would require improved efficiency outcomes and progress on treatment in mental health services.

Evidence: "The assessment is based on the agency's GPRA plan, Healthy People 2010 and PATH program data. The program adopted new data check measures last year to eliminate double counting of contacts. Data are collected from program grantees and validated by program contractors. The periodic evaluation may serve as an additional check of data accuracy. The FY 1999 percentage of contacts who receive mental health services is needed. In 2000, the appropriation was $30,883,000 and the number of enrollees was 46,218. In 1999, the appropriation was $26,000,000 and the number of enrollees was 44,881. As described in Section II, the program has taken steps to improve data collection from grantees and control for outliers in reporting the number of persons contacted. The program estimates over time enrollees will be those who are harder to locate and engage. "

Does the program (including program partners) achieve its annual performance goals?

Explanation: The program sets annual targets and is exceeding the targets. The annual goals relate to outcomes measured in the long-term goals.

Evidence: The data are available in the agency's annual performance plans. As described in Section II, the program has taken steps to improve data collection from grantees and control for outliers in reporting the number of persons contacted. The agency identified data outliers and restated procedures and definitions to correct any inflated numbers. Data indicate progress on the key annual goal related to the percentage of persons contacted who become enrolled to receive services.

Does the program demonstrate improved efficiencies and cost effectiveness in achieving program goals each year?

Explanation: The agency is meeting the standards of a Yes for having incentives and procedures to measure and achieve efficiencies and has realized some improved efficiencies at the Federal program level. The agency is taking further steps to improve efficiency through reductions in deputy manager positions and consolidation of smaller offices. Measuring efficiency is complicated by the program's reliance on the greater service systems for the population and the potential for reaching the easier to treat individuals first. The program's long-term goals will now track the percentage of contacts enrolled, managed and treated. These data will provide evidence of changes in program cost effectiveness. In the future, the data may also be combined with annual measures of the total number of persons contacted and annual appropriation totals to get an idea of how efficiently the program is enrolling the target population into services. A Large Extent or Yes would require additional data on improvements in efficiencies and cost effectiveness in achieving program goals in the last year.

Evidence: Assessment is based on annual performance reports, agency restructuring plans, and discussions with agency managers. The agency's GPRA plan had indicated the number of persons contacted per Federal dollar and percentage contacted who become enrolled have declined over the past three years of available data. However, the program found its FY 1997 data to be unreliable and has improved its data collection efforts through the introduction of new grantee data entry software that detects and rejects the entry of performance information by the grantee that would indicate impossible performance. Program mangers believe that in most cases, such outliers showing highly improbable increases in grantee performance were the result of errors in data entry or a lack of understanding of performance measurement methodology.

Does the performance of this program compare favorably to other programs with similar purpose and goals?

Explanation: Because this program is the only Federal program of its kind as noted in Program Purpose & Design, the question weighting is reduced to zero. However, it is worth noting that other Federal programs do provide services to homeless individuals with mental illness, including Mental Health Block Grant and HUD's Supportive Housing Program. The Block Grant and HUD program have not been evaluated for their ability to reach homeless individuals with mental illness and their performance reports do not track their effectiveness in reaching this population. As noted previously, PATH does have documented evidence of effectiveness for its outreach efforts. However, because of their more broad mandates and lack of specific evaluations, an accurate comparison with these other programs cannot be made at this time. A Large Extent would require additional evidence of improved efficiency at the grantee level.

Evidence: The assessment is based on annual performance reports, GAO report on homelessness, HHS and HUD performance reports. These reports indicate the program is performing well but does not share a similar purpose and goals with other programs for this question. The Supportive Housing Program provides annual competitive grants to communities for housing and supportive services for the homeless. Similar to PATH, the HUD program can support outreach and case management, but it also supports a longer list of services such as child care, employment assistance and outpatient health services.

Do independent and quality evaluations of this program indicate that the program is effective and achieving results?

Explanation: The most recent final evaluation report of PATH was completed by Westat in late 2000. The evaluation indicates the program is meeting its annual output and outcome goals; responds to resource constraints by targeting the most vulnerable population; and supports the overall service delivery system for this population. The evaluation also indicated states and localities on average provide twice the required one to three match. Evaluation data confirm high levels of enrollment, but data on final entry into treatment, housing or other assistance through PATH funding are not available.

Evidence: In addition to results related to PATH's annual GPRA measures, key findings include 35% of clients who received PATH funded services were diagnosed with schizophrenia or some other psychotic disorder and an additional 30% were diagnosed with an effective disorder such as major depression or bipolar disorder; outreach is the leading service supported with PATH funds, followed by medical referrals, screening and diagnostic treatment and mental health services; PATH funds are used most frequently to fund salaries of individuals who offer case management services; the leading referrals are for housing, mental health treatment, and substance abuse treatment.