ExpectMore.gov


Detailed Information on the
Transportation Security Administration: Federal Air Marshal Service Assessment

Program Code 10001070
Program Title Transportation Security Administration: Federal Air Marshal Service
Department Name Dept of Homeland Security
Agency/Bureau Name Department of Homeland Security
Program Type(s) Direct Federal Program
Assessment Year 2008
Assessment Rating Moderately Effective
Assessment Section Scores
Section Score
Program Purpose & Design 80%
Strategic Planning 88%
Program Management 86%
Program Results/Accountability 60%
Program Funding Level
(in millions)
FY2007 $719
FY2008 $770
FY2009 $786

Ongoing Program Improvement Plans

Year Began Improvement Plan Status Comments
2008

Develop an advanced risk analysis tool whose goals include providing FAMS with to the capability to quantify increased/decreased risk levels relative to inputted resource levels.

Action taken, but not completed Develop final operational prototype by 4th Quarter of FY 2008. Improvement action addresses re-PART findings for Question 2.7.
2008

Strengthen the FAMS program's financial performance by remediating undelivered order balances that were contributory to TSA's UDO material weakness.

Action taken, but not completed Provide quarterly to OMB the results of FAMS?? review and validation of past obligations. Improvement action addresses re-PART findings for Question 3.6.
2008

Use findings of the program's general audit by the General Accountability Office (GAO) to determine improvement actions for 2nd through 4th quarters of FY 2009.

Action taken, but not completed GAO audit is anticipated to be completed in the 1st quarter of FY 2009. Improvement action addresses re-PART findings for Question 4.5.
2008

Use findings of the program's general audit by the General Accountability Office (GAO) to determine improvement actions for 2nd through 4th quarters of FY 2009.

Action taken, but not completed GAO audit is anticipated to be completed in the 1st quarter of FY 2009. Improvement action addresses re-PART findings for Question 4.5.

Completed Program Improvement Plans

Year Began Improvement Plan Status Comments
2008

Use findings of the program's general audit by the General Accountability Office (GAO) to determine improvement actions for 2nd through 4th quarters of FY 2009.

Completed GAO audit is anticipated to be completed in the 1st quarter of FY 2009. Improvement action addresses re-PART findings for Question 4.5.

Program Performance Measures

Term Type  
Long-term Outcome

Measure: Number of successful terrorist and other criminal attacks initiated from commercial passenger aircraft cabins on flights in identified risk categories.


Explanation:The measure captures the number of successful terrorist and other criminal attacks that affect control of the aircraft as a whole. Hostile control of an aircraft may be sought for terrorist purposes (e.g., use of an aircraft as a weapon of mass destruction against a target on the ground, or mid-air detonation with an IED) or criminal purposes (e.g., to exact ransom or other demands). Whether attackers' motivations are terrorist or criminal, it is expected that on 100% of covered flights, FAMs will detect and defeat them to prevent hostile control of the aircraft. By virtue of meeting this measure's targets, the program also deters onboard attacks.

Year Target Actual
2005 0 0
2006 0 0
2007 0 0
2008 0
2009 0
2010 0
2011 0
2012 0
2013 0
2014 0
Long-term/Annual Outcome

Measure: Level of public confidence in Federal Air Marshals' ability to promote aviation security.


Explanation:Beginning in FY 2008, participants in the Bureau of Transportation Statistics' scientifically designed and independently administered National Household Transportation Survey were asked about their confidence in FAMs. Respondents chose from one of five options to describe their confidence "in the ability of the Federal Air Marshals to defend an aircraft and its passengers from individuals with hostile intentions": 1) no confidence, 2) a small amount of confidence, 3) a moderate amount of confidence, 4) a great deal of confidence, or 5) total confidence.

Year Target Actual
2008 Baseline 3.33
2009 3.5
2010 3.65
2011 3.78
2012 3.91
2013 4.0
2014 4.0
Annual Outcome

Measure: Average annual rate of accuracy in Federal Air Marshals' firearms re-qualification.


Explanation:To remain a FAM, an employee must demonstrate a high standard of excellence in firearms performance on a recurring basis by scoring, quarterly, at least 255 out of 300 (85 percent) on the agency's requalification course of fire. The measure captures targeted and actual precision rates - among all FAMs, on an average annual basis. Because a very high level of firearms proficiency is required for FAMs to successfully defeat terrorist and criminal attacks onboard commercial passenger aircraft, the "Average annual rate of accuracy in Federal Air Marshals' firearms re-qualification" is a proxy outcome for the number of successful attacks. A flight that is under attack presents a unique crime scene where innocent bystanders are present in very close proximity, without the options of dispersing, taking cover, or fleeing; collateral physical and structural damage must be strictly and expertly limited to prevent decompression of the aircraft. Precision requirements (minimum 85 percent accuracy, maximum 15 percent margin of error) drive a FAM's ultimate ability to defeat an attempted attack. As a proxy outcome, this measure need not meet the 'external event or condition' criterion for a stand-alone outcome. As in other deterrence programs (e.g. Nuclear Regulatory Commission), the proxy outcome measure needn't be a public good unto itself, but rather be linked to the public good that is captured by the parent outcome measure.

Year Target Actual
2007 Baseline 95%
2008 95%
2009 95%
2010 95%
2011 95%
Annual Outcome

Measure: Percentage level in meeting FAM coverage targets for each individual category of identified risk


Explanation:In each of ten (classified) categories of highest risk flights, the FAMS plans flight coverage to a specific target, which represents some percentage of all flights in that category; if the target itself represents less than 100% of all flights in that category, it may be possible to exceed the target, particularly if there was some emergent need for the FAMS to cover additional flights or shift resources within a given category. The measure captures performance in each of the ten risk categories into a composite value, calculated as the average of the percentages of attainment of the ten individual targets. This measure is a "proxy" outcome for the number of successful attacks measure: Because the universe of daily U.S.-flagged flights is so large, the program's ability to identify and deploy its personnel on higher risk flights definitionally reduces the likelihood of catastrophic attack. As in other deterrence programs (e.g. Nuclear Regulatory Commission), the proxy outcome measure needn't be a public good unto itself, but rather be linked to the public good that is captured by the parent outcome measure.

Year Target Actual
2005 Baseline 99.8%
2006 100% 101.7%
2007 100% 96.2%
2008 100%
2009 100%
2010 100%
2011 100%
Annual Efficiency

Measure: Percentage of FAMs providing the targeted number of flying days per year.


Explanation:The measure is expressed as the percentage of mission deployed FAMs who were in mission status for within plus/minus ten percent of the expected minimum number of days. FAMS best demonstrates cost efficiency by optimizing the allocation of its personnel - that is, improving (optimizing) the rate at which they serve in mission status at levels tied to management's focus on long-term and annual performance goals. This is a non-traditional efficiency measure that, given the idiosyncrasies of the FAMS program, is the most meaningful measurement of program efficiency. Since efficiency measures are to reflect the economical utilization of resources to produce outputs, and FAMS has very low "overhead" (non-personnel funding) by benchmarked standards, FAMS best demonstrates cost efficiency by optimizing the allocation of its personnel. Furthermore, a measure of the cost of the program divided by the number of flights staffed is not a measure of efficiency because: (1) FAMS' "airfare" costs are, by law, absorbed by the airlines; (2) FAMS per diem costs are set by an external body (GSA) and there can be no management decision (in the pursuit of efficiency) to pay slightly less than the amount to which the employee is entitled; and (3) FAMS' destinations are risk based (for instance, flying exclusively domestically would be the least expensive approach, however, threat conditions as well as OMB guidance have stressed the importance of international flights - which will cost more per mission, without representing any loss in efficiency). Alternatively, measuring the number of flying days per FTE would be sub-optimal in that it would represent an average rather than a per-FAM assessment of flying output. Long-term sustainability of the program requires avoidance of a scenario where certain FAMs are over-flown while others are under-flown, which is permitted or masked by a flying days per FTE measure. True efficiency is demonstrated by ensuring that each individual FAM provides, if you will, his or her proportionate share of flying output. Note that the targets are not closer to 100% because the number of days is "targeted" rather than required - for instance, certain FAMs by virtue of their prior years of federal service are entitled to a great deal more annual leave than others, which cannot be "held against" the program's efficiency.

Year Target Actual
2007 75% 74%
2008 76%
2009 76.5%
2010 77%
2011 77.5%
Long-term/Annual Efficiency

Measure: Efficiency measure, "Cost per flight per FAM"


Explanation:The result reflects the full cost ( FTE, Per Diem and Program Management Costs ) of the FAM Service, divided by the number of flights per year (classified), divided by the number of FAMs per flight. The result is the full cost to the government per flight per FAM. "Entire budget" refers to the FAMS Appropriation (pre-FY 2009) and the FAMS Program/Project Activity of the Aviation Security Appropriation (PB 2009 and subsequent).

Year Target Actual
2007 SSI SSI
2008 SSI SSI
2009 SSI SSI
2010 SSI SSI
2011 SSI SSI
2012 SSI SSI
2013 SSI SSI

Questions/Answers (Detailed Assessment)

Section 1 - Program Purpose & Design
Number Question Answer Score
1.1

Is the program purpose clear?

Explanation: The program purpose is to promote confidence in our Nation's civil aviation system through the effective deployment of Federal Air Marshals (FAMs) to detect, deter, and defeat hostile acts targeting U.S. air carriers, airports, passengers, and crews.

Evidence: ??The Aviation and Transportation Security Act (ATSA) of 2001 (P.L. 107-71), as codified at Title 49, Section 44917 of the United States Code (U.S.C.), authorizes the deployment of FAMs on commercial passenger aircraft. ??49 U.S.C., Section 114(q) authorizes the Secretary of Homeland Security (and by delegation, the Administrator of the Transportation Security Administration) to designate employees of the Transportation Security Administration as Federal law enforcement officers. The authority provides that these officers may carry a firearm, make arrests, and seek and execute warrants for arrest or seizure of evidence. ??Section 1303 of the Implementing Recommendations of the 9/11 Commission Act of 2007 amended the Homeland Security Act of 2002 to authorize the Secretary of the Department of Homeland Security, acting through the Administrator of the Transportation Security Administration (TSA), to develop Visible Intermodal Prevention and Response (VIPR) Teams for augmenting the security of any mode of transportation at any U.S. location. The Act authorizes the Secretary to utilize any DHS asset, including FAMs, for this purpose. In Budget of the United States Government, Fiscal Year 2009, the President requests VIPR funding within a "Law Enforcement" Program/Project Activity (LE PPA) of TSA's Aviation Security appropriation; the LE PPA sustains funding for ten permanent VIPR teams. For purposes of this Program Assessment Rating Tool, "the program" refers to the FAMS PPA of TSA's Aviation Security appropriation, consistent with the FY 2009 President's Budget. The physical demands and physiological effects of flight are such that any category of aviation professionals (be they in law enforcement, the military, or private industry) do not fly five days per week and twenty days per month on a sustained basis. Training (in tactics and firearms) for conditions and scenarios that may arise in mission status is the program's priority use for non-flying days. With respect to the balance of "non-mission status" days unneeded for training, FAMs in that status may be drawn upon on an ad hoc basis to support VIPRs deployed consistent with the 9/11 Recommendations Act. The program's (FAMS PPA's) "other transportation modes" activities occur in non-mission time without reduction to mission flights or disruption to the program's purpose.

YES 20%
1.2

Does the program address a specific and existing problem, interest, or need?

Explanation: As a result of the September 11th attacks, President George W. Bush ordered the rapid expansion of the Federal Air Marshal Service (FAMS), in order to restore public confidence in air travel and prevent further attacks. In the establishment of the Department of Homeland Security, the Homeland Security Act of 2002 stated, "It is the sense of Congress that the Federal air marshal program is critical to aviation security." On the occasion of TSA's transition from the Department of Transportation to the Department of Homeland Security, Secretary of Transportation Norman Mineta articulated that TSA was "forged on an anvil of cruel necessity." The National Strategy for Aviation Security categorizes protection of the "Air Domain" as a "vital national interest," and cites FAMs' law enforcement presence on commercial passenger aircraft as among the efforts responsible for enhancing aviation security on an ongoing basis. Terrorists have an ongoing interest in and capability to target aviation, as confirmed by intelligence sources, underscored by world events (e.g. Heathrow plot of August 2006), and duly recognized by both the Administration and the Congress.

Evidence: ??Homeland Security Act of 2002, Section 1402 ??Secretary Norman Mineta quotation ??National Security Strategy of the United States of America - Introduction ??National Security Strategy of the United States of America - Overview ??National Strategy for Aviation Security - Prologue ??National Strategy for Aviation Security - Introduction ??Message to DHS employees from Secretary Chertoff, August 10, 2006, regarding Heathrow plot (stated that the FAMS would expand mission coverage for U.K.-U.S. flights as a component of the U.S. Government response to the Heathrow plot and the Department of Homeland Security's responsibility to protect travelers). ??Reuters news article covering Secretary Chertoff's remarks to the World Economic Forum, January 2008. ??FAMs' ability to defeat terrorist attacks is by no means limited to traditionally defined hijacking scenarios. For instance, FAMs' training in surveillance techniques is such that FAMs possess the ability to detect the suspicious activities that would generally precede the onboard assembly of an improvised explosive device (IED).

YES 20%
1.3

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: The design of aviation security is to have layers to create redundancies and duplicate security measures to reinforce each and every layer for the strongest level of security. While other programs within TSA and throughout government overlap in purpose and the targeted beneficiaries, FAMS is important to Aviation Security. In order to prevent dangerous people and cargo from reaching the aircraft, TSA in cooperation with other government agencies has employed a layered approach to aviation security. As a critical layer of aviation security, the Federal Air Marshal Service fills a critical role detecting, deterring, and defeating criminal terrorist activities on the aircraft. For international flights, only the FAMS are authorized to fly armed. Although the presence of other armed law enforcement officers (armed LEOs) and Federal Flight Deck Officers (FFDOs) on domestic flights may at times augment the FAMS, the distinctive differences inherent in the FAMS program prevents redundancy or excessive overlap. These characteristics detailed in the accompanying evidence begin with the process of flight scheduling and extend through tactical positioning on the flight, training, equipping, and focusing during flight to defeat in-flight attacks. FFDOs' primary responsibility is flying aircraft, and their jurisdiction is limited to the cockpit, therefore FFDOs do not protect passengers in any situation that occurs within the cabin's confines. Even if all pilots were FFDOs (which is not the case) or FFDOs were assigned their flights on a risk basis (which is not the case), the need for risk-based armed LEOs (FAMs) with the unique positioning within the cabin would remain. In addition, while internal aircraft security has been increased by hardened cockpit doors, this enhancement does not mitigate that threat posed by IEDs and liquid explosives which can be used to attack an aircraft from the cabin without breaching the cockpit door. The FAMS will continue to be a vital force promoting public confidence in aviation, and protecting passengers in a range of scenarios outside the cockpit (e.g. IEDs, hostage situations) in which a FAM team can discover, surveil, and intercept suspicious activities, as well as contain or defeat perpetrators present in the cabin.

Evidence: Explanation of Unique Facets of the FAMS Scheduling - FAMS is the sole in-flight security measure that the US Government deploys on the basis of risk. This complex process which utilizes knowledge of the FAMS Concept of Operations, risk mitigation, and the airline industry is flexible enough to accommodate emerging intelligence or security events and deploys FAMs in a manner that maximizes both risk reduction and deterrence against onboard attack. In contrast, the presence or absence of other armed LEOs is not based on any determination of risk but is rather a coincidence of carrying out their agencies' missions or personal travel schedules. Participation in the FFDO Program is voluntary, and a pilot's status as an FFDO is not a factor in the airlines' assignment of cockpit crews to flights. Tactical Positioning - In contrast to other armed LEOs, FAMs are seated in a tactically optimal manner with respect to the defeat of an attempted attack. Training - FAMs receive extensive specialized and recurrent training to defeat attacks in the idiosyncratic aircraft cabin environment. This training includes close quarters defensive tactics and a requirement to meet the highest agency-wide firearms proficiency standard in government, preparing FAMs to fire their weapons with confidence, while positioned on a completely unstable platform. Equipping - There is no expectation that other LEOs, though armed, would have other equipment (e.g., baton, handcuffs) useful in neutralizing a threat non-lethally. Focus - other armed LEOs are free to rest, focus on their traveling companions, type reports on their laptops, etc. There is no expectation that other armed LEOs will be surveilling the cabin for suspicious pre-attack activity. July 31, 2007 GAO Testimony "Subject: Aviation Security: Federal Coordination for Responding to In-flight Security Threats Has Matured, but Procedures Can Be Strengthened"

NO 0%
1.4

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: The program has a statutory responsibility to deploy an armed federal law enforcement presence on high risk flights. Therefore, FAMS is a "direct federal" program and the FAMS could not delegate associated responsibilities to state, local, or private entities through a "grant" or "regulatory" program design. Within the direct federal provision of flight coverage, the program design is primarily a function of the allocation of FAM teams to commercial passenger flights based on risk, consistent with the FAMS Concept of Operations. In 2006, the Homeland Security Institute (HSI) conducted an independent evaluation of the development and implementation of the FAMS Concept of Operations. Given the program's objectives and performance goals, HSI found that the FAMS applies a valid approach to analyzing risk and allocating resources on the basis of threat, vulnerability, and consequence -- employing a filtering process along with an allocation tool to optimize resource allocation. HSI identified additional considerations (classified) that may be determined in further (currently ongoing) study to offer incremental improvements to FAMS' program design, but would not alter the essence of the program's existing design or structure. In its inter-organizational comparisons, HSI did not find evidence (in other agencies facing comparable risk management challenges) of significantly better risk management techniques suited to FAMS' objectives. The FAMS has utilized study teams and working groups to study management challenges facing the agency and to make recommendations for consideration by agency leadership. Teams' and groups' methodologies included benchmarking where appropriate. The subjects of study teams and working groups have included (but not been limited to) seating protocols, and the scheduling (quantity and sequence) of ground duty days. The scheduling review resulted in a pilot program currently underway, and results of the review that covered tactical seating are in various stages of implementation.

Evidence: ??Homeland Security Institute independent evaluation of FAMS Concept of Operations. (Document is classified). ??Summary of intent, with respect to underway HSI efforts to study incremental refinements to the program's valid design. ??Scheduling study team and tactical procedures working group: summary of findings and recommendations being implemented (Sensitive Security Information). ??A comparison of the U.S. and Israeli air marshal services would be an "apples to oranges" comparison because Israel's tiny fleet (fewer than 50 aircraft, all state-owned) facilitates 100% coverage (as compared to 29,000 U.S.-flagged flights daily by 82 commercially-owned airlines). Many other international air marshal services come to the United States Federal Air Marshal Service for training and expertise and the sharing of ideas and tactics; their coverage rates and allocation also raise "apples to oranges" issues because the majority face airfare funding constraints (must pay for their marshals' airline tickets), whereas, by contrast, the airfares of U.S. FAMs are absorbed by the commercial airlines. The U.S. FAMS has trained air marshals from ten countries, and its leadership in international air marshal training is recognized in Section 4017 of the National Intelligence Reform Act of 2004. The U.S. FAMS hosted the 2007 International Air Marshal Conference and is a key leader in the international air marshal community.

YES 20%
1.5

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: ??Macro benefit - primary: With respect to deterrence, the flying public as a whole is the (both intended and actual) primary beneficiary of the program's resources. To the terrorist, the presence vs. absence of air marshals is an ominous "unknown" that functions to deter attacks against commercial passenger aircraft flights -- FAM coverage is a deterrent of hostile acts on (all) flights, because any (specific) flight could have a FAM team. ??Others who benefit: Other intended and actual beneficiaries of program resources are: -Airline passengers on FAMS covered flights -Airline crewmembers (cabin and cockpit). -The airline industry, and its critical role in the U.S. economy and international commerce. -Persons, structures, or entities that terrorists may target, on the ground, using a hijacked aircraft. By design, the application of the program's Concept of Operations effectively targets resources by prioritizing and scheduling missions on the basis of risk. Deployments are thus structured such that program benefits reach highest risk beneficiaries at the micro level, while deterrence as a macro-level benefit reaches the flying public (collectively) and other intended beneficiaries. In any instance when a flight categorized as lower risk is covered, coverage of the lower risk flight reaches intended beneficiaries by delivering the aforementioned macro benefit. There is no evidence that activities that would have occurred without the program are subsidized; the Federal Government (and prudently so) - would be unlikely to ever revise aviation security regulations to permit commercial airlines to employ or contract for civilian security personnel to be deployed as passengers.

Evidence: ??FAMS uses a risk-based approach to scheduling that selects from among over 29,000 U.S. (domestic and international) commercial airline flights each day to deploy thousands of FAMs worldwide. This complex process is carried out by teams of technically-skilled professionals who utilize their knowledge of the FAMS Concept of Operations, risk mitigation, and the airline industry to deploy FAMs in a manner that maximizes both risk reduction and deterrence against onboard attack. ??The process of scheduling FAM deployments on a risk basis is flexible with respect to being able to accommodate emerging intelligence or security events. ??The U.S. Bureau of Transportation Statistics' "Pre- and Post-9/11 Econometric Analysis of Travel by Mode" found a "statistically significant negative impact of 9/11 on air travel per person above age 16." The Congressional Research Service report, "9/11: Global Economic Costs" captures the direct and indirect costs of the 9/11 terror attacks.

YES 20%
Section 1 - Program Purpose & Design Score 80%
Section 2 - Strategic Planning
Number Question Answer Score
2.1

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The program has two long-term outcome measures that are specific to the program's purpose to promote confidence in our Nation's civil aviation system through the effective deployment of FAMs to detect, deter, and defeat hostile acts targeting U.S. air carriers, airports, passengers, and crews. The "Number of successful terrorist and other criminal attacks initiated from commercial passenger aircraft cabins on flights in identified risk categories" is a long-term, outcome measure with timeless relevance to the purpose of the program. The measure refers to the number of successful terrorist and other criminal attacks that affect control of the aircraft as a whole. The absence of these successful attacks is, to quote OMB's definition of an outcome measure, "the intended result or consequence" of program activities and is "of direct importance to beneficiaries and the public generally." Measuring programs' effect upon the absence of an activity (deterrence) has historically been a complex challenge in performance measurement; therefore, per OMB guidance, the program has also developed two proxy measures for the "number of successful attacks" outcome. (Proxy measures appear in response to Question 2.3). As the program exists to promote confidence in the Nation's civil aviation system, it has long been the case that no more salient or meaningful program outcome exists than public confidence in FAMs. Until recently, the program did not have in place a means of quantifying public confidence. This changed when multiple Transportation Security Administration programs began appearing in the Federal Government's "National Household Transportation Survey;" a question that measures public confidence in FAMs first appeared in the survey in Calendar Year 2007. The survey is externally and independently administered by the Department of Transportation's Bureau of Transportation Statistics (BTS), and the results will provide a long-term measurement of the "Level of public confidence in Federal Air Marshals' ability to promote aviation security." The program's proposed new outcome measure is (by design, and consistent with OMB guidance) program-specific. Were the public surveyed about its level of confidence in the Nation's civil aviation system, that level would represent the collective outcome of multiple TSA, U.S. Government, and airline industry programs, with the contribution of any one program indistinguishable from the contribution of the others. By contrast, the survey question to which the program's outcome measure is tied is intentionally limited in scope to FAMs, to ensure that the results explicitly depict public confidence in FAMs.

Evidence: ??Expectmore.gov: current program performance measures. ??Discussion of BTS' survey methodology. ??Text of FAMS question that appeared in BTS' National Household Transportation Survey.

YES 12%
2.2

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: With respect to the long-term outcome measure "Number of successful terrorist and other criminal attacks initiated from commercial passenger aircraft cabins on flights in identified risk categories," targets are ambitious in that they are quantified as "zero" for all timeframes. "Zero" is the only acceptable goal for performance. Conversely, an end target of "zero" with previously higher targets on a trajectory of downward increments would be unrealistic from the perspective of the program, the Federal Government, and the taxpayer. Despite the clear necessity for the targets' ambitiousness, the program recognizes that the measure presents one of the common challenges of deterrence measurement (i.e. as an "all or nothing," catastrophe-based measure, failure is a non-option and targets are definitionally non-conducive to annual variances). In such instances, OMB recommends the use of "proxy measures;" the program has developed two proxy measures for the "number of successful attacks" outcome. (Proxy measures appear in response to Question 2.3; proxy measures' targets and baselines appear in response to Question 2.4). The long-term outcome "Number of successful terrorist and other criminal attacks initiated from commercial passenger aircraft cabins with FAM coverage" has an established and published baseline. With respect to the long-term outcome measure "Level of public confidence in Federal Air Marshals' ability to promote aviation security," the program has quantified ambitious targets for a five-year timeframe. These targets promote continued improvement relative to the baseline (1st Quarter FY 2008) annual result. Since this measure's baseline is only one year old, the program will offer (in response to Question 4.1) additional, supplemental evidence of public confidence in the program. On an interim basis (until a longer baseline is established), that supplemental evidence should be considered in accompaniment with the first year of baseline data.

Evidence: ??Expectmore.gov: current program performance measures.

YES 12%
2.3

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: The program's first annual measure is "Average annual rate of accuracy in Federal Air Marshals' firearms re-qualification." Because a very high level of firearms proficiency is required for FAMs to successfully defeat terrorist and criminal attacks onboard commercial passenger aircraft, the associated rate of accuracy is one proxy outcome for the number of successful attacks. As in other deterrence programs (e.g. Nuclear Regulatory Commission), the proxy outcome measure needn't be a public good unto itself, but rather be linked to the public good that is captured by the parent outcome measure. Graduation from the Federal Law Enforcement Training Center (FLETC) requires federal law enforcement candidates to score 210 out of a possible 300 on its Practical Pistol Course (PPC). To graduate from Phase II of the Federal Air Marshal Training Program (FAMTP-II), a FAM candidate must achieve a much higher standard: 255 out of 300 (85 percent). To remain a FAM, an employee must demonstrate the same standard of excellence in firearms performance on a recurring basis by scoring, quarterly, at least 255 out of 300 on the PPC. For their re-qualification course of fire, some federal law enforcement agencies (including FAMS) use the FLETC PPC, while others have courses of fire that represent slight adaptations from that standard. The FAMS knows of no other federal agency that affords its law enforcement officers a smaller margin of error (15 percent) on the agency-wide course of fire suitable for that entity's operational environment; the FAMS' minimum rate of accuracy (85 percent) represents a high standard of excellence on a benchmarked basis with respect to firearms requirements government-wide. This high standard is driven by program need: a flight that is under attack presents a unique crime scene where innocent bystanders are present in very close proximity, without the options of dispersing, taking cover, or fleeing; collateral physical and structural damage must be strictly and expertly limited to prevent decompression of the aircraft. A FAM's successful completion of the program's firearms re-qualification is an output-demonstrated excellence in the scored and benchmarked re-qualification is an outcome and an annual proxy tied closely to the "number of successful attacks" long-term outcome: Precision requirements (high rate of accuracy, small margin of error) drive a FAM's ultimate ability to defeat an attempted attack, and FAMs' renown as elite marksmen contributes strongly to the program's deterrent effect and, secondarily, to the public's confidence in the security of air travel. The program's second annual measure is "Percentage level in meeting FAM coverage targets for each individual category of identified risk." With U.S. air carriers providing over 29,000 flights daily, the ability of the program to identify high risk flights for FAM deployment is one proxy outcome for the number of successful attacks. The absence of a FAM team on a single high-risk flight would be statistically unlikely to lead to a catastrophic failure to prevent an attack; however, the absence of a risk-based resource allocation method for deploying FAMs would, were it the case, drive a substantially higher risk of program failure. In light of recent OMB guidance as to the identification and constitution of proxy measures, the program believes that one of its historical "output" measures (coverage for categories of risk), while not a stand-alone outcome measure, is actually a "proxy" outcome for the number of successful attacks measure: The ability to identify and deploy its personnel on higher risk flights definitionally reduces the likelihood of the catastrophic outcome that the number of successful attacks measure targets to avoid. Actual variances in the data with respect to the attainment of coverage targets are driven by emergent threat conditions, suggesting that the proxy measure should be re-categorized as "annual" in nature.

Evidence: ??Chart: Federal law enforcement agencies' agency-wide courses of fire - required rates of accuracy. Even courses adapted to a specific agency's operational environment possess essential similarity, as the basis for variance in testing pistol proficiency is limited in feasibility. ??FAMS Concept of Operations - executive briefing (Sensitive Security Information). ??Office of Flight Operations memorandum, March 2008, "FAMS Risk Categories Metric" (Sensitive Security Information). ??The measure "FAM coverage on flights with identified threats" no longer reflects a meaningful output and is proposed for deletion. FAMS' 2003 PART contained three annual measures tailored specifically to the program's post-September 11th rapid expansion; the maturity of the program has rendered these measures obsolete, and FAMS requests their deletion: -Level of operational FAMs who successfully complete Phase II training. -Level of operational FAMs verified as meeting recurrent training requirements. With respect to measuring the impact of training activities, the program proposes replacing this obsolete output measure with a new annual outcome measure ("Average annual rate of accuracy in Federal Air Marshals' firearms re-qualification"); as discussed in the above Explanation, the latter measure has the added utility of being a proxy for the program's "number of successfully attacks" approved GPRA measure. -Level of FAM days allocated to core mission (i.e., the number of days FAMs are flying on aircraft versus training and other activity days). The program proposes replacing this obsolete measure with a new measure "Percentage level meeting annual targets for flying days per FAM" (an efficiency measure comprehensively presented for credit in response to Question 3.4). The latter is a stronger measure than the former since targets represent performance levels to which the program aspires, rather than the actual but static provision of a formulaic amount of availability for missions. Additionally, the new measurement is more meaningful because it occurs at the individual FAM level rather than the aggregate field office level. ??OMB form "Request to Delete a Measure in Older PARTs" (four). ??FAMS Policy OPT 8141, "Extended Absence Refresher Training" - FAMs whose extended absence preceded or interrupted completion of FAMTP-II assume duties (at the conclusion of that absence) in non-operational status pending their FAMTP-II class date. This policy's enactment, along with a 100 percent FAMTP-II completion rate among currently operational FAMs, obviates the measure "Level of operational FAMs who successfully complete Phase II training." In addition, the policy requires alumni of FAMTP-II who have an extended absence (in excess of three years) to repeat FAMTP-II. ??FAMS Policy OPT 8142, "Annual Training Requirements for Mission Deployed Federal Air Marshals," evidences the institutionalization of twenty annual training days for FAMS and obviates the usefulness of the measure "Level of operational FAMs verified as meeting recurrent training requirements." ??49 U.S.C. § 44905(b) - "If a decision is made that a particular threat cannot be addressed in a way adequate to ensure, to the extent feasible, the safety of passengers and crew of a particular flight or series of flights, the Under Secretary of Transportation for Security shall cancel the flight or series of flights." The Undersecretary delegated this authority to Federal Security Directors in February 2006 (Department of Homeland Security Delegation of Authority 400.1).

YES 12%
2.4

Does the program have baselines and ambitious targets for its annual measures?

Explanation: FAMS has the highest rate of accuracy (85 percent) required on an agency-wide course of fire in a federal law enforcement agency. The program endeavors to achieve higher rates of accuracy than even this benchmarked high, by striving to achieve an average annual PPC score of 95 percent (285 out of 300) among all FAMs. The targets strike an appropriate balance between challenge (out-performing the benchmarked high) and achievability (an average re-qualification score of 100 percent could not be achieved in any law enforcement population, or even subpopulation such as a SWAT team). The program has an FY 2007 baseline (95 percent) for its PPC score measure. Currently planned hiring initiatives are such that the program would be unrealistic to set overly simplistic incremental targets (89 percent in year one, 90 percent in year two, 91 percent in year three, etc.), as if FAMs were a static population of the same individuals gradually enhancing their marksmanship over time. Because OMB requires that targets be both aggressive and within reason for the program to achieve, the impending introduction of a substantial number of new personnel into the program's population is such that a 95 percent average annual score in fact represents continued improvement and clear effectiveness. Note that no other federal law enforcement agency is known to afford its officers a smaller margin of error (15 percent) on its agency-wide course of fire; the FAMS' minimum rate of accuracy (85 percent) represents a high standard of excellence on a benchmarked basis with respect to firearms requirements government-wide, and FAMS plans to exceed this benchmarked standard, even with the need for near-year targets that realistically account for the assimilation of new hires. The "Percentage level in meeting FAM coverage targets for each individual category of identified risk" has established baselines and specific annual targets. Targets are highly ambitious (never striving for less than 100 percent) because the program, in accordance with FAMS Concept of Operations, pursues flight coverage in its risk categories as a high priority. The ten categories: ??Are based upon vulnerability, threat, and consequence. ??Appear in the FAMS Concept of Operations, which has been independently validated by HSI (see responses to Questions 1.4 and 2.6).

Evidence: ??Expectmore.gov: current program performance measures.

YES 12%
2.5

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: Air carriers and foreign air marshal programs are key partners upon whom FAMS relies to support the program's performance goals. Scheduling data reported by the airline industry support the program's goals in that they contain critical information needed for the risk-based planning of FAM deployments. The program also interacts daily with the airlines to ensure that both planned and emergent reservation requirements are met in order to ensure both ticketing and tactically optimal seating for FAMs on high risk flights. Close and agile partnerships with the airlines are particularly crucial when FAM coverage must be introduced or altered on short notice (e.g. to "surge" FAM coverage in response to a particular emergent threat, or when widespread inclement weather over-taxes reservation systems). Protocols exist for the airlines' cabin and cockpit crews to report suspicious activities to an onboard FAM team, as well as to report suspicious activities that occurred on flights without FAM coverage. Section 4017 of the National Intelligence Reform Act of 2004 encouraged the President "to pursue aggressively international agreements with foreign governments to allow the maximum deployment of Federal air marshals on international flights." As a consequence of numerous such bilateral agreements, U.S. FAMs cover U.S. flights into and out of multiple foreign countries. In some instances, the partner country has air marshals, whose deployment into the United States on foreign airlines is also provided for in the bilateral agreement. The U.S. FAMS conducts training, education, outreach, and liaison efforts that encourage the development and maturity of foreign air marshal programs, and the deployment of foreign air marshals on foreign carrier flights to and from the United States. These bilateral partnerships reduce the threat of terrorism against aircraft of both nations. Partner countries report to the U.S. FAMS all instances of foreign air marshal deployments to the United States. Just as domestic airlines and foreign air marshals partner with FAMS to work toward the program's goals, the program - in turn - partners with other security "layers," as well as the Transportation Security Operations Center, to work toward TSA's collective goals of aviation security. A layered approach to aviation security is consonant with the National Strategy for Aviation Security; furthermore, the program is pursuing a next-generation resource allocation system in which determinations of high risk flights (for FAM deployments) could account for variances in the other layers of aviation security. Variances may include, for example, a flight's historical levels of FFDO coverage, or whether a particular airport has recently seen a high rate of attrition among senior screener personnel.

Evidence: ??Listing of risk factors (Sensitive Security Information) derived from airline data. Samples of airline data used by the program to determine high risk flights. ??Content of FAMS "crew briefing" (Sensitive Security Information) ??National Strategy for Aviation Security - excerpts regarding layered aviation security. ??Advanced Route Evaluation System - abstract. ??National Intelligence Reform Act of 2004, Section 4017. ??Data on bilateral agreements pertaining to air marshal coverage.

YES 12%
2.6

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: In response to OMB's recommendation that the program should obtain an independent evaluation of its methods for risk analysis and resource allocation, FAMS contracted with the Homeland Security Institute (HSI) for that evaluation, entitled "Federal Air Marshal Service Risk Assessment," completed July 14, 2006. ??HSI's independent assessment employed quality evaluation methods to determine the validity of the program's Concept of Operations (CONOPS), the basis of the program's risk analysis and resource allocation. HSI defined "validation" as a test of whether or not the FAMS' risk management processes and the outcome (i.e. impact) of those processes are reasonable and consistent with externally stated guidance and, internally, with the FAMS CONOPS. The assessment culminated in the development and employment of a basic quantitative model to study the impact and implications of alternative strategies for assigning resources. In addition to HSI's quantitative methods, the renowned sophistication of its staff's education and experience represents an additional aspect of quality in assessment. ??The FAMS CONOPS is the agency's strategy for the determination of high risk flights and prioritization of FAM deployments on the basis of threat, vulnerability, and consequence; as such, a determination as to the CONOPS' validity and implementation equates to a determination of agency-wide effectiveness in the program's risk analysis and resource allocation. Accordingly, HSI's independent evaluation was comprehensive in its scope. HSI did identify multiple issues and aspects meriting more micro or targeted follow-on evaluation. ??HSI is a federally funded research and development center (FFRDC) whose independence of the Department of Homeland Security's opinions and policies is established (in both its authorizing legislation and its mission statement) and practiced (in its evaluations). Federal Acquisition Regulations establish FFRDCs' independence, explicitly requiring FFRDCs "to operate in the public interest with objectivity and independence" as a condition of their FFRDC status. ??To ensure adequate frequency of assessment, the program anticipates an ongoing relationship with HSI for independent analyses supporting program improvements. Following HSI's initial independent evaluation in 2006, the program contracted with HSI for a follow-on evaluation that is currently ongoing (to be completed in the 4th quarter of FY 2008); FAMS anticipates that the follow-on evaluation will support program improvements in the areas of scheduling, resource allocation, and budget transparency.

Evidence: ??Quality: -HSI "Standards of Quality." -"Methodology" excerpt from HSI statement of work. -85% of HSI staff members hold advanced degrees, 28% hold doctorate degrees, and analysts average nearly two decades of postgraduate experience. 50% of the analysts conducting the program's risk assessment held doctorate degrees. ??Scope - "Objectives" excerpt from HSI statement of work. ??Independence: -Description of FFRDCs' and HSI's independence, from HSI's website. -Background, vision, and mission statements - HSI Annual Report. -HSI mission statement page that prefaced FAMS risk assessment. ??Frequency - Schedule of completed and ongoing independent evaluations. ??A formal comparison with international air marshal services would have been highly cost prohibitive for the program, due the very large incremental costs (for international travel) that FAMS would have had to provide to HSI. That being said, a comparison of the U.S. and Israeli air marshal services would be an "apples to oranges" comparison because Israel's tiny fleet (fewer than 50 aircraft) facilitates 100% coverage (as compared to 29,000 U.S.-flagged flights daily by 82 airlines). Many of the other international air marshal services come to the United States Federal Air Marshal Service for training and expertise and the sharing of ideas and tactics; their coverage rates and allocation also raise "apples to oranges" issues, because the majority face airfare funding constraints (must pay for their marshals' airline tickets), whereas, by contrast, the airfares of U.S. FAMs are absorbed by the commercial airlines. The U.S. FAMS has trained air marshals from ten countries, and its leadership in international air marshal training is recognized in Section 4017 of the National Intelligence Reform Act of 2004. The U.S. FAMS hosted the 2007 International Air Marshal Conference and is a key leader in the international air marshal community.

YES 12%
2.7

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: The program does not yet meet this question's requirement to put forth budget justifications that define explicitly the relationship between increments/decrements in program performance and increases/decreases in funding levels. The program is working with independent experts of the Homeland Security Institute to develop a means of transparently quantifying and communicating the performance implications (outputs and outcomes) that various resource levels (inputs) would drive.

Evidence: FY 2009 Congressional Justification for FAMS PPA. (Congressional Justifications, at this time, explicitly tie neither direct nor indirect cost variances to specific variances in performance targets).

NO 0%
2.8

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: At the time of the program's initial PART, OMB's rating of "Results Not Demonstrated" cited strategic planning deficiencies that included the absence of baselines, targets, and timeframes associated with performance goals and performance measurements. These deficiencies stemmed primarily from the newness of the Federal Air Marshal Service in its post-9/11 form. With respect to current OMB PART standards, the initial PART also reflected the absence of the following: ??A second long-term outcome measure. ??Proxy measures with respect to deterrence. ??An efficiency measure. The program has taken meaningful steps to correct the aforementioned strategic planning deficiencies by working to adopt: ??A second long-term outcome measure meaningfully reflective of the program's purpose (see response to Question 2.1). ??Proxy measures for its approved GPRA measure (see response to Question 2.3). ??An efficiency measure (see response to Question 3.4). ??Baselines, targets, and timelines for both its approved GPRA measures and its proposed new measures (see responses to Questions 2.2 and 2.4).

Evidence: The PART Web "Measures Screen" contains a package of measures that meet OMB's requirements as to the types, quantity, and characteristics of performance goals and measurements. This reflects the program's greater maturity and its commitment to correcting previous strategic planning deficiencies associated with the program's prior "Results Not Demonstrated" PART rating.

YES 12%
Section 2 - Strategic Planning Score 88%
Section 3 - Program Management
Number Question Answer Score
3.1

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: The use of the program's performance measures is seen in the management and resource allocation activities associated with each and every Roster Period (period of mission scheduling which corresponds to two pay periods). In advance of each Roster Period, the program's field offices make resources (FAM teams) available for missions according to the parameters captured in the program's efficiency measure; managers at the program's headquarters use "availability sheets" to monitor compliance with those parameters, each Roster Period. The program's planning entities then apply the Concept of Operations to allocate resources (FAM teams) to flights according to the categories represented in the program's categories of identified risk performance measure. These two activities, taken together, improve program performance (to detect, deter, and defeat attacks) in that they constitute measured efforts to apply the optimal volume of resources where the most risk lies. With respect to partners, the program actively requests from international air marshal programs their increased coverage on foreign-flagged flights to the U.S., in association with special events that call for added vigilance and the allocation of additional resources. This action improves aviation security with respect to U.S. special events that may be attractive targets for terrorists. Foreign air marshal agencies report to the program (see response to Question 2.5) all deployments to the United States. The impact of partners on resource allocation is also seen in the program's partnership with the TSA Office of Intelligence. Prior to each Roster Period, the program coordinates its international deployments with that office, whose threat assessment criteria identify geographic locations where terrorist activity is ongoing or likely to occur, or where there is specific threat information; the program improves its performance by allocating FAM deployments in greater numbers to those targeted cities.

Evidence: ??Description of stages of Roster creation and relationship to program performance measurements. ??Discussion of introduction of a performance variable recommended by a specific (HSI) evaluation. (Sensitive Security Information) ??Discussion of performance considerations taken into account in the derivation of days-in-mission-status parameters, and management actions to institutionalize those parameters. ??Data on bilateral agreements pertaining to international air marshal coverage.

YES 14%
3.2

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: The program's key managers are subject to the Departmentally-standardized DHS Senior Executive Service (SES) pay-for-performance system. The FAMS Director, Deputy Director, Assistant Directors, SES-level Deputy Assistant Directors, and SES-level Special-Agents-in-Charge each enter into an annual "Executive Performance Agreement" under that system. Under the Executive Performance Agreement, each of these personnel receives an Annual Performance Review that includes both narrative and quantified performance ratings (on a zero to 100 point scale) derived from: ??Demonstration of DHS core leadership competencies (40%) ??Employee-specific performance objectives (60%) that: -Include a minimum of five "key, measurable expectations to be met by the executive during the rating period." -Pertain specifically to the program sub-functions which that employee manages. -Are linked to DHS, TSA, and FAMS strategic goals. The performance agreements are the vehicle for identifying in writing the managers who are responsible for achieving key program results, and for clearly defining each manager's standards of accountability for achieving those results. The program's contract awards and renewals do consider past performance.

Evidence: ??Samples of program managers' Executive Performance Agreements. Excerpt (performance objective) from a Special Agent in Charge's (SAC's) performance agreement: "Effectively deploy Federal Air Marshals in accordance with TSA and FAMS strategic goals and the Concept of Operations, to support programs and processes that are responsive to a dynamic risk-based management strategy." This performance objective is measurable, and measured, throughout the performance cycle by the headquarters Office of Field Operations - with respect to whether that SAC's office is meeting its mission requirements and mission recovery expectations, and contributing to surge operations. ??Request for Proposal excerpts for a recently awarded and large contract - Mission Scheduling and Notification System, $90,154,394. (RFP is provided to demonstrate consideration of past performance in the program's contracting.)

YES 14%
3.3

Are funds (Federal and partners') obligated in a timely manner, spent for the intended purpose and accurately reported?

Explanation: The program obligates its funds in a timely manner and for intended purposes. (Note that the program does not make awards or reimbursements to state and local partners). Success in doing so is monitored both at the program level as well by the program's parent entities, TSA and DHS. Each year, FAMS prepares a comprehensive budget execution plan (with quarterly schedules of obligations) consistent with appropriated purposes, Administration policy, and DHS/TSA/FAMS senior leadership priorities. The program employs the Department of Agriculture's National Finance Center as its payroll agent. The program allocates funding (excluding personnel compensation) to its sub-Directorates, and centrally monitors their actual obligations, using specialized reports, routine controls, and periodic year-to-date reviews to ensure that obligations are occurring at appropriate rates, conform to their intended use and to appropriations law, and are consistent with the program's overall spending plan. The program also updates its budget execution plan as necessary to reflect any required fund reallocations, address resource needs associated with emerging mission priorities, and ensure that very limited unobligated funds remain in the program at year's end. The program has achieved annual obligation rates of at least 99 percent. The program's financial analysts review and certify all procurement requests to ensure funds are available and planned actions comply with established appropriation law standards. The program's contract writing systems promptly report obligations and awards via their automated, real-time interface with the Federal Procurement Data System (FPDS).

Evidence: ??Sample Fund Status Reports - periodic and year-end ??FY 2007 Financial Plan. ??Unobligated balance chart. ??Discussion of FPDS interface.

YES 14%
3.4

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: The program's initial capital asset costs were met with emergency supplemental funding following the September 11th attacks. The program's base appropriation, once established, did not reflect the usual and customary levels of capital and operating expenses associated with sustainability in a federal agency. On a benchmarked basis, the amount of FAMS' budget that is dedicated to purposes other than Personnel Compensation and Benefits ("non-PCB costs") remains small by benchmarked standards. In addition, with respect to Personnel Compensation and Benefits ("PCB"), FAMS has few "support" employees per law enforcement employee, on a benchmarked basis. These facts evidence low "overhead" resources (in both the quantities of support personnel and the levels of non-personnel costs) that have required improving and cumulative efficiency and cost effectiveness in order to: ??Operate and modernize with non-PCB resources levels below those available to comparable programs. ??Operate with a smaller proportion of support personnel than is available to comparable programs. The program's efficiency is not tied to any specific or micro procedure focused on a certain sub-program's line of business. Rather, limited "overhead" and "support" resources have required the program to be highly efficient on a systemic and sustained basis. The program's efficiency measure, "Percentage of FAMs providing the targeted number of flying days per year," is expressed as the percentage of mission deployed FAMs who were in mission status for within plus/minus ten percent of the expected minimum number of days. This measure reflects the effective utilization and management of resources (FAMs) to achieve program outcomes (no successful attacks and increased public confidence) and outputs (risk category targets) through the delivery of program services (mission coverage) to the public. As an agency which has very low "overhead" expenses by benchmarked standards, FAMS best demonstrates cost efficiency by optimizing the allocation of its personnel - that is, improving (optimizing) the rate at which they serve in mission status at levels tied to management's focus on long-term and annual performance goals. This is a non-traditional efficiency measure that is the most meaningful measurement of program efficiency. Since efficiency measures are to reflect the economical utilization of resources to produce outputs, and FAMS has very low "overhead" (non-personnel) by benchmarked standards, FAMS best demonstrates cost efficiency by optimizing the allocation of its personnel. (See measures screen for this measure's relative advantages to more traditional efficiency measures.) The use of the program's efficiency measure is seen in the management and resource allocation activities associated with each and every Roster Period (period of mission scheduling which corresponds to two pay periods). In advance of each Roster Period, the program's field offices make resources (FAM teams) available for missions according to the parameters captured in the program's efficiency measure; managers at the program's headquarters use "availability sheets" to monitor compliance with those parameters, each Roster Period. The efficeincy measure "Cost per flight per FAM" reflects the full cost (entire budget) of the FAM Service, divided by the number of flights per year (classified), divided by the number of FAMs per flight. The result is the full cost to the government per flight per FAM; interpretation of program results on this measure must take into account inflationary changes as well as (only major) policy changes. "Entire budget" refers to the FAMS Appropriation (pre-FY 2009) and the FAMS Program/Project Activity of the Aviation Security Appropriation (PB 2009 and subsequent).

Evidence: ??Chart depicting FAMS' highest rate of PCB levels, among comparable agencies. ??Chart depicting FAMS' smallest number of support employees, among comparable agencies. ??Director's message to the workforce establishing a FAM's "Targeted Range of Flight" (Sensitive Security Information).

YES 14%
3.5

Does the program collaborate and coordinate effectively with related programs?

Explanation: The program's Tactical Information Sharing System (TISS) is a tool which FAMs, TSA security officers (i.e. behavior detection officers), and airport police departments use to report to the program's Tactical Information Branch (TIB) suspicious activities. Suspicious activity may not initially rise to the level of probable cause for an arrest, but is one tool for use in identifying potential criminal and terrorist threats to civil aviation. TIB management determines whether a report of suspicious activity, whether singularly or in the context of other related reports, requires follow-up resources by FAMS and/or should be referred to a partner agency's management. The program allocates significant resources to international mission deployments (FAM team coverage on U.S. airliners transiting foreign countries). FAMs are the only armed U.S. government law enforcement personnel deployed worldwide on every day of the year, however FAMS does not presently have personnel assigned overseas. Managing overseas FAM deployments, therefore, requires strong partnerships with both U.S. Government entities (e.g. Department of State, U.S. Embassy, TSA Transportation Security Area Representative) and each foreign locality's host government (particularly foreign law enforcement authorities). Partnerships address the diplomatic, security, and logistical matters required to accomplish the program's international mission deployments, including passport and visa requirements, cost effective lodging, credential verification protocols, customs and airport egress, area threat assessments, access to emergency medical facilities, and weapons download and storage. Adherence to the host nation's unique procedures and protocols is essential to the program's continued ability to cover U.S. flights transiting the foreign country. Foreign air marshal agencies report to the program (see response to Question 2.5) all deployments to the United States. In addition, the program actively requests their increased coverage (during specific time periods and to particular airports) in association with special events that call for added vigilance and additional resources.

Evidence: ??TISS sample statistics depicting level of contributions from partners: In December 2007, 28.6% of suspicious activity reports to TISS were submitted by partner TSA security officers, and 23.6% were submitted by airport police officers. ??Classified annex to NSPD-46/HSPD-15, "U.S. Policy and Strategy in the War on Terror" - Outlines the unique jurisdiction of FAMs aboard U.S. flagged commercial aircraft in international or foreign airspace. ??Sample "Country Procedures" documents depicting the program's collaboration with other programs to support FAM deployments.

YES 14%
3.6

Does the program use strong financial management practices?

Explanation: Both the program and its parent agency (TSA) do use strong financial management practices; however, in the annual DHS audit (for which DHS has contracted with KPMG), the auditors did report two agency-wide material control weaknesses for FY 2007. One of the two TSA-wide material control weaknesses (on the subject of undelivered order [UDO] balances) did apply to the program, precluding attainment of this question's Elements of Yes. The program's response to Question 3.7 addresses the meaningful steps that FAMS is taking to contribute to the remediation of this TSA-wide material weakness.

Evidence: ??KPMG audit findings, excerpts.

NO 0%
3.7

Has the program taken meaningful steps to address its management deficiencies?

Explanation: OMB-issued President's Management Agenda ratings identify "agency-level" deficiencies, if applicable, within TSA. TSA has developed and implemented a Mission Action Plan to correct the material weakness referenced in response to Question 3.6; TSA anticipates that remediation will be reflected in the FY 2008 financial audit. As a part of TSA's meaningful efforts to properly account for funds and improve financial record accuracy agency-wide, TSA's component programs (including FAMS) are actively conducting quarterly reviews and validation of past obligations. With respect to the "program-level" management deficiencies that are the focus of this question, FAMS does participate in a system for evaluating and correcting program-level management deficiencies: The program's field offices undergo, on a three-year cycle, thorough inspections by the TSA Office of Inspection (OI), whose mission is to ensure the integrity and effectiveness of TSA employees and programs. Inspectors determine whether management deficiencies are present, and recommend courses of action to address any deficiencies identified. (Inspectors are TSA officials independent of field offices' chain of command). TSA OI began inspecting FAMS field offices in January 2006, and 100% of FAMS field offices will have undergone an inspection by the end of Calendar Year 2008. 76% of field offices had been inspected as of March 2008; in those offices, OI made 112 determinations (quantity of offices multiplied by the quantity of rated categories) of effectiveness and efficiency. Among those 112 determinations, most reflected either full compliance, or minor procedural issues to be addressed at the local level; of 112 possible instances, OI inspections identified only three instances of management deficiencies ("Recommendations"). Within 60 days of receiving an OI final report that contains one or more Recommendations, the program must submit to OI a corrective action plan and milestones. Within 120 days of the final report, the program must report to OI that the office has addressed the management deficiency by having taken the plan's corrective actions within the agreed upon timeframes.

Evidence: ??Listing of the seven areas/elements inspected by TSA OI. ??Schedule of completed and upcoming TSA OI inspections of FAM field offices. ??Sample corrective action plan addressing an OI "Recommendation."

YES 14%
Section 3 - Program Management Score 86%
Section 4 - Program Results/Accountability
Number Question Answer Score
4.1

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: The program has made substantial and measurable progress in achieving its long-term performance goals to: ??Detect, deter, and defeat hostile acts targeting U.S. air carriers, airports, passengers, and crews. ??Improve confidence in the security of our Nation's civil aviation system. For all timeframes, the program has met its targets for the "Number of successful terrorist and other criminal attacks initiated from commercial passenger aircraft cabins on flights in identified risk categories." The absence of these successful attacks is the intended result of the program's activities and is of direct, significant benefit to the public. With respect to its public confidence goal, the program has some measurable evidence of public confidence in FAMs: ??In its 2007 National Household Transportation Survey, the Department of Transportation's Bureau of Transportation Statistics (BTS) found that the U.S. adult population is confident in the ability of FAMs to defend an aircraft. 81 percent of BTS-validated responses indicated either a moderate, great deal, or total amount of confidence in FAMs. While these high positives do not represent a pro-forma result (success defined as an 'actual' of 81% and a 'target' of some lower value), the newness of the measure does not diminish its compelling result. To that end, the PART guidance allows for an assessment of whether the program is meeting "or" making measurable progress toward meeting its long-term performance goals - the program must be "on track to" meet all its long-term goals. That state of progress or being on track is evident in the 81% figure, particularly given that the public is assessing confidence in an undercover workforce. The program also offers other clear evidence of public confidence, below. ??In 2007, TSA contracted with an independent public affairs firm ("Blue Lime") for the "Partnership in Security Project" that gauged public perceptions of TSA using dialogue sessions with passengers (flyers) in New York, Minneapolis, and Washington D.C. At the conclusion of this study, Blue Lime reported the following with respect to FAMs (fuller study results are provided in evidence): -Passengers perceive that FAMs are well trained professionals dedicated to protecting the traveling public, and they (passengers) derive high levels of confidence and comfort from that perception. -Passengers associate FAMs with high levels of skill. -Passengers consider FAMs to be proactive, alert, and ready. -Knowledge of FAMs' mission has promoted a sense of confidence in commercial aviation. -Passengers closely associate FAMs with their (passengers') safety. ??The Federal Aviation Administration (FAA) projects that in FY 2009, average daily departures of U.S.-flagged air carriers (driven by the market's passenger load demands) will surpass pre-9/11/2001 levels. Passenger "demand" for aircraft departures ("supply") is not a discrete, long-term indicator of public confidence in FAMS, because other factors (e.g. ticket prices; performance of other aviation security programs) bear upon that demand. Nonetheless, "getting America flying" again after terrorists used American aircraft to carry out the most devastating attack in U.S. history was a national challenge; and, the correlation of these FAA data with BTS and Blue Lime findings indicate that FAMS played a substantial role in that result. In addition, both after September 11th and during the U.S.-U.K. liquid explosives plot, the President made public references to the presence and/or expansion of FAM coverage as a direct Administration assurance to the public, references which themselves evidence the confidence that FAMs elicit.

Evidence: ??Expectmore.gov data for "number of successful terrorist attacks" measure. ??2007 National Household Transportation Survey (conducted 1st quarter FY 2008): FAMS question and FAMS result. ??Blue Lime results (PowerPoint): "Partnership in Security Project." ??FAA daily departure statistics (historical and projected).

SMALL EXTENT 7%
4.2

Does the program (including program partners) achieve its annual performance goals?

Explanation: With respect to the "Percentage level in meeting FAMS coverage targets for each individual category of identified risk," the program performed under its target in FY 2007, and exceeded its target in FY 2006: ??FY 2006: 101.7 percent (Exceeded goal) ??FY 2007: 96.2 percent (Large Extent of goal attainment. Exigent circumstances [i.e., the Heathrow-based liquid explosives plots] factored in heavily when, during the 4th quarter of the Fiscal Year, the program re-directed resources to reflect the domestic/international relative threat environment. Furthermore, on a per-mission basis, international missions consume a greater amount of resources than domestic missions). In the mathematical calculation of the risk categories metric, it is very likely that the FAMS will at times exceed 100%. Bearing in mind that the metric is a composite of performance in ten categories, and that the percentage coverage targets in each of the categories may or may not be 100%, an overall score that exceeds 100% should be viewed as a positive. It would represent that the FAMS exceeded their coverage goals, though it does not necessarily mean that the FAMS had to reallocate resources to do so. It could simply be a factor of working in the complex and dynamic environment of the airline industry. The FAMS sets coverage targets that are aggressive yet achievable, but if changes in the airline industry result in fewer total flights in some particular category, then the FAMS coverage percentage in that category will automatically increase even if the number of covered flights remains exactly the same. The dynamics of the airline industry, and all the factors therein represented that are beyond the FAMS control, are a major part of the reason that the FAMS set an annual timetable for review of the percentage coverage goals. The "Average annual rate of accuracy in Federal Air Marshals' firearms requalification scores" is a new measure, baselined in FY 2007; however, the program met the target in the first and second quarters of FY 2008. The attainment of the target in one of the past two years, the near attainment of the target in the most recent fiscal year, and the exigent, risk-driven circumstances show that the program achieved its annual targets (Large Extent) on the categories of identified risk measure to a Large Extent. Mid-year progress toward the firearms accuracy scores show partial but notable achievement of the annual goal. FAMS received the approval of TSA, DHS, and OMB to delete four annual measures, associated with the program's emergency post September 11th stand-up, that the program was able to meet easily, but had become obsolete given the program's maturity; it could be said that these would represent an easy "YES" with respect to target attainment, but the program did the right thing by focusing on other annual measures that are very meaningful proxies for the program's long-term outcomes. This correctness of action should not be penalized when Large Extent goal attainment for the more focused package of measures can be demonstrated, as above. The program's response to Question 4.3 addresses goal attainment with respect to the program's annual efficiency measure.

Evidence: ??Expectmore.gov data on categories of identified risk measure. ??Comparison of a typical international mission's and a typical domestic mission's relative consumption of the program's human resources. (Sensitive Security Information) ??The program will provide its 2nd quarter FY 2008 PPC data once the quarter has concluded.

SMALL EXTENT 7%
4.3

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: As an agency which (as detailed in response to Question 3.4) has very low "overhead" expenses by benchmarked standards (which is in itself a demonstration of cost efficiency), FAMS best demonstrates cost efficiency by optimizing the allocation of program personnel (FAM teams) to missions. Program management has determined a range of days in mission status appropriate for an individual Federal Air Marshal. FY 2007 was the baseline year for the program's new efficiency measure, "Percentage of FAMs providing the targeted number of flying days per year." Based on a retroactive analysis of historical (pre-FY 2007) data, the program targeted 75% as its performance for the baseline year. The actual performance (74%) was slightly under the target, however, the small degree of variance (-1%) is such that partial achievement of the target still represents notable ("Large Extent") efficiency. In a pilot program begun halfway through the fiscal year (April 1, 2007), the program took direct management action to increase efficiency as regards individual FAMs' days in mission status. A FAM's mission availability had historically been measured strictly on a by-office basis (rather than a by-FAM basis), and a FAM with too few or too many days in mission status could be overlooked in the by-office view. As the efficiency measure is calculated using a "twelve-month look-back" that ends with the last day of the reported quarter, the program's performance to be reported in the 2nd quarter of FY 2008 will represent the impact of the first year of direct, management action tied to this measure (April 1, 2007 to March 31, 2008). The program anticipates (and midyear data indicate) that the targeted management actions have improved efficiency over the past twelve months by optimizing resource allocation on a by-FAM basis. There was an increase in the "Cost per Flight per FAM" between FY 2006 and FY 2007 (data are sensitive security information provided to OMB), reflecting FAMS' carrying out a major policy change within the Administration - to provide a very substantial increase in a certain category of identified risk. In years without major Administration policy changes, efficiency is maintained by targeting growth in the "Cost per flight per FAM" not to exceed the OMB-identified rate of inflation. The program has also provided Evidence outlining instances of cost efficiencies (in hotel and rent costs) achieved by managers in the program's Directorates and field offices in the past year.

Evidence: ??Director's message announcing measurement of flight activity on a per-FAM basis (Sensitive Security Information). ??Field Operations and Flight Operations examples of cost savings. ??Data on "Cost per Flight per FAM" (Sensitive Security Information)

LARGE EXTENT 13%
4.4

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: The most significant aspect of the program is its performance with respect to the risk-based allocation of resources that prioritizes FAM deployments (flight coverage) on the basis of threat, vulnerability, and consequence. For this reason, OMB directed that the program have an independent evaluation of FAMS program performance with respect to: ??flight coverage risk categories ??the distribution of covered flights ??target levels of coverage. The program's independent evaluation by HSI (also discussed in response to Questions 1.4 and 2.6) considered other programs whose main business line is deterrence and/or protection via risk management. According to HSI, "We did not find any other organizations that face a similar challenge and apply significantly better tools or methodologies." Additional evidence of the program comparing favorably to others with similar purposes and goals includes: ??The program benchmarks favorably (in terms of measuring, targeting, baselining, and delivering deterrence) when compared to other programs having deterrence purpose and goals. The Evidence section details those four aspects of like programs: -Protection of Foreign Missions and Officials (Department of State) -Reactor Inspection and Performance Assessment (Nuclear Regulatory Commission) -Domestic Protectees (U.S. Secret Service) ??The program's precision marksmanship requirements (minimum 85 percent rate of accuracy, maximum 15 percent margin of error on the agency-wide requalification course of fire) represent a federally-benchmarked high. And, the program endeavors to achieve even higher rates of accuracy than the benchmarked high, as reflected in the associated performance measure. ??Public confidence in the program compares favorably with other programs in which BTS gauged public confidence. ??The program's limited "overhead" (with respect to both quantities of support personnel and levels of non-personnel costs) is indicative of relative efficiency when compared to DHS' other law enforcement agencies.

Evidence: ??HSI independent evaluation of FAMS Concept of Operations (document is classified). ??Discussion of PART measures, targets, baselines, and results for Protection of Foreign Missions and Officials, Reactor Inspection and Performance Assessment, and Domestic Protectees. ??Chart: Federal law enforcement agencies' agency-wide courses of fire - required rates of accuracy. ??BTS survey summary slides. ??Charts depicting benchmarking of support personnel levels and non-PCB funding.

YES 20%
4.5

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: In response to OMB's recommendation that the program should obtain an independent evaluation of its methods for risk analysis and resource allocation, FAMS contracted with HSI for the "Federal Air Marshal Service Risk Assessment." The independent evaluation (completed July 2006) meets the PART's standards of quality, scope, and independence; independent evaluations will also be conducted regularly (see response to Question 2.6 for details regarding compliance with all four 2.6 standards). HSI determined that the program ensures effective risk reduction by applying a reasonable and valid approach to analyzing risk, and allocating resources on its basis. An inter-agency steering group - convened by HSI and consisting of representatives from an array of law enforcement and counter-terrorism agencies - also considered the program's Concept of Operations to be reasonable. In its inter-organizational comparisons, HSI did not find evidence (in other agencies facing comparable risk management challenges) of significantly better risk management techniques suited to FAMS' objectives.

Evidence: ??HSI independent evaluation of FAMS Concept of Operations (document is classified).

LARGE EXTENT 13%
Section 4 - Program Results/Accountability Score 60%


Last updated: 09062008.2008SPR