Assets for Independence Act Evaluation:
Design Phase Final Report
August 9, 2000

1.	Overview
	1.1	Statutory Mandate
	1.2	Basic Evaluation Design
	1.3	Organization of This Report

 

This report describes the plans for—and estimated costs of—data collection and data analysis to carry out the congressionally mandated evaluation of the Assets for Independence Act (AFIA, Public Law 105-285, enacted on October 27, 1998).

1.1 Statutory Mandate

The Assets for Independence Act provides demonstration funding for state and local individual development account (IDA) programs. IDAs are special savings accounts in which account-holder deposits can be matched when used to purchase homes or automobiles, start or expand businesses, or obtain further post-secondary education. The Act provides federal matching funds, and funds for program operating expenses and for evaluation of the IDA demonstration projects. The evaluation activities are specified in Section 414 of the Act, which calls for the evaluation to be carried out for the Department of Health and Human Services (HHS) under contract with an independent research organization. [1]

Section 414(b) of the Act identifies the following six specific "factors to evaluate," to be addressed by the research organization "in evaluating any demonstration project" conducted under AFIA :

1. The effects of incentives and organizational or institutional support on savings behavior.
2. The savings rates of individuals . . . based on demographic characteristics including gender, age, family size, race or ethnic background, and income.
3. The economic, civic, psychological, and social effects of asset accumulation, and how such effects vary among different populations or communities.
4. The effects of individual development accounts on savings rates, home ownership, level of post-secondary education attained, and self-employment, and how such effects vary among different populations or communities.
5. The potential financial returns to the Federal Government and to other public sector and private sector investors in individual development accounts over a 5-year and 10-year period of time.
6. "The lessons to be learned from the demonstration projects conducted under this title and if a permanent program of individual development accounts should be established."

This section also states that the evaluation shall address "other factors as may be described by the Secretary" of HHS.

Section 414(c) of the Act addresses the methodological requirements of the evaluation, specifying that "for at least one site, [the research organization shall] use control groups to compare participants with nonparticipants."

 top of page

---

1.2 Basic Evaluation Design

This report is one of a series of documents produced during the first-year design phase of the AFIA evaluation. Two previous reports, the Concept Paper and the Evaluation Design Plan, detailed the research activities that would be necessary to address the "factors to evaluate" listed above.[2] As described in these earlier reports, the evaluation activities fall within the following areas or "components":

• The program and participant tracking and monitoring includes collection and analysis of information regularly maintained about the status of program participants, the flow of funds into and out of the accounts, and administrative operational details, including staffing and costs.
• The process analysis includes on-site observation of program operations, interviewing of program staff, and examination of written materials to determine how the program was implemented and how the program operates.
• The experimental impact analysis includes collection and analysis of information on program-eligible persons randomly assigned to a treatment group (participating in the program) and a control group (not participating in the program) for the purpose of estimating the effects of the program on its participants, relying on random assignment as the means of obtaining groups that are comparable on both observable and unobservable traits and thus as the means of enabling one to attribute to the program any differences in observed outcomes between the two groups.
• The nonexperimental impact analysis includes collection and analysis of information on persons participating in the program and a comparison group of persons identified in data as not participating in the program, for the purpose of estimating the effects of the program on its participants, relying on statistical techniques to take account of demographic and socioeconomic differences between groups and thus to isolate the effects of the program.
• The in-depth participant interviewing involves extended personal interviews with program participants to examine their understanding of the program, their reasons for participating, and their experiences as participants.
• The benefit-cost analysis includes collection and analysis of information on the benefits of the program to its participants and the costs of the program to the federal government (i.e., to federal taxpayers), to other public sponsors (including state and local), and to private funders.

Time frame of activities and deliverables

The evaluation activities will extend over a five-year period, from October 2000 through September 2005. The findings will be presented in a Final Evaluation Report submitted in September 2005.[3] Interim Evaluation Reports will be submitted in September of 2001 through 2004. The second and third of these interim reports will be timely with respect to congressional consideration of the Act's reauthorization, which will occur in the context of the fiscal year 2004 budget cycle.[4] Project meetings or briefings with HHS staff are scheduled for October of 2000 through 2004, with a final project briefing planned for September 2005.

Exhibit 1-1 shows the proposed schedule of major planning activities, primary data collection activities, reports, and briefings associated with the evaluation.

Site selection

One aspect of the evaluation will be based on data from all AFIA grantees, as compiled by each grantee for the purpose of submitting to HHS the annual progress reports required under Section 412 of the Act. Specifically, the second factor ("the savings rates of individuals . . . based on demographic characteristics including gender, age, family size, race or ethnic background, and income") will be analyzed using a database that combines the available information from all grantees. This aggregated database will be constructed from the program and participant information collected by grantees through their use of the Management Information System for Individual Development Accounts (MIS IDA) or equivalent MIS software.

All other aspects of the evaluation will involve a selected number of sites (i.e., grantees), as follows:

• The process analysis will involve eleven grantees. In a first phase, the experimental site (discussed below) and another five grantees selected from the first and second cohorts (FY 1999 and FY 2000) will be studied through multi-round site visits during 2001 and 2002. The experimental site along with a newly selected group of five grantees, the latter to be selected from the third and fourth cohorts (FY 2001 and FY 2002), will then be studied during a second phase that occurs in 2003 and 2004.
• The experimental impact analysis and the benefit-cost analysis are both assumed to involve one grantee—the experimental site—to be selected from among the first or second cohort of grantees. This site would also thus become one of the multiple sites for the process analysis (in both phases of the process study) and the in-depth participant interviewing (described below). The experimental research sample is assumed to include 666 cases, with 333 assigned randomly each to the treatment and control groups. The experimental data collection will include a baseline interview (conducted prior to random assignment) and two waves of follow-up interviews (at the 12th and 24th months of participation for each case).[5]
• The nonexperimental impact analysis will involve a randomly selected sample of the grantees in the first and second funded cohorts. Only the experimental site will be excluded from selection for the nonexperimental analysis. (This exclusion is to avoid an undue respondent burden on the program participants in the experimental site.) The nonexperimental sites will be selected on a probability-proportional-to-size (PPS) basis, through a multi-stage cluster design aimed at producing a nationally representative sample of 1,600 AFIA program participants. These program participants will be subject to four waves of follow-up interviews (at the 12th, 24th, 36th, and 48th months of participation for each case).
• The in-depth participant interviewing will involve three sites. A total of 90 interviews will be conducted—at each site 15 interviews at each of two periods, during 2002 and 2004. These three sites will consist of either: (a) the experimental site and two of the ten process analysis sites; or (b) three of the ten process analysis sites, if no experimental site is selected.

Exhibit 1-2 shows the expected configuration of grantees involved in each of the evaluation components. Note that site selection issues will be addressed in a series of memoranda, as shown on the evaluation schedule (Exhibit 1-1).

Exhibit 1-2: Evaluation Role of AFIA Grantees

	Program and Participant Tracking and Monitoring	Process Analysis	Experimental Impact Analysis	Nonexperimental Impact Analysis	In-depth Participant Interviews	Benefit-Cost Analysis
Experimental Site (1)
Other Process Analysis Sites (10)		*		**	***
All Other Grantees				**

* If no experimental site is selected, the process analysis will involve a total of twelve grantees, six in each of two phases of visits.
** Those grantees in the first and second funded cohorts (FY 1999 and FY 2000) will be subject to selection for the nonexperimental impact analysis on a probability-proportional-to-size (PPS) basis.
*** In-depth participant interviews will be conducted at three sites. These sites will be either: (a) the experimental site and two of the process analysis sites; or (b) three of the process analysis sites, if no experimental site is selected.

For background reference, Appendix A contains a profiling of the AFIA-funded programs in the FY 1999 cohort, showing the characteristics of grantees and subgrantees.

Data collection

Primary data will be collected from large numbers of respondents for the process analysis, the experimental and nonexperimental impact analyses, and the in-depth participant interviews. We assume that OMB clearance will be necessary for these data collection activities. As shown in Exhibit 1-1, the OMB clearance package is to be submitted to HHS by the research organization in December 2000, so that clearance can be obtained in time to commence the primary data collection activities in April 2001.

With respect to program and participant tracking and monitoring, the costs properly associated here with the evaluation are the costs of analyzing data aggregated across grantees and the costs of presenting the findings of such analysis on an annual basis. The costs of collecting such data from grantees and the costs of providing the annual progress reports called for in Section 412 are discussed in Chapter 2, but are not considered here as evaluation costs.

Cost estimates

It is important to note at the outset that this evaluation design has been developed with the aim of meeting the statutory mandate for evaluation, as set out in Section 414 of the Assets for Independence Act, without immediate regard to the funding available to support these evaluation activities. The Act itself sets aside for evaluation purposes 2 percent of the annually appropriated amount. There is uncertainty, of course, over the amount of future annual congressional appropriations for AFIA. There is also uncertainty over whether additional evaluation funding might become available, either from within the U.S. Department of Health and Human Services (HHS) or from non-federal sources. The strategy adopted here has been to proceed with the evaluation design as appropriate to meet the statutory requirements, projecting the cost of these planned activities so that HHS can ultimately decide how best to use the available evaluation resources. We do not attempt in this document to establish the level of priority that should be assigned to each of the evaluation components or to the activities within each component.

The cost estimates presented in this report reflect the following assumptions.

• Each year (1 through 5) refers to a 12-month period from October through September.
• An annual inflation adjustment of 4 percent is applied to staff labor; and a 3 percent annual adjustment is applied to other direct costs (ODCs).
• For labor fringe and overhead, a combined rate of 110 percent is applied to the inflation-adjusted staff labor subtotal.
• Travel costs are projected on the basis of assumed trips from Boston to either Washington, DC (for meetings or briefings with HHS staff) or to Kansas City, MO (for on-site data collection).[6]
• For general and administrative (G&A) costs and fee, a combined rate of 25 percent is applied to the sum of total staff labor and total other direct costs.

Hours of staff labor have been classified according to the following categories established by the Program Support Center (PSC) of HHS for its task order contracts:

• Class I – Senior
• Class II – Associate
• Class III – Intermediate
• Class IV – Junior
• Class V – Editorial
• Class VI – Clerical

The hourly rates estimated for all staff are within the maximum rates established by PSC for each labor category.

 top of page

---

1.3 Organization of This Report

The following six chapters of this report address the major planned components of the evaluation:

• Chapter 2: Program and Participant Tracking and Monitoring
• Chapter 3: Process Analysis
• Chapter 4: Experimental Impact Analysis
• Chapter 5: Nonexperimental Impact Analysis
• Chapter 6: In-depth Participant Interviews
• Chapter 7: Benefit-Cost Analysis

Each chapter explains the role that such activities will play in meeting the statutory mandate, describes a plan for data collection and data analysis, and provides an estimate of the associated costs. Where specific data collection instruments are to be used, these appear in the corresponding appendices. Chapter 8 is a summary.

This report represents a collaborative effort by staff at both Abt Associates and the Center for Social Development of Washington University in St. Louis, under the direction of Gregory Mills and Michael Sherraden, respectively. The principal authors by chapter are as follows:

• Chapter 1: Greg Mills (Abt Associates)
• Chapter 2: Lissa Johnson (Center for Social Development)
• Chapter 3: Michelle Ciurea and Doug Welch (Abt Associates)
• Chapters 4 and 5: Donna DeMarco and Greg Mills (Abt Associates)
• Chapter 6: Margaret Sherraden and Amanda Moore (Center for Social Development)
• Chapter 7: Mark Schreiner and Shirley Porterfield (Center for Social Development)
• Chapter 8: Greg Mills (Abt Associates)
  

 top of page

---

Notes

[1] The Act authorizes $25 million for each of five fiscal years (FY 1999 through 2003), with 2 percent of the annually appropriated amount earmarked for the evaluation activities required under Section 414. (An HHS-proposed amendment to the Act would set the annual amount for evaluation at the greater of $500,000 or 2 percent of the appropriation.) The annual appropriation for both FY 1999 and FY 2000 was $10 million. The program is administered by the Office of Community Services (OCS), within the Administration for Children and Families of HHS. OCS has thus far awarded 40 grants (from FY 1999 funds), to 38 local organizations and to the states of Indiana and Pennsylvania. A second cohort of grantees (from FY 2000 funds) will be announced in the summer of 2000. [Return to Text]

[2] See Gregory Mills, Michael Sherraden, et al., Assets for Independence Act Evaluation: Design Phase--Concept Paper, Abt Associates Inc., Cambridge, Mass., December 1, 1999; and Gregory Mills, Michael Sherraden, et al., Assets for Independence Act Evaluation: Design Phase--Evaluation Design Plan, Abt Associates Inc., Cambridge, Mass., February 17, 2000. [Return to Text]

[3] We assume that the Final Report on this evaluation, when submitted by the Secretary to the Congress, will comply with the requirement under Section 414(d)(2) of the Act that "Not later than 12 months after the conclusion of all demonstration projects conducted under this title, the Secretary shall submit to Congress a final report setting forth the results and findings of all reports and evaluations conducted pursuant to this title." We recognize that this interpretation may require a technical amendment to the Act, construing the phrase "after the conclusion of all demonstration projects conducted under this title" to mean "after the conclusion of the last fiscal year of funding authorized by this title." [Return to Text]

[4] We assume that the Interim Reports on this evaluation, when submitted by the Secretary to the Congress, will comply with the requirement for interim evaluation reports under Section 414(d)(1) of the Act. We recognize that this interpretation may also require a technical amendment to the Act, as the proposed timing of these Interim Reports (in September of each year) does not strictly conform to the language of the Act, which calls for reports 90 days after the end of each calendar year. The pending HHS-proposed amendments to the Act address this issue, calling for the Interim Reports to be submitted to the Congress in December of each year. [Return to Text]

[5] We assume that a suitable experimental site can be found and that this mandated aspect of the evaluation will indeed take place. We recognize that there are serious concerns about the feasibility and generalizability of this evaluation component. We have estimated its cost so that an informed decision can be made about whether to proceed with this activity--and, correspondingly, whether to proceed with the proposed benefit-cost analysis, which would be undertaken in the experimental site. If no experimental site is selected, the process analysis would be re-designed to include six grantees in each of the two phases, or twelve grantees in total. [Return to Text]

[6] For trips from Boston to Washington, DC, the following assumptions were used: round-trip coach air fare, $587; per diem for lodging, $118; and per diem for meals and incidental expenses, $46. For trips from Boston to Kansas City, MO, the corresponding assumptions were as follows: round-trip coach air fare, $1,190; per diem for lodging, $85; and per diem for meals and incidental expenses, $42. Ground transportation costs were assumed to be $70 per day. [Return to Text]