Effects on Other Outcomes

Part two of this annotated bibliography offers an overview of literature published since 2014 on the effects of Outcomes-Based Funding (OBF) policies on other OBF-related outcomes, such as changes in enrollment and changes to institutional policies and practice. The literature is presented in two tables:

  1. Literature on OBF’s effects on other outcomes. Table 1 includes literature covering outcomes and effects of OBF other than degrees and attainment.
     

  2. Books that examine OBF implementation and other outcomes. Table 2 includes literature from two preeminent publications on outcomes-based funding. The table presents a summary of each book and identifies key topics covered by the authors.

 

Please note that some studies may not be relevant to current OBF policies given that many states have updated their policies since the studies were conducted.  For example, Indiana, Ohio and Tennessee have each updated their OBF formulas at least once since 2009.

NOTE

Ø – No significant effect

↑ – Significant positive effect

↓ – Significant negative effect

⇄ – Mixed effects: a mix of significant positive and negative findings and/or null findings

Study

0

Treatment Sample
Years Studied
Outcomes
Select Findings (compared to non-OBF states)
Considerations
 

LA

2005 - 2013

Institution finance

↑ In-State tuition and fees

Phase I

Findings

Institutions in LA vs. US Institutions

After the implementation of P/OBF, the in-district/in-state tuition and fees charged at both public two-year and four-year institutions in LA significantly increased compared to their counterparts in non-P/OBF states in all models.

Institutions in LA vs. Southern Regional Education Board (SREB) Institutions

After the implementation of P/OBF, the in-district/in-state tuition and fees charged at both public two-year and four-year institutions in LA significantly increased compared to their counterparts in SREB states based on the first model.

Institutions in LA vs. Tuition-Setting Institutions

After the implementation of P/OBF, the in-district/in-state tuition and fees charged at both public two-year and four-year institutions in LA significantly increased compared to their counterparts in tuition-setting states based on the first model.

Out-of-State Tuition and Fees

The authors found no significant changes of out-of-state tuition and fees charged at both two-year and four-year institutions across all comparison groups.

Considerations

This is a Phase I study that examines the changes in institution finance characteristics before and after the implementation of P/OBF policies. The authors called for future research to examine the variations of tuition and fees charged across academic fields.

Sample Description

The treatment groups include all two-year and four-year public institutions in LA. The comparison groups include all two-year and four-year non-P/OBF public institutions in the SREB, a national comparison group of all institutions in non-P/OBF states, and a national comparison group of institutions in non-P/OBF states where institutions have the authority to set their tuition levels.

AZ, AR, CO, FL, GA, IL, IN, KS, LA, ME, MA, MI, MN, MS, MO, MT, NV, NM, NC, ND, OH, OK, OR, PA, SD, TN, UT, VA

2005 - 2015

Enrollment of at-risk students

Ø Enrollment of at-risk students

Phase III

Findings

(Subgroup information is in the Sample Description below)

Any P/OBF vs. No P/OBF

The author found no significant impact of any kind of P/OBF policies on the enrollment of “all undergraduates” population groups and on all first-time, full-time FAFSA filers. For Hispanic first-time degree-seeking students, having any kind of P/OBF policies has a significant and positive impact on enrollment.

At-Risk P/OBF vs. No P/OBF

States with any at-risk metric in their P/OBF policies experienced a significant increase in the enrollment of African American students, compared to non-P/OBF states.

Having any at-risk metric in their P/OBF policies has no impact on the enrollment of any group of the first-time, full-time FAFSA filer students, but it has a signficant and positive impact on the enrollment of first-time degree-seeking students who are African American.

At-Risk P/OBF vs. P/OBF without At-Risk Metric

States with any at-risk metric in their P/OBF policies experienced a decrease in the enrollment of non-Pell Grant recipients and students aged 24 years and younger, compared to states without any at-risk metric in their P/OBF policies.

States with any at-risk metric in their policy experienced a significant increase on the enrollment of students whose family income is less than $30,000, but no change on all other first-time student populations, compared to states without any at-risk metric in their policy.

P/OBF without At-Risk Metric  vs. No P/OBF

The author found no significant impact of P/OBF policies without any at-risk metric on the enrollment of all eight population groups and all first-time student populations.

 What is the difference in “any” versus “other” when describing comparison groups?

Considerations

This is a Phase III study that examines the metric-specific effect of P/OBF policies on the enrollment of at-risk students.

Sample Description

The whole sample includes 526 four-year public institutions in 49 states (28 P/OBF states). Treatment groups include four-year institutions in 16 P/OBF states with a financial component encouraging underrepresented students. The author examines the following undergraduate population groups: Pell Grant recipients, non-Pell Grant recipients, underrepresented minority students, African American students, Hispanic students, white/Asian students, students aged 25 years or older, and students aged 24 years or younger.

TX, WA

2005 - 2015

Institution finance, Performance metrics

↓ State appropriation per FTE at MSIs in TX

Ø State appropriation per FTE at MSIs in WA

⇄ Performance metrics at MSIs

Phase II

Findings

On average, minority-serving institutions (MSIs) in TX are allocated less and MSIs in WA are allocated the same in per-student state funding after P/OBF compared to non-MSIs.

Texas:

All community colleges in Texas experienced funding increases under P/OBF.

MSIs had experienced higher funding levels in the decade prior to performance funding, but under P/OBF, they received less funding than non-MSIs. However, the downward trend started in the years immediately preceding P/OBF.

MSIs scored higher in the aggregate of the performance metrics than non-MSIs but scored almost equivalently to non-MSIs across different metrics.

MSIs performed slightly better than non-MSIs on transfers and gateway courses in math, while non-MSIs scored better on developmental course in math, 15 credit hour accumulated, and degrees awarded in critical fields.

 

Washington:

Overtime, the funding level of MSIs fluctuates but there is no evidence that MSIs have been disproportionally impacted due to declines in overall state funding.

Though the P/OBF policy in WA does not have a metric for underrepresented students, there is no clear evidence that MSIs experienced disadvantages in their scores based on the performance metrics compared to non-MSIs.

Non-MSIs scored better on the following metrics: credit attainment, credit hour milestone, and college math readiness.

MSIs scored lower than non-MSIs on the total performance metric points.

Considerations

This is a Phase II study which examines TX and WA’s P/OBF policy on the student outcomes included in each state’s formula funding.

Sample Description

The treatment group includes on average, 20 two -year MSIs in TX and 10 MSIs in WA. The comparison group includes 30 non-MSIs in TX and 23 non-MSIs in WA.

Gándara and Rutherford (2017)

CO, FL, IL, IN, ME, MI, MN, MS, MO, NM, OH, OK, OR, PA, TN, TX, VA

years

Enrollment of at-risk students, Admission rates

*The effects of any premium for underserved student populations in P/OBF:

Ø Admission rates

Ø Enrollment of at-risk students

Phase III

P/OBF policies with any premium on outcomes for underserved student populations show no impact on admission rates.

P/OBF policies grouped by the inclusion of a premium on outcomes for minority students, a premium on outcomes for low-income students, or a premium on outcomes for both minority and low-income students showed no significant impact on admission rates and low-income student enrollment.

A premium on outcomes for low-income students in P/OBF policies has a negative impact on black and Hispanic student enrollment.

The number of years P/OBF policies include a premium on outcomes for minority and low-income students has a positive impact on admission rate.

The number of years P/OBF policies include a premium on outcomes for minority students or minority and low-income students has a negative impact on black student enrollment.

The number of years P/OBF policies include a premium on outcomes for low-income students or minority and low-income students has a positive impact on Hispanic and low-income student enrollment.

This is a Phase III study that examines different premiums in P/OBF policies and their effects on underrepresented-minority student enrollment. This study compares the enrollment within P/OBF states and focuses on the variations in the design of premium metrics. The authors called for future research to examine the different weights and institutions’ reactions to these premiums.

The treatment group includes 187 institutions in 17 states with premiums for at-risk students in their P/OBF policies. The comparison group includes 64 institutions in 9 P/OBF states without premiums for at-risk students, including AR, AZ, KS, KY, LA, NJ, SC, SD, and WA.

authors

AR, AZ, FL, IN, IL, LA, ME, MA, MI, MN, MO, MS, NV, NM, ND, OH, OK, OR, PA, TN, UT

years

Institution finance

Ø State appropriation per FTE in four-year institutions in P/OBF states vs. non-P/OBF states

↓ State appropriation per FTE for MSIs in P/OBF states vs. MSIs in non-P/OBF states

Ø State appropriation per FTE in MSIs vs. non-MSIs in P/OBF states

⇄ Mix results among individual P/OBF states

Phase III

On average, the authors found that the introduction of P/OBF has no significant impact on the changes in state appropriations across all institution types.

Full Sample Before and After P/OBF policies

The authors found no significant difference before and after the implementation of P/OBF policies in the all-state sample.

MSIs in P/OBF states vs. non-P/OBF states

Compared to MSIs in non-P/OBF states, MSIs in P/OBF states experienced a decrease in the average state appropriations per FTE.

MSIs vs. Non-MSIs in P/OBF states

Compared to other institutions in P/OBF states, MSIs in P/OBF states experienced no changes in the average state appropriations per FTE.

State-Specific Findings

12 MSIs lost significantly more funding in LA, MO, NM, OH, and TN, while 9 MSIs had significant funding gains in AR, PA, and UT after the implementation of P/OBF policies. Changes in the remaining 13 states are not significant.

This is a Phase III study that addresses the variations across state policies by examining the state-specific effects. Having a national sample poses challenges to the parallel trends assumption so the authors called for synthetic control groups to address this limitation.

Two treatment groups include185 four-year public institutions in 21 P/OBF states, consisting of 31 MSIs and 154 non-MSIs. Three comparison groups include 287 four-year institutions in non-P/OBF states, consisting of 83 MSIs and 204 non-MSIs.

authors

TX

years

Total PBF revenue earned by each student

⇄ State appropriation per metric by student characteristics

Phase I

This study examines the appropriation allocated from student success points. There is a significant gap in appropriation per student across student groups:

female students procured more appropriation than male students;

Asian students procured the most appropriation compared to African American, Hispanic, or white students;

students who are 19 and younger procured more than older students;

students with high school diplomas procured more than GED holders;

full-time students procured more than part-time students;

students with GPA 3.0 and above procured more compared to students with lower GPA;

Pell Grant recipients procured more than non-Pell students;

students who were placed in developmental education procured more than their college-ready peers; and

students with lower levels of math deficiency procured more.

Full-time status and Pell Grant receipt are the two strongest positive predictors of the amount of P/OBF appropriation procured per student.

This is a Phase I study which examines the average rates of P/OBF appropriation procured through success points across student characteristic. This study is more exploratory than explanatory and it only includes student data from one cohort in one district following the implementation of TX’s P/OBF policy.

The full sample includes 5,878 first-time-in-college students from the Fall 2007 cohort in one urban community college system in TX.

authors

A northeastern state

years

Institutional responses

↑Policy encourages diversity

↑Policy promotes evidence-based accountability

↓Limited institutional involvement

↓Data capacity restricts outcomes reporting

Phase I

Positive perceptions

P/OBF encourages diversity and an expanded focus also includes non-Black student population.

P/OBF raises awareness to better report and use institutional data.

Negative perceptions

State did not involve the HBCU during the policy development.

State did not account for the HBCU’s limited resources and the limited data collection capacity that hinders the outcomes reporting.

Outcome measures are not tailored well to the population this HBCU is serving.

Implications on policy design:

to include more institution-specific and mission-aligned measures;

to involve institutions in the development or re-design of policies; and

to consider institutions’ data capacity during the development of policies.

This is a Phase I study that examines the perceptions of institution before and after the P/OBF policy was implemented. Data was collected from one HBCU hence the context of findings is unique to this HBCU, the P/OBF design being implemented, and the state policy environment. The policy type is uncommon because it includes metrics such as private support and faculty diversity.

One public, non-selective four-year HBCU in a northeastern state

authors

AR, CO, HI, IN, KS, KY, LA, MA, MI, NC, NM, NY, OH, OK, OR, PA, SD, TN, TX, VA, WA

years

Insitution finance

Ø Revenues per FTE

Ø Expenditure per FTE

⇄ Institutional grant aid per FTE

⇄ Institutional financial characteristics

Phase I

Public Four-Year Institutions in P/OBF states vs. non-P/OBF states

Four-year institutions in P/OBF states have fewer total per-FTE revenues and per-FTE revenues from tuition and fees and state/local appropriations; fewer total per-FTE expenditures; fewer per-FTE expenditures on research, student services, and institutional support; fewer funded grants but higher tuition discount rate; and lower listed tuition and fees.

Public Two-Year Institutions in P/OBF states vs. non-P/OBF states

Two-year institutions in P/OBF states have fewer total per-FTE revenues and per-FTE revenues from state/local appropriations and auxiliary enterprises; fewer total per-FTE expenditures; fewer per-FTE expenditures on research, public services, student services, institutional support and auxiliary enterprises; no differences in per-FTE institutional grant aid factors; and more Pell Grants per FTE.

This is a Phase I study which examines the impact of P/OBF policies on institution finance. The authors recognized the variations across states in terms of the selection of metrics, the degree to which a policy is designed to be mission-specific, etc. but the implications of these variations weren’t fully addressed in this study. Moreover, the authors pointed out that the expenditure and revenue data reported to IPEDS lacked clarity and consistency for certain categories.

The full sample includes 538 public four-year institutions and 1,113 public two-year institutions in 21 P/OBF states and 29 non-P/OBF states.

authors

OH, PA

years

Institutional response

↑ Funding unpredictability

↑ Shifted focus to retention and completion

↓ Policy do not rewards

⇄ Perceptions of "competition for funding"

Phase I

Reported effects include:

State policymakers and administrative staff voiced concerns for the unpredictability of funding;

Institutions shifted focus to student retention and completion; and

Some reported mixed perceptions of policy as it entices the competition for funding across institutions.

This is a Phase I study that examines the perceptions of institution before and after the P/OBF policy was implemented.

28 interviews in three universities in OH and 19 interviews in two institutions in PA.

authors

OH, PA

years

Institutional response

⇄ Faculty and staff support

↑ Institutional mission alignment

⇄ Perceptions of equity in funding levels

Phase I

Reported effects include:

Faculty and administrative staff reported awareness of policy;

Little evidence of strong resistance was found;

Institutions responded differently to policy by changing internal financial incentives;

There were connections between policy and institutional strategic planning;

Enrollment still matters for institutions to sustain their operations; and

Institutions voiced concerns about inequity in allocation funded through P/OBF models, college being more selective and weakening of academic standards.

This is a Phase I study that examines the perceptions of institution before and after the P/OBF policy was implemented.

28 interviews in three universities in OH and 14 interviews in two institutions in PA.

authors

TN

years

Institutional response

↑ Completion-related programs and initiatives

⇄ Perceptions of institutional mission alignment to policy

⇄ Perceptions of "collaboration vs. competition" among institutions

↑ Concerns of equity in funding levels

Phase I

Reported effects include:

Completion-related programs and activities are occurring at study institutions;

Campus indicated their clear commitment to improving student outcomes;

Institutions reported mixed perceptions on institution-specific mission alignment to policy;

The institutions reported that premiums for at-risk populations are not enough to fund necessary services; and

The authors identified some unintended consequences include: restrictions in admissions to college, a weakening of academic standards, and selective enrollment.

This is a Phase I study that examines the perceptions of institution before and after the P/OBF policy was implemented.

104 participants in Tennessee, working in higher education related fields and 19 event observations

authors

TX, NC

years

Institutional responses

↑ Decision making driven by P/OBF

↑ Programming changes of academic programs and student services

↑ Need for more faculty and staff support

⇄ Institutional awareness

⇄ Levels of changes in internal practice and procedures

⇄ Perceptions of emphasis on performance

Phase I

Reported effects include:

Decision-making and planning are driven by P/OBF policies;

Varied levels of changes of internal awareness of P/OBF from faculty and administrators are identified;

Only two of four colleges made changes to internal practices and processes in response to P/OBF;

P/OBF policies have no influence on students’ performance requirements in individual classes but there are small changes occurred in what student services were offered, e.g. more learning resources or academic supports;

Data capacity requires some staffing changes;

Participants voiced concerns on emphasis on performance might lead to “tunnel vision”; and

Board and public also had concerns about whether a comparison based on performance metrics would bring negative effects on institutional reputation and image.

This is a Phase I study that examines the perceptions of institution before and after the P/OBF policy was implemented.

Four small rural comprehensive community colleges in Texas and North Carolina

authors

IN

years

Admission rates Average ACT scores, Enrollment of at-risk students

↓ Admission rates

↑ Average ACT scores

Ø Enrollment of at-risk students

Phase I

Average Policy Effect

Overall, the admission rates decreased and the average ACT scores increased after the implementation of OBF in IN across all comparison groups. There is no change in the enrollment of minority and low-income students after OBF policy is implemented.

Lagged Policy Effect

With a one-year or a two-year lag, the admission rates decreased and the average ACT scores increased after the implementation of OBF in IN across all comparison groups.

With a two-year lag, a negative effect appeared on the changes in the enrollment of minority students when the outcomes were compared to surrounding out-of-state public institutions and private institutions in IN.

The increase in the enrollment of low-income students appeared after one year of implementation, compared to private institutions in IN.

This is a Phase II study that examines the average policy effect and lagged policy effects in IN, but it only examines a one-year and a two-year lags in outcomes.

The study included 14 public four-year institutions in IN. The authors used three comparison groups: the frist group contained 29 public institutions from KY, MO, and WI; the second group included 21 private institutions in IN; and the third included 55 institutions from IA, IL, KY, MO, MN, and WI.

Books

 
Authors and Year
Topics
Book Summary

Implications for racial equity

Book discusses implications of P/OBF for racial equity in states implementing the policy, as well as in states where the adoption of P/OBF was under discussion. These implications include

  1. potential changes in funding for MSIs, which are dependent on formula design (TN, OH & TX), and

  2. challenge to close opportunity gaps and reduce institutional asymmetries if MSIs’ realities are not acknowledged (FL, TX, CA & MD).

Policy implementation and synthesis of research on P/OBF

The book studies the implementation and impact of P/OBF in Indiana, Ohio and Tennessee. Summary of findings:

  1. Three states focused on funding incentives to improve outcomes. Other strategies include assistance to campus leaders, although they varied in terms of campus penetration,

  2. Institutions use a variety of strategies to determine how to respond to the demands of P/OBF. Key to effective deliberative processes are organizational commitment and leadership, effective communication & collaboration, timely relevant data, and time for deliberation,

  3. The most common institutional responses to P/OBF include changes to developmental education, improvements in college articulation and transfer, revamping advising and counseling services, and enhancing tutoring services and supplemental instruction. Authors caution that it is difficult to determine whether these changes respond exclusively to P/OBF are related to other concurring policies,

  4. Obstacles to institutional responsiveness to the policy include: academic and demographic composition of student body, inappropriate performance-funding metrics, and insufficient institutional capacity (e.g. lack of timely and relevant data). Authors found little evidence of systematic state efforts to improve institutional capacity to respond to P/OBF, and

  5. The most common unintended consequences are increased selectivity and weakening of academic standards.