A quick check
Where does your program sit right now?
Answer four questions to place a program on the curve and see the questions I would ask at that point.
Why programs often get cut on the flat part
As a district director, I watched programs end on the flat part of the curve for reasons that had little to do with whether they would have worked.
The first renewal decision comes before the outcome can move. Budgets run on a single year, and the outcomes a program is meant to move usually take longer, so the first vote on a program often happens while its numbers are still flat.
Startup money ends on its own schedule. A grant or pilot budget covers the first year or two, and the decision to carry the cost comes in the same window where the program looks weakest.
Teachers sometimes stop using a program before anyone decides anything. When they can’t see it working, many set it aside, and by the time the organization cuts it, it has often been gone from classrooms for months.
A change in leadership can end a program anywhere on the curve, including the steep part. A baseline can’t stop that, though it gives a new leader a record of where the program started and where it is now.
The level part
Many programs level off further along, which means the gains have held. Some keep adding gains for years, so check whether yours has leveled off before treating it as a plateau. Once the numbers level off, the risk is quieter. A program whose numbers look acceptable rarely comes up for review, so the organization can keep paying for parts that stopped adding anything.
When it goes well
A program makes it through the flat part when someone can show it is working before the outcome moves: teachers using it as designed, early indicators heading the right way. With that record, leaders renew it, and the program reaches the steep part, where the outcome finally moves. At the top, the gains become the new baseline. The team can then decide on purpose which parts to keep and where the next flat part begins.
What the research says
Open a line to see the research behind it.
Programs often take two to four years to run as designed.
Fixsen and colleagues reviewed implementation research across education, mental health, and social services and found that getting a program fully running in a new site often takes two to four years. Most evaluations happen before that point, so a program can look ineffective because it was measured too early. Fullan calls the early slump the implementation dip.
- Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M., & Wallace, F. (2005). Implementation research: A synthesis of the literature. National Implementation Research Network, University of South Florida. Source
- Fullan, M. (2001). Leading in a culture of change. Jossey-Bass.
Reform effects tend to grow with years of implementation.
Borman and colleagues’ meta-analysis of 29 comprehensive school reform models found particularly strong effects in schools that had implemented a model for five years or more. May and Supovitz followed one reform model in Rochester, New York, for 11 years and found yearly effects in elementary and middle grades reading and math that kept adding up.
- Borman, G. D., Hewes, G. M., Overman, L. T., & Brown, S. (2003). Comprehensive school reform and achievement: A meta-analysis. Review of Educational Research, 73(2). Source
- May, H., & Supovitz, J. A. (2006). Capturing the cumulative effects of school reform: An 11-year study of the impacts of America’s Choice on student achievement. Educational Evaluation and Policy Analysis, 28(3). Source
Programs often continue only in part.
Wiltsey Stirman and colleagues reviewed studies of what happens to new programs after initial support ends. Partial continuation was more common than continuing the whole program, and where fidelity was rated independently, fewer than half of the providers kept the practice up at a high level.
- Wiltsey Stirman, S., Kimberly, J., Cook, N., Calloway, A., Castro, F., & Charns, M. (2012). The sustainability of new programs and innovations: A review of the empirical literature and recommendations for future research. Implementation Science, 7, 17. Source
Short leadership tenures feed a cycle of adopting and dropping reforms.
In his study of urban districts, Hess describes policy churn. Superintendents, whose tenure is usually three years or less, face pressure to show they are making a difference, so new reforms keep arriving, and the churn seldom produces lasting change in classrooms.
- Hess, F. M. (1998). Spinning wheels: The politics of urban school reform. Brookings Institution Press. Source
How well a program is implemented shapes what it achieves.
Durlak and DuPre’s review of implementation research found mean effect sizes two to three times higher in programs that were implemented well. Positive results often came with implementation levels around 60 percent, and few studies reached levels above 80 percent.
- Durlak, J. A., & DuPre, E. P. (2008). Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology, 41(3–4), 327–350. Source
Measurement for improvement starts with a baseline.
Solberg, Mosser, and McDonald separate measurement for improvement from measurement for accountability and for research. Measurement for improvement includes establishing a baseline and checking newly changed processes, which is why the flat part starts with a baseline recorded before anything changes.
- Solberg, L. I., Mosser, G., & McDonald, S. (1997). The three faces of performance measurement: Improvement, accountability, and research. Joint Commission Journal on Quality Improvement, 23(3), 135–147. Source
Required data often can’t answer improvement questions.
Solberg and colleagues write that data for accountability “do not usually illuminate how the outcomes were achieved or how processes might be changed to improve them.” In the RAND studies, state test results generally arrived too late to adjust the current school year, and more than 80 percent of superintendents found local assessments more useful for decisions than state tests.
- Solberg, L. I., Mosser, G., & McDonald, S. (1997). The three faces of performance measurement: Improvement, accountability, and research. Joint Commission Journal on Quality Improvement, 23(3), 135–147. Source
- Marsh, J. A., Pane, J. F., & Hamilton, L. S. (2006). Making sense of data-driven decision making in education: Evidence from recent RAND research. RAND Corporation. Source
Imagining a failure in advance helps teams find its causes.
Klein developed the pre-mortem and reports that research by Mitchell, Russo, and Pennington found that imagining an event had already happened increased people’s ability to identify reasons for future outcomes by 30 percent. The worksheet asks the question both ways, which adapts Klein’s method.
- Klein, G. (2007, September). Performing a project premortem. Harvard Business Review. Source
Take the worksheet into your next team meeting.
This tool can also run as a full-day session with your team. See ways to work together.
If you want support working out where your program sits, I would love to hear about it.
Send a short note about the program or question and the decision you are facing.
I read every note myself and reply within the week.