A Measure of Success

by Justin Boles

I nearly fell out of my chair. I had been invited to advise a startup ministry on how to launch in a way that would give it the best chance for success. I was one of a number of advisors who’d been flown in from across the nation to participate in this task force. The ministry included an incredibly compelling purpose, a talented leader, and a sound business plan. The leader astutely sought the group’s advice on outcomes: What should we measure? How should we measure those things?One outspoken, well-meaning member of the group interjected, “I wouldn’t worry about that right now. You have enough to do without thinking about statistics—you have lives to save.”

Great leaders don’t embark on the outcomes journey primarily to satisfy reporting requirements or to win the next grant. They know outcomes are the gateway to fulfilling the mission that compelled them into the difficult but rewarding work that God called them to.

I couldn’t let go of this statement, so I spoke up—outcomes should have animated our entire conversation that day. An organization can’t accurately determine what activities to engage in or what resources are necessary without first defining what a program’s outcomes should be.

What are outcomes?

It’s challenging to make sense of outcomes with.out discussing the other factors in the equation. Every program, product, or service involves three things: inputs, outputs, and outcomes.

Inputs are resources. They’re people, money, facilities, and such. Those resources, or inputs, make outputs possible.

Outputs are activities. In the life-transformation work that Citygate Network members do, outputs are things such as providing food and shelter, teaching classes, discipling new believers, counseling clients, and arranging career opportunities. Outputs are generally quantitative and demonstrate need. For example, “We taught career employment classes to 22 men” illustrates an output.

Outcomes, on the other hand, are qualitative and demonstrate progress, change, and transformation. Proper outcomes should also be time-limited. An example of an outcomes statement might be, “Seventy-two percent of the graduates of our New Life Program found careers earning living wages within six months of graduation and have maintained employment for at least one year.”

This outcomes statement includes a measurement—72 percent, a qualification—living wages, and a limited time—for at least one year. When placed alongside a statement describing the status of clients at program entry, it can tell a powerful story of life transformation.

So What?

Tracking outputs demonstrates what you do and helps define what inputs are needed. Many good leaders stop with outputs because they’re relatively easy to generate. You simply count people who are within your purview or products that you delivered. How many people passed through your program? How many meals did you dish out?

But outcomes are even more valuable. Outcomes data and the follow-up process itself can help your ministry do a number of other things: Diagnose problems in your programs, motivate staff by letting them know the long-term impact of their work, serve as a connection point between your ministry and former clients (which is valuable in case additional services or referrals are needed), increase accountability, demonstrate legitimacy, affect legislation in positive ways, and show the quality of your results and how those results have been sustained.

In short, outcomes can help you tell the story of life transformation. That, in turn, attracts resources to continue the cycle of life transformation. Foundations seek this information, so a funding boost is a wonderful byproduct of producing solid outcomes as well. Funders want meaningful and measurable results of their grants, not only to highlight grantees’ accomplishments, but also to illustrate their own accountability and appeal to donors.

Last year the representative of a major foundation approached me to help him gather information about a Citygate Network member applicant. “The mission included some interesting outputs in the grant request,” he explained, “but we’re looking for outcomes.” I desperately wanted to send him outcomes data, but because I didn’t have that information available, I did my best to paint a narrative based on what I knew of that mission’s work.

But great leaders don’t embark on the outcomes journey primarily to satisfy reporting requirements or to win the next grant. They know outcomes are the gateway to fulfilling the mission that compelled them into the difficult but rewarding work that God called them to.

Like any worthy endeavor, developing outcomes measurements, collecting them, and tracking them is a challenging process with many pieces.

First, you must decide what should be measured. The outcomes you decide to measure should be relevant to your ministry’s mission. For instance, several potential outcomes could be drawn from this mission statement: “To utilize Christ-centered programs to move men and women from poverty, homelessness, addiction, and trauma toward sustainable, comprehensive life transformation.” This particular organization has comprehensive goals in mind, so it would not want to define its outcomes too narrowly.

In short, outcomes can help you tell the story of life transformation. That, in turn, attracts resources to continue the cycle of life transformation.

Categorizing Outcomes

“Domains” are broad headings that outcomes could be nested within. Domains, in this example, might be economics, housing, sobriety, and health. A number of factors affect how many domains an organization may choose, such as software limitations, ministry philosophy, redundant reporting requirements, and others. For instance, sobriety could be a sub-domain within health.

Citygate Network, through the work of a member-driven outcomes task force and the help of outside consultants, has developed the following outcomes domains that we believe most of our members could benefit from tracking.

Relationships

Relationships with God and other people are vital to the sustainability of any results. A relationship with Christ is not only transformational, but it also offers support through very difficult circumstances. Within this domain, a ministry might develop a tool for evaluating spiritual growth, as well as a connection to a healthy community that supports an individual’s goals and provides some accountability—such as a church, AA group, or rescue mission alumni group.

Health

Health is a particularly broad domain. In the context of a rescue mission or kindred ministry, it relates to physical condition, mental health, and sobriety. Interestingly, it’s not unusual for some outcomes data to dip from a person’s program entry to partway through the program. This happens with health as clients report worsening physical conditions, when in reality they have simply become more aware of existing health problems through medical exams and proper care. Another factor may be their physical and mental response to withdrawal from alcohol and drugs.

Economics

The most significant economic result is progress toward securing an income. For some people, that means government assistance; for others, it’s a job that provides a living wage. Additional outcomes to consider within economics include soft skills, money management skills, and insurance, but the primary measurement should be individuals’ ability to sustain themselves long-term.

Housing

Housing might seem like a no-brainer, but we must consider all the ways someone can be sheltered to create a spectrum, assigning a value to each level of housing. On one end of the spectrum is temporary shelter, and at the other end is home ownership. Subsidized housing, transitional housing, staying with family, and renting fall in between.

While the ability for an individual to pay his or her own rent is an outstanding accomplishment and may be the best scenario, the equity and leverage gained toward home ownership demonstrates another level of achievement.

Each domain you measure will likely have subdomains specific to your goals, so a survey could be developed with multiple-choice questions.

Each answer would be weighted differently, which enables the instrument to demonstrate progress.

For instance, in the Housing domain, an appropriate question to ask in a follow-up interview might be:

Which statement best describes where you stay?

The answer choices and their weights—which would not be displayed on the survey—might look something like this:

I own or am making payments on a home (value: 4).I am renting a home or apartment where my name is on the lease (value: 3).
I am staying with friends or family (value: 2).
I live in a temporary or transitional home with supportive services (value: 2).
I live in permanent housing that is provided or subsidized by the government (value: 1).
I am homeless or living in a shelter (value: 0).

Your organization may word the question another way or value the answer choices differently, but the idea is to demonstrate progress at various intervals in a person’s journey toward independence and wholeness.

Along with collecting outcomes data, record specific client characteristics and relate that information to the data. Over time, patterns will begin to emerge that can help improve your service to particular groups and inform your aftercare process. For instance, addiction recovery programs may benefit from reporting outcomes by characteristics such as gender, age, duration of substance abuse, type of substance abuse, race/ethnicity, and education level.

How Should We Collect Outcomes Data?

While an organization can obtain out.comes data from clients in person, by telephone, via video conference, from an app, or by mailed questionnaires, the most accurate follow-up method is for a trained staff member to meet with the former client face-to-face because this enables the staff member to use his or her senses to better assess the situation. This information can help you identify key influencers and paint a more comprehensive picture.

For example, clients who left one program and then went to another program may have different outcomes when compared with clients who received no further interventions. Other potential influences on client outcomes could include family trauma, interpersonal relationships, living conditions, and so forth.

In addition to gathering the core outcomes data you’re after, surveys can address these influencing factors by asking both direct questions such as, “Have you received services from another program since you left?” and open-ended questions, such as “Did anything else happen after you left that may have affected your success?”

Obvious advantages emerge for the data being captured electronically, even if information is fed into software after being collected by other means. In electronic form, data can more easily be analyzed, transferred, extrapolated, stored, and disseminated. Some software modules have even adopted the outcomes domains that Citygate Network recommends, and we’re aware of more options in development that utilize artificial intelligence to gather all kinds of related data.

While the software piece is important, other significant decisions involve strategies for staying connected to clients after they’ve exited your program. Staying connected to your program graduates drives positive results, reduces recidivism, and supports data validity through increased participation in outcomes surveys.

Staying connected to your program graduates drives positive results, reduces recidivism, and supports data validity through increased participation in outcomes surveys.

When and How Often Should Data Be Collected?

Surveys should certainly be conducted at entry, at least once during the program (although several surveys will tell a more complete story), at program exit, and at least once between 3 and 12 months after program completion. Some Citygate Network member ministries conduct surveys several times throughout a two-year period.

While conducting more than one follow-up survey with former clients increases the potential for collecting important information, it also increases costs and staff time. On the other hand, the sooner clients are contacted after departing from a pro.gram, the easier it is to locate and survey them.

Whom Should We Survey?

To reduce costs, ministries with many clients might only follow up with a sample of their client population. In this case, use a random sampling procedure that gives every client an approximately equal chance of being included in the sample. Random sampling produces more valid findings and makes the follow-up process more credible with foundations and other reviewers. Consider ways to minimize selection bias in outcomes reporting. The clients you’re able to maintain contact with are more likely to report positive results, so foundation leaders may feel that the results are not as positive as indicated.

To address this: (1) Report findings, but footnote the percentage of graduates who could not be reached for a particular time frame; (2) Attempt to follow up with all graduates, then randomly sample the surveys—including those you could not contact—and consider any who could not be contacted as unsuccessful (“unsuccessful” may be represented by the number value not increasing); or (3) Count all those with whom the ministry could not maintain contact as unsuccessful (this last option is the most conservative and probably the least attractive alternative).

Launching outcomes collection can seem overwhelming, but please don’t let that keep you from starting. Begin conservatively and build your outcomes program slowly as you learn what works for your organization.

Through Citygate Network you have access to many fellow members who are well-versed in outcomes collection and analysis. We envision a not-too-distant future when most of our members are collecting data and uploading it to a central repository, where it can be analyzed and used to refine program practices, garner more funding for ministry, and authoritatively tell the story of what God is doing through you.

For more information on outcomes collection, visit www.citygatenetwork.org/outcomes.

Justin Boles is Citygate Network’s vice president. He’s passionate about helping mission-minded organizations become more effective. You can reach him at jboles@citygatenetwork.org.

To read more articles like this from our bi-monthly members-only magazine Instigate email Sam Edwards about membership today.