WREC 2014 Closing and Plenary: How to Build Reliable Evidence and Inform Policy

WREC 2014 Closing and Plenary: How to Build Reliable Evidence and Inform Policy


HILARY: Good morning. I’d like to welcome you
to the final plenary session and the last day of 2014 WREC. Before I make a few
closing remarks, and offer a few final
housekeeping notes, I’d like to invite Vince
Kilduff up to say a word about the upcoming NAWRS
annual workshop. VINCE: Thank you. Thank you. Hi, I’m Vince Kilduff from
the NAWRS Board of Directors. On behalf of the NAWRS
President Michael Bono I want to thank the
OPRE for the opportunity to invite you all to attend
the 54th annual workshop and research academy
that NAWRS is holding. The flyer shows the dates
August 17th through the 20th. And then we are back on
the East Coast this year in Providence, Rhode Island. You also see our theme there,
Putting the Pieces Together. The NAWRS program
committee, co-chaired by George Falco of the
New York State Office of Temporary and
Disability Assistance, and Heather Hahn of
the Urban Institute, are currently fine tuning
this year’s program for you. We appreciate the presentation
proposals we’ve received this year, a number of them
from people in this room. Thank you. Michael and I also want to
thank official sponsor, Public Consulting Group,
for their support. Our web address is on the slide,
and I’ll also have some cards out there at the
registration desk. If you haven’t had the
opportunity to attend in the last few years, I urge
you to check out our past programs that are on the site. In closing, we look forward
to you joining us at the NAWRS workshop this year
for a robust program and many fruitful discussions. Thank you. HILARY: Thank you Vince. I hope that you have
all enjoyed the last two days of productive
conversation and networking. To me the WREC plays many roles. It highlights and charts the
progress of ongoing research. It stimulates ideas
for future research. It provides a forum
for peers to network and learn from one another,
and it offers a dedicated time and space away from
the usual work routine to reflect on the
broad field of research and practice our work
collectively operates in. Over the last 17 years WREC
has grown considerably. In length in terms of time, in
size in terms of participants, and in depth in terms of content
and issue areas addressed. WREC continues it’s evolution
as we move to a bi-annual schedule. As the late Maya Angelou
said “All great achievements require time.” I’m confident this
additional time will allow WREC to continue to fill its many
roles in a meaningful and high quality way. A few housekeeping
points before we move on to our last
plenary session. The conference evaluation
is available online, and will also be
sent to your inbox. We really do take conference
feedback seriously, especially now as we’re
transitioning to a bi-annual schedule and have some
flexibility to make changes. I hope you will take the time
to let us know your thoughts. Conference presentations
are available by request, just send a message to
the email address found throughout your program book. Videos of live stream sessions
will be available online later this year, just check
back to our website. As we have been saying
throughout the conference please stay connected. On the back cover
of your program book you can see a variety of ways
to keep in touch with us, through Twitter and LinkedIn,
by joining the OPRE listserv, and by visiting the
WREC and OPRE websites. Last, I’d just like to take a
moment to thank those who have made this WREC the successful
convening it has been. Thank you to the
Grand Hyatt hotel staff who has done an excellent
job of taking care of us this week. Thank you to ESI, our
conference contractor, who has done an amazing job of
coordinating this conference and making sure we all knew
where we needed to be when. Their good work and
expert assistance assures everything
runs smoothly, and we are truly
appreciative of that. Lastly, thank you to all of you. Both to those who are
present in this room and to those who are
joining us virtually. Your participation is
what fuels this conference and makes it worthwhile. We look forward to
continuing the conversation and seeing you again in 2016. With that I’d like to invite
our panelists up for the last plenary, including our session
moderator Rebecca Maynard. Dr. Maynard is a university
trustee chair professor of education and social
policy at the University of Pennsylvania. She is a leading expert
in the design and conduct of randomized control trials
in the areas of welfare and education policy. She has directed numerous
large scale evaluations of welfare policies
and practices, from visiting programs,
education reforms, and employment and
training programs. She has also been a leader in
the development and application of methods for conducting
systematic reviews of evidence on program effectiveness. She holds a B.A. in Economics from
the University of Connecticut, and a Ph.D. in Economics
from the University of Wisconsin Madison. Her full bio, as well as the
bios or all of our panelists are available on our website. With that I’ll turn
it over to Rebecca. REBECCA: Thank you. It is a real pleasure
to be here this morning. We have one more panelist,
Don where are you? HILARY: He’s in
the lobby somewhere. REBECCA: Yeah, I think we … Okay well let me … I’m going to start the
introduction so we stay on time. This is a really
exciting plenary session. I want to welcome you. We are going to talk about
how to build reliable evidence to reform policy and practice. We are going to be
treated to a conversation among a group of
individuals who have been at the task of building
and using reliable evidence to reform policy for
many years, as signaled by the subtitle of this session,
which is Lessons From 40 Years Of Welfare Research. You may wonder why we’re
still all up here walking around and talking to you today, but
now more than ever the public is demanding evidence that’s
it’s tax dollars are being well spent. And moreover, we have growing
needs and shrinking budgets, which are increasing the
importance of being highly strategic in how we invest our
scarce resources to improve the health and welfare
of our citizens. A good strategy requires that
evidence about the likely returns on these investments
that we’re making be available to us. This morning’s plenary is
about the journey that brought us to the point where
experiments are increasingly looked to first in the quest for
evidence on what works and what doesn’t work, and how well. We are also going to hear about
the challenges that are still before us as we seek to improve
policies and practices based on this kind of evidence. We have four
panelists and we hope that we will have
a fifth commenter by the end of the session. You have full bios on these
distinguished individuals on line and in your packet. I’m going to be brief because
I want us to spend the time on the conversation, which
I think is going to be very interesting for you. Judy Gueron is President
Emerita at MDRC. She was a pioneer in the
field of experimentation with social welfare programs,
and has probably done more than any other person to create
a norm of using experiments to address questions
of cause and effect. Judy recently co-authored a
book with a second panelist, Howard Rolston, that
chronicles the 40 year history of the conduct
and use of experiments in social welfare. Howard is a principal
associate at Abt Associates, and he was a former
long time administrator at the U.S. Department of
Health and Human Services. Our third panelist
is Ron Haskins who is a senior fellow at
the Brookings Institution, but who spent a very
distinguished career as a senior staff member and
Staff Director of the House Ways and Means Sub-committee
on Human Resources. Fourth, we are treated to
have Don Winstead with us. He is presently principal
of Don Winstead Consulting in Tallahassee,
Florida, but Don has done work across all
levels of public service, being case worker in his
early years in the profession, and working up to senior
positions in state and federal government including
serving as Deputy Assistant Secretary of Health
and Human Services. Our fifth panelist, whom
we hope will be joining us in offering some
concluding remarks, is Mark Greenberg who is
acting Assistant Secretary for the Administration
of Children and Families. Mark sent his regrets late last
evening that he is not going to be here early this morning
because he has been called up to the Hill for some very
important work that you all … you know what that is like. Hopefully Mark
will be here and be able to join in the final
part of the discussion. Then following the conversation
among the panelists, we’re going to have 15 to
20 minutes to let the audience engage with these
distinguished individuals. I want to remind you that this
session is being live streamed so we do have some people
who will be participating in that way. We will be able to invite those
of you who are watching us, not in this room, to participate
in the Q&A at the end. With that I’m going to turn
it over to the panelists, and I want to begin the
conversation talking about why randomized experiments have
been sustained over this 40 year period. Why have we been able to keep
this car on the rails here? I’m going to ask Judy
to kick us off here. JUDY: Thank you Rebecca. Is this on? REBECCA: Okay, it’s on. JUDY: There’s many reasons and
I’m going to highlight five. The first is the
interest in evidence. 40 years ago
welfare, called AFDC, was an unpopular, open-ended
entitlement program. Funded in part by the federal
government, and in part by the states. As states got increasing
flexibility, starting in 1981 under Ronald Regan, to
redesign the program both the feds and the states
were interested in learning did reforms increase
work, reduce welfare cost, or save money. Those are the types
of questions that call for determining if reforms
led to changes in behavior over and above what
would have happened due to normal economic changes,
or the behavior of people interested in getting
off the rolls. The second is evidence
of the unique value of random assignment studies
in answering those questions. As these 40 years unfolded
it was increasingly clear that random assignment studies
first were feasible, legal, ethical, and the
most reliable way to answer those
kinds of questions. At the same time, it
became increasingly clear that alternatives fell
short, and could provide misleading information
on who benefited, or what programs were affective,
including the normal statistics that welfare
administrators looked at. Things like how many
people got a job, or how many people
left the rolls. Could they figure
out from such numbers how much was due to the
economy, or the program? Third, the actions of
state administrators. State administrators stepped
forward to volunteer, and in some cases seek out,
participation in such studies. In some cases, particularly the
gentleman sitting over here, Don Winstead,
fighting to defend it when such studies
were under attack. In the book Howard and I wrote,
your colleagues in the states are really the
heroes of the story. Later there was a
prodding by HHS and OMB, who insisted that random
assignment studies were the quid pro quo for getting
waivers to welfare policy, but at the beginning
in the first 5 and maybe even 10 years, the
interest was getting truth and this was the way to do it. Fourth, money. Evaluations cost money
and especially prior to 1996 welfare
studies benefited from two unusual factors. First was the peculiarity
in the AFDC law that basically put up
federal matching money. For every dollar, they put up
a dollar in an open-ended basis for federal research. The second was actions of
the Ford Foundation to take on studies of less popular,
or really to keep such studies alive when the federal
government didn’t choose to. Fifth point, this momentum was
sustained by several factors. The first was a
growing community of supporters,
state legislators, Congressional staff,
OMB, state officials, client advocates like Mark
Greenberg at the time. A small number, and then
ultimately a growing number of academics who believed
that this was the way to find out whether social
programs were effective. Second, that the people who
really believed in such work and fought for it in
HHS, in OMB, in MDRC, and other research firms, stayed
in their jobs for the 30 year period, sometimes
the 40 year period, that we’re talking about,
and therefore could sustain the focus on what
we needed to learn, and the importance
of doing it well. Finally, evidence
that these studies, not just because of some
gimmick random assignment, but because they addressed
important issues, and that the results were
replicated in state after state in very real world
contexts, proved to be uniquely
convincing and had an unusual impact on
policy and practice. The evidence that they
were useful and used created a positive feedback
loop for doing more studies. REBECCA: Howard, you were in
the trenches all that time. HOWARD: I think Judy
has done a wonderful job summarizing the key points. I would just expand on one
thing that she mentioned that I think was really crucial. That was the acceptance,
and the support, of the broader policy
and practice community so that people
really came to expect to see findings portrayed
in a certain kind of way. They began to look for
experimental evidence. That I think has to do with
a really key part of why experiments are so
powerful beyond their power of their rigor to produce
reliable evidence, and that’s their transparency. It’s not like people were
asked to believe a statistician that this was the right
way to interpret the data. That the transparency
of the results … we know underneath
it’s more complicated, but it’s basically a simple
comparison of a group that was exposed to the program, and a
randomly assigned group that was not. I think that … in that sense, the method
along with presentation of the results really
created a powerful force within the broader community
of not just researchers, but of policy makers,
practitioners, and that that was really
critical to why things were able to be
sustained, and why they had such a big influence. REBECCA: Let’s move on and
talk a little bit about the way in which this actually worked
in the states and in the federal government. Did the research influence
legislation, policy, and practice? Don, why don’t you … DON: Based on the experience
that we had in Florida, first of all there’s
the [inaudible], but in a couple of ways. Our first experience with
a large random assignment and evaluation was
a project in Florida called Project Independence. My colleague Judy Moon,
who is in the room, was a key part of that
implementation in Florida also. This was an evaluation
that was conducted by MDRC. There were participants
selected in nine counties, that were randomly selected
to be representative of the state of Florida. Then the participants were
subject to random assignment and over 18,000
participants were involved in the
overall evaluation. Importantly, it was
not a waiver program. We didn’t do it
because Howard made us. We did it because
we thought it was important to answer the
key questions about program effectiveness, and also
because both the state agency and the auditor
general had tried to answer that question
non-experimentally. All we ended up with was
controversy about who’s findings were right. I suspect that the
answer was none of those findings were correct
based on what we later learned. When Judy made reference to some
of the controversy in Florida … Let me briefly
quote from Judy’ and Howard’s book. “The challenge of selling random
assignment in California”, which had some challenges
too, “proved a tepid warm up for what happened next in
Florida where the toxic combination of
gubernatorial politics, a concerned state legislature,
and ill-considered decision in a separate study”,
referencing I think the Texas child care study, “fed an
explosion of inflammatory press that almost led the state
legislature to ban control groups, and in the process
both jeopardize a major federal research project and potentially
poison the well for future studies.’ The first contribution, I think,
was by convincing the Florida legislature … and Judy Gueron did dozens
of mini-tutorials one on one with legislators on the
importance of this research, and by convincing the
Florida legislature that this was a
legitimate and correct way to look at these issues, I
think the first important contribution was not to derail
the research going forward. Also, a key finding of the
Project Independence evaluation was a difference between an
early cohort and a later cohort in terms that had different
access to child care because of budget constraints, and
found that the early cohort with access to child
care had significantly better impact than
a later cohort. This really led to the design
of a later demonstration in Florida, the Family
Transition Program, which was one of the first
time limited welfare reform demonstrations which
was a waiver program. The Project Independence
finding really informed the design of the
Family Transition Program, and particularly the importance
of transitional benefits. There we were able to document
that transitional benefits like an enhanced earnings
disregard, higher asset limits, etc., produced a
positive result in employment and earnings, and
also, did not cost more in terms of welfare payments. That the fact that more
people went to work actually reduced welfare payments
relative to the control group. This then, by having that
information, some of which from a state database
which we used, we were able to then justify
a good design in going state wide in the TANF program when
that was implemented in 1996. You can draw a direct
line between the findings of the Project
Independence evaluation, to the Family
Transition evaluation, to the design of Florida’s
state wide TANF program. I think because of the results
that came out of the Family Transition Program,
and the focus on that, a lot of states who were
interested in things like enhanced
earning disregards, and different asset limits,
and strong work requirements look at Florida as
being at least one of the models for things
that they were thinking about in their TANF program. I think in that way it not
only directly led to the design of Florida’s program today,
but without the evidence from the random assignment evaluation
we would never have been able to prove the case to the
appropriators in Florida for them to see that not only
was this a better way to go, but it was also a cost
effective way to go. REBECCA: What do you say to the
smart econometrician who comes in and says “I’ve got
another way to show you.” Did that not happen? DON: We had some smart
econometricians in Florida, and we also have
some econometricians. In fact, some of the
early findings, both that the state agency did, and
that the auditor general did, and the auditor general
hired a professor from a local university to
do some economic modeling, a lot of the modeling
was done but all it led to was controversy,
and claims, and counter claims in a political
environment where people had a very pronounced
political stake in the outcome. Anything that left wiggle
room, and room for debate produced debate. The findings from both the
Project Independence evaluation and the Family Transition
program evaluation were seen by people on both sides of
the political debate as being authoritative, and as being … there was still some
controversy about what to do, but there wasn’t any lingering
controversy about the evidence base on which those
decisions were made. REBECCA: From where you sat
Ron what did this look like? RON: I first want to clarify
that what I know about this stuff, I didn’t actually
experience this 40 years ago, I wasn’t even born yet,
but these other elderly panel members have informed me, so
I can really talk about real stories that they’ve
told me about. I wanted to clarify that. REBECCA: The age
discrepancy is noted. RON: Let me make a
distinction, there are people like the people on
this panel who read studies and they are really extremely
aware of random assignment. I would say … I wouldn’t doubt
it on this panel, I know all of them personally,
that would say that unless it’s been demonstrated
by random assignment, it hasn’t been demonstrated. Even the National
Academy has taken that position in
a recent report. That was not the case back
then, plus members of Congress do not read research
reports generally. What I think happened was that
the whole series of experiments that the states ran …
and 41 states had … even Florida did, and other
states got waivers’and I think by the time we passed
welfare reform in 96, 40 states, or 41 states, had
actually done experiments. The majority of which I
think were random assignment, Howard can correct me
on that if he wants to. What happened was that a kind
of an atmosphere developed, that the members of Congress
who had never read a study, and probably could not
define random assignment, came to believe that
what works is work. And, of course, that was
the heart of the debate that Republicans said
we can make people work. They want to work. They will work. We just have to arrange
things the right way. We’ve got to punish them
if they don’t because some people are going to
respond to punishment. We’re going to
help them find jobs, so that’s a positive thing,
and we’re going to make sure we’re paid, that was
the goal of the President. I think it was a way of
developing an atmosphere that people do want to work. They will work if you set it
up right, and not only that, but an MDRC study,
and other studies, showed you could
actually save money. In fact, Don described
a situation like that, and boy that’s music
to the ears of anybody in a legislature,
they love that. I think it was this
more general sense. My answer is yes, I think
that the research definitely influenced the policy, and it
can continue to do so today. I hope we have time on this
panel to talk about … I think the situation
has changed a lot now. I think there are many
staffers in Congress, and even members of Congress who are
well aware of random assignment. A lot of them easily say
“evidence-based policy.” I think we’ve had
kind of a revolution, and we should talk
about this more later, on evidence based policy. It’s moved off in a lot
of different directions. This administration I
think is the most evidenced based administration ever. They’ve done more to
push evidence based policy, but it all started with welfare
reform, random assignment, the role of MDRC
[inaudible], and the other companies. Yes, it had a big
influence on legislation, and I believe it will have
an even bigger influence on legislation in years ahead. REBECCA: Yes, absolutely. DON: One comment, Ron
mentioned the National Academy and I think one
of the things that was part of the
debate in Florida, and part of the discussion,
that I think the story would be incomplete
without acknowledging the role of Rob Hollister. Rob Hollister chaired the
National Academy panel on youth employment
programs in the studies that have been done in Florida
before Project Independence. Judy helped us get
in touch with Rob, and sent those studies to him
and asked for his judgment. He wrote back and was fairly
direct in his assessment of the value of the studies
that we had done, then laid out the methodological case
for why those approaches were not valid, and what
we needed to do. We presented that
information from Dr. Hollister to an oversight committee
of the Florida senate. They became convinced, because
of the case that he laid out, of the way that we
should go forward. I think that Hollister’s
description of the importance of the work, and also the
methodologies involved, in a way that the Florida Senate
could understand and accept was a real important part of
what developed in Florida. REBECCA: Can you, or Judy,
in a nutshell recap what it was that Hollister said
that really was convincing? JUDY: First I would just
say that Hollister was almost a voice in the
wilderness in academia. Academics were late to
this table, to this feast, if you will. There were a few
who sang this song, but most academics, led by very
powerful statistically trained leaders in the
econometric field, felt that they could
conquer these subjects through complex
statistical analysis, and really thought that random
assignment was not necessary. Over these years they failed. Becca was one of the
early analysts who looked at an early experiment
and showed that the effort to do this
statistically would not lead you to the right answers. What Hollister looked at in
the late 70s were a plethora of youth studies launched
under a 1977 law … Maybe a billion
dollars was spent on innovative programs,
and evaluations. Close to 500 … in real dollars
… those were big dollars, weighty dollars. That was a lot of money. At the end, when the National
Academy of Sciences got together they said they really
couldn’t figure out whether all these fabricated comparison
groups were showing you what people would have done
without the intervention. They couldn’t sort out whether
the comparison group was really the same and therefore gave you
an honest answer by saying what would have happened had people
not been in the program, or differed in unmeasured
ways such as motivation, or where they were located, or
the context in which they were living. People, as Don said, couldn’t
reach a solid conclusion. This thing is called
selection bias. Are they biased in some
way, those studies. The panel, the National Academy
of Sciences panel and report in 85… it was really a landmark. It was clung to by those of
us defending these studies, it said “We can’t reach a
conclusion for those studies.” At the same time the
Department of Labor, which had funded many studies
on employment and training programs, pulled
together a panel that concluded that with
tens of millions of dollars studied there was
not one number they could defend before Congress. That’s really bleak. You don’t want to go
down that road again. I think those kinds of … those two landmark
studies, the Department of Labor and the
National Academy, just changed the whole
direction of this work. RON: Just a brief
comment that picks up on my theme of how much
things have changed. The scholarly world
certainly was not … didn’t play an important role
in developing this movement we’re describing here. One of the greatest
examples is education. I can remember a report from
the National Academy that described the field
of education research as “The vast wasteland
of education research.” They actually said
something along the lines “We know hardly
anything about education and effective
practices, and so forth, because they do junky studies
if they do studies at all.” So, the creation of the National
Institute of Education Science in the early 2000s
under President Bush and required by
previous legislation so it was bipartisan, is
really a big turning point. I don’t think
whatever happened, and it wouldn’t
have the support, and almost everybody associated
with it played some role in the welfare
reform experiments, or at least knew them very well. We’ve really had a revolution
in education research, and we have many random
assignment studies that have already been reported,
and are going on now. We are gradually beginning … I think it’s like building
a mountain out of pebbles. We are gradually beginning to
accumulate evidence from random assignment studies about
reading, about math, about all sorts of things,
and it’s because … and Becca is grinning because
she was an important member of the National Academy
of Education Sciences. It is really one of the great
institutions in America, and continues to do
spectacular work. As a result we’re learning,
and I don’t think any of that could have happened without
the welfare reform history that we’re describing. REBECCA: There must be something
that these experiments can’t tell us. There must be things they
tell us, and tell us well, and things they don’t
tell us so well. I’m wondering if we can
talk a little bit about that, because I’m confident
that there … If we did nothing but
experiments we wouldn’t get to the end game
where we want to be. Howard you want to
kick us off on this? HOWARD: Sure. I’m going to begin by
very quickly reading a list of things we’ve learned. It’s not a complete list
which would take too much time. I’m going to read it quickly
because I think a lot of people here are familiar with this, but
mostly because my main purpose it to show how many
things we learned, and how many things we
learned in how many different dimensions. * Workforce programs
that include jobs search increased employment
earnings and reduced welfare. * Programs focused
on basic education also did, but their effects
were typically smaller. * The education focused programs
were also more expensive. * With a few
exceptions effects were largest at about 2 to
3 years after people entered the program and
declined to 0 by 5 to 7 years. * The size of effects
varied somewhat but they were markedly
similar across individuals more and less disadvantaged. * By themselves the work first
invasive education programs typically failed to move
family income up or down. * Earnings increases were
offset by welfare decreases. * Wage subsidies combined
with work requirements did increase family
income but only while the subsidies
were in effect. * Positive effects
for wage subsidies typically were larger
for those with weaker expected employment outcomes. * In the absence of
wage subsidies programs had no patterned effects
on young children. * Subsidies produced
positive effects on school outcomes
for these children while they were in effect. * There are indications of
negative effects on adolescents in all types of programs. * Earnings impacts were
bigger in better economies which was not a
forgone conclusion. * With very few exceptions
programs focused on increasing employment retention,
especially those that used case managers to help deal
with employment issues have not been successful,
there is some indication that wage subsidies have been. That’s a lot of information. There are few fields, I think,
where you would get so many people … the consensus … almost everybody would
agree with those things. If you’ve answered
a lot of questions, what haven’t we
answered so well? I want to talk about
three different areas because I think they illustrate
the different kinds of points. One area where I think we
didn’t answer questions so well is related to the value
of occupational skills training for low income mothers. There were some very
good studies of that, so I don’t want to say
that we learned nothing, but they were very limited
compared to the studies that we had on … Related to either basic
education or to job search. I think that’s an area where
more work certainly can be done. A second reason not only was the
work more limited in that area … it was partly limited
because a lot of women on welfare couldn’t get into
occupation skills training programs. That along with other research,
and other experience, really led to new approaches,
to real innovation in trying to overcome some
of these limitations that were observed in earlier
studies of occupational skills training. I think you have an example here
where there weren’t as many studies and the
world changed a lot. Another area I
want to talk about is the area providing subsidized
employment to individuals. Sometimes now called
transitional jobs, it goes under a lot
of different names. Here’s a case where 20 years
ago I think we could have … there would have been
a lesson on the list. The lesson was from 4 or
5 very good experiments, that women who participated
in employment subsidy, wage subsidy … Sorry, subsidized jobs, it
increased their longer term earnings. More recent studies
have been fewer, but they’ve been negative. They don’t show that. I think one of the lessons from
that is that the world changes. I personally think one of the
ways in which the world changed here is that low wage women
have lots more work experience than they did when
the studies were done. It’s harder to increment
that experiment experience by putting somebody
in a subsidized job. That’s the hypothesis. I think this is another
area where I think the world changing suggests if we’re
going to pursue this strategy we need to find more
effective ways of applying it. A third area which illustrates
another thing is we learned a lot about job search. We learned that it
was highly effective, but we learned about
it most frequently in the context of being
combined with something else. Especially in a lot
of the later studies, so that our best studies of
which forms of job search are better than other
forms of job search. Job search is now ubiquitous
related to [canif 00:40:45]. If we want to understand
which forms work better we really have to
go back to studies that were done in the 1970s. There is no reason to believe
that those studies are necessarily particularly
relevant today. I think that part of the … The point I’m trying to
make is it’s not so much … it is in some cases that areas
were addressed less well, but the main thing is if we
really want to have confidence in any of these findings …
and I purposely wrote them in the past tense. I wrote them in the past tense
to emphasize the fact that if we really want to have
confidence in these things we can’t just assume that
the findings hold up. In what is a very dynamic
world changing a lot in terms of demographics,
changing a lot in terms of the
economy of the country and the economies of localities. It’s important to press
forward with further studies. I’m pleased to say that all
three of the areas identified are areas that the Office
of Planning, Research, and Evaluation is pursuing. I think it’s also important,
even to look back at things and think about trying
to replicate them. We could be very confident
in the findings in terms of how something
would work in 1994, where I think we have to
wonder would we actually get the same results in 2014. REBECCA: Judy I know that
MDRC was known for it’s experimentation, and a
difference in means from that method, but I also think you
were known in the research community for something else,
which was that those studies were rarely just about
that difference in means. I’m wondering if you can pick
up on Howard’s point about both changing context, but also
what else you need to be aware of when you do these studies. JUDY: Howard’s
summary was terrific. I’d add that what he’s
talking about is the great progress that was made
in learning, on average, whether these
programs cause change. That’s a triumph of science,
but we’ve made only limited progress, and not
through a lack of effort, in dissecting the factors that
contribute to program success. As a result I think we
don’t know enough about how to replicate, or improve,
programs to make them more effective. In all of these
studies there were persistent efforts to try
to look at, statistically or through various
kinds of implementation, and even at the
graphic research, what were the nature
of these programs, because what we learned
was that impacts often vary across offices for example. When you look at
multi-site studies … I emphasized earlier that
results were replicated over different states, but there’s
also a lot of variation. What explains it? How can we make programs better? The findings that Howard cited,
and that I saluted earlier, basically are about progress. Findings that are modest in
scale, can we ratchet those up? Can we make them larger? What would it take to do that? How can we improve programs? There’s a real effort now
going on to try to look at what explains variation in
impact across sites. I don’t know whether
that methodological effort, looking at all of the
experiments that are out there, and trying to look at ones
that are in many locations, and figure out,
really rigorously, what explains success. The other thing
that I would stress in thinking about what
remains to be done, is making random assignment
a more useful management tool for you. Studies take time, sometimes
because you have to follow people for years to know
whether they succeed in … sometime follow
thousands of people, but you have questions that
are very pressing and now. You are everyday
testing out new things. I would urge you to
run little experiments. I’m very intrigued by
behavioral economics, trying to combine psychology and
economics to realign everything from forms, and letters, and all
the procedures that you do all the time. If Wal-Mart can test the
results experimentally of what the difference is if you put
ladies underwear in the front of the office, or men’s
shirts, or whatever the store. They do that all the
time using experiments. You can do it too. It doesn’t take a lot and the
use of administrative records to track people gives
you a powerful tool, so get involved with people
who will help you do that. The other area where I’d
really like to see progress is that … right now there’s
a real dichotomy between what experiments show and
what performance measures in the whole performance
management movement suggests. That’s not very useful. These are two different
strategies that have as a goal improving results. They really go in different
directions right now. I think one of the
challenges of the future is to try to understand
ways that these two kinds of information can
reinforce each other. Right now they don’t and they
push in different directions. REBECCA: Don, as someone who is
on the front-line and actually is … it seems like one
foot in, one foot out of this experimenting
at the local level. DON: Two points I think. First of all in terms of one of
the points that Judy said about smaller scale, but
there has also been … there was a part of this
conference was on rapid cycle evaluations and
other techniques. One of the lessons that I
learned from the Project Independence
Evaluation was it was very frustrating to be
in this large evaluation and to find out the
impacts 5 years later. As an operational
tool that was really limiting because the life cycle
of major research studies, and the life cycle of policy
change are out of sync. States typically don’t wait
5 years to change things. Every 2 years there’s a new
class of legislators and they want to change things now. One of the things that we did
with the Family Transition program was we developed
a database so that the … and the field offices that were
involved had no knowledge about this, but in our headquarters
office we had a team. One of the things
we were tracking was the administrative
data on a month by month basis for the folks
who had been randomly assigned. We knew in near real time
how things were trending. When it was time to make some
important program decisions we didn’t have the formal
evaluation impact, but we knew the basic
information about that. Having that capability to really
know on a more abbreviated time scale was very important. The other thing I’d like to
stress is that while random assignment evaluations are
important and irreplaceable really in terms of answering
the impact question of what was the difference attributed
to the program, they’re not very good
at telling you why. The why question is critically
important for people who are administering programs. I think one of the things
that certainly MDRC did, and others that do
this research, pay a lot of attention to are
implementation studies. The study of in addition
to what are the impacts, the more context question
about why things are different. There is a … At least when I was a
state administrator, I sometimes found myself
having the illusion that when a memo
came across my desk and I signed off on a
policy change, that somehow within a couple
weeks by magic it would be implemented
all over the state. I was often disappointed to find
out that somebody hadn’t read the memo after a while. That question of to what extent
is your implementation faithful to your program design is a
critically important question. The implementation study
really helps get at that, because if there are not any
differences between the program group and the control group
it may be because the program treatment didn’t work, or
it may be that nobody actually implemented the
program treatment. That’s a really important
part of the story. I think as we go to using
more rapid cycle use of administrative data, and
maybe some smaller scale studies, it’s still going
to be important to keep that context information, and the
implementation information available to help tease
out the why of the observed differences. REBECCA: Ron, did you
want to weigh in on this, or do you want to move on? RON: Let’s move on. REBECCA: Move on, okay. Today we’ve slipped into
the world today is a little different than the world 40
years ago when this train started. To what extent are these
lessons applicable today, and how do we make
that determination? RON: I think there are three
huge lessons that we’re likely to overlook as a result
that they’re so major. The first one is that policy
makers will pay attention, not that they’ll
read the studies, but you can describe this
atmosphere and really have an impact on policy making. We have lots of areas now that
are equally as controversial, well maybe not equally but
are very controversial. Especially birth control
now, which is extremely controversial, and we now
have good evidence … I wish we had more random
assignment evidence, but we know a lot
of things about long acting reversible birth control. They could really make a huge
difference in our non-marital birth rates, we’ve had
tremendous success with teenagers, but we could do the
same thing with 20 somethings, and that’s where the
problem is most important. If we gave unlimited access
and counseled people correctly, we could really reduce the
non-marital birth rate. Which would have an
immediate effect on poverty, and I think a long
term effect as well. Policy makers will
listen to evidence. That’s by far the
broadest lesson. The second that we
learned also has not been applied as
wisely as it should, is that poor mothers will work. If you arrange programs to help
them work, and encourage them to work, help them find
work, and support them, that they will
work at high rates. Why we have not applied these
principals in food stamps, in housing, even
perhaps in Medicaid. I think they should be
much more broadly applied. That’s very
controversial on the Hill, but I think that’s the
second thing that’s extremely important. Now we have lots
of new questions that have been suggested here. Howard brought it
up first, but I think questions about the role
of education, and training, is really important. There is a feeling in the
whole country I think, but including the Congress,
that education programs are the best. One of the flaws
in welfare reform is that actually has reduced
the chance to use education. They’re still
arguing over this, about the extent to which
education will help. Howard said the
long term results, like the Baltimore study,
that there is evidence that an education training
component will work, but now we have huge questions
because the economy has changed so much, as Howard pointed
out, that a lot of … If someone is going
to make $40,000 a year you should have that in mind. We want more people to
make $40,000 a year. They are going to
have to have skills. How we give them those skills
is really a big question. If we apply the lessons
of welfare reform there, with random assignment
experiments, a one year program, a
6 month programs … Bob Lerman
is he here? We should have … REBECCA: He’s
here, he’s here. RON: What else
should we have Bob? I think that these lessons about
how to get people to earn more money, even when they’ve had
a lousy high school education, and especially how we can
incorporate our community colleges. We have experiments
like that going on now. The administration is very big
on that, and they funded … I think they have
over a billion dollars to fund experiments like that. The results are not in yet. Those are big
questions that would profit in the same way
welfare reform did. REBECCA: How does a policy
maker, or a practitioner, know when you pick up
a study whether this is relevant to those questions? There are studies out there
that say yeah, education works, and you can see the
treatment control difference, but if you’re sitting
out in the field, or sitting up on the Hill, are
we making smart judgments about whether this study is the
one that is supportive of my particular
policy or practice? RON: Can I say something? I think your knowledge
of the way things actually happen on the Hill,
and maybe in some states as well is really important. The way things actually
happen on the Hill … I think there are two really
important considerations. Once is Congressional
Budget Office, Congressional Research Service,
General Accounting Office … that’s not it’s
name anymore, it’s still GAO but it’s
something else, anyway, these organizations are really
important because members of Congress actually
listen sometimes, but especially to staff members. They are a filter. I often describe the people
at the Congressional Research Service as full professors
without the attitude. In other words, they really,
really know their stuff. They are often asked to testify,
but even more importantly, behind the scenes many staffers
on the Hill talk to them. They play a very important role,
so they are the ones that can say this is really
valuable information, and it’s correct,
and so forth. The Congress trusts them
because they’re a creature of the Congress, and not
part of the administration. That’s one answer. A second answer is that I think
the big research companies like MDRC, Mathematica, Abt’ have
gotten very good at making their information available,
briefing people on the Hill, having meetings. That’s another source of
policy makers to find out, again, primarily through their
staffs although occasionally policy makers will come
to something like that, but mostly their staffs. That’s another valuable
lesson that I think began, or at least in my experience,
with welfare reform. The big companies really
wanted to help policy makers, and they devised
ways, and they hired people who were good at that. Many researchers that actually
were the ones down there doing the research, and understood the
research design and statistics, could come up to the Hill and
give a nice 5 minute talk, or organize a seminar. I think those are the ways
that this information gets translated from the
researcher themselves, either directly through these
kind of events on the Hill, or through the Congressional
Research Service, Congressional Budget Office, and so forth. REBECCA: Don, did you
want to add something? DON: I’d just say, and I think
it’s a particular challenge at the state level, because
there is nobody in the Florida legislature today, and haven’t
been for any number of years, who were involved in these
discussions in the early 90s to the mid-90s. I think that there is a
generational shift as people who are as old as Ron Haskins
reach retirement or leave government. Folks coming along … I would say to the
next generation of agency administrators,
many of whom are sitting in this room,
you have a responsibility to learn this stuff and
to know this history. It’s as important to know this
history as it is to know all the practice issues you have
to know that are important to the programs
that you administer, because you don’t want to take
the time for your policy heads, your agency heads,
and your legislatures, and governor’s office people,
to have to completely relearn these lessons by repeating them. In every state and
local jurisdiction there are knowledge
intermediaries that include the policy
folks in the agencies. They include the
governor’s staff. They include legislative
staff at the state level, people like CRS, and GAO, folks
who were involved in these discussions, and it’s really
important to know what’s happened in the past so you
can move forward in terms of applying those lessons
in new ways in the future. While I think that a lot of
the organizations that Ron mentioned can be very helpful
in helping to guide that, it’s important that when
your policy makers in the state and counties look to
you to know the answers, that you’re well
grounded in this history. REBECCA: Howard. HOWARD: I would like
to emphasize and add another aspect to this,
which is a general point. It’s not just about
welfare research, but it applies to
welfare research. That is that what gave
these set of studies so much power was their
volume and the coherence in which they eventually
came together. That they really ended up,
although we didn’t envision this at the beginning,
creating an agenda. An agenda of studies
where one set of studies led to
another set of studies. What we learned
in an earlier set were applied to a later set,
created the impetus for others. We’ve seen a really huge
explosion in the use of random assignment, especially
in the last decade. I think part of what’s
critical about it is that as powerful as any
particular study is, and individual studies
can be very powerful, that what really creates the
ability to influence programs to operate at higher levels,
and influence policy makers, is an agenda of studies. A coherent body of work that
leads to subsequent work, and that’s what I think
really creates credibility. That’s what I think we all
need to really be striving for, and that practitioners and
policy makers should be looking for, and that those who fund
research should really aim to achieve. REBECCA: Judy. JUDY: I would also add that the
way the question is framed … How lessons learned are relevant
to welfare research today, I think one of the changes
is that it isn’t just about welfare now. Welfare is not the big program,
or the thing under attack, right now. The solutions lie, as was hinted
at in the discussion here, outside of what directly
goes on in welfare agencies. Relevant to you is also a
discussion about education and what works in education,
which Ron mentioned. The literature on emerging
successful interventions in community colleges, you ought
to really take a look at that. There are some really
striking results emerging from random assignment
studies in community colleges where everyone is poor in
the study and 1/3 of them are mothers. That’s the same population
just looked at through a different lens, and various
ways of supporting people who are working. I think as you look
at research you need to be thinking
about the lessons, and pulling in people
who can integrate those lessons into the
techniques and programs that you want to reach
out to and encourage to occur in your communities. REBECCA: I think what I’m
going to do is give you in the audience a heads up
that we’re going to move to questions and
answers in a little bit. Before we do that I want to
pose to each of the panelists, just to give you an
opportunity to reflect on … particularly given
this conversation, and given where
we are today, what do you think we should be
focusing on moving ahead? I’m thinking about things like
what are the questions that are paramount? What do we know and not know? People raised … I think Don and Howard in
particular raised questions about the changing landscape
of data that we have available. The imperative of speed or
turning out information, the practitioners and policy
offices can’t wait 5 years. How do you think about where
we should be headed from here? We want to capitalize
on this 40 years, and not move backwards,
keep moving forward. Let me just start … Judy, do you want
to take that first? JUDY: Well, I’d
say a few things. First, this fight is not
won about random assignment, this is a 40 year success story,
but there are still many people who think it’s too
expensive, it’s too slow, alternatives are just as good,
we know as much as we need to know, we don’t need to
do research we’ve just got to implement what succeeds. I don’t think any of
those things are true, but the marshaling of a
community of support and using it, and having answers to those
questions, remains important. We call the book Fighting
For Reliable Evidence, we don’t declare victory. In the end I don’t think
that’s where we are now. In moving ahead, as
I said, I think … I mentioned two things
and I just repeat them. I think turning
random assignment … given the fact that performance
measures are not good proxies for what you get from
random assignment studies, figuring out ways to use this
tool, as Don was suggesting, in routine practices
in your offices. Try it out. When you’re designing a
new intake form design two, or keep the present, or do it
in some offices and choose them by … it’s easy to do. You don’t need to turn
everything into a full-fledged research study to use the power
of this very simple technique. On the research staff side, we
have to keep working on the why and how questions, because
I don’t think … People are not satisfied
with the average impacts. It’s not enough. We need to learn more about
how to improve programs. There is an impatience
about that in the field and I think we have
to do a better job. RON: Here’s what I think I
learned from the welfare reform experience that I think
adds enormous application, and that is, you can actually
answer questions about whether a program produces an impact. You could actually know. If you did a good
experiment, you could conclude that if
I do this set of things this will happen under
these circumstances. It’s an amazing thing. It’s way beyond correlation,
and multiple regression, and all that. That’s the heart of everything
we’re trying to do now I think. The second thing is that I think
the future of random assignment evidence, which is
crucial I think, the methods have to change. We cannot be launching
5 year experiments, like Don was saying. We’ve got to learn
to do it better. Judy has said it exactly
right, that administrators have to have the capacity to
do things that take 6 months and they can get
an actual answer. They are going to have to
make a decision of some kind. If we can figure out
ways, especially I think administrative
data is really key here, that if we do not have to
necessarily collect new things. We can use the evidence
we’re already … I call it evidence,
but administrative data that you are already collecting,
that will be a key part of it. We need to work on that. We had a session yesterday
and talked about that. As Judy said, this is
happening in the private sector all the time. It’s extremely impressive,
and they run thousands, thousands of experiments in a
year, not one every 5 years. That’s the second thing. The third thing is I’m
tempted to say even if everybody on this panel tried the
best that they could, and rallied another
100 people behind them, they could not stop the
evidence based movement. We’re not at the beginning
anymore, we’re down the road. Evidence based … Member of Congress
talk about it. It’s gotten to
be a common thing. Somebody wants to support a
certain program they virtually have to … it’s
evidence based. Even if it’s not they
make it up and say it is. In this Administration,
as I said, and especially Office of
Management and Budget, but the agencies as well,
several that I could name have really beefed
up their staff. They are focused on evidence. They want to make more
decisions on evidence, and they are trying to impose
it on people around the country, and have billions of dollars
behind them to actually do it. I think we’re in a growth
phase now and if we can make this work we’ll be able to
improve our public policies. We’ll actually
be able, I think, to move the needle on national
problems as we have on teen pregnancy. We do the same
thing with violence. Do the same thing
with graduation rates, and reading scores,
and so forth. We could do that. REBECCA: Thanks, Don. DON: A couple of things, and I
think one of the lessons that I just don’t want to end
without mentioning is that with the fight over random
assignment that we were involved in in Florida, with the Florida
legislature, and their looking, and the information that
Rob Hollister shared that was presented to the Florida senate,
and the random assignment fight was largely in
the Florida House, these were complex
difficult issues that State representatives and state
senators spent a lot of time trying to understand and
ultimately they got it. I think one of the
lessons was that … I walked away from that
discussion going “Wow.” Ultimately it was
hard, but folks struggled with this very
complex information, and nuanced information,
and ultimately got it and made the right decisions. I think one of the things is
don’t be afraid of making those complex arguments
in that public arena, because folks are capable
of understanding it. The other thing, and to pick
up on what Ron said about administrative data, I
can remember a time … younger people in the
room won’t believe this, but there once was a time when
I had more questions about programs than I had data to
help answer those questions. Now people have
so much more data than they could ever analyze. The world has completely
flipped there. The access to
administrative data is completely different
than it once was. Now I think a real
challenge that we all have is how to bring that
massive data together and make sense of it horizontally across
programs, because families that we work with are
not just influenced by this program,
or this program, but by a whole
range of programs. Understanding those
relationships, and teasing those out, and
then doing the careful research to see what works
best for whom, there is an extra dimension to that. We have tools now that didn’t
exist back then to help sort that out. I think one of the things going
forward to build on the lessons from the past is to utilize
the new analytic capability with the methodologies
that are available. To take the next step in not
only answering what does this program do, or what
does that program do, but what do the array of
programs and services do taken together to influence the lives
of the families that we’re working with. REBECCA: Howard. HOWARD: I would just throw
my vote behind the things that my colleagues have said. Routinize,
institutionalize, make it more common, make
it more flexible, usable, use this tool in more settings. Get in the mode of
thinking about … I had a boss who was terrific
because every time a question would come up even … we
didn’t have the resources to do [inaudible]. He’d think of a neat
experiment that would answer that question if we only had
the time and the resources. That’s the thing to do, is
to get yourself in that mode of thinking. The other thing I
would just quickly say is the why question
is really important. Why did it work? What part of it worked? There are a lot of experimental
ways of getting that. They aren’t
entirely easy to do, they’re not
entirely satisfactory, in some cases there are
non-experimental ways to do it. We’ve practiced a
variety of things. In my lessons I didn’t
talk about implementation, but that was just
for lack of time. Judy, and fortunately others
piped in with some of that. I think this is an area
where actually people are really paying attention. Over the last 10 years we’ve
begun to make some progress in the area of really
understanding more than we were able to 15 or 20 years ago. We’ve been making progress. A lot of progress
was made in the 80s, so it’s not like people
haven’t worked at this question, but I feel that it’s
really a place to be optimistic because I think
it’s moving forward. REBECCA: Thank you, let’s
thank our panelists and we’ll go to questions. I’m going to … Yes, there are microphones. There’s one on each
side of the room. I’m going to ask you to please
line up behind the microphones and if you would please,
when I call on you, state your name and affiliation. Please, so that we have
opportunities for everyone who wants to raise questions
to do so and our panelists to respond, limit yourselves
to the questions, not comments, and one per speaker. Do we have any from our
outside listeners … okay, let’s start over here. Mariana: Hello. My name is Mariana Chilton,
I’m an associate professor at Drexel University School of
Public Health and I’m getting ready to start my first
randomized controlled trial on Monday. This was a very relevant
panel discussion, thank you very much. I have just one comment
that leads to my question. I have to say that I’m so
appreciative of the comments about the importance of
understanding context and of the why. I have to say that from my work
in the ghetto of Philadelphia, and also in Baltimore,
Camden, and Boston, there is so much heterogeneity
in the TANF programs where we are working, not only
by state but also by county, and also by county
assistance office. I really just wanted to give
a plug for those who are … I’m an epidemiologist
and an anthropologist. If you want to do a
randomized control trial and understand the context and
how to scale up, and understand the why, you need to
collaborate with anthropologists and other kinds of qualitative
researchers that can seek, they can give you information
about the context, and the heterogeneity. My question is being a
researcher from the outside, really someone who is
not a member of the … As you would refer to them, the
corporations, Abt, Mathematica, MDRC, the large
research organizations, or Agency for
Children and Families. How does someone like me, and
the other researchers, that understand the
context and also have excellent methods
of epidemiology, and are willing to run
randomized controlled trials, how do we get into the funding
mechanisms of the Agency for Children and Families? What is our
onboarding process so that we can join you
and join the cause? REBECCA: Gee, I’m not sure
which of us up here can … Mark, does anybody from ACF
want to take that question? Would somebody here,
I think Judy … JUDY: I guess what I’d say
is that in almost all of these studies that I’ve been
involved in, there have been collaborators from academia
who brought special expertise to the study. I think there’s lots of
opportunities to work with these firms who have reached
out to all the disciplines you mentioned in designing and
conducting those studies. That’s one way to do it. I would say I don’t think
you can do many of these large studies as a solo
practitioner in a university. I want to make one other point,
that hasn’t been made so far. The power of this research,
it’s not just the agenda that Howard mentioned, it’s
that they aren’t a group of one-off studies. One here on a topic,
scatter shots, they are replicated
in multiple locations. That’s very important because
one study that you do out there it can get someone a Ph.D., but
it’s not going to influence policy on it’s own. It’s got to get … You’ve got to have some
traction and replicate studies. Another thing of the power
of what’s going on now is the opportunity for synthesis. Once you build up a
volume of studies, you then have a whole
industry of people who can pull those results
together across studies. That’s a tremendous leverage. HOWARD: I would just echo
what Judy said throughout, but I think we are always
interested in collaborating with others. I would be glad to
exchange cards with you. REBECCA: Okay, let’s go
over here on the right. GEORGE CAVE: Hi,
I’m George Cave, I’m with Summit
Consulting here in DC, but I worked for Judy for
a number of years at MDRC, mainly though in the
youth development area. My question is about remaining
questions about Work Fair. I heard Howard say that one of
the things we’ve learned is that job search was often
combined with something else. I would add that something
else always included “or else.” Go to job club or you will
be deemed non-compliant and you will lose at least part
of your already meager welfare grant. The question is to what
extent is the job search help responsible for people
getting off the roles and getting jobs, and to what
extent is the fear that they felt when they got the notice
responsible for their leaving welfare? MDRC actually did an experiment
to address this question and the results are
still on the website. The paper is called
Do Mandates Matter. It was found that
when the people who were called in for a
mandatory orientation were sent home with
no further obligation, the impacts on welfare
receipt and earnings were just about the same as the
impacts of the full program. Do you think that that
experiment answered that question, or does
further work still remain to be done to
answer that question? To what extent is
the modest help that people get
with job planning responsible for the
savings, and to what extent is the fear that they
may have felt when they got the notice responsible? Other people have looked at
this issue in other programs and a simple way to get at this
through implementation analysis is to keep track of when
people get their notices, and when they’re scheduled
to report and finish job club, and to see whether they tend
to leave welfare en mass early, when they get the notice, or
late when they finish job club. REBECCA: Thank you. Howard do you want
to answer that? HOWARD: I would start. I don’t think in welfare per
se there is a lot of evidence. There’s the study
that you mentioned. I read the study as saying both. The services matter and
the mandate matters. Plus I think we have
other experience that suggests that the
services matter too when it came after the mandate,
when people were randomly assigned afterwards. I think it is a question
that is not necessarily directly critical,
because I think there is a lot of evidence
that the services matter. As far as I can see there’s
not a lot of appetite within TANF for voluntary job search. I think it’s much more
typically mandatory. I think the question that’s
more critical is which forms of job search actually
add more than others? There is a big research on the
unemployment insurance side which says that the
mandate is very important, at least for that population. In some cases it shows that
is what makes the difference. I think with respect to research
there’s good evidence that both matter. REBECCA: Okay, let’s
go over … thank you. DON: Becca can I have
… one thought on that. First of all, those who would
like to help tease out what some of those strategies
are, are in luck, because ACF now is working
on an evaluation that Abt, and Mathematica, and others
are involved in called the Job Search … JSA, Job Search
Activities evaluation, that there was a work shop on. There’s plenty of room to
help tease out exactly what job search strategies
work best for others. The other thing though that
was just an interesting fact that we found in the Family
Transition evaluation. As folks came into the
random assignment process MDRC gave everybody a
survey form to fill out. A BIF, a background information
form, Judy remembers the BIF. One question was
do you want to work and almost everybody said yes. Another question
was how many jobs have you applied for
in the last 30 days, and the response
was not so much. I think that one of the
things you can tease out, and one of the things
with the mandate, is then when we looked
at differences controlled to experimental, is that
people subject to the mandate were much more likely
to actually participate in those activities
that led to employment. REBECCA: Thank you. Let’s go over here. Bob: Hi, Bob Lerman. Quick point on history,
academics were behind the negative income tax
experiments and those were very experimentally based, but moving
forward it seems to me that one of the strategies is to … We’re always looking at the
individual, which is important, but we’re not looking
at things like employers. We don’t really do a whole
lot of analysis of how employers behave, how you
change that behavior. That’s one thing
going forward, more studies on
employers themselves. Second thing, which I think
we haven’t really teased out very well is the joint impact of
all these financial incentives of all the programs taken
together and the impact on individuals, both on
work and on marriage. REBECCA: Anybody want
to comment on that? Okay, let’s go over here
and get your question. WILSON: Hi, my name
is Wilson Siguda from Riverside County in
California, Social Services. My question is … First I would like to start
by saying that I’m a true believer of experiments
and random assignment, and I think it is the best way
to show causation. However, my question is I also
thing that there are other tools that we should be bringing
into research so I would like to talk about some of those
and I’ll see [inaudible] One of the examples
that came to mind was research in
smoking and cancer. Some of the studies developed
as correlation of studies, or as showing
association, because it was very difficult to
do experiments showing that someone who smokes gets
cancer because of cigarettes. That’s one instance where I
see that experiments might not be 100% applicable,
in that situation. Granted there were some
experiments in that area, but also correlational
research was very useful for seeing how
smoking impacts cancer. The second area that
came to mind was … Let’s say, for instance, you
have a funder who says I want you to examine this program
within 8 months after implementation. One of the problems is that some
programs take time to develop, to show effect, so based on the
literature you might see that the program … You’re going to see cost and
effect within 2 years, 3 years, because it takes time for
that program to actually have an effect. In that instance I would say
if you try to do a random assignment experiment you’re
shooting yourself in the foot because you’re going to show
that there’s no causation, basically what you want to
do I think would be to do a regression study possibly. Then show, okay, there
is some association but we need more time to
show cause and effect. The third area I
was thinking … I think experiments are great
but the third area where I thought they might not
be as applicable is … Let’s say there’s a
possibility that the program that you’re implementing
it’s not powerful enough to actually show
cause and effect. It’s not causing
someone to get a job, but what you might find is
the program is influential in helping someone to get a job. Another thing you
might see is that … Let’s say you do a multi-level
analysis you see that’s there’s nesting effects
within communities. For instance, someone who comes
from a community that doesn’t have many resources doesn’t
succeed in the program, but someone who comes from
another community who maybe has other resources such as
reliable transportation to get to their job, you’ll find that
they succeed in the program. That’s one area where I would
say association would be very powerful for showing the
program doesn’t cause it, but it actually is
very beneficial. Having stated that
I think experiments are the best way to
show cause and effect. What is the role of other
tools such as regression, multi-level analysis, in
showing credible evidence in your opinion. REBECCA: I want to exercise
the chair’s prerogative. You’ve asked a lot of really
important questions there, and we’re a little
bit short on time, so let me make a statement and
I’ll give the panel a chance to, if they want, add anything. I would refer you to a recent
document that was put out by the Institute of
Education Sciences, and NSF, which actually
talks about the progression of research from basic science,
to development of new ideas, to testing of those new
ideas in different settings, and talks about the roles
of these different genres of research in that process. I think that … the reason I’m … I think it would take
us a very long time to fully and responsibly
answer your question, but I think this
really does help. I think we’ve made a lot of
progress in sorting out what is the place of different
types of research. When is it useful to do
correlational research? How does that feed
into the process of understanding problems? Where does work like
experimental research fit in? How does that relate to
program maturity, and the like? I think you’ve raised
really good questions. I think it would take us a
very long time to answer. Unless the panel … DON: I’ll just make
one quick comment. Your comment about
the funder who wanted the results in 8
months reminded me of a saying that a former
colleague of mine used to say in moments like that
is “If you want it bad, you get it bad.” REBECCA: Okay, good point. I’m going to give the
last question over here. I guess it’s the left as
you’re looking forward, because we are
really out of time. You get the last one. Yuhuan: Oh nice. Hi, my name is
Yuhuan Fu, I’m an intern with the OMB,
and also a graduate student at the University of
Virginia Frank Batten School of Leadership and Public Policy. Building reliable evidence and
using that to inform policy is really relevant to me
and my peer right now. My question has to do with
the actual reliability of the evidence. Daniel Moynihan says
that “You are entitled to your own opinions,
but not your own facts.” It seems like that as the
number of research institutions and think tanks rise that
might not be necessarily true. What I mean by that is
with the same set of data, if you analyze it in
slightly different ways … I’m not talking about
fudging the numbers, I’m saying using slightly
different cut off points you can draw very
different conclusions. My question is how do you
guarantee that we’re drawing truly non-partisan conclusions
from this evidence so we can actually inform policy well? As a follow-up to
that, should the burden be on the analysts who are
writing the papers to actively be non-partisan, or
should the burden be on the reader to have
enough statistical literacy to be able to understand
the data behind it? REBECCA: I’m going to
give this one to Judy, because I think you
… the question is, how do you protect
against fudging numbers, or mediate differences in
impact estimates when you have an experiment? JUDY: I always fear that as more
and more people do experiments this gold standard will get
debased, and all that glitters is not in fact gold. I think we have been largely
protected from that in that the people … the community
of people who have done these experiments share a value of not
becoming advocates for policy, and not turning research
into advocacy research, but in fact letting the
chips fall where they fall. I do not see that there
was lots of difference in interpretation
of the evidence. I think there is lots of
difference in the goals that people hold for policy. People give primacy
to reducing poverty. People give primacy to
reducing dependence. That’s a different question. The key … We really have to make
sure that random assignment studies retain their
quality and do not become subject to the death
through a million cuts, and therefore the kind of
debate about methodology that Don Winstead so
eloquently stated. REBECCA: Anybody else want
to weigh in on this one? This is a really
important question, so I think what I’ve been
hearing from the panel, my own experiences, that
one of the best features of experiments is it is not … We don’t see this kind of
variability in the results if we have a well
done experiment. The answer is what
the answer is. The different analysts are
going to generally reproduce the same finding. Let’s thank our panelists. This has been a very
stimulating conversation. A great end to the conference.

Leave a Reply

Your email address will not be published. Required fields are marked *