To tally, to randomize, perchance to dream?

I’m just back from the Equal Justice Conference in Jacksonville, Florida (about which more, another time).  After 15 years of attending this conference, there was a first, from my perspective:  academicians were there to talk about measuring the impact of legal services.  And there were some understated but fascinating fireworks.

All of the panelists made an impassioned case for doing thoughtful quantitative and qualitative research.  First of all, funders are increasingly demanding proof of our impact.  But even more importantly, data about our services offers us the precious opportunity to test some of our widely held beliefs.  Many of our service delivery models are surely extremely effective.  But perhaps some are not; recent research suggests as much.  And if a particular approach is not effective, or is having little impact, don’t we want to know that?  So that we can change our systems and increase our impact? 

The panelists also agreed on the need for randomized studies.  To know what interventions make a difference, we must do a head to head comparison.  We need to compare, for example, people who get full scope representation with those that get limited scope, and then measure the different impacts.  And to do those comparisons, you need to randomize the populations served.  If you triage and select only certain cases for full scope representation, that will affect the outcomes – your comparison is not head to head.[1] We are of course concerned about the ethical issues involved, but the medical community has overcome this objection in order to save lives, and we must do the same. 

The differences surfaced when the discussion turned to what happens to the data, once collected.  Some on the panel argued in favor of academic studies conducted independently of legal services providers.  While it is critical, all would agree, for the academicians to understand the service delivery models and the complex issues that inform our programs, an independent eye – and independent ownership of the data – is vital to the credibility of the study.  Like a medical study, these panelists argued, the data must be made available so that others in the community can test and replicate it, and draw their own conclusions.  The world of studies and evaluation is premised on this openness to external critique and testing. 

Others, however, expressed great concern over this approach.  Outsiders don’t always understand the context in which we operate, or the ways in which conclusions play with government agencies, funders, and critics of legal services.  Valid criticisms of our approaches are welcome, and we want to take them to heart – but if we invite others into our world and then allow them to publish those criticisms without having any control over how they are presented, we are opening ourselves up to the kinds of attacks we faced in the Reagan administration.  Even well meaning legislators, desperately looking for a way to fund other programs we also care about, could see such studies as offering a solution – hey, we can cut legal aid programs, since they have serious problems. 

This debate is in its nascent stages, and I hope we see more open discussion.  The panel in Jacksonville was very polite, and it wasn’t always easy to hear the truly significant differences of opinion brewing.  I believe we will benefit as a community if the discussion becomes a little less polite, and a little more overt. 

I am sympathetic to the concerns about studies that are critical of our work.  We have always struggled to tell our stories in an effective and persuasive way – we are lawyers, after all – and losing control over that is terrifying.  Nonetheless, I believe that open, independent studies are the wave of the future.  Like the advent of limited scope and self-help services, they may seem to us to threaten our programs, but in the end, we will adjust to them, and come to embrace them.  But perhaps, in the process, we will become more educated and sophisticated participants in these studies.  And we will learn, and improve things for our clients – and that is, after all, the point. 

In order to read some of the studies people are talking about, check out Rebecca Sandefur’s article Access Across America, Jim Greiner’s studies on an unemployment clinic, a Massachusetts Housing Court, and a Massachusetts District Court with housing cases, the Boston Bar Foundation’s report on the  Massachusetts studies, and Jessica Steinberg’s paper on a San Mateo court project.   

---

[1] A great illustration of this point is a study of juveniles accused of crimes.  Only some of them had lawyers assigned by the Court – and those with lawyers actually had a much higher rate of incarceration than those without.  But the reason for this was that the Court was selecting the most serious cases to assign attorneys to – so it wasn’t that lawyers were causing the incarcerations, but rather that that incarcerations were causing the lawyers.

Pro Bono Management - not just a job any more

A couple of years ago, a couple of enthusiastic young associates from a law firm came to me.  “We have a great idea!” they said. “We want to help address the terrible fact that 80% of the litigants in family court are representing themselves.  We’ll go to our friends in various firms, and start a new project.  We’ll get lots of attorneys to take cases!”  They were so excited – they wanted to help families in crisis, and kids at risk.  I wanted so badly to hug them both and send them out to fulfill this passion of theirs. 

I did hug them – but then I had to sit them down to have a deflating conversation.  Did they know that most law firms refuse to allow associates to take pro bono family law matters? Had they studied family law, which is notoriously complex, and did they have a plan for training and mentoring?  Who were they thinking would do the extensive intake and triage work it would take to identify cases appropriate for volunteers, not to mention spend the time reaching out to their friends to place each case? Had they thought through what types of cases, at what stage in the proceedings, they wanted to take, and what scope of representation they were going to provide?  Could they articulate the “core competencies” volunteer attorneys could develop with these cases, in order to be able to make the business case for the project?

The good news is that, while these questions may have made these associates somewhat crestfallen, they are a sign of the strength of our community.  Pro Bono has become professionalized.  A corps of intelligent and thoughtful people has developed systems, research, experience and even official Pro Bono Standards (released by the ABA in 1996 and now being updated).  We know a lot about what works and what doesn’t, and have significant expertise in how to run an effective and efficient pro bono program.

This is a little disconcerting for those folks who think of it as something that anyone can do.  People who have an idea over pizza with friends on Tuesday, believe they can implement that idea Wednesday morning – and still have time to bill an hour or two before lunch. And it’s a delicate balance for pro bono professionals:  we absolutely want the commitment and enthusiasm those pizza-planners have; we just need to work with them to contain, shape and direct it, based on all that we know, and are learning. 

And so, yes – pro bono professionals are motivational speakers, and sales people, and advocates, and systems analysts – and we are also diplomats, and educators, and counselors.  We empower people to do what (most of them) feel they should, and do it well.  We have the best job in the world – we just need to gently let them know that it’s a job, and a profession. 

Collaboration - a port in a storm

This week the legal services community and its allies gather in Jacksonville, Florida to learn from each other and plan for the future.  What new programs are they implementing in Cleveland that we might want to replicate?  Are there new ideas about fund development in these challenging times?  How can we involve more transactional attorneys in pro bono?  What do the recent studies on “civil Gideon” efforts tell us about the provision of legal services, and what should our next steps be?

One item not listed on the literal agenda, but very present on the figurative one, is the question of collaborations between organizations that are otherwise competitors.  In this era of diminishing resources, we may feel particularly leery of joining forces with those competitors.  What if another legal services program, working side by side with us, learns enough about us to more successfully distinguish themselves from us in the latest round of community foundation grant applications?  What if the firm across the street uses our ideas for pro bono, and includes them in a proposal to corporate counsel we are also trying to woo?  How can we really have bragging rights about a project, if we only did part of it with another entity?

These are fair concerns.  The foundation probably will be cutting back on grants and looking for any basis for its cuts; corporate counsel is not going to partner with all the firms in the area.  There is less and less room in the shelter to huddle from the cold; we had better seize the warmest corner and defend it stoutly.

In fact, however, we cannot make real headway in this storm unless we work together.  Sharing infrastructure costs allows more of our dollars to go to desperately needed services.  Jointly applying for grant funding increases our credibility in the eyes of funders, who are anxious to avoid duplication.  Collaborations that increase our capacity to assist clients in turn offer us the chance to build service delivery models that can magnify our impact – staffing a court calendar, bringing a clinic to a homeless shelter – doing things we could not have managed on our own.

Which is not to say that there aren’t challenges.  Beyond the territoriality and resources issues, there are fascinating questions about organizational culture.  I’ve recently had the privilege to watch some of the finest legal services organizations in the country work together on an enormous project.  (More on that another time!)  There is no question that everyone involved is highly skilled, extremely intelligent, and devoted to expanding access to justice.  But organizational cultures differ, and we are still figuring out whether we need to do everything the same way, or whether we can each approach the litigation, the data gathering, and the staffing in our own way.  It has opened everyone’s eyes, I believe, to that amazing experience that diversity of opinion and experience always brings – the realization that there are many ways to see the world, and not just the way that we have always assumed, was really the only rational approach.   And so, building collaborations may be prudent; may broaden our impact; may expand the resource pie; may open new opportunities.  And it just might open our minds, as well.