The Evolution of Oxfam's Advocacy Framework: An Interview with Dr. Jagabandhu Acharya, Director of Evalution

While advocacy evaluation has only recently become a focus of the nonprofit sector in the U.S., internationally-focused nonprofit organizations have been developing methods for evaluating advocacy for years.  Oxfam America is an international relief and development agency that both funds other organizations and implements programs, and is an affiliate of the confederation Oxfam International.  It is considered a thought leader in advocacy evaluation among non-governmental organizations.   Jagabandhu Acharya is Director of Evaluation in Oxfam America’s Department of Learning, Evaluation, and Accountability, which has responsibility for ensuring that Oxfam America’s work and that of its grantees is evaluated. 

Dr. Acharya spoke with Innovation Network about recent changes in Oxfam America’s approach to advocacy. A brief summary of interview topics appears at the end of the interview.

Innovation Network:  Please tell us about your advocacy evaluation framework at Oxfam America.

Dr. Jagabandhu Acharya:  At Oxfam America, we have had a paradigm shift in how we view and design our program work and have developed a new strategy framework which recognizes that our work is often done in partnership with other organizations at multiple levels, and that our goals and programs are often long-term.  Oxfam works with its partners as a co-strategist, not only as a grant-giver.  We define a program by the impact we want to produce.  We require that strategies interlink inputs, outputs, and outcomes of the various interventions planned.  With our new framework, we now look at evaluation of advocacy as a continuous process that combines learning with accountability. 

We have begun to differentiate between our two types of advocacy.  With the first, we target institutional structures, policies and practices related to global poverty, inequality and injustice.  We also support the creation of a policy environment that may not have an immediate impact on the lives of the poor, but will provide hooks for partner organizations and others to translate into beneficial policy changes and, ultimately, positive changes in living conditions and in power relations among people.  This type of advocacy is often used in the global campaigns.  The second type of advocacy forms a part of our regional and national programs.  Here, we are looking for changes in policies and practices of governments, public institutions, and corporate business at national, regional, and/or even global levels. 

While advocacy in developing countries is mostly carried out in partnerships with local nongovernmental organizations, that in northern countries often involves direct advocacy campaigning by Oxfam.  We are co-strategists with our partners so that we are able to strategically influence the change process as we go along. 

InnoNet:  How do you look at the impact of policy work?

Acharya:  In evaluating advocacy campaigns, we look at policy outcomes, not necessarily how the policy changes eventually impact people's lives since there are many more levels to evaluate over time before the policy outcomes could get translated into the intended concrete impact on people's lives.  We may not always reach the final destination of changing the policy, but we move towards it by influencing attitudes, beliefs, and even the very nature of the debates.  Intermediate objectives such as these, therefore, are important because they have long-term strategic implications.  We are also aware that with most advocacy, as the campaign gathers tempo, some of the intermediate outcomes will be different from what was originally projected. 

And, advocacy campaigns do not always directly link with a program impact.  In Mozambique, for example, we successfully worked to get the government to include women's rights into the country's legal framework.  Our evaluation of that effort, by the way, made us realize that changing the law does not always translate into practice.  Attitudes and practices of the police, judiciary and others may also have to be changed.  Now we are working with partners to ensure that the law is implemented so that people may extract its full benefit.  As a result of our ongoing evaluations, advocacy to implement the law is another level of work that we are doing.  This work is particularly difficult in many developing countries, and necessitates a different approach to advocacy than is needed in the United States. 

InnoNet:  Is advocacy integrated into your program work?

Acharya:  In the long-run yes, but in the short run it depends.  Sometimes there are stand-alone advocacy campaigns.  At other times, advocacy can be just one part of a regional/national program, and, in that case, must be strategically linked to overall program goals.  For example, our coffee program interventions range from international campaigns to get a fair price for the coffee farmers to strengthening the farmers productivity and market position through the promotion of coffee farmers' cooperatives.  One or our recent campaigns is pitched at yet another level.  An international company is using traditional Ethiopian brand names on their coffee products there to make them appealing to the people. The Ethiopian government has filed for patent rights to those brand names and the company is opposing the government's action. We are part of an advocacy effort to get the company to withdraw their opposition to Ethiopia having the patent rights, in support of the development of native coffee production.  The advocacy portion of this program will be evaluated within the overall context of our achievements in changing the economic conditions and the power relations of the coffee farmers.

InnoNet:  Tell us more about your new evaluation framework.

Acharya:  Given the fast pace of advocacy campaigns, our primary evaluation focus is on obtaining credible, quick feedback in short intervals so that we can improve our work as we go along.  We view advocacy evaluation as a continual learning feedback loop.  We embed a researcher on the advocacy campaign team who documents the work and scans the outside environment.  The researcher placed on the team for evaluation purposes is not managed by the campaign team, but has access to all of the program information. 

The researcher documents the decision-making process, the adequacy of evidence behind team assumptions, alternative scenarios plans, information gathered from scanning the external environment, related research, and the trends and intermediate outcomes being generated. 

The researcher would critically analyze the data, and give feedback to the program team at frequent intervals.  As a result, the campaign team can improve their work, modify plans and even stop activities that may not be useful.  There is no reason to continue work that has negative effects on the campaign by allowing a mistake to continue on.  We are accountable to learn from our problems as much as from successes, and we learn best when learning can be internalized and applied in the situation. 

In the case of regional programs implemented mostly by local partners, however, the approach is slightly different.  Here we tie up with a local research institute to carry out required research in a relatively independent way, and there is a multi-stakeholder body that meets every six months to identify topics for research, help integrate the learning into the program, and ensure accountability to stakeholders.

InnoNet:  Would your advocacy evaluation framework work for small foundations or grantees?

Acharya:  Small organizations can't do all of it themselves.  But they can tie up with other organizations, as for example, we did with our support in countries affected by the tsunami of December [2005] , to assess the work of the entire coalition.  That also allowed us to build on the credibility of our partners.  With coalitions engaged in advocacy, it is more workable to evaluate outcomes with partner organizations.  And the fact is that we don't evaluate everything.  We always need to work out what needs to be researched and evaluated in a given situation, and what does not. 

InnoNet:  Have you evaluated your work in the Gulf Coast after the Katrina/Rita hurricane season?

Acharya:  We have been in the Gulf Coast for one and half years and are in the process of converting our initial investments to programs.  The right to housing is our major thrust. 

Our post-hurricane work in the Gulf Coast has a very strong advocacy component and a strong research component.  However, our evaluation scan is not only for our direct interventions or our funding support because we are working with a number of partners, some of whom are not our grantees.  We are therefore evaluating the work of the whole coalition and we are looking at the larger picture of what value our presence brings.  Regarding our value added there, the questions we are asking are about how much we influence other players, how the environment is changing, and how we modify our interventions. 

Currently we are conducting anecdotal studies, but are preparing for longer-term work gathering evidence.  We will then integrate the research findings properly and systematically and adapt our work accordingly. 

Through this learning we are making ourselves accountable.  And, we are in it for the long haul.

InnoNet:  When will your new advocacy evaluation methodology be publicly available, and who is putting it together?

Acharya:  We plan to come out with the framework publicly later this year.  We also plan to have an open forum on policy advocacy evaluation methods so that different organizations, can come, share, and learn from each other.  Our six-person team -half are professional evaluators and the rest have complementary experience-has developed and continues to develop our methodology with the help of experience, continuous research by independent research groups linked to us, and through talking with multiple stakeholders in the development sector, the public, and others. 

InnoNet Note:
Dr. Acharya’s description of Oxfam's changing framework for evaluating advocacy includes the following concepts:

• Differentiating between two types of advocacy:
  1. Campaigns for influencing policy and institutions at the global level that create an environment that others can use to create change, and
     
  2. Campaigns that are part of a geographically focused (national/regional) program geared towards a rights-based impact on people.
     
• Evaluating the entire work of a coalition working on a specific project, rather than one or more organizations within the coalition.
  
• Focus on learning - by providing continuous and timely evaluation feedback to mark trends and to modify interventions.
  
• Setting intermediate outcomes while making long term program commitments.
  
• Including independent researchers on program teams to provide evaluation feedback.