End-line Evaluation of Building Resilience Livelihoods (BRL) Project

Who we are?

Oxfam is a rights-based organization, bringing together 20 affiliates, in 67 countries globally “to achieve a just world without poverty.”

Oxfam first provided support in Afghanistan in 1961 and has been directly providing humanitarian and development assistance in the country since 1991, including during the Taliban rule. It currently works in Kabul and seven provinces: Balkh, Daikundi, Herat, Kandahar, Kunduz, Nangarhar and Takhar. Through its local partners, Oxfam provides assistance to families and communities affected by natural and man-made disasters. It works with poor communities to help them pull themselves out of poverty and improve their lives. It promotes the rights of women and youth to become agents of change and fight for their rights. To create lasting change, Oxfam and its partners also speak out on behalf of marginalized groups at the highest levels in Afghanistan and among the international community.

1. Project background information:

The Building Resilient Livelihoods (BRL) project is part of the Australia Afghanistan Community Resilience Scheme (AACRS) funded by the Australian Government (Department of Foreign Affiars and Trade, DFAT). BRL project has been implemented in two phases: a foundation phase (2015-2018) and an extension phase (2018-2021). The goal of the programme is to increase and broaden the income and livelihoods assets for households in target communities that will enable them to create a buffer to shocks to their livelihoods system, increase their well-being, and invest in adaptation strategies. The project has three objectives:

· Objective 1: To increase household income from sales of almonds and dairy products

· Objective 2: To increase reliability, volume and quality of dairy, almond and other agriculture products

· Objective 3: To increase income generation potential and adaptation options for the most poor and vulnerable households

BRL project works directly with community producers and enterprises and in partnership with key stakeholders (Government, Afghan local and international NGOs ) to improve household income from sales of almonds and dairy products, ultimately reducing poverty and shifting the poorest and most vulnerable households from farm income generation to diversified livelihoods. The project works with 40 communities in Nili and Sharistan districts of Daikundi province.

The project conducted a mid-term evaluation (MTR) in 2016, as well as an impact evaluation in 2018 of the foundation phase. Over all, the final evaluation concluded that the project succeeded in increasing and broadening the income and livelihood assets of the targeted households and there was improvement in food security compared to the start of the project. The project did comparatively better on objectives II and III, while the progress towards specific objective I was hindered by the delays in the supply and installation of the machinery, delaying enterprise start up.

2. Purpose of endline evaluation:

The purpose of the consultancy is to conduct an independent evaluation of the Building Resilient Livelihoods (BRL) project to assess performance and impact against the outcomes from foundation phase in January 2015 to date (concludes in June 2021), including:

• Assess the overall performance of BRL project to understand the potential impact, sustainability and achievements against project objectives, strengths and challenges and lessons learnt.
• Assess the progress against the intermediate outcomes of the AACRS program logic
• Capture any unintended outcomes or significant changes.
• Apply a methodology that captures behavioral changes, especially women’s economic empowerment, re-distribution of caregiving in househoulds, and women’s leadership as well as changes in key decision-makers
• Assess how the Extension Phase has added-value to the achievements in the Foundation Phase and particularly how it has strengthened both Sustainability and Resilience amongst communities.

• sing the evidence collected, analyse and comment on the project’s impact. This should be framed against the overall Program principles of AACRS:

• Resilience;

• Women’s Empowerment;

• Inclusive Decision Making; and

• Partnership

• In terms of the audience of the final evaluation report, the primary audience with whom the results of the evaluation will be shared includes the local and national authorities and donor (DFAT). The secondary audience of the evaluation will be other key government stakeholders in Afghanistan, the wider development community within Afghanistan including Oxfam’s local partners and other stakeholders as relevant to adopt similar program model in Afghanistan.

3. Tentative Evaluation Questions

The following questions are indicative only. Questions will be finalised during the inception phase of the consultation with the consultant/consulting firm and Oxfam, in collaboration with DFAT and AACRS partners.

3.1. Relevance:

· To what extent was the the project targeted at the most relevent audiences?

· To what extent was the project design and implementation participatory (i.e., were the communities involved in identifying their own needs and priorities)?**

3.2. Appropriateness:

· Were project activities as per the project design appropriate to the cultural and economic realities in the selected villages? **

· To what extent did project activities and implementation strategies contribute to project results and objectives?**

· Were project activities and implementation strategies coordinated with efforts from other actors on the ground?**

3.3. Effectiveness:

· To what extent were the results (impact, outcomes and outputs) achieved. **

· To what extent did the project results contribute to AACRS outcomes for improving resilience, partnership, inclusiveness and women’s economic empowerment, and investment in adaptation strategies?**

· To what extent has the project put in place measures to minimize the negative effects of frequent natural disasters, increasing environmental degradation, decades of conflict and contribute to security on resilience, food security and livelihoods/increase economy of vulnerable groups?

· To what extent has the project contributed to addressing insufficient production of food crops, livestock production, insufficient water sources for irrigation, job creation, low household income and low wage earnings?

· What has been the impact of the project on women and girls, women-headed households and/or women leaders?

· How has the project approach to partnership (public, private, government) and collaboration contributed to the effectiveness of the activities?

· What are the future intervention strategies and issues? **

3.4. Efficiency:

· To what extent was the project delivered on time, and in a cost-effective manner?

· Were there other alternatives that could deliver the same activities more efficiently?**

· Was implementation of activities in line with the seasonal calendars?**

· To what extent did the Project’s M&E mechanism contribute in meeting project results?**

· What was the unit cost of delivering the project per result? **

· How was research and learning integrated in to the project? **

· What are the strengths, weaknesses, opportunities and threats of the project implementation process?

· Did project activities overlap and duplicate other similar interventions? Are there more efficient ways and means of delivering more and better results (outputs and outcomes) with the available inputs?

3.5. Sustainability:

· To what extent are the project outcomes likely to be sustained after completion of all project activities? **

· What changes has the project contributed to regarding women’s economic empowerment, including changes in attitudes and behaviours amongst men and women at the household level and the community level? How sustainable are these changes?**

· What are the key factors that will require attention to improve prospects of sustainability of project outcomes?

· Has the project the potential to be up-scaled and/or replicated? Why? Which component(s) of the project would offer the best opportunity for replication or up scaling?

· How were capacities strengthened at the individual and organizational level (including contributing factors and constraints)?

4. Scheme -level indicators

Along with 3.1 up to 3.6, the below Scheme level indicators should be part of this evaluation. Data will be collected for each indicator and progress against each analysed[1].

A Note on the Evaluation Questions: Oxfam is specifically interested in being able to link project activities to outcomes. The successful consultant should be able to review not only what occurred, but also how and why these results were achieved. The successful proposal for this consultancy will include a description of how the consultant will achieve this goal using the suggested evaluation questions as a basis for a more refined approach. **

5. Methodology:

The evaluation will involve desk review, discussions with key informants (district & provincial government departmental heads, partner organization, community leaders and key project staff), we are expecting to cover 10 % of direct beneficiaries of BRL project throughout this assignment which will be covered by (Individual Household Interviews, Focus Group Discussions (FGD), Key Informant Interviews) and observation. Overall, this is a summative evaluation involving qualitative and quantitative methods to evaluate BRL project implementation and performance.

The evaluation methodology should also include methods that show Oxfam contribution to shifts in decision-maker commitments, practice or policy, changes in attitudes, norms, relationships or capacity.

The evaluation will take place in 3 phases:

5.1. Inception phase and desk review:

The consultant team will have a discussion with Oxfam as implementing agency to agree on the evaluation deliverables, donor requirements, and work plan, documents for review, key informants, FGD participants and detailed evaluation methodologies, include inception report and evaluation questions. The consultant/consulting firm will also liaise with DFAT during the development of the high-level outcome and impact level indicators to ensure an agreed level of consistency between this evaluation and the evaluations undertaken by other AACRS NGOs.

5.2. Field phase:

The consultant will agree with Oxfam on the logistical requirements for the evaluation based on the detailed evaluation work-plan from the inception report. The field phase will involve recruitment & training of enumerators, pre-testing of data collection tools, finalizing tools, discussions with key informants, household individual interviews, focus group discussion, case studies, and observation.

Table 1: Indicative distribution of qualitative data collection

Type of data collection

Target Population

Location and Quantity

FGD

Women and men, female and male youths (direct and indirect beneficiaries)

· 2 FGD with men (ages 25 and older) who are direct project beneficiaries

· 1 FGD with men (ages 25 and older) who are indirect beneficiaries

· 2 FGD with male youth (ages 16-24) who are direct project beneficiaries

· 1 FGD with male youths (ages 16-24) who are indirect beneficiaries

· 2 FGD with women (ages 25 and older) who are direct project beneficiaries

· 1 FGD with women (ages 25 and older) who are indirect beneficiaries

· 2 FGD with female youth (ages 16-24) who are direct project beneficiaries

· 1 FGD with female youths (ages 16-24) who are indirect beneficiaries

KII

District and Provincial Government Departments Heads, Partner Organization, Community Leaders (Jirga, Shura Leaders) and Key Project Staff.

At least 1 key informant interviewed from each category. At least 50% of the KII respondents should be women.

Individual HH Interviews

Women and men (direct beneficiary)

5% of the total direct beneficiaries

Case Studies

Women and men (direct beneficiary)

At least 2 case studies (1 per district)

Personal/result stories

Women and men (direct beneficiary)

4 change stories gathered

5.3. Analysis and Reporting phases:

Preliminary analysis will be done immediately after data collection and an initial debrief including data validation sessions will be conducted by the consultant in Kabul for Oxfam staff. The consultant should undertake quality assurance of all drafts submitted to Oxfam. Evaluation report drafts will be checked by Oxfam quality assessment team, and comments provided to the consultant within ten days of receipt of report.

Below is the schedule:

Activity

Deliverable **

Time allocated **

Inception phase (evaluation design, methodology, detailed work plan, document review, stakeholder consultation and inception meeting)

Inception report

10 days

Field phase (field visit, enumerator recruitment and orientation, data collection, data analysis, debriefing, presentation and validation workshop)

Draft evaluation report

20 days

Finalization of Evaluation report incorporating additions and comments provided by all stakeholders and submission to Oxfam.

Final Report

5 days

6. Key Deliverables:

· Endline evaluation methodology and inception report

· Tools for data collection in consultation with Oxfam

· Raw and Analysed Data

· Final report including an executive summary and adequate recommendations to guide the extension/next phase programming (no more than 35 pages, excluding annexes).

· Visually engaging short report with key evaluation highlights (no more than 6 pages) to be shared externally

· Develop 4 change stories.

· Workshop/session to discuss and validate research findings

Format of the Final Evaluation Report:

The fonts should be Times New Roman size 14 Bold for headings and size 12 normal for regular contents.

1. Cover page clearly identifying the report as a project final evaluation and stating:

· Project Final Evaluation Title

· Project Title

· Geographical Coverage

· Date that the project final evaluation report was finalised

· evaluator(s) name(s) and logo (if available)

· Oxfam Logo

· Implementing Partners Logo

· Appropriate recognition of institutional donor support

· Clear statement in case this report can NOT be used externally

1. Table of contents

2. Glossary

3. List of abbreviations.

4. Executive summary that can be used as a stand-alone document (1 page)

5. Introduction, stating objectives, intervention and context of the project final evaluation and evaluation questions (1-2 pages)

6. Methodology, including an indication of any perceived limitations of the project final evaluation (2-3 pages)

7. Presentation of the findings and their analysis (3-4 pages)

8. Conclusions (1 page)

9. Learning and Recommendations (1-2 pages)

10. Appendices:

· Terms of reference

· Project final evaluation (main features of data and activities carried out).

· A list of interviewees (name, function and working environment) and places visited.

· List of documents and bibliography used.

· Details on composition of evaluation team (names, nationality, expertise, working environment).

· Link to Methodological appendices:

ü The project final evaluation proposal

ü Project Final Evaluation instruments such as questionnaires and interview guides

ü Data collected

7. Key Responsibilities:

· Developing the endline methodology, plan and research tools in consultation with Oxfam programme staff, including suggested number of households to be surveyed, key informant interviews, focus group discussions etc.

· To conduct desk review of project documents.

· To thoroughly review the baseline and MTR reports and make sure to fill the gaps, that have not been measured in those reviews.

· To develop endline survey and interview questionnaires and tools for different stakeholders that match to the baseline and mid-term evaluation to measure the changes.

· To hire qualified enumerators and train them on conducting household interviews, focus group discussions and key informant interviews.

· Oversight of data collection, including feedback meetings with field and Oxfam senior management staff.

· Data entry, translation and analysis.

· To prepare the end line report.

· Conduct de-briefing and validation workshop on the findings out of the evaluation.

8. Timeframe:

The research should commence by 1 January 2021 and the final report must be submitted by the 30 April 2021. Due to inaccessibility of Daikundi province during winter season (Dec-Jan), inception phase and desk review can be undertaken in January and field work to be completed by early March 2021.

9. Payment schedule:

Payment will be made in three instalments: 15% on contract signing; 35% after the Data collection; remaining 50% after submission of approved final report.

1. Logistics:

The consultant is responsible for all the logistical issues including transportation, accommodation, meal etc.

11. Required skills and experience:

· At least 8 years of experience in managing food security and livelihoods (agriculture, livestock, safety nets, resilience) projects with INGOs and government departments.

· At least 5 years demonstrated experience in managing Monitoring and evaluation process for large projects / programs.

· Extensive expertise, knowledge, and experience in the field of evaluation of development programmes, particularly in midterm/impact evaluations in developing and/or conflict –affected context.

· knowledge/ experience of MEL processes for women’s economic empowerment programs

· Knowledge of the Afghan government system, policies and procedures.

· Skill in training to staff on monitoring and evaluation.

· Quality and excellent written and verbal communication skills in English.

· Afghan language skills preferred for fieldwork.