Final Evaluation on Humanitarian Services at the Contact Line in East Ukraine

Final Evaluation on Humanitarian Services at the Contact Line in East Ukraine

Title of Position: Evaluation Team of GFFO Project, Ukraine

Location: Kramatorsk, Ukraine (Government Controlled Area)

Contract term: 15 November 2020 to 29 December 2020 (up to 25 working days – including preparation, fieldwork and report writing)

1. About ADRA Ukraine

ADRA Ukraine belongs to the worldwide ADRA network, comprised of more than 130 supporting and implementing country offices. ADRA’s five core sectors include Food Security; Health; Economic Growth; Education and Emergency Management. Since 1993, ADRA Ukraine has provided assistance to some 1.5 million Ukrainians, many in conflict affected areas of Luhansk and Donetsk regions.

2. About the project funded by German Federal Foreign Office (GFFO)

The current €950,000 project has been implemented from December 2019 and is scheduled to be completed by November 2020 with four components: Psychosocial Support, Social Transportation, WASH, and Legal Aid. The multi-sectoral intervention covers the following activities: both individual and group psychosocial support; installation of water purification systems in settlements and education facilities; provision of primary and secondary legal aid and improvement of access of isolated residents living along the contact-line to basic services (provision of transport services).

The geographical coverage includes locations in close proximity to the contact-line in both Donetsk and Luhansk Oblasts.[1]

The Overall Goal of the project is to: ‘Contribute to emergency relief for people affected by war and ensure safe, non-discriminatory access to essential and quality relief services.’

The Project Goal is that: ‘People affected by the conflict receive access to drinking water, legal advice and a strengthening of their mental health.’

Following project components are part of the evaluation.

1. PSS Component: Critical gaps in the provision of essential and life-saving health services to the conflict-affected population have been closed.
2. Referral Subcomponent: Non-discriminatory access to basic services and legal assistance provided to people in hard-to-reach areas along the contact line.**
3. WASH Component: Access to safe drinking water in a few settlements and educational facilities had been improved.
4. Legal aid component: Access to justice and awareness about rights had been improved

3. Objectives

The purpose of the Final Evaluation is to assess the impact, effectiveness, efficiency and accountability[2] of the GFFO project as well as develop recommendations and lessons learnt for future program design and project implementation. The evaluation should also take into account the influences and impact of COVID-19 on project activities: changing of PSS activities format – from “live” to online sessions, adjustment of social transportation activities due to the restrictions of freedom of movement, distant provision of legal aid. The project evaluation will be externally presented to relevant clusters and humanitarian actors in Ukraine including beneficiaries and local government partners) as well as to the donor German Federal Foreign Office.

The aims of this evaluation are to:

• Examine the relevance and effectiveness of the project in relation to the project goals and objectives;

· Identify good practices;

· Identify gaps and areas of unmet needs both in activities and from a cross-cutting perspective;

· Analyse the effectiveness of changes in the implementation of the project due to COVID-19;

· Examine the appropriateness of project activities for vulnerable groups; and

· Produce “lessons learnt” for review of implementation approach and for planning future projects.

Specifically, the final evaluation must focus on the following:

Psychosocial Support:

· To assess relevance of PSS response from the point of view of children/families, as well as how the response complimented the HRP action plan.

· To assess the appropriateness of the approach as implemented by ADRA and to determine beneficiary and stakeholder perceptions of the overall response, especially children, parents, and communities.**

· To assess the effectiveness of the implementation process of the ADRA’s psychosocial support response in Ukraine and to see if the component achieved planned outcomes and outputs with regard to psychosocial well-being; to assess intended and unintended changes on the psychosocial well-being, to determine how well the response addressed the priority problems of Ukraine (especially elder population).

· To assess the NEXUS opportunities of the results of the response in the absence of ongoing ADRA support, by identifying the degree to which the PSS response has built on existing local capacities and coping mechanisms, and a potential exit strategy that builds on local resources and capacities. How many volunteers/ community members have been trained, mobilized, and sustainably organized? What has been the impact of the component on local networks and community-based groups?

· To assess the quality of ADRA psychosocial response, in relation to the Interagency Steering Committee (IASC) Guidelines on Mental Health and Psychosocial Support (MHPSS) and the CPWG Minimum Standards for Child Protection in Humanitarian Response, based on the Inter-agency Guide to the Evaluation of the Psychosocial Programming in Humanitarian Crises and other quality benchmarks including gender mainstreaming principles.

· To assess in line with CHS 8, the psychosocial supervision of humanitarian workers and their conditions of mental health.

Social Transport:

· To assess relevance of the social transport from the point of view of elder people, people with disabilities and chronical diseases and families.

· To assess relevance to which extent did the project contribute to the implementation of the Humanitarian Response Plan in 2020?

· To assess the appropriateness and social acceptability of the approach as implemented by ADRA and to determine beneficiary and stakeholder perceptions of the overall response, especially children, parents and People living with disabilities pre and post-COVID-19 phases of implementation.

· To assess the effectiveness of the implementation process of the ADRA’s social transportation support response in Ukraine and to see if the project achieved planned outcomes and outputs. How did the social transport increase the accessibility of basic services (administrative, bank services, buying food, medicines and NFI etc.)?

WASH:

· To assess the relevance of WASH activities from a needs-based perspective, and as articulated in the HRP.

· To assess the appropriateness of the approach as implemented by ADRA and to determine beneficiary and stakeholder perceptions of the overall response.

· To assess the effectiveness of the activities in improving access to water.

Legal Aid:

· To assess the relevance of legal aid from the point of view of conflict-affected population

· To assess the appropriateness and social acceptability of the approach as implemented by ADRA and to determine beneficiary and other stakeholders who also provide legal aid, their perceptions of the overall response.

· To assess availability of legal aid, especially from the point of view of residents of remote locations close to the LoC.

· To assess the effectiveness: did beneficiaries resolve their legal problems, did they become more aware of their rights, etc.

· To assess what was effective and what could be improved/strengthened.

The final evaluation should also reflect relevant Core Humanitarians Standards, focusing on (but not limited to) CHS 1, 2, and 3 in evaluating the project’s impact:

CHS 1 Quality Criterion: Humanitarian response is appropriate and relevant.

CHS 2 Quality Criterion: Humanitarian response if effective and timely.

CHS 3 Quality Criterion: Humanitarian response strengthens local capacities and avoids negative effects.

Finally, the evaluation should also reflect:

· Did the project plan and implement an adequate transition and exit strategy that ensures longer-term positive effects and reduces risk of dependency?

· How well are the project’s outputs linked to more long-term focused objectives?

· Have been there any social innovative approach used? Who was the user group and what was the added value of the innovation for the target group? How can this innovative approach be used for the wider humanitarian community?

· Which are the factors for connecting the services provided by ADRA with other institutional actors at the contact line?

· To which extent did the planning and implementation of the interventions take longer-term and interconnected problems into account?

The questions listed above are to be considered as guiding questions only and the evaluation team is not limited to them. The refining and further elaboration of the questions should be done by the evaluation team, which will propose a matrix of detailed evaluation questions. The final evaluation questions will be discussed and agreed upon through consultation with ADRA Ukraine and ADRA Germany.

4. Approach and Methodology of the Evaluation

The evaluation should include the following:

• Development of approach of the evaluation in consultation with ADRA Ukraine/ ADRA Germany.
• Desk review of relevant documents: national policies, relevant ADRA documents including past evaluations, project proposal, etc.
• Data collection tool design (Questionnaires, focus group questions and informed consent forms) which must be reviewed and approved prior to use by ADRA Ukraine/ ADRA Germany.
• Primary data gathering through key informant interviews and focus group discussions among key external stakeholders including beneficiaries as well as government and CSO partners.
• Participatory capacity-development building approach involving ADRA Ukraine staff as part of the evaluation team.

A suggested work plan could consider:

• Desk Review (1-2 day).
• Preparation of evaluation tools (FGDs, interview questions, questionnaires, etc.) (2-3 days).
• Participatory discussion and review with ADRA Ukraine on evaluation approach, methodology as well as the developed evaluation tools (2-3 days).

· Review data and visit ADRA Ukraine in the field to interview relevant stakeholders in cooperation with ADRA Ukraine Program Director, Project Manager, Monitoring & Evaluation team. (4-5 days)

• Presentation and verification of key findings among key (2 days).
• Drafting of final report and concise recommendations to guide ongoing project activities and identify how government, non-government and humanitarian actors could improve their accountability in compliance with the Core Humanitarian Standards. The recommendations will be verified with the relevant stakeholders to ensure their validity (7-9 days).
• Presentation of key findings to ADRA Ukraine project staff, ADRA Germany, cluster representatives and delegation of the German Federal Foreign Office (1 day).

5. Deliverables

Inception report

The inception report should set out the planned approach to meeting the evaluation aims and objectives, methodologies to be used and questions to be answered through the reviews and planned interviews. It should provide a description on how data will be collected and drafts of suggested data collection tools such as questionnaires and interview guidelines. The proposal outlining methodology and work plan should be used as basis for the inception report. The inception report should be submitted after the kickoff workshop.

Draft and Final evaluation reports (max. 30 pages excluding annexes)

The final evaluation report should include (but is not limited to) the following:

· Executive summary (maximum 5 pages in length) of the main findings including the major indicators of the project.

· Methodology (reflection and linking to the ToR and possible constraints leading to deviations from the ToR).

· Findings.

· Learning and Good Practices.

· Conclusion.

· Recommendations.

· Annexes (including evaluation tools).

Language

All the required reports, including the final evaluation should be written in English.

6. Evaluation Duration

A total of a maximum 25 days is envisioned for the purpose of this consultancy to undertake and produce the final evaluation. Expressions of interests should include a rough plan and outline of the proposed approach.

Specific Dates are subject to mutual agreement.

7. Experience and qualifications required

The evaluator/evaluation team should have the combined following qualifications:

• Demonstrable experience (minimum 5 years) in evaluation of humanitarian programs in conflict and displacement settings.
• Excellent understanding of local context.
• Experience in leading program/project evaluations (quantitative and qualitative) in international development and/or relief settings.
• Experience in the design and implementation of multi-methodological evaluation approaches and tools (quantitative and qualitative).
• Fluent Russian and Ukrainian language skills preferred.
• Familiar with the OECD-DAC evaluation criteria for Humanitarian Action, ALNAP, Sphere standards and related international quality (e.g. CHS) standards.
• Excellent organizational and report writing skills (demonstrated through work samples).
• Excellent verbal and written communication skills in English.
• Ability to work effectively and independently in a cross-cultural environment.
• Willingness to travel to project sites within the 0-20 km zone from the contact line.

Asset:

• Experience of working in Ukraine and/or post-Soviet region.

8. Financial Resources:

· A cost-effective financial proposal should include the consultancy rate as well as all expenses associated with field visit (accommodation, meals, transport, etc.). Transportation during the field visits (data gathering) will be provided by ADRA.

9. Human Resources:

· A Participatory/capacity building approach is desired where an identified ADRA Ukraine staff(s) member will also participate in the processes of the final evaluation including development of tools and fieldwork. ADRA places an importance towards staff growth and through such an approach, its envisioned that the arrangement would be mutually beneficial to the final evaluation team as well as to the staff member(s).

· The evaluator / evaluation team will be responsible for translations & enumeration during the field visit.

· The evaluator/evaluation team will receive the required support in terms of organizing the field trip, security issues and access to ADRA documentation.

Evaluator / Evaluation Team required to undergo a background check, and must adhere to the core humanitarian principles, and sign and adhere to ADRA Policies: such as (but not limited to) Anti-Fraud Policy, Protection Policy, as well as the ethical values of ADRA.

[1] Avdiivka,Novoselivka Persha,Krasnohorivka, Myronivskyi, Svitlodarsk, Bakhmutskyi Raion, Popasna,Marinskyi Raion, Zhovanka,Luhanske,Vidrodzhennia,Kodema,Vozdvyzhenka,Verkhniotoretske,Myrne,Hranytne,Lebedynske,Sopyne, Berdianske, Pavlopil, Zolote-3, Novotoshkivka, Hirske, Vrubivka, Nyzhnie

[2] Cf. Evaluation of Humanitarian Action Guide (ALNAP): https://www.alnap.org/system/files/content/resource/files/main/alnap-eva…