Remote
Evaluations

1. Changes in the course of the evaluation

The biggest change in a remote evaluation is replacing the on-site mission by virtual data collection, e.g. in the form of video or telephone conferences. On the one hand, this changes the cooperation in the teamevaluators, as the shared experience of the on-site mission will no longer be possible. On the other hand, the distribution of tasks between the evaluators and the project team should also be reconsidered and adapted. Finally, the virtual implementation has an impact on the data collection phase itself, which differs from an on-site mission both in terms of timing and logistical aspects.

1.1 Cooperation in the evaluation team

In remote evaluations it is a challenge to build up a common understanding of the project within the evaluation team, i.e. between the lead evaluator and the technical evaluator. If possible, a personal or virtual meeting of the team should be organised at the beginning of the process to prepare the evaluation. During the data collection phase, regular debriefing meetings should be scheduled in addition to the interviews, in order to discuss and classify the impressions gained in the interviews. Finally, if possible, a personal or virtual meeting of the evaluation team should take place after the completion of the data collection in order to facilitate the coordination on developing key issues, in particular the ratings, as well as on the joint report.

Without an on-site mission, it is more difficult to assess the context of the project and the accuracyof the evaluation results, as personal observations and an assessment of the situation in the project country or project countries are missing. This is particularly relevant if the evaluation team has no country expertise and is not networked in the country. In order to counteract this, a local co-evaluator can be called to support the evaluation team in the cultural and administrative context, access to interview partners, and linguistic and cross-cultural understanding. If the (local) co-evaluator has little experience in project evaluations, training on interview techniques, taking minutes, evaluation criteria, or similar, may be necessary before the evaluation starts. Alternatively, cooperation with local evaluation institutions could be taken into account. 

The online procedure leads to a different weighting of the time required by the evaluation team. Thus, travel and transportation times are eliminated, but more time must be allowed for the preparation of data collection. The increased expenditure of time results from a more pointed preparation and evaluation of questionnaires, the preparation of a partner workshop for the presentation and discussion of the preliminary results, as well as from the additional time slots for theexchange within the evaluation team and with the project team. This should be accordingly taken into account in the planning of the evaluation process and in the definition of the evaluators' terms of reference.

1.2 Role of the project staff

A virtual evaluation can lead to a reduction in the workload of the project staff, as there is no travel time and the team of evaluators itself can take over some of the appointment arrangements. Nevertheless, the project team must be able to work properly and play an active role.

The project coordination is still involved in the preparation of the project evaluation. However, participation in interviews with the project partners is even less necessary than in on-site evaluations. In order to ensure the internal exchange between the team of evaluators and the project coordinator, sufficient time slots should be scheduled, as an informal exchange, like e.g. during on-site evaluations in the evening, is not possible.

The project assistant does not have to deal with the logistical tasks of travel arrangements. The assistant may, however, support the arrangement of appointments if this is not taken over by the team of evaluators itself, and take care of the technical aspects. This includes, for example, setting up video conference rooms and making test calls with the interview partners to ensure that the selected video conference solution works for all participants.

The potential role of local staff in the evaluation becomes more important, especially if the evaluation team of evaluators has no geographical anchoring and country expertise. If the necessary equipment is not available at the partners' premises, the local staff can provide technical support for interviews or facilitate virtual laboratory visits or similar. Information on country-specific questions can also be provided.

1.3 Virtual interview phase

While the interview phase of project evaluations is usually conducted on site within one or two weeks, remote evaluation allows for a longer interview period. On the one hand, this may be a necessity for evaluations in countries with time shifts, as less time is available per working day and fewer interviews can be conducted per day. A longer evaluation period provides flexibility to react on limited availability of interview partners. At the same time, a virtual evaluation mission eliminates the transfer times between interviews, which allows for a more efficient implementation and allows interviews to be arranged thematically rather than – as is the case in on-site missions – giving priority to practical criteria such as geographical proximity between institutions. In addition, stakeholders from different locations can also be involved in interviews and workshops, who might not be considered in an on-site evaluation.

On the other hand, care should be taken to ensure that the time flexibility gained does not cause the momentum of the evaluation to be lost. In order to avoid that impressions disappear, the evaluation process should also be carried out as compactly as possible from a distance.

If the interviewees join in from their home office, this can contribute to a more relaxed atmosphere and possibly lead to more honesty and openness, and there is more flexibility for arranging a meetingtime. However, it can also make contextualisation more difficult, there may be more distractions and background noises, and the conversation may suffer from an unstable internet connection. In the office, however, wearing a mask could make mutual understanding and interpretation of facial expressions even more difficult.

One advantage of an online connection to the interview partners, e.g. via the smartphone and various apps, is the possibility to get spontaneously in touch again after the interview if an interviewee or the evaluator(s) would like to discuss an additional subject. In virtual meetings, the project co-ordinator, (intermittent) short-term experts, local project staff or a project partner can be involved as "door openers" at the beginning of the conversation, if necessary, in order to establish contact in difficult conversations. 

A possible negative effect of virtual implementation is the often perceived less-binding nature of the appointments when it comes to online meetings or telephone conferences and non-personal meetings. In order to ensure the participation of important interviewees in the evaluation, a recommendation can be made by an active partner institution, e.g. the political partner, in addition to the Inception Report sent by PTB.

The virtual presentation and discussion of the preliminary evaluation results with the partners, which are usually carried out within the framework of the mission, should take place at a later date in order to be able to prepare them better. This reduces the time pressure on the evaluation team compared to the presentation of the preliminary results before the on-site mission has been completed. Due to this time lag between the interviews and a more intensive preparation, better use can be made of a partner workshop, e.g. by sending the presentation to the partners beforehand and also giving them the opportunity to better prepare for the discussion. The workshop is important for the joint interpretation of the results and can thus be used as an additional survey instrument.

In the case of remote evaluations, the cooperation with interpreters will also change. For example, online simultaneous interpretations can be implemented more easily. The involvement of an interpreter should already be announced in the Inception Report and possibly also be promoted so that interview partners are not excluded due to a lack of language skills and more information can be gained without losing face.

2. Adapting the methods of data collection

The data quality of remote evaluations is often not equivalent to data collected on-site, as virtual evaluations make the following aspects more difficult:

• In-depth discussions
• Building up confidence with the interview partners
• Assessing the equipment of the workrooms (e.g. laboratory visits)
• Observing the interview partners in their working environment
• Informal communication with the partners and/or the project team
• Assessing the general situation in the country and in the sector
• Gaining a sense of the context and accuracy of the recommendations.

This can be counteracted on the one hand by close contact between the team of evaluators and the project team, and by local staff accompanying the process. On the other hand, the methods must be adapted in order to create the necessary proximity to the interview partners and to be able to understand the project context despite virtual implementation.

Remote evaluations make it possible to use a variety of methods. The decisive factor is that they are technically easy to implement. For example, a kick-off workshop or a video message from the team of evaluators can be used to introduce the evaluators and the objectives of the evaluation, and toestablish an initial contact with the interview partners.

2.1 Documents and questionnaires

If no data collection can be carried out on site, documents of good quality such as the Capacity Development strategy, organisational analyses or similar, which have been prepared during the project, become even more important. In addition, supplementary information sources, such as photos from previous visits, can provide insight into the situation on site.

Furthermore, an increased use of questionnaires in remote evaluations offers the advantage that a large amount of information can be obtained on the basis of which interviews can be conducted in a more targeted manner. If necessary, the questionnaires can be tailored to different partner groups. In this way, the number of interviews required can be reduced, their sequence defined thematically, and authentic written reactions from the partners can be used in the report and/or at the partner workshop.

2.2 Interviews

In virtually conducted interviews, the gain in information may be limited, since communication is more oriented towards the factual level, i.e. figures, data, facts. The possibilities of including the emotional dimension in how the respective interview partners interpret and experience the numbers, data andfacts are significantly more difficult. All in all, the reality experienced by the interview partners is less well represented and less taken into account in the evaluation. 

If no video can be used during the interview, aspects of non-verbal communication such as facial expression, posture, gestures or how people react during lulls in conversation are not taken into account. Therefore, if possible, a video interview should be preferred, and the use of the video should be pointed out already when inviting to the interview. However, this depends on the working methods and personal preferences of the interview partners – there are evaluators who are more receptive when they concentrate on the sound alone, as well as interviewees who are more open and concentrated on the phone than when they get distracted by their video.

In order to establish closeness to the interview partners in virtual interviews, it can be helpful to create space for informal exchange at the beginning of the conversation. This can be an introductory question, such as "From where are you connecting to the interview?” The project coordinator, thelocal project representative or a representative of a partner organisation may be present at the beginning of the interview in order to give more space to the social level and to facilitate a rapprochement between the evaluation team and the interview partner(s).

In order to make the interview as efficient as possible and to achieve the best possible results, the interview style should be stringent and based on clear, open and systemic questions. It is advisable to start the interview with a very open and creative question, which gives the interviewee the opportunity to place his/her own topics. During the interviews, technically straightforward narrative methods such as storytelling, or online-based tools such as polls for scaling questions can be used. 

This is also relevant for focus group interviews where it is important to visualise questions and discussions if there are several participants. Depending on the online conference app, whiteboards or similar digital tools can be used for this purpose. It can also be helpful to provide the participants with information (e.g. questions or presentations) before the meeting so that they can prepare themselves. The group size should be kept small to allow active participation of all participants. It is advisable to address participants specifically and ask them about their opinion. It should be noted that dissent is expressed even less in virtual space than in face-to-face meetings.

If a conversation is to be recorded, which is easy during virtual interviews, the explicit consent of the interviewee(s) must be obtained. A recording has the advantage that information is also available verbatim at a later time, e.g. for the exchange of information in the evaluation team. However, the analysis is time-consuming and recording the interview can lead to less openness of the interviewee(s) in the conversation. 

2.3 Observations

Observations made by the evaluation team are an important survey instrument in evaluations. Virtual implementation makes such joint observations more difficult, but at the same time opens up new possibilities. 

For example, technical observations, such as laboratory visits or observations made of the implementation of what has been learned in practice, e.g. in companies or on markets, can be replaced by video recordings or live online visits. In the case of recordings, the technical evaluatorcan inform the project partners about a specific task in advance – in written form or by telephone – and the relevant content will then be filmed by the partners on site and, if necessary, commented on or discussed with the evaluation team. Alternatively, the laboratory or site of interest can be visited "live" during a virtual meeting. A representative of the partner institution or the local project staff can guide the evaluation team through the laboratory or site, e.g. with the aid of a smartphone. A prerequisite is the necessary technical equipment and a sufficiently strong internet connection.

In addition, the current situation offers the chance for more project activities to take place virtually. This makes it possible for the team of evaluators to participate in such project activities as observersand thus to gain insights into the concrete cooperation in the project context. In this way, the missing observations in the working environment can at least be partially replaced.

3. Technical aspects of virtual implementation

The virtual implementation of evaluations brings about technical challenges and opportunities and offers all people involved the chance to learn together. The following should be considered:

For the preparation of the interviews it should be initially defined which digital tools can be used by the partners, the team of evaluators and the PTB. For this purpose, IT security and possible restrictions due to internal regulations of the involved organisations as well as access possibilities (e.g. costs, terminal equipment required, additional hardware/software or settings, data consumption) and the functions required for the interview must be considered. The decisive factor should be to enable the partners to take part in the interview. If necessary, the tools or implementation modalities should be adapted to the local situation. For example, a video connection should not be used if the interview partner has only a limited amount of data and has to pay for it himself/herself.

To avoid technical difficulties, an alternative to the preferred video/telephone conference solution should be available and communicated to the discussion partners. If possible, a test run with the interviewees should be organised to ensure that they can connect without any problems. In any case, the telephone numbers of the interview partners should be known in order to have a direct communication channel where technical problems can be discussed and solved ad hoc, if necessary. 

Digital tools can be used to help organise the interview agenda. Calendar invitations with information and a link or other dial-in data to the video/telephone conference tool used are also helpful. It is recommended to provide sufficient time between meetings to be able to extend the interview in case of technical problems.

4. Important considerations for conducting a remote evaluation

In order to conduct a remote evaluation, the support of the partners or their willingness to take part in the remote evaluation and possibly even act as mediators for other interview partners should be ensured. The relationship with the project partners and the sensitivity of the project context are decisive here. The issue of language and the possible involvement of an interpreter should also be considered (see 1.3). 

In addition, the technical conditions on site must allow a virtual implementation. This means that participation in video conferences or at least telephone calls must be possible. Ideally, it should also be possible to use additional survey methods, such as filming laboratories.

Remote evaluations are often cheaper, more resource-efficient and more environmentally friendly than on-site evaluations, as travel distances and travel time are eliminated. In order to take the on-site situation sufficiently into account, it should be considered to add a local co-evaluator or cooperate with local evaluation institutions. This is particularly relevant if there is no project staff on site.

The users of the evaluation should be aware of the fact that certain restrictions are unavoidable in a virtual data collection and therefore the expectations of the results must be accordingly adjusted tothe results. The higher the limitations of an evaluation, the more the evaluation results should be seen as an enrichment for reflections and as an "offer" (rather than an absolute truth). At the same time, the importance of a dialogue between the team of evaluators and users increases, also in order tojointly interpret some of the results. Sufficient space should be provided for this purpose.

Furthermore, the consequences of the Covid-19 pandemic must be taken into account in evaluations. The urgent situation in some partner countries can lead to project partners having other priorities than to participate in project activities and project evaluations. On the other hand, the limited activity in the country may increase the time availability of the interviewees. Furthermore, it is possible that the time elapsed since analogous project activities have taken place may influence the view on them and/or that the project dynamics have been slowed down by the Covid-19 pandemic and thus influence the evaluation results. Therefore, it may make sense to use the evaluation for explicitly addressing the adaptation of the project to Covid-19-related circumstances and to point out improvement possibilities for a virtual project implementation. 

Text: Suzana Lange, Katharina Telfser and Stefan Wallerath

As of October 2020