Reflexive Monitoring Toolbox

1. Action Learning Cycle

One technique a reflexive monitor can implement is an action learning cycle (van Mierlo et al. 2010). Reflection and action are structured to assist the project team achieve their ambition for change by mitigating systemic failures (Nederlof et al., 2011; van Mierlo et al. 2010).

Step 1 - Observe: The process of observation draws on multiple forms of evidence from body language, facial expressions, tone of voice, interpersonal communication, language used, content of the conversations, short interviews, conversations, structured participant reflections, and secondary data sources (Dick, 1991; Forester, 1999; Kitchin and Tate, 2000; van Mierlo et al. 2010). A moderate amount of the data collected will be based on the experience of the reflexive monitor. Van Mierlo (2013 pers. comm.) found that successful reflective monitors were typically experienced facilitators. As a consequence, they are familiar with structuring small group processes of dialogue and decision making.

Step 2 - Analyse and Evaluate: All the data collected during the previous stage undergoes thematic analysis (Flick, 2009). The depth of analysis depends on the speed at which the cycle is moving; the faster the cycle the quicker the thematic analysis. The key questions during analysis are:

  • Are these behaviours and actions consistent with the co-innovation principles (i.e., will it assist the project overcome or change any potential barriers to success within the system?)
  • What will the likely impact of the observed behaviours actions or practice be on the ambition for change if no intervention occurs?
  • What is driving the observed behaviour, practices and action?

Van Mierlo et al. (2010) and Nederlof et al., (2011) provide insights into what behaviours and system characteristics are desirable and what may hinder system change. This literature and the reflexive monitor’s previous facilitation experience provide a reference point against which to evaluate behaviours and activities within the project.

Step 3 - Reflect: Once the data has been analysed, reflection on how behaviours, practice or activities could be altered (or current practice strengthened) to enhance the change ambition or generate system change. Each option should be carefully evaluated based on the benefits and costs of its application. Who is involved in the refection will depend on the speed at which the cycle is moving; the faster the cycle is moving the less people will be involved. If the cycle is occurring rapidly, the reflexive monitor maybe the sole reflector.

Step 4 - Act: All actions and interventions will be undertaken by the most suitable person and will depend on the nature of the issue. For example, it may be the reflexive monitor in a meeting setting or the project manager in consultation with other project members. How these occur will need to be negotiated with the project team at an early stage of the project. There is a wealth of literature and practice which may inform the choice of action and the benefits and trade-offs associated with each alternative (for example: Dick, 1991; Chambers, 2002; van Mierlo et al. 2010; Chevalier and Buckles, 2011).
Finally, the impacts of actions will be observed and monitored, effectively beginning the cycle again.

Speed of the action learning cycle: It is important to note that this cycle occurs at multiple levels within the project, at different frequencies and with different participants. This concept is illustrated below. For example, this cycle may occur several times during the course of a single meeting resulting in small and rapid interventions or challenges. This type of intervention may include challenging participants to ensure stakeholder knowledge is strongly represented, or pointing out trade-offs which may have been overlooked. Short cycles are suited to dealing with less complex issues as they arise while longer cycles will deal with more systemic pervasive issues.

 

2. Data Collection

The primary focus of the data collected by the reflexive monitor is to progress the group towards achieving their change ambition. As a consequence, the data gathered by the reflexive monitor, via observation and conversations at meetings, feedback sheets, project meetings, and team reflections is synthesised quickly by the reflexive monitor and used to either direct the project team during meetings or provided to the project team to inform future decisions. In accordance with action learning approaches, the preliminary analysis is used to guide process and conversations.

The type and method of data collection needs to meet the objectives of the project as well as the innovation project monitoring and evaluation plan. In addition, it is necessary to collect data to track the project as it evolves. The nature of the data collected has been designed to provide triangulation through multiple lines of evidence to support the analysis. In addition to providing evidence to inform Reflexive Monitoring activities and project team decisions it also generates for longitudinal data sets to monitor impact over the length of the project. Finally, it allows for the evolution of the project to be recorded as it occurs rather than in retrospect.

View example of Data Collection Plan
View example of the Reflexive Monitoring Process Checklist

Data Collection Options:

 

3. Analytical methods

Data analysis can be both quantitative and qualitative and follows established social research practice.

Quantitative analysis of feedback sheets

Closed quantitative questions on the event and meeting feedback sheets are rapidly collated in excel and sent out to the project team to inform future planning.

Qualitative thematic analysis

The principal method of data analysis is qualitative thematic analysis (after Kitchin and Tate 2000; Flick, 2009; Bazeley, 2013) which is influenced and shaped by the systems innovation literature; in particular, Nederlof et al., 2011 and Wieczorek et al., 2012. Two types of thematic analysis can occur; rapid and in-depth.

The dominant form of analysis is rapid thematic analysis by the reflexive monitor. However, the longitudinal data sets are being captured and could be used in the future. In particular, longitudinal data sets will be used to:

  1. Demonstrate impacts
  2. Demonstrate changes in thinking
  3. Prepare a time line for the project which captures key decisions, and learning's
  4. Interview data is being collected yearly or as required (at crucial points) to establish more precise context and drivers for different events or behaviours.

Rapid thematic analysis for reflexive monitoring purposes

Rapid thematic analysis has been conducted by the Reflexive Monitor across multiple forms of evidence following (or sometime during) a meeting or event. The rapid thematic analysis follows the principle of all other thematic analysis in that key ideas and themes are identified and supported by the data (see Flick, 2009). Themes of interest are informed by van Mierlo et al. (2010)

The principal function of this type of analysis is to assist the reflexive monitor to perform their functions.

The speed of the analysis and the number of people involved in the analysis and refection will depend entirely on the speed of the action learning cycle. The speed is dictated by the issues in that simple issues of behaviour and practice may be challenged and modified quickly while more systemic and pervasive issues will require a longer timeframe. In some cases where action is deemed urgent the analysis may be extremely rapid and occur within minutes. Where this occurs the action is recorded and discussed with the project team at the earliest possible convenience.

 

4. Timeline Analysis

Timeline analysis workshop

Bringing together project participants to reflect jointly on the challenges, successes and lessons from the project is valuable for the project team to identify the causes of tensions, frictions or different understandings among the research project team and stakeholders. The timeline method provides an opportunity to do this. Depending on the length of the project and the number of participants it will take between 2 to 4 hours to run a timeline workshop.

A timeline analysis involves someone collecting information on project events over the life of the project and constructing a draft timeline as a starting point for discussion. The draft timeline is then shared at a participant workshop, or during interviews. It is important to get involvement from all workshop participants as people will remember different events. The aim is to gain agreement by all participants on the key events during the life of the project. Participants then identify key moments, highs and lows within the project, and moments of friction. Events which appear to be interpreted differently are important to focus on and discuss. Discussing these differences of interpretation will provide insights into the causes of conflicts that may have never been expressed.

The completed timeline can help the project manager to prioritise sources of tension among the project team and stakeholders and make choices about follow-up steps, based on what has been discussed.

Conducting a timeline

Pre-workshop or interview:

  • Work with a key project person to establish/identify:
  • The start point of the project (this is often pre-funding)
  • Key activities or events that occurred to date (or for the duration of the project)

During an interview

  • Have the timeline typed up on a one pager (using A3 paper might be easier than A4) (and/or possibly have a laptop that you can input as you go)
  • Explain to the interviewee that a base timeline has been drafted and key events identified. Ask them to:
    • Check the accuracy of the base timeline
    • Have they got any key events to add
    • Identify where they felt critical points on the timeline occurred: key moments of success, key challenges within the project, high and low points in the project, points of friction or tension.
  • As the interviewee talks they (or interviewer) adds this information into the timeline.
  • Record the discussion either as notes or audio recording the conversation.
  • Pose the reflective question to the interviewee “If you were to draft a set of recommendations for another group embarking on a similar project, what would you recommend in terms of key actions to take to ensure success?”
  • Record the discussion.
  • For each subsequent interview use the updated version.

During a workshop

  • Write the timeline up on A1 paper hung on the wall (or using a sticky wall) where the complete timeline can be viewed.
  • Make the timeline as interactive as possible with pictures that link to the key events identified included on the timeline.
  • The facilitator or key participant who drafted the timeline should talk the timeline through to the rest of the group. Ensure that this is just a presentation of events, not an analysis of these events. Do this by telling the story. E.g. “this happened… and then this…and then this…”
  • Participants make notes during the story of any key event that was missing
  • Once the timeline has been presented, work through the group with each participant suggesting one idea (written on a post-it) and place this up onto the wall/timeline.
  • Review the complete story again once this has been done.
  • Place participants into pairs to discuss the following four items and have them write their ideas on post-it notes (one idea per post-it).
    • What were the key moments?
    • What were the highs? Why?
    • What were the lows? Why?
    • When was there friction? Why?
  • Have someone recording notes through this discussion to collect the details described – or back up with audio recording.
  • Pose the reflective question to the group “If you were to draft a set of recommendations for another group embarking on a similar project, what would you recommend in terms of key actions to take to ensure success?

Capture the answers on post-it notes and put these up on the wall.


Note: This technique has been adapted from van Mierlo et al’s (2010) paper ‘Reflexive Monitoring in Action: a guide for monitoring system innovation projects.'

 

5. ORID

Using ORID’s as an evaluation tool

ORID is the acronym for: Objective, Reflective, Interpretive & Decisional

Purpose: The ORID process provides valuable qualitative information of the strengths and weaknesses of a workshop, or issue, based on the viewpoint of the participants.

Timing: ORID’s can be used at any small group situation, when you are seeking feedback and group reflection on participants’ experiences.

Approach: The ORID process was designed as a progression of questions that enable a group to reflect on their experiences of an event.

The strength of the ORID process is in its structure. The structured conversation allows observations to be teased out into what their meaning and implications are for an event or project. This allows participants to collectively make decisions on what works, what doesn’t and how things can be changed. However, the structured way of thinking does not come easily and can be awkward for all involved. Having the facilitator explain the process up front and provide a prompt reminding participants about the structure may assist in keeping people on track (Coutts, 2014).

ORID Process

  • Start with a question that is quite focused.
  • Divide a sheet of paper into four squares (figure below)
  • Starting with ‘Objective’ get the group to complete each ‘square’ in the following order:
    • Objective – spend 15% of allocated time focusing on this
    • Reflective – spend 15% of allocated time focusing on this
    • Interpretive – spend 30% of allocated time focusing on this
    • Decisional – spend 30% of allocated time focusing on this
    • The remaining 10% of allocated time should be divided evenly between discussing the question you want address and reflecting on the outcomes of the exercise.

​How to use the ORID process

Section, Purpose and Examples of questions to ask

O - Objective - Getting the facts

  • What did we do today?
  • How did we do it?
  • What do you remember from today?
  • What did you hear or see?
  • How many people were there?
  • Who was involved, what was said?

R - Reflective - Emotions, feelings, associations

  • How did you feel?
  • What was your first response?
  • What other feelings did you experience?
  • Did you like this or not?
  • Where do you remember the whole group reacting?
  • How did your apprehension change or your confidence grow?

I - Interpretive - Value, meaning, purpose, learning

  • What would you say were the main points?
  • What did this mean?
  • What were the main messages?
  • What did you learn?
  • Which of these actions should be first priority?

D - Decisional - Future steps

  • In what ways can you apply what you saw today?
  • How might the things you observed today change what you do?
  • What can you or will you use or follow up from today?
  • What would you say about this event to someone who was not there?