One technique a reflexive monitor can implement is an action learning cycle (van Mierlo et al. 2010). Reflection and action are structured to assist the project team achieve their ambition for change by mitigating systemic failures (Nederlof et al., 2011; van Mierlo et al. 2010).
Step 1 - Observe: The process of observation draws on multiple forms of evidence from body language, facial expressions, tone of voice, interpersonal communication, language used, content of the conversations, short interviews, conversations, structured participant reflections, and secondary data sources (Dick, 1991; Forester, 1999; Kitchin and Tate, 2000; van Mierlo et al. 2010). A moderate amount of the data collected will be based on the experience of the reflexive monitor. Van Mierlo (2013 pers. comm.) found that successful reflective monitors were typically experienced facilitators. As a consequence, they are familiar with structuring small group processes of dialogue and decision making.
Step 2 - Analyse and Evaluate: All the data collected during the previous stage undergoes thematic analysis (Flick, 2009). The depth of analysis depends on the speed at which the cycle is moving; the faster the cycle the quicker the thematic analysis. The key questions during analysis are:
Van Mierlo et al. (2010) and Nederlof et al., (2011) provide insights into what behaviours and system characteristics are desirable and what may hinder system change. This literature and the reflexive monitor’s previous facilitation experience provide a reference point against which to evaluate behaviours and activities within the project.
Step 3 - Reflect: Once the data has been analysed, reflection on how behaviours, practice or activities could be altered (or current practice strengthened) to enhance the change ambition or generate system change. Each option should be carefully evaluated based on the benefits and costs of its application. Who is involved in the refection will depend on the speed at which the cycle is moving; the faster the cycle is moving the less people will be involved. If the cycle is occurring rapidly, the reflexive monitor maybe the sole reflector.
Step 4 - Act: All actions and interventions will be undertaken by the most suitable person and will depend on the nature of the issue. For example, it may be the reflexive monitor in a meeting setting or the project manager in consultation with other project members. How these occur will need to be negotiated with the project team at an early stage of the project. There is a wealth of literature and practice which may inform the choice of action and the benefits and trade-offs associated with each alternative (for example: Dick, 1991; Chambers, 2002; van Mierlo et al. 2010; Chevalier and Buckles, 2011).
Finally, the impacts of actions will be observed and monitored, effectively beginning the cycle again.
Speed of the action learning cycle: It is important to note that this cycle occurs at multiple levels within the project, at different frequencies and with different participants. This concept is illustrated below. For example, this cycle may occur several times during the course of a single meeting resulting in small and rapid interventions or challenges. This type of intervention may include challenging participants to ensure stakeholder knowledge is strongly represented, or pointing out trade-offs which may have been overlooked. Short cycles are suited to dealing with less complex issues as they arise while longer cycles will deal with more systemic pervasive issues.
The primary focus of the data collected by the reflexive monitor is to progress the group towards achieving their change ambition. As a consequence, the data gathered by the reflexive monitor, via observation and conversations at meetings, feedback sheets, project meetings, and team reflections is synthesised quickly by the reflexive monitor and used to either direct the project team during meetings or provided to the project team to inform future decisions. In accordance with action learning approaches, the preliminary analysis is used to guide process and conversations.
The type and method of data collection needs to meet the objectives of the project as well as the innovation project monitoring and evaluation plan. In addition, it is necessary to collect data to track the project as it evolves. The nature of the data collected has been designed to provide triangulation through multiple lines of evidence to support the analysis. In addition to providing evidence to inform Reflexive Monitoring activities and project team decisions it also generates for longitudinal data sets to monitor impact over the length of the project. Finally, it allows for the evolution of the project to be recorded as it occurs rather than in retrospect.
View example of Data Collection Plan
View example of the Reflexive Monitoring Process Checklist
Data Collection Options:
Data analysis can be both quantitative and qualitative and follows established social research practice.
Quantitative analysis of feedback sheets
Closed quantitative questions on the event and meeting feedback sheets are rapidly collated in excel and sent out to the project team to inform future planning.
Qualitative thematic analysis
The principal method of data analysis is qualitative thematic analysis (after Kitchin and Tate 2000; Flick, 2009; Bazeley, 2013) which is influenced and shaped by the systems innovation literature; in particular, Nederlof et al., 2011 and Wieczorek et al., 2012. Two types of thematic analysis can occur; rapid and in-depth.
The dominant form of analysis is rapid thematic analysis by the reflexive monitor. However, the longitudinal data sets are being captured and could be used in the future. In particular, longitudinal data sets will be used to:
Rapid thematic analysis for reflexive monitoring purposes
Rapid thematic analysis has been conducted by the Reflexive Monitor across multiple forms of evidence following (or sometime during) a meeting or event. The rapid thematic analysis follows the principle of all other thematic analysis in that key ideas and themes are identified and supported by the data (see Flick, 2009). Themes of interest are informed by van Mierlo et al. (2010)
The principal function of this type of analysis is to assist the reflexive monitor to perform their functions.
The speed of the analysis and the number of people involved in the analysis and refection will depend entirely on the speed of the action learning cycle. The speed is dictated by the issues in that simple issues of behaviour and practice may be challenged and modified quickly while more systemic and pervasive issues will require a longer timeframe. In some cases where action is deemed urgent the analysis may be extremely rapid and occur within minutes. Where this occurs the action is recorded and discussed with the project team at the earliest possible convenience.
Timeline analysis workshop
Bringing together project participants to reflect jointly on the challenges, successes and lessons from the project is valuable for the project team to identify the causes of tensions, frictions or different understandings among the research project team and stakeholders. The timeline method provides an opportunity to do this. Depending on the length of the project and the number of participants it will take between 2 to 4 hours to run a timeline workshop.
A timeline analysis involves someone collecting information on project events over the life of the project and constructing a draft timeline as a starting point for discussion. The draft timeline is then shared at a participant workshop, or during interviews. It is important to get involvement from all workshop participants as people will remember different events. The aim is to gain agreement by all participants on the key events during the life of the project. Participants then identify key moments, highs and lows within the project, and moments of friction. Events which appear to be interpreted differently are important to focus on and discuss. Discussing these differences of interpretation will provide insights into the causes of conflicts that may have never been expressed.
The completed timeline can help the project manager to prioritise sources of tension among the project team and stakeholders and make choices about follow-up steps, based on what has been discussed.
Conducting a timeline
Pre-workshop or interview:
During an interview
During a workshop
Capture the answers on post-it notes and put these up on the wall.
Note: This technique has been adapted from van Mierlo et al’s (2010) paper ‘Reflexive Monitoring in Action: a guide for monitoring system innovation projects.'
Using ORID’s as an evaluation tool
ORID is the acronym for: Objective, Reflective, Interpretive & Decisional
Purpose: The ORID process provides valuable qualitative information of the strengths and weaknesses of a workshop, or issue, based on the viewpoint of the participants.
Timing: ORID’s can be used at any small group situation, when you are seeking feedback and group reflection on participants’ experiences.
Approach: The ORID process was designed as a progression of questions that enable a group to reflect on their experiences of an event.
The strength of the ORID process is in its structure. The structured conversation allows observations to be teased out into what their meaning and implications are for an event or project. This allows participants to collectively make decisions on what works, what doesn’t and how things can be changed. However, the structured way of thinking does not come easily and can be awkward for all involved. Having the facilitator explain the process up front and provide a prompt reminding participants about the structure may assist in keeping people on track (Coutts, 2014).
How to use the ORID process
Section, Purpose and Examples of questions to ask
O - Objective - Getting the facts
R - Reflective - Emotions, feelings, associations
I - Interpretive - Value, meaning, purpose, learning
D - Decisional - Future steps