LGBTQ+ Action Plan for Wales: evaluability assessment (summary)
This report details research findings looking at the data landscape on LGBTQ+ people, and how this informs options for evaluating the LGBTQ+ action plan.
This file may not be fully accessible.
In this page
Research aims
The ʶwants to make Wales the most LGBTQ+ friendly nation in Europe. In 2023, the ʶpublished its LGBTQ+ Action Plan, which outlines how it intends this goal to be reached. The Action Plan falls alongside other ʶaction plans and strategies, such as the Anti-Racist Wales Action Plan, and they all feed into the National Equality Objectives. It is important to consider how these interact. The ʶhas commissioned Alma Economics to carry out an evaluability assessment to provide recommendations on how the LGBTQ+ Action Plan should be evaluated. Evaluation will help determine whether the LGBTQ+ Action Plan is successful and is achieving its intended aims of improving the lives and experiences of LGBTQ+ people in Wales.
Methodology
An assessment of the feasibility and suitability of different evaluation methodologies was carried out to determine the best approach for a future evaluation of the Action Plan. This assessment was guided by which provides guidance on evaluation, best practice and stakeholder engagement. A review of documents relating to the LGBTQ+ Action Plan and interviews with ʶofficials were used to scope the project before a non-systematic literature review was performed. Stakeholders were engaged through interviews, and a Theory of Change was drafted from these conversations. The outputs from the literature review, stakeholder engagement, Theory of Change and data assessment were used to then inform the evaluability assessment.
This began with a review of key documentation relating to the LGBTQ+ Action Plan, including progress updates, recommendations of the Independent LGBTQ+ Expert Panel, and related strategies. Scoping interviews with ʶofficials were also conducted to deepen understanding of the LGBTQ+ Action Plan and how it relates to other ʶplans and objectives. Following this, a literature review was conducted, which focused on existing evaluations of similar LGBTQ+ policies and programmes. Literature included peer-reviewed and non-peer-reviewed literature, with 52 papers identified as relevant. This was a targeted exercise with limited scope and should not be considered comprehensive. This review informed the evaluability assessment by providing an understanding of potential risks, challenges, and solutions related to particular methodologies and evaluation tools.
The ʶand external stakeholders representing LGBTQ+ organisations were engaged throughout the evaluability assessment through semi-structured one-to-one and small group interviews. In total, 36 interviews were conducted with 50 interviewees. This engagement provided an understanding of what outcomes and impact could be expected from the Action Plan, how these should be measured using existing data, and how to define and measure successful implementation. Discussions also included views on potential approaches to a future evaluation.
Findings from the interviews were also used to draft a Theory of Change which visually describes the mechanisms through which the activities and actions included in the LGBTQ+ Action Plan may generate outcomes and impact. The Theory of Change is used to support the evaluability assessment by identifying important impacts and causal mechanisms which would need to be assessed in an evaluation and should therefore be considered in the evaluability assessment. It also helps to identify key indicators to monitor the progress and impact of the LGBTQ+ Action Plan, which in turn determines what data should be collected in advance of a future evaluation for this purpose.
The outputs from the literature review, stakeholder engagement, Theory of Change and data assessment were used to inform the assessment of evaluation options to determine which approach was most feasible and suitable.
Main findings
Literature review
When evaluating complex programmes, the literature emphasises the importance of using a Theory of Change to clarify the underlying causal pathway between a policy intervention and its desired societal outcome within complex systems. The role of external influences in affecting the impact of the Action Plan should be recognised in the choice of evaluation method, as identifying causality with such outcomes is likely to be difficult. The evidence also suggests using a mixed methodologies approach to evaluation (i.e. those that use a combination of quantitative and qualitative techniques) given they will enable a flexible approach that can capture complexities and nuances of the programme, as well as any changes across time. Overall, few examples were found of evaluations of LGBTQ+ programmes and interventions, both domestically and internationally.
Engagement with stakeholders
Interviews were conducted with 38 ʶpolicy officials and 12 representatives of external organisations. Stakeholders consistently emphasised that quantitative measures should be used with caution when evaluating the Action Plan, because there is a risk such metrics would not capture the diversity and nuances of experiences within the LGBTQ+ community and intersectional groups. Stakeholders also emphasised the need to for an evaluation to understand the impact the Action Plan has on the perceptions and lived experiences of LGBTQ+ people and intersectional groups through qualitative methods, such as through interviews and focus groups.
Theory of Change
Informed by interviews with stakeholders and literature review, a Theory of Change was developed. It is divided into a diagram for each of the LGBTQ+ Action Plan’s themes and explicitly highlights key assumptions and anticipated risks, which should be considered during an evaluation. The Theory of Change also includes a column to represent the Overall Vision of the LGBTQ+ Action Plan, which is informed wholly by interviews with internal and external stakeholders. It is also recognised that the Theory of Change may need to be updated nearer the evaluation as outputs or outcomes which are not yet predictable develop and require representation in the diagram.
Assessment of data sources
There is relatively little availability of quantitative evidence that can be used to confidently measure the outcomes and impact of the LGBTQ+ Action Plan. No sources have been identified that could evidence the Action Plan’s objective ‘to make Wales the most LGBTQ+ friendly nation in Europe’. Some quantitative data does exist and is outlined in a Databank (Table 1) within the main report, and this could be used to measure important outcomes of the Action Plan. However, quantitative data should be used with caution given that, as some stakeholders pointed out, there is a risk that such metrics may oversimplify the complexities of the Action Plan and intersectional experiences. Furthermore, there does not exist any systematic data collection which captures the lived experiences and attitudes of LGBTQ+ people in Wales over time, which was consistently viewed by stakeholders as being a key feature of any future evaluation.
Assessment of evaluation options
An assessment of evaluation methodologies listed in the Magenta Book was conducted to arrive at a preferred evaluation option. This included assessing the relative merits and drawbacks of different methodologies based on the findings from the literature review, stakeholder engagement, Theory of Change development and assessment of quantitative data.
This assessment recommended against experimental or quasi-experimental techniques being used in the evaluation due to ethical concerns and limited data availability, and therefore recommended that theory-based approaches be employed. Some theory-based approaches were ruled out as evaluation options, largely due to their high demands on stakeholder engagement and inability to capture the complexities of the LGBTQ+ Action Plan for Wales.
Furthermore, an economic evaluation is not recommended due to the absence of data to quantify and monetise key outcomes for a value-for-money appraisal and the risk of oversimplifying the complexities of the LGBTQ+ Action plan and the variety of LGBTQ+ people’s experiences.
Recommended evaluation options
Separate process and impact evaluations are recommended by this assessment. The process evaluation should be conducted as soon as possible. It should be conducted before the impact evaluation to ensure that any lessons learned from the process evaluation can be incorporated into policymaking and begin to take effect before evaluating impact. Recognising that the impact evaluation will largely be driven by peoples’ perceptions and experiences, which are expected to change slowly over time, it is recommended that the impact evaluation is conducted several years after the Action Plan has been implemented.
For the process evaluation, it is recommended that a mixed-methods approach is used to understand the extent to which the actions have been completed and how this can be improved. It should also consider any further improvements to data and monitoring processes, as well as the adequacy of structures and governance processes to engage with external stakeholders and LGBTQ+ people more generally.
It is recommended that the impact evaluation consists of a mixed-methods approach with principles of realist evaluation. A realist evaluation is useful for understanding “what works, in which circumstances, and for whom”, making it useful for the impact and process evaluation, as well as capturing the outcomes on intersectional groups. In this case, applying a light-touch approach to the realist evaluation of the LGBTQ+ Action Plan will reduce reliance on stakeholders’ time and expertise, which can be significant barriers to conducting realist evaluations.
The impact evaluation should engage with ʶofficials to provide useful insight into how the Action Plan has led to changes in policymaking. It should also understand and represent the views of LGBTQ+ people and intersectional groups to provide insight into how the Action Plan may have a positive impact on their lives and lived experiences. To address existing gaps in available data collection on the lived experiences and perceptions of LGBTQ+ people and intersectional groups, it is recommended that a large-scale survey is conducted to support the evaluation. A survey can maximise an evaluation’s reach and be used to systematically record the attitudes of LGBTQ+ people, which can enhance the availability of qualitative and quantitative information to allow comparison across time and across intersectional groups. The success of the Action Plan is currently defined as “making Wales the most LGBTQ+ friendly nation in Europe”. Recognising that this objective is not measurable, it is recommended that this objective is revisited before it is used to measure success in an impact evaluation.
Recommendations to support a future evaluation
The following recommendations are made to improve the data landscape and therefore support a future evaluation of the LGBTQ+ Action Plan for Wales.
- The ʶshould oversample LGBTQ+ individuals within the National Survey for Wales to increase the quality and useability of data by sexual orientation. The ʶshould make data from the National Survey for Wales available to evaluators for more questions by sexual orientation. Evaluators should use this data cautiously, due to concerns over small sample sizes.
- The ʶshould add relevant questions to the National Survey for Wales, including a question to measure respondent’s awareness of anti-LGBTQ+ hate crimes and the options available to report hate crimes.
- ʶ public services, including the NHS, should consider more frequently collecting data on service user’s gender identities and sexual orientations in order to identify any inequalities in service delivery. Data collection should be standardised and collected consistently.
- The ʶshould contact support services and/or public services to explore the availability of data on conversion practices. If this data is unavailable, ʶofficials should see if data on conversion practices is available from Galop and Victim Support in Wales.
- ʶofficials should collate a list of the equality forums described in “Theme A” (Human Rights and Recognition) of the LGBTQ+ Action Plan for Wales. This could be used to evaluate the implementation of the following action: Strengthen LGBTQ+ representation on equality forums.
- ʶshould seek data from the General Register Office on the number of people who apply for a Gender Recognition Certificate within Wales.
- The ʶshould have a greater focus on equalities issues in Overseas Network reports to inform the process evaluation. ʶshould include a chapter dedicated to equalities issues.
- The ʶshould request and, if necessary, collate monitoring data on hate crimes by protected characteristics provided by the Home Office on ʶ police forces to inform the evaluation.
- The ʶshould request monitoring data from the Wales Hate Support Centre. The data request should include: monitoring data regarding the number of LGBTQ+ referrals, data regarding their engagement with LGBTQ+ people, and data measuring the impact of their services for LGBTQ+ individuals.
- The ʶshould request data from ʶ police forces regarding LGBTQ+ representation among PCSOs and police officers.
- The ʶshould ask for regional breakdowns by sexual orientation and gender identity compared to sex recorded at birth from the Crime Survey for England and Wales. Validating the quality of this data will also be crucial, as low sample sizes are expected.
- ʶshould provide updates on the outcome of the review into VAWDASV (violence against women, domestic abuse and sexual violence) indicators and seek to ensure that indicators will be available according to sexual orientation and gender identity.
- The ʶshould include a question of sexual orientation and gender identity in the Wales Sanctuary Service’s satisfaction survey. The data will need to be obtained and validated. This may be complex considering asylum is not a devolved policy area.
- The ʶshould seek to secure health complaints data on the number of complaints from people who identify as LGBTQ+. If this is not possible, they should seek the number of complaints which reference anti-LGBTQ+ discrimination in healthcare.
- It is unclear if the NHS Wales Staff Survey asks respondents to share their sexual orientation and gender identity. ʶofficials should investigate this, with the goal being that the survey results can be analysed according to sexual orientation, gender identity, and whether an individual’s gender aligns with the gender assigned at birth.
- ʶofficials should request monitoring data from the ʶ Gender Service – especially regarding patient-reported outcome measures (PROMs), patient-reported experience measures (PREMs), and anonymised feedback from meetings between the WGS and trans representatives – and make this data available to the evaluators.
- ʶofficials should liaise with the School Health Research Network to ensure that future Student Health and Wellbeing Surveys ask secondary school pupils questions about both sexual orientation and gender identity and make this data available to the evaluators of the LGBTQ+ Action Plan for Wales. Currently, the survey asks for the respondent’s gender and sex at birth and does not collect data on whether people identify as LGBTQ+ or any group within the community.
- The ʶshould make website analytics data available to evaluators for the “Parenting: Give it time” website pages. To improve this data’s usefulness for the evaluation, the ʶshould consider asking those visiting the webpage for feedback on their experience using these.
- The ʶshould strengthen the Local Government Candidates Survey by asking if people’s gender identity is the same as the gender that was described at birth. In addition, ʶshould seek to increase the response rate to this survey.
- The ʶshould collect data on how many National Governing Bodies of sport in Wales have trans-inclusive policies by reviewing the policies of a sample of sporting bodies. This exercise should be repeated before the evaluation begins. This will allow for a baseline to be collected against which evaluators can determine changes in trans-inclusivity in ʶ sport and the implementation of the following action: Improve the access and participation of transgender people in sport.
- The ʶshould seek monitoring data from ʶ Government-funded cultural organisations (Amgueddfa Cymru, the Arts Council of Wales, the National Library of Wales, the Royal Commission on the Ancient and Historic Monuments of Wales, and Sport Wales) to determine their role in promoting LGBTQ+ inclusion and any measurable impacts.
Contact details
Report authors: Alma Economics
Views expressed in this report are those of the researchers and not necessarily those of the ʶ Government.
For further information please contact:
Victoria Saynor
Equality, Race and Disability Evidence Units
Equality, Poverty and Children’s Evidence and Support Division
ʶ Government
Cathays Park
Cardiff
CF10 3NQ
Email: EqualityEvidenceUnit@gov.wales
Social research number: 83/2024
Digital ISBN 978-1-83625-931-2
