Tag: outcomes measurement

New Offline App – coming soon

Posted on by Emily Lamont

When you’re out and about in the community it’s not always easy or possible to connect to the internet. This can make it challenging for those using the Live Completion feature to complete a Star on-screen using the Star Online. We know from feedback that Live Completion is a great way to help practitioners avoid data entry duplication, cut down on paperwork and reduce printing.

That’s why we’re excited to let you know that we’re working on a brand-new Offline App – which allows practitioners to complete Stars collaboratively on screen with the people they support, without needing an internet connection. The new app will be even more user-friendly and accessible, supporting practitioners to have more helpful and collaborative conversations with the people they support.

We’ve been developing the Offline App with Usertopia and QES, drawing on their expertise in research driven user experience and software development. We plan to launch the Offline App later this year.

What will the Offline App look like?

This new functionality will give organisations the option of ‘Enabling Offline Mode’. Once enabled, practitioners linked to the account will be able to download the Offline App to their device and save engagements for use offline.

A screenshot of the Collaborative Session homepage on the Offline App. It shows the Star version, Stars completed, Open actions and options for Star completion actions like complete a new Star.
Collaborative session homepage

Using the Offline App, practitioners will be able to start a ‘Collaborative Completion’ session where they can complete the Star, create Action Plans and view and compare previous Stars with the people that they support.

View and Compare Stars page
A screenshot of the Action Plan screen on the Offline App. It shows an Action Plan including the in progress actions and closed actions.
Action Plan screen

The new screens will also be used to replace the current ‘Live Completion’ function, allowing the Star to be completed using the Star Online in a more accessible and user-friendly way.

A screenshot of the Star completion screen from the Offline App. It shows a completed Star and a prompt showing a section from the Journey of Change so the reading can be finalised and discussed.
Star completion screen

A few thoughts from our research and designer partner, Usertopia:

We would like to thank everyone who took part in our research sessions and workshops over the past 12 months. It has been a delightful experience acquainting ourselves with Triangle’s expert team and engaging with a number of practitioners to learn how they are currently using the Live Completion feature of the Outcomes Star to help change lives.

We’ve woven the invaluable insights garnered from our research into the fabric of every design decision we’ve made. This has included getting under the skin of how practitioners engage with the people they support, understanding current limitations and difficulties with the current Outcomes Star, and outlining sensitive use cases with various ethical considerations leading to refinements such as one-tap screen locking.

Our latest updates enhance collaboration, interactivity and engagement, aiming to guide best practice, make the Star easier to use, and ensure safer, more productive sessions for service users. We truly believe these updates will unlock new potential for the Outcomes Star and lay a foundation for continuous improvement long into the future.

– Usertopia

We are really excited about this development and plan to launch it later this year.

First Look: Register to Try the New Offline App!​

If you already use Star Online and would like to get a first look and the chance to feed back on the Offline App before launch, please register your interest below.

Register your interest

If you have any requests, or any questions about the Offline App or the Star Online, please contact us on info@triangleconsulting.co.uk.

5 Tips for using Star Data in your Reports and Evaluations

Posted on by Juliet Kemp

April is here and you may be thinking about putting together project evaluations or preparing your Trustee Annual Report and considering how you demonstrate the impact of your services.

Now is an ideal time to think about using your Outcomes Stars data. Even if you haven’t been using Outcomes Stars for long and don’t have lots of review Stars, read on… you can still use Star data to evidence the need for your service.

In this article we have gathered some tips and ideas to help you use Star data in your annual reports.

Use Star Charts to bring case studies alive

You probably have lots of stories of amazing personal change achieved by people you support, and already include individual case studies in your annual reporting. You can illustrate this by adding a Star Chart showing someone’s first and last/latest Star readings.

This is a great way to show the changes that have been made. Along with the narrative in your case study, a Star Chart will help you tell the story of how the support provided facilitated progress on the Journey of Change in relevant outcome areas.

TOP TIP

If you don’t use Star Online why not use a photo of your hand-plotted Star Chart image?​

Paula's Star
A Star Chart showing someone’s first and last/latest Star readings

Taking a snapshot view

Snapshot reports look at Outcomes Stars completed across a cohort of people at a particular point in their support. A great way of using a Snapshot report is to look at everyones’ first Stars to see how the readings are distributed across the Journey of Change in each outcome area (by working out the percentage of people with readings at each Stage, in each outcome area). This will give you a profile of the needs people have when they first start support.

TOP TIP

Use this information to illustrate why you deliver the kind of interventions that you do – this can show that your service is evidence led and is focusing resources where they are needed most.

A snapshot report

Example of how an organisation may use a Snapshot (SS) report:

We use the Wellbeing Star™ to help us identify people’s needs when they start the three-month social prescribing programme. Over 60% of our programme participants are either not yet thinking about or finding out about how they could improve their lifestyle, so our programme focuses on one to one and small group work to help people open up about their lifestyle and consider the impact this might have on their existing health conditions.

Illustrating the distance travelled

Outcomes Stars are designed to show the progress made by the people you support, and it is really satisfying to be able to collate Star readings to show this progress across a service or organisation. If enough people have completed two or more Stars, readings can be collated for a service or organisation to capture distance travelled and demonstrate the percentage of people that have moved forward, maintained or dropped back over the course of a year.

How much progress are people making in each outcome area?
A distance travelled (DT) report

In your report narrative, you might want discuss progress made across all outcome areas around the Star or you might prefer to focus your narrative on a few areas that your service targets through your interventions, by selecting a few headline figures from the data in the report.

Example of adding narrative to a Distance Travelled (DT) report:

As evidenced by our use of the Well-being Star™, which we complete with participants on our social prescribing programme, 80% of participants made positive change in the area Money, following our successful collaboration with our local CAB debt and benefits service – this is a big improvement from the previous year, where Money was the area in which we saw significant need (people were at the lowest two stages of the Journey of Change scale) and we saw the least positive change.

TOP TIP

Explain what your distance-travelled data is showing you, the learnings that have been taken from this and include plans for service delivery going forward based on the findings.

Don’t be afraid to report on people dropping back, maintaining or acknowledging change that isn’t as strong as you would like. Some areas do take longer to make positive change or depend on resources that aren’t available or are out of the control of your service. Reporting on this will show to stakeholders that you understand the data you are getting from Outcomes Stars and that you use it to reflect on your provision and the wider context you work in.

For example:

Although we continue to see less positive change in the outcome area Where you live on the Wellbeing Star™, this is an area where it is often difficult to achieve results within the 3-month programme e.g. referrals to the Council’s aids and adaptation team for an assessment typically take about 10 weeks.

TOP TIP

When using Outcomes Stars to evidence your impact, use the correct title of the Star you use and use the ™ to show you are using a recognised evidence-based Outcomes measurement tool.

Further support

Triangle is here to support you to make the most from your Star data. If you need any support or have any questions, get in touch with your main point of contact at Triangle.

We run regular webinars to help you get the most from your Star data and use the Star Online report dashboards. You can book a slot and find out more information about our webinars below.

Book a Webinar

Share your practice with Triangle

We would love to see examples of where you have used your Outcomes Stars data to evidence your impact or where you have learnt something interesting – please get in touch to share your experience of using Star data.

Why relational services matter

Posted on by Donna

Joy MacKeith shares her thoughts after the inaugural Towards Relational Public Services (TRePubS) Conference

How do you deliver public services that are relational – services that recognise and respond to the unique needs and circumstances of each individual?  That was the topic of the conference ‘Towards Relational Public Services’ that I have just had the pleasure to participate in. 

Sam Game and I gave a workshop which explored how the Outcomes Star helps to make ‘Enabling Help’ a reality in practice.  Working in a relational way is one of the six core principles of Enabling Help, Triangle’s blueprint for effective services.

Sam, who leads the implementation of the Parent and Baby Star in Health Visiting services in Warwickshire, described how using the Star has helped them work in a more relational, collaborative and strengths-based way.  Parents have responded incredibly well.  They are much clearer about the health visitor’s role and they love the way that Star helps them see what is going well as well as where they have needs.  And it isn’t just the parents.  It has also helped commissioners to understand and value the service.

Our workshop was one of more than 40 sessions which took place over two days at Newcastle Business School – part of Northumbria University. 

For me the highlight was Mark Smith’s presentation on service transformation work in Gateshead.  He described how they have implemented an approach to working with people with complex needs in which there are only two rules: ‘do no harm’ and ‘stay legal’.  In this ‘liberated method’ case workers have a low caseload and there are no limits on how long they can work with someone or what kind of support can be provided.  And case workers have budgets and the discretion to spend money to help solve pressing problems. 

What I loved about this approach was that the bureaucracy was stripped out, giving workers the chance to do simple things that could make a big difference and avert much greater problems down the line.

Mark is part of a growing movement of managers, practitioners and academics who are challenging the orthodoxies of New Public Management (sometimes described as the 3 Ms – Markets, Metrics and Management). They are making the case that outcomes emerge from many interacting factors and therefore cannot be ‘delivered’ or dictated.  In this complex environment, learning rather than prescription and control is the most effective management strategy.  And it makes more sense for commissioners to take a relational approach and to hold service providers accountable for learning rather than targets.  This new approach is called Human Learning Systems.

Toby Lowe, Visiting Professor of Public Management and the Centre for Public Impact closed the conference with a call to action – for all of us who want to see services that are more relational, responsive and effective to come together and work with a coalition of the willing to create a paradigm shift in public management.  It is an exciting vision, and one that I think the Outcomes Star, a tool with flexible, relational working at its core, can play a role in realising.

Joy MacKeith, Co-creator of Outcomes Star

****

Watch Joy’s conversation with Toby Lowe about the synergies between Enabling Help and Human Learning Systems 

Find out more about Enabling Help here 

Find out more about Human Learning Systems here

 

 

Triangle reflects on attending the Scotland that cares – Scottish Parliamentary Reception event – 7th March 2023

Posted on by Outcomes Star

I’m Jim Borland, the Implementation Lead for Triangle in Scotland and it was my privilege to attend the recent Scottish Parliamentary Reception event hosted by Oxfam, promoting a new campaign ‘A Scotland that cares’.

The campaign is focused on why making a commitment to valuing and investing in care within the National Performance Framework is so vital to drive progress towards a Scotland that cares. The campaign has been launched to coincide with the first review of the National Outcomes framework in five years. It sets out the argument that to build a fairer and more resilient country, the Scottish government must set a National Outcome for Scotland to fully value and invest in all forms of care and all those who provide it. Too many carers face deep personal and financial costs, including poverty. With the cost-of-living crisis deepening pressures on those who rely on or provide care are increasing. Only once this standard exists, can we then all work together to create positive change for society.

A New National Outcome on Care

The campaign has created a blueprint for a new National Outcome on Care, but it requires public support and the political will to change the current situation and work towards improving the situation for those who experience and provide care across Scotland.

The event was organised by Oxfam Scotland and sponsored by Karen Adam, MSP for Banffshire and Buchan Coast. There were approximately 60 attendees’ from interested agencies, including staff from some of the 55 organisations that currently support the campaign. These included Carers Scotland, Carers Trust, One Parent Families Scotland and Scottish Care, as well as a variety of MSPs from constituencies across Scotland. I am proud to say that Triangle is also one of the supporting organisations.

Support the campaign

Jamie Livingston, the head of Oxfam Scotland opened the event describing his lived experience of caring for his sister who died after battling cancer.  He highlighted the practical issues facing those requiring care and those providing it, but also the issues for the carers once their caring role ends. He talked about his sister’s determination to be active in improving the situation.  Highlighting her efforts of contacting, all the political leaders in Scotland to raise awareness of the lack of support for all those needing and providing care. He has taken his sisters positive action through to this campaign. Whilst many current Scottish political parties already support future change, there is an opportunity for everyone, especially those who experience care and those who provide it to add their voice to the call by taking action via the campaign website: https://ascotlandthatcares.org.

Karen Adam MSP, then highlighted her own lived experience of being a carer, outlining the need for financial support and appropriate resources being made available to help individuals being cared for or fulfilling the very difficult carer’s role itself. She highlighted the need for the carers voice to be heard, valued and rewarded for the work that they do.

Jamie Livingston provided background information on the campaign, which started in 2021 during the Covid-19 pandemic. At the time, it was clear those being cared for, and their carers were excluded from many aspects of the pandemic recovery plans. This was highlighted by the fact care and carers are invisible in all 11 Scottish National Performance framework outcome indicators.  

Together with several partner agencies, the drive to change started with the intention to ensure a permanence to investment in the caring role.  Together with the development and publication of policy via a new National Outcome on Care. Working in conjunction with the University of the West of Scotland (UWS) the partnership developed a blueprint for a new National Outcome. It is hoped to be the basis for potential care indicators specifically linked to carers and the caring role in Scotland. Jamie also highlighted the upcoming consultation phase of the Scottish Government’s review of National Standards and urged individuals and organisations alike to be involved in this consultation and support change to the National Outcomes.

Seven new national indicators have been identified to measure progress:

  • The quality of life of carers, care workers and those experiencing care;
  • The quality of care for all;
  • The financial wellbeing of carers, care workers and those experiencing care;
  • The voice and influence of carers, care workers and those experiencing care;
  • Access to education and training;
  • The adequacy of funding for care;
  • The job quality of social care and childcare workers

Oxfam Scotland: “We want Scotland to fully value and invest in those experiencing care and all those providing it because… Scotland’s one million unpaid carers are the bedrock of health and social care, without whom the care system would collapse. Despite saving Scotland £10.9 billion each year, too often they experience poverty, loss of employment and ill health simply because they care. This must change!”

Satwat Rahman, CEO of One Parent Families Scotland, outlined the importance of this work, and why her organisation has been involved since its inception. Introducing four panel speakers, all with similar negative lived experience of the current support available. These experiences included:

  • The view that the carers’ role is predominately valued and seen as a priority only by family members, peers and caring services. General opinion was that there was little, or no value put on this role by external services, including education, employment, and financial services.
  • There was an expectation from external agencies that carers should ‘know what to do’, without any support.
  • Very limited support available for young carers, especially in the areas of education and finance – thus the carer’s own life and dreams have to be put on hold and suffer whilst they fulfil their caring role.
  • An overly complicated benefits system, which was off-putting to a lot of carers.
  • A lack of resources for those requiring care, or respite for the carers themselves. Demonstrating the need for infrastructure improvements in care provision as one size doesn’t fit all.

Consequently, the need for recognition around the importance of the carers role was emphasised.  Increased and appropriate funding for carers and carers services is needed, to assist carers to provide support and ensure the person requiring care is treated appropriately and with dignity.

Karen Hedge, deputy CEO of Scottish Care concluded the presentations highlighting the view that carers were seen as ‘Cinderella’ within Social Care services, and much more is required to support these individuals. She further emphasised the need for proper policy and legislation which in turn would require achievements to be measured to ensure compliance. Only then would Scotland be able to demonstrate we are a nation that cares.

I found all the personal stories and experiences very moving, particularly from the four panel members sharing their ‘lived’ experience.

As an organisation, Triangle is not directly involved in supporting individuals. We develop Outcomes Star’s for organisations who provide support to individuals in a wide range of social provision settings, including carers. The Carers Star was developed in partnership with the Social Enterprise in East Lothian (SEEL), The Carers Trust and funding from the Scottish Government. It provides a robust framework to assist practitioners work together with carers to help optimise the quality of their lives and assist them in their caring role.

We often discuss the importance of enabling and empowering people by being person-centred and strengths-based and this Star will assist in identifying a carers strengths and challenges to identify support needs to improve the carers situation. However, as highlighted by most of the speakers, the current apathy towards the carer’s role and lack of resources and finance available to support this role directly impacts what is achievable by a person or supporting organisation.

The culture around current care services in Scotland requires change if Scotland is to demonstrate that it values and invests in all individuals experiencing or providing care.

The event gave me plenty to think about and I am personally committed along with my colleagues within Triangle to support this campaign and I would urge anyone reading this post to add their voice to the campaign.

Support the campaign

Please visit the campaign website: https://ascotlandthatcares.org and sign up to the campaign to create change for this vital cohort of our society. Everyone will need to be cared for at some point in their life: as a child, in later life, or due to additional support needs. If you receive care or are a carer yourself, please let our political leaders know why valuing and investing in care matters to you.

Further information

If you would like further information about the Carers Star, please email us at info@triangleconsulting.co.uk

Stay updated

You can subscribe to our newsletter to receive updates directly to your inbox, so there is no need to keep checking for new blogs.

GoLab – Social Outcomes Conference 2022

Posted on by Donna

Dr Anna Good shares her thoughts on #SOC22

Last week, Triangle directors- Graham Randles, Joy MacKeith and I attended the Government Outcomes Lab annual conference #SOC22

One presentation that particularly resonated with me was from Reji Ikeda at the Ministry of Justice in Japan. Talking about ‘value measurement for offender rehabilitation’, he emphasised the importance of the quality of the relationship and the ‘emotional labour of professionals and volunteers’ on rehabilitation outcomes. He also spoke about how poor occupational well-being can negatively impact these relationships. I couldn’t resist mentioning the Justice Star as an excellent tool for developing trusting relationships and boosting staff morale in this context.

Joy’s recent report Enabling help puts human relationships at the heart of effective service delivery, so this is at the forefront of my mind when thinking about the Star. The Enabling help report highlights the problems with borrowing from approaches such as the scientific and economic paradigms in social provision. Although not focused on the Star, the innovative suite of Stars offers a better, more enabling approach. I thought of this when the keynote presenter, Julie Battliana, stated that ‘agitation is alone is not enough to make change happen, innovation is required’.

 Another theme that I noticed running throughout the conference was the desire to use data to improve service delivery. I felt this was encouraging in a conference often focused on meeting targets and payment by results. Miika Vuorinen, a Chief of Evaluation from Finland, commented that one way to know if social outcome frameworks are working is to see if they are being integrated into the decision-making processes. Closer to home, Stan Gilmour from Thames Valley Police gave a powerful presentation, emphasising ethical decisions based on using data to allocate resources and plan social impact.

It’s possible I was looking out for these positive signs, as I feel passionate about encouraging and supporting good use of Star data. By ‘good use’ of data, I mean those closest to the creation of Star data (practitioners and managers) interpreting the data and using the learning to encourage changes needed to maximise the outcomes for those being supported.

Further reading: Going for collaboration at the GoLab Conference

Triangle co-founder and Outcomes Star co-author Joy MacKeith reflects on the theme of collaboration at this year’s Social Outcomes Conference and points to Human Learning Systems as an effective way to make this the norm in service delivery.

Sign up to our quarterly newsletter for product updates, case studies, publications and news. 

The Family Star Plus within the Supporting Families Programme (2022-25)

Posted on by Donna

All local authorities delivering Supporting Families must have an approved Outcome Plan setting out their indicators of eligibility with associated family level outcomes against headline objectives set out in the programme guidance.

Since the start of the supporting families programme (formerly called the ‘Troubled families programme’), many local authorities have been using the Family Star Plus as part of evidencing outcomes within their approved Outcomes Plans. This version of the Star was developed specifically to be aligned with the programme’s objectives.

We are currently putting together guidance to show how the Family Star Plus fits the increased list of headline outcomes in the ‘Supporting Families Outcomes Framework 2022-25’, which will be in effect from 3 October 2022:

The new framework identifies ‘practitioner assessments and contacts’ or ‘Practitioner and/or self-assessment’ as valid data sources as part of the evidence within all of the ten headline outcomes, and the Outcomes Star, while collaboratively completed, fits within this category. 

‘Validated outcome measures’ are also suggested as evidence for meeting some outcomes. The validation of the Family Star Plus as an outcomes measure, including evidence of ‘inter-rater reliability’ and ‘predictive validity’ is reported in a recent peer-reviewed journal article (Good & MacKeith, 2021). 

We will publish the 2nd Edition of the Family Star Plus shortly as well as guidance on how to use the Family Star Plus (2nd Ed.) within the new Supporting Families framework.

Please contact our Research Analyst, Dr Anna Good (anna@triangleconsulting.co.uk) if you would like to find out more.

Making change visible: The Outcomes Star captures important achievements that could be missed by focusing on hard outcomes

Posted on by Donna

As the creators of a suite of measures capturing distance travelled towards ‘hard outcomes’ we are sometimes asked whether there is evidence that Star readings correlate with or predict outcomes such as offending or employment. In some cases, we hear there is resistance to using the Star and instead commissioners, managers or funders are only interested in how many service users have ticked the box of meeting these hard outcomes. This misses out on capturing important achievements, ignores the role of internal change in maintaining concrete achievements and disincentives working with those most in need of support.

This briefing describes some of the evidence we have of the ‘predictive validity’ of the Star – that it does in fact predict outcomes such as school attendance, employment, training and accommodation status. This includes findings reported in two articles recently published in peer-reviewed journals.

In it, we also explain the value of the Outcomes Star in measuring the full journey leading up to and including changes in behaviour or circumstances.

The author of this briefing, Dr Anna Good, draws on her expertise in behaviour change theory to summarise the strong evidence base supporting the importance of the changes assessed by the Star. It is clear from the research literature (and our extensive experience of working with service providers), that early steps on the Star’s ‘Journey of Change’ such as acknowledging problems and accepting help are often essential to subsequent change in hard outcomes. Moreover, change in skills, confidence and beliefs are often key factors in the maintenance of life-changing improvements.

****

Please download our new briefing, ‘The Outcomes Star captures important achievements that could be missed by focusing on hard outcomes’.  If you would like more information or support about the use of Star data, please get in touch with us at info@triangleconsulting.co.uk or +44 (0) 207 272 8765.

The Outcomes Star as a management information tool

Posted on by Donna
The Outcomes Star is well established as a tool for supporting effective keywork and demonstrating achievements. Here, 'Triangle's Research Analyst, Dr Anna Good, discusses a third benefit, the opportunity for internal learning. This new briefing describes how Star data can be used to improve service delivery.

Learning from Star data at all levels of the organisation

Over three-quarters of Outcomes Star users in our client survey said Star data reports were ‘useful for learning how their service was ‘doing’ and ‘helpful in managing or developing the ‘service’. Indeed, Star data can provide meaningful management information at all levels, from a service manager reviewing a single ‘worker’s caseload to a senior management team reviewing data aggregated across services. 

Alongside other data (e.g. satisfaction surveys, output and process data), Star data reports, such as those available from our upgraded Star Online System, allow organisations to ask increasingly focused questions about what is happening with the people they support.

Managers can gain essential insights by looking at differences in starting points and change across outcome areas, client groups, and service settings.  Because these insights are likely to be greatest when compared against prior expectations, Triangle has produced resources to support ‘Star data ‘forecasting’.

Learning from Initial Star readings

The distribution of first Star readings provides a valuable overview of people’s needs coming into the service. Star readings can be compared against expectations to ensure that service users are entering the service appropriately and are offered suitable interventions.

An excellent example of the use of first readings is in Staffordshire County Council, where they look at start readings to see if the families are at the right level of service. In our interview with the Commissioning Manager at the time, she told us that “if we have families in our Family Intervention service that have readings of five, I look a bit deeper to see if we’re really using our resources correctly”.

Learning from change in Star readings

Movement in Star readings for each outcome area also provides an opportunity to learn where things are going well and when further exploration of service delivery may be warranted. 

For example, if one service shows different outcomes to another service, this is a starting point for further investigation:

  • Is there other evidence that one service facilitates better outcomes than another?
  • Are there reasons why one service might be supporting people better than another?
  • Is the service user profile different in the different services?
  • Is practice significantly different in that service, and might there be lessons for other services?

A more in-depth analysis of the movement from each Journey of Change stage is also possible, offering more significant potential for learning than typical numerical outcome scales. Managers can explore which stage transitions are happening frequently and where there may be blockages to making other transitions. For example, a service may be very good at helping service users to begin accepting help but struggle more with moving them towards greater self-reliance, limiting the progress currently being made. Specific changes to service delivery might then need to be developed.

*****

Please download our new briefing, ‘The Outcomes Star as a management information tool‘ for more detail on how Star data reports can be used to improve service delivery. If you would like more information or support about the use of Star data, please get in touch with us at info@triangleconsulting.co.uk or +44 (0) 207 272 8765.

New Outcomes Stars for mental health

Posted on by Outcomes Star

Introducing the new My Mind Star and a new, improved edition of the Recovery Star

My Mind Star – a much needed and timely tool for early intervention with young people

My Mind Star was developed in collaboration with managers, service users and professionals at leading UK children’s charity, Action for Children. It was also piloted by:

The results of the pilot were very positive, with 94% of young people agreeing that their complete Star was ‘a good summary of my life right now”.

94% of practitioners agreed that My Mind Star gave them a better idea of the support needs of the young people they support.

“Often young people have not been listened to or given control: completing the Star gives them space and lets them take the lead.” 

 Grainne Hart, Manager of the Choices Service, part of the My Mind Star pilot

Find out more about the My Mind Star here.

The Recovery Star (4th Edition)

This is a new and improved edition, drawing on independent research and feedback from service users, keyworkers and organisations.

The new edition retains the person-centred, strengths based approach of previous editions but with even more accessible language, incorporating trauma-informed thinking and fuller acknowledgement of the impact of external factors.

There is fuller recognition of the necessity of on-going support for enduring and severe conditions. It is backed by a report on independent research into the psychometric properties and a review of literature supporting the Journey of Change and choice of outcome areas.

Find out more about the new Recovery Star here.

Both Stars were launched at the Govconnect Mental Health 2019 Conference at the Royal Society of Medicine on 26th September.

If you have any questions about our new Stars, any queries about transitioning between the Recovery Star 3rd Edition and the new Edition, or you would just like find out more about how the Stars can support your service users, keyworkers and organisation, please contact us on info@triangleconsulting.co.uk or +44 (0) 207 272 8765.

Psychometric testing of the Outcomes Star

Posted on by Donna

The Outcomes Star has been tested psychometrically. A new set of psychometric factsheets demonstrate the validity of the Outcomes Star, and reveal how the Star can produce informative and valuable outcomes data for commissioners, funders and organisations.

Psychometric testing tells us how confident we can be in the data produced by a measurement tool including whether it measures what it claims to measure and produces consistent scores.

Triangle has published a set of factsheets to demonstrate the psychometric properties of every version of the Star. We are also in the process of having an article validating the Family Star Plus published in a peer reviewed journal. Dr Anna Good has produced a psychometric factsheet for each of the Outcomes Stars, providing the findings from a number of these tests. She explains a bit more about the process and importance of the ensuring the Stars are tested psychometrically.

“At its essence, validity means that the information yielded by a test is appropriate, meaningful, and useful for decision making” (Osterlind, 2010, p. 89).

Psychometric validation has been used in some form for over a hundred years. It involves tests of validity (usefulness and meaningfulness) and reliability (consistency), for example:

  • expert opinion about the content of the measure
  • clustering of ‘items’ or questions into underlying constructs
  • consistency across the readings produced by each item
  • consistency across ‘raters’ using a tool
  • sensitivity to detect change over time
  • correlation with, or predicts of, other relevant outcomes

Why is it important to test the Star psychometrically? What are the benefits of testing the Outcomes Star?  What’s the background to the research?
Triangle recognises the importance of having ‘evidence and theory support the interpretations of test scores’ (AERA, APA & NCME, 1999, p.9), both because we are committed to creating scientifically sound and useful tools and because policy advisors, commissioners and managers require validated outcomes measures and want assurance of a rigorous process of development and testing.

The validation process is an important part of the development of new versions of the Star – we need to know that the outcome areas hang together coherently, whether any outcome areas are unnecessary because of overlap with other areas or have readings that cluster at one end of the Journey of Change.

Once there is sufficient data, we also conduct more extensive psychometric testing using data routinely collected using the published version of the Star. This is beneficial for demonstrating that the Star is responsive to change and that Star readings relate to other outcome measures, which is important both within Triangle and for evidencing the value of our tools externally.

What was involved in producing the psychometric factsheets?
The initial validation work for new Stars is conducted using data from collaborators working with Triangle during the Star development and piloting process. It involves collecting Star readings and asking service users and keyworkers to complete questionnaires about the acceptability and how well the Star captures services users’ situations and needs.

The further testing of the published version uses a larger sample size of routinely collected Star data and assesses the sensitivity of the Star in detecting change occurring during engagement with services. Whenever possible, we collaborate with organisations to assess the relationship between Star readings and validated measures or ‘hard outcome measures’ such as school attendance.

We have also been working to assess consistency in worker’s understanding of the scales using a case study method. This method is described fully in an article published in the Journal of Housing Care and Support (MacKeith, 2014), but essentially involves working with organisations using the Star to develop an anonymised case study or ‘service user profile’, and comparing the readings assigned by trained workers with those agreed by a panel of Star experts. The findings tell us how consistent and accurate workers in applying the Star scales when given the same information. 

Conclusion: An evidence-based tool
The Outcomes Star is an evidence-based tool. The development of new Stars follows a standardized and systematic process of evidence gathering through literature reviews, focus groups, refinement, initial psychometric analyses and full psychometric testing using routinely collected data.

Psychometric validation is useful in the development of new Stars and to provide evidence that the Outcome Star can produce data that meaningfully reflects the construct it is designed to measure.

Organisations can use Triangle’s psychometric factsheets alongside peer reviewed articles to demonstrate the validity of the Outcomes Star to funders and commissioners, and to have confidence that provided it is implemented well, the Star can produce informative and useful data.

Interested in finding out more about psychometrics testing and the validity of the Star?
Take a look at our research library. For more information on the key terms and to read the psychometric factsheets please read the Psychometrics Overview or visit www.outcomesstar.org.uk/about-the-star/evidence-and-research/star-psychometrics. Contact Triangle at info@triangleconsulting.co.uk for more information.

*****

References:

American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing (4th ed.). Washington, DC: American Educational Research Association. 

Mackeith, J. (2014). Assessing the reliability of the Outcomes Star in research and practice. Housing, Care and Support, 17(4), 188-197.

Osterlind, S. J. (2010). Modern measurement: Theory, principles, and applications of mental appraisal (2nd ed.). Boston, MA: Pearson Education.

*****

Dr Anna Good: Dr Anna Good is a Research Analyst at Triangle: a large part of her role involves testing the psychometric properties of the Star, conducting research and supporting organisations to make the best use of Star data. After completing an MSc and a PhD in Psychology with specialisms in behaviour change interventions and psychological research methods, Anna spent a number of years as a post-doctoral researcher, including two years as principal investigator on a prestigious grant examining health behaviour change.

For more information on evidence and research into the Star please visit our Research Library or contact us: email Triangle at info@triangleconsulting.co.uk, or call on +44(0)202  7272 8765.