We are sometimes asked whether changes in Star readings actually reflect the changes that occur during service provision. We have three responses to this: the first two are based on the practices we have in place for the development of new Stars and for training and implementation of the Star. The third is to present the research evidence that the Star readings can be applied accurately and that readings correlate with other measures in the expected way.
Star Development
New versions of the Star are created alongside managers and practitioners to ensure the Journey of Change captures key changes occurring for those using services.
Pilot data is statistically analysed to check that the scales are sensitive enough to detect change.
Service users and practitioners provide end-of-pilot feedback about the extent to which the Star captures the changes made.
Training and Implementation
For the Star to accurately reflect change, practitioners should be well-trained with ongoing support to continue using it well. This is why training is mandatory, we provide free CPD for licensed trainers and encourage refresher training, regular supervision, and auditing.
Research Evidence
Convergent validity: Star readings have been shown to correlate with other validated measures in our own, as well as in external peer-reviewed research.
Predictive validity: Star readings, and change in Star readings, predict hard outcomes such as securing accommodation, employment, and school absenteeism
Inter-rater reliability: different practitioners are able to consistently assign Star readings
We are keen to conduct further analyses of the relationship between Star readings and other measures, so please get in touch if you have linked data and are interested in us exploring it.
To read our three page briefing providing a more detailed version of the above, please download it here (PDF).
Since the publication of our Enabling Helpreport last year, we have been working with Toby Lowe and colleagues at the Centre for Public Impact to articulate how the Outcomes Star can operationalise learning as a management strategy and other core components of the Human Learning Systems (HLS) paradigm.
Both HLS and Enabling help focus on moving on from the failings of New Public Management towards an alternative vision of supporting ‘human freedom and flourishing’ through creating flexible, relational, compassionate service delivery systems in which the role of measuring is to support learning for individuals, organisations and places.
The Star as a tool for turning HLS and Enabling Help principles into practice
We are delighted that the Star is one of the tools identified in the Centre for Public Impact’s recently launched guide: Human Learning Systems: A practical guide for the curious, commissioned by Healthcare Improvement Scotland and Iriss.
The guide highlights three different ways that the Outcomes Star can be used to put the principles of Human Learning Systems into practice:
Person level: the Star empowers people to take an overview of where they are, understand their life as a system, choose their goals and work out the next steps to achieve them
Organisation/place level: the Star enables meaningful purpose-aligned reflection and learning about what is and isn’t working within services, in order to better support those they serve
Connected learning cycles: the Star offers a way of collective sense-making between different actors within or across organisations, for example within a multidisciplinary team supporting an individual, or between commissioners and service providers
The synergy between the Star and human learning systems
HLS started as an analysis of the problems within traditional approaches to public service management. It has gone on to articulate an alternative framework and then look for practical ways of bringing that vision to life.
The Outcomes Star, in contrast, started as a practical tool for service providers to evidence that they are making a difference in a way that really reflects and supports what they do. From there we articulated the vision of service delivery and public management that was implicit in the tool in our Enabling Help report.
The convergence of these approaches, despite their different starting points, speaks volumes. The time has come for this shift in thinking and practice. We look forward to furthering collaboration with the HLS movement and others who share this vision.
Could your use of the Star inspire others? Are you working with the Star as part of a wider HLS approach? Please get in touch with our Research analyst, Dr Anna Good (anna@triangleconsulting.co.uk), as we are keen to create and publicise more case studies.
As the creators of a suite of measures capturing distance travelled towards ‘hard outcomes’ we are sometimes asked whether there is evidence that Star readings correlate with or predict outcomes such as offending or employment. In some cases, we hear there is resistance to using the Star and instead commissioners, managers or funders are only interested in how many service users have ticked the box of meeting these hard outcomes. This misses out on capturing important achievements, ignores the role of internal change in maintaining concrete achievements and disincentives working with those most in need of support.
This briefing describes some of the evidence we have of the ‘predictive validity’ of the Star – that it does in fact predict outcomes such as school attendance, employment, training and accommodation status. This includes findings reported in two articles recently published in peer-reviewed journals.
In it, we also explain the value of the Outcomes Star in measuring the full journey leading up to and including changes in behaviour or circumstances.
The author of this briefing, Dr Anna Good, draws on her expertise in behaviour change theory to summarise the strong evidence base supporting the importance of the changes assessed by the Star. It is clear from the research literature (and our extensive experience of working with service providers), that early steps on the Star’s ‘Journey of Change’ such as acknowledging problems and accepting help are often essential to subsequent change in hard outcomes. Moreover, change in skills, confidence and beliefs are often key factors in the maintenance of life-changing improvements.
The Outcomes Star has been tested psychometrically. A new set of psychometric factsheets demonstrate the validity of the Outcomes Star, and reveal how the Star can produce informative and valuable outcomes data for commissioners, funders and organisations.
Psychometric testing tells us how confident we can be in the data produced by a measurement tool including whether it measures what it claims to measure and produces consistent scores.
Triangle has published a set of factsheets to demonstrate the psychometric properties of every version of the Star. We are also in the process of having an article validating the Family Star Plus published in a peer reviewed journal. Dr Anna Good has produced a psychometric factsheet for each of the Outcomes Stars, providing the findings from a number of these tests. She explains a bit more about the process and importance of the ensuring the Stars are tested psychometrically.
“At its essence, validity means that the information yielded by a test is appropriate, meaningful, and useful for decision making” (Osterlind, 2010, p. 89).
Psychometric validation has been used in some form for over a hundred years. It involves tests of validity (usefulness and meaningfulness) and reliability (consistency), for example:
expert opinion about the content of the measure
clustering of ‘items’ or questions into underlying constructs
consistency across the readings produced by each item
consistency across ‘raters’ using a tool
sensitivity to detect change over time
correlation with, or predicts of, other relevant outcomes
Why is it important to test the Star psychometrically? What are the benefits of testing the Outcomes Star? What’s the background to the research? Triangle recognises the importance of having ‘evidence and theory support the interpretations of test scores’ (AERA, APA & NCME, 1999, p.9), both because we are committed to creating scientifically sound and useful tools and because policy advisors, commissioners and managers require validated outcomes measures and want assurance of a rigorous process of development and testing.
The validation process is an important part of the development of new versions of the Star – we need to know that the outcome areas hang together coherently, whether any outcome areas are unnecessary because of overlap with other areas or have readings that cluster at one end of the Journey of Change.
Once there is sufficient data, we also conduct more extensive psychometric testing using data routinely collected using the published version of the Star. This is beneficial for demonstrating that the Star is responsive to change and that Star readings relate to other outcome measures, which is important both within Triangle and for evidencing the value of our tools externally.
What was involved in producing the psychometric factsheets? The initial validation work for new Stars is conducted using data from collaborators working with Triangle during the Star development and piloting process. It involves collecting Star readings and asking service users and keyworkers to complete questionnaires about the acceptability and how well the Star captures services users’ situations and needs.
The further testing of the published version uses a larger sample size of routinely collected Star data and assesses the sensitivity of the Star in detecting change occurring during engagement with services. Whenever possible, we collaborate with organisations to assess the relationship between Star readings and validated measures or ‘hard outcome measures’ such as school attendance.
We have also been working to assess consistency in worker’s understanding of the scales using a case study method. This method is described fully in an article published in the Journal of Housing Care and Support (MacKeith, 2014), but essentially involves working with organisations using the Star to develop an anonymised case study or ‘service user profile’, and comparing the readings assigned by trained workers with those agreed by a panel of Star experts. The findings tell us how consistent and accurate workers in applying the Star scales when given the same information.
Conclusion: An evidence-based tool The Outcomes Star is an evidence-based tool. The development of new Stars follows a standardized and systematic process of evidence gathering through literature reviews, focus groups, refinement, initial psychometric analyses and full psychometric testing using routinely collected data.
Psychometric validation is useful in the development of new Stars and to provide evidence that the Outcome Star can produce data that meaningfully reflects the construct it is designed to measure.
Organisations can use Triangle’s psychometric factsheets alongside peer reviewed articles to demonstrate the validity of the Outcomes Star to funders and commissioners, and to have confidence that provided it is implemented well, the Star can produce informative and useful data.
American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing (4th ed.). Washington, DC: American Educational Research Association.
Mackeith, J. (2014). Assessing the reliability of the Outcomes Star in research and practice. Housing, Care and Support, 17(4), 188-197.
Osterlind, S. J. (2010). Modern measurement: Theory, principles, and applications of mental appraisal (2nd ed.). Boston, MA: Pearson Education.
*****
Dr Anna Good: Dr Anna Good is a Research Analyst at Triangle: a large part of her role involves testing the psychometric properties of the Star, conducting research and supporting organisations to make the best use of Star data. After completing an MSc and a PhD in Psychology with specialisms in behaviour change interventions and psychological research methods, Anna spent a number of years as a post-doctoral researcher, including two years as principal investigator on a prestigious grant examining health behaviour change.
For more information on evidence and research into the Star please visit our Research Library or contact us: email Triangle at info@triangleconsulting.co.uk, or call on +44(0)202 7272 8765.
Dr Anna Good: Dr Anna Good is a Research Analyst at Triangle: a large part of her role involves testing the psychometric properties of the Star, conducting research and supporting organisations to make the best use of Star data. After completing an MSc and a PhD in Psychology with specialisms in behaviour change interventions and psychological research methods, Anna spent a number of years as a post-doctoral researcher, including two years as principal investigator on a prestigious grant examining health behaviour change.