Our March newsletter round-up includes updates on our licensing options, an introduction to our new Managing Director, and a call for interest and views on a Star for Neurodevelopment as well as updates on our new resources and reports.
New resources and reports
Triangle continues to publish new resources to support organisations in creating positive change by promoting an effective outcomes approach and providing sector-wide solutions:
Contact Triangle at info@outcomesstar.org or +44(0) 20 7272 8765 for more information on our Outcomes Stars or to find out more about how the Star can empower service users and keyworkers to make and measure positive change. Sign up for our newsletter here.
This month’s newsletter round-up includes news on new reports; two new Outcomes Star case studies; new Star research and pyschometric findings; links to Family Nurse Partnership’s newly published blog post on the development of our upcoming New Mum Star and an interview with Clare Burrell, Head of Strategic Commissioning and Policy for Children and Families at Essex County Council on the Family Star (Relationships) the new Star for reducing inter-parental conflict.
New resources and reports
We have recently published a new set of psychometric factsheets as well as a short blog on the role of psychometric testing and why it matters. We also introduced new resources including:
The Family Star (Relationships) was one of two new Stars published by Triangle in September 2018. It is designed for services working with families where inter-parental conflict is a major focus, with the aim of protecting their children from the harmful effects and enabling them to thrive. We recently interviewed Clare Burrell, Head of Strategic Commissioning and Policy for Children and Families at Essex County Council, who was one of the key collaborators spearheading the development, to find out why they invested in the development of this new Outcomes Star and to hear her thoughts on Triangle’s collaborative process.
Clare, could you tell us a bit about your role and how you first came across the Outcomes Stars? I lead on Early Intervention where, like others, we take a holistic approach to our work. Five years ago, we set up our Family Innovation Fund which is delivered by voluntary and community providers across Essex. It is important for commissioners to understand and know that the right outcomes are being delivered and to evaluate the impact of the work. It was then we took the decision to ensure that all the providers of these services would use the same outcomes tools, so that we could measure apples with apples rather than apples with pears! The aim was that doing this would strengthen the case for what works and help us develop the economic case for early intervention. Our intendent evaluators, Traverse, reviewed the outcomes tools available and they recommended we adopt the Outcomes Stars. As a result we prescribed the Family Stars to our providers and over time, they have come to realise the potential and usefulness of them as tools.
How did your focus around inter-parental conflict develop? The primary outcome of our work was family stability and through our work with whole families, it became increasingly evident within the early intervention space that if we were going to improve outcomes for children, we needed to focus more sharply on parental relationships. At the same time, we became one of 11 Local Authorities working with the Department for Work & Pensions, to develop the evidence for what works to reduce inter-parental conflict. It was then that we partnered with colleagues at Hertfordshire County Council and with DWP, to commission an Outcomes Star that would support our work in this area. We needed something different to any of the Stars which already existed, to take in the specific intricacies of relationships.
Did you find anything challenging about the collaborative process to develop a Star? Inter-parental conflict is a relatively underdeveloped area of practice, so there was quite a bit of interesting discussion and some challenges for everyone to agree on the focus and content for the Star. A couple of providers decided not to continue with the process and pilot because the Star didn’t fit with their way of working with families – they didn’t have enough time with parents in conflict or were solely focused on getting an agreement for contact with children rather than working holistically, for example. It was helpful to have a provider with vast experience and knowledge of domestic abuse – they made an invaluable contribution to ensure that the Star didn’t inappropriately stray into that space and that it was clear where practitioners using the Star would draw the line.
What would your top tips be for anyone considering collaborating with Triangle to develop a Star? I would say there are two main areas. The first is getting the right people around the table. It can be hard at times, but it is important that the right voices and heads are in the room for the key Star development sessions. I focused on the ‘unusual suspects’ – people who came from all different perspectives but who would challenge and mix up our thinking. It was also important to get people who were committed to the whole of the process, as it takes a considerable amount of time – over a year in our case.
The second is to plan your pilot upfront and think about how you will get buy-in from practitioners. It’s more than just being in the sessions, it’s about committing to piloting the tool and feeding back throughout the process. We piloted with our commissioned providers, and some of our in-house services. This particular Star brought a challenge because it was used only as and when an inter-parental conflict focused case came up, so that limited the amount of data and feedback and made it more difficult to evaluate in a meaningful way, beyond case studies. That said, practitioners love the new Star and are using it in their work, including in unexpected circumstances – as an example one practitioner has recently used the Star to support a teenager and parent in conflict.
And lastly Clare, what are your hopes for the Family Star (Relationships)? My plan is that this Star is embedded in practice. Essex is a big place, but we are doing a lot to make sure that our practitioners know about this Star and that it is part of their toolkit to use with families and that they can draw on it when inter-parental conflict is the main issue for a family they support.
Many thanks to Clare for her time for this interview.
*****
The Family Star (Relationships) is designed for practitioners working to identify and support the resolution of inter-parental conflict so that outcomes for children and young people are improved. It was developed by Triangle in 2017 – 2018 in collaboration with Essex County Council and Hertfordshire County Council. Please see the Family Stars page for more information about the new variant for supporting parents to manage conflict: the Family Star (Relationships).
94% of practitioners piloting the Star indicated they found the Family Star (Relationships) described the situation, strengths and needs of the service users they worked with either well (either fairly or very well). 80% of parents who completed feedback about the Family Star (Relationships) said it helped them understand what they needed in the way of support.¹ For more information about how Stars are developed, and Triangle’s collaborative process please see our Star Development page.
For an example of Star data in action, please see the recently published evaluation report from OPM Group for Essex County Council’s Early Help Programme, the Family Innovation Fund (FIF). FIF used the Family Star Plus and My Star as primary evaluation tools, with practitioners completing Stars in collaboration with service users, as an integrated part of the Early Help support being provided. Star data was used to demonstrate change, as part of an economic evaluation, and to make the case for early help provision.
Contact Triangle at info@outcomesstar.org or +44(0) 20 7272 8765 for more information.
[1] 44 initial Star readings were completed during the 6 months pilot period, in frontline services in Essex and Hertfordshire.
The Outcomes Star has been tested psychometrically. A new set of psychometric factsheets demonstrate the validity of the Outcomes Star, and reveal how the Star can produce informative and valuable outcomes data for commissioners, funders and organisations.
Psychometric testing tells us how confident we can be in the data produced by a measurement tool including whether it measures what it claims to measure and produces consistent scores.
Triangle has published a set of factsheets to demonstrate the psychometric properties of every version of the Star. We are also in the process of having an article validating the Family Star Plus published in a peer reviewed journal. Dr Anna Good has produced a psychometric factsheet for each of the Outcomes Stars, providing the findings from a number of these tests. She explains a bit more about the process and importance of the ensuring the Stars are tested psychometrically.
“At its essence, validity means that the information yielded by a test is appropriate, meaningful, and useful for decision making” (Osterlind, 2010, p. 89).
Psychometric validation has been used in some form for over a hundred years. It involves tests of validity (usefulness and meaningfulness) and reliability (consistency), for example:
expert opinion about the content of the measure
clustering of ‘items’ or questions into underlying constructs
consistency across the readings produced by each item
consistency across ‘raters’ using a tool
sensitivity to detect change over time
correlation with, or predicts of, other relevant outcomes
Why is it important to test the Star psychometrically? What are the benefits of testing the Outcomes Star? What’s the background to the research? Triangle recognises the importance of having ‘evidence and theory support the interpretations of test scores’ (AERA, APA & NCME, 1999, p.9), both because we are committed to creating scientifically sound and useful tools and because policy advisors, commissioners and managers require validated outcomes measures and want assurance of a rigorous process of development and testing.
The validation process is an important part of the development of new versions of the Star – we need to know that the outcome areas hang together coherently, whether any outcome areas are unnecessary because of overlap with other areas or have readings that cluster at one end of the Journey of Change.
Once there is sufficient data, we also conduct more extensive psychometric testing using data routinely collected using the published version of the Star. This is beneficial for demonstrating that the Star is responsive to change and that Star readings relate to other outcome measures, which is important both within Triangle and for evidencing the value of our tools externally.
What was involved in producing the psychometric factsheets? The initial validation work for new Stars is conducted using data from collaborators working with Triangle during the Star development and piloting process. It involves collecting Star readings and asking service users and keyworkers to complete questionnaires about the acceptability and how well the Star captures services users’ situations and needs.
The further testing of the published version uses a larger sample size of routinely collected Star data and assesses the sensitivity of the Star in detecting change occurring during engagement with services. Whenever possible, we collaborate with organisations to assess the relationship between Star readings and validated measures or ‘hard outcome measures’ such as school attendance.
We have also been working to assess consistency in worker’s understanding of the scales using a case study method. This method is described fully in an article published in the Journal of Housing Care and Support (MacKeith, 2014), but essentially involves working with organisations using the Star to develop an anonymised case study or ‘service user profile’, and comparing the readings assigned by trained workers with those agreed by a panel of Star experts. The findings tell us how consistent and accurate workers in applying the Star scales when given the same information.
Conclusion: An evidence-based tool The Outcomes Star is an evidence-based tool. The development of new Stars follows a standardized and systematic process of evidence gathering through literature reviews, focus groups, refinement, initial psychometric analyses and full psychometric testing using routinely collected data.
Psychometric validation is useful in the development of new Stars and to provide evidence that the Outcome Star can produce data that meaningfully reflects the construct it is designed to measure.
Organisations can use Triangle’s psychometric factsheets alongside peer reviewed articles to demonstrate the validity of the Outcomes Star to funders and commissioners, and to have confidence that provided it is implemented well, the Star can produce informative and useful data.
American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing (4th ed.). Washington, DC: American Educational Research Association.
Mackeith, J. (2014). Assessing the reliability of the Outcomes Star in research and practice. Housing, Care and Support, 17(4), 188-197.
Osterlind, S. J. (2010). Modern measurement: Theory, principles, and applications of mental appraisal (2nd ed.). Boston, MA: Pearson Education.
*****
Dr Anna Good: Dr Anna Good is a Research Analyst at Triangle: a large part of her role involves testing the psychometric properties of the Star, conducting research and supporting organisations to make the best use of Star data. After completing an MSc and a PhD in Psychology with specialisms in behaviour change interventions and psychological research methods, Anna spent a number of years as a post-doctoral researcher, including two years as principal investigator on a prestigious grant examining health behaviour change.
For more information on evidence and research into the Star please visit our Research Library or contact us: email Triangle at info@outcomesstar.org, or call on +44(0)202 7272 8765.
This month’s newsletter round-up includes news on two new Stars, newly published research on using Star data with research and evaluation in Early Help services, job vacancies, future Star development as well as upcoming events.
Outcomes Star events
Triangle are running free events across the country in Brighton, London and Glasgow for service managers and practitioners of frontline services. Morning sessions are for services who have heard about the Outcomes Star and would like to know more, afternoon sessions are for services who already use the and will focus on implementation and getting the most of out of the Star and data. Spaces are limited: find out more and book your place here.
Dr Anna Good: Dr Anna Good is a Research Analyst at Triangle: a large part of her role involves testing the psychometric properties of the Star, conducting research and supporting organisations to make the best use of Star data. After completing an MSc and a PhD in Psychology with specialisms in behaviour change interventions and psychological research methods, Anna spent a number of years as a post-doctoral researcher, including two years as principal investigator on a prestigious grant examining health behaviour change.