Equipment and Modification Services: Evaluation of the Prioritisation Tool Pilot

Summary of Findings and Recommendations of the Report prepared by Synergia Limited

This report is a summary of the results of an independent evaluation undertaken following the Equipment and Modification Services’ (EMS) Prioritisation Tool Pilot. This pilot tested an alternative approach to assessment for access to support based on prioritisation of the disabled person’s need, risk and ability to benefit from equipment or modifications.

The aim of the evaluation was to determine the acceptability and usability of the EMS Prioritisation Tool for disabled people and their families and whānau and EMS Assessors, and measure its impact and validity. This included identifying any financial or policy implications.

The evaluation of the pilot was undertaken for the Ministry of Health by Synergia Limited, a consulting, research and evaluation group specialising in health, social services and the public sector. Synergia Limited was selected following a tender process coordinated by the Ministry.

The Ministry’s project team has now considered the recommendations made in the report and a decision has been made by Disability Support Services to move the project forward to the next stage. This will include revising and updating aspects of the Tool.

A focus for the project will be on engaging with all key stakeholders so that a clearer, more transparent and fairer prioritisation process for equipment and modifications is in place by the end of December 2013.


Summary of the evaluation

Background

The Ministry is responsible for the funding of disability support services for people with disabilities. Equipment and Modification Services (EMS) are part of this support and are available to people with long-term physical, sensory, intellectual and age related disabilities.

The EMS Improvement Programme report, completed in 2009, determined that the current policy and funding decision making processes were unfair. The funding pool for EMS is capped and access to funding is determined by the two tiered urgency based P1/P2 priority system. However, with all applications essentially P1, the solutions are provided on a first come, first served basis with little means of ensuring that people with the greatest and most urgent need would get their services in a timely manner.

As a result an alternative prioritisation tool was developed to prioritise access to EMS services based on client need, risk and ability to benefit.

The EMS Prioritisation Tool was developed through a robust and comprehensive process involving substantial engagement with both consumers and professionals.

The prioritisation tool and pilot

To test the acceptability and usability of the tool in real world settings a pilot was established in late 2010 with implementation across 13 sites initiated in early 2011. The prioritisation tool pilot was run in parallel to the existing assessment and support decision processes and included an IoL questionnaire completed by the consumer. EMS Assessors completed the balance of the tool; an evaluation of Future Risk, Benefit (including effect of the EMS solution to reverse the impact on life, likelihood of achieving that benefit and expected duration), together with a final component ‘Current Status’  that assessed the ability of consumers to contribute financially to their own solutions.

The pilot included a consumer evaluation section which asked consumers to provide feedback on the tool, its acceptability and ease of use. Data from the pilot was entered into a central database, including each person’s NHI to facilitate matching to subsequent applications and solution cost information held by the Ministry’s two EMS providers, Accessable and Enable New Zealand.

The perceptions of the tool for both EMS Assessors and clients appeared to be influenced by how and when the prioritisation tool and IoL questionnaire is used during the assessment process.  Introducing the tool early in discussions, integrating the tool and collaboratively completing the questionnaire with clients during the course of the assessment appeared to be the most effective process. Those who included the tool as an integral part of clinical practice gained more benefit from the tool than those who treated it as a semi research exercise and did not use the tool to engage consumers in the process or thinking within the tool.

The tool design is logical and elegant. The core constructs and domains used in the tool are strongly supported by key stakeholders and, from the pilot data analysis, represent valid domains that are applicable across age and disability types.

Prioritisation tool pilot evaluation

 The evaluation of the prioritisation tool pilot had four goals:

  1. Contribute to DSS understanding of design and application issues.
  2. Contribute to the understanding of the validity of the prioritisation tool as applied in the pilot settings.
  3. Contribute to pilot site understanding, learning and improvement.
  4. Provide a platform of learning that will support wider adoption by explicitly addressing the issues and concerns of potential adoptees. 

The evaluation has drawn on a very rich and complex set of insight, information and data resources.

Key findings of the evaluation

The evaluation outlined several conclusions, which were encompassed in the recommendations, including:

  • The tool is likely to achieve better results than the current system, which in effect doesn’t prioritise.
  • There is a need to refine parts of the tool with further testing planned prior to roll out.
  • There is a benefit to be derived by addressing some of the language / terminology in the client IoL questionnaire.
  • Areas to consider for future inclusion in the tool are the ‘Impact on Carers’, the ‘Impact on consumers’ ability to maintain their health and wellbeing’, and review of the way ‘Benefit’ is calculated.
  • Testing / modelling on eligibility and financial impacts needs to occur before roll out as well as looking at how the tool can be integrated into operating decisions and risk management processes.
  • Preparation for roll out needs to include training of EMS Assessors and what is required for EMS providers, including potential changes to their processes and IT systems.
  • Communication planning needs to occur that includes engaging with EMS Assessors to seek their input regarding adaptation of the tool.

 Feedback from disabled people

  • The majority of disabled people and their family or whānau reported that, although some changes could be made to the language in the IoL questionnaire, they were comfortable with its use.
  • Some families reported that the impact on their life of supporting their disabled family member was not well covered in the questionnaire.

Feedback from EMS Assessors

  • The concept of, and need for, prioritisation is widely accepted among EMS Assessors. The current P1/P2 system was seen to be flawed and unfair; however assessors expressed reservations about the inclusion of Ministry of Health List Equipment (as these applications are not currently subject to any priority rating) and equipment for people with sensory disabilities.
  • Some EMS Assessors indicated that the pilot implementation process was either not adequately planned, or not communicated clearly enough to the participants of the pilot. In some cases this caused initial confusion, wasted time and resources, and a breakdown in engagement with the prioritisation tool pilot.
  • How the tool was implemented and used across the 13 sites showed considerable variation – this resulted in wide ranging levels of participation of pilot sites and engagement with the process (from virtually no assessments completed using the new tool to the majority of applications for equipment and modifications utilising the new tool).
  • Training in the use of the tool was also handled differently in different regions and the importance of well-planned and delivered training for EMS Assessors was highlighted in the evaluation. 

Future directions

Moving from the lessons learned in the pilot to considering the future Synergia believes there are some critical questions to be answered about the future direction for prioritisation, within EMS, that should shape the programme going forward:

Is prioritisation needed, is the case established?

The fundamental proposition for prioritisation is that demand exceeds available resources and that effective, transparent, equitable and sustainable means of prioritising need and allocating resources is required. The view of Synergia is that a better process of prioritisation must be incorporated to enable EMS to respond to continued demand pressure within a capped budget.

Is the pilot prioritisation tool heading in the right direction of an effective and sustainable solution?

In the view of Synergia the answer to this is a qualified ‘Yes’. The basic principles and approach have proven to be sound with a good level of acceptance from consumers. However there are a number of areas that require improvement. EMS Assessors need to be more effectively engaged in the next step developments to ensure that their legitimate concerns are addressed through effective improvements or assistance provided to develop new paradigm thinking and practice where this may be needed.

Pragmatically it is recommended by Synergia that the Ministry of Health List Equipment (List Equipment) is excluded in the first wave of implementation. This is seen as only being temporary as List equipment will need to be included in the future to avoid large inequities of access developing as demand grows.

Will implementation of the prioritisation tool achieve a fair and equitable means of allocating resources?

The financial modelling facilitated by the pilot data demonstrates that the tool and the priority profile that is generated can technically work to allocate a capped budget. There are a number of challenges to be addressed in making the resource allocation processes work fairly across the country and over time that they are included in the financial modelling section. Attention to the technical implementation of a priority based financial management system will need to be a priority of the Ministry and providers in future development plans.

Apart from the sensory disability group, (which represented a very small proportion of the pilot sample) there was no EMS Assessor feedback to suggest that the priority rankings for the over 65 years or age related disability types contained systemic errors. It is reasonable to believe that the differential access rates for these groups are likely to fairly represent relative priority. However this should be the focus of monitoring and evaluation of the system as it is implemented to ensure that this assumption is not in error.

Can the prioritisation tool be effectively and efficiently deployed on an on-going basis?

The short answer is that the pilot process cannot answer this question. The tool was tested on a standalone non-integrated basis. There is work to be done to reduce complexity and facilitate integration into practice and application processes.

The issue is pressing for EMS Assessors operating in List Equipment environments where the high volume of assessments performed annually creates a considerable burden of administrative cost.

For EMS Assessors operating in the Complex Equipment and Modifications space, the existing assessment and application processes are already administratively intensive. With much smaller volumes the relatively shorter additional time to complete the tool is much less of a burden.

A critical component of future development will be the process redesign of the assessment, priority and support decision flow.

Will the prioritisation approach and tool be able to generate and maintain confidence and integrity in on-going use?

Again this is an area which the pilot has not attempted to address but is one that Synergia can point to the experience of similar systems elsewhere in Health.

In the view of Synergia any future development must address these issues within its programme:

  • build understanding of the principles, logic and methods used in prioritisation tools and priority based resource allocation decisions
  • develop approaches, guides and training for clinical practice
  • manage funding and access threshold decision making in a data informed and transparent manner
  • develop systems for moderation and judgement development to improve the confidence of EMS Assessors in the quality of their assessments
  • establish systems to monitor patterns of priority for gaming or abuse
  • establish monitoring and evaluation of the impact on consumers who are below access thresholds to monitor for unintended consequences and to improve the prioritisation approach.

Criteria for successful implementation and roll out

The most critical factors identified by Synergia for successful implementation and roll out of the EMS tool are:

  • the integration of the prioritisation tool within the EMS assessment and application process
  • the tool becoming part of a system that is simple, streamlined, and straightforward
  • a commitment to a programme of continuous improvement of the tool involving clients, EMS Assessors and providers
  • the development of clear communication and feedback loops with EMS Assessors
  • ensuring there is well-planned training for EMS Assessors.

Recommendations

Overall, Synergia concluded that the case for the implementation of a new tool for prioritising EMS had been well established – a better process for the prioritisation of services must be incorporated to enable the EMS Assessors and providers to respond to continued demand pressures within a capped budget.

The evaluation also concluded that the basic principles were sound and that the tool was heading in the right direction as an effective and sustainable solution. However some recommendations for changes to aspects of the tool were made, to both the IoL questionnaire and the assessor completed components. Many of the recommendations are focussed on the importance of dedicated training for EMS Assessors and providers before successful implementation of the tool could be achieved.

A summary of key recommendations made by Synergia is provided below.

Summary of key recommendations

  • The prioritisation tool should be implemented nationally, subject to some adaptation, to ensure integration with operating practice, processes and systems, and streamlining of work flow.
  • All applications for equipment which is either refurbished or available on the Equipment List should not be included in the initial prioritisation tool.
  • No changes should be made initially to the existing eligibility criteria.
  • Training resources for all parties, particularly EMS Assessors, need to be well planned and delivered to ensure successful integration into the assessment and application processes and on-going implementation.
  • Language in the IoL questionnaire needs to be reviewed to ensure that it is as accessible as possible and reflects the needs of all potential users (including carers and people with sensory impairment).
  • Financial management design needs to include the establishment of a common financial threshold that is used nationally.
  • Ability to contribute should be retained in the consumer IoL questionnaire.
  • Weightings, where there are marginal weighing differences, should be combined, making the tool easier for the EMS Assessor to operate.
  • A review of how ‘Benefit’ is estimated in the tool should be undertaken.

Conclusion

In the face of continued growth in demand for EMS within a capped budget it is clear that the current system of prioritising access to support is inadequate. The evaluation by Synergia found that the aim of the pilot to establish a more fair and equitable prioritisation process is well accepted by clients, EMS Assessors, EMS providers and Ministry staff.

The EMS prioritisation tool represents a substantial step forward in New Zealand practice through the inclusion of client generated ‘Impact on Life’ together with a structure that includes assessment of realisable benefit, likelihood, duration and risk. 

The IoL component of the prioritisation tool is seen by Synergia as both the most positive and most challenging component of the prioritisation tool. Support for the approach and the five core domains of the IoL questionnaire are strongly positive.

While the prioritisation tool requires refinement before use in on-going operation the evaluation shows that it can effectively prioritise demand with high reported levels of client acceptability and ease of use.

Acceptability and confidence levels of EMS Assessors show some variation, however solutions to the most immediate concerns should be readily achievable. The evaluation has highlighted a number of areas requiring more fundamental review but Synergia proposes that these should be considered as part of a version 2 release of the tool at a later date. 

EMS Assessors need to be more effectively engaged in the next step developments to ensure their concerns are addressed and that implementation assistance is provided to integrate prioritisation into good assessment practices and processes. 

For some assessors the introduction of the prioritisation tool may require a substantial paradigm shift from their current process of performing assessments. Time and effort will be required for people to understand and gain confidence in the prioritisation tool.

The overall view of Synergia is that the concept and direction of prioritisation is accepted and the tool is a solid step in the right direction for the comprehensive determination of priority.

Back to top