The evaluation’s potential areas of focus
- The changing nature of government policies over time (funding levels, desired outcomes, strategies).
- The volume of outputs delivered, service coverage, adequacy of access.
- Service quality.
- The program’s impact on stakeholders (e.g. clients, their families, the broader community, both intended and unintended effects, interaction with other programs).
- Equity of funding allocation across geographic regions and across different client groups.
- Performance monitoring, evaluation and accountability mechanisms.
Systemic challenges
- The department has not developed a program logic model or theory of intervention to specify how the program is supposed to work, any theories of action are implicit not explicit.
- The target group cannot be serviced at agreed standards within current resource allocation. This leads to waiting lists, queuing, the “unofficial” raising of eligibility requirements or narrowing of the intended target group.
- Infinite demand – no clear target to measure against, demand management (i.e. demand reduction) becomes management’s focus, not meeting demand.
- No clear measure of successful outcomes or of appropriate service provision is available (ie unclear criteria and standards for service delivery).
- Workforce issues are common – staff shortages, a need for training, high turnover, low morale. This often results in an inadequate quality of services.
- Large amounts of administrative program data are collected, but data is of poor quality, limited ad hoc analysis is undertaken of what the data actually means. Client files are often incomplete.
- Decentralised service provision coupled with unclear roles and responsibilities between the central administration and regional offices.
- Contracted service delivery with inadequate contract management and quality oversight.
- State governments often have limited control (the national government has a policy and funding role and alternative service providers are also operating in the sector) but the state has overall responsibility for ensuring good outcomes. The state's role becomes not the efficient and effective delivery of an entire service, but of efficiently and effectively plugging gaps in available services without providing perverse incentives.
Methodological problems
- Limitations in the available data, IT systems are not user friendly
- Privacy/ethical considerations
- Identifying all relevant desired outcomes for the client group
- Evaluators often have a need for independent ‘expert’ advice on substantive matters, for example when assessing the quality-of-service delivery
- Sensitivities relating to evaluators second-guessing “clinical” judgements (although there are ways around this)
- Decentralized service provision means time consuming and expensive evaluations with large samples
- When service delivery has been contracted out evaluators will need the authority to examine the contracted operations (the ability to follow the money).