Open-source LLM benchmarking reveals Australian data performance gaps

Open-source LLM benchmarking reveals performance gaps on Australian data, with frontier models underperforming on local tax law, healthcare guidelines, and financial services regulation compared with US-centric benchmarks. The findings come from a benchmarking study conducted by the University of Sydney and the Australian National University, which evaluated 12 open-source large language models on tasks involving Australian legal, medical, and financial data. The study found that accuracy on Australian-specific tasks was 15 to 25 percent lower than accuracy on equivalent US tasks, with the largest gaps in domains where Australian regulation and practice differ significantly from US norms.

The performance gap is important because Australian enterprises are increasingly using open-source LLMs for internal knowledge work, document analysis, and customer service automation, and they need to understand the limitations of those models on local data before deploying them in production. The gap is particularly acute for regulated industries including financial services, healthcare, and legal services, where inaccurate model output can lead to compliance breaches, financial loss, or harm to customers. The benchmarking study provides empirical evidence that Australian enterprises cannot assume that a model's performance on US benchmarks translates to reliable performance on Australian data.

Benchmark methodology and key findings

The benchmarking study evaluated models including Llama 4, Mistral Large, and Qwen2 on tasks including Australian tax law interpretation, Medicare guideline application, and ASIC regulatory compliance analysis. The tasks were constructed from publicly available Australian government publications, regulatory guidance, and professional practice documents, and they were designed to test the models' ability to extract and apply relevant information from Australian sources. The study found that the models performed well on tasks that required general language understanding and reasoning, but they struggled with tasks that required specific knowledge of Australian regulatory frameworks and professional practices.

The largest performance gaps were in tax law and financial services regulation, where the models frequently cited US tax code provisions or US regulatory frameworks instead of their Australian equivalents. The confusion is a consequence of the training data distribution, which includes a much larger volume of US legal and regulatory text than Australian text. The models have learned statistical patterns from the US text, and they apply those patterns even when the input text is Australian, leading to incorrect citations and inappropriate regulatory references. The problem is not unique to open-source models, because frontier proprietary models exhibit the same bias toward US-centric content.

Implications for enterprise AI deployment

The performance gaps have direct implications for enterprise AI deployment in Australia, where regulated organisations need accurate and compliant AI output for customer-facing and internal decision-making processes. The enterprises that are most affected are those that use open-source LLMs for document analysis, contract review, and regulatory compliance checking, because those tasks require precise knowledge of local regulatory frameworks. The enterprises that are least affected are those that use LLMs for general productivity tasks including email drafting, meeting summarisation, and content generation, where the US bias in training data is less likely to produce incorrect or non-compliant output.

The enterprises that are deploying frontier models in regulated environments are mitigating the performance gap through fine-tuning on Australian regulatory datasets and retrieval-augmented generation that grounds model output in authoritative Australian sources. The fine-tuning and RAG approaches are effective, but they require engineering investment and domain expertise that many enterprises lack. The benchmarking study provides a baseline that enterprises can use to assess the quality of their fine-tuned models and to identify areas where additional training data or retrieval sources are needed.

The case for Australian foundation models

The performance gap has renewed discussion about the need for Australian foundation models that are trained on Australian data and aligned with Australian regulatory and cultural contexts. The argument is that Australian enterprises and government agencies cannot rely on models that are trained primarily on US data for tasks that require knowledge of Australian law, regulation, and practice, and that sovereign AI capability requires models that reflect Australian contexts. The argument is countered by the observation that training a competitive foundation model requires investment at a scale that is beyond the capacity of the Australian market, and that fine-tuning and retrieval-augmented generation are more economically viable strategies for addressing the performance gap.

The Australian Government's AI infrastructure investment at CSIRO Data61 and the Australian National University is aimed at developing domain-specific models for Australian applications, including models for legal, medical, and financial services that are trained on Australian data. The investment is a step toward sovereign AI capability, but it is focused on specialised models rather than general-purpose foundation models. The specialised approach is pragmatic, because it targets the domains where the performance gap is most consequential and where Australian data is most different from global training data distributions. Explore more AI research analysis at the Tech & Ideas hub

For the University of Sydney AI benchmarking study, see University of Sydney. The Australian National University's AI research is at ANU. Hugging Face's open-source model repository is published at Hugging Face.