--- title: "Tokens, Latency and Trust: The AI Metrics Leaders in Australia and New Zealand Need to Understand" description: "A plain English guide to time to first token, time per output token, end to end latency, prefix caching and inference cost, and why these metrics matter for IT, security and operational leaders managing AI vendors under APRA and ASIC expectations." type: blog version: 2 version_id: "1d003a6e-e79d-4aaa-bdaf-4eeca7ec6f91" generated_at: "2026-07-07T07:15:46.566Z" author: "Insicon Cyber" date_published: "2026-07-07T06:23:28.000Z" date_modified: "2026-07-07T06:30:43.445Z" language: en reading_time: "12 min" word_count: 2260 keywords: ["How Insicon Cyber helps"] url: "https://insiconcyber.com/blog/ai-inference-metrics-leaders-anz" --- # Tokens, Latency and Trust: The AI Metrics Leaders in Australia and New Zealand Need to Understand > A plain English guide to time to first token, time per output token, end to end latency, prefix caching and inference cost, and why these metrics matter for IT, security and operational leaders managing AI vendors under APRA and ASIC expectations. ## Key Takeaways - Why AI performance metrics are now a leadership issue - The metrics that describe how fast an AI system responds - Inference: the engine behind every AI answer - Prefix caching: the cost lever vendors don't always explain - The security risk hiding inside prefix caching ## Contents - [Why AI performance metrics are now a leadership issue](#why-ai-performance-metrics-are-now-a-leadership-issue) - [The metrics that describe how fast an AI system responds](#the-metrics-that-describe-how-fast-an-ai-system-responds) - [Inference: the engine behind every AI answer](#inference-the-engine-behind-every-ai-answer) - [Prefix caching: the cost lever vendors don't always explain](#prefix-caching-the-cost-lever-vendors-don-t-always-explain) - [The security risk hiding inside prefix caching](#the-security-risk-hiding-inside-prefix-caching) - [Token costs and the hidden line item on your AI bill](#token-costs-and-the-hidden-line-item-on-your-ai-bill) - [What APRA and ASIC expect in practice](#what-apra-and-asic-expect-in-practice) - [Questions for your next AI vendor review](#questions-for-your-next-ai-vendor-review) - [How Insicon Cyber helps](#how-insicon-cyber-helps) 6 min read # Tokens, Latency and Trust: The AI Metrics Leaders in Australia and New Zealand Need to Understand [Insicon Cyber](https://insiconcyber.com/blog/author/insicon-cyber) : Updated on July 7, 2026 [Cyber Security](https://insiconcyber.com/blog/tag/cyber-security) [Governance](https://insiconcyber.com/blog/tag/governance) [AI](https://insiconcyber.com/blog/tag/ai) [ISO 42001](https://insiconcyber.com/blog/tag/iso-42001) Tokens, Latency and Trust: The AI Metrics Leaders in Australia and New Zealand Need to Understand 10:35 AI SECURITY AND GOVERNANCE You don't need to be an AI engineer to run a good vendor review. But when a vendor pitch is built entirely on numbers your team can't interrogate, that's an operational risk, not a technical detail. ## Why AI performance metrics are now a leadership issue Every AI vendor pitch now comes with a slide of numbers. Time to first token. Tokens per second. Cache hit rate. Cost per million tokens. For most leaders and executives across Australia and New Zealand managing IT, security or operations, these numbers wash past as technical noise, somewhere between the demo and the pricing page. That's a problem regulators have now named directly. The Australian Prudential Regulation Authority's letter to industry on artificial intelligence, published 30 April 2026, observed an overreliance on vendor presentations and summaries without sufficient examination of key AI risks such as unpredictable model behaviour and the impact on critical operations. That observation lands squarely on the desks of the people actually running AI deployments day to day: IT leaders, security teams and operational managers who sign off on vendor contracts and carry the consequences when performance or security assumptions don't hold up. ASIC's companion letter of 8 May 2026 (26-092MR) reinforces the same point, requiring AI and cyber risk to be discussed at the highest governance level of every AFS licensee and market participant, which means the operational detail behind that discussion has to come from somewhere. It comes from the leaders and executives closest to the technology. This blog is a plain English glossary of the metrics that actually matter in AI performance and cost, and why each one carries a practical operational and security implication for your team. ## The metrics that describe how fast an AI system responds Behind every AI chat interface, copilot or customer-facing assistant sits an inference process: the model reading a prompt and generating a response, one token at a time. A token is roughly three-quarters of a word (or is that 'w-o-r' 😆). How fast that process runs, and how consistently, is measured by a small set of standard metrics. Different use cases put weight on different metrics. A customer-facing chat assistant lives or dies on TTFT: research suggests a response has to start appearing in under roughly half a second to feel responsive. A batch process summarising thousands of aged care incident reports overnight cares far more about total throughput than about first-token speed. Knowing which metric matters for which workload is the difference between a sensible SLA and one that sounds impressive but measures the wrong thing. ## Inference: the engine behind every AI answer "Inference" is simply the term for a trained AI model being run against new input to produce an output, as distinct from "training," which is the earlier, far more expensive process of building the model in the first place. Every time a staff member sends a prompt or an AI agent processes a case, that's an inference request. It's the ongoing, per-use cost and performance layer your team should be actively monitoring, because unlike training costs, which the vendor absorbs, inference costs scale directly with how heavily your organisation uses the tool. ## Prefix caching: the cost lever vendors don't always explain When an AI model processes a prompt, it builds an internal working memory of that prompt called a KV cache. If the next request shares the same starting text, such as a recurring system instruction, a policy document, or a long case file being asked several questions in a row, the model can reuse that cached memory instead of reprocessing it from scratch. This is prefix caching, also called prompt caching. The economics are significant. Major providers charge substantially less, in some cases around a 90 per cent discount, for cached tokens compared with fresh ones, and cached responses generate noticeably faster. For a team running AI across repeated workflows, such as a fractional CISO reviewing the same client risk register, or an aSOC analyst re-querying the same incident context, prefix caching is the difference between a sustainable AI cost base and a runaway one. Your team doesn't need to manage cache configuration directly, but procurement and platform owners should be asking whether the vendor's pricing model and architecture actually take advantage of it, because the gap between a well cached and poorly cached deployment can be an order of magnitude in ongoing cost. ## The security risk hiding inside prefix caching This is where a pure performance metric becomes a genuine security question, and it's the reason this topic sits inside Insicon Cyber's [AI Security and Governance](/ai-security-governance) practice rather than purely in the IT budget conversation. Because cached prompts respond measurably faster than uncached ones, the response time itself leaks information. Researchers have demonstrated that in multi-tenant AI deployments, where a cache is shared across different users or customers, an attacker can send crafted prompts and measure the time to first token to work out whether a prefix has been seen before, in effect fingerprinting or partially reconstructing another user's private input purely from timing. This is a timing side-channel attack, and it has been documented against several commercial AI API providers. For teams in [financial services](/industries/finance-and-lending), [aged care and healthcare](/industries/aged-care), where AI is increasingly touching client records, medical notes and case files, this is precisely the kind of AI-specific attack pathway APRA's letter calls out directly, alongside prompt injection, data leakage and insecure integrations. It's also a clean, concrete example of why identity and access management and security testing need to be extended specifically to cover AI implementations, not just traditional application layers. ## Token costs and the hidden line item on your AI bill AI usage is billed by the token, both for what goes in (the prompt, including any documents or context) and what comes out (the response). Input and output tokens are often priced differently, cached and uncached tokens are priced differently again, and a single conversation with a large document attached can consume orders of magnitude more tokens than a short question. This matters in a very practical way: a pilot that looks cheap with five staff and short prompts can become a materially different cost line once it's rolled out across a business unit processing full case files or long compliance documents. When you're approving AI budgets or reviewing a vendor contract, ask for a cost-per-transaction model, not just a per-seat licence fee, because per-seat pricing frequently hides the token economics underneath it. ## What APRA and ASIC expect in practice APRA's letter is explicit that entities should map and maintain visibility over their full AI supply chain, including material third and fourth-party dependencies, with contractual arrangements that address audit rights, model updates and incident notification. It also notes that few entities had tested exit or substitution strategies for critical AI providers, and that some were heavily dependent on a single provider across multiple use cases. Performance and cost metrics sit directly inside that supplier risk work. A vendor that can't explain its latency guarantees, its caching architecture, or its cost-per-transaction model under real production load is a vendor your organisation cannot meaningfully audit, and a concentration risk you cannot substitute out of quickly if something goes wrong. This is operational work, not a once-a-year governance exercise. ## Questions for your next AI vendor review - What is our guaranteed time to first token and end to end latency under expected peak load, not a controlled demo? - Is our prompt or context data cached, and if so, is that cache isolated to our organisation or shared across other tenants? - What is our real cost per transaction once input tokens, output tokens and document context are all accounted for, at expected production volume? - What security testing has been performed specifically against AI-specific attack pathways, including timing-based and prompt injection risks? - If this provider became unavailable or was compromised, what is our tested fallback or substitution plan? ## How Insicon Cyber helps Insicon Cyber's [AI Security and Governance](/ai-security-governance) practice, delivered through our team of Australia-based staff, gives IT, security and operational leaders across Australia and New Zealand the practical assurance APRA and ASIC now expect. This includes reviewing AI vendor architecture and contractual protections, testing for AI-specific attack pathways, and turning technical performance metrics into the vendor questions your team needs answered before the next AI deployment goes live. Secure AI. Governed AI. Compliant AI. [Contact Us](https://cta-ap1.hubspot.com/web-interactives/public/v1/track/click?encryptedPayload=AVxigLK29zjMbPSRERfur4HBDbn2%2BM%2B2%2Fmj%2BrkiYk2JpiUtN7CfJzeqlUFvsg1Mda5W%2BsTBqB92iRRrl4wh4nI5rdzkEOciIidHLthJtsgDwuwSMfh%2BeQDU%2Fg0fw9DuINBGCbDECSB1WshLsSZ4b4Et%2F0ZK9BqaX%2B7rYhDOGjQSq%2BCm4q%2F3K&portalId=22526539) * * * ### Sources - APRA, Letter to Industry on Artificial Intelligence, 30 April 2026: [https://www.apra.gov.au/apra-letter-to-industry-on-artificial-intelligence-ai](https://www.apra.gov.au/apra-letter-to-industry-on-artificial-intelligence-ai) - ASIC, Open Letter to AFS Licensees and Market Participants, 26-092MR, 8 May 2026: [https://download.asic.gov.au/media/xhrf1w0e/26-092mr-open-letter-to-afs-licensees-and-market-participants.pdf](https://download.asic.gov.au/media/xhrf1w0e/26-092mr-open-letter-to-afs-licensees-and-market-participants.pdf) - ASD, Frontier models and their impact on cyber security: [https://www.cyber.gov.au/about-us/view-all-content/news/frontier-models-and-their-impact-on-cyber-security](https://www.cyber.gov.au/about-us/view-all-content/news/frontier-models-and-their-impact-on-cyber-security) - IBM, Time to First Token: [https://www.ibm.com/think/topics/time-to-first-token](https://www.ibm.com/think/topics/time-to-first-token) - Databricks, LLM Inference Performance Engineering, Best Practices: [https://www.databricks.com/blog/llm-inference-performance-engineering-best-practices](https://www.databricks.com/blog/llm-inference-performance-engineering-best-practices) - NVIDIA, Metrics, NIM LLMs Benchmarking: [https://docs.nvidia.com/nim/benchmarking/llm/latest/metrics.html](https://docs.nvidia.com/nim/benchmarking/llm/latest/metrics.html) - BentoML, LLM Inference Handbook, Prefix Caching: [https://bentoml.com/llm/inference-optimization/prefix-caching](https://bentoml.com/llm/inference-optimization/prefix-caching) - DigitalOcean, How KV Caching Slashes LLM Inference Costs at Scale: [https://www.digitalocean.com/community/conceptual-articles/how-kv-caching-slashes-llm-inference-costs-at-scale](https://www.digitalocean.com/community/conceptual-articles/how-kv-caching-slashes-llm-inference-costs-at-scale) - Auditing Prompt Caching in LLM APIs (Stanford, arXiv): [https://arxiv.org/pdf/2502.07776](https://arxiv.org/pdf/2502.07776) - PrefixWall: Mitigating Prefix Caching Side Channels in Shared LLM Systems (arXiv): [https://arxiv.org/html/2603.10726v2](https://arxiv.org/html/2603.10726v2) - InputSnatch: Stealing Input in LLM Services via Timing Side-Channel Attacks (arXiv): [https://arxiv.org/abs/2411.18191](https://arxiv.org/abs/2411.18191) - Akamai, State of the Internet, Financial Services Security Trends 2026: [https://www.akamai.com/security](https://www.akamai.com/security) [](https://insiconcyber.com/blog/ai-inference-metrics-leaders-anz) #### [Tokens, Latency and Trust: The AI Metrics Leaders in Australia and New Zealand Need to Understand](https://insiconcyber.com/blog/ai-inference-metrics-leaders-anz) [Insicon Cyber](https://insiconcyber.com/blog/author/insicon-cyber): 7/07/26 4:23 PM AI SECURITY AND GOVERNANCE You don't need to be an AI engineer to run a good vendor review. But when a vendor pitch is built entirely on numbers... [Read More](https://insiconcyber.com/blog/ai-inference-metrics-leaders-anz) [](https://insiconcyber.com/blog/five-eyes-ai-call-to-action-eu-ai-act-anz-governance) #### [Five Eyes Just Issued an AI Warning. The EU Already Made It Law. Where Does That Leave Australia & New Zealand?](https://insiconcyber.com/blog/five-eyes-ai-call-to-action-eu-ai-act-anz-governance) [Matt Miller](https://insiconcyber.com/blog/author/matt-miller): 23/06/26 10:41 AM Yesterday, the heads of the cyber security agencies of Australia, New Zealand, the United States, the United Kingdom, and Canada signed a joint... [Cyber Security](https://insiconcyber.com/blog/tag/cyber-security) [AI](https://insiconcyber.com/blog/tag/ai) [ISO 42001](https://insiconcyber.com/blog/tag/iso-42001) [Read More](https://insiconcyber.com/blog/five-eyes-ai-call-to-action-eu-ai-act-anz-governance) [](https://insiconcyber.com/blog/oaic-acaps-2026-ai-privacy-shadow-ai-australia-new-zealand) #### [Australians Don't Trust AI Companies. The OAIC's 2026 Privacy Survey Explains Why.](https://insiconcyber.com/blog/oaic-acaps-2026-ai-privacy-shadow-ai-australia-new-zealand) [Insicon Cyber](https://insiconcyber.com/blog/author/insicon-cyber): 16/06/26 5:55 PM Four percent. That is the share of Australians who trust AI companies with their personal information. Not four in ten. Four in one hundred. The... [Read More](https://insiconcyber.com/blog/oaic-acaps-2026-ai-privacy-shadow-ai-australia-new-zealand) [](https://insiconcyber.com/blog/asic-frontier-ai-letter) 1 min read #### [ASIC Has Drawn the Line on Frontier AI. Australian and New Zealand Boards Now Have a Reading List.](https://insiconcyber.com/blog/asic-frontier-ai-letter) [Insicon Cyber](https://insiconcyber.com/blog/author/insicon-cyber) : 11/05/26 10:16 AM On 8 May 2026, ASIC Commissioner Simone Constant issued an open letter to AFS licensees and market participants. It runs to four pages. It is not a... [Cyber Security](https://insiconcyber.com/blog/tag/cyber-security) [ISO 27001](https://insiconcyber.com/blog/tag/iso-27001) [Governance](https://insiconcyber.com/blog/tag/governance) [Essential Eight](https://insiconcyber.com/blog/tag/essential-eight) [APRA](https://insiconcyber.com/blog/tag/apra) [AI](https://insiconcyber.com/blog/tag/ai) [Managed Security Services](https://insiconcyber.com/blog/tag/managed-security-services) [Insicon Cyber](https://insiconcyber.com/blog/tag/insicon-cyber) [ISO 42001](https://insiconcyber.com/blog/tag/iso-42001) [Read More](https://insiconcyber.com/blog/asic-frontier-ai-letter) [](https://insiconcyber.com/blog/apra-cps-230) #### [APRA CPS 230: What You Need to Know](https://insiconcyber.com/blog/apra-cps-230) [Insicon Cyber](https://insiconcyber.com/blog/author/insicon-cyber) : 24/07/24 1:37 PM The Australian Prudential Regulation Authority (APRA) has introduced a new prudential standard, CPS 230, focusing on operational risk management.... [Cyber Security](https://insiconcyber.com/blog/tag/cyber-security) [Governance](https://insiconcyber.com/blog/tag/governance) [Read More](https://insiconcyber.com/blog/apra-cps-230) #### [Five Eyes Just Issued an AI Warning. The EU Already Made It Law. Where Does That Leave Australia & New Zealand?](https://insiconcyber.com/blog/five-eyes-ai-call-to-action-eu-ai-act-anz-governance) [Matt Miller](https://insiconcyber.com/blog/author/matt-miller) : 23/06/26 10:41 AM --- ## About This Content **Source:** [Tokens, Latency and Trust: The AI Metrics Leaders in Australia and New Zealand Need to Understand](https://insiconcyber.com/blog/ai-inference-metrics-leaders-anz) **Author:** Insicon Cyber **Published:** July 7, 2026 *This content is provided for informational purposes. Please visit the original source for the most up-to-date information.*