https://cloud.google.com/blog/products/containers-kubernetes/unifying-real-time-and-async-inference-with-gke-inference-gateway/
Unifying real-time and async inference with GKE Inference Gateway | Google Cloud Blog
GKE Inference Gateway treats accelerator capacity as a fluid resource pool that serves workloads that need deterministic latency and high throughput.
real timeasync inference