Higress is a cloud-native API gateway built on Istio and Envoy, extended through WebAssembly (Wasm) plugins written in Go, Rust, or JavaScript. It began at Alibaba to stop long connections dropping during gateway reloads and to improve gRPC and Dubbo load balancing, and it is now a CNCF Sandbox project. Configuration changes take effect in milliseconds without the traffic jitter of an Nginx reload, which matters for long-lived connections such as streaming LLM responses.
As an AI gateway, Higress connects to mainstream model providers through one unified protocol and adds AI observability, multi-model load balancing, model-level fallback, token rate limiting, and exact and semantic caching of responses. It is a conformant Gateway API Inference Extension implementation, so it supports inference-aware routing on Kubernetes.
Higress also hosts Model Context Protocol (MCP) servers through its plugin mechanism, adding unified authentication, fine-grained rate limiting, audit logs of every tool call, and observability. A companion openapi-to-mcp tool converts OpenAPI specifications into remote MCP servers, and a hosted MCP server platform lets you try Higress-hosted servers.
Beyond AI traffic, the same gateway works as a Kubernetes ingress controller compatible with many ingress-nginx annotations, a Gateway API implementation, a microservice gateway that discovers services from Nacos, ZooKeeper, Consul, or Eureka, and a security gateway with WAF and key, HMAC, JWT, basic, and OIDC authentication. It ships a web console and runs inside or outside Kubernetes.
Features
- Unified model protocol: one calling convention for mainstream model providers, with multi-model load balancing and fallback
- Token governance: per-consumer token tracking and rate limiting
- Response caching: exact and semantic caching of model results to cut tokens and latency
- MCP server hosting: authentication, rate limiting, audit logs, and observability for hosted MCP servers
- OpenAPI to MCP: convert OpenAPI specifications into remote MCP servers
- Inference-aware routing: conformant Gateway API Inference Extension implementation for Kubernetes
- Wasm plugins: sandboxed plugins in Go, Rust, or JavaScript with independent versioning and lossless hot updates
- Streaming processing: full streaming of request and response bodies, including SSE
- Ingress and Gateway API: a Kubernetes ingress controller compatible with ingress-nginx annotations and a conformant Gateway API implementation
- Microservice discovery: services from Nacos, ZooKeeper, Consul, and Eureka, with Dubbo and Sentinel integration
- Security gateway: WAF, IP and cookie CC protection, and key, HMAC, JWT, basic, and OIDC authentication
- Automatic certificates: Let's Encrypt issuance and renewal
