This post will cover a demo working setup of a service mesh architecture using Envoy using a demo application. In this service mesh architecture, we will be using Envoy proxy for both control and data plane. The setup is deployed in a Kubernetes cluster using Amazon EKS.
\nWe will be deploying an echo-grpc test application provided by Google in their article related to gRPC load balancing and was used as a reference to test the service mesh setup with Envoy. The article covers setting up Envoy as an edge proxy only.\nThis is a simple gRPC application that exposes a unary method that takes a string in the content request field and responds with the content unaltered.\nRepo: grpc-gke-nlb-tutorial
\nkubectl create namespace envoygo get github.com/fullstorydev/grpcurlConfiguration of envoy for the sidecar deployment:
\nenvoy-echo.yaml:
\napiVersion: v1\nkind: ConfigMap\nmetadata:\n name: envoy-echo\ndata:\n envoy.yaml: |\n static_resources:\n listeners:\n - address:\n socket_address:\n address: 0.0.0.0\n port_value: 8786\n filter_chains:\n - filters:\n - name: envoy.http_connection_manager\n config:\n access_log:\n - name: envoy.file_access_log\n config:\n path: "/dev/stdout"\n codec_type: AUTO\n stat_prefix: ingress_https\n route_config:\n name: local_route\n virtual_hosts:\n - name: https\n domains:\n - "*"\n routes:\n - match:\n prefix: "/api.Echo/"\n route:\n cluster: echo-grpc\n http_filters:\n - name: envoy.health_check\n config:\n pass_through_mode: false\n headers:\n - name: ":path"\n exact_match: "/healthz"\n - name: "x-envoy-livenessprobe"\n exact_match: "healthz"\n - name: envoy.router\n config: {}\n clusters:\n - name: echo-grpc\n connect_timeout: 0.5s\n type: STATIC\n lb_policy: ROUND_ROBIN\n http2_protocol_options: {}\n load_assignment:\n cluster_name: echo-grpc\n endpoints:\n - lb_endpoints:\n - endpoint:\n address:\n socket_address:\n address: "127.0.0.1"\n port_value: 8081\n health_checks:\n timeout: 1s\n interval: 10s\n unhealthy_threshold: 2\n healthy_threshold: 2\n grpc_health_check: {}\n admin:\n access_log_path: "/dev/stdout"\n address:\n socket_address:\n address: 127.0.0.1\n port_value: 8090A couple things to note here.
\n*, allowing all domains to pass-through.Run:\nkubectl apply -f envoy-echo.yaml -n envoy
Deployment of echo-grpc application with 3 replicas. The config contains two containers, one for application and another being the Envoy image.
\napiVersion: apps/v1\nkind: Deployment\nmetadata:\n name: echo-grpc\nspec:\n replicas: 3\n selector:\n matchLabels:\n app: echo-grpc\n template:\n metadata:\n labels:\n app: echo-grpc\n spec:\n containers:\n - name: echo-grpc\n image: <hub-username>/echo-grpc\n imagePullPolicy: Always\n resources: {}\n env:\n - name: "PORT"\n value: "8081"\n ports:\n - containerPort: 8081\n readinessProbe:\n exec:\n command: ["/bin/grpc_health_probe", "-addr=:8081"]\n initialDelaySeconds: 1\n livenessProbe:\n exec:\n command: ["/bin/grpc_health_probe", "-addr=:8081"]\n initialDelaySeconds: 1\n - name: envoy\n image: envoyproxy/envoy:v1.9.1\n resources: {}\n ports:\n - name: https\n containerPort: 443\n volumeMounts:\n - name: config\n mountPath: /etc/envoy\n volumes:\n - name: config\n configMap:\n name: envoy-echoHere, echo-grpc is test application and envoy is being deployed in the same pod. Config volumes are mounted so that the envoy can read the configmaps.
\nRun:\nkubectl apply -f echo-deployment.yaml -n envoy
We are using headless service for echo-grpc. Using service as headless will expose the Pods IP to the DNS server of kubernetes which will be used by Envoy to do service discovery for the pods.
\necho-service.yaml
\napiVersion: v1\nkind: Service\nmetadata:\n name: echo-grpc\nspec:\n type: ClusterIP\n clusterIP: None\n selector:\n app: echo-grpc\n ports:\n - name: http2-echo\n protocol: TCP\n port: 8786\n - name: http2-service\n protocol: TCP\n port: 8081In the above config file, we are exposing two ports, one for envoy sidecar (this is the same port we mentioned in the config map of sidecar envoy) and one for the service itself.
\nRun:\nkubectl apply -f echo-service.yaml -n envoy
Creating a service of type LoadBalancer so that client can access the backend service.
\nenvoy-service.yaml:
\napiVersion: v1\nkind: Service\nmetadata:\n name: envoy\nspec:\n type: LoadBalancer\n selector:\n app: envoy\n ports:\n - name: https\n protocol: TCP\n port: 443\n targetPort: 443Run:\nkubectl apply -f envoy-service.yaml -n envoy
Since we are deploying front envoy LoadBalancer on port 443, we have to create a self-signed certificate to make it terminate SSL/TLS connection.
\nkubectl describe svc/envoy -n envoynslookup <your load balancer aadess>openssl req -x509 -nodes -newkey rsa:2048 -days 365 -keyout privkey.pem -out cert.pem -subj \"/CN=<ip-address>\"kubectl create secret tls envoy-certs --key privkey.pem --cert cert.pem --dry-run -o yamlConfiguration for the edge Envoy:
\nenvoy-configmap.yaml
\napiVersion: v1\nkind: ConfigMap\nmetadata:\n name: envoy-conf\ndata:\n envoy.yaml: |\n static_resources:\n listeners:\n - address:\n socket_address:\n address: 0.0.0.0\n port_value: 443\n filter_chains:\n - filters:\n - name: envoy.http_connection_manager\n config:\n access_log:\n - name: envoy.file_access_log\n config:\n path: "/dev/stdout"\n codec_type: AUTO\n stat_prefix: ingress_https\n route_config:\n name: local_route\n virtual_hosts:\n - name: https\n domains:\n - "*"\n routes:\n - match:\n prefix: "/api.Echo/"\n route:\n cluster: echo-grpc\n http_filters:\n - name: envoy.health_check\n config:\n pass_through_mode: false\n headers:\n - name: ":path"\n exact_match: "/healthz"\n - name: "x-envoy-livenessprobe"\n exact_match: "healthz"\n - name: envoy.router\n config: {}\n tls_context:\n common_tls_context:\n tls_certificates:\n - certificate_chain:\n filename: "/etc/ssl/envoy/tls.crt"\n private_key:\n filename: "/etc/ssl/envoy/tls.key"\n clusters:\n - name: echo-grpc\n connect_timeout: 0.5s\n type: STRICT_DNS\n lb_policy: ROUND_ROBIN\n http2_protocol_options: {}\n load_assignment:\n cluster_name: echo-grpc\n endpoints:\n - lb_endpoints:\n - endpoint:\n address:\n socket_address:\n address: echo-grpc.envoy.svc.cluster.local\n port_value: 8786\n health_checks:\n timeout: 1s\n interval: 10s\n unhealthy_threshold: 2\n healthy_threshold: 2\n grpc_health_check: {}\n admin:\n access_log_path: "/dev/stdout"\n address:\n socket_address:\n address: 127.0.0.1\n port_value: 8090Since we will be offloading HTTPS, we are using port_value of 443. Most of the configurations are same as of sidecar envoy except for three things:
\nRun:\nkubectl apply -f envoy-configmap.yaml -n envoy
envoy-deployment.yaml
\napiVersion: apps/v1\nkind: Deployment\nmetadata:\n name: envoy\nspec:\n replicas: 2\n selector:\n matchLabels:\n app: envoy\n template:\n metadata:\n labels:\n app: envoy\n spec:\n containers:\n - name: envoy\n image: envoyproxy/envoy:v1.9.1\n resources: {}\n ports:\n - name: https\n containerPort: 443\n volumeMounts:\n - name: config\n mountPath: /etc/envoy\n - name: certs\n mountPath: /etc/ssl/envoy\n readinessProbe:\n httpGet:\n scheme: HTTPS\n path: /healthz\n httpHeaders:\n - name: x-envoy-livenessprobe\n value: healthz\n port: 443\n initialDelaySeconds: 3\n livenessProbe:\n httpGet:\n scheme: HTTPS\n path: /healthz\n httpHeaders:\n - name: x-envoy-livenessprobe\n value: healthz\n port: 443\n initialDelaySeconds: 10\n volumes:\n - name: config\n configMap:\n name: envoy-conf\n - name: certs\n secret:\n secretName: envoy-certsRun:\nkubectl apply -f envoy-deployment.yaml -n envoy
Proto file for the echo-grpc service:
\nccho.proto:
\nsyntax = "proto3";\n\npackage api;\n\nservice Echo {\n rpc Echo (EchoRequest) returns (EchoResponse) {}\n}\n\nmessage EchoRequest {\n string content = 1;\n}\n\nmessage EchoResponse {\n string content = 1;\n}Run the following command to call the server:\ngrpcurl -d '{\"content\": \"echo\"}' -proto echo.proto -insecure -v <load_balancer_or_external_ip>:443 api.Echo/Echo
The output will be similar to something like this:
\nResolved method descriptor:\nrpc Echo ( .api.EchoRequest ) returns ( .api.EchoResponse );\n\nRequest metadata to send:\n(empty)\n\nResponse headers received:\ncontent-type: application/grpc\ndate: Wed, 27 Feb 2019 04:40:19 GMT\nhostname: echo-grpc-5c4f59c578-wcsvr\nserver: envoy\nx-envoy-upstream-service-time: 0\n\nResponse contents:\n{\n "content": "echo"\n}\n\nResponse trailers received:\n(empty)\nSent 1 request and received 1 responseRun the above command multiple times and check the value of the hostname field every time which will contain the pod name of one of the 3 pods deployed.
\n