GPU-Live/template/dev/istio-1.23.1/samples/open-telemetry/tracing

Open Telemetry Tracing

This sample demonstrates the support for the OpenTelemetry tracing provider with the Telemetry API.

Start otel-collector service

First, deploy the otel-collector backend with simple configuration.

kubectl -n <namespace> apply -f ../otel.yaml

In this example, we use observability as the namespace to deploy the otel-collector backend:

kubectl create namespace observability
kubectl -n observability apply -f ../otel.yaml

The otel-collector will create a grpc receiver on port 4317, and later the sidecars will report trace information to this grpc port. You can find more details from here.

Below is the configuration:

receivers:
  otlp:
    protocols:
      grpc:
      http:
processors:
  batch:
exporters:
  logging:
    loglevel: debug
service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [logging]

In this example, Jaeger is the exporter for gathering the traces. Assuming you have already deployed Jaeger as your tracing system with this installation, you are good to go to the next steps. If you already have your own Jaeger deployed, you may need to modify the otel collector config. The configmap name is opentelemetry-collector-conf in the namespace you deployed the otel collector, and the related config is defined as:

exporters:
  jaeger:
    endpoint: jaeger-collector.istio-system.svc.cluster.local:14250
    tls:
      insecure: true
    sending_queue:
      enabled: true
    retry_on_failure:
      enabled: true
service:
  pipelines:
    traces:
      exporters:
      - jaeger

You need to modify the jaeger exporter endpoint with the one you deployed, in this case it's jaeger-collector.istio-system.svc.cluster.local:14250.

If you have not deployed the Jaeger service, you can follow this installation to install the service.

You may also choose any existing tracing system if you have, and you should change the exporter settings in the configmap mentioned above.

You may also choose to use your own otel collector if you have, and the key part is to have the otlp grpc protocol receiver to receive the traces. One important thing is to make sure your otel collector service's grpc port starts with grpc- prefix, which is like:

spec:
  ports:
    - name: grpc-otlp
      port: 4317
      protocol: TCP
      targetPort: 4317

Otherwise the traces may not be reported.

Update mesh config

Install or update Istio with the demo profile to make sure you have the OpenTelemetry tracing provider enabled:

istioctl install --set profile=demo -y

Or ensure you have the following additional mesh config set in your Istio:

mesh: |-
  extensionProviders:
  - name: otel-tracing
    opentelemetry:
      port: 4317
      service: opentelemetry-collector.observability.svc.cluster.local  

Make sure the service name matches the one you deployed if you select a different namespace.

Apply the Telemetry resource to report traces

Next, add a Telemetry resource that tells Istio to send trace records to the OpenTelemetry collector.

kubectl -n <namespace> apply -f ./telemetry.yaml

In this example, we deploy it to the default namespace, which is where the sample apps from the getting started are also deployed.

kubectl apply -f ./telemetry.yaml

The core config is:

tracing:
- providers:
  - name: otel-tracing
  randomSamplingPercentage: 0

As you see, the randomSamplingPercentage is 0, which means the tracing is still not enabled because of 0 sampling percentage. The tracing can be opt-on by increasing the randomSamplingPercentage value to 1-100. The Telemetry resource can also be manipulated in workload/namespace/global levels, you can check here for more config examples.

Check tracing results

If you have followed this getting started steps, you have the sample bookinfo applications installed. Try to make some requests to the productpage to generate some traces.

Then open up the Jaeger dashboard with:

istioctl dashboard jaeger

You will see the requests' trace records.

Cleanup

kubectl -n observability delete -f ./telemetry.yaml
kubectl -n observability delete -f ../otel.yaml