146 lines
4.7 KiB
Markdown
146 lines
4.7 KiB
Markdown
# Open Telemetry Tracing
|
|
|
|
This sample demonstrates the support for the OpenTelemetry tracing provider with the Telemetry API.
|
|
|
|
## Start otel-collector service
|
|
|
|
First, deploy the `otel-collector` backend with simple configuration.
|
|
|
|
```bash
|
|
kubectl -n <namespace> apply -f ../otel.yaml
|
|
```
|
|
|
|
In this example, we use `observability` as the namespace to deploy the `otel-collector` backend:
|
|
|
|
```bash
|
|
kubectl create namespace observability
|
|
kubectl -n observability apply -f ../otel.yaml
|
|
```
|
|
|
|
The otel-collector will create a grpc receiver on port `4317`, and later the sidecars will report trace information to this grpc port. You can find more details from [here](https://github.com/open-telemetry/opentelemetry-collector).
|
|
|
|
Below is the configuration:
|
|
|
|
```yaml
|
|
receivers:
|
|
otlp:
|
|
protocols:
|
|
grpc:
|
|
http:
|
|
processors:
|
|
batch:
|
|
exporters:
|
|
logging:
|
|
loglevel: debug
|
|
service:
|
|
pipelines:
|
|
logs:
|
|
receivers: [otlp]
|
|
processors: [batch]
|
|
exporters: [logging]
|
|
```
|
|
|
|
In this example, `Jaeger` is the exporter for gathering the traces. Assuming you have already deployed Jaeger as your tracing system with [this](https://istio.io/latest/docs/ops/integrations/jaeger/) installation, you are good to go to the next steps. If you already have your own `Jaeger` deployed, you may need to modify the otel collector config. The configmap name is `opentelemetry-collector-conf` in the namespace you deployed the otel collector, and the related config is defined as:
|
|
|
|
```yaml
|
|
exporters:
|
|
jaeger:
|
|
endpoint: jaeger-collector.istio-system.svc.cluster.local:14250
|
|
tls:
|
|
insecure: true
|
|
sending_queue:
|
|
enabled: true
|
|
retry_on_failure:
|
|
enabled: true
|
|
service:
|
|
pipelines:
|
|
traces:
|
|
exporters:
|
|
- jaeger
|
|
```
|
|
|
|
You need to modify the jaeger exporter endpoint with the one you deployed, in this case it's `jaeger-collector.istio-system.svc.cluster.local:14250`.
|
|
|
|
If you have not deployed the `Jaeger` service, you can follow [this](https://istio.io/latest/docs/ops/integrations/jaeger/) installation to install the service.
|
|
|
|
You may also choose any existing tracing system if you have, and you should change the exporter settings in the configmap mentioned above.
|
|
|
|
You may also choose to use your own otel collector if you have, and the key part is to have the `otlp` grpc protocol receiver to receive the traces. One important thing is to make sure your otel collector service's grpc port starts with `grpc-` prefix, which is like:
|
|
|
|
```yaml
|
|
spec:
|
|
ports:
|
|
- name: grpc-otlp
|
|
port: 4317
|
|
protocol: TCP
|
|
targetPort: 4317
|
|
```
|
|
|
|
Otherwise the traces may not be reported.
|
|
|
|
## Update mesh config
|
|
|
|
Install or update Istio with the `demo` profile to make sure you have the OpenTelemetry tracing provider enabled:
|
|
|
|
```bash
|
|
istioctl install --set profile=demo -y
|
|
```
|
|
|
|
Or ensure you have the following additional mesh config set in your Istio:
|
|
|
|
```yaml
|
|
mesh: |-
|
|
extensionProviders:
|
|
- name: otel-tracing
|
|
opentelemetry:
|
|
port: 4317
|
|
service: opentelemetry-collector.observability.svc.cluster.local
|
|
```
|
|
|
|
Make sure the service name matches the one you deployed if you select a different namespace.
|
|
|
|
## Apply the Telemetry resource to report traces
|
|
|
|
Next, add a Telemetry resource that tells Istio to send trace records to the OpenTelemetry collector.
|
|
|
|
```bash
|
|
kubectl -n <namespace> apply -f ./telemetry.yaml
|
|
```
|
|
|
|
In this example, we deploy it to the default namespace, which is where the sample apps
|
|
from the [getting started](https://istio.io/latest/docs/setup/getting-started) are also deployed.
|
|
|
|
```bash
|
|
kubectl apply -f ./telemetry.yaml
|
|
```
|
|
|
|
The core config is:
|
|
|
|
```yaml
|
|
tracing:
|
|
- providers:
|
|
- name: otel-tracing
|
|
randomSamplingPercentage: 0
|
|
```
|
|
|
|
As you see, the `randomSamplingPercentage` is 0, which means the tracing is still not enabled because of `0` sampling percentage. The tracing can be opt-on by increasing the `randomSamplingPercentage` value to `1-100`. The `Telemetry` resource can also be manipulated in workload/namespace/global levels, you can check [here](https://istio.io/latest/docs/reference/config/telemetry/) for more config examples.
|
|
|
|
## Check tracing results
|
|
|
|
If you have followed [this](https://istio.io/latest/docs/setup/getting-started/) getting started steps, you have the sample bookinfo applications installed. Try to make some requests to the productpage to generate some traces.
|
|
|
|
Then open up the `Jaeger` dashboard with:
|
|
|
|
```bash
|
|
istioctl dashboard jaeger
|
|
```
|
|
|
|
You will see the requests' trace records.
|
|
|
|
## Cleanup
|
|
|
|
```bash
|
|
kubectl -n observability delete -f ./telemetry.yaml
|
|
kubectl -n observability delete -f ../otel.yaml
|
|
```
|