<div id="readability-page-1" class="page"><article><div><div><div><a rel="noopener follow" href="/@reis.fabio?source=post_page-----624958d38d4d--------------------------------"><div aria-hidden="false"><p><img alt="Fabio Reis" src="https://miro.medium.com/v2/resize:fill:88:88/2*8BWpQml6js3ZSCPrVnfbTQ.jpeg" width="44" height="44" loading="lazy" data-testid="authorPhoto"></p></div></a><a href="https://medium.com/zigbang?source=post_page-----624958d38d4d--------------------------------" rel="noopener follow"><div aria-hidden="false"><p><img alt="직방 기술 블로그" src="https://miro.medium.com/v2/resize:fill:48:48/1*LbSjHv9t1FgET6vfrG76Ug.png" width="24" height="24" loading="lazy" data-testid="publicationPhoto"></p></div></a></div><p id="b2c6">When we think about monitoring, the first thing that comes to our minds are logs and metrics. We oftentimes underestimate traces because of the extra work we need to set up traces, whereas metrics and logs on the other hand are often automatically provided when using any cloud platform.</p><p id="572c">The combination of metrics, logs, traces, and alarms is known as observability. Different from monitoring, which only tells us that there is something wrong within our system, for example with metrics, the goal of observability is to correlate the data collected trying to help us identify what, where, and why something is wrong.</p><p id="204f">In this blog post, we will show you how to collect traces by setting up a simple observability stack with node js and typescript. Although this is not the focus of this post, Grafana also works perfectly integrating with external metrics and logs you must already have, and we encourage you to take a look at their other integrations.</p><h2 id="6701">Prerequisites</h2><p id="b0ef">We must install Grafana and Loki so we can add telemetry to our test application. For that reason, you should have the following tools installed. Please follow their respective installation guide.</p><ul><li id="b8c4"><a href="https://nodejs.org/en" rel="noopener ugc nofollow" target="_blank">NodeJs</a> ≥ 18</li><li id="f2c8"><a href="https://www.docker.com/products/docker-desktop/" rel="noopener ugc nofollow" target="_blank">Docker for Desktop</a> <br>(Kubernetes is included but you must enable it)</li><li id="1adc"><a href="https://kubernetes.io/docs/tasks/tools/install-kubectl-macos/" rel="noopener ugc nofollow" target="_blank">kubectl</a></li><li id="e098"><a href="https://helm.sh/docs/intro/install/" rel="noopener ugc nofollow" target="_blank">Helm</a></li></ul></div><div><h2 id="384c">Setting Helm Repositories and Containers</h2><p id="93b0">Helm is a package manager that will help us set up our Kubernetes environment with minimal effort. After installing Helm, you can add Grafana and Tempo repositories with the following commands.</p><pre><span id="d406">#adding Grafana charts<br>helm repo add grafana https://grafana.github.io/helm-charts<br>helm repo update</span></pre><h2 id="9942">Installing Tempo</h2><p id="21ef">For this example, we will use the standard configuration of tempo, but if you wish to customize any values please check tempo helm <a href="https://github.com/grafana/helm-charts/blob/main/charts/tempo-distributed/values.yaml" rel="noopener ugc nofollow" target="_blank">documentation</a> and <a href="https://github.com/grafana/tempo/tree/main/example/helm" rel="noopener ugc nofollow" target="_blank">examples</a>. You can easily customize your distribution by creating a YAML file and applying it with Helm.</p><pre><span id="ccc0">helm upgrade --install tempo grafana/tempo</span></pre><p id="1786">Tempo supports many receivers and protocols and for the purpose of this tutorial, we will use OTLP receiver with HTTP protocol (port 4318). If you are interested in other receivers take a look o <a href="https://grafana.com/docs/tempo/latest/configuration/#distributor" rel="noopener ugc nofollow" target="_blank">Tempo's documentation</a>.</p><h2 id="4ca3">Installing Grafana</h2><p id="5a11">In order to install Grafana we need to configure our Helm template to add our tempo container as a data source, so we can visualize our traces on Grafana. For more information on Grafana customization please check Helm <a href="https://github.com/grafana/helm-charts/blob/main/charts/grafana/values.yaml" rel="noopener ugc nofollow" target="_blank">documentation</a>.</p><ul><li id="7469">Add the following content to a file called <strong>grafana-helm.yaml</strong></li></ul><pre><span id="65a6">env:<br>  GF_AUTH_ANONYMOUS_ENABLED: true<br>  GF_AUTH_ANONYMOUS_ORG_ROLE: 'Admin'<br>  GF_AUTH_DISABLE_LOGIN_FORM: true<p>datasources:<br>  datasources.yaml:<br>    apiVersion: 1</p><p>    datasources:<br>      - name: Tempo<br>        type: tempo<br>        access: proxy<br>        orgId: 1<br>        url: http://tempo:3100<br>        basicAuth: false<br>        isDefault: true<br>        version: 1<br>        editable: false<br>        apiVersion: 1<br>        uid: tempo</p></span></pre><ul><li id="9629">Install Grafana with the YAML file we just created</li></ul><pre><span id="b73f">helm upgrade -f grafana-helm.yaml --install grafana grafana/grafana</span></pre><h2 id="2aa2">Configure Nginx-controller</h2><p id="5147">We will need to access both our Grafana and our tempo services so therefore we need to expose the service with Nginx-controller and k8 ingress, so we can use it in our test application.</p><ol><li id="9ca9">Install Nginx-controller</li></ol><pre><span id="445c">helm upgrade --install ingress-nginx ingress-nginx \<br>  --repo https://kubernetes.github.io/ingress-nginx \<br>  --namespace ingress-nginx --create-namespace</span></pre><p id="f900">2. Creating Ingress for the services</p><ul><li id="8fb6">Save the following content to a file called <strong>ingress.yaml</strong></li></ul><pre><span id="63cd">apiVersion: networking.k8s.io/v1<br>kind: Ingress<br>metadata:<br>  name: grafana<br>  namespace: default<br>  annotations:<br>    ingress.kubernetes.io/ssl-redirect: "false"<br>spec:<br>  ingressClassName: nginx<br>  rules:<br>    - http:<br>        paths:<br>          - pathType: Prefix<br>            backend:<br>              service:<br>                name: grafana<br>                port:<br>                  number: 80<br>            path: /<br>    - http:<br>        paths:<br>          - pathType: Prefix<br>            backend:<br>              service:<br>                name: tempo<br>                port:<br>                  number: 4318<br>            path: /v1/traces</span></pre><ul><li id="d053">Create ingress with kubectl</li></ul><pre><span id="ea9a">kubectl create -f ingress.yaml</span></pre><h2 id="d82f">Testing Grafana</h2><p id="1f55">After completing the setup you will be able to access both services in the following addresses. If you wish to use a host instead you can change your ingress file as needed. Check K8 <a href="https://kubernetes.io/docs/concepts/services-networking/ingress/" rel="noopener ugc nofollow" target="_blank">documentation</a> for more details.</p><p id="829d"><strong>Grafana</strong>: <a href="http://localhost/" rel="noopener ugc nofollow" target="_blank">http://localhost</a></p><figure><figcaption>Sample of Grafana dashboard</figcaption></figure><p id="eaea">You can also check all the Kubernetes resources we created with the following command:</p><pre><span id="5b6e">#services and pods<br>kubectl get all -A<p>#for our custom ingress<br>kubectl get ingress</p></span></pre><h2 id="cbaf">Other Observaibility Integrations</h2><p id="27d0">This is not the main focus of this tutorial, but if you wish you can also use metrics, logs, and alerts with Grafana. In order to do that you can use Prometheus for metrics, and Loki for logs for example. The installation process can be done in the same way we did with Helm. For more information check <a href="https://github.com/prometheus-community/helm-charts/tree/main/charts/prometheus" rel="noopener ugc nofollow" target="_blank">Prometheus Helm</a> documentation and <a href="https://github.com/grafana/helm-charts/tree/main/charts/loki-stack" rel="noopener ugc nofollow" target="_blank">Loki Helm</a> documentation.</p></div><div><h2 id="57c4">Setting Up OpenTelemetry with NodsJs</h2><p id="35c7">We want to add traces to any NodeJs API, so you can actually use your own project if you wish. If you don't already have a project you can use any open-source sample available on GitHub, like for example the cats API on NestJs repository:</p><p id="b0b3"><a href="https://github.com/nestjs/nest/tree/master/sample/10-fastify" rel="noopener ugc nofollow" target="_blank">https://github.com/nestjs/nest/tree/master/sample/10-fastify</a></p><h2 id="e06a">Adding OpenTelemetry Dependencies</h2><p id="0874">Add the following dependencies to your project:</p><pre><span id="66b7">npm install @opentelemetry/api @opentelemetry/auto-instrumentations-node @opentelemetry/exporter-trace-otlp-http @opentelemetry/sdk-node @opentelemetry/sdk-trace-node</span></pre><h2 id="6707">Adding Tracer and Auto-instrumentation</h2><p id="2b42">On your src folder add a tracer.ts file with the following content:</p><pre><span id="f139">import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';<br>import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';<br>import { NodeSDK } from '@opentelemetry/sdk-node';<br>import {<br>  BasicTracerProvider,<br>  ConsoleSpanExporter,<br>  SimpleSpanProcessor,<br>} from '@opentelemetry/sdk-trace-node';<br>import { Resource } from '@opentelemetry/resources';<br>import { SemanticResourceAttributes } from '@opentelemetry/semantic-conventions';<p>class Tracer {<br>  private sdk: NodeSDK | null = null;</p><p>  // url is optional and can be omitted - default is http://localhost:4318/v1/traces<br>  private exporter = new OTLPTraceExporter({ url: "http://localhost/v1/traces"});</p><p>  private provider = new BasicTracerProvider({<br>    resource: new Resource({<br>      [SemanticResourceAttributes.SERVICE_NAME]: 'DEMO-APP',<br>    }),<br>  });</p><p>  public init() {<br>    try {<br>      // export spans to console (useful for debugging)<br>      this.provider.addSpanProcessor(<br>        new SimpleSpanProcessor(new ConsoleSpanExporter()),<br>      );</p><p>      // export spans to opentelemetry collector<br>      this.provider.addSpanProcessor(new SimpleSpanProcessor(this.exporter));<br>      this.provider.register();</p><p>      this.sdk = new NodeSDK({<br>        traceExporter: this.exporter,<br>        instrumentations: [<br>          getNodeAutoInstrumentations({<br>            // Lets disable fs for now, otherwise we cannot see the traces we want,<br>            // You can disable or enable instrumentation as needed<br>            '@opentelemetry/instrumentation-fs': { enabled: false },<br>          }),<br>        ],<br>      });</p><p>      this.sdk.start();</p><p>      console.info('The tracer has been initialized');<br>    } catch (e) {<br>      console.error('Failed to initialize the tracer', e);<br>    }<br>  }<br>}</p><p>export default new Tracer();</p></span></pre><h2 id="b66a">Initialize our Tracer</h2><p id="1289">Make sure you initialize your tracer before your server starts, otherwise, auto-instrumentation won't work properly. On your main.ts file (or your entry point class) add the following code.</p><pre><span id="ad2e">import tracer from './tracer'<br>tracer.init()<p>//Other imports<br>...</p><p>async function bootstrap() {<br>...<br>}</p><p>bootstrap()</p></span></pre></div><div><h2 id="a278">Testing our Telemetry</h2><p id="42f2">After running your application let's use some of the APIs so we can generate some traces. For the cat API example, you can run one of the commands below.</p><pre><span id="a12e"># POST cats<br>curl -X POST http://localhost:3000/cats -H "Content-Type: application/json" -d '{<br>    "name":"test",<br>    "age": 18,<br>    "breed": "test"<br>}'<p># GET cats<br>curl  http://localhost:3000/cats</p></span></pre><h2 id="e242">Check Telemetry on Grafana</h2><p id="52b3">After collecting some data, let's check our traces on Grafana. On the <strong>Explore</strong> menu, Tempo will be selected automatically, but if not, select Tempo in the data source drop-down list.</p><p id="6948">On the query options, select search and input the name of the service we configured in our tracer.ts file (DEMO-APP). You will be able to see all the instrumentation collected with auto instrumentation.</p><p id="a602">Note that auto instrumentation collects different information for each library. For example, if you check our HTTP spans you will be able to see many of our request parameters. Now, if we were using MySQL, we would be able to see up to the query executed for that specific trace. For each instrumentation, check their respective documentation specifically.</p><p id="6c62">If you are curious about which libraries are instrumented automatically, check the documentation on the <a href="https://www.npmjs.com/package/@opentelemetry/auto-instrumentations-node" rel="noopener ugc nofollow" target="_blank">@opentelemetry/auto-instrumentations-node npm</a> page.</p><figure><figcaption>Instrumentation sample</figcaption></figure><h2 id="c149">Adding Manual Span</h2><p id="585a">For this tutorial, we've only touched the auto-instrumentation setup, and even though this alone helps us to see the path our users take inside of our system, it does not always provide us with all the information we need.</p><p id="9db8">OpenTelemetry also allows us to define our own custom spans and add them to our traces. This is useful for example when you need to add business information to the telemetry.</p><p id="18f0">You can play around with your own custom spans by adding them to your API code. Follow the <a href="https://github.com/open-telemetry/opentelemetry-js/blob/main/doc/tracing.md#acquiring-a-tracer" rel="noopener ugc nofollow" target="_blank">documentation</a> for more information.</p><h2 id="7079">Extra (Telemetry Propagation)</h2><p id="9ff2">Trace opens a whole lot of possibilities for us. However, we only saw a small part of it in this tutorial. OpenTelemetry also works really well with distributed systems and provides us with tools to integrate APIs together in the same trace. This is extremely useful when working with microservice architecture, where multiple APIs are triggered for a single user action.</p><p id="3548">In order to connect microservices together, you should configure OpenTelemetry to send and receive traces to and from other APIs. You can easily achieve that using custom headers and openTelemetry node SDK.</p><p id="8077">OpenTelemetry provides us with some examples of how to set up our API to communicate with other services. Check the <a href="https://github.com/open-telemetry/opentelemetry-js/blob/6d174e2a0689d9386e1aaf69a3c774ce2a9cbca0/doc/propagation.md" rel="noopener ugc nofollow" target="_blank">documentation</a> on propagation for more details.</p></div></div></article></div>

zigbang - Medium

When we think about monitoring, the first thing that comes to our minds are logs and metrics. We oftentimes underestimate traces because of the extra work we need to set up traces, whereas metrics and logs on the other hand are often automatically provided when using any cloud platform.
The combination of metrics, logs, traces, and alarms is known as observability. Different from monitoring, which only tells us that there is something wrong within our system, for example with metrics, the goal of observability is to correlate the data collected trying to help us identify what, where, and why something is wrong.
In this blog post, we will show you how to collect traces by setting up a simple observability stack with node js and typescript. Although this is not the focus of this post, Grafana also works perfectly integrating with external metrics and logs you must already have, and we encourage you to take a look at their other integrations.
Prerequisites
We must install Grafana and Loki so we can add telemetry to our test application. For that reason, you should have the following tools installed. Please follow their respective installation guide.
NodeJs ≥ 18
Docker for Desktop 
(Kubernetes is included but you must enable it)
kubectl
Helm
Setting Helm Repositories and Containers
Helm is a package manager that will help us set up our Kubernetes environment with minimal effort. After installing Helm, you can add Grafana and Tempo repositories with the following commands.
#adding Grafana charts
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
Installing Tempo
For this example, we will use the standard configuration of tempo, but if you wish to customize any values please check tempo helm documentation and examples. You can easily customize your distribution by creating a YAML file and applying it with Helm.
helm upgrade --install tempo grafana/tempo
Tempo supports many receivers and protocols and for the purpose of this tutorial, we will use OTLP receiver with HTTP protocol (port 4318). If you are interested in other receivers take a look o Tempo's documentation.
Installing Grafana
In order to install Grafana we need to configure our Helm template to add our tempo container as a data source, so we can visualize our traces on Grafana. For more information on Grafana customization please check Helm documentation.
Add the following content to a file called grafana-helm.yaml
env:
  GF_AUTH_ANONYMOUS_ENABLED: true
  GF_AUTH_ANONYMOUS_ORG_ROLE: 'Admin'
  GF_AUTH_DISABLE_LOGIN_FORM: true
datasources:
  datasources.yaml:
    apiVersion: 1
    datasources:
      - name: Tempo
        type: tempo
        access: proxy
        orgId: 1
        url: http://tempo:3100
        basicAuth: false
        isDefault: true
        version: 1
        editable: false
        apiVersion: 1
        uid: tempo
Install Grafana with the YAML file we just created
helm upgrade -f grafana-helm.yaml --install grafana grafana/grafana
Configure Nginx-controller
We will need to access both our Grafana and our tempo services so therefore we need to expose the service with Nginx-controller and k8 ingress, so we can use it in our test application.
Install Nginx-controller
helm upgrade --install ingress-nginx ingress-nginx \
  --repo https://kubernetes.github.io/ingress-nginx \
  --namespace ingress-nginx --create-namespace
2. Creating Ingress for the services
Save the following content to a file called ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: grafana
  namespace: default
  annotations:
    ingress.kubernetes.io/ssl-redirect: "false"
spec:
  ingressClassName: nginx
  rules:
    - http:
        paths:
          - pathType: Prefix
            backend:
              service:
                name: grafana
                port:
                  number: 80
            path: /
    - http:
        paths:
          - pathType: Prefix
            backend:
              service:
                name: tempo
                port:
                  number: 4318
            path: /v1/traces
Create ingress with kubectl
kubectl create -f ingress.yaml
Testing Grafana
After completing the setup you will be able to access both services in the following addresses. If you wish to use a host instead you can change your ingress file as needed. Check K8 documentation for more details.
Grafana: http://localhost
You can also check all the Kubernetes resources we created with the following command:
#services and pods
kubectl get all -A
#for our custom ingress
kubectl get ingress
Other Observaibility Integrations
This is not the main focus of this tutorial, but if you wish you can also use metrics, logs, and alerts with Grafana. In order to do that you can use Prometheus for metrics, and Loki for logs for example. The installation process can be done in the same way we did with Helm. For more information check Prometheus Helm documentation and Loki Helm documentation.
Setting Up OpenTelemetry with NodsJs
We want to add traces to any NodeJs API, so you can actually use your own project if you wish. If you don't already have a project you can use any open-source sample available on GitHub, like for example the cats API on NestJs repository:
https://github.com/nestjs/nest/tree/master/sample/10-fastify
Adding OpenTelemetry Dependencies
Add the following dependencies to your project:
npm install @opentelemetry/api @opentelemetry/auto-instrumentations-node @opentelemetry/exporter-trace-otlp-http @opentelemetry/sdk-node @opentelemetry/sdk-trace-node
Adding Tracer and Auto-instrumentation
On your src folder add a tracer.ts file with the following content:
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { NodeSDK } from '@opentelemetry/sdk-node';
import {
  BasicTracerProvider,
  ConsoleSpanExporter,
  SimpleSpanProcessor,
} from '@opentelemetry/sdk-trace-node';
import { Resource } from '@opentelemetry/resources';
import { SemanticResourceAttributes } from '@opentelemetry/semantic-conventions';
class Tracer {
  private sdk: NodeSDK | null = null;
  // url is optional and can be omitted - default is http://localhost:4318/v1/traces
  private exporter = new OTLPTraceExporter({ url: "http://localhost/v1/traces"});
  private provider = new BasicTracerProvider({
    resource: new Resource({
      [SemanticResourceAttributes.SERVICE_NAME]: 'DEMO-APP',
    }),
  });
  public init() {
    try {
      // export spans to console (useful for debugging)
      this.provider.addSpanProcessor(
        new SimpleSpanProcessor(new ConsoleSpanExporter()),
      );
      // export spans to opentelemetry collector
      this.provider.addSpanProcessor(new SimpleSpanProcessor(this.exporter));
      this.provider.register();
      this.sdk = new NodeSDK({
        traceExporter: this.exporter,
        instrumentations: [
          getNodeAutoInstrumentations({
            // Lets disable fs for now, otherwise we cannot see the traces we want,
            // You can disable or enable instrumentation as needed
            '@opentelemetry/instrumentation-fs': { enabled: false },
          }),
        ],
      });
      this.sdk.start();
      console.info('The tracer has been initialized');
    } catch (e) {
      console.error('Failed to initialize the tracer', e);
    }
  }
}
export default new Tracer();
Initialize our Tracer
Make sure you initialize your tracer before your server starts, otherwise, auto-instrumentation won't work properly. On your main.ts file (or your entry point class) add the following code.
import tracer from './tracer'
tracer.init()
//Other imports
...
async function bootstrap() {
...
}
bootstrap()
Testing our Telemetry
After running your application let's use some of the APIs so we can generate some traces. For the cat API example, you can run one of the commands below.
# POST cats
curl -X POST http://localhost:3000/cats -H "Content-Type: application/json" -d '{
    "name":"test",
    "age": 18,
    "breed": "test"
}'
# GET cats
curl  http://localhost:3000/cats
Check Telemetry on Grafana
After collecting some data, let's check our traces on Grafana. On the Explore menu, Tempo will be selected automatically, but if not, select Tempo in the data source drop-down list.
On the query options, select search and input the name of the service we configured in our tracer.ts file (DEMO-APP). You will be able to see all the instrumentation collected with auto instrumentation.
Note that auto instrumentation collects different information for each library. For example, if you check our HTTP spans you will be able to see many of our request parameters. Now, if we were using MySQL, we would be able to see up to the query executed for that specific trace. For each instrumentation, check their respective documentation specifically.
If you are curious about which libraries are instrumented automatically, check the documentation on the @opentelemetry/auto-instrumentations-node npm page.
Adding Manual Span
For this tutorial, we've only touched the auto-instrumentation setup, and even though this alone helps us to see the path our users take inside of our system, it does not always provide us with all the information we need.
OpenTelemetry also allows us to define our own custom spans and add them to our traces. This is useful for example when you need to add business information to the telemetry.
You can play around with your own custom spans by adding them to your API code. Follow the documentation for more information.
Extra (Telemetry Propagation)
Trace opens a whole lot of possibilities for us. However, we only saw a small part of it in this tutorial. OpenTelemetry also works really well with distributed systems and provides us with tools to integrate APIs together in the same trace. This is extremely useful when working with microservice architecture, where multiple APIs are triggered for a single user action.
In order to connect microservices together, you should configure OpenTelemetry to send and receive traces to and from other APIs. You can easily achieve that using custom headers and openTelemetry node SDK.
OpenTelemetry provides us with some examples of how to set up our API to communicate with other services. Check the documentation on propagation for more details.
Tracing NodeJs Applications with OpenTelemetry was originally published in 직방 기술 블로그 on Medium, where people are continuing the conversation by highlighting and responding to this story.

Tracing NodeJs Applications with OpenTelemetry

댓글