ai newsDi., 3. Nov.
EilmeldungAnthropic's Message Threads cut time to first token by 35 percent for long conversations

LinkedIn Uses OpenTelemetry to Trace the Member Journey from Tap to Backend

LinkedIn has built an OpenTelemetry-based telemetry platform that connects what members do in its iOS, Android, and web applications with the backend services processing those interactions. By Craig Risi

InfoQDienstag 13:00 · 03.11.3 Min. Lesezeit
Bild: InfoQ

The goal is to close a common observability gap. Client analytics can show that a member tapped a button or encountered an error, while backend tracing explains what happened inside the services handling the request. Historically, connecting these views required engineers to correlate separate systems. LinkedIn's approach allows a member interaction to be followed from the client through frontend and backend services to the underlying data stores.

A single page can generate numerous requests, each potentially passing through multiple services. LinkedIn introduced a pageForestId to provide context above individual distributed traces.

Each request retains its OpenTelemetry traceId, while related requests from the same page context can be associated through the pageForestId. Engineers can therefore investigate both an individual request and the wider activity that contributed to a member's experience.

This is important because conventional distributed tracing generally follows one request, whereas a member journey can involve many related requests.

LinkedIn operates a diverse technology stack spanning mobile and web clients alongside backend services written in Java, Python, Go, and Rust. OpenTelemetry provides a common model for instrumentation, trace context, and semantic conventions across those environments.

LinkedIn has built its own wrappers and extensions around the standard OpenTelemetry SDKs, allowing product teams to use consistent instrumentation while the platform handles concerns such as mobile lifecycle, export, and LinkedIn-specific metadata.

Client and backend applications generate OpenTelemetry spans, with trace context propagated as requests move through the system. Shared HTTP and gRPC frameworks automatically instrument common request paths, while teams can add explicit spans around important business operations.

Telemetry is transported through Apache Kafka and stored in ClickHouse, with pageForestId, traceId, and timestamps providing the keys for correlation. This allows engineers to begin with a member-facing interaction and drill into the backend execution responsible for it.

Capturing every span would generate an enormous amount of telemetry, making sampling fundamental to the architecture.

LinkedIn uses coordinated head sampling so related client and server activity receives a consistent sampling decision. A pseudonymised member identifier helps maintain that consistency across the journey, preventing the client from being sampled while the corresponding backend trace is discarded, or vice versa.

The company also distinguishes valuable quality-of-experience telemetry from lower-value debugging information. LinkedIn estimates that up to 90% of some backend tracing volume could potentially be treated as debug-level data.

Client-side observability introduces additional constraints. Mobile applications can be suspended or terminated before telemetry is transmitted, so LinkedIn batches spans but can flush telemetry when an application moves into the background.

The platform also applies controls around what data can enter traces, including attribute allowlists, denylists, and redaction. This is important because distributed traces can otherwise become an unintended channel for sensitive application or member information

Traditional distributed tracing tools such as Jaeger and Grafana Tempo are primarily focused on following requests across backend services. Commercial platforms such as Datadog and New Relic also combine distributed tracing with real-user and application monitoring.

LinkedIn's approach is less about introducing a capability that these platforms cannot provide and more about making OpenTelemetry the common foundation connecting its client and backend telemetry. Its own pageForestId model adds a higher-level grouping around standard traces, while Kafka and ClickHouse provide a customised ingestion and storage architecture.

This also gives LinkedIn greater control over its observability stack and avoids making application instrumentation dependent on a particular commercial backend.

A member does not experience a traceId; they experience a page that loads slowly, an action that fails, or a feature that behaves unexpectedly. Connecting those experiences to the distributed systems behind them gives engineers a much more direct path from user impact to technical cause.

PostgreSQL Built for Time-Series Scale Handle massive time-series data with PostgreSQL, without sacrificing performance. Discover how Tiger Data delivers high-speed ingest, fast queries, and scalable storage. Learn More.

Quelle: InfoQ (InfoQ). Original öffnen