Rendered at 06:39:41 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
osener 21 minutes ago [-]
I like the end result of OpenTelemetry tracing when using Axiom and the like, but the SDKs have been a nightmare. Too much emphasis on automatic instrumentation, Java-isms, everything is stateful and abstracted away.
It can do distributed tracing of otherwise traditional long running microservices, but breaks down when your functions are distributed like in durable execution engines, Cloudflare Workflows, “functions” that span hours/days/weeks and steps that retry many times.
I had to reverse engineer how SDKs work and how tracing UIs display data so I could make simpler functions that fit wider variety of runtimes and more freely parent spans, start spans and end them from different function instances.
I think most of the API and terminology complexity is self inflicted. Would love to see a rebooted developer experience that is less Kubernates-brained.
EdSchouten 4 hours ago [-]
What always puzzles me about OpenTelemetry is that tracing, metrics and logs are all designed independently. I wish there was a way I could just annotate my code base once, and let the ultimate decision to expose something as a metric/log/trace be dynamic at runtime.
For example, if I look at a graph in monitoring dashboard and see something suspicious, I’d like to say: “The next time something like this occurs again, please save me a trace.” I should be able to just do that with a single mouse click.
I remember them releasing the tracing spec/SDKs and saying “now let’s move on to metrics/logs.” That never sat right with me.
MathMonkeyMan 28 minutes ago [-]
Tracing is the most general of them, and the most expensive unless you're careful with the implementation.
Trace spans are time-delimited units of "stuff that happened", with a tree relationship among the spans, and each span can have arbitrary tags (key/value pairs) and events (time/value).
From that, if you chose, you could derive metrics and logs. The trick is to start with tracing and to actually put it in your program, rather than trying to mostly-automatically tack it on later.
veqq 50 minutes ago [-]
You can do that in Lisp, since you can arbitrarily redefine the wrapper to have such or other logic etc.
If I understand that correctly, it means your app always creates traces, and Grafana Cloud is responsible for sampling/aggregating. That may be prohibitively expensive in terms of CPU/network load.
What I’m suggesting is that your apps by default only send metrics to your monitoring system, but that the monitoring system can specifically ask to “upgrade” metrics to traces. Or to log entries.
The same thing with metric cardinality: by default, only report metrics in a fully aggregated manner. But do tell the monitoring system how they can potentially be broken up if needed (i.e., which labels to add).
ffsm8 2 hours ago [-]
You're pitching a solution that's incredible brittle and unnecessarily complicated if you think about it in technical terms.
For your feature to work you need bi-directional communication between the otel receiver and your application - that's still doable in general, but now you want a synchronous "upgrade" to traces.
Now we're talking about a massive performance impact - and you need to somehow cache all otel data locally so they're available for the upgrade and only then submit then.
It is a architecture that's not very smart, honestly. And precisely the reason why you'd simply submit everything and let the receiver figure out which samples it wants to keep - as thorian pointed out earlier.
thorian1828i03 4 hours ago [-]
> The same thing with metric cardinality: by default, only report metrics in a fully aggregated manner. But do tell the monitoring system how they can potentially be broken up if needed (i.e., which labels to add).
How does the monitoring system have any of the context to add labels? That would only exist in application memory.
> That may be prohibitively expensive in terms of CPU/network load.
In practice I've not experienced this even on quite high request rates. While it isn't free, exporting everything has been cheap enough that the real cost in dollars spent is basically marginal (it's _storing_ the data that's expensive)
EdSchouten 3 hours ago [-]
> How does the monitoring system have any of the context to add labels? That would only exist in application memory.
Indeed. If you have a protocol that doesn’t allow exposing that kind of information, then that only lives in application memory. But my suggestion is that it’s exposed.
ragall 57 minutes ago [-]
> If I understand that correctly, it means your app always creates traces
Yes, because otherwise what you propose requires modifying the binary in-place and that's too big of a security hole for lots of (production) environments. Some variants of that could work with an out-of-process method like Dtrace or eBPF, but that means mutating the kernel, even more of a no-no.
brikym 10 hours ago [-]
I've never found instrumentation to be a huge issue. Sure it takes more effort but you get a lot more value once you understand _business_ events.
bilalq 5 hours ago [-]
OTel is so frustrating. If it wasn't shaping to be the clear winner in the space, I wouldn't complain about it as much. But today:
1. Every major vendor is still in some weird alpha/beta support for OTel even after all this time.
2. The performance hit is substantial and makes you question what the point of performance instrumentation is if you need twice as much compute/RAM to run the same workload now.
3. Serverless runtimes pay a heavy penalty for cold starts with OTel.
4. You're basically forced to run both gateway collectors and edge collectors for any realistic usage.
5. You still need to configure destination exporters in unique ways. This leaves you questioning what the value of OTel was.
6. Vendors that go beyond the scope of what OTel covers still need their own bespoke instrumentation. What was the point of any of this then?
arcanemachiner 2 hours ago [-]
So what's the alternative then? (Genuine question, not hypothetical snark.)
bilalq 1 hours ago [-]
There isn't really a great alternative without vendor lock-in. If you go all-in on AWS Cloudwatch/X-Ray, it's a really easy setup with low effort. If you go all-in on Datadog, it's pretty easy. But if you want to mix Sentry, Langfuse, Datadog, etc, OTel is still probably the best option. It's just a letdown that this is the best there is.
I don't mean to disparage anyone working on OTel. I can appreciate that it has ambitious goals and it's not an easy problem to get alignment and interop here. Especially with all the stakeholders involved. But as a user, it feels simultaeneously over-engineered and under-engineered.
cyberax 3 hours ago [-]
> 4. You're basically forced to run both gateway collectors and edge collectors for any realistic usage.
You most certainly don't. You can run your app (especially if it's "serverless") without the collector agent.
App-to-agent and agent-to-sink use the same protocol, so all you need to do is set up the tracing/logging/metrics exporters to directly speak with the sink. These days, it typically means specifying the URL and the DSN header.
bilalq 1 hours ago [-]
Perhaps there's a gap in my understanding. Can you clarify on this a bit more? I run a mix of serverless and non-serverless workloads.
Gateway collectors are unavoidable because various SaaS platforms require you to be running publicly reachable endpoints to send telemetry to.
In a runtime like Lambda, how would you avoid the need to run an edge collector? The only thing that comes to mind is to write to logs and then have a log stream processor that then writes to your gateway collector. Other than that, it seems unavoidable, no? Sure, in something like Fargate you could go app to sink. But even that has its own tradeoffs.
rcleveng 2 hours ago [-]
Sounds a lot like K8s. It's not a framework you use, it's a framework to build a framework on top of.
I wish the observability vendors would move to using it under the covers so it's easier to mix and match.
I wish the otel support wasn't super buggy in most of the frameworks and backends.
cyberax 4 hours ago [-]
I disagree. I'm an observability geek, and OTel is... fine.
It's missing a few things that I'd like, but I was able to implement them myself. I guess the major design issue is that the sampling decision is made at the _start_ of the segment. So I hacked up a few improvements:
1. Ability to mark segments as "boring", so they are dropped before the export. For things like healthchecks, empty "get the pending jobs" queries, etc.
2. Ability to downgrade errors for segments that are expected to return an error (e.g. HEAD on a non-existing object in S3 to check if there's a cached blob).
I understand the author's perspective in the linked article, but none of that data shows a project in trouble? Some languages have more resources than others, but those all look like healthy open source projects
steerpike 59 minutes ago [-]
Oh my god. A Jeremy Morrell sighting in the wild.
Every time I share your blog (and I share it a lot) I tell people:
"This guy started a blog in 2024. Wrote three posts and all three of them would still make my top ten list of 'greatest posts on observability' today".
'A practitioner's guide to wide events' especially is still my number 1.
jauntywundrkind 14 minutes ago [-]
I'd make a wager that things would go better smoother faster if folks tried more stuff, ventures forth more on their own. It's obviously not great that there's no semantic convention that's perfect and just works for everything, and yeah it takes a while. I feel like the real data I'd want is who else, how many people show up to say they've tried something. Is that happening? Whether specs are really good enough advance or not, to me, is often whether enough people have tried it to find out.
The net of this is, otel is a very flexible system you can use and adapt in all kinds of ways and while the spec is important, using the toolkit to FAFO yourself, ahead of any beaten path, should really be encouraged. That's the message I'd want to see being radiated out about otel.
jiggawatts 7 hours ago [-]
The alternative is vendor lockin, $$$, and spotty support for complex environments with zero chance of ever getting 100% coverage.
At least with Open Telemetry, anyone can write an OTLP "source" using free, open specifications, and it'll "just work" with dozens of third-party "sinks". That's huge!
Sure, there's a lot of experimental tags on semantic conventions, but at the end of the day, that's not that critical. It's just data: most sinks don't "interpret" these tags, they just display them as-is, so changes aren't breaking changes.
GauntletWizard 5 hours ago [-]
The alternative is Prometheus (which is freaking great) and Jaegar (which is freaking great), each alone. This is better, because Otel is trying to put two distinct things (monitoring and metrics, distributed tracing) into one package, because they know how to use neither.
Neither Prometheus metrics nor Jaeger traces are magic bullets. Neither of them are complicated, either, and in fact the fact that they're not complicated is their greatest strength. You can and should understand every facet of what they entail. You should build the (very small) shims that they need for your company's framework every time. It's not hard. It's not hard because it's not complicated. The fact that it's not complicated seems to break people's brains. They are accurate because they're simple and they're easy to work with because they're simple, and OTel is neither.
morganherlocker 3 hours ago [-]
Prometheus is so easy to add and if you need more scale, there is mimir and a few other options with similar client semantics. I really can't imagine reaching for a framework APK that tries to anticipate every possible thing I would want telemtered, and is inevitably missing all the domain specific derived channels I need. Even prepackaged Prometheus exporters are usually overkill.
firesteelrain 5 hours ago [-]
I’ve built custom Prometheus metrics very easily and had node exporter pick up the .prom files. Python and bash scripts reading and translating.
Node exporter runs on my Prometheus server next to Blackbox Exporter. Blackbox Exporter handles TLS expiry metrics.
lenkite 3 hours ago [-]
Hard Agree on Prometheus. And esp on the complexity - OTel is dizzyingly complex. You can get started ASAP on Prometheus whereas you get lost in analysis-paralysis when dealing with OTel.
cyberax 4 hours ago [-]
OTEL metrics are a bit awkward, but they work just fine with Prometheus.
Jaeger uses the OTLP protocol nowadays. So it _is_ OTEL.
arcanemachiner 2 hours ago [-]
What, so people don't like OTel, but they like Jaeger, which implements an OTel spec? (I'm a noob to this subject, if that wasn't obvious.)
cute_boi 4 hours ago [-]
I wish otel was never there. It is badly designed abstraction and due to otel the code gets very very messy and bad.
hn_acker 13 hours ago [-]
(TFA author is not me.)
0xbadcafebee 4 hours ago [-]
It is crazy to me how often people don't grok how to design software well.
1. The worst thing you can do is try to stuff too many things into one specification. So you want an API? That's great. What's that? You want a rigid set of types so that any tiny changes over time aren't compatible? You want to try to define every conceivable use case as a new call? You want to combine multiple elements from different domains into one flat set of functions? You don't have any hierarchy or inheritance? You don't support extensions?
2. The second-worst thing you can do is to force a whole lot of different people to go through a single standards body. So you want to support a thousand different 3rd party components. What's that? You want to require everyone get their adapter approved by one group? And there's only one supported adapter per 3rd party component?
If you're trying to feed an entire city, it's logistically incredibly difficult to try to do it all yourself. If instead you just define where food can be dropped off or picked up, and ask volunteers to bring their own food there whenever they can/want, now you don't have a logistical nightmare on your hands anymore. The tech alternative? Add support for "plugins", make the plugin interface incredibly loose/backwards-compatible/layered, and invite people to publish their own plugins. If you under-engineer it, it actually works better.
It can do distributed tracing of otherwise traditional long running microservices, but breaks down when your functions are distributed like in durable execution engines, Cloudflare Workflows, “functions” that span hours/days/weeks and steps that retry many times.
I had to reverse engineer how SDKs work and how tracing UIs display data so I could make simpler functions that fit wider variety of runtimes and more freely parent spans, start spans and end them from different function instances.
I think most of the API and terminology complexity is self inflicted. Would love to see a rebooted developer experience that is less Kubernates-brained.
For example, if I look at a graph in monitoring dashboard and see something suspicious, I’d like to say: “The next time something like this occurs again, please save me a trace.” I should be able to just do that with a single mouse click.
I remember them releasing the tracing spec/SDKs and saying “now let’s move on to metrics/logs.” That never sat right with me.
Trace spans are time-delimited units of "stuff that happened", with a tree relationship among the spans, and each span can have arbitrary tags (key/value pairs) and events (time/value).
From that, if you chose, you could derive metrics and logs. The trick is to start with tracing and to actually put it in your program, rather than trying to mostly-automatically tack it on later.
What I’m suggesting is that your apps by default only send metrics to your monitoring system, but that the monitoring system can specifically ask to “upgrade” metrics to traces. Or to log entries.
The same thing with metric cardinality: by default, only report metrics in a fully aggregated manner. But do tell the monitoring system how they can potentially be broken up if needed (i.e., which labels to add).
For your feature to work you need bi-directional communication between the otel receiver and your application - that's still doable in general, but now you want a synchronous "upgrade" to traces.
Now we're talking about a massive performance impact - and you need to somehow cache all otel data locally so they're available for the upgrade and only then submit then.
It is a architecture that's not very smart, honestly. And precisely the reason why you'd simply submit everything and let the receiver figure out which samples it wants to keep - as thorian pointed out earlier.
How does the monitoring system have any of the context to add labels? That would only exist in application memory.
Grafana went the other way - your app exports all labels, and then you selectively aggregate on ingest: https://grafana.com/docs/grafana-cloud/observe-and-act/adapt...
> That may be prohibitively expensive in terms of CPU/network load.
In practice I've not experienced this even on quite high request rates. While it isn't free, exporting everything has been cheap enough that the real cost in dollars spent is basically marginal (it's _storing_ the data that's expensive)
Indeed. If you have a protocol that doesn’t allow exposing that kind of information, then that only lives in application memory. But my suggestion is that it’s exposed.
Yes, because otherwise what you propose requires modifying the binary in-place and that's too big of a security hole for lots of (production) environments. Some variants of that could work with an out-of-process method like Dtrace or eBPF, but that means mutating the kernel, even more of a no-no.
1. Every major vendor is still in some weird alpha/beta support for OTel even after all this time.
2. The performance hit is substantial and makes you question what the point of performance instrumentation is if you need twice as much compute/RAM to run the same workload now.
3. Serverless runtimes pay a heavy penalty for cold starts with OTel.
4. You're basically forced to run both gateway collectors and edge collectors for any realistic usage.
5. You still need to configure destination exporters in unique ways. This leaves you questioning what the value of OTel was.
6. Vendors that go beyond the scope of what OTel covers still need their own bespoke instrumentation. What was the point of any of this then?
I don't mean to disparage anyone working on OTel. I can appreciate that it has ambitious goals and it's not an easy problem to get alignment and interop here. Especially with all the stakeholders involved. But as a user, it feels simultaeneously over-engineered and under-engineered.
You most certainly don't. You can run your app (especially if it's "serverless") without the collector agent.
App-to-agent and agent-to-sink use the same protocol, so all you need to do is set up the tracing/logging/metrics exporters to directly speak with the sink. These days, it typically means specifying the URL and the DSN header.
Gateway collectors are unavoidable because various SaaS platforms require you to be running publicly reachable endpoints to send telemetry to.
In a runtime like Lambda, how would you avoid the need to run an edge collector? The only thing that comes to mind is to write to logs and then have a log stream processor that then writes to your gateway collector. Other than that, it seems unavoidable, no? Sure, in something like Fargate you could go app to sink. But even that has its own tradeoffs.
I wish the observability vendors would move to using it under the covers so it's easier to mix and match.
I wish the otel support wasn't super buggy in most of the frameworks and backends.
It's missing a few things that I'd like, but I was able to implement them myself. I guess the major design issue is that the sampling decision is made at the _start_ of the segment. So I hacked up a few improvements:
1. Ability to mark segments as "boring", so they are dropped before the export. For things like healthchecks, empty "get the pending jobs" queries, etc.
2. Ability to downgrade errors for segments that are expected to return an error (e.g. HEAD on a non-existing object in S3 to check if there's a cached blob).
I understand the author's perspective in the linked article, but none of that data shows a project in trouble? Some languages have more resources than others, but those all look like healthy open source projects
Every time I share your blog (and I share it a lot) I tell people:
"This guy started a blog in 2024. Wrote three posts and all three of them would still make my top ten list of 'greatest posts on observability' today".
'A practitioner's guide to wide events' especially is still my number 1.
The net of this is, otel is a very flexible system you can use and adapt in all kinds of ways and while the spec is important, using the toolkit to FAFO yourself, ahead of any beaten path, should really be encouraged. That's the message I'd want to see being radiated out about otel.
At least with Open Telemetry, anyone can write an OTLP "source" using free, open specifications, and it'll "just work" with dozens of third-party "sinks". That's huge!
Sure, there's a lot of experimental tags on semantic conventions, but at the end of the day, that's not that critical. It's just data: most sinks don't "interpret" these tags, they just display them as-is, so changes aren't breaking changes.
Neither Prometheus metrics nor Jaeger traces are magic bullets. Neither of them are complicated, either, and in fact the fact that they're not complicated is their greatest strength. You can and should understand every facet of what they entail. You should build the (very small) shims that they need for your company's framework every time. It's not hard. It's not hard because it's not complicated. The fact that it's not complicated seems to break people's brains. They are accurate because they're simple and they're easy to work with because they're simple, and OTel is neither.
Node exporter runs on my Prometheus server next to Blackbox Exporter. Blackbox Exporter handles TLS expiry metrics.
Jaeger uses the OTLP protocol nowadays. So it _is_ OTEL.
1. The worst thing you can do is try to stuff too many things into one specification. So you want an API? That's great. What's that? You want a rigid set of types so that any tiny changes over time aren't compatible? You want to try to define every conceivable use case as a new call? You want to combine multiple elements from different domains into one flat set of functions? You don't have any hierarchy or inheritance? You don't support extensions?
2. The second-worst thing you can do is to force a whole lot of different people to go through a single standards body. So you want to support a thousand different 3rd party components. What's that? You want to require everyone get their adapter approved by one group? And there's only one supported adapter per 3rd party component?
If you're trying to feed an entire city, it's logistically incredibly difficult to try to do it all yourself. If instead you just define where food can be dropped off or picked up, and ask volunteers to bring their own food there whenever they can/want, now you don't have a logistical nightmare on your hands anymore. The tech alternative? Add support for "plugins", make the plugin interface incredibly loose/backwards-compatible/layered, and invite people to publish their own plugins. If you under-engineer it, it actually works better.