Hey everyone, welcome to the Unhandled Exception podcast, I'm Dan Clark and this is episode
number 41 and this episode is all about open telemetry, which is a term that seems to
pop-up every time I turn around at the moment and for good reason too.
And here to join me today to chat all about it is Martin Thwaites, a developer advocate
at Honeycomb.
So welcome to the show Martin, thank you for joining me.
Well, thank you for having me.
It's great to be talking about such a popular term.
I know I'm definitely quite excited about this one.
So could you introduce yourself to the listeners and tell us a bit about what you do?
Yeah, so my name is Martin Thwaites.
I go by martin.net on Twitter.
I am a developer advocate for a company called Honeycomb.
I fairly recently joined them, but I've been involved in the observability and the whole
sort of logging metrics community for the last sort of five or six years.
And that's generally what I like to rant about on stage and will wax lyrical to
anybody about who will hear me, like yourself.
Yeah, as I said before, I'm really looking forward to this, just digging to open
telemetry, observability, all, especially all the tooling around it as well,
which I'm guessing your Honeycomb is going to play into that as well.
But before we dig into that, I'm just going to quickly do this episode's listener mention.
And this one goes to Tim Discriver, who tweeted out, the past month, I've discovered
and enjoyed the Unhandled Exception podcast by Dracan during my runs.
A great mix for mind and body.
I highly recommend this if you want to follow dotnet topics, learn new things and
enjoy some chitchat along the way.
So thank you, Tim.
And based on what you said, you're probably running right now as you're listening to this.
Hopefully you're having a good run and thank you for your kind comments.
And if you want to get mention on the show, just send a tweet using hashtag
UnhandledException.
All feedback is greatly appreciated and I'm Dracan on Twitter, which is at
Dracan.
And I quick reminder that this podcast is sponsored by Everstack, which is my own
company providing software development and consultation services.
For more information, visit Everstack.com.
And also very new, just in case you don't already have enough slack workspaces
already, I've created one for the podcast.
So basically a place for listeners to hang out and chat about both the episodes
and also just software development in general.
So to sign up, pop over to the website UnhandledExceptionPodcast.com
and you'll see a big, shiny, new slack link.
So hopefully see you in there.
So open telemetry.
So let's, if we can just assume that, even though, as I said before, you can't
really turn around without hearing about it, let's assume that the listeners have
maybe never even heard this term open telemetry.
Can we just start off by just defining what that actually means?
Yes, absolutely.
So open telemetry is an effort to standardise around tracing.
Very specifically, that's where it started out.
So it all centers around the observability movement, which is a movement
around software engineering and the site reliability engineering around observing
your systems from the outside.
So being able to know how your systems work and by systems, we're talking
about individual systems or distributed systems, but understanding how they're
running, are they healthy?
Are they not healthy?
And trying to evolve that in a world where we're moving to more distribution,
we're moving to more microservices, we're moving to systems that scale, scale
down to zero, scale up to thousands.
These are all systems that require a different way of thinking than we have
in the past.
That's the observability movement.
An open telemetry is a set of libraries or more specifically a protocol
that was created to be able to bring together all of these things in a coherent
way. So it stems from two individual libraries, one called open tracing
and one called open sensors, which were two competing projects about
I think about four or five years ago now, where they were providing tracing
and tracing is this idea of taking a request or some kind of trigger and
following that through individual systems and individual process boundaries in
your applications to be able to say, here's my request into my website.
That request caused a request into another system, and that one went into
another system. So the idea of a front end and a back end and multiple back
ends, and you want to be able to see that particular request and how that made
it through into lots and lots of different systems.
Now, it was always quite hard.
So Open Telemetry came up with this idea of bringing together open sensors
and open tracing standardising around the W3C tracing standards, which
is a widely accepted standard and building a protocol on top of that to
make things a lot more vendor agnostic.
So it's a in a nutshell, Open Telemetry is a standardised protocol of how
things should be transmitted between two services, but also how that data is
then transmitted on to vendors, which makes it completely vendor agnostic.
What they've done is then layered on top of that, the idea of SDKs.
So there's SDKs for, I believe, nine languages now, that all standardise
around the same approach, and they're very familiar if you're transitioning
between languages.
And then further on to that, there's initiatives around being able to add
things like auto instrumentation.
So things that are common in the dotnet world, so things like SQL client,
being able to say, oh, I'd like to be able to see what queries were on.
So they have these auto instrumentations as well.
It is going a little bit further now.
Open Telemetry is a really evolving project.
It's number two on the CNCS list of projects, so it's getting a lot of
attention right now.
But yeah, it stems from tracing.
It's moving into different signals now, which is the Open Telemetry terminology
for different segments.
So we've got the main signal being tracing, then now moving into
different signals like logs and metrics, which should be GA very soon.
As soon as you started talking about it, in my head, just jumped out
microservices, that kind of distributed system.
And then you actually did use the term microservices.
But I guess thinking about lots of modern architecture, where the lines
blur over to whether you would call something a microservice, because now
you tend to get like a, you might get a spire application, you might get
a few APIs and stuff that do different things.
That might not be a full on microservice architecture, but now
you tend to have multiple services involved, even in just a normal application.
So just having this out of the box standardisation just sounds really, really good.
I think what's really interesting there is that from an Open Telemetry
perspective, we talk about process boundaries rather than microservices.
So a microservice, as I say to quite a lot of people, unfortunately, there
is no IEEE standard.
So you can get somebody to come in and stamp your application and say, yes,
that is absolutely a microservice.
It means a lot of things to a lot of people, which means that we, as a
community, we don't want to get into that debate.
So we talk about process boundaries.
So we talk about a request that transitions to process boundaries.
That might be two process boundaries on the same machine.
It might be two process boundaries that exist on two different VMs.
It might be two different containers.
It could just be something that happens now and then something
that happens in a few days time, because somebody picked something up.
But it's also about correlating all of those things together, the things
that cause something else to happen.
I'm glad to use the term correlating then, because talking about all
this tracing stuff you've been describing, correlation ID is jumped to mind,
where in the past we might have manually tried to do it.
But presumably, this whole concept of you've got one service calling another
calling another and passing some form of correlation idea or whatever it is across.
So then when you're observing and you're looking at the logs and the traces
and things, you can just see exactly what happened across an entire.
I don't want to say call stuck.
It's almost like a transaction transaction.
And transactions, the term that we use a lot of the time in transaction
is now analogous to a trace.
So a trace would be how does this particular initiator.
So the the HTTP request being your initiator that initiates a trace.
How do I trace that through all of the different things that happened
in all of the different systems before that thing then returns to the user?
So think of it like a the idea of a transaction.
So a HTTP request being a transaction from request to response
and everything that's entailed within there would be a trace in its own right.
Yeah, that makes total sense.
And then, like, if you think about a support team, if they get an exception
or something, they can actually see that entire transaction
and find out exactly what happened that caused that one exception.
Yeah. And it's, you know, you mentioned correlation, you mentioned logs
and that's something that's existed for, I would imagine, decades,
probably way before my time coming into development,
but correlating logs has been, you know, a staple for a lot of people.
The difference when you move into tracing
is that you're actually talking about causality.
So it's not saying that a request has 4,000 log entries
because it went through seven systems and all of those seven systems,
each of them produced 300 log lines.
But I've got a correlation ID, so I can do the correlation ID
and I can work out the times and I can say,
well, this thing happened after this thing,
so that probably is the things that happened at the same time.
So when you move into tracing,
what you're actually doing is you're saying that
this thing that I've done was caused by this other thing.
So in a HTTP context, and to use a specific example,
when you've got a HTTP request coming in that then calls a database
and then there's a call after that to a third-party web service,
what you can actually see is that the HTTP request
cause the SQL statement to be executed,
but then it wasn't the SQL statement being executed
that caused the web service to be called after it.
It was done in parallel or it was done in series,
but it wasn't called by the database.
And that is really important
when you start to think about these really complex systems,
because you want to know that these things happened
one after the other,
not that they happened at the same time,
not that they happened at some point within the request.
You want to be able to see those individual things
and what caused each one of those to happen,
which is where a lot of people,
you will say, and this is something that you'll see
if you start googling
and you start looking at the open telemetry docs,
you'll see something called a trace waterfall,
which is the idea of a individual transaction,
this trace that we were talking about,
where each of the things that happen,
they have blocks,
you may get sort of flashbacks to Microsoft projects,
it does look a little bit like a Gantt chart,
but trust me, it's not, it's actually useful.
So you'll see this trace waterfall
and you'll see a HTTP request
and you'll be able to see each of the spans within there,
which is what we call those individual segments within there.
And it starts to provide you with a really good picture
of what's causing a request to either be slow,
what's causing a request
to have thrown an exception, for instance,
so it might throw an exception right up at the top,
but what you actually wanna see
is that it was this third party web service
that caused that exception to happen
and everything else worked fine
and you can see that at a glance
from looking at things like a trace waterfall.
So this sounds like such a complicated problem
if you were trying to solve it yourself,
which actually screams out why
something like open telemetry is so valuable.
Presumably, like for where you're talking about
the observability and having tooling
to view that information.
Presumably, because open telemetry is a standard,
then you can use many different observability tools
and you can basically pick and choose
and it would all just plug into the same data,
is that correct?
So that's exactly what open telemetry is built to do.
It's to built to be 100% vendor agnostic.
It's not built by Honeycomb,
it's not built by any of the other providers.
It is built by the community.
Yes, there's a lot of us
from the observability community
that work for companies
that provide observability products
that are helping to build this open telemetry
as a library.
But the idea is that it's completely vendor agnostic.
You will implement into your code base open telemetry,
you'll then decide later
as to where you will send all of those signals to.
You can even decide
that actually I want to send my tracing data
to one provider,
I want to send my metrics data to another provider
and I want to provide my logging data to another provider.
That idea of being 100% vendor agnostic
is one of the founding principles of open telemetry,
is that we don't want it to be a lock-in
at a vendor level
because you've installed some things
that are very specific to that particular vendor.
We want to make it so that you can choose
and it's up to the vendors to be better
than the other vendors,
not by choosing lock-in.
And it's one of the founding principles around Honeycomb,
is that we are the best in a particular area,
so we push back into open telemetry
and we don't really have our own SDK, for instance.
We use the open telemetry SDK.
You install the open telemetry SDK,
you put our URL in the exporter
and send us the data
and it will then be visualised in there.
There's also open source products that you can use,
especially if you're running locally,
it's really useful to be able to run
some of the open source products
just to get some of those trace profiles
in a very, very quick way.
It's funny you mention about locally
because that was actually,
coincidentally, my next question.
Like, quite often,
for example, if I'm doing something with messaging,
I would use, like, in production,
something like Azure Service Bus,
but locally, I would use RabbitMQ
because I use an abstraction library
and I use Docker very heavily,
being able to spin up all my local stuff,
like, whether it's a database,
whether it's RabbitMQ,
and presumably, because all this is using
this open telemetry standard
and, as you mentioned,
there's open source tooling
that you can then, just as part of your Docker Compose,
you can spin up tools to visualise
your open telemetry data as well,
just as part of that local development experience.
Absolutely.
Jager is a very popular tool for doing that
and it has a Docker image that you can use to do that.
What we found is that
running stuff at scale in production
is a very specific problem,
so trying to run Jager at scale in production
is not a pleasant experience, shall we say.
Not that it's not doable,
but the amount of resources you sink in
to be able to do that,
it's the same as running
something like InfluxDB in the cloud
and trying to scale that,
it's quite hard to do
because it has some certain restrictions on there
and we always recommend that
if you're moving into production,
you want to be able to visualise
vaste amounts of data
very, very quickly,
then push it to a cloud provider
that's built a data store
that's specifically suited to that particular need.
But obviously running locally is a bit of a problem.
It's something that I'm personally invested in doing
is trying to work out
what our local developer experience
looks like for tracing.
I'm on a mission to remove logs
from all applications
and replace them with traces,
but in order to do that,
we need to improve the utility of traces
so that you get the same level of visibility
out of traces locally
as you do from logs locally,
because then you get the byproducts
that the traces that you've just built
for your local visibility,
you're going to get in your production visibility as well,
which is when we get to this idea
of observing production systems,
that ability to say
what's going wrong?
And if I've been able to debug that locally,
I should now be able to debug that in production as well.
Yeah, totally.
I think it also helps developers
understand it if they can just play around locally
without fear of damaging any production data or anything.
It's just a local thing.
Doc Compose up,
play around to your heart's content,
and then it just helps developers
just understand it better.
Actually, what I want to do
is I want to go a level further than that.
I don't even want people using Docker.
I want people to be able to say locally
I've got a console output
and I use my console output
to look at my logs at the moment.
As a lot of people do, you know,
et especially the people who are running in the modern ways
with .NET, so they're not running Visual Studio,
they're running things like VS Code
and they've got, they're used to the terminal now.
They can see the log output from a terminal
and they use that log output
to be able to see what's going on.
I want that same level of visibility of traces
into those sort of environments.
Not I need to do a Docker Compose up.
Because Docker Compose up is great.
You can bring up lots and lots of stuff
and I talked about being in this sort of
observability community for the last sort of five years
and that's how I started out.
I used to use Docker Compose
to bring up an influx DB,
to bring up a cluster
to be able to send my logging data to.
I used to do that,
but it was running something on your machine.
It uses CPU cycles
and if your PC isn't that great
in terms of performance,
it causes a problem trying to run all these things.
So what we're trying,
what I'm trying to do personally
is look at how we can bring that
into that local developer experience
so that people don't go,
oh, I need to do a Docker Compose up
and run another website
to be able to see all this stuff.
I'll just use logs, it's easier.
This whole idea is about
how do we make the right thing the easy thing?
So if looking at traces is the right thing,
let's make that easy for developers to do.
My mission is to bring tracing to developers.
To make sure that developers are the ones that go,
I need tracing
because I care about my production systems.
And I think that's a message
that I'm trying to push in the community at the moment
is that developers need to care about production.
The whole DevOps movement
was around developers and operations
getting closer together,
developers caring about production,
ops caring about development
and working together to do it.
And I think that's core to this idea
of us saying,
great, we've got tracing.
How can I use that?
How can I know that my code
is actually running in production?
Yeah, totally.
And I think with the local development store,
it's all about giving the developer's choice.
So like for those developers
that want to use VS Code
and just see output in the command line,
they can do for those developers
that want to actually have a web portal,
like whether it's Jago or whatever,
so they can actually get the full experience.
And it's just different developers
like to work in different ways
and it's giving everyone the right choice.
So before you mention about these open telemetry SDKs,
who would actually be using those SDKs?
Is that for them,
like for example, I'm a .NET developer,
am I as a developer using those SDKs?
Or is that something that Microsoft
is using as part of .NET?
As a Node.js developer,
would they be explicitly installing
these open telemetry SDKs?
I'm just trying to work out
who these SDKs are actually targeted for.
So there's, let's just say
in the community at the moment
that it's fragmented.
There are a few different ways
that you can extract data,
extract traces, metrics
from your systems using open telemetry.
The main way for .NET developers to do this
is about 10 lines of code in your startup,
which will initialise it,
it will use something called activity in .NET
to then observe all of the things that are happening
and then convert those into spans
and then push those to whatever exporter
it is that you've wanted to do.
That's the main way.
There is a project at the moment
within open telemetry called auto instrumentation for .NET,
which is using the CLR injection
to actually inject all of these things at runtime
so that you don't even need to do the 10 lines of code
to do that,
which is a really interesting project
that I'm involved in.
But the main way for me is doing that idea
of just drop some code, 10 lines,
it's really hard, you know,
drop the 10 lines of coding you start up, job done.
Within a lot of the other communities,
so specifically Java and Python,
there is an agent that you can run
and the agent will wrap your JVM process
or your Python interpreter
and go great, I will give you all of this data just for free.
Now, to me, this is the difference
between auto instrumentation and manual instrumentation
because my feeling is that manual instrumentation
is where the magic happens.
Manual instrumentation is what you understand
about your application
and you saying this thing is interesting to me.
You'll say I've got an algorithm,
these 20 lines of code in my, this method here,
these are really important
that we know whether this thing is running particularly well.
It can have massive implications,
it might be that it changes the performance
based on the number of, I don't know,
you pass in an integer into the method
and if it's a 10, then it's going to take 20 seconds,
if it's a 5, it's going to take 5 seconds
and that might be really interesting to you.
Manual instrumentation is where you say
I'm going to wrap that in a span
and that is where things get really interesting with tracing.
Otherwise, all we're really doing
is we're looking at the out of the box metrics
and the out of the box spans that we get
from SQL client or Redis client or from ASP net,
which provide a certain level of utility,
but it's not where the magic happens.
So to me, the user of these libraries
is actually the developers, the engineers.
And this is one of the reasons why
I've been brought into Honeycomb
is to start this movement of
how do we get developers to care about open telemetry,
how do we get developers to care about
their production systems enough
to be implemented in open telemetry,
to be able to say I need visibility
and availability of these things
at a very, very low level.
I see.
But it sounds like, again,
I'm just using .NET
because I'm a .NET developer,
so that's the platform I'm using.
But if I wanted to create my own span
or activity, I think they call it,
then I would still be using Microsoft's activity class.
I wouldn't be using an open telemetry library
that would be happening
through Microsoft's activity, I guess.
OK, so your mannerisms
when you were saying that
is exactly how everybody's feeling
right now about activity.
Activity has muddied the waters.
There are reasons why,
and I've wrote an entire blog post,
which I'm sure we can link in the show details,
about where activity comes from
and why it's a little bit different
and that kind of stuff.
But yeah, the activity is something
that Microsoft used to,
is analogous to a span,
whereas an activity source,
which is what an activity is created from,
is analogous to a tracer in open telemetry.
So yes, you would use those.
You can use the open telemetry shim,
which has been created
to give you that same idea of tracers and spans
so that you've got consistent terminology.
Under the hood, it's just creating activities.
So that's more about syntactic sugar
on top of the stuff that you're doing.
But yes, you would create an activity
and that activity would run for a particular length of time
and you would add attributes to it or tags in .NET.
And then that thing would be then observed
by open telemetry.
So open telemetry then listens to activity sources.
Under the hood, what it's really doing
is creating an activity listener
and that activity listener
is then listening to certain sources.
And then whatever sources are then producing activities,
it will then convert those into spans
and then it will send those
to whatever exporters you've configured.
Very, very complicated.
And it's hopefully something
that people don't really need to care about.
And that's what we're trying to do
is abstract them away from this idea
that you actually need to know what an activity source is,
what an activity listener is.
You need to know how to convert that
into a tool that somebody else uses.
And we've not, as a .NET community,
done really well with this before.
As a .NET community, what we've done
is push things to vendors.
And we've installed a library
that's very specific to an individual vendor.
And unfortunately, that gives you a lot of locking.
You can't easily move vendors
if there is somebody else
who's doing something in a particular space that's better.
This is hopefully going to allow people to be able to say,
I'm just using open telemetry.
Oh, look, there's a better provider of X.
We will start sending that data to them.
My hope, beyond hope,
is that this keeps vendors honest.
It keeps vendors innovating.
It keeps vendors from getting complacent
because you actually need to be continually the best
because it's easy for people to move.
But it's also easy for people to send it to multiple sources.
And that's a really interesting thing within open telemetry.
Because within open telemetry,
you can actually configure multiple exporters
for the same signal.
So you can say, for my traces,
I would like to send them to Yeager locally.
I'd like to write them to a file.
I'd also like to send them to vendor X, Y, and Z.
And what we've seen specifically within Honeycomb
is people actually using that
as the equivalent of the Great British Bake Off of vendors.
We'll send it to all of them and see which one does best.
It's the technical challenge
of the Great British Open Telemetry.
So how do we then take that to all of these vendors
and say, well, we've got some traces.
Who's going to provide us with the best level of visibility?
Because all of the vendors do something a little bit different.
For Honeycomb, for instance,
we focus on high cardinality, high dimensionality,
high performance and significant load.
Those are the things that we really focus on.
We've got other things that are coming, but that's our niche.
That's where we've built our brand around,
being able to say,
send us as many attributes as you want on every one of your spans
and we'll be able to do great things with it.
There are other vendors who focus on different things,
like service mapping, for instance,
which is a very specific idea,
which is a different thing
that you might want to send all of your traces to them
to be able to say,
let's do some really fancy service maps.
Let's do some really fancy things in different areas.
And that's a really interesting thing
that I don't think we push as much
in the Open Telemetry community,
the ability to do multiple exporters
so that you can either use them for different things
or you could do a trade-off and say,
right, which one's right for us?
I definitely like that idea of getting different vendors
to actually...
It's like with the cloud platforms now as well,
with Azure, AWS, Google Cloud,
where they've got to compete,
otherwise they're going to just lose all the customers
to another one, like with all these new services,
whatever they are.
And also another thing I like about Kubernetes as well,
that makes it so much easier for each of the cloud providers
to have their own Kubernetes thing,
but it's quite easy to take my architecture
and put it on someone else's.
If I'm not happy with AKS, for example,
or the way Microsoft do it,
I can easily swap it over.
And it sounds like it's very similar here.
And also you touched on earlier about
sometimes you might want to send different types of telemetry
to different places,
maybe logs to one place and traces to another place.
So it just feels like this.
Again, going back to this choice.
Yeah.
The open telemetry term is signals.
So we call them signals.
So traces is a signal, logs is a signal,
metrics are a signal as well.
Yes, I see, that makes sense.
So just so I understand this SDK thing again,
and I'll keep on going back to it,
but if we move away from .NET then,
and look at another language like,
I don't know, Go or Node,
when I'm using Spons there,
am I directly using the open source telemetry SDKs
in that case?
OK, so .NET is unique.
And the reason that .NET is unique is
because we already had this kind of stuff,
so ETW and all of those things in the past,
they already existed.
So .NET is very much about backwards compatibility,
which is why that exists in that particular world.
We also, as a community of .NET people,
which I don't like,
we don't like taking dependency on third party libraries,
especially as part of core dependencies.
So .NET is unique in that way.
The rest of the community are not doing that.
They are using open telemetry directly
in a lot of their libraries.
The Go community has welcomed this with open arms.
JavaScript is, it works,
but there is still some nuances
with the way that that works.
Java is very much on the auto instrumentation bandwagon,
so they're very much reliant on the agent
to inject a lot of these things,
but it is still using open telemetry directly
rather than bringing in sort of BCL type stuff.
I think it's really because .NET,
we're reliant so much on the BCL.
So the base class library that is produced by Microsoft,
which has all of those bits of functionality in it.
Whether that's a good thing or a bad thing
is open to debate
and that's probably an entirely separate podcast.
But that, I think, is where .NET comes from,
is that we don't want to take those dependencies.
In my blog post about this,
I talk about, for me, use activity directly
if you're writing a library,
something that's going to be used by other people,
even if it's an inner source type project
where it's even people inside of your organisation,
use something like activity
because it leaves things a little bit more open
and means you're not bringing in that dependency.
If you're writing an application from scratch,
use OpenCelemetry.
If you're writing an actual application
that does something, use OpenCelemetry.
Because that's going to allow you things like baggage,
a lot of the other concepts that are existing
within the OpenCelemetry SDKs,
so the auto instrumentation and things like that.
Use those things.
Whether you should use Tracers inside of your application
or whether you should use Activity,
it's a personal choice thing.
For me, the reason to do it
is if you're a polyglot developer
that's jumping between lots and lots of different systems
in lots and lots of different languages,
it could make sense to just use the OpenCelemetry shim
so that you're jumping between them going,
oh yeah, Tracer.createSpan.
Tracer.createSpan, Tracer.createSpan,
in all of those different languages,
and it might help with that idea.
But if you're really just writing .NET applications,
Activity's good enough,
and you can add as many attributes,
you can do all of the things that you need to be able to do.
Yeah, that makes total sense.
I think my confusion was
when I started reading about OpenCelemetry
and they were talking about all these SDKs and everything,
and I was thinking, well,
I'm kind of using OpenCelemetry,
but I've never seen these,
so I got confused about who actually uses it.
But as you say, if kind of .NET is slightly different
and with other languages,
you would be using those SDKs,
then that makes total sense.
Yeah, the SDK for Microsoft,
well for .NET,
is really about your startup class.
It's really about using services.addOpenCelemetry,
configuring what things you want to monitor,
and then configuring where you want to send that data.
So saying, I would like to monitor SQL client,
and I'd like to see the command text,
I'd like to see the connection times,
I'd like to see x, y, and z,
and then choosing where you want to send
that particular bit of data.
So do you want to send all of your SQL data to one source
and all your other data to another source?
I see, I see.
So you used the term baggage before,
and just in case the listeners don't know what that means,
am I right that that's pretty much a key value per
that persists across an entire trace or transaction?
Okay, so baggage, as I put,
I've wrote an entire blog post on this,
because baggage is the foot gun you never wanted,
was the title of the article.
So it's very much a concept that people misuse
and misunderstand.
Baggage is a W3C standard
that is the idea of transmitting additional data
between your process boundaries.
So we talked about this idea of microservices
being one type of process boundary,
or even just using different services,
using service orientation,
it doesn't have to be a microservice,
but transitioning process boundaries.
Now what you might want to do when you transition
between those service boundaries is send additional data,
that would allow you to then manage
what happens in that other service.
Now what people see is that baggage is just the equivalent
of a span attribute that transitions over every single span
in your entire process.
That isn't what baggage is done,
it's what a lot of people use it for.
I'll be honest and even honeycomb have perpetuated this idea
by creating a span processor that just takes baggage
and adds it to every one of the spans seamlessly,
which is not the right idea in my opinion,
but it is what's been created.
Baggage is incredibly dangerous,
because all baggage is a list of strings,
it's a key value pair string of string,
which means that you can send any data you want.
It also sends it on every single process boundary change.
So that means every single HTTP request
that your application makes
will provide all of that baggage.
Now if you imagine the problem with that could be,
well what if I put a user's first name on there
because I actually want the first name
to be available in that second system.
That sounds like a reasonable thing to do
if you've got two services that you want to do.
Now what happens if then that first service
also decides that it needs to call out
to a third party service to get some other data?
Well that's transitioning a process boundary,
because that third party service is another process boundary,
which means that now your third party service
has got the first name of every single one of your users
that's been hitting your system.
So now you've got data leakage problem.
That's problem one and hopefully
we're trying to make that a little bit better.
Problem two becomes that it's just a string.
And we all know that developers love to abuse systems
and I have literally seen somebody go,
oh I need this object in this other system.
So what I'm going to do is I'm going to do
jasonserialiser.serialise object into baggage.
And all of a sudden now all of our HTTP requests
have now got an extra 500 kilobytes of data
that has been transmitted between them.
Just so that somebody can deserialise it on the other end.
Now it's not the wrong usage of baggage
because baggage is just a string.
But it does mean it's dangerous
and it does mean that it will be misused.
So to me, never use baggage.
But if you are going to use it, be really specific about why
and be really cautious about where that's going to be sent.
Yeah, I totally see what you mean.
It's like the example you said where it gets sent to one place
but then also you might not realise
it's also being sent to another place,
which might be a third party provider
and you might have really sensitive information,
maybe way more sensitive than first name.
Yeah, it's a very good, very good point.
So I think it's important that people understand
what they're doing when they're using the stuff and how it works.
And it's probably important anyway
because as you said earlier,
developers really should own a feature
and changes to production.
So they kind of need to understand how to observe it as well.
So they kind of need to understand this stuff anyway.
But it sounds like all this makes things so much easier
than if you were trying to do this kind of stuff yourself.
Yeah, I think the biggest win with open telemetry
is the cross-platform nature of it.
It's the idea that if I want to send a trace
and I'm in a polyglot environment
where some things are going to go services,
some things are running Python,
some things are running .NET,
then I don't need to think about
how does one system communicate with another system
it's telemetry data?
How do I make sure that?
Well, if I want to use Honeycomb,
how do I make sure that I can use that in Java?
I can use that in .NET, I can use that in Go.
Well, because we're using a protocol for this,
you don't need to worry.
As long as there is an implementation of open telemetry
inside of that language,
then you can send your telemetry data to any of the services
that accept open telemetry line protocol data,
OTLP.
As long as they accept that, then you're golden.
But it also means that if I want to use something called...
So let's step back a little bit
because I think I did miss the idea of something called propagation.
So the idea that if I'm running a trace from my top-end service,
my front-end, my SPAR app with its BFF,
in the front of there, which is going to create our initial trace,
how do I get that data into the next service down,
which is calling into an API?
And that's something called propagation.
So the idea that I'm going to send over my trace ID
in the olden days, that would have been correlation ID.
It's now used the term trace ID.
I would send over that.
I would also send over what my parent span was.
And that's where we get that causality.
So it's this specifics process in this specific system.
This is the thing that caused us to call this other system.
And that's all called propagation.
And it's also where package comes in as well,
where we'll use that propagation system
to propagate that baggage over all of those different services.
That protocol is a W3C standard.
And all of the open telemetry SDKs
that you will install in your applications
are compliant with that methodology.
So again, this idea of it being cross-platform
means that, well, yeah, I could absolutely write something
in every one of my applications that inspects the headers
and says, oh, yes, I'm using FUBAR for my trace ID field.
So, oh, I write something in every one of my...
You just don't want to do that.
You want to just install an SDK
and know that all of that trace propagation
is all going to just work out of the box.
And that is simplifying so much of our industry.
When we talk about using microservices,
when we talk about being polyglot,
when we talk about using the right tools for the right job.
So Python might be the right job for the thing that you're writing.
Well, just use that.
It supports open telemetry,
so you're not going to lose any of your production visibility
by choosing a different language.
And you get that trace right through the system,
regardless of what the service is.
Yeah, that really makes sense.
So you've used the term agents quite a few times before.
And if I just think of a typical service running on its own,
where it might be...
Let's think of one example,
where it might be pushing logs or something to a third-party service.
I drop an application in sites, for example.
Or I know I've worked on systems where...
So Kubernetes environment,
where we've got something like Permethius installed,
which is scraping against an endpoint on my API
to get the metrics from there.
And then Permethius is like a data store,
which is something that Gryphana might visualise, that data.
This term agents in this context.
If we look at the bigger picture,
what does an agent look like and where does that sit?
OK, so there's a slightly different concept.
We'll talk about agents,
and then we'll talk about something called the collector,
which is a slightly different concept.
Agents are about things that sit alongside your application,
or they wrap the startup of your application
and inject stuff into your application
for auto-instrumentation purposes.
The reason why they're really useful
is if you don't have the ability to change the code,
to add in the SDK,
to add those 10 lines of code
that we were talking about from the .NET environment.
If you don't have the ability to do that,
because maybe it's not your application,
or if you're running in a mode
where you've got a separate dev and ops team,
and your ops team run the application,
it might be that the ops team still want tracing data,
but they don't have the ability to change the code,
because the dev team are the ones that control the code.
Agents come in that scenario
where they will inject into your process
the ability to extract that data.
So to be able to say, in the .NET world,
it will inject into the CLR the ability,
some IL essentially that says,
extract all of these activity sources for me
and send them to this location.
That's where agents come in,
because it's really interesting in that scenario
to be able to say,
I don't want to change the code,
but I do want to get that level of visibility
to see what SQL query it's doing,
what amount of time is it spending in serialisation
in the ASP.NET frameworks.
So where would that sit in?
Is that part of your build pipeline?
Is that just an extra step after compilation?
Or is that something that's...
I'm just trying to visualise where that actually sits?
Yeah, so that's an executable that is done at runtime.
So that's something that you would do actually
when you're running your application.
So think about this,
if you threw over your application over the wall,
so let's imagine that we're in the 80s
and you write your application and you throw it over the wall
and somebody else will deploy your application.
It's the people who would deploy their application would say,
right, we're going to run it
and then I'm going to attach to that application
X, Y and Z profilers.
So think of it like the profiler
that you would attach to your application
when it's running.
So like in a Kubernetes environment, for example,
would that be a sidecar container
or would that be something actually having to sit inside my container?
It would actually sit inside of your container
because it's a process that needs to have access directly to the process
to do that.
So in the sort of APM world within .NET
and obviously we're both .NET developers,
the clues in my Twitter handle, artin.net.
So in the APM scenario for .NET,
what you would generally have done if you were running full framework
is you would run an application alongside your main application,
which would inject into the app domain
and it would inject,
do maybe do a wrapping of each one of the methods
to be able to extract some data out.
That's the agent scenario.
So you don't make changes to your code,
but you still get that same level of visibility out.
Now, the second concept is something called the collector.
Now, when we talk about an agent,
we're talking about extracting that data from your application.
Now, extracting data from an application
could have memory implications,
it could have performance implications,
especially if you try to do things in a synchronous way,
where you're actually dependent on now your third party telemetry provider
to respond quickly enough
to be able to say that your request can finish.
Or if you're using a batching process,
it could be your third party provider
has been down for the last hour.
So your application is now backed up
with a billion different telemetry calls
that it needs to send that data off.
And now we've got a memory issue inside of that application,
but all of that's happening inside of that process.
And now we've got a problem that we're going to have to drop some events.
Now, OpenTelemetry,
we've done is creating something called the OpenTelemetry Collector.
Now, the collector is a service that you can run inside of your infrastructure.
You can run it as a sidecar in Kubernetes.
You could run it as its own pod entirely in Kubernetes.
You could run it inside of ECS, for instance,
in Amazon or inside of Container Apps, in Azure.
I'm sure there's something that you can do in that Google thing
that they do, that cloud.
I believe they've got a cloud.
Don't really know it, but who's Google?
I don't know.
Some search system, I think, a bit like Lucine, I think.
But yeah, they would have inside of that environment,
you would have a collector.
The collector can receive all of your spans in real time.
And then can distribute those to your third party providers.
So it works as a kind of internal cash or an internal proxy.
It can be really good from a security perspective,
because if you're talking about a really hyper secure environment
where let's just use Kubernetes as the example here,
where you've got lots and lots of pods
and you don't want any of those pods to have external connectivity to the internet.
But you do want to send telemetry data.
One of the really cool things about the collector is,
well, you can run your collector inside of that environment.
You can have all of your pods pushing their telemetry data
to that one collector instance.
And you can actually use that one collector instance
to then push all of those logs out to one source,
which means you can limit your egress,
you can limit where things can go just to that particular collector,
which makes things really cool.
You can also use service meshes and stuff like that to have the locations.
Becomes a really scalable environment inside of Kubernetes to say,
I only want my telemetry data going to one particular source.
It's also really cool if you don't want all of your applications
to have your API keys to all of your telemetry providers,
because you can just centralise those inside the collector.
I see. Could you say that the collector is almost like a buffer
for your telemetry data?
Or does it keep as in, presumably,
that's got to then purge data as it's pushing it out?
Yeah, so it does a couple of things.
There are what's called pipelining or processes.
So that you can, as your traces come in, you can enrich them.
So in a Kubernetes world,
one of the really cool things it does is it will actually enhance
the span data with your pod ID, with the namespace that that exists in.
It can also do things like redactions.
So it can say, you sent me over a load of stuff,
but on here it's got the person's national insurance number,
because we're in the UK.
If you want to internationalise this, social security number for the people in the US,
which is really, really sensitive data,
what you can actually do is you can say,
you know what, inside of my centralised collector process,
I'd like to remove or redact all of those social security numbers
or national insurance numbers from that data.
So it provides more than just it being a buffer.
It is a pass-through.
So it should take those spans in and then send them straight back out again
once it's run through its processing stuff.
It can also do, there is a tail-based sampler,
but we don't recommend people use it.
So that's the idea of sampling data
to say I only want one of these to come through,
but I want one of them, but tell them there was a thousand of them.
So that's the idea of sampling that data.
It can do those.
It's not a great experience inside the collector at the moment.
It doesn't work particularly well,
but I mention it because it is mentioned in the open telemetry docs.
So just be careful with that and look into what it does.
It sounds so, so powerful.
So you mentioned service meshes before.
I'm kind of thinking like, for example, take one service mesh linkerd,
if I install linkerd, it installs like Prometheus, Grafano and things.
Is there any work going on with service mesh providers?
I think you got SDO, linkerd, open service mesh and a few others.
Is there any work going on with those that when you install the service mesh,
you're also getting open telemetry stuff like these collectors as part of it?
Yeah.
The open telemetry collector can actually use Prometheus endpoints
to scrape its data.
So that's already a thing that it can do.
So things like Leakadie that will have a Prometheus scrape endpoint.
Because it already has those, you can actually use the collector
and just point it at your linkerd instances
and it will extract all of that information out.
So it will do that discovery and find all of those things.
Envoy Proxy, as recently announced, they've got a PR in now for actually exporting traces via OTLP.
So you can actually then configure Envoy Proxy
to be able to send that tracing data to a collector
and then outbound to other providers.
So what we're seeing is a lot of these providers,
whether it's the ones that do service meshes,
the ones that do metrics and all of that kind of stuff,
they're all gravitating towards open telemetry as being that mechanism.
They're all gravitating to using the open telemetry line protocol
as their export mechanism for their traces, for their metrics, for their logs
so that they can go through a collector
and then they can go to whatever sources we need.
We don't need proprietary stuff anymore.
We don't need to say that we're going to use Provider X on day one.
One of the things that I used to talk about quite a lot
when .NET Core started was the introduction of iLogger.
It's one of my favorite things in .NET
because I don't know about you.
But when we used to start projects maybe 10 years ago,
the first six months of the project was deciding which login frame
what we're going to use.
Are we going to use N-Log, are we going to use Log for Net?
And which IOC container you're going to use as well for the same thing.
Exactly, exactly.
And what was really interesting about that was the reason why you had to make that decision
is because every piece of code that you wrote
to get that visibility, you needed to use that library.
Now when iLogger came along, that killed that debate.
Right, we'll just use iLogger and we'll decide where we're going to put the data later
because we don't need to make that decision now.
We'll just depend on iLogger.
Hopefully with OpenCelemetry, with the OpenCelemetry line protocol becoming such a big thing
is we can do the same when we talk about traces, when we talk about metrics.
Because all we do is say, well, we'll just use OpenCelemetry,
we'll decide where we put it later.
We'll just use OpenCelemetry and we'll push it to the collector
because then we can push it even further down the line
because it's not until the collector is built and the collector will manage where all those things go.
So it's really making engineers more productive by saying
you don't need to have those debates right now.
It's probably annoying developers because those debates were always really interesting
and it delayed us actually doing real work.
But it is going to be hopefully a productivity boost now
because everybody's going to be standardised.
When you go into a new organisation or a new project,
you don't need to say, oh, well, I only know how to use LogFanet.
So how do you do this in the N-Log,
which were all the things we used to have in the past?
I mean, now it's just you, SerriLog, obviously,
because everybody uses SerriLog, but obviously we're hoping that then
people gravitate towards the same with OpenCelemetry.
Yeah, it sounds that way.
As you say, with the whole dependency injection,
stuffin.net, in iLog, all this kind of stuff,
you just have to think about it now.
It's all follows the same pattern. It's quite neat.
It was interesting when you mentioned about the announcement for the Envoy Proxy.
Ironically, the service mesh we've mentioned the most so far,
LinkoD, is probably the only service mesh that doesn't use the Envoy Proxy.
All the other ones use Envoy Proxy, but then LinkoD has written their own.
A really lightweight specific proxy.
But hopefully they do something as well.
LinkoD, actually, does use Jager as its mechanism within the collector.
So one of the really interesting things in the collector
is that you have something called receivers.
And receivers are where we receive data in.
Some of those are push and some of those are pull.
So the push ones were obviously our OpenCelemetry line protocol.
So our SDKs in our applications or our agents push data into the collector
in OpenCelemetry line protocol format.
And you set up receivers in the collector to do that.
One of the receivers that you can set up is to receive Jager tracing data.
So LinkoD, the one that you've mentioned quite a lot,
would also be able to use the OpenCelemetry proxy, which would then convert them out.
My hope is that obviously projects like LinkoD,
which are very popular, will move towards the OpenCelemetry line protocol
like everyone else, and then we will have a really standardised approach everywhere.
Awesome, awesome.
I feel like I need to rename this podcast to like a Kubernetes podcast or something.
It comes up that often.
So going back to the observability side of things,
obviously if you've got dashboards, we can go and view charts and...
Oh, you mentioned the dirty word.
You mentioned the dirty word of dashboards.
Oh, no.
But something you've actually got to physically go and have a look at.
This concept of alerting, does that form part of the story?
Or is that something that goes into the tools?
The actual vendor tools?
So observability in general is the idea of what can I learn about my system
from the outputs that it delivers me.
It's based in science.
It's a scientific concept of control theory, I believe.
But that's where it comes from.
When we talk about alerts, triggers, that kind of stuff,
they're all based on those signals that you get out of your application.
So that's a layer on top of being able to alert when things happen.
What I would say on that is that, again, things are evolving.
We used to just say, did I get more than 500 errors in the last 10 seconds?
OK, go and get somebody out of bed.
The problem is when you're talking about high load systems,
it's no longer relevant.
Those 500 requests just could be somebody sending us duff data.
It might not be really important to us.
There's a movement towards something called service level objectives,
which use service level indicators to be able to say, give people budgets.
I can have 500 errors every day, and that's perfectly fine.
I can have 500 errors providing they're distributed over different users.
Or I want to be able to say that a certain number of users have been affected
before this particular thing is triggered.
And what that's doing is helping us in the on-call scenarios of developers
to be able to say that we don't want to get somebody out of bed for noisy alerts.
Those noisy alerts are the reason why people hate being on-call.
If you can get rid of noisy alerts, being on-call is basically free money
when people pay extra for people being on-call.
So triggers in general are great,
but try and augment them with SLOs
to be able to say that it's not just X happened.
It's that X is going to cause you a problem in the next couple of days.
It's this idea of saying to your product owner,
if we don't fix this now, in four hours it's going to be a problem.
So why don't we fix it now?
If we don't fix it now, it's going to be a problem in five days.
So we should really get something on the backlog and get it sorted right now.
Because I can tell you that they're going to have a problem.
So it's providing the engineering teams with their ability to go back to product owners,
to the business and say, we have an SLO.
We have an objective that we've worked together to do.
I can tell you that this SLO is going to be breached in the next five days.
So I really need to sort something now.
So that feature that you wanted is going to be delayed a little bit.
Because otherwise it's probably going to be delayed anyway,
because I'm going to be up in the middle of the night sorting this thing.
So you're going to lose two days of my time.
Et ça me fait total sense.
Mais encore, ça me semble un problème très compliqué.
Ça me semble assez facile à dire,
mais pour en faire des règles business,
on va dire que ce n'est pas une alerte qui devrait vous attendre en milieu de la nuit.
Et que vous pouvez dire que ce n'est pas une alerte qui devrait vous attendre en milieu de la nuit.
Mais comment vous déterminez programmatiquement
si quelque chose est important ou quelque chose que vous pouvez attendre?
Ça semble vraiment un problème compliqué.
C'est absolument possible.
Et c'est pourquoi j'ai hâte de l'idée de l'autoroute,
j'ai hâte d'avoir l'idée d'automatique.
L'observatrice pour moi est de ne pas être purposeful
sur ce que vous faites.
Ce n'est pas que les agents ne soient pas installés,
et que l'OPS team peut regarder des dashboards.
Parce que les dashboards ne vous permettent pas de la visibilité.
Pour moi, l'observatrice est de avoir une conversation avec votre système,
une conversation inscritique,
où vous vous demandez comment vous faites.
OK, quelque chose va bien, mais où?
Dis-moi où, qui vous a hâte?
C'est l'idée d'avoir une conversation avec votre système.
Automatique n'a pas vraiment donné vous ça,
mais ça vous donne des choses basées.
On a vu une grande quantité de gens
qui ont fait l'exploitation de Honnico.
Le premier niveau d'automatique que ils ont fait,
on le appelle MTTWTF,
qui est le temps de la main pour quelqu'un qui ne devrait pas faire ça.
Et l'automatique peut vous donner un certain niveau.
C'est ceci, l'instrument manual qui vous donne ce niveau.
Vous parlez de la complication.
Vous pensez que c'est compliqué pour un système auto-instrumenté
de voir votre système et savoir ce qui est important.
C'est à vous de dire que ce sont les choses importantes.
C'est à vous de dire que je vais donner
une spannerise pour tous les choses.
Je vais ajouter les deux milliers de attributes
pour les spanners,
parce que c'est important de savoir ce qui se passe dans ce service.
Je vais samplez des données,
je ne vais pas décharger les providers.
Mais c'est à dire que je vais pouvoir
prendre 2000 attributes pour jouer avec.
Je sais tout ce qui se passe dans mon système de production.
Je sais que si quelque chose se passe,
ce sera ces 5 attributes.
Je vais mettre un monétaire
sur ces choses pour voir si elles vont faire mal.
Oui, c'est ça.
C'est un peu comme ce que nous avons dit avant
que les développeurs ne voient pas de choses à faire.
Parce que si c'est encore le
modèle de la production,
si on se met sur les opérations de l'eau
et que le travail est terminé,
ils ne vont pas être motivés
à faire cet instrumentation dans l'application.
Mais si le développeur
sait que si quelque chose se passe,
ils vont devoir investir en ce qui s'est passé.
Ils vont peut-être être réveillés en nuit,
et ils veulent trouver des problèmes
comme bientôt possible.
Il y a plus de motivation
pour faire
cet instrumentation,
partie de votre définition de faire
et partie de développer ceci.
Oui, c'est un très important point.
Le développement s'est impacté
par ce niveau de visibilité
et de l'observité de leur plateforme.
C'est vraiment important.
J'ai une histoire très contente
de mettre un développeur
comme le point de contact
sur la équipe de support
pour une semaine entière sur la rotation.
Le travail est d'aider
la équipe de support
et quand la équipe de support
a des questions,
ils sont les pointes,
ils vont s'y aller dans le salle avec eux.
Et ce qui était très intéressant
était que le producteur de biotec
était l'amount de visibilité
qui était ajouté à l'application
sans personne en demandant.
Je lui ai ajouté ce extra...
C'était tout pour des logs.
J'ai ajouté les 5 loglines
et j'ai créé un petit peu de vue
dans l'application
pour que vous puissiez voir les loglines
sans aller dans le database.
Ce niveau de visibilité
dans les premiers jours
de ce mouvement de observabilité
les gens se faisaient ça
comme des produits biotec
parce qu'ils étaient affectés par ça.
Et quand vous avez un team de ops
qui ne contrôlent pas ce niveau de visibilité
et un team de devs qui n'ont pas été affectés
par ce niveau de visibilité,
vous vous enduriez avec les systèmes de support.
En faisant le travail de développeur
pour pouvoir vous voir ce data
et en les faire être impactés par ça,
c'est-à-dire que ce qui se passe
est que vous vous en avez à un niveau massive
de visibilité.
Avec cette whole
telemetrie électorale,
c'est un peu plus populaire
que ça.
Est-ce qu'il y a quelque chose de nouveau
dans la pipeline
et dans le roadmap?
Absolument.
La telemetrie électorale
est en train de faire des cylindres
dans plusieurs de différents moyens.
Donc, le protocole,
la protocole électorale électorale
pour les logs
a été finalisé quelques mois
et ça a été implémenté
dans les STKs.
Maintenant,
il y a des projets
sur Autoinstrumentation.net.
Il y a aussi
un mouvement en faisant
un brum
ou l'idée de pouvoir
voir une tour de l'end-usage.
Et comment ça se fait partie
de la telemetrie électorale.
Comment nous commençons à penser
sur les sessions
comme un premier classité,
par exemple,
parce que c'est très différent
de faire des tracés de l'HTTP.
Donc, il y a un groupe d'interessants
qui travaillent sur ce spécifique
et de comment nous standardiserons ces choses.
Et ce sont tous les committees publics.
N'importe quoi, ce n'est pas fait
en cas de l'end-usage.
Ce n'est pas fait par des vendeurs spécifiques.
Évidemment, il y a beaucoup de vendeurs
qui ont un intérêt investi
en en déterminant que ce sont
toutes les choses qu'ils utilisent
dans ces spécifications.
Mais comme je l'ai dit,
c'est le projet n°2
sur les contributions et l'activité
des contributaires.
Parce que je sais que c'est maintenant
le podcast des Kubernetes.
Je vais mentionner les Kubernetes encore.
Mais c'est le projet n°2 sur les CNCF.
Donc, il y a beaucoup de choses qui se passent.
On est toujours en train de voir plus de gens
qui sont invités
en aidant pas seulement les bugs
et la code,
mais aussi, qu'est-ce que nous devons faire ?
Nous voulons des utilisateurs
pour venir aux committees
pour nous dire ce qui se passe,
pour nous dire comment nous pouvons l'improver.
Mais il y a beaucoup de choses
que nous travaillons sur.
Nous avons fait un truc à KubeCon en Espagne
quelques mois plus tard.
Nous avons regardé ce que la priorité
devrait être pour ces équipes
pour voir.
La réel utilisation de la monitoring
était une grande partie de cela.
Donc, oui, nous absolument
voulons plus de gens être invités.
C'est génial.
C'est un moment incroyable
de faire ça.
Donc, nous sommes
presque en train de faire un moment.
Mais avant de commencer la recording,
vous avez fait un grand grand
truc de papier.
Je n'ai pas compris ce que c'était.
C'est un petit feu.
C'est ce que nous referions.
Oui, c'est un petit feu.
C'est ce que nos ancêtres
ont été en train de prendre une
chelon et de apprendre des choses.
C'est un livre.
Non, je ne comprends pas.
Je n'ai pas compris tout ce qu'il y a
dans cet épisode, mais maintenant je suis confusé.
Donc, oui,
quelques gens de Honeycombe,
donc George Miranda,
Lisbonne Jones et les majors de la Chérité
ont écrit un livre
sur la monitoring et l'engineering.
Et ça a été un long temps de faire.
C'est un livre qui a été créé,
mais c'est finit.
Et oui, j'ai un livre que je peux
envoyer comme un livre
actualement tangible pour un de vos
listeners pour pouvoir lire et comprendre
beaucoup de ce que les concepts fondamentaux
sont qu'on a parlé aujourd'hui.
Fantastique.
Donc, je pense que normalement,
quand nous faisons ce genre de prière,
on dit normalement que les listeners
peuvent envoyer un tweet,
utiliser le hashtag
et prendre l'exception
pour que je puisse trouver le tweet.
J'ai mentionné le podcast,
j'ai mentionné le livre,
j'ai mentionné Martin et Honeycombe.
C'est beaucoup de choses à m'en rappeler,
je vais inclure les liens à tout ce que les notes de la chanson.
Et peut-être, dans une semaine ou deux,
je vais prendre un en randonne.
Et puis on va avoir ce type de chanson.
Ça ressemble absolument à ça.
Et j'aimerais entendre
d'autres questions de vos utilisateurs
et des trucs comme ça.
Donc, utilise les mêmes handels
et achetez-vous.
Fantastique.
Donc, je vais inclure les liens à toutes les moyens
pour vous contacter dans les notes de la chanson
et les liens à la chanson,
et tout ce que nous avons parlé aujourd'hui.
Donc, on va faire des picks de la chanson ?
Absolument.
Donc, une des choses que j'ai travaillé
sur dans l'espace de l'opinion de l'amélioration de l'Eventurier
que je parlais de,
c'est de les utilisateurs de la chanson beaucoup plus.
Donc, une des choses que nous voulions pouvoir faire
c'est de voir les traces
localement
sans avoir à installer
et faire un composé de docker
et mettre un Jager, etc.
Donc, je suis travaillé sur un outil
pour l'opinion de l'amélioration de l'opinion de l'Eventurier
qui a été utilisé un outil de package called Spectre Console
qui est un outil d'amélioration
un outil d'amélioration pour provider des outils d'outils réels.
Et j'ai vraiment aimé
juste essayer de faire des choses qui semblent vraiment intéressantes
dans l'outil,
parce que les traces sont vraiment réelles.
Et de pouvoir voir ce genre de charte de l'eau.
Donc, de pouvoir produire un table
et de pouvoir avoir l'exact nombre de votre console
et de pouvoir obtenir le nombre de votre console
et de pouvoir faire tout ça.
C'est juste un projet vraiment cool
et l'API est juste tellement facile à utiliser.
Oui, on avait, en fait,
en mai 14, Patrick en parlant de Spectre Console
et, comme vous le savez, c'est absolument incroyable.
J'ai joué avec ça beaucoup plus tard
et j'ai même fait un talk sur ça.
Dans le talk, je n'avais pas écrit tout ça.
Je l'ai juste dévoilé la documentation
et j'ai essayé de faire des choses différentes.
C'était un peu de frein.
Et, comme vous le savez, c'est juste tellement facile à utiliser.
C'est un peu de base de la documentation.
C'est comme des meilleures documentations que j'ai vécu,
juste pour pouvoir faire quelque chose de très rapidement et de très facilement.
Donc, je vais inclut le lien dans les notes de la show.
Donc, votre tool, est-ce que ça vient de Docker ?
Je suis désolé pour ça.
Je suis en train de jeter.
Ça s'amuse bien.
Et je vais définitivement inclut le lien dans les notes de la show.
Je suis plus célèbre, comme dans la concentrate.

j'aiै 190 cl池ces et isolated.
Tu regarde et tu écoute et tu écoutes.

tu vois, c'est quand même plus en require.

Tu as écouté un adresse.
Mais tu n'as pas découvert les instaurations ?
En unique.
Le site de l'application que l'on a appris est super utile pour aider à écrire ces queries.
Donc, je vais vous le donner dans les notes de la show.
C'est absolument cool.
J'ai toujours l'air de l'entendre JQ comme j'ai l'aimé d'ailleurs.
Je faisais quelque chose avec de l'enviendre de Jason et de la filtrer.
Je pense que c'était sur la commande GitHub pour essayer de filtrer sur les noms de repo.
Donc, l'entendre comment faire JQ syntaxe a toujours été un peu de bugbear.
C'est vraiment comme les tools de regex que vous utilisez.
Vous pouvez faire le regex dans le browser. C'est un cool tool.
Mon défi pour le regex est le regex101.
C'est absolument fantastique.
J'ai utilisé le regex buddy.
C'est un tool de desktop qui vous aille payer pour le licence.
Mais maintenant, vous pouvez juste le faire au browser.
C'est vraiment utile.
Je dirais que c'est un peu mieux que les regex.
Il y a 4000 différents esclaves de regex.
Toutes les différents tools font de différentes manières.
C'est pourquoi, quand vous regardez les tools de browser,
ils ont un défi.
C'est comme, quel de 4000 tools vous utilisez aujourd'hui ?
Mais oui, JQ est assez cool.
Le regex101,
qu'est-ce qu'il a ?
Flavours.
Il y a toujours un, deux, trois, quatre, cinq, six,
il y a encore sept.
Mais le dotnet est dans ça.
Le dotnet est sur le liste.
Pourquoi le dotnet a sa propre flèvre ? C'est ridicule.
Ça va passer à notre conversation plus tard, n'est-ce pas ?
Oui, je pense qu'il y a des choses spécifiques
sur le dotnet, sur comment ça se tracera.
Je ne comprends pas le regex.
Je peux faire l'opinion de la telemetrie,
je peux faire le tracage,
je peux faire les systèmes distribués.
Le regex est où je dérange la ligne.
Je peux les faire, les lire dans la même matière.
Même si je l'ai écrit un week-august,
c'est un endroit où je ferais des commentaires condamnés.
Cool.
Donc, je me demande à ce point
où est-ce le meilleur endroit pour les listeners
pour vous atteindre ?
Mais on a déjà couvert ça.
On va mettre tous les links et tout dans les notes de la show,
including Honeycomb,
aussi, pour que vous puissiez vérifier ce tour.
Et avec ça, c'est juste un grand merci pour vous joindre.
Je suis vraiment en train de récolter ça.
Et ça n'a pas été déjeuné.
J'ai vraiment aimé ça.
Donc, merci.
Pas de problème.
Merci pour avoir moi.
J'aime vraiment la conversation aussi.
C'est génial.
Merci pour vous joindre.
C'est le podcast de Kubernetes.
Oh, ça va être un peu dur.
Je pense que ça a été déjà pris.
Oui, je pense que ça peut déjà être un podcast de Kubernetes
quelque part sur Internet.
Je pense qu'il y a un ou deux.
Mais il y a un appel qui je suis un éclairant.
En fait, je vais mettre ceci dans les notes de la show.
Cool.
Et merci à tous d'avoir écouté.
Et je vous remercie de ce podcast
sponsorisé par Everstack,
qui est mon propre company,
qui a donné des services de développement et de consultation.
Pour plus d'informations, visite Everstack.com.
Et si vous aimez le podcast,
s'il vous plait,
me laissez me séparer le nom sur les médias sociaux.
Je n'ai pas utilisé le hashtag
d'un handel d'exception.
Et je peux être trouvé sur Twitter.
D-R-A-C-A-N.
Et mes D-O-M-S-A-O-N.
Mon blog d'Angloch.com
a des liens à tous mes trucs sociaux.
Et oui, comme j'ai mentionné
quelques fois avant,
nous allons inclure un lien à tout ceci
dans les notes de la show,
qui peuvent être trouvé
sur le podcast de Kubernetes.
Oh, je veux dire,
c'est-à-dire,
le podcast d'Angloch.com.