How to use System.Diagnostics.Metrics and dotnet-counters to measure and observe a Nethermind node
Nethermind can be configured to publish its metrics using System.Diagnostics.Metrics. This mechanism is a native tool embedded in .NET Platform. It allows for a low overhead monitoring and reporting. Once .NET Platform metrics are enabled, they can be monitored and collected using
dotnet-countersand other tools.
Reporting metrics as System.Diagnostics.Metrics is enabled by passing and additional argument
--Metrics.CountersEnabled trueto the Docker containers,
./Nethermind.Runner --Metrics.CountersEnabled true.
This flag can be configured separately from Using Prometheus and Grafanaas this two reporting modes are treated separately.
Metrics reported by a Nethermind node follow the module convention. Whenever there's a module
X, its metrics will be reported under meter
Evmmodule will be repoted under
Nethermind.Evmand so on.
dotnet-countersis a tool provided by the .NET team to monitor and collect metrics for further analysis. The usage of it is different when used on the same machine or in the Dockerized environment. To learn more about the tool, please visit the official documentation page of metrics collection with dotnet-counters.
When a node is running on the same machine,
dotnet-counters, given that the .NET runtime is already installed, can be installed with the following
dotnet tool install -g dotnet-counters
This will install the tool globally and will allow the user to monitor and to collect metrics from any .NET process that is run on the same machine. For further information how to monitor and collect, please refer to the original documentation of this command.
When running in a Dockerized environment, the most common way is to create a separate docker image for .NET diagnostics. This can be done with the following
FROM mcr.microsoft.com/dotnet/sdk:7.0 AS base
RUN dotnet tool install -g dotnet-counters; \
dotnet tool install -g dotnet-trace; \
echo 'export PATH="$PATH:/root/.dotnet/tools"' >> /root/.bashrc
Once it's built, as
dotdiagimage, it will enable running
The second part is connecting the dockerized node with the
dotdiag. Whether using
docker coposeor images run manually, it's important to remember that
dotnet-counterscommunicate over a named pipe (Windows) or an IPC socked (Linux, macOS). To make it work, volume mapping should be provided so that the two images share the directory used for the communication. Similarly
pid namespaceneeds to be shared between them.
Let's visit an extract of a
docker-compose.yamlthat would provide such configuration.
- ./dotnet-tmp:/tmp # /tmp is used to create the IPC socket, expose it as ./dotnet-tmp
# ports omitted as they are not changed
# make counters enabled so that reporting happens by setting the flag
# the created dotdiag
stdin_open: true # docker run -i, so that it runs
tty: true # docker run -t, so that it runs
- ./dotnet-tmp:/tmp # map to the same directory, to make IPC socket connection
pid: "service:execution" # make pid namespaces are shared - processes are visible
- execution # make the dependency explicit