Sampling data at a fixed interval means we can potentially miss data
from events occurring between sampling intervals. For example, say we
sample data every 15 seconds but Unicorn workers get killed after 10
seconds. In this particular case it's possible to miss interesting data
as the sampler will never get to actually submitting data.
To work around this (at least for the most part) the sampling interval
is randomized as following:
1. Take the user specified sampling interval (15 seconds by default)
2. Divide it by 2 (referred to as "half" below)
3. Generate a range (using a step of 0.1) from -"half" to "half"
4. Every time the sampler goes to sleep we'll grab the user provided
interval and add a randomly chosen "adjustment" to it while making
sure we don't pick the same value twice in a row.
For a specified timeout of 15 this means the actual intervals can be
anywhere between 7.5 and 22.5, but never can the same interval be used
twice in a row.
The rationale behind this change is that on dev.gitlab.org I'm sometimes
seeing certain Gitlab::Git/Rugged objects being retained, but only for a
few minutes every 24 hours. Knowing the code of Gitlab and how much
memory it uses/leaks I suspect we're missing data due to workers getting
terminated before the sampler can write its data to InfluxDB.
Where a vew is called from doesn't matter as much. We already know what
action they belong to and this is more than enough information. By
removing the file/line number from the list of tags we should also be
able to reduce the number of series stored in InfluxDB.
This gives a very rough estimate of how much memory is allocated during
a transaction. This only works reliably when using a single-threaded
application server and a Ruby implementation with a GIL as otherwise
memory allocated by other threads might skew the statistics. Sadly
there's no way around this as Ruby doesn't provide a reliable way of
gathering accurate object sizes upon allocation on a per-thread basis.
Without this it's impossible to find out what methods/views/queries are
executed by a certain controller or Sidekiq worker. While this will
increase the total number of series it should stay within reasonable
limits due to the amount of "actions" being small enough.
Suppress e-mails on failed builds if allow_failure is set
Every time I push to GitLab, I get > 2 emails saying a spec failed when I don't care about the benchmarks and others that have `allow_failure` set to `true`.
@ayufan mentioned creating a summary e-mail to prevent getting one e-mail per build, but the latter might actually be desirable. For example, I do want to know if Rubocop errors fail right away.
See merge request !2178
Every time I push to GitLab, I get > 2 emails saying a spec failed when
I don't care about benchmarks and other specs that have `allow_failure` set to `true`.
Since filtering by these values is very rare (they're mostly just
displayed as-is) we don't need to waste any index space by saving them
as tags. By storing them as values we also greatly reduce the number of
series in InfluxDB.
This reverts commit 7549102bb7.
Apparently I was wrong about
ActiveSupport::Notifications::Event#duration returning the duration in
seconds, instead it returns it in milliseconds already.
Tuning of metrics data to store
This removes data we don't really need, as well as making sure we don't overload any cache stores or databases.
See merge request !2265
This will be used to store/increment the total query/view rendering
timings on a per transaction basis. This in turn can greatly reduce the
amount of metrics stored.
This particular setup had 3 problems:
1. Storing SQL queries as tags is very inefficient as InfluxDB ends up
indexing every query (and they can get pretty large). Storing these
as values instead means we can't always display the SQL as easily.
2. We already instrument ActiveRecord query methods, thus we already
have timing information about database queries.
3. SQL obfuscation is difficult to get right and I'd rather not expose
sensitive data by accident.
Nokogiri produces inefficient XPath expressions when given CSS
expressions such as "a.gfm". Luckily these expressions can be optimized
quite easily while still achieving the same results.
In the two cases where this optimization is applied the run time has
been reduced from around 170 ms to around 15 ms.
While it's useful to keep track of the different versions (Ruby, GitLab,
etc) doing so for every point wastes disk space and possibly also RAM
(which InfluxDB is all to eager to gobble up). If we want to see the
performance differences between different GitLab versions simply looking
at the performance since the last release date should suffice.
This removes the need for Sidekiq and any overhead/problems introduced
by TCP. There are a few things to take into account:
1. When writing data to InfluxDB you may still get an error if the
server becomes unavailable during the write. Because of this we're
catching all exceptions and just ignore them (for now).
2. Writing via UDP apparently requires the timestamp to be in
nanoseconds. Without this data either isn't written properly.
3. Due to the restrictions on UDP buffer sizes we're writing metrics one
by one, instead of writing all of them at once.
Storing of application metrics in InfluxDB
This adds support for tracking metrics in InfluxDB, which in turn can be visualized using Grafana. For more information see #2936.
See merge request !2042
Yorick Peterse
Robert Speicher
Stan Hu
Robert Speicher
Douwe Maan
Douglas Barbosa Alexandre
Dmitriy Zaporozhets
Sytse Sijbrandij
Patricio Cano
Drew Blessing