From 790c68680893b333c0a9bedca7c528147b8417b1 Mon Sep 17 00:00:00 2001 From: Yorick Peterse Date: Mon, 18 Jan 2016 16:55:14 +0100 Subject: [PATCH 1/5] Added documentation for GitLab Metrics --- doc/README.md | 7 ++ doc/metrics/gitlab_configuration.md | 12 ++++ doc/metrics/influxdb_configuration.md | 96 +++++++++++++++++++++++++++ doc/metrics/influxdb_schema.md | 75 +++++++++++++++++++++ doc/metrics/introduction.md | 55 +++++++++++++++ 5 files changed, 245 insertions(+) create mode 100644 doc/metrics/gitlab_configuration.md create mode 100644 doc/metrics/influxdb_configuration.md create mode 100644 doc/metrics/influxdb_schema.md create mode 100644 doc/metrics/introduction.md diff --git a/doc/README.md b/doc/README.md index 7d4f84857e..b9ee7de534 100644 --- a/doc/README.md +++ b/doc/README.md @@ -49,6 +49,13 @@ - [Test Clojure applications](ci/examples/test-clojure-application.md) - Help your favorite programming language and GitLab by sending a merge request with a guide for that language. +## GitLab Metrics + +- [Introduction](metrics/introduction.md) +- [GitLab Configuration](metrics/gitlab_configuration.md) +- [InfluxDB Configuration](metrics/influxdb_configuration.md) +- [InfluxDB Schema](metrics/influxdb_schema.md) + ## Administrator documentation - [Custom git hooks](hooks/custom_hooks.md) Custom git hooks (on the filesystem) for when web hooks aren't enough. diff --git a/doc/metrics/gitlab_configuration.md b/doc/metrics/gitlab_configuration.md new file mode 100644 index 0000000000..3b44b4394f --- /dev/null +++ b/doc/metrics/gitlab_configuration.md @@ -0,0 +1,12 @@ +# GitLab Configuration + +By default GitLab Metrics is disabled. To enable GitLab Metrics and change any +of its settings open a web browser and navigate to +`http://YOUR_GITLAB_HOST/admin/application_settings`, the settings can be found +in the "Metrics" section. A restart of all GitLab processes is required for any +changes to take effect. + +## Pending Migrations + +When any migrations are pending the metrics are disabled until the migrations +have been performed. diff --git a/doc/metrics/influxdb_configuration.md b/doc/metrics/influxdb_configuration.md new file mode 100644 index 0000000000..c67c0f1510 --- /dev/null +++ b/doc/metrics/influxdb_configuration.md @@ -0,0 +1,96 @@ +# InfluxDB Configuration + +The default settings provided by InfluxDB are not sufficient for a high traffic +GitLab environment. The settings discussed in this document are based on the +settings GitLab uses for GitLab.com, depending on your own needs you may need to +further adjust them. + +## Requirements + +* InfluxDB 0.9 or newer +* A fairly modern version of Linux +* At least 4GB of RAM +* At least 10GB of storage for InfluxDB data + +Note that the RAM and storage requirements can differ greatly depending on the +amount of data received/stored. To limit the amount of stored data users can +look into [InfluxDB Retention Policies][influxdb-retention]. + +## InfluxDB Server Settings + +Since InfluxDB has many settings that users may wish to customize themselves +(e.g. what port to run InfluxDB on) we'll only cover the essentials. + +### Storage Engine + +InfluxDB comes with different storage engines and as of InfluxDB 0.9 a new +storage engine is available called "tsm1". All users _must_ use the new tsm1 +storage engine (this will be the default engine in upcoming InfluxDB engines). + +### Admin Panel + +Production environments should have the InfluxDB admin panel _disabled_. This +feature can be disabled by adding the following to your InfluxDB configuration +file: + + [admin] + enabled = false + +### HTTP + +HTTP is required when using the InfluxDB CLI or other tools such as Grafana, +thus it should be enabled. When enabling make sure to _also_ enable +authentication: + + [http] + enabled = true + auth-enabled = true + +### UDP + +GitLab writes data to InfluxDB via UDP and thus this must be enabled. Enabling +UDP can be done using the following settings: + + [udp] + enabled = true + bind-address = ":8089" + database = "gitlab" + batch-size = 1000 + batch-pending = 5 + batch-timeout = 1s + read-buffer = 209715200 + +This does the following: + +1. Enable UDP and bind it to port 8089 for all addresses. +2. Store any data received in the "gitlab" database. +3. Define a batch of points to be 1000 points in size and allow a maximum of + 5 batches _or_ flush them automatically after 1 second. +4. Define a UDP read buffer size of 200 MB. + +One of the most important settings here is the UDP read buffer size as if this +value is set too low packets will be dropped. You must also make sure the OS +buffer size is set to the same value, the default value is almost never enough. + +To set the OS buffer size to 200 MB on Linux you can run the following command: + + sysctl -w net.core.rmem_max=209715200 + +To make this permanent, add the following to `/etc/sysctl.conf` and restart the +server: + + net.core.rmem_max=209715200 + +It is **very important** to make sure the buffer sizes are large enough to +handle all data sent to InfluxDB as otherwise you _will_ lose data. The above +buffer sizes are based on the traffic for GitLab.com. Depending on the amount of +traffic users may be able to use a smaller buffer size, but we highly recommend +using _at least_ 100 MB. + +When enabling UDP users should take care to not expose the port to the public as +doing so will allow anybody to write data into your InfluxDB database (as +InfluxDB's UDP protocol doesn't support authentication). We recommend either +whitelisting the allowed IP addresses/ranges, or setting up a VLAN and only +allowing traffic from members of said VLAN. + +[influxdb-retention]: https://docs.influxdata.com/influxdb/v0.9/query_language/database_management/#retention-policy-management diff --git a/doc/metrics/influxdb_schema.md b/doc/metrics/influxdb_schema.md new file mode 100644 index 0000000000..a8e69b5e17 --- /dev/null +++ b/doc/metrics/influxdb_schema.md @@ -0,0 +1,75 @@ +# InfluxDB Schema + +The following measurements are currently stored in InfluxDB: + +* `PROCESS_file_descriptors` +* `PROCESS_gc_statistics` +* `PROCESS_memory_usage` +* `PROCESS_method_calls` +* `PROCESS_object_counts` +* `PROCESS_transactions` +* `PROCESS_views` + +Here `PROCESS` is replaced with either "rails" or "sidekiq" depending on the +process type. In all series any form of duration is stored in milliseconds. + +## PROCESS_file_descriptors + +This measurement contains the number of open file descriptors over time. The +value field `value` contains the number of descriptors. + +## PROCESS_gc_statistics + +This measurement contains Ruby garbage collection statistics such as the amount +of minor/major GC runs (relative to the last sampling interval), the time spent +in garbage collection cycles, and all fields/values returned by `GC.stat`. + +## PROCESS_memory_usage + +This measurement contains the process' memory usage (in bytes) over time. The +value field `value` contains the number of bytes. + +## PROCESS_method_calls + +This measurement contains the methods called during a transaction along with +their durations and a name of the transaction action that invoked the method (if +available). The method call duration is stored in the value field `duration` +while the method name is stored in the tag `method`. The tag `action` contains +the full name of the transaction action. Both the `method` and `action` fields +are in the following format: + + ClassName#method_name + +For example, a method called by the `show` method in the `UsersController` class +would have `action` set to `UsersController#show`. + +## PROCESS_object_counts + +This measurement is used to store retained Ruby objects (per class) and the +amount of retained objects. The number of objects is stored in the `count` value +field while the class name is stored in the `type` tag. + +## PROCESS_transactions + +This measurement is used to store basic transaction details such as the time it +took to complete a transaction, how much time was spent in SQL queries, etc. The +following value fields are available: + +* `duration`: the total duration of the transaction. +* `allocated_memory`: the amount of bytes allocated while the transaction was + running. This value is only reliable when using single-threaded application + servers. +* `method_duration`: the total time spent in method calls. +* `sql_duration`: the total time spent in SQL queries. +* `view_duration`: the total time spent in views. + +## PROCESS_views + +This measurement is used to store view rendering timings for a transaction. The +following value fields are available: + +* `duration`: the rendering time of the view. +* `view`: the path of the view, relative to the application's root directory. + +The `action` tag contains the action name of the transaction that rendered the +view. diff --git a/doc/metrics/introduction.md b/doc/metrics/introduction.md new file mode 100644 index 0000000000..007fff9e7b --- /dev/null +++ b/doc/metrics/introduction.md @@ -0,0 +1,55 @@ +# Introduction to GitLab Metrics + +GitLab comes with its own application performance measuring system as of GitLab +8.4, simply called "GitLab Metrics". GitLab Metrics is available in both the +Community and Enterprise editions. + +GitLab Metrics makes it possible to measure a wide variety of statistics +including (but not limited to): + +* The time it took to complete a transaction (a web request or Sidekiq job). +* The time spent in running SQL queries and rendering HAML views. +* The time spent executing (instrumented) Ruby methods. +* Ruby object allocations, and retained objects in particular. +* System statistics such as the process' memory usage and open file descriptors. +* Ruby garbage collection statistics. + +Metrics data is written to [InfluxDB][influxdb] over [UDP](influxdb-udp). Stored +data can be visualized using [Grafana][grafana] or any other application that +supports reading data from InfluxDB. Alternatively data can be queried using the +InfluxDB CLI. + +## Metric Types + +Two types of metrics are collected: + +1. Transaction specific metrics. +2. Sampled metrics, collected at a certain interval in a separate thread. + +### Transaction Metrics + +Transaction metrics are metrics that can be associated with a single +transaction. This includes statistics such as the transaction duration, timings +of any executed SQL queries, time spent rendering HAML views, etc. These metrics +are collected for every Rack request and Sidekiq job processed. + +### Sampled Metrics + +Sampled metrics are metrics that can't be associated with a single transaction. +Examples include garbage collection statistics and retained Ruby objects. These +metrics are collected at a regular interval. This interval is made up out of two +parts: + +1. A user defined interval. +2. A randomly generated offset added on top of the interval, the same offset + can't be used twice in a row. + +The actual interval can be anywhere between a half of the defined interval and a +half above the interval. For example, for a user defined interval of 15 seconds +the actual interval can be anywhere between 7.5 and 22.5. The interval is +re-generated for every sampling run instead of being generated once and re-used +for the duration of the process' lifetime. + +[influxdb]: https://influxdata.com/time-series-platform/influxdb/ +[influxdb-udp]: https://docs.influxdata.com/influxdb/v0.9/write_protocols/udp/ +[grafana]: http://grafana.org/ From d4c05766c7a058cbf62186723216d754eebad92e Mon Sep 17 00:00:00 2001 From: Achilleas Pipinellis Date: Tue, 19 Jan 2016 22:18:51 +0100 Subject: [PATCH 2/5] Refactor GitLab Metrics docs [ci skip] --- doc/README.md | 8 +- doc/integration/README.md | 1 + .../metrics/gitlab_configuration.md | 39 ++++ .../metrics_gitlab_configuration_settings.png | Bin 0 -> 45148 bytes .../metrics/influxdb_configuration.md | 192 ++++++++++++++++++ .../metrics/influxdb_schema.md | 54 +++-- doc/{ => integration}/metrics/introduction.md | 29 ++- doc/metrics/gitlab_configuration.md | 12 -- doc/metrics/influxdb_configuration.md | 96 --------- 9 files changed, 285 insertions(+), 146 deletions(-) create mode 100644 doc/integration/metrics/gitlab_configuration.md create mode 100644 doc/integration/metrics/img/metrics_gitlab_configuration_settings.png create mode 100644 doc/integration/metrics/influxdb_configuration.md rename doc/{ => integration}/metrics/influxdb_schema.md (60%) rename doc/{ => integration}/metrics/introduction.md (66%) delete mode 100644 doc/metrics/gitlab_configuration.md delete mode 100644 doc/metrics/influxdb_configuration.md diff --git a/doc/README.md b/doc/README.md index b9ee7de534..bee9ab0590 100644 --- a/doc/README.md +++ b/doc/README.md @@ -49,13 +49,6 @@ - [Test Clojure applications](ci/examples/test-clojure-application.md) - Help your favorite programming language and GitLab by sending a merge request with a guide for that language. -## GitLab Metrics - -- [Introduction](metrics/introduction.md) -- [GitLab Configuration](metrics/gitlab_configuration.md) -- [InfluxDB Configuration](metrics/influxdb_configuration.md) -- [InfluxDB Schema](metrics/influxdb_schema.md) - ## Administrator documentation - [Custom git hooks](hooks/custom_hooks.md) Custom git hooks (on the filesystem) for when web hooks aren't enough. @@ -74,6 +67,7 @@ - [Reply by email](incoming_email/README.md) Allow users to comment on issues and merge requests by replying to notification emails. - [Migrate GitLab CI to CE/EE](migrate_ci_to_ce/README.md) Follow this guide to migrate your existing GitLab CI data to GitLab CE/EE. - [Git LFS configuration](workflow/lfs/lfs_administration.md) +- [GitLab Metrics](integration/metrics/introduction.md) Configure GitLab and InfluxDB for measuring performance metrics ## Contributor documentation diff --git a/doc/integration/README.md b/doc/integration/README.md index 5edac746c7..5ba0d0fb24 100644 --- a/doc/integration/README.md +++ b/doc/integration/README.md @@ -15,6 +15,7 @@ See the documentation below for details on how to configure these services. - [OAuth2 provider](oauth_provider.md) OAuth2 application creation - [Gmail actions buttons](gmail_action_buttons_for_gitlab.md) Adds GitLab actions to messages - [reCAPTCHA](recaptcha.md) Configure GitLab to use Google reCAPTCHA for new users +- [GitLab Metrics](metrics/introduction.md) Configure GitLab and InfluxDB for measuring performance metrics GitLab Enterprise Edition contains [advanced Jenkins support][jenkins]. diff --git a/doc/integration/metrics/gitlab_configuration.md b/doc/integration/metrics/gitlab_configuration.md new file mode 100644 index 0000000000..019db1ad02 --- /dev/null +++ b/doc/integration/metrics/gitlab_configuration.md @@ -0,0 +1,39 @@ +# GitLab Configuration + +GitLab Metrics is disabled by default. To enable it and change any of its +settings, navigate to the Admin area in **Settings > Metrics** +(`/admin/application_settings`). + +The minimum required settings you need to set are the InfluxDB host and port. +Make sure _Enable InfluxDB Metrics_ is checked and hit **Save** to save the +changes. + +--- + +![GitLab Metrics Admin Settings](img/metrics_gitlab_configuration_settings.png) + +--- + +Finally, a restart of all GitLab processes is required for the changes to take +effect: + +```bash +# For Omnibus installations +sudo gitlab-ctl restart + +# For installations from source +sudo service gitlab restart +``` + +## Pending Migrations + +When any migrations are pending, the metrics are disabled until the migrations +have been performed. + +--- + +Read more on: + +- [Introduction to GitLab Metrics](introduction.md) +- [InfluxDB Configuration](influxdb_configuration.md) +- [InfluxDB Schema](influxdb_schema.md) diff --git a/doc/integration/metrics/img/metrics_gitlab_configuration_settings.png b/doc/integration/metrics/img/metrics_gitlab_configuration_settings.png new file mode 100644 index 0000000000000000000000000000000000000000..14d82b6ac98c5c942b93a83d17f6f82987a5c7b3 GIT binary patch literal 45148 zcmdSBcUaR~*DZ>os3^^bfRv4H5fD&%FN%OjFVaCpdWn<}X+gw-ARy9v?~$w)^> z$8<+a-H48ko}Z5H@YCZA;FbG?ZWcN^KDs;VY9>$8S8y=c+4ch>nN&__TAm`)^+IqY z1?oyuQoEf`+4!U-p%mKp~v~}@DFF`tHlIb zayw}#F%ritn^YyZOG_Kdma#S?Bo*a@U-m6u|NV%c+f>ocpu6&~M`5!JTln=G|9TWC z&mQ5A6VR8gt*zDh50Bevh5A`MJmKZGO&D+sbgDNzFg34LkI?=^9apEllpp=q)208l zr2`x_?)hyf3VP|G18x zujOuU3f$=)DI4ns7kygnF5h9=m6UbS$L6$5v82y;CY1SQK0Lxt5@N?}O@16O>JOk@ zt8bGilloKuS&8dnx~zXEEVJ`yWcMevTitglw!*TkDe@jyLXU&@_22S!;S&$&MmVm$ zg>}c5NCeFf482>F=;~Inds6>h_0h`56Gy-=OQ*^*P!Rep91?QI}`#uMcc%o}|=6m+NqeqcV>c8s-*shdCb3g~~|9a1H{c)9r zt*Hm6n9Vkm`sVIQvF3;EDJzrp*qXXJjEp6>0{lzRTMI;{PEFlZPcOB@sfE75Q7-i5 z6{VvC#ut=w>H~K?dv~T()x#Du_#W*)S6RWw@F;!w+&YF2avHjN2~Fp=D|w=sEG5+kE}`k^($88i4VubK?vLwoy$ zAllHGcm8Cp_nD6lM!omMF_4j*+j#0)H6_G=Z-YSQJ?2;MU3fb((Q!WJO!$$gLnrp8 z?%XnY>P49JA3{)76$t8|BQrn#Bn4gpL#f)uYV+}PHdefBMwz+v*SFbffl)BaxA^H} zkDg3=G*66B{d(tCU|`@Qcv>1F?M{eZ?V;C+9Fo60svpc0ewzhjJZ-*OtEvu3F@lx< zc{dR&a7=GT`6?^NHsMvB+bGy4_8th~!)yg=S#IuH)|`PK0{fInyX&rK8V(f~@ zBU{Td^1=AgQO|}v!Q?|=ptr){Djd7hDJw~%`sv;9^Jl}Yot>-HA~SR26U@eCOcn(r z7!U}HhNx-;+`F5o2bfZ zW4HvLxwV=1jBx0OlvHiToLtbyUU(9OkDu}CGk8~iCE$zG;DLtZYgN7Zg07amYG{2OnL-5`*ra7qb{$Ou~$qYU4hp+o_Djh0pY5HJ&{k zu7w-0pKCWnkJ~*SLv1Y+rtI#Gl|A38XP9>Tv?p_^Stf92HKtnyjNZ=rcl=AxD4?$nc8s}IJ@kd&{U(c}7Mzo2jw4nWKr1|?B_@KXo)Bhz;&EBa# z`elYs1u-bHdCG_kqKIy5W2cco-rcPiirXggh7v>^wqEEW;(V-QJaS5l|)!eN=l?uVtCkhIZ)`-)aeyu@zz;XMo=PgN%awY zmY+n4QE{Rw2j%xNTxm5hZrMEocD(Ti)kn9tN^cHkl+K!tWN_YrXXTKEN8yo{`Z?GR zr`<|sb;rbnv>W#APShzR^{M~th}vxnnfWtxbiw)^hhsr(!BkqJ^7iJeh{;XlZ6|cd zueF@j=@A5V*|GmSQnz)o(;XRAyDXjj`PS48TCU*O#pxHW56*sbn0zOy;@|0urF>nq zE!_VVJomGUlCK$putE>K;gNp#N-cC}7fML$52bB79QM$|_9|k?a(6gWG2Yo^hWr)x?zy zex&|}glnv{tveC&{eGd>pl4^Nu=^$44aZ4{Sj+ku_sk(S=WaI#om{L z?!ECafOJYGBW&ClZQ4ZevrvKYK;$S<0PquS)$J0cL2h9 zD8eg{!Kluo6!0jlU>&X7%^>8pb7$C}bxZxSyZ_OjSwVXR_5J<#Ev<;DQf^~u0EC#& zo}i-($a;5_e<|XJY#^c0<5XE}k;0f*SwXD^PubF zY(vl=q2Xi*OnJn7n7X_y_6SWw0zDvc-WpmmOAIJw6A)umUXd1)e1sk{PPs`R`w<(n zF;slRQLlCX8HwYbpBHE2lNzG3vrh54n13e+47d3 zQsDCHj=Ri<;@O!(pS8CS=E@M77)tTeYkC%~B16ThVBQ6&aQ%?!cmX=PgyqnJ5cdC~O%?8)=Jp}a`-_`^lW%i1`e}7JP+@ayb>rxoOzdt~*cWg}b)DTDbH2aMfjpb5eFDe5dOgQTBq=0P$&o zQpH0c!40h*W-|;^$LS5agSl-DH`D!57-!z&nKmYDGV~b#UXDCd z=RNy-OkFX`yZnZHr(t#ro=tUS`)*bZJ1Y{9#$CS^FmVbG9OiKlufZQk4>W~$tEtoP z>o?w&3+JQy2JU-ArL1vDW{B@jbU&wA` zq^#N=w%@rN7NcB!2ozik87K$7+?R=*N}LN7NaCAc+s71Sk+roWI+^k<%{R2{>4vMk zP~t~LW)g`TmU=iF-yMug|K9H(CM|sP{FS1Uk(^M}OohfHbIX=aaUrn)Z2V|}=lxr+ zg73E*exzr^)gOFs(f4SUtg>Xn=Z9*g2mf(5@fwnTb8Gr=lv_Xd{^d|D<@VSA+RbOS znmETTiP@iTty)#X)2??gJcn=S7Lm7_tm-_gKZi#nKawxIH_L`x|M)fSH+<#l|w`#e`81Yp>gI2U@(T8fCD6xv)Yh6B9 zi9oL>QZim=kEQvYqwP7m_`|=epQpoYzX#Rg{=ej@5p1^eoUx6K$9YlfWamL~%k$u7 zlz2mr8x=jB#?-nMHZ>^?_--wkR(sEOzRUOj3vGPHM(;J7%gf7mC%>QRxl3V47BqIs zuft_d&*(Qxo+Am}%{WCz$4XBdriklK7^~CZ2RKOlxcwPBF=#4}3fwwg5@yPm>nofc z?naucn2VCZOM}#P14f(@0Z3A`1x`7Zia_u7uHYI}9%r)K1&o{u+__c#q(hvvxj6I* z+PvSXIy5?(N%9Bimej{?D=pF0X;kfYQ45*&khWN65xr$Yv8oZaKjVnA-=>zKFxgtY*Z)NJwA+a7Y`?hvw z^+YpM+T!f*Vg*+p8>(fMRxdtl=CdDiz&o+nhA9O|U1}WaWEb@?3h{(g0qdR$4uL7i zLzikeqyoyD?yB7XeqSP3!$J>sNkk>sF@@?RN&Gnf%pUZxdgLg6I5QsC*zyE*S}3R z`ga}Jn8CUmxA$0wFaa`Er%dNCg>ei!ic6ja&{G$FITi`<5cWl^n!glwP4pJrP3mE{ zgWDJCrKNzfE@9%Q;E>No1ytUlVD(^a#@Xtgu5M38XP$`9wFI;7YV6ohh8J3|$_Ddj z6~#oO5mpW^(m>daI8jh4@f>E$?>(VDyt?#d4c1eiBbGKA*PeOo50;);JiL>EPvBsT z>QRZXem8a@HbVjCgf^o$jp}WAwd_uhDsTKEy}`{T?I~`BT&-dnqm@jYb%>OVhV7r% zRD8$DbKi4Xwii`<3enx@16-e^v~4Gx=t&pDv6IF@>H4UXM=i`~=2&mCsCI0EpqfIp z%;obk!$Arb_ZmFgun+J?_c4%4>fCYXlh40G*_Naw+Q|v48azF#P7<@M8SWO%avOEv zJP=2F$J%5KPBW{2>C_?lXN6xSLeQj8Ij`w_CF%XnTZb|-n`sqb=-aw7QHP$MF%QhS zZkG_x&iIL1H{H-(ut{4r`JN5Sg?`IuAy*%2FQM)y)x&-?y6o>Rqr=`(4??;tGOQxR zHNIk1m1qz~OM%H;4EkrP@eP&7{sW(!nTEIF2!-W;FpZz>(d(|}ngt3%87~<4wc|3S zog47MDngPX(Q09@Fh$3v{Amyhpa{%|LF1DusUe@g{8`w1@JzR?k$ljelAS%Gvv2P$ z&H7rXI$QNo$6zTy6g#)BXqil8AP~g(%D^4Mgs9Wf#_SZR-L+UiWZ2m`6MN5xP?2=N ztaz|PO(R|YIa-HNhxBNl4wtlcuZc1#Vk-?PE4`N*i>E%~T0&V>dfrs)>WayO-xwsuNe=m^Hx&8Nbi%Z2hRv7Esd>Q0t;Zdw{$nmsMBNdk=jgC!DuB53P<=^#39sX`Q#wP_xNR$Ux@q^ItwwT7P?ZAn? zNxY1tbq#EXh>vL=vRzSC-!(_)Tp=Ud{65mh?ww?k?ib3E=nL9z9g~alMT-xgIVLrn zF-F|$$rIJp&KuJ0NmBuYiaA@9z(n)R#5gC6@3g67b?4mp4Mk7G&~TLf(Sj~rtjXIv z3Lr-ZZ-M3Df2<6To^;mRArsyCa*+6{QknS5$ zyUrX?>|nN^({~b$hD$if?lf2`Hby+#ks>9 zym4Ea)v$jZa84fCx!>>VUriJxH@L;^Y&DhLtEi=~r(zmMU_8(sBl{45P24AbQ4+%y zSnIv=TA35aE--l=<2Q6D6d$<29YuQlETLFcy_vf1L~31Q^LyHuIAZnbtSq;Z-(+{X zsV5cTrN)WKMLs!1_jHpM3E_|)6zt$l33Y*KND`u97}9URfB%KN}dUqJ=6jc#45HhU3}vpJEzUbIbFj*Zu!JZ z)U%SnRQUZH&S=@t@FW@56T%Wvg1bV-bDc?Nu75qDe)dsmkW}w&F}`2WeRZ;ZgeCrG z_dMg31hb>p0KQ(+r9ofLBm7!6X8RxH!FCQf58~Qid@$`*D73adV9Rxy?UD}bVM@kyJXCs(2HhnL30Tap)0w>`kPLY|Nrr6eq}{di=-Xe zp;Ef`U$pqvfXn)z5NWjkf!vwU!dJ69SuNwr><1f1AKhD{|20}eN%Fi3JQBaa>#u0cB z9nCD^xbg&$6o(u;hdCVo;Y;3~N8C*kO$*sy{?mt6fy2x^%i;svngLfawtOa3E0wAL z;R0b%*yIL%Y(44x*UXUr3a9+ZD)?+FRScx*t$;^-;-33N>Fn?_VXkcLNY3=sy)9nn z=7TwfY8dh0lfP}yqN&6Rj-_Y%&#vyFBK_d83DYPA590bv`)xajQNdEG=?&!IzG8s& zHso;Zm#xttu;wer1P;emx<5q?dC&K9S39l-kg}4FB}12G_ImS*BtXq}Bka{Fi4kE+ zS6vw=43Kqlvj##1j5G=aD_|d8>8@^9?_F!m_aQgHG|9azo;IDT&3o#6g9%Iq=o%9< zM4i4SkKz+n**@YKs$_^x>U2YIrE85SqLg478V*xgf*gHn->zGQTbl}6=B|d}tBZ_* zE|411yym^#5&uk$xN+-cvzx1%%)jZDZ7M)G(v)MT#AP5;A40}PxFrxx%3lFfT^r(N zr@!ap>ezsAc-9SDxiQbAUf6d{A`i~vmK7oBq2Nbg?8XAlpa(EfDKiUD3FkGebn@s> zd9iQ0JQVYWQ)yIG20)tz=AAehD~)C8asT*eB!2f(V(XIBvN)E*HOT^1F60E~j_lp< z!~)*s@Q#`RiUOpOl#K}gy2*2mF=Xho;D@{AioBSOL3xH#K|?yfg5T8#ll`V>NfoD$ zIIgZeLAP~M9D~KKN@qxZoOf|^LoOT4bmFpojQ7rqSh`+0xG^E^j@ykb*1%wN=yjPi zr1~}BJ8>?Jz7sl`;z~sq;|q-zJ3+ix2GnVykA?pW5d}>XRZhQ)-@fX6FP_a(^na-Q zYt0zY?sY4$ybum2W$LsW`Leox+nh-^@oe}h?(Jr|Cks_fmaftihl4F5yzu?{JIGYq z8*)!2X|5;DZ_n}Tj$=>iR_x;VKkUfQEWnQNr~r1Pqk{zKq!j!1Xza`ajbWX6gxRzJ zgaz4=9Bs2LTMwsb=%HA6ORASmP+P$*tr~doQ+JU`y<_^yltCKqn2*c6vh+Gfr0+MS z(q-E)`lwH7cBi@z{lULifEN^381p^C>LD}AfI8|!8-v)akk=<0izQ0@F&x?BFZjqC+|a+ z0(HIGyxB{bB-R;BKJyBj(~lRuF226TBE!RGS&`;xA2m?EBSlIBf_i+p4Zl!f!~)6!vvtNDF}dS+UtL9*nhN)PBZhovFMnnPO}2dXFhiOsw%h62sUV5>Q#LzQFIO-W+gYmH z+#Y^wgsjZ=-@JTl+0E0F5Rb0f|LT%2*w3NzgJme>6PM$cuPPp9e-vFjuKp~BUMhJE zQrvM+L3m7O`D`L; zoldc$?+5{k5Yy$6;)8`au@CK`+q(i;Ovv`vocp}+O@F`ZHJtNHmH9>1jMqn$2*?w0 z@<;jQ{f5uoSu}(RG8OwVIEAbYIQdZ@4ZWMtx6FFR#jOc|3g)T6HSPe;LJuAQbLJlG zWCquH**f!y?=QznKVG{FD-u7Yg|y>A{|z;yorf;4HU0d&s8JkzMlHicqk0*Rh1t+djyIR z+>v`$_F0<@LfqAyK|Bf3K($IUv*U7azYsH*B~DpsH!1C9h>oBVk`8a%Oi@jsZy|8qdl^x|0FKVATk&Hy-& z0^kFqz>^@)0V4R-KfrgUI5vT02NYePc15Be78Al=S^dx(sQ>cxKpWsLcHPpNd z)Y?-)H-etq=ut<yaE z>m;1g19K!!Q9?pvtwJEn zf!{2mtnUavFdp>u<+VFdMg91u^JjZA6zUF@DJ(6ILN+rr^_}!Gl6vgF*ZICa)kRP-n>?gBl*L#Av-Ko9c)R%ZQi8~`F|-+B)$Y@FJV(3j%017t36 zXJbeB8Lw>~c?g&#fH=}1P4?S_z)>Yv!0*tAvR=Qa!J*?2K5<}@D6I4I*@i?3J^?z} z{~Nef+dKiX$X?;?|0fo@aD7b|`)ez2v!dA6fvaz+Q|ie#1v!#3(4uv+EF2k_;Mht{ znF)#;o&6zpmcu&*@>N|!sCrJPS zW_dL^rfL>YwjLG`{QE9~iqan-@6M$46lFPlpfZJISXJY4C z-N|d;cWq#@0oaEAa=|JM+bsGdhLAH@x?i6W@W{gmhD~j#tK$%REKgdL-}lSL44~gg z00FHEuXIvd4Ex(f(~3DrRmMs+{pl-sfW23=q*m;eBfsaaAM=K)kxF5l-O z^O|_TqY2A4fz|MB#EqU!NzT9?=RaeZ112j&xvO$WseeLDB)3vI;Ms46WZ}-s(IQbb zTX^WV-{gw8i$PTmG!XxBYZOa#pyx%;1{(9_ivI!xrH`@|I;V}sS=6nZi`iq>=HGe@mHKE}Vr&be}i4LV|&H?nwmHtGDp&~fM? zBboQv{=*g8F1~1-;MHJ8`LoUo5&{sjO6wjtLT#NB|9#)~r5&GmnY*6fCqKb+msky} zaw=h+c^mNEjdP^Wnpr6&Lyj2Y*6F7Z%>!pUe1rYGC}~5Oo+d))bP28%C!4v@w*aki z+vS^?U5w7wxh^mz!MC%R)HQ<0VZm|l?1%g8S3h}5nQ(E6rqYxbuJEf1zkXK#kTw{w zN>tuo@TvBlz~eF-5k#9>*9s=aNtO8d)zkuf#f2Li$GC=nN%N?XXT8H^P6J7)k7on|C>j}L6Tr`8qKSp34WOkVGHB|J9vG%-T}(m$y(qu18A1kIM_k(6o!pJ=SwB3x>E$E+sr+T6AmIe!p`l z^tg7Z0t0_^*sJLAhPSJ$+@Yj&rNAlaQ-dFuFV;T$an1PcgetQOhs+)1G{hH4kQ9as+(?P5(v%^`?)Uol2>6h~cN46$Cq!dY3yqFX7A`YGG617FR1y9V(1Okk^<=$l!I0W!p#Z_7{yHs9g#yq;UH zyn;NBLtQPl&cs%~^#t!}QIvcI-Ou2MW!AneTR&!uaKy;onhzm?IT zBPMMtw0Gl&5}{>q4jI$re^Q~1laq6G*Ozz@Ooq)AbUE&I%gz00k;d0(jr*4pYnW#T zf(b7cS9iA*ioQfU995J`lg0*^&mT+$A6xbAT|aH=^otvNr;gWTUiM3GNtGMGgt~Q} zqtZn^SCLpV_OvRfysInHH{f*T@*9%H@b0;y?k`JCK}!#a@=2hKIa^*@_CD^_rBsTmd(c7H<95-@aKiO#<~52|pIvr0B96{=CY05kQit(>AvFF9 z=B?#MR%xSZw+Q${>V;mpyXhwlCMiFf;@;_Vd~|6@IGL2fC1dG6qf6E%Um>7sL&R(w ze+Is}9n70JU*)^IdI{ZTIbJt!D%xQ^>^IJcmsTuFavFS$Fy1}lA=Yu@$Jlk2yH>+z zo)y1*%Bg8->ynZrrw`n&XMj+&qqr5VG}J0lIhARL_7Pe!ADutRZk7V@EcWHlsE5L; zG~)^NX-}2%Uayn#MPDla(2U8bl6hw$W9Q~t_c7+7*I%kd% zkCa7E$6FXQB|(%{Y@B8ag#FU=n(}LU(xt{EtsCQ?*x1@wh-qd(459YQ4q1Kgr*+M8 zcB8AQd6SC${c~{zuf`r^+6`ylzmu}65Oy<+L2?hu)SoRXGW4z^Y;>l%p(rwSwCrS2bt6F; z1uPNM?(HJw?>ybys%>u8*iAZQJqr0KTou&yDNQD7>!QEtm^X6f=VN2QC0 z1L@F%>^1jfh>BxaJ$DHoUFE}H-edSpO85@nyS`l=zGv?u6g1S?n)J?cH_CfaranqJ5x(dx z56tvoda&}rm5k%Tnk~d<2+a~pgin&N3*HN@of9x?N&Yt25w^ozLvWO3I)$E%W8n9y zn>lZKr!2PGlX67@8c=V6em}~bt@}Y`@|=0I18V{@Xl(UA%$gQw_>-Z+S*Z1LWfB3d z+iOCofDzj;wuAQkr4I(gb0k)8+3zC+v_MP5Y^wXZZl3{vi?ac`&^X*wmbO&6Yf~T8IxT8~Eid(iKhvu~NRnbRFPE z6{IIMgF&g(1l_SWPq`UPAKQIYpP~}>1RYb&En|8fs;U2os!SYm4ycRvvsz!`PyBnq z@ueZV{c3|Ei~624FYBESe?BXB&8F^D(Ds8tRrTQ4P&JbsR;3i5VB+ zjlW8qw|VmSI{Q6yI+okJJi>&nvzPGj#(8z;w7jQo_kimui&K?T0Hrww6P-3U!l#6Tnxf8q zl}p$xxNEj@zs)J^F0T?1+x%nKdwu&A@^DwtsmU5Kc9D_~Sfk7>jX#Re?t@T)6M#3n zm?(-4j+bP84|~rk56jyyUbtcC+!8dgk+9ucc4_6p&C_`8KVF8`FdnQrWfkK+7f?oY|0&bo$bACSE znkjkhg5XJyo>^X({ z<;I$lC2UqXA@h7(Mz%ZEt-(zWQsDI@L{^8wkx|J)@PEg^U+V7bEquY~=nq0z3FF;_ zl);~kw_E%JqwP4p0p-KPcKE`210R`|jATsjql#VqnaYi4mrVvOk)r0M<0cxVAJa9b zymtPcUfg8*F4;DADn!t|7q~V7?r$N?&CCUfEt*iU( zbNgTJ{UE~#$dYzvx z-yk<>c@C!cg1gIAa>3e+yHKrhDVOG4*qBE)UBRH}WTTrKR zmVIkO=1tP-)V=J-xFo|MvMF1*isB=S3?-FQBwCgO%6L zE(pnFvD{|~!1>ta0(PJ;TW0hRo8TtTl@br;!))1+o5R|NaA}!qE^j^+beXLzpWE`) zw+Y+~JWFWaS2bIUW?PIL&d>9jm=oM?u;(; zO5MLlEM^JV;x_>NVO#Wo;m$dz{%g`Z=KLbDe?9^Vk^gD?_=4BqlbB%rU|tw3X&b_7 zB5LmHw=y|2i(S-hkA(3Sn7+JfJ?u5kzU`2%n79|TdV-u&HZ-0Q64RCjBd!~D&J+!| zd%pj;qY`>NAU*%wIM5bX#eQ5dT#8f5l1%b$@+x>woII}n?h@-A&7Af+{D#t>tc#ri z-_qKHI!g=;cO&E4Do?|#a3ZXCs^4gHfRI&j1BqzIOe4K#(y=?VDR((|ABD`6n&6JKL*rxlv~$M>7?9qEbO@| z6XG%D$$Mwd1wyB_;Q<;Avcv=VW$euz_;w*Uk4gtNaz<+b;u)^d5+ z*4Xup$@p4&?qpIxb+@#F$Lwgg^jIOQvjp)`QAyRyxays!6QDRuGb0AHbP9W>KHGN7 zS&qSI-sv)MF~V$SR1+m_+#7#3ge)yZV<{zy31-)>n<1o9S~R0mQ>y1Mlm-Yj$9(pJ ziL_dlCby>H?l%FE=$n1(*|T}kw}{Pew2J~coOoT}Bgdliw_erFByIEUrbp`WkVJ}# z6%!KrWZL!9I}dEKEc`laCUUTw!b6uy>@$idgjpcmL=gKx2{KxZowFg8cU&V^RrJ8w z=VgUvYqQ_lU+u+bY@3<~b9uYBDxLp=x#ETB*&U9Qx04=q@3rVltWa?0$*tqAqn=PV zI1Lc%^&;_eSwcuR-Jb35ip%mTllLlYA?HOa#gXC(ZJ^`J!(pNFQ|@R1{Z!zqKU_I7 zR;uSsQeVE*2s{jG%+7lNap;d$&01LzF`J7wa~z9k$4LtZ%A(y_H8Ogqz=E9a|^l+F~y+1DZ=+<8>SW|gu7pyWDg$Wx@YS( z6i@UzS9?$#9A9(lH}t;>8;>-iRauzL4V_fNQ9{n=uc`MEZfldFaL4vzJ^f8yV%=!xCfUnQ>%sPx%a#!J*{N&XSXmn#nf_ zj_AD>u~oGVId)vToAT`0grX|d%X$aX`2z|eH+ti$Hr5E2Ae^Ayw#>2if zO=(w(*7g^2)EJi5xW_TN=o;xmM-FUWswTiaS zJagpF;Fdpa4yu)fZS8F(t~eP6-)&ls@EIV9&X^4pPY>GE6whKY7|`M8u+3H`dYa%{ zChi+j*r_g1mL$_7(nnfp9vYsS^1R|OS`=Ag)xS-$L3BPOHix|?8I}gmpNAfDaYcC! z%~%KF(qBdtPmi@Wmj*;ptP++ZbnHJh*(MWt7*6_fNr=cEEnv;iFb| zNbJ{c%rA@i3fZBdi_c{3@7?ZIbbVaHb}_Lp`@;zZ_QuI}LlfnVJCSC8%zwmhoc6mY zk5`!GmdSVkeW1LhX@F_Dti(QZgkNp;tbl&OarFqV%;yQOV7&Fn#rQkb%R!I7%0;7E zey?#CzSAZuZE|(*cur2I$Iw>zXk_lwUz7C#*?)6ff2dqMGS9B4-Mq9QKAC1{ub;>NBwX*a~1wIbIUv~BEMb&OI zKNaGHv9ma(O28fz1qBYRTw89EC}lK-6*lhIWf23K9N1)j#}-3&R@}(Q20Yxbrr}f#f}Bm{OL*p3s+Vue?Aip1FFBcek*$qTffcSdF6&(ui-r<794=tWRN! z-@g7LJ@QVMpkAgnclcTF%&&qhL?<2UeK=c)Lu>gC1v0DVp?@y^uT*r;HKtxh>* z1%;#D$*F0N*X3=43}sXlW_YZK*^xV2o)voYmnmQG=K%6{hF+FSz3ejPT}jaJ{Ed$7 zCV>VaX4CU~*Y4~=k+%{e=0+}g;BNFg88*832}v-m?FHhvw?)l^KRr221gi>C6d~oP ztX)f-1U0g6yN(T;Z>SNky+vJ6K~q;=Ge~EY$d*wqQ`9$8t%aBp&^H?W%=%rh^`*rl1{5__}Hi@5PllJ#|5Zx!l#*50M7 zs}C6E*y9q7(-jxTOB4e5$rOJG-|1Mitjno1^jp$pJ^v>vL-M9qL}V0VD34G6YkJIq z>D^v<@%>vLpRMH@8 zLaE)++&G;W?yV;V6u}XsA$?vty=*Va_t}!VDf(W?(u0e03UysC zm9oTwt`W1eb|6}k1%Fz17p3QJ#i@=xI)g>|Hl1<~)k~@7xb!UG6AvZOpsGoL;V-w8CN4L2hZLy=`sGC&o@EgNf$} z7Ql?kBON<`PwjX6sr^>J z49+LzFVhrQ*RhU+k}rF*i2kRMiu=mOmuj~W6?Hnknn50O=>uI{(>$65&F8h6+uvs9 zIydyjog(5NbX`%Zo4$DRfk&L|*xb#f+lEN>*~R`JsPw)TT7It%{T2xn>ZgY%T|A_G z=0!T|cB#j@X}0qJ0YO8cq)y!gXtqw zB7a`G_;KYQFF>Aw%Mt#74d3035m6k&Q^Mb^_WxaNo~9rCZ#LrpH%tKkU#b3^r*w67 z9VvTTj;#^r-e2h{bC~{@Su%j(=y`yLq}pS6Wt{plnKJa!WgpDrsvg`bho1SzNT3v! z>76d;(^i8-wx{Z4DSITn4*N&BS>QNpA}J~Pm04X<3^+GMXhO@=%uv-MO^mdb7S%r1 zv*EH~LnlCmz)EX%**pbQ!;RUlo~j2eEBVwj;o0rKT^im%#rm zOB{}k58CfYDtu_`P!boL^usRLt@K{`H5X2x>;rH$zB19!M{DztaM`HQ!;c-*wgM+y z=uouCjVom`}@$be(3kM%s8>BAiptJyjphcacE&tt;!opFSce5nU zP_jvo)r;nYFH-&4#G?G3%nase*Vw?&&W=}sb)XR^#7b+%Ak7rpG1}@2UPF{#VHNxU z+Ey(sJuyX>Y3(mde&8c}oRklH19_S~wQghL9_5MscSz1to}{iN;A@wp?!i=i*7|>Z z+ojnj!x+#W|Ix^}9VnGOR)KeJi%z?5yCn0ead%X~k=RUm4|-j%S08XVfUd={RcX)z z(0qAcn($A(6b$uo_Z-H{-rX|m4ckw>(a&9*n(5ZXZ%B%tM!n>hPj<9U^>lTKX3 z=mH0&N54XDGp__eMq)3?tIjRC5F(DX1YH;uy+dCFvfwU57)-!eIp~o0_4OSm0tYNC zfPdPfn?2Q!h@$o4T~s!5qR&s&4Vk$Ajl-F{wnElraWyo&Q|~ho^x=&2mDlg0aOmnT zfJ_^}l{aM^W>Y%%nk(GdR^*Ik5>T+W*xmUChtFON2TdU#NY>s=wqtbmO{>n-(6DU? zgnnZM$id=!@{y4_Ga$ z_LzSiYG%S`^rq%m1vY103bao|Ox{y=5qtqw-lk+3Xk(r0PN|;g1Df7@{RnXXtXaQ9 z#1&wi2b2N~@p&o06v0p)D97Yj^q>;yCI@W8_uI8rupyrgY_$Gr;QYhP;ODF__t4N0 zm4Kn7Hn)%UNYmfnD0u1U>Ah>NR=fEHX9u5vqY36`BY#|SM_z*(2kuZ}JdzZCua%h) z&7TkL^O|l|9GV6c>U;WjnbBd4#``mGt{QLvxgGo11Py1pV1|Yl1?mDng-ytn$9=G+#l^Qs zmODr4c^Vp44S5=Yxi`G?k3n$^el3w^9}nL5FDkMQ_#^ZxIv}+mbcxpRS7KdNJQPP0 zoS!eBLzh?HSC4wsRI3e%-t+qz<5&LtR#cD_|1n@m!f$T^LEW#aHm@HXE}>)8ViMMt zz##Q4qf%jqfRivT1hpl+1n)LCwti++_Ez z7+Soa;ZMT4upu^L4-V`m(qe?7D=xy@zt1l4WkT~xf&&ifIi%f_Gd8K7l(3iKeg44^ zhz3oL@0U12`G6uNmU~jWmckJVZ+Hsd0b67G z+jS2T$+@kqt@re7J$FGp2>K4dl62}9E8|rP7`ATk>#3=!d9IhLN%BjxyLYVdug|6`1t&Xob#^?^`ANRVtH2n)j0U(D+Z(Bemz&<(!UPV3HGsUtm$l& z@-^=T-t}8oSyL1sY9(VoQzYL-~5|OY? zmGP`e2Dbl#^hMB^j>Ibk{Q|xvL&v?X5Zo!w{koE5D;HXa;)i`OJ{|BnSc6t~T1b?p zIqpIHy`JVOljb&-x&kApeqC*g;&S}-=`+yC4aqw+PaMr91g!5hL$ot2+(ygwrGQ_W z?kN|rNr3RKL_5f&y0KKAL?z4NcaQSZRDHV;QbvmL4g5hxkpSq(((e6mjD@@IGsw4e z20};pl|Hq$X-L4o@D@^^%5*<}9g?a5BaHV^Xy+LurRXbjWv&L)Qa^pxv#FD03fyTK zCefxw7WgmUl6Qz{pwqG1rs6ACcrkgDb}Z0ugUu*txmIQy_yFFh3RB}7?D?nbru_xY zF8?~NFxjr{4H92Z&lbh&58{iCuC7NB6j;tiX%1}AoMP$7z3R6TF7cfkjB)ko;aJbg zzk7z6mB-9ui6Q$F)@XPOe4bf-_|GmZWyy|8a6$*Jeb1q}-|&qEGEPyT)z}A6aPMdx z_jJE;10Ago#P@DwQswny%sjwo5pc07A9U|T&GiXZ>#W*YMWRHv@5uk#h|$mFpY#IAdqi=Ka!bVsk-r&q!P}MV2gz-;dw>cwB*f!zK~H9I_k9FYx-FE|>Iy z!@S^D$339{$_D7%+pse=0Uxw6B}&0QBc7VqK!ehcfsohgJ{2XGvu zw0#Ruz}Ilh`f(@`d?7`ZgpV1TxR6fa^WzG{S1}i*B%FF`yH>E&9o*1%w;!-ncp~Ng z3w7@q)#TQ_{bB(WM6jYDidaC9AYBMmnsks}0viSCAT`p0z!p&iRC@14TBt!1ijCf+ z1_&W2H3WjRV5n!sz4uf8?>ojh=N;pGIDTRvklc6Hz2;ok{LO2+wqB}6D7p5k8yU;V z7ltDH^8A3xS&xJIPn4SNxGX(~TBkbIrGV9LkGV#T*Bqt5$>XVmJ=)YK>k>;}27c5* z-03eQV1WuL>a*?QwL)FNF8K5h+V={xUO`a{RA^0+>nnMmWI>Ccblm)4;U3{v0S1qQ zFQJGJ5JOyT#EkQ`gwLHky<0YIkMz)?l)Ab!W^GJ{1X2B6^vbMJI_3s_{xv6zo5w(r zjMRMc!1QE{%`AI-Gq*oI#bkg%7!m zyC7x93*0)tvVdMv5-_z&M_=nJvHXF#yhRYNwQw2xaO>kw#N0X1q9%PZrxS}IOkdK~ z$F^tr{EF8d-5>Tew<6Sc?L3^$*KyPsKN;bb4jG(Y{zfrQ}?aLo7$DFAAK2WqhqI|0sDy* z7ss9Xu{5?Wg`>et5LnYgp;zdD;#c>16A)gr-N{}k5r@VAY;Utu4+B8$Gy*(J-j?w) zFrOFf%lP2I&zJs&UC3&!CsAjp0?Mn8(_K@(u4Dc=1Jo&ubRjhj0}?`B-=ThsT7Q^d z(ujd5_sUF0qCg9Z^i6(th$z_TN1v&Y^b^bP&Ym|quTOZv6+K4GRR7F;?(!kF{i+MCjJyG>0_rU*=~3gOzqt5Ru<4)+LfK9%+lQ2+SNqzI0&NQ z8HOl;XEd8w<%BNLybCuo8@*%(AbKo5#k5M|sw(KtujIZ?K=u;QqgEw8j&Gxc-4@f)B zH8g9htBTQK+L zX>La-JlT5WBrko)``$u-whj~5Zt>I_i*lNn!^e1~ zm>8eJukW*J8qvADRfJ4kjtE=Gokb*(x0flyXIH63~r4Q!>9Kq(vcOO0(q4eePzP)HFqg>0TwOycL-gzxX6^=7e(G*$K2le zH98JeW-%1XMM`I`vnVUGo3NN}c}A;5v)D@=Ji3-CE%wjCDy+=I$_2(aNbZGaXFuk`enmQ?V!yS2_^5e^)7T_HD zwMI5Q4}I+VapjG4NX?=U8Sbg(+AMJ7tzDN%3g}{=&AY*ebh@(;oyqTFr5dQ`RRPWiYZQS^4!p`_r+F^&I1mkEu>S|{ktc6Ryp3{YE+LsiB<0Mmx0;3 z=0+TN7!x6|j-@uPKE~T}{&H^iN`+PHnrcJ7_#b&5s{>9L)_92Qw#mWMdAq|PKp9b_ zQQn^@xvHsXe+RmJ1gqL!i_exm>9#7BLiv5y`TOeEm4%^gFe-R4>W)`IYA-#EQQ7#>|QZLRnsqt3lXvqrq`^_^C4c{)VFh_Ll@D#tWx9eHnOF0i_nzQ!y=k8;01yVao0e zly5p~MRB!>-SA{vD03IdbQ)r+y$L z<-0EQHoK`lz2VBJ44ZGingYoyOu>bt=p(7uF z7z0^#^4~8;T}^|!>+2v7$0u{(?E8ijAvQm0sQAJIF}}B~(W&-BM%s9^*yDT6S3;in zGv%#@`*R_!UX<}FEZ!_7pgd#D6t)!p^XJ8UbzoaVMiPrR=$@+Lv0_d4U?F1d*3D*T zHi}*1&i|j8?qr#)H3c_$etJ;-Xg&jivob+8OA;3e3=!rB|9Pya63cSmXE?g45I9+- z;oqR#W8J=5h*p%pY|5`5lbhi?vDt#ozOXUhO&!jyLZ*!fcuAuL}R1du?Gtto@+A~ zMXOyhXbn$(qa(&&gYye{?%N25;@D0bwIdr6D7zRSMPX<7kfe7~1 zD*VyaB_fTG9+@2lGr#FmzH$St1G#1)K} zJ^qnLL|rX$bbTz9hhB~b{UV;aaDtBrnNPhH+Mc#*)+%O-_R`SX-(;oBH^oiUTB*P z_~1d#oK$d^II;~e_&+3EcDuAP#-&#D?Fxf@cfY=0B5Wqx6>6n&eL&)HP_3SV9$yQM z82CVA&=6pz!6| zHrENyW76j*mJsF83{dy}A9#lk85hh~D(S zQ_k#|#)I{e5RcU#Hvm6zt=b0%v2gUO1tzq{(oT9Mr-!7|WlB5q{^7Lk3PNQQV9oq^ z0L6`ON?M^G3TW}hXuRP65gwU%8YysF+hd??Q#;el24eYPA$6;$N|Pgk&-!olITiQ_ zjECC+t;*=6;?C{c%^m|J2%*UP?6t-gzInfRO%e7UE>!=wr^Kpc36Lv?2S`zwaD9`3 zN7G}TXJ~?KKi&xiXxkf&H!qELZp&I*yF@E3Kx zF#$UD?oKW0k%Y>r$AbXRrkzzdQ-=kxD|kCCxr~JLVn^Fx!EEk z=9EJd;u^9>MO-0m6SHCk4h6uRwzuLO(t0~3r5tCM79N@lI{vhh%^zPO7XR>ezz5ro z%e(J6Ds5|J>X5Fiq2j zcJmBn`-e9b($>3UtYImAtDyNn(MMy#`#C+2dSYUhv%)(Wp%QbuX67^VDN}RWCA}pV ztN8&_aAa&xOt_S`V)Fnn>&5F7Ttu!={VTC z$PluYV-UG*TQTJRfagzgb=L09#-}Rzxfh6ns#2s&MSzo-j2e0#=vmOe)O3ww>jEqE zN~8y*fg;8qY4+-*fe&paX+Dg@x2SntJ8)s-DfP7__J|oT?I-I6M!J?8 zlxvdd7Mp#TRpofFQZl>))Ja7}oAoVzo)@;#HZ=TS|Cx^&#a?!A3hn%S%6{i4kHXs` z{UQw&_B-Ndhq;GZBuWqWcf3C{K)p>{FvC6fcmpGKgGqNC2TQny=Z6IC_xc2`u)Oju zB0VEF9ECZ2KK!;B+wUeIoUiTo4R$SPu5E>9kA0mN*!Gmbw|YJN$Na?ElQf+)<1pYVKN(g9W6eld-vK%6Kv4{)9BY zG52ZDjHg??iL;_QXfLp~$R_B9pAgQPpu?xQ)rMvjcqig1CvooUfWSg|{xs$$6_Z@@ z^I}cmD!$1ik8x$uciwA53v*WH_~!kkxcE`%1YNX6&F z_c#>lvmaM4&zCVh2(uioscrDnNGGl=4!%7%dXsE=Wq z!u3wS)AMQGwhvl!W|Sm+9*!cO$H!JI1MfMhJh?88iQ+@f=*Vn0#z`h+-fWITi)-0g zU+^YSm!`>A+0f&?_~(UJc49`YZw^ms;QMhY_rpZLN!)h*^5qLf(9k>>-CE0kF=&1@ zRqAr{-aQr@_;#+?K#d@5#wWxm*hBnv8d<3Lgpsi0le9^W9A+fL619ww-PSqY3rsc) zM9~w&B-M<|}UXCNKbE8xku+t!s5v*j5a3r=h`au*v;#9iA73y+-M z%ewlk`&2~Mb>MB3Pa3avgFZ7ywitfVhOrF6{r5v}+?w`y=9vfW+ul1)M+IhY?|?H! zoJ3=p>R$SAk2dn_5cFqm(`imVyG2xoQ2vqU09pnjde+qMQViYMOMfN9p?^6h>4An^ z@V|6Se~%pa^p{U1-`&HfUTaPgE}GS%$1llvUw6lKZEbgHR|e0|&h=Q#C61k=Y|#zW zBO2S5@WDmT-h-ELJIZq*aqVJ+jQ>WvOyI9Ov(+yzMf{polRC+#{GeqXWUV=R2fNy-oZSttGz=;$I(Jd?&z65oKjEk{ zf80Bv@_6;#xw#)LBctnsd$JO~giCjSVcwUTJ0jBSMu*&Sj^S#m_XbhUTXLe#y@zq_ z!!_covdx<@(NW2=0jxM)m^_rJ&3`USnXegQbUF6~t1=(LE?t#@Z9?o}zO+E2Nj!*RK*V}Wl zplkce;ZtW-rt2SjUF~NxQuu}+bmn|hR+bpMKCB0;@{T7iELC--IivsJY>?b`SIxL` zHiD1NwDxfruvsIVV2v3KUj#fYPri24LU=sC)$C|x|4*xbqjkvj?d9Ao9Z6J z%IbZbKTcpIUBa|@*^YNA=}RrBmjb_nv*0|VaSuP|idtRxNgB^R^j#--|HC?vs^l-q zf5XAA<<@|Nqjwt*$E&HS^`u#|kB8S=Z~OaiFTopF_TIid1%ejvk(_CAM*8&Wl=khJ z1q2kjHb6<(UyOA(j7qxmHC-tO=`fRJ14jDm$)#O_Fa%!@kMB3ZfKCRddm}ySs;@(ApNUU0>5z_ANd= zL(P-NfmMJ_$bDd7e>^&);a9${uIC0QoCj+z219N@f z2{B`ZCvzZ1I!bb}KJ_Yo_h#_&E4vk9N4d_(8VwN0l5pVTw=i@3hZegf$YI}qkjH6pljO#1IK6HMr5+b0S&i|U z%k;9_8G}zK<-vn8GDgKhccO)Q&5o(dA=`voVZbgt{ZWbwlpKRZ$E$ z6?K*;(35%VPE}A-^pwqBxb%*gvZF_rEzD!>+HJPu8cL(W`fSB+e%s}((|mp6#|X!j zL)Y{g!iG&s&HGrn4%6v@iY^~h6l;;Go*knYiHt&bJw0+teDd7w?sRL%u#;O!(G}Ff zKCy9LwqAqd?q5uzaQNos>LaMlxIdhESd^5M-VR8}$hZ*hskHz2Qk@+W+tyt&|F_e*IXv7zoi=?jk z-6Xw=y`q2kZI;-dTxaMnc5=qkvxK@k*4wtt@X?+U*>e_TzaS)wGKIEZzi|ieUEL;% z@bEx~zLk;wZu1o>t0h}Hx;t0@3MAZwA#Jy@SG6bs0Fdy>0QvFy;ZrlIi0f}5rr=Qn zuA3kPJo~>73A9K-|Chtw|ND6I|Lm9fKD_N09>l@BELVv3|8)4LWDTMh&DTd}ZruX5 zfomFy?W3c64|##11f2?)*KjtC=MV6umJ981`0W4yy4C-Rcb!VQx-pIp4tF>u-Cm11 zng#JPfi3iR3{>#b`@>^nL4OdiNahw6M+)cW9%Gg$%eZT0sR~E`x=g*x8*#b$`7N)b zqg7KNRt*l#XXrzY{S_>;WcZ34+}&d?ML3v&J9_&!5w`{~1`l9l1Ol#19UU(;y9|IhSJyJ?LT*fVynxByK-}@EFjjeqF5^;u zUy-VNv=ILL2US#_vnZrBZw3dca=&ZcB;g-#>x%YhA{*THDZ2u5UC1VJu%$(&m+H|R zL;#dg47vo&+;Km=5`+B+lw zByjUILy7(O@mkW{ESMb-DUg@~deKwBrdsW{Zd{#x)12YlU~f96XUm|F9ePnJu|&bP zZ*p%=H`uRNXHvm-PROgfj-V}C=f;wx^a3H-*Z31Yh{GkuE`#gPI?KdAK0eOF$YY+M zF!OSy>7YMw9nadLqpIrUL^`I-lO!JCroo+4c(qSvCx^JSq=BFeV+pCX&Lv)G8s)dD z(_A_^I_iMNV6=mSgQxQBYHl}|W;|J$te|#TTi@Z;O+N{9`;?=xKAu(d*_bM@?2!tM>5(Ng|PPiLrKzP#KNDisrI*kHv(0U7fBsU$I7^y^5Q*T;I$|UpFF%iZ=az%xtb|Aoj8MXl631d8;`t72)S>tPF1#QPO0iH z7AJMX4T8IgG)k})laX;5pf5-gOKOdaySsayPYg+6Mk4^@`t^lf>yV>-a-ZA*|EZSE zByfk59p!uP9BlO$f*y?*zD*qKv6F#qpqWVG)ZN!9oyB%2#g${J$Vw)I;3PLpxZ}3< zLyOH#7T=7+f3hB|S89_$oHl5#{q#C2YRoT^za#^oZY;`G+5y5dP&TREowv$=Z1u(2 z#SsqnX4cbaFxejsz`&)D2Y4{G)7S=(K@&!cxYr6)x3GIu~U|)t=o%AFJ!l}3Aqm$usT6KshpD9hxt@OjOqVvVl9qT!PfrFJb(bZwn^!Q8! zU=>Drd6-@`q?JD+D-OE>L=A2k7GP0-|MzX{|8Pgayv~@|qwkdE^cyv69A2zdk6lhJL}W;{o!s@c z<&0Q3*@!lC{PtnsCzw%`nvh@&b$4)R2P1>Nxmn;}B2t9AHns>g>kN#HJ%yP;j+sw( z@3@AqpnfkLem3eB;}zNAS#7_o{vvtohxk@D!oqoVGCKd>C;)3#`mbHBkCh|GWoG*& zEuO4h!wZ!_-8+6H4(l8LK74d4p%t#uRG@- z2%=y(h)bNW7dnoe`TO7C*R&BMY)aitUG-$JYmL`?)lm_P8xMJ}Ri#6_1RB9DxX}ER z=iT#^adef_XrhHb*bnX?PDR){kQ?ad`ub$t_cs#fm&Afvg(&O5Ve|8l+eDnDPOv); zg%6b6R9j#25=sR&rit}WDqCFM6CPWoij6g_EnYWh%$BI%{rR)PtfFF|;pc$->2-nx z=*-seEkwhGgx&S?_jcn=2zNw-lOHiKFuWrKd97g{Z;@p?nq#=+ZL9*RwI6AgV{|Rx zW;|DkH`0~00W&Zr({^>e$d<7z*PEzhM$R{ew@qEb!^tT}7+G<2*~pb}7-D=QvuEf^ zK`Pj*#)-k1!FG!z&ZUO;g9wq#8~bE3S-l%$SFm^Y%Pv7JGvx*Am2s*f$btX;T1B)` zDD+GRPxs2=R`~8n1>+oApr%1sPo(m$CT4{2hCw9D?44kEm;10wr z>Rjs%cxhfl3iaG<9iM-6=KC(au#gD39}w^&h$n}a%W9b`U4jn`bnsuRra6%p-r6qb z`@?axNmS|p4Dq-X5@AjI)Sf*AzaDz{b%(*EXjmBh-6tjRO!MjeqF@~7wm02GMMRu> z(_Cv{sHKLLV6n_#YAtbjxjJm9@IY)Z@KRKi5!J~5l%dW2pmXQkO`QLwr2EH|^$cm{ktllPOOA*%XK8l{SI72WI{*so zSlnrei&6NRdS!=lMdZcFD?MY|)9rT2kVNpkgZh)oX*CWmv>MA&Y$7ZL-=RnRniFKzpj~YVE93geJyz?<2WJ9dhL7^}-v~D`;YR%#Tmy}$7 zkxITma;uG)gUQ5FI#`~!UC37jnG35 zU`!2hfOGQ~S^mjgS9S$$9rpCBJ*=E%!>E(Qh->Z6lO4(*o;YwOal;&uGB+2(ZB90b zzuDCyk(L@#EKT&1DH5HbYjQ3y{dK-h4^IHtg*A~(E;sI6@NNRpIfYh-$mT_IV4b_! zZ<{-XzfcPtHdq39H%nrY#^4_0wUMNSn?Sr|n!!h0= z_lNdgSp=&!y4f1^sl9JIv3lQ{;kD;$&}b9u`ldJ}57StJ9l6fws}E*#ltZP_fm@nUt65M-8=UTzvT`%m zv>3xVI=(2KTxmWbad%K#12s1Uj?VDO)UKPhsrhG$5zex@d}KmH?BP(YDvKacNi^Kk zrb#_9E9luu>6m5pP~kQ!V#!iR3hqINrB_!Px(dy6OBm{7`Hw>J&%9GG6=@i zQO97{eJTfA^w>FGkTVu@P8YbXOW>w_+}na|23zdD|FFA7=zm_8TivxGfAPyylc>D3 zwA0-6fgFZx<>Sf|IIoq&qp*)TF7hyL5fPDxm{LAnJoS8BVlM?mYumVltRSPG*#}#j zn}vBT$CU^2C0}p-3J3G9?9_LjY>al);JiW)nE^e_^_K%@4nuFN@|X5gh6U+V9R&Ss zP@E6_aVTTx$n5Vs;BxpB7dt+F#MfD}z$$Wpo(Fw~d{X%xnCu`Y2DHZIeNuU?iCkB} z=&n|u(sAdCd7foAkFLO`BX5f~;X@=<$f6&pC;&G3tkIA#kK}2VW3Wr!r}|qyB1?LJ zu9E?}6HUU(ZuUp$hpBcE-OSL-MP1>OYtseHuOVW+zIeR3GW(;#?`I_0JpQ?8hu0At zHaHI7WGZv9)|Y5J<(9IZRRPXwsOIQ$3AthMfb4c^^3x)MxXA=<0%@?R858GVR=GY0 z9kK=qvvyuNOL|&zs1S-=|2C;L;g@JOVlk}PIq21Cw z$);{OHB1=Ojayc`SXv%9sPce%IxaD#aCvj*bx*`Q@WJ|oMu&vpI;h0Slvy35Mcg#! zd9YCPD-@LBVZ(N+Q9_%Ui9kVeQ8L{dbg$p;ZF4MQpnaxbRRfD^+G0++^>pBArg(X| zFv9VaY~zEe#fgqB3st=9?wUmAtgH*7U_SfT@Woqpw-_DaT?O53Ov?8tT`Maex1m3L zuCZ?h>dBtuJ_j~_{#XTN!;X^3=NwhOY)l)8y}M$$keMc6-I(L5 ziMoUT6gaVk2yn0}{$7sB*tT#Suy~nP$zkJoeB1<#HLJ744Rbk_DIK|&u5g{N(IXm3 zu&5UM3ELqynB8(X_yR1uBSn+;L4{yv48$jwt+QNebst!Uvl?Q?pLEY$YPaXkVIToj z@y5YPX=_7OW8ThKjzHRTUZ#8T=~7V83aFKHi!oG>IVKw3R1{Dp8SK;%WRRd`;y1F^{f`hAf)dO)s(BUAr!OhtH)B}!m zRSjzb3nmj2jLWRDoCtT)n{0Eu+bYWlDi#Y6^x9-S?vi&C--2nQuhQDB`?~I#T*t~> z?>c&*v3v(5nG^9_1z0mx_?O{))tz$Sd>z}jmZ6<~J`U0>J^jA!v0bt+QmdsP>m zZ)BBqC21uB8|6asYp%g5*n+WHQ@@=s{|~A#Af5aFndAl(u)mZEGwe zW`(?hf>y0$krth8`H=e|fj|o?@SQG5D?^r$im8Lv88o$O#wFcta$`<gS@?IC0Yw**p$>k33y(~DZYxN{zP%`6Cu&f>Rv((2FMS_d{} z&IQ^BV{>2*)_+NH?%WUb^=*!@&AabeXP%0X$yS(op15xVIxv#f>N%#iWt_OHyyo53 z_lAhzRYlmqaR%*2U5V`m=bbq)6yoJar>`!P4Z|))FMXfv4L~yvzm*Nx!Zi$n7T3r7 zB5i>?c(GLEi<5h983Er{3hL^FcyaV}bV+;{N-xJAetV^`uZBgxj;1hCZN^`>huqh8 zO?T(6s&w<}9Vq_e8jO1lxLc>VH|<<4*^HT2_$us7nTS(R)>Wb%>juZh;&+!y06OQ4 zHJX;eVAVGAkcA@g)X6?br8O53C)dnM-6`vW-GMYAlj}e}dxV$z-S7)}#c+FC9`h{VC zi+sXNnFa;vM=V ziNb~HZrWpXtt$hP4(HS*>A>qZ8qWy+rS!3F1EtSf>tLs2UJ4K9qkVje9UOf0yXV#> zHi5;tS-RDgyUM@Ee{jUNMWO+$Lb*F(VPUBL#iKkN8)Xm~P~D3q44*khf9)NFO+Qgk zjg;Rd@yEAZJ|Fsl1B_)@$FTMS?QqTj(UB}g2`qz<-5SBSyxma9J09X)1WgI$g!}V8 z=3aB_6tP&WOO81XSWvo$`v0vrp7J{&Tp0_0-)aGdLFqr4@@@p3sm!rJI-g*nX@5Ih z^rGgOXs>#VYR6x)APP@&SAvibXH3uv%7WMFtGE~7So?E(A#2T*% z@^F-ROikJB)Ir&RfmDUA^3cLoZk>@|+qt8lnOq zqqyd$=Vuh^r9z?)bI@xqS`~1sk&=YHD1Wjli#4h0W_!zA>g?JiIeY%CVe5jUkU0;l zar4tvp0MF`g&C0cuMY?ylPfEQX|H^Wo@22$Z%W~j9FP{Y_H|xxEX5W!*fTajKWXT3 zKg1>;yCiq8=DbsE%JKl~{%pY&#>8<}A@Jw-Hs1ga`5ZeS|M9wV5A=^YGQync_C9`e zg;pTidhrr0=GbrF-t3WI#umCv?`&Ob>n6E3q@Rum+;749X8EFxW%cLo?iwmK^* zVa*C)%3u>!J8%c-`d6Vm!l3m}2pHTkH0+OXmTt`55+4#@un;Sl8SHDS(G>%u%#SlG zpRM3yd+Iu1yK`MNBs}!wi`Kxv&7Lik%`OFcP^IhSy(Ncd<>0@d|0K6YcI&eM%A?HV z)=B&<);-FBZtr!beR6U&Li@O~n8ZCIJY(SL0VLcKxOMDLi+)zNA)g;vEzH- z?}Euy;4RE%7Z$cLobR=$`g~Imhr@9?;{S1e)op)&m1~ci&++Rzh)yUyZatAHpY3G7b%;cH4iFuIT8e|?J&1GMqF_{d6EQS z+Ve(TI|y=?#+0j$_cjK+^nn!DkOF3hCA+(==aND~LgKu^06ie%B}+B;){zD%=Z%c> zS(J@SE%$~Dn9FowwZHdz5^iO;3rtCPh#GR&%(L<2VTsk+3?H9aBT`5pcuQOXnrEx@ zWpIrORoJ@7fla$P?lNT1EmCg$G@8H4c%R!w3;89>; zP58q7JHNbgpec-nXZ-tt6 zTMPPAFT?++hMuzAo!+c_kf8sE=XUql$32ZY1bFqVZ{S1rkZi|6fsQ-r_#|Ln;Lgtg zW^5dsP`57r_+e}J*j&Dem87Wso>QjrpRQ^oFx$|_i)K?0JPy&HT*(AGI|55Qd2@_` z$N%BlHm7^x?e!mBNHDTW_b53raDNs%H$)WGbp@p>I=(y)uD`1{{tr(nuWw?w7SEqU zam^q24}8~S=aYT=ZlATi zyCESUd(|T&Wnmd+egGt4C&oeNC-tpyzvy|GZk}B@y8PG6a1homBUeE7rf{6EL&26O zU^gs0cAXLyUV<&{qxyH@|5Ae|qEhwsw!4;IDSz}AvL~H&mkt|fbn>YM!^Q6F)LprW zr6y@=;NBb9S6k_pN4sXP6|>E9oA{-=;q%Q4TB{$<1ysgRpGr&eN^d>@CB84a((aXG zd$^|3ky%+Q8|<_vw2A6er>w~RAavuXJuWt6r9S5jNE78MU|%#k7Hc`RT0)CwN@CWv z@q-OG9=47st@Zwuq#2Ho_D42;=^d5aY<{I|vmf++%mhg6z#_Ik7}i<4u?#- zH{IyZFt=%2JsI=bHvatKx4T!_ZYeHSsZfW)864q z!I4VlE|OL>J-C?Xd~MH9^Af7ZR_+b8x;)}7t)^WN zIIoZ0B}HP9{E0XWkNFm`RR-&od8wno5eV%an^&a#Kf4fCQ^5kOd~aDbVCl9q&B&*A zhJ*wDk}#8&&-n#5JLI@nA8+q&2E*<0jGskon(%uuQFz_Fc{teb1}GL~x_O`cYyEj9acerw+z1N1#tlAWJ5+H=>k@t$!3kmi%^Qv)c z7iM?cfkaBt{@j@Yjj1i;f^Ovn1_L+3j%^W>O~t&1Nd4%~o%x-f2J6pf2(B-^drC}C z6HT8Fo7mt?xMq6c%P%Yn=Culf3rki)qg_XQ?kUtC!=O~&XLYg2&>0%QZ#**LH!$Vl+zQW&H-L&BU>bA|wFe~d$3e!f>Mdi6EC6{+K<4*#z2e@Rw0h;n_X z19y-|$lGyy#(`zZ@y;GlFQqy?g$``idXEh6%=mQLEK0an$19-&BRC=!wruBmkTp9C zz3k03`8eI?<@)toux;-)b#Gpswr=F11acC{lm;ZrPQv&9oGlxHi}afsU-Q$>sqzJ# zNNcC(M_&y-Jte;8eYK>lq%N5Bxk}h0+pCZlmdq^h#is2Oh5&-86=GJzZBq4C?Bs^M zh3YvzS-FhhxVP~o*wijplY4`tQfQHu|2jm&zp-mOo}fj67DaDZSe?B~Qhx?VOh}8n zP_tybQ0e4}pJ$A98G_`=lw7ifsoA<_AxBFhq;~8;;7NbF+i+riDQm;$HAvAVm0O84 ze65CYSWO>w<$Q##SMOIU6c)tp?cxRF{Qzn_&aFOp8$IL+P7-oZp-r7Ri5-B^;Wdcp^s$ zab)%zC5=v-{ikvC!E#_+jRdURrZjr`Xb{>&i1__0;N28|sXtHGFczSZoY) z@TT$FG;gQy@vh8@1|RatGGuu*d)r&yn7Xh|P2kD6XjLgxnQ8ubdD}byt3UhTTr*jY z_%;~5w=UpC?0honajdfq6r%ZPH&(+md80s{-Og|a70`zWa2fiTl=@**8ZI+E7hpQ6 zqV7Igl??I#u4FfBZ?#dWas>5qtqzNY9ADq=z?^Kj2dN`#{n{W&F)oOi-x+}esE%*| zqveob$8k1KvnyYFFnrkYEYFLQY^w#bMNO}&_CQH-!s{lPfcU*;ebT3TaeP=5-=-J9 zDYB;0+7NK^?rM^Gl(-LUm6UO=@bkB8-D9Y(#Ne|?hRW?f8OjJjpQR753Xm6Ul0=_bo-=q7uq3vm#fB>#GP{ zOGRQr(O>eIwAu5T#B>F6F2z>t9`CfU59%^0ua24W2{CC|3>1R-z4UTX(N);E06X-e z^>urAiZo!T%@3Na{@X<86*+}eDxoDK>DE`q93;gzP z=38h?1a7U(!QwX1b?N21kHJ2`Y2@^+=+tyz!n+k=(oj(weg%VY^t$pzYq8^_!gcZ{ zcWV=Uym<3+5h)U_K^!e^S_mG{hJX$IrXTy$vM4=BZDVy*PX{$|nt|ugrcvB-;sfqA zHI9QK+9qC3Itrg`tw+%h_toD6XSBhNY{(_6;wM9OlLBtfvo{ysR_ln+-{s(5lKDL3 z@BPUrSaW`cwJ|cq6dRt>Bk^oNz4*{0rzgb^vGkTOjvCp`!66Y2v4rx<;&N#)t9*lx z1psQ5?A&;j5qLSEWf}1Kz=2#2Ep*yxXtCT3-8?vJW`Y%m{?y0yL{++sKfWg5$Pcny7K`I(&7%(3K?eUaQ&`FL+jV7Ns2W=%mU7cS#_a? zZ-Z=H7Il_GjL9>wU$qh+ES;JbgI#s)Tkx$eUI$#3a$^m-oavKo`m^Ra?OyRG;Yz9@ci%b|At zF=M?SK705rbsJa`r+V5O%|3{#8sQE5llN6c7SmXE>7jnWT(wKdp3=)S+7uk-cFP(a zg6SL)+C=L#j%mWUhJu4=T_{2MyoW*Ro%8q4*Uzo9FX!6Io{tf!IKy|;=agZlmyhDN z$g>-e{UYSxW!+=SYcW*}%1Ycc@aX?5dKmoa|6MB!?c4vsE$cl2vPnVRN<#hQVBbHZ zK*babeTHl2$~p`lfGJ`#@u6=j|4oJQ0*p-MBls&xqyH*lfRguL-7;YQ_&>C;zrVkh zl2X{%2aiX2>RpRa{eP)FonikQ{1Ld>UH-Q;1d1Akmjnd`AA5U?4R`}|ap%7ZCWz@}J~ot&L})UE{C z0terRyuV8d${QVp)=A@0sjt#s=kPrb`pr!JE`H?XeE!=^Q(#QN*2SgU&{$9unBJlM zDD{aw!Sef=fZ=i(A@vwSIHzf4k+~l03w~7E{AAFkJ7bGWzMvZtv9{XY|LAciW+he1 zOW?trU~#EP@OGEiEraR$f;1NzhnmB;^J&rP?EY44pSG?pQhO@W7Y6rk@lYqJk^i;#7m5kJsmV}%$xl5=UiP~-vXG`;z39|X643+ zL9-xfb4CMuTln2DAJIDs<4p5p%UaL`#ns>#aaCZM!WnUf- z<=X$>;Y6oW+E1iJqR76cLZPznvhPA>WEo?ZoKV7%rDW@5U&fFc3}zxr)?^!u8M1FP z!`NaB-|JSL=Xw75yh{kiVzy?uH=%u|rtPNJ6XWpy175-t*HMjrXv zW+4uwsR=qH2_SCAoJXlwa($hyrs*xT8kT6^JWB$F;frA_*8KAMk%v0y9<1ItziR%mU$5GzXB_j)4Yq6aZ zK}d%npbn()tU%hpyDvsYB z5Y}gqTxF`a*j8-QBf3sO_COuI5<&yHzW_;_#4nwpC)-Dc45m*U!Ieeb0Q&e2sV_Je zR<7n2!#tSys7N&q+n+stDyp)A^m^d_b>_pze}eFCBxC=z5=6Ha}%jk0WeW<@}*MwfR%h#H#bQX3KeGP?Okq#$;LYS z7lh^8s_Js6#w-pEX+t9Y4>4{XhA;UdE0&5&Gy+&(re+)kac=cU|F4F}ud6I|Dus+q z7bH5~=9$M_{8;0)Fd!4_Dsfe_m(<4N($LXnFQm#lCL`zhJa@(mGF@fe@A#6l~{0Y)GJd&2d!`}UMqq5EHqQ4m=z23?= z)UgG?h#~;IKbmRXM5;;$SgSlH+e!mEk4+#|mXP{#=SGWcVq`>w`z>Zei=i)>b~|>E zm!GISk?;j4ZF== zhDA-eZat&d|8YZdaQpraBNVjnjNI9f-!Mz8S)uN#wFk?*D^jjf1XDAwooqfMT{t*7iGuC5u|YH9KwaxF)uXXxv(E%F?okly^VcjGb>T zB5Onx(p$syys9X1lv$XC+bip8U(z;8y&9|Ks#i{y&1u521u$SlVK}d(yu8auP!45B z9H!m`hU+wnz<4%P`s28MX{d9w8HKj+jc0vpmtL){GDAZh!zIDZ+A#hP_J%o}72$l)-@SI!wB?f9Lz1PXHJup*J9B<6v!)V@dP@TSvhQo~3m)7)yKp@) z$QWeU&J=tG1?&|)1(#==U0&X{H7oWdr46in6_6w9IBVFQr;7(TrE?wr8*qmx_=;N*Y-#k(izyM=@5+#KdYbF%n7=)ZFwL)Q}2!61HQljpXNi9Q#)o-IUy}8AJoFhYoMCEz%ZPw`Xa1*Xh~2ErB&S zh$;|(4)of#%BL3p9hFag5BjHJOz^g{5x9V=x(XN=G2$1C6=Y*%(stCeIYXKe~fDvUVNuyzQSWYP<$F;Uuoo z@b2YrfY+1ly0jNhQLq&j!MieIM;0U}AJQB2Tf?H74(+XVy)v>-RTBSjp6|WK&bNlD z+WPjC>d(birVp;NSewLq5;vpoJ)U7J}8R+rac z{Fm(K4VS~WZx9C@Y0gFHJY;ARJjZ5v*lKVK`>nz37cBo`-aA%JZ`&<+s3HboLD}p# zC8R)*bG99ToFO3ovwwQY+vlme@Y&elE?f*i;kTH(hIN8X=I6uLsmWg()}g2(ykXhH z0+Q0Ynw$tME6d?29oL+EKS%Qv=lT;xtAKEK*npGR)NJw_8A+QKCOV~0(?Id%_E>hAVxF}V;9ps_N5BCpwJ zBQk(hetN`@J~tr*HpXvaV?{@pR%LF%Z@sR$2p?fO1D5JCqtIE&lYlx9wce=(T{<^g zkaA5bL6^_JuC<+sub!#TQ@fLeXwN!=hC zVD%KWJM*WpfdujsgHXHao9B{cVcs2BbH$Td=+XYYtyK@%bL67>vm&BfUvu$VJ$3kI zvfy<#k)X2i?F7bsAY>CGJwE5JZbT_3l`^@y%7SQd-p--E3)Hk** z7={Tox7{t8JInKcM=Fmxt1I0SS(H z^R8PT`*}u36IC2(}8H__oq5Vv!}_PM38~M;XYeTAB8Gw^|V7b*)r&_q-3mV=oHXNY%>z)z_$x)()v*aA8WcT7A;I z(u$T%hr|11X660j9waJXD5m})d@7}PoJ;|l1TD1S*!7SUYu2&f`34ie1VIr3YCQaJ z6-%xbAXp0O7X!!pe~86H+C42kiAsg4B)JZh5y@3YP`3qxprC=T#kBbnD=cL`zKX2% zEmz|mrNzM=->QjZO-F!K*E8v=*|ezB^Yjb`#GyQn_BAUQ*ojPj$h5 z^{W}zVlF#FWkhJ>Thu#f9?A^WOAa<;4l*!RqMZ%J0MHTlK1MP@!0B*UN0wB#rs#i|$rf`HdqO_eADO@{-VqXS5+yU`LQiu@J_L7pvy7 zwPT$9AbRk!p=Yb|F!uaItJ7(QZ}W~~iKwgUF$A16GA*3mcZK@le7JqZcQ!v&6o!2@yM)6O zDTbLxVht2dQQxc->S9IBtu^qiqhR}g?p(F%cE(ji|C$*R8+f8REe|k-y9-!rNI;uk z;?D{=Gka~wxp_l~r77>k#d1qRcCI>-UOi-+e zAI{kuwy!8UoPHa>oKP~KJ%VFHL4(Y=8+54CW{HhY=hXT2SC}{Bli3I;8mX@ZO^wf= zH%KPx5U)%&-P z9CWAu&meap8n5M@K~fJ6Mid90`Nvf6KZ9%)IVI+|S)hdt<}ioucep}l-^ToUzesJL zhR2pS%cIg|WmjXe@SVk?zzZ5xQ3g|wG2FraW;A>2QSOnf!1hOtM2VneRO8u zoOL1!NIEd}zDhdAPPx(A^9jF=DW9JxPh-_Ds4HkP#>%lzX<`s%)WZ2D7J3%v~Yiuv7SY>YFCob5wS`7c~h$Ti04YI<9l=iO?y)`+U%~a66_FwvJfZ zkUr81GBd$8cm7bhge$_|LeO-o@V=2kF}yl+rSIbkS42Rk0#eNcK1NL~V7 z=6Co}^H^wKuUv2*C>@0&Bj{DR%g#N?5x}ay1-)4<;7&h@XoagB*d6=lK+==A{~mmT4gCWAG4 z1o8ENFS3vZMR6fO_+_#5OX3j!G};2Y@)yOTb?N!vhgD_)Kz0lyo;$)vW=N=>(eM1|^MpuHguz};}R-0Pi46V#wBe0WCg(|{CAL$PyB7*$3XkIkMzSE8sajoCS%QaLAGhqF^g&YX z157VX95M%nv(bFRyJ`J?C4-iCeizrcjnqYR2$@@mc2!9RruGuE4Xn$-~AblrFXYCmS<#4Sk^F^Jfw60?G#2D{d2!lW2>3A#4OOXaWPJGT#{Pr70-7UV*BQZ07I!X&MRE1JX(?t>NTHu&i4WytTseN<;n#NX zEmk8YEFBI3Fe*NkW#3+_O-q$0bk(q%y`W8YXwTi2FZ&qD+BbH+BM>vm^?i<>O;zX8 zedD>GYj^=JR8)uFaMLPc=-i(+CcdK1VV&2giZ0E3E?W&C>Z?6@ld`7x0kI0H2ddMW zqJZ7tq5}CmFlv-wJK1F1c#DJ+*`6t~l0Ou)977TRcp)p$T?s-2N$fdRtP?Uijs9HU zNa}iBJQz#e-w|1W{Zobmf>C#O zY>(@>@Y37F@UKAcG%1M~#wx4ZYr`@k~ zAbb1E-&J|*e-U{N4MvMkmH&GB@2g<6XNUb{S)=^TvZA}7-tM`prlVSN)AsTI0TN*! A8UO$Q literal 0 HcmV?d00001 diff --git a/doc/integration/metrics/influxdb_configuration.md b/doc/integration/metrics/influxdb_configuration.md new file mode 100644 index 0000000000..13227f1e88 --- /dev/null +++ b/doc/integration/metrics/influxdb_configuration.md @@ -0,0 +1,192 @@ +# InfluxDB Configuration + +The default settings provided by [InfluxDB] are not sufficient for a high traffic +GitLab environment. The settings discussed in this document are based on the +settings GitLab uses for GitLab.com, depending on your own needs you may need to +further adjust them. + +If you are intending to run InfluxDB on the same server as GitLab, make sure +you have plenty of RAM since InfluxDB can use quite a bit depending on traffic. + +Unless you are going with a budget setup, it's advised to run it separately. + +## Requirements + +- InfluxDB 0.9.5 or newer +- A fairly modern version of Linux +- At least 4GB of RAM +- At least 10GB of storage for InfluxDB data + +Note that the RAM and storage requirements can differ greatly depending on the +amount of data received/stored. To limit the amount of stored data users can +look into [InfluxDB Retention Policies][influxdb-retention]. + +## Installation + +Installing InfluxDB is out of the scope of this document. Please refer to the +[InfluxDB documentation]. + +## InfluxDB Server Settings + +Since InfluxDB has many settings that users may wish to customize themselves +(e.g. what port to run InfluxDB on), we'll only cover the essentials. + +The configuration file in question is usually located at +`/etc/influxdb/influxdb.conf`. Whenever you make a change in this file, +InfluxDB needs to be restarted. + +### Storage Engine + +InfluxDB comes with different storage engines and as of InfluxDB 0.9.5 a new +storage engine is available, called [TSM Tree]. All users **must** use the new +`tsm1` storage engine as this [will be the default engine][tsm1-commit] in +upcoming InfluxDB releases. + +Make sure you have the following in your configuration file: + +``` +[data] + dir = "/var/lib/influxdb/data" + engine = "tsm1" +``` + +### Admin Panel + +Production environments should have the InfluxDB admin panel **disabled**. This +feature can be disabled by adding the following to your InfluxDB configuration +file: + +``` +[admin] + enabled = false +``` + +### HTTP + +HTTP is required when using the [InfluxDB CLI] or other tools such as Grafana, +thus it should be enabled. When enabling make sure to _also_ enable +authentication: + +``` +[http] + enabled = true + auth-enabled = true +``` + +_**Note:** Before you enable authentication, you might want to [create an +admin user](#create-a-new-admin-user)._ + +### UDP + +GitLab writes data to InfluxDB via UDP and thus this must be enabled. Enabling +UDP can be done using the following settings: + +``` +[[udp]] + enabled = true + bind-address = ":8089" + database = "gitlab" + batch-size = 1000 + batch-pending = 5 + batch-timeout = "1s" + read-buffer = 209715200 +``` + +This does the following: + +1. Enable UDP and bind it to port 8089 for all addresses. +2. Store any data received in the "gitlab" database. +3. Define a batch of points to be 1000 points in size and allow a maximum of + 5 batches _or_ flush them automatically after 1 second. +4. Define a UDP read buffer size of 200 MB. + +One of the most important settings here is the UDP read buffer size as if this +value is set too low, packets will be dropped. You must also make sure the OS +buffer size is set to the same value, the default value is almost never enough. + +To set the OS buffer size to 200 MB, on Linux you can run the following command: + +```bash +sysctl -w net.core.rmem_max=209715200 +``` + +To make this permanent, add the following to `/etc/sysctl.conf` and restart the +server: + +```bash +net.core.rmem_max=209715200 +``` + +It is **very important** to make sure the buffer sizes are large enough to +handle all data sent to InfluxDB as otherwise you _will_ lose data. The above +buffer sizes are based on the traffic for GitLab.com. Depending on the amount of +traffic, users may be able to use a smaller buffer size, but we highly recommend +using _at least_ 100 MB. + +When enabling UDP, users should take care to not expose the port to the public, +as doing so will allow anybody to write data into your InfluxDB database (as +[InfluxDB's UDP protocol][udp] doesn't support authentication). We recommend either +whitelisting the allowed IP addresses/ranges, or setting up a VLAN and only +allowing traffic from members of said VLAN. + +## Create a new admin user + +If you want to [enable authentication](#http), you might want to [create an +admin user][influx-admin]: + +``` +influx -execute "CREATE USER thedude WITH PASSWORD '1234' WITH ALL PRIVILEGES" +``` + +## Create the `gitlab` database + +Once you get InfluxDB up and running, you need to create a database for GitLab. +Make sure you have changed the [storage engine](#storage-engine) to `tsm1` +before creating a database. + +_**Note:** If you [created an admin user](#create-a-new-admin-user) and enabled +[HTTP authentication](#http), remember to append the username (`-username thedude`) +and password (`-password 1234`) to the commands below._ + +Run the following command to create a database named `gitlab`: + +```bash +influx -execute 'CREATE DATABASE gitlab' +``` + +The name **must** be `gitlab`, do not use any other name. + +Next, make sure that the database was successfully created: + +```bash +influx -execute 'SHOW DATABASES' +``` + +The output should be similar to: + +``` +name: databases +--------------- +name +_internal +gitlab +``` + +That's it! Now your GitLab instance should send data to InfluxDB. + +--- + +Read more on: + +- [Introduction to GitLab Metrics](introduction.md) +- [GitLab Configuration](gitlab_configuration.md) +- [InfluxDB Schema](influxdb_schema.md) + +[influxdb-retention]: https://docs.influxdata.com/influxdb/v0.9/query_language/database_management/#retention-policy-management +[influxdb documentation]: https://docs.influxdata.com/influxdb/v0.9/ +[influxdb cli]: https://docs.influxdata.com/influxdb/v0.9/tools/shell/ +[udp]: https://docs.influxdata.com/influxdb/v0.9/write_protocols/udp/ +[influxdb]: https://influxdata.com/time-series-platform/influxdb/ +[tsm tree]: https://influxdata.com/blog/new-storage-engine-time-structured-merge-tree/ +[tsm1-commit]: https://github.com/influxdata/influxdb/commit/15d723dc77651bac83e09e2b1c94be480966cb0d +[influx-admin]: https://docs.influxdata.com/influxdb/v0.9/administration/authentication_and_authorization/#create-a-new-admin-user diff --git a/doc/metrics/influxdb_schema.md b/doc/integration/metrics/influxdb_schema.md similarity index 60% rename from doc/metrics/influxdb_schema.md rename to doc/integration/metrics/influxdb_schema.md index a8e69b5e17..a9ef0b446c 100644 --- a/doc/metrics/influxdb_schema.md +++ b/doc/integration/metrics/influxdb_schema.md @@ -2,16 +2,16 @@ The following measurements are currently stored in InfluxDB: -* `PROCESS_file_descriptors` -* `PROCESS_gc_statistics` -* `PROCESS_memory_usage` -* `PROCESS_method_calls` -* `PROCESS_object_counts` -* `PROCESS_transactions` -* `PROCESS_views` +- `PROCESS_file_descriptors` +- `PROCESS_gc_statistics` +- `PROCESS_memory_usage` +- `PROCESS_method_calls` +- `PROCESS_object_counts` +- `PROCESS_transactions` +- `PROCESS_views` -Here `PROCESS` is replaced with either "rails" or "sidekiq" depending on the -process type. In all series any form of duration is stored in milliseconds. +Here, `PROCESS` is replaced with either `rails` or `sidekiq` depending on the +process type. In all series, any form of duration is stored in milliseconds. ## PROCESS_file_descriptors @@ -32,13 +32,15 @@ value field `value` contains the number of bytes. ## PROCESS_method_calls This measurement contains the methods called during a transaction along with -their durations and a name of the transaction action that invoked the method (if -available). The method call duration is stored in the value field `duration` +their duration, and a name of the transaction action that invoked the method (if +available). The method call duration is stored in the value field `duration`, while the method name is stored in the tag `method`. The tag `action` contains the full name of the transaction action. Both the `method` and `action` fields are in the following format: - ClassName#method_name +``` +ClassName#method_name +``` For example, a method called by the `show` method in the `UsersController` class would have `action` set to `UsersController#show`. @@ -55,21 +57,31 @@ This measurement is used to store basic transaction details such as the time it took to complete a transaction, how much time was spent in SQL queries, etc. The following value fields are available: -* `duration`: the total duration of the transaction. -* `allocated_memory`: the amount of bytes allocated while the transaction was - running. This value is only reliable when using single-threaded application - servers. -* `method_duration`: the total time spent in method calls. -* `sql_duration`: the total time spent in SQL queries. -* `view_duration`: the total time spent in views. +| Value | Description | +| ----- | ----------- | +| `duration` | The total duration of the transaction | +| `allocated_memory` | The amount of bytes allocated while the transaction was running. This value is only reliable when using single-threaded application servers | +| `method_duration` | The total time spent in method calls | +| `sql_duration` | The total time spent in SQL queries | +| `view_duration` | The total time spent in views | ## PROCESS_views This measurement is used to store view rendering timings for a transaction. The following value fields are available: -* `duration`: the rendering time of the view. -* `view`: the path of the view, relative to the application's root directory. +| Value | Description | +| ----- | ----------- | +| `duration` | The rendering time of the view | +| `view` | The path of the view, relative to the application's root directory | The `action` tag contains the action name of the transaction that rendered the view. + +--- + +Read more on: + +- [Introduction to GitLab Metrics](introduction.md) +- [GitLab Configuration](gitlab_configuration.md) +- [InfluxDB Configuration](influxdb_configuration.md) diff --git a/doc/metrics/introduction.md b/doc/integration/metrics/introduction.md similarity index 66% rename from doc/metrics/introduction.md rename to doc/integration/metrics/introduction.md index 007fff9e7b..bf388aa2bb 100644 --- a/doc/metrics/introduction.md +++ b/doc/integration/metrics/introduction.md @@ -1,20 +1,29 @@ -# Introduction to GitLab Metrics +# GitLab Metrics GitLab comes with its own application performance measuring system as of GitLab 8.4, simply called "GitLab Metrics". GitLab Metrics is available in both the Community and Enterprise editions. +Apart from this introduction, you are advised to read through the following +documents in order to understand and properly configure GitLab Metrics: + +- [GitLab Configuration](gitlab_configuration.md) +- [InfluxDB Configuration](influxdb_configuration.md) +- [InfluxDB Schema](influxdb_schema.md) + +## Introduction to GitLab Metrics + GitLab Metrics makes it possible to measure a wide variety of statistics including (but not limited to): -* The time it took to complete a transaction (a web request or Sidekiq job). -* The time spent in running SQL queries and rendering HAML views. -* The time spent executing (instrumented) Ruby methods. -* Ruby object allocations, and retained objects in particular. -* System statistics such as the process' memory usage and open file descriptors. -* Ruby garbage collection statistics. +- The time it took to complete a transaction (a web request or Sidekiq job). +- The time spent in running SQL queries and rendering HAML views. +- The time spent executing (instrumented) Ruby methods. +- Ruby object allocations, and retained objects in particular. +- System statistics such as the process' memory usage and open file descriptors. +- Ruby garbage collection statistics. -Metrics data is written to [InfluxDB][influxdb] over [UDP](influxdb-udp). Stored +Metrics data is written to [InfluxDB][influxdb] over [UDP][influxdb-udp]. Stored data can be visualized using [Grafana][grafana] or any other application that supports reading data from InfluxDB. Alternatively data can be queried using the InfluxDB CLI. @@ -24,7 +33,7 @@ InfluxDB CLI. Two types of metrics are collected: 1. Transaction specific metrics. -2. Sampled metrics, collected at a certain interval in a separate thread. +1. Sampled metrics, collected at a certain interval in a separate thread. ### Transaction Metrics @@ -41,7 +50,7 @@ metrics are collected at a regular interval. This interval is made up out of two parts: 1. A user defined interval. -2. A randomly generated offset added on top of the interval, the same offset +1. A randomly generated offset added on top of the interval, the same offset can't be used twice in a row. The actual interval can be anywhere between a half of the defined interval and a diff --git a/doc/metrics/gitlab_configuration.md b/doc/metrics/gitlab_configuration.md deleted file mode 100644 index 3b44b4394f..0000000000 --- a/doc/metrics/gitlab_configuration.md +++ /dev/null @@ -1,12 +0,0 @@ -# GitLab Configuration - -By default GitLab Metrics is disabled. To enable GitLab Metrics and change any -of its settings open a web browser and navigate to -`http://YOUR_GITLAB_HOST/admin/application_settings`, the settings can be found -in the "Metrics" section. A restart of all GitLab processes is required for any -changes to take effect. - -## Pending Migrations - -When any migrations are pending the metrics are disabled until the migrations -have been performed. diff --git a/doc/metrics/influxdb_configuration.md b/doc/metrics/influxdb_configuration.md deleted file mode 100644 index c67c0f1510..0000000000 --- a/doc/metrics/influxdb_configuration.md +++ /dev/null @@ -1,96 +0,0 @@ -# InfluxDB Configuration - -The default settings provided by InfluxDB are not sufficient for a high traffic -GitLab environment. The settings discussed in this document are based on the -settings GitLab uses for GitLab.com, depending on your own needs you may need to -further adjust them. - -## Requirements - -* InfluxDB 0.9 or newer -* A fairly modern version of Linux -* At least 4GB of RAM -* At least 10GB of storage for InfluxDB data - -Note that the RAM and storage requirements can differ greatly depending on the -amount of data received/stored. To limit the amount of stored data users can -look into [InfluxDB Retention Policies][influxdb-retention]. - -## InfluxDB Server Settings - -Since InfluxDB has many settings that users may wish to customize themselves -(e.g. what port to run InfluxDB on) we'll only cover the essentials. - -### Storage Engine - -InfluxDB comes with different storage engines and as of InfluxDB 0.9 a new -storage engine is available called "tsm1". All users _must_ use the new tsm1 -storage engine (this will be the default engine in upcoming InfluxDB engines). - -### Admin Panel - -Production environments should have the InfluxDB admin panel _disabled_. This -feature can be disabled by adding the following to your InfluxDB configuration -file: - - [admin] - enabled = false - -### HTTP - -HTTP is required when using the InfluxDB CLI or other tools such as Grafana, -thus it should be enabled. When enabling make sure to _also_ enable -authentication: - - [http] - enabled = true - auth-enabled = true - -### UDP - -GitLab writes data to InfluxDB via UDP and thus this must be enabled. Enabling -UDP can be done using the following settings: - - [udp] - enabled = true - bind-address = ":8089" - database = "gitlab" - batch-size = 1000 - batch-pending = 5 - batch-timeout = 1s - read-buffer = 209715200 - -This does the following: - -1. Enable UDP and bind it to port 8089 for all addresses. -2. Store any data received in the "gitlab" database. -3. Define a batch of points to be 1000 points in size and allow a maximum of - 5 batches _or_ flush them automatically after 1 second. -4. Define a UDP read buffer size of 200 MB. - -One of the most important settings here is the UDP read buffer size as if this -value is set too low packets will be dropped. You must also make sure the OS -buffer size is set to the same value, the default value is almost never enough. - -To set the OS buffer size to 200 MB on Linux you can run the following command: - - sysctl -w net.core.rmem_max=209715200 - -To make this permanent, add the following to `/etc/sysctl.conf` and restart the -server: - - net.core.rmem_max=209715200 - -It is **very important** to make sure the buffer sizes are large enough to -handle all data sent to InfluxDB as otherwise you _will_ lose data. The above -buffer sizes are based on the traffic for GitLab.com. Depending on the amount of -traffic users may be able to use a smaller buffer size, but we highly recommend -using _at least_ 100 MB. - -When enabling UDP users should take care to not expose the port to the public as -doing so will allow anybody to write data into your InfluxDB database (as -InfluxDB's UDP protocol doesn't support authentication). We recommend either -whitelisting the allowed IP addresses/ranges, or setting up a VLAN and only -allowing traffic from members of said VLAN. - -[influxdb-retention]: https://docs.influxdata.com/influxdb/v0.9/query_language/database_management/#retention-policy-management From 6f32459f2c0ebc0aa10c728e5562375db9cf4c52 Mon Sep 17 00:00:00 2001 From: Yorick Peterse Date: Wed, 20 Jan 2016 12:05:03 +0100 Subject: [PATCH 3/5] Renamed "Metrics" to "Performance Monitoring" --- doc/README.md | 2 +- doc/integration/README.md | 2 +- doc/integration/metrics/gitlab_configuration.md | 6 +++--- doc/integration/metrics/influxdb_configuration.md | 2 +- doc/integration/metrics/influxdb_schema.md | 2 +- doc/integration/metrics/introduction.md | 10 +++++----- 6 files changed, 12 insertions(+), 12 deletions(-) diff --git a/doc/README.md b/doc/README.md index bee9ab0590..05c0d6fa0f 100644 --- a/doc/README.md +++ b/doc/README.md @@ -67,7 +67,7 @@ - [Reply by email](incoming_email/README.md) Allow users to comment on issues and merge requests by replying to notification emails. - [Migrate GitLab CI to CE/EE](migrate_ci_to_ce/README.md) Follow this guide to migrate your existing GitLab CI data to GitLab CE/EE. - [Git LFS configuration](workflow/lfs/lfs_administration.md) -- [GitLab Metrics](integration/metrics/introduction.md) Configure GitLab and InfluxDB for measuring performance metrics +- [GitLab Performance Monitoring](integration/metrics/introduction.md) Configure GitLab and InfluxDB for measuring performance metrics ## Contributor documentation diff --git a/doc/integration/README.md b/doc/integration/README.md index 5ba0d0fb24..297e4f6e7b 100644 --- a/doc/integration/README.md +++ b/doc/integration/README.md @@ -15,7 +15,7 @@ See the documentation below for details on how to configure these services. - [OAuth2 provider](oauth_provider.md) OAuth2 application creation - [Gmail actions buttons](gmail_action_buttons_for_gitlab.md) Adds GitLab actions to messages - [reCAPTCHA](recaptcha.md) Configure GitLab to use Google reCAPTCHA for new users -- [GitLab Metrics](metrics/introduction.md) Configure GitLab and InfluxDB for measuring performance metrics +- [GitLab Performance Monitoring](metrics/introduction.md) Configure GitLab and InfluxDB for measuring performance metrics GitLab Enterprise Edition contains [advanced Jenkins support][jenkins]. diff --git a/doc/integration/metrics/gitlab_configuration.md b/doc/integration/metrics/gitlab_configuration.md index 019db1ad02..b856e7935a 100644 --- a/doc/integration/metrics/gitlab_configuration.md +++ b/doc/integration/metrics/gitlab_configuration.md @@ -1,6 +1,6 @@ # GitLab Configuration -GitLab Metrics is disabled by default. To enable it and change any of its +GitLab Performance Monitoring is disabled by default. To enable it and change any of its settings, navigate to the Admin area in **Settings > Metrics** (`/admin/application_settings`). @@ -10,7 +10,7 @@ changes. --- -![GitLab Metrics Admin Settings](img/metrics_gitlab_configuration_settings.png) +![GitLab Performance Monitoring Admin Settings](img/metrics_gitlab_configuration_settings.png) --- @@ -34,6 +34,6 @@ have been performed. Read more on: -- [Introduction to GitLab Metrics](introduction.md) +- [Introduction to GitLab Performance Monitoring](introduction.md) - [InfluxDB Configuration](influxdb_configuration.md) - [InfluxDB Schema](influxdb_schema.md) diff --git a/doc/integration/metrics/influxdb_configuration.md b/doc/integration/metrics/influxdb_configuration.md index 13227f1e88..df5e4af7fa 100644 --- a/doc/integration/metrics/influxdb_configuration.md +++ b/doc/integration/metrics/influxdb_configuration.md @@ -178,7 +178,7 @@ That's it! Now your GitLab instance should send data to InfluxDB. Read more on: -- [Introduction to GitLab Metrics](introduction.md) +- [Introduction to GitLab Performance Monitoring](introduction.md) - [GitLab Configuration](gitlab_configuration.md) - [InfluxDB Schema](influxdb_schema.md) diff --git a/doc/integration/metrics/influxdb_schema.md b/doc/integration/metrics/influxdb_schema.md index a9ef0b446c..a5a8aebd2d 100644 --- a/doc/integration/metrics/influxdb_schema.md +++ b/doc/integration/metrics/influxdb_schema.md @@ -82,6 +82,6 @@ view. Read more on: -- [Introduction to GitLab Metrics](introduction.md) +- [Introduction to GitLab Performance Monitoring](introduction.md) - [GitLab Configuration](gitlab_configuration.md) - [InfluxDB Configuration](influxdb_configuration.md) diff --git a/doc/integration/metrics/introduction.md b/doc/integration/metrics/introduction.md index bf388aa2bb..f2460d3130 100644 --- a/doc/integration/metrics/introduction.md +++ b/doc/integration/metrics/introduction.md @@ -1,19 +1,19 @@ -# GitLab Metrics +# GitLab Performance Monitoring GitLab comes with its own application performance measuring system as of GitLab -8.4, simply called "GitLab Metrics". GitLab Metrics is available in both the +8.4, simply called "GitLab Performance Monitoring". GitLab Performance Monitoring is available in both the Community and Enterprise editions. Apart from this introduction, you are advised to read through the following -documents in order to understand and properly configure GitLab Metrics: +documents in order to understand and properly configure GitLab Performance Monitoring: - [GitLab Configuration](gitlab_configuration.md) - [InfluxDB Configuration](influxdb_configuration.md) - [InfluxDB Schema](influxdb_schema.md) -## Introduction to GitLab Metrics +## Introduction to GitLab Performance Monitoring -GitLab Metrics makes it possible to measure a wide variety of statistics +GitLab Performance Monitoring makes it possible to measure a wide variety of statistics including (but not limited to): - The time it took to complete a transaction (a web request or Sidekiq job). From 82bc7679e782059568402a282905d3d607dbb97d Mon Sep 17 00:00:00 2001 From: Achilleas Pipinellis Date: Wed, 20 Jan 2016 23:02:24 +0100 Subject: [PATCH 4/5] Move integration/metrics to monitoring/performance [ci skip] --- doc/README.md | 2 +- .../performance}/gitlab_configuration.md | 0 .../img/metrics_gitlab_configuration_settings.png | Bin .../performance}/influxdb_configuration.md | 0 .../performance}/influxdb_schema.md | 0 .../performance}/introduction.md | 0 6 files changed, 1 insertion(+), 1 deletion(-) rename doc/{integration/metrics => monitoring/performance}/gitlab_configuration.md (100%) rename doc/{integration/metrics => monitoring/performance}/img/metrics_gitlab_configuration_settings.png (100%) rename doc/{integration/metrics => monitoring/performance}/influxdb_configuration.md (100%) rename doc/{integration/metrics => monitoring/performance}/influxdb_schema.md (100%) rename doc/{integration/metrics => monitoring/performance}/introduction.md (100%) diff --git a/doc/README.md b/doc/README.md index 05c0d6fa0f..1efed2871f 100644 --- a/doc/README.md +++ b/doc/README.md @@ -67,7 +67,7 @@ - [Reply by email](incoming_email/README.md) Allow users to comment on issues and merge requests by replying to notification emails. - [Migrate GitLab CI to CE/EE](migrate_ci_to_ce/README.md) Follow this guide to migrate your existing GitLab CI data to GitLab CE/EE. - [Git LFS configuration](workflow/lfs/lfs_administration.md) -- [GitLab Performance Monitoring](integration/metrics/introduction.md) Configure GitLab and InfluxDB for measuring performance metrics +- [GitLab Performance Monitoring](monitoring/performance/introduction.md) Configure GitLab and InfluxDB for measuring performance metrics ## Contributor documentation diff --git a/doc/integration/metrics/gitlab_configuration.md b/doc/monitoring/performance/gitlab_configuration.md similarity index 100% rename from doc/integration/metrics/gitlab_configuration.md rename to doc/monitoring/performance/gitlab_configuration.md diff --git a/doc/integration/metrics/img/metrics_gitlab_configuration_settings.png b/doc/monitoring/performance/img/metrics_gitlab_configuration_settings.png similarity index 100% rename from doc/integration/metrics/img/metrics_gitlab_configuration_settings.png rename to doc/monitoring/performance/img/metrics_gitlab_configuration_settings.png diff --git a/doc/integration/metrics/influxdb_configuration.md b/doc/monitoring/performance/influxdb_configuration.md similarity index 100% rename from doc/integration/metrics/influxdb_configuration.md rename to doc/monitoring/performance/influxdb_configuration.md diff --git a/doc/integration/metrics/influxdb_schema.md b/doc/monitoring/performance/influxdb_schema.md similarity index 100% rename from doc/integration/metrics/influxdb_schema.md rename to doc/monitoring/performance/influxdb_schema.md diff --git a/doc/integration/metrics/introduction.md b/doc/monitoring/performance/introduction.md similarity index 100% rename from doc/integration/metrics/introduction.md rename to doc/monitoring/performance/introduction.md From dd59fc213c6cf2a2efb39eda241f453bbd1e82c7 Mon Sep 17 00:00:00 2001 From: Achilleas Pipinellis Date: Wed, 20 Jan 2016 23:10:27 +0100 Subject: [PATCH 5/5] Change InfluxDB admin username [ci skip] --- doc/monitoring/performance/influxdb_configuration.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/doc/monitoring/performance/influxdb_configuration.md b/doc/monitoring/performance/influxdb_configuration.md index df5e4af7fa..3a2b598b78 100644 --- a/doc/monitoring/performance/influxdb_configuration.md +++ b/doc/monitoring/performance/influxdb_configuration.md @@ -135,7 +135,7 @@ If you want to [enable authentication](#http), you might want to [create an admin user][influx-admin]: ``` -influx -execute "CREATE USER thedude WITH PASSWORD '1234' WITH ALL PRIVILEGES" +influx -execute "CREATE USER jeff WITH PASSWORD '1234' WITH ALL PRIVILEGES" ``` ## Create the `gitlab` database @@ -145,8 +145,8 @@ Make sure you have changed the [storage engine](#storage-engine) to `tsm1` before creating a database. _**Note:** If you [created an admin user](#create-a-new-admin-user) and enabled -[HTTP authentication](#http), remember to append the username (`-username thedude`) -and password (`-password 1234`) to the commands below._ +[HTTP authentication](#http), remember to append the username (`-username `) +and password (`-password `) you set earlier to the commands below._ Run the following command to create a database named `gitlab`: