Reduce Rack Attack false positives causing 403 errors during HTTP authentication
### What does this MR do?
This MR reduces false positives causing `403 Forbidden` messages after HTTP authentication.
A Git client may attempt to access a repository without a password. If it receives a 401 error, the client often will try again, this time supplying a password. The problem is that `grack_auth.rb` considers a blank password an authentication failure and increases a Redis counter each time this happens. With enough requests, an IP can be banned temporarily even though previous attempts may have been successful. This leads users to see `403 Forbidden` errors until the ban times out (default: 1 hour).
To reduce the chance of a false positive, this MR resets the counter upon a successful authentication from an IP.
In addition, this MR logs when a user has been banned and introduces the ability to disable Rack Attack via a config variable.
### Are there points in the code the reviewer needs to double check?
rack-attack v4.2.0 doesn't support the ability to clear counters out of the box, so `rack_attack_helpers.rb` includes a number of monkey patches to make it work. It looks like this functionality may be added in v4.3.0. I've also sent pull requests to rack-attack to add the functionality necessary to delete a key.
Each time an authentication is successful, the Redis counter for that IP is cleared. I deemed it better to clear the counter than to allow for blank passwords, since the latter seems like a security risk.
### Why was this MR needed?
It was quite difficult to figure out why users were seeing `403 Forbidden`, which is why the log message was added. Users were getting a lot of false positives when accessing repositories with HTTPS. Including the username in the HTTPS URL (e.g. `https://username@mydomain.com/account/repo.git`) caused authentication failures because while the git client provided the username, it left the password blank, leading to an authentication failure.
### What are the relevant issue numbers / [Feature requests](http://feedback.gitlab.com/)?
See Issue #1171https://github.com/kickstarter/rack-attack/issues/113
See merge request !392
Fix nested task lists
When nesting task list items, the parent item is wrapped in a `<p>` tag. Update the task list parser to handle these paragraph wrappers.
cc @sytse
See merge request !413
Replace commits calendar with contributions calendar
* count opening of issues and merge requests
* dont trigger git repository - use events from database
* count pushes instead of commits for faster and easier counting
* much-much faster since does not affected by repository size
See merge request !420
Return a `SafeBuffer` instead of a `String` from the `#gfm_with_options`
method so that Rails doesn't escape our markup.
Also add `<span>` to the sanitization whitelist to avoid breaking syntax
highlighting in code blocks.
Disable reference generation in preformatted/code blocks
### Summary
If a user adds text in code or preformatted text via Markdown or HTML that contains `#XXX`, the system adds a note that issue `XXX` was mentioned. This is particularly annoying because we often list gdb backtrace dumps into our issues, and many issues get mentioned as a result. For example:
```
(gdb) bt
#0 0x00000000004004c4 in second () at main.cc:6
#1 0x00000000004004d2 in first () at main.cc:11
#2 0x00000000004004dd in main () at main.cc:17
(gdb)
```
### Steps to reproduce
1. In an issue, write the above text using Markdown or HTML tags (e.g. `<code>`, `<pre>`).
2. Observe that [issue 1](https://gitlab.com/gitlab-org/gitlab-ce/issues/1) and [issue 2](https://gitlab.com/gitlab-org/gitlab-ce/issues/2) have a note that says they were mentioned.
### Expected behavior
Everything enclosed in the code blocks should be ignored as references.
### Observed behavior
Issues get referenced unnecessarily.
### Fix
I've made `reference_extractor.rb` strip out HTML and Markdown blocks before processing. I considered running the raw text through the entire Markdown processor, but this seems overkill and perhaps could lead to some unintended side effects.
See merge request !365
* count opening of issues and merge requests
* dont trigger git repository - use events from database
* much-much faster since does not affected by repository size
The Net::LDAP::Filter.escape function can not be used to escape the DN name because the backslash is required to escape special chars in the DN name. This leads to the error message "Access denied for your LDAP account." and prevents the user from logging in to gitlab.
Example DN:
CN=Test\, User,OU=Organization,DC=Company
CN=Test User,OU=Organization,DC=Company
http://www.ietf.org/rfc/rfc4514.txt
Fix invalid Atom feeds when using emoji, horizontal rules, or images
This is a fix for issues #880, #723, #1113.
Markdown must be rendered to XHTML, not HTML, when generating summary content for Atom feeds. Otherwise, content-less tags like *img* and *hr* are not terminated and make the Atom XML invalid. Such tags are generated when issue descriptions, merge request descriptions, comments, or commit messages use emoji, horizontal rules, or images.
To pass this option through from the relevant Haml templates to the proper place in the `gfm()` method, a new method `gfm_with_options()` is introduced. It reuses the options dictionary passed to `markdown()` and interprets options `xhtml` and `parse_tasks` from it (the latter was a convenient replacement for `gfm_with_tasks()`). `xhtml` is already interpreted by Redcarpet::Render::HTML, but that alone was not sufficient, because the post-processing in `gfm()` would convert its XHTML tags back to HTML.
I found no way of passing additional optional options to the existing `gfm()` method without requiring updates to existing callers and without getting in the way of the existing optional arguments, but maybe someone who knows more about Ruby than I can think of one.
Thorough review appreciated since this is the first time I have used Ruby.
See merge request !344
Fixes issues #880, #723, #1113: Markdown must be rendered to XHTML, not HTML, when generating summary content for Atom feeds. Otherwise, content-less tags like <img> and <hr>, generated when issue descriptions, merge request descriptions, comments, or commit messages use emoji, horizontal rules, or images, are not terminated and make the Atom XML invalid.
Restricted visibility levels - bug fix and new feature
This allows admin users to override restricted visibility settings when creating and updating projects and snippets, and moves the restricted visibility configuration from gitlab.yml to the web UI. See #1903.
## Move configuration location
I added a new section to the application settings page for restricted visibility levels. Each level has a checkbox, styled with Bootstrap to look like a toggle button. A checked box means that the level is restricted. I added a glowing text shadow and changed the background color for checked buttons because the default styles made it hard to distinguish between checked and unchecked. This image shows the new section with the "Public" box checked:
## Allow admins to override
To allow admin users to override the restricted visibility levels, I had to remove the `visibility_level` validation from the `Project` class. The model doesn't know about the `current_user`, which should determine whether the restrictions can be overridden. We could use the creator in the validation, but that wouldn't work correctly for projects where a non-admin user is the creator and an admin tries to change the project to a restricted visibility level.
The `Project::UpdateService` and `Project::CreateService` classes already had code to determine whether the current user is allowed to use a given visibility level; now all visibility level validation is done in those classes. Currently, when a non-admin tries to create or update a project using a restricted level, these classes silently set the visibility level to the global default (create) or the project's existing value (update). I changed this behavior to be more like an Active Model validation, where using a restricted level causes the entire request to be rejected.
Project and personal snippets didn't have service classes, and restricted visibility levels weren't being enforced in the model or the controllers. The UI disabled radio buttons for restricted levels, but that wouldn't be difficult to circumvent. I created the `CreateSnippetService` and `UpdateSnippetService` classes to do the same restricted visibility check that the project classes do. And since I was dealing with snippet visibility levels, I updated the API endpoints for project snippets to allow users to set and update the visibility level.
## TODO
* [x] Add more tests for restricted visibility functionality
cc @sytse @dzaporozhets
See merge request !1655
Execute hooks and services when branch or tag is created or deleted through web interface.
Fixes#2095.
Split up into commits to make it easier to see why what was changed :)
See merge request !1692
Dmitriy Zaporozhets
nicklegr
Vinnie Okada
Douwe Maan
Stan Hu
Douwe Maan
Michael Alt
Kamil Trzcinski
Marin Jankovski
Vinnie Okada
Dmitriy Zaporozhets
Hannes Rosenögger
Christian Walther
