Add lib/gitlab/emoji.rb instead of gitlab_emoji gem and upgrade Gemojione
- No reason to split it into a separate gem when the gem barely did anything. We can use gemojione directly, making updating gemojione that much easier. Unless there's a particularly good reason we were using the gem?
- Fixes the Rake task since it broke after all the AwardEmoji changes.
- Update gemojione to 2.6.1.
Spring Update changes! http://emojione.com/releases/#2.2.0
See merge request !4919
IssueReferenceFilter will end up processing internal issue references
when a project uses an external issues tracker while still using
internal issue references (in the form of `#\d+`). This commit ensures
that these links are rendered as external issue links, regardless of
whether the project one currently views uses an internal or external
issues tracker.
Fixesgitlab-org/gitlab-ce#19036, gitlab-com/performance#16
No reason to split it into a separate gem when the gem barely did anything. We can use gemojione directly, making updating gemojione that much easier.
Also fix the Rake task and update gemojione to 2.6.1. This adds the EmojiOne Spring update.
Changelog: https://github.com/jonathanwiesel/gemojione/blob/master/CHANGELOG.md
This commit changes the way certain documents are rendered (currently
only Notes) and how documents are redacted. Previously both rendering
and redacting would run on a per document basis. The result of this was
that for every document we'd have to run countless queries just to
figure out if we could display a set of links or not.
This commit changes things around so that redacting Markdown documents
is no longer tied into the html-pipeline Gem. This in turn allows it to
redact multiple documents in a single pass, thus reducing the number of
queries needed.
In turn rendering issue/merge request notes has been adjusted to take
advantage of this new setup. Instead of rendering Markdown somewhere
deep down in a view the Markdown is rendered and redacted in the
controller (taking the current user and all that into account). This has
been done in such a way that the "markdown()" helper method can still be
used on its own.
This particular commit also paves the way for caching rendered HTML on
object level. Right now there's an accessor method Note#note_html which
is used for setting/getting the rendered HTML. Once we cache HTML on row
level we can simply change this field to be a column and call a "save"
whenever needed and we're pretty much done.
Handle external issues in IssueReferenceFilter
Handling of external issues was broken when I refactored
`IssueReferenceFilter` to use fewer SQL queries.
Fixes#18827
See merge request !4789
Move pre_process into render_result
This MR moves `Banzai::Renderer.pre_process` into `Banzai::Renderer.render_result`.
The `pre_process` method was called even when its output would be ignored. See 11a5a4f359ee57029dbfcc9185fc6b47243ea2aa for more details.
See merge request !4830
In the past this class would use Project#get_issue to retrieve an issue
by its ID. This method would automatically determine whether to return
an Issue or ExternalIssue.
This commit changes IssueReferenceFilter to handle external issues again
and in a somewhat more explicit manner than before.
Fixesgitlab-org/gitlab-ce#18827
The method Banzai::Renderer.pre_process would always be called,
regardless of whether the Markdown to render was already cached or not.
In cache the document _was_ cached the output of the pre-processing
pipeline was ignored resulting in it doing nothing but wasting CPU
cycles.
This commit moves Banzai::Renderer.pre_process into
Banzai::Renderer.render_result so that it's _only_ used when needed.
A lot of git operations were being repeated, for example, to build a url
you would ask if the path was a Tree, which would call a recursive routine
in Gitlab::Git::Tree#where, then ask if the path was a Blob, which would
call a recursive routine at Gitlab::Git::Blob#find, making reference to
the same git objects several times. Now we call Rugged::Tree#path, which
allows us to determine the type of the path in one pass.
Some other minor improvement added, like saving commonly used references
instead of calculating them each time.
Fix RangeError exceptions when referring to issues or merge requests outside of max database values
When using #XYZ in Markdown text, if XYZ exceeds the maximum value of a signed 32-bit integer, we get an exception when the Markdown render attempts to run `where(iids: XYZ)`. Introduce a method that will throw out out-of-bounds values.
Closes#18777
See merge request !4777
When using #XYZ in Markdown text, if XYZ exceeds the maximum value of a signed 32-bit integer, we
get an exception when the Markdown render attempts to run `where(iids: XYZ)`. Introduce a method
that will throw out out-of-bounds values.
Closes#18777
1. An exception would be raised if the filter was called with an invalid
URI. Mainly because we weren't catching the `Addressable` exception.
2. This commit fixes it and adds a spec for the filter.
Reduce SQL query counts in IssueReferenceFilter
## What does this MR do?
This MR adds a preparation phase for reference filters that allows them to prepare/create data structures used while iterating over HTML nodes. In this particular case the preparation phase is used for issue references to greatly cut down the amount of queries executed to get projects/issues for Markdown references.
## Are there points in the code the reviewer needs to double check?
No.
## Why was this MR needed?
Rendering Markdown containing issue references would run at most two queries for every issue reference: one to get the project and one to get the issue from said project. When rendering Markdown with lots of issue references this would result in _a lot_ of queries being executed.
## What are the relevant issue numbers?
#18042
See merge request !4410
This reduces the number of queries executed in IssueReferenceFilter by
retrieving the various projects/issues that may be referenced in batches
_before_ iterating over all the HTML nodes.
A chunk of the logic resides in AbstractReferenceFilter so it can be
re-used by other filters in the future.
By eager loading these associations we can greatly cut down the number
of SQL queries executed when processing documents with lots of
references, especially in cases where there are references belonging to
the same project.
Since these associations are so specific to the reference parsing
process and the permissions checking process that follows it I opted to
include them directly in IssueParser instead of using something like a
scope. Once we have a need for it we can move this code to a scope or
method.
Consider this command:
bundle exec rails r "include GitlabMarkdownHelper
puts markdown('<span>this is a span</span>', pipeline: :description)
puts markdown('<span>this is a span</span>')"
And the same in the opposite order:
bundle exec rails r "include GitlabMarkdownHelper
puts markdown('<span>this is a span</span>')
puts markdown('<span>this is a span</span>', pipeline: :description)"
Before this change, they would both output:
<p><span>this is a span</span></p>
<p>this is a span</p>
That's because `span` is added to the list of whitelisted elements in
the `SanitizationFilter`, but this method tries not to make the same
changes multiple times. Unfortunately,
`HTML::Pipeline::SanitizationFilter::LIMITED`, which is used by the
`DescriptionPipeline`, uses the same Ruby objects for all of its hash
values _except_ `:elements`.
That means that whichever of `DescriptionPipeline` and `GfmPipeline` is
called first would have `span` in its whitelisted elements, and the
second wouldn't.
Fix this by creating an entirely separate hash, before either pipeline
is invoked.
Fix wiki linking behavior for markdown wiki pages
Related to #18019
- As per the documentation in !4372
## TODO
- [ ] !4432 Have wiki linking behave as per the documentation
- [x] Move `WikiLinkFilter` specs to the pipeline level
- [x] Verify current behavior on wiki `show` page
- [x] Fix current behavior on wiki `show` page
- [x] Verify current behaviour on wiki preview
- [x] Fix current behaviour on wiki preview
- [x] Rewrite all links and get preview links working
- [x] Make sure all links are on-par with the wiki `show` page
- [x] TDD `WikiLinkFilter` and get it working
- [x] Hook `WikiLinkFilter` up
- [x] Fix tests
- [x] Fix `markdown_spec`
- [x] Fix `wiki` spinach feature
- [x] Wait for [build](https://gitlab.com/gitlab-org/gitlab-ce/commit/4f50dd2/builds) to pass
- [x] Make sure all wiki-related pages are working as expected (history, all pages, etc.)
- [x] Test in different ruby versions
- [x] GitLab instances hosted on a relative URL
- [x] Non-markdown rendering formats?
- [x] RDoc
- [x] ASCIIDoc
- [x] Create issues to fix things for RDoc and ASCIIDoc
- [x] Gauge performance impact
- [x] Refactor
- [x] Re-organize commits
- [x] Make sure [build](https://gitlab.com/gitlab-org/gitlab-ce/commit/f860e9a8dcabe7d5f160c32fc549807c98baa4a1/builds) passes
- [x] Respond to @rymai's comments
- [x] `class WikiLinkFilter < HTML::Pipeline::Filter`
- [x] blank line after guard clause
- [x] keyword arguments for `wiki` and `slug`
- [x] invert the condition
- [x] inline `user` in spec
- [x] Make sure spec names are not out of date
- [x] Comment for each rewrite rule
- [x] Add CHANGELOG entry
- [x] Reorganize commits
- [x] Make sure [build](https://gitlab.com/gitlab-org/gitlab-ce/commit/19b91e749a6320d12fb299d33f1f6440777e0e26/builds) passes
- [ ] Wait for merge
See merge request !4432
Original Comments
=================
- Linking behaves as per rules documented here:
https://gitlab.com/gitlab-org/gitlab-ce/blob/16568-document-wiki-linking-behavior/doc/markdown/wiki.md
- All links (to other wiki pages) are rewritten to be at the level of
the app root. We can't use links relative to the current
page ('./foo', 'foo', '../foo'), because they won't work in the
markdown preview, where the current page is suffixed with `/edit`
- Move existing `WikiLinkFilter` specs to `WikiPipeline` spec. It makes
sense to run these tests on the combined output of the pipeline,
rather than a single filter, since we can catch issues with
conflicting filters.
- Add more tests to cover the new linking
@rymai's Review
===============
- Classes nested under `WikiLinkFilter` should declare `WikiLinkFilter`'s
inherit, so nothing changes if the nested class is loaded first.
- Add a blank line after a guard clause
- Use keyword arguments for the `Rewriter` constructor
- Invert a condition - use `if` instead of `unless`
- Inline a `let` in `WikiPipeline` spec - it was only used in a single place
- Change out of date spec names
- Add a comment for every rewrite rule in `Rewriter`
This changes UserReferenceFilter so it operates using the following
steps:
1. Grab all username references from the input document.
2. Query the corresponding Namespace objects using a single query.
3. Iterate over all nodes to build links while re-using the objects
queried in step 2.
The impact of these changes is that a comment mentioning 5 different
usernames no longer runs 5 different queries (1 for every username),
instead it only runs a single query.
This method returns an Array of the HTML nodes as yielded by
ReferenceFilter#each_node. The method's return value is memoized to
allow multiple calls without having to re-query the input document.
Nokogiri's `node.replace` was being unnecessarily called for every text node in
the document due to a comparison bug. The code previously was comparing the
HTML representation of the full document against the text node, which would
always fail. Fix the comparison to just compare the modified text.
Closes#18011
Separate reference gathering from rendering
This is a required step to allow batch processing when gathering references. This in turn would allow grabbing (for example) all mentioned users of an issue/merge request using a single query.
cc @rspeicher @DouweM
See merge request !3969
A user viewing the TODOs page will see a 404 if there are mentioned labels
in multiple different projects. This is likely a caching bug and only occurs
when Markdown rendering occurs across multiple projects, which is why it's so
tricky to reproduce. This is what I think is happening:
1. LabelReferenceFilter#references_in encounters label ~X for ProjectA and finds the label in the DB as id = 1.
2. LabelReferenceFilter.references_in yields [1, 'X', nil, ...]
3. Since project_ref is nil, AbstractReferenceFilter#project_from_ref_cache caches nil => ProjectA.
4. LabelReferenceFilter#references_in encounters label ~Y for ProjectB and finds the label in the DB as id = 2.
5. LabelReferenceFilter.references_in yields [2, 'Y', nil, ...]
6. AbstractReferenceFilter#project_from_ref_cache lookups nil and returns ProjectA. It was supposed to be ProjectB.
7. A is the wrong project, so the label lookup fails.
This MR caches Markdown references if the key is present.
Closes#17898
This splits the Markdown rendering and reference extraction phases into
two distinct code bases. The reference extraction phase no longer relies
on the html-pipeline Gem (and any related code) and allows for
extracting of references from multiple HTML nodes in a single pass. This
means that if you want to extract user references from 200 comments you
no longer need to run 200 times N number of queries, instead only a
handful of queries may be needed.
Douglas Barbosa Alexandre
Robert Speicher
Grzegorz Bizon
Paco Guzman
Yorick Peterse
Connor Shea
Alejandro Rodríguez
Stan Hu
Timothy Andrew
Douwe Maan
Sean McGivern
Rémy Coutable
Alfredo Sumaran
Douwe Maan
Adam ButlerandRémy Coutable
Dmitriy Zaporozhets
Benedikt Huss