Reduce SQL query counts in IssueReferenceFilter
## What does this MR do?
This MR adds a preparation phase for reference filters that allows them to prepare/create data structures used while iterating over HTML nodes. In this particular case the preparation phase is used for issue references to greatly cut down the amount of queries executed to get projects/issues for Markdown references.
## Are there points in the code the reviewer needs to double check?
No.
## Why was this MR needed?
Rendering Markdown containing issue references would run at most two queries for every issue reference: one to get the project and one to get the issue from said project. When rendering Markdown with lots of issue references this would result in _a lot_ of queries being executed.
## What are the relevant issue numbers?
#18042
See merge request !4410
This reduces the number of queries executed in IssueReferenceFilter by
retrieving the various projects/issues that may be referenced in batches
_before_ iterating over all the HTML nodes.
A chunk of the logic resides in AbstractReferenceFilter so it can be
re-used by other filters in the future.
By eager loading these associations we can greatly cut down the number
of SQL queries executed when processing documents with lots of
references, especially in cases where there are references belonging to
the same project.
Since these associations are so specific to the reference parsing
process and the permissions checking process that follows it I opted to
include them directly in IssueParser instead of using something like a
scope. Once we have a need for it we can move this code to a scope or
method.
Consider this command:
bundle exec rails r "include GitlabMarkdownHelper
puts markdown('<span>this is a span</span>', pipeline: :description)
puts markdown('<span>this is a span</span>')"
And the same in the opposite order:
bundle exec rails r "include GitlabMarkdownHelper
puts markdown('<span>this is a span</span>')
puts markdown('<span>this is a span</span>', pipeline: :description)"
Before this change, they would both output:
<p><span>this is a span</span></p>
<p>this is a span</p>
That's because `span` is added to the list of whitelisted elements in
the `SanitizationFilter`, but this method tries not to make the same
changes multiple times. Unfortunately,
`HTML::Pipeline::SanitizationFilter::LIMITED`, which is used by the
`DescriptionPipeline`, uses the same Ruby objects for all of its hash
values _except_ `:elements`.
That means that whichever of `DescriptionPipeline` and `GfmPipeline` is
called first would have `span` in its whitelisted elements, and the
second wouldn't.
Fix this by creating an entirely separate hash, before either pipeline
is invoked.
Fix wiki linking behavior for markdown wiki pages
Related to #18019
- As per the documentation in !4372
## TODO
- [ ] !4432 Have wiki linking behave as per the documentation
- [x] Move `WikiLinkFilter` specs to the pipeline level
- [x] Verify current behavior on wiki `show` page
- [x] Fix current behavior on wiki `show` page
- [x] Verify current behaviour on wiki preview
- [x] Fix current behaviour on wiki preview
- [x] Rewrite all links and get preview links working
- [x] Make sure all links are on-par with the wiki `show` page
- [x] TDD `WikiLinkFilter` and get it working
- [x] Hook `WikiLinkFilter` up
- [x] Fix tests
- [x] Fix `markdown_spec`
- [x] Fix `wiki` spinach feature
- [x] Wait for [build](https://gitlab.com/gitlab-org/gitlab-ce/commit/4f50dd2/builds) to pass
- [x] Make sure all wiki-related pages are working as expected (history, all pages, etc.)
- [x] Test in different ruby versions
- [x] GitLab instances hosted on a relative URL
- [x] Non-markdown rendering formats?
- [x] RDoc
- [x] ASCIIDoc
- [x] Create issues to fix things for RDoc and ASCIIDoc
- [x] Gauge performance impact
- [x] Refactor
- [x] Re-organize commits
- [x] Make sure [build](https://gitlab.com/gitlab-org/gitlab-ce/commit/f860e9a8dcabe7d5f160c32fc549807c98baa4a1/builds) passes
- [x] Respond to @rymai's comments
- [x] `class WikiLinkFilter < HTML::Pipeline::Filter`
- [x] blank line after guard clause
- [x] keyword arguments for `wiki` and `slug`
- [x] invert the condition
- [x] inline `user` in spec
- [x] Make sure spec names are not out of date
- [x] Comment for each rewrite rule
- [x] Add CHANGELOG entry
- [x] Reorganize commits
- [x] Make sure [build](https://gitlab.com/gitlab-org/gitlab-ce/commit/19b91e749a6320d12fb299d33f1f6440777e0e26/builds) passes
- [ ] Wait for merge
See merge request !4432
Original Comments
=================
- Linking behaves as per rules documented here:
https://gitlab.com/gitlab-org/gitlab-ce/blob/16568-document-wiki-linking-behavior/doc/markdown/wiki.md
- All links (to other wiki pages) are rewritten to be at the level of
the app root. We can't use links relative to the current
page ('./foo', 'foo', '../foo'), because they won't work in the
markdown preview, where the current page is suffixed with `/edit`
- Move existing `WikiLinkFilter` specs to `WikiPipeline` spec. It makes
sense to run these tests on the combined output of the pipeline,
rather than a single filter, since we can catch issues with
conflicting filters.
- Add more tests to cover the new linking
@rymai's Review
===============
- Classes nested under `WikiLinkFilter` should declare `WikiLinkFilter`'s
inherit, so nothing changes if the nested class is loaded first.
- Add a blank line after a guard clause
- Use keyword arguments for the `Rewriter` constructor
- Invert a condition - use `if` instead of `unless`
- Inline a `let` in `WikiPipeline` spec - it was only used in a single place
- Change out of date spec names
- Add a comment for every rewrite rule in `Rewriter`
This changes UserReferenceFilter so it operates using the following
steps:
1. Grab all username references from the input document.
2. Query the corresponding Namespace objects using a single query.
3. Iterate over all nodes to build links while re-using the objects
queried in step 2.
The impact of these changes is that a comment mentioning 5 different
usernames no longer runs 5 different queries (1 for every username),
instead it only runs a single query.
This method returns an Array of the HTML nodes as yielded by
ReferenceFilter#each_node. The method's return value is memoized to
allow multiple calls without having to re-query the input document.
Nokogiri's `node.replace` was being unnecessarily called for every text node in
the document due to a comparison bug. The code previously was comparing the
HTML representation of the full document against the text node, which would
always fail. Fix the comparison to just compare the modified text.
Closes#18011
Separate reference gathering from rendering
This is a required step to allow batch processing when gathering references. This in turn would allow grabbing (for example) all mentioned users of an issue/merge request using a single query.
cc @rspeicher @DouweM
See merge request !3969
A user viewing the TODOs page will see a 404 if there are mentioned labels
in multiple different projects. This is likely a caching bug and only occurs
when Markdown rendering occurs across multiple projects, which is why it's so
tricky to reproduce. This is what I think is happening:
1. LabelReferenceFilter#references_in encounters label ~X for ProjectA and finds the label in the DB as id = 1.
2. LabelReferenceFilter.references_in yields [1, 'X', nil, ...]
3. Since project_ref is nil, AbstractReferenceFilter#project_from_ref_cache caches nil => ProjectA.
4. LabelReferenceFilter#references_in encounters label ~Y for ProjectB and finds the label in the DB as id = 2.
5. LabelReferenceFilter.references_in yields [2, 'Y', nil, ...]
6. AbstractReferenceFilter#project_from_ref_cache lookups nil and returns ProjectA. It was supposed to be ProjectB.
7. A is the wrong project, so the label lookup fails.
This MR caches Markdown references if the key is present.
Closes#17898
This splits the Markdown rendering and reference extraction phases into
two distinct code bases. The reference extraction phase no longer relies
on the html-pipeline Gem (and any related code) and allows for
extracting of references from multiple HTML nodes in a single pass. This
means that if you want to extract user references from 200 comments you
no longer need to run 200 times N number of queries, instead only a
handful of queries may be needed.
Wiki preview URL converting problem [via Markdown]
Current implementation when rendering the preview, thinks relative links are for project repository files.
We are creating a new preview route that will define correct context data to render for wikis instead.
Fixes#2380, #1184
See merge request !3461
This fixes the problem where Markdown such as:
### 31st
Would get rendered as a link tag pointing to issue number 31 inside a
header tag.
See gitlab-org/gitlab-ce#14936 for more information.
Every object processed by ExternalIssueReferenceFilter can return a
different Project instance when calling "project". For example, every
note processed will have it's own associated Project. If we were to
cache Project#default_issues_tracker? on Project level this would have
no impact on Markdown rendering timings as the cache would have to be
built for every Project instance without it ever being re-used.
To work around this we cache Project#default_issues_tracker? in
Banzai::Filter::ExternalIssueReferenceFilter using the project's _id_
instead of the whole object. This setup allows re-using of the cached
data even when the Project instances used are different, as long as the
actual project IDs are the same.
Douwe Maan
Paco Guzman
Yorick Peterse
Sean McGivern
Rémy Coutable
Timothy Andrew
Alfredo Sumaran
Douwe Maan
Stan Hu
Adam ButlerandRémy Coutable
Robert Speicher
Grzegorz Bizon
Dmitriy Zaporozhets
Benedikt Huss
Alejandro Rodríguez
Gabriel Mazetto
connorshea