Reduce SQL query counts in IssueReferenceFilter
## What does this MR do?
This MR adds a preparation phase for reference filters that allows them to prepare/create data structures used while iterating over HTML nodes. In this particular case the preparation phase is used for issue references to greatly cut down the amount of queries executed to get projects/issues for Markdown references.
## Are there points in the code the reviewer needs to double check?
No.
## Why was this MR needed?
Rendering Markdown containing issue references would run at most two queries for every issue reference: one to get the project and one to get the issue from said project. When rendering Markdown with lots of issue references this would result in _a lot_ of queries being executed.
## What are the relevant issue numbers?
#18042
See merge request !4410
This reduces the number of queries executed in IssueReferenceFilter by
retrieving the various projects/issues that may be referenced in batches
_before_ iterating over all the HTML nodes.
A chunk of the logic resides in AbstractReferenceFilter so it can be
re-used by other filters in the future.
Fix wiki linking behavior for markdown wiki pages
Related to #18019
- As per the documentation in !4372
## TODO
- [ ] !4432 Have wiki linking behave as per the documentation
- [x] Move `WikiLinkFilter` specs to the pipeline level
- [x] Verify current behavior on wiki `show` page
- [x] Fix current behavior on wiki `show` page
- [x] Verify current behaviour on wiki preview
- [x] Fix current behaviour on wiki preview
- [x] Rewrite all links and get preview links working
- [x] Make sure all links are on-par with the wiki `show` page
- [x] TDD `WikiLinkFilter` and get it working
- [x] Hook `WikiLinkFilter` up
- [x] Fix tests
- [x] Fix `markdown_spec`
- [x] Fix `wiki` spinach feature
- [x] Wait for [build](https://gitlab.com/gitlab-org/gitlab-ce/commit/4f50dd2/builds) to pass
- [x] Make sure all wiki-related pages are working as expected (history, all pages, etc.)
- [x] Test in different ruby versions
- [x] GitLab instances hosted on a relative URL
- [x] Non-markdown rendering formats?
- [x] RDoc
- [x] ASCIIDoc
- [x] Create issues to fix things for RDoc and ASCIIDoc
- [x] Gauge performance impact
- [x] Refactor
- [x] Re-organize commits
- [x] Make sure [build](https://gitlab.com/gitlab-org/gitlab-ce/commit/f860e9a8dcabe7d5f160c32fc549807c98baa4a1/builds) passes
- [x] Respond to @rymai's comments
- [x] `class WikiLinkFilter < HTML::Pipeline::Filter`
- [x] blank line after guard clause
- [x] keyword arguments for `wiki` and `slug`
- [x] invert the condition
- [x] inline `user` in spec
- [x] Make sure spec names are not out of date
- [x] Comment for each rewrite rule
- [x] Add CHANGELOG entry
- [x] Reorganize commits
- [x] Make sure [build](https://gitlab.com/gitlab-org/gitlab-ce/commit/19b91e749a6320d12fb299d33f1f6440777e0e26/builds) passes
- [ ] Wait for merge
See merge request !4432
Original Comments
=================
- Linking behaves as per rules documented here:
https://gitlab.com/gitlab-org/gitlab-ce/blob/16568-document-wiki-linking-behavior/doc/markdown/wiki.md
- All links (to other wiki pages) are rewritten to be at the level of
the app root. We can't use links relative to the current
page ('./foo', 'foo', '../foo'), because they won't work in the
markdown preview, where the current page is suffixed with `/edit`
- Move existing `WikiLinkFilter` specs to `WikiPipeline` spec. It makes
sense to run these tests on the combined output of the pipeline,
rather than a single filter, since we can catch issues with
conflicting filters.
- Add more tests to cover the new linking
@rymai's Review
===============
- Classes nested under `WikiLinkFilter` should declare `WikiLinkFilter`'s
inherit, so nothing changes if the nested class is loaded first.
- Add a blank line after a guard clause
- Use keyword arguments for the `Rewriter` constructor
- Invert a condition - use `if` instead of `unless`
- Inline a `let` in `WikiPipeline` spec - it was only used in a single place
- Change out of date spec names
- Add a comment for every rewrite rule in `Rewriter`
This changes UserReferenceFilter so it operates using the following
steps:
1. Grab all username references from the input document.
2. Query the corresponding Namespace objects using a single query.
3. Iterate over all nodes to build links while re-using the objects
queried in step 2.
The impact of these changes is that a comment mentioning 5 different
usernames no longer runs 5 different queries (1 for every username),
instead it only runs a single query.
This method returns an Array of the HTML nodes as yielded by
ReferenceFilter#each_node. The method's return value is memoized to
allow multiple calls without having to re-query the input document.
This splits the Markdown rendering and reference extraction phases into
two distinct code bases. The reference extraction phase no longer relies
on the html-pipeline Gem (and any related code) and allows for
extracting of references from multiple HTML nodes in a single pass. This
means that if you want to extract user references from 200 comments you
no longer need to run 200 times N number of queries, instead only a
handful of queries may be needed.
Wiki preview URL converting problem [via Markdown]
Current implementation when rendering the preview, thinks relative links are for project repository files.
We are creating a new preview route that will define correct context data to render for wikis instead.
Fixes#2380, #1184
See merge request !3461
This fixes the problem where Markdown such as:
### 31st
Would get rendered as a link tag pointing to issue number 31 inside a
header tag.
See gitlab-org/gitlab-ce#14936 for more information.
Add label description in tooltip to labels in issue index and sidebar
The tooltip can get quite big, but it’s so useful to inform people about what these labels mean
new
@JobV writes on Slack:
> yeah I think it’s better than not, but I’d have to try it 😃
>
> cautious yes, revert if it’s annoying 😉
See merge request !3124
Add support for cross project references for labels
## Summary
Support for cross project references for labels.
## Rationale
1. Cross project label references are currently not supported in GitLab
1. `to_reference` method signature in `Label` model breaks the abstraction introduced in `Referable`.
`concerns/referable.rb: def to_reference(_from_project = nil)`
Signatures:
```
label.rb: def to_reference(format = :id)
commit_range.rb: def to_reference(from_project = nil)
commit.rb: def to_reference(from_project = nil)
external_issue.rb: def to_reference(_from_project = nil)
group.rb: def to_reference(_from_project = nil)
issue.rb: def to_reference(from_project = nil)
merge_request.rb: def to_reference(from_project = nil)
milestone.rb: def to_reference(from_project = nil)
project.rb: def to_reference(_from_project = nil)
snippet.rb: def to_reference(from_project = nil)
user.rb: def to_reference(_from_project = nil)
```
This MR suggests using `def to_reference(from_project = nil, format: :id)` which makes use of keyword arguments and preserves abstract interface.
1. We need support for cross project label references when we want to move issue to another project
It may happen that issue description, system notes or comments contain reference to label and this reference will be invalid after moving issue to another project and will not be displayed correctly unless we have support for cross project references.
Merge request that needs this feature: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/2831
I think that cross project label references may be useful, (example: `Hey, see our issues for CI in GitLab CE! - gitab-org/gitlab-ce~"CI"`).
cc @JobV @DouweM @rspeicher
See merge request !2966
Add a spec for WikiPipeline
Removes the specs from GollumTagsFilter that were more like integration
tests for the pipeline than unit tests of the filter.
See merge request !3054
Douwe Maan
Paco Guzman
Yorick Peterse
Douglas Barbosa Alexandre
Rémy Coutable
Timothy Andrew
Douwe Maan
Adam ButlerandRémy Coutable
Robert Speicher
Grzegorz Bizon
Stan Hu
Felipe Artur
Alfredo Sumaran
Alejandro Rodríguez
Gabriel Mazetto
connorshea
Arinde Eniola
Robert Speicher