Separate reference gathering from rendering
This is a required step to allow batch processing when gathering references. This in turn would allow grabbing (for example) all mentioned users of an issue/merge request using a single query.
cc @rspeicher @DouweM
See merge request !3969
A user viewing the TODOs page will see a 404 if there are mentioned labels
in multiple different projects. This is likely a caching bug and only occurs
when Markdown rendering occurs across multiple projects, which is why it's so
tricky to reproduce. This is what I think is happening:
1. LabelReferenceFilter#references_in encounters label ~X for ProjectA and finds the label in the DB as id = 1.
2. LabelReferenceFilter.references_in yields [1, 'X', nil, ...]
3. Since project_ref is nil, AbstractReferenceFilter#project_from_ref_cache caches nil => ProjectA.
4. LabelReferenceFilter#references_in encounters label ~Y for ProjectB and finds the label in the DB as id = 2.
5. LabelReferenceFilter.references_in yields [2, 'Y', nil, ...]
6. AbstractReferenceFilter#project_from_ref_cache lookups nil and returns ProjectA. It was supposed to be ProjectB.
7. A is the wrong project, so the label lookup fails.
This MR caches Markdown references if the key is present.
Closes#17898
This splits the Markdown rendering and reference extraction phases into
two distinct code bases. The reference extraction phase no longer relies
on the html-pipeline Gem (and any related code) and allows for
extracting of references from multiple HTML nodes in a single pass. This
means that if you want to extract user references from 200 comments you
no longer need to run 200 times N number of queries, instead only a
handful of queries may be needed.
Wiki preview URL converting problem [via Markdown]
Current implementation when rendering the preview, thinks relative links are for project repository files.
We are creating a new preview route that will define correct context data to render for wikis instead.
Fixes#2380, #1184
See merge request !3461
This fixes the problem where Markdown such as:
### 31st
Would get rendered as a link tag pointing to issue number 31 inside a
header tag.
See gitlab-org/gitlab-ce#14936 for more information.
Every object processed by ExternalIssueReferenceFilter can return a
different Project instance when calling "project". For example, every
note processed will have it's own associated Project. If we were to
cache Project#default_issues_tracker? on Project level this would have
no impact on Markdown rendering timings as the cache would have to be
built for every Project instance without it ever being re-used.
To work around this we cache Project#default_issues_tracker? in
Banzai::Filter::ExternalIssueReferenceFilter using the project's _id_
instead of the whole object. This setup allows re-using of the cached
data even when the Project instances used are different, as long as the
actual project IDs are the same.
These filters now use a single iteration over all the document nodes
instead of multiple ones. This in turn allows variables to be re-used
(e.g. links only have to be unescaped once). Combined with some other
refactoring this can drastically reduce render timings.
This was originally suggested by @ayufan and modified to be a bit
cleaner and use RequestStore instead of a regular Hash.
By caching the output of the two methods involved the number of queries
is reduced significantly. For example, for an issue with 200 notes (of
which 100 reference a number of merge requests) this cuts down the
amount of queries from around 6300 to around 3300.
Rails' "url_helpers" method creates an anonymous Module (which a bunch
of methods) on every call. By caching the output of this method in a
dedicated method we can shave off about 10 seconds of loading time for
an issue with around 200 comments.
* master: (121 commits)
Dedupe labels in labels selector in Dashboard pages
Refactor colors and lists
Add a safeguard in MergeRequest#compute_diverged_commits_count
Fix an issue when the target branch of a MR had been deleted
Add avatar to issue and MR pages header
Cleanup somce css colors
Re-group scss variables
Refactor `Todo#target`
Fixes issue with filter label missing on labels & milestones
Rename `Todo#to_reference` to `Todo#target_reference`
Fixed failing tests
Updated controller with before_action Fixed other issues based on feedback
Fixes issue on dashboard issues
Full labels data in JSON
Fixed issue with labels dropdown getting wrong labels
Update CHANGELOG
Use `Note#for_project_snippet?` to skip notes on project snippet
Use `Commit#short_id` instead of `Commit.truncate_sha`
Reuse `for_commit?` on conditional validations
Update schema info comment on todo related files
...
Conflicts:
app/models/issue.rb
db/schema.rb
spec/models/issue_spec.rb
Douwe Maan
Stan Hu
Yorick Peterse
Douwe Maan
Rémy Coutable
Adam ButlerandRémy Coutable
Timothy Andrew
Robert Speicher
Grzegorz Bizon
Dmitriy Zaporozhets
Benedikt Huss
Alfredo Sumaran
Alejandro Rodríguez
Gabriel Mazetto
connorshea
Robert Speicher
Douglas Barbosa Alexandre