Commit Graph
3826 Commits
Author SHA1 Message Date
Douwe Maan c19795dff5 Merge branch 'ci/build_dependencies' into 'master'
Let the CI runner know about builds that this build depends on

This allows us to implement artifacts passing: runner will download artifacts from all prior builds. It will happen automatically, and always, as long as artifacts are enabled.

## The changes:

This MR exposes list of prior builds in CI::API::Builds.

**The API response when asking for builds**

```json
{
  "id": 48584,
  "ref": "0.1.1",
  "tag": true,
  "sha": "d63117656af6ff57d99e50cc270f854691f335ad",
  "status": "success",
  "name": "pages",
  "token": "9dd60b4f1a439d1765357446c1084c",
  "stage": "test",
  "project_id": 479,
  "project_name": "test",
  "commands": "echo commands",
  "repo_url": "http://gitlab-ci-token:token@gitlab.example/group/test.git",
  "before_sha": "0000000000000000000000000000000000000000",
  "allow_git_fetch": false,
  "options": {
    "image": "docker:image",
    "artifacts": {
      "paths": [
        "public"
      ]
    },
    "cache": {
      "paths": [
        "vendor"
      ]
    }
  },
  "timeout": 3600,
  "variables": [
    {
      "key": "CI_BUILD_TAG",
      "value": "0.1.1",
      "public": true
    }
  ],
  "dependencies": {
    "builds": [
      {
        "id": 48584,
        "ref": "0.1.1",
        "tag": true,
        "sha": "d63117656af6ff57d99e50cc270f854691f335ad",
        "status": "success",
        "name": "build",
        "token": "9dd60b4f1a439d1765357446c1084c",
        "stage": "build",
        "project_id": 479,
        "project_name": "test",
        "artifacts_file": {
          "filename": "artifacts.zip",
          "size": 0
        }
      }
    ]
  }
}
```

## How it will work?

**Example:**
```
build:
  type: build
  script:
  - echo TEST > test_file
  artifacts:
    untracked: true

rspec:
  type: test
  script:
  - test-my-project

staging:
  type: deploy
  script:
  - scp test_file root@server.com:
```

**The flow:**
1. We run `build`. The `build` creates a file `test_file`. This file gets archived and send us build artifacts.
2. We run `rspec`. The `rspec` downloads build artifacts from `build`. Uses the `test_file`.
3. We run `staging`. The `staging` downloads build artifacts from `build` and `rspec`, but since the `rspec` doesn't have build artifacts we skip that build. Deploys the `test_file`.

This partially implements the https://gitlab.com/gitlab-org/gitlab-ce/issues/3423.

In the next release we will introduce option to configure what artifacts are received.

/cc @grzesiek @DouweM @sytse @rspeicher


See merge request !2437
2016-01-15 15:54:35 +00:00
Kamil Trzcinski 1ce766367e Change dependencies.builds to depends_on_builds 2016-01-15 15:35:33 +01:00
Tomasz Maczukin 768721dc58 Merge branch 'master' into ci/api-triggers
* master: (150 commits)
  fixes margin and padding
  Update mailroom/postfix documentation [ci skip]
  Fix css mess around git clone panel. Align it properly
  Fix missing padding for user/group pages
  Fix parse_gollum_tags matcher
  Update documentation on Banzai::Filter::GollumTagsFilter
  Add tests for the wiki pipeline
  Refactoring Banzai::Filter::GollumTagsFilter
  Make sure the .git is at the end on Gitlab::GithubImport::WikiFormatter
  Remove GollumTagsPipeline
  Refactoring Gitlab::GithubImport::Importer
  Remove unnecessary brackets on WIKI_SLUG_ID route constraints
  Move js function to removing accents to vendor/assets/javascripts
  Update CHANGELOG
  Use the WikiPipeline when rendering the wiki markdown content
  Add Banzai::Filter::GollumTagsFilter for parsing Gollum's tags in HTML
  Relax constraints for wiki slug
  Import GitHub wiki into GitLab
  Move Ci::Build#available_statuses to AVAILABLE_STATUSES constant in CommitStatus
  Revert changes to how the notes are paginated in the API
  ...

Conflicts:
	doc/api/README.md
	lib/api/entities.rb
2016-01-14 21:58:17 +01:00
Kamil Trzcinski ac652d82f1 Let the CI runner know about builds that this build depends on
This allows us to implement artifacts passing: runner will download artifacts from all prior builds
2016-01-14 19:45:55 +01:00
Douwe Maan ba42b03348 Merge branch 'ci/api-builds' into 'master'
Add builds API

References #4264

See merge request !2207
2016-01-14 16:29:15 +00:00
Dmitriy Zaporozhets 0eef82761f Merge branch 'add-pagination-headers-to-api' into 'master'
Add pagination headers to already paginated API resources

Following #3045, I've added pagination headers using already available Kaminari methods.

## Pagination headers in action

![Screen_Shot_2016-01-14_at_12.14.53](/uploads/88807f754dd9aafbb24da85f6bdf9521/Screen_Shot_2016-01-14_at_12.14.53.png)

---

I've also added a shared example to test paginated API resources.

A possible next step would be to paginate all the API resources that return an array.

See merge request !2426
2016-01-14 15:20:16 +00:00
Douwe Maan b6454fef56 Merge branch 'import-gh-wiki' into 'master'
Import GitHub wiki into GitLab

Closes #2834 

GitHub doesn’t apply any constraints to theirs wiki slug allowing chars like ,, :, *, etc, we need to remove our constraints or some wiki pages will not be available after they are imported.

Some wikis use the Gollum's tags to link its internal/external resources like: images, urls, wiki pages, etc. So, to avoid that wiki links/images displays completely broken after they were imported, we added a new `WikiPipeline`, that for now will parse only simple links, and image tags.

##### Before `WikiPipeline`:

![Screenshot_2016-01-11_20.14.48](/uploads/46fd5dbb5acfc70aa8ecca3050b675e4/Screenshot_2016-01-11_20.14.48.png)

##### After `WikiPipeline`:

![Screenshot_2016-01-11_20.15.56](/uploads/b1d94aa852f385f867a7868c358b4257/Screenshot_2016-01-11_20.15.56.png)

See merge request !2324
2016-01-14 15:08:42 +00:00
Douwe Maan b6ae2def2c Merge branch 'unsubscribe-from-thread-from-email-footer' into 'master'
Unsubscribe from thread through link in email footer

Closes #3437 

![Screenshot_from_2015-12-11_15-35-06](/uploads/cfb3d8737d4757f527995411f103d0ce/Screenshot_from_2015-12-11_15-35-06.png)

![Screenshot_from_2015-12-11_15-35-56](/uploads/9b7121be7ce4b05e5995ca6d38c5bea4/Screenshot_from_2015-12-11_15-35-56.png)

See merge request !2068
2016-01-14 14:38:55 +00:00
Douglas Barbosa Alexandre 2450916dc6 Fix parse_gollum_tags matcher 2016-01-14 12:09:31 -02:00
Douglas Barbosa Alexandre a8c836c371 Add tests for the wiki pipeline 2016-01-14 12:09:31 -02:00
Douglas Barbosa Alexandre 765a2c7327 Refactoring Banzai::Filter::GollumTagsFilter 2016-01-14 12:09:31 -02:00
Douglas Barbosa Alexandre 89e8b82b63 Make sure the .git is at the end on Gitlab::GithubImport::WikiFormatter 2016-01-14 12:09:31 -02:00
Douglas Barbosa Alexandre 4872b319c8 Use the WikiPipeline when rendering the wiki markdown content 2016-01-14 12:09:31 -02:00
Douglas Barbosa Alexandre a6a5990ee5 Add Banzai::Filter::GollumTagsFilter for parsing Gollum's tags in HTML 2016-01-14 12:09:31 -02:00
Douglas Barbosa Alexandre 78f5eb94fb Import GitHub wiki into GitLab 2016-01-14 12:09:31 -02:00
Rémy Coutable 1b08cd811a Merge branch 'master' of https://gitlab.com/gitlab-org/gitlab-ce into add-pagination-headers-to-api 2016-01-14 14:23:42 +01:00
Grzegorz Bizon f03da18e3a Merge branch 'ci/view-build-artifacts' into 'master'
Add browser for build artifacts

Discussion in #3426, closes #3426.

See merge request !2123
2016-01-14 12:57:33 +00:00
Tomasz Maczukin 405b82af23 Merge branch 'master' into ci/api-builds
* master: (51 commits)
  Fix version
  Fix specs and rubocop warnings
  Improve the consistency of commit titles, branch names, tag names, issue/MR titles, on their respective project pages
  fixed LDAP activation on login to use new ldap_blocked state
  Fix Admin/Users view to position buttons without spacing magic
  Update to Go 1.5.3
  Fix the undefinded variable error in Project's safe_import_url method
  Update CHANGELOG [ci skip]
  Add some cosmetic changes to variables API documentation [ci skip]
  Fix misaligned edit button in milestone collection partial
  Update button styles for Milestones#show
  Modify builds API documentation style [ci skip]
  Modify :ci_variable factory
  Ensure the API doesn't return notes that the current user shouldn't see
  Add 'Build' prefix to Variables entry name in API docs index
  Fix some typos
  Add spec for Note#cross_reference_not_visible_for?
  Remove (invalid) timestamp formatting
  Move `BroadcastMessage#status` to a helper since it's presentational
  Update CHANGELOG
  ...

Conflicts:
	doc/api/README.md
	lib/api/api.rb
	lib/api/entities.rb
2016-01-14 13:30:18 +01:00
Tomasz Maczukin af3b97cdde Merge branch 'master' into ci/api-triggers
* master:
  Fix version
  Improve the consistency of commit titles, branch names, tag names, issue/MR titles, on their respective project pages
  Update CHANGELOG [ci skip]
  Add some cosmetic changes to variables API documentation [ci skip]
  Modify builds API documentation style [ci skip]
  Modify :ci_variable factory
  Add 'Build' prefix to Variables entry name in API docs index
  Fix some typos
  Add some fixes after review
  Remove blank line
  Update ./doc/api
  Change :variable_id to :key as resource ID in API
  Fix a typo in method description
  Add create feature to variables API
  Add missing 'not_found' checks in variables API
  Add delete feature to variables API
  Add update feature for variables API
  Add features for list and show details of variables in API

Conflicts:
	doc/api/README.md
	lib/api/entities.rb
2016-01-14 13:09:54 +01:00
Grzegorz Bizon 0d6e7b9d3d Use Hash to store paths and entries metadata in artifacts browser 2016-01-14 12:48:17 +01:00
Grzegorz Bizon ad2b0358e0 Improve readability of artifacts Metadata related code 2016-01-14 12:48:17 +01:00
Grzegorz Bizon 6b0a43aff3 Improve readability of artifacts browser Entry related code 2016-01-14 12:48:17 +01:00
Grzegorz Bizon 154b8ceba4 Refactor build artifacts upload API endpoint 2016-01-14 12:48:17 +01:00
Grzegorz Bizon 0b946029a1 Update build artifacts API
We do not want to allow runners to upload a metadata file. This needs to
be generated by Workhorse only.
2016-01-14 12:48:16 +01:00
Grzegorz Bizon e8995f9fd5 Modify artifacts upload API endpoint, add artifacts metadata 2016-01-14 12:48:16 +01:00
Grzegorz Bizon cf00a808cc Fix specs for artifacts metadata after changing fixture content 2016-01-14 12:48:16 +01:00
Grzegorz Bizon 2be76355ca Support only valid UTF-8 paths in build artifacts browser 2016-01-14 12:48:16 +01:00
Grzegorz Bizon f80d7a868e Update build model specs 2016-01-14 12:48:16 +01:00
Grzegorz Bizon 09a4a5aff8 Render only valid paths in artifacts metadata
In this version we will support only relative paths in artifacts
metadata. Support for absolute paths will be introduced later.
2016-01-14 12:48:16 +01:00
Grzegorz Bizon 61fb47a432 Simplify implementation of build artifacts browser (refactoring) 2016-01-14 12:48:15 +01:00
Grzegorz Bizon 387b27813d Change format of artifacts metadata from text to binary 0.0.1
This changes the format of metadata to handle paths, that may contain
whitespace characters, new line characters and non-UTF-8 characters.

Now those paths along with metadata in JSON format are stored as
length-prefixed strings (uint32 prefix).

Metadata file has a custom format:

1.   First string field is metadata version field (string)
2.   Second string field is metadata errors field (JSON strong)
3.   All subsequent fields is pair of path (string) and path metadata
     in JSON format.

Path's metadata contains all fields that where possible to extract from
ZIP archive like date of modification, CRC, compressed size,
uncompressed size and comment.
2016-01-14 12:48:15 +01:00
Grzegorz Bizon cfffc9eff2 Update build specs for artifacts browser support 2016-01-14 12:48:15 +01:00
Grzegorz Bizon 1b1793c253 Show file size in artifacts browser using metadata 2016-01-14 12:48:15 +01:00
Grzegorz Bizon cd3b8bbd2f Add method that checks if path exists in StringPath 2016-01-14 12:48:15 +01:00
Grzegorz Bizon a5e1905d28 Render 404 when artifacts path is invalid 2016-01-14 12:48:15 +01:00
Grzegorz Bizon f948c00757 Do not depend on universe when checking parent in StringPath 2016-01-14 12:48:15 +01:00
Grzegorz Bizon a7f99b67a0 Extract artifacts metadata implementation to separate class 2016-01-14 12:48:15 +01:00
Grzegorz Bizon df41148662 Improve path sanitization in StringPath 2016-01-14 12:48:15 +01:00
Grzegorz Bizon 3de8a4620a Parse artifacts metadata stored in JSON format 2016-01-14 12:48:15 +01:00
Grzegorz Bizon e3ef0ac8f4 Update artifacts metadata fixture 2016-01-14 12:48:14 +01:00
Grzegorz Bizon 662f4b9e1d Add artifacts metadata uploader filed
Artifacts metadata field will be used to store a filename of gzipped
file containing metadata definition for given artifacts archive.
2016-01-14 12:48:14 +01:00
Grzegorz Bizon ece114e630 Update artifacts download specs 2016-01-14 12:48:14 +01:00
Grzegorz Bizon 5ff7ec42dc Add method that checks if artifacts browser is supported
This is needed because of backward compatibility. Previously artifacts
archive had `.tar.gz` format, but artifacts browser requires ZIP format
now.
2016-01-14 12:48:14 +01:00
Grzegorz Bizon 8eeed761a9 Update specs for CI Build, add artifacts? method
`artifacts?` method checks if artifacts archive is available.
2016-01-14 12:48:14 +01:00
Grzegorz Bizon 9e0e9342a4 Rename method that returns url to CI build artifacts download 2016-01-14 12:48:14 +01:00
Grzegorz Bizon 87df1df3cd Seed db with CI build artifacts using a zip archive 2016-01-14 12:48:13 +01:00
Grzegorz Bizon 304c39b6dc Fix rubocop offenses in StringPath specs 2016-01-14 12:48:13 +01:00
Grzegorz Bizon b19e958d86 Add support for parent directories in StringPath
This support is not completed though, as parent directory that is first
in collection returned by `directories!` is not iterable yet.
2016-01-14 12:48:13 +01:00
Grzegorz Bizon 37b2c5dd55 Add support for root path for StringPath 2016-01-14 12:48:13 +01:00
Grzegorz Bizon d382335dcd Add implementation of remaining methods in StringPath 2016-01-14 12:48:13 +01:00