home/gitlabhq
2
0
Fork 0
mirror of https://github.com/wahyd4/gitlabhq.git synced 2026-08-09 04:36:11 +10:00
Code Issues Packages Projects Releases Wiki Activity

Introduce CI documentation for services and languages

Browse Source
This commit is contained in:
Kamil Trzcinski
2015-12-08 18:25:24 +02:00
committed by Achilleas Pipinellis
parent 033947de90
commit 4f074aaa14
11 changed files with 559 additions and 68 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/README.md
+12
View File
@@ -24,9 +24,21 @@
- [Using Docker Images](ci/docker/using_docker_images.md)
- [Using Docker Build](ci/docker/using_docker_build.md)
- [Using Variables](ci/variables/README.md)
- [Using SSH keys](ci/ssh_keys/README.md)
- [User permissions](ci/permissions/README.md)
- [API](ci/api/README.md)
### CI Languages
+ [Testing PHP](ci/languages/php.md)
## CI Services
+ [Using MySQL](ci/services/mysql.md)
+ [Using PostgreSQL](ci/services/postgres.md)
+ [Using Redis](ci/services/redis.md)
+ [Using Other Services](ci/docker/using_docker_images.html#how-to-use-other-images-as-services)
### CI Examples
- [Test and deploy Ruby applications to Heroku](ci/examples/test-and-deploy-ruby-application-to-heroku.md)
doc/ci/README.md
+12
View File
@@ -9,6 +9,18 @@
+ [Using Docker Images](docker/using_docker_images.md)
+ [Using Docker Build](docker/using_docker_build.md)
+ [Using Variables](variables/README.md)
+ [Using SSH keys](ssh_keys/README.md)
### Languages
+ [Testing PHP](languages/php.md)
### Services
+ [Using MySQL](services/mysql.md)
+ [Using PostgreSQL](services/postgres.md)
+ [Using Redis](services/redis.md)
+ [Using Other Services](docker/using_docker_images.html#how-to-use-other-images-as-services)
### Examples
doc/ci/docker/using_docker_images.md
+47 -68
View File
@@ -42,7 +42,44 @@ So, **to access your database service you have to connect to host: `mysql` inste
### How to use other images as services?
You are not limited to have only database services.
You can hand modify `config.toml` to add any image as service found at [Docker Hub](https://registry.hub.docker.com/).
You can add the services to `.gitlab-ci.yml` or hand modify the `config.toml`.
You can use any image as service found at [Docker Hub](https://registry.hub.docker.com/).
### Define image and services from `.gitlab-ci.yml`
You can simply define image or list services that you want to use for the build time.
```
image: ruby:2.2
services:
- postgres:9.3
before_install:
- bundle install
test:
script:
- bundle exec rake spec
```
It's possible to define image and service per-job:
```
before_install:
- bundle install
test:2.1:
image: ruby:2.1
services:
- postgres:9.3
script:
- bundle exec rake spec
test:2.2:
image: ruby:2.2
services:
- postgres:9.4
script:
- bundle exec rake spec
```
### Define image and services in `config.toml`
Look for `[runners.docker]` section:
```
[runners.docker]
@@ -50,13 +87,16 @@ Look for `[runners.docker]` section:
services = ["mysql:latest", "postgres:latest"]
```
The image and services defined these way will be added to all builds run by that runner.
### Accessing the services
For example you need `wordpress` instance to test some API integration with `Wordpress`.
You can for example use this image: [tutum/wordpress](https://registry.hub.docker.com/u/tutum/wordpress/).
This is image that have fully preconfigured `wordpress` and have `MySQL` server built-in:
You can for example use this image: [tutum/wordpress](https://registry.hub.docker.com/u/tutum/wordpress/).
```
[runners.docker]
image = "ruby:2.1"
services = ["mysql:latest", "postgres:latest", "tutum/wordpress:latest"]
# .gitlab-ci.yml
services:
- tutum/wordpress:latest
```
Next time when you run your application the `tutum/wordpress` will be started
@@ -64,7 +104,7 @@ and you will have access to it from your build container under hostname: `tutum_
Alias hostname for the service is made from the image name:
1. Everything after `:` is stripped,
2. '/' is replaced with `__`.
2. '/' is replaced to `__`.
### Configuring services
Many services accept environment variables, which allow you to easily change database names or set account names depending on the environment.
@@ -99,67 +139,6 @@ or README page for any other Docker image.
**Note: All variables will passed to all service containers. It's not designed to distinguish which variable should go where.**
### Overwrite image and services
It's possible to overwrite `docker-image` and specify services from `.gitlab-ci.yml`.
If you add to your YAML the `image` and the `services` these parameters
be used instead of the ones that were specified during runner's registration.
```
image: ruby:2.2
services:
- postgres:9.3
before_install:
- bundle install
test:
script:
- bundle exec rake spec
```
It's possible to define image and service per-job:
```
before_install:
- bundle install
test:2.1:
image: ruby:2.1
services:
- postgres:9.3
script:
- bundle exec rake spec
test:2.2:
image: ruby:2.2
services:
- postgres:9.4
script:
- bundle exec rake spec
```
#### How to enable overwriting?
To enable overwriting you have to **enable it first** (it's disabled by default for security reasons).
You can do that by hand modifying runner configuration: `config.toml`.
Please go to section where is `[runners.docker]` definition for your runner.
Add `allowed_images` and `allowed_services` to specify what images are allowed to be picked from `.gitlab-ci.yml`:
```
[runners.docker]
image = "ruby:2.1"
allowed_images = ["ruby:*", "python:*"]
allowed_services = ["mysql:*", "redis:*"]
```
This enables you to use in your `.gitlab-ci.yml` any image that matches above wildcards.
You will be able to pick only `ruby` and `python` images.
The same rule can be applied to limit services.
If you are courageous enough, you can make it fully open and accept everything:
```
[runners.docker]
image = "ruby:2.1"
allowed_images = ["*", "*/*"]
allowed_services = ["*", "*/*"]
```
**It the feature is not enabled, or image isn't allowed the error message will be put into the build log.**
### How Docker integration works
1. Create any service container: `mysql`, `postgresql`, `mongodb`, `redis`.
1. Create cache container to store all volumes as defined in `config.toml` and `Dockerfile` of build image (`ruby:2.1` as in above example).
doc/ci/languages/README.md
+3
View File
@@ -0,0 +1,3 @@
### Languages
+ [Testing PHP](php.md)
doc/ci/languages/php.md
+178
View File
@@ -0,0 +1,178 @@
## Testing PHP projects
This guide covers basic of building PHP projects.
Is it possible to test PHP apps on any system.
However, it will require manual configuration.
The simplest is to use Docker executor as described below.
### PHP projects on Docker executor
It's possible to official [PHP](https://hub.docker.com/_/php/) repositories on Docker Hub.
They allow to test PHP projects against different versions of the runtime.
However, they require additional configuration.
To build PHP project you need to create valid `.gitlab-ci.yml` describing the build environment:
1. First you need to specify PHP image as described here: http://doc.gitlab.com/ce/ci/docker/using_docker_images.html#what-is-image. To your `.gitlab-ci.yml` add:
image: php:5.6
2. The official images are great, but they are lacking a few useful tools for testing. We need to install them first in build environment. Create `ci/docker_install.sh` file with following content:
#!/bin/bash
# We need to install dependencies only for Docker
[[ ! -e /.dockerinit ]] && exit 0
set -xe
# Install git (the php image doesn't have it) which is required by composer
apt-get update -yqq
apt-get install git -yqq
# Install phpunit, the tool that we will use for testing
curl -o /usr/local/bin/phpunit https://phar.phpunit.de/phpunit.phar
chmod +x /usr/local/bin/phpunit
# Install mysql driver
# Here you can install any other extension that you need
docker-php-ext-install pdo_mysql
3. From your `.gitlab-ci.yml` run the created script:
before_script:
- bash ci/docker_install.sh > /dev/null
4. Now you can run your tests. Usually it will be `phpunit` with arguments:
test:app:
script:
- phpunit --configuration phpunit_myapp.xml --coverage-text
5. Commit your files, and push them to GitLab to see if it works. With GitLab Runner 1.0 you can also test the changes locally. From your terminal execute:
# Check using docker executor
gitlab-runner exec docker test:app
# Check using shell executor
gitlab-runner exec shell test:app
The final `.gitlab-ci.yml` should look similar to this:
# Select image from https://hub.docker.com/_/php/
image: php:5.6
before_script:
# Install dependencies
- ci/docker_install.sh > /dev/null
test:app:
script:
- phpunit --configuration phpunit_myapp.xml --coverage-text
#### Test against different PHP versions in Docker builds
You can also test against multiple version of PHP runtime:
before_script:
# Install dependencies
- ci/docker_install.sh > /dev/null
# We test PHP5.6
test:5.6:
image: php:5.6
script:
- phpunit --configuration phpunit_myapp.xml --coverage-text
# We test PHP7.0
test:7.0:
image: php:7.0
script:
- phpunit --configuration phpunit_myapp.xml --coverage-text
#### Custom PHP configuration in Docker builds
You can customise your PHP environment by putting your .ini file into `/usr/local/etc/php/conf.d/`:
before_script:
- cp my_php.ini /usr/local/etc/php/conf.d/test.ini
### Test PHP projects using Shell
Shell executor runs your builds in terminal session of your server. Thus in order to test your projects you need to have all dependencies installed as root.
1. Install PHP dependencies:
sudo apt-get update -qy
sudo apt-get install phpunit php5-mysql -y
This will install the PHP version available for your distribution.
2. Now you can run your tests. Usually it will be `phpunit` with arguments:
test:app:
script:
- phpunit --configuration phpunit_myapp.xml --coverage-text
#### Test against different PHP versions in Shell builds
The [phpenv](https://github.com/phpenv/phpenv) allows you to easily manage different PHP with they own configs.
This is specially usefull when testing PHP project with Shell executor.
Login as `gitlab-runner` user and follow [the installation guide](https://github.com/phpenv/phpenv#installation).
Using phpenv also allows to easily configure PHP environment with: `phpenv config-add my_config.ini`.
#### Install custom extensions
Since we have pretty bare installation of our PHP environment you may need some extensions that are not present on your installation.
To install additional extensions simply execute.:
pecl install <extension>
It's not advised to add this to the `.gitlab-ci.yml`.
You should execute this command once, only to setup the build environment.
### Extend your tests
#### Using atoum
Instead of PHPUnit, you can use any other tool to run unit tests. For example [atoum](https://github.com/atoum/atoum):
before_script:
- wget http://downloads.atoum.org/nightly/mageekguy.atoum.phar
test:atoum:
script:
- php mageekguy.atoum.phar
#### Using Composer
Majority of the PHP projects use Composer for managing the packages.
It's very simple to execute the Composer before running your tests.
To your `.gitlab-ci.yml` add:
# The composer stores all downloaded packages in vendor/
# Remove it if you committed the vendor/ directory
cache:
paths:
- vendor/
before_script:
# Install composer dependencies
- curl -sS https://getcomposer.org/installer | php
- php composer.phar install
### Access private packages / dependencies
You need to configure [the SSH keys](../ssh_keys/README.md) in order to checkout the repositories.
### Use databases or other services
Please checkout the docs about configuring [the CI services](../services/README.md).
### Example project
You maybe interested in our [Example Project](https://gitlab.com/gitlab-examples/php) that runs on [GitLab.com](https://gitlab.com) using our publically available shared runners.
Want to hack it? Simply fork it, commit and push changes. Within a few moments the changes will be picked and rebuilt by public runners.
doc/ci/services/README.md
+6
View File
@@ -0,0 +1,6 @@
## GitLab CI Services
+ [Using MySQL](mysql.md)
+ [Using PostgreSQL](postgres.md)
+ [Using Redis](redis.md)
+ [Using Other Services](../docker/using_docker_images.html#how-to-use-other-images-as-services)
doc/ci/services/docker-services.md
+5
View File
@@ -0,0 +1,5 @@
## GitLab CI Services
+ [Using MySQL](mysql.md)
+ [Using PostgreSQL](postgres.md)
+ [Using Redis](redis.md)
doc/ci/services/mysql.md
+72
View File
@@ -0,0 +1,72 @@
## Using MySQL
It's possible to use MySQL database test your apps during builds.
### Use MySQL with Docker executor
If you are using our Docker integration you basically have everything already.
1. Add this to your `.gitlab-ci.yml`:
services:
- mysql
variables:
# Configure mysql service (https://hub.docker.com/_/mysql/)
MYSQL_DATABASE: hello_world_test
MYSQL_ROOT_PASSWORD: mysql
2. Configure your application to use the database:
Host: mysql
User: root
Password: mysql
Database: hello_world_test
3. You can also use any other available on [DockerHub](https://hub.docker.com/_/mysql/). For example: `mysql:5.5`.
Example: https://gitlab.com/gitlab-examples/mysql/blob/master/.gitlab-ci.yml
### Use MySQL with Shell executor
It's possible to use MySQL on manually configured servers that are using GitLab Runner with Shell executor.
1. First install the MySQL server:
sudo apt-get install -y mysql-server mysql-client libmysqlclient-dev
# Pick a MySQL root password (can be anything), type it and press enter
# Retype the MySQL root password and press enter
2. Create an user:
mysql -u root -p
# Create a user which will be used by your apps
# do not type the 'mysql>', this is part of the prompt
# change $password in the command below to a real password you pick
mysql> CREATE USER 'runner'@'localhost' IDENTIFIED BY '$password';
# Ensure you can use the InnoDB engine which is necessary to support long indexes
# If this fails, check your MySQL config files (e.g. `/etc/mysql/*.cnf`, `/etc/mysql/conf.d/*`) for the setting "innodb = off"
mysql> SET storage_engine=INNODB;
# Create the database
mysql> CREATE DATABASE IF NOT EXISTS `hello_world_test` DEFAULT CHARACTER SET `utf8` COLLATE `utf8_unicode_ci`;
# Grant necessary permissions on the database
mysql> GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, CREATE TEMPORARY TABLES, DROP, INDEX, ALTER, LOCK TABLES ON `hello_world_test`.* TO 'runner'@'localhost';
# Quit the database session
mysql> \q
3. Try to connect to database:
sudo -u gitlab-runner -H mysql -u runner -p -D hello_world_test
4. Configure your application to use the database:
Host: localhost
User: runner
Password: $password
Database: hello_world_test
doc/ci/services/postgres.md
+70
View File
@@ -0,0 +1,70 @@
## Using PostgreSQL
It's possible to use PostgreSQL database test your apps during builds.
### Use PostgreSQL with Docker executor
If you are using our Docker integration you basically have everything already.
1. Add this to your `.gitlab-ci.yml`:
services:
- postgres
variables:
# Configure postgres service (https://hub.docker.com/_/postgres/)
POSTGRES_DB: hello_world_test
POSTGRES_USER: postgres
POSTGRES_PASSWORD: ""
2. Configure your application to use the database:
Host: postgres
User: postgres
Password: postgres
Database: hello_world_test
3. You can also use any other available on [DockerHub](https://hub.docker.com/_/postgres/). For example: `postgres:9.3`.
Example: https://gitlab.com/gitlab-examples/postgres/blob/master/.gitlab-ci.yml
### Use PostgreSQL with Shell executor
It's possible to use PostgreSQL on manually configured servers that are using GitLab Runner with Shell executor.
1. First install the PostgreSQL server:
sudo apt-get install -y postgresql postgresql-client libpq-dev
2. Create an user:
# Install the database packages
sudo apt-get install -y postgresql postgresql-client libpq-dev
# Login to PostgreSQL
sudo -u postgres psql -d template1
# Create a user for runner
# Do not type the 'template1=#', this is part of the prompt
template1=# CREATE USER runner CREATEDB;
# Create the database & grant all privileges on database
template1=# CREATE DATABASE hello_world_test OWNER runner;
# Quit the database session
template1=# \q
3. Try to connect to database:
# Try connecting to the new database with the new user
sudo -u gitlab-runner -H psql -d hello_world_test
# Quit the database session
hello_world_test> \q
4. Configure your application to use the database:
Host: localhost
User: runner
Password:
Database: hello_world_test
doc/ci/services/redis.md
+40
View File
@@ -0,0 +1,40 @@
## Using Redis
It's possible to use Redis database test your apps during builds.
### Use Redis with Docker executor
If you are using our Docker integration you basically have everything already.
1. Add this to your `.gitlab-ci.yml`:
services:
- redis
2. Configure your application to use the database:
Host: redis
3. You can also use any other available on [DockerHub](https://hub.docker.com/_/redis/). For example: `redis:2.6`.
Example: https://gitlab.com/gitlab-examples/redis/blob/master/.gitlab-ci.yml
### Use Redis with Shell executor
It's possible to use Redis on manually configured servers that are using GitLab Runner with Shell executor.
1. First install the Redis server:
sudo apt-get install redis-server
2. Try to connect to the server:
# Try connecting the the Redis server
sudo -u gitlab-runner -H redis-cli
# Quit the session
127.0.0.1:6379> quit
4. Configure your application to use the database:
Host: localhost
doc/ci/ssh_keys/README.md
+114
View File
@@ -0,0 +1,114 @@
# Using SSH keys
GitLab currently doesn't have built-in support for SSH keys in build environment.
The SSH keys can be useful when:
1. You want to checkout internal submodules,
2. You want to download private packages using your package manager (ie. bundler),
3. You want to deploy your app (ex. to Heroku or own server),
4. You want to execute ssh commands from build environment on remote server,
5. You want to rsync files from your build to remote server.
If anyone of the above holds true, then you most likely need SSH key.
There are two possibilities to add SSH keys to build environment.
## Inject keys in your build environment
The most widely supported is to inject SSH key into your build environment by extending your .gitlab-ci.yml.
This is the universal solution which works with any type of executor (docker, shell, etc.).
### How it works?
1. We create a new SSH private key with [ssh-keygen](http://linux.die.net/man/1/ssh-keygen).
2. We add the private key as the Secure Variable to project.
3. We run the [ssh-agent](http://linux.die.net/man/1/ssh-agent) during build to load the private key.
The example [.gitlab-ci.yml](https://gitlab.com/gitlab-examples/ssh-private-key/blob/master/.gitlab-ci.yml) looks like this.
### Make it work?
1. First, go to terminal and generate a new SSH key:
```bash
$ ssh-keygen -t rsa -f my_key
Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in my_key.
Your public key has been saved in my_key.pub.
The key fingerprint is:
SHA256:tBJEfyJUGTMNmPCiPg4UHywHs67MxlM2iEBAlI/W+TY fingeprint
The key's randomart image is:
+---[RSA 2048]----+
|=*. .o++*= |
|..= +o..o. |
|.+++o + + . |
|+o*=.. + + |
|o+.=. . S |
|*.o .E . |
|o*o . . |
|.o.. |
| . |
+----[SHA256]-----+
```
2. Create a new **Secure Variable** in your project settings on GitLab and name it: `SSH_PRIVATE_KEY`.
3. Copy the content of `my_key` and paste it as a **Value** of **SSH_PRIVATE_KEY**.
4. Next you need to modify your `.gitlab-ci.yml` and at the top of the file add:
```
before_script:
# install ssh-agent (it is required for Docker, change apt-get to yum if you use CentOS-based image)
- 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
# run ssh-agent (in build environment)
- eval $(ssh-agent -s)
# add ssh key stored in SSH_PRIVATE_KEY variable to the agent store
- ssh-add <(echo "$SSH_PRIVATE_KEY")
# for Docker builds disable host key checking, by adding that you are suspectible to man-in-the-middle attack
- mkdir -p ~/.ssh
- '[[ -f /.dockerinit ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config`
```
5. Add the public key from `my_key.pub` to services that you want to have an access from build.
6. If your builds are run using `shell` executor, you may need to login to server and execute the `ssh <address-of-my-server>` to store the fingerprint of remote server.
## SSH keys when using Shell executor
If use `shell`, not `docker` it can be easier to have the SSH key.
We can generate the SSH key for the machine that holds `gitlab-runner` and use that key for all projects that are run on this machine.
1. First, login to server that runs your builds.
2. From terminal login as `gitlab-runner` user and generate the SSH private key:
```bash
$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in ~/.ssh/id_rsa.
Your public key has been saved in ~/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:tBJEfyJUGTMNmPCiPg4UHywHs67MxlM2iEBAlI/W+TY fingeprint
The key's randomart image is:
+---[RSA 2048]----+
|=*. .o++*= |
|..= +o..o. |
|.+++o + + . |
|+o*=.. + + |
|o+.=. . S |
|*.o .E . |
|o*o . . |
|.o.. |
| . |
+----[SHA256]-----+
```
3. Add the public key from `~/.ssh/id_rsa.pub` to services that you want to have an access from build.
4. Try to login for the first time and accept fingerprint:
```bash
ssh <address-of-my-server
```