With Visual Builder Studio you can easily create your build pipelines using the GUI configurable tools available in the "Builds" section of Visual Builder Studio.

On the other hand, if your team is practicing Infrastructure as Code, we still have you covered.

In VBS, you can define your build jobs and pipelines using YAML files that can be checked into your git repository with all of the same features as the GUI tools.  This will allow your team to develop your CICD pipeline following the same development processes you use for the rest of your application code.  You can track changes with issue tracking, develop your pipeline in separate branches, run it through proper code reviews and use merge requests to deploy the changes.

Let's take a look at some examples and build a CICD Pipeline using YAML.  We'll keep the examples as simple as possible to make it easy to get started, but you can make your actual pipelines and jobs as advanced as you need.

Quick Links

Create a Git Repository

The YAML files will need to be in a Git repository under a directory named ".ci-build".  They can be in the same Git repository as your application code but you could also have a separate Git repository just for your YAML pipelines.

  1. Log into Visual Builder Studio
  2. Create/Open your project
  3. On the right side of the Home page for your project, select Repositories and press the "+ Create Repository" button to create a Git repo
    Create Git Repo
  4. In the Details window, enter a name for your new Repo
    app-for-yml-build
  5. Select the option to Initialize repository with README file
  6. Click the "Create" button
    Git Repo Details
  7. The repository will be created and opened.

Create a Build Job

  1. Click "+ File"
    New File
  2. Name the file
    /.ci-build/first-build-job.yml
  3. Add the following YAML 
    job: 
  name: first-build-job-yaml
  description: YAML Job example 1
  executor-template: System Default OL7 for Visual Builder
  steps: 
  - shell: 
      script: |
        echo "My first YAML build job"
  4. Commit the file
    Edit Build Job

Test the Build Job

You can manually run your new build job the same way you would run a GUI based job.

  1. Go the the Builds Section
  2. Click on YAML Jobs
    Show YAML Jobs
  3. Open your job
    It may take a few seconds for the system to sync the .yml file into the Builds Section.  If doesn't show up, go back and make sure your YAML is correct and verify the directory name.
    Open Job 1
  4. Click Build Now
    Start Build Job 1
  5. Wait for the job to complete
  6. Click the Build Log button
    Open Build Job 1 Log 1
  7. In the log you should see entries similar to this 
    [2024-07-01 17:06:07] BEGIN shell script execution with /bin/sh -ex
[2024-07-01 17:06:07] + echo 'My first YAML build job'
[2024-07-01 17:06:07] My first YAML build job
[2024-07-01 17:06:07] END shell script execution

If the build fails, go back and make sure your YAML is correct.

Build on Git Commit

The goal is to create a pipeline that will automatically run anytime there is a commit to the main branch of the repository.  Next you'll connect the build job to the Git repo. 

  1. Go back to the Git repository
  2. Open the file /.ci-build/first-build-job.yml
  3. Click the Edit File Pencil icon
    Edit Git File
  4. Add a git section before the steps section
    *Notice the exceptions entry.  This is not a required entry, but it will prevent changes to your .yml files from executing the build.  I often make frequent commits and I prefer to run my builds manually after I make all of my edits to the build jobs. 
    	git:
	  - url: app-for-yml-build.git
	    branch: "main"
	    exceptions: .ci-build/*
	    build-on-commit: true
  5. To verify that the Git repository was cloned into your workspace you'll display the contents of the README.md file by adding the following below the echo statement
    cat README.md
  6. Commit the file

Your file should now look like this

job:
  name: first-build-job-yaml
  description: YAML Job example 1
  executor-template: System Default OL7 for Visual Builder
  git: 
  - url: app-for-yml-build.git
    branch: "main"
    exceptions: .ci-build/*
    build-on-commit: true
  steps: 
  - shell: 
      script: |
        echo "My first YAML build job"
        cat README.md

Auto Build After a Commit

Modify the README file to trigger the build job.

  1. Go to the root of the Git repository
  2. Open the file /README.md
    Open README
  3. Click the Edit File Pencil icon
  4. Make a change to the file
  5. Commit the file
  6. Go the the Builds Section
  7. The build job should run automatically (It may take a few seconds to start)
    The double arrow icon indicates that this job was started by a change to the Git repo.
    Build Started By SCM
  8. Once the job finishes open the build log

Your Git repository main branch is checked out into your workspace.

[2024-07-01 17:14:46] Git: Checkout directory is the workspace root.
[2024-07-01 17:14:46] Git: git version 2.31.1
[2024-07-01 17:14:46] Git: Fetching from remote repository app-for-yml-build.git
[2024-07-01 17:14:48] Git: Checking out branch main
[2024-07-01 17:14:49] Git: Done

The Shell script is executed with the echo statement and the README.md file contents are displayed in the log.

[2024-07-01 17:14:49] BEGIN shell script execution with /bin/sh -ex
[2024-07-01 17:14:49] + echo 'My first YAML build job'
[2024-07-01 17:14:49] My first YAML build job
[2024-07-01 17:14:49] + cat README.md
[2024-07-01 17:14:49] app-for-yml-build.git
[2024-07-01 17:14:49] Change 1
[2024-07-01 17:14:49] END shell script execution

Securing Keys and Passwords

When you're using the GUI interface for your build jobs, you can populate the Key and Password fields directly and VBS will store them securely without exposing them through the build log.

With a .yml file the definition of the build job is stored in your Git repository and you SHOULD NOT include Keys and Passwords in your Git repository.

Visual Builder Studio has a solution.  You can store your credential at the project level and then reference them as needed in your build jobs.

Create Named Credentials

  1. Open Project Administration
  2. Select "Builds"
    Open Admin Builds
  3. Open the "Named Credentials" tab
  4. Click on "+ Create Named Credentials"
    Create Credential
  5. Enter a Name
    MyNamedCredential
  6. Enter the Password value or Check the Private Key checkbox and paste the Private Key
  7. Click Create
    Credential Details
  8. Click on your new Named Credential to see an example of how to reference it
    Reference Example

Use Named Credentials in a Build Job

You are able to use a direct reference to a Named Credential with the following format #{CredentialName} as seen in the example above.

A direct reference can only be used in secure fields such as a Password field or a Private Key field.

- password:
    name: myPassword
    password: "#{MyNamedCredential}"
    required: true

- ssh: 
     config: 
       private-key: "#{MyNamedCredential}"

If you want to use the Named Credential elsewhere, you'll need to create a Password Parameter in your Build job and use the Named Credential as its value.

Add a Password Parameter

  1. Go back to the Git repository
  2. Open the file /.ci-build/first-build-job.yml
  3. Click the Edit File Pencil icon
  4. Add a params section before the steps section 
    	  params:
	  - password:
	      name: myPassword
	      password: "#{MyNamedCredential}"
	      required: true
  5. Use the parameter in the shell step
    You could use this in something like a curl command or an SCP transfer, but to keep the example simple, you will use it in an echo statement.
    Edit the echo statement
    echo "My first YAML build job parameter is ${myPassword}"
  6. Commit the file

Your file should now look like this

job:  
  name: first-build-job-yaml
  description: YAML Job example 1
  executor-template: System Default OL7 for Visual Builder
  git:
  - url: app-for-yml-build.git
    branch: "main"
    exceptions: .ci-build/*
    build-on-commit: true
  params:
  - password:
      name: myPassword
      password: "#{MyNamedCredential}"
      required: true
  steps: 
  - shell: 
      script: |
        echo "My first YAML build job parameter is ${myPassword}"
        cat README.md

Manually Run the Job

  1. Go the the Builds Section
  2. Click on YAML Jobs
  3. Click on your Build Job
  4. Click the "Build Now" button
  5. A window will pop up where you can enter a different value for your parameter
    Click "Build Now" to accept the default parameter value
    Accept Default Value
  6. Wait for the job to complete
  7. Click the Build Log button

You can see that the parameter is defined at the top of the build log but it's value is obfuscated.

[2024-07-01 17:18:00] Parameters in this job:
[2024-07-01 17:18:00] ------------------------------------------------------------------
[2024-07-01 17:18:00] myPassword = ********

Notice that the value for the parameter is also obfuscated in the shell step.

[2024-07-01 17:18:05] BEGIN shell script execution with /bin/sh -ex
[2024-07-01 17:18:05] + echo 'My first YAML build job parameter is ********'
[2024-07-01 17:18:05] My first YAML build job parameter is ********
[2024-07-01 17:18:05] + cat README.md
[2024-07-01 17:18:05] app-for-yml-build.git
[2024-07-01 17:18:05] Change 1
[2024-07-01 17:18:05] END shell script execution

Export Artifacts to be used in other Build Jobs

It's a good idea to work with the same artifacts throughout the build pipeline, rather than re-pulling them from repositories at each step.  This will help ensure that every step is consistent and you wont run the risk of pulling a file that may have been changed externally while your pipeline was running.  To help facilitate this you can archive artifacts in a build job, making them available to other build jobs.

  1. Go back to the Git repository
  2. Open the file /.ci-build/first-build-job.yml
  3. Click the Edit File Pencil icon
  4. Add an after section after the steps section
  5. Add an artifacts list and include the README.md file
  after: 
  - artifacts: 
      include: "README.md"

Your file should now look like this.

job:  
  name: first-build-job-yaml
  description: YAML Job example 1
  executor-template: System Default OL7 for Visual Builder
  git:
  - url: app-for-yml-build.git
    branch: "main"
    exceptions: .ci-build/*
    build-on-commit: true
  params:
  - password:
      name: myPassword
      password: "#{MyNamedCredential}"
      required: true
  steps: 
  - shell: 
      script: |
        echo "My first YAML build job parameter is ${myPassword}"
        cat README.md
  after: 
  - artifacts: 
      include: "README.md"

Second Build Job

You are able to create a single giant build job with many steps, but that's not usually a great idea.  It's a better idea to break your build procedures into smaller more manageable pieces.

For simplicity this job will be very similar to the previous job, but it will not pull from the Git repository.  Instead, it will copy the artifact from the first build job.

This copy artifacts process will happen before the build job is started and the artifacts will be available in the working directory.

The YAML look like this.

  before:
  - copy-artifacts:
      from-job: first-build-job-yaml
  1. In your Git repository, open the .ci-build folder
    Open cibuild Folder
  2. Click "+ File
  3. Name the file
    /.ci-build/second-build-job.yml
  4. Add the following YAML
job:  
  name: second-build-job-yaml
  description: YAML Job example 2
  executor-template: System Default OL7 for Visual Builder
  before:
    - copy-artifacts:
        from-job: first-build-job-yaml
  steps: 
  - shell: 
      script: |
        echo "My second YAML build job"
        cat README.md

You're welcome to manually run the build job the same way you manually ran the first build job.  Be aware, you were not asked to test the previous job after adding the artifact section.  The new job may fail since there won't be an artifact to copy.

Rather than manually running both of these jobs, let's connect the two jobs with a Build Pipeline.

Create a Pipeline

This is a relatively simple pipeline.  In the start section, list the build jobs you want to run in to order you want to run them.  More complex options can be found here.

  1. In your Git repository, open the .ci-build folder
  2. Click "+ File"
  3. Name the file
    /.ci-build/yaml-pipeline.yml
  4. Add the following YAML 
    pipeline:
  name: yaml-pipeline
  description: Pipeline to Package and Deploy a Visual Application
  auto-start: true
  allow-external-builds: false
  start:
  - first-build-job-yaml
  - second-build-job-yaml
  5. Commit the file

Run the pipeline manually

  1. Go the the Builds Section
  2. Open the Pipelines tab
  3. Click on YAML Pipelines
  4. Select your new pipeline
  5. Click the Run button
  6. Click "Build Now" to accept the default parameter value
  7. Once the pipeline has started running, click on the "Run" number under "Pipeline Run History"
    Click Pipeline Run Number
  8. You'll notice there is a spinning circle in the bubble of whichever build jobs are currently running.  Once the first job is complete it will turn green and the second job will start running.  Right click on the second build job and select "Open Job second-build-job-yaml in a new tab
    Open Second Pipeline Job
  9. Wait for the job to complete
  10. Click the Build Log button

Notice in the build log that the "myPassword" parameter is included in this build job.  When you use build job parameters in a pipeline they will automatically be made available to other jobs in the pipeline.  If you were to run second-build-job-yaml manually, this parameter would not be available because it was not defined it in this job.

[2024-07-01 17:25:56] Parameters in this job:
[2024-07-01 17:25:56] ------------------------------------------------------------------
[2024-07-01 17:25:56] myPassword = ********
[2024-07-01 17:25:56] EXECUTOR_TYPE = DOCKER
[2024-07-01 17:25:56] DEPLOYMENT_VM = 150.136.78.44
[2024-07-01 17:25:56] TASKID = ade6911c-9028-4f9c-a5dc-bc11934547c7
[2024-07-01 17:25:56] ------------------------------------------------------------------

Next, you can see that the README.md artifact was Successfully downloaded from the previous build job and it is now available in the workspace.

[2024-07-01 17:25:58] Copy Artifacts: Job first-build-job-yaml build #5 selected
[2024-07-01 17:26:01] Copy Artifacts: Successfully downloaded artifact README.md -> workspace/README.md

When you cat the README.md file the contents are displayed in the build log.

[2024-07-01 17:26:01] BEGIN shell script execution with /bin/sh -ex
[2024-07-01 17:26:01] + echo 'My second YAML build job'
[2024-07-01 17:26:01] My second YAML build job
[2024-07-01 17:26:01] + cat README.md
[2024-07-01 17:26:01] app-for-yml-build.git
[2024-07-01 17:26:01] Change 1
[2024-07-01 17:26:01] END shell script execution

Trigger the pipeline with a Git commit

Now, make a change to the README.md file and trigger the full pipeline.

  1. Go to the root of the Git repository
  2. Open the file /README.md
  3. Click the Edit File Pencil icon
  4. Make a change to the file
  5. Commit the file
  6. Go the the Builds Section
  7. Open the Pipelines tab
  8. Click on YAML Pipelines
  9. Select your pipeline
  10. The pipeline should run automatically (It may take a few seconds to start)
  11. Once it finishes look at the build log for the second build job

You can see your changes in the build log. 

[2024-07-01 17:47:28] BEGIN shell script execution with /bin/sh -ex
[2024-07-01 17:47:28] + echo 'My second YAML build job'
[2024-07-01 17:47:28] My second YAML build job
[2024-07-01 17:47:28] + cat README.md
[2024-07-01 17:47:28] app-for-yml-build.git
[2024-07-01 17:47:28] Change 1
[2024-07-01 17:47:28] Change 2
[2024-07-01 17:47:28] END shell script execution

What's Next?

If this is your fist experience with YAML pipelines don't worry, you can start out slow.  It's perfectly fine to create a pipeline using the GUI tools and convert one build job at a time to YAML.  Visual Builder Studio Pipelines can contain both GUI and YAML jobs in the same pipeline.

Visual Builder Studio also has a REST API to help you.  You can export GUI defined jobs and pipelines as YAML then save them in files using the REST endpoints found here (https://docs.oracle.com/en/cloud/paas/visual-builder/visualbuilder-rest-apis/op-v1-yaml-pipelines-pipelinename-get.html, https://docs.oracle.com/en/cloud/paas/visual-builder/visualbuilder-manage-development-process/rest-api-accessing-yaml-files.html).

For Example:

curl -X GET -u USERNAME:PASSWORD YourServerURL/v1/yaml/jobs/JobName

Replace USERNAME, PASSWORD, YourServerURL and JobName with your values to export a GUI defined build job as YAML.

More information on configuring YAML jobs and pipelines can be found here (https://docs.oracle.com/en/cloud/paas/visual-builder/visualbuilder-manage-development-process/configure-jobs-and-pipelines-yaml.html).

Using the above examples, you can include the definition of your CICD build process in the same Git repository as your code, or even a separate Git repository.  This will allow your team to include the CICD pipeline definition in your standard development processes.

For Example:

  1. Make a ticket for a CICD change
  2. Make the change using YAML files
  3. Check your files into a Git branch
  4. Follow the same Merge Request process that you use for the rest of your project
  5. Accept the changes and merge them into your main branch

The great thing is, it's all just code.  You can have multiple branches, each with their own pipeline, you can even use all of the Git processes you use for application code changes.  If you are working in an Infrastructure as Code environment, YAML pipelines will fit in perfectly.