Monday Feb 06, 2017

Part 4: Making an Oracle Linux Docker Vagrant Box

So now we have a Vagrant Oracle Linux 7 base image which can now become the foundation for future derived images. In this article, we will manually build a Docker Box based on the Linux base image ol7-base.

Then in the next article (Part 4.5), we will do exactly the same as the below but fully automate it. In other words no logging in, no editing files and configuring the base image but rather applying all the changes through the Vagrantfile via its provisioning capability as we move to Infrastructure as Code.

The true value of this becomes apparent in the future when you want to create/re-create a new base image and you can now automatically re-generate the Docker box.

But first, the manual way:

  1. Setup Yum
  2. Install and enable Docker
  3. Clean up the image (see Part 3)
  4. Create a new, derived, Docker box

1. Setup Yum

Oracle provides a number of yum channels, some of which are not enabled by default. Docker lives in one of these, the [ol7-addons] section. The curious can take a look at the file below:

/etc/yum.repos.d/public-yum-ol7.repo

So let's enable the ol7-addons:

$ sudo yum install yum-utils
$ sudo yum-config-manager --enable ol7_addons

Installing yum-utils is not strictly necessary but greatly assists automation by eliminating the need to open and edit the repo files manually which can be both error prone but also harder to script/automate.

(An alternative would have been to use the Vagrant file provisioner to move a new repo file into /etc/yum.repos.d - a trick we'll cover the mechanics of in the next Blog article.)

2. Install and enable Docker

Now on to install and enable Docker.

$ sudo yum install docker
$ sudo systemctl start docker
$ sudo systemctl status docker
$ sudo systemctl enable docker
$ sudo init 0

The wise will probably at this point restart the finished box and check that Docker is actually configured and running.

$ vagrant up
$ vagrant ssh
$ sudo docker run whalesay

Now we are ready to repeat the steps from the previous article, Part 3 Sections 3 and 4:

3. Clean the Image

Either su to root or use sudo to perform the tasks below, we'll use sudo. As a reminder this task allows us to shrink the image, somewhat. In a future post i plan to look at 2 approaches to making much smaller images.

$ sudo yum update
$ sudo yum clean all
$ sudo dd if=/dev/zero of=/foo bs=100k
$ sudo rm /foo
$ sudo  rm -rf /tmp/*
$ sudo init 0

4. Create a new, derived, Docker box

Final stage and quite straightforward. One you have done this for Docker you will realize how simple it is to derive new boxes and can use the same techniques to create other new boxes.

Basically, this follows the steps in the last part of the previous post with the exception of the box names. I'll call the new box ol7-docker. Of course, you can name it anything you want but this will be the box name you would use with the vagrant init ol7-docker command.

So having exited your guest machine let's just ensure it is fully closed down before packing it. Make sure you do this from the directory containing the existing Vagrantfile.

$ vagrant halt
$ vagrant package 
$ vagrant box add ol7-docker package.box

5. Test the new Vagrant Oracle Linux Docker Box

All that remains is to verify the box works as expected:

$ mkdir test-docker
$ vagrant init ol7-docker
$ vagrant up
$ vagrant ssh

In your new login session to the ol7-docker box:

$ sudo docker run hello-world

All being well you should see Docker pull in hello-world and run it.

Wednesday Jan 25, 2017

Part 3: Using Vagrant, VirtualBox and Oracle Linux for Docker: Building the Vagrant Box

So following on from Part Two we now have a basic Oracle Linux VM with the VirtualBox additions installed. Next we want to customize that image, prepare it for vagarnt, and finally run the conversion process to become the Vagrant box. This is actually quite a straightforward process but you want to make sure you do each of the steps carefully otherwise you may end up with a box that won’t boot, or perhaps won’t allow SSH, or you’ll find sudo won’t work. Basically this can be broken down into a 4 Step process:

  1. Create the vagrant user, setup SSH
  2. Setup sudo
  3. Final Linux updates if required and minimize your system image and cleanup
  4. Convert the VirtualBox VM to a Vagrant Box

1. Create the vagrant user,  setup SSH and sudo

# useradd vagrant  
# su – vagrant 
$ mkdir .ssh 
$ chmod 700 .ssh 
$ cd .ssh 
$ cu‌rl -k https://raw.githubusercontent.com/mitchellh/vagrant/master/keys/vagrant.pub \
 -o authorized_keys 
$ chmod 600 authorized_keys

Basically we’ve just added the vagrant user, setup their environment for ssh and provided them with well known keys i.e. insecure keys. It is actually straightforward to add your own keys but for now we’ll leave that as a future exercise more information can be found here. In fact both the user and ssh settings can be overridden in the Vagrantfile which is well documented.

2. Setup passwordless sudo access for your new vagrant user

# visudo 
Insert the line: 

vagrant ALL=(ALL) NOPASSWD:ALL

Again we are trading away Security for simplicity as this will give you passwordless sudo ability however if this is a concern you can modify to taste, for example omit the above step and use the root login and password.

At this point if you did want additional packages or configuration you can add them now. Remember the purpose of this exercise is simply to create a basic box which you can modify and build on in the future. So smaller is better. It’s simple to then take this box and add some packages in the future and derive new boxes from that image. For example you could create both a node.js box and a http box from the original base image.

3. Minimize your system image

Now is a great time to remove extra packages, or stop services from starting up. Just don’t stop sshd or you won’t be able to login! Final preparation, as root. First we run yum again in case there are any last updates, then it is time to clean up and overwrite the free space with 0s which will help in the shrink phase.

# yum update
# yum clean all
# dd if=/dev/zero of=/foo bs=100k
# rm /foo
# rm -rf /tmp/*
# shutdown -h

Now you are finished with preparing VM

4. Convert the VirtualBox VM to a Vagrant Box

Now comes the conversion process which takes place outside of the guest VM, back in your host operating system. You’ll need the name of your VirtualBox VM which you can either find in the VirtualBox GUI or you can simply list from the command line, in this case the VM is called ol7-base:

$ vboxmanage list vms

In this case the image is called ol7-base. Now create the vagrant image in a 2 step process:

$ vagrant package --base ol7-base
$ vagrant box add ol7-base package.box

The first command, vagrant package, actually converts the VirtualBox VM and outputs it in the current directory as a package.box file. Then this is added in the 2nd command to your collection of vagrant boxes as ol7-base. So now we have our newly created Vagrant box let’s test it actually works. In an appropriate sub directory, remember the Vagrantfile will be placed there so you'll want at least 1 directory for each box you work with.

$ mkdir test
$ cd test

$ vagrant init ol7-base
$ vagrant up
$ vagrant ssh 

So now you are up and running and hopefully you’ll find yourself in a Oracle Linux shell. In my next post we'll add Docker and customize our Vagrant box a little more.

Monday Jun 20, 2016

Part 2: Using Vagrant, VirtualBox and Oracle Linux for Docker: Building the Oracle Linux VM

In this second post in the series we will build a minimal Oracle Linux VM running on VirtualBox. This will provide the foundation for our Vagrant box and ultimately a platform to run Docker. You’ll need to download both unless you already have them installed:

One compelling consideration about the above stack is that it is completely free to download and deploy, not only in test and dev but also in production setting them apart from many of their competitors. So zero cost but combined with the ability to take out Commercial Support should you want those benefits. Same for VirtualBox’s big brother Oracle VM Server.

We’re going to be doing a minimal install which is pretty straightforward. At this point we won’t do any customizations for vagrant however there are a couple of prerequisites and best practices that we’ll touch on. Particularly when you are going to use vagrant it makes no sense to install a full-blown desktop environment. In fact I almost always do a minimal install, setup ssh, and from then run all my VM’s headless. I prefer the native host OS look and feel, tooling, simple cut-and-paste etc. Vagrant also makes it extremely easy to set up shared folders between the host and guest OS. Also these VM’s are lighter on local resources, something that can become noticeable when you start running several simultaneously. (I work primarily from iTerm on OSX and as I await Skylake Macs my aging Air has a meagre 4GB RAM.) 

Networking

A note on networking, normally I set networking to Bridged Adapter which allows for both inbound and outbound access from a single interface. So you can ssh into the VM and run outbound connections such as ping, yum etc and have all the  connectivity you need from that one interface. Vagrant actually insists on the first network adapter being set to NAT so we will be resetting this later.

Storage

On the storage side it is clearly a good practice to keep both applications and data outside the virtual machine’s base disk image. So I tend to go light on this, perhaps 2, or 4GB (In retrospect I'd say 4GB, 2GB and you may end up juggling space). I recommend you size to taste but allow at least 4GB. In the final part of the Vagrant box creation we will take steps to crush the image down as small as possible.

So let's get started, we can break this into 3 phases:

  1. Create a new Oracle Linux Virtual Box
  2. Configure Networking in Oracle Linux
  3. Update Linux and install necessary additional packages and the VirtualBox Additions

Create a New VirtualBox

Since you are planning on running vagrant and docker I'm assuming you are fairly comfortable installing and configuring a Linux VM. So just a bare bones list of the necessay steps below. If you want a more detailed step by step account you can refer to this entry and modify to suit. In the future I may do a full screen by screen entry, however it is pretty straightforward.

  1. Accept all the defaults:
  2. Set disk to whatever size you feel you need (4GB will work).
  3. Enable 1st Network device and set to Bridged Adapter (for now)
  4. Install Oracle Linux
  5. Minimal install, no user, set a root password

Configure Networking in Oracle Linux 

Networking is not setup to use DHCP by default on boot so some minor, but straightforward, configuration is needed here.

  • Find the uninitialized 2nd interface typically: enp0s3

# ip a

ip a gives network interfaces
  • Edit the necessary configuration file in  etc/sysconfig/network-scripts/ using the interface you just discovered above
  • # vi  /etc/sysconfig/network-scripts/ifcfg-enp0s3

  • Set ONBOOT=yes  #this will ensure interface gets IP address via DHCP at boot time.

ifcfg-enp0s3

  • Bring up the interface and check for the new IP address

# ifup enp0s3

# ip a  

ip address

Note the guest VM's IP address, in the above case 10.0.1.16 and you should be able to initiate both outbound and inbound connections to/from the VM.

  • From your guest Linux VM: $ ping oracle.com
  • From your host OS ssh into the guest $ ssh root@10.0.1.16

Update Linux and install necessary additional packages and the VirtualBox Additions

Now we have network connectivity it is time to:

  • Update Oracle Linux via yum
  • Install missing packages needed by the VirtualBox Extensions (kernel-headers and bzip2)
  • Install the VirtualBox Extensions

Updating and installing the missing packages

# yum update
# yum install bzip2 kernel-uek-devel

Install the VirtualBox Extensions

Whilst your VM will run and connect without installing the VirtualBox Extensions, you will get superior performance and experience by adding these, thus allowing the guest VM and VirtualBox to more deeply integrate and work together. First you need to load the .iso file from VirtualBox, whilst in your VirtualBox VM, not the VirtualBox Manager, Select the Devices Menu option and then "Insert Guest Additions CD Image..."

VirtualBox Guest Additions

Then from the command line either via ssh or in your console session you need to mount the CD/.iso file so enter the following commands as root: 

# mount /dev/cdrom /mnt
# /mnt/VBoxLinuxAdditions.run

Update 2017-01-19: With Oracle Linux 7.3 the above failed until a quick reboot, after which it ran fine.


# init 6 (old Unix habits die hard, you can use your reboot command of choice)

So now Part 2 is complete. We now have an up-to-date minimal install of Oracle Linux running on VirtualBox. In part 3 we will take this VM and convert it into a Vagrant box and start getting into the code habits enforced by Configuration Management and Infrastructure as Code.

(The above was tested on VirtualBox 5.0.22 and Oracle Linux 7.2)



Wednesday May 25, 2016

Part 1: Using Vagrant, VirtualBox and Oracle Linux for Docker

For my first posts I originally wanted  to jump in and take a look at running a "hello world” type Microservice type application with Docker. A quick look around the hard drives and I found a relatively fresh Oracle Linux VM (7.2), and even had the correct ssh keys. A dive into /etc and a few commands later and there seemed to be a couple of strange packages plus other strange additions - what is up in /data2? Some weirdness in a couple of configuration files and it seemed much faster to throw it away start over from scratch.

I think anyone who works with VM’s extensively knows this feeling, you’ve got an older working VM but you can’t remember 100% all the customizations you made so you feel safer throwing it away and starting again with a clean install…

So instead what I want to cover in the first series of posts is eliminating this problem altogether and actually practice Infrastructure as Code to build a repeatable known good configuration which is simple to apply, update, and share. No more “it works on my machine”, at least that’s the theory. The toolchain I want to explore is quite simple and entirely open source: 

Vagrant, VirtualBox, Oracle Linux, Docker

The Goal

Over the course of 4 posts we’ll build a 100% reproducible environment which creates an up-to-date Docker environment running on Oracle Linux (7.2 at this point) and requires no user intervention at all other than a single command to provision and run: vagrant up 


A quick introduction to vagrant

If you're not familiar with vagrant I can highly recommend it. Very lightweight, it provides great Infrastructure As Code tooling and effectively acts in a similar manner to a Makefile except for operating environments though you can also use it to install and configure applications as well. If you’re primarily using Docker for your development then vagrant can perform the role of automatically building and configuring the underlying operating environment to host the Docker containers. That is what we intend to do here.

Vagrant comprises 3 main components

Vagrant command, Vagrantfile, Providers

Vagrant Command

In Part 3 and Part 4 we will take a look at a handful of everyday vagrant commands to create, manage, and destroy boxes but in day-to-day operation it’s pretty simple. It will soon become clear why their domain is vagrantup.com

Vagrantfile

The Vagrantfile is very powerful not only allowing us to instantiate one, or more, virtual machines or containers but also allows us to heavily customize them and control attributes such as memory, networking, shared folders etc. It effectively acts in a similar manner to a Makefile except for operating environments though it can also install and configure applications. Docker users can see it performs a similar role to a Dockerfile. If the syntax doesn’t look familiar it is Ruby.

Vagrant Providers

Vagrant Providers give tremendous deployment flexibility. The default Provider, not surprisingly given its popularity, is VirtualBox however many other providers exist allowing Vagrant to support a multitude of Type I and II hypervisors, Docker, and various Cloud Platforms. In fact in a future post I hope to cover an example where we develop a very simple application on top of Linux running on VirtualBox and then deploy it to Oracle VM running on the operating system itself, and then later onto Docker.

In addition to building and configuring the operating environment Vagrant can be used to invoke Puppet, Chef etc. So whilst in some scenarios it can eliminate the need for these tools in others it becomes the first step in the Configuration Management workflow. Perhaps a good topic to explore at a later date but in the interests of space and time let’s get on with creating a reproducible Docker runtime environment using Vagrant, VirtualBox and Oracle Linux.

An Introduction to a DevOps, Security and Cloud Architecture Blog

I want to take a couple of paragraphs to briefly introduce the goals and typical content I want to write about here. With the rise of the cloud and particularly areas such as PaaS coupled with the tooling and the culture of DevOps it’s a very exciting time for software development. For so many of the people I’m lucky to meet and talk with I think Gene Kim’s quote “IT is the factory of the 21st century” sums it up well and that this whole area provides an abundance of topics on both a technical and a business level. In addition from an architectural perspective it is also a fascinating time as we see the evolution of Cloud Native Architectures and trends like Microservices. That said if there’s one thing this industry teaches us it’s that there are no silver bullets.

In conjunction with this the sheer abundance of connectivity and online data have seen an explosion in the challenges around security. Security has always been important however it’s probably fair to say that only in the last couple of years has that awareness spread far beyond the world of InfoSec and the Chief Security Officer and the magnitude and diversity of the problem start to be understood. Cloud clearly offers a fabulous foundation to create the agility and the security we seek however we need the right tools, practices, and processes to attain these goals as they don’t come for free. I hope to keep the style of posts quite broad varying from the more technical tutorial or HOWTO to higher-level discussion of architectures, trends and thoughts. 

Ultimately I hope that some of these posts will become useful either as a resource or as food for thought and discussion.
About

Welcome to a wide ranging blog on the world of DevOps, Application Architectures, the underlying tooling and keeping it more secure. The views expressed on this blog are my own and do not necessarily reflect the views of Oracle.

Search


Categories
Archives
« March 2017
SunMonTueWedThuFriSat
   
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
 
       
Today