Friday Oct 09, 2015

DevOps on Oracle Solaris 11

There's no doubting the popularity of the DevOps movement these days. We're seeing it in many of our customers as the need to move faster in their business becomes more important. More often than not, it's being combined with the move to cloud computing and self-service everything. The traditional application development model with infrastructural and organizational silos is dead....well, almost!

DevOps promotes an open culture of collaboration, merging these silos into central teams with a more agile development methodology. Everything from development to production is automated as much as possible, allowing applications to be continuously developed, built, and released into production. In this environment everything is monitored and measured allowing for faster feedback back into the development cycles, with many incremental changes over short time periods. While the key to success for a DevOps environment is really the work environment itself, we've certainly seen some changes to tools that have made such an agile methodology much, much easier.

Many folks connect DevOps with Linux on commodity x86 based systems in the cloud. Not necessarily so! In my latest technical article Automated Application Development and Deployment with DevOps on Oracle Solaris 11, I put a simple application pipeline together to demonstrate a typical DevOps environment on Oracle Solaris 11. In this article, we'll take a look at Git distributed version control, Apache Maven build automation tool, Jenkins continuous integration server, and Puppet configuration management. I'll also show some integration with IPS using a Maven IPS plugin to automatically generate new packages that can be quickly deployed on a successful test run.

Let me know what you think!

Wednesday Sep 09, 2015

Integrated technologies FTW

In the latest articles I've been writing, I've been trying to link some of the Oracle Solaris technologies together and show how they can be used for a more complete story. The nice thing about Oracle Solaris is that we really care about the integration between technologies - for example, Oracle Solaris Zones is pretty seamlessly linked with ZFS, the entire network space, IPS packaging, Unified Archives and SMF services. It's absolutely our point of differentiation, and it's a hell of a lot less frustrating an administration experience as a result. Linux really is a poor cousin that regard.

Which is why I was really thrilled to see Thorsten Mühlmann latest blog, Deploying automated CVE reporting for Solaris 11.3. He talks through how to provide regular reporting of CVE (Common Vulnerability Exploits) for his systems. Not only does he use the integrated CVE meta-data in IPS, a core part of our wider compliance framework, but he provides the integration in IPS and SMF to make this easily deployable across the systems he manages with Puppet. It's a really nice example of how to engineer things that are reliable, repeatable and integrated. Thanks Thorsten!

Monday Jul 13, 2015

Periodic and scheduled services with SMF

With the release of Oracle Solaris 11.3 Beta last week, we've introduced a metric ton of new features. I'm really excited by the direction Oracle Solaris has been taking ad we continue to modernise the platform, include software administrators and developers are using on other platforms, and generally ensure we're ready to support the next generation of applications and infrastructure. If you've not really been following along, I'd strongly suggest you download Oracle Solaris 11.3 and have a play.

Back in 2005, we took the brave step to move away from /etc/init.d and introduced the Service Management Facility (SMF) as the main way to manage application and system services. SMF provided us with automatic service dependencies, central logging, structured configuration management, reliable application restart in the event of hardware or software failures as part of the overall fault management architecture in Oracle Solaris, and a much, much easier way of administering services. Better still, we converted all the system services over to SMF straight away and improved startup performance as we could now graph service dependencies and identify issues. You can under estimate the significance of this work, especially if you've read the turbulent history of systemd.

That was then, and this is now. One of the exciting enhancements in Oracle Solaris 11.3 relates to SMF, the introduction of the periodic and scheduled services. In another bold move, we're hoping to knock cron off it's block. There's no doubt cron is a foundation of scheduling in UNIX and Linux environments, and will be for years to come. But with scheduled SMF services we take all the ability of cron and combine them with all the benefits of SMF.

Creating an SMF periodic service is easy, with a simple addition to your SMF manifest to describe a periodic method (or using svcbundle):

        <method_credential user='oracle' group='dba' />
In the above snippet, we can see that we're executing /usr/local/bin/db_check every 10-11 minutes (as indicated by a jitter attribute of 60 seconds) with a maximum of 30 seconds delay after the service has been transitioned to the online state. We've also given it a method credential to run the script as the oracle user with dba group. The svc:/system/svc/periodic-restarter:default service instance will be responsible for restarting this service periodically.

Scheduled services are services that are run at a specific time, perhaps at an off-peak time. Similarly these are easy to create with a simple addition to your SMF manifest (or again by using svcbundle):

        <method_credential user='oracle' group='db' />
In the above snippet, we can see that we're executing /usr/local/bin/db_backup every day at 2am (as indicated by the hour and minute attributes). In this case the frequency is set as a default value of 1, meaning that we will run this every day. Like the previous example, we have given it a method credential to run the script as the oracle user with dba group. The svc:/system/svc/periodic-restarter:default service instance is also responsible for ensuring this services runs to its defined schedule.

One of the outstanding gaps with the Image Packaging System (IPS) was the ability to associate cron jobs during package install time by locating . Some other platforms have solved this with the introduction of /etc/cron.d using a process of self-assembly of the system's cron entries. We don't support this ability with the cron version included in Oracle Solaris 11. But now using periodic or scheduled services, administrators can simply install their SMF manifests into /lib/svc/manifest/site and restart the svc:/system/manifest-import:default service instance. You can achieve this with an IPS manifest fragment that uses an IPS actuator similar to the following:

file lib/svc/manifest/site/db-backup.xml \
    path=lib/svc/manifest/site/db-backup.xml owner=root group=sys \
    mode=0444 restart_fmri=svc:/system/manifest-import:default

So take the plunge and move your cron entries over to SMF today - you'll not regret it! Our plan is to convert the existing system cron entries over in future releases. For more information, see the following chapters in the excellent Oracle Solaris 11.3 Product Docs:

Friday Aug 15, 2014

Mirroring IPS repositories

Out of the many changes introduced in packaging with the Oracle Solaris 11.2 release, one of really good ones was the introduction of a repository mirroring service. This provides administrators with an easy, automated way of mirroring repository contents. For example, let's say you had a package repository set up locally that was serving the clients in your data center. While we provide a few different ways to sync up the contents of this repository with the Oracle Solaris 11 support repository hosted by Oracle through the pkgrecv utility or incremental ISO images, it's a pretty manual process. Now it's a case of simply configuring and starting an SMF service, svc:/application/pkg/mirror:default.

I've written a short article on this new IPS feature - How to Set Up a Repository Mirroring Service with the Oracle Solaris 11 Image Packaging Service.

You'll also notice that I also include a sneaky mention of pkg exact-install, another new feature that allows administrators to essentially reset a system to a known software boundary. Bart Smaalders has already covered this in a great blog post.

Friday Aug 01, 2014

Secure, compliant application deployment with Oracle Solaris 11

One of the really exciting features that was introduced in Oracle Solaris 11.2 is called Unified Archives. Unified Archives provide system cloning and disaster recovery capabilities for the platform. Built on the foundations of Oracle Solaris ZFS, an archive can quickly be taken on a live running system thanks to snapshot and cloning. A single archive can be created for a complete system that includes a number of virtual environments. Once captured, it can be deployed using Automated Installer or using the existing zonecfg(1M) and zoneadm(1M) utilities during Oracle Solaris Zone creation. Thanks to integration with the IPS packaging system, an archive can be partially deployed with complete flexibility - across different systems of the same architecture, or using physical-to-virtual or virtual-to-physical transforms. They're completely flexible. Jesse Butler, the architect for Unified Archives, has already covered a lot of the basics in two blog posts: Introducing Unified Archives in Oracle Solaris 11.2 and Cloning Zones with Unified Archives.

Unified Archives are a pretty critical piece of the overall application lifecycle. Combined with Oracle Solaris Zones, Immutable Zones (read-only VMs), and our new compliance framework, we have a very nice set of technologies that can be combined to really aid developers and administrators in creating and deploying compliant application environments, from development through to test and eventually production. I've written an article that helps explain how you can achieve this, and greatly cut down the cost of ensuring certified and compliant applications and reducing the cost of human error or security exploits.

Take a look at How to Ensure Secure, Compliant Application Deployment with Oracle Solaris 11.

Monday Jul 21, 2014

Understanding IPS versioning

During the lead up to Oracle Solaris 11.2 GA, I noticed that I had written an article back last year that never got published about understanding IPS package versioning. If you haven't yet had a chance to look at Oracle Solaris 11, one of the really great changes that we introduced was completely replacing the packaging mechanism from the rather legacy SVR4 packaging system to the network based Image Packaging System. IPS relies on the fact that ZFS is the underlying file system using a feature called ZFS Boot Environments, allowing us to take advantage of snapshots and clones while updating systems. This means that administrators can perform a system update while still having the old environment to fallback to if something goes wrong. There was a similar concept in Oracle Solaris 10, but was quite primitive by comparison.

And so to the document in question. IPS uses a pretty comprehensive versioning system to allow it to calculate how to go about performing a system update, or indeed any individual software package. We use a series of package constraints on the system to ensure that administrators are updating their software to a well known, and tested state. By contrast, Oracle Solaris 10 essentially let you update or apply any patches you wanted, often leading our customers down a very un-tested path. It's useful to understand this versioning system at times so I've written a useful article that covers some of this.

Take a read of Understanding Oracle Solaris 11 Package Versioning.

Tuesday Mar 26, 2013

The Building Blocks of an Oracle Solaris 11 Application Deployment

A while back I blogged about a simple deployment example using some of the new and old technologies included in Oracle Solaris 11. Together they provide some of the basic building blocks that you can use for a more complex deployment in your data center. I've been meaning to put some audio to the presentation and got some time today to do just that. So if you haven't seen the original set of slides, watch the 30 min presentation below with voice over from yours truly. Enjoy!

Friday Nov 23, 2012

A simple deployment example using Oracle Solaris 11

Last week I was over in Melbourne and Sydney to present at an Oracle Technology Network System Administration Day. During a set of morning sessions, I presented on some of the new things that we've done in Oracle Solaris 11 and our focus for the future. One of the sessions I presented was giving a quick technical overview of what a typical application deployment scenario would look like using technologies like the Image Packaging System (IPS), Service Management Facility (SMF), Oracle Solaris Zones, and network virtualization. While it's just a simple example, it provides the basic building blocks for a more advanced configuration that a data center would typically deploy. Given these would likely be of general interest, I thought I'd upload the slides for others to view - enjoy!

You can download the original source PPTX without some of the formatting errors in the above.

Monday Jul 16, 2012

The basics of administration using the Image Packaging System

One of the major new changes that was introduced in Oracle Solaris 11 is a new package management system, replacing the legacy SVR4 packaging that we've been using in Oracle Solaris for many years. Image Packaging System (IPS) modernizes the software lifecycle with an easy to use, network based packaging system built on the foundations of the Oracle Solaris ZFS filesystem. I've been working with the IPS engineering team to produce some content that will help administrators understand the basics of IPS and how best to take advantage of it updating systems in the data center.

An overview of IPS

In previous releases of the Oracle Solaris platform, administrators used SVR4 packaging to install software onto a system, and then they used a different set of commands to install patches to update the system. As Oracle Solaris evolved to include new technologies, such as Oracle Solaris zones, Oracle Solaris ZFS, and Solaris Service Management Facility (SMF), previously used processes for managing system updates and upgrades became more complex. With thousands of operating system instances installed in some of today's large virtualized data centers, manual methods of tracking and installing patches can result in errors that negatively affect application availability and security.

IPS is an integrated solution that helps automate and ease the complexity of managing system software on Oracle Solaris 11 by integrating patching with package updates. It relies on a network-centric and efficient approach with automatic software dependency checking and validation, and it builds on the foundation of ZFS as the default root file system. Using IPS, administrators can easily and reliably install or replicate an exact set of software package versions across many different client machines, and get a much clearer understanding of any differences between software versions installed on a system.

With ZFS's ability to snapshot and clone a given file system with little or no overhead, IPS establishes a much safer system update by applying changes to a clone or alternate boot environment so that updates can be done while a system is running services in a production environment. When a planned maintenance window can be scheduled, administrators can simply reboot the system into the new boot environment to get up and running faster with much lower system downtime. If administrators experience any problems with a new environment, they can simply bring the system down and back into the older boot environment.

Additionally, administration across any zones created on the system is much improved because the system automatically ensures that software package versions within a non-global zone are consistent with the global zone.

For the rest of the article, head on over to OTN and read Introducing the Basics of the Image Packaging System on Oracle Solaris 11.


To learn more about Oracle Solaris 11, check out an extensive list of resources including technical articles, cheat sheets and screencasts on Oracle Technology Network


« October 2015