Friday Oct 09, 2015

DevOps on Oracle Solaris 11

There's no doubting the popularity of the DevOps movement these days. We're seeing it in many of our customers as the need to move faster in their business becomes more important. More often than not, it's being combined with the move to cloud computing and self-service everything. The traditional application development model with infrastructural and organizational silos is dead....well, almost!

DevOps promotes an open culture of collaboration, merging these silos into central teams with a more agile development methodology. Everything from development to production is automated as much as possible, allowing applications to be continuously developed, built, and released into production. In this environment everything is monitored and measured allowing for faster feedback back into the development cycles, with many incremental changes over short time periods. While the key to success for a DevOps environment is really the work environment itself, we've certainly seen some changes to tools that have made such an agile methodology much, much easier.

Many folks connect DevOps with Linux on commodity x86 based systems in the cloud. Not necessarily so! In my latest technical article Automated Application Development and Deployment with DevOps on Oracle Solaris 11, I put a simple application pipeline together to demonstrate a typical DevOps environment on Oracle Solaris 11. In this article, we'll take a look at Git distributed version control, Apache Maven build automation tool, Jenkins continuous integration server, and Puppet configuration management. I'll also show some integration with IPS using a Maven IPS plugin to automatically generate new packages that can be quickly deployed on a successful test run.

Let me know what you think!

Wednesday Sep 09, 2015

Another OTN Virtual Sys Admin Day

Next week we'll be hosting our next Oracle Technology Network virtual technology summit event. This is an opportunity for folks to tune into a bunch of technical sessions, including content on DBaaS, Java, WebLogic, Oracle Solaris and ZFS storage, Puppet and Linux. For the Oracle Solaris session Duncan Hardie and I will be talking about some of the new things that we've introduced in Oracle Solaris 11.3, including OpenStack, Puppet and ZFS and how we're continuing to work to make Oracle Solaris a great cloud platform capable of both horizontal and vertical scale. Check out the rest of the agenda here.

We'll be running 3 separate events for different timezones - Americas, Europe and Asia/Pacific. Register now and join us!

Integrated technologies FTW

In the latest articles I've been writing, I've been trying to link some of the Oracle Solaris technologies together and show how they can be used for a more complete story. The nice thing about Oracle Solaris is that we really care about the integration between technologies - for example, Oracle Solaris Zones is pretty seamlessly linked with ZFS, the entire network space, IPS packaging, Unified Archives and SMF services. It's absolutely our point of differentiation, and it's a hell of a lot less frustrating an administration experience as a result. Linux really is a poor cousin that regard.

Which is why I was really thrilled to see Thorsten Mühlmann latest blog, Deploying automated CVE reporting for Solaris 11.3. He talks through how to provide regular reporting of CVE (Common Vulnerability Exploits) for his systems. Not only does he use the integrated CVE meta-data in IPS, a core part of our wider compliance framework, but he provides the integration in IPS and SMF to make this easily deployable across the systems he manages with Puppet. It's a really nice example of how to engineer things that are reliable, repeatable and integrated. Thanks Thorsten!

Tuesday Sep 01, 2015

Applying read-only protection with Immutable Zones

A while back, I wrote an article that focus on how you can achieve a secure and compliant application deployment from development, through test and production. This article took advantage of a number of Oracle Solaris technologies including Oracle Solaris Zones, Unified Archives, Immutable Zones, Automated Installer and the integrated compliance framework.

We've had a number of customers get really excited by Immutable Zones and being able to lock down their environments. Not only does this provide an additional layer of security, but also protects against the potential cost of human error and ensures that organisations can meet their compliance requirements routinely. Darren Moffat and Casper Dik have already written great blog entries on how to do this, but I've also recently published another How-To article on Applying read-only protection with Oracle Solaris Immutable Zones. In this article we cover immutable non-global and global zones, and show how we can make administrative changes such as applying critical security fixes using the Trusted Path. Hope you find it useful!

Friday Jul 31, 2015

Secure Remote RESTful Administration with RAD

I've written before about the work we've been doing to provide a set of programmatic interfaces to Oracle Solaris using RAD. This allows developer and administrators to administer systems remotely over C, Java, Python and REST based interfaces. For anyone wanting to get their hands dirty, I've written a useful article: Getting Started with the Remote Administration Daemon on Oracle Solaris 11.

One of the areas I didn't tackle in this initial article was providing secure REST based administration interfaces over TLS. Thanks to the help of Gary Pennington, we now have a new article: Secure Remote RESTful Administration with RAD. In this article we'll use the automatically generated self-signed certificates, but this could be easily changed to point to certificates that have been signed by a Certificate Authority.

With the various announcements that we've been making recently about Oracle joining the Open Container Initiative and bringing Docker into Oracle Solaris, we're in a great position of being able to design a platform to handle the next wave of cloud deployment and delivery - whether that's traditional enterprise applications or micro services. We see the huge advantage of streamlining IT operations and facilitating methodologies such as DevOps, and it's time to take Oracle Solaris into that next wave.

Monday Jul 13, 2015

Periodic and scheduled services with SMF

With the release of Oracle Solaris 11.3 Beta last week, we've introduced a metric ton of new features. I'm really excited by the direction Oracle Solaris has been taking ad we continue to modernise the platform, include software administrators and developers are using on other platforms, and generally ensure we're ready to support the next generation of applications and infrastructure. If you've not really been following along, I'd strongly suggest you download Oracle Solaris 11.3 and have a play.

Back in 2005, we took the brave step to move away from /etc/init.d and introduced the Service Management Facility (SMF) as the main way to manage application and system services. SMF provided us with automatic service dependencies, central logging, structured configuration management, reliable application restart in the event of hardware or software failures as part of the overall fault management architecture in Oracle Solaris, and a much, much easier way of administering services. Better still, we converted all the system services over to SMF straight away and improved startup performance as we could now graph service dependencies and identify issues. You can under estimate the significance of this work, especially if you've read the turbulent history of systemd.

That was then, and this is now. One of the exciting enhancements in Oracle Solaris 11.3 relates to SMF, the introduction of the periodic and scheduled services. In another bold move, we're hoping to knock cron off it's block. There's no doubt cron is a foundation of scheduling in UNIX and Linux environments, and will be for years to come. But with scheduled SMF services we take all the ability of cron and combine them with all the benefits of SMF.

Creating an SMF periodic service is easy, with a simple addition to your SMF manifest to describe a periodic method (or using svcbundle):

        <method_credential user='oracle' group='dba' />
In the above snippet, we can see that we're executing /usr/local/bin/db_check every 10-11 minutes (as indicated by a jitter attribute of 60 seconds) with a maximum of 30 seconds delay after the service has been transitioned to the online state. We've also given it a method credential to run the script as the oracle user with dba group. The svc:/system/svc/periodic-restarter:default service instance will be responsible for restarting this service periodically.

Scheduled services are services that are run at a specific time, perhaps at an off-peak time. Similarly these are easy to create with a simple addition to your SMF manifest (or again by using svcbundle):

        <method_credential user='oracle' group='db' />
In the above snippet, we can see that we're executing /usr/local/bin/db_backup every day at 2am (as indicated by the hour and minute attributes). In this case the frequency is set as a default value of 1, meaning that we will run this every day. Like the previous example, we have given it a method credential to run the script as the oracle user with dba group. The svc:/system/svc/periodic-restarter:default service instance is also responsible for ensuring this services runs to its defined schedule.

One of the outstanding gaps with the Image Packaging System (IPS) was the ability to associate cron jobs during package install time by locating . Some other platforms have solved this with the introduction of /etc/cron.d using a process of self-assembly of the system's cron entries. We don't support this ability with the cron version included in Oracle Solaris 11. But now using periodic or scheduled services, administrators can simply install their SMF manifests into /lib/svc/manifest/site and restart the svc:/system/manifest-import:default service instance. You can achieve this with an IPS manifest fragment that uses an IPS actuator similar to the following:

file lib/svc/manifest/site/db-backup.xml \
    path=lib/svc/manifest/site/db-backup.xml owner=root group=sys \
    mode=0444 restart_fmri=svc:/system/manifest-import:default

So take the plunge and move your cron entries over to SMF today - you'll not regret it! Our plan is to convert the existing system cron entries over in future releases. For more information, see the following chapters in the excellent Oracle Solaris 11.3 Product Docs:

Wednesday Jul 08, 2015

Remote Administration with RAD and Oracle Solaris 11

As organisations look for increased agility in their IT operations, many are turning towards more cloud like environments with shared compute, network and storage, and the ability for self-service users to quickly provision new virtualised environments on demand. With this increased virtualization sprawl, it's imperative to have a set of tools to allow administrators to effectively manage these environments, ensure they remain highly available, secure and observable.

There's hundreds of tools that have been created to help administrators manage their environments more effectively. Many tools such as Puppet and Chef, have inspired administrators to shift legacy enterprise management models over towards more rapid, agile and 'dev ops' like models. In Oracle Solaris 11, we've worked hard to modernise the operating system to adapt to this change and transform it into a highly capable cloud platform. We've included tools like Puppet as a response to customer demand, but we've also created our own - in this case RAD.

RAD (or Remote Administration Daemon) provides a set of programmatic interfaces to allow administrators to manage Oracle Solaris 11 subsystems using Python, C, Java, and RESTful APIs. RAD is also intended for developers as a complete development framework for creating their own custom interfaces to manage systems. I've written a getting started article that covers the basics of RAD, including some examples of using a few of the Oracle Solaris RAD modules. RAD is a very strategic technology for us because it provides a standardised set of interfaces to allow Oracle and other 3rd parties to write their own management interfaces on top of RAD. In fact we've already used RAD extensively in our port of OpenStack to Oracle Solaris.

Getting Started with the Remote Administration Daemon on Oracle Solaris 11.

Oracle Solaris 11.3 Beta Now Available!

We've done it again! Oracle Solaris 11.3 beta has been released today! The beta program is a great opportunity to download the latest release, try it out, and give us some feedback.

We've crammed in hundreds of new features into this release including some of my favourites: an updated OpenStack distribution (Juno), live migration support for Oracle Solaris Kernel Zones and hosting them over NFS using shared storage, bigger compression ratios with LZ4 support in the ZFS file system, PVLAN support, REST APIs and additional RAD modules (see here), Hiera to allow easy variable substitution in your Puppet manifests, faster Oracle Database 12c startups and SGA resize with Optimised Shared Memory, and everything that goes into supporting Oracle's next generation systems based on the SPARC M7 processor including Application Data Integrity (ADI) that helps prevent illegal memory access during a malicious attack.

There's a lot more, so I'd encourage you to check out the Oracle Solaris 11.3 Beta What's New and see for yourself.

Wednesday Apr 29, 2015

Managing Oracle Solaris systems with Puppet

This morning I gave a presentation to the IOUG (Independent Oracle Users Group) about how to manage Oracle Solaris systems using Puppet. Puppet was integrated with Oracle Solaris 11.2, with support for a number of new resources types thanks to Drew Fisher. The presentation covered the challenges in today's data center, some basic information about Puppet, and the work we've done to integrate it as part of the platform. Enjoy!

Wednesday Feb 25, 2015

New Solaris articles on Oracle Technology Network

I haven't had much time to do a bunch of writing for OTN, but here's a few articles that have been published over the last few weeks that I've had a hand in. The first is a set of hands on labs that we organised for last year's Oracle Open World. We walked participants through how to create a complete OpenStack environment on top of Oracle Solaris 11.2 and a SPARC T5 based system with attached ZFS Storage Appliance. Once created, we got them to create a golden image environment with the Oracle DB to upload to the Glance image repository for fast provisioning out to VMs hosted on Nova nodes.

The second article I teamed up with Ginny Henningsen to write. We decided to write an easy installation guide for Oracle Database 12c running on Oracle Solaris 11, covering some of the tips and tricks, along with some ideas for what additional things you could do. This is a great complement to the existing white paper, which I consider an absolute must read for anyone deploying the Oracle Database on Oracle Solaris.


Friday Aug 15, 2014

Mirroring IPS repositories

Out of the many changes introduced in packaging with the Oracle Solaris 11.2 release, one of really good ones was the introduction of a repository mirroring service. This provides administrators with an easy, automated way of mirroring repository contents. For example, let's say you had a package repository set up locally that was serving the clients in your data center. While we provide a few different ways to sync up the contents of this repository with the Oracle Solaris 11 support repository hosted by Oracle through the pkgrecv utility or incremental ISO images, it's a pretty manual process. Now it's a case of simply configuring and starting an SMF service, svc:/application/pkg/mirror:default.

I've written a short article on this new IPS feature - How to Set Up a Repository Mirroring Service with the Oracle Solaris 11 Image Packaging Service.

You'll also notice that I also include a sneaky mention of pkg exact-install, another new feature that allows administrators to essentially reset a system to a known software boundary. Bart Smaalders has already covered this in a great blog post.

Friday Aug 01, 2014

Oracle Solaris 11.2 Available

We got there in the end, and today, Oracle Solaris 11.2 has been officially made available.

This is the most significant release I've had the pleasure to work on, and we've made huge strides in terms of overall usability, performance, and functionality. There are some really incredible base technologies included in this release - everything from independent kernel versions and patching with Oracle Solaris Kernel Zones, fast and portable clone and disaster recovery images with Unified Archives, simple to use compliance framework built on OpenSCAP, and open cloud infrastructure with OpenStack. With 21 months of hard work, Oracle Solaris 11.2 represents a huge milestone as it shifts from an enterprise-grade operating system, to a comprehensive cloud platform.

Of course we're still ensuring we're the best platform for enterprise applications and ensuring an engineered solution for Oracle - those go without saying.

Download Oracle Solaris 11.2 today!

Secure, compliant application deployment with Oracle Solaris 11

One of the really exciting features that was introduced in Oracle Solaris 11.2 is called Unified Archives. Unified Archives provide system cloning and disaster recovery capabilities for the platform. Built on the foundations of Oracle Solaris ZFS, an archive can quickly be taken on a live running system thanks to snapshot and cloning. A single archive can be created for a complete system that includes a number of virtual environments. Once captured, it can be deployed using Automated Installer or using the existing zonecfg(1M) and zoneadm(1M) utilities during Oracle Solaris Zone creation. Thanks to integration with the IPS packaging system, an archive can be partially deployed with complete flexibility - across different systems of the same architecture, or using physical-to-virtual or virtual-to-physical transforms. They're completely flexible. Jesse Butler, the architect for Unified Archives, has already covered a lot of the basics in two blog posts: Introducing Unified Archives in Oracle Solaris 11.2 and Cloning Zones with Unified Archives.

Unified Archives are a pretty critical piece of the overall application lifecycle. Combined with Oracle Solaris Zones, Immutable Zones (read-only VMs), and our new compliance framework, we have a very nice set of technologies that can be combined to really aid developers and administrators in creating and deploying compliant application environments, from development through to test and eventually production. I've written an article that helps explain how you can achieve this, and greatly cut down the cost of ensuring certified and compliant applications and reducing the cost of human error or security exploits.

Take a look at How to Ensure Secure, Compliant Application Deployment with Oracle Solaris 11.

Monday Jul 21, 2014

Understanding IPS versioning

During the lead up to Oracle Solaris 11.2 GA, I noticed that I had written an article back last year that never got published about understanding IPS package versioning. If you haven't yet had a chance to look at Oracle Solaris 11, one of the really great changes that we introduced was completely replacing the packaging mechanism from the rather legacy SVR4 packaging system to the network based Image Packaging System. IPS relies on the fact that ZFS is the underlying file system using a feature called ZFS Boot Environments, allowing us to take advantage of snapshots and clones while updating systems. This means that administrators can perform a system update while still having the old environment to fallback to if something goes wrong. There was a similar concept in Oracle Solaris 10, but was quite primitive by comparison.

And so to the document in question. IPS uses a pretty comprehensive versioning system to allow it to calculate how to go about performing a system update, or indeed any individual software package. We use a series of package constraints on the system to ensure that administrators are updating their software to a well known, and tested state. By contrast, Oracle Solaris 10 essentially let you update or apply any patches you wanted, often leading our customers down a very un-tested path. It's useful to understand this versioning system at times so I've written a useful article that covers some of this.

Take a read of Understanding Oracle Solaris 11 Package Versioning.

Thursday Jun 12, 2014

Interactive manifest editing with the Automated Installer Manifest Wizard

Oracle Solaris 11.2 adds a new Automated Installer (AI) Manifest Wizard to allow administrators to more easily create AI manifests for use in provisioning new client systems in the data center. The AI Manifest Wizard is a web web based interface that steps administrators through the basics of the AI manifest - target disks and layout selection, additional ZFS pools and datasets, IPS publisher and package selection, and the creation of any Oracle Solaris Zone virtual environments. The end result is an AI manifest without having to directly edit XML, and this can then be associated with an appropriate AI service.

To get started, check out How To Create an Automated Installer Manifest with an Interactive Wizard


To learn more about Oracle Solaris 11, check out an extensive list of resources including technical articles, cheat sheets and screencasts on Oracle Technology Network


« October 2015