Friday Jul 25, 2008

SMF snapshots, or why didn't my property change show up in svcprop?

One area of SMF that is a bit confusing is the whole notion of snapshots, refresh and how property updates occur. I find this a bit tricky myself, so if there's mistakes below, I'll happily correct them.

A snapshot is essentially a read-only picture of a set of property groups associated with an instance and its parent service. The most important snapshot is the 'running' snapshot, as it is the place applications (including svcprop) and service methods should read their configuration from. Why is this?

My understanding is that the snapshot model is there (in part at least) to allow atomic commit of multiple changes. This is particularly important for services that consume such configuration information. Imagine I want to change two properties in an inetd service. I'd like to change a tcp service from a wait-type (only handling one connection at a time) to a nowait-type service with a specific maximum connection rate. I need to change two properties - the 'wait' property, and the 'max_con_rate' property, since nonzero values for the latter only makes sense for nowait services. The snapshot model allows me to change each individually with svccfg, then refresh. The refresh action updates the running snapshot to contain a read-only picture of the latest values of my properties (it also runs the refresh method associated with the service instance, if there is one). Then entities consuming these values will see both changes at once as the running snapshot is updated atomically.

Sometimes you'll see the actual service and instance property values referred to as the 'configuration' or 'editing' snapshot, but that's not quite accurate, since as we saw above, snapshots are read-only and we make our changes to the actual configuration values, not a frozen picture of them.

If you select an instance using svccfg, you can see there's a bunch of snapshots in addition to 'running':

# svccfg -s nwam
svc:/network/physical:nwam> listsnap

Here we see another aspect of snapshots. svccfg(1M) allows us to revert back to earlier snapshots. The names of the various snapshots are pretty self-explanatory. The idea is, if I mess up my configuration, I just need to revert to the initial snapshot, or the one associated with last manifest import, etc.

You'll note something that seems weird - 'refresh' can be carried out both by svcadm and svccfg. Why is this? Well as we saw earlier, refresh has a configuration component - update the running snapshot - and an action component - run the refresh method. Configuration actions belong in svccfg, and actions in svcadm, so it makes sense to have refresh in both. Also, it's possible you may need to update the running snapshot in a SMF repository that's not currently running - maybe you made a configuration change that makes the system unbootable, and it's in the running snapshot, so you can't fix it with svccfg. Having refresh in svccfg makes such problems fixable, since it can operate on an inactive SMF repository. More on this another time.

So finally, why didn't my svccfg property change show up in svcprop? Because svcprop reads from the running snapshot, and until refresh is run, the changes haven't hit that snapshot yet.

Monday Jul 21, 2008

SMF property value ordering

With a lot of help from the SMF OpenSolaris community, I recently introduced a change in how the Service Management Framework (SMF) handles property values. I'll try to describe it here.

Service data in SMF is internally represented as groupings of property values (which are named, funnily enough property groups). Some specify dependencies, others the restarter that is responsible for handling starting and stopping the service, while others specify how a service is started, stopped or refreshed. In addition to specifying service data of relevance to the framework, property groups can specify configuration data that is consumed by the service too.

Each property in a property group must be of one specific type - a string, a boolean, an integer etc., but can have multiple values associated with it. Up until recently (before build 95 of OpenSolaris), these values were unordered - i.e. a consumer of the values could assume nothing about the relationship between the order in which they were stored and the order in which they were later retrieved. The putback of 6715372 - SMF repository should store (and allow retrieval of) property values in the order they were added - changes this. Now if property values are specified in a particular order via libscf(3scf) interfaces or via svccfg, they will be stored in that order, and are also retrieved in that order. I'll give a brief description of how this was solved.

Firstly, we need to understand how properties were stored previously. All SMF data is stored in a sqlite database file - in fact there are two database files. The persistent data is stored in /etc/svc/respository.db, and the transient data (temporary property groups and properties that do not survive reboot) is stored in /etc/svc/volative/svc_nonpersist.db. We can examine the content of these databases using /lib/svc/bin/sqlite, but it's not a good idea to do such a thing on a production system of course. Property values are stored in the 'value_tbl' database table. Let's display a single row in that table prior to my putback:

# lib/svc/bin/sqlite /etc/svc/repository.db
SQLite version 2.8.15-repcached-nightly
Enter ".help" for instructions
sqlite> SELECT \* FROM value_tbl LIMIT 1; 

The columns displayed are the associated property id (16), the type ('s' for string) and the value itself ('Unstable'). The value table can have multiple values with the same id - this is how multivalued properties are stored. When property vaues were retrieved in the past, the SMF backend would translate requests into an SQL statement like this 'SELECT \* FROM value_tbl WHERE value_id = 16'. Note that there is no ordering here, we just select all values with order unspecified.

Trying the same query on the upgraded repository reveals an extra column:

# lib/svc/bin/sqlite /etc/svc/repository.db
SQLite version 2.8.15-repcached-nightly
Enter ".help" for instructions
sqlite> SELECT \* FROM value_tbl LIMIT 1; 

The extra column specifies value_order for a given set of values sharing the same id - the first value is 0, the second 1, etc. When storing a set of values, the backend now specifies a value_order value for each. On retrieval, an exta clause is added to the SQL statement: 'SELECT \* FROM value_tbl WHERE value_id = 16 ORDER BY value_order'. This ensures backend retrieval is done in order. Note that this is not the whole story in getting value ordering to work - some libscf(3scf) library changes were needed too, along with some changes to svccfg(1M).

So if you are upgrading to snv_95, and you notice a message like this during boot:

svc.configd: upgrading SMF repository format...
svc.configd: SMF repository upgrade is complete.'re seeing the backend database being upgraded as the extra 'value_order' column is being added. It takes a few seconds as the version of sqlite used does not support the 'ADD COLUMN' statement - instead we create a new temporary table with the extra column, import the values from the old table, remove the old table and replace it with the new one.

So finally, what is the practical value of all this? Well, if your service needs to specify configuration data that is order-sensitive - for example a set of nameservices that should be tried in a particular order - you can now specify this using multiple property values, and be sure that you get the values back in the order in which they were set. Here's an example of setting multiple values for a string property, and seeing the order preserved:

# svccfg -s rdisc:default setprop routing/send_address = \\( \\"A\\" \\"B\\" \\"C\\" \\)
# svccfg -s rdisc:default listprop routing/send_address
routing/send_address  astring  "A" "B" "C" 

In the past, since value ordering was not supported, order-sensitive values had to be stuffed into a single string value with a character separator delimiting each value, which is a pain for consumers to handle, since they had to parse the stored value themselves. A small change, but I suspect it'll come in handy moving forward.

If you want more details, check out the bug report, the ARC case, and the design discussion on smf-discuss.




« June 2016