config/SPEC-yaml.md

HOCON (Human-Optimized Config Object Notation)

Very informal spec for a HOCON based on YAML.

In this, "application" really could mean "library," it means "a thing defining a configuration, such as Akka"

Some Java-specific stuff is in here, though a Java-independent version would also be possible, without the system properties parts.

Goals

The primary goal is: keep the semantics (tree structure; set of types; encoding/escaping) from JSON, but make it more convenient as a human-editable config file format.

The following features are provided by YAML:

• less noisy / less pedantic syntax
• ability to refer to another part of the configuration (set a value to another value)
• ability to write comments

The following features are defined on top of YAML:

• import/include another configuration file into the current file
• a mapping to a flat properties hierarchy such as Java's System.properties
• ability to get values from environment variables

The first implementation should have these properties:

• pure Java with no external dependencies other than YAML parser
• API easily supports "nesting" (get a subtree then treat it as a root)
• API supports a "root" name which is used to scope Java properties. So if you say your config is for "akka" then your config key "foo" would go with -Dakka.foo=10, but in the config file people don't have to write the root akka object.
• application can define the search path for include statements (with a function from string to input stream)
• API supports treating values as "Durations" and "Memory sizes" in addition to basic JSON types (for example "50ms" or "128M")
• API should attempt to perform reasonable type conversions. This might include: treating numbers 0 or 1 as booleans, treating strings yes/no/y/n as booleans, integer to and from float, any number or boolean to a string. This puts "figure out what people mean" in the API rather than in the syntax spec. (Note: this is pretty much how YAML works anyway; the types are not always apparent from the syntax as they are in JSON.)

YAML background information

The Wikipedia article about YAML is quite good as a brief introduction: http://en.wikipedia.org/wiki/YAML

YAML 1.2 is a minor revision of 1.1 which makes JSON a subset of YAML. With 1.1, "most" JSON is valid YAML, while with 1.2 all JSON is valid YAML.

The official YAML site is: http://yaml.org/

Java YAML parsers

There is a good discussion at: http://stackoverflow.com/questions/450399/which-java-yaml-library-should-i-use

There appear to be only two options:

• SnakeYAML http://code.google.com/p/snakeyaml/
• YamlBeans http://code.google.com/p/yamlbeans/

The SnakeYAML maintainer expressed some interest in a Scala wrapper: http://www.scala-lang.org/node/5497 There's also an apparently-unmaintained wrapper HelicalYAML: http://code.google.com/p/helicalyaml/

According to the .pom, all SnakeYAML deps are in test scope, so no dependencies at runtime: http://repo2.maven.org/maven2/org/yaml/snakeyaml/1.9/snakeyaml-1.9.pom

Both SnakeYAML and YamlBeans are YAML 1.1, so some JSON may not parse.

SnakeYAML appears to be more popular and more flexible, while YamlBeans is the simplest API that could possibly work to serialize and deserialize beans.

Java properties mapping

See the Java properties spec here: http://download.oracle.com/javase/7/docs/api/java/util/Properties.html#load%28java.io.Reader%29

There is a mapping from Java properties to HOCON, foo.bar.baz=10 to foo={ bar={ baz=10 } }. If an HOCON key has a . in it then there is no way to refer to it as a Java property; it is not recommended to name HOCON keys with a . in them.

For an application's config, Java System properties override HOCON found in the configuration file. This supports specifying config options on the command line.

When loading a configuration, all System properties should be merged in.

Generally an application's configuration should be under a root namespace, to avoid merging every system property in the whole process.

For example, say your config is for "akka" then your config key "foo" would go with -Dakka.foo=10. When loading your config, any system properties starting with akka. would be merged into the config.

System properties always have string values. No substitution or unescaping is performed on the value.

New YAML tags

To achieve HOCON on top of YAML, we assume the JSON schema (http://www.yaml.org/spec/1.2/spec.html#id2803231) and then add a couple more tags for includes and substitutions.

Tags

In YAML, standardized data types are tagged with the !! prefix:

d: !!float 123

The !!float is an explicit type tag; without it 123 would be an integer instead.

Applications can define their own types with a single exclamation point prefix:

d: !mytype 123

Substitution

The substitution type is tagged with !subst and will have expressions like ${some.path} in the value.

Substitutions are a way of referring to other parts of the configuration tree.

If the entire substitution value is a ${} expression, the substitution is interpreted as the type of the original configuration subtree being substituted.

If there are any characters outside of a single ${} the substitution is interpreted as a string, with substrings of the form ${some.path} replaced. There is no way to escape literal ${ in a !subst value; you could do it by having another value containing the ${ and substituting that in, if you ever needed to.

Substitution processing is performed as the last parsing step, so a substitution can look forward in the configuration file and even retrieve a value from a Java System property. This also means that a substitution will evaluate to the last-assigned value for a given option (since identical keys later in the stream override those earlier in the stream).

Circular substitutions are an error, implementations might try to detect them in a nicer way than stack overflow, for bonus points.

A substitution is replaced with any value type (number, object, string, array, true, false, null). If the substitution is the only part of a value, then the type is preserved. If the substitution is part of a string (needs to be concatenated) then it is an error if the substituted value is an object or array. Otherwise the value is converted to a string value as follows:

• null is converted to an empty string, not the string null
• strings are already strings
• numbers are converted to a string that would parse as a valid number in HOCON
• booleans are converted to true or false

Substitutions are looked up in the root object, using the regular key syntax, so foo.bar is split to be "key bar inside the object at key foo", but "foo.bar" is not split because it's quoted.

Recall that the root object already has system properties merged in as overrides. So looking up foo.bar in root object akka would get the system property akka.foo.bar if that system property were present.

Substitutions are looked up in three namespaces, in order:

• the application's normal root object.
• System properties directly, without the root namespace. So ${user.home} would first look for a user.home in the root configuration (which has a scoped system property like akka.user.home merged in!) and if that failed, it would look at the system property user.home without the akka. prefix.
  • the intent is to allow using generic system properties like user.home and also to allow overriding those per-app.
  • the intent is NOT to allow accessing config from other apps, only to allow access to global system properties
• system environment variables

If a substitution is not resolved, it evaluates to JSON value null, which would then be converted to an empty string if the substitution is part of a string.

If a substitution evaluates to null (which may mean it was either unset, or explicitly set to null), then environment variables should be used as a fallback.

Note that environment variables and global system properties are fallbacks, while app-scoped system properties are an override.

It's recommended that HOCON keys always use lowercase, because environment variables generally are capitalized. This avoids naming collisions between environment variables and configuration properties. (While on Windows getenv() is generally not case-sensitive, the lookup will be case sensitive all the way until the env variable fallback lookup is reached.)

An application can explicitly block looking up a substitution in the environment by setting a non-null value in the configuration, with the same name as the environment variable. But there is no way to set a key to null if a non-empty environment variable is present.

Environment variables are interpreted as follows:

• present and set to empty string: treated as not present
• System.getenv throws SecurityException: treated as not present
• encoding is handled by Java (System.getenv already returns a Unicode string)
• Always result in a string value.

YAML references vs. substitutions

YAML has a built-in mechanism using node anchors and references to avoid duplication, see http://en.wikipedia.org/wiki/YAML#References

Substitutions are different because they don't require premeditation, so you can refer to system properties or environment variables or the default config, even though those things may not have node anchors.

Includes

• The type tag !include means to interpret the value as a filename and merge the object defined in that file into the current object, overriding any keys defined earlier and being overridden by any keys defined later. The search path for the file (which may include the classpath or certain directories in the filesystem) will be application-defined.
• An included filename can refer to any of a HOCON file or a Java properties file. These are distinguished by extension:
  • .properties for Java properties
  • .yml for HOCON If the included filename has no extension, then any of the above extensions are allowed. If the included filename has an extension already then it refers to precisely that filename and the format is not flexible.

Duplicate Keys

YAML does not allow duplicate keys, so specifying the same key twice is an error; it is not allowed to "override" a previous key.

However, the !include mechanism allows overriding keys because the included file may specify keys that appear in the parent document. The included keys will override those earlier in the parent document, and be overridden by those later in the parent document.

FIXME: need to see if SnakeYAML actually allows this (is it possible to get keys in the order they appeared, or is it only possible to get a Map).

Examples

To get this JSON:

{
    "foo" : {
        "bar" : 10,
        "baz" : 12
    }
}

You could write the JSON itself, or condense it to:

foo:
   bar: 10
   baz: 12

You can also mix the syntaxes, like:

foo: { bar: 10, baz: 12 }

The colon can't have spaces before it unless you quote the key, so foo: not foo :.

To do an include, you write:

foo:
   bar: !include something.yml

To do a substitution, you write:

foo:
   bar: !subst ${user.home}/.myconf