Schedulers

Scheduling storage one resource at a time...


Overview

This page reviews the scheduling systems supported by REX-Ray.

Docker

The majority of the documentation for the Docker integration driver has been relocated to the libStorage project.

External Access

By default, REX-Ray's embedded Docker Volume Plug-in endpoint handles requests from the local Docker service via a UNIX socket. Doing so restricts the endpoint to the localhost, increasing network security by removing a possible attack vector. If an externally accessible Docker Volume Plug-in endpoint is required, it's still possible to create one by overriding the address for the default-docker module in REX-Ray's configuration file:

rexray:
  modules:
    default-docker:
      host: tcp://:7981

The above example illustrates how to override the default-docker module's endpoint address. The value tcp://:7981 instructs the Docker Volume Plug-in to listen on port 7981 for all configured interfaces.

Using a TCP endpoint has a side-effect however -- the local Docker instance will not know about the Volume Plug-in endpoint as there is no longer a UNIX socket file in the directory the Docker service continually scans.

On the local system, and in fact on all systems where the Docker service needs to know about this externally accessible Volume Plug-in endpoint, a spec file must be created at /etc/docker/plugins/rexray.spec. Inside this file simply include a single line with the network address of the endpoint. For example:

tcp://192.168.56.20:7981

With a spec file located at /etc/docker/plugins/rexray.spec that contains the above contents, Docker instances will query the Volume Plug-in endpoint at tcp://192.168.56.20:7981 when volume requests are received.

Volume Management

The volume sub-command for Docker 1.12+ should look similar to the following:

$ docker volume

Usage:  docker volume [OPTIONS] [COMMAND]

Manage Docker volumes

Commands:
  create                   Create a volume
  inspect                  Return low-level information on a volume
  ls                       List volumes
  rm                       Remove a volume

List Volumes

The list command reviews a list of available volumes that have been discovered via Docker Volume Plug-in endpoints such as REX-Ray. Each volume name is expected to be unique. Thus volume names must also be unique across all endpoints, and in turn, across all storage platforms exposed by REX-Ray.

With the exception of the local driver, the list of returned volumes is generated by the backend storage platform to which the configured driver communicates:

$ docker volume ls
DRIVER              VOLUME NAME
local               local1
scaleio             Volume-001
virtualbox          vbox1

Inspect Volume

The inspect command can be used to retrieve details about a volume related to both Docker and the underlying storage platform. The fields listed under Status are all generated by REX-Ray, including Size in GB, Volume Type, and Availability Zone.

The Scope parameter ensures that when the specified volume driver is inspected by multiple Docker hosts, the volumes tagged as global are all interpreted as the same volume. This reduces unnecessary round-trips in situations where an application such as Docker Swarm is connected to hosts configured with REX-Ray.

$ docker volume inspect vbox1
[
    {
        "Name": "vbox1",
        "Driver": "virtualbox",
        "Mountpoint": "",
        "Status": {
            "availabilityZone": "",
            "fields": null,
            "iops": 0,
            "name": "vbox1",
            "server": "virtualbox",
            "service": "virtualbox",
            "size": 8,
            "type": ""
        },
        "Labels": {},
        "Scope": "global"
    }
]

Create Volume

Docker's volume create command enables the creation of new volumes on the underlying storage platform. Newly created volumes are available immediately to be attached and mounted. The volume create command also supports the CLI flag -o|--opt in order to support providing custom data to the volume creation workflow:

$ docker volume create --driver=virtualbox --name=vbox2 --opt=size=2
vbox2

Additional, valid options for the -o|--opt parameter include:

option description
size Size in GB
IOPS IOPS
volumeType Type of Volume or Storage Pool
volumeName Create from an existing volume name
volumeID Create from an existing volume ID
snapshotName Create from an existing snapshot name
snapshotID Create from an existing snapshot ID

Remove Volume

A volume may be removed once it is no longer in use by a container, running or otherwise. The process of removing a container actually causes the volume to be removed if that is the last container to leverage said volume:

$ docker volume rm vbox2

Containers with Volumes

Please review the Applications section for information on configuring popular applications with persistent storage via Docker and REX-Ray.

Mesos

In Mesos the frameworks are responsible for receiving requests from consumers and then proceeding to schedule and manage tasks. While some frameworks, like Marathon, are open to run any workload for sustained periods of time, others are use case specific, such as Cassandra. Frameworks may also receive requests from other platforms in addition to schedulers instead of consumers such as Cloud Foundry, Kubernetes, and Swarm.

Once a resource offer is accepted from Mesos, tasks are launched to support the associated workloads. These tasks are eventually distributed to Mesos agents in order to spin up containers.

REX-Ray enables on-demand storage allocation for agents receiving tasks via two deployment configurations:

  1. Docker Containerizer with Marathon

  2. Mesos Containerizer with Marathon

Docker Containerizer with Marathon

When the framework leverages the Docker containerizer, Docker and REX-Ray should both already be configured and working. The following example shows how to use Marathon in order to bring an application online with external volumes:

{
    "id": "nginx",
    "container": {
        "docker": {
            "image": "million12/nginx",
            "network": "BRIDGE",
            "portMappings": [{
                "containerPort": 80,
                "hostPort": 0,
                "protocol": "tcp"
            }],
            "parameters": [{
                "key": "volume-driver",
                "value": "rexray"
            }, {
                "key": "volume",
                "value": "nginx-data:/data/www"
            }]
        }
    },
    "cpus": 0.2,
    "mem": 32.0,
    "instances": 1
}

Mesos Containerizer with Marathon

Mesos 0.23+ includes modules that enable extensibility for different portions of the architecture. The dvdcli and mesos-module-dvdi projects are required to enable external volume support with the native containerizer.

The next example is similar to the one above, except in this instance the native containerizer is preferred and volume requests are handled by the env section.

{
  "id": "hello-play",
  "cmd": "while [ true ] ; do touch /var/lib/rexray/volumes/test12345/hello ; sleep 5 ; done",
  "mem": 32,
  "cpus": 0.1,
  "instances": 1,
  "env": {
    "DVDI_VOLUME_NAME": "test12345",
    "DVDI_VOLUME_DRIVER": "rexray",
    "DVDI_VOLUME_OPTS": "size=5,iops=150,volumetype=io1,newfstype=xfs,overwritefs=true"
  }
}

This example also illustrates several important settings for the native method. While the VirtualBox driver is being used, any validated storage platform should work. Additionally, there are two options recommended for this type of configuration:

Property Recommendation
libstorage.integration.volume.operations.mount.preempt Setting this flag to true ensures any host can preempt control of a volume from other hosts
libstorage.integration.volume.operations.unmount.ignoreUsedCount Enabling this flag declares that mesos-module-dvdi is the authoritative source for deciding when to unmount volumes

Please refer to the libStorage documentation for more information on Volume Configuration options.

note

The libstorage.integration.volume.operations.remove.disable property can prevent the scheduler from removing volumes. Setting this flag to true is recommended when using Mesos with Docker 1.9.1 or earlier.

libstorage:
  service: virtualbox
  integration:
    volume:
      operations:
        mount:
          preempt: true
        unmount:
          ignoreusedcount: true
        remove:
          disable: true
virtualbox:
  volumePath: $HOME/VirtualBox/Volumes