Schedulers

Scheduling storage one resource at a time...


Overview

This page reviews the scheduling systems supported by REX-Ray.

Docker

REX-Ray has a Docker Volume Driver which is compatible with 1.7+.

It is suggested that you are running Docker 1.10.2+ with REX-Ray especially if you are sharing volumes between containers, or you want interactive volume commands through docker volume.

Example Configuration

Below is an example config.yml that can be used. The volume.mount.preempt is an optional parameter here which enables any host to take control of a volume irrespective of whether other hosts are using the volume. If this is set to false then plugins should ensure safety first by locking the volume from to the current owner host. We also specify docker.size which will create all new volumes at the specified size in GB.

rexray:
  storageDrivers:
  - virtualbox
  volume:
    mount:
      preempt: true
docker:
  size: 1
virtualbox:
  endpoint: http://yourlaptop:18083
  volumePath: /Users/youruser/VirtualBox Volumes
  controllerName: SATA

Extra Global Parameters

These are all valid parameters that can be configured for the service.

parameter description
docker.size Size in GB
docker.iops IOPS
docker.volumeType Type of Volume or Storage Pool
docker.fsType Type of filesystem for new volumes (ext4/xfs)
docker.availabilityZone Extensible parameter per storage driver
linux.volume.rootPath The path within the volume to private mount (/data)
rexray.volume.mount.preempt Forcefully take control of volumes when requested

Starting Volume Driver

REX-Ray must be running as a service to serve requests from Docker. This can be done by running rexray start. Make sure you restart REX-Ray if you make configuration changes.

$ sudo rexray start
Starting REX-Ray...SUCESS!

  The REX-Ray daemon is now running at PID 18141. To
  shutdown the daemon execute the following command:

    sudo rexray stop

Following this you can now leverage volumes with Docker.

Creating and Using Volumes

There are two ways to interact with volumes. You can use the docker run command in combination with --volume-driver for new volumes, or specify -v volumeName by itself for existing volumes. The --volumes-from will also work when sharing existing volumes with a new container.

The docker volume sub-command enables complete management to create, remove, and list existing volumes. All volumes are returned from the underlying storage platform.

  1. Run containers with volumes (1.7+)

    docker run -ti --volume-driver=rexray -v test:/test busybox
    
  2. Create volume with options (1.8+)

    docker volume create --driver=rexray --opt=size=5 --name=test
    

Extra Volume Create Options

option description
size Size in GB
IOPS IOPS
volumeType Type of Volume or Storage Pool
volumeName Create from an existing volume name
volumeID Creat from an existing volume ID
snapshotName Create from an existing snapshot name
snapshotID Create from an existing snapshot ID

Caveats

If you restart the REX-Ray instance while volumes are shared between Docker containers then problems may arise when stopping one of the containers sharing the volume. It is suggested that you avoid stopping these containers at this point until all containers sharing the volumes can be stopped. This will enable the unmount process to proceed cleanly.

Mesos

In Mesos the frameworks are responsible for receiving requests from consumers and then proceeding to schedule and manage tasks. Some frameworks are open to run any workload for sustained periods of time (ie. Marathon), and others are use case specific (ie. Cassandra). Further than this, frameworks can receive requests from other platforms or schedulers instead of consumers such as Cloud Foundry, Kubernetes, and Swarm.

Once frameworks decide to accept resource offers from Mesos, tasks are launched to support workloads. These tasks eventually make it down to Mesos agents to spin up containers.

REX-Ray provides the ability for any agent receiving a task to request storage be orchestrated for that task.

There are two primary methods that REX-Ray functions with Mesos. It is up to the framework to determine which is most appropriate. Mesos (0.26) has two containerizer options for tasks, Docker and Mesos.

Docker Containerizer with Marathon

If the framework uses the Docker containerizer, it is required that both Docker and REX-Ray are configured ahead of time and working. It is best to refer to the Docker page for more information. Once this is configured across all appropriate agents, the following is an example of using Marathon to start an application with external volumes.

{
    "id": "nginx",
    "container": {
        "docker": {
            "image": "million12/nginx",
            "network": "BRIDGE",
            "portMappings": [{
                "containerPort": 80,
                "hostPort": 0,
                "protocol": "tcp"
            }],
            "parameters": [{
                "key": "volume-driver",
                "value": "rexray"
            }, {
                "key": "volume",
                "value": "nginx-data:/data/www"
            }]
        }
    },
    "cpus": 0.2,
    "mem": 32.0,
    "instances": 1
}

Mesos Containerizer with Marathon

Mesos 0.23+ includes modules that enables extensibility for different portions the architecture. The dvdcli and mesos-module-dvdi projects are required for this method to enable external volume support with the native containerizer.

The following is a similar example to the one above. But here we are specifying to use the the native containerizer and requesting volumes through The env section.

{
  "id": "hello-play",
  "cmd": "while [ true ] ; do touch /var/lib/rexray/volumes/test12345/hello ; sleep 5 ; done",
  "mem": 32,
  "cpus": 0.1,
  "instances": 1,
  "env": {
    "DVDI_VOLUME_NAME": "test12345",
    "DVDI_VOLUME_DRIVER": "rexray",
    "DVDI_VOLUME_OPTS": "size=5,iops=150,volumetype=io1,newfstype=xfs,overwritefs=true"
  }
}

This example also comes along with a couple of important settings for the native method. This is a config.yml file that can be used. In this case we are showing a virtualbox driver configuration, but you can use anything here.
We suggest two optional options for the mesos-module-dvdi. Setting the volume.mount.preempt flag ensures any host can preempt control of a volume from other hosts. Refer to the User-Guide for more information on preempt. The volume.unmount.ignoreusedcount ensures that mesos-module-dvdi is authoritative when it comes to deciding when to unmount volumes.

rexray:
  storageDrivers:
  - virtualbox
  volume:
    mount:
      preempt: true
    unmount:
      ignoreusedcount: true
virtualbox:
  endpoint: http://yourlaptop:18083
  volumePath: /Users/youruser/VirtualBox Volumes
  controllerName: SATA