Page tree
Skip to end of metadata
Go to start of metadata

DAOS can now be configured using YAML files, this enables daos_server to derive environment variables and commandline options needed to pass to daos_io_server instances (forked within daos_server process) from the daos_server.yml config file.

This should eventually result in more efficient usage and remove the need to set environment variables in the users shell prior to starting DAOS servers, specifying all configuration details in a single file.

Points of note:

  • Configuration File Examples can be found in the DAOS repo (example configurations and default config in daos/utils/config/)
  • server config is read from daos/install/etc/daos_server.yml by default, config path set with "-odaos_server commandline option
  • only server side config files are supported for now, any client tools e.g. dmg will need environment variables set in shell
  • daos_server will attempt to run with parameters specified in OS env vars in case of no/incomplete config
  • command-line options passed to daos_server take precedence over configuration file parameters
  • If user shell executing daos_server has environment variable CRT_PHY_ADDR_STR set, user os environment will be used when spawning daos_io_server instances. In this situation a "Warning: using os env vars..." message will be printed to the console and no environment variables will be used as specified in the env_vars list within the per-server section of the server config file. This behaviour provides backward compatibility with historic mechanism of specifying all parameters through environment variables.
  • the active configuration that is generated by the control plane and used to start IO server(s) is written to a temporary file (location should be displayed on daos_server stdout. Example: 

    2018/11/30 10:35:29 Active config could not be saved (open /home/tanabarr/projects/daos_m/install/etc/.daos_server.active.yml: permission denied)
    2018/11/30 10:35:29 Active config saved to /tmp/.daos_server.active.yml (read-only)


Step-by-step guide (with config)


  1. Create SCM Mount at the location specified in the config:

    [root@boro-45 d]# mkdir -p /mnt/daos/1
    [root@boro-45 d]# mount -t tmpfs -o size=68719476736 tmpfs /mnt/daos/1
  2. Run daos_server with specified configuration file path (dependency on orterun is temporary should go away soon):

    [root@boro-45 daos_m]# install/bin/orterun -np 1 --hostfile hostfile --enable-recovery --report-uri /tmp/urifile --allow-run-as-root install/bin/daos_server -c 1 -o $(pwd)/utils/config/examples/daos_server_sockets.yml
    2018/12/04 15:48:46 DAOS config read from /home/tanabarr/projects/daos_m/utils/config/examples/daos_server_sockets.yml
    2018/12/04 15:48:46 Warning: active config could not be saved (open /home/tanabarr/projects/daos_m/utils/config/examples/.daos_server.active.yml: permission denied)
    2018/12/04 15:48:46 os env detected ABT_MAX_NUM_XSTREAMS=64, skipping...
    2018/12/04 15:48:46 os env detected CRT_PHY_ADDR_STR=ofi+sockets, skipping...
    2018/12/04 15:48:46 os env detected OFI_INTERFACE=ib0, skipping...
    2018/12/04 15:48:46 DAOS server listening on 0.0.0.0:10001
    DAOS I/O server (v0.0.2) process 79605 started on rank 0 (out of 1) with 1 xstream(s)

Step-by-step guide (without config)

  1. Create SCM Mount at default assumed location:

    [root@boro-45 d]# mkdir -p /mnt/daos
    [root@boro-45 d]# mount -t tmpfs -o size=68719476736 tmpfs /mnt/daos
  2. Run daos_server (dependency on orterun is temporary should go away soon), parameters are used from commandline and OS env vars:

    [root@boro-45 daos_m]# install/bin/orterun -np 1 --hostfile hostfile --enable-recovery --report-uri /tmp/urifile --allow-run-as-root install/bin/daos_server -c 1 -s /mnt/daos/1
    2018/12/04 16:01:36 Configuration could not be read (open /home/tanabarr/projects/daos_m/install/etc/daos_server.yml: no such file or directory), proceeding without
    2018/12/04 16:01:36 Warning: incomplete config file, specify params in config as per /home/tanabarr/projects/daos_m/install/utils/config/examples
    2018/12/04 16:01:36 Warning: active config could not be saved (open /home/tanabarr/projects/daos_m/install/etc/.daos_server.active.yml: permission denied)
    2018/12/04 16:01:36 DAOS server listening on 0.0.0.0:10000 as access point (bootstrap)
    DAOS I/O server (v0.0.2) process 80346 started on rank 0 (out of 1) with 1 xstream(s)

Step-by-step guide (with bdev_class=nvme; bdev_list=["0000:81:00.0"] in config)

  1. Create SCM Mount at default assumed location:

    [root@boro-45 d]# mkdir -p /mnt/daos
    [root@boro-45 d]# mount -t tmpfs -o size=68719476736 tmpfs /mnt/daos
  2. Run daos_server (dependency on orterun is temporary should go away soon), parameters are used from config file:

    [root@boro-45 daos_m]# install/bin/orterun -np 1 --hostfile hostfile --enable-recovery --report-uri /tmp/urifile --allow-run-as-root install/bin/daos_server -c 1 -o /home/tanabarr/projects/daos_m/utils/config/examples/daos_server_sockets.yml
    2018/12/10 14:23:27 DAOS config read from /home/tanabarr/projects/daos_m/utils/config/examples/daos_server_sockets.yml
    2018/12/10 14:23:27 Warning: active config could not be saved (open /home/tanabarr/projects/daos_m/utils/config/examples/.daos_server.active.yml: permission denied)
    2018/12/10 14:23:27 Active config saved to /tmp/.daos_server.active.yml (read-only)
    2018/12/10 14:23:27 DAOS server listening on 0.0.0.0:10001
    Starting SPDK v18.07-pre / DPDK 18.02.0 initialization...
    [ DPDK EAL parameters: daos -c 0x1 --file-prefix=spdk_pid154066 ]
    EAL: Detected 72 lcore(s)
    EAL: No free hugepages reported in hugepages-1048576kB
    EAL: Multi-process socket /var/run/.spdk_pid154066_unix
    EAL: Probing VFIO support...
    EAL: PCI device 0000:81:00.0 on NUMA socket 1
    EAL:   probe driver: 8086:2701 spdk_nvme
    DAOS I/O server (v0.0.2) process 154066 started on rank 0 (out of 1) with 1 xstream(s)
  • No labels