Attaching virtual persistent memory to guests¶

Dependencies¶

The following are required to support the vPMEM feature:

• Persistent Memory Hardware

  One such product is Intel® Optane™ DC Persistent Memory. ipmctl is used to configure it.

• Linux Kernel version >= 4.18 with the following modules loaded:

  dax_pmem, nd_pmem, device_dax, nd_btt

• QEMU version >= 3.1.0

• Libvirt version >= 5.0.0

• ndctl version >= 62

• daxio version >= 1.6

The vPMEM feature has been verified under the software and hardware listed above.

Configure PMEM namespaces (Compute)¶

1. Create PMEM namespaces as vPMEM backends using the ndctl utility.

   For example, to create a 30GiB namespace named ns3:

   $ sudo ndctl create-namespace -s 30G -m devdax -M mem -n ns3
   {
     "dev":"namespace1.0",
     "mode":"devdax",
     "map":"mem",
     "size":"30.00 GiB (32.21 GB)",
     "uuid":"937e9269-512b-4f65-9ac6-b74b61075c11",
     "raw_uuid":"17760832-a062-4aef-9d3b-95ea32038066",
     "daxregion":{
       "id":1,
       "size":"30.00 GiB (32.21 GB)",
       "align":2097152,
       "devices":[
       {
         "chardev":"dax1.0",
         "size":"30.00 GiB (32.21 GB)"
       }
       ]
     },
     "name":"ns3",
     "numa_node":1
   }
   

   Then list the available PMEM namespaces on the host:

   $ ndctl list -X
   [
     {
       ...
       "size":6440353792,
       ...
       "name":"ns0",
       ...
     },
     {
       ...
       "size":6440353792,
       ...
       "name":"ns1",
       ...
     },
     {
       ...
       "size":6440353792,
       ...
       "name":"ns2",
       ...
     },
     {
       ...
       "size":32210157568,
       ...
       "name":"ns3",
       ...
     }
   ]
   

2. Specify which PMEM namespaces should be available to instances.

   Edit libvirt.pmem_namespaces:

   [libvirt]
   # pmem_namespaces=$LABEL:$NSNAME[|$NSNAME][,$LABEL:$NSNAME[|$NSNAME]]
   pmem_namespaces = 6GB:ns0|ns1|ns2,LARGE:ns3
   

   Configured PMEM namespaces must have already been created on the host as described above. The conf syntax allows the admin to associate one or more namespace $NSNAMEs with an arbitrary $LABEL that can subsequently be used in a flavor to request one of those namespaces. It is recommended, but not required, for namespaces under a single $LABEL to be the same size.

3. Restart the nova-compute service.

   Nova will invoke ndctl to identify the configured PMEM namespaces, and report vPMEM resources to placement.

Configure a flavor¶

Specify a comma-separated list of the $LABELs from libvirt.pmem_namespaces to the flavor’s hw:pmem property. Note that multiple instances of the same label are permitted:

$ openstack flavor set --property hw:pmem='6GB' my_flavor
$ openstack flavor set --property hw:pmem='6GB,LARGE' my_flavor_large
$ openstack flavor set --property hw:pmem='6GB,6GB' m1.medium

Based on the above examples, an openstack server create request with my_flavor_large will spawn an instance with two vPMEMs. One, corresponding to the LARGE label, will be ns3; the other, corresponding to the 6G label, will be arbitrarily chosen from ns0, ns1, or ns2.

Verify inventories and allocations¶

This section describes how to check that:

• vPMEM inventories were created correctly in placement, validating the configuration described above.

• allocations were created correctly in placement for instances spawned from flavors configured with vPMEMs.

1. Get the list of resource providers

   $ openstack resource provider list
   +--------------------------------------+--------+------------+
   | uuid                                 | name   | generation |
   +--------------------------------------+--------+------------+
   | 1bc545f9-891f-4930-ab2b-88a56078f4be | host-1 |         47 |
   | 7d994aef-680d-43d4-9325-a67c807e648e | host-2 |         67 |
   --------------------------------------+---------+------------+
   

2. Check the inventory of each resource provider to see resource classes

   Each $LABEL configured in libvirt.pmem_namespaces is used to generate a resource class named CUSTOM_PMEM_NAMESPACE_$LABEL. Nova will report to Placement the number of vPMEM namespaces configured for each $LABEL. For example, assuming host-1 was configured as described above:

   $ openstack resource provider inventory list 1bc545f9-891f-4930-ab2b-88a56078f4be
   +-----------------------------+------------------+----------+----------+-----------+----------+--------+
   | resource_class              | allocation_ratio | max_unit | reserved | step_size | min_unit |  total |
   +-----------------------------+------------------+----------+----------+-----------+----------+--------+
   | VCPU                        |             16.0 |       64 |        0 |         1 |        1 |     64 |
   | MEMORY_MB                   |              1.5 |   190604 |      512 |         1 |        1 | 190604 |
   | CUSTOM_PMEM_NAMESPACE_LARGE |              1.0 |        1 |        0 |         1 |        1 |      1 |
   | CUSTOM_PMEM_NAMESPACE_6GB   |              1.0 |        3 |        0 |         1 |        1 |      3 |
   | DISK_GB                     |              1.0 |      439 |        0 |         1 |        1 |    439 |
   +-----------------------------+------------------+----------+----------+-----------+----------+--------+
   

   Here you can see the vPMEM resource classes prefixed with CUSTOM_PMEM_NAMESPACE_. The LARGE label was configured with one namespace (ns3), so it has an inventory of 1. Since the 6GB label was configured with three namespaces (ns0, ns1, and ns2), the CUSTOM_PMEM_NAMESPACE_6GB inventory has a total and max_unit of 3.

3. Check allocations for each server that is using vPMEMs

   $ openstack server list
   +--------------------------------------+----------------------+--------+-------------------+---------------+-----------------+
   | ID                                   | Name                 | Status | Networks          | Image         | Flavor          |
   +--------------------------------------+----------------------+--------+-------------------+---------------+-----------------+
   | 41d3e139-de5c-40fd-9d82-016b72f2ba1d | server-with-2-vpmems | ACTIVE | private=10.0.0.24 | ubuntu-bionic | my_flavor_large |
   | a616a7f6-b285-4adf-a885-dd8426dd9e6a | server-with-1-vpmem  | ACTIVE | private=10.0.0.13 | ubuntu-bionic | my_flavor       |
   +--------------------------------------+----------------------+--------+-------------------+---------------+-----------------+
   
   $ openstack resource provider allocation show 41d3e139-de5c-40fd-9d82-016b72f2ba1d
   +--------------------------------------+------------+------------------------------------------------------------------------------------------------------------------------+
   | resource_provider                    | generation | resources                                                                                                              |
   +--------------------------------------+------------+------------------------------------------------------------------------------------------------------------------------+
   | 1bc545f9-891f-4930-ab2b-88a56078f4be |         49 | {u'MEMORY_MB': 32768, u'VCPU': 16, u'DISK_GB': 20, u'CUSTOM_PMEM_NAMESPACE_6GB': 1, u'CUSTOM_PMEM_NAMESPACE_LARGE': 1} |
   +--------------------------------------+------------+------------------------------------------------------------------------------------------------------------------------+
   
   $ openstack resource provider allocation show a616a7f6-b285-4adf-a885-dd8426dd9e6a
   +--------------------------------------+------------+-----------------------------------------------------------------------------------+
   | resource_provider                    | generation | resources                                                                         |
   +--------------------------------------+------------+-----------------------------------------------------------------------------------+
   | 1bc545f9-891f-4930-ab2b-88a56078f4be |         49 | {u'MEMORY_MB': 8192, u'VCPU': 8, u'DISK_GB': 20, u'CUSTOM_PMEM_NAMESPACE_6GB': 1} |
   +--------------------------------------+------------+-----------------------------------------------------------------------------------+
   

   In this example, two servers were created. server-with-2-vpmems used my_flavor_large asking for one 6GB vPMEM and one LARGE vPMEM. server-with-1-vpmem used my_flavor asking for a single 6GB vPMEM.