Integrate with Nutanix Storage

Nutanix Objects in Kubeflow Pipeline

You can use standard s3 boto api to upload and download objects from a Kubeflow Pipeline. See Nutanix Objects Docs for more details on creating object store and buckets.

import boto3

bucket_name="ml-pipeline-storage"
object_name="models"
object_store_access_key_id="<key_id>"
object_store_secret_access_key="<access_key>"
host="http://<Nutanix Objects Store Domain Name>"
region_name='us-west-1'
s3_client = boto3.client(
                 's3',
                 endpoint_url=host,
                 aws_access_key_id=object_store_access_key_id,
                 aws_secret_access_key=object_store_secret_access_key,
                 region_name=region_name,
                 verify=False)
response = s3_client.upload_file(f'./test_upload_data.txt', bucket_name, object_name)

Nutanix Volumes in Kubeflow Pipeline

Nutanix volumes are created with the default storage class configured in the NKE cluster. See Default Storage Class of Nutanix Kubernetes Engine for more details about creating storage classes.

vop = dsl.VolumeOp(
   name="Create a volume to share data between stages on Nutanix Volumes",
   resource_name="data-volume",
   size="1Gi",
   modes=dsl.VOLUME_MODE_RWO)

Nutanix Files in Kubeflow Pipeline

Create a storage class to dynamically provision Nutanix File shares. See Files Storage Class of Nutanix Kubernetes Engine for more details on creating storage classes for dynamic NFS Share provisioning with Nutanix Files. Once storage class is setup, you can use VolumeOp operation to create volume on Nutanix Files.

vop = dsl.VolumeOp(
   name="Create a volume to share data between stages on Nutanix Files",
   resource_name="data-volume",
   size="1Gi",
   modes=dsl.VOLUME_MODE_RWM,
   storage_class="files-sc")

Using Nutanix Objects as an artifact store

In order to use Nutanix Objects as an underlying artifact store, install the kubeflow with Nutanix object store.

To verify this is working correctly, you can check Nutanix Objects browser to see if your artifacts are created and show up inside your buckets.