azurebatch Configuration

To be used with the azurebatch profile by specifying the -profile azurebatch when running nf-core pipelines. Custom queue and storage need to be supplied with params.az_location, params.batch_name, params.batch_key, params.storage_name, params.sas_token.

Required Parameters

--storage_name

Name of Azure blob storage account.

--storage_key

Access key to Azure blob storage account. --storage_key or --storage_sas are required, but not both.

--storage_sas

SAS for access to Azure blob storage account (see relevant permissions on Nextflow documentation). --storage_key or --storage_sas are required, but not both.

--az_location

The Azure Batch region where the computation is executed in VMs. Default (westus2).

--batch_name

The Azure Batch account name.

--batch_key

The Azure Batch account key.

-w

The Azure Blob container to be used as Nextflow work directory (-w az://work).

--vm_type

VM size to use with Nextflow autopool or when creating a worker pool in Azure Batch. Make sure your Azure account has sufficient quota. Defaults to Standard_D8s_v3. See Azure VM Size documentation for more information.

--autopoolmode

Whether to use Nextflow autopool mode which creates an autoscaling pool for running Nextflow jobs. Defaults to true.

--allowpoolcreation

Allow Nextflow to create a pool for running Nextflow jobs. Defaults to true.

--deletejobs

Allow Nextflow to delete pools after completion. Defaults to true.

--az_worker_pool

Select an existing pool by name to run Nextflow jobs on. Defaults to auto (matching --autopoolmode).

--acr_registry

URL to Azure container registry for private docker images.

--acr_username

Username to access private Azure container registry.

--acr_password

Password to access private Azure container registry.

Azure Batch Setup

Please refer to the Nextflow documentation which describe how to setup the Azure Batch environment.

Config file

See config file on GitHub

azurebatch.config
//Nextflow config file for running on Azure batch
params {
    config_profile_description = 'Azure BATCH Cloud Profile'
    config_profile_contact     = 'Venkat Malladi (@vsmalladi) & Adam Talbot (@adamrtalbot)'
    config_profile_url         = 'https://azure.microsoft.com/services/batch/'
 
    // Storage
    storage_name               = null
    storage_key                = null
    storage_sas                = null
 
    // Batch
    az_location                = "westus2"
    batch_name                 = null
    batch_key                  = null
 
    vm_type                    = "Standard_D8s_v3"
    autopoolmode               = true
    allowpoolcreation          = true
    deletejobs                 = true
    deletepools                = true
    az_worker_pool             = "auto"
 
    // ACR
    acr_registry               = null
    acr_username               = null
    acr_password               = null
}
 
process {
    executor = "azurebatch"
}
 
azure {
    process {
        queue = params.az_worker_pool
    }
    storage {
        accountName = params.storage_name
        accountKey  = params.storage_key
        sasToken    = params.storage_sas
    }
    batch {
        location                = params.az_location
        accountName             = params.batch_name
        accountKey              = params.batch_key
        tokenDuration           = "24h"
        autoPoolMode            = params.autopoolmode
        allowPoolCreation       = params.allowpoolcreation
        deleteJobsOnCompletion  = params.deletejobs
        deletePoolsOnCompletion = params.deletepools
        pools {
            auto {
                vmType     = params.vm_type
                autoScale  = true
                vmCount    = 1
                maxVmCount = 12
            }
        }
    }
    registry {
        server   = params.acr_registry
        userName = params.acr_username
        password = params.acr_password
    }
}