Overview

The .takeoff/config.yaml is the place put any variables that should not be hardcoded in python. Then you can use these variables in your component to authenticate against Cloud services.

By default, Takeoff assumes the configuration is available in the `.takeoff` directory. You can configure this with the optional CLI parameter: `--takeoff_dir my_path`. For example: `takeoff --takeoff_dir my_path` will read both of Takeoff's yaml files from the `my_path` directory

The basic setup contains:

environment_keys:
common:
plugins:
ci_environment_keys_dev:
ci_environment_keys_tst:
ci_environment_keys_acp:
ci_environment_keys_prd:
azure:

field	description	more info
`environment_keys`	Mandatory Takeoff variables	Jump to values
`common` [optional]	Evironment agnostic variables	Jump to values
`plugins` [optional]	List of paths to Takeoff plugins	Read more
`ci_environment_keys_dev` [optional]	Environment variables for your development environment	Jump to values
`ci_environment_keys_tst` [optional]	Environment variables for your test environment	Jump to values
`ci_environment_keys_acp` [optional]	Environment variables for your acceptance environment	Jump to values
`ci_environment_keys_prd` [optional]	Environment variables for your production environment	Jump to values
`azure` [optional]	Microsoft Azure specific values	Jump to values

environment_keys

The mandatory fields are

environment_keys:
  application_name: "CI_PROJECT_NAME"
  branch_name: "CI_BRANCH_NAME"

field	description
`application_name`	A CI environment variable containing your application name, advised is to use your repository name. This variable should be available on all CI providers
`branch_name`	A CI environment variable containing the current branch name

common

The optional fields are

common:
  databricks_fs_libraries_mount_path: "dbfs:/mnt/libraries"

field	description
`databricks_fs_libraries_mount_path`	Path on `dbfs` where libraries (such as wheels and jars) are stored. Usually this is a mounted cloud storage path.

ci_environment_keys

These keys are meant to authenticate to cloud vaults using service accounts. Currently supported values are Azure service principals.

The optional fields are

ci_environment_keys_*:
  service_principal: 
    tenant: "AZURE_TENANT_ID"
    client_id: "AZURE_SP_USERNAME"
    secret: "AZURE_SP_PASSWORD"

field	description
`service_principal`	Values used to authenticate against an Azure Service Principal
`service_principal.tenant`	CI environment variable containing the Azure Tenant ID (usually a UUID)
`service_principal.client_id`	CI environment variable containing the client id (usually a UUID)
`service_principal.secret`	CI environment variable containing the secret

Example

Assume you have two enviroments, a dev and prd. The following snippet could be part of your .takeoff/config.yml

ci_environment_keys_dev:
  service_principal: 
    tenant: "AZURE_TENANT_ID"
    client_id: "AZURE_SP_USERNAME_DEV"
    secret: "AZURE_SP_PASSWORD_DEV"
ci_environment_keys_prd:
  service_principal: 
    tenant: "AZURE_TENANT_ID"
    client_id: "AZURE_SP_USERNAME_PRD"
    secret: "AZURE_SP_PASSWORD_PRD"

azure

This section contains all specific values for deploying to Azure services.

azure:
  resource_group_naming: "rg{env}"
  keyvault_naming: "https://keyvault{env}.vault.azure.net"
  location: "west europe"
  keyvault_keys: 
  common: 

field	description	more info
`resource_group_naming`	Naming convention for Azure resource groups, must contain `{env}`	See deployment environments for more info
`keyvault_naming`	Naming convention for Azure Keyvaults, must contain `{env}`	See deployment environments for more info
`location` [optional]	Location of your Azure Data Center
`keyvault_keys` [optional]	Names of keys in the Azure KeyVault containing values for other Azure services	Jump to values
`common` [optional]	Names of common Azure names	Jump to values

azure-keyvault_keys

Every key contains a mapping of credential parameters to KeyVault keys. Possible values are:

azure:
  keyvault_keys: 
    active_directory_user:
      username: "azure-username"
      password: "azure-password"
    databricks:
      host: "azure-databricks-host"
      token: "azure-databricks-token"
    container_registry:
      username: "registry-username"
      password: "registry-password"
      registry: "registry-server"
    storage_account:
      account_name: "azure-shared-blob-username"
      account_key: "azure-shared-blob-password"
    artifact_store:
      repository_url: "artifact-store-upload-url"
      username: "artifact-store-username"
      password: "artifact-store-password"
    subscription_id: "subscription-id"

Concretely this means that, for example azure-username and azure-password must be valid secret keys in your Azure KeyVault.

field	description	more info
`active_directory_user` [optional]	A registered AAD user, used for application to application authentication
`active_directory_user.username`	Username of the AAD user
`active_directory_user.password`	Password of the AAD user
`databricks` [optional]	A registered AAD user, used for application to application authentication
`databricks.host`	Host of the Databricks (for “West Europe” this is `https://westeurope.azuredatabricks.net/`)
`databricks.token`	Token for Databricks authentication
`container_registry` [optional]	An Azure Container Registry (ACR)
`container_registry.username`	Username of the ACR
`container_registry.password`	Password of the ACR
`container_registry.registry`	The url for the ACR (looks like: `*.azurecr.io/`)

azure-common

azure:
  common:
    artifacts_shared_storage_account_container_name: "libraries"

field	description
`artifacts_shared_storage_account_container_name` [optional]	Container name for an Azure Storage Account V1. Useful for storing artifacts such as wheels and jars.

Takeoff config