Enable network access for custom tasks¶

Now available for preview, you can set up custom tasks to have public network access. The code examples on this page showcase a binary estimator task that uses an API endpoint with credentials to gain network access.

Get the credentials ID¶

Before configuring network access, you must be able to provide your credentials ID. After setting up your own credentials, open your DataRobot user profile and select Credentials Management.

Select your credentials. Once selected, you can copy the credentials ID. The ID is the string that follows /credentials-management/ in the URL.

Set credentials¶

Review the example below of how to add credentials to model-metadata.yaml. You can access the example file in the DRUM repo. The typeSchema is copied from another example in the DRUM repo.

name: 13_python_credentials_and_internet_access
type: training
environmentID: 5e8c889607389fe0f466c72d
targetType: binary

# These must be actual DataRobot credentials that the author owns
userCredentialSpecifications:
  - key: MY_CREDENTIAL  # A legal POSIX env var key
    valueFrom: 655270e368a555f026e2512d  # A credential ID from DataRobot for which you are the owner
    reminder: my super-cool.com/api api-token  # Optional: any string value that you for a reminder

typeSchema:
  input_requirements:
    - field: data_types
      condition: IN
      value:
        - NUM
    - field: number_of_columns
      condition: NOT_LESS_THAN
      value: 2
    - field: sparse
      condition: EQUALS
      value: SUPPORTED
# This requirement is only ignored because this is an example using test data

Enable public IP address access¶

In order to have network access from within a custom task, you need to specifically enable it in the Custom Task Version using the outgoingNetworkPolicy field. Any new versions will inherit the previous version's outgoingNetworkPolicy unless you specify a different one. To do so, you must the the REST API.

from datarobot.enums import CustomTaskOutgoingNetworkPolicy

task_version = dr.CustomTaskVersion.create_clean(
    custom_task_id=custom_task_id,
    base_environment_id=execution_environment.id,
    folder_path=custom_task_folder,
    outgoing_network_policy=CustomTaskOutgoingNetworkPolicy.PUBLIC,
)

For more information, reference the Python client documentation.

Test locally with DRUM¶

If you want to test in DRUM with your credentials, you can fake the data by making a secrets directory and putting all of your secrets there. You can view the example in the DRUM repo.

Example command¶

drum fit -cd task_templates/2_estimators/13_python_credentials_and_internet_access/ \
--input tests/testdata/10k_diabetes.csv --target-type binary --target readmitted \
--user-secrets-mount-path task_templates/2_estimators/13_python_credentials_and_internet_access/ \
--verbose --logging-level info --show-stacktrace

Secrets details¶

Each secret file should have corresponding credentials with the same name. The contents of a secret file should be a JSON string that can be cast to one of the secrets objects. All secrets objects are in custom_model_runner/datarobot_drum/custom_task_interfaces/user_secrets.py. Your secret response must contain a credential_type, which is another name for datarobot_drum.custom_task_interfaces.user_secrets.SecretType; however, the value must be all lowercase (SecretType.SNOWFLAKE_KEY_PAIR_USER_ACCOUNT corresponds to {"credential_type": "snowflake_key_pair_user_account"}).

For example:

@dataclass(frozen=True)
class SnowflakeKeyPairUserAccountSecret(AbstractSecret):
    username: Optional[str]
    private_key_str: Optional[str]
    passphrase: Optional[str] = None
    config_id: Optional[str] = None

would be:

{
  "credential_type": "snowflake_key_pair_user_account",
  "username": "bob@bob.com",
  "private_key_str": "shhhhhhhh"
}