Create custom models¶

Custom models are user-created, pretrained models that you can upload to DataRobot (as a collection of files) via the Model workshop. You can assemble custom models in either of the following ways:

• Create a custom model without the environment requirements and a start_server.sh file on the Assemble tab. This type of custom model must use a drop-in environment. Drop-in environments contain the requirements and start_server.sh file used by the model. They are provided by DataRobot in the workshop.

• Create a custom model with the environment requirements and a start_server.sh file on the Assemble tab. This type of custom model can be paired with a custom or drop-in environment.

Be sure to review the guidelines for assembling a custom model before proceeding. If any files overlap between the custom model and the environment folders, the model's files will take priority.

Create a new custom model¶

To create a custom model in preparation for assembly:

1. Click Registry > Model workshop. This tab lists the models you have created:

   

2. Click + Add model (or the button when the custom model panel is open):

   

3. On the Add a model page, enter the fields under Configure the model:

   

   Field	Description
   Model name	A descriptive name for the custom model.
   Target type	The type of prediction the model makes. Depending on the prediction type, you must configure additional settings: Binary: For a binary classification model, enter the Positive class label and the Negative class label. Regression: No additional settings. Time Series (Binary): Preview feature. For a binary classification model, enter the Positive class label and the Negative class label and configure the time series settings. Time Series (Regression): Preview feature. Configure the time series settings. Multiclass: For a multiclass classification model, enter or upload (.csv, .txt) the Target classes for your target, one class per line. To ensure that the classes are applied correctly to your model's predictions, enter classes in the same order as your model's predicted class probabilities. Text Generation: Premium feature. No additional settings. Anomaly Detection: No additional settings. Unstructured: No additional settings. Unstructured models do not need to conform to a specific input/output schema and may use a different request format. Prediction drift, accuracy tracking, challengers, and humility are disabled for deployments with a "Unstructured" target type. Service health, deployment activity, and governance remain available.
   Target name	The dataset's column name that the model will predict on. This field isn't available for multiclass or anomaly detection models.
   Advanced configuration
   Language	The programming language used to build the model.
   Description	A description of the model's contents and purpose.

4. After completing the fields, click Add model.

   The custom model opens to the Assemble tab.

Configure time series settings¶

You can create time series custom models by configuring the following time series-specific fields, in addition to the fields required for binary classification and regression models. To create a time series custom model, select Time Series (Binary) or a Time Series (Regression) as a Target type, and configure the following settings while creating the model:

When you make real-time predictions with a time series custom model, in the CSV serialization of the prediction response, any extra columns (beyond the prediction results) returned from the model have a column name suffixed with _OUTPUT.

Configure an anomaly detection model¶

You can create custom models that support anomaly detection problems. If you choose to build one, reference the DRUM template. (Log in to GitHub before clicking this link.) When deploying custom anomaly detection models, note that the following functionality is not supported:

• Data drift
• Accuracy and association IDs
• Challenger models
• Humility rules
• Prediction intervals

Assemble the custom model¶

After you create a custom model, you can provide the required environment, dependencies, and files:

1. In the model you want to assemble, on the Assemble tab, navigate to the Environment section and select a model environment from the Base environment dropdown menu.

   

   Model environments

   The model environment is used for testing and deploying the custom model. The Base Environment drop-down list includes drop-in model environments and any custom environments that you can create.

2. To populate the Dependencies section, you can upload a requirements.txt file in the Files section, allowing DataRobot to build the optimal image.

3. In the Files section, add the required custom model files. If you aren't pairing the model with a drop-in environment, this includes the custom model environment requirements and a start_server.sh file. You can add files in several ways:

   

   	Element	Description
   1	Files	Drag files into the group box for upload.
   2	Choose from source	Click to browse for Local Files or a Local Folder.
   3	Upload	Click to browse for Local Files or a Local Folder or to pull files from a remote repository.
   4	Create	Create a new file, empty or as a template, and save it to the custom model: Create model-metadata.yaml: Creates a basic, editable example of a runtime parameters file. Create blank file: Creates an empty file. Click the edit icon () next to Untitled to provide a file name and extension, then add your custom contents.

   File replacement

   If you add a new file with the same name as an existing file, when you click Save, the old file is replaced in the Files section.

   Model file location

   If you add files from a Local Folder, make sure the model file is already at the root of the custom model. Uploaded folders are for dependent files and additional assets required by your model, not the model itself. Even if the model file is included in the folder, it will not be accessible to DataRobot unless the file exists at the root level. The root file can then point to the dependencies in the folder.

If you incorrectly add one or more files or folders to the Assemble tab, in the Files section, you can click the delete () icon next to each file or folder to remove it from the custom model.

Remote repositories¶

In the Files section, if you click Upload > Remote Repository, the Add the content from remote repository panel opens. You can click Remote Repositories to register a new remote repository (A), or select an existing remote repository (B):

After registering and selecting a repository, you can select the check box for each file or folder you want to upload, and then click Select content:

The selected files are pulled into the Files section.

Drop-in environments¶

DataRobot provides drop-in environments in the model workshop, defining the required libraries and providing a start_server.sh file. The following table details the drop-in environments provided by DataRobot and links to the template in the DRUM repository. Each environment is prefaced with [DataRobot] in the Environment section of the Model workshop's Assemble tab.

When using a drop-in environment, your custom model code can reference several environment variables injected to facilitate access to the DataRobot Client and MLOps Connected Client:

Configure evaluation and moderation¶

Evaluation and moderation guardrails help your organization block prompt injection and hateful, toxic, or inappropriate prompts and responses. It can also prevent hallucinations or low-confidence responses and, more generally, keep the model on topic. In addition, these guardrails can safeguard against the sharing of personally identifiable information (PII). Many evaluation and moderation guardrails connect a deployed text generation model (LLM) to a deployed guard model. These guard models make predictions on LLM prompts and responses and report these predictions and statistics to the central LLM deployment. To use evaluation and moderation guardrails, first, create and deploy guard models to make predictions on an LLM's prompts or responses; for example, a guard model could identify prompt injection or toxic responses. Then, when you create a custom model with the Text Generation target type, define one or more evaluation and moderation guardrails.

To select and configure evaluation and moderation guardrails, on the Assemble tab for a custom model with the Text Generation target type, scroll to Evaluation and Moderation and click Configure:

In the Configure evaluation and moderation panel, click a metric card, configure the required properties, and then click Add to return the the Evaluation tab:

Select another metric or click Next to proceed to the Moderation tab to configure a Moderation strategy:

After adding guardrails to a text generation custom model, you can test, register, and deploy the model to make predictions in production. After you make predictions, you can view the evaluation metrics on the Custom metrics tab and prompts, responses, and feedback (if configured) on the Data exploration tab.

For more information, see Configure evaluation and moderation.

Define connected deployments¶

Connected deployments report predictions and statistics to a central text generation model deployment. To use connected deployments, first, create and deploy audit models to make predictions on an LLM's prompts or outputs; for example, an audit model could identify prompt injection or toxic responses. Then, when you create a custom model with the Text Generation target type, define one or more connected deployments to generate predictions and report them to the central model. The connected deployment predictions can be combined into the prediction output of the central, text generation model deployment.

To define connected deployments, upload or create a model-metadata.yaml file with one or more deployment-type runtime parameters. If you click Create > Create model-metadata.yaml the template includes an example:

runtimeParameterDefinitions:
- fieldName: DEPLOYMENT_PARAMETER
    type: deployment
    description: An example of a deployment parameter (for various purposes, such as connected deployments)
    defaultValue: null

In the example above replace null with the deployment ID of the deployment you want to connect, or, after saving the model-metadata.yaml file, click the Edit icon () to enter the deployment ID:

To use connected deployments, ensure that the custom model has the Text Generation target type. The target type can be defined in the model-metadata.yaml file as follows:

targetType: textgeneration

After you assemble and register a custom model, you can view the connected deployments in the registered model version's Related Items section:

Manage runtime parameters¶

If you defined any runtime parameters through runtimeParameterDefinitions in the model-metadata.yaml file, you can manage them in the Runtime Parameters section:

If any runtime parameters have allowEmpty: false in the definition without a defaultValue, you must set a value before registering the custom model.

For more information on how to define runtime parameters and use them in your custom model code, see the Define custom mode runtime parameters documentation.

Assign custom model datasets¶

To enable feature drift tracking for a model deployment, you must add training data. To do this, assign training data to a model version. The method for providing training and holdout datasets for unstructured custom inference models requires you to upload the training and holdout datasets separately. Additionally, these datasets cannot include a partition column.

To assign training data to a custom model after you define the Environment and Files:

1. On the Assemble tab, in the Settings section, under Datasets:

   • If the model version doesn't have training data assigned, click Assign.

     

   • If the model version does have training data assigned, click () Change, and then, under Training Data, click the delete () icon to remove the existing training data.

     

2. In the Training Data section, click + Select dataset and do either of the following:

   

   • To add a new dataset to the Data Registry, click Upload, select a file from your local storage, and then click Open.

   • To select an existing dataset from the Data Registry, in the Data Registry list, locate and click the training dataset you previously uploaded to DataRobot, and then click Select dataset.

   Include features required for scoring

   The columns in a custom model's training data indicate which features are included in scoring requests to the deployed custom model. Therefore, once training data is available, any features not included in the training dataset aren't sent to the model. Available as a preview feature, you can disable this behavior using the Column filtering setting.

3. (Optional) In the Partition column section, specify the column name (from the provided training dataset) containing partitioning information for your data (based on training/validation/holdout partitioning). If you plan to deploy the custom model and monitor its data drift and accuracy, specify the holdout partition in the column to establish an accuracy baseline.

   

   Specify partition column

   You can track data drift and accuracy without specifying a partition column; however, in that scenario, DataRobot won't have baseline values. The selected partition column should only include the values T, V, or H.

4. Click Assign.

   Training data assignment error

   If the training data assignment fails, an error message appears in the new custom model version under Datasets. While this error is active, you can't create a model package to deploy the affected version. To resolve the error and deploy the model package, reassign training data to create a new version, or create a new version and then assign training data.

Disable column filtering for prediction requests¶

Now available for preview, you can enable or disable column filtering for custom model predictions. The filtering setting you select is applied in the same way during custom model testing and deployment. By default, the target column is filtered out of prediction requests and, if training data is assigned, any additional columns not present in the training dataset are filtered out of any scoring requests sent to the model. Alternatively, if the prediction dataset is missing columns, an error message appears to notify you of the missing features.

You can disable this column filtering when you assemble a custom model. In the Model workshop, open a custom model to the Assemble tab, and, in the Settings section, under Column filtering, clear Exclude target column from requests (or, if training data is assigned, clear Exclude target and extra columns not in training data):

As with other changes to a model's environment, files, or settings, changing this setting creates a new minor custom model version.

The following behavior is expected when Exclude target column from requests / Exclude target and extra columns not in training data is enabled or disabled:

Configure custom model resource settings¶

After creating a custom inference model, you can configure the resources the model consumes to facilitate smooth deployment and minimize potential environment errors in production.

To configure the resource allocation and access settings:

1. On the Assemble tab, in the Settings section, next to Resources, click () Edit:

   

2. In the Update resource settings dialog box, configure the following settings:

   

   Resource settings access

   Users can determine the maximum memory allocated for a model, but only organization admins can configure additional resource settings.

   Setting	Description
   Memory	Determines the maximum amount of memory that can be allocated for a custom inference model. If a model is allocated more than the configured maximum memory value, it is evicted by the system. If this occurs during testing, the test is marked as a failure. If this occurs when the model is deployed, the model is automatically launched again by Kubernetes.
   Replicas	Sets the number of replicas executed in parallel to balance workloads when a custom model is running. Increasing the number of replicas may not result in better performance, depending on the custom model's speed.
   Network access	Premium feature. Configures the egress traffic of the custom model: Public: The default setting. The custom model can access any fully qualified domain name (FQDN) in a public network to leverage third-party services. None: The custom model is isolated from the public network and cannot access third-party services. When public network access is enabled, your custom model can use the DATAROBOT_ENDPOINT and DATAROBOT_API_TOKEN environment variables. These environment variables are available for any custom model using a drop-in environment or a custom environment built on DRUM.

   Imbalanced memory settings

   DataRobot recommends configuring resource settings only when necessary. When you configure the Memory setting above, you set the Kubernetes memory "limit" (the maximum allowed memory allocation); however, you can't set the memory "request" (the minimum guaranteed memory allocation). For this reason, it is possible to set the "limit" value too far above the default "request" value. An imbalance between the memory "request" and the memory usage allowed by the increased "limit" can result in the custom model exceeding the memory consumption limit. As a result, you may experience unstable custom model execution due to frequent eviction and relaunching of the custom model. If you require an increased Memory setting, you can mitigate this issue by increasing the "request" at the organization level; for more information, contact DataRobot Support.

   Premium feature: Network access

   Every new custom model you create has public network access by default; however, when you create new versions of any custom model created before October 2023, those new versions remain isolated from public networks (access set to None) until you enable public access for a new version (access set to Public). From this point on, each subsequent version inherits the public access definition from the previous version.

3. Once you have configured the resource settings for the custom model, click Save.

   This creates a new version of the custom model with edited resource settings applied.

   Frozen versions

   You can only update resource settings in the most recent model version. Additionally, if the most recent model version was registered or deployed, you cannot apply the updated resource settings, and you'll encounter a "frozen version" warning if you try to save the changes. If you encounter this warning, create a new custom model version, update the resource settings, and then register and deploy the model with the correct settings.

Select a resource bundle¶

You can select a Resource bundle—instead of Memory—when you assemble a model and configure the resource settings. Resource bundles allow you to choose from various CPU and GPU hardware platforms for building and testing custom models. In a custom model's Settings section, open the Resources settings to select a resource bundle. In this example, the model is built to be tested and deployed on an NVIDIA A10 device.

Click Edit to open the Update resource settings dialog box and, in the resource Bundle field, review the CPU and NVIDIA GPU devices available as build environments:

DataRobot can deploy models onto any of the following NVIDIA resource bundles:

Along with the NVIDIA GPU resource bundles, this feature introduces custom model environments optimized to run on NVIDIA GPUs. When you assemble a custom model, you define a Base Environment. The model in the example below is running on an [NVIDIA] Triton Inference Server:

For more information on the DataRobot integration with NVIDIA, see Generative AI with NeMo Guardrails on NVIDIA GPUs.