Batch monitoring¶

You can view monitoring statistics organized by batch, instead of by time, with batch-enabled deployments. To do this, you can create batches, add predictions to those batches, and view service health, data drift, and accuracy statistics for the batches in your deployment.

Enable batch monitoring for a new deployment¶

When you initiate model deployment, the Deployments tab opens to the deployment configuration page:

To enable batch monitoring for this deployment, in the Prediction History and Service Health section, locate Batch Monitoring and click Enable batch monitoring:

Enable batch monitoring for an existing deployment¶

If you already created the deployment you want to configure for batch monitoring, navigate to that deployment's Predictions > Settings tab. Then, under Predictions Settings and Batch Monitoring, click Enable batch monitoring and Save the new settings:

View a batch monitoring deployment in the inventory¶

When a deployment has batch monitoring enabled, on the Deployments tab, in the Deployment Name column, you can see a BATCH badge. For these deployments, in the Prediction Health lens, the Service, Drift, and Accuracy indicators are based on the last few batches of predictions, instead of the last few days of predictions. The Activity chart shows any batch predictions made with the deployment over the last seven days:

To configure the number of batches represented by the Prediction Health indicators, you select a batch range on the deployment's settings tabs for Service Health, Data Drift, and Accuracy:

Create batches with batch predictions¶

To make a batch and add predictions to it, you can make batch predictions or schedule a batch prediction job for your deployment. Each time a batch prediction or scheduled batch prediction job runs, a batch is created automatically, and every prediction from the batch prediction job is added to that batch.

In addition, on the Predictions > Predictions API tab, you can copy and run the Python code snippet to make a DataRobot API request creating a batch on the Batch Management tab and adding predictions to it:

Create a batch and add agent-monitored predictions¶

In a deployment with the BATCH badge, you can access the Predictions > Batch Management tab, where you can add batches. Once you have the Batch ID, you can assign predictions to that batch. You can create batches using the UI or API:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

import requests

API_KEY = <api_key>
DATAROBOT_KEY = <datarobot_key>

BATCH_CREATE_URL = "https://app.datarobot.com/api/v2/deployments/<deployment_id>/monitoringBatches/"

headers = {
    'Content-Type': 'application/json; charset=UTF-8',
    'Authorization': 'Bearer {}'.format(API_KEY),
    'DataRobot-Key': DATAROBOT_KEY,
}

requests_body = {'batchName': '<batch_name>'}

response = requests.post(url=BATCH_CREATE_URL, headers=headers, json=request_body)

print(response.json())

You can now add predictions to the new batch. To add predictions from an agent-monitored external model to an existing batch, you can follow the batch monitoring example provided in the MLOps agent tarball. Before you run the example, on the Predictions > Batch Management tab, copy the Batch ID of the batch you want to add external predictions to, saving it for use as an input argument in the agent example.

In the BatchMonitoringExample/binary_classification.py example script included in the MLOps agent tarball, the batch ID you provide as an input argument defines the batch_id in the report_deployment_stats and report_predictions_data calls on the highlighted lines below:

118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159

 # MLOPS: report deployment metrics: number of predictions and execution time
    m.report_deployment_stats(
        predictions_array.shape[0],
        (end_time - start_time) * 1000,
        batch_id=args.batch_id,
    )

    target_column_name = columns[len(columns) - 1]
    predictions = []

    orig_labels = test_df[target_column_name].tolist()
    # Based on prediction value and the threshold assign correct label to each prediction
    target_labels = []
    for value in predictions_array.tolist():
        if len(value) == 1:
            # Random forest classifier from scikit-learn can return a single probability value
            # instead of 2 values.  You need to infer the other one before reporting predictions,
            # because, 'report_predictions_data' expects probability for each class.
            value.append(1 - value[0])
        predictions.append(value)
        if value[0] < prediction_threshold:
            target_labels.append("0.0")
        else:
            target_labels.append("1.0")

    # Write csv file with labels and association Id, when output file is provided
    if args.actuals_filename is not None:
        write_actuals_file(args.actuals_filename, orig_labels, association_ids)

    # MLOPS: report test features and predictions and association_ids
    m.report_predictions_data(
        features_df=test_df,
        predictions=predictions,
        association_ids=association_ids,
        batch_id=args.batch_id,
    )

    m.shutdown()


if __name__ == "__main__":
    main()

Predictions reported by the agent are added to the batch you defined in the code.

Create batches with the batch CLI tools¶

To make batch predictions using the DataRobot command line interface (CLI) for batch predictions, download the CLI tools from the Predictions > Prediction API tab of your deployment. To locate it make sure the Prediction Type is set to Batch and the Interface is set to CLI, then click download CLI tools (or Copy script to clipboard):

Open the downloaded predict.py file (or paste the code you copied into a Python file) and add the highlighted code below to the appropriate location in the file, defining monitoringBatchPrefix to assign a name to your batches:

408
409
410
411
412
413
414
415
416
417
418
419
420
421

    elif args.predictions_start_date:
        logger.info("Cannot use --predictions_start_date without --predictions_end_date")
        return 1
    elif args.predictions_end_date:
        logger.info("Cannot use --predictions_end_date without --predictions_start_date")
        return 1

    payload['monitoringBatchPrefix'] = 'Batch From predict.py'

    try:
        make_datarobot_batch_predictions(input_file, output_file, payload)
    except DataRobotPredictionError as err:
        logger.exception("Error: {}".format(err))
        return 1

You can now use the CLI as normal, and the batch predictions are added to a batch with the batch prefix defined in this file.

Manage batches¶

On the Predictions > Batch Management tab, in addition to creating batches, you can perform a number of management actions, and view your progress towards the daily batch creation limit for the current deployment.

At the top of the Batch Management page, you can view the Batches created progress bar. The default batch creation limit is 100 batches per deployment per day. Below the progress bar, you can view the when your daily limit resets:

In the list below, you can view basic batch information, including the Batch Name, Batch ID, Creation Date, Earliest Prediction, Latest Prediction, and the number of Predictions.

Additionally, you can view advanced information, such as:

In the Job Status and Actions columns, you can perform several management actions:

Click a batch to open the Info tab, where you can view and edit () the Batch Name, Description, and External URL. In addition, you can click the copy icon () next to the Batch ID to copy the ID for use in prediction code snippets:

Click the Model History tab to view the Model Id of any model used to make predictions for a batch. You can also view the prediction Start Time and End Time fields. You can edit () these dates to identify predictions made on historical data, allowing the deployments charts to display prediction information for when the historical data is from, instead of displaying when DataRobot received the predictions:

Monitoring for batch deployments¶

Batch deployments support batch-specific service health, data drift, and accuracy monitoring. These visualizations differ from the visualizations for standard deployments:

In addition to service health, data drift, and accuracy monitoring, you can configure data exploration and custom metrics for batch deployments. On both of these tabs, you can use the Batch Name selector to determine which batches are included:

In the Select batches modal, click to select a batch, moving it to the right (selected) column. You can also click to remove a selected batch, returning it to the left (unselected) column.