Databricks API Essential Guide

What type of API does Databricks provide?

Based on the search results provided, Databricks primarily offers REST APIs. The Databricks documentation specifically mentions a "REST API reference" multiple times.

There is no mention of GraphQL or SOAP APIs for Databricks in the provided search results.

REST (Representational State Transfer) is a widely used architectural style for building web services and APIs. REST APIs are stateless, resource-based, have a uniform interface, are cacheable, and designed as layered systems. REST APIs are well-suited for building web and mobile applications, microservices architectures, and IoT systems.

Here's a general example of how you might interact with a Databricks REST API using Python:

import requests

# Make a GET request to a Databricks REST API endpoint
response = requests.get('https://your-databricks-instance.com/api/2.0/endpoint', 
                        headers={'Authorization': 'Bearer YOUR_ACCESS_TOKEN'})

# Check if the request was successful
if response.status_code == 200:
    # Process the JSON response
    data = response.json()
    print(data)
else:
    print(f"Request failed with status code: {response.status_code}")

When working with Databricks REST APIs, it's important to follow best practices such as using appropriate authentication methods, handling rate limiting and pagination, properly managing API keys and tokens, implementing error handling, and considering the use of a Databricks SDK if available for your programming language.

Does the Databricks API have webhooks?

Availability of Webhooks

Yes, the official Databricks API does have webhooks, specifically for the MLflow Model Registry. These are called MLflow Model Registry Webhooks.

Types of Events You Can Subscribe To

The MLflow Model Registry webhooks allow you to subscribe to the following events:

1. MODEL_VERSION_CREATED: Triggered when a new model version is created
2. MODEL_VERSION_TRANSITIONED_STAGE: Triggered when a model version's stage is changed
3. TRANSITION_REQUEST_CREATED: Triggered when a user requests a model version's stage to be transitioned
4. COMMENT_CREATED: Triggered when a user writes a comment on a registered model
5. REGISTERED_MODEL_CREATED: Triggered when a new registered model is created (only for registry-wide webhooks)
6. MODEL_VERSION_TAG_SET: Triggered when a user sets a tag on the model version
7. MODEL_VERSION_TRANSITIONED_TO_STAGING: Triggered when a model version is transitioned to staging
8. MODEL_VERSION_TRANSITIONED_TO_PRODUCTION: Triggered when a model version is transitioned to production
9. MODEL_VERSION_TRANSITIONED_TO_ARCHIVED: Triggered when a model version is archived
10. TRANSITION_REQUEST_TO_STAGING_CREATED: Triggered when a user requests a model version to be transitioned to staging
11. TRANSITION_REQUEST_TO_PRODUCTION_CREATED: Triggered when a user requests a model version to be transitioned to production
12. TRANSITION_REQUEST_TO_ARCHIVED_CREATED: Triggered when a user requests a model version to be archived

Key Points to Consider

1. There are two types of webhooks based on their trigger targets:

   • HTTP registry webhooks: Send triggers to an HTTP endpoint
   • Job registry webhooks: Trigger a job in a Databricks workspace

2. There are also two types of webhooks based on their scope:

   • Model-specific webhooks: Apply to a specific registered model
   • Registry-wide webhooks: Triggered by events on any registered model in the workspace

3. Webhooks can be created, tested, updated, and deleted using the Databricks REST API or the Python client databricks-registry-webhooks.

4. Webhooks are not available when using Models in Unity Catalog.

5. For security, Databricks includes an X-Databricks-Signature in the header computed from the payload and the shared secret key associated with the webhook.

6. Webhooks can be used to automate and integrate your machine learning pipeline with existing CI/CD tools and workflows.

Best Practices

1. Use different credentials for each webhook destination to improve security.
2. Consider using OAuth tokens or personal access tokens belonging to service principals instead of workspace users for authentication.
3. When setting up webhooks, ensure you have the necessary permissions. For model-specific webhooks, you need CAN MANAGE permissions on the registered model. For registry-wide webhooks, you need workspace admin permissions.

In summary, the Databricks API offers robust webhook functionality for the MLflow Model Registry, allowing you to subscribe to a variety of events related to model versioning, transitions, and comments. This can be particularly useful for automating workflows and integrating with other tools in your machine learning pipeline.

Rate Limits and other limitations

Based on the search results provided, here are the key points regarding API rate limits for the Databricks API:

General API Rate Limits

The search results do not provide specific information about general API rate limits for the Databricks API. However, they do mention some limits for specific services and features:

1. Jobs API:

   • Created per hour: 10,000 (per workspace) [1][2]
   • Tasks running simultaneously: 1,000 (per workspace) [1][2]
   • Saved jobs: 10,000-12,000 (per workspace) [1][2]

2. MLflow API:

   • Experiment tracking total metric steps: 1M (per MLflow run) [1][2]
   • Experiment tracking parameters: 1,600 (per MLflow run) [1][2]
   • Experiment tracking tags: 1,600 (per MLflow run) [1][2]
   • Experiment tracking runs: 500k (per MLflow experiment) [1][2]

3. MLflow Workspace Model Registry API:

   • /mlflow/webhooks/test requests per second: 40 (per workspace) [2]

Model Serving API Rate Limits

For Model Serving, there are more specific rate limits mentioned:

1. Queries per second (QPS):

   • Default: 200 per workspace
   • Can be increased to 25,000 or more by contacting the Databricks account team [5]

2. Foundation Model APIs (pay-per-token) rate limits:

   • DBRX Instruct model: 1 query per second
   • Other chat and completion models: 2 queries per second (default)
   • Embedding models: 300 embedding inputs per second (default)
   • These limits can be increased by contacting the Databricks account team [5]

3. Foundation Model APIs (provisioned throughput) rate limits:

   • 200 per workspace [5]

4. Provisioned concurrency:

   • 200 concurrency per workspace
   • Can be increased by contacting the Databricks account team [5]

Key Points to Consider

• Many of these limits are per workspace, not per account or user.
• Some limits can be increased by contacting the Databricks account team.
• For Foundation Model APIs, only workspace admins can change governance settings like rate limits.
• There are additional limitations on payload size, model execution duration, and memory usage for model serving endpoints.

Best Practices

1. Monitor your API usage to ensure you stay within the limits.
2. If you need higher limits, contact your Databricks account team to discuss increasing them.
3. For high-throughput applications, consider using the provisioned throughput option for Foundation Model APIs.
4. Be aware of the differences in limits between pay-per-token and provisioned throughput models, especially for compliance-sensitive workloads.

It's important to note that these limits may change over time, and the most up-to-date information should be obtained directly from Databricks or their official documentation.

Latest API Version

The most recent version of the Databricks API is 2.0, based on the information provided in the search results. Here are the key points to consider:

API Version

• The Databricks API is currently at version 2.0 ^2.

Databricks API Client

• The latest version of the databricks-api Python package is 0.9.0, released on June 8, 2023 ^2.
• However, it's important to note that this package is auto-generated from the official databricks-cli package ^2.

Official Databricks SDK

• The search results recommend switching to the official Databricks SDK for Python ^2.
• You can install it using the command: pip install databricks-sdk ^2.

Databricks Runtime Versions

• While not directly related to the API version, it's worth noting that Databricks regularly releases new Runtime versions ^5.
• The most recent non-beta release mentioned in the search results is Databricks Runtime 15.3, scheduled for release on June 24, 2024 ^5.

Best Practices

1. Use the official Databricks SDK for Python when possible, as it provides the most up-to-date and supported interface to the Databricks API.

2. Keep your Databricks Runtime version up to date to ensure compatibility with the latest API features and improvements.

3. When using the API, always refer to the official Databricks documentation for the most accurate and current information on endpoints and functionality.

4. Be aware of the compatibility between different Databricks components, such as the Runtime version, MLflow version, and Feature Engineering version, when developing your applications ^5.

Remember that API versions and related software are frequently updated, so it's always a good idea to check the official Databricks documentation for the most current information on API versions and best practices.

How to get a Databricks developer account and API Keys?

To get a developer account for Databricks and create an API integration, you need to follow these steps:

1. Sign up for a Databricks account

First, you need to sign up for a Databricks account if you don't already have one. You can do this by visiting the Databricks website and signing up for a free trial or a paid plan.

2. Generate an access token

Once you have a Databricks account, you need to generate an access token to authenticate your API requests. Here's how to do it:

1. Log in to your Databricks workspace
2. Click on your username in the top-right corner
3. Select "User Settings"
4. Go to the "Access Tokens" tab
5. Click "Generate New Token"
6. Provide a comment for the token and set its lifetime
7. Copy the generated token and store it securely

3. Set up authentication

To use the Databricks API, you need to set up authentication. There are two main methods:

1. OAuth for service principals (recommended for automated tools and applications)
2. OAuth for users (for individual user authentication)

For OAuth authentication, you'll need to configure your Databricks CLI or use the appropriate authentication method in your API requests.

4. Install the Databricks CLI (optional but recommended)

Installing the Databricks CLI can make it easier to interact with the API. You can install it using pip:

pip install databricks-cli

5. Configure the Databricks CLI

After installation, configure the CLI with your access token:

databricks configure --token

Enter your Databricks host (e.g., https://your-account.cloud.databricks.com) and your access token when prompted.

6. Start using the API

Now you can start using the Databricks API. Here are a few examples:

• List clusters:

  databricks clusters list
  

• Create a job:

  databricks jobs create --json-file job_config.json
  

• Run a notebook:

  databricks runs submit --json-file notebook_run_config.json
  

What can you do with the Databricks API?

Based on the search results provided, here are the key data models you can interact with using the Databricks API, along with what is possible for each:

Custom Models

• Deploy and query custom Python models packaged in MLflow format
• Can be registered in Unity Catalog or workspace model registry
• Examples include scikit-learn, XGBoost, PyTorch, and Hugging Face transformer models
• Deploy generative AI agents as custom models

Foundation Model APIs

• Access and query state-of-the-art open models from a serving endpoint
• Available in two pricing modes:
  • Pay-per-token: For getting started, not designed for high-throughput production use
  • Provisioned throughput: For production workloads requiring high performance
• Supported models include DBRX Instruct, Meta-Llama models, Mixtral, MPT models, and embedding models
• Can deploy base models of all sizes, fine-tuned variants, and fully custom weights/tokenizers

External Models

• Serve generative AI models hosted outside of Databricks
• Centrally govern and establish rate limits and access control
• Examples include OpenAI's GPT-4 and Anthropic's Claude

Key Capabilities

• Unified REST API and MLflow Deployment API for CRUD and querying tasks
• Single UI to manage all models and serving endpoints
• Access models directly from SQL using AI functions
• Secure customization with private data through Feature Store and Vector Search integration
• Governance features like permissions management, usage limits, and quality monitoring
• Automatic scaling and optimized inference for large models
• High availability and security for production use

Additional Features

• AI Playground for testing and comparing LLMs
• Batch inference capabilities
• Function calling support

This overview covers the main data models and capabilities available through the Databricks API for model serving and interaction. The API provides a comprehensive set of tools for deploying, managing, and querying various types of AI models within the Databricks ecosystem.