> azure-databricks

Expert knowledge for Azure Databricks development including troubleshooting, best practices, decision making, architecture & design patterns, limits & quotas, security, configuration, integrations & coding patterns, and deployment. Use when building, debugging, or optimizing Azure Databricks applications. Not for Azure HDInsight (use azure-hdinsight), Azure Synapse Analytics (use azure-synapse-analytics), Azure Machine Learning (use azure-machine-learning), Azure Data Explorer (use azure-data-ex

fetch
$curl "https://skillshub.wtf/MicrosoftDocs/Agent-Skills/azure-databricks?format=md"
SKILL.mdazure-databricks

Azure Databricks Skill

This skill provides expert guidance for Azure Databricks. Covers troubleshooting, best practices, decision making, architecture & design patterns, limits & quotas, security, configuration, integrations & coding patterns, and deployment. It combines local quick-reference content with remote documentation fetching capabilities.

How to Use This Skill

IMPORTANT for Agent: Use the Category Index below to locate relevant sections. For categories with line ranges (e.g., L35-L120), use read_file with the specified lines. For categories with file links (e.g., [security.md](security.md)), use read_file on the linked reference file

IMPORTANT for Agent: If metadata.generated_at is more than 3 months old, suggest the user pull the latest version from the repository. If mcp_microsoftdocs tools are not available, suggest the user install it: Installation Guide

This skill requires network access to fetch documentation content:

  • Preferred: Use mcp_microsoftdocs:microsoft_docs_fetch with query string from=learn-agent-skill. Returns Markdown.
  • Fallback: Use fetch_webpage with query string from=learn-agent-skill&accept=text/markdown. Returns Markdown.

Category Index

CategoryLocationDescription
TroubleshootingL37-L127Diagnosing and fixing Databricks errors and failures across compute, SQL, Delta, connectors, Lakeflow, AI/Genie, VS Code/CLI, and performance using logs, query history, and debugging tools.
Best PracticesL128-L328End-to-end Databricks best practices: cost, security, governance, compute, performance tuning, streaming, Lakehouse design, ML/GenAI, Lakeflow, and troubleshooting for production workloads.
Decision MakingL329-L410Guides for choosing Databricks runtimes, compute, storage and connectors, and for planning/migrating workloads, ML/LLM, data, and Lakebase/Unity Catalog features across versions and services
Architecture & Design PatternsL411-L448Patterns and reference architectures for Databricks lakehouse, AI agents, RAG, LLMOps/MLOps, Lakebase, governance, performance, and data modeling on Azure.
Limits & Quotaslimits-quotas.mdLimits, quotas, constraints, and requirements for Databricks runtimes, compute (serverless/standard/GPU), AI/BI, connectors, Lakeflow, Model Serving, Unity Catalog, and related APIs.
Securitysecurity.mdIdentity, access control, encryption, networking, compliance, and secure integrations for Azure Databricks, Unity Catalog, Lakeflow, Lakebase, Delta Sharing, and Databricks Apps.
Configurationconfiguration.mdConfiguring and administering Azure Databricks: accounts, workspaces, security, networking, compute, storage, SQL/UC, Lakeflow, ML/GenAI, Lakebase, Marketplace, and CLI/app-based automation.
Integrations & Coding Patternsintegrations.mdPatterns and code for integrating Databricks with external systems: BI tools, storage, databases, streaming, agents/LLMs, Lakeflow, Lakebase, MLflow, and SDK/CLI/Terraform-based automation.
Deploymentdeployment.mdDeploying and managing Azure Databricks workspaces, apps, jobs, ML/GenAI models and agents via CI/CD, Terraform, VS Code, Model Serving, and migration/region/platform readiness guidance.

Troubleshooting

TopicURL
Monitor Genie space activity with audit log querieshttps://learn.microsoft.com/en-us/azure/databricks/ai-bi/admin/audit
Interpret Databricks enhanced security audit log schemashttps://learn.microsoft.com/en-us/azure/databricks/archive/security/monitor-log-schemas
Migrate and troubleshoot Databricks Serverless GPU workloadshttps://learn.microsoft.com/en-us/azure/databricks/compute/serverless/sgc-guides
Troubleshoot Azure Databricks compute startup issueshttps://learn.microsoft.com/en-us/azure/databricks/compute/troubleshooting/
Resolve Databricks classic compute termination error codeshttps://learn.microsoft.com/en-us/azure/databricks/compute/troubleshooting/cluster-error-codes
Debug Spark applications using Databricks Spark UIhttps://learn.microsoft.com/en-us/azure/databricks/compute/troubleshooting/debugging-spark-ui
Troubleshoot common Delta Sharing errorshttps://learn.microsoft.com/en-us/azure/databricks/delta-sharing/troubleshooting
Drop Delta features to fix compatibility issueshttps://learn.microsoft.com/en-us/azure/databricks/delta/drop-feature
Troubleshoot common Databricks CLI errors and issueshttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/troubleshooting
Use Databricks app details page for monitoring and debugginghttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-apps/view-app-details
Troubleshoot Databricks Connect for Python issueshttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-connect/python/troubleshooting
Troubleshoot Databricks Connect for Scala problemshttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-connect/scala/troubleshooting
Troubleshoot common Databricks Terraform provider errorshttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/terraform/troubleshoot
Resolve common issues with Databricks VS Code extensionhttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/vscode-ext/faqs
Troubleshoot Databricks VS Code extension errorshttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/vscode-ext/troubleshooting
Handle ARITHMETIC_OVERFLOW errors in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/arithmetic-overflow-error-class
Resolve CAST_INVALID_INPUT errors in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/cast-invalid-input-error-class
Understand DC_GA4_RAW_DATA_ERROR in Databricks connectorshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/dc-ga4-raw-data-error-error-class
Understand DC_SFDC_API_ERROR in Databricks connectorshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/dc-sfdc-api-error-error-class
Understand DC_SQLSERVER_ERROR in Databricks connectorshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/dc-sqlserver-error-error-class
Handle DELTA_ICEBERG_COMPAT_V1_VIOLATION errors in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/delta-iceberg-compat-v1-violation-error-class
Handle DIVIDE_BY_ZERO errors in Databricks SQLhttps://learn.microsoft.com/en-us/azure/databricks/error-messages/divide-by-zero-error-class
Reference Databricks error conditions for programmatic handlinghttps://learn.microsoft.com/en-us/azure/databricks/error-messages/error-classes
Diagnose EWKB_PARSE_ERROR issues in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/ewkb-parse-error-error-class
Diagnose EWKT_PARSE_ERROR issues in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/ewkt-parse-error-error-class
Diagnose GEOJSON_PARSE_ERROR issues in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/geojson-parse-error-error-class
Resolve GROUP_BY_AGGREGATE errors in Databricks SQLhttps://learn.microsoft.com/en-us/azure/databricks/error-messages/group-by-aggregate-error-class
Handle H3_INVALID_CELL_ID errors in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/h3-invalid-cell-id-error-class
Handle H3_INVALID_GRID_DISTANCE_VALUE errors in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/h3-invalid-grid-distance-value-error-class
Handle H3_INVALID_RESOLUTION_VALUE errors in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/h3-invalid-resolution-value-error-class
Handle H3_NOT_ENABLED errors and tier requirementshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/h3-not-enabled-error-class
Understand INSUFFICIENT_TABLE_PROPERTY errors in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/insufficient-table-property-error-class
Resolve INVALID_ARRAY_INDEX errors in Databricks SQLhttps://learn.microsoft.com/en-us/azure/databricks/error-messages/invalid-array-index-error-class
Resolve INVALID_ARRAY_INDEX_IN_ELEMENT_AT errorshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/invalid-array-index-in-element-at-error-class
Resolve MISSING_AGGREGATION errors in Databricks SQLhttps://learn.microsoft.com/en-us/azure/databricks/error-messages/missing-aggregation-error-class
Understand ROW_COLUMN_ACCESS errors in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/row-column-access-error-class
Map Databricks errors to SQLSTATE codeshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/sqlstates
Resolve TABLE_OR_VIEW_NOT_FOUND errors in Databricks SQLhttps://learn.microsoft.com/en-us/azure/databricks/error-messages/table-or-view-not-found-error-class
Fix UNRESOLVED_ROUTINE function resolution errors in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/unresolved-routine-error-class
Handle UNSUPPORTED_TABLE_OPERATION errors in Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/unsupported-table-operation-error-class
Handle UNSUPPORTED_VIEW_OPERATION errors in Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/unsupported-view-operation-error-class
Troubleshoot WKB_PARSE_ERROR geometry parsing in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/wkb-parse-error-error-class
Troubleshoot WKT_PARSE_ERROR geometry parsing in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/error-messages/wkt-parse-error-error-class
Troubleshoot Mosaic AI Agent Evaluation issueshttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/agent-evaluation/troubleshooting
Troubleshoot and debug Databricks AI agent deploymentshttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/agent-framework/debug-agent
Troubleshoot common issues in Databricks Genie spaceshttps://learn.microsoft.com/en-us/azure/databricks/genie/troubleshooting
Resolve common Databricks Auto Loader issueshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/cloud-object-storage/auto-loader/faq
Diagnose and fix Databricks Confluence ingestion issueshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/confluence-troubleshoot
Troubleshoot Dynamics 365 data ingestion with Lakeflow Connecthttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/d365-troubleshoot
Troubleshoot Google Ads connector ingestion issueshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/google-ads-troubleshoot
Troubleshoot Google Analytics raw data ingestion issueshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/google-analytics-troubleshoot
Troubleshoot HubSpot connector ingestion problemshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/hubspot-troubleshoot
Troubleshoot Jira connector authentication and OAuth issueshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/jira-troubleshoot
Troubleshoot Meta Ads Lakeflow ingestion issueshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/meta-ads-troubleshoot
Troubleshoot MySQL ingestion with Lakeflow Connecthttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/mysql-troubleshoot
Troubleshoot PostgreSQL ingestion with Lakeflow Connecthttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/postgresql-troubleshoot
Troubleshoot Salesforce ingestion with Lakeflow Connecthttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/salesforce-troubleshoot
Diagnose and fix Databricks ServiceNow connector issueshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/servicenow-troubleshoot
Troubleshoot Microsoft SharePoint ingestion in Lakeflowhttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/sharepoint-troubleshoot
Troubleshoot SQL Server ingestion with Lakeflow Connecthttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/sql-server-troubleshoot
Troubleshoot TikTok Ads connector in Lakeflowhttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/tiktok-ads-troubleshoot
Troubleshoot Workday HCM connector in Lakeflowhttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/workday-hcm-troubleshoot
Diagnose and fix Databricks Workday connector issueshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/workday-reports-troubleshoot
Troubleshoot Databricks Zendesk Support connector errorshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/zendesk-support-troubleshoot
Handle Zerobus Ingest errors and retrieshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/zerobus-errors
Use logging to troubleshoot Databricks init scriptshttps://learn.microsoft.com/en-us/azure/databricks/init-scripts/logs
Troubleshoot and repair Lakeflow Job failureshttps://learn.microsoft.com/en-us/azure/databricks/jobs/repair-job-failures
Monitor and troubleshoot Lakeflow pipelineshttps://learn.microsoft.com/en-us/azure/databricks/ldp/observability
Use query history to debug and optimize pipelineshttps://learn.microsoft.com/en-us/azure/databricks/ldp/query-history
Recover pipelines from streaming checkpoint corruptionhttps://learn.microsoft.com/en-us/azure/databricks/ldp/recover-streaming
Troubleshoot Databricks Feature Store issues and limitationshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/feature-store/troubleshooting-and-limitations
Debug common issues in Databricks Model Serving endpointshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/model-serving/model-serving-debug
Diagnose and resolve Databricks model serving timeoutshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/model-serving/model-serving-timeouts
Monitor Lakebase system operations and healthhttps://learn.microsoft.com/en-us/azure/databricks/oltp/projects/operations
Troubleshoot failing Spark jobs and removed executorshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/failing-spark-jobs
Diagnose and fix Spark memory issues on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/spark-memory-issues
Troubleshoot Databricks Partner Connect issueshttps://learn.microsoft.com/en-us/azure/databricks/partner-connect/troubleshoot
Troubleshoot common Databricks Git folders errorshttps://learn.microsoft.com/en-us/azure/databricks/repos/errors-troubleshooting
Fetch cursor rows and handle SQLSTATE in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/control-flow/fetch-stmt
Use GET DIAGNOSTICS for SQL error handling in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/control-flow/get-diagnostics-stmt
Open cursors and handle errors with OPEN in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/control-flow/open-stmt
Re-raise handled conditions with RESIGNAL in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/control-flow/resignal-stmt
Raise custom conditions with SIGNAL in Databricks SQLhttps://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/control-flow/signal-stmt
Validate UTF-8 strings and handle INVALID_UTF8_STRINGhttps://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/functions/validate_utf8
Understand Databricks SQL query performance insightshttps://learn.microsoft.com/en-us/azure/databricks/sql/user/queries/performance-insights
Use query history UI to debug Databricks SQLhttps://learn.microsoft.com/en-us/azure/databricks/sql/user/queries/query-history
Interpret Databricks SQL query profiles for performancehttps://learn.microsoft.com/en-us/azure/databricks/sql/user/queries/query-profile

Best Practices

TopicURL
Tag Databricks resources for cost attribution and trackinghttps://learn.microsoft.com/en-us/azure/databricks/admin/account-settings/usage-detail-tags
Use Databricks default compute policy families effectivelyhttps://learn.microsoft.com/en-us/azure/databricks/admin/clusters/policy-families
Apply identity best practices in Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/admin/users-groups/best-practices
Apply best practices for Databricks serverless workspaceshttps://learn.microsoft.com/en-us/azure/databricks/admin/workspace/serverless-workspaces-best-practices
Migrate Databricks library installs from init scriptshttps://learn.microsoft.com/en-us/azure/databricks/archive/compute/libraries-init-scripts
Apply best practices for Databricks compute policieshttps://learn.microsoft.com/en-us/azure/databricks/archive/compute/policies-best-practices
Use DBIO for transactional writes to cloud storage in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/archive/legacy/dbio-commit
Optimize skewed joins in Databricks using skew hintshttps://learn.microsoft.com/en-us/azure/databricks/archive/legacy/skew-join
Apply Azure Databricks platform administration best practiceshttps://learn.microsoft.com/en-us/azure/databricks/cheat-sheet/administration
Optimize BI performance with Databricks SQL warehouseshttps://learn.microsoft.com/en-us/azure/databricks/cheat-sheet/bi-serving
Prepare and model data for high-performance BI on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/cheat-sheet/bi-serving-data-prep
Configure Databricks SQL warehouses for optimal BI servinghttps://learn.microsoft.com/en-us/azure/databricks/cheat-sheet/bi-serving-sql-serving
Follow best practices for Azure Databricks compute creationhttps://learn.microsoft.com/en-us/azure/databricks/cheat-sheet/compute
Implement best practices for Azure Databricks production jobshttps://learn.microsoft.com/en-us/azure/databricks/cheat-sheet/jobs
Best practices for Power BI dashboards on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/cheat-sheet/power-bi
Apply Databricks compute configuration recommendationshttps://learn.microsoft.com/en-us/azure/databricks/compute/cluster-config-best-practices
Use flexible node types for reliable Databricks computehttps://learn.microsoft.com/en-us/azure/databricks/compute/flexible-node-types
Apply best practices for Databricks poolshttps://learn.microsoft.com/en-us/azure/databricks/compute/pool-best-practices
Apply serverless compute best practices in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/compute/serverless/best-practices
Optimize data loading on Databricks Serverless GPU computehttps://learn.microsoft.com/en-us/azure/databricks/compute/serverless/sgc-dataloading
Track experiments and monitor Serverless GPU workloads with MLflowhttps://learn.microsoft.com/en-us/azure/databricks/compute/serverless/sgc-tracking-observability
Tune Databricks SQL warehouses for BI workloadshttps://learn.microsoft.com/en-us/azure/databricks/compute/sql-warehouse/bi-workload-settings
Control large interactive queries with Query Watchdoghttps://learn.microsoft.com/en-us/azure/databricks/compute/troubleshooting/query-watchdog
Optimize Databricks dashboard performance with cachinghttps://learn.microsoft.com/en-us/azure/databricks/dashboards/caching
Apply observability best practices for Databricks jobs and pipelineshttps://learn.microsoft.com/en-us/azure/databricks/data-engineering/observability-best-practices
Apply schema evolution strategies in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/data-engineering/schema-evolution
Best practices for UDFs in Unity Catalog ABAC policieshttps://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/abac/udf-best-practices
Apply Unity Catalog best practices for data governancehttps://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/best-practices
Monitor fairness and bias for Databricks classification modelshttps://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/data-quality-monitoring/data-profiling/fairness-bias
Update Databricks jobs after Unity Catalog upgradehttps://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/jobs-update
Work with legacy Hive metastore database objectshttps://learn.microsoft.com/en-us/azure/databricks/database-objects/hive-metastore
Apply safe usage patterns for DBFS roothttps://learn.microsoft.com/en-us/azure/databricks/dbfs/dbfs-root
Use and migrate off DBFS mounts safelyhttps://learn.microsoft.com/en-us/azure/databricks/dbfs/mounts
Apply best practices for DBFS and Unity Cataloghttps://learn.microsoft.com/en-us/azure/databricks/dbfs/unity-catalog
Optimize Delta Sharing egress costshttps://learn.microsoft.com/en-us/azure/databricks/delta-sharing/manage-egress
Apply Delta Lake best practices on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/delta/best-practices
Use liquid clustering for Delta layouthttps://learn.microsoft.com/en-us/azure/databricks/delta/clustering
Add custom metadata to Databricks tableshttps://learn.microsoft.com/en-us/azure/databricks/delta/custom-metadata
Improve queries with Delta data skippinghttps://learn.microsoft.com/en-us/azure/databricks/delta/data-skipping
Use deletion vectors to speed up Delta updateshttps://learn.microsoft.com/en-us/azure/databricks/delta/deletion-vectors
Safely drop or replace tables in Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/delta/drop-table
Use Delta table history and time travel safelyhttps://learn.microsoft.com/en-us/azure/databricks/delta/history
Optimize Delta table layout with OPTIMIZEhttps://learn.microsoft.com/en-us/azure/databricks/delta/optimize
Handle Delta Lake limitations when using AWS S3https://learn.microsoft.com/en-us/azure/databricks/delta/s3-limitations
Use selective overwrite patterns with Delta Lakehttps://learn.microsoft.com/en-us/azure/databricks/delta/selective-overwrite
Control Delta data file size on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/delta/tune-file-size
Use VACUUM to remove stale Delta fileshttps://learn.microsoft.com/en-us/azure/databricks/delta/vacuum
Optimize VARIANT queries with shreddinghttps://learn.microsoft.com/en-us/azure/databricks/delta/variant-shredding
Apply Databricks-recommended CI/CD workflows and patternshttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/ci-cd/best-practices
List Databricks cluster policy families via CLIhttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/reference/policy-families-commands
Best practices for secure and performant Databricks Appshttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-apps/best-practices
Test Scala code using Databricks Connect and ScalaTesthttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-connect/scala/testing
Run Python tests on Databricks via VS Codehttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/vscode-ext/pytest
Choose patterns for external access to Databricks datahttps://learn.microsoft.com/en-us/azure/databricks/external-access/
Choose between volumes and workspace files in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/files/files-recommendations
Customize AI judges for Databricks Agent Evaluationhttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/agent-evaluation/advanced-agent-eval
Design effective evaluation sets for Databricks agentshttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/agent-evaluation/evaluation-set
Synthetically generate agent evaluation setshttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/agent-evaluation/synthesize-evaluation-set
Build and evaluate Databricks retrieval agentshttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/tutorials/agent-framework-notebook
Measure RAG performance with Databricks metricshttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/tutorials/ai-cookbook/evaluate-assess-performance
Create evaluation sets for Databricks RAG appshttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/tutorials/ai-cookbook/evaluate-define-quality
Evaluate and monitor RAG apps on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/tutorials/ai-cookbook/fundamentals-evaluation-monitoring-rag
Optimize Databricks RAG application qualityhttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/tutorials/ai-cookbook/quality-overview
Improve Databricks RAG chain qualityhttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/tutorials/ai-cookbook/quality-rag-chain
Configure Genie Code custom instructionshttps://learn.microsoft.com/en-us/azure/databricks/genie-code/instructions
Best practices for effective Genie Code promptshttps://learn.microsoft.com/en-us/azure/databricks/genie-code/tips
Evaluate Genie spaces using benchmarkshttps://learn.microsoft.com/en-us/azure/databricks/genie/benchmarks
Curate effective Azure Databricks Genie spaceshttps://learn.microsoft.com/en-us/azure/databricks/genie/best-practices
Build Genie knowledge stores for accurate responseshttps://learn.microsoft.com/en-us/azure/databricks/genie/knowledge-store
Use trusted assets to provide verified Genie answershttps://learn.microsoft.com/en-us/azure/databricks/genie/trusted-assets
Migrate existing Auto Loader streams to file eventshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/cloud-object-storage/auto-loader/migrating-to-file-events
Apply common Auto Loader data loading patternshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/cloud-object-storage/auto-loader/patterns
Configure Databricks Auto Loader for production workloadshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/cloud-object-storage/auto-loader/production
Configure Auto Loader with Unity Catalog for secure ingestionhttps://learn.microsoft.com/en-us/azure/databricks/ingestion/cloud-object-storage/auto-loader/unity-catalog
Apply common COPY INTO data loading patternshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/cloud-object-storage/copy-into/examples
Ingest local and internet files into Databrickshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/file-upload/
Download and store internet data in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/file-upload/download-internet-files
Apply common patterns to Lakeflow ingestion pipelineshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/common-patterns
Perform full refreshes of Lakeflow target tableshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/full-refresh
Analyze Lakeflow ingestion costs with billing tableshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/monitor-costs
Perform ongoing maintenance for Lakeflow pipelineshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/pipeline-maintenance
Operate and maintain PostgreSQL Lakeflow ingestion pipelineshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/postgresql-maintenance
Optimize incremental ingestion of Salesforce formula fieldshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/salesforce-formula-fields
Use init scripts to customize Databricks clustershttps://learn.microsoft.com/en-us/azure/databricks/init-scripts/
Reference external files safely in Databricks init scriptshttps://learn.microsoft.com/en-us/azure/databricks/init-scripts/referencing-files
Test applications using Databricks JDBC Driver (Simba)https://learn.microsoft.com/en-us/azure/databricks/integrations/jdbc/testing
Test applications using the Databricks ODBC Driverhttps://learn.microsoft.com/en-us/azure/databricks/integrations/odbc/testing
Configure compute resources for Lakeflow Jobs efficientlyhttps://learn.microsoft.com/en-us/azure/databricks/jobs/compute
Set up recurring, backfillable jobs with parametershttps://learn.microsoft.com/en-us/azure/databricks/jobs/how-to/create-recurring-job
Apply best practices to classic Lakeflow Jobshttps://learn.microsoft.com/en-us/azure/databricks/jobs/run-classic-jobs
Apply cost optimization best practices on Databricks lakehousehttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/cost-optimization/best-practices
Implement data and AI governance best practices on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/data-governance/best-practices
Apply interoperability and usability best practices on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/interoperability-and-usability/best-practices
Apply operational excellence best practices on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/operational-excellence/best-practices
Apply performance efficiency best practices on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/performance-efficiency/best-practices
Apply reliability best practices on Databricks lakehousehttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/reliability/best-practices
Implement security, compliance, and privacy best practices on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/security-compliance-and-privacy/best-practices
Optimize pipeline clusters with enhanced autoscalinghttps://learn.microsoft.com/en-us/azure/databricks/ldp/auto-scaling
Apply best practices for Lakeflow Spark Declarative Pipelineshttps://learn.microsoft.com/en-us/azure/databricks/ldp/best-practices
Use advanced AUTO CDC features and monitor processing metricshttps://learn.microsoft.com/en-us/azure/databricks/ldp/cdc-advanced
Apply development and testing best practices to Lakeflow pipelineshttps://learn.microsoft.com/en-us/azure/databricks/ldp/develop
Manage Python dependencies in Lakeflow pipelineshttps://learn.microsoft.com/en-us/azure/databricks/ldp/developer/external-dependencies
Apply advanced expectation patterns and scaling strategieshttps://learn.microsoft.com/en-us/azure/databricks/ldp/expectation-patterns
Reduce pipeline initialization latency by restructuring flowshttps://learn.microsoft.com/en-us/azure/databricks/ldp/fix-high-init
Develop and debug ETL pipelines with the Lakeflow Pipelines Editorhttps://learn.microsoft.com/en-us/azure/databricks/ldp/multi-file-editor
Use legacy notebook experience to develop Lakeflow pipelineshttps://learn.microsoft.com/en-us/azure/databricks/ldp/notebook-devex
Optimize stateful streaming with watermarks in pipelineshttps://learn.microsoft.com/en-us/azure/databricks/ldp/stateful-processing
Design CDC and snapshot patterns in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/ldp/what-is-change-data-capture
Restart Python process to refresh Databricks librarieshttps://learn.microsoft.com/en-us/azure/databricks/libraries/restart-python-process
Apply Hyperopt best practices and troubleshooting on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/automl-hyperparam-tuning/hyperopt-best-practices
Implement point-in-time correct feature joinshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/feature-store/time-series
Load and prepare data for ML on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/load-data/
Perform batch inference on Spark DataFrames with registered modelshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/model-inference/dl-model-inference
Configure Locust-based load tests for Databricks endpointshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/model-serving/configure-load-test
Validate models before Databricks Model Serving deploymenthttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/model-serving/model-serving-pre-deployment-validation
Optimize Databricks Model Serving endpoints for productionhttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/model-serving/production-optimization
Plan and execute load testing for Databricks serving endpointshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/model-serving/what-is-load-test
Tune and scale Ray clusters on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/ray/scale-ray
Implement distributed image inference on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/reference-solutions/images-etl-inference
Follow deep learning best practices on Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/train-model/dl-best-practices
Fine-tune Hugging Face models on a single GPU in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/train-model/huggingface/fine-tune-model
Prepare datasets for Hugging Face fine-tuning on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/train-model/huggingface/load-data
Model Unity Catalog metric view data effectivelyhttps://learn.microsoft.com/en-us/azure/databricks/metric-views/data-modeling/
Apply composability patterns in metric viewshttps://learn.microsoft.com/en-us/azure/databricks/metric-views/data-modeling/composability
Define joins in Databricks metric view YAMLhttps://learn.microsoft.com/en-us/azure/databricks/metric-views/data-modeling/joins
Use semantic metadata in Databricks metric viewshttps://learn.microsoft.com/en-us/azure/databricks/metric-views/data-modeling/semantic-metadata
Implement window measures in metric viewshttps://learn.microsoft.com/en-us/azure/databricks/metric-views/data-modeling/window-measures
Use materialization to optimize metric view querieshttps://learn.microsoft.com/en-us/azure/databricks/metric-views/materialization
Adapt existing Apache Spark workloads to Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/migration/spark
Align MLflow LLM judges with human evaluatorshttps://learn.microsoft.com/en-us/azure/databricks/mlflow3/genai/eval-monitor/align-judges
Developer workflow for MLflow code-based scorershttps://learn.microsoft.com/en-us/azure/databricks/mlflow3/genai/eval-monitor/custom-scorer-dev-workflow
Automatically optimize prompts with MLflow GEPAhttps://learn.microsoft.com/en-us/azure/databricks/mlflow3/genai/prompt-version-mgmt/prompt-registry/automatically-optimize-prompts
Evaluate and compare MLflow prompt versionshttps://learn.microsoft.com/en-us/azure/databricks/mlflow3/genai/prompt-version-mgmt/prompt-registry/evaluate-prompts
Use manual MLflow tracing for production GenAI appshttps://learn.microsoft.com/en-us/azure/databricks/mlflow3/genai/tracing/app-instrumentation/manual-tracing/
Analyze GenAI traces for errors and performancehttps://learn.microsoft.com/en-us/azure/databricks/mlflow3/genai/tracing/observe-with-traces/analyze-traces
Run Databricks notebooks safely and efficientlyhttps://learn.microsoft.com/en-us/azure/databricks/notebooks/run-notebook
Monitor and analyze active Lakebase querieshttps://learn.microsoft.com/en-us/azure/databricks/oltp/projects/active-queries
Implement branch-based development in Lakebasehttps://learn.microsoft.com/en-us/azure/databricks/oltp/projects/dev-workflow-tutorial
Analyze Lakebase query performance historyhttps://learn.microsoft.com/en-us/azure/databricks/oltp/projects/query-performance
Follow Databricks performance optimization guidancehttps://learn.microsoft.com/en-us/azure/databricks/optimizations/
Use adaptive query execution on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/aqe
Use archival support for Delta on Azurehttps://learn.microsoft.com/en-us/azure/databricks/optimizations/archive-delta
Leverage cost-based optimizer in Databricks SQLhttps://learn.microsoft.com/en-us/azure/databricks/optimizations/cbo
Improve read performance with Databricks disk cachehttps://learn.microsoft.com/en-us/azure/databricks/optimizations/disk-cache
Speed up queries with dynamic file pruninghttps://learn.microsoft.com/en-us/azure/databricks/optimizations/dynamic-file-pruning
Optimize Delta MERGE with low shuffle mergehttps://learn.microsoft.com/en-us/azure/databricks/optimizations/low-shuffle-merge
Accelerate data access with predictive I/Ohttps://learn.microsoft.com/en-us/azure/databricks/optimizations/predictive-io
Tune Azure Databricks range join performancehttps://learn.microsoft.com/en-us/azure/databricks/optimizations/range-join
Diagnose Databricks Spark cost and performance in UIhttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/
Use Spark jobs timeline to debug Databricks workloadshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/jobs-timeline
Diagnose long-running Spark jobs in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/long-spark-stage
Analyze high I/O Spark stages in Databricks UIhttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/long-spark-stage-io
Debug skew and spill in Databricks Spark stageshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/long-spark-stage-page
Handle Databricks spot instance losses effectivelyhttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/losing-spot-instances
Resolve long Spark stages with a single taskhttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/one-spark-task
Debug slow Spark stages with low I/O in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/slow-spark-stage-low-io
Optimize many small Spark jobs on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/small-spark-jobs
Identify expensive reads in Databricks Spark DAGshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/spark-dag-expensive-read
Mitigate overloaded Spark driver on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/spark-driver-overloaded
Diagnose gaps between Spark jobs in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/spark-job-gaps
Detect unnecessary data rewriting in Databricks Spark writeshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/spark-ui-guide/spark-rewriting-data
Best practices for setting up Databricks Partner Connecthttps://learn.microsoft.com/en-us/azure/databricks/partner-connect/best-practice
Configure networking for Databricks Lakehouse Federationhttps://learn.microsoft.com/en-us/azure/databricks/query-federation/networking
Optimize performance of Databricks Lakehouse Federation querieshttps://learn.microsoft.com/en-us/azure/databricks/query-federation/performance-recommendations
Encrypt inter-node traffic in Databricks clustershttps://learn.microsoft.com/en-us/azure/databricks/security/keys/encrypt-otw
Optimize transformations on complex and nested data typeshttps://learn.microsoft.com/en-us/azure/databricks/semi-structured/complex-types
Use higher-order functions to process arrays in Databricks SQLhttps://learn.microsoft.com/en-us/azure/databricks/semi-structured/higher-order-functions
Use VOID (NULL) type correctly in Databricks SQLhttps://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/data-types/null-type
Work with OBJECT type and VARIANT schemas in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/data-types/object-type
Use TIMESTAMP_NTZ type and Delta support in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/data-types/timestamp-ntz-type
Use VARIANT type and Iceberg compatibility in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/data-types/variant-type
Collect table statistics with ANALYZE TABLE for optimizationhttps://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/sql-ref-syntax-aux-analyze-compute-statistics
Optimize Databricks SQL queries using hintshttps://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/sql-ref-syntax-qry-select-hints
Benchmark Databricks SQL with TPC-DS sample datasetshttps://learn.microsoft.com/en-us/azure/databricks/sql/tpcds-eval
Use Databricks SQL query caching for performancehttps://learn.microsoft.com/en-us/azure/databricks/sql/user/queries/query-caching
Use Databricks SQL query filters effectivelyhttps://learn.microsoft.com/en-us/azure/databricks/sql/user/queries/query-filters
Optimize queries using primary key constraints in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/sql/user/queries/query-optimization-constraints
Work with query parameters in Databricks SQLhttps://learn.microsoft.com/en-us/azure/databricks/sql/user/queries/query-parameters
Create and use query snippets in Databricks SQLhttps://learn.microsoft.com/en-us/azure/databricks/sql/user/queries/query-snippets
Use Structured Streaming checkpoints correctly on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/structured-streaming/checkpoints
Implement Delta Lake streaming reads and writes in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/structured-streaming/delta-lake
Choose Structured Streaming output modes on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/structured-streaming/output-mode
Configure Databricks Structured Streaming for production workloadshttps://learn.microsoft.com/en-us/azure/databricks/structured-streaming/production
Optimize stateless Structured Streaming queries on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/structured-streaming/stateless-streaming
Monitor Structured Streaming queries using Databricks toolshttps://learn.microsoft.com/en-us/azure/databricks/structured-streaming/stream-monitoring
Combine Unity Catalog with Structured Streaming workloadshttps://learn.microsoft.com/en-us/azure/databricks/structured-streaming/unity-catalog
Apply watermarks for efficient stateful streaminghttps://learn.microsoft.com/en-us/azure/databricks/structured-streaming/watermarks
Optimize partition discovery for Unity Catalog external tableshttps://learn.microsoft.com/en-us/azure/databricks/tables/external-partition-discovery
Analyze Databricks table size and storage costshttps://learn.microsoft.com/en-us/azure/databricks/tables/size
Aggregate data with batch, streaming, and viewshttps://learn.microsoft.com/en-us/azure/databricks/transform/aggregation
Design data models optimized for Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/transform/data-modeling
Use joins effectively in Databricks batch and streaminghttps://learn.microsoft.com/en-us/azure/databricks/transform/join
Optimize join performance for Azure Databricks workloadshttps://learn.microsoft.com/en-us/azure/databricks/transform/optimize-joins
Implement data cleaning and validation on Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/transform/validate
Optimize Mosaic AI Vector Search performancehttps://learn.microsoft.com/en-us/azure/databricks/vector-search/vector-search-best-practices
Design and run load tests for vector search endpointshttps://learn.microsoft.com/en-us/azure/databricks/vector-search/vector-search-endpoint-load-test
Improve Mosaic AI Vector Search retrieval qualityhttps://learn.microsoft.com/en-us/azure/databricks/vector-search/vector-search-retrieval-quality

Decision Making

TopicURL
Plan migration from Databricks Standard to Premium tierhttps://learn.microsoft.com/en-us/azure/databricks/admin/account-settings/standard-tier
Evaluate and create Azure Databricks serverless workspaceshttps://learn.microsoft.com/en-us/azure/databricks/admin/workspace/serverless-workspaces
Decide and migrate from dbx to Databricks bundleshttps://learn.microsoft.com/en-us/azure/databricks/archive/dev-tools/dbx/dbx-migrate
Migrate optimized LLM endpoints to provisioned throughputhttps://learn.microsoft.com/en-us/azure/databricks/archive/machine-learning/migrate-provisioned-throughput
Decide and migrate to Databricks Runtime 10.xhttps://learn.microsoft.com/en-us/azure/databricks/archive/runtime-release-notes/10.x-migration
Migrate workloads to Databricks Runtime 11.xhttps://learn.microsoft.com/en-us/azure/databricks/archive/runtime-release-notes/11.x-migration
Migrate workloads to Databricks Runtime 12.xhttps://learn.microsoft.com/en-us/azure/databricks/archive/runtime-release-notes/12.x-migration
Plan migration to Databricks Runtime 13.xhttps://learn.microsoft.com/en-us/azure/databricks/archive/runtime-release-notes/13.x-migration
Plan migration to Databricks Runtime 14.xhttps://learn.microsoft.com/en-us/azure/databricks/archive/runtime-release-notes/14.x-migration
Use Databricks Runtime 6.4 Extended Support strategicallyhttps://learn.microsoft.com/en-us/azure/databricks/archive/runtime-release-notes/6.4x
Plan migration to Databricks Runtime 7.3 LTShttps://learn.microsoft.com/en-us/azure/databricks/archive/runtime-release-notes/7.3-migration
Migrate workloads from Databricks Runtime 6.x to 7.xhttps://learn.microsoft.com/en-us/azure/databricks/archive/runtime-release-notes/7.x-migration
Plan migration to Databricks Runtime 9.1 LTShttps://learn.microsoft.com/en-us/azure/databricks/archive/runtime-release-notes/9.1-migration
Plan migration of Databricks workloads to Spark 3.xhttps://learn.microsoft.com/en-us/azure/databricks/archive/spark-3.x-migration/
Migrate from Deep Learning Pipelines to newer Databricks MLhttps://learn.microsoft.com/en-us/azure/databricks/archive/spark-3.x-migration/deep-learning-pipelines
Select and manage the default Unity Catalog cataloghttps://learn.microsoft.com/en-us/azure/databricks/catalogs/default
Select the right Databricks compute type for workloadshttps://learn.microsoft.com/en-us/azure/databricks/compute/choose-compute
Decide when and how to use GPU Databricks computehttps://learn.microsoft.com/en-us/azure/databricks/compute/gpu
Decide when to use Databricks pools vs serverlesshttps://learn.microsoft.com/en-us/azure/databricks/compute/pool-index
Plan Databricks SQL warehouse sizing and queuinghttps://learn.microsoft.com/en-us/azure/databricks/compute/sql-warehouse/warehouse-behavior
Choose between Databricks SQL warehouse typeshttps://learn.microsoft.com/en-us/azure/databricks/compute/sql-warehouse/warehouse-types
Choose and configure Azure Databricks data connectionshttps://learn.microsoft.com/en-us/azure/databricks/connect/
Plan and execute upgrade to Unity Cataloghttps://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/upgrade/
Choose between Delta Sharing, Marketplace, and Clean Roomshttps://learn.microsoft.com/en-us/azure/databricks/data-sharing/
Choose Delta Lake protocol and featureshttps://learn.microsoft.com/en-us/azure/databricks/delta/feature-compatibility
Choose local development tools for Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/
Migrate from legacy to new Databricks CLI versionshttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/migrate
Manage Databricks account budget policies via CLIhttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/reference/account-budget-policy-commands
Configure Databricks account budgets using CLIhttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/reference/account-budgets-commands
Manage Databricks account usage dashboards via CLIhttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/reference/account-usage-dashboards-commands
Choose appropriate compute size for Databricks Appshttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-apps/compute-size
Migrate Python projects to new Databricks Connect runtimeshttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-connect/python/migrate
Migrate from legacy to new Scala Databricks Connecthttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-connect/scala/migrate
Choose and use Databricks SDKs for automationhttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/sdks
Select SQL connectors and tools for Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/sql-drivers-tools
Decide between CDKTF and Databricks Terraform providerhttps://learn.microsoft.com/en-us/azure/databricks/dev-tools/terraform/cdktf
Select Unity Catalog integration approach by enginehttps://learn.microsoft.com/en-us/azure/databricks/external-access/integrations
Migrate Databricks Community Edition to Free Editionhttps://learn.microsoft.com/en-us/azure/databricks/getting-started/ce-migration
Choose between Databricks Free Edition and free trialhttps://learn.microsoft.com/en-us/azure/databricks/getting-started/free-trial-vs-free-edition
Choose between Auto Loader directory listing and file notificationhttps://learn.microsoft.com/en-us/azure/databricks/ingestion/cloud-object-storage/auto-loader/file-detection-modes
Plan migration of existing data to Delta Lake on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/ingestion/data-migration/
Migrate from Simba Spark to Databricks ODBC Driverhttps://learn.microsoft.com/en-us/azure/databricks/integrations/odbc/migration
Migrate from Spark Submit task to supported Lakeflow taskshttps://learn.microsoft.com/en-us/azure/databricks/jobs/spark-submit
Select a development language for Databrickshttps://learn.microsoft.com/en-us/azure/databricks/languages/overview
Choose between triggered and continuous pipeline modeshttps://learn.microsoft.com/en-us/azure/databricks/ldp/pipeline-mode
Migrate online feature tables to Lakebasehttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/feature-store/migrate-from-online-tables
Migrate Databricks models and workflows to Unity Cataloghttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/manage-model-lifecycle/migrate-to-uc
Upgrade Databricks ML workflows to Unity Cataloghttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/manage-model-lifecycle/upgrade-workflows
Choose Databricks options for batch model inferencehttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/model-inference/
Migrate from legacy MLflow serving to Mosaic AI Model Servinghttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/model-serving/migrate-model-serving
Decide when to use Spark vs. Ray on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/ray/spark-ray-overview
Decide when to use distributed XGBoost with Ray on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/sgc-examples/tutorials/sgc-raytune-xgboost
Decide when and how to use distributed training on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/train-model/distributed-training/
Plan migration of data applications to Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/migration/
Assess options for migrating ETL pipelines to Databrickshttps://learn.microsoft.com/en-us/azure/databricks/migration/etl
Choose a migration path from Parquet to Delta Lakehttps://learn.microsoft.com/en-us/azure/databricks/migration/parquet-to-delta-lake
Migrate enterprise data warehouses to the Databricks lakehousehttps://learn.microsoft.com/en-us/azure/databricks/migration/warehouse-to-lakehouse
Decide and migrate from Agent Evaluation to MLflow 3https://learn.microsoft.com/en-us/azure/databricks/mlflow3/genai/agent-eval-migration
Quick reference for migrating to MLflow 3https://learn.microsoft.com/en-us/azure/databricks/mlflow3/genai/agent-eval-migration-reference
Use Lakebase Postgres OLTP within the Databricks Lakehousehttps://learn.microsoft.com/en-us/azure/databricks/oltp/
Plan and adjust Lakebase instance capacityhttps://learn.microsoft.com/en-us/azure/databricks/oltp/instances/create/capacity
Evaluate Lakebase Postgres Autoscaling capabilities and use caseshttps://learn.microsoft.com/en-us/azure/databricks/oltp/projects/about
Choose Lakebase backup and restore methodshttps://learn.microsoft.com/en-us/azure/databricks/oltp/projects/backup-methods
Choose how to connect applications to Lakebase Postgreshttps://learn.microsoft.com/en-us/azure/databricks/oltp/projects/connect-application
Understand default autoscaling behavior for new Lakebase instanceshttps://learn.microsoft.com/en-us/azure/databricks/oltp/upgrade-to-autoscaling
Choose and configure incremental refresh for materialized viewshttps://learn.microsoft.com/en-us/azure/databricks/optimizations/incremental-refresh
Choose pandas options and patterns on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/pandas/
Use pandas API on Spark effectively on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/pandas/pandas-on-spark
Migrate legacy Databricks query federation to Lakehouse Federationhttps://learn.microsoft.com/en-us/azure/databricks/query-federation/migrate
Choose appropriate Azure Databricks preview release typehttps://learn.microsoft.com/en-us/azure/databricks/release-notes/release-types
Decide on Databricks runtime and feature lifecycle supporthttps://learn.microsoft.com/en-us/azure/databricks/release-notes/runtime/databricks-runtime-ver
Interpret serverless DBU consumption by Azure Databricks SKUhttps://learn.microsoft.com/en-us/azure/databricks/resources/pricing
Decide between VARIANT and JSON strings for semi-structured datahttps://learn.microsoft.com/en-us/azure/databricks/semi-structured/variant-json-diff
Decide between Spark Connect and Spark Classichttps://learn.microsoft.com/en-us/azure/databricks/spark/connect-vs-classic
Choose between SparkR and sparklyr on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/sparkr/sparkr-vs-sparklyr
Migrate to the latest Databricks SQL REST APIhttps://learn.microsoft.com/en-us/azure/databricks/sql/dbsql-api-latest
Choose synchronous vs asynchronous state checkpointing in Databrickshttps://learn.microsoft.com/en-us/azure/databricks/structured-streaming/async-checkpointing
Optimize and manage Mosaic AI Vector Search costshttps://learn.microsoft.com/en-us/azure/databricks/vector-search/vector-search-cost-management

Architecture & Design Patterns

TopicURL
Implement fan-in and fan-out patterns in Lakeflow pipelineshttps://learn.microsoft.com/en-us/azure/databricks/data-engineering/fan-in-fan-out
Design multi-agent supervisor systems with Agent Brickshttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/agent-bricks/multi-agent-supervisor
Build Databricks multi-agent orchestrator appshttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/agent-framework/multi-agent-apps
Create Genie-based multi-agent systems on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/agent-framework/multi-agent-genie
Build non-conversational Databricks AI agents with MLflowhttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/agent-framework/non-conversational-agents
Implement AI agent memory with Databricks Lakehousehttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/agent-framework/stateful-agents
Implement AI agent memory on Databricks Model Servinghttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/agent-framework/stateful-agents-model-serving
Apply Databricks design patterns for AI agentshttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/guide/agent-system-design-patterns
Design and tune Databricks RAG inference chainshttps://learn.microsoft.com/en-us/azure/databricks/generative-ai/tutorials/ai-cookbook/fundamentals-inference-chain-rag
Architect cost-optimized Databricks lakehouse solutionshttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/cost-optimization/
Design data and AI governance architecture for the lakehousehttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/data-governance/
Apply guiding architectural principles for Databricks lakehousehttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/guiding-principles
Architect interoperability and usability for Databricks lakehousehttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/interoperability-and-usability/
Architect operational excellence for the Databricks lakehousehttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/operational-excellence/
Architect performance efficiency for Databricks lakehousehttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/performance-efficiency/
Use Databricks lakehouse reference architectures on Azurehttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/reference
Architect reliability for the Databricks data lakehousehttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/reliability/
Apply Databricks well-architected lakehouse frameworkhttps://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/well-architected
Apply Databricks data lakehouse architecture patternhttps://learn.microsoft.com/en-us/azure/databricks/lakehouse/
Apply medallion lakehouse architecture on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/lakehouse/medallion
Replicate external RDBMS tables to Databricks using AUTO CDChttps://learn.microsoft.com/en-us/azure/databricks/ldp/database-replication
Choose Databricks ML model deployment patternshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/mlops/deployment-patterns
Design LLMOps workflows on Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/mlops/llmops
Implement MLOps workflows on Azure Databrickshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/mlops/mlops-workflow
Choose and train deep-learning recommender models on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/machine-learning/train-recommender-models
Use Lakebase branches for safe database evolutionhttps://learn.microsoft.com/en-us/azure/databricks/oltp/projects/branches
Understand Lakebase autoscaling, branches, and read replicashttps://learn.microsoft.com/en-us/azure/databricks/oltp/projects/core-concepts
Design high availability for Lakebase Postgres computeshttps://learn.microsoft.com/en-us/azure/databricks/oltp/projects/high-availability
Scale reads with Lakebase Postgres read replicashttps://learn.microsoft.com/en-us/azure/databricks/oltp/projects/read-replicas
Understand and apply Databricks catalog federationhttps://learn.microsoft.com/en-us/azure/databricks/query-federation/catalog-federation
Plan Hive metastore federation with Unity Cataloghttps://learn.microsoft.com/en-us/azure/databricks/query-federation/hms-federation-concepts
Choose patterns for modeling semi-structured data on Databrickshttps://learn.microsoft.com/en-us/azure/databricks/semi-structured/
Decide when to partition Databricks tableshttps://learn.microsoft.com/en-us/azure/databricks/tables/partitions
Choose interactive vs non-interactive transactionshttps://learn.microsoft.com/en-us/azure/databricks/transactions/transaction-modes

> related_skills --same-repo

> azure-well-architected

Expert guidance for designing, assessing, and optimizing Azure workloads using Azure Well Architected. Covers design review checklists, recommendations, design principles, tradeoffs, service guides, workload patterns, and assessment questions. Use when architecting new solutions, reviewing existing workloads, or applying Well-Architected principles.

> azure-web-pubsub

Expert knowledge for Azure Web PubSub development including troubleshooting, best practices, decision making, architecture & design patterns, limits & quotas, security, configuration, integrations & coding patterns, and deployment. Use when building, debugging, or optimizing Azure Web PubSub applications. Not for Azure SignalR Service (use azure-signalr-service), Azure Event Hubs (use azure-event-hubs), Azure Service Bus (use azure-service-bus), Azure Relay (use azure-relay).

> azure-web-application-firewall

Expert knowledge for Azure Web Application Firewall development including troubleshooting, best practices, decision making, architecture & design patterns, limits & quotas, security, configuration, integrations & coding patterns, and deployment. Use when building, debugging, or optimizing Azure Web Application Firewall applications. Not for Azure Application Gateway (use azure-application-gateway), Azure Front Door (use azure-front-door), Azure Firewall (use azure-firewall), Azure DDos Protectio

> azure-vpn-gateway

Expert knowledge for Azure VPN Gateway development including troubleshooting, best practices, decision making, architecture & design patterns, limits & quotas, security, configuration, integrations & coding patterns, and deployment. Use when building, debugging, or optimizing Azure VPN Gateway applications. Not for Azure Virtual Network (use azure-virtual-network), Azure Virtual WAN (use azure-virtual-wan), Azure ExpressRoute (use azure-expressroute), Azure Application Gateway (use azure-applica

┌ stats

installs/wk0
░░░░░░░░░░
github stars425
██████████
first seenMar 17, 2026
└────────────

┌ repo

MicrosoftDocs/Agent-Skills
by MicrosoftDocs
└────────────

┌ tags

└────────────