Databricks

db

Databricks is a user-friendly platform for working with large datasets. It is like a shared space where people who analyze data and those who work with it can all collaborate. Built on Apache Spark and hosted in the cloud, Databricks offers convenient access from anywhere. It features useful tools like collaborative notebooks for writing and sharing, task scheduling, and data exploration capabilities. Designed to streamline the entire data analytics process, Databricks covers everything from data preparation to model training and deployment. Plus, it supports various programming languages such as Python, SQL, R, and Scala, and seamlessly integrates with popular machine learning and data visualization tools.

CHARISMA can be utilized on Databricks. This method clones (copies) the original CHARISMA GitHub contents and save it into your workspace. Thus, users are free to modify, add, or remove the codes and data. Also, all the necessary Python packages are automatically installed, eliminating the need for users to install anything themselves.

  1. Log in into Databricks.

URL: https://dbc-76d86e80-ff34.cloud.databricks.com/

  1. Click Workspace.

2

  1. Click Create on the right top blue panel and click Git folder.

3

  1. Paste the following address into the “Git repository URL”:

https://github.com/TFHRCFASTNDElab/CHARISMA/

Then the Git provider and Git folder name will be automatically filled.

3

  1. Databricks does not allow Git operations on a GitHub repository larger than 200MB. Therefore, we need to specify only the directories we use. Enable “Sparse checkout mode” and specify the following directories:

environment
ground-penetrating-radar
new

  1. Click Create Git folder.


  1. Click the branch icon with “main”.

3

  1. Click “Branch: main” and change it as DataBricks.

3

  1. Exit the white screen after confirming the branch has set with Databricks.

3

  1. Click ground-penetrating-radar, docs, Rebar mapping, and notebooks.

3

  1. Click one of the notebooks, and type in your DOT email address, to specify the directory path.

3

  1. Request a computational resource by clicking Connect, and click Run all.

3