Enrich Metadata and Discovery of Lakehouse Data

ARC123 — Enrich Metadata and Discovery of Lakehouse Data: Challenge Lab

Estimated time: 45 minutes

# Enrich Metadata and Discovery of Lakehouse Data Challenge Lab Follow the execution sequence below exactly as shown to successfully complete all tasks. ## 🚀 Step-by-Step Raw Execution (Recommended) ### 🔹 Step 1 — Execute Command (1 of 2) 1. Open **Google Cloud Shell**. 2. Copy and paste **Command (1 of 2)** into the terminal. 3. Press **Enter**. This command will automatically: - ✅ Enable all required Google Cloud APIs. - ✅ Create the **ecommerce** BigQuery dataset. - ✅ Create the requi

GREEN='\e[1;32m'
CYAN='\e[1;36m'
YELLOW='\e[1;33m'
BLUE='\e[1;34m'
MAGENTA='\e[1;35m'
RESET='\e[0m'
BOLD='\e[1m'

clear
echo -e "${CYAN}${BOLD}"
cat << "EOF"
  ____       _     _ _            __    ___            
 / __ \     | |   (_) |          / _|  / _ \           
| |  | |_ __| |__  _| |_   ___  | |_  | | | |_ __  ___ 
| |  | | '__| '_ \| | __| / _ \ |  _| | | | | '_ \/ __|
| |__| | |  | |_) | | |_ | (_) || |   | |_| | |_) \__ \
 \____/|_|  |_.__/|_|\__| \___/ |_|    \___/| .__/|___/
                                            | |        
                                            |_|        
EOF
echo -e "${RESET}"
echo -e "${MAGENTA}${BOLD}>>> ORBIT OF OPS: ARC123 TERMINAL AUTOMATION INITIALIZED (PART 1) <<<${RESET}\n"

export PROJECT_ID=$(gcloud config get-value project 2>/dev/null)
echo -e "${YELLOW}${BOLD}[*] Project ID: ${PROJECT_ID}${RESET}\n"

echo -e "${BLUE}${BOLD}[Orbit of Ops] Step 1: Enabling Required APIs...${RESET}"
gcloud services enable datacatalog.googleapis.com bigqueryconnection.googleapis.com dataplex.googleapis.com --quiet

echo -e "${CYAN}${BOLD}[Orbit of Ops] Step 2: Creating BigQuery Dataset...${RESET}"
bq mk --location=US ecommerce 2>/dev/null || true

echo -e "${YELLOW}${BOLD}[Orbit of Ops] Step 3: Creating Cloud Resource Connection...${RESET}"
bq mk --connection --location=US --project_id=$PROJECT_ID --connection_type=CLOUD_RESOURCE customer_data_connection 2>/dev/null || true

echo -e "${MAGENTA}${BOLD}[Orbit of Ops] Step 4: Granting Service Account Storage Permissions...${RESET}"
export SERVICE_ACCOUNT=$(bq show --format=json --connection $PROJECT_ID.US.customer_data_connection | jq -r '.cloudResource.serviceAccountId')
gcloud projects add-iam-policy-binding $PROJECT_ID --member=serviceAccount:$SERVICE_ACCOUNT --role=roles/storage.objectViewer --quiet

echo -e "\n${GREEN}${BOLD}>>> PART 1 COMPLETE! Proceed to Command (2 of 2). <<<${RESET}"

GREEN='\e[1;32m'
CYAN='\e[1;36m'
YELLOW='\e[1;33m'
BLUE='\e[1;34m'
MAGENTA='\e[1;35m'
RESET='\e[0m'
BOLD='\e[1m'

export PROJECT_ID=$(gcloud config get-value project 2>/dev/null)

echo -e "${BLUE}${BOLD}[Orbit of Ops] Step 5: Creating BigLake Table Definition...${RESET}"
bq mkdef --autodetect --connection_id=$PROJECT_ID.US.customer_data_connection --source_format=CSV gs://$PROJECT_ID-bucket/customer-online-sessions.csv > /tmp/tabledef.json

echo -e "${CYAN}${BOLD}[Orbit of Ops] Step 6: Creating External Lakehouse Table...${RESET}"
bq mk --external_table_definition=/tmp/tabledef.json --project_id=$PROJECT_ID ecommerce.customer_online_sessions 2>/dev/null || true

rm /tmp/tabledef.json 2>/dev/null

echo -e "\n${GREEN}${BOLD}>>> INFRASTRUCTURE COMPLETE! Follow the UI steps below to finish Task 3. <<<${RESET}"