{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "yh5jzMGT9Dnh"
   },
   "source": [
    "***We recommend working with this notebook on Google Colab***\n",
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/ridatadiscoverycenter/riddc-jbook/blob/main/riddc/notebooks/fox-kemper/volume_viewer_converter.ipynb)\n",
    "\n",
    "# Visualize OSOM data using the OSOM 3D volume viewer\n",
    "\n",
    "\n",
    "<img src=\"https://github.com/ridatadiscoverycenter/riddc-jbook/raw/main/riddc/notebooks/fox-kemper/images/volume-viewer-2.png\" >\n",
    "\n",
    "The following example goes through how to export OSOM datasets and visualize them in 3D using the Volume Viewer application.\n",
    "**NOTE: These scripts create a large amount of data. We recommend using your Google Drive to save them.**\n",
    "\n",
    "You can use the Volume Viewer either from you computer or using Brown's Oscar HPC.\n",
    "\n",
    "## 1. Get the Volume Viewer Application\n",
    "\n",
    "You can either dowload the Volume Viewer and work locally (option a), or load it as a module in Oscar and work there (option b, recommended).\n",
    "\n",
    "### a. Download Volume Viewer\n",
    "\n",
    "If you wish to work locally, you can find the latest version of the Volume Viewer application on [this link](https://github.com/brown-ccv/VR-Volumeviewer/releases). Download the file `volume-viewer.zip` and unzip it on your local machine.\n",
    "\n",
    "\n",
    "### b. Using Oscar HPC\n",
    "\n",
    "Log in to Oscar using [OOD](https://ood.ccv.brown.edu/) and open the desktop application with at least 1 GPU.\n",
    "\n",
    "<img src=\"https://raw.githubusercontent.com/ridatadiscoverycenter/riddc-jbook/main/riddc/notebooks/fox-kemper/images/ood-desktop-session.png\" >\n",
    "\n",
    "Go to the terminal inside the virtual desktop and type the following commands:\n",
    "\n",
    "`module load volumeviewer/2.0 gcc/10.2`\n",
    "\n",
    "`VR-Volumeviewer`\n",
    "\n",
    "## 2. Mount Google Drive\n",
    "\n",
    "Execute the rest of the commands below directly in this notebook to connect to your Google Drive\n",
    "\n",
    "**NOTE: The following command will open a pop-up window requesting access to your Google Drive file system**\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "GCpPVMBlYJnx"
   },
   "outputs": [],
   "source": [
    "\n",
    "from google.colab import drive\n",
    "drive.mount('/content/gdrive/', force_remount=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "yBxXaGPG9Cpf"
   },
   "source": [
    "Optional: If you experience any issues and want to verify that the notebook can access your Google Drive file system, execute the following code. It should display a list of files located in your Google Drive."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "gsVRTnfzaGvE"
   },
   "outputs": [],
   "source": [
    "%ls gdrive/MyDrive"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2_Bn8otEzO0L"
   },
   "source": [
    "Create a folder in Google Drive to store the configuration, osom-data, and results files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Y9wjAAO5z0JS"
   },
   "outputs": [],
   "source": [
    "%mkdir ./gdrive/MyDrive/osom3d/\n",
    "%mkdir ./gdrive/MyDrive/osom3d/grid-data/\n",
    "%mkdir ./gdrive/MyDrive/osom3d/configurations/\n",
    "%mkdir ./gdrive/MyDrive/osom3d/osom-data/\n",
    "%mkdir ./gdrive/MyDrive/osom3d/results/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HDwcgzexF_6g"
   },
   "source": [
    "\n",
    "## 3. Download Grid File\n",
    "\n",
    "Now, we need to download the nc grid file to map our global data points to the application coordinates. Execute the following command.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "1Be7a0FwEr2I"
   },
   "outputs": [],
   "source": [
    "! wget -P ./gdrive/MyDrive/osom3d/grid-data/ https://github.com/brown-ccv/osom-data-volume-viewer/raw/main/grid-data/osom_grid4_mindep_smlp_mod7-netcdf4.nc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5e5VaaDauQ6d"
   },
   "source": [
    "## 4. Install Volume Converter package\n",
    "\n",
    "We have to install the `volume_data_converter` library that converts the OSOM nc files to Volume Viewer data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "uwtTzNQFbsFV"
   },
   "outputs": [],
   "source": [
    "! pip install volume-data-converter@git+https://github.com/brown-ccv/volume-data-converter.git@v3.0.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ANYL6euUuvwe"
   },
   "source": [
    "## 4. Download OSOM data from ERDDAP\n",
    "\n",
    "First we need to acquire data from the ERDDAP server. In order to configure a connection and set up constrains for your search, create a json containing the connection url, protocol type and dataset id. You can also specify a time range and coordinates. The following is an example on how to create a json file with the mentioned properties. \n",
    "The command will create a file in your virtual local folder. We will use the result output in the next step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "AMgj9F91cEuc"
   },
   "outputs": [],
   "source": [
    "import json \n",
    "dictionary ={ \n",
    "  \"erddap_connection\": {\"server\": \"https://pricaimcit.services.brown.edu/erddap\",\"protocol\": \"griddap\",\"dataset_id\": \"model_data_57db_4a85_81d9\"}, \n",
    "  \"erddap_constrainsts\": {\"variables\":[\"SalinityBottom\",\"SalinitySurface\"],  \"time>=\": \"2019-12-30T00:00:00Z\",\n",
    "        \"time<=\": \"2019-12-31T00:00:00Z\",\n",
    "        \"time_step\": 1,\n",
    "        \"eta_rho>=\": 0,\n",
    "        \"eta_rho<=\": 1099,\n",
    "        \"eta_rho_step\": 1,\n",
    "        \"xi_rho>=\": 0,\n",
    "        \"xi_rho<=\": 999,\n",
    "        \"xi_rho_step\": 1\n",
    "        }, \n",
    "} \n",
    "\n",
    "with open('./gdrive/MyDrive/osom3d/configurations/erddap_connection.json', 'w') as f:\n",
    "    json_object = json.dumps(dictionary, indent = 4) \n",
    "    f.write(json_object)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "gsx2Kmhuwv-F"
   },
   "source": [
    "After running the previous script, if we list the files in our Google Drive folder, we can see the recently created `'erddap_connection.json'` file.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "No4P_adz0t6v"
   },
   "outputs": [],
   "source": [
    "%ls ./gdrive/MyDrive/osom3d/configurations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xaVjrrSw1xte"
   },
   "source": [
    "Using the `volume_data_converter` library, we will pass `erddap_connection.json` to the `my_erddap_query` function to start the download process and retrieve the data from ERDDAP."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ghE2zS3Rb-xl"
   },
   "outputs": [],
   "source": [
    "from volume_data_converter.erddap_query import erddap_query as my_erddap_query\n",
    "my_erddap_query(\"./gdrive/MyDrive/osom3d/osom-data/\",\"./gdrive/MyDrive/osom3d/configurations/erddap_connection.json\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8AGEdXSG0Tun"
   },
   "source": [
    "If we list the files in our Google Drive folder, we can see the ouput file `model_data_57db_4a85_81d9.nc` which is the name of the dataset id specified in the `erddap_connection.json`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "8shshwQs5AG5",
    "outputId": "a784fec7-9dcb-4606-e72c-6bdbec8e484e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "model_data_57db_4a85_81d9.nc\n"
     ]
    }
   ],
   "source": [
    "%ls ./gdrive/MyDrive/osom3d/osom-data"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "wisVJhej1bv-"
   },
   "source": [
    "## 5. Convert NC-OSOM data file to Volume Viewer file format\n",
    "\n",
    "Now we will create another json file containing the configuration to convert the nc file to raw data that the Volume Viewer app will read. The following code is an example of the type of configuration file the `volume_data_converter` expects to work with:\n",
    "\n",
    "- `osom_grid_file`: Path to the nc grid file (the first downloaded file)\n",
    "- `osom_data_file`: Path to the OSOM nc data file (`model_data_57db_4a85_81d9.nc`)\n",
    "- `ouput_folder`: Path where the resulting package will be saved (a place in your Google Drive file system)\n",
    "- `data_descriptors`: A list of  OSOM variables to export to the Volume Viewer (e.g., `[\"temperature\", \"salt\"]`)\n",
    "- `time_frames`: If the dataset has multiple time frames, it will export all of them (time sequence). Otherwise, specify the required frame. (i.e: Null, [0,1],[5,6,7])\n",
    "- `layers`: List of OSOM-supported layers: `all`(default), `surface`, `bottom`. (e.g., `[\"surface\", \"bottom\"]`)\n",
    "- `to_texture_atlas`:  Flag to convert the volume viewer dataset to a texture atlas image (jpg) or 3D matrix strucure (.raw): `True` or `False` (default).\n",
    "\n",
    "### IMPORTANT:\n",
    "MacOS users must set the `to_texture_atlas` flag to `True`. Windows and Linux systems support both (jpg and raw) modes.\n",
    "\n",
    "The following code will create the `volume_viewer_data.json` file with the given configuration.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "2ZSz1sJ2ck41"
   },
   "outputs": [],
   "source": [
    "import json \n",
    "dictionary ={ \n",
    "  \"osom_grid_file\": \"./gdrive/MyDrive/osom3d/grid-data/osom_grid4_mindep_smlp_mod7-netcdf4.nc\",\n",
    "  \"osom_data_file\": \"./gdrive/MyDrive/osom3d/osom-data/model_data_57db_4a85_81d9.nc\",\n",
    "  \"output_folder\": \"./gdrive/MyDrive/osom3d/results/\",\n",
    "  \"data_descriptor\" : [\"SalinityBottom\"],\n",
    "  \"time_frames\": [0],\n",
    "  \"layers\" : [\"bottom\"],\n",
    "  \"to_texture_atlas\": False\n",
    "} \n",
    "\n",
    "with open('./gdrive/MyDrive/osom3d/configurations/volume_viewer_data.json', 'w') as f:\n",
    "    json_object = json.dumps(dictionary, indent = 4) \n",
    "    f.write(json_object)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "qLtckh8o_u0f"
   },
   "source": [
    " Finally, we import the `create_osom_data` script and pass the `volume_viewer_data.json` file to start the conversion process.\n",
    "\n",
    " ==NOTE: If you get errors with the below command, please restart the runtime using the menu options `Runtime -> restart runtime `=="
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "pl9qiK41cwX1"
   },
   "outputs": [],
   "source": [
    "from volume_data_converter.create_volume_viewer_data import create_osom_data as my_create_volume_viewer_data\n",
    "my_create_volume_viewer_data(\"./gdrive/MyDrive/osom3d/configurations/volume_viewer_data.json\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "gKY-6EkeAWwP"
   },
   "source": [
    "Take into account that the defined ouput file path points to `./gdrive/myDrive/osom`. That is a location inside **your** Google Drive. From there you can download to the environment in which you are running the Volume Viewer application."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "hnDKgpRE3jq-",
    "outputId": "8057762b-04e0-4d6c-f895-68181d5ca063"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "data/  labels/  osom-loader.txt  textures/\n"
     ]
    }
   ],
   "source": [
    "%ls ./gdrive/MyDrive/osom3d/results/osom-data-SalinityBottom"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "y18TazFKBiDx"
   },
   "source": [
    "## 6. Load data into the Volume Viewer\n",
    "\n",
    "\n",
    "### If you are working on Brown-Oscar\n",
    "\n",
    "load the Google Chrome module and open a chrome browser to access your Google Drive. Open a new terminal tab and type the following to get a Chrome browser:\n",
    "\n",
    "`module load chrome`\n",
    "`google-chrome --no-sandbox`\n",
    "\n",
    "When the browser opens, sign in to Google and navigate to your Google Drive. In the left side panel you will find the result output folder (see image below). Right-click on the folder and select **Download**.\n",
    "\n",
    "![google drive folder](https://raw.githubusercontent.com/ridatadiscoverycenter/riddc-jbook/main/riddc/notebooks/fox-kemper/images/Screen%20Shot%20google%20drive%20file%20system.png)\n",
    "\n",
    "\n",
    "- Unzip the osom3d file and navigate to the open Volume Viewer application.\n",
    "\n",
    "- If you are on Oscar, make sure the osom3d folder gets extracted to `/gpfs/home/<username>/Downloads/osom3d`\n",
    "\n",
    "- Click on `load file` and navigate to `/gpfs/home/<username>/Downloads/osom3d/results/osom-data-SalinityBottom`. Double click the `osom-loader.txt` file.\n",
    "\n",
    "![volume viewer clip](https://raw.githubusercontent.com/ridatadiscoverycenter/riddc-jbook/main/riddc/notebooks/fox-kemper/images/clip-volume-viewer.gif)\n",
    "\n",
    "\n",
    "- After a few seconds, the volume will be loaded.\n",
    "\n",
    "![volume viewer loaded](https://raw.githubusercontent.com/ridatadiscoverycenter/riddc-jbook/main/riddc/notebooks/fox-kemper/images/volume-viewer-4.png)\n",
    "\n",
    "- Check the box next to TF1 to turn on the transfer function and have fun playing around with the Volume Viewer!\n",
    "\n",
    "![transfer function clip](https://raw.githubusercontent.com/ridatadiscoverycenter/riddc-jbook/main/riddc/notebooks/fox-kemper/images/transfer_function_example.png)\n",
    "\n",
    "- For any support or questions with this product, please reach out to The Center for Computation and Visualization via any of the mediums listed [on our help page](https://ccv.brown.edu/help)  \n",
    "* Email us at support@ccv.brown.edu\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.9"
  },
  "vscode": {
   "interpreter": {
    "hash": "9432b285d580434117d6b346f70478cdd15262786192a944c0a29a25c909307d"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}