User Guides
MAAP User Documentation
https://maap-project.readthedocs.io/en/latest/
Pilot User Documentation
FAQ
- How do I register my algorithm to MAS?
- How do I delete my algorithm from MAS?
- How do I view algorithms already registered in MAS?
- How do I run a DPS job?
- How do I check the status of my DPS job?
- How do I get the results from my DPS job?
- How do I name my workspace something useful?
- How do I share my workspace with another MAAP user?
- How do I copy my EarthData search into my Jupyter notebook?
- How do I import granules over from my EarthData search into my Jupyter notebook?
- Which files in my workspace are persistent, and which ones are only available in my workspace
Platform Technical Documentation
ADE Workspace
Create a Workspace
Workspaces
Creating workspaces
Navigate to the side panel tab Workspaces
and click Add Workspace
. Name your workspace, choose your
stack and the amount of RAM you will need. Make sure you pay close attention to the Organization you start the
workspace in. This is something that cannot be changed, and will affect who you are able to share the workspace
with! (more info here).
If you want to add a git project, you may do this when creating the workspace, or add after creation.
Choosing a stack
There are many stack options that are defaults of Eclipse Che, but in order to use the custom MAAP jupyter interface,
please select the stack maap-jupyter-ide
.
Sharing workspaces
See documentation on sharing.
here
For more info on things you should not do, seeShare a Workspace
Sharing
Organizations
We are utilizing Eclipse Che’s Organizations feature to allow users to share workspaces with other MAAP users. If a workspace is created in an organization, a user can easily share that workspace with other members of the organization.
All users are in one shared organization
Users can be in multiple organizations, but there is one organization, called maap-users
that
all users are in. This allows for users to be able to see all the possible MAAP users to
share with.
All users have their own personal organization they can customize
Users all also have their own personal organization. Each user is the admin of their own organization so they can easily curate groups and individuals that they share their work with often.
To add a user to your personal organization, use the Add Member
button. Now you can share
any workspace in your personal organization with them!
To create a group of users you want to share with, create a sub-organization in your personal organization. In order to add someone to your sub-organization, you first must add them to your personal organization. Organizations can be created in layers of nesting sub-organizations, but for all organizations, you can only add members of it's parent organization.
As you can see, your personal organization is really just a sub-organization of
maap-users
/
For example - if I wanted to create a group frequent collaborators of mine.
- Add each of them to my personal organization
maap-users\<my-name-here>
. - Create a sub-organization under
maap-users\<my-name-here>
, lets's call itmaap-users\<my-name-here>\my-collaborators
. - Add each of my collaborators to my new sub-organization.
Creating workspaces in an organization's namespace
All workspaces are created under the namespace of an organization. NOTE: Once a workspace is created under an organization it cannot be moved to a different organization.
To specify which organization you want a workspace to be a part of, specify using the dropdown on workspace creation.
Once a workspace is created, it is not inherent that all memebers of that organization
are able to see it. You must share with them after it is created. This can be found in
workspace's Share
tab.
More documentation
For more documentation help on using organizations, check out Che's guide. It is not specific to MAAP's implementation, but most of the info will still apply!
Sharing Data
Users who have access to a workspace have access to all the files contained in that workspace.
All users have their own personal s3-bucket folder mounted in the FileBrowser home
In /projects
, each user has a personal s3-hosted folder with the same name as their CAS username. Files in this
folder are automatically uploaded to s3 and will accessible from any workspace a user signs into.
The intention of this mounted folder is that you can use this to share data with others, and also store files you want to access across different workspaces. It is not intended that you do all of your work in this directory. Because this directory is mounted to s3, you will notice that processes are slower when working in this directory.
Generating a shareable link to s3-hosted files
Users can create a shareable link for any files in their folder that is hosted on s3. To do this, go to
Command Palette
-> User
-> Get Presigned S3 URL
and enter the relative path to the file you want to share.
The link will expire after 12 hours.
See Notebook Magics for how to get the presigned s3 url from an inline notebook command.
ADE Projects
Creating a Project
Creating a project
The first step in getting started with projects in MAAP is going to MAAP's GitLab and creating a new project. When you create a MAAP account, an account is automatically configured for you on MAAP's (GitLab instance)[https://repo.maap-project.org/].
To sign in, click on Sign in with CAS
, then use your URS or ESA login. This will direct
you to your projects.
Click on Create a project
. Name your project and select it's visibility level:
- Projects will default to
private
, meaning only you can see it, and if you want to allow another user to contribute, you will have to explicitly add them in the gitlab interface. NOTE: Adding a user to a workspace that has a project in it does not automatically give them access to your gitlab project, you must also add them to the private project if you want to allow them to make changes. - Projects set to
internal
can be seen by any authenticated MAAP user. - Projects set to
public
can be seen by anyone, regardless of whether they are a MAAP user or not.
Once you have created a project, copy the https url for the project the can be found under the clone button's dropdown. You will use this to add your project to you Che workspace.
Adding a project to your workspace
Projects can be added to a workspace during or after workspace creation. If a project is added after a workspace exists, the workspace must be restarted when the projects are added. After restart, your projects will automatically appear in your workspace.
Updating your project from inside your workspace
All projects imported from GitLab can be found in /projects
.
Using the left side panel in the jupyter interface, you can push changes to your gitlab project.
If you are more comfortable using the command line to interact with git, you do not need to use the side panel. It will work the same way in the terminal, once you navigate to the project's filepath.
When you are ready to update your project with your changes, navigate to the git panel. Add the files you want to change to the list of list of staged changes. Then write a commit message, and click the check. Now you need to push your changes by selecting the push changes button on the toolbar.
If you want to check your commit history, look at branches, and confirm your updates have been pushed, you can see this on the history tab.
Adding a Project to your Workspace
Adding a project to your workspace
Projects can be added to a workspace during or after workspace creation. If a project is added after a workspace exists, the workspace must be restarted when the projects are added. After restart, your projects will automatically appear in your workspace.
Updating your Project from Inside your Workspace
Updating your project from inside your workspace
All projects imported from GitLab can be found in /projects
.
Using the left side panel in the jupyter interface, you can push changes to your gitlab project.
If you are more comfortable using the command line to interact with git, you do not need to use the side panel. It will work the same way in the terminal, once you navigate to the project's filepath.
When you are ready to update your project with your changes, navigate to the git panel. Add the files you want to change to the list of list of staged changes. Then write a commit message, and click the check. Now you need to push your changes by selecting the push changes button on the toolbar.
If you want to check your commit history, look at branches, and confirm your updates have been pushed, you can see this on the history tab.
ADE EarthData Search
Finding Data
Finding data
To find data, open the Earthdata Search client using the Data Search
tab on the toolbar. When the window
opens as a tab in jupyter, take the tour for instructions on how to use the platform to search for data.
To see what search parameters you have selected in your EDSC session, select
View Selected Search Parameters
on the dropdown menu.
Adding Data to a New Notebook
Adding data to a notebook
Once you have selected the data that you want to work with, it's time to transfer that information to a notebook. There are 2 options for this - you can either insert a search query, or the results of that query (a list of links to the data) to your notebook.
To choose and insert, select an option in the dropdown menu:
In order to run the search command, and make other helpful calls for interacting with the data, you need to have the MAAP library (maap-py) imported into your notebook. On the notebook toolbar there is a helpful button that will insert all the MAAP defaults.
Visualizing Data in a Notebook
Visualizing data in a notebook
To visualize this data, check out ipycmc.
ADE Common Mapping Client
Getting Started
ipycmc
ipycmc
provides a data mapping and plotting package for datasets ingested into the MAAP ecosystem. With it you can create a widget in your python notebook where you can visualize and interact with rasterized geospatial data on a 2D and 3D map. From the widget you can select specific regions and time periods and retrieve dynamically calculated analyses and plot them inline with an interactive charting widget.
Getting Started
import the module - This will load ipycmc into your notebook for use.
import ipycmc
create a mapping widget - This will create and display a widget inline in your notebook. This widget includes some controls for navigating the map, switching between 2D and 3D, changing the displayed time, and manipulating layer displays.
w = ipycmc.MapCMC()
w
load data into the mapping widget - This will load all the layers specified in this WMS server's capabilities into the widget for display. You can then use the controls in the widget to display some of those layers on the map, then select an area and generate commands for plotting the data corresponding to your selection in the notebook below.
w.load_layer_config(
"http://geoserver.biomass-maap.com/geoserver/gwc/service/wms?REQUEST=GetCapabilities",
"wms/xml"
)
For loading 3D data, see documentation for load_layer_config below.
API Documentation
ipycmc
Module: .MapCMC()
- creates a new instance of a MapCMC widget. This widget widget is meant to be dislayed inline in the notebook and functions called on the instance will not perform as expected until the widget is displayed.
Example
w = ipycmc.MapCMC()
w
.retrieve_data(plotType, startDate, endDate, ds, geometry)
- retrieves data meant for plotting from the MAAP data analysis platform
-
plotType
: string - ['timeseries', 'timeAvgMap', 'hovmoller_lat', 'hovmoller_lon'] the type of plot data to be calculated -
startDate
: string - ISO 8601 string specifying the start time bound of the analysis -
endDate
: string - ISO 8601 string specifying the end time bound of the analysis -
ds
: list<string> - list of dataset identifiers that should be included in this analysis -
geometry
: dict - dictionary of type, projection, and coordinates of the area that should be included in the analysis
Example
plotType = "timeseries"
startDate = "2019-08-25T07:00:00.000Z"
endDate = "2019-09-01T07:00:00.000Z"
ds = ["dataset_id"]
geometry = {"type":"Box","proj":"EPSG:4326","coordinateType":"Cartographic","coordinates":[-180,-90,180,90]}
data = ipycmc.retrieve_data(plotType, startDate, endDate, ds, geometry)
data
.plot_data(plotType, data)
- plots the data from retrieve_data()
inline in a charting widget
-
plotType
: string - ['timeseries', 'timeAvgMap', 'hovmoller_lat', 'hovmoller_lon'] the type of plot to be generated -
data
: dict - the output fromretieve_data()
for this plot type
Example
plotType = "timeseries"
startDate = "2019-08-25T07:00:00.000Z"
endDate = "2019-09-01T07:00:00.000Z"
ds = ["dataset_id"]
geometry = {"type":"Box","proj":"EPSG:4326","coordinateType":"Cartographic","coordinates":[-180,-90,180,90]}
data = ipycmc.retrieve_data(plotType, startDate, endDate, ds, geometry)
ipycmc.plot_data(plotType, data)
MapCMC
Class: NOTE: All methods of the MapCMC
class expect the widget to be rendered so make sure you display the widget before calling any of these methods beforehand like so
w = ipycmc.MapCMC()
w
.load_layer_config(url, handle_as, default_ops={})
- load a layer config into the mapping widget
-
url
: string - the url endpoint of the config (such as WMS or WMTS GetCapabilities endpoint) -
handle_as
: string - ["json", "wmts/xml", "wms/xml"] the type of config endpoint to be loaded -
default_ops
: dict - a dictionary of default options to apply to the loaded layers. If loading from a GIBS endpoint, you might use:{"handleAs": "GIBS_raster"}
.
Example
w.load_layer_config(
"https://gibs.earthdata.nasa.gov/wmts/epsg4326/best/wmts.cgi?SERVICE=WMTS&request=GetCapabilities",
"wmts/xml",
{"handleAs": "GIBS_raster"}
)
When loading 3D layers from MAAP's CMR you must include "json"
and {"handleAs": "vector-3d-tile"}
.
Example
w.load_layer_config(
"https://cmr.maap-project.org/search/concepts/G1200354094-NASA_MAAP.json”,
"json”,
{“handleAs": "vector-3d-tile"})
.set_date(date_str, format_str="")
- set the display date of the mapping widget
-
date_str
: string - date or datetime that the widget should be set to -
format_str
: string - the format the of the date string. See token formats for MomentJS: https://momentjs.com/docs/#/parsing/string-format/
Example
w.set_date("2019-Jan-03", "YYYY-MMM-DD")
.get_date()
- get the current widget display date as an ISO 8601 string
.get_layers()
- get the list of ingested layer information dictionaries
.get_area_selections()
- the the list of selected layer information dictionaries
.get_active_layers()
- the the list of layer information dictionaries for layers currently active on the map
API Documentation
API Documentation
ipycmc
Module: .MapCMC()
- creates a new instance of a MapCMC widget. This widget widget is meant to be dislayed inline in the notebook and functions called on the instance will not perform as expected until the widget is displayed.
Example
w = ipycmc.MapCMC()
w
.retrieve_data(plotType, startDate, endDate, ds, geometry)
- retrieves data meant for plotting from the MAAP data analysis platform
-
plotType
: string - ['timeseries', 'timeAvgMap', 'hovmoller_lat', 'hovmoller_lon'] the type of plot data to be calculated -
startDate
: string - ISO 8601 string specifying the start time bound of the analysis -
endDate
: string - ISO 8601 string specifying the end time bound of the analysis -
ds
: list<string> - list of dataset identifiers that should be included in this analysis -
geometry
: dict - dictionary of type, projection, and coordinates of the area that should be included in the analysis
Example
plotType = "timeseries"
startDate = "2019-08-25T07:00:00.000Z"
endDate = "2019-09-01T07:00:00.000Z"
ds = ["dataset_id"]
geometry = {"type":"Box","proj":"EPSG:4326","coordinateType":"Cartographic","coordinates":[-180,-90,180,90]}
data = ipycmc.retrieve_data(plotType, startDate, endDate, ds, geometry)
data
.plot_data(plotType, data)
- plots the data from retrieve_data()
inline in a charting widget
-
plotType
: string - ['timeseries', 'timeAvgMap', 'hovmoller_lat', 'hovmoller_lon'] the type of plot to be generated -
data
: dict - the output fromretieve_data()
for this plot type
Example
plotType = "timeseries"
startDate = "2019-08-25T07:00:00.000Z"
endDate = "2019-09-01T07:00:00.000Z"
ds = ["dataset_id"]
geometry = {"type":"Box","proj":"EPSG:4326","coordinateType":"Cartographic","coordinates":[-180,-90,180,90]}
data = ipycmc.retrieve_data(plotType, startDate, endDate, ds, geometry)
ipycmc.plot_data(plotType, data)
MapCMC
Class: NOTE: All methods of the MapCMC
class expect the widget to be rendered so make sure you display the widget before calling any of these methods beforehand like so
w = ipycmc.MapCMC()
w
.load_layer_config(url, handle_as, default_ops={})
- load a layer config into the mapping widget
-
url
: string - the url endpoint of the config (such as WMS or WMTS GetCapabilities endpoint) -
handle_as
: string - ["json", "wmts/xml", "wms/xml"] the type of config endpoint to be loaded -
default_ops
: dict - a dictionary of default options to apply to the loaded layers. If loading from a GIBS endpoint, you might use:{"handleAs": "GIBS_raster"}
Example
w.load_layer_config(
"https://gibs.earthdata.nasa.gov/wmts/epsg4326/best/wmts.cgi?SERVICE=WMTS&request=GetCapabilities",
"wmts/xml",
{"handleAs": "GIBS_raster"}
)
.set_date(date_str, format_str="")
- set the display date of the mapping widget
-
date_str
: string - date or datetime that the widget should be set to -
format_str
: string - the format the of the date string. See token formats for MomentJS: https://momentjs.com/docs/#/parsing/string-format/
Example
w.set_date("2019-Jan-03", "YYYY-MMM-DD")
.get_date()
- get the current widget display date as an ISO 8601 string
.get_layers()
- get the list of ingested layer information dictionaries
.get_area_selections()
- the the list of selected layer information dictionaries
.get_active_layers()
- the the list of layer information dictionaries for layers currently active on the map
ADE Jupyter Notebook Magics
Load the Inline Magics in Notebook
In-line Notebook Magics
Load the inline magics in notebook
WARNING: This will not work for a python script. This is for use in notebooks only.
You must enable the extension in the notebook: %load_ext dps_magic
You can run a command by entering %
followed by a command and its arguments into a notebook cell.
Available Magics
You can list the available magics by running %help
Output:
Magic Command | Description |
---|---|
%capabilities | get information about MAAP API services |
%list | list algorithms registered in MAS |
%describe | describe the selected algorithm |
%execute | submit a job to DPS using an algorithm registered in MAS |
%status | check the status of a submitted job |
%result | get the results for a completed job |
%delete_algorithm | remove a registered algorithm from MAS |
%delete_job | remove a completed job from DPS |
%s3_url | get a presigned s3 url for an object |
%help | print this help info |
Pass the argument "help" to a command for more specific instructions on how to use it:
%execute help
Output:
List
Cell:
%list
Output:
Algorithms:
example_plot_ubuntu:1.0
gdal_service:master
george_demo_ubuntu:master
george_esdis_demo_ubuntu:1.0
liz-plot2_ubuntu:1.0
plant_algo:master
plant_test:master
plot_algo:master
plot_demo3:master
plot_demo4:master
plot_demo_ubuntu:master
plot_example_ubuntu:1.0
plot_test2_ubuntu:master
plot_test3_ubuntu:master
plot_test5_ubuntu:master
plot_test_ubuntu:master
plot_ubuntu:master
Describe
Cell:
%describe help
Output:
Describe Algorithm Help
Check the inputs required for an algorithm stored in MAS. You need to know your algorithm name and version.
Example Describe Call:
%describe plot_algo:master
Example Dictionary Call:
d = {'algo_id':'plot_algo','version':'master'}
%describe $d
Cell:
%describe george_demo_ubuntu:master
OR
Cell:
d = {'algo_id':'george_demo_ubuntu','version':'master'}
%describe $d
Output:
Algorithm: george_demo_ubuntu
Version: master
Identifier: george_demo_ubuntu:master
Input
Title: pass_number
Identifier: pass_number
DataType: string
Input
Title: username
Identifier: username
DataType: string
Output: []
Execute
Cell:
%execute help
Output:
Execute Job Help
Execute a job through DPS. You need to know the parameters of your algorithm. Use %describe to check if you are unsure.
Example String Call:
%execute plot_algo:master(pass_number=3,username=liz)
Example Dictionary Call:
d = {'pass_number':'6','username':'liz','algo_id':'plot_algo','version':'master'}
%execute $d
Cell:
%execute plot_algo:master(pass_number=3,username=eyam)
OR
d = {'pass_number':'6','username':'eyam','algo_id':'plot_algo','version':'master'}
%execute $d
Output:
JobID is cd36b6ff-5477-4cbf-80b8-772a77a44c58
- make sure you use your username, or else the Job ID will not be saved anywhere else and will not be displayed in the jobs table
Status
Cell:
%status help
Output:
Job Status Help
Check the status of a job in DPS. You need to know your job ID.
Example Status Call:
%status ef6fde9e-0975-4556-b8a7-ee52e91d8e61
Cell:
%status cd36b6ff-5477-4cbf-80b8-772a77a44c58
Output:
JobID is cd36b6ff-5477-4cbf-80b8-772a77a44c58
Status: Succeeded
Result
Cell:
%result help
Output:
Job Result Help
Check the result of a completed job in DPS. You need to know your job ID.
Example Result Call:
%result ef6fde9e-0975-4556-b8a7-ee52e91d8e61
Cell:
%result cd36b6ff-5477-4cbf-80b8-772a77a44c58
Output:
Job Results | |
---|---|
JobID: | cd36b6ff-5477-4cbf-80b8-772a77a44c58 |
ProductName: | output-2019-09-13_00-21-53 |
Locations: | http://maapdev.s3.amazonaws.com/products/plot/v1.0/2019/09/13/output-2019-09-13_00-21-53 • s3://s3.amazonaws.com:80/maapdev/products/plot/v1.0/2019/09/13/output-2019-09-13_00-21-53 • https://s3.console.aws.amazon.com/s3/buckets/maapdev/products/plot/v1.0/2019/09/13/output-2019-09-13_00-21-53/?region=us-east-1&tab=overview |
- the call will return an error if the job was not successfully completed (includes started, revoked, failed, and deleted)
Delete
Cell:
%delete_job help
Output:
Delete Job Help
Delete a finished (completed or failed) job or queued job stored in DPS. You need to know your Job ID.
Example Delete Call:
%delete_job ef6fde9e-0975-4556-b8a7-ee52e91d8e61
Cell:
%delete_job 5bcac3c8-9958-4c54-bb73-99d5eba09879
Output:
JobID is 5bcac3c8-9958-4c54-bb73-99d5eba09879
Status: Deleted
Dismiss
Cell:
%dismiss help
Output:
Dismiss Job Help
Dismiss a running (started, NOT queued) job on DPS. You need to know your Job ID.
Example Dismiss call:
%dismiss_job ef6fde9e-0975-4556-b8a7-ee52e91d8e61'
Cell:
%dismiss c9737ace-b143-49cc-96e4-9a6a7665932d
Output:
JobID is c9737ace-b143-49cc-96e4-9a6a7665932d
Status: Dismissed
s3 Url
Cell:
%s3_url help
Output:
Presigned S3 Url Help
Get a presigned s3 url for an object. You need to know the path of the file relative to the bucket or your /projects directory.
Example s3_url call:
%s3_url eyam/file-on-bucket
Cell:
%s3_url eyam/file-on-bucket
Output:
https://maap-mount-dev.s3.amazonaws.com/eyam/file-on-bucket?AWSAccessKeyId=AKIAJTSO23RLALBN45NQ&Signature=vgK%2FvwuSSYc4WUf39o4KWtbdVtw%3D&Expires=1568351409
DPS and MAS Notebook Magic
404 - Not found
Document not foundADE Workspace and Data Collaboration
Organizations
Sharing
Organizations
We are utilizing Eclipse Che’s Organizations feature to allow users to share workspaces with other MAAP users. If a workspace is created in an organization, a user can easily share that workspace with other members of the organization.
All users are in one shared organization
Users can be in multiple organizations, but there is one organization, called maap-users
that
all users are in. This allows for users to be able to see all the possible MAAP users to
share with.
All users have their own personal organization they can customize
Users all also have their own personal organization. Each user is the admin of their own organization so they can easily curate groups and individuals that they share their work with often.
To add a user to your personal organization, use the Add Member
button. Now you can share
any workspace in your personal organization with them!
To create a group of users you want to share with, create a sub-organization in your personal organization. In order to add someone to your sub-organization, you first must add them to your personal organization. Organizations can be created in layers of nesting sub-organizations, but for all organizations, you can only add members of it's parent organization.
As you can see, your personal organization is really just a sub-organization of
maap-users
/
For example - if I wanted to create a group frequent collaborators of mine.
- Add each of them to my personal organization
maap-users\<my-name-here>
. - Create a sub-organization under
maap-users\<my-name-here>
, lets's call itmaap-users\<my-name-here>\my-collaborators
. - Add each of my collaborators to my new sub-organization.
Creating workspaces in an organization's namespace
All workspaces are created under the namespace of an organization. NOTE: Once a workspace is created under an organization it cannot be moved to a different organization.
To specify which organization you want a workspace to be a part of, specify using the dropdown on workspace creation.
Once a workspace is created, it is not inherent that all memebers of that organization
are able to see it. You must share with them after it is created. This can be found in
workspace's Share
tab.
More documentation
For more documentation help on using organizations, check out Che's guide. It is not specific to MAAP's implementation, but most of the info will still apply!
Sharing Data
Users who have access to a workspace have access to all the files contained in that workspace.
All users have their own personal s3-bucket folder mounted in the FileBrowser home
In /projects
, each user has a personal s3-hosted folder with the same name as their CAS username. Files in this
folder are automatically uploaded to s3 and will accessible from any workspace a user signs into.
The intention of this mounted folder is that you can use this to share data with others, and also store files you want to access across different workspaces. It is not intended that you do all of your work in this directory. Because this directory is mounted to s3, you will notice that processes are slower when working in this directory.
Generating a shareable link to s3-hosted files
Users can create a shareable link for any files in their folder that is hosted on s3. To do this, go to
Command Palette
-> User
-> Get Presigned S3 URL
and enter the relative path to the file you want to share.
The link will expire after 12 hours.
See Notebook Magics for how to get the presigned s3 url from an inline notebook command.
Sharing Data
Sharing Data
Users who have access to a workspace have access to all the files contained in that workspace.
All users have their own personal s3-bucket folder mounted in the FileBrowser home
In /projects
, each user has a personal s3-hosted folder with the same name as their CAS username. Files in this
folder are automatically uploaded to s3 and will accessible from any workspace a user signs into.
The intention of this mounted folder is that you can use this to share data with others, and also store files you want to access across different workspaces. It is not intended that you do all of your work in this directory. Because this directory is mounted to s3, you will notice that processes are slower when working in this directory.
Generating a shareable link to s3-hosted files
Users can create a shareable link for any files in their folder that is hosted on s3. To do this, go to
Command Palette
-> User
-> Get Presigned S3 URL
and enter the relative path to the file you want to share.
The link will expire after 12 hours.
See Notebook Magics for how to get the presigned s3 url from an inline notebook command.
ADE SSH
SSH Access
SSH
As an alternative to using the jupyter interface, you can ssh directly into the container that your workspace set up.
In order to get the IP and port information, navigate to the command palette of the jupyter interface. Find the
command Display SSH Info
(you can easily search for ssh) which will display the information you need. Your public
SSH key that you added to your account will be added to any workspace you create. If you did not upload an SSH key
to your profile, you will not be able to SSH in and must use the jupyter interface.
In the workspace administration section of che, you will see and SSH tab, nothing you do here will affect your workspace, keys generated here are not valid.
here
For more info on things you should not do, seeADE User Interface Do’s/Don’t
What Not to Do!
We tried to use as many out of the box technologies as possible in this project to get as many features available as possible. But with that comes a lot of features that are not relevant to our platform or have not been implemented on our side yet. As we continue to build MAAP this list should become shorter and shorter, but for now here are the things that you should not mess with.
If you happen to mess with one of these things, you'll probably be fine - you'll just end up with a bunch of errors you don't understand or just nothing happening when you click a button.
Side Panel
What's relevant: Workspaces, Organizations.
What's not: Stacks, Factories, Administration.
Workspace Administration
Adding a workspace
What to do
Configure the name, organization, RAM, and projects.
What NOT to do
Use the wrong stack. For now, only run the stack maap-jupyter-ide stable build
When adding projects, only add projects with the Git
tab. Do not use any of the other project tabs.
Managing a workspace
Here are the features allowed/not allowed on all of the workspace administration tabs.
Tab | Features |
---|---|
Overview | Do not bother with the Export buttons - these are not relevant. |
Projects | When adding projects, only add projects with the Git tab. Do not use any of the other project tabs. |
Machines | Do not add a machine! Do not play with any of the actions! The only thing you can do here is change the RAM. |
Installers | The Exec and SSH installers must be turned on here in order to get SSH access to your machine. All other installers are not relevant and should not be turned on. |
Servers | Do not touch! |
Env Variables | This is allowed! |
Volumes | Do not touch! |
Config | Do not touch! |
SSH | Do not touch! SSH has been handled for you and adding a key here will not work. |
Tools | Do not touch! |
Plugins | Do not touch! |
Share | Do this! All of the features should be implemented here. |
DPS Algorithms
List Algorithms
List Algorithms
A user can execute a job with any algorithm that has been published to the MAAP Algorithm Store (MAS). To view what algorithms are available, click on Command Palette
-> DPS
-> List Algorithms
. Algorithms are listed in the format <algorithm-name>:<version>
.
Users will be shown the same list in a dropdown menu when trying to execute a job.
Describe an Algorithm
Describe an Algorithm
A user can request more information about available algorithms. Click on Command Palette
-> DPS
-> Describe Algorithm
.
A popup window will appear, asking the user to select an available algorithm.
A second popup window will follow, listing the algorithm's registered name, version, identifier, any required inputs, and expected outputs.
Register an Algorithm
Register an Algorithm
Go to Command Palette
-> DPS
-> Register Algorithm
. The user is be shown a dropdown menu with available Python source files or Python notebook files.
After selecting the desired script, the user is prompted with a new dialog box to populate required information for registering the script as an algorithm. The name is pre-populated from the filename, and environment is set to ubuntu
, which most ADEs are running on.
Enter the command to run the script in run_cmd
(e.g. python3 plot.py
) and required parameters for the algorithm in inputs
(e.g. pass_number
), each on its own line. The algorithm name and description can be changed if desired. Click OK
to finish registering. The user is then given a response if the request for registration was successful, or if some error occurred.
NOTE: After registering a new algorithm, a corresponding docker image must build successfully in GitLab before the algorithm is available for job submission.
Delete an Algorithm
Delete an Algorithm
Go to Command Palette
-> DPS
-> Delete Algorithm
. The user is show a dropdown menu with available containerized algorithms to remove.
If at any point the user does not get the expected output or behavior, it is possible that user's session cookie has expired and will need to refresh the page.
DPS Job Submission
List Previous Jobs
List Previous Jobs
The Jobs
pane will need to be manually (re)loaded. This can be done by going to the Command Palette
-> DPS
-> Refresh Job List
. Once this happens, the Jobs
pane will display a scrollable, up-to-date table of the user's job history with Job Id, Job Status, and Algorithm, sorted by the most recently submitted job. From here, users can go directly to the s3-hosted results of a completed job or delete the currently viewed job.
Click on the row containing the job of interest for more information about inputs and results.
Execute a Job
Execute a Job
To execute a job, click on Command Palette
-> DPS
-> Execute DPS Job
.
A popup window will appear, asking the user to select an available algorithm. Next, the user will be prompted to enter the required inputs for the selected algorithm. Upon submission, a Job ID will be returned.
Note: The Job Submissions
must be manually reloaded to show the newly submitted job.
Check Job Status
List Previous Jobs
The Jobs
pane will need to be manually (re)loaded. This can be done by going to the Command Palette
-> DPS
-> Refresh Job List
. Once this happens, the Jobs
pane will display a scrollable, up-to-date table of the user's job history with Job Id, Job Status, and Algorithm, sorted by the most recently submitted job. From here, users can go directly to the s3-hosted results of a completed job or delete the currently viewed job.
Click on the row containing the job of interest for more information about inputs and results.
Check Job Results
Check Job Results
The easiest way to check a job's status is to find and click on it on the Jobs
pane.
Alternatively, if the job was submitted under a different user, and the Job ID is known, the user can check the job status by going to Command Palette
-> DPS
-> Get DPS Job Result
. A series of dialogs similar to Get Status
will popup.
Delete Job
Delete Job
The easiest way to delete a job is to view it on the Jobs
pane and click the Delete Job
button.
Deleting a job means removing any record of a job that has already been attempted and failed or completed. To delete a job, go to Command Palette
-> DPS
-> Delete DPS Job
. Similar to Get Status
and Get Results
, a popup will appear, prompting the user to enter the job id.
Dismiss Job
Dismiss Job
The easiest way to delete a job is to view it on the Jobs
pane and click the Delete Job
button.
Dismissing a job is to stop a job that is currently running already. To dismiss a job, go to Command Palette
-> DPS
-> Dismiss DPS Job
. A popup will appear, prompting the user to enter the job id.
If at any point the user does not get the expected output or behavior, it is possible that user's session cookie has expired and will need to refresh the page.