• Skip to primary navigation
  • Skip to main content
NASA MAAP Logo

NASA MAAP

Multimission Algorithm and Analytics Platform

  • News
  • Explore
    • Search Catalog
    • Visualize
  • Communities
  • Help
    • User Guides
    • Terms of Service
    • Open Policies
    • Bug/Feature Reports
  • Login
    • Profile
  • Show Search
Hide Search

User Guides

MAAP User Documentation

https://maap-project.readthedocs.io/en/latest/

Pilot User Documentation

Google Docs

FAQ

  • How do I register my algorithm to MAS?
  • How do I delete my algorithm from MAS?
  • How do I view algorithms already registered in MAS?
  • How do I run a DPS job?
  • How do I check the status of my DPS job?
  • How do I get the results from my DPS job?
  • How do I name my workspace something useful?
  • How do I share my workspace with another MAAP user?
  • How do I copy my EarthData search into my Jupyter notebook?
  • How do I import granules over from my EarthData search into my Jupyter notebook?
  • Which files in my workspace are persistent, and which ones are only available in my workspace

Platform Technical Documentation

ADE Workspace
Create a Workspace

Workspaces

Creating workspaces

Navigate to the side panel tab Workspaces and click Add Workspace. Name your workspace, choose your stack and the amount of RAM you will need. Make sure you pay close attention to the Organization you start the workspace in. This is something that cannot be changed, and will affect who you are able to share the workspace with! (more info here).

If you want to add a git project, you may do this when creating the workspace, or add after creation.

Choosing a stack

There are many stack options that are defaults of Eclipse Che, but in order to use the custom MAAP jupyter interface, please select the stack maap-jupyter-ide.

Sharing workspaces

See documentation on sharing.


For more info on things you should not do, see here
Share a Workspace

Sharing

Organizations

We are utilizing Eclipse Che’s Organizations feature to allow users to share workspaces with other MAAP users. If a workspace is created in an organization, a user can easily share that workspace with other members of the organization.

All users are in one shared organization

Users can be in multiple organizations, but there is one organization, called maap-users that all users are in. This allows for users to be able to see all the possible MAAP users to share with.

User's organizations

All users have their own personal organization they can customize

Users all also have their own personal organization. Each user is the admin of their own organization so they can easily curate groups and individuals that they share their work with often.

To add a user to your personal organization, use the Add Member button. Now you can share any workspace in your personal organization with them!

Adding a user to a personal organization

To create a group of users you want to share with, create a sub-organization in your personal organization. In order to add someone to your sub-organization, you first must add them to your personal organization. Organizations can be created in layers of nesting sub-organizations, but for all organizations, you can only add members of it's parent organization.

As you can see, your personal organization is really just a sub-organization of maap-users/

For example - if I wanted to create a group frequent collaborators of mine.

  1. Add each of them to my personal organization maap-users\<my-name-here>.
  2. Create a sub-organization under maap-users\<my-name-here>, lets's call it maap-users\<my-name-here>\my-collaborators.
  3. Add each of my collaborators to my new sub-organization.

Creating workspaces in an organization's namespace

All workspaces are created under the namespace of an organization. NOTE: Once a workspace is created under an organization it cannot be moved to a different organization.

To specify which organization you want a workspace to be a part of, specify using the dropdown on workspace creation.

Select organization in workspace creation

Once a workspace is created, it is not inherent that all memebers of that organization are able to see it. You must share with them after it is created. This can be found in workspace's Share tab.

Share workspace with organization members

More documentation

For more documentation help on using organizations, check out Che's guide. It is not specific to MAAP's implementation, but most of the info will still apply!

Sharing Data

Users who have access to a workspace have access to all the files contained in that workspace.

All users have their own personal s3-bucket folder mounted in the FileBrowser home

In /projects, each user has a personal s3-hosted folder with the same name as their CAS username. Files in this folder are automatically uploaded to s3 and will accessible from any workspace a user signs into.

The intention of this mounted folder is that you can use this to share data with others, and also store files you want to access across different workspaces. It is not intended that you do all of your work in this directory. Because this directory is mounted to s3, you will notice that processes are slower when working in this directory.

s3-hosted folder

Generating a shareable link to s3-hosted files

Users can create a shareable link for any files in their folder that is hosted on s3. To do this, go to Command Palette -> User -> Get Presigned S3 URL and enter the relative path to the file you want to share.
The link will expire after 12 hours.

s3-link-gif

See Notebook Magics for how to get the presigned s3 url from an inline notebook command.

ADE Projects
Creating a Project

Creating a project

The first step in getting started with projects in MAAP is going to MAAP's GitLab and creating a new project. When you create a MAAP account, an account is automatically configured for you on MAAP's (GitLab instance)[https://repo.maap-project.org/].

To sign in, click on Sign in with CAS, then use your URS or ESA login. This will direct you to your projects.

Welcome screen

Click on Create a project. Name your project and select it's visibility level:

  • Projects will default to private, meaning only you can see it, and if you want to allow another user to contribute, you will have to explicitly add them in the gitlab interface. NOTE: Adding a user to a workspace that has a project in it does not automatically give them access to your gitlab project, you must also add them to the private project if you want to allow them to make changes.
  • Projects set to internal can be seen by any authenticated MAAP user.
  • Projects set to public can be seen by anyone, regardless of whether they are a MAAP user or not.

Settings for your new project

Once you have created a project, copy the https url for the project the can be found under the clone button's dropdown. You will use this to add your project to you Che workspace.

Git clone url dropdown

Adding a project to your workspace

Projects can be added to a workspace during or after workspace creation. If a project is added after a workspace exists, the workspace must be restarted when the projects are added. After restart, your projects will automatically appear in your workspace.

Add project on workspace creation Add project to existing workspace

Updating your project from inside your workspace

All projects imported from GitLab can be found in /projects.

Using the left side panel in the jupyter interface, you can push changes to your gitlab project.

If you are more comfortable using the command line to interact with git, you do not need to use the side panel. It will work the same way in the terminal, once you navigate to the project's filepath.

When you are ready to update your project with your changes, navigate to the git panel. Add the files you want to change to the list of list of staged changes. Then write a commit message, and click the check. Now you need to push your changes by selecting the push changes button on the toolbar.

Stage/Add changed file Stage and commit Push changes

If you want to check your commit history, look at branches, and confirm your updates have been pushed, you can see this on the history tab.

Check git history

Adding a Project to your Workspace

Adding a project to your workspace

Projects can be added to a workspace during or after workspace creation. If a project is added after a workspace exists, the workspace must be restarted when the projects are added. After restart, your projects will automatically appear in your workspace.

Add project on workspace creation Add project to existing workspace

Updating your Project from Inside your Workspace

Updating your project from inside your workspace

All projects imported from GitLab can be found in /projects.

Using the left side panel in the jupyter interface, you can push changes to your gitlab project.

If you are more comfortable using the command line to interact with git, you do not need to use the side panel. It will work the same way in the terminal, once you navigate to the project's filepath.

When you are ready to update your project with your changes, navigate to the git panel. Add the files you want to change to the list of list of staged changes. Then write a commit message, and click the check. Now you need to push your changes by selecting the push changes button on the toolbar.

Stage/Add changed file Stage and commit Push changes

If you want to check your commit history, look at branches, and confirm your updates have been pushed, you can see this on the history tab.

Check git history

ADE EarthData Search
Finding Data

Finding data

To find data, open the Earthdata Search client using the Data Search tab on the toolbar. When the window opens as a tab in jupyter, take the tour for instructions on how to use the platform to search for data.

Open EDSC

To see what search parameters you have selected in your EDSC session, select View Selected Search Parameters on the dropdown menu.

Current search parameters

Adding Data to a New Notebook

Adding data to a notebook

Once you have selected the data that you want to work with, it's time to transfer that information to a notebook. There are 2 options for this - you can either insert a search query, or the results of that query (a list of links to the data) to your notebook.

To choose and insert, select an option in the dropdown menu: Paste search in notebook

In order to run the search command, and make other helpful calls for interacting with the data, you need to have the MAAP library (maap-py) imported into your notebook. On the notebook toolbar there is a helpful button that will insert all the MAAP defaults.

Insert MAAP Defaults

Visualizing Data in a Notebook

Visualizing data in a notebook

To visualize this data, check out ipycmc.

ADE Common Mapping Client
Getting Started

ipycmc

ipycmc provides a data mapping and plotting package for datasets ingested into the MAAP ecosystem. With it you can create a widget in your python notebook where you can visualize and interact with rasterized geospatial data on a 2D and 3D map. From the widget you can select specific regions and time periods and retrieve dynamically calculated analyses and plot them inline with an interactive charting widget.

Getting Started

import the module - This will load ipycmc into your notebook for use.

import ipycmc

create a mapping widget - This will create and display a widget inline in your notebook. This widget includes some controls for navigating the map, switching between 2D and 3D, changing the displayed time, and manipulating layer displays.

w = ipycmc.MapCMC()
w

load data into the mapping widget - This will load all the layers specified in this WMS server's capabilities into the widget for display. You can then use the controls in the widget to display some of those layers on the map, then select an area and generate commands for plotting the data corresponding to your selection in the notebook below.

w.load_layer_config(
    "http://geoserver.biomass-maap.com/geoserver/gwc/service/wms?REQUEST=GetCapabilities",
    "wms/xml"
)

For loading 3D data, see documentation for load_layer_config below.

API Documentation

Module: ipycmc

.MapCMC() - creates a new instance of a MapCMC widget. This widget widget is meant to be dislayed inline in the notebook and functions called on the instance will not perform as expected until the widget is displayed.

Example

w = ipycmc.MapCMC()
w

.retrieve_data(plotType, startDate, endDate, ds, geometry) - retrieves data meant for plotting from the MAAP data analysis platform

  • plotType: string - ['timeseries', 'timeAvgMap', 'hovmoller_lat', 'hovmoller_lon'] the type of plot data to be calculated
  • startDate: string - ISO 8601 string specifying the start time bound of the analysis
  • endDate: string - ISO 8601 string specifying the end time bound of the analysis
  • ds: list<string> - list of dataset identifiers that should be included in this analysis
  • geometry: dict - dictionary of type, projection, and coordinates of the area that should be included in the analysis

Example

plotType = "timeseries"
startDate = "2019-08-25T07:00:00.000Z"
endDate = "2019-09-01T07:00:00.000Z"
ds = ["dataset_id"]
geometry = {"type":"Box","proj":"EPSG:4326","coordinateType":"Cartographic","coordinates":[-180,-90,180,90]}
data = ipycmc.retrieve_data(plotType, startDate, endDate, ds, geometry)
data

.plot_data(plotType, data) - plots the data from retrieve_data() inline in a charting widget

  • plotType: string - ['timeseries', 'timeAvgMap', 'hovmoller_lat', 'hovmoller_lon'] the type of plot to be generated
  • data: dict - the output from retieve_data() for this plot type

Example

plotType = "timeseries"
startDate = "2019-08-25T07:00:00.000Z"
endDate = "2019-09-01T07:00:00.000Z"
ds = ["dataset_id"]
geometry = {"type":"Box","proj":"EPSG:4326","coordinateType":"Cartographic","coordinates":[-180,-90,180,90]}
data = ipycmc.retrieve_data(plotType, startDate, endDate, ds, geometry)
ipycmc.plot_data(plotType, data)

Class: MapCMC

NOTE: All methods of the MapCMC class expect the widget to be rendered so make sure you display the widget before calling any of these methods beforehand like so

w = ipycmc.MapCMC()
w

.load_layer_config(url, handle_as, default_ops={}) - load a layer config into the mapping widget

  • url: string - the url endpoint of the config (such as WMS or WMTS GetCapabilities endpoint)
  • handle_as: string - ["json", "wmts/xml", "wms/xml"] the type of config endpoint to be loaded
  • default_ops: dict - a dictionary of default options to apply to the loaded layers. If loading from a GIBS endpoint, you might use: {"handleAs": "GIBS_raster"}.

Example

w.load_layer_config(
    "https://gibs.earthdata.nasa.gov/wmts/epsg4326/best/wmts.cgi?SERVICE=WMTS&request=GetCapabilities",
    "wmts/xml",
    {"handleAs": "GIBS_raster"}
)

When loading 3D layers from MAAP's CMR you must include "json" and {"handleAs": "vector-3d-tile"}.

Example

w.load_layer_config(
    "https://cmr.maap-project.org/search/concepts/G1200354094-NASA_MAAP.json”, 
    "json”,  
    {“handleAs": "vector-3d-tile"})

.set_date(date_str, format_str="") - set the display date of the mapping widget

  • date_str: string - date or datetime that the widget should be set to
  • format_str: string - the format the of the date string. See token formats for MomentJS: https://momentjs.com/docs/#/parsing/string-format/

Example

w.set_date("2019-Jan-03", "YYYY-MMM-DD")

.get_date() - get the current widget display date as an ISO 8601 string


.get_layers() - get the list of ingested layer information dictionaries


.get_area_selections() - the the list of selected layer information dictionaries


.get_active_layers() - the the list of layer information dictionaries for layers currently active on the map

API Documentation

API Documentation

Module: ipycmc

.MapCMC() - creates a new instance of a MapCMC widget. This widget widget is meant to be dislayed inline in the notebook and functions called on the instance will not perform as expected until the widget is displayed.

Example

w = ipycmc.MapCMC()
w

.retrieve_data(plotType, startDate, endDate, ds, geometry) - retrieves data meant for plotting from the MAAP data analysis platform

  • plotType: string - ['timeseries', 'timeAvgMap', 'hovmoller_lat', 'hovmoller_lon'] the type of plot data to be calculated
  • startDate: string - ISO 8601 string specifying the start time bound of the analysis
  • endDate: string - ISO 8601 string specifying the end time bound of the analysis
  • ds: list<string> - list of dataset identifiers that should be included in this analysis
  • geometry: dict - dictionary of type, projection, and coordinates of the area that should be included in the analysis

Example

plotType = "timeseries"
startDate = "2019-08-25T07:00:00.000Z"
endDate = "2019-09-01T07:00:00.000Z"
ds = ["dataset_id"]
geometry = {"type":"Box","proj":"EPSG:4326","coordinateType":"Cartographic","coordinates":[-180,-90,180,90]}
data = ipycmc.retrieve_data(plotType, startDate, endDate, ds, geometry)
data

.plot_data(plotType, data) - plots the data from retrieve_data() inline in a charting widget

  • plotType: string - ['timeseries', 'timeAvgMap', 'hovmoller_lat', 'hovmoller_lon'] the type of plot to be generated
  • data: dict - the output from retieve_data() for this plot type

Example

plotType = "timeseries"
startDate = "2019-08-25T07:00:00.000Z"
endDate = "2019-09-01T07:00:00.000Z"
ds = ["dataset_id"]
geometry = {"type":"Box","proj":"EPSG:4326","coordinateType":"Cartographic","coordinates":[-180,-90,180,90]}
data = ipycmc.retrieve_data(plotType, startDate, endDate, ds, geometry)
ipycmc.plot_data(plotType, data)

Class: MapCMC

NOTE: All methods of the MapCMC class expect the widget to be rendered so make sure you display the widget before calling any of these methods beforehand like so

w = ipycmc.MapCMC()
w

.load_layer_config(url, handle_as, default_ops={}) - load a layer config into the mapping widget

  • url: string - the url endpoint of the config (such as WMS or WMTS GetCapabilities endpoint)
  • handle_as: string - ["json", "wmts/xml", "wms/xml"] the type of config endpoint to be loaded
  • default_ops: dict - a dictionary of default options to apply to the loaded layers. If loading from a GIBS endpoint, you might use: {"handleAs": "GIBS_raster"}

Example

w.load_layer_config(
    "https://gibs.earthdata.nasa.gov/wmts/epsg4326/best/wmts.cgi?SERVICE=WMTS&request=GetCapabilities",
    "wmts/xml",
    {"handleAs": "GIBS_raster"}
)

.set_date(date_str, format_str="") - set the display date of the mapping widget

  • date_str: string - date or datetime that the widget should be set to
  • format_str: string - the format the of the date string. See token formats for MomentJS: https://momentjs.com/docs/#/parsing/string-format/

Example

w.set_date("2019-Jan-03", "YYYY-MMM-DD")

.get_date() - get the current widget display date as an ISO 8601 string


.get_layers() - get the list of ingested layer information dictionaries


.get_area_selections() - the the list of selected layer information dictionaries


.get_active_layers() - the the list of layer information dictionaries for layers currently active on the map

ADE Jupyter Notebook Magics
Load the Inline Magics in Notebook

In-line Notebook Magics

Load the inline magics in notebook

WARNING: This will not work for a python script. This is for use in notebooks only.
You must enable the extension in the notebook: %load_ext dps_magic
You can run a command by entering % followed by a command and its arguments into a notebook cell.

Available Magics

You can list the available magics by running %help
Output:

Magic Command Description
%capabilities get information about MAAP API services
%list list algorithms registered in MAS
%describe describe the selected algorithm
%execute submit a job to DPS using an algorithm registered in MAS
%status check the status of a submitted job
%result get the results for a completed job
%delete_algorithm remove a registered algorithm from MAS
%delete_job remove a completed job from DPS
%s3_url get a presigned s3 url for an object
%help print this help info

Pass the argument "help" to a command for more specific instructions on how to use it:
%execute help
Output:
execute_help_menu

List

Cell:
%list
Output:

Algorithms:
	example_plot_ubuntu:1.0
	gdal_service:master
	george_demo_ubuntu:master
	george_esdis_demo_ubuntu:1.0
	liz-plot2_ubuntu:1.0
	plant_algo:master
	plant_test:master
	plot_algo:master
	plot_demo3:master
	plot_demo4:master
	plot_demo_ubuntu:master
	plot_example_ubuntu:1.0
	plot_test2_ubuntu:master
	plot_test3_ubuntu:master
	plot_test5_ubuntu:master
	plot_test_ubuntu:master
	plot_ubuntu:master

Describe

Cell:
%describe help
Output:

Describe Algorithm Help

Check the inputs required for an algorithm stored in MAS.  You need to know your algorithm name and version.

Example Describe Call:
    %describe plot_algo:master
Example Dictionary Call: 
    d = {'algo_id':'plot_algo','version':'master'} 
    %describe $d



Cell:
%describe george_demo_ubuntu:master
OR
Cell:

d = {'algo_id':'george_demo_ubuntu','version':'master'}
%describe $d

Output:
Algorithm: george_demo_ubuntu
Version: master
Identifier:	george_demo_ubuntu:master
Input
	Title:	pass_number
	Identifier:	pass_number
	DataType:	string

Input
	Title:	username
	Identifier:	username
	DataType:	string

Output:	[]

Execute

Cell:
%execute help
Output:

Execute Job Help
Execute a job through DPS.  You need to know the parameters of your algorithm.  Use %describe to check if you are unsure.

Example String Call: 
    %execute plot_algo:master(pass_number=3,username=liz)
Example Dictionary Call: 
    d = {'pass_number':'6','username':'liz','algo_id':'plot_algo','version':'master'} 
    %execute $d



Cell:
%execute plot_algo:master(pass_number=3,username=eyam)
OR

d = {'pass_number':'6','username':'eyam','algo_id':'plot_algo','version':'master'} 
%execute $d

Output:
JobID is cd36b6ff-5477-4cbf-80b8-772a77a44c58
  • make sure you use your username, or else the Job ID will not be saved anywhere else and will not be displayed in the jobs table

Status

Cell:
%status help
Output:

Job Status Help

Check the status of a job in DPS.  You need to know your job ID.
Example Status Call:
    %status ef6fde9e-0975-4556-b8a7-ee52e91d8e61



Cell:
%status cd36b6ff-5477-4cbf-80b8-772a77a44c58

Output:

JobID is cd36b6ff-5477-4cbf-80b8-772a77a44c58
Status: Succeeded

Result

Cell:
%result help
Output:

Job Result Help

Check the result of a completed job in DPS.  You need to know your job ID.
Example Result Call:
    %result ef6fde9e-0975-4556-b8a7-ee52e91d8e61



Cell:
%result cd36b6ff-5477-4cbf-80b8-772a77a44c58

Output:

Job Results
JobID: cd36b6ff-5477-4cbf-80b8-772a77a44c58
ProductName: output-2019-09-13_00-21-53
Locations: http://maapdev.s3.amazonaws.com/products/plot/v1.0/2019/09/13/output-2019-09-13_00-21-53
• s3://s3.amazonaws.com:80/maapdev/products/plot/v1.0/2019/09/13/output-2019-09-13_00-21-53
• https://s3.console.aws.amazon.com/s3/buckets/maapdev/products/plot/v1.0/2019/09/13/output-2019-09-13_00-21-53/?region=us-east-1&tab=overview
  • the call will return an error if the job was not successfully completed (includes started, revoked, failed, and deleted)

Delete

Cell:
%delete_job help
Output:

Delete Job Help

Delete a finished (completed or failed) job or queued job stored in DPS.  You need to know your Job ID.
Example Delete Call:
    %delete_job ef6fde9e-0975-4556-b8a7-ee52e91d8e61



Cell:
%delete_job 5bcac3c8-9958-4c54-bb73-99d5eba09879
Output:

JobID is 5bcac3c8-9958-4c54-bb73-99d5eba09879
Status: Deleted

Dismiss

Cell:
%dismiss help
Output:

Dismiss Job Help

Dismiss a running (started, NOT queued) job on DPS.  You need to know your Job ID.
Example Dismiss call:
	%dismiss_job ef6fde9e-0975-4556-b8a7-ee52e91d8e61'



Cell:
%dismiss c9737ace-b143-49cc-96e4-9a6a7665932d
Output:

JobID is c9737ace-b143-49cc-96e4-9a6a7665932d
Status: Dismissed

s3 Url

Cell:
%s3_url help
Output:

Presigned S3 Url Help

Get a presigned s3 url for an object.  You need to know the path of the file relative to the bucket or your /projects directory.
Example s3_url call:
	%s3_url eyam/file-on-bucket



Cell:
%s3_url eyam/file-on-bucket
Output:

https://maap-mount-dev.s3.amazonaws.com/eyam/file-on-bucket?AWSAccessKeyId=AKIAJTSO23RLALBN45NQ&Signature=vgK%2FvwuSSYc4WUf39o4KWtbdVtw%3D&Expires=1568351409
DPS and MAS Notebook Magic

404 - Not found

Document not found
ADE Workspace and Data Collaboration
Organizations

Sharing

Organizations

We are utilizing Eclipse Che’s Organizations feature to allow users to share workspaces with other MAAP users. If a workspace is created in an organization, a user can easily share that workspace with other members of the organization.

All users are in one shared organization

Users can be in multiple organizations, but there is one organization, called maap-users that all users are in. This allows for users to be able to see all the possible MAAP users to share with.

User's organizations

All users have their own personal organization they can customize

Users all also have their own personal organization. Each user is the admin of their own organization so they can easily curate groups and individuals that they share their work with often.

To add a user to your personal organization, use the Add Member button. Now you can share any workspace in your personal organization with them!

Adding a user to a personal organization

To create a group of users you want to share with, create a sub-organization in your personal organization. In order to add someone to your sub-organization, you first must add them to your personal organization. Organizations can be created in layers of nesting sub-organizations, but for all organizations, you can only add members of it's parent organization.

As you can see, your personal organization is really just a sub-organization of maap-users/

For example - if I wanted to create a group frequent collaborators of mine.

  1. Add each of them to my personal organization maap-users\<my-name-here>.
  2. Create a sub-organization under maap-users\<my-name-here>, lets's call it maap-users\<my-name-here>\my-collaborators.
  3. Add each of my collaborators to my new sub-organization.

Creating workspaces in an organization's namespace

All workspaces are created under the namespace of an organization. NOTE: Once a workspace is created under an organization it cannot be moved to a different organization.

To specify which organization you want a workspace to be a part of, specify using the dropdown on workspace creation.

Select organization in workspace creation

Once a workspace is created, it is not inherent that all memebers of that organization are able to see it. You must share with them after it is created. This can be found in workspace's Share tab.

Share workspace with organization members

More documentation

For more documentation help on using organizations, check out Che's guide. It is not specific to MAAP's implementation, but most of the info will still apply!

Sharing Data

Users who have access to a workspace have access to all the files contained in that workspace.

All users have their own personal s3-bucket folder mounted in the FileBrowser home

In /projects, each user has a personal s3-hosted folder with the same name as their CAS username. Files in this folder are automatically uploaded to s3 and will accessible from any workspace a user signs into.

The intention of this mounted folder is that you can use this to share data with others, and also store files you want to access across different workspaces. It is not intended that you do all of your work in this directory. Because this directory is mounted to s3, you will notice that processes are slower when working in this directory.

s3-hosted folder

Generating a shareable link to s3-hosted files

Users can create a shareable link for any files in their folder that is hosted on s3. To do this, go to Command Palette -> User -> Get Presigned S3 URL and enter the relative path to the file you want to share.
The link will expire after 12 hours.

s3-link-gif

See Notebook Magics for how to get the presigned s3 url from an inline notebook command.

Sharing Data

Sharing Data

Users who have access to a workspace have access to all the files contained in that workspace.

All users have their own personal s3-bucket folder mounted in the FileBrowser home

In /projects, each user has a personal s3-hosted folder with the same name as their CAS username. Files in this folder are automatically uploaded to s3 and will accessible from any workspace a user signs into.

The intention of this mounted folder is that you can use this to share data with others, and also store files you want to access across different workspaces. It is not intended that you do all of your work in this directory. Because this directory is mounted to s3, you will notice that processes are slower when working in this directory.

s3-hosted folder

Generating a shareable link to s3-hosted files

Users can create a shareable link for any files in their folder that is hosted on s3. To do this, go to Command Palette -> User -> Get Presigned S3 URL and enter the relative path to the file you want to share.
The link will expire after 12 hours.

s3-link-gif

See Notebook Magics for how to get the presigned s3 url from an inline notebook command.

ADE SSH
SSH Access

SSH

As an alternative to using the jupyter interface, you can ssh directly into the container that your workspace set up. In order to get the IP and port information, navigate to the command palette of the jupyter interface. Find the command Display SSH Info (you can easily search for ssh) which will display the information you need. Your public SSH key that you added to your account will be added to any workspace you create. If you did not upload an SSH key to your profile, you will not be able to SSH in and must use the jupyter interface.

Search for SSH Display SSH info


In the workspace administration section of che, you will see and SSH tab, nothing you do here will affect your workspace, keys generated here are not valid.

For more info on things you should not do, see here
ADE User Interface Do’s/Don’t

What Not to Do!

We tried to use as many out of the box technologies as possible in this project to get as many features available as possible. But with that comes a lot of features that are not relevant to our platform or have not been implemented on our side yet. As we continue to build MAAP this list should become shorter and shorter, but for now here are the things that you should not mess with.

If you happen to mess with one of these things, you'll probably be fine - you'll just end up with a bunch of errors you don't understand or just nothing happening when you click a button.

Side Panel

side panel options

What's relevant: Workspaces, Organizations.

What's not: Stacks, Factories, Administration.


Workspace Administration

Adding a workspace

What to do

Configure the name, organization, RAM, and projects.

What NOT to do

Use the wrong stack. For now, only run the stack maap-jupyter-ide stable build

When adding projects, only add projects with the Git tab. Do not use any of the other project tabs.

Add_project_options

Managing a workspace

Workspace admin tabs

Here are the features allowed/not allowed on all of the workspace administration tabs.

Tab Features
Overview Do not bother with the Export buttons - these are not relevant.
Projects When adding projects, only add projects with the Git tab. Do not use any of the other project tabs.
Machines Do not add a machine! Do not play with any of the actions! The only thing you can do here is change the RAM.
Installers The Exec and SSH installers must be turned on here in order to get SSH access to your machine. All other installers are not relevant and should not be turned on.
Servers Do not touch!
Env Variables This is allowed!
Volumes Do not touch!
Config Do not touch!
SSH Do not touch! SSH has been handled for you and adding a key here will not work.
Tools Do not touch!
Plugins Do not touch!
Share Do this! All of the features should be implemented here.
DPS Algorithms
List Algorithms

List Algorithms

A user can execute a job with any algorithm that has been published to the MAAP Algorithm Store (MAS). To view what algorithms are available, click on Command Palette -> DPS -> List Algorithms. Algorithms are listed in the format <algorithm-name>:<version>.

Users will be shown the same list in a dropdown menu when trying to execute a job.

List Algorithms - Command Palette

List Algorithms - List

Describe an Algorithm

Describe an Algorithm

A user can request more information about available algorithms. Click on Command Palette -> DPS -> Describe Algorithm.

A popup window will appear, asking the user to select an available algorithm.
A second popup window will follow, listing the algorithm's registered name, version, identifier, any required inputs, and expected outputs.

Describe Algorithm - Command Palette

Describe Algorithm - Select Algorithm

Describe Algorithm

Register an Algorithm

Register an Algorithm

Go to Command Palette -> DPS -> Register Algorithm. The user is be shown a dropdown menu with available Python source files or Python notebook files.

Register Algorithm - Command Palette Register Algorithm - Select File

After selecting the desired script, the user is prompted with a new dialog box to populate required information for registering the script as an algorithm. The name is pre-populated from the filename, and environment is set to ubuntu, which most ADEs are running on.

Enter the command to run the script in run_cmd (e.g. python3 plot.py) and required parameters for the algorithm in inputs (e.g. pass_number), each on its own line. The algorithm name and description can be changed if desired. Click OK to finish registering. The user is then given a response if the request for registration was successful, or if some error occurred.

Register Algorithm - Algorithm Info, Inputs Register Algorithm - Algorithm Submitted for Registration

NOTE: After registering a new algorithm, a corresponding docker image must build successfully in GitLab before the algorithm is available for job submission.

Delete an Algorithm

Delete an Algorithm

Go to Command Palette -> DPS -> Delete Algorithm. The user is show a dropdown menu with available containerized algorithms to remove.

List Algorithms - Command Palette

List Algorithms - Select Algorithm

List Algorithms - Deleted Algorithm


If at any point the user does not get the expected output or behavior, it is possible that user's session cookie has expired and will need to refresh the page.

DPS Job Submission
List Previous Jobs

List Previous Jobs

The Jobs pane will need to be manually (re)loaded. This can be done by going to the Command Palette -> DPS -> Refresh Job List. Once this happens, the Jobs pane will display a scrollable, up-to-date table of the user's job history with Job Id, Job Status, and Algorithm, sorted by the most recently submitted job. From here, users can go directly to the s3-hosted results of a completed job or delete the currently viewed job.

List Jobs - Command Palette Jobs Panel

Click on the row containing the job of interest for more information about inputs and results.

Jobs Panel - Completed Job Results Jobs Panel - Failed Job

Execute a Job

Execute a Job

To execute a job, click on Command Palette -> DPS -> Execute DPS Job.

A popup window will appear, asking the user to select an available algorithm. Next, the user will be prompted to enter the required inputs for the selected algorithm. Upon submission, a Job ID will be returned.

Note: The Job Submissions must be manually reloaded to show the newly submitted job.

Execute Job - Command Palette

Execute Job - Select Algorithm

Execute Job - Enter Parameters

Execute Job - Return Job ID

Check Job Status

List Previous Jobs

The Jobs pane will need to be manually (re)loaded. This can be done by going to the Command Palette -> DPS -> Refresh Job List. Once this happens, the Jobs pane will display a scrollable, up-to-date table of the user's job history with Job Id, Job Status, and Algorithm, sorted by the most recently submitted job. From here, users can go directly to the s3-hosted results of a completed job or delete the currently viewed job.

List Jobs - Command Palette Jobs Panel

Click on the row containing the job of interest for more information about inputs and results.

Jobs Panel - Completed Job Results Jobs Panel - Failed Job

Check Job Results

Check Job Results

The easiest way to check a job's status is to find and click on it on the Jobs pane.

Alternatively, if the job was submitted under a different user, and the Job ID is known, the user can check the job status by going to Command Palette -> DPS -> Get DPS Job Result. A series of dialogs similar to Get Status will popup.

Job Results - Command Palette

Job Results - Job ID

Job Results

Delete Job

Delete Job

The easiest way to delete a job is to view it on the Jobs pane and click the Delete Job button.

Deleting a job means removing any record of a job that has already been attempted and failed or completed. To delete a job, go to Command Palette -> DPS -> Delete DPS Job. Similar to Get Status and Get Results, a popup will appear, prompting the user to enter the job id.

Dismiss Job

Dismiss Job

The easiest way to delete a job is to view it on the Jobs pane and click the Delete Job button.

Dismissing a job is to stop a job that is currently running already. To dismiss a job, go to Command Palette -> DPS -> Dismiss DPS Job. A popup will appear, prompting the user to enter the job id.


If at any point the user does not get the expected output or behavior, it is possible that user's session cookie has expired and will need to refresh the page.

API Documentation

https://api.maap-project.org/api/

NASA MAAP

MAAP is a collaboration between ESA and NASA

  • Privacy Policy
  • ESA MAAP