Giosystem-orders¶
Giosystem includes an ordering service that processes order requests for products of the processing lines.
The ordering service uses the following components:
- An OGC OSEO server
- An Order database where order information is stored
- A User authentication module that implements custom auth
- An Order processing queue that is used to manage order requests
- An Order processing module module, that uses giosystemcore
OGC OSEO server¶
The ordering service implements a subset of the OGC OSEO specification, which defines a standard for ordering earth observation products.
The giosystem-orders server is based on the pyoseo project, while incorporating some custom logic in order to work with the giosystem processing lines.
Note
The OSEO server has the following restrictions:
- Implements only the following OSEO operations:
- Submit
- GetStatus
- DescribeResultAccess
- GetOptions
- Does not implement the asynchronous interface for the Submit operation
- Does not support FTP push as a delivery method
Supported OSEO operations¶
Submit¶
The Submit operation allows a client to place an order on the server. Upon having been successfully authenticated, the order is placed in the order database and is sent to the processing queue, where it will be processed in due time. The following image provides a rough workflow of the Submit operation:
Since the server does not support the OSEO asynchronous interface, the response to a Submit request is sent only once, immediately after the order has been requested. Further status changes must be explicitly asked for by the user, by issuing Getstatus requests.
Example requests and further information on the Submit operation is available at http://pyoseo.readthedocs.org/en/latest/submit.html
GetStatus¶
The Getstatus operation allows a client to ask the server what is the status of an order. The server queries its order database and returns the status information back to the client.
As orders get processed by the queue, their status is dynamically updated. After an order has been successfully downloaded, its status is also updated.
Example requests and further information on the GetStatus operation is available at http://pyoseo.readthedocs.org/en/latest/getstatus.html
DescribeResultAccess¶
When an order has finished processing, a client can issue a DescribeResultAccess request in order to discover the URLs where the ordered items are available. The OSEO server supports both HTTP and FTP pull as delivery methods for an order.
Generally, completed orders will remain available for download for a period of 10 days.
GetOptions¶
Server configuration¶
Being based on django, the server is configured by editing the settings.py and settings_local.py files. Besides the main django configuration, these files define the following settings, specific to the OSEO server:
- GIOSYSTEM_SETTINGS_URL - The full URL to the API for the giosystem settings web application;
- OSEOSERVER_ORDER_DELETION_THRESHOLD - An integer specifying the number of days that ordered items will remain on the server before being deleted;
- OSEOSERVER_ONLINE_DATA_ACCESS_HTTP_PROTOCOL_ROOT_DIR - The path on the file system to the root directory for placing ordered items which are to be available via HTTP;
- OSEOSERVER_ONLINE_DATA_ACCESS_FTP_PROTOCOL_ROOT_DIR - The path on the file system to the root directory for placing ordered items which are available via FTP;
- OSEOSERVER_AUTHENTICATION_CLASS - The Python import path to the custom authentication class that is used when authenticating incoming requests;
- OSEOSERVER_PROCESSING_CLASS - The Python import path to the custom module that handles the actual processing of orders
- OSEOSERVER_ADMIN_MAILS - a list of e-mail addresses that will be notified whenever an order fails
- OSEOSERVER_MAIL_SERVER - name of the server that is used to send e-mail notifications to the server’s administrators whenever an order fails
- OSEOSERVER_MAIL_SERVER_PORT - port of the mail server
- OSEOSERVER_MAIL_DOMAIN - domain of the mail server
- OSEOSERVER_MAIL_ACCOUNT - user account that is to be used when sending e-mails
- OSEOSERVER_MAIL_ACCOUNT_PASSWORD - password of the account that is to be used when sending e-mails
- OSEOSERVER_AUTH_VITO_ATTRIBUTE - Custom setting used in the authentication module;
- OSEOSERVER_AUTH_VITO_TOKEN - Custom setting used in the authentication module;
- OSEOSERVER_AUTH_VITO_PASSWORD - Custom setting used in the authentication module.
The settings_local.py file holds site specific settings that override the ones in settings.py.
Order database¶
The giosystem ordering service is backed by a django application that includes a database. The database stores information on each order’s contents, such as:
- status
- requesting user
- ordered items
- delivery options
The stored information is suficciently detailed in order to reconstruct an order request and monitor the complete order status life cycle.
The database is dynamically updated as orders are being processed. It is controlled by the OSEO server. Additionally, there is also a web based client used to inspect and manage the database. It can be accessed at
User authentication module¶
The authentication module runs whenever there is a request for ordering. Currently, the implemented authentication scheme uses the SOAP headers to supply authentication information between the ordering client and the server.
The authentication scheme agreed between IPMA and VITO operates as follows:
- ordering requests must send in a wsse:Security/wsse:Password element with the correct value.
- ordering requests must send in a wsse:Security/wsse:Username element with the username of the user that made the order on the PDF portal. The user’s password is never sent to the ordering server. When the user tries to access the FTP server to download the data, the FTP server communicates with an LDAP server stored at VITO and authenticates the user using its username and the user supplied password.
Order processing queue¶
The giosystem ordering service is backed by a celery task queue. This enables the service to manage task execution processes asynchronously. The queue is populated with specialized order processing tasks, which are responsible for doing the actual processing of orders
There are two daemon processes responsible for processing orders and maintenance stuff. They are:
pyoseo-worker¶
The pyoseo-worker daemon is a celery queue that is responsible for processing orders.
pyoseo-beat¶
The pyoseo-beat daemon is responsible for performing periodic tasks, such as monitoring the number of FTP downloads for each order item, removing old orders, etc.
Both these daemons are configured to start automatically at boot, so if there ever is a need to reboot the machine, they will be back online as soon as it reboots.
Managing the services¶
The daemons are managed using SystemV-style init scripts. The scripts are located in the /etc/init.d directory and can be used like this:
sudo service pyoseo-worker status # check if the service is running
sudo service pyoseo-worker start # start the service
sudo service pyoseo-worker stop # stop the service
sudo service pyoseo-worker restart # stop the service and immediately start it again
The previous commands can be used with both daemons. Both init scripts are exact copies of the files supplied by the celery development team. The original can be found at celery’s github repository.
Each service can be customized by editing the files in:
/etc/default/pyoseo-worker
/etc/default/pyoseo-beat
For more information on the options for each service, consult celery’s documentation.
Log files¶
The log files for these services are located in /var/log/celery. They can be monitored in real time by issuing the command:
tail -f /var/log/celery/worker1.log # for the pyoseo-worker daemon
tail -f /var/log/celery/celeryd.log # for the pyoseo-beat daemon
Order processing module¶
The order processing module performs the job of actually fetching the ordered
items and placing them in the correct directories on the web or the FTP server,
making them available for download. This module makes use of the helper
functions defined in the giosystemcore
.
Order processing consists in:
- retrieving the ordered items and options
- fetching the products from the giosystem host network
- apply some post-processing routines
- placing the ordered items in their web accessible destination
Note
Update this documentation to reflect the current status of the ordering server.
In order to handle order requests that specify delivery as HTTP GET requests, the django-giosystem-downloads app includes a get_ordered_file view that will fetch the files from disk. In order for it to work correctly, you must install the xsendfile apache module.
sudo apt-get install libapache2-mod-xsendfile
Debugging failed orders¶
Orders can fail if the user placing the order is not actually registered in the PDF portal. In order to verify that a given user is present in VITO’s LDAP database, one can use the ldapsearch command. For example, to see if the user demouser exists in VITO’s LDAP server, issue the command:
ldapsearch -v -x -D <bind_dn> -w <password> -s sub -b <search_base> -P 3 -H <ldap_uri> uid=demouser
Note
The variables bind_dn, password search_base and ldap_uri are sensitive information. They’re values are stored at IPMA’s internal network. Look for them in the passwords section of the collaborative platform.