Welcome to GRID_LRT’s documentation!¶
This package is built by Alexandar Mechev and the LOFAR e-infra group at Leiden University with the support of SURFsara. The goals of this package is to enable High Throughput processing of LOFAR data on the Dutch GRID infrastructure. We do this by making a set of tools designed to wrap around several different LOFAR processing strategies. These tools are responsible for staging data at the LOFAR Long Term Archives, creating and launching GRID jobs, as well as managing intermediate data on the GRID storage.
Installation¶
Via Python Package Index¶
Install the package (or add it to your requirements.txt file):
pip install GRID_LRT
Via Git or Download¶
Download the latest version from https://www.github.com/apmechev/GRID_LRT. To install, use
python setup.py build
python setup.py install
In the case that you do not have access to the python system libraries, you can use --prefix= to specify install folder. For example if you want to install it into a folder you own (say /home/apmechev/software/python) use the following command:
python setup.py build
python setup.py install --prefix=${HOME}/software/python
Note
NOTE: you need to have your pythonpath containing
“${HOME}/software/python/lib/python[2.6|2.7|3.4]/site_packages”
and that folder needs to exist beforehand or setuptools will complain
Tokens¶
The GRID_LRT.Token module is responsible for interactions with CouchDB using the PiCaS token framework. It contains a Token_Handler object which manages a single _design document on CouchDB, intended for a set of jobs that are logically linked together. In the LOFAR Surveys case, this holds the jobs of a single Observation. Additionally a Token_Set object can create batch tokens, upload attachments to them in bulk and change Token fields in bulk as well. This module is used in combination with the srmlist class to automatically create sets of jobs with N files each.
Staging Modules¶
These modules are located in GRID_LRT.Staging and can be used to batch stage or check the status of the files on the GRID Storage.
GRID_LRT.Staging.srmlist¶
-
GRID_LRT.Staging.srmlist.slice_dicts(srmdict, slice_size=10)[source]¶ Returns a dict of lists that hold 10 SBNs (by default). Missing Subbands are treated as empty spaces, if you miss SB009, the list will include 9 items from SB000 to SB008, and next will start at SB010
-
class
GRID_LRT.Staging.srmlist.srmlist(checkOBSID=True, link=None)[source]¶ Bases:
listThe srmlist class is an extension of Python lists that can hold a list of srm links to data on GRID Storage (LOFAR Archive, Intermediate Storage, etc).
In addition to the regular list capabilities, it also has internal checks for the location and the OBSID of the data. When a new item is appended, these checks are done automatically. Checking OBSID is an optional argument set to True by default.
-
__init__(checkOBSID=True, link=None)[source]¶ x.__init__(…) initializes x; see help(type(x)) for signature
-
count(value) → integer -- return number of occurrences of value¶
-
extend()¶ L.extend(iterable) – extend list by appending elements from the iterable
-
gfal_links()[source]¶ Returns a generator that can be used to generate links that can be staged/stated with gfal
-
gsi_links()[source]¶ Returns a generator which can be iterated over, this generator will return a set of gsiftp:// links which can be used with globus-url-copy and uberftp
-
http_links()[source]¶ Returns a generator that can be used to generate http:// links that can be downloaded using wget
-
index(value[, start[, stop]]) → integer -- return first index of value.¶ Raises ValueError if the value is not present.
-
insert()¶ L.insert(index, object) – insert object before index
-
pop([index]) → item -- remove and return item at index (default last).¶ Raises IndexError if list is empty or index is out of range.
-
remove()¶ L.remove(value) – remove first occurrence of value. Raises ValueError if the value is not present.
-
reverse()¶ L.reverse() – reverse IN PLACE
-
sbn_dict(pref='SB', suff='_')[source]¶ Returns a generator that creates a pair of SBN and link. Can be used to create dictionaries
-
sort()¶ L.sort(cmp=None, key=None, reverse=False) – stable sort IN PLACE; cmp(x, y) -> -1, 0, 1
-
GRID_LRT.Staging.stage_all_LTA¶
GRID_LRT.Staging.state_all¶
Python module to check the state of files using gfal and return their locality # ===================================================================== # # author: Ron Trompert <ron.trompert@surfsara.nl> – SURFsara # # helpdesk: Grid Services <grid.support@surfsara.nl> – SURFsara # # # # usage: python state.py # # description: # # Display the status of each file listed in “files”. The paths # # should have the ‘/pnfs/…’ format. Script output: # # ONLINE: means that the file is only on disk # # NEARLINE: means that the file in only on tape # # ONLINE_AND_NEARLINE: means that the file is on disk # # and tape # # ===================================================================== #
-
GRID_LRT.Staging.state_all.check_status(surl_link, verbose=True)[source]¶ Obtain the status of a file from the given surl.
- Args:
param surl: the SURL pointing to the file. type surl: str parame verbose: print the status to the terminal. type verbose: bool - Returns:
(filename, status): a tuple containing the file and status as stored in the ‘user.status’ attribute.
-
GRID_LRT.Staging.state_all.load_file_into_srmlist(filename)[source]¶ Helper function that loads a file into an srmlist object (will be added to the actual srmlist class later)
-
GRID_LRT.Staging.state_all.main(filename, verbose=True)[source]¶ Main function that takes in a file name and returns a list of tuples of filenames and staging statuses. The input file can be both srm:// and gsiftp:// links.
- Args:
param filename: The filename holding the links whose have to be checked type filename: str param verbose: A toggle to turn off printing out the status of each file. True by default will print everything out :type verbose: bool
- Returns:
ret results: A list of tuples containing the file_name and the State
Usage:
>>> from GRID_LRT.Staging import state_all >>> filename='/home/apmechev/GRIDTOOLS/GRID_LRT/GRID_LRT/tests/srm_50_sara.txt' >>> results=state_all.main(filename) >>> results=state_all.main(filename, verbose=False) >>> results[0] ('L229507_SB150_uv.dppp.MS_f6fc7fc5.tar', 'ONLINE_AND_NEARLINE')
-
GRID_LRT.Staging.state_all.percent_staged(results)[source]¶ Takes list of tuples of (srm, status) and counts the percentage of files that are staged (0->1) and retunrs this percentage as float
Usage:
>>> from GRID_LRT.Staging import state_all >>> filename='/home/apmechev/GRIDTOOLS/GRID_LRT/GRID_LRT/tests/srm_50_sara.txt' >>> results=state_all.main(filename, verbose=False) >>> state_all.percent_staged(results)
GRID_LRT.Staging.stager_access¶
It uses an xmlrpc proxy to talk and authenticate to the remote service. Your account credentials will be read from the awlofar catalog Environment.cfg, if present or can be provided in a .stagingrc file in your home directory.
!!Please do not talk directly to the xmlrpc interface, but use this module to access the provided functionality. !! This is to ensure that when we change the remote interface, your scripts don’t break and you will only have to upgrade this module.
-
GRID_LRT.Staging.stager_access.stage(surls)[source]¶ Stage list of SURLs or a string holding a single SURL
Parameters: surls (either a list() or a str()) – Either a list of strings or a string holding a single surl to stage Returns: An integer which is used to refer to the stagig request when polling the API for a staging status
-
GRID_LRT.Staging.stager_access.get_status(stageid)[source]¶ Get status of request with given ID
- Args:
param stageid: The id of the staging request which you want the status of type stageid: int - Returns:
status: A string describing the staging status: ‘new’, ‘scheduled’, ‘in progress’ or ‘success’
-
GRID_LRT.Staging.stager_access.get_surls_online(stageid)[source]¶ Get a list of all files that are already online for a running request with given ID
-
GRID_LRT.Staging.stager_access.get_srm_token(stageid)[source]¶ Get the SRM request token for direct interaction with the SRM site via Grid/SRM tools
-
GRID_LRT.Staging.stager_access.reschedule(stageid)[source]¶ Reschedule a request with a given ID, e.g. after it was put on hold due to maintenance
Sandbox Module¶
The Sandbox module creates a tar archive of the scripts to be distributed to the worker nodes at the launch of a PiCaS job. The location of the sandbox is stored in the PiCaS token and upon launch, it is downloaded and extracted. The sandbox is created from a configuration file which defines its name, location scrts repository and any additional processing scripts, such as prefactor.
GRID_LRT.sandbox¶
Sandbox building and uploading module
-
class
GRID_LRT.sandbox.Sandbox(cfgfile=None, **kwargs)[source]¶ Bases:
objectA set of functions to create a sandbox from a configuration file. Uploads to grid storage and ssh-copies to a remote ftp server as a fallback location.
Usage with a .cfg file:
>>> from GRID_LRT import sandbox >>> s=sandbox.Sandbox() >>> s.build_sandbox('GRID_LRT/data/config/bash_file.cfg') >>> s.upload_sandbox()
This will build the sandbox according to the recipe in bash_file.cfg and upload it to grid storage
-
__init__(cfgfile=None, **kwargs)[source]¶ Creates a ‘sandbox’ object which builds and uploads the sanbox. An optional argument is the configuration file which is a yaml file specifying the repositories to include, the type of the sanbox, and its name.
Example configuration files are included in GRID_LRT/data/config.
Parameters: cfgfile (str) – The name of the configuration file to build a sandbox from
-
build_sandbox(sbx_config)[source]¶ A comprehensive function that builds a Sandbox from a configuration file and creates a sandbox tarfile.
-
check_token()[source]¶ This function does the necessary linkage between the sandbox and token most importantly it saves the tokvar.cfg file in the sbx, but also checks if the token variables are all existing. If so, tokvar is created and put inside the SBX
-
copy_git_scripts()[source]¶ Reads the location of the sandbox base scripts repository and clones in the current directory. Checks out the appropriate branch
-
load_git_scripts()[source]¶ Loads the git scripts into the sandbox folder. Top dir names are defined in the yaml, not by the git name
-
Bases:
GRID_LRT.sandbox.SandboxCreates a ‘sandbox’ object which builds and uploads the sanbox. An optional argument is the configuration file which is a yaml file specifying the repositories to include, the type of the sanbox, and its name.
Example configuration files are included in GRID_LRT/data/config.
Parameters: cfgfile (str) – The name of the configuration file to build a sandbox from
A comprehensive function that builds a Sandbox from a configuration file and creates a sandbox tarfile.
This function does the necessary linkage between the sandbox and token most importantly it saves the tokvar.cfg file in the sbx, but also checks if the token variables are all existing. If so, tokvar is created and put inside the SBX
Backwards compatible
Reads the location of the sandbox base scripts repository and clones in the current directory. Checks out the appropriate branch
Makes an empty sandbox folder or removes previous one
Removes the sandbox folder and subfolders
Changes directory to the (temporary) sandbox folder)
Loads the git scripts into the sandbox folder. Top dir names are defined in the yaml, not by the git name
Helper function to parse the sandbox configuration options from the yaml .cfg file. Loads the options in a dictionary stored in an internal variable
Parameters: yamlfile (str) – The name of the sandbox configuration file
Uploads the sandbox to the relative folders
Uploads sandbox to all possible locations
Error Codes¶
Here are a list of errors that the GRID_Sandbox or GRID_Launcher return when processing data on a worker node. The error code is saved in the ‘output’ field of the PiCaS token.