dl package¶

Subpackages¶

dl.helpers package

Submodules¶

dl.Util module¶

dl.Util.add_doc(value)[source]¶: Decorator to set the ‘Call docstring’ ipython field.

dl.Util.def_token(tok)[source]¶

Get a default token. If no token is provided, check for an existing $HOME/.datalab/id_token.<user> file and return that if it exists, otherwise default to the ANON_TOKEN.

If a token string is provided, return it directly. The value may also simply be a username, in which case the same check for a token ID file is done.

dl.Util.encode_multipart(fields, files, boundary=None)[source]¶

Encode dict of form fields and dict of files as multipart/form-data. Return tuple of (body_string, headers_dict). Each value in files is a dict with required keys ‘filename’ and ‘content’, and optional ‘mimetype’ (if not specified, tries to guess mime type or uses ‘application/octet-stream’).

..code-block:: python

>>> body, headers = encode_multipart({'FIELD': 'VALUE'},
...                                  {'FILE': {'filename': 'F.TXT', 'content': 'CONTENT'}},
...                                  boundary='BOUNDARY')
>>> print('\n'.join(repr(l) for l in body.split('\r\n')))
'--BOUNDARY'
'Content-Disposition: form-data; name="FIELD"'
''
'VALUE'
'--BOUNDARY'
'Content-Disposition: form-data; name="FILE"; filename="F.TXT"'
'Content-Type: text/plain'
''
'CONTENT'
'--BOUNDARY--'
''
>>> print(sorted(headers.items()))
[('Content-Length', '193'), ('Content-Type', 'multipart/form-data; boundary=BOUNDARY')]
>>> len(body)
193

dl.Util.multimethod(module, nargs, cm)[source]¶: Wrapper function to implement multimethod for functions. The identifying signature in this case is the number of required method parameters. When methods are called, all original arguments and keywords are passed.

dl.Util.readTokenFile(tok_file)[source]¶

dl.authClient module¶

dl.authClient.acToString(s)[source]¶: acToString – Force a return value to be type ‘string’ for all Python versions.

class dl.authClient.authClient[source]¶

Bases: object

AUTHCLIENT – Client-side methods to access the Data Lab: Authentication Service.

debug(debug_val)[source]¶: Toggle debug flag.

getConfig(section, param)[source]¶: Get a value from the configuration file.

getFromURL(svc_url, path, token)[source]¶: Get something from a URL. Return a ‘response’ object.

getHeaders(token)[source]¶: Get default tracking headers.

get_profile()[source]¶

Get the requested service profile.

Parameters: None –
Returns: profile – The currently requested service profile.
Return type: str

Example

from dl import authClient
profile = authClient.client.get_profile()

get_svc_url()[source]¶

Return the currently-used Authentication Service URL.

Parameters: None –
Returns: service_url – The currently-used Authentication Service URL.
Return type: str

Example

from dl import authClient
service_url = authClient.get_svc_url()

hasAccess(token, resource)[source]¶

See whether the given token has access to the named Resource.

Parameters

token (str) – User login token
resource (str) – Resource identifier to check

Returns

status – True if user owns or has access, False otherwise

Return type

bool

Example

from dl import authClient
status = authClient.hasAccess(token,'vos://test.dat')

isAlive(svc_url='https://datalab.noao.edu/auth')[source]¶

Check whether the AuthManager service at the given URL is: alive and responding. This is a simple call to the root service URL or ping() method.

Parameters: service_url (str) – The Query Service URL to ping.
Returns: result – True if service responds properly, False otherwise
Return type: bool

Example

from dl import authClient
if authClient.isAlive():
    print("Auth Manager is alive")

isTokenLoggedIn(token)[source]¶

See whether the user identified by the token is currently logged in.

Parameters: token (str) – User login token
Returns: status – True if token is marked as logged-in, False otherwise
Return type: bool

Example

isUserLoggedIn(user)[source]¶

See whether the user identified by the token is currently logged in.

Parameters: user (str) – User login name
Returns: status – True if user is currently logged-in, False otherwise
Return type: bool

Example

isValidPassword(user, password)[source]¶

See whether the password is valid for the user.

Parameters

user (str) – User login name
password (str) – Password for named user

Returns

status – True if password is valid for the user, False otherwise

Return type

bool

Example

isValidToken(token)[source]¶

See whether the current token is valid.

Parameters: token (str) – User login token
Returns: status – True if token is valid, False otherwise
Return type: bool

Example

isValidUser(user)[source]¶

See whether the specified user is valid.

Parameters: user (str) – User login name
Returns: status – True if ‘user’ is a valid user name, False otherwise
Return type: bool

Example

list_profiles(token, profile=None, format='text')[source]¶

List the service profiles which can be accessed by the user.

Parameters: token (str) – Valid auth service token.
Returns: profiles
Return type: JSON string

Example

from dl import authClient
profiles = authClient.client.list_profiles(token, profile, format)

loadConfig()[source]¶: Read the $HOME/.datalab/dl.conf file.

login(username, password, debug=False, verbose=False)[source]¶

Authenticate the user with the Authentication Service.: We first check for a valid login token in the user’s $HOME/.datalab/ directory and simply return that rather than make a service call to get a new token. If a token exists but is invalid, we remove it and get a new token. In either case, we save the token for later use.

Parameters

username (str) – User login name.
password (str) – User password. If not given, a valid ID token will be searched for in the $HOME/.datalab directory.
debug (bool) – Method debug flag.
verbose (bool) – Initialize session to print verbose output messages.

Returns

token – One-time security token for valid user(identified via ‘username’ and ‘password’).

Return type

str

Example

from dl import authClient
token = authClient.login('dldemo', 'dldemo')   # get security token

logout(token=None)[source]¶

Log the user out of the Data Lab.

Parameters: token (str) – User login token
Returns
Return type: ‘OK’ string on success, or exception message

Example

from dl import authClient
status = authClient.logout(token)

passwordReset(token, username, password)[source]¶

Reset a user password reset. We require that the user provide: either a valid ‘root’ token or the token for the account being reset.

Parameters

token (str) – User login token
username (str) – User login name
password (str) – User password

Returns

Return type

‘OK’ string on success, or exception message

Example

from dl import authClient
status = authClient.logout(token)

retBoolValue(url)[source]¶: Utility method to call a boolean service at the given URL.

setConfig(section, param, value)[source]¶: Set a value and save the configuration file.

set_profile(profile)[source]¶

Set the requested service profile.

Parameters: profile (str) – Requested service profile string.
Returns
Return type: Nothing

Example

from dl import authClient
token = authClient.client.set_profile("dev")

set_svc_url(svc_url)[source]¶

Set the URL of the Authentication Service to be used.

Parameters: svc_url (str) – Authentication service base URL to call.
Returns
Return type: Nothing

Example

from dl import authClient
authClient.set_svc_url("http://localhost:7001/")

whoAmI()[source]¶

Return the currently logged-in user identity.

Parameters: None –
Returns: name – Currently logged-in user name, or ‘anonymous’ if not logged-in
Return type: str

Example

from dl import authClient
name = authClient.whoAmI()

writeConfig()[source]¶: Write out the configuration file to disk.

exception dl.authClient.dlAuthError(message)[source]¶

Bases: Exception

A throwable error class.

dl.authClient.getClient()[source]¶

Get a new instance of the authClient client.

Parameters: None –
Returns: client – An authClient object
Return type: authClient

Example

dl.authClient.get_profile()[source]¶

Get the requested service profile.

Parameters: None –
Returns: profile – The currently requested service profile.
Return type: str

Example

from dl import authClient
profile = authClient.client.get_profile()

dl.authClient.get_svc_url()[source]¶

Return the currently-used Authentication Service URL.

Parameters: None –
Returns: service_url – The currently-used Authentication Service URL.
Return type: str

Example

from dl import authClient
service_url = authClient.get_svc_url()

dl.authClient.hasAccess(user, resource)[source]¶

See whether the given token has access to the named Resource.

Parameters

token (str) – User login token
resource (str) – Resource identifier to check

Returns

status – True if user owns or has access, False otherwise

Return type

bool

Example

from dl import authClient
status = authClient.hasAccess(token,'vos://test.dat')

dl.authClient.isAlive(svc_url='https://datalab.noao.edu/auth')[source]¶

Check whether the AuthManager service at the given URL is: alive and responding. This is a simple call to the root service URL or ping() method.

Parameters: service_url (str) – The Query Service URL to ping.
Returns: result – True if service responds properly, False otherwise
Return type: bool

Example

from dl import authClient
if authClient.isAlive():
    print("Auth Manager is alive")

dl.authClient.isTokenLoggedIn(token)[source]¶

See whether the user identified by the token is currently logged in.

Parameters: token (str) – User login token
Returns: status – True if token is marked as logged-in, False otherwise
Return type: bool

Example

dl.authClient.isUserLoggedIn(user)[source]¶

See whether the user identified by the token is currently logged in.

Parameters: user (str) – User login name
Returns: status – True if user is currently logged-in, False otherwise
Return type: bool

Example

dl.authClient.isValidPassword(user, password)[source]¶

See whether the password is valid for the user.

Parameters

user (str) – User login name
password (str) – Password for named user

Returns

status – True if password is valid for the user, False otherwise

Return type

bool

Example

dl.authClient.isValidToken(token)[source]¶

See whether the current token is valid.

Parameters: token (str) – User login token
Returns: status – True if token is valid, False otherwise
Return type: bool

Example

dl.authClient.isValidUser(user)[source]¶

See whether the specified user is valid.

Parameters: user (str) – User login name
Returns: status – True if ‘user’ is a valid user name, False otherwise
Return type: bool

Example

dl.authClient.list_profiles(token, profile=None, format='text')[source]¶

dl.authClient.login(user, password=None, debug=False, verbose=False)[source]¶

Authenticate the user with the Authentication Service.: We first check for a valid login token in the user’s $HOME/.datalab/ directory and simply return that rather than make a service call to get a new token. If a token exists but is invalid, we remove it and get a new token. In either case, we save the token for later use.

Parameters

username (str) – User login name.
password (str) – User password. If not given, a valid ID token will be searched for in the $HOME/.datalab directory.
debug (bool) – Method debug flag.
verbose (bool) – Initialize session to print verbose output messages.

Returns

token – One-time security token for valid user(identified via ‘username’ and ‘password’).

Return type

str

Example

from dl import authClient
token = authClient.login('dldemo', 'dldemo')   # get security token

dl.authClient.logout(token=None)[source]¶

Log the user out of the Data Lab.

Parameters: token (str) – User login token
Returns
Return type: ‘OK’ string on success, or exception message

Example

from dl import authClient
status = authClient.logout(token)

dl.authClient.passwordReset(token, username, password)[source]¶

Reset a user password reset. We require that the user provide: either a valid ‘root’ token or the token for the account being reset.

Parameters

token (str) – User login token
username (str) – User login name
password (str) – User password

Returns

Return type

‘OK’ string on success, or exception message

Example

from dl import authClient
status = authClient.logout(token)

dl.authClient.set_profile(profile)[source]¶

Set the requested service profile.

Parameters: profile (str) – Requested service profile string.
Returns
Return type: Nothing

Example

from dl import authClient
token = authClient.client.set_profile("dev")

dl.authClient.set_svc_url(svc_url)[source]¶

Set the URL of the Authentication Service to be used.

Parameters: svc_url (str) – Authentication service base URL to call.
Returns
Return type: Nothing

Example

from dl import authClient
authClient.set_svc_url("http://localhost:7001/")

dl.authClient.whoAmI()[source]¶

Return the currently logged-in user identity.

Parameters: None –
Returns: name – Currently logged-in user name, or ‘anonymous’ if not logged-in
Return type: str

Example

from dl import authClient
name = authClient.whoAmI()

dl.dlinterface module¶

class dl.dlinterface.DLInteract[source]¶

Bases: object

Main class for Data Lab interactions

get(section, param)[source]¶: Get a value from the configuration file.

save(section, param, value)[source]¶: Save the configuration file.

class dl.dlinterface.Dlinterface(verbose=True)[source]¶

Bases: object

Data Lab python interface super-class with methods for each command.

copyurl(url=None, name=None)[source]¶

Copy a file to VOSpace using a URL.

Parameters

url (str) – The URL location of the file.
name (str) – The name of the file in VOSpace. The vos:// prefix is not necessary.

Example

Copy the file http://www.mywebsite.com/file1.fits to output1.fits in VOSpace.

dl.copyurl('http://www.mywebsite.com/file1.fits','output1.fits')

cp(source=None, destination=None, verbose=True)[source]¶

Copy a file in Data Lab VOSpace.

Parameters

source (str) – The name of the file in VOSpace to copy, e.g. file1.txt.
destination (str) – The new name of the file in VOSpace, e.g. newfile1.txt.

Example

Copy the file file.txt to newfile.txt.

dl.ls()
file1.txt

dl.cp('file1.txt','newfile.txt')

dl.ls()
file1.txt, newfile.txt

droptable(table=None)[source]¶

Drop a user’s MyDB table.

Parameters: table (str) – The name of a specific table in mydb to drop.
Returns: list – The list of properties of table or all tables in mydb.
Return type: str

Example

Drop the MyDB table called table.

print dl.listdb()
table
table2

dl.droptable('table')
table
table2

print dl.listdb()
table2

exporttable(table=None, name=None, fmt=None)[source]¶

Copy a user’s MyDB table to a file in VOSpace.

Parameters

table (str) – The name of a specific table in mydb to drop.
name (str) – The file name to save the table to.
fmt (str) – The output file format. The available formats are ‘csv’, ‘fits’ and ‘hdf5’. If this is not specified then the file extension is used to identify the format type.

Example

Export the MyDB table called table to file test.csv.

dl.exporttable('table','test.csv')

get(source=None, destination=None, verbose=True)[source]¶

Get one or more files from Data Lab.

Parameters

source (str) – The name of the source file on VOSpace, e.g. file2.txt.
destination (str) – The name of the local destination file, e.g. file1.txt.

Example

Get a query output table called table1_output.txt from VOSpace.

dl.get('table1_output.txt','table1_output.txt')
(1/1) [====================] [   9.1K] table1_output.txt

help(command=None)[source]¶: Print out useful help information on the Data Lab python interface and it’s commands.

listdb(table='')[source]¶

List the user’s MyDB tables.

Parameters: table (str) – The name of a specific table in mydb. If this is blank then all tables will be listed.
Returns: list – The list of properties of table or all tables in mydb.
Return type: str

Example

List the MyDB tables.

print dl.listmydb()
table
table2

ln(target=None, link=None)[source]¶

Link a file in Data Lab VOSpace.

Parameters

target (str) – The name of the file in VOSpace to link to, e.g. file1.txt.
link (str) – The name of the link, e.g. file1link.

Example

Create a link called iamlink to the file file1.txt.

dl.ls()
file1.txt

dl.ln('file1.txt','iamlink')

dl.ls()
file1.txt, iamlink

load(name=None, inpfmt=None, fmt='pandas', ext=None)[source]¶

Save the string representation of a data object to a file in VOSpace.

Parameters

name (str) – The name of the file in load into memory. The vos:// prefix is necessary otherwise it is assumed a local file that should be read. Currently only FITS binary tables and CSV file formats are supported.
inpfmt (str) – Tne input format type. The currently supported types are FITS binary tables, HDF5, CSV, and ASCII files (‘string’). If this is not specified then the file extension of name is used to attempt to figure out the format.
fmt (str) –
The data type format to output. Permitted values are:
- ’csv’ the returned result is a comma-separated string that looks like a
  csv file (newlines at the end of every row)
- ’ascii’ same as csv but tab-delimited
- ’string’ just a straight read of ASCII into a string
- ’array’ Numpy array
- ’structarray’ Numpy structured / record array
- ’pandas’ a Pandas data frame
- ’table’ in Astropy Table format
- ’votable’ result is a string XML-formatted as a VO table
The output type for a FITS image is a numpy array. For other data ‘pandas’ is the default format.
ext (int) – The FITS extension to load for images. The default is 1 for a FITS binary table and 0 for a FITS image.

Example

Load the file “output.fits” into a pandas data frame.

df = dl.load('output.fits',fmt='pandas')

Load a FITS image “im1.fits”.

im,head = dl.load('im1.fits')

login(user=None)[source]¶

Login to Data Lab using username.

Parameters: user (str) – The Data lab username. If this is not given, then the user will be prompted for the information.

Example

Login and give the username,

dl.login(‘myusername’) Enter password: *** Welcome to the Data Lab, myusername

or,

dl.login() Enter user: myusername Enter password: ** Welcome to the Data Lab, myusername

logout(unmount=None, verbose=True)[source]¶

Logout out of the Data Lab.

Example

Logout of Data Lab.

dl.logout()
'myusername' is now logged out of the Data Lab

ls(name='vos://', format='csv', verbose=False)[source]¶

List files in VOSpace.

Parameters

name (str) – The name of a specific file to list. If name is blank then all files will be listed.
format (str) – The format to use.
verbose (bool) – Give more verbose output, or just a list of files. The default is verbose=True.

Returns

results – The list of files in VOSpace.

Return type

str

Example

List the files.

dl.ls()
test2  test1

Verbose listing of the files in the public/ directory.

dl.ls('public',verbose=True)
lrw-rw----  demo15      0B  17 May 2017 14:04:25  thisisalsoalink -> /public/smash2
lrw-rw----  demo15      0B  17 May 2017 13:58:04  thisisalink -> /smash1
-rw-rw-r--  demo15    3.4K  17 May 2017 09:40:13  smash2
-rw-rw-r--  demo15    3.4K  17 May 2017 07:34:54  smash1
drw-rw----  demo15      0B  17 May 2017 14:05:02  data/  tableingester,downloader,runner

mkdir(name=None)[source]¶

Create a directory in Data Lab VOSpace.

Parameters: name (str) – The name of the directory in VOSpace to create, e.g. results.

Example

Create the directory data1/.

dl.mkdir('data1')

mv(source=None, destination=None, verbose=True)[source]¶

Move a file in Data Lab VOSpace.

Parameters

source (str) – The name the file in VOSpace to move/rename, e.g. file1.txt.
destination (str) – The new name of the file in VOSpace (e.g. newfile1.txt) or the directory to move it to.

Example

Rename the file file.txt to newfile.txt.

dl.ls()
file.txt

dl.mv('file.txt','newfile.txt')

dl.ls()
newfile.txt

Move the file output.fits to the results/ directory.

dl.ls()
output.txt, results

dl.mv('output.fits','results/output.fits')

dl.ls()
results/output.txt

put(source=None, destination=None, verbose=True)[source]¶

Put files into Data Lab VOSpace.

Parameters

source (str) – The name of a local file to upload to VOSpace, e.g. file1.txt.
destination (str) – The name of the destination file with, e.g. file2.txt. The destination file can have the vos:// prefix but it is not required.

Example

Put a catalog called cat.fits into VOSpace.

dl.put('cat.fits','cat.fits')
(1 / 1) cat.fits -> vos://cat.fits

query(query=None, qtype='sql', fmt='csv', out=None, async_=False, drop=False, profile='default', verbose=True, **kw)[source]¶

Send a query to a remote query service.

Parameters

query (str) –
The query string that will be passed to the queryClient and then to the DB query manager. This can either be in the SQL or ADQL format (specified by the “type” parameter). For example,
```
'select ra,dec from gaia_dr1.gaia_source limit 3'
```
qtype (str) – The query format, SQL or ADQL. SQL is used by default.
fmt (str) –

Format of the result to be returned by the query. Permitted values are.
For file output and direct output to python: * ‘csv’ the returned result is a comma-separated string that looks like a csv file (newlines at the end of every row) * ‘ascii’ same as csv but tab-delimited * ‘votable’ result is a string XML-formatted as a VO table Only for direct output to python: * ‘array’ Numpy array * ‘structarray’ Numpy structured / record array * ‘pandas’ a Pandas data frame * ‘table’ in Astropy Table format Only for file output: * ‘fits’ FITS binary table. Only if the results are saved to a file with out=. * ‘hdf5’ HDF5 file. Only if the results are saved to a file with out=.
out (str or None) – The output name if the results are to be saved to mydb (mydb://tablename), to VOSpace (vos://filename), or the local file system (file:// and other names with no prefix). The files are in csv format.
async_ (bool) –
If True, the query is asynchronous, i.e. a job is submitted to the DB, and a jobID is returned. The jobID must be then used to check the query’s status and to retrieve the result (when status is COMPLETE). Default is False, i.e. synchroneous query.

async_ replaces the previous async parameter, because async was promoted to a keyword in Python 3.7. Users of Python versions prior to 3.7 can continue to use the async keyword.
drop (bool) – If True, then if the query is saving to mydb where the same table name already exists, it will overwrite the old mydb table.

Returns

result – If async_=False and out is not used, then the return value is the result of the query in the requested format (see fmt). If out is given then the query result is saved to a file or mydb. If async_=True the jobID is returned with which later the asynchronous query’s status can be checked (dl.querystatus()), and the result retrieved (see dl.queryresults().

Return type

str

Example

A simple query returned as a pandas data frame.

data = dl.query('SELECT * from smash_dr1.source LIMIT 100',fmt='pandas')
Returning Pandas dataframe

type(data)
pandas.core.frame.DataFrame

print data['ra'][0:3]
0    103.068355
1    103.071774
2    103.071598

Perform a query and save the results to a table called “table1.txt” in mydb.

res = dl.query('SELECT * from smash_dr1.source LIMIT 100',out='mydb://table1.txt')

 dl.listmydb()

Perform the same query and save it to a local file.

res = dl.query('SELECT * from smash_dr1.source LIMIT 100',out='table1.txt')

ls
table1.txt

queryhistory(async_=None, **kw)[source]¶

Report the history of queries made so far.

Parameters

async (bool) –
A boolean (True/False) of whether to only show the ASYNC queries. By default all queries are shown.

async_ replaces the previous async parameter, because async was promoted to a keyword in Python 3.7. Users of Python versions prior to 3.7 can continue to use the async keyword.
Results –
------- –
information on part queries is output to the screen with the following (The) –
columns (query ID, submission time, query type (sql/adql), sync or async query, jobid (for async queries),) – output format, status of query (or number of returned rows if sync query), query string

Examples

Perform some queries and then list the history.

data1 = dl.query('select ra,dec from smash_dr1.source limit 100',fmt='csv')
Returning CSV formatted table as a string

data2 = dl.query('select ra,dec from smash_dr1.source limit 500',fmt='pandas')
Returning Pandas dataframe

data3 = dl.query('select ra,dec from smash_dr1.source limit 1000',fmt='structarray')
Returning Numpy structured / record array

dl.queryhistory()
1  2017-05-16 13:27:34  sql  SYNC  pandas  100  --  'select ra,dec,gmag from smash_dr1.object limit 100'
2  2017-05-16 13:27:40  sql  SYNC  csv  500  --  'select ra,dec,gmag from smash_dr1.object limit 500'
3  2017-05-16 13:27:46  sql  SYNC  structarray  1000  --  'select ra,dec,gmag from smash_dr1.object limit 1000'

queryprofiles(profile=None)[source]¶

List the available Query Manager profiles to use with a dl.query().

Parameters: profile (str) – The name of a specific Query Manager profile to check. If this is blank then all of the available profile names will be listed.
Returns: results – The list of properties of profile profile, or a list of all available profiles.
Return type: str

Example

List of available profiles.

dl.queryprofiles()
default,IRSA,HEASARC,Vizier,GAVO,SIMBAD,zeus1,SDSS-DR9,STScI-RegTAP,GALEX-DR6,dldb1

Get profile information on profile dldb1.

dl.queryprofiles(‘dldb1’) {u’accessURL’: u’http://dldb1.sdm.noao.edu:8080/ivoa-dal/tap’, u’dbport’: 5432, u’password’: u’datalab’, u’description’: u’Development NOAO Data Lab TAP Service / Database on dldb1’, u’database’: u’tapdb’, u’host’: u’dldb1.sdm.noao.edu’, u’vosRoot’: u’vos://datalab.noao!vospace’, u’vosEndpoint’: u’http://dldb1.sdm.noao.edu:8080/vospace-2.0/vospace’, u’user’: u’dlquery’, u’vosRootDir’: u’/data/vospace/users’, u’type’: u’datalab’}

queryresults(jobid=None)[source]¶

Get the async query results.

Parameters: jobid (str) – This can be either (1) the Query ID (QID) returned by the dl.queryhistory() command, or (2) the unique job identifier for the asynchronous query which was returned by ql.query() when the query job was submitted.
Returns: result – The result of the query in the requested format (see fmt in dl.query().
Return type: str

Example

Submit an asynchronous query and then check the status.

jobid = dl.query('SELECT ra,dec from smash_dr1.source LIMIT 3',async_=True)
Asynchronous query JobID = uqrcs8a5n8s6d0je

dl.querystatus(jobid)
COMPLETED

results = dl.queryresults(jobid)
print results
ra,dec
103.068354922718,-37.973538878907299
103.071774116284,-37.973599429479599
103.071597827998,-37.972329108796401

querystatus(jobid=None)[source]¶

Get the async query job status.

Parameters: jobid (str) – This can be either (1) the Query ID (QID) returned by the dl.queryhistory() command, or (2) the unique job identifier for the asynchronous query which was returned by ql.query() when the query job was submitted.
Returns: status – The status of the query, which can be one of the following: QUEUED the query job is in queue and waiting to be executed. EXECUTING the query job is currently running. COMPLETED the query is done and the results are ready to be retrieved with dl.queryresults(). ERROR there was a problem with the query.
Return type: str

Example

Submit an asynchronous query and then check the status.

jobid = dl.query('SELECT ra,dec from smash_dr1.source LIMIT 100',async_=True)
Asynchronous query JobID = uqrcs8a5n8s6d0je

dl.querystatus(jobid)
COMPLETED

rm(name=None, verbose=True)[source]¶

Delete files in Data Lab VOSpace.

Parameters: name (str) – The name of the file in VOSpace to delete, e.g. file1.txt.

Example

Delete the file file1.txt.

dl.ls()
file1.txt, file2.txt

dl.rm('file1.txt')

dl.ls()
file2.txt

rmdir(name=None)[source]¶

Delete a directory in Data Lab VOSpace.

Parameters: name (str) – The name of the directory in VOSpace to delete, e.g. results.

Example

Delete the directory data1/.

dl.rmdir('data1')

save(data=None, name=None, fmt=None, clobber=False)[source]¶

Save the string representation of a data object to a file in VOSpace.

Parameters

data (str) – The data object such as a pandas data frame or numpy structured array.
name (str) – The name of the file in VOSpace to create. The vos:// prefix is not necessary.
fmt (str) –
The format to use for the output file. If this is not specified then the file extension of name is used to attempt to figure out the format. The currently supported input and output formats are:

Data type Format csv csv ascii ascii array csv/fits structarray csv/fits pandas csv/fits/hdf5 table csv/fits/hdf5 votable csv/fits/hdf5
clobber (bool) – Whether to overwrite an existing file. The default is False.

Example

Save the pandas data frame called “df” to a file called “data1.csv” in VOSpace.

dl.save(df,'data1.csv')

schema(val='', fmt='text', profile='default')[source]¶

Print information about data service schema.

Parameters

val (str) – Value to list ([[<schema>][.<table>][.<col>]]).
fmt (str) – Output format (csv|text|json).
profile (str) – Service profile.

Returns

results – The schema information is printed to the screen.

Return type

str

Example

Print out all the DL tables.

 datalab schema

 Schema Name   Description
-----------   -----------
   gaia_dr1   GAIA Data Release 1
       ivoa   IVOA ObsCore tables
   des_sva1   DES SVA1 Data Products
 tap_schema   TAP Schema Tables
       usno   USNO Astrometry Catalogs
  sdss_dr13
    neo_dr1   NEO Survey Data Release 1
     ls_dr3   The DECam Legacy Survey Data Release 3
  smash_dr1   SMASH Data Release 1

List all tables in a schema/catalog.

datalab schema val=smash_dr1

Schema: smash_dr1

Table Name   Description
----------   -----------
      chip   Info on each chip in the frame
  exposure   Info on each exposure
     field   Info on each target field (position, Num exposures, etc)
    object   Average photometry of each unique object
    source   All of the individual source measurements
     stars   View of object table to select for stars
  galaxies   View of object table to select for galaxies
    xmatch   Crossmatch of object against GAIA DR1 and WISE

servicestatus()[source]¶: This checks on the status of the DL services.

siaquery(ra=None, dec=None, dist=None, verbose=False)[source]¶

Perform a SIA query with a set of coordinates or from an uploaded file.

Parameters

ra (float) – The right ascension (in degrees) of the point to use for the search.
dec (float) – The declination (in degrees) of the point to use for the search.
dist (float) – The search distance (radius) in degrees. The default is 0.0085 deg.
verbose (bool) – Use verbose output. The default is False.

Returns

images – The list of images in Astropy table format.

Return type

votable

Example

Perform a simple SIA search.

itab = dl.siaquery(0.5,10.0,0.1)
The image list contains 6 entries

Download the first image using copyurl().

dl.copyurl(itab['access_url'][0],'im1.fits')

status()[source]¶

Print the status of the Data Lab connection.

Example

The “myusername” is logged in.

dl.status()
User myusername is logged into the Data Lab

No user is currently logged in.

dl.status()
No user is currently logged into the Data Lab

whoami()[source]¶

Print the current active user.

Example

dl.whoami()
myusername

dl.dlinterface.addFormatMapping(self)[source]¶: Add the format mapping information to the DL object

dl.dlinterface.areLoginsWorking()[source]¶: This checks if the Authentication Manager is returning proper tokens.

dl.dlinterface.areSyncQueriesWorking()[source]¶: This checks if the Query Manager is returning proper Sync queries.

dl.dlinterface.checkLogin(self)[source]¶: Check if the user is already logged in. If not, give a warning message

dl.dlinterface.convertTableToFormat(t, format)[source]¶: Convert Astropy table to a different format using StringIO and write method.

dl.dlinterface.convert_vospace_time_to_seconds(str_date)[source]¶

A convenience method that takes a string from a vospace time field and converts it to seconds since epoch.

Parameters: str_date (str) – string to parse into a VOSpace time
Returns: A datetime object for the provided string date
Return type: datetime

dl.dlinterface.getNodeInfo(self, xnode, lenpathbase, verbose=True)[source]¶: Get information on a node. The input is a “node” element of a XML ElementTree.

dl.dlinterface.getUserName(self)[source]¶: Get the currently logged-in user token. If we haven’t logged in return the anonymous username.

dl.dlinterface.getUserToken(self)[source]¶: Get the currently logged-in user token. If we haven’t logged in return the anonymous token.

dl.dlinterface.isListWorking()[source]¶: This checks if the Storage Manager is returning proper list queries:

dl.dlinterface.isTapWorking()[source]¶: This checks if the TAP service and Tomcat are running.

dl.dlinterface.readAscii(filename)[source]¶: Read in an ASCII file and return the data

dl.dlinterface.reformatQueryOutput(self, res=None, fmt='csv', verbose=True)[source]¶: Reformat the output of a query based on a format.

dl.dlinterface.writeAscii(filename, txt)[source]¶: Write data to ASCII file

dl.dltasks module¶

class dl.dltasks.AddCapability(datalab)[source]¶

Bases: dl.dltasks.Task

Add a capability to a VOSpace container

run()[source]¶

class dl.dltasks.Broadcast(datalab)[source]¶

Bases: dl.dltasks.Task

Broadcast a SAMP message

run()[source]¶

class dl.dltasks.Copy(datalab)[source]¶

Bases: dl.dltasks.Task

Copy a file in Data Lab

run()[source]¶

class dl.dltasks.DataLab[source]¶

Bases: object

Main class for Data Lab interactions

get(section, param)[source]¶: Get a value from the configuration file.

save(section, param, value)[source]¶: Save the configuration file.

class dl.dltasks.Delete(datalab)[source]¶

Bases: dl.dltasks.Task

Delete files in Data Lab

run()[source]¶

class dl.dltasks.DropMyDB(datalab)[source]¶

Bases: dl.dltasks.Task

Drop a user’s MyDB table. [DEPRECATED]

run()[source]¶

class dl.dltasks.Get(datalab)[source]¶

Bases: dl.dltasks.Task

Get one or more files from Data Lab.

run()[source]¶

class dl.dltasks.Init(datalab)[source]¶

Bases: dl.dltasks.Task

Print the current service URLS in use.

run()[source]¶: Initialize the Data Lab configuration.

class dl.dltasks.Launch(datalab)[source]¶

Bases: dl.dltasks.Task

Launch a plugin in Data Lab

run()[source]¶

class dl.dltasks.LaunchJob(datalab)[source]¶

Bases: dl.dltasks.Task

Execute a remote processing job in the Data Lab

getJob(job)[source]¶

run()[source]¶

validJob(job)[source]¶

class dl.dltasks.Link(datalab)[source]¶

Bases: dl.dltasks.Task

Link a file in Data Lab

run()[source]¶

class dl.dltasks.List(datalab)[source]¶

Bases: dl.dltasks.Task

List files in Data Lab

run()[source]¶

class dl.dltasks.ListCapability(datalab)[source]¶

Bases: dl.dltasks.Task

Add a capability to a VOSpace container

run()[source]¶

class dl.dltasks.ListMyDB(datalab)[source]¶

Bases: dl.dltasks.Task

List the user’s MyDB tables. [DEPRECATED]

run()[source]¶

class dl.dltasks.Login(datalab)[source]¶

Bases: dl.dltasks.Task

Log into the Data Lab

do_login()[source]¶

login()[source]¶: Login to the Data Lab.

run()[source]¶: Execute the Login Task.

class dl.dltasks.Logout(datalab)[source]¶

Bases: dl.dltasks.Task

Logout out of the Data Lab

run()[source]¶

class dl.dltasks.MkDir(datalab)[source]¶

Bases: dl.dltasks.Task

Create a directory in Data Lab

run()[source]¶

class dl.dltasks.Mountvofs(datalab)[source]¶

Bases: dl.dltasks.Task

Mount a VOSpace via FUSE

run()[source]¶

class dl.dltasks.Move(datalab)[source]¶

Bases: dl.dltasks.Task

Move files in Data Lab

run()[source]¶

class dl.dltasks.MyDB_Copy(datalab)[source]¶

Bases: dl.dltasks.Task

Copy a user MyDB table.

run()[source]¶

class dl.dltasks.MyDB_Create(datalab)[source]¶

Bases: dl.dltasks.Task

Create a user MyDB table.

run()[source]¶

class dl.dltasks.MyDB_Drop(datalab)[source]¶

Bases: dl.dltasks.Task

Drop a user’s MyDB table.

run()[source]¶

class dl.dltasks.MyDB_Import(datalab)[source]¶

Bases: dl.dltasks.Task

Import data into a user MyDB table.

run()[source]¶

class dl.dltasks.MyDB_Index(datalab)[source]¶

Bases: dl.dltasks.Task

Index data in a user’s MyDB table.

run()[source]¶

class dl.dltasks.MyDB_Insert(datalab)[source]¶

Bases: dl.dltasks.Task

Insert data into a user MyDB table.

run()[source]¶

class dl.dltasks.MyDB_List(datalab)[source]¶

Bases: dl.dltasks.Task

List the user’s MyDB tables.

run()[source]¶

class dl.dltasks.MyDB_Rename(datalab)[source]¶

Bases: dl.dltasks.Task

Rename a user MyDB table.

run()[source]¶

class dl.dltasks.MyDB_Truncate(datalab)[source]¶

Bases: dl.dltasks.Task

Truncate a user MyDB table.

run()[source]¶

class dl.dltasks.Option(name, value, description, display=None, default=None, required=False)[source]¶

Bases: object

Represents an option

class dl.dltasks.Put(datalab)[source]¶

Bases: dl.dltasks.Task

Put files into Data Lab.

run()[source]¶

class dl.dltasks.Query(datalab)[source]¶

Bases: dl.dltasks.Task

Send a query to a remote query service (OLD VERSION - NOT USED))

dbquery(h, url, token)[source]¶

httpquery(h, url, token)[source]¶

ivoquery(h, uri, token, out)[source]¶

run()[source]¶

class dl.dltasks.Query2(datalab)[source]¶

Bases: dl.dltasks.Task

Send a query to a remote query service. [Note: placeholder name until we figure out what to do with the old Query() functionality.]

run()[source]¶

class dl.dltasks.QueryProfiles(datalab)[source]¶

Bases: dl.dltasks.Task

List the available Query Manager profiles.

run()[source]¶

class dl.dltasks.QueryResults(datalab)[source]¶

Bases: dl.dltasks.Task

Get the async query results.

run()[source]¶

class dl.dltasks.QueryStatus(datalab)[source]¶

Bases: dl.dltasks.Task

Get the async query job status.

run()[source]¶

class dl.dltasks.Receiver(client)[source]¶

Bases: object

SAMP listener

point_select(private_key, send_id, mtype, params, extra)[source]¶

receive_notifications(private_key, sender_id, mtype, params, extra)[source]¶

receiver_call(private_key, sender_id, msg_id, mtype, params, extra)[source]¶

class dl.dltasks.Resolve(datalab)[source]¶

Bases: dl.dltasks.Task

Resolve a vos short form identifier – FIXME

run()[source]¶

class dl.dltasks.RmDir(datalab)[source]¶

Bases: dl.dltasks.Task

Delete a directory in Data Lab

run()[source]¶

class dl.dltasks.Schema(datalab)[source]¶

Bases: dl.dltasks.Task

Print information about data servicce schema

run()[source]¶

class dl.dltasks.Services(datalab)[source]¶

Bases: dl.dltasks.Task

Print the available data services.

run()[source]¶

class dl.dltasks.SiaQuery(datalab)[source]¶

Bases: dl.dltasks.Task

SIA query with an uploaded file

getUID()[source]¶: Get the UID for this user

run()[source]¶

class dl.dltasks.Status(datalab)[source]¶

Bases: dl.dltasks.Task

Status of the Data Lab connection

run()[source]¶

class dl.dltasks.StorageProfiles(datalab)[source]¶

Bases: dl.dltasks.Task

List the available Storage Manager profiles.

run()[source]¶

class dl.dltasks.SvcURLs(datalab)[source]¶

Bases: dl.dltasks.Task

Print the current service URLS in use.

run()[source]¶

class dl.dltasks.Tag(datalab)[source]¶

Bases: dl.dltasks.Task

Tag a file in Data Lab

run()[source]¶

class dl.dltasks.Task(datalab, name, description)[source]¶

Bases: object

Superclass to represent a Task

addLogger(logLevel, logFile)[source]¶: Add a Logger to the Task.

addOption(name, option)[source]¶: Add an option to the Task.

addStdOptions()[source]¶

broadcast(client, messageType, params, app=None)[source]¶: Broadcast a SAMP message.

getSampConnect()[source]¶: Get a SAMP listening client connection.

listen(client)[source]¶: Setup a listener for a specific SAMP message.

report(resp)[source]¶: Handle call response

run()[source]¶

setLogger(logLevel=None)[source]¶: Set the logger to be used.

setOption(name, value)[source]¶: Set a Task option.

class dl.dltasks.Version(datalab)[source]¶

Bases: dl.dltasks.Task

Print the task version.

run()[source]¶

class dl.dltasks.WhoAmI(datalab)[source]¶

Bases: dl.dltasks.Task

Print the current active user.

run()[source]¶

dl.dltasks.getUserName(self)[source]¶: Get the currently logged-in user token. If we haven’t logged in return the anonymous token.

dl.dltasks.getUserToken(self)[source]¶: Get the currently logged-in user token. If we haven’t logged in return the anonymous token.

dl.dltasks.parseSelf(obj)[source]¶

dl.queryClient module¶

dl.queryClient.abort[source]¶

Abort the specified asynchronous job.

Usage:: abort (token=None, jobId=None)

queryClient.abort (token, jobId) queryClient.abort (jobId) queryClient.abort (token, jobId=<id>) queryClient.abort (jobId=<str>)

Parameters

token (str) – Authentication token (see function authClient.login())
jobId (str) – The jobId to abort.

Returns

results

Return type

str

Example

# issue an async query (here a tiny one just for this example)
query = 'select ra,dec from gaia_dr1.gaia_source limit 3'
jobId = queryClient.query(token, adql=query, fmt='csv', async=True)

# ensure job completes...then check status and retrieve results
time.sleep(4)
if queryClient.status(token, jobId) == 'COMPLETED':
    results = queryClient.results(token,jobId)
    print type(results)
    print results

This prints

<type 'str'>
ra,dec
301.37502633933002,44.4946851014515588
301.371102372343785,44.4953207577355698
301.385106974224186,44.4963443903961604

dl.queryClient.drop[source]¶

Drop the specified table from the user’s MyDB

Usage:: drop (table=None, token=None)

queryClient.drop (token, table) queryClient.drop (table) queryClient.drop (token, table=<id>) queryClient.drop (table=<str>)

Parameters

token (str) – Authentication token (see function authClient.login())
table (str) – The specific table to drop

Example

# List the tables
queryClient.drop('foo1')

dl.queryClient.error[source]¶

Retrieve the error of an asynchronous query, once completed.

Usage:: error (token=None, jobId=None)

queryClient.error (jobId) queryClient.error (token, jobId) queryClient.error (token, jobId=<id>) queryClient.error (jobId=<str>)

Parameters

token (str) – Authentication token (see function authClient.login())
jobId (str) – The jobId returned when issuing an asynchronous query via queryClient.query() with async=True.

Returns

error

Return type

str

Example

# issue an async query (here a tiny one just for this example)
query = 'select ra,dec from gaia_dr1.gaia_source limit 3'
jobId = queryClient.query(token, adql=query, fmt='csv', async=True)

# ensure job completes...then check status and retrieve error
time.sleep(4)
if queryClient.status(token, jobId) == 'ERROR':
    error = queryClient.error(token,jobId)
    print type(error)
    print error

This prints

<type 'str'>
ra,dec
301.37502633933002,44.4946851014515588
301.371102372343785,44.4953207577355698
301.385106974224186,44.4963443903961604

dl.queryClient.getClient(profile='default', svc_url='https://datalab.noao.edu/query')[source]¶: Create a new queryClient object and set a default profile.

dl.queryClient.get_profile()[source]¶

Get the current query profile.

Parameters: None –
Returns: profile – The name of the current profile used with the Query Manager service
Return type: str

Example

print ("Query Service profile = " + queryClient.get_profile())

dl.queryClient.get_svc_url()[source]¶

Get the currently-used Query Manager serice URL.

Parameters: None –
Returns: service_url – The currently-used Query Service URL.
Return type: str

Example

print (queryClient.get_svc_url())

dl.queryClient.get_timeout_request()[source]¶

Get the current Sync query timeout value.

Parameters: None –
Returns: result – Current sync query timeout value.
Return type: int

Example

# get the current timeout value
print (queryClient.get_timeout_request())

dl.queryClient.isAlive(svc_url='https://datalab.noao.edu/query', timeout=5)[source]¶

Check whether the QueryManager service at the given URL is: alive and responding. This is a simple call to the root service URL or ping() method.

Parameters

svc_url (str) – The Query Service URL to ping.
timeout (int) – Call will assume to have failed if ‘timeout’ seconds pass.

Returns

result – True if service responds properly, False otherwise

Return type

bool

Example

if queryClient.isAlive():
    print ("Query Manager is alive")

dl.queryClient.jobs[source]¶

Get a list of the user’s Async jobs.

Usage:: jobs (token=None, jobId=None, format=’text’, status=’all’)

queryClient.jobs (jobId) queryClient.jobs (token, jobId) queryClient.jobs (token, jobId=<id>) queryClient.jobs (jobId=<str>)

Use the authentication token and the jobId of a previously issued asynchronous query to check the query’s current status.

Parameters

token (str) – Authentication token (see function authClient.login())
jobId (str) – The jobId returned when issuing an asynchronous query via queryClient.query() with async=True.
format (str) – Format of the result. Support values include ‘text’ for a simple formatted table suitable for printing, or ‘json’ for a JSON string of the full matching record(s).
status (str) –
If status=’all’ then all async jobs are returned, otherwise this value may be used to return only those jobs with the specified status. Allowed values are:

all Return all jobs EXECUTING Job is still running COMPLETED Job completed successfully ERROR Job exited with an error ABORTED Job was aborted by the user
option (str) – If ‘list’ then the matching records are returned, if ‘delete’ then the records are removed from the database (e.g. to clear up long job lists of completed jobs).

Returns

joblist – Returns a list of async query jobs submitted by the user in the last 30 days, possibly filtered by the ‘status’ parameter. The ‘json’ format option allows the caller to format the full contents of the job record beyond the supplied simple ‘text’ option.

Return type

str

Example

print (queryClient.jobs(token, jobId))

This prints

JobID              Start              End                 Status
tfu8zpn2tkrlfyr9e  07-22-20T13:10:22  07-22-20T13:34:12   COMPLETED
k8uznptrkkl29ryef  07-22-20T14:09:45                      EXECUTING
      :                   :                 :                :

dl.queryClient.list[source]¶

List the tables or table schema in the user’s MyDB.

Usage:: list (table=None, token=None)

queryClient.list (token, table) queryClient.list (table) queryClient.list (token, table=<id>) queryClient.list ()

Parameters

token (str) – Authentication token (see function authClient.login())
table (str) – The specific table to list (returns the table schema), or an empty string to return a list of the names of all tables.

Returns

listing – The list of tables in the user’s MyDB or the schema of the named table

Return type

str

Example

# List the tables
queryClient.list()

dl.queryClient.list_profiles[source]¶

Retrieve the profiles supported by the query manager service.

Usage:: list_profiles (token=None, profile=None, format=’text’)

queryClient.list_profiles (token) queryClient.list_profiles ()

Parameters

token (str) – Authentication token (see function authClient.login())
profile (str) – A specific profile configuration to list. If None, a list of profiles available to the given auth token is returned.
format (str) – Result format: One of ‘text’ or ‘json’

Returns

profiles – A list of the names of the supported profiles or a dictionary of the specific profile

Return type

list/dict

Example

profiles = queryClient.list_profiles()
profiles = queryClient.list_profiles(token)

dl.queryClient.mydb_copy[source]¶

Copy a table in the user’s MyDB to a new name

Usage:: mydb_copy (source, target, token=None)

queryClient.mydb_copy (token, source, target) queryClient.mydb_copy (source, target)

Parameters

token (str) – Authentication token (see function authClient.login())
source (str) – The old table name, i.e. the table to be copied
target (str) – The new table name, i.e. the table to be created

Example

# Copy table 'foo' to a new table, 'bar'
queryClient.mydb_copy ('foo', 'bar')

dl.queryClient.mydb_create[source]¶

Create a table in the user’s MyDB

Usage:: mydb_create (table, schema, token=None, **kw)

queryClient.mydb_create (token, table, <schema_dict>) queryClient.mydb_create (table, <schema_dict>)

Parameters

token (str) – Authentication token (see function authClient.login())
table (str) – The name of the table to create
schema (str or dict) – The schema is CSV text containing the name of the column and it’s PostgreSQL data type. If set as a ‘str’ type it is either a CSV string, or the name of a file containing the CSV. If passed as a ‘dict’ type, it is a dictionary object where keys are the column names and values are the data types.
drop (bool (optional)) – Drop any existing table of the same name before creating new one.

Example

# Create table in MyDB named 'foo' with columns defined by
# the file 'schema.txt'.  The schema file contains:
#
#     id,text
#     ra,double precision
#     dec,double precision
#
queryClient.mydb_create ('foo', 'schema.txt')

dl.queryClient.mydb_drop[source]¶

Drop the specified table from the user’s MyDB

Usage:: mydb_drop (table, token=None)

queryClient.mydb_drop (token, table) queryClient.mydb_drop (table) queryClient.mydb_drop (token, table=<id>)

Parameters

token (str) – Authentication token (see function authClient.login())
table (str) – The specific table to drop

Example

# Drop the 'foo1' table
queryClient.drop('foo1')

dl.queryClient.mydb_flush(token=None)[source]¶

dl.queryClient.mydb_import[source]¶

Import data into a table in the user’s MyDB

Usage:: mydb_import (table, data, token=None, **kw)

queryClient.mydb_import (token, table, data) queryClient.mydb_import (table, data)

Parameters

token (str) – Authentication token (see function authClient.login())
table (str) – The name of the table to be loaded
data (str or data object) –
The data file or python object to be loaded. The ‘data’ value may be one of the following types:

filename A CSV file of data string A string containing CSV data Pandas DataFrame A Pandas DataFrame object

: : : :

Additional object types can be added provided the data can be converted to a CSV format.
schema (str [OPTIONAL]) – If set, this is a filename or string containing a schema for the data table to be created. A schema contains a comma-delimited row for each column containing the column name and it’s Postgres data type. If not set, the schema is determined automatically from the data.
drop (bool [Optional]) – Drop any existing table of the same name before creating new one.
verbose (bool [Optional]) – Be verbose about operations.

Returns

schema A string containing the table schema
data_obj The CSV data to be imported (possibly converted)

Example

# Import data into a MyDB table named 'foo' from file 'data.csv'.
schema, data = queryClient.mydb_import ('foo', 'data.csv')

dl.queryClient.mydb_index[source]¶

Index the specified column in a table in the user’s MyDB

queryClient.mydb_index (table, colunm) queryClient.mydb_index (token, table, column) queryClient.mydb_index (table, column, token=None)

Parameters

token (str) – Authentication token (see function authClient.login())
table (str) – The table to be indexed
column (str) – The column
q3c (str) – A comma-delimited list of two column names giving the RA and Dec positions (decimal degrees) to be used to Q3C index the table. If None, no Q3C index will be computed.
cluster (bool) – If enabled, data table will be rewritten to cluster on the Q3C index for efficiency. Only used when ‘q3c’ columns are specified.
async_ (bool) – If enabled, index commands will be submitted asynchronously.

Returns

Return type

command status

Example

# Index the table's "id" column
queryClient.index('foo1', 'id')

# Index and cluster the table by position
queryClient.index('foo1', q3c='ra,dec', cluster=True)

dl.queryClient.mydb_insert[source]¶

Insert data into a table in the user’s MyDB

Usage:: mydb_insert (table, data, token=None, **kw)

queryClient.mydb_insert (token, table, <filename>) queryClient.mydb_insert (token, table, <data_object>) queryClient.mydb_insert (table, <filename>) queryClient.mydb_insert (table, <data_object>)

Parameters

token (str) – Authentication token (see function authClient.login())
table (str) – The name of the table to append
data (str or data object) – The schema is CSV text containing the name of the column and it’s PostgreSQL data type. If set as a ‘str’ type it is either a CSV string, or the name of a file containing the CSV data. If passed as a tabular data object, it is converted to CSV and sent to the service.
csv_header (bool [OPTIONAL]) – If True, then the CSV data object contains a CSV header line, i.e. the first line is a row of column names. Otherwise, no column names are assumed and the column order must match the table schema.

Example

# Insert data into a MyDB table named 'foo'.
queryClient.mydb_insert ('foo', 'data.csv')

dl.queryClient.mydb_list[source]¶

List the tables or table schema in the user’s MyDB.

Usage:: mydb_list (table=None, token=None)

queryClient.mydb_list (table) queryClient.mydb_list (token, table=<str>) queryClient.mydb_list (table=<str>)

Parameters

token (str) – Authentication token (see function authClient.login())
table (str) – The specific table to list (returns the table schema), or an empty string to return a list of the names of all tables.

Returns

listing – The list of tables in the user’s MyDB or the schema of the named table

Return type

str

Example

# List the tables
queryClient.mydb_list()

dl.queryClient.mydb_rename[source]¶

Rename a table in the user’s MyDB to a new name

Usage:: mydb_rename (source, target, token=None)

queryClient.mydb_rename (token, source, target) queryClient.mydb_rename (source, target)

Parameters

token (str) – Authentication token (see function authClient.login())
source (str) – The old table name
target (str) – The new table name

Example

# Copy table 'foo' to a new table, 'bar'
queryClient.mydb_rename ('foo', 'bar')

dl.queryClient.mydb_truncate[source]¶

Truncate the specified table in the user’s MyDB

Usage:: mydb_truncate (table, token=None)

queryClient.mydb_truncate (token, table) queryClient.mydb_truncate (table) queryClient.mydb_truncate (token, table=<id>)

Parameters

token (str) – Authentication token (see function authClient.login())
table (str) – The specific table to drop

Example

# Truncate the table 'foo'
queryClient.truncate('foo')

dl.queryClient.qcToString(s)[source]¶: qcToString – Force a return value to be type ‘string’ for all Python versions.

dl.queryClient.query[source]¶

Send an SQL or ADQL query to the database or TAP service.

Usage:

query (token=None, adql=None, sql=None, fmt=’csv’, out=None,: async_=False, drop=False, profile=’default’, **kw):

queryClient.query (token, query, <args>) queryClient.query (token | query, <args>)

Parameters

token (str) – Secure token obtained via authClient.login()
adql (str or None) –
ADQL query string that will be passed to the DB query manager, e.g.
```
adql='select top 3 ra,dec from gaia_dr1.gaia_source'
```
sql (str or None) –
SQL query string that will be passed to the DB query manager, e.g.
```
sql='select ra,dec from gaia_dr1.gaia_source limit 3'
```
fmt (str) –
Format of result to be returned by the query. Permitted values are:
- ’csv’ The returned result is a comma-separated string
  that looks like a csv file (newlines at the end of every row) [DEFAULT]
- ’csv-noheader’ A csv result with no column headers (data only)
- ’ascii’ Same, but the column separator is a tab
- ’array’ Returns a NumPy array
- ’pandas’ Returns a Pandas DataFrame
- ’structarray’ Numpy structured array (aka ‘record array’)
- ’table’ Returns an Astropy Table object
  
  The following formats may be used when saving a file to virtual storage on the server:
- ’fits’ FITS binary
- ’hdf5’ HDF5 file (NOT YET IMPLEMENTED)
- ’votable’ An XML-formatted VOTable
out (str or None) – The output filename to create on the local machine, the URI of a VOSpace or MyDB resource to create, or None if the result is to be returned directly to the caller.
async_ (bool) –
If True, the query is Asynchronous, i.e. a job is submitted to the DB, and a jobID token is returned the caller. The jobID must be then used to check the query’s status and to retrieve the result (when the job status is COMPLETED) or the error message (when the job status is ERROR). Default is False, i.e. the task runs a Synchroneous query.

async_ replaces the previous async parameter, because async was promoted to a keyword in Python 3.7. Users of Python versions prior to 3.7 can continue to use the async keyword.
drop (bool) – If True, then if the query is saving to mydb where the same table name already exists, it will overwrite the old mydb table.
profile (str or None) – The Query Manager profile to use for this call. If None then the default profile is used. Available profiles may be listed using the queryClient.list_profiles()
**kw (dict) –
Optional keyword arguments. Supported keywords currently include:

wait = False
Wait for asynchronous queries to complete? If enabled, the query() method will submit the job in async mode and then poll for results internally before returning. the default is to return the job ID immediately and let the client poll for job status and return results.

timeout = 300
Requested timeout (in seconds) for a query. For a Sync query, this value sets a session timeout request in the database that will abort the query at the specified time. A maximum value of 600 seconds is permitted. If the wait option is enabled for an ASync query, this is the maximum time the query will be allowed to run before an abort() is issued on the job. The maximum timeout for an ASync job is 24-hrs (86400 sec).

poll = 1
ASync job polling time in seconds.

verbose = False
Print verbose messages during ASync job.

Returns

result – If async=False, the return value is the result of the query as a formatted string (see fmt). Otherwise the result string is a job token, with which later the Asynchroneaous query’s status can be checked (queryClient.status()), and the result retrieved (see queryClient.result().

Return type

str

Example

Get security token first, see authClient.login(). Then:

query = 'select ra,dec from gaia_dr1.gaia_source limit 3'
response = queryClient.query(token, adql=query, fmt='csv')
print response

This prints

ra,dec
002571989537842,35.2662974820284489
00408275885701,35.2665448169895797
996334457679438,35.2673478725552698

class dl.queryClient.queryClient(profile='default', svc_url='https://datalab.noao.edu/query')[source]¶

Bases: object

QUERYCLIENT – Client-side methods to access the Data Lab: Query Manager Service.

abort(**kw)[source]¶: Call the appropriate instance of the function.

chunked_upload(token, local_file, remote_file)[source]¶: A streaming file uploader.

conequery(token, input=None, out=None, schema=None, table=None, ra=None, dec=None, search=0.5)[source]¶: Send a cone search query to the consearch service

dataType(val, current_type)[source]¶: Lexically scan a value to determine the datatype.

drop(**kw)[source]¶: Call the appropriate instance of the function.

error(**kw)[source]¶: Call the appropriate instance of the function.

getFromURL(svc_url, path, token)[source]¶: Get something from a URL. Return a ‘response’ object

getHeaders(token)[source]¶: Get default tracking headers,

getSchema(data, **kw)[source]¶: Generate a schema for mydb_create() from a CSV file or data object.

getStreamURL(url, headers, fname=None, chunk_size=1048576)[source]¶: Get the specified URL in a streaming fashion. This allows for large downloads without hitting timeout limits.

get_profile()[source]¶

Get the current query profile.

Parameters: None –
Returns: profile – The name of the current profile used with the Query Manager service
Return type: str

Example

print ("Query Service profile = " + queryClient.get_profile())

get_svc_url()[source]¶

Get the currently-used Query Manager serice URL.

Parameters: None –
Returns: service_url – The currently-used Query Service URL.
Return type: str

Example

print (queryClient.get_svc_url())

get_timeout_request()[source]¶

Get the current Sync query timeout value.

Parameters: None –
Returns: result – Current sync query timeout value.
Return type: int

Example

# get the current timeout value
print (queryClient.get_timeout_request())

isAlive(svc_url=None, timeout=5)[source]¶

Check whether the QueryManager service at the given URL is: alive and responding. This is a simple call to the root service URL or ping() method.

Parameters

svc_url (str) – The Query Service URL to ping.
timeout (int) – Call will assume to have failed if ‘timeout’ seconds pass.

Returns

result – True if service responds properly, False otherwise

Return type

bool

Example

if queryClient.isAlive():
    print ("Query Manager is alive")

jobs(**kw)[source]¶: Call the appropriate instance of the function.

list(**kw)[source]¶: Call the appropriate instance of the function.

list_profiles(**kw)[source]¶: Call the appropriate instance of the function.

mydb_copy(**kw)[source]¶: Call the appropriate instance of the function.

mydb_create(**kw)[source]¶: Call the appropriate instance of the function.

mydb_drop(**kw)[source]¶: Call the appropriate instance of the function.

mydb_import(**kw)[source]¶: Call the appropriate instance of the function.

mydb_index(**kw)[source]¶: Call the appropriate instance of the function.

mydb_insert(**kw)[source]¶: Call the appropriate instance of the function.

mydb_list(**kw)[source]¶: Call the appropriate instance of the function.

mydb_rename(**kw)[source]¶: Call the appropriate instance of the function.

mydb_truncate(**kw)[source]¶: Call the appropriate instance of the function.

static pretty_print_POST(req)[source]¶

At this point it is completely built and ready to be fired; it is “prepared”.

However pay attention at the formatting used in this function because it is programmed to be pretty printed and may differ from the actual request.

query(**kw)[source]¶: Call the appropriate instance of the function.

results(**kw)[source]¶: Call the appropriate instance of the function.

schema(**kw)[source]¶: Call the appropriate instance of the function.

services(name=None, svc_type=None, mode='list', profile='default')[source]¶: Usage: queryClient.services ()

set_profile(profile)[source]¶

Set the service profile to be used.

Parameters: profile (str) – The name of the profile to use. The list of available profiles can be retrieved from the service (see function func:queryClient.list_profiles())
Returns
Return type: Nothing

Example

queryClient.set_profile('test')

set_svc_url(svc_url)[source]¶

Set the Query Manager service URL.

Parameters: svc_url (str) – The service URL of the Query Manager to call.
Returns
Return type: Nothing

Example

queryClient.set_svc_url ("http://localhost:7002")

set_timeout_request(nsec)[source]¶

Set the requested Sync query timeout value (in seconds).

Parameters: nsec (int) – The number of seconds requested before a sync query timeout occurs. The service may cap this as a server defined maximum.
Returns
Return type: Nothing

Example

# set the sync query timeout request to 30 seconds
queryClient.set_timeout_request(30)

siaquery(token, input=None, out=None, search=0.5)[source]¶: Send a SIA (Simple Image Access) query to the query manager service

status(**kw)[source]¶: Call the appropriate instance of the function.

wait(**kw)[source]¶: Call the appropriate instance of the function.

exception dl.queryClient.queryClientError(message)[source]¶: Bases: Exception

dl.queryClient.results[source]¶

Retrieve the results of an asynchronous query, once completed.

Usage:: results (token=None, jobId=None, delete=True)

queryClient.results (jobId) queryClient.results (token, jobId) queryClient.results (token, jobId=<id>) queryClient.results (jobId=<str>)

Parameters

token (str) – Authentication token (see function authClient.login())
jobId (str) – The jobId returned when issuing an asynchronous query via queryClient.query() with async=True.

Returns

results

Return type

str

Example

# issue an async query (here a tiny one just for this example)
query = 'select ra,dec from gaia_dr1.gaia_source limit 3'
jobId = queryClient.query(token, adql=query, fmt='csv', async=True)

# ensure job completes...then check status and retrieve results
time.sleep(4)
if queryClient.status(token, jobId) == 'COMPLETED':
    results = queryClient.results(token,jobId)
    print type(results)
    print results

This prints

<type 'str'>
ra,dec
301.37502633933002,44.4946851014515588
301.371102372343785,44.4953207577355698
301.385106974224186,44.4963443903961604

dl.queryClient.schema[source]¶

Return information about a data service schema.

Usage:: schema (value=’’, format=’text’, profile=None)

Parameters

value (str) – Schema object to return: Of the form <schema>[.<table>[.<column]]
profile (str) – The name of the service profile to use. The list of available profiles can be retrieved from the service (see function queryClient.list_profiles())
format (str) – Result format: One of ‘text’ or ‘json’ (NOT CURRENTLY USED)

Example

# List the available schema
queryClient.schema("", "text", "default")

# List the tables in the USNO schema
queryClient.schema("usno", "text", "default")

# List the columns of the USNO-A2 table
queryClient.schema("usno.a2", "text", "default")

# List the attributes of the USNO-A2 'raj2000' column
queryClient.schema("usno.a2.raj2000", "text", "default")

dl.queryClient.services(name=None, svc_type=None, mode='list', profile='default')[source]¶: Usage: queryClient.services ()

dl.queryClient.set_profile(profile)[source]¶

Set the service profile to be used.

Parameters: profile (str) – The name of the profile to use. The list of available profiles can be retrieved from the service (see function func:queryClient.list_profiles())
Returns
Return type: Nothing

Example

queryClient.set_profile('test')

dl.queryClient.set_svc_url(svc_url)[source]¶

Set the Query Manager service URL.

Parameters: svc_url (str) – The service URL of the Query Manager to call.
Returns
Return type: Nothing

Example

queryClient.set_svc_url ("http://localhost:7002")

dl.queryClient.set_timeout_request(nsec)[source]¶

Set the requested Sync query timeout value (in seconds).

Parameters: nsec (int) – The number of seconds requested before a sync query timeout occurs. The service may cap this as a server defined maximum.
Returns
Return type: Nothing

Example

# set the sync query timeout request to 30 seconds
queryClient.set_timeout_request(30)

dl.queryClient.status[source]¶

Get the status of an asynchronous query.

Usage:: status (token=None, jobId=None)

queryClient.status (jobId) queryClient.status (token, jobId) queryClient.status (token, jobId=<id>) queryClient.status (jobId=<str>)

Use the authentication token and the jobId of a previously issued asynchronous query to check the query’s current status.

Parameters

token (str) – Authentication token (see function authClient.login())
jobId (str) – The jobId returned when issuing an asynchronous query via queryClient.query() with async=True.

Returns

status – Either ‘QUEUED’ or ‘EXECUTING’ or ‘COMPLETED’. If the token & jobId combination does not correspond to an actual job, then a HTML-formatted error message is returned. If there is a problem with the backend, the returned value can be ‘ERROR’.

When status is ‘COMPLETED’, you can retrieve the results of the query via queryClient.results()

Return type

str

Example

import time
query = 'select ra,dec from gaia_dr1.gaia_source limit 200000'
jobId = queryClient.query(token, adql=query, fmt='csv', async=True)
while True:
    status = queryClient.status(token, jobId)
    print "time index =", time.localtime()[5], "   status =", status
    if status == 'COMPLETED':
        break
    time.sleep(1)

This prints

time index = 16    status = EXECUTING
time index = 17    status = EXECUTING
time index = 18    status = COMPLETED

dl.queryClient.wait[source]¶

queryClient.wait (jobID=<str>)

Type: Usage

dl.resClient module¶

Client methods for the Data Lab Resource Management Service.

createUser (username, password, email, name, institute)
getUser (token, username, keyword) setUser (token, username, keyword, value)

deleteUser (token, username)

passwordReset (token, user, password)

sendPasswordLink (token, user)

listFields ()

createGroup (token, group)
getGroup (token, group, keyword) setGroup (token, group, keyword, value)

deleteGroup (token, group)

createResource (token, resource)
getResource (token, resource, keyword) setResource (token, resource, keyword, value)

deleteResource (token, resource)

createJob (token, jobid, job_type, query=None, task=None)
getJob (token, jobid, keyword) setJob (token, jobid, keyword, value)

deleteJob (token, jobid)

findJobs (token, jobid, format=’text’, status=’all’,
option=’list’)

set_svc_url (svc_url) get_svc_url () set_profile (profile) get_profile ()

list_profiles (token, profile=None, format=’text’)

Import via

from dl import resClient

dl.resClient.createGroup(token, group, profile='default')[source]¶

Create a new Group in the system.

Parameters

token (str) – User identity token
group (str) – Group name to create

Returns

resp – Service response

Return type

str

dl.resClient.createJob(token, jobid, job_type, query=None, task=None, profile='default')[source]¶

dl.resClient.createResource(token, resource, profile='default')[source]¶

Create a new Resource in the system.

Parameters

token (str) – User identity token
resource (str) – Resource URI to create

Returns

resp – Service response

Return type

str

dl.resClient.createUser(username, password, email, name, institute, profile='default')[source]¶

Create a new user in the system.

Parameters

username (str) – Account username
password (str) – Account password
email (str) – User’s contact email address
name (str) – User’s full name
institute (str) – User’s home institution

Returns

Return type

Service response

dl.resClient.deleteGroup(token, group, profile='default')[source]¶

Delete a Group in the system.

Parameters

token (str) – User identity token.
group (str) – Name of Group to be deleted. The token must identify the owner of the Group to be deleted.

dl.resClient.deleteJob(token, jobid, profile='default')[source]¶

dl.resClient.deleteResource(token, resource, profile='default')[source]¶

Delete a Resource in the system.

Parameters

token (str) – User identity token. The token must identify the owner of the Resource to be deleted.
resource (str) – Name of Resource to be deleted. The token must identify the owner of the Group to be deleted.

dl.resClient.deleteUser(token, username, profile='default')[source]¶

Delete a user in the system.

Parameters

token (str) – User identity token.
username (str) – Name of user to be deleted.

Returns

Return type

Service response

exception dl.resClient.dlResError(message)[source]¶

Bases: Exception

A throwable error class.

dl.resClient.findJobs(token, jobid, format='text', status='all', option='list')[source]¶

dl.resClient.getClient()[source]¶

dl.resClient.getGroup(token, group, keyword, profile='default')[source]¶

Read info about a Group in the system.

Parameters

token (str) – User identity token
group (str) – Group name to create
keyword (str) – Group record field to retrieve

Returns

value – Value of record field

Return type

str

dl.resClient.getJob(token, jobid, keyword, profile='default')[source]¶

dl.resClient.getResource(token, resource, keyword, profile='default')[source]¶

Read info about a Resource in the system.

Parameters

token (str) – User identity token
resource (str) – Resource URI to create
keyword (str) – Resource record to retrieve

Returns

value – Resource record value

Return type

str

dl.resClient.getUser(token, username, keyword, profile='default')[source]¶

Read info about a user in the system.

Parameters

username (str) – User name
keyword (str) – User record field to be set

Returns

Return type

Nothing

Example

from dl import resClient
resClient.client.set_svc_url("http://localhost:7001/")

dl.resClient.get_profile()[source]¶

Get the requested service profile.

Parameters: None –
Returns: profile – The currently requested service profile.
Return type: str

Example

from dl import resClient
profile = resClient.client.get_profile()

dl.resClient.get_svc_url()[source]¶

Return the currently-used Resource Management Service URL.

Parameters: None –
Returns: service_url – The currently-used Resource Management Service URL.
Return type: str

Example

from dl import resClient
service_url = resClient.client.get_svc_url()

dl.resClient.listFields(profile='default')[source]¶

List available user fields.

Parameters: None –
Returns
Return type: Service response

dl.resClient.list_profiles(token, profile=None, format='text')[source]¶

List the service profiles which can be accessed by the user.

Returns: profiles
Return type: JSON string

Example

from dl import resClient
profiles = resClient.client.list_profiles(token, profile, format)

dl.resClient.passwordReset(token, user, password, profile='default')[source]¶

Change a user’s password.

Parameters

token (str) – User identity token
user (str) – User account name. Token must match the user or root token.
password (str) – New password

Returns

Return type

Nothing

Example

from dl import resClient
resClient.client.set_svc_url("http://localhost:7001/")

class dl.resClient.resClient[source]¶

Bases: object

RESCLIENT – Client-side methods to access the Data Lab: Resource Management Service.

approveUser(token, user, profile='default')[source]¶

Approve a pending user request.

Parameters

token (str) – User identity token
user (str) – User account to approve. Token must have authority to manage users.

Returns

Return type

Service response

clientDelete(token, what, query_args)[source]¶: Generic method to call a /delete service.

clientRead(token, what, key, keyword, profile='default')[source]¶: Generic method to call a /get service.

clientUpdate(token, what, key, keyword, value, profile='default')[source]¶: Generic method to call a /set service.

createGroup(token, group, profile='default')[source]¶

Create a new Group in the system.

Parameters

token (str) – User identity token
group (str) – Group name to create

Returns

resp – Service response

Return type

str

createJob(token, jobid, job_type, query=None, task=None, profile='default')[source]¶

Create a new Job record in the system.

Parameters

token (str) – User identity token
jobid (str) – Job ID to create
type (str) – Type of job: currently only ‘query’ or ‘compute’
query (str) – If ‘type’ is ‘query’, the SQL/ADQL query string
task (str) – If ‘type’ is ‘compute’, the name of the task being run

Returns

resp – Service response

Return type

str

createResource(token, resource, profile='default')[source]¶

Create a new Resource in the system.

Parameters

token (str) – User identity token
resource (str) – Resource URI to create

Returns

resp – Service response

Return type

str

createUser(username, password, email, name, institute, profile='default')[source]¶

Create a new user in the system.

Parameters

username (str) – Account username
password (str) – Account password
email (str) – User’s contact email address
name (str) – User’s full name
institute (str) – User’s home institution

Returns

Return type

Service response

debug(debug_val)[source]¶: Set the debug flag.

deleteGroup(token, group, profile='default')[source]¶

Delete a Group in the system.

Parameters

token (str) – User identity token.
group (str) – Name of Group to be deleted. The token must identify the owner of the Group to be deleted.

deleteJob(token, jobid, profile='default')[source]¶

Delete a Job in the system.

Parameters

token (str) – User identity token. The token must identify the owner of the Job to be deleted.
jobid (str) – ID of Job to be deleted. The token must identify the owner of the Job to be deleted.

deleteResource(token, resource, profile='default')[source]¶

Delete a Resource in the system.

Parameters

token (str) – User identity token. The token must identify the owner of the Resource to be deleted.
resource (str) – Name of Resource to be deleted. The token must identify the owner of the Group to be deleted.

deleteUser(token, username, profile='default')[source]¶

Delete a user in the system.

Parameters

token (str) – User identity token.
username (str) – Name of user to be deleted.

Returns

Return type

Service response

disapproveUser(token, user, profile='default')[source]¶

Disapprove a pending user request.

Parameters

token (str) – User identity token
user (str) – User account to decline. Token must have authority to manage users.

Returns

Return type

Service response

findJobs(token, jobid, format='text', status='all', option='list')[source]¶

Find job records. If jobid is None or ‘*’, all records for the user: identified by the token are returned, otherwise the specific job record is returned.

Parameters

token (str) – User identity token. The token must identify the owner of the Job to be deleted.
jobid (str) – Job ID to match.
format (str) – Output format: ‘text’ or ‘json’
status (str) – Job phase status to match. Default to ‘all’ but may be one of EXECUTING, COMPLETED, ERROR, ABORT or WAITING.
option (str) – Processing option: ‘list’ will return a listing of the matching records in the format specified by ‘format’; ‘delete’ will delete all matching records from the server except for EXECUTING jobs.

Returns

Return type

A JSON string of Job records matching the user or jobid.

getGroup(token, group, keyword, profile='default')[source]¶

Read info about a Group in the system.

Parameters

token (str) – User identity token
group (str) – Group name to create
keyword (str) – Group record field to retrieve

Returns

value – Value of record field

Return type

str

getJob(token, jobid, keyword, profile='default')[source]¶

Read info about a Job in the system.

Parameters

token (str) – User identity token
jobid (str) – Job ID to create
keyword (str) – Job record to retrieve

Returns

value – Job record value

Return type

str

getResource(token, resource, keyword, profile='default')[source]¶

Read info about a Resource in the system.

Parameters

token (str) – User identity token
resource (str) – Resource URI to create
keyword (str) – Resource record to retrieve

Returns

value – Resource record value

Return type

str

getUser(username, keyword, profile='default')[source]¶

Read info about a user in the system.

Parameters

username (str) – User name
keyword (str) – User record field to be set

Returns

Return type

Nothing

Example

from dl import resClient
resClient.client.set_svc_url("http://localhost:7001/")

get_profile()[source]¶

Get the requested service profile.

Parameters: None –
Returns: profile – The currently requested service profile.
Return type: str

Example

from dl import resClient
profile = resClient.client.get_profile()

get_svc_url()[source]¶

Return the currently-used Resource Management Service URL.

Parameters: None –
Returns: service_url – The currently-used Resource Management Service URL.
Return type: str

Example

from dl import resClient
service_url = resClient.client.get_svc_url()

isAlive(svc_url)[source]¶

Check whether the ResManager service at the given URL is: alive and responding. This is a simple call to the root service URL or ping() method.

Parameters: svc_url (str) – Resource Management service base URL to call.
Returns
Return type: Nothing

Example

from dl import resClient
resClient.client.set_svc_url("http://localhost:7001/")

listFields(profile='default')[source]¶

List available user fields.

Parameters: None –
Returns
Return type: Service response

listPending(token, verbose=False, profile='default')[source]¶

List all pending user accounts.

Parameters

token (str) – User identity token
verbose (bool) – Return verbose listing?

Returns

Return type

List of use accounts pending approval

list_profiles(token, profile=None, format='text')[source]¶

List the service profiles which can be accessed by the user.

Returns: profiles
Return type: JSON string

Example

from dl import resClient
profiles = resClient.client.list_profiles(token, profile, format)

passwordReset(token, user, password, profile='default')[source]¶

Change a user’s password.

Parameters

token (str) – User identity token
user (str) – User account name. Token must match the user or root token.
password (str) – New password

Returns

Return type

Nothing

Example

from dl import resClient
resClient.client.set_svc_url("http://localhost:7001/")

retBoolValue(url)[source]¶: Utility method to call a boolean service at the given URL.

sendPasswordLink(token, user, profile='default')[source]¶

Send a password-reset link to the user.

Parameters

token (str) – User identity token
user (str) – User account name.

Returns

Return type

Service response

setField(token, user, field, value, profile='default')[source]¶

Set a specific user record field

Parameters

token (str) – User identity token
user (str) – User name to modify. If None then identity is take from token, a root token is required to modify other users.
field (str) – Record field to be set
value (str) – Field value

Returns

Return type

‘OK’ is field was set, else a service error message.

setGroup(token, group, keyword, value, profile='default')[source]¶

Update info about a Group in the system.

Parameters

token (str) – User identity token
group (str) – Group name to create
keyword (str) – Group record field to be set
value (str) – Value of field to set

setJob(token, jobid, keyword, value, profile='default')[source]¶

Update info about a Job in the system.

Parameters

token (str) – User identity token
jobid (str) – Job ID to create
keyword (str) – Job record field to be set
value (str) – Value of field to set

Returns

status – ‘OK’ if record was set

Return type

str

setResource(token, resource, keyword, value, profile='default')[source]¶

Update info about a Resource in the system.

Parameters

token (str) – User identity token
resource (str) – Resource URI to create
keyword (str) – Resource record field to be set
value (str) – Value of field to set

Returns

status – ‘OK’ if record was set

Return type

str

setUser(username, keyword, value, profile='default')[source]¶

Update info about a user in the system.

Parameters

username (str) – User name
keyword (str) – User record field to be set
value (str) – Value of field to set

Returns

Return type

Service response

set_profile(profile)[source]¶

Set the requested service profile.

Parameters: profile (str) – Requested service profile string.
Returns
Return type: Nothing

Example

from dl import resClient
token = resClient.client.set_profile("dev")

set_svc_url(svc_url)[source]¶

Set the URL of the Resource Management Service to be used.

Parameters: svc_url (str) – Resource Management service base URL to call.
Returns
Return type: Nothing

Example

from dl import resClient
resClient.client.set_svc_url("http://localhost:7001/")

svcGet(token, url)[source]¶

Utility method to call a Resource Manager service.

Parameters

token (str) – User identity token
url (str) – URL to call with HTTP/GET

Returns

Return type

Service response

userRecord(token, user, value, format, profile='default')[source]¶

Get a value from the User record.

Parameters

token (str) – User identity token
user (str) – User account name to retrieve. Token must match the use name or be a root token to access other records
value (str) – Value to retrieve. The special ‘all’ value will return all fields accessible to the token.
format (str) – ‘text’ for a single value, or ‘json’ for a complete record

Returns

Return type

User record

dl.resClient.sendPasswordLink(token, user, profile='default')[source]¶

Send a password-reset link to the user.

Parameters

token (str) – User identity token
user (str) – User account name.

Returns

Return type

Service response

dl.resClient.setGroup(token, group, keyword, value, profile='default')[source]¶

Update info about a Group in the system.

Parameters

token (str) – User identity token
group (str) – Group name to create
keyword (str) – Group record field to be set
value (str) – Value of field to set

dl.resClient.setJob(token, jobid, keyword, value, profile='default')[source]¶

dl.resClient.setResource(token, resource, keyword, value, profile='default')[source]¶

Update info about a Resource in the system.

Parameters

token (str) – User identity token
resource (str) – Resource URI to create
keyword (str) – Resource record field to be set
value (str) – Value of field to set

Returns

status – ‘OK’ if record was set

Return type

str

dl.resClient.setUser(token, username, keyword, value, profile='default')[source]¶

Update info about a user in the system.

Parameters

username (str) – User name
keyword (str) – User record field to be set
value (str) – Value of field to set

Returns

Return type

Service response

dl.resClient.set_profile(profile)[source]¶

Set the requested service profile.

Parameters: profile (str) – Requested service profile string.
Returns
Return type: Nothing

Example

from dl import resClient
token = resClient.client.set_profile("dev")

dl.resClient.set_svc_url(svc_url)[source]¶

Set the URL of the Resource Management Service to be used.

Parameters: svc_url (str) – Resource Management service base URL to call.
Returns
Return type: Nothing

Example

from dl import resClient
resClient.client.set_svc_url("http://localhost:7001/")

dl.specClient module¶

dl.specClient.airtovac(l)[source]¶: Convert air wavelengths (greater than 2000A) to vacuum wavelengths.

dl.specClient.catalogs(context='default', profile='default', fmt='text')[source]¶: List available catalogs for a given dataset context

exception dl.specClient.dlSpecError(message)[source]¶

Bases: Exception

A throwable error class.

dl.specClient.getClient(context='default', profile='default')[source]¶

Get a new instance of the specClient client.

Parameters

context (str) – Dataset context
profile (str) – Service profile

Returns

client – An specClient object

Return type

specClient

Example

dl.specClient.getSpec(id_list, fmt='numpy', out=None, align=False, cutout=None, context=None, profile=None, **kw)[source]¶

Get spectra for a list of object IDs.

Usage:

getSpec(id_list, fmt=’numpy’, out=None, align=False, cutout=None,: context=None, profile=None, **kw)

Parameters

id_list (list object) – List of object identifiers.
fmt (str) – Return format of spectra
out – Output file or return to caller if None
align – Align spectra to common wavelength grid with zero-padding
cutout – Wavelength cutout range (as “<start>-<end>”)
context (str) – Dataset context.
profile (str) – Data service profile.
**kw (dict) –
Optional keyword arguments. Supported keywords currently include:

values = None
Spectrum vectors to return.

token = None
Data Lab auth token.

id_col = None
Name of ID column in input table.

verbose = False
Print verbose messages during retrieval

debug = False
Print debug messages during retrieval

Returns

result

Return type

object or array of objects or ‘OK’ string

Example

Retrieve spectra individually:

Retrieve spectra in bulk:

dl.specClient.get_context()[source]¶

Get the requested dataset context.

Parameters: None –
Returns: context – The currently requested dataset context.
Return type: str

Example

from dl import specClient
context = specClient.client.get_context()

dl.specClient.get_profile()[source]¶

Get the requested service profile.

Parameters: None –
Returns: profile – The currently requested service profile.
Return type: str

Example

from dl import specClient
profile = specClient.client.get_profile()

dl.specClient.get_svc_url()[source]¶

Return the currently-used Spectroscopic Data Service URL.

Parameters: None –
Returns: service_url – The currently-used Spectroscopic Data Service URL.
Return type: str

Example

from dl import specClient
service_url = specClient.get_svc_url()

dl.specClient.isAlive(svc_url='https://datalab.noao.edu/spec', timeout=5)[source]¶

dl.specClient.list_contexts[source]¶

Retrieve the contexts supported by the spectro data service.

Usage:: list_contexts ([context], fmt=’text’)

specClient.list_contexts (context) specClient.list_contexts ()

Parameters

contexts (str) – A specific contexts configuration to list. If None, a list of available contexts is returned.
format (str) – Result format: One of ‘text’ or ‘json’

Returns

contexts – A list of the names of the supported contexts or a dictionary of the specific contexts

Return type

list/dict

Example

contexts = specClient.list_contexts(context)
contexts = specClient.list_contexts()

dl.specClient.list_profiles[source]¶

Retrieve the profiles supported by the spectro data service.

Usage:: list_profiles ([profile], fmt=’text’)

specClient.list_profiles (profile) specClient.list_profiles ()

Parameters

profile (str) – A specific profile configuration to list. If None, a list of available profiles is returned.
format (str) – Result format: One of ‘text’ or ‘json’

Returns

profiles – A list of the names of the supported profiles or a dictionary of the specific profile

Return type

list/dict

Example

profiles = specClient.list_profiles(profile)
profiles = specClient.list_profiles()

dl.specClient.plot(spec, context=None, profile=None, out=None, **kw)[source]¶

Utility to batch plot a single spectrum.

Usage:: spec.plot(id, context=None, profile=None, **kw)

Parameters

spec (object ID or data array) – Spectrum to be plotted. If ‘spec’ is a numpy array or a Spectrum1D object the data are plotted directly, otherwise the value is assumed to be an object ID that will be retrieved from the service.
out (str) – Output filename. If specified, plot saved as PNG.
context (str) – Dataset context.
profile (str) – Data service profile.
**kw (dict) –
Optional keyword arguments. Supported keywords currently include:

rest_frame - Whether or not to plot the spectra in the

rest-frame (def: True)

z - Redshift value

xlim - Set the xrange of the plot ylim - Set the yrange of the plot

values - A comma-delimited string of which values to plot,
a combination of ‘flux,model,sky,ivar’

mark_lines - Which lines to mark. No lines marked if None or

an empty string, otherwise one of ‘em|abs|all|both’

grid - Plot grid lines (def: True) dark - Dark-mode plot colors (def: True)

em_lines - List of emission lines to plot. If not given,
all the lines in the default list will be plotted.

abs_lines - Lines of absorption lines to plot. If not given,
all the lines in the default list will be plotted.

spec_args - Plotting kwargs for the spectrum

model_args - Plotting kwargs for the model

ivar_args - Plotting kwargs for the ivar
sky_args - Plotting kwargs for the sky

Returns

Return type

Nothing

Example

Plot a single spectrum, save to a virtual storage file

dl.specClient.plotGrid(id_list, nx, ny, page=0, context=None, profile=None, **kw)[source]¶

Get a grid of preview plots of a spectrum list.

Usage:

image = plotGrid(id_list, nx, ny, page=0,: context=None, profile=None, **kw):

Parameters

id_list (list object) – List of object identifiers.
nx (int) – Number of plots in the X dimension
ny (int) – Number of plots in the Y dimension
page (int) – Dataset context.
context (str) – Dataset context.
profile (str) – Data service profile.
**kw (dict) –
Optional keyword arguments. Supported keywords currently include:

verbose = False
Print verbose messages during retrieval

debug = False
Print debug messages during retrieval

Returns

image

Return type

A PNG image object

Example

Display a 5x5 grid of preview plots for a list:

dl.specClient.preview(spec, context=None, profile=None, **kw)[source]¶

Get a preview plot of a spectrum

Usage:: spec.preview(spec, context=None, profile=None, **kw):

Parameters

id_list (list object) – List of object identifiers.
context (str) – Dataset context.
profile (str) – Data service profile.
**kw (dict) –
Optional keyword arguments. Supported keywords currently include:

N/A

Returns

image

Return type

A PNG image object

Example

Display a preview plot a given spectrum:

dl.specClient.prospect(spec, context=None, profile=None, **kw)[source]¶

Utility wrapper to launch the interactive PROSPECT tool.

Usage:: stat = prospect(spec, context=None, profile=None, **kw)

Parameters

spec (object ID or data array) – Spectrum to be plotted. If ‘spec’ is a numpy array or a Spectrum1D object the data are plotted directly, otherwise the value is assumed to be an object ID that will be retrieved from the service.
context (str) – Dataset context.
profile (str) – Data service profile.
**kw (dict) –
Optional keyword arguments. Supported keywords currently include:

TBD

Returns

result – Status ‘OK’ string or error message.

Return type

str

Example

Plot ….

dl.specClient.query[source]¶

Query for a list of spectrum IDs that can then be retrieved from

the service.

Usage:

id_list = query(ra, dec, size, constraint=None, out=None,: context=None, profile=None, **kw)
id_list = query(pos, size, constraint=None, out=None,: context=None, profile=None, **kw)
id_list = query(region, constraint=None, out=None,: context=None, profile=None, **kw)
id_list = query(constraint=None, out=None,: context=None, profile=None, **kw)

Parameters

ra (float) – RA search center specified in degrees.
dec (float) – Dec of search center specified in degrees.
size (float) – Size of search center specified in degrees.
pos (Astropy Coord object) – Coordinate of search center specified as an Astropy Coord object.
region (float) – Array of polygon vertices (in degrees) defining a search region.
constraint (str) – A valid SQL syntax that can be used as a WHERE constraint in the search query.
out (str) – Output filename to create. If None or an empty string the query results are returned directly to the client. Otherwise, results are writeen to the named file with one identifier per line. A Data Lab ‘vos://’ prefix will save results to the named virtual storage file.
context (str) – Dataset context.
profile (str) – Data service profile.
**kw (dict) –
Optional keyword arguments. Supported keywords currently include:

For context=’sdss_dr16’ | ‘default’:

fields:
specobjid # or ‘bestobjid’, etc tuple # a plate/mjd/fiber tuple

Service will always return array of ‘specobjid’ value, the p/m/f tuple is extracted from the bitmask value by the client.

catalog:

<schema>.<table> # alternative catalog to query e.g. a
# VAC from earlier DR (must support an # ra/dec search and return a specobjid- # like value)

For all contexts:

verbose = False
Print verbose messages during retrieval

debug = False
Print debug messages during retrieval

Returns

result – An array of spectrum IDs appropriate for the dataset context.

Return type

array

Example

Query by position:

dl.specClient.set_context(context)[source]¶

Set the requested dataset context.

Parameters: context (str) – Requested dataset context string.
Returns
Return type: Nothing

Example

from dl import specClient
context = specClient.client.set_context("dev")

dl.specClient.set_profile(profile)[source]¶

Set the requested service profile.

Parameters: profile (str) – Requested service profile string.
Returns
Return type: Nothing

Example

from dl import specClient
profile = specClient.client.set_profile("dev")

dl.specClient.set_svc_url(svc_url)[source]¶

Set the URL of the Spectroscopic Data Service to be used.

Parameters: svc_url (str) – Spectroscopic Data service base URL to call.
Returns
Return type: Nothing

Example

from dl import specClient
specClient.set_svc_url("http://localhost:7001/")

dl.specClient.spcToString(s)[source]¶: spcToString – Force a return value to be type ‘string’ for all Python versions.

class dl.specClient.specClient(context='default', profile='default')[source]¶

Bases: object

SPECCLIENT – Client-side methods to access the Data Lab: Spectroscopic Data Service.

catalogs(context='default', profile='default', fmt='text')[source]¶: Usage: specClient.client.catalogs (…)

curl_get(url)[source]¶: Utility routine to use cURL to return a URL

debug(debug_val)[source]¶: Toggle debug flag.

extractIDList(id_list, id_col=None)[source]¶: Extract a 1-D array or single identifier from an input ID type.

getFromURL(svc_url, path, token)[source]¶: Get something from a URL. Return a ‘response’ object.

getHeaders(token)[source]¶: Get default tracking headers.

getSpec(id_list, fmt='numpy', out=None, align=False, cutout=None, context=None, profile=None, **kw)[source]¶

Get spectra for a list of object IDs.

Usage:

getSpec(id_list, fmt=’numpy’, out=None, align=False, cutout=None,: context=None, profile=None, **kw)

Parameters

id_list (list object) – List of object identifiers.
format – Return format of spectra
out – Output filename or return to caller if None
align – Align spectra to common wavelength grid with zero-padding
cutout – Wavelength cutout range (as “<start>-<end>”)
context (str) – Dataset context.
profile (str) – Data service profile.
**kw (dict) –
Optional keyword arguments. Supported keywords currently include:

values = None
Spectrum vectors to return.

token = None
Data Lab auth token.

id_col = None
Name of ID column in input table.

verbose = False
Print verbose messages during retrieval

debug = False
Print debug messages during retrieval

Returns

result

Return type

object or array of objects or ‘OK’ string

Example

Retrieve spectra individually:

Retrieve spectra in bulk:

get_context()[source]¶

Get the requested dataset context.

Parameters: None –
Returns: context – The currently requested dataset context.
Return type: str

Example

from dl import specClient
context = specClient.client.get_context()

get_profile()[source]¶

Get the requested service profile.

Parameters: None –
Returns: profile – The currently requested service profile.
Return type: str

Example

from dl import specClient
profile = specClient.client.get_profile()

get_svc_url()[source]¶

Return the currently-used Spectroscopic Data Service URL.

Parameters: None –
Returns: service_url – The currently-used Spectroscopic Data Service URL.
Return type: str

Example

from dl import specClient
service_url = specClient.get_svc_url()

isAlive(svc_url='https://datalab.noao.edu/spec')[source]¶

Check whether the service at the given URL is alive and responding.: This is a simple call to the root service URL or ping() method.

Parameters: service_url (str) – The Query Service URL to ping.
Returns: result – True if service responds properly, False otherwise
Return type: bool

Example

from dl import specClient
if specClient.isAlive():
    print("Spec Server is alive")

list_contexts(**kw)[source]¶: Call the appropriate instance of the function.

list_profiles(**kw)[source]¶: Call the appropriate instance of the function.

plot(spec, context=None, profile=None, out=None, **kw)[source]¶

Utility to batch plot a single spectrum.

Usage:: spec.plot(id, context=None, profile=None, **kw)

Parameters

spec (object ID or data array) – Spectrum to be plotted. If ‘spec’ is a numpy array or a Spectrum1D object the data are plotted directly, otherwise the value is assumed to be an object ID that will be retrieved from the service.
out (str) – Output filename. If specified, plot saved as PNG.
context (str) – Dataset context.
profile (str) – Data service profile.
**kw (dict) –
Optional keyword arguments. Supported keywords currently include:

rest_frame - Whether or not to plot the spectra in the

rest-frame. If True, the wavelengths are assumed to have been already shifted and no ‘z’ value is required. If False, wavelengths are assumed to be in observed frame and a ‘z’ value is required to overplot absorption/emission lines. (def: True)

z - Redshift value (def: None)

xlim - Set the xrange of the plot ylim - Set the yrange of the plot

title - Plot title (def: object ID)

xlabel - Plot x-axis label (def: wavelength) ylabel - Plot y-axis label (def: flux units)

out - Saved output filename.

values - A comma-delimited string of which values to plot,
a combination of ‘flux,model,sky,ivar’

mark_lines - Which lines to mark. No lines marked if None or

an empty string, otherwise one of ‘em|abs|all|both’

grid - Plot grid lines (def: True) dark - Dark-mode plot colors (def: True)

em_lines - List of emission lines to plot. If not given,
all the lines in the default list will be plotted.

abs_lines - Lines of absorption lines to plot. If not given,
all the lines in the default list will be plotted.

spec_args - Plotting kwargs for the spectrum

model_args - Plotting kwargs for the model

ivar_args - Plotting kwargs for the ivar
sky_args - Plotting kwargs for the sky

Returns

Return type

Nothing

Example

Plot a single spectrum, save to a virtual storage file

plotGrid(id_list, nx, ny, page=0, context=None, profile=None, **kw)[source]¶

Get a grid of preview plots of a spectrum list.

Usage:

image = plotGrid(id_list, nx, ny, page=0,: context=None, profile=None, **kw):

Parameters

id_list (list object) – List of object identifiers.
nx (int) – Number of plots in the X dimension
ny (int) – Number of plots in the Y dimension
page (int) – Dataset context.
context (str) – Dataset context.
profile (str) – Data service profile.
**kw (dict) –
Optional keyword arguments. Supported keywords currently include:

verbose = False
Print verbose messages during retrieval

debug = False
Print debug messages during retrieval

Returns

image

Return type

A PNG image object

Example

Display a 5x5 grid of preview plots for a list:

preview(spec, context=None, profile=None, **kw)[source]¶

Get a preview plot of a spectrum

Usage:: spec.preview(spec, context=None, profile=None, **kw):

Parameters

spec (objectID) – Object identifiers.
context (str) – Dataset context.
profile (str) – Data service profile.
**kw (dict) –
Optional keyword arguments. Supported keywords currently include:

N/A

Returns

image

Return type

A PNG image object

Example

Display a preview plot a given spectrum:

prospect(spec, context=None, profile=None, **kw)[source]¶

Utility wrapper to launch the interactive PROSPECT tool.

Usage:: stat = prospect(spec, context=None, profile=None, **kw)

Parameters

spec (object ID or data array) – Spectrum to be plotted. If ‘spec’ is a numpy array or a Spectrum1D object the data are plotted directly, otherwise the value is assumed to be an object ID that will be retrieved from the service.
context (str) – Dataset context.
profile (str) – Data service profile.
**kw (dict) –
Optional keyword arguments. Supported keywords currently include:

TBD

Returns

result – Status ‘OK’ string or error message.

Return type

str

Example

Plot ….

query(**kw)[source]¶: Call the appropriate instance of the function.

retBoolValue(url)[source]¶: Utility method to call a boolean service at the given URL.

set_context(context)[source]¶

Set the requested dataset context.

Parameters: context (str) – Requested dataset context string.
Returns
Return type: Nothing

Example

from dl import specClient
context = specClient.client.set_context("dev")

set_profile(profile)[source]¶

Set the requested service profile.

Parameters: profile (str) – Requested service profile string.
Returns
Return type: Nothing

Example

from dl import specClient
profile = specClient.client.set_profile("dev")

set_svc_url(svc_url)[source]¶

Set the URL of the Spectroscopic Data Service to be used.

Parameters: svc_url (str) – Spectroscopic Data service base URL to call.
Returns
Return type: Nothing

Example

from dl import specClient
specClient.set_svc_url("http://localhost:7001/")

stackedImage(id_list, align=False, yflip=False, context=None, profile=None, **kw)[source]¶

Get …

Usage:

Parameters

id_list (list object) – List of spectrum identifiers.
context (str) – Dataset context.
profile (str) – Data service profile.
**kw (dict) –
Optional keyword arguments. Supported keywords currently include:

verbose = False
Print verbose messages during retrieval

debug = False
Print debug messages during retrieval

Returns

result

Return type

Example

Query ….

to_Spectrum1D(npy_data)[source]¶: Convert a Numpy spectrum array to a Spectrum1D object.

to_Table(npy_data)[source]¶: Utility method to convert a Numpy array to an Astropy Table object.

to_pandas(npy_data)[source]¶: Utility method to convert a Numpy array to a Pandas DataFrame

dl.specClient.stackedImage(id_list, align=False, yflip=False, context=None, profile=None, **kw)[source]¶

Get …

Usage:

Parameters

id_list (list object) – List of spectrum identifiers.
context (str) – Dataset context.
profile (str) – Data service profile.
**kw (dict) –
Optional keyword arguments. Supported keywords currently include:

verbose = False
Print verbose messages during retrieval

debug = False
Print debug messages during retrieval

Returns

result

Return type

Example

Query ….

dl.specClient.to_Spectrum1D(npy_data)[source]¶: Utility method to convert a Numpy array to Spectrum1D

dl.specClient.to_Table(npy_data)[source]¶: Utility method to convert a Numpy array to an Astropy Table object.

dl.specClient.to_pandas(npy_data)[source]¶: Utility method to convert a Numpy array to a Pandas DataFrame

dl.storeClient module¶

dl.storeClient.access[source]¶

Determine whether the file can be accessed with the given node.

Usage:: access(path, mode=None, token=None, verbose=True)

storeClient.access(token, path, mode) storeClient.access(path, mode) storeClient.access(path)

Parameters

path (str) – A name or file template of the file status to retrieve.
mode (str) – Requested access mode. Modes are ‘r’ (read access), ‘w’ (write access), or ‘rw’ to test for both read/write access. If mode is None a simple existence check is made.
token (str) – Authentication token (see function authClient.login())
verbose (bool) – Verbose output flag.

Returns

result – True if the node can be access with the requested mode.

Return type

bool

Example

if storeClient.access('/mydata.csv')
    print('File exists')
elif storeClient.access('/mydata.csv','rw')
    print('File is both readable and writable')

dl.storeClient.chunked_upload(token, local_file, remote_file)[source]¶: A streaming file uploader.

dl.storeClient.cp[source]¶

Copy a file/directory within the store manager service

Usage:: cp(token=None, fr=’’, to=’’, verbose=False)

storeClient.cp(token, fr, to) storeClient.cp(fr, to) storeClient.cp(fr) storeClient.cp(fr=’’,to=’’)

Parameters

token (str) – Authentication token (see function authClient.login())
fr (str) – Name of the file to be copied (may not be a directory).
to (str) – Name of the file to be created

Returns

result – An ‘OK’ message or error string for each uploaded file.

Return type

str

Example

# Copy a file in vospace
storeClient.cp('foo', 'bar')
storeClient.cp('vos://foo', 'vos:///new/bar')

dl.storeClient.expandFileList(svc_url, token, pattern, format, full=False)[source]¶: Expand a filename pattern in a VOSpace URI to a list of files. We do this by getting a listing of the parent container contents from the service and then match the pattern on the client side.

dl.storeClient.get[source]¶

Retrieve a file from the store manager service

Usage:

get(token=None, fr=’’, to=’’, mode=’text’, verbose=True, debug=False,: timeout=30)

storeClient.get(token, fr, to) storeClient.get(fr, to) storeClient.get(fr) storeClient.get(token, fr, to)

Parameters

token (str) – Authentication token (see function authClient.login())
fr (str) – A name or file template of the file(s) to retrieve.
to (str) – Name of the file(s) to locally. If not specified, the contents of the file are returned to the caller.
mode ([binary | text | fileobj]) – Return data type if note saving to file. If set to ‘text’ the file contents are converted to string – this is appropriate when dealing with unicode but may fail with general binary data. If set to ‘binary’ the raw content of the HTTP response is returned – for Python 2 this will be a ‘string’, for Python 3 it will be a ‘bytes’ data type (the caller is responsible for conversion).
verbose (bool) – Print verbose output, e.g. progress indicators.
debug (bool) – Print debug output.
timeout (integer) – Retry timeout value. When processing long lists, download will pause every timeout files to lessen server load. For individual files, transfer will retry for timeout seconds before aborting. Failed transfers are automatically appended to the file list so they may be transferred again later.

Returns

result – A list of the names of the files retrieved, or the contents of a single file.

Return type

str

Example

# get a single file to a local file of a different name
data = storeClient.get('vos://mydata.csv', 'data.csv')

# get the contents of a single file to a local variable
data = storeClient.get('vos://mydata.csv')

# get a list of remote files to a local directory
flist = storeClient.get('vos://*.fits', './data/')
flist = storeClient.get('*.fits', './data/')

dl.storeClient.getClient(profile='default', svc_url='https://datalab.noao.edu/storage')[source]¶: Create a new storeClient object and set a default profile.

dl.storeClient.get_profile()[source]¶

Get the profile

Parameters: None –
Returns: profile – The name of the current profile used with the Storage Manager
Return type: str

Example

print('Store Service profile = ' + storeClient.get_profile())

dl.storeClient.get_svc_url()[source]¶

Return the currently-used Storage Manager service URL.

Parameters: None –
Returns
Return type: Current Storage Manager service URL

Example

print(storeClient.get_scv_url())

dl.storeClient.hasmeta(s)[source]¶: Determine whether a string contains filename meta-characters.

dl.storeClient.isAlive(svc_url='https://datalab.noao.edu/storage')[source]¶

Check whether the StorageManager service at the given URL is: alive and responding. This is a simple call to the root service URL or ping() method.

Parameters: svc_url (str) – The service URL of the storage manager to use

Example

# Check if service is responding
if storeClient.isAlive():
   .....stuff

dl.storeClient.is_vosDir(svc_url, token, path)[source]¶: Determine whether ‘path’ is a ContainerNode in the VOSpace.

dl.storeClient.list_profiles[source]¶

Retrieve the profiles supported by the storage manager service

Usage:: list_profiles(token=None, profile=None, format=’text’)

storeClient.list_profiles(token) # list default profile storeClient.list_profiles(profile) # list named profile storeClient.list_profiles() # list default profile

Parameters

token (str) – Authentication token (see function authClient.login())
profile (str) – A specific profile to list

Returns

profiles – A list of the names of the supported profiles or a dictionary of the specific profile

Return type

list/dict

Example

# get the list of profiles
profiles = storeClient.list_profiles(token)

dl.storeClient.ln[source]¶

Create a link to a file/directory in the store manager service

Usage:: ln(token, fr=’’, target=’’, verbose=False)

storeClient.ln(token, fr, target) storeClient.ln(fr, target) storeClient.ln(fr, target)

Parameters

token (str) – Authentication token (see function authClient.login())
fr (str) – Name of the file to be linked (may not be a directory).
to (str) – Name of the link to be created

Returns

result – An ‘OK’ message or error string for each uploaded file.

Return type

str

Example

# Link a file in vospace
storeClient.ln('foo', 'bar')
storeClient.ln('vos://foo', 'vos:///new/bar')

dl.storeClient.load[source]¶

Load a file from a remote endpoint to the Store Manager service

Usage:: load(name, endpoint, token=None, is_vospace=False)

storeClient.load(token, name, endpoint) storeClient.load(name, endpoint)

Parameters

token (str) – Authentication token (see function authClient.login())
name (str) – Name or template of local files to upload.
endpoint (str) – Name of the file for destination directory on remote VOSpace.

Returns

result – An ‘OK’ message or error string

Return type

str

Example

# Load a file from a remote URL
storeClient.load('mydata.vot', 'http://example.com/data.vot')

dl.storeClient.ls[source]¶

Get a file/directory listing from the store manager service

Usage:: ls(name=’vos://’, token=None, format=’csv’, verbose=False)

storeClient.ls(token, name) storeClient.ls(name) storeClient.ls()

Parameters

token (str) – Secure token obtained via authClient.login()
name (str) – Valid name of file or directory, e.g. vos://somedir .. todo:: [20161110] currently doesn’t seem to work.
format (str) – Default csv. The long option produces an output similar to ‘ls -l’.

Example

listing = storeClient.ls(token, name='vos://somedir')
listing = storeClient.ls(token, 'vos://somedir')
listing = storeClient.ls('vos://somedir')
print(listing)

This prints for instance:

bar2.fits,foo1.csv,fancyfile.dat

dl.storeClient.mkdir[source]¶

Make a directory in the storage manager service

Usage:: mkdir(optval, name=’’, token=None)

storeClient.mkdir(token, name) storeClient.mkdir(name)

Parameters

token (str) – Authentication token (see function authClient.login())
name (str) – Name of the container (directory) to create.

Returns

result – An ‘OK’ message or error string for each uploaded file.

Return type

str

Example

# Create a directory in vospace
storeClient.mkdir('foo')

dl.storeClient.mv[source]¶

Move/rename a file/directory within the store manager service

Usage:: mv(token=None, fr=’’, to=’’, verbose=False)

storeClient.mv(token, fr, to) storeClient.mv(fr, to) storeClient.mv(token) storeClient.mv(fr=’’,to=’’)

Parameters

token (str) – Authentication token (see function authClient.login())
fr (str) – Name of the file to be moved
to (str) – Name of the file or directory to move the ‘fr’ file. If given as a directory the original filename is preserved.

Returns

result – An ‘OK’ message or error string for each uploaded file.

Return type

str

Example

# Move a file in vospace
storeClient.mv('foo', 'bar')             # rename file
storeClient.mv('foo', 'vos://newdir/')   # move to new directory
storeClient.mv('foo', 'newdir')

dl.storeClient.pull[source]¶

Load a file from a remote endpoint to the Store Manager service

Usage:: pull(name, endpoint, token=None, is_vospace=False)

storeClient.pull(token, name, endpoint) storeClient.pull(name, endpoint)

Parameters

token (str) – Authentication token (see function authClient.login())
name (str) – Name or template of local files to upload.
endpoint (str) – Name of the file for destination directory on remote VOSpace.

Returns

result – An ‘OK’ message or error string

Return type

str

Example

# Load a file from a remote URL
storeClient.load('mydata.vot', 'http://example.com/data.vot')

dl.storeClient.put[source]¶

Upload a file to the store manager service

Usage:: put(fr=’’, to=’vos://’, token=None, verbose=True, debug=False)

storeClient.put(token, fr, to) storeClient.put(fr, to) storeClient.put(fr) storeClient.put(fr=’’,to=’’)

Parameters

token (str) – Authentication token (see function authClient.login())
fr (str) – Name or template of local files to upload.
to (str) – Name of the file for destination directory on remote VOSpace.

Returns

result – An ‘OK’ message or error string for each uploaded file.

Return type

str

Example

# get the contents of a single file

dl.storeClient.rm[source]¶

Delete a file from the store manager service

Usage:: rm(name=’’, token=None, verbose=False)

storeClient.rm(token, name) storeClient.rm(name)

Parameters

token (str) – Authentication token (see function authClient.login())
name (str) – Name of the file to delete

Returns

result – An ‘OK’ message or error string for each uploaded file.

Return type

str

Example

# Remove a file from vospace
storeClient.rm('foo.csv')
storeClient.rm('vos://foo.csv')

dl.storeClient.rmdir[source]¶

Delete a directory from the store manager service

Usage:: rmdir(name=’’, token=None, verbose=False)

storeClient.rmdir(token, name) storeClient.rmdir(name)

Parameters

token (str) – Authentication token (see function authClient.login())
name (str) – Name of the container (directory) to delete.

Returns

result – An ‘OK’ message or error string for each uploaded file.

Return type

str

Example

# Remove an empty directory from VOSpace.
storeClient.rmdir('datadir')
storeClient.rmdir('vos://datadir')

dl.storeClient.saveAs[source]¶

Save the string representation of a data object as a file.

Usage:: saveAs(data, name, token=None)

storeClient.saveAs(token, data, name) storeClient.saveAs(data, name)

Parameters

token (str) – Authentication token (see function authClient.login())
data (str) – Data object to be saved.
name (str) – Name of the file to create containing the string representation of the data.

Returns

result – An ‘OK’ message or error string for each uploaded file.

Return type

str

Example

# Save a data object to VOSpace
storeClient.saveAs(pandas_data, 'pandas.example')
storeClient.saveAs(json_data, 'json.example')
storeClient.saveAs(table_data, 'table.example')

dl.storeClient.scToString(s)[source]¶: scToString – Force a return value to be type ‘string’ for all Python versions. If there is an error, return the original.

dl.storeClient.services(name=None, svc_type='vos', format=None, profile='default')[source]¶

dl.storeClient.set_profile(profile='default')[source]¶

Set the service profile to be used.

Parameters: profile (str) – The name of the profile to use. The list of available ones can be retrieved from the service (see function storeClient.list_profiles())
Returns
Return type: Nothing

Example

storeClient.set_profile('test')

dl.storeClient.set_svc_url(svc_url='https://datalab.noao.edu/storage')[source]¶

Set the Storage Manager service URL.

Parameters: svc_url (str) – The service URL of the Storage Manager to use
Returns
Return type: Nothing

Example

storeClient.set_scv_url("http://demo.datalab.noao.edu:7003")

dl.storeClient.stat[source]¶

Get file status information, similar to stat().

Usage:: stat(path, token=None, verbose=True)

storeClient.stat(token, path) storeClient.stat(path)

Parameters

path (str) – A name or file template of the file status to retrieve.
token (str) – Authentication token (see function authClient.login())
verbose (bool) – Verbose output flag.

Returns

stat –

A dictionary of node status values. Returned fields include:

name Name of node groupread List of group/owner names w/ read access groupwrite List of group/owner names w/ write access publicread Publicly readable (0=False, 1=True) owner Owner name perms Formatted unix-like permission string target Node target if LinkNode size Size of file node (bytes) type Node type (container|data|link)

Return type

dictionary

Example

# get status information for a specific node
stat = storeClient.stat('vos://mydata.csv')

if stat['type'] == 'container':
    print('This is a directory')
else:
    print('File size is: ' + stat['size'])

class dl.storeClient.storeClient(profile='default', svc_url='https://datalab.noao.edu/storage')[source]¶

Bases: object

STORECLIENT – Client-side methods to access the Data Lab: Storage Manager Service.

access(**kw)[source]¶: Call the appropriate instance of the function.

cp(**kw)[source]¶: Call the appropriate instance of the function.

get(**kw)[source]¶: Call the appropriate instance of the function.

getFromURL(svc_url, path, token)[source]¶: Get something from a URL. Return a ‘response’ object

get_profile()[source]¶

Get the profile

Parameters: None –
Returns: profile – The name of the current profile used with the Storage Manager
Return type: str

Example

print('Store Service profile = ' + storeClient.get_profile())

get_svc_url()[source]¶

Return the currently-used Storage Manager service URL.

Parameters: None –
Returns
Return type: Current Storage Manager service URL

Example

print(storeClient.get_scv_url())

isAlive(svc_url=None, timeout=2)[source]¶

Check whether the StorageManager service at the given URL is: alive and responding. This is a simple call to the root service URL or ping() method.

Parameters: svc_url (str) – The service URL of the storage manager to use

Example

# Check if service is responding
if storeClient.isAlive():
   .....stuff

list_profiles(**kw)[source]¶: Call the appropriate instance of the function.

ln(**kw)[source]¶: Call the appropriate instance of the function.

load(**kw)[source]¶: Call the appropriate instance of the function.

ls(**kw)[source]¶: Call the appropriate instance of the function.

mkdir(**kw)[source]¶: Call the appropriate instance of the function.

mv(**kw)[source]¶: Call the appropriate instance of the function.

pull(**kw)[source]¶: Call the appropriate instance of the function.

put(**kw)[source]¶: Call the appropriate instance of the function.

rm(**kw)[source]¶: Call the appropriate instance of the function.

rmdir(**kw)[source]¶: Call the appropriate instance of the function.

saveAs(**kw)[source]¶: Call the appropriate instance of the function.

services(name=None, svc_type='vos', format=None, profile='default')[source]¶

set_profile(profile)[source]¶

Set the service profile to be used.

Parameters: profile (str) – The name of the profile to use. The list of available ones can be retrieved from the service (see function storeClient.list_profiles())
Returns
Return type: Nothing

Example

storeClient.set_profile('test')

set_svc_url(svc_url)[source]¶

Set the Storage Manager service URL.

Parameters: svc_url (str) – The service URL of the Storage Manager to use
Returns
Return type: Nothing

Example

storeClient.set_scv_url("http://demo.datalab.noao.edu:7003")

stat(**kw)[source]¶: Call the appropriate instance of the function.

tag(**kw)[source]¶: Call the appropriate instance of the function.

exception dl.storeClient.storeClientError(message)[source]¶: Bases: Exception

dl.storeClient.tag[source]¶

Annotate a file/directory in the store manager service

Usage:: tag(token, name=’’, tag=’’)

storeClient.tag(token, name, tag) storeClient.tag(name, tag) storeClient.tag(token, name=’foo’, tag=’bar’)

Parameters

token (str) – Authentication token (see function authClient.login())
name (str) – Name of the file to tag
tag (str) – Annotation string for file

Returns

result – An ‘OK’ message or error string for each uploaded file.

Return type

str

Example

# Annotate a file in vospace
storeClient.tag('foo.csv', 'This is a test')

dl package¶

Subpackages¶

Submodules¶

dl.Util module¶

dl.authClient module¶

dl.dlinterface module¶

dl.dltasks module¶

dl.queryClient module¶

dl.resClient module¶

dl.specClient module¶

dl.storeClient module¶

Table of Contents

Previous topic

Next topic

This Page