api package

api.client module

Client module for SPARCL.

This module interfaces to the SPARC-Server to get spectra data.

Todo

  • Data Discovery

    Allow retrieval of spectra based upon matches to metadata values.

class api.client.SparclApi(url='http://sparc1.datalab.noirlab.edu:8000', verbose=False, internal_names=False, connect_timeout=1.1, read_timeout=5400)[source]

Bases: object

Provides interface to SPARCL Server.

When using this to report a bug, set verbose to True. Also print your instance of this. The results will include important info about the Client and Server that is usefule to Developers.

Parameters
  • url – Base URL of SPARC Server

  • verbose – (True,[False]) Default verbosity for all client methods.

  • verbose – (True,[False]) True:: override field renaming

  • [1.1] (connect_timeout) – Number of seconds to wait to establish connection with server.

  • [5400] (read_timeout) – Number of seconds to wait for server to send a response. (generally time to wait for first byte)

Example

>>> client = SparclApi(verbose=True)
>>> print(client)

Object creation compares the version from the Server against the one expected by the Client. Throws error if the Client is a major version or more behind.

KNOWN_GOOD_API_VERSION = 4.0
client_field(structure, orig_name)[source]

Get field name used in Client methods

Parameters
  • structure – Name of Data Set

  • orig_name – Original field name as provided in Data Set.

Returns

Client field name

Return type

string

Example

>>> client.client_field('BOSS-DR16', 'spectra.coadd.FLUX')
'flux'
get_field_names(structure)[source]

List field names available for retreive.

Parameters

structure – List field names of this Data Set.

Returns

list of field names

Return type

list

Example

>>> client.get_field_names('DESI-everest')
missing_specids(specid_list, countOnly=False, verbose=False)[source]

Return the subset of the given specid list that is NOT stored in the database.

Parameters
  • specid_list (list) – List of specids.

  • verbose (boolean, optional) – (default: False)

Returns

The subset of the given specid list that is NOT stored in the database.

Example

>>> si = [1858907533188556800, 6171312851359387648, 1647268306035435520]
>>> client.missing_specids(si)
[1858907533188556800, 6171312851359387648]
normalize_field_names(recs)[source]

Return copy of records with all field names converted to the names used by the Data Set provider.

Parameters

recs – List of dictionaries representing spectra records

Returns

new list of dicts of spectra records (with diff field names)

Return type

list

Example

>>> recs = client.sample_records(1)
>>> client.normalize_field_names(recs)
orig_field(structure, client_name)[source]

Get original field name as provided in Data Set.

Parameters
  • structure – Name of Data Set

  • client_name – Field name used in Client methods.

Returns

Original field name

Return type

string

Example

>>> client.orig_field('BOSS-DR16', 'flux')
'spectra.coadd.FLUX'
retrieve(specid_list, include='DEFAULT', rtype=None, structure=None, verbose=False)[source]

Get spectrum from specid list.

Parameters
  • specid_list (list) – List of specids.

  • include (list, 'DEFAULT', 'ALL') – List of paths to include in each record. (default: ‘DEFAULT’)

  • rtype (str, optional) – Data-type to use for spectra data. One of: json, numpy, pandas, spectrum1d. (default: None)

  • structure (str) – The Data Set (DS) name associated with the specids. Or None to retrieve from any DS that contains the specid. (default: None)

  • verbose (boolean, optional) – (default: False)

Returns

List of records. Each record is a dictionary of named fields.

Example

>>> ink = ['flux']
>>> sdss_ids = [849044290804934656, 309718438815754240]
>>> res_sdss = client.retrieve(sdss_ids, structure='SDSS-DR16', include=ink)
sample_records(count, structure=None, include='DEFAULT', **kwargs)[source]

Return list of random records from given STRUCTURE (Data Set).

Parameters
  • count (int) – Number of sample records to get from database.

  • structure (str, optional) – (default: None means ANY) The Data Set from which to get sample records.

  • include (list, 'DEFAULT', 'ALL') – List of paths to include in each record. (default: ‘DEFAULT’)

Returns

List of random records from given STRUCTURE (Data Set).

Example

>>> samrec = client.sample_records(1, structure='BOSS-DR16')
>>> pprint.pprint(samrec,depth=2)
[{'data_release_id': 'BOSS-DR16',
  'dec_center': 47.193549,
 'dirpath': '/net/mss1/archive/hlsp/sdss/dr16/eboss/spectro/redux/v5_13_0/spectra/lite/7399',
  'filename': 'spec-7399-57162-0376.fits',
  'specid': 1429845755960551,
  'spectra': {...},
  'updated': '2021-04-28T20:16:20.399464Z'}]
sample_specids(samples=5, structure=None, random=True, **kwargs)[source]

Return a small list of specids.

This is intended to make it easy to get just a few specids to use for experimenting with the rest of the API.

Parameters
  • samples (int, optional) – (default: 5) The number of sample specids to get.

  • structure (str, optional) – (default: None means ANY) The Data Set from which to get specids.

  • random (True,False,None) – Randomize sample

Returns

List of specids

Rtype: list

Example

>>> client.sample_specids(samples=3, structure='DESI-everest')
property version

Return version of Server Rest API used by this client.

If the Rest API changes such that the Major version increases, a new version of this module will likely need to be used.

Returns

API version.

Return type

float

Example

>>> SparclApi('http://localhost:8030').version
1.0
api.client.fields_available(records)[source]

Get list of fields used in records. One list per Data Set.

Parameters

records – List of records (each is a dictionary)

Returns

dict[Structure] = [field_name1, …]

Return type

dict

Example

>>> from api.client import fields_available
>>> recs = client.sample_records(3)
>>> flds = fields_available(recs)
api.client.get_metadata(records)[source]

Get records of just metadata used in records.

Metadata is considered to be any field whose type is a Number or String. Therefore, this will not include vectors, lists, tuples, etc.

Parameters

records – List of records (dictionaries)

Returns

new list of dictionaries. Each dict contains only metadata fields.

Return type

list(dict)

Example

>>> from api.client import get_metadata
>>> recs = client.sample_records(3)
>>> metadata = get_metadata(recs)
api.client.record_examples(records)[source]

Copy one record for each Data Set type.

Parameters

records – List of records (each is a dictionary)

Returns

dict[Structure] = rec

Return type

dict

Example

>>> from api.client import record_examples
>>> recs = client.sample_records(3)
>>> rex = record_examples(recs)
api.client.rename_fields(rename_dict, records)[source]

Rename some field names in all given records.

Parameters
  • rename_dict – The key is current field name, value is new.

  • records – List of records (dictionaries) to transform

Returns

new_records

Return type

list

Example

>>> from api.client import rename_fields
>>> recs = client.sample_records(1)
>>> renamed = rename_fields({'ra':'Right_Ascension'},recs)

api.exceptions module

exception api.exceptions.BadInclude(error_message, error_code=None)[source]

Bases: api.exceptions.BaseSparclException

Include list contains invalid data field(s).

error_code = 'BADINCL'
exception api.exceptions.BadPath(error_message, error_code=None)[source]

Bases: api.exceptions.BaseSparclException

A field path starts with a non-core field.

error_code = 'BADPATH'
exception api.exceptions.BaseSparclException(error_message, error_code=None)[source]

Bases: Exception

Base Class for all SPARCL exceptions.

error_code = 'UNKNOWN'
error_message = '<NA>'
get_subclass_name()[source]
to_dict()[source]

Convert a SPARCL exception to a python dictionary

traceback = None
exception api.exceptions.ReadTimeout(error_message, error_code=None)[source]

Bases: api.exceptions.BaseSparclException

The server did not send any data in the allotted amount of time.

error_code = 'RTIMEOUT'
exception api.exceptions.UnkDr(error_message, error_code=None)[source]

Bases: api.exceptions.BaseSparclException

The Data Release is not known or not supported.

error_code = 'UNKDR'
exception api.exceptions.UnknownServerError(error_message, error_code=None)[source]

Bases: api.exceptions.BaseSparclException

Client got a status response from the SPARC Server that we do not know how to decode.

error_code = 'UNKNOWN'
exception api.exceptions.UnknownSparcl(error_message, error_code=None)[source]

Bases: api.exceptions.BaseSparclException

Unknown SPARCL error. If this is ever raised (seen in a log) create and use a new BaseSparcException exception that is more specific.

error_code = 'UNKSPARC'
api.exceptions.genSparclException(response, verbose=False)[source]

Given status from Server response.json(), which is a dict, generate a native SPARCL exception suitable for Science programs.

api.type_conversion module

class api.type_conversion.BossDr16[source]

Bases: api.type_conversion.Convert

to_numpy(record, o2nLUT)[source]
to_pandas(record, o2nLUT)[source]
to_spectrum1d(record, o2nLUT)[source]
class api.type_conversion.Convert[source]

Bases: abc.ABC

Convert JSON record to mix of plain python and selected data record type.

abstract to_numpy(record, o2nLUT)[source]
abstract to_pandas(record, o2nLUT)[source]
abstract to_spectrum1d(record, o2nLUT)[source]
class api.type_conversion.Desi[source]

Bases: api.type_conversion.Convert

to_numpy(record, o2nLUT)[source]
to_pandas(record, o2nLUT)[source]
to_spectrum1d(record, o2nLUT)[source]
class api.type_conversion.DesiDenali[source]

Bases: api.type_conversion.Desi

class api.type_conversion.DesiEverest[source]

Bases: api.type_conversion.Desi

class api.type_conversion.NoopConvert[source]

Bases: api.type_conversion.Convert

to_numpy(record, o2nLUT)[source]
to_pandas(record, o2nLUT)[source]
to_spectrum1d(record, o2nLUT)[source]
class api.type_conversion.Rtype(value)[source]

Bases: enum.Enum

An enumeration.

JSON = 1
NUMPY = 2
PANDAS = 3
SPECTRUM1D = 4
class api.type_conversion.SdssDr16[source]

Bases: api.type_conversion.Convert

to_numpy(record, o2nLUT)[source]

Convert FitsFile record to a structure that uses Numpy

to_pandas(record, o2nLUT)[source]
to_spectrum1d(record, o2nLUT)[source]
api.type_conversion.convert(record, rtype, client, include, verbose=False)[source]

api.utils module

api.utils.dict2tree(obj, name=None, prefix='')[source]

Return abstracted nested tree. Terminals contain TYPE. As a special case, a list is given as a dict that represents a compound type. E.G. {‘<list(3835)[0]>’: float} means a list of 3835 elements where the first element is of type ‘float’. NB: Only the type of the first element in a list is given. If the list has hetergeneous types, that fact is invisible in the structure!!

api.utils.here_now()[source]

Used to track info for benchmark. Probably OBE?

Parameters

None.

Returns

Time, date, and hostname.

api.utils.objform(obj)[source]

Nested structure of python object. Avoid spewing big lists. See also: https://code.activestate.com/recipes/577504/

Parameters

obj – Python object.

Returns

Length if list, objforms of dict contents if dict, type if anything else.

Example

>>> res = client.sample_records(1)[0]
>>> objform(res)
<class 'api.client.AttrDict'>
api.utils.tic()[source]

Start tracking elapsed time. Works in conjunction with toc().

Parameters

None.

Returns

Elapsed time.

api.utils.toc()[source]

Return elapsed time since previous tic().

Parameters

None.

Returns

Elapsed time since previous tic().

Module contents