api package
api.client module
Client module for SPARCL.
This module interfaces to the SPARC-Server to get spectra data.
Todo
Data Discovery
Allow retrieval of spectra based upon matches to metadata values.
- class api.client.SparclApi(url='http://sparc1.datalab.noirlab.edu:8000', verbose=False, internal_names=False, connect_timeout=1.1, read_timeout=5400)[source]
Bases:
object
Provides interface to SPARCL Server.
When using this to report a bug, set verbose to True. Also print your instance of this. The results will include important info about the Client and Server that is usefule to Developers.
- Parameters
url – Base URL of SPARC Server
verbose – (True,[False]) Default verbosity for all client methods.
verbose – (True,[False]) True:: override field renaming
[1.1] (connect_timeout) – Number of seconds to wait to establish connection with server.
[5400] (read_timeout) – Number of seconds to wait for server to send a response. (generally time to wait for first byte)
Example
>>> client = SparclApi(verbose=True) >>> print(client)
Object creation compares the version from the Server against the one expected by the Client. Throws error if the Client is a major version or more behind.
- KNOWN_GOOD_API_VERSION = 4.0
- client_field(structure, orig_name)[source]
Get field name used in Client methods
- Parameters
structure – Name of Data Set
orig_name – Original field name as provided in Data Set.
- Returns
Client field name
- Return type
string
Example
>>> client.client_field('BOSS-DR16', 'spectra.coadd.FLUX') 'flux'
- get_field_names(structure)[source]
List field names available for retreive.
- Parameters
structure – List field names of this Data Set.
- Returns
list of field names
- Return type
list
Example
>>> client.get_field_names('DESI-everest')
- missing_specids(specid_list, countOnly=False, verbose=False)[source]
Return the subset of the given specid list that is NOT stored in the database.
- Parameters
specid_list (list) – List of specids.
verbose (boolean, optional) – (default: False)
- Returns
The subset of the given specid list that is NOT stored in the database.
Example
>>> si = [1858907533188556800, 6171312851359387648, 1647268306035435520] >>> client.missing_specids(si) [1858907533188556800, 6171312851359387648]
- normalize_field_names(recs)[source]
Return copy of records with all field names converted to the names used by the Data Set provider.
- Parameters
recs – List of dictionaries representing spectra records
- Returns
new list of dicts of spectra records (with diff field names)
- Return type
list
Example
>>> recs = client.sample_records(1) >>> client.normalize_field_names(recs)
- orig_field(structure, client_name)[source]
Get original field name as provided in Data Set.
- Parameters
structure – Name of Data Set
client_name – Field name used in Client methods.
- Returns
Original field name
- Return type
string
Example
>>> client.orig_field('BOSS-DR16', 'flux') 'spectra.coadd.FLUX'
- retrieve(specid_list, include='DEFAULT', rtype=None, structure=None, verbose=False)[source]
Get spectrum from specid list.
- Parameters
specid_list (list) – List of specids.
include (list, 'DEFAULT', 'ALL') – List of paths to include in each record. (default: ‘DEFAULT’)
rtype (str, optional) – Data-type to use for spectra data. One of: json, numpy, pandas, spectrum1d. (default: None)
structure (str) – The Data Set (DS) name associated with the specids. Or None to retrieve from any DS that contains the specid. (default: None)
verbose (boolean, optional) – (default: False)
- Returns
List of records. Each record is a dictionary of named fields.
Example
>>> ink = ['flux'] >>> sdss_ids = [849044290804934656, 309718438815754240] >>> res_sdss = client.retrieve(sdss_ids, structure='SDSS-DR16', include=ink)
- sample_records(count, structure=None, include='DEFAULT', **kwargs)[source]
Return list of random records from given STRUCTURE (Data Set).
- Parameters
count (int) – Number of sample records to get from database.
structure (str, optional) – (default: None means ANY) The Data Set from which to get sample records.
include (list, 'DEFAULT', 'ALL') – List of paths to include in each record. (default: ‘DEFAULT’)
- Returns
List of random records from given STRUCTURE (Data Set).
Example
>>> samrec = client.sample_records(1, structure='BOSS-DR16') >>> pprint.pprint(samrec,depth=2) [{'data_release_id': 'BOSS-DR16', 'dec_center': 47.193549, 'dirpath': '/net/mss1/archive/hlsp/sdss/dr16/eboss/spectro/redux/v5_13_0/spectra/lite/7399', 'filename': 'spec-7399-57162-0376.fits', 'specid': 1429845755960551, 'spectra': {...}, 'updated': '2021-04-28T20:16:20.399464Z'}]
- sample_specids(samples=5, structure=None, random=True, **kwargs)[source]
Return a small list of specids.
This is intended to make it easy to get just a few specids to use for experimenting with the rest of the API.
- Parameters
samples (int, optional) – (default: 5) The number of sample specids to get.
structure (str, optional) – (default: None means ANY) The Data Set from which to get specids.
random (True,False,None) – Randomize sample
- Returns
List of specids
Rtype: list
Example
>>> client.sample_specids(samples=3, structure='DESI-everest')
- property version
Return version of Server Rest API used by this client.
If the Rest API changes such that the Major version increases, a new version of this module will likely need to be used.
- Returns
API version.
- Return type
float
Example
>>> SparclApi('http://localhost:8030').version 1.0
- api.client.fields_available(records)[source]
Get list of fields used in records. One list per Data Set.
- Parameters
records – List of records (each is a dictionary)
- Returns
dict[Structure] = [field_name1, …]
- Return type
dict
Example
>>> from api.client import fields_available >>> recs = client.sample_records(3) >>> flds = fields_available(recs)
- api.client.get_metadata(records)[source]
Get records of just metadata used in records.
Metadata is considered to be any field whose type is a Number or String. Therefore, this will not include vectors, lists, tuples, etc.
- Parameters
records – List of records (dictionaries)
- Returns
new list of dictionaries. Each dict contains only metadata fields.
- Return type
list(dict)
Example
>>> from api.client import get_metadata >>> recs = client.sample_records(3) >>> metadata = get_metadata(recs)
- api.client.record_examples(records)[source]
Copy one record for each Data Set type.
- Parameters
records – List of records (each is a dictionary)
- Returns
dict[Structure] = rec
- Return type
dict
Example
>>> from api.client import record_examples >>> recs = client.sample_records(3) >>> rex = record_examples(recs)
- api.client.rename_fields(rename_dict, records)[source]
Rename some field names in all given records.
- Parameters
rename_dict – The key is current field name, value is new.
records – List of records (dictionaries) to transform
- Returns
new_records
- Return type
list
Example
>>> from api.client import rename_fields >>> recs = client.sample_records(1) >>> renamed = rename_fields({'ra':'Right_Ascension'},recs)
api.exceptions module
- exception api.exceptions.BadInclude(error_message, error_code=None)[source]
Bases:
api.exceptions.BaseSparclException
Include list contains invalid data field(s).
- error_code = 'BADINCL'
- exception api.exceptions.BadPath(error_message, error_code=None)[source]
Bases:
api.exceptions.BaseSparclException
A field path starts with a non-core field.
- error_code = 'BADPATH'
- exception api.exceptions.BaseSparclException(error_message, error_code=None)[source]
Bases:
Exception
Base Class for all SPARCL exceptions.
- error_code = 'UNKNOWN'
- error_message = '<NA>'
- traceback = None
- exception api.exceptions.ReadTimeout(error_message, error_code=None)[source]
Bases:
api.exceptions.BaseSparclException
The server did not send any data in the allotted amount of time.
- error_code = 'RTIMEOUT'
- exception api.exceptions.UnkDr(error_message, error_code=None)[source]
Bases:
api.exceptions.BaseSparclException
The Data Release is not known or not supported.
- error_code = 'UNKDR'
- exception api.exceptions.UnknownServerError(error_message, error_code=None)[source]
Bases:
api.exceptions.BaseSparclException
Client got a status response from the SPARC Server that we do not know how to decode.
- error_code = 'UNKNOWN'
- exception api.exceptions.UnknownSparcl(error_message, error_code=None)[source]
Bases:
api.exceptions.BaseSparclException
Unknown SPARCL error. If this is ever raised (seen in a log) create and use a new BaseSparcException exception that is more specific.
- error_code = 'UNKSPARC'
api.type_conversion module
- class api.type_conversion.BossDr16[source]
Bases:
api.type_conversion.Convert
- class api.type_conversion.Convert[source]
Bases:
abc.ABC
Convert JSON record to mix of plain python and selected data record type.
- class api.type_conversion.Desi[source]
Bases:
api.type_conversion.Convert
- class api.type_conversion.DesiDenali[source]
Bases:
api.type_conversion.Desi
- class api.type_conversion.DesiEverest[source]
Bases:
api.type_conversion.Desi
- class api.type_conversion.NoopConvert[source]
Bases:
api.type_conversion.Convert
- class api.type_conversion.Rtype(value)[source]
Bases:
enum.Enum
An enumeration.
- JSON = 1
- NUMPY = 2
- PANDAS = 3
- SPECTRUM1D = 4
- class api.type_conversion.SdssDr16[source]
Bases:
api.type_conversion.Convert
api.utils module
- api.utils.dict2tree(obj, name=None, prefix='')[source]
Return abstracted nested tree. Terminals contain TYPE. As a special case, a list is given as a dict that represents a compound type. E.G. {‘<list(3835)[0]>’: float} means a list of 3835 elements where the first element is of type ‘float’. NB: Only the type of the first element in a list is given. If the list has hetergeneous types, that fact is invisible in the structure!!
- api.utils.here_now()[source]
Used to track info for benchmark. Probably OBE?
- Parameters
None. –
- Returns
Time, date, and hostname.
- api.utils.objform(obj)[source]
Nested structure of python object. Avoid spewing big lists. See also: https://code.activestate.com/recipes/577504/
- Parameters
obj – Python object.
- Returns
Length if list, objforms of dict contents if dict, type if anything else.
Example
>>> res = client.sample_records(1)[0] >>> objform(res) <class 'api.client.AttrDict'>