readthedocs

abcunit-backend

Storage backends for an ABCUnit Framework, logging success and failures of process units.

Database Backend

To use this backend you will need to contact the JASMIN help desk (support@jasmin.ac.uk) and ask them to setup a postgresql database for you. Specify a name for the database and a username to login with. JASMIN support will get back to you with the user password and host name.

After you have got your database, you’ll need to export an environment variable called $ABCUNIT_DB_SETTINGS and set it to a connection string for psycopg2:

ABCUNIT_DB_SETTINGS="dbname=<name> user=<user> host=<host> password=<pwd>"

DatabaseHandler class construction looks like this:

DatabaseHandler(table_name="results")

Where

• table_name is the name of the table logs will be insert into

Connects to an existing database and creates a table to store results:

<table_name> (id varchar(255) PRIMARY KEY, result varchar(255) NOT NULL)

id	result
facet1.facet2.facet3	success
facet1.facet2.facet3	bad_file
⋮	⋮

File System Backend

FileSystemHandler class construction looks like this:

FileSytemHandler(base_log_dir, n_facets, sep)

Where;

• base_log_dir is the string path to top level directory for logs

• n_facets is the number of facets used to describe each unit result

• sep is the separator used for a result identifier

Uses the file system to create log files marking success and failures, categorised by directory structure:

<log_base_dir>/success/facet1/facet2/facet3

<log_base_dir>/failure/error_type/facet1/facet2/facet3

API

Backend

class abcunit_backend.database_handler.DataBaseHandler(table_name='results')

Bases: abcunit_backend.base_handler.BaseHandler

count_failures()

Returns

(int) Number of failed results in the table

count_results()

Returns

(int) Number of results in the table

count_successes()

Returns

(int) Number of successfull results in the table

delete_all_results()

Deletes all entries from the table

delete_result(identifier)

Deletes entry specified by the given identifier from the database

Parameters

identifier – (str) Identifier of the job

get_all_results()

Returns

(dict) Dictionary of all job identifiers mapped to their respective results

get_failed_runs()

Returns

(dict) Dictionary of error types mapped to lists of job identifiers which result in them

get_result(identifier)

Selects the result of the job with the identifier parsed and returns it

Parameters

identifier – (str) Identifier of the job result

Returns

(str) Result of job

get_successful_runs()

Returns

(str list) Returns a list of the identifiers of all successful runs

insert_failure(identifier, error_type='failure')

Inserts an entry into the table with a given identifier and erroneous result

Parameters

• identifier – (str) Identifier of the job

• error_type – (str) Result of the job

insert_success(identifier)

Inserts an entry into the table with a given identifier and the result ‘success’

Parameters

identifier – (str) Identifier of the job

ran_successfully(identifier)

Returns true / false on whether the result with this identifier is successful

Parameters

identifier – (str) Identifier of the job result

Returns

(bool) Boolean on if job ran successfully

class abcunit_backend.file_system_handler.FileSystemHandler(base_log_dir, n_facets, sep)

Bases: abcunit_backend.base_handler.BaseHandler

count_failures()

Returns

(int) Number of failure result files

count_results()

Returns

(int) Number of results in the table

count_successes()

Returns

(int) Number of successful result files

delete_all_results()

Deletes all result files in the file system under the base_log_dir

delete_result(identifier)

Deletes result file from the file system given its identifier

Parameters

identifier – (str) Identifier of the job

get_all_results()

Returns

(dict) Dictionary of all job identifiers mapped to their respective results

get_failed_runs()

Returns

(dict) Dictionary of error types mapped to lists of job identifiers which result in them

get_result(identifier)

Returns the value of a result given its identifier

Parameters

identifier – (str) Identifier of the job

Returns

(str) Result of job

get_successful_runs()

Returns

(str list) Returns a list of the identifiers of all successful runs

insert_failure(identifier, error_type)

Creates a failure result file using the identifier and error type parsed

Parameters

• identifier – (str) Identifier of the job

• error_type – (str) Result of the job

insert_success(identifier)

Creates a successful result file with the identifier passed

Parameters

identifier – (str) Identifier of the job

ran_successfully(identifier)

Returns true / false on whether the result with this identifier is successful

Parameters

identifier – (str) Identifier of the job result

Returns

(bool) Boolean on if job ran successfully

validate()

Decorator to check if an identifier is of the correct format

History

1.2.0 (2021-03-17)

• Database conections now opened and closed for each method call

1.1.0 (2021-02-09)

• Changed module name from backend to abcunit_backend

• Removed need to parse error_types to either handler

1.0.0 (2020-11-06)

• First release