Ensure you have Python and pip installed. As this is platform dependent, you need to figure out how to install them by yourself. This will get you started:
This module is hosted on the Python Package Index (PyPi). You can install and update it and all of its dependencies via the Python Package Manager (pip):
python -m pip install --user --upgrade coscine
Depending on your install you may have to substitute python
with py
or python3
.
You need an API token to use the Coscine API. If you have not already, create a new API token. Once you have an API token you are ready to use the API.
The token represents sensible data and grants anyone in possesion of it full access to your data in coscine. Do not leak it publicly on online platforms such as github! Do not include it within your sourcecode if you intend on uploading that to the internet. Take precautions and follow best practices to avoid corruption, theft or loss of data!
There are two simple and safe methods of using the API token without exposing it to unintended audiences.
Simply put your API token in a file on your harddrive and read the file when initializing the Coscine client. This has the advantage of keeping the token out of the sourcecode and offering the user an easy way to switch between tokens by changing the filename.
import os
fd = open("token.txt", "rt")
token = fd.read()
fd.close()
However it comes at the disadvantage of potentially exposing the token by accidentially leaking the file together with the sourcecode. Therefore precautions must be taken i.e. when using git as a versioning system. A .gitignore file including any possible token name or file extension should be mandatory. You could for example exclude the filename token.txt. A better way would be to agree upon a common token file extension such as .token and exclude that file extension. Then you can safely push your code to online platforms such as GitLab or GitHub.
This method does not rely on any files but instead on environment variables. Simply set an environment variable containing your token and use that variable from your python program.
import os
# Set environment variable
os.environ["COSCINE_API_TOKEN"] = "My Token Value"
# Get environment variable
token = os.getenv("COSCINE_API_TOKEN")
This is certainly a little more complex for some users who may want to use your program. They can easily share tokens by sending a file to colleagues but sharing environment variables requires each user to additionally create the environment variable on their local PC.
Find out how to temporarily or permanently set environment variables on certain Operating Systems:
Before you can use this python library you obviously have to import it in your sourcecode.
import coscine
Initializing the Coscine API client is done by calling the CoscineClient constructor:
coscine.Client(token: str, verbose: bool = False, lang: str = "en", persistent_cache: bool = False, loglevel)
The constructor takes one mandatory argument - the Coscine API token. A minimal usage example would thus look like this:
client = coscine.Client(token)
Note how we call the CoscineClient constructor with a variable called token. This variable should contain a string with your Coscine API token. You can set this variable by following one of the steps described in Creating an API token.
The constructor takes a few optional arguments such as:
verbose
(Boolean) : Enable/Disable command line output such as progress bars and status information. Exceptions are printed regardless. This option is disabled by default.lang
(String) : Set the language used for input forms. Possible values are "en" for english and "de" for german. By default it is set to "en".loglevel
(List of strings) : See Logging.persistent_caching
(Boolean) : Enable to save the request cache in a file for later use. Makes frequently stopped and subsequently restarted programs much faster.import coscine
TOKEN = "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
client = coscine.Client(TOKEN)
The python SDK provides a handful of logging functionality to inspect data sent to/from Coscine and get updated on up-/download progress.
You can enable/disable and configure it in the coscine.Client()
constructor.
coscine.Client(TOKEN, verbose: bool = False, loglevel: Iterable[str] = ["LOG", "INFO", "WARN", "REQUEST", "DATA"])
Setting verbose
to True
enables command line output. By default this option is disabled. Furthermore you can specify what kind of messages are printed by setting the loglevel
. You can choose between any combination of the aforementioned values. loglevel
expects a list of strings and could for examples be set to loglevel = ["REQUEST", "DATA"]
to inspect all incoming and outgoing http traffic aswell as the data that is coming in and getting sent out.
projects = client.projects() # Optional filtering possible e.g. client.projects(Author = "Jane")
for project in projects:
print(project.name)
If we know which project we require beforehand, we can just query it by its name. We now get a single object of type Project, instead of a list:
project = client.project("My Project") # We could also filter by different keys such as Author: client.project(Author="Joe")
print(project)
Here we can see some more project metadata. When printing a project object, its metadata is output nicely formatted inside of a table.
The keys and values are human readable and easy to understand. Under the hood its not that simple and sometimes we might need to access the Coscine internal identifier for a certain field. In that case we can use the data dictionary, which grants us access to the metadata as seen by the Coscine server. Printing it yields a JSON representation of our projects metadata:
print(project.data)
Once again a plain and simple function call. Be aware though, that this function may fail due to unsufficient privileges. The call may yield an error, which we should catch and handle:
try:
project.delete()
except coscine.AuthorizationError:
print("We are not authorized.")
The python module provides a simplified way of programmatically creating a project. All data formatting and error detection is performed under the hood.
form = client.ProjectForm()
# or: form = coscine.project.ProjectForm(client)
print(form)
form = client.ProjectForm()
form["Projektname"] = "test"
form["Anzeigename"] = "test"
form["Projektbeschreibung"] = "test"
form["Principal Investigators"] = "test"
form["Projektstart"] = "test"
form["Projektende"] = "test"
form["Disziplin"] = ["Informatik 409"]
form["Teilnehmende Organisation"] = ["RWTH Aachen University"]
form["Sichtbarkeit"] = "Public"
client.create_project(form)
Looking at the use of a dictionary, one might ask why we do not just use a function with named arguments for each dictionary field. Multiple benefits arise from the use of a dictionary:
The 2nd benefit ultimately enables easy inclusion in GUI applications.
To download a project and all of the resources contained within the project, call the download method. You need to specify a storage location using the path argument, otherwise the current directory will be used.
project.download(path="./")
members = project.members()
for member in members:
print(member.name)
print(member.email)
# Set project role for a member
member.set_role("Owner")
# Delete a member
member.remove()
Inviting new members:
EMAIL_ADDRESS: str = "john@example.com"
ROLE: str = "Member"
project.invite(EMAIL_ADDRESS, ROLE)
Resources store all of your data and metadata. As such they represent a key data structure, which you most certainly will interact a lot with.
Analoguous to getting a list of projects, you can get a list of resoures. The only difference being, that the resources()
method is part of the project object and does only query resources contained within that project. Just like for projects, you can specify a filter to filter by certain resource properties.
resources = project.resources()
for resource in resources:
print(resource.name)
If we know which resource we require beforehand, we can just query it by its name. We now get a single object of type Resource, instead of a list:
resource = project.resource("RessourcenNameS3")
print(resource)
Again, this does not work with our public token - you do not have privileges to delete our sample resource. Try deleting a resource you have created, but be careful not to delete anything of value.
try:
resource.delete()
except coscine.UnauthorizedError:
print("Not authorized.")
Downloading a resource and all of the data contained within the resource is just as simple as downloading a project. In fact internally project.download()
just calls Resource.download()
for all resources contained within the project.
resource.download(path="./")
You can fetch the used up quota of a resource as an integer indicating the size used in Bytes.
quota = resource.quota()
print(quota)
An application profile specifies a template for metadata. There may be times where you need to interact with that profile. To get the application profile of a resource you simple call the application_profile()
method. You can either get the raw application profile in JSON-LD format or a more readable (and easier to interact with) parsed version, by setting the parse
argument to True
.
# Print a raw and a parsed application profile
profile = resource.applicationProfile()
print(profile)
print("-------------------------------------------------------------")
profile = resource.applicationProfile(parse=True)
print(profile)
Once again we are using InputForms to set metadata of a Coscine object.
form = project.ResourceForm()
# or: form = coscine.resource.ResourceForm(project)
form = project.ResourceForm()
form["Resource Type"] = "rds"
form["Resource Size"] = "31"
form["Resource Name"] = "My Cool Resource"
form["Display Name"] = "Cool"
form["Resource Description"] = "Testing Coscine Client Resources"
form["Discipline"] = ["Computer Science 409"]
form["Application Profiles"] = "RADAR"
form["Visibility"] = "Project Members"
project.create_resource(form)
RDS-S3 resources can be directly accessed via an S3-client. Direct connections require S3 credentials, which s3 resource instances happily provide to us. Only works for s3 resources.
access_key: str = resource.s3.access_key
secret_key: str = resource.s3.secret_key
endpoint: str = resource.s3.endpoint
bucket: str = resource.s3.bucket
Files are stored inside of resources. However Resources do not necessarily contain files. It depends on the resource type. RDS and RDS-S3 contain files, but Linked Data resources contain references to files. Therefore we cannot just talk about files, but have to use a more abstract term such as 'object'. Objects represent files and file-like instances in Coscine. Nonetheless the methods of interacting with files and file-like objects is always the same - just don't expect file contents for objects of linked data resources, as those merely contain links or whatever has been specified as their content.
files = resource.objects()
for file in files:
print(file.name)
file = resource.object("Messung (1).bin")
print(file.name)
data = file.content()
file = resource.object("Messung (1).bin")
path = "./"
file.download(path)
metadata = coscine.MetadataForm(resource)
# or: metadata = resource.MetadataForm()
metadata["Title"] = "..."
# fill in fields
filename = "messung.bin" # filename as it should appear in Coscine
path = "./data/messung.bin" # path on harddrive
resource.upload(filename, path, metadata)
file = resource.object("Messung (1).bin")
file.delete()
# The file object is still valid until garbage collected.
# The file on the Coscine server has already been removed though.
We can interact with metadata using a MetadataForm.
form = file.form()
print(form)
The form fields change depending on the selected application profile for the resource.
form["Title"] = "My Title"
# Update metadata
file.update(form)
class CoscineException(Exception):
"""
Coscine base Exception class.
"""
pass
###############################################################################
class ConnectionError(CoscineException):
"""
In case the client is not able to establish a connection with
the Coscine servers, a ConnectionError is raised.
"""
pass
###############################################################################
class ClientError(CoscineException):
"""
An error has been made or detected on the client side.
"""
pass
###############################################################################
class ServerError(CoscineException):
"""
An error has been made or detected on the Coscine server.
"""
pass
###############################################################################
class VocabularyError(CoscineException):
"""
Raised in InputForms when a supplied value is not contained within
a controlled vocabulary.
"""
pass
###############################################################################
class RequirementError(CoscineException):
"""
Commonly raised in InputForms when a required field has not been set.
"""
pass
###############################################################################
class AuthorizationError(CoscineException):
"""
AuthorizationErrors are thrown when the owner of the Coscine API
token does not hold enough privileges.
"""
pass
###############################################################################
class AmbiguityError(CoscineException):
"""
An AmbiguityError is raised in cases where two objects could
not be differentiated between.
"""
pass
###############################################################################
class ParameterError(CoscineException):
"""
Invalid (number of) function parameters provided. In some cases
the user has the option of choosing between several optional arguments,
but has to provide at least one.
"""
pass
###############################################################################
You can generate detailed documentation using pydoc.
Install pydoc with py -m pip install pydoc
.
Install coscine with py -m pip install coscine
.
Download the coscine python sdk repository as a zip or with git.
Navigate to the coscine/doc
directory inside of the repository and either use one of the supplied scripts or generate documentation yourself with py -m pydoc -w coscine
, py -m pydoc -w coscine.project
, etc.