Problems when uploading via s3
I'm running into various problems when uploading data to Coscine via S3 in the Python SDK.
Using the following code, everything appears to be OK:
import requests
requests.packages.urllib3.util.ssl_.DEFAULT_CIPHERS += 'DEFAULT:!DH'
from pathlib import Path
from datetime import datetime
import coscine
TOKEN_COS = 'mytoken'
client_settings = coscine.Settings(verbose=True)
client = coscine.Client(TOKEN_COS, settings=client_settings)
# enter project name
PROJECT: str = "NikkisSandbox"
# enter resource name
RESOURCE: str = 'DSA Test'
project = client.project(PROJECT, toplevel=False)
resource = project.resource(RESOURCE)
ACCESS_KEY: str = resource.s3.write_access_key
SECRET_KEY: str = resource.s3.write_secret_key
ENDPOINT: str = resource.s3.endpoint
BUCKETNAME: str = resource.s3.bucket
resource.s3.upload(str(filepath.absolute()), str(filepath))
Returns:
/Users/nicoleparks/sciebo/Dokumente/ITC/IT-RPDM/coscine/testfile.txt: 100% 3.00/3.00 [00:47<00:00, 4.22B/s]
However, at this point, I run into various problems:
- Cannot update metadata via API (but that's probably Issue #35 (closed))
- Listing the contents of the resource returns:
HTTPError Traceback (most recent call last)
File /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/coscine/client.py:382, in Client._request(self, method, uri, **kwargs)
381 response = self.session.request(method, uri, **kwargs)
--> 382 response.raise_for_status()
383 logger.debug("response: %s", str(response.content))
File /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/requests/models.py:1021, in Response.raise_for_status(self)
1020 if http_error_msg:
-> 1021 raise HTTPError(http_error_msg, response=self)
HTTPError: 400 Client Error: Bad Request for url: https://coscine.rwth-aachen.de/coscine/api/Coscine.Api.Tree/Tree/0e2ff439-8b05-478d-bd80-0ee9530296b9
The above exception was the direct cause of the following exception:
RuntimeError Traceback (most recent call last)
Cell In[85], line 1
----> 1 resource.contents()
File /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/coscine/resource.py:402, in Resource.contents(self)
400 contents = []
401 directories = []
--> 402 files = self.objects()
403 while more_folders:
404 more_folders = False
File /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/coscine/resource.py:441, in Resource.objects(self, path)
439 dirpath = posixpath.dirname(path) if path else None
440 args = {"path": path} if dirpath and dirpath != "/" else None
--> 441 data = self.client.get(uri, params=args).json()
442 file_storage: List[dict] = data["data"]["fileStorage"]
443 metadata_storage: List[dict] = data["data"]["metadataStorage"]
File /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/coscine/client.py:428, in Client.get(self, uri, **kwargs)
397 def get(self, uri: str, **kwargs) -> requests.Response:
398 """
399 Performs a GET request to the Coscine API.
400
(...)
425 The response of the Coscine server as a requests.Response object.
426 """
--> 428 return self._request("GET", uri, **kwargs)
File /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/coscine/client.py:390, in Client._request(self, method, uri, **kwargs)
388 if exc.response.status_code == 401:
389 raise PermissionError("Invalid Coscine API token!") from exc
--> 390 raise RuntimeError(
391 "Unspecified error occurred when communicating "
392 "with the Coscine servers"
393 ) from exc
RuntimeError: Unspecified error occurred when communicating with the Coscine servers
This worked before the file was uploaded via s3
- Using Cyberduck, the uploaded file is not there (I can upload files without a problem using Cyberduck)
- Looking at the resource via the web interface results in a
400 error bad request
popup. The resource does not load (or shows the content of the last functioning resource I looked at).
The folliwing does not give the same problems, works just fine:
import requests
requests.packages.urllib3.util.ssl_.DEFAULT_CIPHERS += 'DEFAULT:!DH'
import boto3
s3 = boto3.client(
's3',
endpoint_url=ENDPOINT,
aws_access_key_id=ACCESS_KEY,
aws_secret_access_key=SECRET_KEY
)
with open("testfile.txt", "rb") as f:
s3.upload_fileobj(f, BUCKETNAME, f.name)