Commit 639cf05b authored by Mayr, Hannes's avatar Mayr, Hannes
Browse files

Merge branch 'dev' into 'main'

Create v0.2.2

Closes #77, #94, #105, #95, #85, #86, #65, and #80

See merge request !52
parents c64d0385 c7fb8b7e
Pipeline #871219 canceled with stages
......@@ -19,7 +19,7 @@ PEP8:
stage: linting
script:
- pip install flake8
- flake8 --count .
- flake8 --count --max-line-length=88 .
Pylint:
stage: linting
......@@ -27,6 +27,19 @@ Pylint:
- pip install pylint
- find . -type f -name '*.py' | xargs pylint -rn --rcfile='plotid/.pylintrc' # Find all python files and check the code with pylint
Autoformatting:
stage: linting
script:
- pip install black
- black --check --verbose --diff --color .
Typechecker:
stage: linting
script:
- pip install mypy
- mypy --ignore-missing-imports --strict plotid tests examples
allow_failure: true
test:
stage: testing
tags:
......@@ -53,7 +66,7 @@ sast:
before_script:
- echo "No before script pls"
variables:
SAST_EXCLUDED_PATHS: spec, test, tmp
SAST_EXCLUDED_PATHS: spec, tests, tmp
stage: security
tags:
- docker
......
......@@ -59,24 +59,26 @@ Optional parameters can be used to customize the tag process.
- *prefix* : str, optional
Will be added as prefix to the ID.
- *id_method* : str, optional
id_method for creating the ID. Create an ID by Unix time is referenced as 'time', create a random ID with id_method='random'. The default is 'time'.
id_method for creating the ID. Create an ID by Unix time is referenced as "time", create a random ID with id_method="random". The default is "time".
- *location* : string, optional
Location for ID to be displayed on the plot. Default is 'east'.
Location for ID to be displayed on the plot. Default is "east".
- *qrcode* : boolean, optional
Experimental support for encoding the ID in a QR Code.
- *id_on_plot* : boolean, optional
Print ID on the plot. Default: True.
Example:
```python
FIG1 = plt.figure()
FIG2 = plt.figure()
FIGS_AS_LIST = [FIG1, FIG2]
FIGS_AND_IDS = tagplot(FIGS_AS_LIST, 'matplotlib', prefix='XY23_', id_method='random', location='west')
FIGS_AND_IDS = tagplot(FIGS_AS_LIST, "matplotlib", prefix="XY23_", id_method="random", location="west")
```
### publish()
Save plot, data and measuring script. It is possible to export multiple figures at once.
`publish(figs_and_ids, src_datapath, dst_path, plot_name)`
`publish(figs_and_ids, src_datapath, dst_path)`
- *figs_and_ids* must be a PlotIDTransfer object. Therefore, it can be directly passed from tagplot() to publish().
- *src_datapath* specifies the path to (raw) data that should be published. It can be a string or a list of strings that specifies all files and directories which will be published.
......@@ -88,15 +90,17 @@ Optional parameters can be used to customize the publish process.
Method how the data should be stored. Available options:
- *centralized*: The raw data will copied only once. All other plots will reference this data via sym link.
- *individual*: The complete raw data will be copied to a folder for every plot, respectively.
- *plot_names* : str or list of str, optional
Name for the exported plot. If not provided, the corresponding IDs will be used.
Example:
`publish(figs_and_ids, '/home/user/Documents/research_data', '/home/user/Documents/exported_data', 'EnergyOverTime-Plot')`
`publish(figs_and_ids, "/home/user/Documents/research_data", "/home/user/Documents/exported_data", plot_names=["EnergyOverTime-Plot", "TimeOverEnergy-Plot")`
## Build
If you want to build plotID yourself, follow these steps:
1. Download the source code from [Gitlab](https://git.rwth-aachen.de/plotid/plotid_python):
`git clone https://git.rwth-aachen.de/plotid/plotid_python.git`
`cd plotid_python`
2. *[Optional]* Create a virtual environment (see [Installation](#Installation)).
2. *[Optional]* Create a virtual environment (see [Installation](#installation)).
3. *[Optional]* Run unittests and coverage:
`python3 tests/runner_tests.py`
4. Build the package
......@@ -110,4 +114,4 @@ To install all optional dependencies use `pip install .[test,docs]` or `pip inst
## Documentation
If you have more questions about plotID, please have a look at the [documentation](https://plotid.pages.rwth-aachen.de/plotid_python).
Also have a look at the [examples](./examples) that are shipped with plotID.
Also have a look at the [examples](https://git.rwth-aachen.de/plotid/plotid_python/-/tree/main/examples) that are shipped with plotID.
About plotID
==================================
plotID is developed at the Institut Fluidsystemtechnik within nfdi4ing at the Technische Universität Darmstadt.
Acknowledgements
----------------
The authors would like to thank the Federal Government and the Heads of Government of the Länder, as well as the Joint Science Conference (GWK), for their funding and support within the framework of the NFDI4Ing consortium. Funded by the German Research Foundation (DFG) - project number 442146713.
\ No newline at end of file
......@@ -14,15 +14,16 @@
import os
import sys
sys.path.insert(0, os.path.abspath('../..'))
sys.path.insert(0, os.path.abspath("../.."))
from plotid import __version__, __author__ # noqa: E402
# -- Project information -----------------------------------------------------
project = 'plotID'
copyright = '2022, ' + __author__
project = "plotID"
copyright = "2022, " + __author__
author = __author__
# The full version, including alpha/beta/rc tags
......@@ -34,16 +35,14 @@ release = __version__
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = [
'sphinx.ext.napoleon',
'autoapi.extension',
"myst_parser"
]
autoapi_type = 'python'
autoapi_dirs = ['../../plotid', '../../tests']
extensions = ["sphinx.ext.napoleon", "autoapi.extension", "myst_parser"]
autoapi_type = "python"
autoapi_dirs = ["../../plotid", "../../tests"]
# autoapi_keep_files = True
myst_heading_anchors = 2
# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']
templates_path = ["_templates"]
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
......@@ -56,10 +55,10 @@ exclude_patterns = []
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
#
html_theme = 'sphinx_rtd_theme'
html_logo = '_static/plotID.jpg'
html_theme = "sphinx_rtd_theme"
html_logo = "_static/plotID.jpg"
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ['_static']
html_static_path = ["_static"]
......@@ -8,10 +8,11 @@ The plotID package depends on other software for its full funcionality. Since pl
- `qrcode <https://github.com/lincolnloop/python-qrcode>`_: Creates QR codes from input data. qrcode is used to create the QR code which is printed on the plot in the end.
- `numpy <https://numpy.org/doc/stable/>`_: Package for scientific computing. Right now it is only used to generate sample data in the examples. Since it is a dependency of matplotlib, it has to be installed anyway.
Optional depedencies that are not necessary to run plotID but more to contribute to the code are the following:
Optional dependencies that are not necessary to run plotID but more to contribute to the code quality are the following:
- `coverage <https://coverage.readthedocs.io/en/6.5.0/#>`_: Program to measure the code coverage. It is used in the development of plotID to monitor the coverage of the code when the unit tests are run.
- `sphinx <https://www.sphinx-doc.org/en/master/>`_: Program to build documentation. For example, this documentatio is generated with the help of sphinx.
- `sphinx-autoapi <https://sphinx-autoapi.readthedocs.io/en/latest/>`_: Generates documentation files automatically from documentation inside the code. This has the advantage that code can be documented inside the code but can also be read in the documentation without having two places with the same content.
- `sphinx-rtd-theme <https://sphinx-rtd-theme.readthedocs.io/en/stable/>`_: Readthedocs theme for nicer documentation style.
- `myst-parser <https://myst-parser.readthedocs.io/en/latest/>`_: Extenstion for sphinx, e.g. to include markdown files in the documentation. That allows to include plotID's README.md in the documentation without storing it at two different places.
\ No newline at end of file
- `myst-parser <https://myst-parser.readthedocs.io/en/latest/>`_: Extension for sphinx, e.g. to include markdown files in the documentation. That allows to include plotID's README.md in the documentation without storing it at two different places.
......@@ -10,9 +10,11 @@ Welcome to plotID's documentation!
:maxdepth: 2
:caption: Contents:
README <readme_link.rst>
Structure and architecture <structure.rst>
Overview <readme_link.rst>
Installation <installation.rst>
Dependencies <dependencies.rst>
Structure and architecture <structure.rst>
About <about.rst>
......
Installation of plotID
==================================
For installation instructions, first have a look at the general :doc:`readme_link` or the `README.md <https://git.rwth-aachen.de/plotid/plotid_python/-/blob/main/README.md>`_ file. More detailed information about the installation process can then be found here.
Wrong python version
--------------------
To install and run plotID a python version ≥ 3.10 is required. If your linux distribution is shipped with a python version < 3.10, you still can install a newer python version. Intalling python 3.10 on Debian or Ubuntu would look like this
.. code:: bash
sudo apt install python3.10 python3.10-venv
You can then create a virtual environment "venv" to run plotID inside it
.. code:: bash
python3.10 -m venv venv
Now activate the venv
.. code:: bash
source venv/bin/activate
The virtual environment now runs with python 3.10 and plotID can be installed the usual way as described in :doc:`Overview`.
Installation on Windows via pip
-------------------------------
If you're trying
.. code:: cmd
pip install .
and following error is thrown
.. code:: cmd
Fatal error in launcher: Unable to create process using '"...env\Scripts\python.exe" "..\venv\Scripts\pip.exe" install .': The system cannot find the specified file.
Try to forcefully reinstall pip
.. code:: cmd
python -m pip install --upgrade --force pip
Or use
.. code:: cmd
python -m pip install .
instead.
plotid
======
.. toctree::
:maxdepth: 4
plotid
plotid package
==============
Submodules
----------
plotid.create\_id module
------------------------
.. automodule:: plotid.create_id
:members:
:undoc-members:
:show-inheritance:
plotid.example module
---------------------
.. automodule:: plotid.example
:members:
:undoc-members:
:show-inheritance:
plotid.filecompare module
-------------------------
.. automodule:: plotid.filecompare
:members:
:undoc-members:
:show-inheritance:
plotid.hdf5\_external\_link module
----------------------------------
.. automodule:: plotid.hdf5_external_link
:members:
:undoc-members:
:show-inheritance:
plotid.plotoptions module
-------------------------
.. automodule:: plotid.plotoptions
:members:
:undoc-members:
:show-inheritance:
plotid.publish module
---------------------
.. automodule:: plotid.publish
:members:
:undoc-members:
:show-inheritance:
plotid.save\_plot module
------------------------
.. automodule:: plotid.save_plot
:members:
:undoc-members:
:show-inheritance:
plotid.tagplot module
---------------------
.. automodule:: plotid.tagplot
:members:
:undoc-members:
:show-inheritance:
plotid.tagplot\_matplotlib module
---------------------------------
.. automodule:: plotid.tagplot_matplotlib
:members:
:undoc-members:
:show-inheritance:
Module contents
---------------
.. automodule:: plotid
:members:
:undoc-members:
:show-inheritance:
```{include} ../../README.md
:relative-docs: docs/
:relative-images:
```
\ No newline at end of file
Readme File
-----------
.. include:: ../../README.md
:parser: myst_parser.sphinx_
\ No newline at end of file
Structure and architecture of plotID
==================================
====================================
A visualization of the internal structure of plotID is provided here. The structure tries to follow Object Oriented Programming (OOP) principles. First tagplot() is depicted with matplotlib as example plot engine:
......
......@@ -15,8 +15,8 @@ from plotid.publish import publish
PROJECT_ID = "MR05_"
# %% Read example images
IMG1 = 'example_image1.png'
IMG2 = 'example_image2.png'
IMG1 = "example_image1.png"
IMG2 = "example_image2.png"
# %% TagPlot
......@@ -24,8 +24,9 @@ IMG2 = 'example_image2.png'
IMGS_AS_LIST = [IMG1, IMG2]
# Example for how to use tagplot with image files
FIGS_AND_IDS = tagplot(IMGS_AS_LIST, 'image', prefix=PROJECT_ID,
id_method='time', location='west')
FIGS_AND_IDS = tagplot(
IMGS_AS_LIST, "image", prefix=PROJECT_ID, id_method="time", location="west"
)
# Required arguments: tagplot(images as list, desired plot engine)
......@@ -34,7 +35,6 @@ FIGS_AND_IDS = tagplot(IMGS_AS_LIST, 'image', prefix=PROJECT_ID,
# Export your tagged images, copy the research data that generated the images,
# specify the destination folder and give a name for the exported image files.
publish(FIGS_AND_IDS, ['../README.md', '../docs', '../LICENSE'],
'./data', 'my_image')
publish(FIGS_AND_IDS, ["../README.md", "../docs", "../LICENSE"], "./data")
# Required arguments: publish(output of tagplot(), list of files,
# path to destination folder, name(s) for the resulting images)
# path to destination folder)
......@@ -25,13 +25,13 @@ y_2 = np.sin(x) + 2
# 1. figure
FIG1 = plt.figure()
plt.plot(x, y, color='black')
plt.plot(x, y_2, color='yellow')
plt.plot(x, y, color="black")
plt.plot(x, y_2, color="yellow")
# 2. figure
FIG2 = plt.figure()
plt.plot(x, y, color='blue')
plt.plot(x, y_2, color='red')
plt.plot(x, y, color="blue")
plt.plot(x, y_2, color="red")
# %% tagplot
......@@ -39,13 +39,13 @@ plt.plot(x, y_2, color='red')
FIGS_AS_LIST = [FIG1, FIG2]
# Example for how to use tagplot with matplotlib figures
FIGS_AND_IDS = tagplot(FIGS_AS_LIST, 'matplotlib', location='west',
id_method='random', prefix=PROJECT_ID)
FIGS_AND_IDS = tagplot(
FIGS_AS_LIST, "matplotlib", location="west", id_method="random", prefix=PROJECT_ID
)
# Required arguments: tagplot(images as list, desired plot engine)
# %% Publish
publish(FIGS_AND_IDS, ['../README.md', '../docs', '../LICENSE'],
'data', 'my_plot')
publish(FIGS_AND_IDS, ["../README.md", "../docs", "../LICENSE"], "data")
# Required arguments: publish(output of tagplot(), list of files,
# path to destination folder, name(s) for the resulting images)
# path to destination folder)
......@@ -10,5 +10,5 @@ research data, the plot is based on. Additionally, the script that created the
plot will also be copied to the directory.
"""
__version__ = '0.2.2'
__author__ = 'Institut Fluidsystemtechnik within nfdi4ing at TU Darmstadt'
__version__ = "0.2.2"
__author__ = "Institut Fluidsystemtechnik within nfdi4ing at TU Darmstadt"
......@@ -27,19 +27,20 @@ def create_id(id_method):
figure_id : str
"""
match id_method:
case 'time':
case "time":
figure_id = time.time() # UNIX Time
figure_id = hex(int(figure_id)) # convert time to hexadecimal str
time.sleep(1) # break for avoiding duplicate IDs
case 'random':
time.sleep(1) # break for avoiding duplicate IDs
case "random":
figure_id = str(uuid.uuid4()) # creates a random UUID
figure_id = figure_id[0:8] # only use first 8 numbers
figure_id = figure_id[0:8] # only use first 8 numbers
case _:
raise ValueError(
f'Your chosen ID method "{id_method}" is not supported.\n'
'At the moment these methods are available:\n'
"At the moment these methods are available:\n"
'"time": Unix time converted to hexadecimal\n'
'"random": Random UUID')
'"random": Random UUID'
)
return figure_id
......
......@@ -40,17 +40,20 @@ class PlotOptions:
The default is 'time'.
qrcode : bool, optional
Experimental status. Print qrcode on exported plot. Default: False.
id_on_plot: bool, optional
Print ID on the plot. Default: True.
"""
def __init__(self, figs, rotation, position, **kwargs):
self.figs = figs
self.figure_ids = kwargs.get('figure_ids', [])
self.figure_ids = kwargs.get("figure_ids", [])
self.rotation = rotation
self.position = position
self.prefix = kwargs.get('prefix', '')
self.id_method = kwargs.get('id_method', 'time')
self.qrcode = kwargs.get('qrcode', False)
self.prefix = kwargs.get("prefix", "")
self.id_method = kwargs.get("id_method", "time")
self.qrcode = kwargs.get("qrcode", False)
self.id_on_plot = kwargs.get("id_on_plot", True)
def __str__(self):
"""Representation if an object of this class is printed."""
......@@ -76,7 +79,7 @@ class PlotOptions:
raise TypeError("Prefix is not a string.")
if not isinstance(self.id_method, str):
raise TypeError('The chosen id_method is not a string.')
raise TypeError("The chosen id_method is not a string.")
# Store figs in a list, even if it is only one.
if not isinstance(self.figs, list):
......@@ -145,14 +148,19 @@ def validate_list(list_var, elt_type=str, is_file=False):
if isinstance(list_var, list):
for elt in list_var:
if not isinstance(elt, elt_type):
raise TypeError(f'The list of {list_var} contains an '
f'object which is not of type {elt_type}.')
raise TypeError(
f"The list of {list_var} contains an "
f"object which is not of type {elt_type}."
)
if is_file:
# Check if directory and files exist
if not os.path.exists(elt):
raise FileNotFoundError('The specified directory'
f'/file {elt} does not exist.')
raise FileNotFoundError(
"The specified directory" f"/file {elt} does not exist."
)
else:
raise TypeError(f'The specified {list_var} are neither a '
f'{elt_type} nor a list of {elt_type}.')
raise TypeError(
f"The specified {list_var} are neither a "
f"{elt_type} nor a list of {elt_type}."
)
return list_var
......@@ -33,19 +33,17 @@ class PublishOptions:
Export the plot and copy specified files to the destiantion folder.
"""
def __init__(self, figs_and_ids, src_datapaths, dst_path, plot_names,
**kwargs):
def __init__(self, figs_and_ids, src_datapaths, dst_path, **kwargs):
if not isinstance(figs_and_ids, PlotIDTransfer):
raise RuntimeError('figs_and_ids is not an instance of '
'PlotIDTransfer.')
raise RuntimeError("figs_and_ids is not an instance of PlotIDTransfer.")
self.figure = figs_and_ids.figs
self.figure_ids = figs_and_ids.figure_ids
self.src_datapaths = src_datapaths
self.dst_path = os.path.abspath(dst_path)
self.plot_names = plot_names
self.data_storage = kwargs.get('data_storage', 'individual')
self.data_storage = kwargs.get("data_storage", "individual")
self.dst_path_head, self.dst_dirname = os.path.split(self.dst_path)
self.plot_names = kwargs.get("plot_names", self.figure_ids)
def __str__(self):
"""Representation if an object of this class is printed."""
......@@ -75,23 +73,25 @@ class PublishOptions:
self.plot_names = validate_list(self.plot_names)
if not os.path.isfile(sys.argv[0]):
raise FileNotFoundError('Cannot copy original python script. '
'Running publish from a shell is not '
'possible.')
raise FileNotFoundError(
"Cannot copy original python script. "
"Running publish from a shell is not "
"possible."
)
# Check if self.src_datapaths are strings and existing files.
self.src_datapaths = validate_list(self.src_datapaths,
is_file=True)
self.src_datapaths = validate_list(self.src_datapaths, is_file=True)
# Check if destination directory is allowed path
if not os.path.exists(self.dst_path_head):
raise FileNotFoundError('The specified destination directory '
f'{self.dst_path_head} does not exist.')
raise FileNotFoundError(
"The specified destination directory "
f"{self.dst_path_head} does not exist."
)
# Check if data_storage is a string
if not isinstance(self.data_storage, str):
raise TypeError('The specified data_storage method is not a '
'string.')
raise TypeError("The specified data_storage method is not a string.")
def export(self):
"""
......@@ -112,55 +112,66 @@ class PublishOptions:
# Export plot figure to picture.
plot_paths = save_plot(self.figure, self.plot_names)
match self.data_storage:
case 'centralized':
case "centralized":
self.centralized_data_storage()
case 'individual':
case "individual":
for i, plot in enumerate(plot_paths):
try:
# Create folder with ID as name
dst_path = os.path.join(self.dst_path,
self.figure_ids[i])
dst_path_invisible = os.path.join(self.dst_path, '.'
+ self.figure_ids[i])
dst_path = os.path.join(self.dst_path, self.figure_ids[i])
dst_path_invisible = os.path.join(
self.dst_path, "." + self.figure_ids[i]
)
# If dir with the same ID already exists ask user
# if it should be overwritten.
if os.path.isdir(dst_path):
warnings.warn(f'Folder "{dst_path}" already exists'
' – plot has already been published.'
)
overwrite_dir = input('Do you want to overwrite '
'the existing folder? '
'(yes/no[default])\n')
if overwrite_dir in ('yes', 'y', 'Yes', 'YES'):
warnings.warn(