Skip to content
Snippets Groups Projects
Select Git revision
  • 972df78b66dbcae22cdbe63cd56448a520b77714
  • main default protected
2 results

nfdi-in-emacs-with-orgmode.tex

    • Code owners
    Assign users and groups as approvers for specific file changes. Learn more.
    nfdi-in-emacs-with-orgmode.tex 21.25 KiB 
    % Created 2023-09-15 Fri 13:21
% Intended LaTeX compiler: lualatex
\documentclass[11pt]{article}
\usepackage{graphicx}
\usepackage{longtable}
\usepackage{wrapfig}
\usepackage{rotating}
\usepackage[normalem]{ulem}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{capt-of}
\usepackage{hyperref}
\usepackage{minted}
\usepackage[french, english, american]{babel}
\usepackage[a4paper,top=3cm,bottom=3cm]{geometry}
\usepackage{minimalist}
\usepackage{fontspec}
\usepackage{caption} \captionsetup{labelfont=bf,font={sf,small}}
\setmainfont{TeX Gyre Pagella}
\usepackage{enumitem} \setlist{nosep}
\usepackage{longtable}
\usepackage{microtype}
\AtBeginEnvironment{longtable}{\footnotesize}
\setminted{fontsize=\small,baselinestretch=.75}
\usepackage[backend=biber, style=ext-verbose]{biblatex}
\addbibresource{emacs-references.bib}
\addbibresource{~/.config/doom/bib/RDM-RWTH.bib}
\addbibresource{~/.config/doom/bib/Data-Literacy-RDM.bib}
\addbibresource{~/.config/doom/bib/auf-dem-forum-thesis.bib}
\usepackage[norule,marginal,hang]{footmisc}
\author{Jonathan A. Hartman | Lukas C. Bossert}
\date{\today}
\title{Data Processing, Code Documentation and Beyond \\  (Emacs and org-mode)}
\hypersetup{
 pdfauthor={Jonathan A. Hartman | Lukas C. Bossert},
 pdftitle={Data Processing, Code Documentation and Beyond \\  (Emacs and org-mode)},
 pdfkeywords={},
 pdfsubject={},
 pdfcreator={Emacs 29.1 (Org mode 9.7)}, 
 pdflang={English}}
\begin{document}

\maketitle
\setcounter{tocdepth}{2}
\tableofcontents


\section{Overview}
\label{sec:org761ab24}
This document provides insights into an efficient way handling data.
We show not only how to retrieve data from an publicly accesible webpge but also
how the data can be processed afterwards. We admit that in the examples shown
below we definetly drawing from the full, but we consider this as a proof of
concept for how in our modern technological world plain text is still a great
way of processing and documenting data workflow and analyses.

The paper is divided into three main steps, focussing on first preparing, second
processing and last presevering the data and its documentation (fig. \ref{workflow}).

\begin{figure}[htbp]
\centering
\includegraphics[width=\linewidth]{img/nfdi-in-emacs-best-practice-overview.png}
\caption{\label{workflow}Workflow of the document. Source \href{https://excalidraw.com/\#room=8617c3374a9c2c2c895b,a\_SoKClI-tyAxWfSgzThWQ}{Excalidraw}.}
\end{figure}


\section{Introduce}
\label{sec:org94dfc11}
What is Emacs and \textbf{org-mode}? Well, where to start? You may not have heard of
Emacs or org-mode, yet. Usually it is considered to a tool for geeks, \ldots{}.. this
    might be kind of true, but once you noticed the myriard ways of using
Emacs\autocite{Hahn2016Emacs,Kitchin2016DataSharing,Strobel1996Linux} and
espeically its module org-mode you never ever won’t to use anything else.\footnote{There might be people having a different opinion.}
Emacs has been around for decades (no kidding) and is free software.

Org-mode is quite younger but the killing feature in Emacs. Or let’s express it with
the words of the original creator Carsten Dominik:

\begin{quote}
Org-mode does outlining, note-taking, hyperlinks, spreadsheets, TODO lists,
project planning, GTD, HTML and \LaTeX{} authoring, all with plain text files in
Emacs.
\end{quote}

or in a nutshell:
\begin{quote}
Back to the future for plain text\\[0pt]
(Carsten Dominik)
\end{quote}

Let’s make an executive summary of org-mode:
\begin{itemize}
\item Module for \href{https://emacs.org}{Emacs}
\item Plain text based
\item Around since 2003
\item Meant for (scientific) text production and organisation
\begin{itemize}
\item project management
\item agenda, diary, journaling
\item personal knowledge management
\item presentation
\item single-source-publishing
\item literate programming
\end{itemize}
\item Extensible and fully customizable

Org-mode is a magnificent tool when it comes to reproducible research,\autocite{Stanisic2014}
since this combines a well documented way of analysing a data set.
\end{itemize}


\section{Prepare}
\label{sec:org3b374bd}

For our demonstration, we are going to create a dataset from openly available
data on the German National Research Data Infrastructure (\textbf{NFDI}) and perform
some simple analysis tasks on it.


\subsection{Data retrieval using SPARQL}
\label{sec:org1a79980}
The data we are interested in exists on Wikidata. Wikidata is similar to
Wikipedia, but rather than long form articles, the data is stored as structured
data. This allows machines to easily access and traverse these pages with query
langauges. Here, we are going to submit a \texttt{SPARQL} query to the Wikidata query
endpoint.


SPARQL will look familiar to anyone familar with SQL, however it is slightly
more cryptic at first glance. Take a look at the below query – things like
``Q98270496'' refer to specific items in wikidata, where things like ``P31'' are
more akin to concepts. In English, this query translates to something like

\begin{quote}
Give me the Names for items that has a property (P31) of NFDI Consortia
(Q98270496), and return all items you find on each of those entries under the
property ``affiliations'' (P1416).
\end{quote}

If you like how to do this in more detail, have a look at \autocite{Bossert2023Wikidata}.
    
\begin{listing}[htbp]
\begin{minted}[linenos,firstnumber=1]{sparql}
SELECT ?wLabel ?pLabel
WHERE
{
  ?p wdt:P31 wd:Q98270496 .                                                (consortium)
  ?p wdt:P1416 ?w .                                                        (affiliations)
  SERVICE wikibase:label { bd:serviceParam wikibase:language "en" . }
}
ORDER BY ASC(?wLabel) ASC(?pLabel)
LIMIT 50
\end{minted}
\caption{\label{raw-dataset}Retrieving the dataset from wikidata}
\end{listing}

\begin{longtable}{ll}
\caption{\label{result-raw-dataset}Result of the query for NFDI consortia and their institutions.}
\\[0pt]
wLabel & pLabel\\[0pt]
\hline
\endfirsthead
\multicolumn{2}{l}{Continued from previous page} \\[0pt]
\hline

wLabel & pLabel \\[0pt]

\hline
\endhead
\hline\multicolumn{2}{r}{Continued on next page} \\
\endfoot
\endlastfoot
\hline
Q105775472 & NFDI4Health\\[0pt]
Q1117007 & NFDI4Health\\[0pt]
Q115254989 & NFDI4Objects\\[0pt]
Q1205424 & NFDI4Objects\\[0pt]
Q17575706 & NFDI4Objects\\[0pt]
Academy of Sciences and Humanities in Hamburg & Text+\\[0pt]
Academy of Sciences and Literature Mainz & NFDI4Culture\\[0pt]
Academy of Sciences and Literature Mainz & NFDI4Memory\\[0pt]
Academy of Sciences and Literature Mainz & NFDI4Objects\\[0pt]
Academy of Sciences and Literature Mainz & Text+\\[0pt]
Alfred Wegener Institute for Polar and Marine Research & NFDI4Biodiversity\\[0pt]
Alfred Wegener Institute for Polar and Marine Research & NFDI4DataScience\\[0pt]
Alfred Wegener Institute for Polar and Marine Research & NFDI4Earth\\[0pt]
Anthropological Society (Munich) & NFDI4Objects\\[0pt]
Arachnologische Gesellschaft & NFDI4Biodiversity\\[0pt]
Arbeitskreis Provenienzforschung e.V. & NFDI4Memory\\[0pt]
Archivschule Marburg & NFDI4Memory\\[0pt]
Archäologische Kommission für Niedersachsen & NFDI4Objects\\[0pt]
Archäologisches Museum Hamburg und Stadtmuseum Harburg & NFDI4Objects\\[0pt]
Arthistoricum & NFDI4Culture\\[0pt]
Association for Data-Intensive Radio Astronomy & PUNCH4NFDI\\[0pt]
Association for Technology and Construction in Agriculture & FAIRAgro\\[0pt]
Association of German Architects & NFDI4Culture\\[0pt]
Association of Population Based Cancer Registries in Germany & NFDI4Health\\[0pt]
Association of states archaeologists & NFDI4Objects\\[0pt]
BERD@NFDI & Base4NFDI\\[0pt]
Bach-Archiv Leipzig & NFDI4Culture\\[0pt]
Bauhaus-Universität Weimar & NFDI4Ing\\[0pt]
Bavarian Academy of Sciences and Humanities & BERD@NFDI\\[0pt]
Bavarian Academy of Sciences and Humanities & NFDI4Earth\\[0pt]
Bavarian Academy of Sciences and Humanities & NFDI4Memory\\[0pt]
Bavarian Academy of Sciences and Humanities & NFDI4Objects\\[0pt]
Bavarian Academy of Sciences and Humanities & NFDIxCS\\[0pt]
Bavarian Academy of Sciences and Humanities & PUNCH4NFDI\\[0pt]
Bavarian Academy of Sciences and Humanities & Text+\\[0pt]
Bavarian Forest National Park & NFDI4Biodiversity\\[0pt]
Bavarian Natural History Collections & NFDI4Biodiversity\\[0pt]
    Bavarian Natural History Collections & NFDI4Objects\\[0pt]
Bavarian State Archaeological Collection & NFDI4Objects\\[0pt]
Bavarian State Archives & FAIRAgro\\[0pt]
Bavarian State Archives & NFDI4Biodiversity\\[0pt]
Bavarian State Archives & NFDI4Earth\\[0pt]
Bavarian State Archives & NFDI4Objects\\[0pt]
Bavarian State Library & NFDI4Culture\\[0pt]
Bavarian State Library & NFDI4Memory\\[0pt]
Bavarian State Research Center for Agriculture & FAIRAgro\\[0pt]
Beethoven House & NFDI4Culture\\[0pt]
Beilstein Institute for the Advancement of Chemical Sciences & NFDI4Chem\\[0pt]
Berlin State Library & Base4NFDI\\[0pt]
Berlin State Library & NFDI4Memory\\[0pt]
\end{longtable}




\subsection{Data cleaning using shell}
\label{sec:org56020cc}
The data we got from listing \ref{raw-dataset} is good but it needs further cleaning.

We can see several entries in our data that look like ``Q1234567'' - These are Q
Ids for items which no label has been defined. Let’s remove those from our
dataset.

We’re going to include the output from the previous cell, where we executed the
SPARQL query, as an input variable to this cell (\texttt{:var input=raw-dataset}).

\begin{listing}[htbp]
\begin{minted}[]{sh}
echo "$input" | sed -E '/Q[0-9]+/d'
\end{minted}
\caption{\label{clean-dataset}Cleaning the raw data using good old \texttt{sed} and a regex pattern.}
\end{listing}

\begin{longtable}{ll}
\caption{\label{result-clean-dataset}Cleaned data set which will be used for ruther processing.}
\\[0pt]
wLabel & pLabel\\[0pt]
\hline
\endfirsthead
\multicolumn{2}{l}{Continued from previous page} \\[0pt]
\hline

wLabel & pLabel \\[0pt]

\hline
\endhead
\hline\multicolumn{2}{r}{Continued on next page} \\
\endfoot
\endlastfoot
\hline
Academy of Sciences and Humanities in Hamburg & Text+\\[0pt]
Academy of Sciences and Literature Mainz & NFDI4Culture\\[0pt]
Academy of Sciences and Literature Mainz & NFDI4Memory\\[0pt]
Academy of Sciences and Literature Mainz & NFDI4Objects\\[0pt]
Academy of Sciences and Literature Mainz & Text+\\[0pt]
Alfred Wegener Institute for Polar and Marine Research & NFDI4Biodiversity\\[0pt]
Alfred Wegener Institute for Polar and Marine Research & NFDI4DataScience\\[0pt]
Alfred Wegener Institute for Polar and Marine Research & NFDI4Earth\\[0pt]
Anthropological Society (Munich) & NFDI4Objects\\[0pt]
Arachnologische Gesellschaft & NFDI4Biodiversity\\[0pt]
Arbeitskreis Provenienzforschung e.V. & NFDI4Memory\\[0pt]
Archivschule Marburg & NFDI4Memory\\[0pt]
Archäologische Kommission für Niedersachsen & NFDI4Objects\\[0pt]
Archäologisches Museum Hamburg und Stadtmuseum Harburg & NFDI4Objects\\[0pt]
Arthistoricum & NFDI4Culture\\[0pt]
Association for Data-Intensive Radio Astronomy & PUNCH4NFDI\\[0pt]
Association for Technology and Construction in Agriculture & FAIRAgro\\[0pt]
    Association of German Architects & NFDI4Culture\\[0pt]
Association of Population Based Cancer Registries in Germany & NFDI4Health\\[0pt]
Association of states archaeologists & NFDI4Objects\\[0pt]
BERD@NFDI & Base4NFDI\\[0pt]
Bach-Archiv Leipzig & NFDI4Culture\\[0pt]
Bauhaus-Universität Weimar & NFDI4Ing\\[0pt]
Bavarian Academy of Sciences and Humanities & BERD@NFDI\\[0pt]
Bavarian Academy of Sciences and Humanities & NFDI4Earth\\[0pt]
Bavarian Academy of Sciences and Humanities & NFDI4Memory\\[0pt]
Bavarian Academy of Sciences and Humanities & NFDI4Objects\\[0pt]
Bavarian Academy of Sciences and Humanities & NFDIxCS\\[0pt]
Bavarian Academy of Sciences and Humanities & PUNCH4NFDI\\[0pt]
Bavarian Academy of Sciences and Humanities & Text+\\[0pt]
Bavarian Forest National Park & NFDI4Biodiversity\\[0pt]
Bavarian Natural History Collections & NFDI4Biodiversity\\[0pt]
Bavarian Natural History Collections & NFDI4Objects\\[0pt]
Bavarian State Archaeological Collection & NFDI4Objects\\[0pt]
Bavarian State Archives & FAIRAgro\\[0pt]
Bavarian State Archives & NFDI4Biodiversity\\[0pt]
Bavarian State Archives & NFDI4Earth\\[0pt]
Bavarian State Archives & NFDI4Objects\\[0pt]
Bavarian State Library & NFDI4Culture\\[0pt]
Bavarian State Library & NFDI4Memory\\[0pt]
Bavarian State Research Center for Agriculture & FAIRAgro\\[0pt]
Beethoven House & NFDI4Culture\\[0pt]
Beilstein Institute for the Advancement of Chemical Sciences & NFDI4Chem\\[0pt]
Berlin State Library & Base4NFDI\\[0pt]
Berlin State Library & NFDI4Memory\\[0pt]
\end{longtable}

\section{Process}
\label{sec:org4470386}

\subsection{Data Aggregation with Python}
\label{sec:org0e78222}

The great thing about org mode is that we can seamlessly switch between
languages! Our original query (listing \ref{raw-dataset}) was written in SPARQL, which returned
a kind of table (tab. \ref{result-raw-dataset}). We then took that table and ran a shell command on it. Now,
we’re going to take the output of that shell command (cf. tab.
\ref{result-clean-dataset}) and run
some python code on it.


\begin{minted}[]{sh}
python -m pip install pandas --user
\end{minted}

\begin{listing}[htbp]
\begin{minted}[linenos,firstnumber=1]{python}
import pandas as pd

# The data comes into the cell as a list of lists.
# We can pick it apart into a DataFrame object
df = pd.DataFrame(clean_df[1:], columns=clean_df[0])

# Perform a groupby operation on wLabel and
# rename the resulting new column "Count"
institutions_by_consortia = (
    df
    .groupby("wLabel")
    .size()
    .reset_index(name="Count"))

# Return our dataframe in a way that org will
# display it as an org table
return [list(institutions_by_consortia.columns),
        None, *map(list, institutions_by_consortia.values)]
\end{minted}
\caption{\label{python-aggregation}Counting the number of consortia involved in one institution.}
    \end{listing}

\begin{longtable}{lr}
\caption{\label{result-python-aggregation}Overview of institutions and the count of their associated consortia.}
\\[0pt]
wLabel & Count\\[0pt]
\hline
\endfirsthead
\multicolumn{2}{l}{Continued from previous page} \\[0pt]
\hline

wLabel & Count \\[0pt]

\hline
\endhead
\hline\multicolumn{2}{r}{Continued on next page} \\
\endfoot
\endlastfoot
\hline
Academy of Sciences and Humanities in Hamburg & 1\\[0pt]
Academy of Sciences and Literature Mainz & 4\\[0pt]
Alfred Wegener Institute for Polar and Marine Research & 3\\[0pt]
Anthropological Society (Munich) & 1\\[0pt]
Arachnologische Gesellschaft & 1\\[0pt]
Arbeitskreis Provenienzforschung e.V. & 1\\[0pt]
Archivschule Marburg & 1\\[0pt]
Archäologische Kommission für Niedersachsen & 1\\[0pt]
Archäologisches Museum Hamburg und Stadtmuseum Harburg & 1\\[0pt]
Arthistoricum & 1\\[0pt]
Association for Data-Intensive Radio Astronomy & 1\\[0pt]
Association for Technology and Construction in Agriculture & 1\\[0pt]
Association of German Architects & 1\\[0pt]
Association of Population Based Cancer Registries in Germany & 1\\[0pt]
Association of states archaeologists & 1\\[0pt]
BERD@NFDI & 1\\[0pt]
Bach-Archiv Leipzig & 1\\[0pt]
Bauhaus-Universität Weimar & 1\\[0pt]
Bavarian Academy of Sciences and Humanities & 7\\[0pt]
Bavarian Forest National Park & 1\\[0pt]
Bavarian Natural History Collections & 2\\[0pt]
Bavarian State Archaeological Collection & 1\\[0pt]
Bavarian State Archives & 4\\[0pt]
Bavarian State Library & 2\\[0pt]
Bavarian State Research Center for Agriculture & 1\\[0pt]
Beethoven House & 1\\[0pt]
Beilstein Institute for the Advancement of Chemical Sciences & 1\\[0pt]
Berlin State Library & 2\\[0pt]
\end{longtable}

\subsection{Counting Elements with awk}
\label{sec:orgb1e796e}

We’re not limited to python though. Here we’re going to perform a very similar
aggregation, but grouping by consortia to get the number of institutes at each.
Like the listing  \ref{python-aggregation} above, we are going to use the output of
listing \ref{clean-dataset} (cf. tab.
\ref{result-clean-dataset})  to perform this operation.
Instead of python, we’re going to use \texttt{awk} for our data processing.


As an additional bonus, we’re going to paramaterize this cell by defining a
variable called \texttt{consortium}. With this we could reuse the code in this cell
over and over, changing the desired consortium name to show only the desired
results.


\begin{listing}[htbp]
\begin{minted}[linenos,firstnumber=1]{awk}
BEGIN {
# before the evaluating process of the data begins
    # this block is taken in account
# set the separator to tab
  FS =  "\t"
}
  # MAIN section of the evaluating process
  #----------------------------------------
  # while going through the rows of the input
  # check only for the second column
  # step a counter for equal values and store it in 'counts'
  $2 == consortium { ++counts[$2] }
END {
  # final part where no evaluation is done anymore
  # only collecting and printing results
  # going through the counts from above
  for (k in counts)
  # check for the amount of associated institutions
      if (counts[k] == 1)
  # if only one institution, then use the singular version
         print consortium " (" counts[k]  " institution)";
  # otherwise we need the plural form.
     else print consortium " (" counts[k]  " institutions)"
}
\end{minted}
\caption{\label{institutions-count}Calculating the number of involved institions in one specific consortium.}
\end{listing}

Having created the source block we can also use it in our text with executing
the the function \texttt{call\_institutions-count('NFDI4Objects')}. The
result will be blended in smoothly in the text and if there are any changes to
the initial data set updated automatically.

Back to our example: So, now we know of many institutions are involved in
NFDI4Objects (9 institutions)
or in
NFDI4Earth (3 institutions).



\subsection{Network Disply with R}
\label{sec:org9b7cffc}

How about something a little more visual than some tables? We can also create
plots and visuals, generating them with the code contained in the document and
embedding the results in the output.


And while we’re at it, how about another language? This time we’ll use R to make
a simple network plot of our data. Again, we’re still using the output from
listiing \ref{clean-dataset} (which is tab.
\ref{result-clean-dataset}) to do this.

The result is a nice visualization of a network (fig. \ref{result-nfdi-network}).
Such a visualization can help to detect outliers faster.



\begin{listing}[htbp]
\begin{minted}[linenos,firstnumber=1]{r}
# making sure the required package is installed
if (!require("igraph")) install.packages("igraph")
library("igraph")
# making a more robust outcome by stating a seed number
set.seed(123456789)
# convert the tabular data into a data frame which is required
# for creating a network
NFDI_network <- graph_from_data_frame(NFDI_edges,
                                      directed = FALSE)
plot(NFDI_network,                   # loading data frame
    main  = "NFDI Network",          # adding a title
    # adding a color to all nodes from the second column.
        vertex.color = c("blue", "red")#
            [1 + names(V(NFDI_network)) %in% NFDI_edges[,2]],
    vertex.size        = 4,         # size of the node
    vertex.frame.color = NA,        # no frame for nodes
    vertex.label       = NA,        # no color of the description
    edge.curved        = 0.2,       # factor of "curvity"
    )
\end{minted}
\caption{\label{nfdi-network}Network of all institutions and their related consortia.}
\end{listing}

\begin{figure}[htbp]
\centering
\includegraphics[width=.5\linewidth]{img/nfdi-network.png}
\caption{\label{result-nfdi-network}Network of NFDI consortia (red) and institutions (blue).}
\end{figure}



\section{Preserve}
\label{sec:org8435f75}


There are two ways exporting this document in multiple documents.
The concept of this is called ``single-source-publishing''.
This means we have on document, our org-file, and we will export it into
different formats, which are more suitable for different occasions.


\subsection{Manual export}
\label{sec:org91bc9ff}
The common approach is to invoke the commands for exporting into a certain
format individually and by hand.
Org-mode has a great build in exporting mechanism which converts the document
into all mainly used formats. You get to the menue by calling \texttt{SPC m e} or \texttt{C-c
C-e} and then select which export format you would like to have.


In tab. \ref{export-options} you find a quick overview of some basic formats.

\begin{table}[htbp]
\caption{\label{export-options}Overview of various individual export functions.}
\centering
\begin{tabular}{lll}
\hline
 & evil & normal\\[0pt]
\hline
PDF & \texttt{SPC m e l o} & \texttt{C-c C-e l o}\\[0pt]
HTML & \texttt{SPC m e h o} & \texttt{C-c C-e h o}\\[0pt]
ASCII & \texttt{SPC m e t a} & \texttt{C-c C-e t a}\\[0pt]
\hline
\end{tabular}
\end{table}



\subsection{Automatic batch process}
\label{sec:org4b60c04}
In a batch process the file is opened with a clean and neutral version of emacs
and will be exported (see listing \ref{batch-export}).

\begin{listing}[htbp]
\begin{minted}[linenos,firstnumber=1]{common-lisp}
(let ((org-file (find-file-noselect filename)))
  (with-current-buffer org-file
    (org-html-export-to-html)
    (message "HTML export successful.")
    )
  (with-current-buffer org-file
    (org-ascii-export-to-ascii)
        (message "ASCII export successful.")
    )
  (with-current-buffer org-file
    (org-latex-export-to-pdf)
    (message "PDF export successful.")
    ))
\end{minted}
\caption{\label{batch-export}Exporting file into various formats}
\end{listing}

\printbibliography
\end{document}