Skip to content
Snippets Groups Projects
Commit 1b0782b0 authored by Lukas C. Bossert's avatar Lukas C. Bossert
Browse files

reduce aggregate table example

parent 1473f25a
Branches
No related tags found
No related merge requests found
Show whitespace changes
Inline Side-by-side
nfdi-in-emacs-with-orgmode.html
+ 83
− 260
View file @ 1b0782b0
......@@ -3,7 +3,7 @@
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
<head>
<!-- 2023-09-17 Sun 00:06 -->
<!-- 2023-09-19 Tue 08:23 -->
<meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<title>Data Processing, Code Documentation and Beyond @@latex:\\@@ @@html:&lt;br&gt;@@ (Emacs and org-mode)</title>
......@@ -24,33 +24,32 @@
<h2>Table of Contents</h2>
<div id="text-table-of-contents" role="doc-toc">
<ul>
<li><a href="#orge8fbcab">1. Overview</a></li>
<li><a href="#orgd7eeaf8">2. Introduce</a></li>
<li><a href="#org6d162bd">3. Prepare</a>
<li><a href="#orgde4d7b5">1. Overview</a></li>
<li><a href="#org10d5fad">2. Introduce</a></li>
<li><a href="#orgd02aeb1">3. Prepare</a>
<ul>
<li><a href="#org3e7a80a">3.1. Data retrieval using SPARQL</a></li>
<li><a href="#org23a687d">3.2. Data cleaning using shell</a></li>
<li><a href="#org8b2ad7a">3.1. Data retrieval using SPARQL</a></li>
<li><a href="#org456dfd2">3.2. Data cleaning using shell</a></li>
</ul>
</li>
<li><a href="#org4b6e209">4. Process</a>
<li><a href="#orga472d58">4. Process</a>
<ul>
<li><a href="#orge3daa90">4.1. Data Aggregation with Python</a></li>
<li><a href="#org40afcd3">4.2. Counting Elements with awk</a></li>
<li><a href="#org56ee722">4.3. Network Disply with R</a></li>
<li><a href="#org73867f1">4.1. Data Aggregation with Python</a></li>
<li><a href="#org932557b">4.2. Counting Elements with awk</a></li>
<li><a href="#orga213ff1">4.3. Network Disply with R</a></li>
</ul>
</li>
<li><a href="#orge20e0aa">5. Preserve</a>
<li><a href="#orgc41a15f">5. Preserve</a>
<ul>
<li><a href="#org89ae948">5.1. Manual export</a></li>
<li><a href="#orgccb6683">5.2. Automatic batch process</a></li>
<li><a href="#org1b0da9d">5.1. Manual export</a></li>
<li><a href="#org75ba140">5.2. Automatic batch process</a></li>
</ul>
</li>
</ul>
</div>
</div>
<div id="outline-container-orge8fbcab" class="outline-2">
<h2 id="orge8fbcab"><span class="section-number-2">1.</span> Overview</h2>
<div id="outline-container-orgde4d7b5" class="outline-2">
<h2 id="orgde4d7b5"><span class="section-number-2">1.</span> Overview</h2>
<div class="outline-text-2" id="text-1">
<p>
This document provides insights into an efficient way handling data.
......@@ -63,21 +62,19 @@ way of processing and documenting data workflow and analyses.
<p>
The paper is divided into three main steps, focussing on first preparing, second
processing and last presevering the data and its documentation (fig. <a href="#org2abe7ec">1</a>).
processing and last presevering the data and its documentation (fig. <a href="#org53cdeb3">1</a>).
</p>
<div id="org2abe7ec" class="figure">
<div id="org53cdeb3" class="figure">
<p><img src="img/nfdi-in-emacs-best-practice-overview.png" alt="nfdi-in-emacs-best-practice-overview.png" width="100%" />
</p>
<p><span class="figure-number">Figure 1: </span>Workflow of the document. Source <a href="https://excalidraw.com/#room=8617c3374a9c2c2c895b,a_SoKClI-tyAxWfSgzThWQ">Excalidraw</a>.</p>
</div>
</div>
</div>
<div id="outline-container-orgd7eeaf8" class="outline-2">
<h2 id="orgd7eeaf8"><span class="section-number-2">2.</span> Introduce</h2>
<div id="outline-container-org10d5fad" class="outline-2">
<h2 id="org10d5fad"><span class="section-number-2">2.</span> Introduce</h2>
<div class="outline-text-2" id="text-2">
<p>
What is Emacs and <b>org-mode</b>? Well, where to start? You may not have heard of
......@@ -138,10 +135,8 @@ since this combines a well documented way of analysing a data set.
</ul>
</div>
</div>
<div id="outline-container-org6d162bd" class="outline-2">
<h2 id="org6d162bd"><span class="section-number-2">3.</span> Prepare</h2>
<div id="outline-container-orgd02aeb1" class="outline-2">
<h2 id="orgd02aeb1"><span class="section-number-2">3.</span> Prepare</h2>
<div class="outline-text-2" id="text-3">
<p>
For our demonstration, we are going to create a dataset from openly available
......@@ -149,10 +144,8 @@ data on the German National Research Data Infrastructure (<b>NFDI</b>) and perfo
some simple analysis tasks on it.
</p>
</div>
<div id="outline-container-org3e7a80a" class="outline-3">
<h3 id="org3e7a80a"><span class="section-number-3">3.1.</span> Data retrieval using SPARQL</h3>
<div id="outline-container-org8b2ad7a" class="outline-3">
<h3 id="org8b2ad7a"><span class="section-number-3">3.1.</span> Data retrieval using SPARQL</h3>
<div class="outline-text-3" id="text-3-1">
<p>
The data we are interested in exists on Wikidata. Wikidata is similar to
......@@ -183,7 +176,7 @@ If you like how to do this in more detail, have a look at (<a href="#citeproc_bi
</p>
<div class="org-src-container">
<label class="org-src-name"><span class="listing-number">Listing 1: </span>Retrieving the dataset from wikidata</label><pre class="src src-sparql" id="org20d5b71"><span class="linenr">1: </span><span class="org-keyword">SELECT</span> <span class="org-variable-name">?wLabel</span> <span class="org-variable-name">?pLabel</span>
<label class="org-src-name"><span class="listing-number">Listing 1: </span>Retrieving the dataset from wikidata</label><pre class="src src-sparql" id="org1eeccfa"><span class="linenr">1: </span><span class="org-keyword">SELECT</span> <span class="org-variable-name">?wLabel</span> <span class="org-variable-name">?pLabel</span>
<span class="linenr">2: </span><span class="org-keyword">WHERE</span>
<span class="linenr">3: </span>{
<span id="coderef-consortium" class="coderef-off"><span class="linenr">4: </span> <span class="org-variable-name">?p</span> wdt:P31 wd:Q98270496 . (consortium)</span>
......@@ -195,7 +188,7 @@ If you like how to do this in more detail, have a look at (<a href="#citeproc_bi
</pre>
</div>
<table id="org5079e0a" border="2" cellspacing="0" cellpadding="6" rules="groups" frame="hsides">
<table id="orge0b2468" border="2" cellspacing="0" cellpadding="6" rules="groups" frame="hsides">
<caption class="t-above"><span class="table-number">Table 1:</span> Result of the query for NFDI consortia and their institutions.</caption>
<colgroup>
......@@ -463,15 +456,11 @@ If you like how to do this in more detail, have a look at (<a href="#citeproc_bi
</table>
</div>
</div>
<div id="outline-container-org23a687d" class="outline-3">
<h3 id="org23a687d"><span class="section-number-3">3.2.</span> Data cleaning using shell</h3>
<div id="outline-container-org456dfd2" class="outline-3">
<h3 id="org456dfd2"><span class="section-number-3">3.2.</span> Data cleaning using shell</h3>
<div class="outline-text-3" id="text-3-2">
<p>
The data we got from listing <a href="#org20d5b71">1</a> is good but it needs further cleaning.
The data we got from listing <a href="#org1eeccfa">1</a> is good but it needs further cleaning.
</p>
<p>
......@@ -486,11 +475,11 @@ SPARQL query, as an input variable to this cell (<code>:var input=raw-dataset</c
</p>
<div class="org-src-container">
<label class="org-src-name"><span class="listing-number">Listing 2: </span>Cleaning the raw data using good old <code>sed</code> and a regex pattern.</label><pre class="src src-sh" id="org6353068"><span class="org-type"><span class="org-tree-sitter-hl-faceXfunctionXcall"><span class="org-tree-sitter-hl-faceXfunctionXbuiltin">echo</span></span></span> <span class="org-tree-sitter-hl-faceXstring">"</span><span class="org-tree-sitter-hl-faceXstring"><span class="org-tree-sitter-hl-faceXpunctuationXspecial"><span class="org-constant">$</span></span></span><span class="org-tree-sitter-hl-faceXstring"><span class="org-tree-sitter-hl-faceXproperty"><span class="org-variable-name">input</span></span></span><span class="org-tree-sitter-hl-faceXstring">"</span> <span class="org-tree-sitter-hl-faceXoperator">|</span> <span class="org-tree-sitter-hl-faceXfunctionXcall">sed</span> <span class="org-tree-sitter-hl-faceXconstant">-E</span> <span class="org-tree-sitter-hl-faceXstring">'/Q[0-9]+/d'</span>
<label class="org-src-name"><span class="listing-number">Listing 2: </span>Cleaning the raw data using good old <code>sed</code> and a regex pattern.</label><pre class="src src-sh" id="org152bf6e"><span class="org-type">echo</span> <span class="org-string">"</span><span class="org-string"><span class="org-constant">$</span></span><span class="org-string"><span class="org-variable-name">input</span></span><span class="org-string">"</span> | sed -E <span class="org-string">'/Q[0-9]+/d'</span>
</pre>
</div>
<table id="org0251484" border="2" cellspacing="0" cellpadding="6" rules="groups" frame="hsides">
<table id="orgfdb797e" border="2" cellspacing="0" cellpadding="6" rules="groups" frame="hsides">
<caption class="t-above"><span class="table-number">Table 2:</span> Cleaned data set which will be used for ruther processing.</caption>
<colgroup>
......@@ -734,53 +723,52 @@ SPARQL query, as an input variable to this cell (<code>:var input=raw-dataset</c
</div>
</div>
</div>
<div id="outline-container-org4b6e209" class="outline-2">
<h2 id="org4b6e209"><span class="section-number-2">4.</span> Process</h2>
<div id="outline-container-orga472d58" class="outline-2">
<h2 id="orga472d58"><span class="section-number-2">4.</span> Process</h2>
<div class="outline-text-2" id="text-4">
</div>
<div id="outline-container-orge3daa90" class="outline-3">
<h3 id="orge3daa90"><span class="section-number-3">4.1.</span> Data Aggregation with Python</h3>
<div id="outline-container-org73867f1" class="outline-3">
<h3 id="org73867f1"><span class="section-number-3">4.1.</span> Data Aggregation with Python</h3>
<div class="outline-text-3" id="text-4-1">
<p>
The great thing about org mode is that we can seamlessly switch between
languages! Our original query (listing <a href="#org20d5b71">1</a>) was written in SPARQL, which returned
a kind of table (tab. <a href="#org5079e0a">1</a>). We then took that table and ran a shell command on it. Now,
languages! Our original query (listing <a href="#org1eeccfa">1</a>) was written in SPARQL, which returned
a kind of table (tab. <a href="#orge0b2468">1</a>). We then took that table and ran a shell command on it. Now,
we’re going to take the output of that shell command (cf. tab.
<a href="#org0251484">2</a>) and run
<a href="#orgfdb797e">2</a>) and run
some python code on it.
</p>
<div class="org-src-container">
<pre class="src src-sh"><span class="org-tree-sitter-hl-faceXfunctionXcall">python</span> <span class="org-tree-sitter-hl-faceXconstant">-m</span> pip install pandas <span class="org-tree-sitter-hl-faceXconstant">--user</span>
<pre class="src src-sh">python -m pip install pandas --user
</pre>
</div>
<div class="org-src-container">
<label class="org-src-name"><span class="listing-number">Listing 3: </span>Counting the number of consortia involved in one institution.</label><pre class="src src-python" id="org4c8d7fe"><span class="linenr"> 1: </span><span class="org-tree-sitter-hl-faceXkeyword">import</span> pandas <span class="org-tree-sitter-hl-faceXkeyword">as</span> pd
<label class="org-src-name"><span class="listing-number">Listing 3: </span>Counting the number of consortia involved in one institution.</label><pre class="src src-python" id="org1cc200d"><span class="linenr"> 1: </span><span class="org-keyword">import</span> pandas <span class="org-keyword">as</span> pd
<span class="linenr"> 2: </span>
<span class="linenr"> 3: </span><span class="org-tree-sitter-hl-faceXcomment"># The data comes into the cell as a list of lists.</span>
<span class="linenr"> 4: </span><span class="org-tree-sitter-hl-faceXcomment"># We can pick it apart into a DataFrame object</span>
<span class="linenr"> 5: </span><span class="org-tree-sitter-hl-faceXvariable">df</span> <span class="org-tree-sitter-hl-faceXoperator">=</span> pd.<span class="org-tree-sitter-hl-faceXproperty"><span class="org-tree-sitter-hl-faceXmethodXcall"><span class="org-tree-sitter-hl-faceXconstructor">DataFrame</span></span></span>(clean_df[<span class="org-tree-sitter-hl-faceXnumber">1</span>:], <span class="org-tree-sitter-hl-faceXlabel">columns</span><span class="org-tree-sitter-hl-faceXoperator">=</span>clean_df[<span class="org-tree-sitter-hl-faceXnumber">0</span>])
<span class="linenr"> 3: </span><span class="org-comment-delimiter"># </span><span class="org-comment">The data comes into the cell as a list of lists.</span>
<span class="linenr"> 4: </span><span class="org-comment-delimiter"># </span><span class="org-comment">We can pick it apart into a DataFrame object</span>
<span class="linenr"> 5: </span><span class="org-variable-name">df</span> <span class="org-operator">=</span> pd.DataFrame(clean_df[<span class="org-highlight-numbers-number">1</span>:], columns<span class="org-operator">=</span>clean_df[<span class="org-highlight-numbers-number">0</span>])
<span class="linenr"> 6: </span>
<span class="linenr"> 7: </span><span class="org-tree-sitter-hl-faceXcomment"># Perform a groupby operation on wLabel and</span>
<span class="linenr"> 8: </span><span class="org-tree-sitter-hl-faceXcomment"># rename the resulting new column "Count"</span>
<span class="linenr"> 9: </span><span class="org-tree-sitter-hl-faceXvariable">institutions_by_consortia</span> <span class="org-tree-sitter-hl-faceXoperator">=</span> (
<span class="linenr"> 7: </span><span class="org-comment-delimiter"># </span><span class="org-comment">Perform a groupby operation on wLabel and</span>
<span class="linenr"> 8: </span><span class="org-comment-delimiter"># </span><span class="org-comment">rename the resulting new column "Count"</span>
<span class="linenr"> 9: </span><span class="org-variable-name">institutions_by_consortia</span> <span class="org-operator">=</span> (
<span class="linenr">10: </span> df
<span class="linenr">11: </span> .<span class="org-tree-sitter-hl-faceXproperty"><span class="org-tree-sitter-hl-faceXmethodXcall">groupby</span></span>(<span class="org-tree-sitter-hl-faceXstring">"wLabel"</span>)
<span class="linenr">12: </span> .<span class="org-tree-sitter-hl-faceXproperty"><span class="org-tree-sitter-hl-faceXmethodXcall">size</span></span>()
<span class="linenr">13: </span> .<span class="org-tree-sitter-hl-faceXproperty"><span class="org-tree-sitter-hl-faceXmethodXcall">sort_values</span></span>(<span class="org-tree-sitter-hl-faceXlabel">ascending</span><span class="org-tree-sitter-hl-faceXoperator">=</span><span class="org-tree-sitter-hl-faceXconstantXbuiltin">False</span>)
<span class="linenr">14: </span> .<span class="org-tree-sitter-hl-faceXproperty"><span class="org-tree-sitter-hl-faceXmethodXcall">reset_index</span></span>(<span class="org-tree-sitter-hl-faceXlabel">name</span><span class="org-tree-sitter-hl-faceXoperator">=</span><span class="org-tree-sitter-hl-faceXstring">"Count"</span>))
<span class="linenr">11: </span> .groupby(<span class="org-string">"wLabel"</span>)
<span class="linenr">12: </span> .size()
<span class="linenr">13: </span> .sort_values(ascending<span class="org-operator">=</span><span class="org-constant">False</span>)
<span class="linenr">14: </span> .reset_index(name<span class="org-operator">=</span><span class="org-string">"Count"</span>))
<span class="linenr">15: </span>
<span class="linenr">16: </span><span class="org-tree-sitter-hl-faceXcomment"># Return our dataframe in a way that org will</span>
<span class="linenr">17: </span><span class="org-tree-sitter-hl-faceXcomment"># display it as an org table</span>
<span class="linenr">18: </span><span class="org-tree-sitter-hl-faceXkeyword">return</span> [<span class="org-tree-sitter-hl-faceXfunctionXcall"><span class="org-tree-sitter-hl-faceXfunctionXbuiltin">list</span></span>(institutions_by_consortia.<span class="org-tree-sitter-hl-faceXproperty">columns</span>),
<span class="linenr">19: </span> <span class="org-tree-sitter-hl-faceXconstantXbuiltin">None</span>, <span class="org-tree-sitter-hl-faceXoperator">*</span><span class="org-tree-sitter-hl-faceXfunctionXcall"><span class="org-tree-sitter-hl-faceXfunctionXbuiltin">map</span></span>(list, institutions_by_consortia.<span class="org-tree-sitter-hl-faceXproperty">values</span>)]
<span class="linenr">16: </span><span class="org-comment-delimiter"># </span><span class="org-comment">Return our dataframe in a way that org will</span>
<span class="linenr">17: </span><span class="org-comment-delimiter"># </span><span class="org-comment">display it as an org table</span>
<span class="linenr">18: </span><span class="org-keyword">return</span> [<span class="org-builtin">list</span>(institutions_by_consortia.columns),
<span class="linenr">19: </span> <span class="org-constant">None</span>, <span class="org-operator">*</span><span class="org-builtin">map</span>(<span class="org-builtin">list</span>, institutions_by_consortia.values)]
</pre>
</div>
<table id="org25d7232" border="2" cellspacing="0" cellpadding="6" rules="groups" frame="hsides">
<table id="orge0ed8f8" border="2" cellspacing="0" cellpadding="6" rules="groups" frame="hsides">
<caption class="t-above"><span class="table-number">Table 3:</span> Overview of institutions and the count of their associated consortia.</caption>
<colgroup>
......@@ -941,177 +929,23 @@ some python code on it.
There is also a &ldquo;native way&rdquo; getting the counting done by using the package
<code>org-aggregate</code><sup><a id="fnr.2" class="footref" href="#fn.2" role="doc-backlink">2</a></sup>.
</p>
<div class="exports" id="orgfba396c">
<p>
</p>
<table border="2" cellspacing="0" cellpadding="6" rules="groups" frame="hsides">
<colgroup>
<col class="org-left" />
<col class="org-right" />
</colgroup>
<thead>
<tr>
<th scope="col" class="org-left">wLabel</th>
<th scope="col" class="org-right">count()</th>
</tr>
</thead>
<tbody>
<tr>
<td class="org-left">Bavarian Academy of Sciences and Humanities</td>
<td class="org-right">7</td>
</tr>
<tr>
<td class="org-left">Academy of Sciences and Literature Mainz</td>
<td class="org-right">4</td>
</tr>
<tr>
<td class="org-left">Bavarian State Archives</td>
<td class="org-right">4</td>
</tr>
<tr>
<td class="org-left">Alfred Wegener Institute for Polar and Marine Research</td>
<td class="org-right">3</td>
</tr>
<tr>
<td class="org-left">Bavarian Natural History Collections</td>
<td class="org-right">2</td>
</tr>
<tr>
<td class="org-left">Bavarian State Library</td>
<td class="org-right">2</td>
</tr>
<tr>
<td class="org-left">Berlin State Library</td>
<td class="org-right">2</td>
</tr>
<tr>
<td class="org-left">Academy of Sciences and Humanities in Hamburg</td>
<td class="org-right">1</td>
</tr>
<tr>
<td class="org-left">Anthropological Society (Munich)</td>
<td class="org-right">1</td>
</tr>
<tr>
<td class="org-left">Arachnologische Gesellschaft</td>
<td class="org-right">1</td>
</tr>
<tr>
<td class="org-left">Arbeitskreis Provenienzforschung e.V.</td>
<td class="org-right">1</td>
</tr>
<tr>
<td class="org-left">Archivschule Marburg</td>
<td class="org-right">1</td>
</tr>
<tr>
<td class="org-left">Archäologische Kommission für Niedersachsen</td>
<td class="org-right">1</td>
</tr>
<tr>
<td class="org-left">Archäologisches Museum Hamburg und Stadtmuseum Harburg</td>
<td class="org-right">1</td>
</tr>
<tr>
<td class="org-left">Arthistoricum</td>
<td class="org-right">1</td>
</tr>
<tr>
<td class="org-left">Association for Data-Intensive Radio Astronomy</td>
<td class="org-right">1</td>
</tr>
<tr>
<td class="org-left">Association for Technology and Construction in Agriculture</td>
<td class="org-right">1</td>
</tr>
<tr>
<td class="org-left">Association of German Architects</td>
<td class="org-right">1</td>
</tr>
<tr>
<td class="org-left">Association of Population Based Cancer Registries in Germany</td>
<td class="org-right">1</td>
</tr>
<tr>
<td class="org-left">Association of states archaeologists</td>
<td class="org-right">1</td>
</tr>
<tr>
<td class="org-left">BERD@NFDI</td>
<td class="org-right">1</td>
</tr>
<tr>
<td class="org-left">Bach-Archiv Leipzig</td>
<td class="org-right">1</td>
</tr>
<tr>
<td class="org-left">Bauhaus-Universität Weimar</td>
<td class="org-right">1</td>
</tr>
<tr>
<td class="org-left">Bavarian Forest National Park</td>
<td class="org-right">1</td>
</tr>
<tr>
<td class="org-left">Bavarian State Archaeological Collection</td>
<td class="org-right">1</td>
</tr>
<tr>
<td class="org-left">Bavarian State Research Center for Agriculture</td>
<td class="org-right">1</td>
</tr>
<tr>
<td class="org-left">Beethoven House</td>
<td class="org-right">1</td>
</tr>
<tr>
<td class="org-left">Beilstein Institute for the Advancement of Chemical Sciences</td>
<td class="org-right">1</td>
</tr>
</tbody>
</table>
</div>
</div>
<div id="outline-container-org40afcd3" class="outline-3">
<h3 id="org40afcd3"><span class="section-number-3">4.2.</span> Counting Elements with awk</h3>
</div>
<div id="outline-container-org932557b" class="outline-3">
<h3 id="org932557b"><span class="section-number-3">4.2.</span> Counting Elements with awk</h3>
<div class="outline-text-3" id="text-4-2">
<p>
We’re not limited to python though. Here we’re going to perform a very similar
aggregation, but grouping by consortia to get the number of institutes at each.
Like the listing <a href="#org4c8d7fe">3</a> above, we are going to use the output of
listing <a href="#org6353068">2</a> (cf. tab.
<a href="#org0251484">2</a>) to perform this operation.
Like the listing <a href="#org1cc200d">3</a> above, we are going to use the output of
listing <a href="#org152bf6e">2</a> (cf. tab.
<a href="#orgfdb797e">2</a>) to perform this operation.
Instead of python, we’re going to use <code>awk</code> for our data processing.
</p>
......@@ -1125,7 +959,7 @@ results.
<div class="org-src-container">
<label class="org-src-name"><span class="listing-number">Listing 4: </span>Calculating the number of involved institions in one specific consortium.</label><pre class="src src-awk" id="orgb0021f3"><span class="linenr"> 1: </span><span class="org-keyword">BEGIN</span> {
<label class="org-src-name"><span class="listing-number">Listing 4: </span>Calculating the number of involved institions in one specific consortium.</label><pre class="src src-awk" id="orgd18dc1e"><span class="linenr"> 1: </span><span class="org-keyword">BEGIN</span> {
<span class="linenr"> 2: </span><span class="org-comment-delimiter"># </span><span class="org-comment">before the evaluating process of the data begins</span>
<span class="linenr"> 3: </span><span class="org-comment-delimiter"># </span><span class="org-comment">this block is taken in account</span>
<span class="linenr"> 4: </span><span class="org-comment-delimiter"># </span><span class="org-comment">set the separator to tab</span>
......@@ -1143,7 +977,7 @@ results.
<span class="linenr">16: </span> <span class="org-comment-delimiter"># </span><span class="org-comment">going through the counts from above</span>
<span class="linenr">17: </span> <span class="org-keyword">for</span> (k <span class="org-keyword">in</span> counts)
<span class="linenr">18: </span> <span class="org-comment-delimiter"># </span><span class="org-comment">check for the amount of associated institutions</span>
<span id="coderef-singular" class="coderef-off"><span class="linenr">19: </span> <span class="org-keyword">if</span> (counts[k] == <span class="org-highlight-numbers-number">1</span>)</span>
<span id="coderef-singular" class="coderef-off"><span class="linenr">19: </span> <span class="org-keyword">if</span> (counts[k] == <span class="org-highlight-numbers-number">1</span>) (singular)</span>
<span class="linenr">20: </span> <span class="org-comment-delimiter"># </span><span class="org-comment">if only one institution, then use the singular version</span>
<span class="linenr">21: </span> <span class="org-preprocessor">print</span> consortium <span class="org-string">" ("</span> counts[k] <span class="org-string">" institution)"</span>;
<span class="linenr">22: </span> <span class="org-comment-delimiter"># </span><span class="org-comment">otherwise we need the plural form.</span>
......@@ -1167,11 +1001,8 @@ NFDI4Earth (3 institutions).
</p>
</div>
</div>
<div id="outline-container-org56ee722" class="outline-3">
<h3 id="org56ee722"><span class="section-number-3">4.3.</span> Network Disply with R</h3>
<div id="outline-container-orga213ff1" class="outline-3">
<h3 id="orga213ff1"><span class="section-number-3">4.3.</span> Network Disply with R</h3>
<div class="outline-text-3" id="text-4-3">
<p>
How about something a little more visual than some tables? We can also create
......@@ -1183,19 +1014,19 @@ embedding the results in the output.
<p>
And while we’re at it, how about another language? This time we’ll use R to make
a simple network plot of our data. Again, we’re still using the output from
listiing <a href="#org6353068">2</a> (which is tab.
<a href="#org0251484">2</a>) to do this.
listiing <a href="#org152bf6e">2</a> (which is tab.
<a href="#orgfdb797e">2</a>) to do this.
</p>
<p>
The result is a nice visualization of a network (fig. <a href="#orgbb3a07d">2</a>).
The result is a nice visualization of a network (fig. <a href="#org995d029">2</a>).
Such a visualization can help to detect outliers faster.
</p>
<div class="org-src-container">
<label class="org-src-name"><span class="listing-number">Listing 5: </span>Network of all institutions and their related consortia.</label><pre class="src src-R" id="org223485e"><span class="linenr"> 1: </span><span class="org-comment-delimiter"># </span><span class="org-comment">making sure the required package is installed</span>
<label class="org-src-name"><span class="listing-number">Listing 5: </span>Network of all institutions and their related consortia.</label><pre class="src src-R" id="org3581a36"><span class="linenr"> 1: </span><span class="org-comment-delimiter"># </span><span class="org-comment">making sure the required package is installed</span>
<span class="linenr"> 2: </span><span class="org-ess-keyword">if</span> (!<span class="org-ess-modifiers">require</span>(<span class="org-string">"igraph"</span>)) install.packages(<span class="org-string">"igraph"</span>)
<span class="linenr"> 3: </span><span class="org-ess-modifiers">library</span>(<span class="org-string">"igraph"</span>)
<span class="linenr"> 4: </span><span class="org-comment-delimiter"># </span><span class="org-comment">making a more robust outcome by stating a seed number</span>
......@@ -1218,7 +1049,7 @@ Such a visualization can help to detect outliers faster.
</div>
<div id="orgbb3a07d" class="figure">
<div id="org995d029" class="figure">
<p><img src="img/nfdi-network.png" alt="nfdi-network.png" width="70%" />
</p>
<p><span class="figure-number">Figure 2: </span>Network of NFDI consortia (red) and institutions (blue).</p>
......@@ -1226,11 +1057,8 @@ Such a visualization can help to detect outliers faster.
</div>
</div>
</div>
<div id="outline-container-orge20e0aa" class="outline-2">
<h2 id="orge20e0aa"><span class="section-number-2">5.</span> Preserve</h2>
<div id="outline-container-orgc41a15f" class="outline-2">
<h2 id="orgc41a15f"><span class="section-number-2">5.</span> Preserve</h2>
<div class="outline-text-2" id="text-5">
<p>
There are two ways exporting this document in multiple documents.
......@@ -1239,10 +1067,8 @@ This means we have on document, our org-file, and we will export it into
different formats, which are more suitable for different occasions.
</p>
</div>
<div id="outline-container-org89ae948" class="outline-3">
<h3 id="org89ae948"><span class="section-number-3">5.1.</span> Manual export</h3>
<div id="outline-container-org1b0da9d" class="outline-3">
<h3 id="org1b0da9d"><span class="section-number-3">5.1.</span> Manual export</h3>
<div class="outline-text-3" id="text-5-1">
<p>
The common approach is to invoke the commands for exporting into a certain
......@@ -1254,10 +1080,10 @@ C-e</code> and then select which export format you would like to have.
<p>
In tab. <a href="#org76c8487">4</a> you find a quick overview of some basic formats.
In tab. <a href="#org9e59508">4</a> you find a quick overview of some basic formats.
</p>
<table id="org76c8487" border="2" cellspacing="0" cellpadding="6" rules="groups" frame="hsides">
<table id="org9e59508" border="2" cellspacing="0" cellpadding="6" rules="groups" frame="hsides">
<caption class="t-above"><span class="table-number">Table 4:</span> Overview of various individual export functions.</caption>
<colgroup>
......@@ -1296,19 +1122,16 @@ In tab. <a href="#org76c8487">4</a> you find a quick overview of some basic form
</table>
</div>
</div>
<div id="outline-container-orgccb6683" class="outline-3">
<h3 id="orgccb6683"><span class="section-number-3">5.2.</span> Automatic batch process</h3>
<div id="outline-container-org75ba140" class="outline-3">
<h3 id="org75ba140"><span class="section-number-3">5.2.</span> Automatic batch process</h3>
<div class="outline-text-3" id="text-5-2">
<p>
In a batch process the file is opened with a clean and neutral version of emacs
and will be exported (see listing <a href="#orge662ffc">6</a>).
and will be exported (see listing <a href="#orgbc80be8">6</a>).
</p>
<div class="org-src-container">
<label class="org-src-name"><span class="listing-number">Listing 6: </span>Exporting file into various formats</label><pre class="src src-emacs-lisp" id="orge662ffc"><span class="linenr"> 1: </span><span class="org-rainbow-delimiters-depth-1">(</span><span class="org-keyword">let</span> <span class="org-rainbow-delimiters-depth-2">(</span><span class="org-rainbow-delimiters-depth-3">(</span>org-file <span class="org-rainbow-delimiters-depth-4">(</span><span class="org-constant">find-file-noselect</span> filename<span class="org-rainbow-delimiters-depth-4">)</span><span class="org-rainbow-delimiters-depth-3">)</span><span class="org-rainbow-delimiters-depth-2">)</span>
<label class="org-src-name"><span class="listing-number">Listing 6: </span>Exporting file into various formats</label><pre class="src src-emacs-lisp" id="orgbc80be8"><span class="linenr"> 1: </span><span class="org-rainbow-delimiters-depth-1">(</span><span class="org-keyword">let</span> <span class="org-rainbow-delimiters-depth-2">(</span><span class="org-rainbow-delimiters-depth-3">(</span>org-file <span class="org-rainbow-delimiters-depth-4">(</span><span class="org-constant">find-file-noselect</span> filename<span class="org-rainbow-delimiters-depth-4">)</span><span class="org-rainbow-delimiters-depth-3">)</span><span class="org-rainbow-delimiters-depth-2">)</span>
<span class="linenr"> 2: </span> <span class="org-rainbow-delimiters-depth-2">(</span><span class="org-keyword">with-current-buffer</span> org-file
<span class="linenr"> 3: </span> <span class="org-rainbow-delimiters-depth-3">(</span><span class="org-function-name">org-html-export-to-html</span><span class="org-rainbow-delimiters-depth-3">)</span>
<span class="linenr"> 4: </span> <span class="org-rainbow-delimiters-depth-3">(</span><span class="org-constant">message</span> <span class="org-string">"HTML export successful."</span><span class="org-rainbow-delimiters-depth-3">)</span>
......@@ -1351,7 +1174,7 @@ There might be people having a different opinion.
</div></div>
<div id="postamble" class="status">
<p class="author">Author: Jonathan A. Hartman | Lukas C. Bossert</p>
<p class="date">Created: 2023-09-17 Sun 00:06</p>
<p class="date">Created: 2023-09-19 Tue 08:23</p>
</div>
</body>
</html>
\ No newline at end of file
nfdi-in-emacs-with-orgmode.org
+ 8
− 37
View file @ 1b0782b0
......@@ -313,41 +313,12 @@ return [list(institutions_by_consortia.columns),
There is also a "native way" getting the counting done by using the package
=org-aggregate=[fn:2].
#+begin_exports org
#+ATTR_LaTeX: :environment longtable
#+BEGIN: aggregate :table "result-clean-dataset" :cols "wLabel count();^A" :exports none
| wLabel | count() |
|--------------------------------------------------------------+---------|
| Bavarian Academy of Sciences and Humanities | 7 |
| Academy of Sciences and Literature Mainz | 4 |
| Bavarian State Archives | 4 |
| Alfred Wegener Institute for Polar and Marine Research | 3 |
| Bavarian Natural History Collections | 2 |
| Bavarian State Library | 2 |
| Berlin State Library | 2 |
| Academy of Sciences and Humanities in Hamburg | 1 |
| Anthropological Society (Munich) | 1 |
| Arachnologische Gesellschaft | 1 |
| Arbeitskreis Provenienzforschung e.V. | 1 |
| Archivschule Marburg | 1 |
| Archäologische Kommission für Niedersachsen | 1 |
| Archäologisches Museum Hamburg und Stadtmuseum Harburg | 1 |
| Arthistoricum | 1 |
| Association for Data-Intensive Radio Astronomy | 1 |
| Association for Technology and Construction in Agriculture | 1 |
| Association of German Architects | 1 |
| Association of Population Based Cancer Registries in Germany | 1 |
| Association of states archaeologists | 1 |
| BERD@NFDI | 1 |
| Bach-Archiv Leipzig | 1 |
| Bauhaus-Universität Weimar | 1 |
| Bavarian Forest National Park | 1 |
| Bavarian State Archaeological Collection | 1 |
| Bavarian State Research Center for Agriculture | 1 |
| Beethoven House | 1 |
| Beilstein Institute for the Advancement of Chemical Sciences | 1 |
#+END:
#+BEGIN: aggregate :table "result-clean-dataset" :cols "wLabel count();^A"
#+END:
#+end_exports
** Counting Elements with awk
......@@ -367,7 +338,7 @@ results.
#+caption: Calculating the number of involved institions in one specific consortium.
#+name: institutions-count
#+begin_src awk -n -r :stdin clean-dataset :var consortium="NFDI4Ing" :results value drawer :exports code
#+begin_src awk -n :stdin clean-dataset :var consortium="NFDI4Memory" :wrap org :exports code
BEGIN {
# before the evaluating process of the data begins
# this block is taken in account
......@@ -395,9 +366,9 @@ END {
#+end_src
#+RESULTS: institutions-count
:results:
NFDI4Ing (1 institution)
:end:
#+begin_org
NFDI4Memory (6 institutions)
#+end_org
Having created the source block we can also use it in our text with executing
the the function =call_institutions-count('NFDI4Objects')=. The
......
nfdi-in-emacs-with-orgmode.pdf
+ 0
− 0
View file @ 1b0782b0
No preview for this file type
nfdi-in-emacs-with-orgmode.tex
+ 18
− 75
View file @ 1b0782b0
% Created 2023-09-17 Sun 00:10
% Created 2023-09-19 Tue 07:23
% Intended LaTeX compiler: lualatex
\documentclass[11pt]{article}
\usepackage{graphicx}
......@@ -11,7 +11,7 @@
\usepackage{capt-of}
\usepackage{hyperref}
\usepackage{minted}
\usepackage[french, english, american]{babel}
\usepackage[french, english]{babel}
\usepackage[a4paper,top=3cm,bottom=3cm]{geometry}
\usepackage{minimalist}
\usepackage{fontspec}
......@@ -36,7 +36,7 @@
pdftitle={Data Processing, Code Documentation and Beyond \\ (Emacs and org-mode)},
pdfkeywords={},
pdfsubject={},
pdfcreator={Emacs 29.1 (Org mode 9.7)},
pdfcreator={Emacs 29.1 (Org mode 9.7–9.7-??-e90a8a69a)},
pdflang={English}}
\begin{document}
......@@ -44,9 +44,8 @@
\setcounter{tocdepth}{2}
\tableofcontents
\section{Overview}
\label{sec:orgb1baff2}
\label{sec:org55072bf}
This document provides insights into an efficient way handling data.
We show not only how to retrieve data from an publicly accesible webpge but also
how the data can be processed afterwards. We admit that in the examples shown
......@@ -62,10 +61,8 @@ processing and last presevering the data and its documentation (fig. \ref{workfl
\includegraphics[width=\linewidth]{img/nfdi-in-emacs-best-practice-overview.png}
\caption{\label{workflow}Workflow of the document. Source \href{https://excalidraw.com/\#room=8617c3374a9c2c2c895b,a\_SoKClI-tyAxWfSgzThWQ}{Excalidraw}.}
\end{figure}
\section{Introduce}
\label{sec:org72a1101}
\label{sec:org97417ed}
What is Emacs and \textbf{org-mode}? Well, where to start? You may not have heard of
Emacs or org-mode, yet. Usually it is considered to a tool for geeks, \ldots{}.. this
might be kind of true, but once you noticed the myriard ways of using
......@@ -107,18 +104,14 @@ Let’s make an executive summary of org-mode:
Org-mode is a magnificent tool when it comes to reproducible research,\autocite{Stanisic2014}
since this combines a well documented way of analysing a data set.
\end{itemize}
\section{Prepare}
\label{sec:org2fdb5a0}
\label{sec:orge2abf86}
For our demonstration, we are going to create a dataset from openly available
data on the German National Research Data Infrastructure (\textbf{NFDI}) and perform
some simple analysis tasks on it.
\subsection{Data retrieval using SPARQL}
\label{sec:org1e83c85}
\label{sec:org725ba10}
The data we are interested in exists on Wikidata. Wikidata is similar to
Wikipedia, but rather than long form articles, the data is stored as structured
data. This allows machines to easily access and traverse these pages with query
......@@ -222,12 +215,8 @@ Beilstein Institute for the Advancement of Chemical Sciences & NFDI4Chem\\[0pt]
Berlin State Library & Base4NFDI\\[0pt]
Berlin State Library & NFDI4Memory\\[0pt]
\end{longtable}
\subsection{Data cleaning using shell}
\label{sec:org3315362}
\label{sec:org0371d9f}
The data we got from listing \ref{raw-dataset} is good but it needs further cleaning.
We can see several entries in our data that look like ``Q1234567'' - These are Q
......@@ -307,12 +296,11 @@ Beilstein Institute for the Advancement of Chemical Sciences & NFDI4Chem\\[0pt]
Berlin State Library & Base4NFDI\\[0pt]
Berlin State Library & NFDI4Memory\\[0pt]
\end{longtable}
\section{Process}
\label{sec:org7477678}
\label{sec:org7c38de4}
\subsection{Data Aggregation with Python}
\label{sec:org650a249}
\label{sec:org8e409c1}
The great thing about org mode is that we can seamlessly switch between
languages! Our original query (listing \ref{raw-dataset}) was written in SPARQL, which returned
......@@ -400,45 +388,11 @@ Association of states archaeologists & 1\\[0pt]
There is also a ``native way'' getting the counting done by using the package
\texttt{org-aggregate}\footnote{\url{https://github.com/tbanel/orgaggregate}}.
\begin{code}
\begin{center}
\begin{tabular}{lr}
wLabel & count()\\[0pt]
\hline
Bavarian Academy of Sciences and Humanities & 7\\[0pt]
Academy of Sciences and Literature Mainz & 4\\[0pt]
Bavarian State Archives & 4\\[0pt]
Alfred Wegener Institute for Polar and Marine Research & 3\\[0pt]
Bavarian Natural History Collections & 2\\[0pt]
Bavarian State Library & 2\\[0pt]
Berlin State Library & 2\\[0pt]
Academy of Sciences and Humanities in Hamburg & 1\\[0pt]
Anthropological Society (Munich) & 1\\[0pt]
Arachnologische Gesellschaft & 1\\[0pt]
Arbeitskreis Provenienzforschung e.V. & 1\\[0pt]
Archivschule Marburg & 1\\[0pt]
Archäologische Kommission für Niedersachsen & 1\\[0pt]
Archäologisches Museum Hamburg und Stadtmuseum Harburg & 1\\[0pt]
Arthistoricum & 1\\[0pt]
Association for Data-Intensive Radio Astronomy & 1\\[0pt]
Association for Technology and Construction in Agriculture & 1\\[0pt]
Association of German Architects & 1\\[0pt]
Association of Population Based Cancer Registries in Germany & 1\\[0pt]
Association of states archaeologists & 1\\[0pt]
BERD@NFDI & 1\\[0pt]
Bach-Archiv Leipzig & 1\\[0pt]
Bauhaus-Universität Weimar & 1\\[0pt]
Bavarian Forest National Park & 1\\[0pt]
Bavarian State Archaeological Collection & 1\\[0pt]
Bavarian State Research Center for Agriculture & 1\\[0pt]
Beethoven House & 1\\[0pt]
Beilstein Institute for the Advancement of Chemical Sciences & 1\\[0pt]
\end{tabular}
\end{center}
\end{code}
\subsection{Counting Elements with awk}
\label{sec:orgf0aaa45}
\label{sec:org6966a91}
We’re not limited to python though. Here we’re going to perform a very similar
aggregation, but grouping by consortia to get the number of institutes at each.
......@@ -474,7 +428,7 @@ END {
# going through the counts from above
for (k in counts)
# check for the amount of associated institutions
if (counts[k] == 1)
if (counts[k] == 1) (singular)
# if only one institution, then use the singular version
print consortium " (" counts[k] " institution)";
# otherwise we need the plural form.
......@@ -493,11 +447,8 @@ Back to our example: So, now we know of many institutions are involved in
NFDI4Objects (9 institutions)
or in
NFDI4Earth (3 institutions).
\subsection{Network Disply with R}
\label{sec:org5c9cd23}
\label{sec:orgb6cd15c}
How about something a little more visual than some tables? We can also create
plots and visuals, generating them with the code contained in the document and
......@@ -544,21 +495,16 @@ plot(NFDI_network, # loading data frame
\includegraphics[width=.5\linewidth]{img/nfdi-network.png}
\caption{\label{result-nfdi-network}Network of NFDI consortia (red) and institutions (blue).}
\end{figure}
\section{Preserve}
\label{sec:org36352f8}
\label{sec:org0d7ae0d}
There are two ways exporting this document in multiple documents.
The concept of this is called ``single-source-publishing''.
This means we have on document, our org-file, and we will export it into
different formats, which are more suitable for different occasions.
\subsection{Manual export}
\label{sec:org56ebae6}
\label{sec:orgc845442}
The common approach is to invoke the commands for exporting into a certain
format individually and by hand.
Org-mode has a great build in exporting mechanism which converts the document
......@@ -581,11 +527,8 @@ ASCII & \texttt{SPC m e t a} & \texttt{C-c C-e t a}\\[0pt]
\hline
\end{tabular}
\end{table}
\subsection{Automatic batch process}
\label{sec:org64e2378}
\label{sec:orgaabd198}
In a batch process the file is opened with a clean and neutral version of emacs
and will be exported (see listing \ref{batch-export}).
......
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment