Skip to content
Snippets Groups Projects
Commit 972df78b authored by Lukas C. Bossert's avatar Lukas C. Bossert
Browse files

improve getting filename

not obscurring using the buffer name
parent 944f5411
No related branches found
No related tags found
No related merge requests found
......@@ -3,7 +3,7 @@
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
<head>
<!-- 2023-09-15 Fri 13:05 -->
<!-- 2023-09-15 Fri 13:21 -->
<meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<title>Data Processing, Code Documentation and Beyond @@latex:\\@@ @@html:&lt;br&gt;@@ (Emacs and org-mode)</title>
......@@ -24,33 +24,33 @@
<h2>Table of Contents</h2>
<div id="text-table-of-contents" role="doc-toc">
<ul>
<li><a href="#orgb7d045a">1. Overview</a></li>
<li><a href="#org6545e67">2. Introduce</a></li>
<li><a href="#orge434f25">3. Prepare</a>
<li><a href="#org6b5605a">1. Overview</a></li>
<li><a href="#orga86c532">2. Introduce</a></li>
<li><a href="#org52ebe0c">3. Prepare</a>
<ul>
<li><a href="#org9ecabce">3.1. Data retrieval using SPARQL</a></li>
<li><a href="#org2931e83">3.2. Data cleaning using shell</a></li>
<li><a href="#org0e0765e">3.1. Data retrieval using SPARQL</a></li>
<li><a href="#org03738ff">3.2. Data cleaning using shell</a></li>
</ul>
</li>
<li><a href="#orgdb456dc">4. Process</a>
<li><a href="#orgffb61d7">4. Process</a>
<ul>
<li><a href="#org8d45caa">4.1. Data Aggregation with Python</a></li>
<li><a href="#orgcca651f">4.2. Counting Elements with awk</a></li>
<li><a href="#org72bb622">4.3. Network Disply with R</a></li>
<li><a href="#org1c52f17">4.1. Data Aggregation with Python</a></li>
<li><a href="#org7dc971a">4.2. Counting Elements with awk</a></li>
<li><a href="#orga6ceed9">4.3. Network Disply with R</a></li>
</ul>
</li>
<li><a href="#org407d616">5. Preserve</a>
<li><a href="#org86686dd">5. Preserve</a>
<ul>
<li><a href="#org64f39b4">5.1. Manual export</a></li>
<li><a href="#orgd8e14dd">5.2. Automatic batch process</a></li>
<li><a href="#org92d8632">5.1. Manual export</a></li>
<li><a href="#org0970726">5.2. Automatic batch process</a></li>
</ul>
</li>
</ul>
</div>
</div>
<div id="outline-container-orgb7d045a" class="outline-2">
<h2 id="orgb7d045a"><span class="section-number-2">1.</span> Overview</h2>
<div id="outline-container-org6b5605a" class="outline-2">
<h2 id="org6b5605a"><span class="section-number-2">1.</span> Overview</h2>
<div class="outline-text-2" id="text-1">
<p>
This document provides insights into an efficient way handling data.
......@@ -63,11 +63,11 @@ way of processing and documenting data workflow and analyses.
<p>
The paper is divided into three main steps, focussing on first preparing, second
processing and last presevering the data and its documentation (fig. <a href="#orgb3c77ae">1</a>).
processing and last presevering the data and its documentation (fig. <a href="#org1044a82">1</a>).
</p>
<div id="orgb3c77ae" class="figure">
<div id="org1044a82" class="figure">
<p><img src="img/nfdi-in-emacs-best-practice-overview.png" alt="nfdi-in-emacs-best-practice-overview.png" width="100%" />
</p>
<p><span class="figure-number">Figure 1: </span>Workflow of the document. Source <a href="https://excalidraw.com/#room=8617c3374a9c2c2c895b,a_SoKClI-tyAxWfSgzThWQ">Excalidraw</a>.</p>
......@@ -76,8 +76,8 @@ processing and last presevering the data and its documentation (fig. <a href="#o
</div>
<div id="outline-container-org6545e67" class="outline-2">
<h2 id="org6545e67"><span class="section-number-2">2.</span> Introduce</h2>
<div id="outline-container-orga86c532" class="outline-2">
<h2 id="orga86c532"><span class="section-number-2">2.</span> Introduce</h2>
<div class="outline-text-2" id="text-2">
<p>
What is Emacs and <b>org-mode</b>? Well, where to start? You may not have heard of
......@@ -140,8 +140,8 @@ since this combines a well documented way of analysing a data set.
</div>
<div id="outline-container-orge434f25" class="outline-2">
<h2 id="orge434f25"><span class="section-number-2">3.</span> Prepare</h2>
<div id="outline-container-org52ebe0c" class="outline-2">
<h2 id="org52ebe0c"><span class="section-number-2">3.</span> Prepare</h2>
<div class="outline-text-2" id="text-3">
<p>
For our demonstration, we are going to create a dataset from openly available
......@@ -151,8 +151,8 @@ some simple analysis tasks on it.
</div>
<div id="outline-container-org9ecabce" class="outline-3">
<h3 id="org9ecabce"><span class="section-number-3">3.1.</span> Data retrieval using SPARQL</h3>
<div id="outline-container-org0e0765e" class="outline-3">
<h3 id="org0e0765e"><span class="section-number-3">3.1.</span> Data retrieval using SPARQL</h3>
<div class="outline-text-3" id="text-3-1">
<p>
The data we are interested in exists on Wikidata. Wikidata is similar to
......@@ -183,7 +183,7 @@ If you like how to do this in more detail, have a look at (<a href="#citeproc_bi
</p>
<div class="org-src-container">
<label class="org-src-name"><span class="listing-number">Listing 1: </span>Retrieving the dataset from wikidata</label><pre class="src src-sparql" id="orgc290221"><span class="linenr">1: </span><span class="org-keyword">SELECT</span> <span class="org-variable-name">?wLabel</span> <span class="org-variable-name">?pLabel</span>
<label class="org-src-name"><span class="listing-number">Listing 1: </span>Retrieving the dataset from wikidata</label><pre class="src src-sparql" id="orgd510de7"><span class="linenr">1: </span><span class="org-keyword">SELECT</span> <span class="org-variable-name">?wLabel</span> <span class="org-variable-name">?pLabel</span>
<span class="linenr">2: </span><span class="org-keyword">WHERE</span>
<span class="linenr">3: </span>{
<span id="coderef-consortium" class="coderef-off"><span class="linenr">4: </span> <span class="org-variable-name">?p</span> wdt:P31 wd:Q98270496 . (consortium)</span>
......@@ -195,7 +195,7 @@ If you like how to do this in more detail, have a look at (<a href="#citeproc_bi
</pre>
</div>
<table id="orgc4ffc61" border="2" cellspacing="0" cellpadding="6" rules="groups" frame="hsides">
<table id="org45520e2" border="2" cellspacing="0" cellpadding="6" rules="groups" frame="hsides">
<caption class="t-above"><span class="table-number">Table 1:</span> Result of the query for NFDI consortia and their institutions.</caption>
<colgroup>
......@@ -467,11 +467,11 @@ If you like how to do this in more detail, have a look at (<a href="#citeproc_bi
<div id="outline-container-org2931e83" class="outline-3">
<h3 id="org2931e83"><span class="section-number-3">3.2.</span> Data cleaning using shell</h3>
<div id="outline-container-org03738ff" class="outline-3">
<h3 id="org03738ff"><span class="section-number-3">3.2.</span> Data cleaning using shell</h3>
<div class="outline-text-3" id="text-3-2">
<p>
The data we got from listing <a href="#orgc290221">1</a> is good but it needs further cleaning.
The data we got from listing <a href="#orgd510de7">1</a> is good but it needs further cleaning.
</p>
<p>
......@@ -486,11 +486,11 @@ SPARQL query, as an input variable to this cell (<code>:var input=raw-dataset</c
</p>
<div class="org-src-container">
<label class="org-src-name"><span class="listing-number">Listing 2: </span>Cleaning the raw data using good old <code>sed</code> and a regex pattern.</label><pre class="src src-sh" id="org5056ef3"><span class="org-type"><span class="org-tree-sitter-hl-faceXfunctionXcall"><span class="org-tree-sitter-hl-faceXfunctionXbuiltin">echo</span></span></span> <span class="org-tree-sitter-hl-faceXstring">"</span><span class="org-tree-sitter-hl-faceXstring"><span class="org-tree-sitter-hl-faceXpunctuationXspecial"><span class="org-constant">$</span></span></span><span class="org-tree-sitter-hl-faceXstring"><span class="org-tree-sitter-hl-faceXproperty"><span class="org-variable-name">input</span></span></span><span class="org-tree-sitter-hl-faceXstring">"</span> <span class="org-tree-sitter-hl-faceXoperator">|</span> <span class="org-tree-sitter-hl-faceXfunctionXcall">sed</span> <span class="org-tree-sitter-hl-faceXconstant">-E</span> <span class="org-tree-sitter-hl-faceXstring">'/Q[0-9]+/d'</span>
<label class="org-src-name"><span class="listing-number">Listing 2: </span>Cleaning the raw data using good old <code>sed</code> and a regex pattern.</label><pre class="src src-sh" id="org23ed5d1"><span class="org-type"><span class="org-tree-sitter-hl-faceXfunctionXcall"><span class="org-tree-sitter-hl-faceXfunctionXbuiltin">echo</span></span></span> <span class="org-tree-sitter-hl-faceXstring">"</span><span class="org-tree-sitter-hl-faceXstring"><span class="org-tree-sitter-hl-faceXpunctuationXspecial"><span class="org-constant">$</span></span></span><span class="org-tree-sitter-hl-faceXstring"><span class="org-tree-sitter-hl-faceXproperty"><span class="org-variable-name">input</span></span></span><span class="org-tree-sitter-hl-faceXstring">"</span> <span class="org-tree-sitter-hl-faceXoperator">|</span> <span class="org-tree-sitter-hl-faceXfunctionXcall">sed</span> <span class="org-tree-sitter-hl-faceXconstant">-E</span> <span class="org-tree-sitter-hl-faceXstring">'/Q[0-9]+/d'</span>
</pre>
</div>
<table id="orge3164f5" border="2" cellspacing="0" cellpadding="6" rules="groups" frame="hsides">
<table id="org731089c" border="2" cellspacing="0" cellpadding="6" rules="groups" frame="hsides">
<caption class="t-above"><span class="table-number">Table 2:</span> Cleaned data set which will be used for ruther processing.</caption>
<colgroup>
......@@ -735,19 +735,19 @@ SPARQL query, as an input variable to this cell (<code>:var input=raw-dataset</c
</div>
</div>
<div id="outline-container-orgdb456dc" class="outline-2">
<h2 id="orgdb456dc"><span class="section-number-2">4.</span> Process</h2>
<div id="outline-container-orgffb61d7" class="outline-2">
<h2 id="orgffb61d7"><span class="section-number-2">4.</span> Process</h2>
<div class="outline-text-2" id="text-4">
</div>
<div id="outline-container-org8d45caa" class="outline-3">
<h3 id="org8d45caa"><span class="section-number-3">4.1.</span> Data Aggregation with Python</h3>
<div id="outline-container-org1c52f17" class="outline-3">
<h3 id="org1c52f17"><span class="section-number-3">4.1.</span> Data Aggregation with Python</h3>
<div class="outline-text-3" id="text-4-1">
<p>
The great thing about org mode is that we can seamlessly switch between
languages! Our original query (listing <a href="#orgc290221">1</a>) was written in SPARQL, which returned
a kind of table (tab. <a href="#orgc4ffc61">1</a>). We then took that table and ran a shell command on it. Now,
languages! Our original query (listing <a href="#orgd510de7">1</a>) was written in SPARQL, which returned
a kind of table (tab. <a href="#org45520e2">1</a>). We then took that table and ran a shell command on it. Now,
we’re going to take the output of that shell command (cf. tab.
<a href="#orge3164f5">2</a>) and run
<a href="#org731089c">2</a>) and run
some python code on it.
</p>
......@@ -758,7 +758,7 @@ some python code on it.
</div>
<div class="org-src-container">
<label class="org-src-name"><span class="listing-number">Listing 3: </span>Counting the number of consortia involved in one institution.</label><pre class="src src-python" id="org7b7a3ec"><span class="linenr"> 1: </span><span class="org-tree-sitter-hl-faceXkeyword">import</span> pandas <span class="org-tree-sitter-hl-faceXkeyword">as</span> pd
<label class="org-src-name"><span class="listing-number">Listing 3: </span>Counting the number of consortia involved in one institution.</label><pre class="src src-python" id="org4655d8f"><span class="linenr"> 1: </span><span class="org-tree-sitter-hl-faceXkeyword">import</span> pandas <span class="org-tree-sitter-hl-faceXkeyword">as</span> pd
<span class="linenr"> 2: </span>
<span class="linenr"> 3: </span><span class="org-tree-sitter-hl-faceXcomment"># The data comes into the cell as a list of lists.</span>
<span class="linenr"> 4: </span><span class="org-tree-sitter-hl-faceXcomment"># We can pick it apart into a DataFrame object</span>
......@@ -779,7 +779,7 @@ some python code on it.
</pre>
</div>
<table id="org0bf5867" border="2" cellspacing="0" cellpadding="6" rules="groups" frame="hsides">
<table id="org5cd2a55" border="2" cellspacing="0" cellpadding="6" rules="groups" frame="hsides">
<caption class="t-above"><span class="table-number">Table 3:</span> Overview of institutions and the count of their associated consortia.</caption>
<colgroup>
......@@ -938,15 +938,15 @@ some python code on it.
</div>
</div>
<div id="outline-container-orgcca651f" class="outline-3">
<h3 id="orgcca651f"><span class="section-number-3">4.2.</span> Counting Elements with awk</h3>
<div id="outline-container-org7dc971a" class="outline-3">
<h3 id="org7dc971a"><span class="section-number-3">4.2.</span> Counting Elements with awk</h3>
<div class="outline-text-3" id="text-4-2">
<p>
We’re not limited to python though. Here we’re going to perform a very similar
aggregation, but grouping by consortia to get the number of institutes at each.
Like the listing <a href="#org7b7a3ec">3</a> above, we are going to use the output of
listing <a href="#org5056ef3">2</a> (cf. tab.
<a href="#orge3164f5">2</a>) to perform this operation.
Like the listing <a href="#org4655d8f">3</a> above, we are going to use the output of
listing <a href="#org23ed5d1">2</a> (cf. tab.
<a href="#org731089c">2</a>) to perform this operation.
Instead of python, we’re going to use <code>awk</code> for our data processing.
</p>
......@@ -960,7 +960,7 @@ results.
<div class="org-src-container">
<label class="org-src-name"><span class="listing-number">Listing 4: </span>Calculating the number of involved institions in one specific consortium.</label><pre class="src src-awk" id="org32107b0"><span class="linenr"> 1: </span><span class="org-keyword">BEGIN</span> {
<label class="org-src-name"><span class="listing-number">Listing 4: </span>Calculating the number of involved institions in one specific consortium.</label><pre class="src src-awk" id="orgb6b644f"><span class="linenr"> 1: </span><span class="org-keyword">BEGIN</span> {
<span class="linenr"> 2: </span><span class="org-comment-delimiter"># </span><span class="org-comment">before the evaluating process of the data begins</span>
<span class="linenr"> 3: </span><span class="org-comment-delimiter"># </span><span class="org-comment">this block is taken in account</span>
<span class="linenr"> 4: </span><span class="org-comment-delimiter"># </span><span class="org-comment">set the separator to tab</span>
......@@ -1005,8 +1005,8 @@ NFDI4Earth (3 institutions).
<div id="outline-container-org72bb622" class="outline-3">
<h3 id="org72bb622"><span class="section-number-3">4.3.</span> Network Disply with R</h3>
<div id="outline-container-orga6ceed9" class="outline-3">
<h3 id="orga6ceed9"><span class="section-number-3">4.3.</span> Network Disply with R</h3>
<div class="outline-text-3" id="text-4-3">
<p>
How about something a little more visual than some tables? We can also create
......@@ -1018,19 +1018,19 @@ embedding the results in the output.
<p>
And while we’re at it, how about another language? This time we’ll use R to make
a simple network plot of our data. Again, we’re still using the output from
listiing <a href="#org5056ef3">2</a> (which is tab.
<a href="#orge3164f5">2</a>) to do this.
listiing <a href="#org23ed5d1">2</a> (which is tab.
<a href="#org731089c">2</a>) to do this.
</p>
<p>
The result is a nice visualization of a network (fig. <a href="#org6b96e28">2</a>).
The result is a nice visualization of a network (fig. <a href="#org379f7dc">2</a>).
Such a visualization can help to detect outliers faster.
</p>
<div class="org-src-container">
<label class="org-src-name"><span class="listing-number">Listing 5: </span>Network of all institutions and their related consortia.</label><pre class="src src-R" id="orgb1cc777"><span class="linenr"> 1: </span><span class="org-comment-delimiter"># </span><span class="org-comment">making sure the required package is installed</span>
<label class="org-src-name"><span class="listing-number">Listing 5: </span>Network of all institutions and their related consortia.</label><pre class="src src-R" id="orgc4cafd9"><span class="linenr"> 1: </span><span class="org-comment-delimiter"># </span><span class="org-comment">making sure the required package is installed</span>
<span class="linenr"> 2: </span><span class="org-ess-keyword">if</span> (!<span class="org-ess-modifiers">require</span>(<span class="org-string">"igraph"</span>)) install.packages(<span class="org-string">"igraph"</span>)
<span class="linenr"> 3: </span><span class="org-ess-modifiers">library</span>(<span class="org-string">"igraph"</span>)
<span class="linenr"> 4: </span><span class="org-comment-delimiter"># </span><span class="org-comment">making a more robust outcome by stating a seed number</span>
......@@ -1053,7 +1053,7 @@ Such a visualization can help to detect outliers faster.
</div>
<div id="org6b96e28" class="figure">
<div id="org379f7dc" class="figure">
<p><img src="img/nfdi-network.png" alt="nfdi-network.png" width="70%" />
</p>
<p><span class="figure-number">Figure 2: </span>Network of NFDI consortia (red) and institutions (blue).</p>
......@@ -1064,8 +1064,8 @@ Such a visualization can help to detect outliers faster.
<div id="outline-container-org407d616" class="outline-2">
<h2 id="org407d616"><span class="section-number-2">5.</span> Preserve</h2>
<div id="outline-container-org86686dd" class="outline-2">
<h2 id="org86686dd"><span class="section-number-2">5.</span> Preserve</h2>
<div class="outline-text-2" id="text-5">
<p>
There are two ways exporting this document in multiple documents.
......@@ -1076,8 +1076,8 @@ different formats, which are more suitable for different occasions.
</div>
<div id="outline-container-org64f39b4" class="outline-3">
<h3 id="org64f39b4"><span class="section-number-3">5.1.</span> Manual export</h3>
<div id="outline-container-org92d8632" class="outline-3">
<h3 id="org92d8632"><span class="section-number-3">5.1.</span> Manual export</h3>
<div class="outline-text-3" id="text-5-1">
<p>
The common approach is to invoke the commands for exporting into a certain
......@@ -1089,10 +1089,10 @@ C-e</code> and then select which export format you would like to have.
<p>
In tab. <a href="#orgb405333">4</a> you find a quick overview of some basic formats.
In tab. <a href="#org8786fd7">4</a> you find a quick overview of some basic formats.
</p>
<table id="orgb405333" border="2" cellspacing="0" cellpadding="6" rules="groups" frame="hsides">
<table id="org8786fd7" border="2" cellspacing="0" cellpadding="6" rules="groups" frame="hsides">
<caption class="t-above"><span class="table-number">Table 4:</span> Overview of various individual export functions.</caption>
<colgroup>
......@@ -1134,16 +1134,16 @@ In tab. <a href="#orgb405333">4</a> you find a quick overview of some basic form
<div id="outline-container-orgd8e14dd" class="outline-3">
<h3 id="orgd8e14dd"><span class="section-number-3">5.2.</span> Automatic batch process</h3>
<div id="outline-container-org0970726" class="outline-3">
<h3 id="org0970726"><span class="section-number-3">5.2.</span> Automatic batch process</h3>
<div class="outline-text-3" id="text-5-2">
<p>
In a batch process the file is opened with a clean and neutral version of emacs
and will be exported (see listing <a href="#orga81324c">6</a>).
and will be exported (see listing <a href="#orgf59a8d4">6</a>).
</p>
<div class="org-src-container">
<label class="org-src-name"><span class="listing-number">Listing 6: </span>Exporting file into various formats</label><pre class="src src-emacs-lisp" id="orga81324c"><span class="linenr"> 1: </span><span class="org-rainbow-delimiters-depth-1">(</span><span class="org-keyword">let</span> <span class="org-rainbow-delimiters-depth-2">(</span><span class="org-rainbow-delimiters-depth-3">(</span>org-file <span class="org-rainbow-delimiters-depth-4">(</span><span class="org-constant">find-file-noselect</span> filename<span class="org-rainbow-delimiters-depth-4">)</span><span class="org-rainbow-delimiters-depth-3">)</span><span class="org-rainbow-delimiters-depth-2">)</span>
<label class="org-src-name"><span class="listing-number">Listing 6: </span>Exporting file into various formats</label><pre class="src src-emacs-lisp" id="orgf59a8d4"><span class="linenr"> 1: </span><span class="org-rainbow-delimiters-depth-1">(</span><span class="org-keyword">let</span> <span class="org-rainbow-delimiters-depth-2">(</span><span class="org-rainbow-delimiters-depth-3">(</span>org-file <span class="org-rainbow-delimiters-depth-4">(</span><span class="org-constant">find-file-noselect</span> filename<span class="org-rainbow-delimiters-depth-4">)</span><span class="org-rainbow-delimiters-depth-3">)</span><span class="org-rainbow-delimiters-depth-2">)</span>
<span class="linenr"> 2: </span> <span class="org-rainbow-delimiters-depth-2">(</span><span class="org-keyword">with-current-buffer</span> org-file
<span class="linenr"> 3: </span> <span class="org-rainbow-delimiters-depth-3">(</span><span class="org-function-name">org-html-export-to-html</span><span class="org-rainbow-delimiters-depth-3">)</span>
<span class="linenr"> 4: </span> <span class="org-rainbow-delimiters-depth-3">(</span><span class="org-constant">message</span> <span class="org-string">"HTML export successful."</span><span class="org-rainbow-delimiters-depth-3">)</span>
......@@ -1182,7 +1182,7 @@ There might be people having a different opinion.
</div></div>
<div id="postamble" class="status">
<p class="author">Author: Jonathan A. Hartman | Lukas C. Bossert</p>
<p class="date">Created: 2023-09-15 Fri 13:05</p>
<p class="date">Created: 2023-09-15 Fri 13:21</p>
</div>
</body>
</html>
\ No newline at end of file
......@@ -459,7 +459,7 @@ and will be exported (see listing [[batch-export]]).
#+name: batch-export
#+caption: Exporting file into various formats
#+BEGIN_SRC emacs-lisp -n :results none :var filename=(buffer-name)
#+BEGIN_SRC emacs-lisp -n :var filename=(file-name-nondirectory (buffer-file-name))
(let ((org-file (find-file-noselect filename)))
(with-current-buffer org-file
(org-html-export-to-html)
......@@ -475,6 +475,9 @@ and will be exported (see listing [[batch-export]]).
))
#+END_SRC
#+RESULTS: batch-export
: nfdi-in-emacs-with-orgmode.org
#+print_bibliography:
* Footnotes
......
No preview for this file type
% Created 2023-09-15 Fri 13:02
% Created 2023-09-15 Fri 13:21
% Intended LaTeX compiler: lualatex
\documentclass[11pt]{article}
\usepackage{graphicx}
......@@ -46,7 +46,7 @@
\section{Overview}
\label{sec:orgec35bc5}
\label{sec:org761ab24}
This document provides insights into an efficient way handling data.
We show not only how to retrieve data from an publicly accesible webpge but also
how the data can be processed afterwards. We admit that in the examples shown
......@@ -65,7 +65,7 @@ processing and last presevering the data and its documentation (fig. \ref{workfl
\section{Introduce}
\label{sec:org2c31cfc}
\label{sec:org94dfc11}
What is Emacs and \textbf{org-mode}? Well, where to start? You may not have heard of
Emacs or org-mode, yet. Usually it is considered to a tool for geeks, \ldots{}.. this
might be kind of true, but once you noticed the myriard ways of using
......@@ -110,7 +110,7 @@ since this combines a well documented way of analysing a data set.
\section{Prepare}
\label{sec:org61254cc}
\label{sec:org3b374bd}
For our demonstration, we are going to create a dataset from openly available
data on the German National Research Data Infrastructure (\textbf{NFDI}) and perform
......@@ -118,7 +118,7 @@ some simple analysis tasks on it.
\subsection{Data retrieval using SPARQL}
\label{sec:org7f8618d}
\label{sec:org1a79980}
The data we are interested in exists on Wikidata. Wikidata is similar to
Wikipedia, but rather than long form articles, the data is stored as structured
data. This allows machines to easily access and traverse these pages with query
......@@ -227,7 +227,7 @@ Berlin State Library & NFDI4Memory\\[0pt]
\subsection{Data cleaning using shell}
\label{sec:org5e11845}
\label{sec:org56020cc}
The data we got from listing \ref{raw-dataset} is good but it needs further cleaning.
We can see several entries in our data that look like ``Q1234567'' - These are Q
......@@ -309,10 +309,10 @@ Berlin State Library & NFDI4Memory\\[0pt]
\end{longtable}
\section{Process}
\label{sec:org71a6f0b}
\label{sec:org4470386}
\subsection{Data Aggregation with Python}
\label{sec:org65e7422}
\label{sec:org0e78222}
The great thing about org mode is that we can seamlessly switch between
languages! Our original query (listing \ref{raw-dataset}) was written in SPARQL, which returned
......@@ -398,7 +398,7 @@ Berlin State Library & 2\\[0pt]
\end{longtable}
\subsection{Counting Elements with awk}
\label{sec:org91535c1}
\label{sec:orgb1e796e}
We’re not limited to python though. Here we’re going to perform a very similar
aggregation, but grouping by consortia to get the number of institutes at each.
......@@ -457,7 +457,7 @@ NFDI4Earth (3 institutions).
\subsection{Network Disply with R}
\label{sec:orgd53bd53}
\label{sec:org9b7cffc}
How about something a little more visual than some tables? We can also create
plots and visuals, generating them with the code contained in the document and
......@@ -508,7 +508,7 @@ plot(NFDI_network, # loading data frame
\section{Preserve}
\label{sec:org659e884}
\label{sec:org8435f75}
There are two ways exporting this document in multiple documents.
......@@ -518,7 +518,7 @@ different formats, which are more suitable for different occasions.
\subsection{Manual export}
\label{sec:org456fd1f}
\label{sec:org91bc9ff}
The common approach is to invoke the commands for exporting into a certain
format individually and by hand.
Org-mode has a great build in exporting mechanism which converts the document
......@@ -545,7 +545,7 @@ ASCII & \texttt{SPC m e t a} & \texttt{C-c C-e t a}\\[0pt]
\subsection{Automatic batch process}
\label{sec:org6ffc2dd}
\label{sec:org4b60c04}
In a batch process the file is opened with a clean and neutral version of emacs
and will be exported (see listing \ref{batch-export}).
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment