Data Processing, Code Documentation and Beyond
(Emacs and org-mode)

The data we are interested in exists on Wikidata. Wikidata is similar to Wikipedia, but rather than long form articles, the data is stored as structured data. This allows machines to easily access and traverse these pages with query langauges. Here, we are going to submit a SPARQL query to the Wikidata query endpoint.

SPARQL will look familiar to anyone familar with SQL, however it is slightly more cryptic at first glance. Take a look at the below query – things like “Q98270496” refer to specific items in wikidata, where things like “P31” are more akin to concepts. In English, this query translates to something like

Give me the Names for items that has a property (P31) of NFDI Consortia (Q98270496), and return all items you find on each of those entries under the property “affiliations” (P1416).

If you like how to do this in more detail, have a look at (Bossert et al. 2023).

1: SELECT ?wLabel ?pLabel
2: WHERE
3: {
4:   ?p wdt:P31 wd:Q98270496 . (consortium)
5:   ?p wdt:P1416 ?w . (affiliations)
6:   SERVICE wikibase:label { bd:serviceParam wikibase:language "en" . }
7: }
8: ORDER BY ASC(?wLabel) ASC(?pLabel)
9: LIMIT 50

The data we got from listing 1 is good but it needs further cleaning.

We can see several entries in our data that look like “Q1234567” - These are Q Ids for items which no label has been defined. Let’s remove those from our dataset.

We’re going to include the output from the previous cell, where we executed the SPARQL query, as an input variable to this cell (:var input=raw-dataset).

echo "$input" | sed -E '/Q[0-9]+/d'

The great thing about org mode is that we can seamlessly switch between languages! Our original query (listing 1) was written in SPARQL, which returned a kind of table (tab. 1). We then took that table and ran a shell command on it. Now, we’re going to take the output of that shell command (cf. tab. 2) and run some python code on it.

python -m pip install pandas --user

 1: import pandas as pd
 2: 
 3: # The data comes into the cell as a list of lists.
 4: # We can pick it apart into a DataFrame object
 5: df = pd.DataFrame(clean_df[1:], columns=clean_df[0])
 6: 
 7: # Perform a groupby operation on wLabel and
 8: # rename the resulting new column "Count"
 9: institutions_by_consortia = (
10:     df
11:     .groupby("wLabel")
12:     .size()
13:     .sort_values(ascending=False)
14:     .reset_index(name="Count"))
15: 
16: # Return our dataframe in a way that org will
17: # display it as an org table
18: return [list(institutions_by_consortia.columns),
19:         None, *map(list, institutions_by_consortia.values)]

There is also a “native way” getting the counting done by using the package org-aggregate².

We’re not limited to python though. Here we’re going to perform a very similar aggregation, but grouping by consortia to get the number of institutes at each. Like the listing 3 above, we are going to use the output of listing 2 (cf. tab. 2) to perform this operation. Instead of python, we’re going to use awk for our data processing.

As an additional bonus, we’re going to paramaterize this cell by defining a variable called consortium. With this we could reuse the code in this cell over and over, changing the desired consortium name to show only the desired results.

 1: BEGIN {
 2: # before the evaluating process of the data begins
 3: # this block is taken in account
 4: # set the separator to tab
 5:   FS =  "\t"
 6: }
 7:   # MAIN section of the evaluating process
 8:   #----------------------------------------
 9:   # while going through the rows of the input
10:   # check only for the second column
11:   # step a counter for equal values and store it in 'counts'
12:   $2 == consortium { ++counts[$2] }
13: END {
14:   # final part where no evaluation is done anymore
15:   # only collecting and printing results
16:   # going through the counts from above
17:   for (k in counts)
18:   # check for the amount of associated institutions
19:       if (counts[k] == 1) (singular)
20:   # if only one institution, then use the singular version
21:          print consortium " (" counts[k]  " institution)";
22:   # otherwise we need the plural form.
23:      else print consortium " (" counts[k]  " institutions)"
24: }

Having created the source block we can also use it in our text with executing the the function call_institutions-count('NFDI4Objects'). The result will be blended in smoothly in the text and if there are any changes to the initial data set updated automatically.

Back to our example: So, now we know of many institutions are involved in NFDI4Objects (9 institutions) or in NFDI4Earth (3 institutions).

How about something a little more visual than some tables? We can also create plots and visuals, generating them with the code contained in the document and embedding the results in the output.

And while we’re at it, how about another language? This time we’ll use R to make a simple network plot of our data. Again, we’re still using the output from listiing 2 (which is tab. 2) to do this.

The result is a nice visualization of a network (fig. 2). Such a visualization can help to detect outliers faster.

 1: # making sure the required package is installed
 2: if (!require("igraph")) install.packages("igraph")
 3: library("igraph")
 4: # making a more robust outcome by stating a seed number
 5: set.seed(123456789)
 6: # convert the tabular data into a data frame which is required
 7: # for creating a network
 8: NFDI_network <- graph_from_data_frame(NFDI_edges,
 9:                                       directed = FALSE)
10: plot(NFDI_network,                   # loading data frame
11:     main  = "NFDI Network",          # adding a title
12:     # adding a color to all nodes from the second column.
13:     vertex.color = c("blue", "red")#
14:             [1 + names(V(NFDI_network)) %in% NFDI_edges[,2]],
15:     vertex.size        = 4,         # size of the node
16:     vertex.frame.color = NA,        # no frame for nodes
17:     vertex.label       = NA,        # no color of the description
18:     edge.curved        = 0.2,       # factor of "curvity"
19:     )

The common approach is to invoke the commands for exporting into a certain format individually and by hand. Org-mode has a great build in exporting mechanism which converts the document into all mainly used formats. You get to the menue by calling SPC m e or C-c C-e and then select which export format you would like to have.

In tab. 4 you find a quick overview of some basic formats.

In a batch process the file is opened with a clean and neutral version of emacs and will be exported (see listing 6).

 1: (let ((org-file (find-file-noselect filename)))
 2:   (with-current-buffer org-file
 3:     (org-html-export-to-html)
 4:     (message "HTML export successful.")
 5:     )
 6:   (with-current-buffer org-file
 7:     (org-ascii-export-to-ascii)
 8:     (message "ASCII export successful.")
 9:     )
10:   (with-current-buffer org-file
11:     (org-latex-export-to-pdf)
12:     (message "PDF export successful.")
13:     ))