An Abstract Syntax Tr… What Now?

Imagine you have a simple document called text.Rmd:

---
title: AST-what-now?
author: Ulrik Lyngs

---

## my header

Some text with *italics* and also **bold**.

When pandoc reads this, it splits it up into a Meta block containing the stuff in the YAML header (in this case title and author), and a list of Block elements containing the actual content (in this case two blocks, a ‘Header’ and a ‘Para’).

If you’ve got pandoc accessible at the terminal, you can see what this looks like by creating a file named text.Rmd, putting the content above into it, navigating to its location, and then typing in the command pandoc -s -t native test.Rmd:

~ ulyngs$ pandoc -s -t native text.Rmd

Pandoc (Meta {unMeta = fromList [("author",MetaInlines [Str "Ulrik",
Space,Str "Lyngs"]),("title",MetaInlines [Str "AST-what?"])]})
[Header 2 ("my-header",[],[]) [Str "my",Space,Str "header"],
Para [Str "Some",Space,Str "text",Space,Str "with",Space,
Emph [Str "italics"],Space,Str "and",Space,Str "also",Space,
Strong [Str "bold"],Str "."]]

Languages with which to write pandoc filters

So basically, a pandoc filter walks through our document’s elements in this form and makes some changes we might want.

The original way to create filters used the functional programming language Haskell. However, as Haskell is not widely known, John MacFarlane also created the python package pandocfilters, and others made the more accessible alternative (still in python) panflute package. Adaptations of the pandocfilters package have been developed for other languages, including JavaScript and R).

A potential problem with these options is that they create new dependencies, for example on a local Python installation, which means that additional stuff besides pandoc is necessary to get them to work.

Since pandoc v2.0, this problem can be bypassed: Now, pandoc has a built-in ability to understand filters written in the lightweight programming language lua (the pandoc executable now includes a lua interpreter and relevant libraries).

tl;dr: pandoc filters written in lua should ‘just work’. Yay, let’s use lua!

Manipulate Metadata from YAML header

Imagine we want to use a pandoc filter to change the author name in the final output of text.Rmd.

First, create an empty text file and save it with the name toy_filter.lua. Then, in text.Rmd’s YAML header, add a line that tells pandoc to use this file as a lua filter:

---
title: AST-what-now?
author: Ulrik Lyngs
output:
  pdf_document:
    pandoc_args: ["--lua-filter=toy_filter.lua"]

---

## my header

Some text with *italics* and also **bold**.

If we leave toy_filter.lua empty and knit text.Rmd, the PDF output is unaltered:

Now let’s use toy_filter.lua as a filter to change the author name to ‘Hello, World!’. The filter should find the YAML metadata in the abstract syntax tree (AST) and manipulate the author, before pandoc goes on to convert it to a .tex file from which a PDF can be built.

To do this, we create a function with the name of the element in the AST we want to manipulate, in this case Meta. This function should manipulate the author and return a modified version:

function Meta(m)
  m.author = "Hello, World!"
  return m
end

If we save toy_filter.lua and now knit text.Rmd to PDF, this is the result:

Manipulate text formatting

Another silly example: Say we wanted to change italicised text to bold.

First, we look at the documentation for pandoc lua filters. We notice that italicised text passages are parsed by pandoc as Emph inline elements in the AST, and text passages in bold as Strong inline elements.

Then we add this to toy_filter.lua:

function Emph(element)
  return pandoc.Strong(element.content)
end

Now, when we save toy_filter.lua and knit text.Rmd to PDF, we get this:

Solving for HTML

When outputting to HTML, highlighting these passages was then simply a matter of styling with CSS: I created a new text file style.css and in it added

.correction {
  background-color: #CCCCFF;
}

From a document like this…

---
title: AST-what-now?
author: Ulrik Lyngs
output:
  html_document:
    css: style.css
  pdf_document:
    pandoc_args: ["--lua-filter=toy_filter.lua"]
    
---

## my header

Some text with *italics* and also **bold**.

And <span class="correction">this text</span>, but not this, 
<span class="correction">should be highlighted</span>.

I now got this when outputting to HTML:

Solving for PDF

For PDF output, I needed the lua filter to find Span elements, check if the class was “correction”, and if so wrap the content in the \mccorrect command. This would do the trick:

function Span (el)
  if el.classes[1] == "correction" then
    table.insert(el.content, 1, pandoc.RawInline('latex', '\\mccorrect{'))
    table.insert(el.content, pandoc.RawInline('latex', '}'))
    end
  return el.content
end

If I included this in toy_filter.lua, knitted text.Rmd to PDF and inspected the intermediary .tex file (add keep_tex: true in your YAML header) I saw that it worked as expected:

Some text with \textbf{italics} and also \textbf{bold}.

And \mccorrect{this text}, but not this, \mccorrect{should be
highlighted}.

\end{document}

Hooray!