utilities/file_location_listing#65: Add support for YAML frontmatter to markdown parsing

The commit above introduces a new metadata attribute to storage files.

The idea is that this is an extensible section which can have arbitrary attributes added under it - in this case, we've added one called frontmatter.

Although it's not currently implemented (and won't be under the heading of this issue), the plan is that we'll add a new index key which indicates what metadata attributes a stored item has.

In this case that might look something like

METADATA-ATTRIBS: ['frontmatter']

That'll allow indexes to be used to speed up searches which match against something that only exists in metadata. Not that there's currently syntax to support it, but if we're doing a search for documents who's frontmatter includes the key "foo" with value "bar" we'd be able to quickly narrow the search set down to only include documents that actually have frontmatter.

It might also be that we want to make the storage headers extensible, perhaps including something like

X-METADATA-ATTRIB-frontmatter: ['foo:bar']

But I think that path probably leads to pain once you start thinking about how to incorporate that into indexes

For now, we keep this simple - the changes should mean than indexing files doesn't break if they've got front-matter.

That's the main change that I need.

Once that's tested and definitely working, we can look at collapsing key-pairs into tag values to make them searchable, but the main thing is I want to be able to release soon (this weekend ideally) so that I can start adding front-matter to my notes.

Edit: actually, it looks like that injection is a 1 line change, so I'll do it now.

OK, test crawl running now

There's a test file in there with the following content:

---
Category: foo
CatName: foo Bar Sed
age: 40
---
# Test file

This is my test file

With a couple of typos fixed, it looks to have worked:

Additional things checked

• searches/indexing still working correctly for pages without front-matter
• search parsing doesn't treat those tags as a dork, I can search for them

There is one small problem though - the current dork processing looks like this

    dorks = [
        'content-type', 
        'domain', 
        'ext', 
        'hastitle', 
        'matchtype', 
        'mode',
        'prefix'
        ]

...

        # It's a search-fu
        t_sp = t.split(":")
        if t_sp[0] in dorks:
            filters["fu"][t_sp[0]] = t_sp[1].strip()

In the example above, the values are Category:foo, CatName:foo_Bar_Sed and age:40. None of those prefixes exist in dorks, so it's treated as a normal search.

However, that may not always be true. If our markdown doc looked like this

---
Category: foo
domain: www.bentasker.co.uk
---
# Some Doc

We'd run into trouble. It wouldn't be possible to search for tag value domain:foo because it would get interpreted as a dork and only results from www.bentasker.co.uk would be displayed (so if this file were on a different domain, it wouldn't appear in results)

The result would potentially be quite confusing: At best, you'd get no results back despite knowing that there was something there, at worst you'd get incomplete results back (meaning you may not notice it was inaccurate).

It might be better to collapse frontmatter into something more akin to scoped tags - i.e. Category::foo rather than Category:foo. It'd be trivial to have the dork logic skip anything with a double colon

The other thing that we need to consider, though, is that the current implementation doesn't actually parse the YAML.

So there are all sorts of YAML supported things which aren't currently handled, for example:

---
foo: ['bar','sed']
a:
 - b
 - c
 - d

It might instead be prudent to pass it into Python's YAML parser, then we can infer type when injecting tags (injecting multiple for relevant lists).

We should also add special support for tag and tags as Obsidian explicitly includes special handling for those.

The test file now has the following contents

---
Category: foo
CatName: foo Bar Sed
age: 40
tagset:
  - a
  - b
  - c
taglist: ['foo', 'bar', 'sed']


---
# Test file

This is my test file

As of the commit above, the stored tags for this page are

TAGS:["Category::foo", "CatName::foo_Bar_Sed", "tagset::a", "tagset::b", "tagset::c", "taglist::foo", "taglist::bar", "taglist::sed"]

Within the JSON payload, the metadata attribute has the following:

{"frontmatter": {"Category": "foo", "CatName": "foo Bar Sed", "age": 40, "tagset": ["a", "b", "c"], "taglist": ["foo", "bar", "sed"]}}

Which is pretty much what we wanted.

We don't currently support nested objects in the YAML though:

foo:
  bar: 1
  sed: 2
  zoo: 3

Although it'll still appear under metadata, it won't result in any injected tags.

We should also add special support for tag and tags as Obsidian explicitly includes special handling for those.

I'm intending to do this once I've special cased :: in search term parsing.

OK, on to special casing of YAML items.

Obsidian's doc lists a set of defaults:

It also notes that it used to support singular versions of these (tag,alias and cssclass) but that these were deprecated and should not be used.

Realistically, deprecated or not, we should probably special case them to ensure historic docs are still supported.

Although not listed in the defaults table, the doc also references title - it'd make sense to support setting the page title from this.

Down the line, it'd be nice if we could index aliases too, but that's not for today - for now, we just won't special case (then they'll be indexed as tags)

So, there are three changes that need to be made as a result of this

• if the attribute name is tag or tags, don't prefix the tag value
• if the title attribute exists, set the page title from that
• if the attribute is cssclasses or cssclass, don't inject a tag

I think we might now be done here.

The following test file works just fine

---
title: Real title
Category: foo
CatName: foo Bar Sed
age: 40
tags:
  - a
  - b
  - c
tag: ['foo', 'bar', 'sed']
cssclass: foobar
cssclasses: ['class1', 'class2']
date: 2024-07-16T00:00:00Z
float: 1.3
mixeds:
  - 2024-07-16T00:00:00Z
  - a
  - 1.5
  - 1
  - true

---
# Test file

This is my test file

The new functionality is controlled by environment variables:

• YML_LOAD_FRONTMATTER: should frontmatter be parsed (default: True)
• YML_NO_PREFIX: which YAML properties shouldn't be prefixed when injecting tags (default: tag,tags)
• YML_NO_TAG: which YAML properties shouldn't result in a tag being injected (default: cssclass,cssclasses,title,date)
• YML_LOAD_TITLE: should a YAML title property be used to set the page title?