# Web Annotation Discovery

This document describes a simple mechanism for using [Web Annotation][] in web browsers. The envisioned outcome is that people can discover annotation sources and subscribe to annotation ‘feeds’ while browsing the web, then view the annotations on other pages they visit.

## Introduction

### Context

Various tools, often in the form of browser extensions, have attempted to make possible on the web what is easy enough on paper: highlighting passages and scribbling on the document while reading. Moreover, several such tools go beyond what its paper equivalent permits, and enable sharing such annotations with others via a web service. One could for example view their friends’ comments on a blog post, or their grandma’s personal tips on a recipe — without requiring involvement of the website that was annotated.

However, annotations created in one annotation tool are often only viewable with exactly the same tool. And sharing annotations, when possible, requires that people use exactly the same tool & service. Ideally, people could use any tool of their choice to view annotations from any source of their choice, and regardless of the software and/or service chosen by the annotator.

To facilitate interoperability among annotation tools, the [Web Annotation Data Model][] was specified and standardised by W3C in 2017. The specification outlines an extensible JSON-based format for annotations, which encodes the target document (and the targeted passage/part within it) that is being annotated, the ‘body’ of the annotation (i.e. one’s scribbles, if any), and metadata such as the author, creation time, and motivation of the annotation.

The accompanying [Web Annotation Protocol][] specifies an [LDP][]-based protocol for exchanging (collections of) annotations between a client (typically an annotation viewing/editing tool) and a server. That server could e.g. be a personal data store for creating and viewing one’s own annotations, or a read-only collection of annotations that were created/generated by other parties.

Together, the two specifications provide a standarised way for any annotation viewer to get a collection of annotations from any annotation server. However, they have not specified a way to *discover* such annotation servers.

### Goal

The current proposal addresses the discovery of annotations and annotation sources, so that web browsers can offer users to ‘import’ or ‘subscribe to’ their annotations.

The goal is that the web browser becomes an annotation viewer (possibly through an extension), that shows known annotations on pages the user visits. The displayed annotations can come from any source, and importantly need not be supplied or endorsed by the publisher of the visited website itself; allowing for diverse use case like commentary, fact-checking, and many others.

In a sense, it introduces the concept of reverse links: the browsers shows documents (annotations) linking *to* the visited page. Ideally, the browser will also supply ways to create annotations and share/publish them, but this is not part of (and orthogonal to) the current proposal.

### Approach

To show annotations on a visited page, the web browser needs to somehow obtain these annotations. Various previous annotation projects depend on a single global service to index the annotations by their target, which browsers would query for annotations targeting a particular page. To avoid such centralisation and cater for the diversity of use cases, the browser could instead query any annotation services of the user’s choice.

However, querying services for annotations on visited pages has an enormous impact on reader privacy: to find for annotations on pages you read, you have to tell the service which pages you read. Subscribing to multiple sources would reveal this information to even more parties.

In many usage scenarios, the annotations a person is actually interested in is limited and from a known source. Centralised services (e.g. [Hypothes.is][]) can help discover annotations from any other user, but are often used for annotating in well-defined groups: in classrooms, among colleagues, etc.

In such cases, there is no need for a central global index, and moreover the total set of annotations of interest could easily fit on the user’s device. This would solve the reader privacy issue as no querying is needed — the browser can simply look up if it has any relevant annotations for any visited page (and can thereby be much quicker too).

Also for somewhat larger-scale annotation consumption, the total size may well remain managable. For example, if an investigative journalist subscribes to a thousand colleagues each writing ten annotations per day of 1KB each, this produces roughly 4GB in a year — significant, but perhaps worth it for their work (at which privacy may be more important than disk storage). This size could still be reduced by an order of magnitude if, of each annotation, only its own URL and the URL it targets are stored (with a tradeoff for latency and privacy, see [further below](#compacted-storage)).

The current proposal omits any querying mechanism and adopts this approach of a local ‘annotation library’. The mechanisms defined below serve to populate this library: How to discover annotation sources and import their current annotations, and subscribe to a source/‘feed’ to obtain their future annotations. To this end, it selects and combines existing parts of the Web Annotation specifications.

Two discovery mechanisms are defined:

1. Annotations encountered directly, either served as a file or embedded in a web page.
2. Annotation ‘feeds’: collections of annotations discovered via links in web pages.

Both of these are described and specified below.


## Encountering Annotations

The simplest way for a web browser to discover annotations is by stumbling upon them. The Web Annotation Data Model defines the canonical serialisation of annotations in the JSON format, either individually or in an [Annotation Collection][]. An individual annotation or a collection can be encountered in two ways:

- served directly.
- embedded in a `<script>` element in a web page.

### Example: proofreading use case
You read my book published as a web page at `https://my.example/book`, in order to give feedback on unclear passages and grammatical mistakes. Instead of returning me a copy of the book with your comments, or perhaps placing a copy in collaborative editing software, you annotate the book at its original URL in your web browser. You then share the created annotation collection with me, e.g. via your cloud storage or attached in an email.

When I open the annotation collection file in my browser, it lets me import the annotations; possibly storing them, or just applying them while I keep the collection open in a tab. Either way, it will display your annotations in context when I visit `https://my.example/book` — possibly along with other annotations that I got from others or made myself.

### Annotations served directly

Somebody could open an annotation or collection in their browser, for example by opening the JSON file from local disk, or by following a link to such a file. The browser can then offer to import the contained annotation(s) into its library, and henceforth display it/them where applicable.

When possible, annotations and annotation collections SHOULD be presented with the media type set to `application/ld+json;profile="http://www.w3.org/ns/anno.jsonld"`. Otherwise, e.g. when opened from the file system without media type information, detection of the appropriate `@context` and `type` attributes in the JSON object can be used to recognise the annotation (collection).

**Example**:
You follow a link to `http://an.example/annotations/comment-2022-01-09-349`, and get this single annotation as the response:

```
HTTP/1.1 200 OK
Content-Type: application/ld+json;profile="http://www.w3.org/ns/anno.jsonld"

{
  "@context": "http://www.w3.org/ns/anno.jsonld",
  "id": "http://an.example/annotations/comment-2022-01-09-349",
  "type": "Annotation",
  "creator": {
      "name": "Somebody"
  },
  "bodyValue": "This page is good.",
  "target": "https://some.example/page"
  },
}
```

The browser would detect this response is an annotation, and likely offer to import it or open the page it targets to show it in its intended context.

If you follow a link to a collection of annotations, the result will look exactly like the [Example: annotation feed](#example-annotation-feed) further below. A feed is simply an annotation collection, and when opened directly in the browser, it could offer both to import the annotations it embeds, and (perhaps depending on its URL scheme) offer to subscribe to the feed.

### Annotations embedded in a web page

To really discover annotations as one browses the web, annotations can be embedded inside web pages. The format used for this is defined in the W3C Note [Embedding Web Annotations in HTML][], section [Annotations Embedded as JSON-LD][]. Unlike the example cases given in that Note, the annotation can of course target any other page on the web, rather than parts of the same page.

The format amounts to the same JSON representation as above being wrapped in `<script>` tags with the appropriate media type.

#### Example: annonation embedded in HTML

The same annotation as above, now embedded in a web page, would look like this:

```
<script type='application/ld+json;profile="http://www.w3.org/ns/anno.jsonld"'>
{
  "@context": "http://www.w3.org/ns/anno.jsonld",
  "id": "http://an.example/annotations/comment-2022-01-09-349",
  "type": "Annotation",
  "creator": {
      "name": "Somebody"
  },
  "bodyValue": "This page is good.",
  "target": "https://some.example/page"
  },
}
</script>
```

To embed multiple annotations, one could either use multiple `<script>` elements, or embed an Annotation Collection in the same manner, with the annotations embedded in the collection.


## Annotation feeds

While the mechanism of gathering annotations, described above, enables people to import collections of annotations into their browser, a likely wish is to also display annotations from that source that will be published later on.

Much like how one can subscribe to a website’s [RSS][]/[Atom][] feed(s) to obtain its future publications, this section proposes a mechanism to subscribe to a website’s ‘annotation feed(s)’.

### Format

Technically, an annotation feed is an [Annotation Container][] as specified in the [Web Annotation Protocol][]. This means the feed is represented as an [Annotation Collection][] that can be obtained in its JSON representation via an HTTP request. Its annotations can be spread across multiple pages for piecemeal access, like with a [paged feed][] for Atom. (Note the terms Container and Collection are practically interchangeable here.)

As the Annotation Protocol is a flavour of [JSON-LD][] and [LDP][], an annotation feed could naturally use various extensions and alternative formats. However, as the protocol requires a canonical JSON-LD serialisation to be supported by both clients and servers, mere mortal implementers can freely ignore any linked data concepts and simply consider it a HTTP & JSON protocol with a couple of extra fields and headers that can be copy-pasted from the example below.

#### Example: annotation feed

The client has subscribed to an annotation feed `https://an.example/annotations/`. Periodically, the client updates the annotations from this source by sending an HTTP request:

```
GET /annotations/ HTTP/1.1
Host: an.example
Accept: application/ld+json;profile="http://www.w3.org/ns/anno.jsonld"
Prefer: return=representation;include="http://www.w3.org/ns/oa#PreferContainedDescriptions"
```

The response to the above could be (shortened to its essentials):

```
HTTP/1.1 200 OK
Content-Type: application/ld+json;profile="http://www.w3.org/ns/anno.jsonld"
Link: <http://www.w3.org/ns/ldp#Resource>; rel="type"
Allow: GET,OPTIONS,HEAD

{
  "@context": [
    "http://www.w3.org/ns/anno.jsonld",
    "http://www.w3.org/ns/ldp.jsonld"
  ],
  "id": "http://an.example/annotations/",
  "type": ["BasicContainer", "AnnotationCollection"],
  "modified": "2022-01-23T12:00:00Z",
  "label": "My annotation feed",
  "total": 48,
  "first": {
    "id": "http://an.example/annotations/?page=0",
    "type": "AnnotationPage",
    "next": "http://an.example/annotations/?page=1",
    "items": [
      {
        "id": "http://an.example/annotations/comment-2022-01-09-349",
        "type": "Annotation",
        "creator": {
            "name": "Somebody"
        },
        "bodyValue": "This page is good.",
        "target": "https://some.example/page"
      },
      {
        "id": "http://an.example/annotations/comment-2022-01-07-925",
        "type": "Annotation",
        "creator": {
            "name": "Somebody"
        },
        "bodyValue": "Quite quaint indeed."
        },
        "target": {
            "source": "https://bogus.example/page",
            "selector": {
                "type": "TextQuoteSelector",
                "exact": "This is a quaint but quotable sentence.",
            }
        }
      },
      …
    ]
  },
  "last": "http://an.example/annotations/?page=5"
}
```

In this example, the first page of annotations is embedded in the response, with a link to the next page that could be requested subsequently. The last page is also linked to.

#### Formatting recommendations

While any Annotation Container can be used as an annotation feed, it is helpful to use reverse chronological ordering of annotations across the pages of a container (if it has multiple pages). The server SHOULD also embed the first page of annotations in the response, as exemplified here, unless asked otherwise. This enables clients to fetch the most recent feed additions in a single HTTP request.

A client SHOULD set the `Prefer` header, as exemplified here, to request that this first page of annotations is indeed embedded (i.e. [Representations with Annotation Descriptions][]). Alternatively it could ask to only get the annotations’ URIs (`PreferContainedIRIs`), or no annotations at all (`PreferMinimalContainer`). However, the client cannot rely on a server respecting these preferences.

### Feed discovery

Discovery of annotation feeds works analogous to Atom and [RSS Autodiscovery][]. A page can announce an annotation feed through a `<link>` element in a web page, or the equivalent `Link` HTTP header, with the relationship (`rel`) set to `alternate` and the `type` set to `application/ld+json;profile="http://www.w3.org/ns/anno.jsonld"`.

#### Example: feed discovery link

When visiting `fredsfrets.example`, your browser might find:

```
<link
  rel="alternate"
  type='application/ld+json;profile="http://www.w3.org/ns/anno.jsonld"'
  href="https://annotations.fredsfrets.example/all/"
  title="Fred’s frets"
/>
```

Your browser offers you the option to subscribe to this annotation feed. Once you do, the browser imports the feed’s annotations into its library, periodically refreshes the feed content (at configurable intervals), and will henceforth display Fred’s notes on the pages they target.


## Interaction with annotations

While this proposal makes certain assumptions about how a browser would let the user interact with annotations, there is significant freedom in this regard. Two extremes in the range of possible behaviour are:

- Highly automatic: The browser could automatically import all annotations it discovers, and automatically display them on the document(s) they target.
- Highly manual: The browser could let the user choose manually which (collections of) annotations to import, and when to display them.

For a broad range of cases, the more pleasant user experience will likely be somewhere between these extremes. For example, annotations embedded in bookmarked pages might be imported automatically, while others are not.

It may often be desirable to display annotations on documents besides the exact URI that they target, because the same document content may be published with multiple identifiers. An obvious example is the same content being served via both the `http` and `https` protocols. The [`canonical` link relationship][] can be taken into account to establish document sameness, but implementers should be aware of potential confusion as described [further below](#misdirecting-annotations).

Also, throughout this document it is usually assumed that the browser ‘imports’ annotations for future display. For various use cases, other storage or processing solutions may be chosen, possibly not even without persisting them at all. For example, the browser could display annotations embedded in any webpage that is currently opened (in any tab) in any other tabs where they are applicable.


## Privacy concerns

As described in the introduction, the discovery-based annotation system proposed here avoids the reader privacy problem of query-based web annotation systems — where the annotation source knows which page one is visiting at which moment. An open ecosystem with many annotation sources would exacerbate this issue.

The feed-based approach presented here avoids the information leakage resulting from annotation search. Instead of querying a server for any relevant annotations it may have, one downloads all its annotations, so search can be done locally.

Note that feeds are not privacy-perfect either: without further anonymisation (proxies) the publisher (and intermediaries) gets some information about the fact that one is subscribed, to which feeds (if there are multiple), and when one refreshes the feed. However this reveals much less information than when normally browsing the web. Importantly, the annotation publisher gets no information regarding which pages the user visits, or which annotations they view, nor when.

### Compacted storage

For storage efficiency, the browser could decide to store only the targets and URLs of annotations. In this situation, the browser could determine when an annotation is relevant, but would have to request the content of an annotation to display it.

This would lead to the annotation publisher learning about the (likely) fact that the user visited the page the requested annotation targets. However, no information is leaked until the annotation is displayed, and only pages that have annotations from a publisher are revealed to that publisher.

### Misdirecting annotations

As described above, a web browser might use the `canonical` link relationship to establish sameness of documents, in order to apply annotations targeting either of the documents to the other. This could let a web page ‘claim’ the annotations of another. While web pages have no access to the content or even the presence of annotations on them, this misdirection may confuse the visitor enough to create security issues.

For example, a virtual sticky note, stored as a private annotation, might be used to remember a password for a website — like its physical counterpart, more often than people are willing to admit. Displaying this note on a fake copy of that website would make such a scam extra likely to succeed.

Using the usual trust models on the web mitigate this issue. A `rel=canonical` link pointing to another origin could be treated with less trust than one to the same domain; possibly by ignoring it completely, or requiring user interaction before displaying the potentially misdirected annotations.


## Implementations

The example code in [discovery.ts](./example-code/discovery.ts) implements the basics of the described protocol:

- Recognise and import annotations served directly to the browser.
- Discover annotations embedded in `<script>` tags in a page.
- Detect `<link>` tags for annotation feed discovery.

The [Web Annotation Discovery WebExtension][] browser extension uses this example code to demonstrate a complete browser-side annotation viewer, supporting subscription to annotation feeds, importing discovered annotations (served directly or embedded), and displaying them all besides any page they target. (Moreover, it can create and edit annotations and store/publish them using the Web Annotation Protocol.)

On the server side, there is the [Web Annotation Discovery Server][]. It implements the Web Annotation Protocol for publishing annotation collections, enables the discovery mechanisms described above by including the `<link>` tags (on collections) and `<script>` elements (on individual annotations).


## Related work

The Web Annotation Protocol specification provides a section on [Discovery of Annotation Containers][], which suggests that documents link to annotation containers with the relationship `rel="http://www.w3.org/ns/oa#annotationService"`. However, the intended relation is that the linked container is used to store annotations *about the page that links to it*:

> “Any resource MAY link to an Annotation Container when Annotations on the resource SHOULD be created within the referenced Container.”

While looking similar at first sight, this is a very different concept, and roughly the reverse of the current proposoal in two senses:

1. Annotations in the linked container should *target* the linking resource; whereas in the discovery mechanism in this proposal, annotations in the linked container can target anything — and more likely their annotation *body* is represented in the current page.
2. The linked container is intended (primarily) for *creating* annotations, not *retrieving* them.

A browser may still choose to use this discovery mechanism in parallel, possibly to retrieve and display additional annotations targeting the page currently being viewed.


## Potential future work

The proposed annotation discovery mechanism could be extended in various ways as the need appears. Below a few such possibilities are mentioned.

### Unify with Atom/RSS feeds

Given the similarity between annotation feeds and RSS/Atom feeds, a logical, alternative proposal would combine the XML-model of RSS/Atom feeds with the possibility of representing Web Annotations as RDF+XML. The items of an RSS/Atom feed could simply contain the annotations themselves, perhaps even along with usual RSS/Atom content in order to make a single feed consumable both by existing feed readers and annotation viewers.

### Filtered collections

To only fetch subsets of a large annotation selection, it would be helpful to have filtering options; in particular, a filter could enable requesting annotations whose target matches a given domain name or other pattern.

Such a flexible filtering system would allow various trade-offs between privacy and data size. The service will get to know domains the user is interested in, but not exact information on which page the user visits at which moment.

### Annotation Search

If the above filter function is usable to request only annotations targeting a specific URL, it could be used as an annotation search mechanism, to efficiently query a large collection for annotations on a specific page.

As described above, such annotation search can have a large impact on people’s privacy. If it would be pursued, a combination of the following (and other) methods could help reduce the information leaking via queries:

- Using Bloom filters provide a privacy and possibly efficiency gain, by skipping most of the requests that would certainly not return any results. An Annotation Container could define a Bloom filter in its metadata, which would have to be updated regularly when annotations are added to the collection.

- Annotation sources could announce the domains/scopes they provide annotations for; e.g. a university system might provide annotations targeting some academic websites and journals, and need not be queried about gossip magazines.

- [Oblivious HTTP][], [TOR][] or similar methods could hide the identity of the client from the annotation server. (Note this mitigation is also applicable when using the discovery mechanism.)

- Querying annotation sources only on manual request of the user would avoid many queries, and leaves privacy tradeoff decisions to the user (for better and worse).

### Remote library

To have annotation feeds available on multiple devices, or on devices with limited storage, one’s browser could instruct a designated server to subscribe to feeds instead of subscribing locally. An annotation search mechanism as described above could be used to query that server for annotations, though an additional mechanism would be needed for connecting to the server and instructing it.

### Access control

Since the Annotation Protocol is based on the LDP protocol, its access control mechanisms could be naturally applied to annotation feeds to create non-public feeds. For collections the user has write access to, the browser could offer the option to add or modify annotations in the feed.

### Interpreting HTML content as annotations

Besides explicitly embedding annotation objects inside a web page, existing structured content with annotation-like semantics may be considered as equivalent to a Web Annotation.

For example, the web browser could automatically interpret a web page containing a `<q>`/`<blockquote>` as an annotation targeting the quoted phrase (using a [TextQuoteSelector][]) within the resource linked in its `cite` attribute.

Similarly, pages that link to phrases within other pages using the WICG draft [Text Fragments][] specification could be treated as if they were annotations on those phrases.

More or less the same idea is also explored in the [Embedding Web Annotations in HTML][] note, section [Web Annotation-based Citation URLs][].

### Generalised back-link discovery

More generally, many pages may be interesting to discover while browsing pages that they link to. Generalising the approach of the current proposal, one can for example imagine a web browser with a “what links here?” button, analagous to the ones found on Wikipedia. In a sense, annotation discovery is merely a method for creating a reverse link mechanism on the web, a concept which could be considered more broadly.


<!-- References -->

[Web Annotation]: https://www.w3.org/annotation/
[Web Annotation Data Model]: https://www.w3.org/TR/annotation-model/
[Web Annotation Protocol]: https://www.w3.org/TR/annotation-protocol/
[LDP]: https://www.w3.org/TR/ldp/
[Hypothes.is]: https://hypothes.is/
[Annotation Collection]: https://www.w3.org/TR/annotation-model/#collections
[Annotations Embedded as JSON-LD]: https://www.w3.org/TR/annotation-html/#embed-json-ld
[Embedding Web Annotations in HTML]: https://www.w3.org/TR/annotation-html/
[RSS]: https://www.rssboard.org/rss-specification
[Atom]: https://datatracker.ietf.org/doc/html/rfc4287
[Annotation Container]: https://www.w3.org/TR/annotation-protocol/#annotation-containers
[JSON-LD]: https://www.w3.org/TR/json-ld/
[paged feed]: https://www.rfc-editor.org/rfc/rfc5005#section-3
[Representations with Annotation Descriptions]: https://www.w3.org/TR/annotation-protocol/#representations-with-annotation-descriptions
[RSS Autodiscovery]: https://www.rssboard.org/rss-autodiscovery
[`canonical` link relationship]: https://www.rfc-editor.org/rfc/rfc6596.html
[Web Annotation Discovery WebExtension]: https://code.treora.com/gerben/web-annotation-discovery-webextension
[Web Annotation Discovery Server]: https://code.treora.com/gerben/web-annotation-discovery-server
[Discovery of Annotation Containers]: https://www.w3.org/TR/annotation-protocol/#discovery-of-annotation-containers
[Oblivious HTTP]: https://datatracker.ietf.org/group/ohai/
[TOR]: https://www.torproject.org/
[TextQuoteSelector]: https://www.w3.org/TR/annotation-model/#text-quote-selector
[Web Annotation-based Citation URLs]: https://www.w3.org/TR/annotation-html/#web-annotation-based-citation-urls
[Text Fragments]: https://wicg.github.io/scroll-to-text-fragment/