SoftspaceAR | Prototype04
See the true shape of your ideas.
Softspace harnesses spatial computing to help thinkers and makers better see, understand, and develop the ideas that form the heart of their creative projects.
This is the fourth in a series of prototypes weβre releasing as a part of our open development process to build the augmented reality version of Softspace.
Help us invent a powerful new kind of tool for thought by testing these prototypes π§π½βπ¬, following us on Twitter π¦, and joining the Softspace Discord π―ββοΈ.
SoftspaceAR Prototype04 is out! You can download the APK here, then install it on your Quest 2 using SideQuest or the Oculus Developer Hub.
This prototype is an answer to the question: Could a spatial interface let you work with the true shape of your ideas?
To see how this question leads to the design of Prototype04, we need to take a step back and understand what this question even means.
In what sense do ideas have a shape, and why does their shape matter?
The Structure of Knowledge
What is the structure of knowledge? In other words: what are the actual things that constitute knowledge, and how are they related to each other?
There are many possible answers to this question. It could be that knowledge is one very long sequence of information; or a scattered field of points that donβt relate to each other at all; or a Borgesian state space of permutational nonsense through which we rifle for nuggets of Truth.
(Iβm going to dodge the notoriously divisive question of what constitutes knowledge in the first place, as opposed to other kinds of information like data, or sensory experience. Letβs go with a Stewartian definition of knowledge for now. Itβs the thing that PKM folks think about all the time. Itβs what knowledge workers are busy working on. Itβs the kind of information that tools for thought help you better create, understand, and harness.)
How knowledge is structured matters, because its structure defines what we consider to be legitimate strategies for seeking and creating it.
For example, if knowledge were actually a single long sequence of information (e.g. the revealed word of a deity), then we would seek knowledge within the canon, and create knowledge by expanding on canonical points.
The biblical reference might make this model seem epistemologically unsoundβ¦ but consider that this is how a lot of writing is produced. Pieces start out as a small list of points, each of which is then expanded into a collection of points, some of which need further elaborationβall while maintaining the overall single-sequence structure.
Another candidate for the structure of knowledge is the tree: information is grouped into topics, which are themselves grouped into topics, going from the specific to the general. This model is the basis for the Dewey Decimal System, which is used to organize books in libraries. Itβs also how most people think of species as being organized: within an evolutionary Tree of Life.
If knowledge is structured as a tree, you should seek knowledge by determining the general category that information would be found in, then follow the path of narrowing subtopics until you get to your answer. The work of creating knowledge would be the work of discovering new pieces of information and deciding which sub-sub-sub-branch it belongs to, or filling in categorical βmissing linksβ between something higher up in the tree with something lower down. This is literally how librarians and taxonomists work.
Knowledge is a Network
The problem is that these models prove incomplete under closer scrutiny.
Biblical hermeneutics is the discipline of interpreting and elaborating on the received word, and is a critical bridge between whatβs written in the Bible and important practical questions like βshould capital punishment be allowedβ? These interpretations are core theological knowledge, but they donβt simply slot into the existing sequence of the Bible. Rather, they can draw on information from across the textβor indeed, other authoritative interpretationsβto arrive at their conclusions.
The librarianβs dream is every book in its proper place, neatly arranged according to the single totalizing hierarchy of the Dewey Decimal System. But underneath their calm, hardbound surfaces, books are seething with non-hierarchicial irregularities. Footnotes and citations in the text refer to texts that could be on the next shelf down, or in a totally different category two floors up, forming a vast, nonlinear web of meaning.
Even the evolutionary Tree of Life, which seems to hold true for the vast majority of species that reproduce sexually, proves more convoluted than taxonomists would like. Consider the case of the hoatzin, a bird that defies all attempts to paint an orderly picture of the tree of bird species.
These examples hint at a different model.
If knowledge can be said to have a global structure, thereβs a lot of evidence pointing to that structure being a network. Technically, itβs a graph in which pieces of information are the nodes and relationships between those pieces are the edges.
Why might we think this?
Footnotes and citations in texts referencing other texts, or even other sections in the same text, form a nonlinear graph structure.
Mind maps almost always take on a graph-like structure, with node-like concepts connected by lines.
A new generation of graph-based note-taking apps (spearheaded by Roam Research) have become very popular, very quickly, in academic and professional communities that work intensively with knowledge.
Anecdotally, a lot of people who were uncommonly good at working with knowledge also seem to intuit its underlying structure as graph-like.
"Creativity is just connecting thingsβ¦ When you ask creative people how they did something, they feel a little guilty because they didnβt really do it, they just saw something. It seemed obvious to them after a while. Thatβs because they were able to connect experiences theyβve had and synthesize new things.β
βThe physical entities which seem to serve as elements in thought are certain signs and more or less clear images which can be βvoluntarilyβ reproduced and combinedβ¦ this combinatory play seems to be the essential feature in productive thoughtβ¦ the play with the mentioned elements is aimed to be analogous to certain logical connections one is searching for.β
Last, and never least, James Webb Young:
βAn idea is nothing more nor less than a new combination of old elements [and] the capacity to bring old elements into new combinations depends largely on the ability to see relationships.β
Graphs Are Hard
With the graph-like structure of knowledge (hopefully) established, letβs get back to the original question:
Could a spatial interface let you work within the true shape of your ideas?
Ideas are knowledge, and the βtrue shapeβ of your ideas is the way that the graph-like structure of knowledge is manifested within the ideas youβre working on.
Analogous to the way that language shapes what we can think, or architectural drawing techniques shape what we can build, our tools and methods for working with knowledge shape what we can come to know.
If knowledge really is graph-like, then the most effective strategies for seeking and creating it would be those that treat it as such. Tools that obscure or ignore its underlying structure would limit our ability to do knowledge work.
Unfortunately, graphs are notoriously difficult to represent usably in the kind of bounded, 2D interfaces that computer monitors encourage. Graphs are large, nonlinear, and multi-scalar, three qualities that 2D UIs struggle with.
And if our flat tools are limiting our ability to work with the true shape our ideas, then theyβre limiting how well we can understand and develop them.
Spatial Knowledge Graphs
SoftspaceAR Prototype04 proposes a new paradigm for representing knowledge graphsβone thatβs free of the limitations of 2D UIs by virtue of its use of spatial computing.
This paradigm is a 3D spatial canvas in which user can create text blocks.
Any terms in the text which are surrounded by double square brackets are interpreted as topics to which that text refers (syntax borrowed from Roam).
Text and topics are our informational nodes, and the references from text to topic are one of two kinds of possible relation.
The other kind of relation that is possible is inclusion: topics can be expanded, and text added to that topic as a way of defining it.
A 3D forced-directed graph simulation running in the background automatically tries to move related items together, while pushing unrelated ones apart. As the user creates, relates, unrelates, and deletes pieces of information, the system constantly updates the overall shape of the knowledge contained within.
Prototype04 shows some of the advantages of working with knowledge graphs spatially:
A spatial interface offers much more room to display the nodes of a graph. In fact, this even allows the graph to display individual text blocks as nodes, instead of the entire pages that many 2D graph views settle for.
The global shape of the graph is generated from the local, node-to-node interactions. This global form is an important part of the meaning of the knowledge graph, because it suggests what the higher-level cluster or syntheses of the details could be.
Object permanence makes it expected and intuitive that items would be includable in multiple contexts. This makes it straightforward to implement true transclusion.
While working on a given node, you can see direct and indirect connections receding into the distance. This creates ambient contextual awareness, so you always know βwhereβ you are within your ideas.
How to use Prototype04
When you launch the prototype for the first time, a pre-constructed knowledge graph loads.
Click on any text block to start editing it.
Press Return at the end of a text block to create a new text block, and press Backspace when typing on an empty text block to delete it.
As you write on a text block, terms enclosed in double square brackets are interpreted as topic references.
Prototype04 will find or create the topic with the matching title (case-sensitive) and link it to the text block where you typed out the term.
As more topics are created, and more text blocks refer to different topics, the system automatically draws related items together and pushes unrelated ones apartβbuilding up the shape of your ideas as you write them out.
Topics are more than just tags for drawing related text blocks together. Topics are epistemic entities that can contain content.
Click on a topicβs title to expand it into a board. Click anywhere on the board to create a new, pre-contained text block.
You can also grab an existing text item and add it to the topic board, or pull text items off and let them float away.
Clicking on the topic title again, or expanding a different topic, will collapse it and allow the contained text items to float awayβwhile remaining connected with a bold connection line.
A text item can be contained in any number of topicsβthe system permits true transclusion.
Topics are created automatically when referenced for the first time, and deleted automatically if they are no longer referenced and contain no text.
Your work isnβt stuck in the headset! Each topic automatically exports itself as a markdown file, making it easy to bring your work into the rest of your workflow. These .md files are called βLensesβ.
Known Issues
Text blocks and topic titles that are far away become difficult to readβthis is a major limitation. I have an idea for a text LODing system that would address this.
Links are currently represented using a simple straight line between the linked items. These get in the way of text, and donβt tell you much when you canβt see the other end. They generally need design work.
Prototype01 took eight weeks start-to-finish. P02 took four weeks; P03 took five weeks; and this prototype took me six weeks. I want to reverse this trend and get the build cycles back toward four weeks or less.
Next Steps
Currently, topic boards use a single column layout, but thereβs no reason why each topicβs internal layout system couldnβt be a full ordinospatial layout like the one in Prototype03.
I think this paradigm will really shine with the (re)introduction of images and PDFs.
The only way to create new content right now is by typing out text, but I will port over the very flexible in-app web browser from SoftspaceVR. This allows you to save out full-res images, copy/paste text, snip any part of the browser window into PNG, and create website bookmarks.
Only one topic can be expanded at time right now, but weβll probably want to let users keep several topics open at once; this raises the question of how to represent content blocks that are transcluded across multiple expanded topics.
These prototypes write to, and read from, the headsetβs local storage, which is a real pain to access. Production builds will use a Dropbox backend at first, and expand to other backend options later (IPFS anyone?).
Thereβs currently a hard distinction between text blocks and topic items, which isnβt necessary. It would be far more elegant and flexible to allow any content block to become a topic container, so that ideas can be defined recursively.
Thatβs it for now! This post is the longest one yet, coming in at just under 2500 words, so if you made it this farβthanks and congrats π.
If youβd like to support this work, please consider following us on Twitter, trying out Prototype04, and joining our Discord to offer your questions, ideas, and criticisms.
Until next time,
Yiliu
