Magrathea

gemini/gemtext version

In July 2020, I finally got frustrated enough by existing software that I decided to look for a new software environment. Existing solutions not quite fitting the bill, I have since then been designing one. I came to call it Magrathea, after the planet in The Hitch Hiker's Guide to the Galaxy.

The ideas are not particularly unique. They are drawn from existing platforms - UNIX, Android, and Emacs are particular inspirations. Very late in the design process, I discovered that Plan 9 has many similar ideas, too.

Additionally, the impact of malleable.systems on this endeavor cannot be overstated. Their seven principles opened my eyes to what I actually liked about Emacs (my previous environment of choice), to what it lacks, and to the world of malleable software beyond it. The malleable.systems principles are the spirit that guides Magrathea -

  1. Software must be as easy to change as it is to use it
  2. All layers, from the user interface through functionality to the data within, must support arbitrary recombination and reuse in new environments
  3. Tools should strive to be easy to begin working with but still have lots of open-ended potential
  4. People of all experience levels must be able to retain ownership and control
  5. Recombined workflows and experiences must be freely sharable with others
  6. Modifying a system should happen in the context of use, rather than through some separate development toolchain and skill set
  7. Computing should be a thoughtfully crafted, fun, and empowering experience

Here is an overview of the primary ideas and goals of Magrathea.

System-wide requirements

  1. It must be implemented in a Lisp. 1 I really can't stand non-Lisp languages, largely because of -
    1. the complex and inconsistent syntax,
    2. lack of structural editing, and
    3. lack of means to extend the language.
    Moreover, Lisp is the original memory-safe language - I really hate applications which crash, or languages which want me to bother with irrelevant details which computers are perfectly capable of handling. Common Lisp brings, in addition, the ability to modify a running application, a unique object system, and - when desired - the ability to add type information and run static type checks.
    • Common Lisp is my top choice here.
  2. Scale - it must be possible for a single person to (re)implement it within a reasonable timeframe, and to maintain and understand it completely.
  3. Immediate response - it must be possible for the user to inspect, reconfigure, and modify any component of the system as it runs, without restarting anything.
  4. The system and application user interface is primarily graphical.
    • Textual interfaces are something I am avoiding, unless there is something which cannot be achieved without them.
  5. Window management is tiled.
    • I have not seen any particular benefits of overlapping windows. It only seems to increase the burden of window management for the user.
  6. All GUIs can be modified by the user - similar to Firefox's toolbar customization, but more powerful. All applications can be inspected and customized by the user, similar to Emacs.
  7. The system is object-oriented.
    • I will probably use the Common Lisp Object System (CLOS).
  8. Consistency is important. It simplifies both use and maintenance (#2).
  9. Operations depend on objects, not context. Operations for specific classes of objects are registered with the system by applications. Wherever in the system an object appears, the operations applicable to it can be performed. This greatly improves consistency (#8).
  10. Means of combination in GUI - applications 2 Yes, GUI applications 🙂 can pass data to each other. They can be composed to create pipelines, similar to UNIX shell pipelines. Consequently, an application need not implement something another application does, keeping the size of the system down (#2). Components are modular and the user has the power to trivially recombine them as they wish.
    • I have numerous examples in my notes, but the one that started it all was working in a video editor and wanting to open an audio track in a separate audio editor, then having the changes reflected in the video editor. Currently this requires much fiddling with files, exporting, and importing. In Magrathea, I want it to be a trivial operation.
    • Unlike shell pipelines, Magrathea pipelines are not necessarily unidirectional - it must be possible not only for data to flow from application A → B → C, but also from A → B → C → B → A, akin to a call stack.
    • Whether data passed in a pipeline is updated in source/destination programs only at application launch and exit, or as soon as it change on either side, is something I've not yet been able to decide on.
  11. Means of abstraction in GUI - pipelines can be abstracted into GUI elements, such as buttons, and added to existing applications.
  12. GUI elements can be combined to create new applications. As long as the user does not need to create new primitives from scratch, this requires no knowledge other than the knowledge required to use the system - it is done through clicking, menus, and drag-and-drop. 3 Now you know why it's called Magrathea—the program which creates new programs, named after the planet which creates new planets.

  13. The default applications and their configuration must target the majority of users. Design the UX to -

    1. Help the new user use the system, and perhaps become an experienced user;
    2. Help the experienced user tinker with the system, and perhaps become a new tinkerer;
    3. Help the tinkerer program the system, and perhaps become a new programmer.

    In particular, this means that we do not simply give a bunch of components which the end user must learn to combine before they can use them - we give them a software environment which is familiar to them, one they can start using right away.

  14. Wherever possible, move common functionality from applications to the system. This aids consistency (#8) and keeps the size down (#2).
    • For instance, how to render data such as mathematical notation or musical staff notation, or how to syntax highlight code, is not the purview of individual programs, but of the system itself. Additional packages can install new data types and/or ways of rendering them.
  15. It must be designed for multiple devices - laptops, phones, tablets, et al. Among other things, this means -
    • It must be possible to run the exact same programs on another device, rather than having to reinvent reduced-functionality versions of the same. 4 This is the situation we have with existing mobile operating systems.
    • All GUIs must be able to adapt to different devices.

Minor ideas

Some things I have given relatively less thought to -

  1. Omnipresent undo and redo - it should be possible to undo or redo most operations in the system, including -

    • launching or closing applications,
    • changing window layout,
    • jumping to a new position in an audio/video player,

    …and so on. In a way, it is similar to the forward/back buttons in a web browser, or the back button in Android.

  2. Per-application capabilities/permissions, sandboxing, and resource limits, so the user can run untrusted applications fearlessly.
  3. Adapting to hardware - I can't stand slow software; but rather than forgo human-friendliness for the sake of performance, I'd prefer a system which adapts to the available hardware resources.
  4. It must not be possible for any single application to consume resources to the point that the system becomes unresponsive…unless the user explicitly permits it. Related to #17 and #18.
  5. Structured documentation at the system package level. Each package must provide a tutorial, how-to guides, an explanation, and a reference - this way, the four essential types of documentation are covered.

Some application-level desires

WYSIWYG word processor

…one that is at par with WYSIWYM systems like LaTeX. I detest edit-compile-view cycles (see #3 - immediate response). This is the tool I would like to use to write literate programs. Some aspects that set it apart from existing solutions -

  • The final document format must be a simpler and smaller specification than the ODF standard (to serve #2).
  • The document must not make any network requests nor run any scripts when rendered, to make it safer than mainstream formats (alternatively, see #17).
  • I really like Org's outlines, tags, properties, and "sparse trees", and this word processor will support these features.
  • The ability to link to arbitrary elements of a document, especially to verbatim text or a range of verbatim text.

  • "Layers", similar to those in a raster image editing program. Each layer can contain either -

    1. content (anchored to content in another layer), or
    2. presentation information (applied to content in another layer), or
    3. modifications to content in another layer.

    Thus, content/style can be layered onto a document, while keeping it separate from the older content/style. This is useful for annotations, editorial changes, and other things. Layers can be individually hidden or displayed differently by the program.

  • There may be a concept of (local) transclusion.
  • While scripting in a document is useful, I'm not sure how to add it without turning it into an application platform like the WWW 🙁

Sheet music viewer

The dominant mode of viewing music on a screen is via PDFs or images, and I absolutely hate it - it is equivalent to storing text as an image. You cannot reflow, copy, (easily) annotate, or modify the appearance. The navigation commands understand images or pages, not bars or systems of music.

The sheet music viewer I have in mind will read structured data (e.g. MusicXML 5 Or better yet, s-expressions/EDN. ) and render it according to user-specified constraints (e.g. screen size, font size; it could even transpose the music if necessary, hide one or more instruments, and so on).

The design of the system (see #7-#9) means one can select and operate on (copy, etc) individual musical elements (notes, stems, beams, bars, systems, etc) from this viewer, and indeed anywhere sheet music is displayed (e.g. when embedded in a document, chat message, or email), without each application having to support it separately.

The concept of layers from the word processor is available in the sheet music too, so it is trivial to make annotations and editorial changes.

A multitrack metaformat for audio

Every audio format I know of is a squashing together of multiple tracks into one. I'm imagining a metaformat where each track is preserved separately. As a result -

  • the volume of each instrument could be changed (e.g. you want to hear something more prominently, perhaps for transcription - this is currently done by EQ, which is an inexact hack), and
  • specific tracks could be extracted and unwanted tracks could be removed (e.g. remove vocal tracks to make a karaoke version).

Conclusion

So that's the essentials of Magrathea. I have tried to avoid mentioning implementation details for now. It is not as ambitious at its foundation as CLOSOS or Dynamicland, perhaps. While I would love it if it empowered lay users as I want it to, it is designed primarily to cater to me.

If you have some feedback, or want to help, I'd love to hear from you. Get in touch!

Comments

khinsen, 2021-05-20T19:27:32+Z

A nice vision - and a nice name!

It doesn't look easy though. You have two items in your wish list that could well turn out to be contradictory: understandable by a single person, and graphics everywhere. Graphics support means a lot of essential complexity, just because of the sheer number of items you need to provide and make composable. Even text is very complicated nowadays, thanks to Unicode.

Even though your top priority is Common Lisp, you should probably take a close look at today's Smalltalk systems, which check off many points on your list, probably more than any other. Finally, on a slightly provocative note, I'll disagree with your footnote on Lisps. There are languages with even less syntax: the concatenative languages of the Forth family. Structural editing in Forth is just the "move to next word" function that every text editor has. Consider Factor, a modern descendant of Forth. It has the equivalent of Common Lisps macros and even reader macros, plus a CLOS-like object system. It's also a live programming environment. Of course it's a niche language, with nothing near the support you get in the Common Lisp world, but I just wanted to point out that your criteria don't necessarily lead to Lisp.