One thing I've been thinking of: how about making typed file descriptors?
Besides stdio and stdout, one of the best-accepted conventions on modern operating systems running a graphical shell seems to be the clipboard.
Programs can offer different mime types trough a clipboard, so why not do so trough a pipe or file descriptor? That would bring us a bit closer to powershell, while remaining backwards-compatible.
Instead of xclip in the above example, one could use an image viewer, a multiplexer a bit like tee (1) but to redirect to standard input, etc. One could force a pipe to filter out some mimetypes, or use a program dedicated to that. One could imagine bidirectional communications over a channel.
"first" starts up a HTTP server, "second" is a HTTP client which makes a standard HTTP request to "first" declaring in the headers what content types it can support, and "first" returns a standard HTTP response declaring in the headers what kind of content it is sending. "Second" is also a HTTP server for "third" in the same way, and so on down the pipeline. All the OS has to do is coordinate a free TCP port for each pipe. The HTTP client can simply be a lightweight headless Chrome Electron bundle, and the whole pipeline gets all the benefits of HTTP (standard UTF-8 encoding, compression), with only a small overhead of cookie policies, cross pipe scripting, modern auth tokens, 2-factor authentication between client and server, SAML, and a tiny surface area increase for security concerns but only a few dozen MB or so.
Plus the pipeline then has ongoing two-way communication up and down it.
Obviously the following can't be taken at face value:
> lightweight headless Chrome Electron [...] with only a small overhead of cookie policies, cross pipe scripting, modern auth tokens, 2-factor authentication between client and server, SAML, and a tiny surface area increase for security concerns but only a few dozen MB or so.
However, there's probably much to be learned from the history of the web when it comes to security and compatibility between different implementations and successive evolutions of a protocol. http certainly has merits, given its long history. It isn't flawless either, of course.
Even features such as compression could make sense at first glance, though they would be a better fit for a dedicated tool like piping to zstd, since you control both ends of a pipe. Range requests, multiplexing, server-sent events, etc, could have some use. And it isn't like making a http server or client is so hard with a small library, given you already pull in libc or worse.
http probably wouldn't be suitable as-is, certainly not over TCP (though the fopen/fclose dance is quite close to it), but making it a fully fledged bidirectional protocol could be interesting, especially if you can somehow build on top of the system IPC on a microkernel. It could be a way to forward hardware access and such.
The problem isn't so much finding a way of passing metadata but how do you retain compatibility with existing GNU / UNIX tools? It would be easy to create an entirely new shell (like Powershell) that throws out backwards compatibility but that wouldn't be very practical on Linux/UNIX when so much of it is powered by old POSIX standards.
I addressed this by having a wrapper around pipes. Anything that connects out to a classic executable is a traditional POSIX byte stream with no type data sent. Anything sent to a command that is written to support my shell can make use of said meta data.
As for handling graphical components, that's a little easier. Many command line programs already alter their behaviour depending on whether STDOUT is a TTY or not. I take things a little further and have a media handler function, called `open`, that inlines stuff in a shell that can be (it will render images in the terminal) but if STDOUT is a TTY then it will pipe the image data instead.
> The problem isn't so much finding a way of passing metadata but how do you retain compatibility with existing GNU / UNIX tools
Honestly? I'd have suggested a new syscall that provides an interface similar to isatty(). It seems a matter of adding a few constants to the ioctl syscall, prototype it and prove it is useful, then upstream it. Better make a crude prototype, see if that's any useful, and ask upstream what they think of it.
If the new ioctl isn't available, nothing changes, programs continue exchanging whatever they expect. If the new interface is available, most programs won't know about it, and would continue their business as usual. However, programs that know about it could get more metadata trough it, and communicate in a smarter way. Looks like a standard way of introducing new functionality, can be deprecated if needed, and keeps backwards-compatibility, at the expense of a tiny bit of complexity (dwarfed by having to support multiple formats, which some tools already do).
The way you do it seems quite fine too, not sure which would be better.
Staying with traditional file descriptors also has the issues that you are limited to unidirectional communication, and only one file descriptor, so maybe you need a protocol like html, as a sibling joked.
> As for handling graphical components, that's a little easier. Many command line programs already alter their behaviour depending on whether STDOUT is a TTY or not.
Well, that's kind of my point: you could output more than curses interface if you wanted, something like an xml ui file, or a list of commands on a side channel. But I make no judgement on this: it was just to illustrate what kind use-cases this would enable.
A new syscall would definitely be a better engineered way of doing things but pragmatically I'm unlikely to convince Linux, let alone Apple and the BSDs that they should all introduce this new syscall when there isn't a wealth of software that would use it. And I'm definitely not going to get any software written for it if there isn't a syscall for developers to code against. And that's before I've even addressed the problem that most people are happy enough with bash/zsh that any new shell is already fighting an uphill battle for adoption.
So with those points in mind I think this is one of those cases where worse is better. But kludging things a little it means I can have greater ownership of development which means I can actually deliver something that people can use right now. If it gains popularity then maybe I'd have more clout to push through new syscalls, but I certainly wouldn't have that with "only" 500 Github stars :)
> Well, that's kind of my point: you could output more than curses interface if you wanted, something like an xml ui file, or a list of commands on a side channel. But I make no judgement on this: it was just to illustrate what kind use-cases this would enable.
You can do that with my shell right now. The issue is you'd have to write an `open` handler (because how would the shell or terminal know what to do with pseudo-random data if you don't have some kind of handler?) but PNGs, JPEGs, etc will be rendered with sixel/or whatever proprietary support the terminal emulator has for inlining images if `open` is used and STDOUT is a TTY. So it's not just curses interfaces though the principle is similar.
In one experimental module I even went as far as having web pages render in the terminal if STDOUT was a TTY. So instead of making a HTTP(S) request and displaying the bytes, it would render the webpage, images and all. Only falling back to a standard `curl` operation if the command was piped. This never made it to production code though, it was very Linux specific code and required a heap of 3rd party software to be installed just to work. But it was a great example of just how much you can already do with some terminals.
Come to think of it, you could abuse NTFS Alternate Data Streams for this.
Imagine:
> # See alternate data streams
> Get-Item diagram.dot -Stream *
Stream Length
------ ------
:$DATA 123
text/plain 4444
image/svg+xml 65432
image/png 456134
> # Send the whole bulk
> Get-Item diagram.dot -Stream * | ...
> #... or pick just one:
> Get-Content diagram.dot -Stream "image/svg+xml" | Send-Email ....
Would have to ensure everything is aware of the streams and the convention, though, so that you wouldn't accidentally truncate them when copying files around.
Not just NTFS, but pre OSX Macs would do stuff like this. Like a formatted document would have a text/plain fork, and the formatting information in another fork. Or an executable could keep it's symbol table or relocation information in other forks.
I've tried it with ancient font suitcases -- the ones that pre-date dfont (data fork only) fonts.
It's somewhat ironic that all desktop file systems these days support myriad data streams on a file. I've implemented some on NTFS, played with extended attributes across tar archives between Linux and Mac and Windows. It's all there, but no longer used.
Ah, that's interesting, thanks for mentioning it. That's pretty much what I mean, but on arbitrary file descriptors instead of files.
I agree that both ends need to be aware of these extra formats to make use of them, but just like with clipboards, programs could always fall back to pretending only text/plain exists :)
Typing, in the programming language sense, is a major can of worms that a general purpose shell usually avoids with a deliberately bare bones type system (e.g. text, binary, directory and symlink).
How can a shell do anything serious with asserted file types that can be missing, wrong, malformed, unknown or mendacious? It's hard to go beyond offering a best-effort "open with the associated program" application as an alternative to vi, cat etc.
> How can a shell do anything serious with asserted file types that can be missing, wrong, malformed, unknown or mendacious? It's hard to go beyond offering a best-effort "open with the associated program" application as an alternative to vi, cat etc.
How would that be any worse than the status quo, though?
Programs already output, depending on their invocation and the utility at hand, binary data in arbitrary formats, shell escape sequences, text, ASCII art, etc.
The only issue I can think of is for portability of the scripts, if a program gets updated to support a new type as input or output, and that causes issues compared to the testing environment. But that's arguably already an issue too, and it would be rather simple to annotate expected types.
The cost of specifying MIME types or the like can easily exceed its value, which is very low because we are already specifying file types (to applications that can deal with more than one) well enough with file extensions (undependable, but vastly more convenient than new infrastructure for new metadata), with self-describing or recognizable file formats, and often explicitly.
A possible way of doing what you describe might be to add an extra file descriptor or two when invoking processes. Along side the normal stdin, stdout, and stderr, you might have a couple more where a protocol can be used to negotiate and share typed data. It might even be possible to do it a backwards compatible way. (If in an app the extra file descriptors are missing, then fall back to old school stdin/stdout.)
Thank you for the blog post, it resonates quite well with me.
I thought a bit about this, but that becomes a bit complicated without additional metadata on a file descriptor (is it really that additional interface, or just a random FD we inherited?). It might be possible to achieve in a robust way with mount namespaces and named pipes, but that's another can of worms...
That said, it would likely work 90% of the time that way.
I'd imagine that in a shell context, most of your FDs would be closed on exec unless explicitly passed.
That said, why not use env vars which tell the child process what to expect on which file descriptor numbers?
Depending how crazy you want to get, you could also use unix sockets to pass around FDs instead of inheritance. In that scenario, a process would be passed a unix socket either by a socket pair file descriptor or the shell would have a named socket which it tells child processes about using an env var. You can then use the sendfd functionality of unix sockets to pass around arbitrary file descriptors with arbitrary metadata.
You could also use the unix socket to instead have the child process optionally ask the shell what the descriptor's metadata is. That way you'd keep backwards compatibility with unix tools, but the child process could sendfd the file descriptor back to its parent to ask for data about it. The parent could look it up by calling fstat on it and comparing the fstat data to the other file descriptors it sent to the child.
But regardless, when you can't add a new syscall and need to do file descriptor stuff on behalf of another process, unix sockets + sendfd can really start to look like a userspace syscall interface which you can do a lot with.
I pay attention to things in this area as I've been working a terminal which aims to do new and interesting things with text and also non-text datatypes. See "Shell integration" and below at https://extraterm.org/features.html
Integrating some new shell projects like murex and nushell with Extraterm is on my TODO list.
The clipboard nowadays is populated lazily (you do not want that dot call to generate Postscript, svg, pdf, png, gif, and whatever else it can generate while, in the end, you’ll need only one of them), so you would have to keep those processes running until you know nobody needs the data anymore.
That would be quite a change from the traditional pipe model.
Oh, but the traditional pipe model can also produce data lazily (wait until a given FD is open before producing output), so that wouldn't be that much of an issue.
That wouldn't change much from the pipe model. At least on wayland, wl-copy forks to the background. If you wanted the producer to finish producing data and exit, surely you could make a flag for it in the receiving program, or pipe it trough one that does.
This is different. You need communication back through the pipeline to inform the data producers what data you want.
Also, in your example ending in | xclip, xclip cannot know what format data it has to ask from dot until it gets pasted somewhere. Alternatively, it would either have to guess what format(s) the user would like to paste later or it would have to cover all bases and ask dot for output in every format it supports.
I'm interested in progress in this area, but I'm disappointed the article seems to ignore the existence of powershell, which does some things different to traditional linux shells -- I would at least like a discussion of how this related to powershell.
I work in a Windows shop (security compliance reasons for federal government contracts). Before I got this job I was pretty much only developing in Linux.
Honestly? Powershell is pretty cool, though it feels a little strange to admit that. It has it's own very powerful syntax and virtually all of the common Bash commands are now aliased by default, so it's pretty smooth transition for the basic stuff.
The "key issues to address" list explicitly calls out many of the challenges Powershell manages with its object-based view of the world. It looks like Arcan's SHMIF layer is meant to let developers using Arcan capture rich data and metadata before it gets smooshed into the VTxxx emulation layer and smeared away in a sea of screen formatting characters.
Where Arcan cannily (IMHO) differs from Powershell is it acknowledges the transition from existing toolchains will be organic instead of demanding wholesale operational changes to a different processing model. So it splits the data and metadata processing pipeline, exposes it at key points to the developers using Arcan, and lets an emulation layer cover legacy use cases, while enabling Arcan developers to create much richer data representations underneath the covers that can be tapped if desired by users via Lua manipulating the many different Arcan components (I think). This opens up hybrid GUI/TUI/other-interface best-of-breed/mixed-mode options that are unreasonably challenging to implement (especially with sync logic) with current UI stacks.
I like this approach more than Powershell's, but it remains to be seen how Arcan handles the guidance of the data and metadata ontology and resultant taxonomy; it is a much deeper problem space than just defining a default namespace, a way to modularly plug into it with extensions, then standing back and letting developers have at it. The way Powershell handled it, introspection and discoverability are out of band instead of a batteries included experience, and module management is still split out of band between Nuget, PrivateGallery, custom solutions, hand-curated Git+Jenkins lash-ups, etc.
For my small-scale, personal projects like for my family's intranet, the lack of an opinionated default for at-scale management of these issues is not a problem. But at my clients with dozens, hundreds and even thousands of developers, it presents software engineering management challenges. Solvable, just a developer experience hygiene factor I'd rather not deal with because solving it adds zero business value, it's just the cost of doing business.
There was no point in bringing up powershell at the time, its legacy is an entirely different beast from vtXYZ and would invite the next "why isn't hyper.is covered" and "why isn't upterm covered and ..."
The fabric under attack was the bastard display server protocol itself and not the parts that tie in with the strong points of Powershell, i.e. shell UI design or data exchange format specifics.
I feel that the closest we can get to a Lisp machine with modern components right now would be to get PowerShell, Emacs and StumpWM (or EXWM) to deeply interoperate.
I mean it's possible to run PowerShell on Linux, but PowerShell has some pretty significant rough edges, including a pretty painful performance story for something I want to use as a UI.
It is also a matter of culture, as mentioned you could have a UNIX shell being structured, able to directly interact with shared objects, IPC with running applications, yet very few do it.
Is it possible to return structured data in PowerShell from non-.Net languages? I am genuinely asking here, it’s been years since I’ve poked it. If not though that seems like an enormous stumbling block.
That sounds interesting, but either programs deal with schema transforms, or we'll have to write a lot of that kind of logic in XSLT or jq (kind of the same stuff we do with cut, sed, awk, sort and grep for unstructured streams). Not sure it's that useful a feature unless the domain of the programs is narrow.
I think it could be useful - from PowerShell's perspective, all you need is to wrap your "classical" executables to somehow accept and emit .NET objects, and you can carry on from there. Importantly, no more writing parsers in the pipeline or in your scripts.
I could see a whole library of such wrappers being built for common executables (and new popular OSS CLI tools could provide their own wrapper along with the tool).
You can wrap your .NET, JVM, Python, Perl or C functionality in a small program that links to the library you want, that deserializes your XML or JSON, reads your command, and acts on the incoming object stream, spitting it to stdout in a neutral, preferably human readable, format that can then be piped into the next command.
Oh! Wait! This is what tools like cut, jq, grep, awk, and sed to.
> Oh! Wait! This is what tools like cut, jq, grep, awk, and sed to.
Yes! Except, you have to use them to write one-shot parsers that are specific to both what you're piping from and what you're piping to! For n commands you may want to arbitrarily mix, that's n² parsers to write.
> And they won't tie you to a certain object model.
From the point of view of a system user, tying to a certain object model is a feature. It facilitates commands interacting well with no parsers in between. But even if you don't have a common object model, just wrapping the commands to produce structured output can already be helpful, because the parsers you'll be writing to glue commands will operate on structured, higher-level concepts. It's easier to map between two different representations of what a "file" or "process" is, than it is to manually dig them out of an unstructured text stream.
To use a stupid example, assuming hypothetical "kill-and-announce" that echoes a process name and sends a SIGKILL to its PID:
ps aux | awk '{ ...extract columns... }' | sed 'rename columns' | xargs kill-and-announce ...
This is one of those eternal debates I think: on the one hand you have OOP and certain sorts of FP advocates advocating for extremely rich data models that eliminate certain classes of errors by design. On the other hand, you have the Unix notion of using a very generic data type (see also Perlis’s “rather 100 functions on one data structure than ten functions on ten data structures”) to maximize flexibility, at the cost of a certain amount of footguns.
Personally, I’m discovering more and more that CL’s REPL in SLIME is my preferred shell in part because the CL ecosystem has a rich array of operations on a small number of data-types (without forcing everything into binary streams), but also the ability to define ad-hoc datatypes as needed and a powerful system for manipulating them and integrating the custom data types into existing protocols.
Also, the claim that you need n^2 encoder/decoders isn’t actually true: conventions around data formats can reduce it to O(n) in the number of conventions.
Myself I know I'm stuck here. I've been thinking in circles for a while. At the moment, I'm leaning towards rich type system and structured data, but I'm not sure if it can be made fully convenient for the shell.
Do you know if anyone attempted to summarize this debate and distill some conclusions, or reframe it in a way that would allow to make progress?
> I’m discovering more and more that CL’s REPL in SLIME is my preferred shell
In terms of command line or scripting? SLIME is awesome, particularly with its interactive Inspector :). And I wrote my fair share of "scripts" as CL functions. But I can't seem to get used to using the REPL as a shell; I feel something is missing. Most likely convenience functions (that I didn't bother to write for myself yet).
That whole paragraph you wrote applies to another Lisp system, though: Emacs. In recent years, I've been leaning on it more and more for day-to-day operations. For instance, I rarely use shell for file operations - I mostly stick to dired. I end up scripting common operations straight in Emacs Lisp, and bind them to ad-hoc UIs (using the Hydra package). And of course utilize well-known packages like Magit (for Git). This recent experience makes me think that we could pull more common shell interactions and turn them from 1D to 2D interface. And I keep dreaming about merging it with PowerShell, in a way that allows Emacs to ingest and manipulate .NET objects nicely, while at the same time exposing its internals to PowerShell.
> Also, the claim that you need n^2 encoder/decoders isn’t actually true: conventions around data formats can reduce it to O(n) in the number of conventions.
I think this assumes conversions are lossless, which isn't the case in UNIX piping. When you invoke "tools like cut, jq, grep, awk, and sed", it's almost always to filter out most of the text in the pipeline.
To be honest, I’ve been thinking about a shell based on the JVM. Especially now with GraalVM, you can have truly polyglot programs running on top of it, with inlinig cross-language boundaries! But of course native code execution should be “bootstrapped” into that world somehow. (Actually, it can use LLVM IR and that as well can run a C function on js objects which is phenomenal, but people do want to run native software from a shell)
While you'll be able to invoke methods on returned objects, if you are piping stuff, you'll still need to deal with hammering the output of one program into the input of another.
And neither of those are native parts of shells but external applications, dare I say, just like the C# wrappers. (Especially parallel, since it's nowhere close to default install, and has very pesky license notice).
Yeah, but & is a thing and will do the job nicely as long as you don't try to pipe output to the same file... also, you can invoke pretty much any command with xargs... the issue was escaping escaped stuff with PowerShell. Hopefully things have gotten better.
Yeah... shouldn't need to but... there are issues with escaping command line parameters with invoke-command and ForEach-Object that prevent parallelizing older cmd-style command line applications. The work around is to write a wrapper in a compiled language and use PowerShell to invoke the wrapper... There's a lot of really cool stuff in PowerShell.
> there are issues with escaping command line parameters with invoke-command and ForEach-Object that prevent parallelizing older cmd-style command line applications.
Can you tell me more? So I know what to expect when I hit such issue myself.
> The work around is to write a wrapper in a compiled language and use PowerShell to invoke the wrapper...
Wonder if you need to really break out of PowerShell for this.
Recently when doing some scripting at work, I ended up writing .NET code straight in the script. It looks like this:
$someCSharp = @"
using System;
//...
public class Foo : Bar {
public override object Baz(Quux arg) {
//...
}
}
"@
Add-Type $someCSharp -ReferencedAssemblies C:\Path\To\Some\dependency-I-need.dll
This makes PowerShell compile your code into an in-memory assembly and make it available in the session, just as if it was an external .NET type. You could probably make a module out of it. Hell, cmdlets are .NET types underneath, you can make new ones from C#, so you could probably make such ad-hoc .NET classes available as PowerShell cmdlets.
I suppose using a proper .NET development environment for this would be more kosher; I arrived at the solution described above because my needs were simple, like binding .NET regex engine to SQLite (to get REGEXP operator in queries), or convincing Invoke-RestMethod to stop checking certificate validity.
Still, I love the capability, the fact that I can just drop into C# in the middle of the script. It's one of those things that make me think of Lisp machines and Common Lisp the experience of being able to deeply interact with your environment, and the joy of writing "scripts" that are automatically AOT-compiled (like with SBCL implementation of Common Lisp).
The general structure of the quip/aphorism/joke/snark pre-dates the industrial revolution dating back (at least!) to Edmund Burke's [1] "Those who don't know history are doomed to repeat it."
LOL, I knew I knew the quote, didn’t remember from where and WikiQuote gave me that source. Should be corrected but I’m on mobile at the moment, I’ll do so later if no one does in the meantime. That’ll teach me to trust the internet.
Programming languages are tools. Tools need to be used to build wonderful works of art or industry to be useful or have any realistic shot at being considered great tools.
The fact that the overwhelming majority of great applications, services, any kind of software we create is not written in Lisp should at some point in the first century after the creation of Lisp, lead Lisp afficionados to some kind of enlightenment, one would hope.
Unlikely, since it's already been 60 years, so they only have 40 years left.
The OP had identified problems with the existing approach, and proposed a concrete solution in detail. Dismissing that with a pithy quote because it vaguely matches a common pattern is low effort and needlessly dismissive. The quote stinks anyway. Unix is not the pinnacle of computing.
Honestly one could argue that DOS has some merit beyond Unix. It’s simple command line interface expects users will program in other languages like C or assembly. It’s more directly interaction with the machine. Sure it had limitations like file name lengths etc back in the day, but I think many Unix shells are overly complex and ought to yield to more suitable languages like Perl.
> It’s simple command line interface expects users will program in other languages like C or assembly.
I think the point here is that DOS explicitly rejects the composition that pipes enable, in exchange for the ability to just hand "the whole system" over to a programmer.
But the main point, that DOS was basically a thin program loader that "got out of the way" is sound. Think of it like a UEFI shell environment, with 640k+ limits.
I feel towards the unx shell the same way I feel towards Vim: there's quite a bit of cruft, some poor decisions, quite a bit I'd do differently if I decided to write my own implementation from scratch. But on the other hand it mostly works well and I've learned over the years how to work around the awkward parts.
It's easy to make a laundry list of the ways the traditional unix shell falls short. The problem is creating an alternative that's superior enough to warrant learning something new and consensual enough that it can get mainstream appeal.
The article acknowledges that:
>There is arguably a rift here between those that prefer the ‘shove it in a browser’ or flashy UIs that animate and morph as you interact, and those that prefer staring into a text editor. It seems to me that the former category gets all the fancy new toys, while the latter mutters on about insurmountable levels of legacy. What I personally want is many more “one- purpose” TUIs and for them to be much easier to develop. They need to be simpler, more consistent, obvious to use, and more configurable. That’s nice and dreamy, but how are “we” supposed to get there?
But then I argue that they jumps the shark a bit. As far as I can tell the Arcan project seems more preoccupied with reaching feature-parity with Xorg than actually improving the CLI. And I'm sure for some people that's great, but for me personally that makes it a funny novelty thing that I don't see myself using. It reminds me when I first saw compiz ~15 years ago, where you had virtual desktops on a rotating 3D cube with all sorts of weird visual effects[1]. It looked cool, but I never actually used it because it wasn't practical and I had no use for it.
On the other hand I'd love for a better CLI implementation, better pipes, easier debugging, a more expressive scripting language, better task control, better feedback.
But then we went full circle to the core of the issue: everybody agrees that the unx CLI sort of sucks, but we don't really agree on what exactly needs fixing.
That's odd, I found the Compiz cube to be helpful actually.
As a graphics effect it's about as basic as it can get -- even the venerable GLX Gears is more complex. And I found it to actually increase usability because it visually and very intuitively showed you the movement, so it made organization a bit neater. It made it easier to keep a mental map of what was where, because it could be arranged spatially.
If anything, it didn't go far enough. I wanted more faces, and the possibility of working in a "zoomed out" state, for when I need to look at a whole bunch of stuff at once.
The real novelty stuff to me were the Enlightenment effects like the water effect on the bottom of the screen -- now that chewed up CPU and did nothing of actual use.
I liked the cube for the same reason - virtual desktops just kinda clicked when I could rotate the cube to get to the stuff running just off screen. I've never really bothered with them before (or since).
You could make the 'cube' have an arbitrary number of faces, but I don't think there was the zoom mode you talk about.
True, you could not when using compiz with gnome either, as I did back then.
You could add more faces to the cube so that it became a slice of a pentagonal prism, hexagonal prism, heptagonal, octagonal etc. Or take one away so you were working on the sides of a triangular prism.
What I find an interesting is command interfaces in GUI applications. For instance ctrl + shift + p / cmd + shift + p menu of visual studio code or the new windows terminal, or the commands in slack. A GUI gives way more options for displaying information especially for auto-complete even while staying in a keyboard-only interface.
This seems to be more about text-based user interfaces rather than command-line interfaces. I wonder if the author has considered Emacs? A lot of people think of Emacs as a "text editor", but that's far from the truth of what it is. It's really a Lisp interpreter and toolkit especially designed for making text-based user interfaces.
It looks like the problem space the author is addressing is broader than Emacs; they seem to want to make TUI's one of many different, first-class-citizen presentation layer options, interacting with many different input mechanisms. A Lego approach to UI frameworks basically as far as I can tell.
It would be interesting if there was an Arcan-philosophy UI framework in CL for an updated Climacs on top of a bare metal Lisp environment, for a turtles-all-the-way-down CL experience. It would open up interesting interactions with Emacs that are an unreasonably big pain to implement today, like buffers that manipulate view ports into observability data under a pluggable UI that renders graphically a graph data-oriented view of the discovered structure of the data under an investigation taking notes into a Jupiter-like notebook with rich multimedia data, for instance.
I'd actually be surprised if you couldn't do some of this with emacs today. Falling back to the text is just hella repeatable in ways that most binary forms are not.
All the Arcanspace stuff is cool. The problems are real.
The pieces and principles force me to think about many of the "skills" I have acquired in order to work on various shells and desktops as burdens.
I can't bring myself to change to (or develop) pipeworld/durden-esque environments (yet?), I am drawn to them because I have a strong, natural alignment to the principles the author espouses and develops against.
One issue with windows command line I’ve run into recently is text rendering performance. Running cxxmatrix at a high resolution brings even a modestly powerful machine to its knees. cxxmatrix crashes alacritty, but I have found that running VcXsrv and urxvt from wsl yields much better cxxmatrix performance than when running from wsl.
Here is a related issue. There are many cousin issues.
This is one of my favourite itches to scratch. Initially it was inspired by TermKit and since then I pay attention to such endevours.
Intuitively I feel that we can do better on the presentation side. It is also tempting to refactor pipelines to use structured data - at first sight this would help avoiding all this plaintext parsing glue which hides otherwise elegant logic in your pipelines.
However, digging deeper into possible solution frustration creeps in :( In my case it started when I tried to collect real world use cases for "rich terminal":
cat some_data | plot .... # hm... not that I use this frequently
.....
psql -c "select * from customer;" # hm.. I do need some kind of grid control here.
Heck, do I really need to bring such a complex thing here?
Ok, I'm a bit stuck, lets postpone this and think about the implementation.
The obvious choice to make some shortcut such as HTML. However, here comes JS and I'm totally unsure that this behemot is what I wanted. Improving VT100? Most likely this will end up with a brand new 15th standard. No luck, lets postpone.
Structured data. Lets scratch another itch - model some basic commands using Clojure to build commands and represent structured data. ls, find, cat, grep. After a good start with producing file tree, flattening and filtering it with core Clojure function it becomes clear that I need some kind of DSL for "find" so that it would be as concise and easy type as with "find".
Блин!!!!
Lets wait for the next such post on HN to start over.
I'm not sure I agree with some of the key issues. In particular, the points on code/data and user input versus pasted input.
By and large, I like knowing that I am in control of what I send a program on the terminal. It is actually maddening when I go to paste into a word processor and it doesn't just bring over the text. Or when I go to script a program, but it doesn't act sanely when driven by a script instead of by a user. Where sanely is, exactly the same.
There are times when I have taken advantage of eshell to inline images and such. That said, I don't use many tui style programs, other than emacs. And I greatly appreciate that the majority of emacs buffers are textual buffers.
I think emacs is a good example of a terminal program (not really command line) that made a good success out of becoming more gui and I think it shows a lot of the limitations of the terminal emulators. In a gui you easily get all the colours and not many weird duplicate keyboard shortcuts (exceptions are eg TAB vs C-i and M- vs ESC). You can have images and clicking and middle click paste and multiple fonts and some OS integration. But the interface still remains a fundamentally mutable text-based thing.
yes, but that's exactly why we are stuck and can't evolve.
if we want to get something better than current terminals, a new protocol and client are needed.
take mosh for example. it uses a different protocol than ssh too, which adds some interesting features. and it replaces ssh (except for relying on ssh to make the initial authentication)
there is no reason why another alternative to ssh could not be created. if it's good, it will catch on. just like mosh which is slowly growing, and also like ssh was in the very beginning when it slowly grew to replace telnet.
fair point. actually, i think i may have misunderstood the problem. ssh can tunnel anything, so whatever new protocol is devised, it should actually work though ssh.
also i am not disputing that a new system could not be decoupled. i am just concerned that it should not be required to work with ssh if that becomes a limiting factor.
architecture astronautitis is certainly bad, but maybe it's possible to create an implementation with a limited subset of features that are actually practical.
this is interesting -- I was expecting to see a replacement for existng ttys, and was planning to complain about ssh forwarding, redirection, etc... instead, they explicitly say:
> Recall that the prime target now is local text-oriented, command line interfaces – not changing or tampering with the awk | sed | grep | … flow, that’s an entirely different beast.
So I'd call this a "TUI" (text user interface), and the proposed schema replaces X11 or Wayland, not xterm.
Besides stdio and stdout, one of the best-accepted conventions on modern operating systems running a graphical shell seems to be the clipboard.
Programs can offer different mime types trough a clipboard, so why not do so trough a pipe or file descriptor? That would bring us a bit closer to powershell, while remaining backwards-compatible.
Instead of xclip in the above example, one could use an image viewer, a multiplexer a bit like tee (1) but to redirect to standard input, etc. One could force a pipe to filter out some mimetypes, or use a program dedicated to that. One could imagine bidirectional communications over a channel. Basically, building on the strengths of UNIX.