Unix / CLI / shell-like tools for iOS

So you have an iPad, but you are also used to command-line tools from the Unix world, or you have colleagues who use these tools, and you need to interact with them, or... In short, you'd like to access command-line tools from iOS.

Well, I've got good news and I've got bad news.

The iPad as a computer

The iPad is a powerful tool. Especially for the "Pro" versions,it has a powerful processor, more than enough memory and storage space and a sizeable screen. All this while being very light.

But the iPad also runs iOS, which enforces limits on the apps you can run. iPad users are face with a dilemn: they'd like to use it "like any other computer", but they can't. There are three limitations:

Sandbox limitations: an app can only read and write files that are inside its own space. It may do whatever it wants inside this space, but all the other apps are protected. So is the system, although some files from the system are readable, e.g. fonts.
API limitations: iOS runs a version of freeBSD, but some calls are disabled. Mostly, system(), fork() and exec(). In layman's terms, an app cannot start a new app, or launch a binary tool to process data. An app can still start a new thread, but it will be inside the main app and share its memory.
AppStore restrictions: once you have written an app, if you want to distribute it you need to go through the Apple AppStore, which places more restrictions. Originally, apps could not run external APIs or languages, nor load external binaries. These restrictions have been partly lifted, but so far no-one has managed to get a full freeBSD app through the AppStore review process.

In the old times, a possible solution was to jailbreak, at which point you could run any kind of shell and compiler, and thus install any apps you needed. I may or may not have done this with an iPad 1, and installed texlive on it using gammalevel instructions.

Jailbreaking is becoming harder. To the best of my knowledge, there is no solution available for iOS versions above 10.2.1. The good news are:

iOS 8 introduced the possibility to sideload, that is to load any app you want onto any iOS device you own. Assuming you have the source code, that is. Sideloading is possible without a paid Apple Developer license (so, for free) but you have to re-install the app every 7 days.
iOS 11 introduced the possibility for apps to open files inside another app sandbox, and for the user to move files from one app to another. The restriction is that you need user input to open and move files. An app cannot spontaneously create files in another app sandbox, or move them on its own; only the user is empowered; it has to be a conscious choice. You can also open entire directories from another app sandbox, but the user interface for that is a bit hidden: click Select, then click on the directory, then Open.
iOS 13 introduced the possibility for apps to open multiple windows, of which at most three are in the foreground (but any number can be in the background). It's not yet full multitasking (because your app is still using a single process space) but it allows new things for terminals, such as edit in one window and process in another one.

Together, these removed several restrictions. So what's the current state of affairs?

Tools from the AppStore

There are several tools on the AppStore that reproduce command-line tools or programming languages:

WorkingCopy, an iOS port of git. Excellent user interface, easy integration with the big sites (github, gitlab, bitbucket...). With iOS 11, you can edit files in a different app if you want: clone the repository, drag it to the other app (inside the Files app), work on it, and drag it back to WorkingCopy. The next commit will only push files that you have actually changed. (free, ability to push unlocked by in-app purchase. Well worth the price).
Pythonista, an iOS port of python 3. Powerful. You can install stash, a python shell, and use standard shell commands such as ls. You can even use git and pip, to install more modules. Python 2 modules are executed using 2to3. (around 10 $).
Codea, an iOS port of lua. Beautiful user interface. You can write programs in lua, execute them locally, then export to Xcode to transform them into full apps. Codea also includes a GLSL shader editor, with interactive preview. (around 15 $, and worth it).
Swift Playgrounds, which lets you code in Swift, Apple's programming language.
TexPad, an iOS port of TeX, with local typesetting. It produces pdf from TeX, but not using pdftex because it is GPL. So you might have to edit your source to remove references to the pdftex driver. With iOS11, TexPad can typeset files inside another app sandbox (yeah) but not include files from these (because sandbox restrictions). For pictures or multiple files project, you need to copy the directory to TexPad sandbox. (around 15 $).
Carnets, which lets you edit and run Jupyter Notebooks on iOS, locally. Jupyter notebooks are a combination of Python source code, result of code execution and explanatory text. Very useful to test hypothesis or run small snippets of code. Or for more serious work.
Juno, another tool to run Jupyter Notebooks locally on iOS devices (15$).
Shadertweak, shadertoy for Metal, which lets you edit Metal shaders and see the results in real-time. (not on the AppStore, you'll need to sideload it).
iVim, an iOS port of the Vim editor. With iOS 11, you can edit files in other apps sandbox, and then use the original apps for compilation. You can also write small programs in vimscript, and import Vim plugins using other apps, such as WorkingCopy. Inside iVim, netrw lets you organize your files, move them around... (free ; you can also install from source code: https://github.com/terrychou/iVim) (Edit: as of october 2019, I am unable to compile iVim from source with Xcode 11).
Blinkshell, an iOS ssh/mosh client. Lets you connect into your other computer, to run commands, edit files, etc. It has great user interface for key management and storage. (around 20 $, and worth it. You can also install from source code, free: https://github.com/blinksh/blink/).
iSH, a full Linux running on iOS, using software emulation of x86.
OpenTerm, a command line interface for iOS. It lets you execute basic file commands (type "help" to get the full list: ls, mkdir, rm, tar...) (Edit: OpenTerm was removed from the AppStore when its author was hired by Apple)
LibTerm, which is pretty similar to OpenTerm (but is still on the AppStore).
a-Shell, a multi-window command line interface for iOS, with TeX, Vim, Lua and Python (full disclosure: I'm the author).

Sideloading

All these apps are great, but they don't do exactly what I need. Most of my colleagues use hg for source control, I use luaTeX for typesetting, etc. So I looked into sideloading. The good news is, you can port almost anything to iOS, if you are ready to do a bit of code editing. The bad news is, well, you'll have to look into the code of your favorite tool or language.

The idea is simple: first, you need a host app, which will take care of the user interface. Second, you need the source of your favorite tool (ls, tar, curl, TeX...). Edit the tool source so it creates a library instead of an executable, and link with the host app.

You will have to change the tool source in the following ways:

Rename the main function to something different, like ls_main.
Change all calls to exit() to calls to pthread_exit().
Rewrite calls to err and the like so they don't call exit.
Initialize all variables and flags at the beginning if they are not already initialized.
Explicitly release all memory at the end of the program if it is not already released.
Edit the Makefile or project so it creates a library instead of an executable.

The idea behind all these changes is that your tool will run inside the host app, not outside. So a call to exit() kills both the tool and the host app. Similarly, the memory allocated for the tool remains active inside the host app. If you don't release it or set it to zero, it will be there the next time you run the tool. With the values from the previous run.

Once you've done this, you need to edit the source of the host app, so that when you type ls, it calls the function ls_main in a thread (for exit), and link it with the library you created in the first step.

I've done this porting with three different apps, two of which (Blink and OpenTerm) are on the AppStore. The AppStore versions follow the restrictions imposed by Apple and other license holder: a smaller set of commands are available. The full versions have more commands and are available as systems to clone, compile and install:

Blinkshell, my fork of blinkshell using their open-sourced code. This gives a complete shell, with only command-line interface.
iVim, my fork of iVim, using their open-sourced code. Here, the focus is on file edition, but iVim can run commands on the files as I edit them (TeX or Python compilation, for example). (Edit: does not compile with Xcode 11)
OpenTerm, my fork of OpenTerm, using their open-sourced code. Compared to the AppStore version, more commands are available, including Python, Lua, pdflatex and lualatex.
a-Shell, my own terminal, now available on the AppStore. It uses hterm for xterm emulation, making it possible to run Vim and iPython. The full set of commands is available: python, pdftex, luatex, lua...

What sort of commands are available?

Well, you need the source, and you need to cross-compile. So open-sourced tools, with a reasonably modern implementation and not relying too much on autoconfig tools. Apple OpenSource initiative is a great starting point. You want the source for the MacOS tools, obviously.

What I've already done:

File commands: ls, touch, cp, rm, ln, mv, mkdir, rmdir, df, du, chksum, chmod, chflags, chgrp, stat, readlink, compress, uncompress, gzip, gunzip.
Shell commands: cd, setenv, pwd, env, printenv, date, uname, id, groups, whoami, uptime.
Other commands: awk, sed, tr, grep, cat, wc, curl, scp, sftp, tar.
Python: a scripting language on iOS. And with it, the ability to run mercurial (hg). You can run pip and install new packages. (Edit: Python2 is EOL on 31/12/2019, and the status of mercurial after that date is unclear).
Lua: another programming language available. See my lua_ios package. .
TeX: namely, pdftex, luatex and bibtex. In my lib-tex package.

All these commands are available as a single framework, iOS_system, which provides a drop-in replacement for the system() call in the source.

For programs like python and TeX, what you have with the library is only the equivalent of the executable. You still have to install the packages and files, probably by transferring them from an existing installation, because running make install is not possible. Which is why you have to start with the other commands: curl and tar to transfer the files, mkdir and mv to move them where you need them, rm to remove failed attempts.

Also, with TeX, you'll have to re-generate the format files with pdftex --ini --etex pdflatex.ini and the likes.

What are the limitations?

You cannot write in the ~ directory, only in ~/Documents/ and ~/Library/. Most Unix programs assume the configuration files are in ~. So you'll have to redefine $HOME or tell your tools to look for their configuration files elsewhere (e.g. setenv SSH_HOME = $HOME/Documents/).
Anything that requires root privilege (traceroute) is impossible.
You can't access files outside your app sandbox, but you can transfer files and directories from one app to the other (using Files or open-in-place).
You cannot run multiple instances of the same command simultaneously. Which includes some python packages where pip runs python on the setup.py file, then executes the result of the modification. This means a python script running inside a python script, and is currently not possible. You can still install the packages by hand, using python setup.py install.
I did not manage to compile perl, which is a bummer because many TeXlive scripts depend on it.

Possible extensions

Having Haskell or OCaml with the same setup seems quite feasible. Both have been cross-compiled to create apps for iOS. I don't need them, so I won't do them, but they're low hanging fruits.
I've managed to port LLVM and Clang, and compile C source files. I've also ported LLI, and interpreted the bytecode generated by Clang. So I can compile and execute (sort of) source files entirely on the iPad. Right now, the interpreter is slow, and it only works for C files. LibTerm makes this available on the AppStore.