The setup of completion is an essential part of creating a well-behaved script. Unfortunately, the setup has several obstacles, mainly that each shell handles completion differently. In this post, I hope to unravel the problems and show solutions to setting up completion for a custom script in several frequently used unix shells. The first section simply states the solution, its usage, and references used to accomplish these. In the second section we show a detailed progression which lead to the solution.

First, the final solution

When the completion is invoked we want to consider the command written up to that point, and complete the parameter under the cursor.

In the following example, we are trying to invoke completion while having the cursor at the location of | (pipe symbol). The second line shows us what is the desired input for the completion.

$ test first |  # at the end
in: ('test', 'first', '')

$ test first  "second second"    th|ird  fourth  # in the middle
in: ('test', 'first', 'second second', 'th')

$ test first \
> "second second"    th|ird  fourth  # multiline
in: ('test', 'first', 'second second', 'th')

The solution consists of two main parts. First part handles the shell input and transforms it to the array shown above. The array elements are then passed as arguments to a custom script. We then have to handle script’s output and pass it to completion system.

Second part is shell-independent and consists of the script which is given the written command and its arguments for completion and prints all the words which should be shown in completion.

The major advantage of this setup is that you may write the completion scipt in anything invokable (bash, python, c++, …) and don’t have to care for what shell is running it.

file structure

completion_example/
├── test_completion.py
├── test_completion_setup.bash
└── test_completion_setup.zsh

test_completion_setup.bash

#!/usr/bin/env bash
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" >/dev/null 2>&1 && pwd)"
SCRIPT="$DIR/test_completion.py" # relative to the script location

function _my_custom_completion_func {
    trimmed="${COMP_LINE:0:$COMP_POINT}"
    comp=($trimmed)
    if [ "${trimmed: -1}" == ' ' ]; then comp+=(''); fi
    COMPREPLY=()
    reply=($("$SCRIPT" "${comp[@]}"))
    for reply_word in "${reply[@]}"; do COMPREPLY+=("$reply_word "); done
}

complete -o nospace -F _my_custom_completion_func test

test_completion_setup.zsh

#!/usr/bin/env zsh
DIR=$(dirname $0:A)
SCRIPT="$DIR/test_completion.py" # relative to the script location

function _my_custom_completion_func {
    local cmd_string
    cmd_string="$PREBUFFER$LBUFFER"
    cmd_string=${cmd_string/$'\\\n'/' '}
    IFS=$' ' comp=($(echo $cmd_string))
    if [[ "${cmd_string: -1}" == ' ' ]]; then comp+=(''); fi
    ans_str=($($SCRIPT "${comp[@]}"))
    local ans
    IFS=$' ' ans=($(echo $ans_str))
    _describe 'test' ans
}

compdef _my_custom_completion_func test

References

How it works

Whenever you invoke the completion (tapping <tab> key once or twice) the unix shell (bash, zsh, etc.) invokes its own command which deals with completion (e.g. bash has complete command, zsh has compdef). This command then finds function which is bound to the completed command and invokes it. The invoked function returns a list of words which will be suggested, and potentially some other information. If there is only one suggestion, then it is often substituted right away.

If you require only simple completion, the function for suggestions may be auto-generated. On the other hand, if you want you may write the suggestion function yourself.

There are several unix shells, such as bash, zsh, so it would be nice to write the completion code only once, and make only its registration shell-dependent. The flip-side is that some shells, for example zsh, provide additional functionality, which we might not be able to use with this approach.

Setup script for BASH

BASH provides a basic command-line completion capabilities via the complete command.

Register completion

One of the following commands should be enough for the most basic cases. By adding invocation of one of these on bash startup you will be able to invoke completion of that command.

complete -W "word1 word2" <command_name>  # complete static set of words
complete -A file <command_name>  # complete files from the current directory
complete -A directory <command_name>  # complete directories from the current directory
complete -F _my_custom_completion_func <command_name>  # invoke the given function for completion

Whenever the completion is non-trivial you will choose the last (function) variant. Let’s discuss how to write this function.

The following shows a command with cursor at the position of | (pipe symbol). We tap <tab> key twice and our function will be invoked. Our function will get all the necessary information via several variables, whose content is shown in the example below. (Note the weird spacing to distinguish raw input from the array.)

$ test first    "second second" th|ird       fourth

COMP_LINE:    test first    "second second" third       fourth  # content of the input line
COMP_POINT:   32  # index into COMP_LINE where the cursor is located

COMP_WORDS[]: test first "second second" third fourth  # array of command and its arguments
COMP_CWORD:   3  # index into COMP_WORDS where the cursor is located

You may now extract the word for completion by ${COMP_WORDS[$COMP_CWORD]} and from COMP_CWORD deduce what exactly you should complete.

#!/usr/bin/env bash
# this file: completion_setup.sh
function _my_custom_completion_func {
    COMPREPLY=($(compgen -W "thursday third fourth" "${COMP_WORDS[$COMP_CWORD]}"))
    # compgen is a command provided to make custom completion creation easier
}
complete -F _my_custom_completion_func test

The result is set to the COMPREPLY variable with array of possible word completions.

Now run:

$ source completion_setup.sh
$ test <tab><tab>
fourth     third     thursday  
$ test th<tab><tab>
third     thursday  
$ test thu<tab>
$ test thursday

This is workable only if you can tolerate imperfections. There is an ugly behavior waiting for you, if you try completion from the middle. Let | be the cursor and see what doesn’t work!

$ test | thursday friday
$ test thursday| thursday friday  # :( I would expect list of all options

$ test th|ursday friday
$ test thursday|rsday friday  # :| again

Note, that space after the completed parameter would be nice because by tapping <tab><tab> now, we get nothing since the cursor is still at the end of the parameter

We will fix these issues now by adding 2 lines of code.

Fixing the completion in the middle of arguments

Note that COMP_WORDS and COMP_CWORD are unusable if you want the completion to work if the cursor is not at the end of an arugment. The solution is to take COMP_LINE and truncate it to COMP_POINT.

function _my_custom_completion_func {
    comp_words=(${COMP_LINE:0:$COMP_POINT})
    COMPREPLY=($(compgen -W "thursday third fourth" "${comp_words[$COMP_CWORD]}"))
}
complete -F _my_custom_completion_func test

You will notice that when trying the completion now it works, but does not append a space after completion which is not at the end of the line. This is very weird behavior, which is intrinsic to the complete implementation. We can fix it by appending each word with a space, and disabling the default space adding.

function _my_custom_completion_func {
    comp_words=(${COMP_LINE:0:$COMP_POINT})
    COMPREPLY=()
    reply=($(compgen -W "this thursday third fourth" "${comp_words[$COMP_CWORD]}"))
    # add the space manually
    for reply_word in "${reply[@]}"; do COMPREPLY+=("$reply_word "); done
}
# disable default spaces with -o nospace
complete -o nospace -F _my_custom_completion_func test

The same attempts for completion as before give us the following results.

$ test | thursday friday
fouth      third      thursday

$ test th|ursday friday
third      thursday

$ test thu|rsday friday
$ test thursday |rsday friday

One last thing we notice about bash completion is that it doesn’t work with multiline command, i.e., ls \<newline> --<tab><tab> doesn’t really run the ls completion script. We can test this via printing some debugging message in our completion script, and notice it doesn’t print when invoked from `test <newline> This seems to be problem of bash itself, so there is nothing we can do.

Binding the BASH completion to a script

Our setup can be simply changed to invoke a script, which can be shell-independent. The first option takes more work, but gives more power to the script.

The script should be placed next to this setup script ↗.

However, how do we distinguish case where there is a space after the last argument? The trick is to check if the command ends in space, and append an empty argument if it is so.

DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" >/dev/null 2>&1 && pwd)"
SCRIPT="$DIR/test_completion.py" # relative to the script location

function _my_custom_completion_func {
    trimmed="${COMP_LINE:0:$COMP_POINT}"
    comp=($trimmed)
    if [ "${trimmed: -1}" == ' ' ]; then comp+=(''); fi
    COMPREPLY=()
    reply=($("$SCRIPT" "${comp[@]}"))
    for reply_word in "${reply[@]}"; do COMPREPLY+=("$reply_word "); done
}

complete -o nospace -F _my_custom_completion_func test

Setup script for ZSH

ZSH provides advanced capabilities for command-line completion (documentation ↗). This means that not only it shows possible completions, but can also display additional information to each option in the form of description string.

There seem to be two different ways to setup completion in zsh. Either we create a standardized file, used solely for defining the completion, and autoload it. Or we create a setup script which we will invoke from .zshrc and register completion manually.

First, registering the function is quite easy.

compdef _my_custom_completion_func test

To extract what this complex machinery gives us to work with, we can run the set > vars.txt command and compare results from default zsh terminal with that of completion script function. This gives the following additional variables.

$ test first \
> "second second" th|ird       fourth   # this is second line of the command

BUFFER='"second second" third       fourth'  # only the current line of the command
CURSOR=18  # cursor position in the BUFFER
PREBUFFER=$'test first \\\n'  # contains the command parts before the BUFFER
LBUFFER='"second second" th'  # part of the BUFFER before the cursor (only last line)
RBUFFER='ird       fourth'  # part of the BUFFER after the cursor (only last line)
words=( test first '"second second"' third fourth )

We can now do a process similar to the one used for bash. One cumbersome cookie is to tackle multiline commands. We can acheve this by merging the PREBUFFER which contains already entered lines and the LBUFFER which contains left part of the current line. We substitute newlines with spaces, to ignore them properly. We push this through our script and parse the result into an array. Finally, we can set list of completion words into the system via _describe.

#!/usr/bin/env zsh
DIR=$(dirname $0:A)
SCRIPT="$DIR/test_completion.py" # relative to the script location

function _my_custom_completion_func {
    local cmd_string
    cmd_string="$PREBUFFER$LBUFFER"
    cmd_string=${cmd_string/$'\\\n'/' '}
    IFS=$' ' comp=($(echo $cmd_string))
    if [[ "${cmd_string: -1}" == ' ' ]]; then comp+=(''); fi
    ans_str=($($SCRIPT "${comp[@]}"))
    local ans
    IFS=$' ' ans=($(echo $ans_str))
    _describe 'test' ans
}

compdef _my_custom_completion_func test

If you want to delve into the sophisticated way of creating completion for zsh you may. However, you will have to learn a entirely new tagging language used only for this purpose. Hence, we delegated the work to a completion script.

Writing a completion function

We may create this script in any language of choice. It will take the command and its arguments as arguments and output space-separated completion options into standard output.

invocation: ./script <cmd> <cmd_arg1> <cmd_arg2> ... <last,partial arg>
std output: <completion_option1> <completion_option2> ...

TODO: we will show more details on how to setup the script which is invoked by the completion setups

Command-line completion