Making LLM helpers for the CLI

I read about the LLM v0.27 release. Its added the ability to associating tool calling with a template. Cool! I made a ZSH function using LLM templates to help me quickly get aid on the CLI.

Playing around with templates

I don’t really use LLMs too much when I’m on the CLI. Mostly I use a plugin in my editor (Avante and CodeCompanion are my two favorite NeoVim plugins at the moment). If I need some more extensive help I’ll jump into the OpenAI or Gemini web interface. Or if my task involves a lot of context, I’ll use repomix to gather relevant files, and pipe them to the LLM cli.

But I have noticed that I often lookup CLI commands, and when I do that, I go to the browser.

I’ve seen there are a few CLI tools out there to fill this particular use case, such as sgpt and shell_gpt. I’ve taken both for a spin and they work as expected…but I don’t really know what the prompts are without digging. I want more control over the output. I want it to be aware of context of my current shell session. And I want to use LLM, my normal CLI swiss army knife.

First I just setup a template in LLM:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

# Setup a simple CLI helper template using a fast model:
llm prompt \
  -m gemini-1.5-flash-latest \
  -s 'act as a lookup tool for the CLI man pages/manual. \
If possible return the exact command requested of the user, or corrected command only. \
If the command does not exist, or cannot be achieved provide no more than one sentence describing the conundrum. \
The output is intended to be used directly on the CLI. \
Do not use markdown or code fences.' \
  --save clilookup

# Test it out
llm -t clilookup 'whats the command to find markdown files that use yyyy-mm-dd pattern files in their name?'

find . -name "*[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]*.md"

The main thing I want though, is to have a command that already has a default behavior: help me with my last command. I am almost always running commands inside tmux, so I thought I could provide the following context and get some useful responses:

  • The last command I ran and its exit code (!! and $?)
  • The last X lines of the current tmux pane (so that the LLM can see what I was trying to do, and output)
  • If the output is something I like, I’d like to just run it. I thought for this, just copy the output to my clipboard using pbcopy.
  • Since I make use of a number of python environments, use a specific one when running LLM with whatever plugins I end up using.

After some playing around I came up with the following ZSH pattern for a function:

1
2
3
4
5
6

# ask for something I want to immediately run:
llmhelp() (
  export PYENV_VERSION=3.11.9
  local query="$*"
  pyenv exec llm -t clilookup "$query" | tee >(pbcopy)
)

Final Product

Ultimately I decided that I wanted to have two modes: get a usable command back, or get some explanation (pretty printed).

I particularly like that I can just type llmhelp and it’ll attempt to fix whatever error it sees in the tmux pane.

I was able to hack this together in about an hour, and I loved that (like my editor) its tuned to how I like to work. I have a lot of control over how it works, what LLM provider I’m using, what the prompt is, etc:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

# a subtle error in my find command, the shell is expanding all the globs:
> find . -name [0-9]*-[0-9]*-[0-9]*.txt
zsh: no matches found: [0-9]*-[0-9]*-[0-9]*.txt

> llmhelp
find . -name "[0-9]*-[0-9]*-[0-9]*.txt"

> llmhelp oops this should be for markdown files not text
find . -name "[0-9]*-[0-9]*-[0-9]*.md"

> find . -name [0-9]*-[0-9]*-[0-9]*.md

> llmhelp explain why this does not find markdown files I know are in this directory


  The command `fd -g '^.*[0-9]{4}-[0-9]{2}-[0-9]{2}.*\.md'` uses the `-g` flag, which  performs a literal glob search.
  This means it's not using regular expression matching, but a shell-style glob.  The regular expression `^.*[0-9]{4}-[0-
  9]{2}-[0-9]{2}.*\.md` would work correctly with `find` using `-regex`, but not with `fd`'s `-g` option.  If you want to
  use regular expressions with `fd`, you need to use the `-E` or `--extended-regex` option instead of the `-g` option.
  However, you likely want to use a more appropriate glob or regex.


    To find markdown files with a date in the filename, you should try this command:

    ```bash
    fd -H  '^.*[0-9]{4}-[0-9]{2}-[0-9]{2}.*\.md$'

  This will find all files ending in  .md  that contain a date formatted as YYYY-MM-DD.  The  -H  flag ensures that
  hidden files are included in the search.   If files are excluded by  .gitignore  or similar, add  --no-ignore  to
  override.

> fd -H  '^.*[0-9]{4}-[0-9]{2}-[0-9]{2}.*\.md$'

... yay, I see the list of files I'm looking for ...

Here is the final helper function that I put into my .zshrc:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

llmhelp() {
  local rc=$?                           # exit code of previous command
  local -a ps=("${pipestatus[@]}")      # per-stage pipeline statuses
  local cmd; cmd=$(fc -ln -1)           # literal previous command
  local lines=20

  # Optional user-supplied prompt text; else a default
  local query="${*:-Can you correct the previous command or explain why it won't work?}"

  # Set useglow to true if query starts with 'explain'
  local useglow=false; [[ "$query" == explain* ]] && useglow=true

  # Base info
  local info
  info=$(
    printf 'Previous command:\n%s\n\nExit code: %d\nPipeline statuses: %s\nCWD: %s\nShell: zsh\n' \
      "$cmd" "$rc" "${ps[*]:-n/a}" "$PWD"
  )

  # If in tmux, append the active pane context
  if [[ -n "$TMUX" ]]; then
    local id meta content
    id=$(tmux list-panes -F '#{?pane_active,#{pane_id},}' | grep -v '^$' | head -n1)
    if [[ -n "$id" ]]; then
      meta=$(tmux display-message -p -t "$id" \
        '* pane #{pane_index} #{pane_width}x#{pane_height} | cmd=#{pane_current_command} | title="#{pane_title}" | tty=#{pane_tty}')
      content=$(tmux capture-pane -ep -S -"${lines}" -t "$id" 2>/dev/null || echo "<unable to capture>")
      info+=$'\n\n'"=== tmux active pane (last '"$lines"' lines) ==="$'\n'"--- ${meta} ---"$'\n'"${content}"
    fi
  fi

  # One clean string to llm
  local payload="$query"$'\n\n'"$info"
  if [[ "$useglow" == false ]]; then
    PYENV_VERSION=3.11.9 pyenv exec llm -t clilookup "$payload" | tee >(pbcopy)
  else
    PYENV_VERSION=3.11.9 pyenv exec llm -t clilookup "$payload" | glow
  fi
}

And the final prompt (edited with llm templates edit clilookup):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

model: gemini-1.5-flash-latest
system: |
    Act in two ways: lookup mode or explain mode.

    lookup mode (default behavior): Act as a lookup tool for the CLI man pages/manual. If possible return the
    exact command requested of the user, or corrected command only. If the command
    does not exist, or cannot be achieved provide no more than one sentence describing
    the conundrum. Do not use markdown or code fences. The output is intended to be used directly on the CLI. 

    explain mode: If the prompt starts with 'explain', provide a paragraph of guidance. Limit the response to
    no more than two paragraphs (and a list, if applicable). The output is intended to be output directly
    to a shell console. Use markdown. If sample commands are provided they should be on their own 
    line so that they're easy to copy and paste.

Notes:

  • Using llm as the CLI tool is also nice for debugging. I used llm logs frequently to see what I was sending to the LLM and what it was returning.

Reading

“God doesn’t use the scientific method”
- Camp Damascus

I finished listening to the Camp Damascus audiobook. Considering the truly outlandish short story titles, such as Space Raptor Butt Invasion, I wasn’t sure what to expect (Yes there is a back story there, see wikipedia). Overall great read. I’ll probably read Lucky Day someday soon.