Step 2: Split a line into words

The shell receives text but the operating system wants an array of words (argv): the program name and its arguments, ending with a NULL pointer. Writing the tokenizer teaches careful pointer work in C: walk the line, skip spaces, and for each word store a pointer to its start and put a '\0' at its end. It changes the string in place, so no memory is allocated. Double quotes keep several words together.

ExampleC

#include <stdio.h>
#include <string.h>

#define MAX_ARGS 16

// Split a line into words in place. Words are separated by spaces; "double quotes" keep spaces together.
int tokenize(char *line, char *argv[]) {
  int argc = 0;
  char *p = line;
  while (*p != '\0' && argc < MAX_ARGS - 1) {
    while (*p == ' ') p++;                 // skip spaces between words
    if (*p == '\0') break;
    if (*p == '"') {                        // quoted word
      p++;
      argv[argc++] = p;
      while (*p != '\0' && *p != '"') p++;
    } else {                                // plain word
      argv[argc++] = p;
      while (*p != '\0' && *p != ' ') p++;
    }
    if (*p != '\0') *p++ = '\0';           // end the word
  }
  argv[argc] = NULL;                       // exec functions want a NULL at the end
  return argc;
}

int main(void) {
  char line1[] = "ls -l  /tmp";
  char line2[] = "echo \"hello big world\" done";
  char line3[] = "   ";
  char *argv[MAX_ARGS];

  char *lines[] = { line1, line2, line3 };
  for (int i = 0; i < 3; i++) {
    int argc = tokenize(lines[i], argv);
    printf("argc=%d:", argc);
    for (int j = 0; j < argc; j++) printf(" [%s]", argv[j]);
    printf("\n");
  }
  return 0;
}

Output

argc=3: [ls] [-l] [/tmp]
argc=3: [echo] [hello big world] [done]
argc=0:

The output shows ls -l /tmp (with two spaces) becomes three words, the quoted text stays as one word, and a blank line gives zero words. Always limit the array size (MAX_ARGS): a very long line must not write past the array.

Quick check

Why must argv end with a NULL pointer?

The exec functions use it to know where the list endsC strings need itIt makes the shell faster

Finish the quick check (and any code exercise) to complete this lesson, or mark it done yourself.

Tip: use the left and right arrow keys to move between lessons.