1 YouTube Video

https://www.youtube.com/watch?v=OxgObpqgUDc

Official YouTube video for this article. Check it out if you’d like.

2 Introduction

If I were to ask you what you thought jq stood for, then you might answer or you might assume that it is the short form of JSON query. However, that assumption is false according to one of the authors of jq! This is just one of the many ways that this programming language is misunderstood. One of these misunderstandings is that jq is just for handling JSON. To the contrary, it can even serve as a rudimentary general-purpose programming language, a stream editor, and even a PEG parser. jq is so complex that it can be implemented in itself. It is a Turing-complete language like any other.

Simple FizzBuzz program in jq:

range(1; 101)
  | if   . % 15  == 0 then "FizzBuzz"
    elif . % 3  == 0 then  "Fizz"
    elif . % 5 == 0 then   "Buzz"
    else . end

2.1 Disclaimer

First, I must admit that I am not a jq expert, just wanted to make this after completing the Exercism jq track for fun. I urge anyone interested to go look into the documentation and manual. The main purpose of this is to show off a cool thing. Any errors are my own.

3 Basics

Basic scripting flags:

jq -n  '...'                     - evaluate pure jq without passing a JSON file
jq -f  'program.jq' input.json   - evaluate input.json using program.jq
jq -nf 'program.jq'              - evaluate a pure jq file without passing a json file
jq '<jq code>' input.json        - evaluate input.json using command-line code

3.1 Hello, World!

If I had to say something about the general “vibe” of jq, I’d say it is a functional programming language out of necessity rather than by choice. The first thing you need to understand is that everything is either a filter or a construct that acts on filters. Let’s look at a quick “hello world”:

"Hello World" | .

Here, the default filter that returns what is passed to it is .. Any value passed to it gets returned, it is the identity operator.

"hello world"

Fun fact: literals in jq are filters that ignore their input and produce their own value as output.

3.2 Object Indexing

We use object indexing in order to get the value of a key from an object.

{
    "John": 50,
    "Bob":  75,
    "Tom":  80,
    "John Smith": 100
}

We pass this object to the code blocks.

.Bob

75

We can even use square brackets for more complicated keys.

.["John Smith"]

100

Note: We use ? when we do not know if the attribute exists.

.jrrom?

null

3.3 Pipe Operator

We use it to chain values, and apply many filters in sequence.

.["John"]
 | tostring
 | split("")

[
 "5",
 "0"
]

3.4 Arrays

Arrays are a language construct for arranging a sequence of values in jq.

[{"John":50},{"Bob":75},{"Tom":80},{"John Smith":100}]

We can use array indexing to query them by index position, this is helpful in iteration and arrays created by range.

.[1]

jq is a 0-indexed language!

{
  "Bob": 75
}

You can also take in an array and output a stream of objects by themselves. This is the main way that we create streams in jq.

.[]

{
  "John": 50
}
{
  "Bob": 75
}
{
  "Tom": 80
}
{
  "John Smith": 100
}

3.5 Comma Operator

To evaluate multiple filters on an input to produce multiple outputs.

{"name": "Joltik", "type": "Electric"}
  | (.name, .type)
  | "hi " + .      

"hi Joltik"
"hi Electric"

3.6 Default Operator

It acts on null values or keys that do not exist. It produces a fallback value.

[{"name": "Joltik", "type": "Electric"}, {"name": "Ash"}]
  | .[]
  | (.name, .type // "Unknown")

"Joltik"
"Electric"
"Ash"
"Unknown"

3.7 Object Construction & Assignment

We can create new JSON objects inside of jq and we can also modify existing ones.

[{"name": "Joltik", "type": "Electric"}, {"name": "Ash"}]
  | .[]
  | {"name": .name, "object": "entity"}

{
  "name": "Joltik",
  "object": "entity"
}
{
  "name": "Ash",
  "object": "entity"
}

[{"name": "Joltik", "type": "Electric"}, {"name": "Ash"}]
  | .[]
  | .type = "entity"     

{
  "name": "Joltik",
  "type": "entity"
}
{
  "name": "Ash",
  "type": "entity"
}

4 Variables

Variables are pretty complex in jq. Their main use case comes from the fact that it is hard to preserve values in jq. Variables were made specifically to solve this. We really don’t need to use variables in most situations but there are some situations where they are vital. See the following:

[1, 2, 3]
  | .[2] as $saved_value
  | [.[] + 1]
  | $saved_value # Without variables, it would be difficult to access the old value before the change

3

5 Data Manipulation

These are generally the foundational data manipulation constructs in functional programming languages including jq. We can do most of the important querying with their help.

Trinity of Functional Programming (jq style)
│
├── Map
│   ├── Equivalent to map in most languages
│   ├── [a] → (a → b) → [b]
│   └── <data> │ map(<filter>)
│
├── Select
│   ├── Similar to filter in most languages
│   ├── [a] → (a → boolean) → [a]
│   └── <data> │ map(select(<filter>))
│
└── Reduce
    ├── Equivalent to reduce or fold
    ├── Complex construct in jq
    └── <data> │ reduce .[] as $var (<initial>; <update>)

5.1 Map

[{name: "A"}, {name: "B"}, {name: "C"}]
  | map(.name)

[
 "A",
 "B",
 "C"
]

5.2 Filter

[{name: "A", marks: 55}, {name: "B", marks: 66}, {name: "C", marks: 77}]
  | map(select(.marks > 60))

[
  {
    "name": "B",
    "marks": 66
  },
  {
    "name": "C",
    "marks": 77
  }
]

5.3 Reduce

[{name: "A", marks: 55}, {name: "B", marks: 66}, {name: "C", marks: 77}]
  | reduce .[].marks as $m (0; . + $m)

198

6 Functions

To call jq functions “functions” is a bit misleading in my opinion. They are closer to glorified macros if you do not use variables. They are a complex type of reusable filter that is more ergonomic than repeating the same code.

def give_msg(greeting; msg):
  "\(greeting) \(.)! \(msg)" # String interpolation
;
def say_hello:
  give_msg("Good morning"; "How are you?")
  # . | give_msg("Good morning"; "How are you?")
;
"John"
  | say_hello

"Good morning John! How are you?"

6.1 Important footgun!

Arguments in jq are not evaluated values, they remain as filters! Keep this in mind before you make a complicated function.

def addvalue(f): map(. + f);

[[1,2], [10,20]] | addvalue(.[0])

You might expect the result to be [[1, 2, 1, 2], [10, 20, 1, 2]], however, you must remember that the filter itself is passed, so it’s more like map(. + .[0]) which leads to:

jq: error (at <unknown>): array ([1,2]) and number (1) cannot be added

Since it becomes map(. + .[0]) which is map([1, 2] + 1).

7 Miscellaneous

7.1 Some built-ins

Check the manual for more!

7.1.1 tonumber

"123" | tonumber

123

7.1.2 tostring

123 | tostring

"123"

7.1.3 length

It works on arrays, strings and even objects.

"tabby" | length

5

7.1.4 has(key)

{"name": "Ash"} | has("name")

true

7.1.5 del(key)

{"name": "Ash", "age": 10} | del(.name)

{
  "age": 10
}

7.1.6 split, join

"Hello World"
  | split("")
  | join("")

"Hello World"

8 Conclusion

I hope this serves as a good reference point to jq. My objective for this is to put all the most important basic beginner knowledge all in one place. To any readers, I would also highly recommend reading the manual and continuing your jq journey over there!

Thank you for reading.