9 uses for cURL worth knowing

Working with HTTP from the command-line is a valuable skill for HTTP architects and API designers to have. The cURL library and

curl

command
give you the ability to design a Request, put it on the pipe, and explore the Response. The downside to the power of

curl

is
how much breadth its options cover. Running

curl --help

spits
out 150 different flags and options. This article demonstrates nine basic, real-world applications of

curl

.

In this tutorial we’ll use the httpkit echo service as our end point. The echo server’s Response is a JSON representation of the HTTP request it receives.

Make a Request

Let’s start with the simplest

curl

command possible.

Request

curl http://echo.httpkit.com

Response

{
  "method": "GET",
  "uri": "/",
  "path": {
    "name": "/",
    "query": "",
    "params": {}
  },
  "headers": {
    "host": "echo.httpkit.com",
    "user-agent": "curl/7.24.0 ...",
    "accept": "*/*"
  },
  "body": null,
  "ip": "28.169.144.35",
  "powered-by": "http://httpkit.com",
  "docs": "http://httpkit.com/echo"
}

Just like that we have used

curl

to make an HTTP Request.
The method, or “verb”,

curl

uses, by default, is

GET

.
The resource, or “noun”, we are requestion is addressed by the URL pointing to the httpkit echo service,

http://echo.httpkit.com

.

You can add path and query string parameters right to the URL.

Request

curl http://echo.httpkit.com/path?query=string

Response

{ ...
  "uri": "/path?query=string",
  "path": {
    "name": "/path",
    "query": "?query=string",
    "params": {
      "query": "string"
    }
  }, ...
}

Set the Request Method

The

curl

default HTTP method,

GET

,
can be set to any method you would like using the

-X

option.
The usual suspects

POST

,

PUT

,

DELETE

,
and even custom methods, can be specified.

Request

curl -X POST echo.httpkit.com

Response

{
    "method": "POST",
    ...
}

As you can see, the

http://

protocol prefix can be dropped
with

curl

because it is assumed by default. Let’s give

DELETE

a
try, too.

Request

curl -X DELETE echo.httpkit.com

Response

{
    "method": "DELETE",
    ...
}

Set Request Headers

Request headers allow clients to provide servers with meta information about things such as authorization, capabilities, and body content-type. OAuth2 uses an

Authorization

header
to pass access tokens, for example. Custom headers are set in

curl

using
the

-H

option.

Request

curl -H "Authorization: OAuth 2c4419d1aabeec" \
     http://echo.httpkit.com

Response

{...
"headers": {
    "host": "echo.httpkit.com",
    "authorization": "OAuth 2c4419d1aabeec",
  ...},
...}

Multiple headers can be set by using the

-H

option multiple
times.

Request

curl -H "Accept: application/json" \
     -H "Authorization: OAuth 2c3455d1aeffc" \
     http://echo.httpkit.com

Response

{ ...
  "headers": { ...
    "host": "echo.httpkit.com",
    "accept": "application/json",
    "authorization": "OAuth 2c3455d1aeffc" 
   }, ...
}

Send a Request Body

Many popular HTTP APIs today

POST

and

PUT

resources
using

application/json

or

application/xml

rather
than in an HTML form data. Let’s try

PUT

ing some JSON
data to the server.

Request

curl -X PUT \
     -H 'Content-Type: application/json' \
     -d '{"firstName":"Kris", "lastName":"Jordan"}'
     echo.httpkit.com

Response

{
   "method": "PUT", ...
   "headers": { ...
     "content-type": "application/json",
     "content-length": "40"
   },
   "body": "{\"firstName\":\"Kris\",\"lastName\":\"Jordan\"}",
   ...
 }

Use a File as a Request Body

Escaping JSON/XML at the command line can be a pain and sometimes the body payloads are large files. Luckily, cURL’s

@readfile

macro
makes it easy to read in the contents of a file. If we had the above example’s JSON in a file named “example.json” we could have run it like this, instead:

Request

curl -X PUT \
     -H 'Content-Type: application/json' \
     -d @example.json
     echo.httpkit.com

POST HTML Form Data

Being able to set a custom method, like POST, is of little use if we can’t also send a request body with data. Perhaps we are testing the submission of an HTML form. Using the

-d

option
we can specify URL encoded field names and values.

Request

curl -d "firstName=Kris" \
     -d "lastName=Jordan" \
     echo.httpkit.com

Response

{
  "method": "POST", ...
  "headers": {
    "content-length": "30",
    "content-type":"application/x-www-form-urlencoded"
  },
  "body": "firstName=Kris&lastName=Jordan", ...
}

Notice the method is

POST

even though we did not specify
it. When

curl

sees form field data it assumes

POST

.
You can override the method using the

-X

flag discussed
above. The “Content-Type” header is also automatically set to “application/x-www-form-urlencoded” so that the web server knows how to parse the content. Finally, the request body is composed by URL encoding each of the form fields.

POST HTML Multipart / File Forms

What about HTML forms with file uploads? As you know from writing HTML file upload form, these use a

multipart/form-data

Content-Type,
with the

enctype

attribute in HTML. In cURL we can pair
the

-F

option and the

@readFile

macro
covered above.

Request

curl -F "firstName=Kris" \
     -F "publicKey=@idrsa.pub;type=text/plain" \
     echo.httpkit.com

Response

{
  "method": "POST",
  ...
  "headers": {
    "content-length": "697",
    "content-type": "multipart/form-data;
    boundary=----------------------------488327019409",
    ... },
  "body": "------------------------------488327019409\r\n
           Content-Disposition: form-data;
           name=\"firstName\"\r\n\r\n
           Kris\r\n
           ------------------------------488327019409\r\n
           Content-Disposition: form-data;
           name=\"publicKey\";
           filename=\"id_rsa.pub\"\r\n
           Content-Type: text/plain\r\n\r\n
           ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAkq1lZYUOJH2
           ... more [a-zA-Z0-9]* ...
           naZXJw== krisjordan@gmail.com\n\r\n
           ------------------------------488327019409
           --\r\n",
...}

Like with the

-d

flag, when using

-F

curl

will
automatically default to the

POST

method, the

multipart/form-data

content-type
header, calculate length, and compose the multipart body for you. Notice how the

@readFile

macro
will read the contents of a file into any string, it’s not just a standalone operator. The “;text/plain” specifies the MIME content-type of the file. Left unspecified,

curl

will
attempt to sniff the content-type for you.

Test Virtual Hosts, Avoid DNS

Testing a virtual host or a caching proxy before modifying DNS and without overriding hosts is useful on occassion. With cURL just point the request at your host’s IP address and override the default

Host

header
cURL sets up.

Request

curl -H "Host: google.com" 50.112.251.120

Response

{
  "method": "GET", ...
  "headers": {
    "host": "google.com", ...
  }, ...
}

View Response Headers

APIs are increasingly making use of response headers to provide information on authorization, rate limiting, caching, etc. With cURL you can view the headers and the body using the

-i

flag.

Request

curl -i echo.httpkit.com

Response

HTTP/1.1 200 OK
Server: nginx/1.1.19
Date: Wed, 29 Aug 2012 04:18:19 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 391
Connection: keep-alive
X-Powered-By: http://httpkit.com 
{
  "method": "GET",
  "uri": "/", ...
}

Shameless plug:
Do you hack on REST API integrations or implementations? Wiretap is
an HTTP debugger you can use to see every request and response between any client and HTTP API in real time. It's entering private beta soon. Help
test it!