A lot of developers who have worked with REST APIs, have used tools like Postman and Hoppscotch, but only a few know about alternatives that can do the same job.
This article is one for getting started with the Curl command line tool, often written as cURL. A tool that has been for decades, which was born from an open source contribution to a pre-existing tool.
Prerequisites
Knowledge and understanding of the REST API.
Knowledge and understanding of HTTP.
Ability to use the Command-Line interface (CLI).
Previous usage of Postman or a similar tool.
Introduction
cURL stands for client URL. It is an open source project that makes a command line tool and a library libcurl for transferring data using internet protocols. It supports over 20 protocols, among which are HTTP and FTP.
cURL was first released in 1998 by Daniel Stenberg. It is an offshoot of a tool named httpget developed by Rafael Sagula.
In this article, I will be focusing specifically on the curl
command line tool and its options. Also how we can use it for API calls and authentication by harnessing the HTTP methods.
Characteristics
There are several attributes that I touched on below, which curl possesses that has made it valuable in web development.
It is open source.
It is language agnostic : meaning that it supports a variety of programming languages.
Cross-Platform Compatible: It runs on a vast array of operating systems. Linux, Windows, Mac OS and more all support the use of curl without significant modifications.
It is backwards compatible: It has a characteristic that makes it compatible with older versions and with older hardware. It means we can use files created in previous versions in a new version without issues.
Installation
Below is a step by step guide to installing curl
on various operating systems.
Windows method 1 If you are on Windows 10 version 1803 or later, it is already built into your OS. So you don't need to download anything. Default path is
C:\Windows\System32\curl.exe
.Windows method 2
Steps to set-up
Enter and access the URL https://curl.se/dlwiz/?type=bin to download the curl executable wizard.
Under the "Select Operating System" section, select windows.
Next select the parameters applicable to your version of windows.
Once you've finished the on-screen steps, download the ZIP file generated. To download it, simply press "Download".
Next, open the .zip file and enter to the folder called "src". Inside the src folder you will find the
curl
executable file. At this point, you need to copy the executable file and paste it inside a local folder on your PC to be able to run curl.From the Command Prompt, enter to the location where the executable file was pasted with the cd command. Now run the command below to verify the version of curl you have.
curl -v
- Windows method 3 : Download git for Windows and it will install curl with it.
Steps to set-up
You can find curl installed under
c:\Program Files\Git\mingw64\bin\
.Add it to your Windows path and you will be able to execute it from anywhere.Follow these steps to do so.
a. In the search bar by the start menu, enter "Edit the system environment variables".
b. Click on environment variables.
c. Click on New.
d. In the
variable name
field, enter curl.e. In the
variable value
field, enterc:\Program Files\Git\mingw64\bin\curl.exe
.f. Click on Ok to exit.
g. Enter
curl --version
in your command line to confirm you have it installed.
- Linux method You do not need to download curl on Linux. It is built-in by default. However if it is not installed follow the steps below to set up curl on Linux.
Steps to set-up
Open the terminal
Run the command
sudo apt-get install curl
If a password is required after running the command, enter your computers' user password to continue. Then, wait until the installation finishes.
curl
is now available for use on your pc.
- Mac OS method: In the MacOs
curl
is built-in by default. So we don't need to download it. However to install it, we use thebrew
(Homebrew) package manager.
Steps to set-up
Open your terminal.
Run the command
brew install curl
You now have curl
installed on your macOS.
Command-Line Options
Curl, sometimes referred to as cURL offers a vast array of options, often called flags. These options, numbering over 200, are essential building blocks for constructing curl
commands. When these commands are paired with HTTP methods, they become powerful tools for data transfer.
Given the extensive number of options, which exceeds 200, it is impractical to delve into each curl
command individually. Instead, this discussion will focus on some of the most crucial options. Several of these options are essential for specific use cases, such as managing authentication and cookies.
These options usually have a dash or two before them. They have a single dash when an alphabet is used and two dashes when they are spelt out. Let us look at the options and their implementations to make commands below.
- -V or --version: The
curl -V
( with an uppercase v) or thecurl --version
are used to show the cURL version which we have installed. See the command and response below.
C:\Users\user>curl -V
curl 8.0.1 (Windows) libcurl/8.0.1 Schannel WinIDN
Release-Date: 2023-03-20
Protocols: dict file ftp ftps http https imap imaps pop3 pop3s smtp smtps telnet tftp
Features: AsynchDNS HSTS HTTPS-proxy IDN IPv6 Kerberos Largefile NTLM SPNEGO SSL SSPI threadsafe Unicode UnixSockets
- -X:
curl
by default sends aGET
request. We use-X
to make our request explicit with the formatcurl -X <method> URL
. Below I am going to illustrate twoGET
requests with thecurl
command. One with the-X
and the other without the-X
, both fetching data from our URL @ https://my-burger-api.herokuapp.com/burgers.
C:\Users\user>curl -X GET https://my-burger-api.herokuapp.com/burgers
[
{
"id": 0,
"name": "Tribute Burger",
"restaurant": "Honest Burgers",
"web": "www.honestburgers.co.uk",
"description": "A mouth-watering honest beef burger",
"ingredients": [
"beef",
"american cheese",
"burger sauce",
"french mustard",
"pickes",
"onion",
"lettuce"
],
"addresses": [
{
"addressId": 0,
"number": "75",
"line1": "Venn Street",
"line2": "Clapham",
"postcode": "SW4 0BD",
"country": "United Kingdom"
}
]
},
{
"id": 1,
"name": "Pulled Mooshie",
"restaurant": "Mooshies",
"web": "www.veganburger.org",
"description": "Spicy vegan burger with jackfruit",
"ingredients": [
"jackfruit",
"coleslaw",
"gluten free bun"
],
"addresses": [
{
"addressId": 0,
"number": "104",
"line1": "Brick Lane",
"line2": "Shoreditch",
"postcode": "E1 6RL",
"country": "United Kingdom"
}
]
}
]
C:\Users\user>curl https://my-burger-api.herokuapp.com/burgers
[
{
"id": 0,
"name": "Tribute Burger",
"restaurant": "Honest Burgers",
"web": "www.honestburgers.co.uk",
"description": "A mouth-watering honest beef burger",
"ingredients": [
"beef",
"american cheese",
"burger sauce",
"french mustard",
"pickes",
"onion",
"lettuce"
],
"addresses": [
{
"addressId": 0,
"number": "75",
"line1": "Venn Street",
"line2": "Clapham",
"postcode": "SW4 0BD",
"country": "United Kingdom"
}
]
},
{
"id": 1,
"name": "Pulled Mooshie",
"restaurant": "Mooshies",
"web": "www.veganburger.org",
"description": "Spicy vegan burger with jackfruit",
"ingredients": [
"jackfruit",
"coleslaw",
"gluten free bun"
],
"addresses": [
{
"addressId": 0,
"number": "104",
"line1": "Brick Lane",
"line2": "Shoreditch",
"postcode": "E1 6RL",
"country": "United Kingdom"
}
]
}
]
Note: The responses which I have received and will be receiving from our sample burger URL are more than I display in the code. I had to minimize space, so I didn’t display all. You can try the same request to see the full responses.
- -i or --include: This option is one we add when we want our HTTP response headers included as part of our output. The headers are displayed above the body. Below is how we use it to carry out a command for a
GET
request to our sample URL and our response.
C:\Users\user>curl -i https://my-burger-api.herokuapp.com/burgers
HTTP/1.1 200 OK
Server: Cowboy
Report-To: {"group":"heroku-nel","max_age":3600,"endpoints":[{"url":"https://nel.heroku.com/reports?ts=1698313173&sid=e11707d5-02a7-43ef-b45e-2cf4d2036f7d&s=sPATZ5KYepOdWKwPamcdwPn4GJk5l%2BR9WBOmqw52K9g%3D"}]}
Reporting-Endpoints: heroku-nel=https://nel.heroku.com/reports?ts=1698313173&sid=e11707d5-02a7-43ef-b45e-2cf4d2036f7d&s=sPATZ5KYepOdWKwPamcdwPn4GJk5l%2BR9WBOmqw52K9g%3D
Nel: {"report_to":"heroku-nel","max_age":3600,"success_fraction":0.005,"failure_fraction":0.05,"response_headers":["Via"]}
Connection: keep-alive
X-Powered-By: Express
Vary: Origin, Accept-Encoding
Access-Control-Allow-Credentials: true
Cache-Control: no-cache
Pragma: no-cache
Expires: -1
X-Content-Type-Options: nosniff
Content-Type: application/json; charset=utf-8
Content-Length: 20826
Etag: W/"515a-nWi+LSEz+N/8zkJUxW8tpZwF4y0"
Date: Thu, 26 Oct 2023 09:39:33 GMT
Via: 1.1 vegur
[
{
"id": 0,
"name": "Tribute Burger",
"restaurant": "Honest Burgers",
"web": "www.honestburgers.co.uk",
"description": "A mouth-watering honest beef burger",
"ingredients": [
"beef",
"american cheese",
"burger sauce",
"french mustard",
"pickes",
"onion",
"lettuce"
],
"addresses": [
{
"addressId": 0,
"number": "75",
"line1": "Venn Street",
"line2": "Clapham",
"postcode": "SW4 0BD",
"country": "United Kingdom"
}
]
},
{
"id": 1,
"name": "Pulled Mooshie",
"restaurant": "Mooshies",
"web": "www.veganburger.org",
"description": "Spicy vegan burger with jackfruit",
"ingredients": [
"jackfruit",
"coleslaw",
"gluten free bun"
],
"addresses": [
{
"addressId": 0,
"number": "104",
"line1": "Brick Lane",
"line2": "Shoreditch",
"postcode": "E1 6RL",
"country": "United Kingdom"
}
]
}
]
- -I: We use this option as part of our command when we want only the response headers returned back. It is particularly useful when we want to inspect the headers to determine if we will download the body content with a
GET
request.
C:\Users\user>curl -I https://my-burger-api.herokuapp.com/burgers
HTTP/1.1 200 OK
Server: Cowboy
Report-To: {"group":"heroku-nel","max_age":3600,"endpoints":[{"url":"https://nel.heroku.com/reports?ts=1698313173&sid=e11707d5-02a7-43ef-b45e-2cf4d2036f7d&s=sPATZ5KYepOdWKwPamcdwPn4GJk5l%2BR9WBOmqw52K9g%3D"}]}
Reporting-Endpoints: heroku-nel=https://nel.heroku.com/reports?ts=1698313173&sid=e11707d5-02a7-43ef-b45e-2cf4d2036f7d&s=sPATZ5KYepOdWKwPamcdwPn4GJk5l%2BR9WBOmqw52K9g%3D
Nel: {"report_to":"heroku-nel","max_age":3600,"success_fraction":0.005,"failure_fraction":0.05,"response_headers":["Via"]}
Connection: keep-alive
X-Powered-By: Express
Vary: Origin, Accept-Encoding
Access-Control-Allow-Credentials: true
Cache-Control: no-cache
Pragma: no-cache
Expires: -1
X-Content-Type-Options: nosniff
Content-Type: application/json; charset=utf-8
Content-Length: 20826
Etag: W/"515a-nWi+LSEz+N/8zkJUxW8tpZwF4y0"
Date: Thu, 26 Oct 2023 09:39:33 GMT
Via: 1.1 vegur
- --json: We use this to send specified json data to a URL. Requests like the POST and PUT usually employ this option. We don't need to specify the method or send
Content-Type:application/json
for header as it is implied by the--json
option. The command format iscurl --json <data> URL
. We can bundle our data in two ways to send.
By writing the data explicitly.
curl --json '{"key1":"value1","key2":"value2"}' https://my-burger-api.herokuapp.com/burgers
Secondly we can place our data in a file with a json extension as burger.json
. We can send it like this below.
curl --json @/full/path/to/burger.json https://my-burger-api.herokuapp.com/burgers
- -H or --header: We use this to submit our request headers to the URL. When sending headers like
Content-Type
,Accept
andAuthorization
, we use this.
curl -H "Accept: application/json" https://my-burger-api.herokuapp.com/burgers
- -d or --data: Similar to --json which we discussed above. We use this to send data to our URL. This works with the
POST
,PATCH
andPUT
HTTP methods. When using this we add the-H
option with it in order to specify theContent-Type
as a header.
curl -d '{"key1": "value1","key2":"value2" }' -H "Content-Type: application/json" https://my-burger-api.herokuapp.com/burgers
When using Windows command prompt, we use double quotes to encapsulate the argument. Inner double quotes are also escaped. See example below.
curl -d "{\"key\":\"value\", \"key\":\"value\"}" -H "Content-Type: application/json" https://my-burger-api.herokuapp.com/burgers
We can also place the data in a file with a json extension to send it.
curl -d @/full/path/to/burger.json -H "Content-Type: application/json" https://my-burger-api.herokuapp.com/burgers
However when we want to send a PUT
or PATCH
request, we have to be explicit.
curl -d '{"key1": "value1","key2":"value2" }' -H "Content-Type: application/json" -X PUT https://my-burger-api.herokuapp.com/burgers/7
- -v or --verbose: A lowercase v in this scenario. We use this option to include the request headers and response headers as part of our response alongside the body. It shows detailed information about our request and response. See below.
Host: my-burger-api.herokuapp.com
> User-Agent: curl/8.0.1
> Accept: */*
>
* schannel: failed to decrypt data, need more data
* schannel: failed to decrypt data, need more data
< HTTP/1.1 200 OK
< Server: Cowboy
< Report-To: {"group":"heroku-nel","max_age":3600,"endpoints":[{"url":"https://nel.heroku.com/reports?ts=1698313911&sid=e11707d5-02a7-43ef-b45e-2cf4d2036f7d&s=CVmqnZcTiDimL%2BcyjZIirsp6AG2dVa0cqvFtRgpwr00%3D"}]}
< Reporting-Endpoints: heroku-nel=https://nel.heroku.com/reports?ts=1698313911&sid=e11707d5-02a7-43ef-b45e-2cf4d2036f7d&s=CVmqnZcTiDimL%2BcyjZIirsp6AG2dVa0cqvFtRgpwr00%3D
< Nel: {"report_to":"heroku-nel","max_age":3600,"success_fraction":0.005,"failure_fraction":0.05,"response_headers":["Via"]}
< Connection: keep-alive
< X-Powered-By: Express
< Vary: Origin, Accept-Encoding
< Access-Control-Allow-Credentials: true
< Cache-Control: no-cache
< Pragma: no-cache
< Expires: -1
< X-Content-Type-Options: nosniff
< Content-Type: application/json; charset=utf-8
< Content-Length: 20826
< Etag: W/"515a-nWi+LSEz+N/8zkJUxW8tpZwF4y0"
< Date: Thu, 26 Oct 2023 09:51:51 GMT
< Via: 1.1 vegur
<
[
{
"id": 0,
"name": "Tribute Burger",
"restaurant": "Honest Burgers",
"web": "www.honestburgers.co.uk",
"description": "A mouth-watering honest beef burger",
"ingredients": [
"beef",
"american cheese",
"burger sauce",
"french mustard",
"pickes",
"onion",
"lettuce"
],
"addresses": [
{
"addressId": 0,
"number": "75",
"line1": "Venn Street",
"line2": "Clapham",
"postcode": "SW4 0BD",
"country": "United Kingdom"
}
]
}
]
- -o: The
curl -o URL
command is used to create a file which downloaded content should be saved. These files could be varieties of extensions. see below.
curl -o myAudio.mp3 URL
curl -o myFile.txt URL
With our sample URL we can also download the body
from a GET
request. See below.
C:\Users\user>curl -o burgerjoints.json https://my-burger-api.herokuapp.com/burgers
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 20509 100 20509 0 0 12149 0 0:00:01 0:00:01 --:--:-- 12164
This downloads the body
of the response to a file named burgerjoints.json
on my pc.
- -O: The capital o flag. This option helps us preserve the file name of the downloaded content.
curl -O URL/filename.mp4
The downloaded file will be saved as filename.
- -F: This option is used to submit forms that contain both text data and files. With this we use
curl
to interact with web applications that accept form based data. Forms that involve uploading files. Such as videos, images, documents like multipart forms. See below.
curl -F "image:/path/to/image.jpg" -F "description: my pic"
- -L or --location: The
-L
option withcurl
is an instruction tocurl
to follow any redirects automatically. This is for handling situations where web pages have been moved. It also comes in handy to access resources that use multiple URLs to load content.
curl -L https://my-burger-api.herokuapp.com/burgers
So if a server responds with a redirect (e.g. 302 status code) and provides a new URL, curl
will make a request to the new URL and get the content from there. I have two scenarios for you to try out. Send two GET
requests to https://api.currencyscoop.com/v1. One with a -L
option and one without. Like these.
curl -L https://api.currencyscoop.com/v1
curl https://api.currencyscopp.com/v1
The above URL is one that has been changed to "https://api.currencybeacon.com".
While at it, I also want you to try any of them with the verbose option(-v) like this below. To get a verbose response.
C:\Users\user>curl -L https://api.currencyscoop.com/v1 -v
* Trying 172.67.155.40:443...
* Connected to api.currencyscoop.com (172.67.155.40) port 443 (#0)
* schannel: disabled automatic use of client certificate
* ALPN: offers http/1.1
* ALPN: server accepted http/1.1
* using HTTP/1.1
> GET /v1 HTTP/1.1
> Host: api.currencyscoop.com
> User-Agent: curl/8.0.1
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently ⬅️
>
< Date: Sat, 28 Oct 2023 02:43:00 GMT
< Transfer-Encoding: chunked
< Connection: keep-alive
< Cache-Control: max-age=3600
< Expires: Sat, 28 Oct 2023 03:43:00 GMT
< Location: https://api.currencybeacon.com/v1 ⬅️
< Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=AM3Oo5DX%2BSZzT9gdsb5c6vcOvpem%2FwREAKlxsfxyhgUB2Llk1N2mNWs%2FcsujfB3bs93lmj0sxleWh2ESnTneNtNdgkhvC6N4bHc50KHPk%2FJuZg35hKL3ZGjIN7Xe5vXTy%2FRuuiPAIX8%3D"}],"group":"cf-nel","max_age":604800}
< NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
< Server: cloudflare
< CF-RAY: 81cfd42a3bb30ea7-AMS
< alt-svc: h3=":443"; ma=86400
<
* Ignoring the response-body
* Connection #0 to host api.currencyscoop.com left intact
* Issue another request to this URL: 'https://api.currencybeacon.com/v1' ⬅️
* Trying 104.21.33.73:443...
* Connected to api.currencybeacon.com (104.21.33.73) port 443 (#1)
* schannel: disabled automatic use of client certificate
* ALPN: offers http/1.1
* ALPN: server accepted http/1.1
* using HTTP/1.1
> GET /v1 HTTP/1.1
> Host: api.currencybeacon.com
> User-Agent: curl/8.0.1
> Accept: */*
See information in places I marked with arrows above, from the --verbose
response.
Authentication
While we have previously looked at some options for creating curl
commands, there are a few that are multifaceted(even though they are mostly specialized).
In this section, I will be bringing back an option or two you have previously seen along with a new option to show how to perform authentication and authorization using curl
commands.
- -H or --header: Like we saw previously, this option is used to send headers while making requests. Among the headers which we didn't use for demonstration is the
Authorization
header. This header comes in handy when certain credentials are required to access a URL. Some of these could be web tokens, API keys e.t.c.
For web tokens, we sign in with a POST request like this.
curl --json '{"email":"youremail@mail.com","password":"your_password"} https://sampleURL.com
In the above, it might not be email though, might be username. When you sign in, you receive a token. This token will be used for Authorization
like below.
curl - H "Authorization: Bearer your_token" https://sampleURL.com
For sites that require API keys, they usually have their own custom headers for this. These headers usually start with X- ( not to be confused with -X). See below.
curl -H "X-SiteAPI-key:your_key_here" https://sampleURL.com
- -u or --user: The
curl -u
command takes a username and password when sending HTTP requests. This command is used for basic authentication when making HTTP requests. Below is how we use the command.
curl -u username:password https://sampleURL.com/protected/rrsource
When you specify the -u
option, curl
will include the Authorization
header in the request to authenticate you with the server.
Please note that this form of authentication is not the most secure and it is typically used over HTTPS to help protect the credentials from being transmitted in plain text. More secure authentication methods like tokens are preferred, because the credentials are visible in the request when we don't use HTTPS.
Handling Cookies
Cookies are used for tracking user session data on the web. We can use curl to send and receive these cookies as curl provides options for managing these cookies. Let us now take a look at these options and how they can be implemented with curl
commands.
- -c or --cookie-jar: This option is used for saving cookies on the client or to set cookies as is the genarlly used term. These cookies are gotten from the server when the client makes a request to a server that has these cookies. To illustrate We typically send a file with a .txt extension along with the request. This file is where the received cookies will be saved. It takes the format below.
curl -c cookies.txt https://sampleURL.com
- -b or --cookie: We use this option when we want to send the saved cookie in the client to the server. This is usually used for subsequent requests when the cookie must have been saved on the client. It takes the format below.
curl -b cookies.txt https://sampleURL.com
The cookies are read from the cookies.txt
file to send.
we can also send specific cookies among the save cookies with this option. See below.
curl -b "cookieName=valueOfCookie" https://sampleURL.com
Below I want to make a simple app we can use to create this in real time.See steps below.
Go to your vs code. Create a directory name it cookie-setter.
create an app.js file within it.
open up notepad and create a cookies.txt file
open a terminal within vs code.
Run npm init to create a package.json file.
In your terminal, run npm install express.
Next run npm install cookie-parser.
After installing both packages go down to your app.js file and add the code below.
const express = require('express')
const app = express()
const cookieParser = require('cookie-parser')
app.use(cookieParser())
app.get('/setcookie',(req,res)=>{
let maxAge = 24 * 60 *60* 1000 //milliseconds
res.cookie('user','your_name',maxAge)
res.send('cookie has been set')
})
app.get('/getcookie',(req,res)=>{
const myCookie = req.cookies.user
res.json(`value of 'user' cookie:${myCookie}`)
})
let port = 4500
app.listen(4500,()=>{
console.log(`server is running on port ${port}`)
})
Above we have a setcookie
route which when we visit triggers the setting of cookies in our device. We also have a getcookie
route which when we visit shows us the cookie we set.
Note:your package.json file should look like below.
"name": "cookie-setter",
"version": "1.0.0",
"description": "",
"main": "app.js",
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1"
},
"author": "",
"license": "ISC",
"dependencies": {
"cookie-parser": "^1.4.6",
"express": "^4.18.2"
}
}
open up another terminal in vs code or open up your Command prompt. cd
to the folder containing the cookies.txt
file. Mine is in the Documents folder. Run the command below with the --verbose
or -v
flag.
C:\Users\user\Documents>curl -c Cookies.txt http://localhost:4500/setcookie -v
C:\Users\user\Documents>curl -c Cookies.txt http://localhost:4500/setcookie -v
* Trying 127.0.0.1:4500...
* Connected to localhost (127.0.0.1) port 4500 (#0)
> GET /setcookie HTTP/1.1
> Host: localhost:4500
> User-Agent: curl/8.0.1
> Accept: */*
>
< HTTP/1.1 200 OK
< X-Powered-By: Express
* Added cookie user="steve" for domain localhost, path /, expire 0⬅️
< Set-Cookie: user=steve; Path=/
< Content-Type: text/html; charset=utf-8
< Content-Length: 19
< ETag: W/"13-UbqCgG6k5thPk0DHAkVScaoIJp8"
< Date: Sun, 29 Oct 2023 09:17:17 GMT
< Connection: keep-alive
< Keep-Alive: timeout=5
<
cookie has been set* Connection #0 to host localhost left intact
Above I visited the setcookie
route which triggered the setting of cookies in our cookies.txt
file. You can see the information at the line where the arrow is pointing to above.
Now let us visit the getcookie
route to see our cookie on the server.Run command below.
C:\Users\user\Documents>curl -b Cookies.txt http://localhost:4500/getcookie -v
C:\Users\user\Documents>curl -b Cookies.txt http://localhost:4500/getcookie -v
* Trying 127.0.0.1:4500...
* Connected to localhost (127.0.0.1) port 4500 (#0)
> GET /getcookie HTTP/1.1
> Host: localhost:4500
> User-Agent: curl/8.0.1
> Accept: */*
> Cookie: user=steve ⬅️
>
< HTTP/1.1 200 OK
< X-Powered-By: Express
< Content-Type: text/html; charset=utf-8
< Content-Length: 28
< ETag: W/"1c-8P5w3huyD6e9oztUcohhLAM8ogo"
< Date: Sun, 29 Oct 2023 09:25:39 GMT
< Connection: keep-alive
< Keep-Alive: timeout=5
<
value of 'user' cookie:steve* Connection #0 to host localhost left intact
From the line where I put an arrow, you can see my cookie header, as well as the key and value.
- -H or --header: We can also use this option to create a
cookie
header which to send specific cookies.
curl -H "cookie: user=your_name"
URL Globbing
The practice of using using wildcard characters in URLs to match multiple resources at once, and get the content of those resources. URL globbing allows us specify sets of URLs using special characters like square brackets [] and curly braces {}. These characters are usually not allowed in standard URLs, except in specific cases like Internet Protocol Version 6 (IPV6) addresses.
curl
leverages these characters for its own URL globbing feature, allowing users to specify ranges and sets of URLs more conveniently. We will now take a look at specific ways to use these characters.
- Square Brackets []: It is used to specify a range of characters. See an illustration below of a way we can use it.
curl http://localhost:4500/folder/file[1-3].html
From the above, curl
will retrieve files like file1.html
, file2.html
, and file3.html
. If there is a file4.html
, it won't retrieve it.
- Curly Braces {}: It is used to specify a set of characters.The characters are usually separated by commas. See an illustration below of a way it can be used.
curl http://localhost:4500/folder/file{1,2,5}.html
From the above curl
will retrieve file1.html
, file2.html
and file5.html
.
Download Content We might need to download the content from the URLs we accessed. We add a -O
flag. This should download content from the specified URLS. See the code below.
curl -O http://localhost:4500/folders/{index,support,about}.html
This downloads content from index.html
, support.html
and about.html
.
Literal Characters But we know that in IPV6 addresses we might find curly braces and square brackets as part of url characters. We will now take a look at ways to handle such.
Turning off globbing: If the square brackets or curly braces appear in a URL and aren't intended for globbing, you can disable globbing by using the
--globoff
or-g
flag with the curl command.URL Encoding: Globbing characters must be URL encoded if we want to use them as literal characters in the URL. If we want ti use square brackets or curly braces as real URL characters, we need to encode them using URL encoding.
In this URL encoding, we replace special characters with a percent sign "%" followed by two hexadecimal digits representing the ASCII code of the character. See code lists below.
[ becomes %5B
] becomes %5D
{ becomes %7B
} becomes %7D
Note: Ensure your Curl version supports URL globbing, as older versions may not have this feature.
Conclusion
We have learnt abouthow to execute HTTP requests with curl
. The following links were instrumental to my understanding of the tool. You can study them to gain more clarity.