cURL: How to Add Headers and Params When Making HTTP Requests

Introduction

cURL is a versatile tool and library (libcurl) that is used to transfer data with URLs. It supports a variety of protocols, including HTTP, HTTPS, FTP, and more. In the context of HTTP, it’s often needed to send custom headers or URL parameters within a request for tasks such as API interactions, web scraping, or automating downloads.

In this tutorial, we will go over how to add headers and parameters to your cURL requests to interact with HTTP-based services effectively.

Understanding cURL Syntax

Before we add headers and parameters, let’s understand the basic syntax of cURL. The simplest form of a cURL command is:

curl [options] [URL]

Options in the cURL command can control aspects such as headers, request method, data, etc.

Adding HTTP Headers

To add a header to your HTTP request, you can use the -H or --header flag followed by the header you’d like to add. Here’s the general syntax:

curl -H "Header-Name: header value" [URL]

For instance, to add a custom user-agent, you could use:

curl -H "User-Agent: MyCustomAgent" https://example.com

If you need to add multiple headers, you can repeat the -H flag as many times as necessary:

curl -H "Accept: application/json" -H "Content-Type: application/json" https://api.example.com/data

Headers are often used for authentication, content negotiation, or providing metadata about the request body.

Adding URL Parameters

While parameters are usually added directly to the URL, cURL allows for easier assembly of URL parameters through the -G flag when using the GET method. Parameters can then be added using the --data-urlencode option. This approach ensures proper URL-encoding of query strings.

curl -G --data-urlencode "param1=value1" --data-urlencode "param2=value2" https://example.com/api

This will result in a URL that looks like https://example.com/api?param1=value1&param2=value2.

Mixing Headers and Params

Here’s how to mix together headers and URL parameters in a single cURL request:

curl -G -H "Authorization: Bearer YourToken" --data-urlencode "query=keyword" https://api.example.com/search

In the above case, you are sending a GET request with an authorization header and a URL parameter for search.

POST Requests with cURL

With POST requests, headers are used the same way, but data is sent via the --data or -d flag. Use the Content-Type header to specify the content you’re sending.

curl -X POST -H "Content-Type: application/json" -d '{"username":"admin","password":"12345"}' https://api.example.com/login

This sends a JSON object with username and password fields.

Advanced Usage

For more advanced situations, your headers may include additional elements like tokens that frequently change. One way to handle this is to set the header via a shell variable:

TOKEN="Your-API-Token"
curl -H "Authorization: Bearer $TOKEN" https://api.example.com/data

This is particularly useful in scripts where the token might be refreshed or updated.

cURL can also emulate browsers by sending a set of common headers. For instance, you can impersonate as if a request is made by a typical Chrome browser by setting the user-agent and other headers typically associated with the browser.

Example:

curl -X GET \
  -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36" \
  -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8" \
  -H "Accept-Language: en-US,en;q=0.9" \
  -H "Referer: https://www.example.com" \
  -H "Connection: keep-alive" \
  https://example.com/your-resource

In this command:

• The -H option is used to set various headers.
• The “User-Agent” header is set to mimic a Chrome browser.
• Other common headers like “Accept,” “Accept-Language,” “Referer,” and “Connection” are also included.

You can adjust the headers as needed to match the behavior of the specific browser you want to emulate. This can be useful when interacting with web services or APIs that require specific headers to be set, such as those that expect requests from browsers.

Final Words

Fine-tuning requests with custom headers and parameters is a core part of working with APIs and web-based resources. Use the techniques shown above to enhance your cURL requests as needed.

Lastly, remember to review API documentation for any service you are interfacing with, as it will specify any required headers, acceptable parameters, and the proper request format. Using cURL effectively means adhering to these requirements to ensure successful communication with web services.

That concludes our guide on using cURL to add headers and parameters when making HTTP requests. Whether you’re working with APIs, automating downloads, or interacting with web services, mastering cURL is an essential skill for modern developers and systems administrators.