Sending a raw HTTPS request

Today I needed to work out why a remote API endpoint wasn't responding as expected to a multipart/form-data HTTP POST request.

A cURL request like this was successful:

curl 'https://example.com/foo' \
  -H 'Authorization: Bearer foo' \
  -F 'id=example' \
  -F 'file=@example.pdf'

But a theoretically identical request sent from a different application was causing an error in the API endpoint.

To debug the request, I needed to capture what cURL was sending, by setting up netcat to listen on a local port and writing out what it received to a text file:

nc -l 8000 > request.txt

I could then make the same HTTP request with cURL but to the local address:

curl 'http://localhost:8000/foo' \
  -H 'Authorization: Bearer foo' \
  -F 'id=foo' \
  -F 'file=@example.pdf'

request.txt then contained the contents of the HTTP request:

POST /foo HTTP/1.1
Host: localhost:8000
User-Agent: curl/7.69.1
Accept: */*
Authorization: Bearer foo
Content-Length: 123
Content-Type: multipart/form-data; boundary=------------------------368700648dd5f4b5

--------------------------368700648dd5f4b5
Content-Disposition: form-data; name="id"

foo
--------------------------368700648dd5f4b5
Content-Disposition: form-data; name="file"; filename="example.pdf"
Content-Type: application/pdf

[binary data here]
--------------------------368700648dd5f4b5--

Then I needed to edit that request to see what succeeded or failed, sending it to the remote endpoint after each edit.

Before sending it, the Host: line needed to be changed from localhost:8000 to example.com. In many cases it would be fine to make this edit in a text editor, but because VS Code was set up to normalise line endings (LF or CRLF) when a file was saved, it was messing with the line endings within the binary data of the file attachment in the request. I used sed to make the change instead, setting LC_ALL=C to allow it to work with the binary data in the file:

LC_ALL=C && sed -i.bak 's/localhost:8000/example.com/' request.txt

If the remote service was HTTP, the request could have been sent by reading it back into netcat:

nc example.com 80 < request.txt

However, netcat doesn't support HTTPS. I found several ways of sending a raw request to an endpoint over HTTPS:

Using openssl: cat request.txt | openssl s_client -connect example.com:443
Using ncat: cat request.txt | ncat --ssl example.com 443

The problem was that both of those produced an error when sending the stored request (which turned out to be due to the line endings problem described above), with not much in the error message to work out what was wrong.

I decided to make a script using Node's tls library that would send the stored request data over HTTPS:

const tls = require('tls')
const fs = require('fs')

const socket = tls.connect(
  {
    port: 443,
    host: 'example.com',
    servername: 'example.com', // for SNI
  },
  () => {
    const data = fs.readFileSync('request.txt') // read the stored HTTP request
    socket.write(data) // send the HTTP request
  }
)

socket.on('close', () => {
  console.log('Connection closed')
})

socket.on('data', (data) => {
  console.log(`Received ${data.length} bytes`)
  console.log(data.toString('utf8'))
})

socket.on('error', (error) => {
  console.log('Error', error)
})

After a bit of experimenting with sending minimal file attachments and editing the headers in the request, we were able to get to the reason for the error responses: the multipart/form-data parser in the remote API was being too strict about the headers it would accept.