W3cubDocs

/Nim

Module httpclient

This module implements a simple HTTP client that can be used to retrieve webpages and other data.

Retrieving a website

This example uses HTTP GET to retrieve http://google.com:

var client = newHttpClient()
echo client.getContent("http://google.com")

The same action can also be performed asynchronously, simply use the AsyncHttpClient:

var client = newAsyncHttpClient()
echo await client.getContent("http://google.com")

The functionality implemented by HttpClient and AsyncHttpClient is the same, so you can use whichever one suits you best in the examples shown here.

Note: You will need to run asynchronous examples in an async proc otherwise you will get an Undeclared identifier: 'await' error.

Using HTTP POST

This example demonstrates the usage of the W3 HTML Validator, it uses multipart/form-data as the Content-Type to send the HTML to be validated to the server.

var client = newHttpClient()
var data = newMultipartData()
data["output"] = "soap12"
data["uploaded_file"] = ("test.html", "text/html",
  "<html><head></head><body><p>test</p></body></html>")

echo client.postContent("http://validator.w3.org/check", multipart=data)

You can also make post requests with custom headers. This example sets Content-Type to application/json and uses a json object for the body

import httpclient, json

let client = newHttpClient()
client.headers = newHttpHeaders({ "Content-Type": "application/json" })
let body = %*{
    "data": "some text"
}
let response = client.request("http://some.api", httpMethod = HttpPost, body = $body)
echo response.status

Progress reporting

You may specify a callback procedure to be called during an HTTP request. This callback will be executed every second with information about the progress of the HTTP request.

import asyncdispatch, httpclient

proc onProgressChanged(total, progress, speed: BiggestInt) {.async.} =
  echo("Downloaded ", progress, " of ", total)
  echo("Current rate: ", speed div 1000, "kb/s")

proc asyncProc() {.async.} =
  var client = newAsyncHttpClient()
  client.onProgressChanged = onProgressChanged
  discard await client.getContent("http://speedtest-ams2.digitalocean.com/100mb.test")

waitFor asyncProc()

If you would like to remove the callback simply set it to nil.

client.onProgressChanged = nil

Warning: The total reported by httpclient may be 0 in some cases.

SSL/TLS support

This requires the OpenSSL library, fortunately it's widely used and installed on many operating systems. httpclient will use SSL automatically if you give any of the functions a url with the https schema, for example: https://github.com/.

You will also have to compile with ssl defined like so: nim c -d:ssl ....

Timeouts

Currently only the synchronous functions support a timeout. The timeout is measured in milliseconds, once it is set any call on a socket which may block will be susceptible to this timeout.

It may be surprising but the function as a whole can take longer than the specified timeout, only individual internal calls on the socket are affected. In practice this means that as long as the server is sending data an exception will not be raised, if however data does not reach the client within the specified timeout a TimeoutError exception will be raised.

Proxy

A proxy can be specified as a param to any of the procedures defined in this module. To do this, use the newProxy constructor. Unfortunately, only basic authentication is supported at the moment.

Exports

Http417, Http503, Http431, ==, contains, Http304, Http406, ==, $, clear, Http408, $, Http411, is3xx, Http418, Http206, HttpPost, HttpMethod, Http101, Http505, Http413, newHttpHeaders, HttpVer11, HttpTrace, Http200, []=, Http414, add, Http401, Http205, ==, Http407, Http500, Http404, Http416, Http302, HttpHeaders, Http300, Http428, Http410, is2xx, Http202, Http502, headerLimit, HttpHeaderValues, contains, newHttpHeaders, $, [], HttpHead, Http305, Http451, Http409, Http504, Http426, HttpConnect, hasKey, del, HttpPut, pairs, Http429, HttpVersion, HttpVer10, []=, Http421, HttpOptions, Http307, Http301, HttpPatch, is4xx, Http203, getOrDefault, Http100, Http501, HttpDelete, len, Http400, Http403, HttpGet, is5xx, Http415, toString, Http412, Http405, Http303, Http204, Http201, HttpCode, httpcore, Http422, []

Imports

net, strutils, uri, parseutils, strtabs, base64, os, mimetypes, math, random, httpcore, times, tables, streams, asyncnet, asyncdispatch, asyncfile, nativesockets

Types

Response = ref object
  version*: string
  status*: string
  headers*: HttpHeaders
  body: string
  bodyStream*: Stream
AsyncResponse = ref object
  version*: string
  status*: string
  headers*: HttpHeaders
  body: string
  bodyStream*: FutureStream[string]
Proxy = ref object
  url*: Uri
  auth*: string
MultipartEntries = openArray[tuple[name, content: string]]
MultipartData = ref object
  content: seq[string]
ProtocolError = object of IOError
exception that is raised when server does not conform to the implemented protocol 
HttpRequestError = object of IOError
Thrown in the getContent proc and postContent proc, when the server returns an error 
ProgressChangedProc[ReturnType] = proc (total, progress, speed: BiggestInt): ReturnType {...}{.
    closure, gcsafe.}
HttpClientBase[SocketType] = ref object
  socket: SocketType
  connected: bool
  currentURL: Uri
  headers*: HttpHeaders        ## Headers to send in requests.
  maxRedirects: int
  userAgent: string
  timeout: int                 ## Only used for blocking HttpClient for now.
  proxy: Proxy                 ## ``nil`` or the callback to call when request progress changes.
  when SocketType is Socket:
    onProgressChanged
  else:
    onProgressChanged
  when false:
    sslContext
  contentTotal: BiggestInt
  contentProgress: BiggestInt
  oneSecondProgress: BiggestInt
  lastProgressReport: float
  when SocketType is AsyncSocket:
      bodyStream
      parseBodyFut

  else:
    bodyStream
  getBody: bool                ## When `false`, the body is never read in requestAux.
Where we are currently connected. 
HttpClient = HttpClientBase[Socket]
AsyncHttpClient = HttpClientBase[AsyncSocket]

Consts

defUserAgent = "Nim httpclient/0.19.0"

Procs

proc code(response: Response | AsyncResponse): HttpCode {...}{.
    raises: [ValueError, OverflowError].}

Retrieves the specified response's HttpCode.

Raises a ValueError if the response's status does not have a corresponding HttpCode.

proc contentType(response: Response | AsyncResponse): string

Retrieves the specified response's content type.

This is effectively the value of the "Content-Type" header.

proc contentLength(response: Response | AsyncResponse): int

Retrieves the specified response's content length.

This is effectively the value of the "Content-Length" header.

A ValueError exception will be raised if the value is not an integer.

proc lastModified(response: Response | AsyncResponse): DateTime

Retrieves the specified response's last modified time.

This is effectively the value of the "Last-Modified" header.

Raises a ValueError if the parsing fails or the value is not a correctly formatted time.

proc body(response: Response): string {...}{.raises: [Exception], tags: [ReadIOEffect].}

Retrieves the specified response's body.

The response's body stream is read synchronously.

proc body=(response: Response; value: string) {...}{.deprecated, raises: [], tags: [].}

Setter for backward compatibility.

This is deprecated and should not be used.

proc body(response: AsyncResponse): Future[string] {...}{.raises: [FutureError],
    tags: [RootEffect].}
Reads the response's body and caches it. The read is performed only once. 
proc newProxy(url: string; auth = ""): Proxy {...}{.raises: [], tags: [].}
Constructs a new TProxy object. 
proc newMultipartData(): MultipartData {...}{.raises: [], tags: [].}
Constructs a new MultipartData object. 
proc add(p: var MultipartData; name, content: string; filename: string = "";
        contentType: string = "") {...}{.raises: [ValueError], tags: [].}
Add a value to the multipart data. Raises a ValueError exception if name, filename or contentType contain newline characters. 
proc add(p: var MultipartData; xs: MultipartEntries): MultipartData {...}{.discardable,
    raises: [ValueError], tags: [].}
Add a list of multipart entries to the multipart data p. All values are added without a filename and without a content type.
data.add({"action": "login", "format": "json"})
proc newMultipartData(xs: MultipartEntries): MultipartData {...}{.raises: [ValueError],
    tags: [].}
Create a new multipart data object and fill it with the entries xs directly.
var data = newMultipartData({"action": "login", "format": "json"})
proc addFiles(p: var MultipartData; xs: openArray[tuple[name, file: string]]): MultipartData {...}{.
    discardable, raises: [ValueError, IOError], tags: [ReadIOEffect].}
Add files to a multipart data object. The file will be opened from your disk, read and sent with the automatically determined MIME type. Raises an IOError if the file cannot be opened or reading fails. To manually specify file content, filename and MIME type, use []= instead.
data.addFiles({"uploaded_file": "public/test.html"})
proc `[]=`(p: var MultipartData; name, content: string) {...}{.raises: [ValueError], tags: [].}
Add a multipart entry to the multipart data p. The value is added without a filename and without a content type.
data["username"] = "NimUser"
proc `[]=`(p: var MultipartData; name: string;
          file: tuple[name, contentType, content: string]) {...}{.raises: [ValueError],
    tags: [].}
Add a file to the multipart data p, specifying filename, contentType and content manually.
data["uploaded_file"] = ("test.html", "text/html",
  "<html><head></head><body><p>test</p></body></html>")
proc request(url: string; httpMethod: string; extraHeaders = ""; body = "";
            sslContext = getDefaultSSL(); timeout = -1; userAgent = defUserAgent;
            proxy: Proxy = nil): Response {...}{.deprecated: "use HttpClient.request instead", raises: [
    OSError, OSError, HttpRequestError, SslError, TimeoutError, ProtocolError,
    KeyError, Exception, OverflowError, ValueError], tags: [ReadIOEffect,
    WriteIOEffect, TimeEffect].}

Requests url with the custom method string specified by the
httpMethod parameter.
Extra headers can be specified and must be separated by \c\L
An optional timeout can be specified in milliseconds, if reading from the

server takes longer than specified an ETimeout exception will be raised.

Deprecated since version 0.15.0: use HttpClient.request instead.

proc request(url: string; httpMethod = HttpGET; extraHeaders = ""; body = "";
            sslContext = getDefaultSSL(); timeout = -1; userAgent = defUserAgent;
            proxy: Proxy = nil): Response {...}{.deprecated, raises: [OSError,
    HttpRequestError, SslError, TimeoutError, ProtocolError, KeyError, Exception,
    OverflowError, ValueError], tags: [ReadIOEffect, WriteIOEffect, TimeEffect].}

Requests url with the specified httpMethod.
Extra headers can be specified and must be separated by \c\L
An optional timeout can be specified in milliseconds, if reading from the

server takes longer than specified an ETimeout exception will be raised.

Deprecated since version 0.15.0: use HttpClient.request instead.

proc get(url: string; extraHeaders = ""; maxRedirects = 5;
        sslContext: SSLContext = getDefaultSSL(); timeout = -1;
        userAgent = defUserAgent; proxy: Proxy = nil): Response {...}{.deprecated, raises: [
    OSError, HttpRequestError, SslError, TimeoutError, ProtocolError, KeyError,
    Exception, OverflowError, ValueError],
    tags: [ReadIOEffect, WriteIOEffect, TimeEffect].}

GETs the url and returns a Response object
This proc also handles redirection
Extra headers can be specified and must be separated by \c\L.
An optional timeout can be specified in milliseconds, if reading from the

server takes longer than specified an ETimeout exception will be raised.

Deprecated since version 0.15.0: use HttpClient.get instead.

proc getContent(url: string; extraHeaders = ""; maxRedirects = 5;
               sslContext: SSLContext = getDefaultSSL(); timeout = -1;
               userAgent = defUserAgent; proxy: Proxy = nil): string {...}{.deprecated, raises: [
    OSError, HttpRequestError, SslError, TimeoutError, ProtocolError, KeyError,
    Exception, OverflowError, ValueError],
    tags: [ReadIOEffect, WriteIOEffect, TimeEffect].}

GETs the body and returns it as a string.
Raises exceptions for the status codes 4xx and 5xx
Extra headers can be specified and must be separated by \c\L.
An optional timeout can be specified in milliseconds, if reading from the

server takes longer than specified an ETimeout exception will be raised.

Deprecated since version 0.15.0: use HttpClient.getContent instead.

proc post(url: string; extraHeaders = ""; body = ""; maxRedirects = 5;
         sslContext: SSLContext = getDefaultSSL(); timeout = -1;
         userAgent = defUserAgent; proxy: Proxy = nil; multipart: MultipartData = nil): Response {...}{.
    deprecated, raises: [OSError, HttpRequestError, SslError, TimeoutError,
                       ProtocolError, KeyError, Exception, OverflowError, ValueError],
    tags: [ReadIOEffect, WriteIOEffect, TimeEffect].}

POSTs body to the url and returns a Response object.
This proc adds the necessary Content-Length header.
This proc also handles redirection.
Extra headers can be specified and must be separated by \c\L.
An optional timeout can be specified in milliseconds, if reading from the

server takes longer than specified an ETimeout exception will be raised. | The optional multipart parameter can be used to create multipart/form-data POSTs comfortably.

Deprecated since version 0.15.0: use HttpClient.post instead.

proc postContent(url: string; extraHeaders = ""; body = ""; maxRedirects = 5;
                sslContext: SSLContext = getDefaultSSL(); timeout = -1;
                userAgent = defUserAgent; proxy: Proxy = nil;
                multipart: MultipartData = nil): string {...}{.deprecated, raises: [OSError,
    HttpRequestError, SslError, TimeoutError, ProtocolError, KeyError, Exception,
    OverflowError, ValueError], tags: [ReadIOEffect, WriteIOEffect, TimeEffect].}

POSTs body to url and returns the response's body as a string
Raises exceptions for the status codes 4xx and 5xx
Extra headers can be specified and must be separated by \c\L.
An optional timeout can be specified in milliseconds, if reading from the

server takes longer than specified an ETimeout exception will be raised. | The optional multipart parameter can be used to create multipart/form-data POSTs comfortably.

Deprecated since version 0.15.0: use HttpClient.postContent instead.

proc downloadFile(url: string; outputFilename: string;
                 sslContext: SSLContext = getDefaultSSL(); timeout = -1;
                 userAgent = defUserAgent; proxy: Proxy = nil) {...}{.deprecated, raises: [
    IOError, OSError, HttpRequestError, SslError, TimeoutError, ProtocolError,
    KeyError, Exception, OverflowError, ValueError],
    tags: [WriteIOEffect, ReadIOEffect, TimeEffect].}

Downloads url and saves it to outputFilename
An optional timeout can be specified in milliseconds, if reading from the

server takes longer than specified an ETimeout exception will be raised.

Deprecated since version 0.16.2: use HttpClient.downloadFile instead.

proc newHttpClient(userAgent = defUserAgent; maxRedirects = 5;
                  sslContext = getDefaultSSL(); proxy: Proxy = nil; timeout = -1): HttpClient {...}{.
    raises: [], tags: [].}

Creates a new HttpClient instance.

userAgent specifies the user agent that will be used when making requests.

maxRedirects specifies the maximum amount of redirects to follow, default is 5.

sslContext specifies the SSL context to use for HTTPS requests.

proxy specifies an HTTP proxy to use for this HTTP client's connections.

timeout specifies the number of milliseconds to allow before a TimeoutError is raised.

proc newAsyncHttpClient(userAgent = defUserAgent; maxRedirects = 5;
                       sslContext = getDefaultSSL(); proxy: Proxy = nil): AsyncHttpClient {...}{.
    raises: [], tags: [].}

Creates a new AsyncHttpClient instance.

userAgent specifies the user agent that will be used when making requests.

maxRedirects specifies the maximum amount of redirects to follow, default is 5.

sslContext specifies the SSL context to use for HTTPS requests.

proxy specifies an HTTP proxy to use for this HTTP client's connections.

proc close(client: HttpClient | AsyncHttpClient)
Closes any connections held by the HTTP client. 
proc request(client: AsyncHttpClient; url: string; httpMethod: string; body = "";
            headers: HttpHeaders = nil): Future[AsyncResponse] {...}{.
    raises: [FutureError], tags: [RootEffect, TimeEffect].}

Connects to the hostname specified by the URL and performs a request using the custom method string specified by httpMethod.

Connection will be kept alive. Further requests on the same client to the same hostname will not require a new connection to be made. The connection can be closed by using the close procedure.

This procedure will follow redirects up to a maximum number of redirects specified in client.maxRedirects.

proc request(client: HttpClient; url: string; httpMethod: string; body = "";
            headers: HttpHeaders = nil): Response {...}{.raises: [HttpRequestError,
    SslError, OSError, IOError, TimeoutError, ProtocolError, KeyError, Exception,
    OverflowError, ValueError], tags: [ReadIOEffect, WriteIOEffect, TimeEffect].}

Connects to the hostname specified by the URL and performs a request using the custom method string specified by httpMethod.

Connection will be kept alive. Further requests on the same client to the same hostname will not require a new connection to be made. The connection can be closed by using the close procedure.

This procedure will follow redirects up to a maximum number of redirects specified in client.maxRedirects.

proc request(client: AsyncHttpClient; url: string; httpMethod = HttpGET; body = "";
            headers: HttpHeaders = nil): Future[AsyncResponse] {...}{.
    raises: [FutureError], tags: [RootEffect, TimeEffect].}

Connects to the hostname specified by the URL and performs a request using the method specified.

Connection will be kept alive. Further requests on the same client to the same hostname will not require a new connection to be made. The connection can be closed by using the close procedure.

When a request is made to a different hostname, the current connection will be closed.

proc request(client: HttpClient; url: string; httpMethod = HttpGET; body = "";
            headers: HttpHeaders = nil): Response {...}{.raises: [HttpRequestError,
    SslError, OSError, IOError, TimeoutError, ProtocolError, KeyError, Exception,
    OverflowError, ValueError], tags: [ReadIOEffect, WriteIOEffect, TimeEffect].}

Connects to the hostname specified by the URL and performs a request using the method specified.

Connection will be kept alive. Further requests on the same client to the same hostname will not require a new connection to be made. The connection can be closed by using the close procedure.

When a request is made to a different hostname, the current connection will be closed.

proc get(client: AsyncHttpClient; url: string): Future[AsyncResponse] {...}{.
    raises: [FutureError], tags: [RootEffect, TimeEffect].}

Connects to the hostname specified by the URL and performs a GET request.

This procedure will follow redirects up to a maximum number of redirects specified in client.maxRedirects.

proc get(client: HttpClient; url: string): Response {...}{.raises: [HttpRequestError,
    SslError, OSError, IOError, TimeoutError, ProtocolError, KeyError, Exception,
    OverflowError, ValueError], tags: [ReadIOEffect, WriteIOEffect, TimeEffect].}

Connects to the hostname specified by the URL and performs a GET request.

This procedure will follow redirects up to a maximum number of redirects specified in client.maxRedirects.

proc getContent(client: AsyncHttpClient; url: string): Future[string] {...}{.
    raises: [FutureError], tags: [RootEffect, TimeEffect].}

Connects to the hostname specified by the URL and performs a GET request.

This procedure will follow redirects up to a maximum number of redirects specified in client.maxRedirects.

A HttpRequestError will be raised if the server responds with a client error (status code 4xx) or a server error (status code 5xx).

proc getContent(client: HttpClient; url: string): string {...}{.raises: [HttpRequestError,
    SslError, OSError, IOError, TimeoutError, ProtocolError, KeyError, Exception,
    OverflowError, ValueError], tags: [ReadIOEffect, WriteIOEffect, TimeEffect].}

Connects to the hostname specified by the URL and performs a GET request.

This procedure will follow redirects up to a maximum number of redirects specified in client.maxRedirects.

A HttpRequestError will be raised if the server responds with a client error (status code 4xx) or a server error (status code 5xx).

proc post(client: AsyncHttpClient; url: string; body = "";
         multipart: MultipartData = nil): Future[AsyncResponse] {...}{.
    raises: [FutureError], tags: [RootEffect, TimeEffect].}

Connects to the hostname specified by the URL and performs a POST request.

This procedure will follow redirects up to a maximum number of redirects specified in client.maxRedirects.

proc post(client: HttpClient; url: string; body = ""; multipart: MultipartData = nil): Response {...}{.raises: [
    HttpRequestError, SslError, OSError, IOError, TimeoutError, ProtocolError,
    KeyError, Exception, OverflowError, ValueError],
    tags: [ReadIOEffect, WriteIOEffect, TimeEffect].}

Connects to the hostname specified by the URL and performs a POST request.

This procedure will follow redirects up to a maximum number of redirects specified in client.maxRedirects.

proc postContent(client: AsyncHttpClient; url: string; body = "";
                multipart: MultipartData = nil): Future[string] {...}{.
    raises: [FutureError], tags: [RootEffect, TimeEffect].}

Connects to the hostname specified by the URL and performs a POST request.

This procedure will follow redirects up to a maximum number of redirects specified in client.maxRedirects.

A HttpRequestError will be raised if the server responds with a client error (status code 4xx) or a server error (status code 5xx).

proc postContent(client: HttpClient; url: string; body = "";
                multipart: MultipartData = nil): string {...}{.raises: [HttpRequestError,
    SslError, OSError, IOError, TimeoutError, ProtocolError, KeyError, Exception,
    OverflowError, ValueError], tags: [ReadIOEffect, WriteIOEffect, TimeEffect].}

Connects to the hostname specified by the URL and performs a POST request.

This procedure will follow redirects up to a maximum number of redirects specified in client.maxRedirects.

A HttpRequestError will be raised if the server responds with a client error (status code 4xx) or a server error (status code 5xx).

proc downloadFile(client: HttpClient; url: string; filename: string) {...}{.raises: [
    HttpRequestError, SslError, OSError, IOError, TimeoutError, ProtocolError,
    KeyError, Exception, OverflowError, ValueError],
    tags: [ReadIOEffect, WriteIOEffect, TimeEffect].}
Downloads url and saves it to filename. 
proc downloadFile(client: AsyncHttpClient; url: string; filename: string): Future[void] {...}{.
    raises: [FutureError, Exception], tags: [RootEffect, TimeEffect].}
Downloads url and saves it to filename.

© 2006–2018 Andreas Rumpf
Licensed under the MIT License.
https://nim-lang.org/docs/httpclient.html