REST API development

API, REST, WEB

The article presents a set of tools required to create a REST API. The tools are platform independent, meaning they are applicable to REST APIs built using any technology stack. The purpose of this article is to introduce beginner API developers to the various stages of API development and to introduce tools that help in these stages. Detailed coverage of these tools can be found online. The API development steps are listed below.

1. Design. The main goal here is to define the form of the API, document interfaces, and provide endpoints.
2. Testing. At this stage, functional testing of the API is carried out by sending a request and analyzing the response at different levels of visibility, namely: application, HTTP, network.
3. Web Hosting. When deployed on the web, there are HTTP tools that help with hosting APIs to improve performance, security, and reliability.
4. Performance Before we go to work, we use API performance testing tools to inform us about the load that the APIs can support.
5. Observable. Once the API is deployed to production, testing in production ensures the overall health of the APIs and alerts us to any issues.
6. Management. Finally, we'll look at some API management tools such as traffic shaping, deployment, and so on.

The following figure shows the different steps and the tools used.

Let's illustrate the use of the API tools provided by the web application in the development of each phase of the API. product catalog
is a Spring Boot web application that manages a product catalog. It provides a REST API to perform CRUD operations on a product catalog. The code is available here .

Design

During the design phase, the API developer interacts with the API clients and the data provider to obtain the API form. The REST API essentially consists of JSON messaging over HTTP. JSON is the dominant format in REST APIs because it is compact, easy to understand, and has a flexible format that doesn't require a schema description in advance. In addition, different clients can use the same API and read the data they need.
Consider an API project using Swagger. This tool uses an open format to describe the API, combined with a web interface to visualize information about the interfaces. There is no separation between design and implementation. The benefit of Swagger is that the API and documentation will also be kept in sync, with the documentation hosted alongside the API. The disadvantage is that only API developers can change the structure of the API. Since the documentation is generated from the API, we first need to create a skeleton of our API. Used by Spring Boot for API development and Springfox package for documentation generation.
Add swagger 2 and swagger-ui maven dependencies to your pom.xml.

<dependency>
    <groupId>io.springfox</groupId>
    <artifactId>springfox-swagger2</artifactId>
    <version>2.6.1</version>
</dependency>
<dependency>
    <groupId>io.springfox</groupId>
    <artifactId>springfox-swagger-ui</artifactId>
    <version>2.5.0</version>
</dependency>

Add SwaggerConfig.java to the project with the following content:

package com.rks.catalog.configuration;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import springfox.documentation.builders.PathSelectors;
import springfox.documentation.builders.RequestHandlerSelectors;
import springfox.documentation.spi.DocumentationType;
import springfox.documentation.spring.web.plugins.Docket;
import springfox.documentation.swagger2.annotations.EnableSwagger2;
@Configuration
@EnableSwagger2
public class SwaggerConfig {
    @Bean
    public Docket api() {
        return new Docket(DocumentationType.SWAGGER_2)
        .select()
        .apis(RequestHandlerSelectors.any())
        .paths(PathSelectors.any()).build();
    }
}

This configuration tells Swagger to crawl all controllers and include all URLs defined in those controllers for API documentation.
After running the application, you can view the Swagger API documentation at the URL:

http://localhost:8080/swagger-ui.html

Click on each API to see details - URL, HTTP headers and HTTP body. Useful is the "Try it out!" button, which provides a sandbox, allowing users to play around with the API to learn how it works before they start plugging it into their applications.

Testing

Functional testing of REST APIs entails sending HTTP requests and validating responses so that we can test whether the APIs work as we expect them to. REST uses HTTP as a transport that defines API request and response formats. TCP/IP in turn receives HTTP messages and decides how to send them over the wire. Introducing three toolkits for testing APIs at three layers of the protocol stack, namely REST clients for the REST layer, web debuggers for the HTTP layer, and packet sniffers for the TCP/IP layer.

• Postman is a REST client that allows us to test the REST API. This allows us to:
• Create HTTP requests and generate equivalent cURL commands that can be used in scripts.
• Create multiple environments for Dev, Test, Pre-Prod as each environment has different configurations.
• Create a test collection that has multiple tests for each area of the product. Different parts of the test can be parameterized, allowing us to switch between environments.
• Create code snippets in JavaScript to extend tests. For example, to validate return codes or set environment variables.
• Automate running tests with the Newman command line tool.
• Import / export test collections and environments.

• cURL is a command line tool that uses its own HTTP stack and is available cross-platform.

curl -X POST \
  http://localhost:8080/books \
  -H 'Cache-Control: no-cache' \
  -H 'Content-Type: application/json' \
  -d '{
"id":"1",
"author":"shakespeare",
"title":"hamlet"
}'

• Burp is an HTTP debugger that allows us to see the web traffic that passes between the client and the API. It acts as a proxy between the client and the server, allowing us to intercept the request and response and modify them to create scenarios that are otherwise hard to test without modifying the client. Burp is a toolkit that is mainly used for security testing, but can also be very useful for API testing. Set up your Postman to send the request to the Burp proxy, and then set up Burp to intercept the client's request and the server's response.
  Intercept the request and response as shown below.

• Wireshark - Testing some API functions like encryption, compression, etc. will require us to take a deeper look at what is being sent and received on the network. Wireshark is a tool that monitors a network interface and keeps a copy of all TCP packets that pass through it. Traffic is divided into layers - HTTP, TCP, IP, etc. It is an assistant in troubleshooting problems that require more in-depth study, such as TLS handshaking.

Web Hosting

In this section, we'll look at some of the features of the HTTP protocol that, when used correctly, help provide high performance, highly available, reliable, and secure APIs. Specifically, let's look at the three parts of the HTTP protocol - caching for performance, DNS for high availability and scalability, and TLS for transport security.

• Caching is one of the best ways to improve client performance and reduce API load. HTTP allows clients to store a copy of a resource locally by sending a caching header in response, so that the next time the client sends an HTTP request for the same resource served from the local cache. This saves network traffic and computational load on the API.
• HTTP 1.0 expiration caching provides an Expires header in an HTTP response indicating the time a resource will expire. This can be useful for a shared resource with a fixed expiration date.
• HTTP 1.1 expiration caching provides more flexible expiration header caching control that instructs the client to cache the resource for the period specified in the max-age value. There is another s-maxage value that can be set for intermediaries, such as a caching proxy.
• HTTP validation caching. There is an issue with caching with a client having an outdated resource or two clients having different versions of the same resource. If this is not acceptable, or if there are personalized resources that cannot be cached, such as authentication tokens, HTTP provides validation caching. With HTTP validation caching, provides headers in the response Etag or last modified timestamp. If the API returns one of the two headers, clients cache it and include it in subsequent GET calls to the API.

GET http://api.endpoint.com/books
If-none-match: "4v44ffgg1e"

• DNS - Domain Name System finds IP addresses for a domain name so that clients can route their request to the correct server. When an HTTP request is made, clients first query the DNS server to find the host address and then send the query directly to the IP address. DNS is a layered system that is heavily cached to ensure queries don't get slowed down. Clients maintain a DNS cache, then there are intermediate DNS servers leading to the nameserver. DNS provides CNAMEs (canonical names) for accessing different parts of a server, for example an API and a web server can be hosted on the same server with two different CNAMEs - api.endpoint.com and www.endpoint.com - or CNAMEs can point to different servers. CNAMEs also allow us to separate parts of our API. For HTTP GET requests, we can have a separate CNAME for static and transactional resources, allowing us to set up an external proxy for resources we know might be cached. We can also have a CNAME for HTTP POST requests to separate reads and writes so we can scale them independently. Or we can provide a fast lane for priority customers.

When using extended DNS such as Route53, a single CNAME can point to multiple servers instead of just pointing to one server. The routing policy can then be configured for weighted routing, delay routing, or failover.

• TLS - We can secure our APIs with TLS, which allows us to serve our request over HTTPS. HTTPS works on the basic principle of key pair security. To enable HTTPS in our API, we need a certificate on our server that contains a public/private key pair. The server sends the public key to the client, which uses it to encrypt data, and the server uses its private key to decrypt it. When a client first connects to an HTTPS endpoint, a "handshake" occurs, meaning the client and server agree on how to encrypt the traffic. A key is exchanged that is unique to the session and is used to encrypt and decrypt data for the duration of the session. During the initial handshake, there is a performance hit due to asymmetric encryption, but once the connection is established, symmetric encryption is used, which is quite fast.

In order for proxies to cache TLS traffic, we must download the same certificate that is used to encrypt the traffic. The proxy must be able to decrypt the traffic, store it in its cache, encrypt it with the same certificate, and send it to the client. Some proxy servers do not allow this. In this case, the solution is to have two CNAMEs - one for static cached resources over HTTP and one for non-cached personalized resources whose requests over a secure TLS channel will be serviced by the API directly.

Performance

In this section, we'll look at tools to load test our API to quantify the amount of traffic our infrastructure can handle. The main idea of performance testing is to send a large number of requests to an API at the same time and determine at what point performance drops and crashes.

Here are the questions we need answers to:

• What response time can the API give under different load conditions?
• How many simultaneous requests can the API process without errors?
• What infrastructure is required to achieve the desired performance?

loader.io is a free cloud-based load testing service that allows us to stress test our APIs. To get the basic performance of the API, you can run various kinds of load tests with increasing loads, measured in requests per second to determine performance metrics, quantified by errors and response times, for:

• endurance test - average load over long periods, such as 48 hours at 1 request per second. This will detect any memory leaks or other similar hidden errors.
• Load test - peak load, for example, running 2K requests per second with 6 API instances.
• Stress test - peak load during overload, for example, executing 10,000 requests per second for 10 minutes.

It also allows us to define an infrastructure that will allow us to provide APIs with the required performance and linear scaling of our solution.

Surveillance Capability

Deploying an API in production doesn't mean we can forget about the API. Production rollout kicks off another phase of testing, in-production testing, which can uncover issues that were not discovered in earlier phases. Testing in the production process includes a set of activities combined into a single whole: observability (include logging), monitoring, tracking. Tools for these actions will help diagnose and resolve problems found in production.

• Logging should be done explicitly by developers using their preferred logging environment and logging standard. For example, one log statement for every 10 or more lines of code if the code is complex with log levels divided into - 60% DEBUG, 25% INFO 10% WARN and 5% ERROR.
• Monitoring - performed at a higher level than registration. While logging tells us explicitly what is going on with the API, monitoring ensures the overall health of the API using common metrics provided by the platform and the API itself. Metrics are typically provided by an agent deployed on a server, or they can be part of a solution and collected periodically by a monitoring solution deployed remotely.

The solution can include diagnostic endpoints that report the general state of the API.

• Trace - Zipkin is a distributed tracing system. Helps collect the timing data needed to troubleshoot latency issues in microservice architectures.

Enabling centralized logging covers logging and tracing. For monitoring, interesting metrics can be stored in a time series store such as Prometheus and visualized using Grafana.

Management

API management tools serve as a gateway providing services:

• API clients secure themselves by receiving an API key;
• API providers configure DNS, caching, throttling policies, API versioning, etc.

These features and more are available on AWS API Gateway.