Synchronize your tests with your documentation

When you develop your API, a part of the job is to write the documentation. At Fabernovel, we used to write our documentation on a tool called Slate. However, by lack of time, clear process or experience, it happened that API were modified without the documentation. The longer our API lived, the more outdated or incorrect the documentation was.

Our documentation tool

To solve this problem, we decided to automate the detection of outdated documentation by developing our own tool called Pericles. One of the main feature of Pericles is a proxy. You can plug Pericles between your client and your server. Because Pericles acts as middle man, it can analyse the requests and responses and check if they are conformed to the specifications. It’s very useful to detect typo errors, wrong types or evolutions that were not specified. If you want to learn more about why and how we develop this tool you can check this awesome presentation made by Hugo Hache and Julie Rollin: French / English

But is this proxy feature enough to solve our problem of outdated or incorrect documentation? Well, the answer is no. If we only rely on this proxy, we need to wait that a request is performed to be analysed. So, we need to wait for our API to be deployed somewhere and for someone to use it. And to allow someone to use it, the documentation must be up to date.

In order to have earlier feedbacks, we can use automated tests to check that our API is correct according to the documentation.

Automated tests

To analyse the response and the request, Pericles uses json schemas, we can do the same in our tests.

The first step is to add the json-schema gem in our Gemfile

gem 'json-schema'

Then we add the json schemas of our requests and responses in our tests folders. For example, if we have a Product resource in our API our tests folder will look like this:

├── json_schemas
│   ├── product
│   │   ├── get_products_id_detailedproduct_200.json_schema
│   │   ├── get_products_shortproduct_200.json_schema
│   │   ├── post_products_detailedproduct_201.json_schema
│   │   ├── put_products_id_detailedproduct_200.json_schema
│   │   ├── request_post_products_createproduct.json_schema
│   │   └── request_put_products_id_updateproduct.json_schema

Then in our test_helper.rb, we add two new helpers to validate the response and the request.

def validate_json(json_schema_path, json)
  path = "test/json_schemas#{json_schema_path}.json_schema"
  assert_empty JSON::Validator.fully_validate(path, json).join("\n")
end

def validate_json_response_body(json_schema_path)
  validate_json(json_schema_path, response.body)
end

def validate_json_request_body(json_schema_path)
  validate_json(json_schema_path, request.request_parameters)
end

Finally in our controller tests, we can use those helpers to validate that the generated response and the sent request are correct according to the specification.

test 'should get products' do
  get products_path

  assert_response :success
  validate_json_response_body '/product/get_products_shortproduct_200'
  assert_not_empty response.parsed_body['products']
end

test 'should get product' do
  get product_path(@product)

  assert_response :success
  validate_json_response_body '/product/get_products_id_detailedproduct_200'
end

test 'should create product' do
  assert_difference -> { Product.count } do
    post_product
  end

  assert_response :created
  validate_json_request_body '/product/request_post_products_createproduct'
  validate_json_response_body '/product/post_products_detailedproduct_201'
end

In order to detect extra fields in the json that are not allowed, we set additionalProperties to false in our json schemas. You can learn more about it here.

There is one more step to do: automate the synchronization between our documentation and our json schemas folder. Luckily, Pericles provides a route to download all the json-schemas so we can create a rake task to automate it.

namespace :pericles do
  desc 'Load all json schema in test json schemas folder'
  task schemas: :environment do
    config = YAML.load_file('.pericles.yml')
    project_id = config['project_id']
    session = config['session']
  
    url = URI.parse("https://ad-pericles.herokuapp.com/projects/#{project_id}.json_schema")
    request = Net::HTTP::Get.new(url, { 'Cookie': "_Pericles_GW_gw_session=#{session}" })
    zip_data = Net::HTTP.start(url.host, url.port, use_ssl: true) do |http|
      http.request request
    end

    Zip::InputStream.open(StringIO.new(zip_data.body)) do |io|
      while (entry = io.get_next_entry)
        folders = entry.to_s.split('/')[0...-1].join('/')
        FileUtils.mkdir_p "test/json_schemas/#{folders}"
        IO.binwrite("test/json_schemas/#{entry}", io.read)
      end
    end
  end
end

And voilà! By running this task when the documentation is updated, we can guarantee that our code is correctly implementing the specifications.

Conclusion

By writing those tests, we can detect many common mistakes that are made when we document our API:

• Numbers are serialized as string
• Fields are nullable but marked as non-nullable
• Update requests should have optional arguments but are marked as required.

We got a feedback early and solves the issues before deploying our API.

When we update our serializers, we can do it without fear of breaking anything. If it were used somewhere we were not aware of or we did not update the documentation, a test will fail. Furthermore if we update the documentation, for example by adding a new required attribute, we can rely on our tests to tell us where our serializers should be updated.

Before, we used to write code, push the code to our version control system and then write the documentation. Now the CI will prevent the code to be pushed if the documentation is not updated. We can no longer forget this step.

However, this solution is not perfect. Pericles does not handle versioning. If we have a long feature branch or several developers working at the same time, the automated synchronization can be painful to use as it will download all modifications made.

If you are using OpenAPI and not Pericles for your documentation, you can still use this technique because OpenAPI also relies on json schemas.