3dverse-testtask/README.md

3Dverse test task

Solution

Technology choosen

1. deno instead of nodejs as platform.
2. deno kv for database.
3. nix flake as build system

Development environment

If the developper has nix flake installed. nix develop will install all the system dependencies. Otherwise, deno is required; hey, and psrecord are two useful tool to benchmark the server.

Running the tests.

The best way to build the test is to run nix build .#test which will output a report in ./result. Otherwise, deno test --unstable tests should run all the tests.

Running the project.

The main file to run the server is src/server.ts. To command line deno run --unstable --allow-net src/server.ts will run the server locally. One can add the flag --v8-flags=--max-heap-size=50 to limit the allowed memory to, in this example, 50mb.

In a similar fashion, it is possible to run the application without checking out the code using the command line

deno run  --v8-flags=--max-heap-size=50  --unstable --allow-net https://gitlab.com/didiercrunch/3dverse-testtask/-/raw/master/src/server.ts

Architecture

The two main layers of abstraction are the FileService and BlobService. The BlobService persist data as naivly as possible. It does so by chuncking the data and saves it in Deno KV. The FileService manages the file metadata such as the path, creation date, ...

sequenceDiagram
    title: Persist file
    client->>RestAPI: HTTP request
    RestAPI->>FileService: Persist file metadata
    FileService->>BlobService: Persist bytes

The transformation part is done by the WaveFrontService and is 100% agnostic of the persistace layer.

Api

The REST api is defined in the restapi.ts file. The Api tries to be as RESTful as possible.

endpoint	method	params	description
/v1/files	GET	none	Lists all the files (no pagination)
/v1/files/:fileId	GET	none	Get the full file description
/v1/files/:fileId	DELETE	none	Delete the file
/v1/files/	POST	path in url params and the file content as body	Create a new file with specified path and content.
/v1/files/:fileId	PUT	{path: newPath}	Change the path of the file.
/v1/content/:fileId/raw	GET	none	Download the file
/content/:fileId/transform	GET	scaleX, scaleY, scaleZ, translateX, translateY, translateZ url params	Will download the transformed file

Todo

Data deletion

Deleting data in these types of application creates a massive race condition. Basically, if someone is deleting a file while someone else is accessing it, the user that deletes the file should not break the user that access the file.

The solution to implement here is a kind of garbage collector of the "blobs". This is not trivial as it is required to add two new indexed; an index that specify if a blob is refered by a (not deleted) file and an index that specify if a blob is scheduled to be read.

HTTP Cache

Caching is very important in these type of applications. The api is made to be easy to add proper cache.

Swagger

Apparently this needs to be done.. god knows how much the author of this line hates swagger.

Frontend

Hooking the frontend is not yet done.

Implementing sqlite backend

Deno KV has been choosen by mistake. It was designed to only be used for testing. Once the author discovered that Deno KV was a thin layer about FoundationDB, the most perfect database, it was necessary to use it.

Nonetheless, it is important to notice that Deno KV is backed by a wasm verison of sqlite locally and suffers from lock contention when multiple requests read from the database.

A good ol' WAL sqlite should do the job.

A CLI (Or something like it)

The end user should be able to change on which port the process is runnning wihout changing the code. A good CLI is a good fit for this.