1 Abstract¶

Rubin Observatory has adopted an “IVOA-first” approach to APIs for the Rubin Science Platform, and are therefore implementing SODA as the protocol for providing image cutouts. The preferred project implementation language is Python, and we have adopted FastAPI as our primary web framework. This document collects the implementation experience of someone familiar with Python web service development but new to the IVOA standards. It documents hurdles and roadblocks ranging from minor clarity issues to significant impediments to implementing IVOA web services using the FastAPI web framework.

See DMTN-208 for the technical architecture of our image cutout service.

2 Standards used¶

The implementation discussed here is an image cutout service written to the SODA standard and supporting both sync and async APIs. The versions of the IVOA standards consulted while writing this implementation were:

We did not implement SSOAuth (IVOA Single-Sign-On Profile: Authentication Mechanisms 1.01 (2008-01-24)) because it does not support API bearer tokens, which is the authentication mechanism we are using for the Rubin Science Platform. (Nor apparently does the current 2.0 draft.)

3 Web security concerns¶

3.1 POST and simple requests¶

UWS makes extensive use of POST with the default application/x-www-form-urlencoded content type, but doesn’t provide a mechanism for a client to get a form token that has to be included in all POST requests. This makes UWS inherently vulnerable to CSRF attacks, since cross-site POST with a content-type of application/x-www-form-urlencoded is allowed without pre-flight checks. See the rules for simple requests.

In this context, a simple request is bad; you want state-changing requests to not be simple requests so that CORS pre-flight checks are forced. If simple requests cannot be avoided (if, for example, the page must be usable as the target of a form submission), there should be a mechanism to require form tokens, or some other defense against forged cross-site requests.

If client input must be in JSON or XML with an appropriate Content-Type header, it would no longer qualify as a simple request and thus force a pre-flight check.

This is primarily a concern for services that may be authenticated via cookie. Inclusion of an Authorization header also forces pre-flight checks.

3.2 GET for state-changing operations¶

DALI requires synchronous resources be available via both GET and POST. Implementing any resource that creates state changes in the server (such as creation of a job) via GET violates the HTTP security model and thus makes the service more vulnerable to CSRF attacks. See, for example, the OWASP CSRF Prevention Cheat Sheet: “Do not use GET requests for state changing operations.”

Not all sync requests will be state-changing, depending on the nature of the service, so it’s reasonable to allow GET requests as an option, but requiring GET be supported means the only options for an implementation concerned about this risk is to not support sync requests or not follow the standard.

3.3 `http` in schema references¶

XML schema references in standards examples use http instead of https. Depending on the nature of the XML processing library, this can exacerbate security vulnerabilities by allowing an active in-path attacker to replace the remote schema document with one that may trigger XML processor vulnerabilities.

4 Implementation issues¶

4.1 Case-insensitive parameters¶

DALI states, “Parameter names are not case sensitive; a DAL service must treat upper-, lower-, and mixed-case parameter names as equal.” This is a highly unusual provision for a web service and makes implementing IVOA standards with common web application frameworks unnecessarily difficult.

For example, FastAPI’s normal query and form parameter parsing with its associated automated generation of OpenAPI schemas cannot be used because they (like every other Python web framework I’ve used) treat parameter names as case-sensitive. This means that FastAPI cannot automatically impose restrictions on allowable values for parameters and automatically generate error messages. Parameters have to be recovered from a case-canonicalized copy of the query parameters and all the normally-automatic verification of required parameters, valid parameter names, and valid parameter values has to be tediously reimplemented manually, solely because of this requirement.

Implementing and testing this case-insensitivity added noticeably to the cost to the implementation. Worse, it resulted in code that was harder to understand, debug, and maintain; made the implementation work more tedious and irritating; and made the generated OpenAPI documentation less accurate and useful.

4.2 XML¶

XML is used for all responses. Modern web architectures have largely abandoned XML in favor of JSON or (for high performance APIs) Protobufs. Generating XML is tedious and lacks first-class support in web frameworks, which significantly increases implementation costs because of the need to add an XML templating or generation layer. JSON generation and parsing is built-in and transparent in modern web frameworks and is generally the default, so returning XML also requires overriding all of the defaults. Parsing XML is similarly tedious and prone to a large number of security issues.

4.3 VOTable error structure¶

VOTable error messages as specified in DALI do not separate the required error code from the additional details in the required <INFO> tag contents. Since it is using XML, this seems like a missed opportunity. It also doesn’t provide a mechanism for separating a short error message from extended error details (such as a backtrace), even though UWS indicates this is desirable and provides its own mechanism to lift an error summary into the job list.

4.4 SODA multiple cutout results¶

SODA requires each cutout filter parameter produce a separate result file, which forbids returning a single FITS file with all cutouts included (which seems like a better data model for services that can handle it).

4.5 SODA async error reporting¶

SODA requires accepting invalid filter parameters for a given ID and indicating that they are invalid solely by having the corresponding result be a text/plain document starting with an error code. This seems needlessly opaque and requires the client intuit that some of their requests fail by noticing the MIME type of some of the responses. It also creates potential confusion with SODA requests that may legitimately return a text/plain document as a valid response, and assumes structure in text/plain (which is contrary to the definition of text/plain). None of this seems correct.

An implementation should be able to fail the job with an error if the given parameters are inconsistent. This would use the much clearer error handling behavior of marking the job as errored and including the error information in the job metadata.

4.6 UWS async API errors¶

There is no specification in SODA or UWS for error replies from the async API other than job errors. (For example, posting an invalid time to the destruction endpoint or an invalid phase to the phase endpoint, or requesting a job that doesn’t exist.) The HTTP status code is specified in some cases, but not the contents of the message or a clear statement that the contents don’t matter.

Should this return text/plain errors as specified for the sync API, either text/plain or VOTable per DALI, the implementer’s choice as long as the HTTP status code is correct, or something else?

4.7 Use of empty replies¶

The /{jobs}/{job-id}/destruction and /{jobs}/{job-id}/quote UWS routes are specified as returning an empty string if the job has no destruction time or quote, respectively. This is a poor choice of special value, since an empty body can occur by accident or error for many other reasons, such as misconfigured intermediate web servers.

Since all valid values will be ISO 8601 dates, another, less error-prone special value should be used, such as none.

4.8 Mixing query and `POST` parameters¶

UWS says that PHASE=RUN can be added to the query portion of the URL when creating a new job, indicating that the job should automatically be started. This mixes query parameters with a POST body, which is unusual and generally discouraged. Any parameters provided to a POST should be sent in the body of hte POST (and PHASE should then be reserved so that it’s not used as a job parameter).

5 Standard inconsistencies¶

5.1 SODA UWS errors¶

The UWS standard for error messages says, “It is the responsibility of the implementing service to specify the form that such an error message may take.” The SODA standard does not do this. Error documents are only specified for the sync API.

5.2 SODA sync VOTable errors¶

DALI says that errors may be either VOTables or plain text. SODA requires that errors from the sync API be plain text and doesn’t allow for VOTables, but claims that it’s following DALI.

5.3 SODA error code specification¶

SODA section 5.2 says, “Error codes are specified in DALI,” but DALI does not specify any error codes that I could see, only a VOTable representation of errors.

(Perhaps this refers to the brief discussion of HTTP error codes? If so, this is far from a full specification of possible error codes.)

6 Clarity issues¶

6.1 `jobs` XML example¶

There is no example of the jobs XML document returned by the UWS Job List API. The correct form of this document has to be reconstructed from the schema.

6.2 UWS `isPost` attribute¶

The isPost attribute of <uws:parameter> in the UWS standard is never mentioned in the text and has no <xs:documentation> element in the schema, leaving its purpose to the imagination of the reader.

6.3 DALI VOTable error example¶

There is no full example of a VOTable error reply in DALI.

6.4 Ambiguous use of “filter”¶

SODA refers to the parameters controlling the shape of a cutout as “filtering parameters” and, in some cases, as a “filter.” Filter is an overloaded term in astronomy so this terminology could create some confusion with, for example, optical filters. We used the word “stencils” instead for our implementation.

7 Formatting issues¶

7.1 `job` XML example¶

The job XML example in the UWS standard has lost all of its indentation in the HTML version of the document, making it difficult to follow. The UWS schema has the same issue, but at least includes a link to the same schema as a separate XML document, which will be indented properly by a modern web browser.

7.2 IVOA standard cross-references¶

References to other IVOA standards documents are not hyperlinks, but instead are textual academic citations whose associated URLs are only listed in the References section. This makes it tedious to jump back and forth between related documents and find the relevant section being cited in a different document, something that’s unfortunately very frequently needed to understand IVOA standards.

SQR-063: IVOA SODA implementation experience