Pandora's Xbox

In 2012 Mark Nottingham wrote a blog post entitled "JSON or XML: Just Decide". It arguments that instead of trying to support both formats, web APIs should simply just use JSON. His advice seems to have been followed widely, as there are very few XML APIs left nowadays. However...

XQuery is a language that was originally developed as a DSL to query and transform XML documents. Since JSON has pushed XML aside as a data format on the web, the authors sought to support other data structures as well: JSON-like arrays and maps. This was realized in XQuery version 3.1, and a recommendation candidate was published in 2015. I know what you're thinking: too bad, too late, and against MNot's advice. This would indeed be my TL;DR, but just let me elaborate a bit more...

Together with XPath and XSLT, XQuery is a powerful tool to handle semi-structured data-"documents" based on the InfoSet datamodel. However, maps and arrays (JavaScripts' native data structures), are not part of the XML infoset. This means that maps and arrays exist in XQuery on their own, separately from other data types, as so-called "function items". When trying to serialize an XML element containing a map, XQuery will throw an error like "XQTY0105 Enclosed expression contains function item". The W3C XQuery workgroup admits that function item types were tacked on to the language as an afterthought, and has spent a lot of time and resources debating how to fit them into the syntax and datamodel. The solution that was settled upon seems more of a truce than a sound consensus. It is doubtful that after this struggle XQuery will ever see a version 3.2.

It should be mentioned that due to the stateless (or side-effect free) nature of XQuery, maps and arrays are required to be persistent (AKA immutable), which means that "function items" aren't available as native types in common host languages for XQuery, like Java, and also not in JavaScript, and are in that sense not really JSON at all. Implementations are often borrowed from popular functional programming languages like Clojure or Scala, where they have been developed by people like Rich Hickey and Phil Bagwell. Like all non-native types, performance depends heavily on the choice of algorithm, and there's no "general-purpose" immutable data type. Each implementation has its own set of characteristics, like faster appends over slower concatenation or iteration, for example. Strangely enough, there's no mention of this in the XQuery specification. Well, not so strange, since XML seems to be all about "don't explain how it works, just how to use it"...

Although not the first inconsistencies in XQuery (consider the higher-order function notation, for example), function items are a blemish upon the "immaculate" datamodel and performance concerns. But more than anything else, they open Pandora's Box, or rather, a means for XQuery to break out of the DSL and try to do stuff formerly reserved for general-purpose programming languages. Sure enough, smart developers have always found ways to work around the intentional limitations of XML tools, but still... maps and arrays have allowed me to parse, serialize and execute XQuery and XSLT within XQuery! Parsing speed is of course hampered by the implementation's performance characteristics, but this goes to demonstrate that it can be done, and that XQuery has grown out of its DSL boundaries.

So, how to continue? On one hand it's clear that XQuery will not gain more popularity by adding JSON to the stack, since most developers don't require XML at all anymore. But on the other hand, it does have some neat tricks for working with data we rarely see in other languages. So maybe there's still hope for XQuery? One of the problems of JSON is that there's no way to retrieve data from complex structures of maps and arrays conveniently. This has always been a forte of XML tools, because they all include XPath, the standard syntax for traversing the XML tree. But alas, since functional items remain alien to the XML infoset, they can't be traversed using XPath! There's some "syntactic sugar" to select something from a hierarchy of "JSON" data in XQuery, but this is not only alien to XQuery, but also very unlike anything in JavaScript, and won't become popular as such.

So what about modifying "JSON" data in XQuery? XML fragments are simply recreated, but most implementations can optimize this process somewhat behind the scenes. However, doing this with maps and arrays straightforwardly would be a performance disaster! For XML documents there's also the XUpdate extension to XQuery, but again inconsistent and not for JSON.

Ironically, modern JavaScript solutions for traversal and modification of JSON data (or HTML DOM) trees shy away from complex traditional APIs and move towards functional-style, stateless techniques, some reminiscent of XPath, some of CSS selectors, and again others of XQuery-style for-comprehensions, in the form of LINQ, a query DSL that has its origins in the C# .NET tool set. It seems that we should probably look to proprietary solutions nowadays for doing what we need to do with data.

To conclude, JSON is a latecomer in XQuery, very likely too late. Had the focus been on developing a more intuitive syntax for handling maps and arrays, the W3C XQuery Working Group could have had the answer for creating JSON in the best possible way, pouring years of knowledge on treating hierarchical data into a solid model. As it now stands, the feature is a half-hearted attempt to save the language from its impending demise.

Comments

Post a Comment

Popular posts from this blog

Abandoning hope... and XForms

Sometimes a decision must be made. I have invested many hours in a standard that is quite complex and tedious to maintain, and that was great fun, but enough is enough. Why did I invest in XForms when it took so much effort and time? Because I believe standards are good and good standards come from the W3C and the XML community. Also, I always had somewhat of an inferiority complex in IT development that I tried to compensate by using techniques supposedly invented for common folk, i.e. non-programmers. Both are bad reasons from where I now stand. I took up XForms because I was infected by the enthusiasm of a friend (who was even quicker to abandon it) about 3 years ago. At first I assumed it just used a model from which to infer a form. It turned out it was like a superset of HTML forms with options for displaying and validating data. This was largely taken up by HTML5, as we all know. The Model remains the core of XForms, but the W3C is slow to respond to innovations and t...
Read more

JS Journey into outer space - Day 5

This article was written as part of a knowledge sharing program I created for front-end developers. It builds up to creating reusable abstractions by studying async transducers. Day 5 It's time to add more operators. We already have map , but what if the mapping function returns another "container" value, like an array or observable? 🦉 In RxJS some operators like switchMap , concatMap , or mergeMap will expect a "project" function that returns an observable (or promise or array), but what happens then? First map is applied with the project function and after that the result is "flattened". In other words: when the returned (inner) observables emit, the mapped (outer) observable emits those values instead. Since RxJs streams have a specific async behaviour the flattening process takes concurrency into account: mergeMap will flatten inner observables whenever they complete, regardless of the order, but concatMap will preserve the order of the ou...
Read more

JS Journey into outer space - Day 4

This article was written as part of a knowledge sharing program I created for front-end developers. It builds up to creating reusable abstractions by studying async transducers. Day 4 Now we have modified transduce to handle async it's time to figure out how it can handle observables. Kickoff 🏈 First we need to pass the onNext function which kicks off the callback chain. In RxJs this is called subscribe instead, so let's rename the parameter and the callbacks to be more in line with RxJs : function transduce ( input , fn , subscribe , onError , onComplete , init ) { subscribe ( input , { next : ( cur ) => { init = fn ( init , cur ) ; } , error : ( err ) => { onError ( init , err ) ; } , complete : ( ) => { onComplete ( init ) ; } } ) ; return init ; } ; 🦉 Observables are considered a push-style of dealing with many values, while generators are pull-based. In...
Read more