Shared publicly  - 
 
I have few comments on XForms, from someone who has invested pretty heavily in the technology over the years -

I personally think XForms 1.0 failed. As a proponent of the specification for a number of years and a working group member, that's a bitter pill to swallow. I think some of that failure was technical, some of it was sociological.

On the technical side, it was too dependent upon the XML data model. This isn't surprising, given when it was proposed, but the reality has been that for any number of reasons XML never achieved the uptake in adoption on the client that I and many others had hoped for. That's a lesson that's not been lost on the W3C Forms WG, as one of the key areas that we are working on now with XForms 2.0 is that JSON may in fact be a valid flavor of data model.

A second technical factor had to do with the notion of working with data models in toto, rather than simply binding properties to components directly. I don't think this was a flaw, but it was a fairly radical departure away from "the way things were done", to the extent that its taken several years for programming methodologies to catch up. Today, it's no big deal to package a set of objects that are maintained as an internal model and post them up an XHP pipeline, but it took a fairly significant change in the way that Javascript itself was thought about for that to happen.

That XForms had to wait for JavaScript to mature (and increasingly for the emergence of a platform such as jQuery to create a more effective binding language on the client side) before it could really achieve anything approaching cross platform adoption. Putting together an XForms based application layer on a contemporary browser such as Chrome or Firefox in JavaScript should actually be fairly easy, given the building blocks of HTML5, but the need for backward compatibility made universal adoption problematic at best.

A final factor was the lag in adoption of XPath 2.0 on the client, and consequently the inability for developers to extend the XForms language in a consistent manner. XForms is an application layer, and as such, needs to be extensible. Some of the work we're doing with XForms 2.0 is moving in that direction, finally, but its taken nearly ten years to get there.

The sociological factors are perhaps just as important. XForms, like SVG, venture out of the foundational space and into the application space, which meant that support for either spec by principle vendors has always been problematic, as these typically competed with proprietary technologies - and more importantly as product differentiators. The demand for an XForms like solution is certainly there among (especially commercial and governmental) customers, but as these frequently represented big contracts the benefits of an open (and most importantly a vendor-neutral) UI platform remained dubious among the large players.

Moreover, XForms was intended to create an abstraction layer that could then be implemented across different platforms differently, but in my experience, the web UI experience is a bigger factor in application development than cross platform compatibility. People want their web pages to look and act the same from platform to platform, and the idea that one vendor would implement forms pages that differed in presentation from one platform to the next did not sit well with UI designers.

These two factors led to the third factor - no large vendor has specifically carried the XForms torch from the beginning as a concentrated effort. Certainly, IBM has been significant in pushing XForms, but their overall investment in XForms has been comparatively tiny compared to investments in other technologies, especially Java tools. This has meant that over time, XForms based solutions have always been marginal projects, carried out by small companies that were often little more than a couple of people in a garage, and typically without the marketing or investment muscle of a Microsoft, Oracle or even IBM.

This has been true of the other W3C tech as well - indeed, SVG would probably have remained a dead technology if Google hadn't started carrying its banner with Chrome. MathML gained support primarily because science powerhouse Mathematica was willing to promote and integrate it into their offerings.

Finally, XForms exists at the upper end of technical complexity. You have to understand rich hierarchical data structures, the concept of constraint bindings, validation and declarative programming. Name/value pair manipulation can seem so much simpler, and for the typical web form (usually encountered entering in registration information) the overhead of all of these things can be daunting at best, especially if you are not used to thinking about model driven architectures. For web applications, the former is useful, but comparatively speaking, these aren't consumer driven applications - they are b2b type apps. I've been developing XForms applications for years, and my predominant customers are clients developing internal applications, not consumer facing ones.

It's easy to point to the browser vendors and say that it's their fault that XForms never developed, but it's a disingenuous argument at best. They certainly didn't help, but in reality, XForms has not, yet, proved itself to be worth integrating. This is not to say that the need isn't there - it most assuredly is - but the question ultimately comes down to understanding what that need really is.

XForms 2.0 development is underway now, and its shaping up to be an interesting project. While a lot is still sketchy, there are a few things that I think will make a difference down the road, and there are a few factors that I think work in XForms 2.0's favour that weren't there before. Among these:

1) Less X. JSON, inbound and outbound, is seen as a core requirement for any new version of XForms. Internally, this changes things somewhat - you go to a model where attributes don't factor in, for instance. It's possible that this may also translate into a different query language (most likely Javascript, or at a minimum jQuery type selectors) being supportable in that instance, something we're currently discussing.

2) XPath2/XQuery support. It will be possible to use XPath 2.0 constructs, along with XQuery FLOWR statements, within XForms 2.0. This provides much greater support for dates, ordering and grouping, regular expression processing and so forth.

3) Shadow trees and rich widgets. One of the key problems that XForms 1.0 has is that it fails to handle cases such as rich text widgets or components where several elements interact in a control in different ways than the elements individually. One idea being explored right now is the use of shadow trees on data elements that could then be bound to complex controls, such as treeviews, rich text controls and the like, without necessarily being part of the publicly exposed data model.

4) Internal user variables. One key assumption with internal XForms content was the idea that you should be able to store computed information into the model via bindings, but binding mechanism can be complex even for those people familiar with the language. One idea that will likely make its way into the spec is internal user variables that can retain state and be referenceable from within XPath expressions.

5) Attribute Value Templates (AVTs). One of the more frustrating aspects of working with XForms is that it's not possible for output values to directly affect user interfaces. AVTs provide that bridge. You can, for instance, use an AVT to change classes in an interface:

<div contenteditable="{if (instance('foo')//state = 'live') then 'true' else 'false'}">This is a block of text.</div>

This would then make the text editable if a certain state is set, without necessarily needing to reflect this in the data model directly.
6) Functional extensibility. One goal is to provide an extensibility framework for XForms that make it possible for users to add new functions or service libraries to the EXPath (or JSON) evaluation language. Some of them may be simple as the doc() function, which will make it possible for a component binding to load in a document on the fly then access the underlying XML without necessarily going through the process of setting up a formal instance to do so. Similarly date and number formatters and ingesters can process and modify the presentation of dates and numbers within an XForms page. By creating a systematic interface for doing so, XForms can become much more robust.

7) Namespaceless elements. One common complaint with XForms is that namespaces can be cumbersome to work with. An idea being floated is the notion of an XForms implementation that can work without namespace declarations or namespace requirements in bindings. These might have slightly different names than the existing specification, perhaps such as <xfinput> to differentiate it from the HTML <input> element, but the idea is that it should be possible to work with these without having to spend time getting caught up debugging namespace logic.

8) Templates, etc. There are actually a number of changes that have made their way into the XForms 1.1 specification (and are nearly universally adopted) such as the origin attribute, the use of templates within XForms, dynamic switches and toggles and so forth that dramatically improve on the 1.0 language. However, because many people encountered XForms in its 1.0 incarnation, they aren't aware that these improvements now exist. XForms 2.0 will carry these improvements through and work to simplify other rough edges from the original spec.

9) jQuery and HTML5 compatibility. A big problem for the evolution of XForms has been the lack of a consistent and compelling platform for development, coupled with a backward compatibility issue. In reality, it may be necessary to cut the chord and make the assumption that any XForms 2.0 implementation will only work on HTML5 compatible browsers. This might also force a shift in underlying implementations to integrate with at least jQuery, as this tends to be a problem that many current implementations are struggling with, even as jQuery is becoming a near universal default for rich websites.

10) Finally, I know that we're looking at an outreach effort with some of the major vendors to help support the next version of XForms, preferably via a client-side solution but worst case via a relatively easily portable server side solution. More to come on that.

Keep in mind that none of these are set in stone, though they do represent a general lean in the direction being taken by the working group, as well as looking at what customers are demanding of the technology.

I think there is still a need for XForms, but its a need that itself needs to be tempered with the reality that what seemed like a good idea in 2002 may not necessarily be as attractive in 2012. I think that's what's happening now within the Forms working group, and as such XForms might actually end up becoming more significant in the long run than it is now.
5
3
Erik Bruchez's profile photoChimezie Ogbuji's profile photoIvan Herman's profile photoKevin Clarke (ksclarke)'s profile photo
4 comments
 
My $0.02 after using XForms extensively (with @John Clark) for patient record data collection (http://copia.posterous.com/desiderata-and-architectural-constraints-of-w)

XForms exists at the upper end of technical complexity: I'd definitely agree the learning curve is significantly cheap, but I think the potential for automation and templating is much more powerful than anything else out there in my experience

Less X. JSON, inbound and outbound: Seems to sacrifice the power of declarative programming for closer integration with the browser framework. A balance would need to be struck

XPath2/XQuery support: I disagree, I have yet to find a motivation for adopting the next generation of XML standards (which have their own steep learning curve that is the main reason why I haven't bothered moving off XPath/XSLT). But this may just be due to the requirements I was working with

Internal user variables: I definitely agree and this was a major pain point for us.

Attribute Value Templates (AVTs): Same as above

Functional extensibility: I agree and this might be, perhaps, a reason for the lack of browser adoption

Namespaceless elements: I don't find this argument compelling and it seems to be more an issue of the learning curve of XML as a deterrent to its adoption for use in the web.

jQuery and HTML5 compatibility: The backward compatibility issue is one I don't understand. You can use javascript - and thus jQuery - with XForms (even if it is counter to the style of XForms) and the need to be compatible with HTML5 seems more like a political failing than a technical one (given the nature of the community developing HTML5)

Re: the relevance of XForms in 2012,: I cannot imagine using jQuery/DHTML instead of XForms to meet the requirements we had (automation, templating, etc.) that emphasized the value of declarative programming. Perhaps our requirements were an exception.
 
I'm surprised to get a response back so quickly, and for the most part I'm actually in agreement with you, but let me comment inline:

>> My $0.02 after using XForms extensively (with @John Clark) for patient record data collection (http://copia.posterous.com/desiderata-and-architectural-constraints-of-w)

I find it significant that you've HAD experience working with XForms. I'm not disagreeing with their utility - far from it - only trying to figure out how to get it out of the XML-only ghetto that seems to be a gating factor for adoption.

>> XForms exists at the upper end of technical complexity: I'd definitely agree the learning curve is significantly cheap, but I think the potential for automation and templating is much more powerful than anything else out there in my experience

HTML5 + JavaScript by itself can certainly be used to put together complex web forms, but it has been my experience the models become too complex for managing the application once you reach more than perhaps a dozen controls on the page. For anything involving any complexity, I can put together an XForms application in a small fraction of the time for me to do the same with either JavaScript or Java, but that may reflect deep knowledge in one domain and less deep knowledge in another (though I'm inclined to disbelieve that was the overriding factor). Working on XForms apps for the US National Archives proved an interesting experience, because for the most part the initial architecture on those forms came from Java developers who were used to thinking in the HTML forms name/value mode ... and as such the XForms became very brittle.

The problem I'm highlighting here is that the domain involved is generally at the enterprise level, for use in data-centric applications. These currently represent a comparatively small portion of all websites on the web, and as such tends to command a low priority for web developers looking to build the latest and greatest in the consumer sphere. As data-centric applications become more of a factor on the web (which I believe will be the case), the demand for XForms will likely rise as well.

>> Less X. JSON, inbound and outbound: Seems to sacrifice the power of declarative programming for closer integration with the browser framework. A balance would need to be struck

What I'm seeing at this stage is JSON endpoints, possibly converted to an internal XML representation within the system, possibly not. JSON is not JavaScript - it is simply another way of representing a data model, one that in all honesty is not quite as robust, but that, again in the consumer sphere probably commands a larger overall percentage of data traffic (this is not true at the enterprise level, where JSON is quite rare).


>> XPath2/XQuery support: I disagree, I have yet to find a motivation for adopting the next generation of XML standards (which have their own steep learning curve that is the main reason why I haven't bothered moving off XPath/XSLT). But this may just be due to the requirements I was working with

It's worthwhile. XPath 1.0 suffers from the fact that the underlying data model assumed that all nodes were in fact part of a single document, and that restriction prevents a great deal of functionality. Sequences, the logical successor to XPath 1.0 nodesets, are considerably more robust on all fronts, and they work especially well when dealing with heterogeneous data sources. The curve is also not all that steep - the basic expression language doesn't change in XPath 2.0, you just have a larger API of available functions to work with, and more significantly you have the ability to define modular extensions in a consistent manner not only in the host language, but within XQuery or XSLT2. FWIW, I find XSLT 1.0 far more limiting now, though I use XSLT 2 and XQuery both far more extensively.

>> Internal user variables: I definitely agree and this was a major pain point for us.

Me as well. You CAN get by with node bindings, but they are not quite the same things, and in many cases the biggest value of variables is the ability to collapse extensive XPath expressions.

>> Namespaceless elements: I don't find this argument compelling and it seems to be more an issue of the learning curve of XML as a deterrent to its adoption for use in the web.

If you are used to working with namespaces, they serve an obvious function. However, for many people, namespaces are complex and counterintuitive, and they don't translate well into JavaScript. Since this, and not some XML-based language, seems to predominate on the web, trying to overcome that institutionalized fear of namespaces is more of a challenge than simplifying the XForms language and recognizing that at some stage you can't change that many minds fast enough to make a difference.

>> jQuery and HTML5 compatibility: The backward compatibility issue is one I don't understand. You can use javascript - and thus jQuery - with XForms (even if it is counter to the style of XForms) and the need to be compatible with HTML5 seems more like a political failing than a technical one (given the nature of the community developing HTML5)

This is a point I've thought about long and hard, and I'd stand by it, although I agree that the motivation is more political than technical. When it comes right down to it, most XForms implementation, even those that are "server side" like Orbeon, actually use JavaScript quite heavily. Unfortunately, one consequence of this is that they often rely upon home-grown code that may or may not trample on specific global names found within libraries like jQuery, YUI or similar packages. The HTML5 issues come down to the fact that at this stage there's a fairly strong bias against XForms at a political level because it is seen (perhaps legitimately) as a threat to the use of HTML5 forms based solutions by those on the WHATWG group and their backers. However, there are a great number of changes that the HTML5 WG (which is almost but not quite the same as WHATWG) that could in fact be co-opted in the development of a new solution, and in general I've found that with regards to standards it's better to be forward thinking, targeting what likely will be, rather than retroactively thinking, targeting what's known but transient. I see it as turning a potential weakness into a strength.

>> Re: the relevance of XForms in 2012: I cannot imagine using jQuery/DHTML instead of XForms to meet the requirements we had (automation, templating, etc.) that emphasized the value of declarative programming. Perhaps our requirements were an exception.

It won't by itself. HTML5+ARIA (which is a term used by the HTML5 working group and pretty much only them to talk about the JavaScript/CSS side of the solution) is reasonably solid construction materials for building an XForms framework, but is not actually all that well suited to handling the abstractions that XForms enables. I feel the same about Java - you can build incredibly applications with Java, but it is simply too cumbersome to work with application data abstractions and becomes brittle when you try to force it too far up the stack. I see this all the time in applications where architects who were heavily immersed in Java try to build highly fungible business logic into their code, then fail when business requirements change. XML/XQuery is better at that, SemWeb Triples/SPARQL are possibly even better still but require an intermediate pipelining architecture in order to work.

I'm not saying that XForms failed because it was inadequate, though the 1.0 release was problematic for more complex models. I'm saying it failed because it represented too radical a change in thinking given the community of developers, and blaming the developers, while satisfying for those of us in the XML community, has proven to be counterproductive. The central challenge moving forward for the technology is recognizing that there is still a sizeable gulf that it needs to cross in order to be accessible to the JavaScript crowd, and that if the standard is to survive and thrive that's the next frontier that XForms has to conquer.

Kurt
 
Thanks Kurt for this discussion. I think we need more of this to finally lift XForms into the position it deserves.

I largely agree with the points you've outlined though i'm a bit concerned that the use cases that XForms already serves very well might become buried a bit under the pressure towards client-side and more broad use on the internet. While it's true that XForms suffers from the fact that there are very few public applications on the public internet we have to ask why this is the case. I think to a certain extend XForms is too complicated (powerful?) to be applied to simple use cases. You'll hardly use XForms if you just want a simple contact form on your site (even if there were simple to use implementations which i think do not exist today). And probably those use cases should simply be left for HTML5. More complex applications on the other hand are most of the time not public but are used in B2B or internal applications that serve very specific business needs and problem domains. We have used XForms in applications ranging from Air Traffic control, via tunnel construction and risk management to eHealth and insurance solutions to name just a few. (For a more complete list see http://betterform.wordpress.com/usages-of-betterform/). The declarative nature of XForms and therefore the possibility to generate the markup with XSLT was a major strongpoint here. So to a certain extend asking why there are not more public XForms apps on the public internet is like asking why there not more public accessible SAP installations. That's maybe a bad comparison but you get the point.

>> On the technical side, it was too dependent upon the XML data model.
True but again it also opened the door to support many of the complex standards that already exist out there. Examples from the real world being AIXM (Aeronautical Information Exchange Model) or HL7v3. Though i personally don't like JSON too much i agree that direct support of it in XForms opens up new use cases and certainly makes integration with the more interesting JavaScript components out there much easier.

>>Putting together an XForms based application layer on a contemporary
>>browser such as Chrome or Firefox in JavaScript should actually be >>fairly easy (...)
Then why do we don't see those implementations? Actually the only active client-side implementation i know of is XSLTForms and this suffers from only partial support for the standard. From a high-level view the task of implementing XForms on the client looks quite doable but when you really approach the task you're facing some problems at least when a broad support of the features is your goal. For one there are the Schema types which are hard to support in a browser. Second and much more serious i don't know of XPath 2 implementations on the client. When starting today you'll hardly be happy with XPath 1 as an alternative. And to complete this surely incomplete list there is the mismatch in the event model in JavaScript and the XMLEvents used in XForms (this being a worthwhile topic in itself). XMLEvents are extremely powerful but also hard to master sometimes. Who really uses all this power and what are the really useful features? Maybe a major problem with Specware - it's hard to get rid of features once they are there.

>>Moreover, XForms was intended to create an abstraction layer that >>could then be implemented across different platforms differently, but >>in my experience, the web UI experience is a bigger factor in >>application development than cross platform compatibility.
Again i agree but with huge success of mobiles and tablets we again should be careful to not drop the device-independence ahead of time. As always with the arrival of new technology it takes some time to learn that old practises are not appropriate any more - the same form on a desktop browser and on a mobile simply can't look the same without seriously damaging the respective platform-specific conventions and user-expectations.

>>1) Less X. JSON, inbound and outbound, is seen as a core requirement
I hope this can be rephrased as 'more alternative data models' ;) as the 'big' applications need it and want it.

>>2) XPath2/XQuery support. It will be possible to use XPath 2.0 (...)
Nice but probably a new hurdle for client-side implementations.

>>5) Attribute Value Templates (AVTs). (...)
yes yes yes. At the same time the use of AVT could replace some of the IMO unnecessary markup variants in XForms. Think for instance of the resource attribute and the resource child elements and similar constructs in XForms. These child elements could be completely dropped in favor of AVTs.

>>7) Namespaceless elements. (..)
Would be very happy to see that. Actually it would make JavaScript implementations much easier.

9) jQuery and HTML5 compatibility.(..)
I would be a bit cautious with relying on certain libraries like jQuery though adopting certain principles is certainly not a problem. E.g. instead of just relying on XPath i would like to see the option for an author to use an alternative expression language for the binding. E.g. jQuery (and not to forget Dojo) have shown that CSS 3 matchers can be powerful enough for a lot of use cases.

>> I think there is still a need for XForms, (...)
Couldn't agree more. In the many years we're developing with and for XForms we again and again got the feedback from customers that they couldn't even imagined their solution to be build with a different technology in a reasonable time.

Let me add some points i'm missing in the current discussion:
- markup simplification. The aforementioned child elements that are mainly duplicates of the equally named attributes should be dropped and replaced by the use of AVTs. IMO these confuse more than they help. Also the diversity of options makes implementers life much harder which is ok but e.g. we've not a single use case in many years that justified the diversity of child elements for select(1) - choices with itemsets along with items, several itemsets in one select(1), nesting all these etc. Instead it would make much more sense to finally define a tree control.

- refactoring of the insert action. This has been claimed by Mark Birbeck before but i'd like to emphasize that again. Even after years of XForms usage we ourselves have to consult the Spec again and again to choose the right combination of attributes to get the desired result. Instead new actions like move, replace, append etc. should be introduced.

- mentioned that before in the above text but i think it would be nice if the author would be allowed to use an alternative expression language for the data bindings. E.g. instead of using JSON in- and outbound people might like to work with it also internally. XPath would then surely not be an appropriate binding language - you would most likely want to use a JavaScript-like syntax.
 
Good points here. Some quick comments.

- Today XForms (plus extensions) shines where you have XML. This is IMO the big plus that XForms offers. Sure XML is not trendy among geeks right now, but it is massively used and will remain so for a long time. Obviously, XForms still needs to integrate well with JSON (and JavaScript, see below). It's not clear to me whether moving XForms away from XML would do any good beyond that kind of things.

- On the next generation of XML standards: it's impossible for us at Orbeon to imagine anything less than XPath 2.0. If anything, we need XQuery 3 with updates (but Saxon's update implementation is not open source I believe).

- I fully agree on the widgets/components side. Having a solid and extensible set of controls/component is absolutely required nowadays.

- To echo the namespaces discussion, a while back, we had the idea of a completely namespace-free XForms. Some initial thoughts here: http://wiki.orbeon.com/forms/projects/xforms/xforms-without-namespaces

- I agree with Joern on the markup simplification.

- I partially agree on the actions, but there is more to that I think.

Initially (years ago), we were excited about XForms actions because such as small number of them allowed to achieve so many things, and yes "without JavaScript". But over the last 2 years we have come to realize that for complex logic, you need a real programming language with all that implies.

Even programmers working with JavaScript have a better language than XForms actions, and that environment can be made more palatable by using CoffeeScript.

Case in point, a lot of the logic of our Form Builder for example, which were previously written in XForms actions, have now been rewritten in Scala (https://github.com/orbeon/orbeon-forms/tree/master/src/scala/org/orbeon/oxf/fb). This gives us a (way) more expressive language with actual reusability (and static typing, which I like).

One thought is that XForms needs a solid JavaScript API (which would also lend itself to CoffeeScript-ization). We toyed with the idea here: https://gist.github.com/790257

A cool thing about a JavaScript API is that this works for client-side implementations, but also server-side implementations, which can run on top of Rhino for example. With this form authors have a choice to use XForms actions, JavaScript, or CoffeeScript. It would be a big plus.

(For server-side implementations, using Java or Scala is also a good option. Further, you can already compile Java to JavaScript, and there is hope that Scala to JavaScript is not too far ahead, so these languages could even work for full client-side implementations of XForms.)

XQuery (in particular XQuery with updates, which could then handle everything that the current XForms insert/delete/setvalue do and more) can also be an option on the client and on the server. The issue here is implementations: we have full JavaScript implementations on the client and the server, but XQuery implementations are just starting to appear on the client through compilation to JavaScript. And on the server I am not sure there is a Java-compatible implementation of the update facility which is not part of an XML database.
Add a comment...