Shared publicly  - 
 
Web SQL Database needs to die. The sooner IndexedDB is in the hands of developers the better.
17
2
Simon Boddy's profile photoKevin Dangoor's profile photoUmar Turk's profile photoBob Obringer's profile photo
97 comments
 
Yeah, well, it's not like IndexedDB does everything we need yet either. Still stuff left to implement in Firefox, for sure.
 
"Regardless the fact Web SQL Database has been deprecated by W3C thanks to Mozilla influence, and probably somebody else as well, SQLite is present by default everywhere, starting from your mobile phone, tablet, Operating System ... etc etc. SQLite is a de-facto standard only W3C could drop so easily."

"Mozilla talked about advantages of IndexedDB over Web SQL Database and there isn't a single example where IndexedDb is faster, smaller, easier, or better, than SQLite. I call it epic-fail due somebody with pretending technicals knowledge able to make such political decision and the proof is under everybody eyes."
 
SQLite is great, to be sure. But when defining a standard that's going to have multiple implementations, you have to fully define the standard. I gather that both Microsoft and Mozilla were not keen on a standard that says "SQLite v3.23.10" or some such (because the SQLite query language itself is not static!) Firefox certainly ships with SQLite (and our IndexedDB implementation is based on it), but I don't think Microsoft ships SQLite or wants to. That means that someone would need to specify the entire SQL subset that need to be supported by the browser.

Apparently, now leveldb is being considered for things like IndexedDB implementation as well.

Sure, there are people who will remain bitter about Web SQL DB being deprecated in favor of IndexedDB. But, that ship has sailed... and the sooner we get to the future, the better.

Both Firefox and Chrome ship with IndexedDB. IE has a "prototype implementation".
 
SQLite's development I think does make it different than just an implementation. Realistically is a specification more stable than SQLite? And SQLite, as an implementation with an impressive test suite, is certainly a more portable reliable and consistent option than a specification.

I'm not saying SQLite is necessarily the right option, but I think it's unfair to treat concrete source as inferior to comparatively vague natural language specifications.
 
From a web developer's perspective rather from than that of a politician, IndexedDB is a huge step backward for the web. Realistically, how many backward compatible changes do you think SQLite makes between versions? Probably almost none. SQL is a proven technology and is well understood by developers everywhere, making it much easier to learn and deal with than a completely new API like IndexedDB. It is also a whole lot more powerful. Complex queries are a lot easier to do: joins, transformations, sub queries, etc... IndexedDB is basically a glorified key-value store.

I actually think it would have been far better for the W3C to specify the SQL subset for browsers to support than to just go invent an entirely new technology. All they would have to do is basically copy and paste the SQLite documentation! That way, SQLite wouldn't necessarily have to be the implementation behind the scenes if a browser like IE didn't want to use that. As long as they supported the SQL subset that was specified, everything would be OK.

It sounds like someone at the W3C was just feeling lazy and didn't want to go to this effort to do what really should have been done, and so just came up with something else instead. Either that or they are one of those SQL haters of the NoSQL movement.

Anyway, those are my thoughts on the situation. Probably not going to change now, and we'll be stuck with their bad decision making for a long time to come, with some browsers supporting WebSQL and others supporting IndexedDB. Great, twice as much code for me to write. Or, put another way, time to write another JavaScript library. This shouldn't be necessary. But so it goes. Politics prevails over end user (developer) happiness and productivity. :-(
 
If Web SQL were ratified as standard Microsoft would probably implement it with it's SQL Server Compact or some such. So webdevelopers would have at least two SQL dialects to design for.
 
Web SQL wasn't ever specified. It's whatever happens to be in SQLite. Also we ship SQLite, we use it in our products every day. It does change, often in pretty subtle ways. So it might be easy now, but you're just kicking that can down the road.
 
SQLite is impressively complete. It implements SQL constructs (such as an aggregation in a subquery) that even recent MSSQL doesn't.
 
By the way, the W3C spec never says that browsers must use SQLlite just that: "User agents must implement the SQL dialect supported by Sqlite 3.6.19". Then, just because all browsers were going to use SQLlite as a backend to implement the SQlite dialect (SURPRISE!), the W3C said: "The specification reached an impasse: all interested implementors have used the same SQL backend (Sqlite), but we need multiple independent implementations to proceed along a standardisation path." PSH!
 
To be honest, the thing I care about is that the browsers start converging. A usable standard is better than a perfect one. WebSQL is dead. I'd like to either see it magically come back from the dead, or for broad support for IndexedDB to arrive. There are other people who are more invested than I in one or the other of the choices.
 
I am sorry, but with that kind of data structure are we working in the frontend, since about 2005? That is right, a simple clean and fast Key-Value structure. It is called JSON FFS! SQL was never the right tool for the Web-Development Job. It was the only tool that was there so everybody used it and nobody questioned it. SQL-DBs might be good for some things, but certainly NOT for websites. I dont know to much about IndexedDB, so i can not say IndexedDB is the right way, but it is without question that any kind of SQL is the wrong way. IMHO.
 
I replicate tables from the production database into sqlite on browser based apps on smart phones. Replicating tables is easy and I get to re-use queries from the server code since some of the app lives on the phone, some on the server, and I have to support offline operation with lazy sync.  Transforming data from one storage paradigm to another sounds like a wasteful PITA.

Our data is very complex and needs sql things like actual joins.  Indexed storage is not a substitute.  Even if the browser drops support for sqlite, I'll end up just compiling it into the app and accessing it via phonegap.  

Dropping websql is stupid.  It is much more useful than the proposed alternatives.  At least if you've bothered to learn sql.  Which is almost no developers I've ever worked with.  So it looks to me like WebSQL is a victim of developer laziness and slacktitude.  Very sad.
 
Question 1: Is this an either/or issue? Sometimes Web SQL DB is a better tool for the job, sometimes IndexedDB makes more sense. Why can't we have both?

Question 2: I understand that Web SQL DB was moved out of the HTML5 spec into it's own spec, but I don't see that it was deprecated anywhere. Am I missing something? 

Cheers!
j
 
+Jonathan Stark good questions...

Alas, it is an either/or issue. Or, rather, it's a Web SQL DB must die issue :)

Essentially, Web SQL DB needs to die because it is not standardizable, or at least not in a way in which the browser vendors want to implement. See my comment above starting with "SQLite is great, to be sure."

Finally, regarding deprecation, I think you are missing something... Here's the spec (top link on google for "web sql database"):

http://www.w3.org/TR/webdatabase/

scroll down and look at the giant banner under "status of this document": Beware. This specification is no longer in active maintenance and the Web Applications Working Group does not intend to maintain it further.

Thankfully, support on desktop is growing:

http://caniuse.com/#search=indexeddb

Sadly, support on mobile (one place where we really need IndexedDB) is not growing fast enough.
 
I realize that the Web SQL DB spec is "longer in active maintenance" but that is not the same thing as saying it has been "deprecated" which would mean that browser vendors have announced that they are going to remove support for it. Have any browser vendors indicated that they are going to deprecate Web SQL DB?
 
What it also says is:

"This document was on the W3C Recommendation track but specification work has stopped. The specification reached an impasse: all interested implementors have used the same SQL backend (Sqlite), but we need multiple independent implementations to proceed along a standardisation path."

So it appears the main problem is everybody used exactly the same code and thus the code itself is the spec. This is actually GREAT because there will be excellent consistency across platforms.

Furthermore, I do not care a whit about client side storage on desktops. I write mobile apps that have to work offline. No implementation on the phones means no value. I don't care. I don't much care what mozilla does because the phone implementors have chosen WebKit with SQLite. Mozilla is nowhere in this space so I guess they can do what they want - but I have lots of code that replicate tables from server to client and the easy way to make this work well is to stay in the SQL paradigm.

So to summarize - IndexDB is a) missing where I need it and b) doesn't solve my problems. Mozilla needs to just man up and add SQLite support (which I think they've done now).
 
FWIW, you can use the IndexedDB API today on mobile webkit using a polyfill.

https://hacks.mozilla.org/2012/07/using-indexeddb-api-today-the-indexeddb-polyfills/

You can continue to use a deprecated API if you wish, and that's honestly not an unreasonable thing to do today. IndexedDB will only grow in support over time and eventually take over.

Finally, I think the mobile landscape is not yet fully settled, especially outside the US, as featurephones get replaced by smartphones all over the world. Every app developer will have to decide what is important to support.
 
For the last time, it isn't a deprecated API and furthermore the api is inferior to SQL and by inferior I mean less flexible and powerful. Using SQL I can conjure any set of objects I like using arbitrary joins with very little code and I use this extensively in the product catalog for a sales app I have in which what I can offer for sale depends on a large number of environmental factors. Doing this in IndexedDB would about triple the size of the app.

I've been through the object db years, the ORM years (I've written a few) and in the end, I've dumped it all for plain old SQL. It works best if you have the discipline to actually learn it.

All browsers currently have SQLite. Browsers that drop SQLite will be listed as "unsupported" in our domains and links to browsers that do support SQLite will be shown, same as we did to IE 6. If mozilla wants to cede the market to the other vendors, go ahead. There are plenty of other browsers we can direct our users to.
 
What about this page from someone from Google developer relations?

http://www.html5rocks.com/en/tutorials/webdatabase/websql-indexeddb/

"Since November 18, 2010, the W3C announced that Web SQL database is a deprecated specification. This is a recommendation for web developers to no longer use the technology as effectively, the spec will receive no new updates and browser vendors aren't encouraged to support this technology."

Internet Explorer does not and will not have SQLite. Firefox does use SQLite internally, but does not expose it.

I am not personally involved in the standards work that Mozilla does... but this was decided more than 18 months ago.

I think having a widespread, speedy, easy-to-use client side storage standard is important. To me, it's more important to have a standard than which standard is adopted. The W3C decided 18 months ago to drop support, so I'd like to see the web "get on with it". IndexedDB is what the standards setters (browser vendors) have agreed upon. The sooner we can use it everywhere, the better.

I would imagine that one of two things will likely happen in the future:

1. Microsoft and Mozilla have non-trivial mobile marketshare and browsers all support IndexedDB

Browser support for IndexedDB is coming. Firefox for Android has more than 10 million downloads. Windows 8 is coming and might make more inroads on mobile than Microsoft has thus far. Firefox OS is coming.

2. Support for Web SQL Database pulls it back from the dead

Mozilla is a non-profit that operates for the benefit of the web. The people at Mozilla who studied the issues feel that IndexedDB is the better solution for the web. The spec[1] was authored by people from Mozilla, Microsoft and Google. But, the web is a living and thriving ecosystem and we will ultimately do what's best for the users.

--

The only thing that is sure is that browser support for IndexedDB is growing. Everything else is just a guess.

[1]: http://www.w3.org/TR/IndexedDB/
 
I think this guy is all wet. FWIW, I used to be something of an expert on OODBs as well as ORMs. I've come to prefer the relational database model after 20-some years of jacking around with the half baked alternatives.

His list of "disadvantages" for WebSQL include:

* The spec is deprecated.
* Overhead of SQL language you need to master and transform your JavaScript objects into relational schema
* Not object driven

1) Wrong, the spec is not maintained because the spec is the code and the code is the same and if it is all the same code having a spec is pointless. So the spec itself might be dead, but the feature of WebSQL is not deprecated.

2) I have no sympathy for lazy developers who can't come to grips with SQL and write crazy amounts of code trying to avoid learning it. I prefer working with well rounded competent people.

3) This is a feature as it provides more flexibility.

I have no problem if people want to use indexdb. I'm all for choice. But it isn't as good a fit for what I do and I will not be using it.

To me, it seems the height of stupidity that we end up with a zero incompatibility state by virtue of everybody using exactly the same code library, and then because it works so well and was so easy we abandon it for something that will generate more activity, potentially more problems with minor variances in behavior, and whose programming model is generally less understood than good old tables and queries. But I suppose this is the industry model. Once a working solution is arrived at, the industry abandons it for the next shiny thing that doesn't quite work yet.

Consider I was using NextStep PLists for IPC and web requests in 1998. Simple, parsers were easy to write. Then the XML madness which was both a total waste of time and bandwidth. Now JSON is the cool thing. The only difference between JSON and PList is the delimiters.

Now we are going to do it with data stores? If browsers drop support for SQLite I'll just be putting it back via PhoneGap for my
 
Oh and this restriction - I can only create tables in the onsuccess callback from open? Unacceptable restriction. I create temporary tables all the time in SQL. What genius decided that was a good idea?
 
Web SQL DB and Indexed DB are both good tools that are useful in different situations. In a perfect world, browser vendors would support both. If anyone on this thread thinks that NoSQL will "kill" SQL - in the browser or elsewhere - you should watch this:

http://www.xtranormal.com/watch/6995033/mongo-db-is-web-scale

Since Mozilla hasn't implemented Web SQL DB yet, they are in a great position to do so with something besides SQLite. This would revive the spec and bring FF in line with the rest of the big name browsers. 

Regarding the suggestion to use an Indexed DB polyfill: Are you suggesting that I add 23k of javascript to my mobile web app just to use a API syntax that probably won't be implemented natively in the default mobile browsers during the lifecycle of my current codebase? I don't see the rationale in that but I'm open to discussion. 

When IndexedDB gains some traction in mobile browsers, I'll start using it. In the meantime, I'll advocate for Web SQL DB because it's implemented in all the default mobile browsers and it hasn't been deprecated by any browser vendors. 

Just my $0.02 :)
 
+Jonathan Stark good point about the polyfill being a bit heavy of a solution for mobile.

Mozilla already uses SQLite, and we certainly don't have the resources to implement WebSQL DB using some other solution. If it came down to someone wanting to make a second implementation that was not SQLite (thus forcing the WebSQL DB spec to be fully specified), I think that burden would fall on Microsoft. I doubt that would happen.

+Todd Blanchard if you have specific suggestions on making IndexedDB better, now is a good time to air them (and in places other than this thread! I have nothing to do with this standard-setting). The IndexedDB standard is still in "Working Draft" state, but the implementations are coming along.
 
I've read that some of the people involved with the IndexedDB spec actually intend it to be a relatively low-level API with libraries adding higher-level sugar. Here's an example of a library that makes it a bit easier to work with:

http://brian.io/lawnchair/

(In addition to making it easier to work with, Lawnchair also supports multiple kinds of persistence)
 
There is no making IndexedDB easier... Who in their right minds thought of making the standard version an asynchronous API? It does not appear to be easy to switch arbitrarily between asynchronous and synchronous calls on a per need basis... which means you pretty much have to invest in one or the other, and the poor documentation out there seems to indicate everyone using the Asynchronous. MDN states sync code not supported by other browsers and only intended for Web Workers... doh!

This makes for terrible looking code. Look at the GWT wrapper for IndexedDB (the part that says transactional task) http://code.google.com/p/indexeddb-gwt/wiki/HowTo

The equivalent code on Google Bigtable is 1 line: database.put(object)

Okay, so GWT is out the question, let's just use plain native javascript.

Lawnchair, that you post has ZERO querying functionality whatsoever!!! And just like the GWT effort found out, it's not possible to wrap asynchronous API in a way that is useable.

 Trying to write Unit Tests on Asynchronous code is an absolute nightmare!!! Oh, I forget, most web developers don't write unit tests. Don't think IndexedDB advocates thought about enterprise developers. Well, in case one uses Buster&Sinon or whatever javascript framework you choose, it is still sucks ass... forcing you to structure your code deep deep deep in onsuccess handlers. And no, 'promises' don't actually make the testing of this any better.

Most code out there demonstrating the use of IndexedDB is broken. (Thanx Microsoft for actually having a decent working demo). IndexedDB is a failure of an API. I predict that even should the advocates insist on it, that adoption rates will be so poor as to make that part of the spec a failure. Somebody should be spidering the web for HTML5 usage by feature... I guarantee that IndexedDB will be at the bottom of the pile. I'd be very happy for WebSQL to come back (spec or no spec), but one cannot risk developing for a product only to not have it supported by the major browser vendors. I vote that Google should bring GoogleGears back from the dead, so that WebSQL can have a supported implementation on all browsers.
 
For those who think IndexedDB API is not good enough, should check out my wrapper library http://dev.yathit.com/ydn-db/getting-started.html.

It is super easy by creating schema on the fly:

db = new ydn.db.Storage('test-store');
db.put('store1', {test: 'Hello World!'}, 123);
db.get('store1', 123).done(function(record) {
  console.log(record);
}); 


Basic, but very effective query. Here querying child table 'article' from parent table of key 'bolinfest@gmail.com'

blog_db.keys('article', ydn.db.KeyRange.starts(['bolinfest@gmail.com'])).done(function(keys){
  for (var i = 0; i < keys.length; i++) {
    console.log(keys[i][1]);
  }
});

Query for sorting on 'title' with filtering in 'license' for 'article' store:

iterator_primary = new ydn.db.Iterator('article', 'title');
iterator_license = new ydn.db.Iterator('article', 'license', ydn.db.KeyRange.only(license));
var streamer = new ydn.db.Streamer('article', 'title'); // output
streamer.setSink(function(key, value) {
  // got sorted filtered result here
});
var join = new ydn.db.algo.ZigzagMerge(streamer);
db.scan([iterator_primary, iterator_license], join);

Query is quite verbose, but faster than WebSQL and far more responsive. Here we are getting response for each request (or row result in SQL). 

Alternative for more simple query will be using compound index, keyPath: ['license', 'title']

db.list('article', 'license-title', ydn.db.KeyRange.starts(['license']));
 
"IndexedDB is basically a glorified key-value store."

Precisely.

I can query ranges on singular indexes, but not anything robust with composite indexes. It's tremendously frustrating.
 
Just my 5 cents. I feel like this decision is rather political one. Just think about it, what  happens when sqlite is shipped as part of specification.
The power would be in hand of "Richard Hipp" creator of sqlite. Browser vendors would have less power. Providing own(N.I.H.)  implementation is safer. Where is web democracy nowadays?

Anyway, thank you god for amazin library like this http://dev.yathit.com/ydn-db/getting-started.html. I use it as websql/indexeddb adapter.
 
Marcin, thanks for the pointer to the ydn library.  

Nobody is demanding SQLite. We want WebSQL and by WebSQL I was thinking that the processing model and api (excluding SQL dialect) was specified - kind of at the level of ODBC.  The calls are all the same - you just send slightly different SQL.  This would have been tolerable.  Losing SQL entirely hurts.  

I'm rather suspicious as to whether the ydn library can handle queries of the complexity we routinely process using multiple cross table joins and in some cases left outer joins (as not all associations are populated).

My current strategy is that the database either lives on the local device, or it lives at the end of a URL on the main server.  This provides superior performance for WebSQL supporters and penalizes performance for non-WebSQL supporters (and eliminates off-line mode for them).  This is the right solution for now.
 
Not to intrude, but since this conversation shows up pretty high on the Google search for "websql firefox," I thought it might be useful to include a link to this project: https://github.com/yradtsevich/pure-js-websql

It is a pure JS implementation of WebSQL that works with Firefox. If one was developing a webapp using WebSQL, one could check for native WebSQL support and use this as a fallback.

Obviously, the pure JS solution will be (much?) slower than a native implementation, and it might be more beneficial to write two storage backends (one for IndexedDB and one for WebSQL) with an abstraction layer, but this'll do in a pinch.
 
+Ryan Marcus although a js implementation is cool, the number one problem with this library is that it is implemented using localstorage which has a limit that is quite small, on top of the slow issuse, it really won't be a viable alternative to WebSQL or IndexedDB
 
When I saw Mozilla's arguments to drop WebSQL in favor of IndexedDb I just thought: "Well, they only think about javascript in there and not in typical databases...".
I would rater use localStorage instead of IndexedDb since I can store data there in form of JS Objects/Arrays and easily access them! Only WebSQL supports Foreign Keys for example.
I sincerely disagree with Mozilla on that... Maybe supporting both systems at least?  
 
What is insult to injury is Firefox uses SQLite to implement indexDB. It is ridiculous, we get 5% of the functionality and can't access the features behind the curtain. 

Lets stop using obfuscating words like "object database" it is a just a lame flat file data base... the kind we had before we had relational databases 40 years ago. 
 
Taking WebSQL/SQLite out is just a terrible mistake. This decision was obviously made from the POV that "a website need not have all that data storage power anyway" but they have forgotten that HTML5 does not only power websites now, but it ALSO powers offline and mobile applications.

In so doing, they have insulted the very initiative we as a tech community have been vying for, and that is PUSHING WEB TECHNOLOGIES FURTHER and not backwards to the pre-2k stone age. It would be nice if MS joins us, but then again, they have grown quite adept at stifling Web innovation as evident by IE's track record. This is nothing new... 

For actual client-side data processing beyond the scope of a mere "twitter app", the power and maturity of SQL cannot be rivaled.
 
One more reason why firefox is dead. Can't believe they blew this one so comprehensively by having a bunch of tech-heads making the decision to use couchdb and by getting too cosy with Microsoft. It's beyond comprehension. One more reason to recommend Chrome on my website.
 
It's been basically a year since my last comment. What's changed? Nothing! In 10 years I've never seen an API garner so much hate.  Currently switched to building apps for android... and what a pleasure to use robust object store solutions. WebSQL should be brought back regardless, but in the spirit of choice for a more javascript centric api, browsers should adopt something like mongodb... then let the best api win! Even better yet would be to promote FileWriter apis to a sandbox (with some restrictions) as part of the web File APIs... then let ALL the databases of the world implement web solutions. Don't pick winners, let them emerge!

In any case IndexedDB is going to die. FACT.
 
What is the current update on this. Is there any plan to support this in future?
 
As usual, an empty article with trivial examples and no relationships.  I don't see the advantage over localStorage with JSON stores.

This is also what will keep FireFox OS from being taken seriously.  WebKit has SQLite, FireFox OS doesn't.  We will steer our customers to Android/iOS and away from FireFox at every opportunity and we will not develop for a platform lacking a proper relational database..
 
There are advantages over localStorage with JSON stores: IndexedDB has indexes, for example. Additionally, it is not a synchronous API so it doesn't have potential performance issues. (I will grant that, in practice, localStorage works well enough for a variety of uses. No reason to not use localStorage in those instances.)

Firefox OS shipping with only IndexedDB and not some form of exposed SQLite is not going to be the deciding factor of whether it's taken seriously. If it has users, then developers will make sure they support it. Firefox + Firefox OS + IE + Windows devices collectively are a large enough set of users running viable web browsers that don't have SQLite exposed that people writing apps for the web will keep them in mind. Plus, all modern browsers have IndexedDB. If you build an app with IndexedDB, it will work everywhere.

You can certainly choose to not develop for "a platform lacking a proper relational database". People choose to build apps that target a subset of the possible users all the time. shrug everyone has their requirements for the platforms they target.
 
That's a disingenuous comment. The point is not that developers target a subset of platforms. The point is that platforms provide APIs to developers to maximize ease of deployment. If a platform messes up their API, then if they want to stay relevant then they should provide an alternative. The alternative on the web unfortunately is simply not to provide any form local storage and use the server for all ajax storage, to the detriment of user experience.  

Here is a list of problems cross browser.. http://goo.gl/XamCy7 ... although the author feels this can be solved by libraries, I think not. I personally have run into problems on chrome, that were difficult to reproduce... and unlike refreshing a webpage, leaving data on a client's machine that can potentially lock user experience on upgrade is simply not a risk worth taking for enterprise development.

IndexedDb will never be adopted by corporates for this reason. It will only make appearances in introduction blog posts and science experiments.
 
That blog post, despite talking about current cross-browser issues with IndexedDB was actually fairly bullish on the technology. jQuery is a great example of how the low-level nature of the built-in facilities of the browser can be improved by libraries. Libraries can certainly help.

It does seem reasonable that the browsers should offer a simple async key/value store without the issues of localStorage.

Really, my point going all the way back to the beginning of this post is simple: we developers need reliable, cross-browser, performant mechanisms for storing data locally in the browser. The browser vendors have collectively decided to get behind IDB. Odds are that fixing the issues with IDB will be easier than resurrecting Web SQL DB and fixing those issues.

So, the sooner this stuff solidifies, the better.
 
I disagree it "seems reasonable that browsers should offer a simple async key value store". Every modern database asynchronous api is wrapped by a locking synchronous api. Whether you use BigTable, db40, a hibernate wrapped relational solution, or fast modern maps like java's mapdb... they all provide synchronous apis. Many expose the asynchronous apis also, but maybe 1% of developers actually use this. The only reason would be to optimise incredibly heavy loads at incredible scale of millions of transactions per second. Yet IndexedDb forces the web developer to make this complex optimization, at great cost of application code complexity, for absolutely no gain in user experience.
 
+Daniel Gerson Now you're arguing for something that does not fit in the browser programming model. JavaScript is not Java. It is single threaded(*). Using generators, you can make async code look more synchronous, but you're not fundamentally going to change the nature of the beast.

() JavaScript *could have threads and locks, but that's not the way the browser works and I'm 99.99% certain you won't see browsers heading in that direction. Just look at how they added multiple threads to the browser content environment (Web Workers).
 
I have read most of the comments.
As dead as webSql is, indexedDB is not support by the most popular device from Apple.
So if I am targetting mobile device give me a reason to use indexedDB?
http://caniuse.com/indexeddb
 
+Kevin Dangoor Generators are a trick to allow functions to yield. It isn't useable on top of the IndexedDb's asynchronousity.

 I am not arguing for javascript to accommodate threads and locks as first order citizens. However, there are many locks underneath the hood of the javascript implementation. When LocalStorage writes data to disk, that data is lazily written to disk by the IO system.  

http://msdn.microsoft.com/en-us/library/windows/desktop/ms686358(v=vs.85).aspx

However, no-one feels the need to specifically expose that async behaviour for the coder to handle in the case of LocalStorage and probably many other javascript calls.

That is the issue under debate. The wisdom of forcing this behaviour onto the web coder. I am citing the java world stats (or could do any other programming model, because it's similar), to show that ~99% of the time asynchronous database behaviour is avoided. The advocates who pushed an async database api into the web standards, lacked this common sense.
 
+Frederick Tshimanga That's a good point. Apple may be taking a wait-and-see approach there (wait to see if anyone actually uses IDB). That's not great for the web, in the meantime.

+Daniel Gerson As far as the UI thread is concerned, localStorage reads cannot be synchronous because JS may need the value that's being read. Writes could appear synchronous to JS without blocking the UI thread.

The thing is that asynchronous APIs are the norm for JS because it has a single threaded model... look at XHR(*). Or Web SQL DB. Or Web Workers. Or Node.

() There is synchronous XHR, but everyone knows that's a bad idea because it blocks the UI *on a network call.
 
Its not about the web.  Its about applications built using web technology - specifically the approach being advocated by FireFox OS.  I don't really care if SQLite (or some other relational db) is available in a web browser pulling pages from server.

I care VERY VERY MUCH that the embedded browser components support it so I can build offline capable applications.  And key value stores aren't adequate for anything beyond trivial bits of storage.  Business systems require more power.  So long as SQLite remains available on other platforms - FireFox OS is going to remain a "toy" system. 

On more point...on Android and iOS, if the browser component is somehow missing a key feature, I can drop into native code.   Not really seeing that in FireFox OS.  Its a toy - unsuited for serious applications.  When you're number - (4? 5? 7?) in the market, you have to try harder and offer more, not less.
 
+Kevin Dangoor Thanx for your thoughtful response. I agree that I've been cussing too much about the pain of asynchronousity with little regard for the real concern of blocking the UI thread. However a couple of points...

1) It is notable that I've seen very few if any killer apps based on an asynchronous storage api. One should reflect on why this is...

2) The power of sql minimizes the pain of an async api. If I want to join the results out of 3 tables before the business logic begins, I can do this in one sql statement. To do the equivalent with IndexedDB while keeping the data normalized (maintainable), would require 3 separate async calls (async dependency hell), and adding the promises pattern just adds to the complexity of debugging.

3) Synchronous is only bad if the blocking time is potentially long and non-deterministic. There are alternative roads here not explored. For example, if the browsers implemented a version of BigTable, the DB technology enforces that all queries are range scan-able. This would enable true use of a generator/yield model to get at one's data, and also have syncable writes.... greatly simplifying the usage of dbs, and allowing the application developer to actually write meaningful code... without worrying about long blocks of the UI thread (if there were caps on the size of the individual object size stored).

Cheers.
 
+Todd Blanchard In the case of Mozilla and Firefox OS, it is about the web and open standards. Mozilla is a non-profit with the open standards as a core part of the mission. SQLite is not a standard, it's an implementation. I understand what you're saying, and time will tell how client side storage ends up evolving... I know that folks at Mozilla care about the platform being great, but they also care about it being a true standard with multiple implementations.
 
+Daniel Gerson it depends on your definition of "killer app". There's a lot of Node code out there, in production... And it all uses async APIs.

People have been building async REST-based client side apps for some time.

And you can be sure that people will build convenience libs on IDB.

Someone could build an SQL layer with IDB as a storage engine (if they haven't already!)
 
+Kevin Dangoor I have no issue with async apis for network use, e.g REST based clients etc. There are many examples but citing them does not address my point, which is of an html5 offline killer app heavily using an asynchronous db technology. Async apis for network use was never the issue.

I.e. An example of Offline Gmail/Facebook/Evernote/Game with lots of dynamic data etc using a web storage technology only.  When these kind of products are written (and they exist), they always come with proprietary plugin tech to enable them to save their dynamic data.
 
+Daniel Gerson The problem is that there isn't a good cross-browser technology for that today. Here's a library that tries to at least provide a key/value store that you can store a big chunk of data in cross-browser:

https://github.com/tantaman/LargeLocalStorage

At no point have I said that where we're at today is where the web needs to be.

My point about Node is that there are many developers building apps that access databases via async APIs today in Node. They've come up with a bunch of techniques to make it so that you don't have a giant nested ball of callbacks, and generators will make it a bit cleaner. Reasonable async APIs for databases are out there and in use today in Node.

IndexedDB's initial evolution predates a lot of what's gone on in Node. It predates generators getting added to JavaScript (and promises, for that matter).

The two sentences that started this whole thread are basically just that I want to see the web have a viable API for this, as soon as it can. Unless someone on this thread wants to try to resurrect Web SQL DB and fix the issues with it, Web SQL DB as deployed today is not the answer... If IndexedDB is not the answer either, then we'd better get crackin' on building something that will do the job!
 
"resurrect" WebSQL? unlike IndexDB it's not dead, it works fine for many of us who use it daily on multiple platforms. It really seems dumb that Mozilla would make a decision to alienate a significant subset of its developers, who now have a practical reason to hope that Firefox tanks sooner rather than later
 
"resurrect Web SQL DB and fix the issues with it"

Issues?  I'm unaware of any issues with it.  It works.  I've built half a dozen apps on it. Its on all the phone OS's that are likely to dominate for the next decade.  As long as nobody mistakes the W3C's statement that the "standard" isn't worth working on as "the feature should be killed" (which Mozilla seems to have done) and removes the feature, I'm fine.  If Mozilla wants to marginalize their appeal and remain a niche player, I don't care.  

Slavish adherence to "standards" doesn't attract developers.  Useful capabilities do.
 
If useful capabilities are all you want, you might as well build native apps. One significant difference between the web and native is that the web is built upon standards and the only reasonable way to have a standard is to have more than one implementation (or, at the very least, something that could reasonably have more than one implementation). Otherwise, what you have is not a standard...

Back in the IE6 days, that's what the web was like. There was one implementation that mattered. Today, there's Trident, Blink, Gecko and WebKit and they all represent a significant number of users and the web moves forward much faster than it did back then.
 
+Kevin Dangoor , I think it's naive to believe that "the web is built upon standards". Technologies mature into standards, they are not built apon them. That's true of the history of the web, OpenGL, SQL, etc, etc.

When standards boards are laggards, that technology withers on the vine. It is absolutely incumbent on providers to evolve their own products balancing adherence with innovation to maximize utility of experience. 

Chrome supports NativeClient apps. This is an innovation of browser technologies, which they are still iterating on. In their own words it's too early to try and standardize it... but it's available across Chrome and ChromeOS. Good!

Tech innovation is necessarily messy, here you can see the evolution of OpenGL/D3D specifically via the features that NVIDIA and others were iterating on.  I.e. features first, standards second. In fact in this vein it's a pity that 3dLabs (on the board) pushed something as standard that wasn't ready, and the entire community suffers as a consequence.

http://programmers.stackexchange.com/questions/60544/why-do-game-developers-prefer-windows

To the extent that browsers support supersets of HTML to maximize utility, that's a good thing. It should be encouraged. Don't confuse the cart and the horse.
 
+Daniel Gerson Totally agree that the technology comes first and the standards follow.

I would personally be fine if WebSQL DB "won". I don't care what technology wins as long as it's a fast-enough, ideally-transactional technology upon which we can reliably build apps for all browser that hold a "reasonable" amount of data.

I get it: those of you arguing here like WebSQL DB and don't like IndexedDB. Fine.

For me, it's the current state that is the problem. IndexedDB is supported by 81% of the browsers[1]. WebSQL DB is supported by ~30% of the browsers. The problem is that the 19% that doesn't support IndexedDB includes 11.2% for Safari, which is an important browser.

The only technology that reaches across all of the browsers today is localStorage.

[1]: http://arstechnica.com/information-technology/2013/10/internet-explorer-6-usage-drops-below-5-percent-in-september/
 
i think the entire thing is in a bit of a mess, for offline storage on ios you are stuck with local storage cache or websql, however you can have a max of 50mb which isn't exactly a lot! IndexedDB, doesn't work on ios and as a web developer isn't as easy to move to as websql would be. (mysql to websql is almost identical)

SQLite would be ideal still as i'd love to be able to generate the sql file to download before hand rather than have to build the db on the fly.

Something needs to be done to sort this mess out - and a proper standard that everyone will adopt needs to come about, don't really care if its indexeddb, websql or something else it just needs to be done.
 
+Kevin Dangoor

This is insane. I mean, I don't even...

"we developers need reliable, cross-browser, performant mechanisms for storing data locally in the browser."

...You mean like WebSQL?

"The browser vendors have collectively decided to get behind IDB."

...The vendors have collectively decided a LOT of things, and computer usability has suffered for it. Customers are clamoring for 'simple' solutions and vendors are trying to reduce the need for 'complicated' tech support, and that's leading us into a crisis in software usability/reliability.

"Odds are that fixing the issues with IDB will be easier than resurrecting Web SQL DB and fixing those issues."

...WebSQL isn't dead, and has been stated here IndexedDB on Mozilla is implemented on SQLite, so those 'issues' could very well be shared with IndexedDB.

"Apple may be taking a wait-and-see approach there"

Apple has always been a 'best of breed' technology company. None of their market leading merchandise was created based on a 'wait and see' approach.

"SQLite is not a standard, it's an implementation.
*
the only reasonable way to have a standard is to have more than one implementation"

...Despite the minor differences in dialect SQL remains a stable language and effective use model. And while 10 SQL programmers working individually will create remarkably similar (if not identical) models for the same problem, that is not the case for any other 'DB' technology I have ever encountered.

The consistency you get with SQL is a product of the relational model itself. On the other hand, the consistency you get with IndexedDB is a product of the IMPLEMENTATION, which serves as the schema/standard for your data model. So much for reasonable.

Finally, unit tests serve as a more effective description of a standard than a standard, er, standards description, and I wish this line of thought would be pursued further.

"Unless someone on this thread wants to try to resurrect Web SQL DB and fix the issues with it"

...Resurrect. You keep using that word. I do not think it means what you think it means.

"I don't care what technology wins as long as it's a fast-enough, ideally-transactional technology upon which we can reliably build apps for all browser that hold a "reasonable" amount of data."

...That's EXTREMELY vague, and kinda undermines your credibility when it comes to arguing about standards. But lets see, fast-'enough', 'ideally'-transactional, 'reasonable' amount of data? Looks like WebSQL fits the bill!

As for 'reliably' building apps, when you have a cumbersome and complicated standard that lacks any intrinsic consistency, you might be able to write apps for all browsers IN THEORY, but in practice those apps will be harder to understand/debug, and reliability/security in general will suffer.

"those of you arguing here like WebSQL DB and don't like IndexedDB. Fine."

...If that's your conclusion, then you haven't read a damn thing that's been said here, and/or have nothing to support your argument. I don't know what you do at Adobe, but I hope it isn't writing specs, determining product features, customer support, or anything else which will directly affect me or my work.

It's not about 'like', its about FUNCTIONALITY. Roman Numerals are horrible for bookkeeping and accounting, and yet they remained in use 500 YEARS after the introduction of Arabic Numerals. And it doesn't matter if you have a bridge between Roman and Arabic numerals, because fundamentally Roman Numerals just don't have the semantic capacity to address the problems we want as effectively, IF AT ALL. And any changes to extend them will just result in a poor imitation of Arabic Numerals.

Computer science is ultimately a practical discipline, and so there ARE objectively better ways of modeling things. Dismissing all the counterpoints as mere issues of taste is not just disrespectful, it's deeply ignorant.
 
+Anon Adderlan Your quoting was a bit selective. In the end, I don't think anyone here disagrees with my fundamental point, so let me try to state it even more clearly:

As web developers, we rely on the platform to provide us with the capabilities we need to build our applications. There is a fundamental gap today in that we don't have any cross-browser capability to store an acceptable-for-modern-apps amount of data in a performant way on the user's computer.

Me: "I don't care what technology wins as long as it's a fast-enough, ideally-transactional technology upon which we can reliably build apps for all browser that hold a "reasonable" amount of data."

You: "...That's EXTREMELY vague, and kinda undermines your credibility when it comes to arguing about standards. But lets see, fast-'enough', 'ideally'-transactional, 'reasonable' amount of data? Looks like WebSQL fits the bill!"

This is what I mean by selective quoting. The sentence before the part you quoted was this: "I would personally be fine if WebSQL DB "won"."

Yes, Web SQL DB fits the bill, except for the fact that it's dead. I know what resurrect means, but perhaps we disagree on the definition of dead. Look at the honking big warning that was posted to the spec 3 years ago:

http://www.w3.org/TR/webdatabase/

That sounds like dead to me, which is why I say that if someone can successfully resurrect that, then awesome, problem solved.

I've done tons of work with relational databases, including sqlite. They're great. But 2/3 of the browsers in use don't support it and that is the issue.

I care quite a bit about developer ergonomics and have a pretty good history there. Where I'm coming from is that the ergonomics of "there's no way to do what you want to do" is the worst scenario. Think back to 2004/2005... the native DOM APIs were really crappy and verbose. jQuery, Prototype, Dojo, MochiKit and friends all provided much more humane APIs, some of which ultimately led to nice changes in the browser APIs.

But, imagine if the browser didn't even have the DOM manipulation APIs?

So, I am coming at this from the perspective of practical consideration. Having the capability to build the apps you want to build is objectively better than not.
 
Except, you know, you're all over the map.

You: "Web SQL Database needs to die"

later

You:  "I don't care what technology wins as long as it's a fast-enough, ideally-transactional technology upon which we can reliably build apps for all browser that hold a "reasonable" amount of data."

so....not exactly a consistent message.  No wonder you're getting multiple pages of grief.

You know there was a group of developers who worked on web browsers who took a look at where w3.org was going a few years ago and decided that they were on the wrong path and so they formed WHATWG and decided w3 standards be damned - lets build something cool.

The WebSQL "standard" is only "dead" because there is no point in writing a standard for something that has only one implementation.  

I get that Microsoft said they didn't want to play.  So what?  They don't exactly have a track record for implementing the standards that are there, why would this be different?  That's why we put "go get firefox because IE is broken and we're not going to work on making it work" links on our pages.  

Of course lately its "go get Safari or Chrome because firefox developers are being morons and our software with the nice local database won't run on their crappy browser".

Not everything the w3 does makes sense.  De facto standards are better than standards without implementations.
 
My mistake was in not being clearer earlier on that the reason I said "WebSQL Database needs to die" is because the standard is not moving forward and we need the functionality that it would give us. I made things worse by trying to explain what the rationale is for not adopting WebSQL DB, when my biggest issue is just around having the ability to store and retrieve data well.

That problem with the statement "de facto standards are better than standards without implementations." is that we don't have a de facto cross-browser standard. Most web developers don't have the luxury of telling their users "Go use Browser X".
 
If sqlite/websql is so bad can anyone tell me how indexeddb could be better when it can't be used offline ? (yes indexeddb can't be used offline)
One of the idea to push data client side is not only to reduce server roundtrips but also being able to work within a browser even when you've lost you web connection right ?
 
Well, SURELY my bad.....But if you go to http://caniuse.com/#feat=indexeddb and click on KNOWN ISSUES (the second tab from the left on the page bottom) you'll read that: "IndexedDB is only available on websites with http or https URL schemes (not locally)", my interpretation of that was it's not working offline.. but if someone could point me how to use it offline i'd agree that sqlite/websql should die. But for all ipad users (and there's a lot you know)...we don't know what apple's decision for  ios7.x or 8: sqlite/websql or indexeddb....PLEASE CORRECT ME  thnx...
 
Ahh, I see what you mean. Theoretically, offline web apps still use the http or https URL schemes. They are made available through the app cache (which is where a lot of the trouble lies). As long as you can make your app work offline, indexed db should be available (as should websql db, where available)

And, yep, as noted earlier iOS is the biggest hole in indexed db support today and someone wanting broadest support should use one of the libraries that uses the "best available" storage for each browser.
 
Great ! as long as i don't save the app itself (on a dektop for example), and then reopen it by double clicking (which transforms it in a "file://" scheme)  i should be fine right ?

As for ipad: if someone installs chrome there's still no support for indexeddb...because Chrome or any other browser on ios is actually like a "skin for safari" isn't it ?

That would explain why we can't apply themes on chrome on ios....but wouldn't explain why we can't apply themes on chrome for android (which btw can apply themes on firefox)....But thanks for the really important point: as long as my browser thinks he is in such a domain (http or https scheme), it's happy with indexed db, being there a web connection or not ? (affimative question) :)
 
That's right. The offline application cache is supposed to let you use http://foo.bar even when you're offline. The web app itself sets up for offline use. The only problem is that app cache is hard to use in practice.

Chrome on IOS is limited to the WebKit engine that Apple provides. On Android, Chrome and Firefox both are unrestricted in what they can offer.
 
"The only problem is that app cache is hard to use in practice" ..., which means a lot of troubles ?.....I'm about to start a big development client-side in the next weeks,, and i tought that websql would be there for at least two years....especially on ipad. On the other hand i think that w3c dropped websql because the giants want to sell stores applications, not applications in browsers....
 
after reading the douchebag article, i think i'll take the chance for a while with sqlite : my core business is on ipads: with safari i should be fine for 2 years while indexeddb stabilizes. Chrome should be able to assume the interim on workstations what do you think of that approach?
 
My core is on ipads....but The Core is always volatile....i must be ready for everything.... pouchdb ? I'll have loot at it...many thnx
 
+Kevin Dangoor Many people simply suggest PouchDB. Has anybody successfully use PouchDB in real web app? Horrible first time initialization,  full table scan for query, fragile sync, could not serve more than two clients simutinuous sync, currently design for few records (say less than 10,000 records in the database).  
 
>I've done tons of work with relational databases, including sqlite. They're great. But 2/3 of the browsers in use don't support it and that is the issue.

You know, if Firefox got off their asses and supported it, it would only be 1/3 of the browsers not supporting it.

As it is - I specify WebKit browsers on mobile devices.  Which is fine since that's where they are.

Bottom line is I can't build what I want without databases that can do proper joins.  So

>Having the capability to build the apps you want to build is objectively better than not.

doesn't ring true either. Since Firefox is busy making sure I don't have the capability to build the apps I want and trying to bait and switch me into some stupid toy key store.

The right thing to do to empower developers is support a sql database in the browser.  The wrong thing to do is tell us we don't need relational databases. Because we do.
 
right on Todd...i've read a bit since Kevin Dangoor shared his opinion....and there's no great hope there.....I think that the idea is simply to kill the disconnected web app: Apple and android soon Microsoft ecosystems want their slice of OUR pie : this is why they promote applications instead of web apps (that are better for everyone but them)....But you're right : " The wrong thing to do is tell us we don't need relational databases. Because we do."
 
+Kevin Dangoor at this moment i think mozilla is trying to enter the ecosystem mode with their firefox os: by doing so they're will be a supposedely genius that will "out of the blue" will come out whit a firefox store...and then...i don't need to explain. I wish you're right though: webb apps (with available disconnected mode), would be supposed to be the democratic future of applications....according to my vision .....but it's only my opinion....just google a bit "end of web app" there's a big debate around html5 that would be in fact just a swansong ("chant du cygne" in french), for the webapp.
 
Currently the problem is indexdb doesn't support with safari. and web sql doesn't support with Firefox and IE. I'm planing to develop a web application that runs on every platforms(desktop and mobile). How do i chose which local storage to use.
 
+Simon Mayrand I missed your reply when you posted it. The Firefox Marketplace is not at all out of the blue, but I don't think it's what you are expecting it would be. The apps in there are webapps, the APIs they use are developed in the open with intent to standardize and even the marketplace itself is open source. https://github.com/mozilla/zamboni   The Marketplace very much fits into the Mozilla Foundation's mission.

+shasheeka samaraweera Yep, that's a problem. There are libraries out there that provide consistent cross-browser APIs. I'd recommend just googling for that and finding one that fits what your application needs.
 
I'm working on localizing android phone app and all of the newer phones don't run the queries properly. I don't even know why any of the stuff that the previous devs implemented with SQL commands couldn't have been done more directly with JavaScript, but it looks like it will be a big PITB to move to a different metaphor.
 
Geoffrey: they don't want democratic web based anymore: they want you (US), to program custom applications for android, ios, soon firefox OS.
They don't want you to be free to program disconnected applications with data more than 5MB. I've been for our business exploring al the avenues and the obvious conclusion is don't rely anymore on sqlite, don't rely on indexeddb that is a "thing" to make you think that they support web local storage, don't waiste your time. I completely disagree with the opinion in this post , that indexedDb is the way to go: if they killed sqlite it's because they didn't want you to program for browsers with local persistence anymore.
 
+Simon Mayrand Regardless of your opinion on websql vs. indexeddb, you are 100% wrong in your opinion about "they didn't want you to program from browsers with local persistence anymore". There is plenty of evidence to the contrary (I'm sure the debate on whatwg was lengthy), including the fact that Mozilla is building a mobile platform entirely on web standards and clearly the ability to store data is important.

Also worth noting: Safari 8 (both Mac and iOS) supports IndexedDB. Assuming IE 12 goes from partial support to full support, that means all modern browsers fully support IndexedDB soon.
 
Well...future will tell, hope you're right, in imho the obvious agenda is to bring people in the stores not in a browser.
 
That's almost true about Mozilla, but sometimes, if you want to survive you must alter your original mission, it's called adaptation. And if mozilla that is only making a browser was completely coherent with their manifest, they wouldn't sneak around the OS for portable devices. But since applications stores (apple rules), are the new paradigm for portable devices they have to adapt or simply die. I don't disagree for fun: i truly prefer "democratic" (browser) development....it's just that investment in developing browser applications is a risky business.....at least for a while. I've been making an extensive study about what is coming, and sadly, what people want for their pad or phone is an app...not a bookmark. Actually the demand is so strong that when you're in the business, and you don't offer an app for an ipad (60% ot the tablet market), your competitor who's making the pad app will win over you: even if you're web application is full indexeddb or works perfectly disconnected. Even Google drive is an application !!!!.....and if you chose to edit documents in safari in google drive connected it's a nightmare....imagine  disconnected. Spinning indexeddb is a mistake according to the current of apps wave.
 
Yep, there's definitely huge demand for iOS and Android apps. Mozilla is certainly adapting with all of their work on Firefox OS, but it's a steep hill to climb.

This is getting way off-topic for this thread, but iOS 8 is a good step forward for web technology because apps that use web views on iOS won't have the same JavaScript performance penalty they've had to date.
 
good news: we've chosen after long reflection process to program an ipad app. Some parts use webview....off topic but sympathetic :) 
 
+Todd Blanchard I have just come across this thread as I am relatively new to mobile development, but have recently successfully deployed  apps for iOS and Android.

I have to say I am in full agreement with Todd on this.  I am staggered to read ".... lazy developers who can't come to grips with SQL and write crazy amounts of code trying to avoid learning it. I prefer working with well rounded competent people." w.r.t. developers avoiding the use of WebSQl/SQLite

Todd is absolutely right about this.  What's so difficult about learning SQL?

I come from a background of php/mySQL, but it is all self-taught and I got to grips with mySQL in a matter of a few days - I had to, because many years ago I took over a website which had been abandoned by the previous developer, and had to get it back up and running urgently.  Looking at WebSQL syntax, I know if I was put in the same position with it as I was with mySQL, it would take me a similarly short time.  It is clear, intuitive and powerful.
 
Hey Kevin. How's it going after all this time ? You still in IT ? Have you written any great IndexedDB apps lately ? Or seen any ?
 
Hi Simon. Yep, I'm still in software. I have not needed substantial client side storage in my webapps yet.

Many people don't talk about how their apps are built, but I think that usage of IndexedDB is increasing either directly or through wrappers like PouchDB. The Financial Times wrote specifically about how they use IndexedDB and fallback to Web SQL as needed, which is what PouchDB does. I'm guessing that's pretty common.

All major web implementations support IndexedDB now, though with varying degrees of quality (iOS) and completeness (IE). It's a shame that it took 3 years to get there with the release of iOS 8.
Add a comment...