Shared publicly  - 
 
I've been working on an idea over the last few weeks which I would like to share with the community.  It's a proposal for a way to share designs, projects and "things" within the maker community.  To try and present the idea I have created an information website: http://thingtracker.net

To be clear, this is really just a starting point to promote discussion, with the hope that the whole community can provide input, and together we can produce the start of a decentralised Network of Things.

The idea percolated from the discussions following my creating githubiverse [1], and also from the forum thread of +Richard Horne  about a RepRap Object Repository [2].  And whilst it certainly isn't an original idea, I think the time is ripe for something to develop in this area.

I can envision a future where people can develop apps and tools that hook into the Network of Things and offer all kinds of services: search engines; boutique object repositories; personalised lists; and so on.  All open and decentralised.  Pretty idealistic perhaps, but still pragmatic enough to become reality.

There is a whole host of use cases and ideas surrounding the proposal, as well as many potential issues, all of which I hope we can discuss further, leading to something that really benefits the community.  So, if you can find the time, please have a glance over the website and share your opinions.

-----

[1] http://garyhodgson.com/reprap/2012/09/githubiverse-a-github-pages-template-for-3d-printing-projects/#comment-6578
[2] http://forums.reprap.org/read.php?1,160251,161357#msg-161357
Rationale. Makers like to share their creations, but choosing where to share their work, so that it is discovered by the right people, can be prove difficult. Anyone wishing to build a creation-sharin...
19
9
Adrian Stabiszewski's profile photoNelson Neves's profile photoDiego Trapero's profile photoJuan Gonzalez Gomez's profile photo
45 comments
 
Wow, you have put a lot of effort in to that. Well done. I'm going to keep reading... 
 
Thanks! I'm excited by the idea, and the possibilities it might bring - it could be something to counter the growing preponderance of walled communities.  

A use case that keeps popping into my head is that groups, such as hackerspaces, or teams, or simply groups of friends, could put together a simple site which partakes in the network, and so their things are widely discoverable and yet they can format and run the site as they wish.
 
I can totally see this for hackerspaces, as well as everything else :)
 
I especially like the networked approach as someone who loves to build something from the ground up and tinker with APIs -- it makes it possible to use that energy for something useful if other people can directly use it without it having to reach some critical mass in its userbase first :)
 
right, and you don't have to compromise optimizing for one case just to gain breadth.  Also I just want to be able to find stuff before I go re-inventing the wheel.
 
Wheel inventing rules! Just not always ;)
 
I dislike the generic idea of "tags".  Either nobody uses them or they get bloated with concepts from too many aspects and don't mean much.  If we wanted to track pieces of art we would probably want categories of tags like:  Medium, Subject, Style.  For physible things why shouldn't we have multiple lists of specific tags like:

 Subject - is it a tool, piece of art, kitchen appliance, part of another tool, an abstract piece?
 Relationships - whether something is a child, remix, or required part of something else (URL/#UUID of another tracker JSON file)
 Media - what are the preferred media?  Plastic, Wood, Metal
 Area of Interest:  Is this related to games, scientific engineering, cooking?  A spatula that is 1 cm tall and fits into the hand of a miniature figure is still a spatula, but the area of interest is not "the home" or "the kitchen" but "gaming" or "figurine" or "art".  A propeller mount is a propeller mount, but one for a real prop plane and one for a miniature RC helicopter have different areas of interest.

I think these extra meta data would allow for a lot more filtering/searching of things to find stuff someone is interested in.
 
 
+Mark Kimsal These are exactly the kind of thoughts I was hoping for.  You're quite right in that one of the most important roles that the metadata plays is to allow people (and systems/apps/etc) to discover, rank and categorise the content.

The advantage of the term "tags" is that it is generic, the disadvantage of the term "tags" is that it is generic.  :)

I'm thinking generally there would be a core set of required attributes, and then a set of optional additional attributes. This leaves quite a bit of flexibility I think. Plus allows each participant of the network to provide the amount of information they wish.

Additionally, I think it would be useful for some more of the attributes to be arrays.  For example "license" and "author" would make sense as a thing could be dual-licensed and have multiple authors, etc.
 
I think some systems can combine all the specific tags into a generic "tags" if they wish.  But if the schema forgoes the opportunity to be specific then we are left with only generalities.  I guess what I'm saying is that you can always go in reverse to get more generic.

Also, a schema version tag would be nice for scrapers and spiders.
 
I would say, make all the meta-data tags optional (the sort of ones I outlined), and see how the community uses them.  Nothing will be perfect the first time around.  But, it should at least instill guidance as to how it should be used.  If the purpose is to help distributed teams build projects, perhaps a field like "last-updated" or "file format" would help automate some source file updates or compilation steps.  If the purpose  is to discover new things in a social way, then some meta-data tags about area of interest would keep people's updates from being cluttered (and in turn might be required as an attribute).  If the purpose is undefined or amorphous then allowing unlimited optional tag lists would allow people to visualize their own benefit and create systems to meet their needs (at the expense of having an immediate, tangible benefit).
 
This is a fantastic idea +Gary Hodgson (currently reading the full doc, and boy do I like what I see)! Thanks for working on & sharing this! Very promising

On the subject of tags: all this sounds very similar to semantic web /RDF type relationship and concept definitions, perhaps existing systems/formats could be reused in our context (although, to be honest  last time I checked, all the semantic web stuff was very confusing and felt more like drafts/experiments than anything mature)
 
Gary, I glanced over it - looks nice, as far I understand you propose an umbrella, a meta layer in front of all actual hosted things, correct? A tracker responses with all things it knows about . . . that can be a lot, e.g. I write a tracker which represents all of thingiverse.com (30k+ items), partial retrieval or updates-since (I would write such tracker using the rss feed of thingiverse).

Ok, a little brain storming from me:

We have 3 layers:

1) front: non-existant yet, catalog, search-engine, rss feed, harvests / walks from thing-trackers to thingi-trackers
2) thing-tracker: knows about things (your proposal)
3) thing-hoster: thingiverse, github, personal blogs etc

Now, level 2 & 3 is diverse, yet 2) links to other 2), so we might end up with a web of thing-trackers, what is most interesting for me is, level 1).

TODO:

- front-end as of level 1, I would propose catchy domain-name, and have it hosted on every continent, and DNS responds to the nearest one, should be overseen by a group, you Gary, perhaps OSHWA and EFF (having their legal expertise on board).

- thingi-tracker for thingiverse, shapeways and so forth.

- open source thingi-tracker (python, perl) (e.g. one creates local directories for each thing, each directory has multiple .stl, .txt, .html, creating previews per stl, and responds to thingi-tracker requests, a quasi lightweight DIY thingiverse)

SUGGESTION:

Consider to add following fields per thing:
- *"creation-time/ctime" (timezone aware)
- *"modification-time/mtime" (timezone aware)
- *"version" (x.y[.z])

this would simplify track updates, new(est), rss-feed (e.g. a tracker actually saves the meta info of things, so a tracker knows which thing is updated often, is actively developed).

CHALLENGES:

For me it all makes sense when the trackers finally have a frontend, there then searching things will be easy since we have title, tags etc, but the catalog will be challenging as it relies on the data provided, some people will not care about a proper catalog tagging - perhaps it's worth to think of creating a catalog along with the proposal and make it mandatory to have each thing have a catalog tag set.
 
+Mark Moissette Which is why I didn't dare to mention ontologies yet, as the academic nature of all this stuff has rendered it pretty much unusable for the general public :/

I like the idea of defining a basic sets of attributes (as outlined by +Mark Kimsal) and then just going what else will become out of that. Given that it should be quite easy to crawl the things on the trackes and therefore allow data mining and clustering of the associated metadata, it shouldn't take long until someone finds THE descriptive tag cloud for the Maker community while playing around with this ;)
 
I agree +Gina Häußge, I think the sooner there is something to play with the sooner we'll know what works, what doesn't and where to spend energy on the next iteration :)
 
+Mark Moissette I think the problem with Semantic Web is that it's trying to get into an already battered HTML space.  People are more interested on getting their sites to render properly in desktop and mobile browsers than they are in letting non-existent spiders "understand" their site.  But, Semantic Web is pretty big in the scientific/medical fields for bibliographies and stuff.

In this case, there would be no crusty standard to try to make people change their habits to fit:  this standard is the only standard.  Plus, trying to mark-up physical things (or digital files that describe physical things) is a lot easier than trying to add navigation, citations, and relationships to any Web page that talks about anything.
 
+Mark Kimsal +Gina Häußge +Rene K. Mueller +Mark Moissette  - This is a great discussion already!  I'm having to do some regular work at the moment :( but will respond more fully asap.  

In short though I think if we make the requirements of participation to the network flexible enough it will provide an incubator-like environment where people will hopefully feel welcome to play and experiment, allowing de-facto standards to emerge.  If it's open, and and participants embrace the idea, I think there are a lot of possibilities.  And I would add that this should include the opportunity to explore perhaps more academic or advanced topics - such as the semantic web and ontology.
 
schema({
  "version": 1.0,
  "things" : Array.of(
      {
          "title": String,
          "URL": String,
          "UUID": String,
          "*author": Array.of(String),
          "*license": Array.of(String),
          "*thumbnailURL": Array.of(String),
          "*description": String,
          "*x-metadata": Array.of(Array),
          "*subject": Array.of(String),
          "*media": Array.of(String),
          "*aoi": Array.of(String),
          "*relationships": Array.of(String),
          "*history":
              { "initial release" : datetime,
                 "second release" : datetime
              }
      }
  ),
  "*trackers" : Array.of(
      {
          "URL": String
      }
  )
})

where: UUID is only guaranteed unique within the file, external systems should use URL + UUID for guaranteed uniqueness.  History is an object where string keys can be version numbers of no particular format and the values are unique datetime values with standardized time zones.  X-metadata is a free-form object where keys can indicate new categories of meta-data and the values are arrays of strings.  Relationships is an array of URLs to other thing tracker files with a hash and the UUID of the specific thing http://foobar/mythings.json#111-222-333 

If it seems useful, it should be standardized so we don't end up with competing formats like "I use 'materials', and you use 'media'" or something like Atom time zone formats.
 
Linus has a simple metadata format for git pull requests and commits that is amenable to the building of a collective consensus prior to the institution of standards. In a free form text field of arbitrary length, precede the raw description with lines of the form

attribute: value

Easy to enter by hand or machine even in systems that provide no support, easy to strip out programmatically, and easy to extend or retrofit.

As to relationships in Mark's proposal, they need to be pairs, one string for the relationship, one for the url.

Personally, I would prefer the identity be provided by a URI that is never dereferenced, with the URL different and subject to change without breaking links (I.e. to reviews on a review site, or comments, likes, plusses, etc on a social site), but there would need to be some form of limitation to ensure that Joe Black Hat can't post his own entry under my URI and steal my reviews, etc. I guess that is something of a benefit to the URL model, but URLs are worryingly impermanent, and it would be nice if a tracker could be moved without breaking everyone else's databases.
 
dude, turn off whoisguard for your domain, nobody can figure out your email address and you're breaking things
 
+Bryan Bishop the URL / URI references a web-page, where you can link all files. I don't know if it's wise to link all files in the thingitracker metadata already.
 
Some elements, such as author and license should support being given as URLs. Thingiverse gives author pages, many social sites provide individuals with URLs that effectively lead to a page that designates an identity. It would be nice if multiple URLs could be provided in a way that makes it clear they are the same individual, though this is certainly less vital. It would certainly help search sites to provide links to all items submitted by a given author, though.
 
Anybody know how big the thingiverse database is?
 
+Bryan Bishop re: "turn off whoisguard..." - what things are being broken exactly?  I added an email address to the thingtracker.net page, but it's obviously not visible enough: contact@thingtracker.net

Also - feel free to expand on your comment on how json wouldn't be sufficient to represent "things". 
 
Incidentally, while I don't object to web services or JSON queries, I would recommend that some support be offered for simple files from an http GET or ftp... could be formatted like a JSON response; but that would be ideal for a git hub repository, where the tracker file would be easily kept in sync with updates to the objects, with no need to obtain hosting for running a web app.
 
+Howard C. Shaw III Storing a tracker file in github was one of the first places I thought of, simply because of the hosting, and also for versioning and the ability to accept pull requests, etc.  It would allow people to create and manage a tracker really easily.

re: formatting.  My initial idea was to suggest that trackers be made available in a range of formats (txt, json,xml) and perhaps identified simply by their extension, or a URL parameter.  This of course is still a perfectly valid option - I chose to only use json in the current draft to simplify the message and get the discussion started, and also because of json's ubiquity in web interfaces.  Having an attribute:value text file would be ideal for hand-crafted trackers, again for example hosted on github.
 
+Rene K. Mueller +Bryan Bishop +Howard C. Shaw III 

This initial proposal follows a model whereby the tracker acts as a pointer to an existing repository, represented by a URL.  This repository may exist anywhere of course, or even be a part of the tracking service which hosts the tracker, an all-in-one service if you will.  The flexibility here of course is that the page pointed at could take any form, allowing a wide range of websites to participate. The disadvantage is that the contents of that page are opaque, unless information is provided through more attributes.

The proposal could be expanded to encompass a way to give detailed information about each thing, for example including a bill of materials, instructions, detailed plans, etc.  Perhaps utilising the SKDB, as mentioned by Bryan above, in some way.

I see no reason why multiple levels of information cannot be represented by the proposal within the network. Allowing some to track at a relative superficial level, whilst others use it to distribute complete, detailed design packages.
 
Based on that last comment, Gary, it seems to me that you have two different situations in terms of the level 3, the actual object storage, and these are supporting sites, and non-supporting sites.

For example, non-supporting sites include (at the moment, until and unless they add support), Shapeways, physible.com, Thingiverse, etc. Supporting sites include github and private hosting that have files that follow the format.

So if someone wanted to host their data on Thingiverse, and only keep it in one place, but still be able to have it found by trackers prior to Thingiverse implementing support for it, then they need to be able to host a file (JSON, as we've been discussing), that has in itself all the relevant metadata for their Thingiverse entries). Based on Mark's example, this would be one file containing data on all the models they are tracking on Thingiverse.

Now, if someone else has a github, it would make a lot more sense for them to have one tracker file with URL's, maybe names, and very, very little else; and then to have a second file (perhaps the one pointed at by the URL, perhaps merely one on that HTML page linked to in a canonical or recognizable way, perhaps one tagged in some secondary way in the initial JSON file) for each item, stored with that item, so that they can comfortably edit it when they edit the item, and not have to do additional work.

Given a reasonable standard for that file (such as Mark's JSON format, but with only one entry each time), a script could easily be written to create and update that primary file that is submitted to or linked to by other trackers.

So based on that, it seems to me that we want both - to have an utterly simple link, or a complete set of metadata, but that we also want to have a sort of import by reference.

All this, it seems to me, is pretty closely handled by Mark's format already, assuming (not being familiar with JSON), that those attributes in the schema preceeded by an asterisk are optional? There is only one major exception I see, that is probably important to get in early, and that is some indicator in the initial link level as to whether the link should be followed as part of building an index, searching, etc.

That is, we need to be able to say, if, for example, we are pointing at someone's site that prohibits crawlers, or merely where the linked file is not of a useful format, whether or not the URL provided should be pulled and processed for further metadata.

Particularly, it would be nice in the github case to be able to tell a search engine or other crawler that you really must process this URL to have a complete entry, to force it to pull in the linked JSON entries for the individuals, and likewise in the Thingiverse case to be able to say that I've included all the necessary metadata right here, please don't crawl further and annoy my hosting provider. Or in the case of private hosting of files, if someone provides a service that lets you upload tracker files publicly, it would be nice to be able to say I've included everything in this tracker, no need to crawl the linked URL's and push me over my bandwidth cap, let's save that for the interested parties that are actually looking for my particular item.
 
With respect to bill of materials, there is a whole other website for someone to make. A registry of vitamins, with a canonical URL for each, and an interface allowing the community to provide links to and ratings of local or locally appropriate sources for each. Then a website could pass in, for example, a list of vitamins from the Wallace tracker, and get an interface showing each item and the highest rated, closest, and or cheapest sources for each, and let you choose. Maybe you could eventually have one sophisticated enough to have internal BoMs so that it could offer prepackaged vitamin packs... I.e. evaluate, say, a Prusa BoM and notice that buying so and so's motor package and some other outfits nuts and bolts bundle plus a few singles is your best deal.
 
I don't think that we necessarily need a format that has a description and URL for each entry, perhaps it is sufficient that we should just agree on a standard such that any given text field may end with a semicolon followed by a URL.
 
Hi +Howard C. Shaw III   I want to make a fuller response later today, when I get the time, but wanted to briefly say that your comments seem to closely match, or could be covered by, the proposal, either as it is, or with some further development.  

I think your point about having the tracker be able to inform others how deep to search, etc. could be covered by Policies. http://thingtracker.net/specification.html#policy  These could be an agreed upon set of values which allow clients t make informed decision about how to process the data.

I also think that by keeping the specification very flexible we would be able to create a network structure as you describe, with perhaps an index tracker which points to more detailed information which in turn point to BOMs etc.  It makes me think that having a "tracker type" attribute would aid clients in knowing how best to process each tracker, i.e. index, bom.

I also think that all of this is ripe for automation and integrating in build processes etc.  So long as the specification is light-handed it should allow many types of experimentation.

Tonight I will try and find time to mock up a few example use cases, as I think these would really aid people's understanding of just how powerful such a network could be.  I wanted to add a "use case" page to the site in any case.
 
+Gary Hodgson I just miss the disregard for this project with a multilingual perspective. For example, the tag could be standardize in a way that was independent of the language used, so "steel" and "acero" mean same regardless of using English or Spanish.
 
I'm tagging +Simon Day because he and I have been discussing this very topic over the past couple of weeks.

This looks to me to be very similar in concept to that of BitTorrent trackers- am I wrong in that assumption?
 
Hi +Alex Fielder I'm not to familiar with the technical details of BitTorrent trackers - but these were certainly at the forefront of my mind when I was drafting the proposal.  I imagine the ideal Network of Things would be very similar to a p2p infrastructure.

Would love to hear your ideas on the subject.
 
+Gary Hodgson I love this proposal. I think there is a lot of structure there that could be built on in a lot of ways. I think the key to this proposal's success would be how easy it would be to participate in the network. So an established seed site that helps people make trackers for their own work and search for others would be key. Perhaps you could create a circle and add interested parties to it for further discussion? This thread has raised great ideas and it might help the project move forward if there was a way interested parties could interact.

It would also be interesting if you could have a node on the tracker framework that would allow you to fully describe your project (so the STLs involved, instruction files, BOM etc.). That way you'd have the tracker point to a URL (where you've hosted your content) and then using this "thing" node you could then grab out the 3D files if you were trying to "crawl" through the network and grab out specific information. Almost like a more descriptive meta-data layer for interested parties above the URL and its meta-data (all the tags etc.).
ThantiK
+
2
3
2
 
Just like CSS/HTML, separating the data from the presentation here is an awesome idea.  Thing Tracker would basically be the back end, and could allow lots of participation.
 
Hi +Juli Fowler  - thanks for the link! This is a good example of the kind of sites that I think would benefit from a Network of Things - allowing the projects to be discoverable by a wider audience.  And joining the network should be as simple as adding a tracker, which is updated with new projects, and when key metadata is changed on existing projects.

It makes me think that the barrier to entry would be dropped even lower if there is a way to hook in atom/rss feeds into a tracker mechanism... hmmm, one to think about.
 
+Juli Fowler I've had that ode link bookmarked for a looong time. Good to see it finally made it to Public Beta. :)
 
Hi everyone. Thanks again for the amazing response about the proposal.  I will collate everything and start revising the proposal.

I have also created a dedicated G+ page (+Thing Tracker Network ) to act as a blog/discussion site, so people can use that rather than having to wade through posts on my profile.

(EDIT: I was going to add everyone who commented into a circle for the page, but the rules say "Pages cannot circle you until you follow them first", so feel free to opt to follow the page if you wish.)
Add a comment...