Discussion  - 
Hi guys,

I guess I should post something about my recent activities and thoughts before it gets too long between posts.  I've been thinking about how to implement a thing tracker repository, i.e. a way of storing things which uses the TTN spec. as a basis.

Initial ideas revolved around a web service/app which handles the indexing of metadata, plus the humdrum aspects such as user access etc.  Content distribution would be based on bittorrent - largely following +Derrick Oswald's  thoughts on the matter.  

This idea felt a bit lacking however as, even with the interesting BT aspect, it would be "just another web site" and suffer from many of the problems TTN is aimed at addressing.  So the idea moved on to some form of web app which others could install and run, and which would join up to form the network.  This intuitively feels a step in the right direction, but further discussions and thinking leads to a more extreme, and interesting, proposition: a fully decentralised p2p network client.

The aim is for a client which can discover content from the TTN network, and can be used to publish one's own designs in the network.  The client would ideally cover identification and authentication of participants (though not necessarily personal information) along with versioning as outlined by Derrick in his recent post.

Details are still being worked out, but at the time of writing I am leaning towards a Node-webkit based application using the KadOH Kademlia DHT library for the overlay network (distributing node, thing and tracker information), and using Derricks Bittorrent work as a basis for distributing the actual content.  All open to feedback and opinions of course.

If anyone wants further information just say the word and I will try and find time to knock up some pictures and further explanations.
Alex Fielder's profile photoRoss Hendrickson's profile photoGary Hodgson's profile photoJason Gullickson's profile photo
so would this work like utorrent and it's DHT?
+Alex Fielder Yeah, similar.  I imagine the TTN DHT would be used to exchange and store three types of information: details of other nodes, indexes of things (one or more trackers) which originate from a node, and metadata about things as mentioned in the TTN spec.  I'm still considering whether the TTN DHT should/could be used to share torrent information, or whether it makes more sense to keep the overlay (TTN) network distinct from the content sharing (BitTorrent) network.
I would suggest also that we make sure this 'deep network' would be easy to index. A search engine is going to have to be mandatory for the general populace to be able to find anything. +Gary Hodgson are you working off the thing-tracker-network repo for this right now? Are you planning on having the nodes share and index things across themselves or each application instance only cares about itself? The distributed node idea is powerful, I however have little experience with coding such things so pardon the ignorance.
+Ross Hendrickson I think that indexing of the p2p network would involve a node joining and acting as an agent, collating the trackers and things in the swarm.  For these kind of things I am thinking that the DHT might have particular commands that can be issued which floods the swarm to gather such info (all under controlled circumstances of course).  The agent would be used by a web app or service as a data source, allowing the things to be browsed, searched etc.

The important distinction is that this web app would simply be another participant in the network.  One of the exciting aspects of TTN to me is the possibilities it might create to develop this kind of ecosystem!

The nature of the DHT (and I'm learning this myself as I go along) is that the data is stored across all participants of the network.  With it being duplicated across a certain amount of nodes to add redundancy.  

By the way, if you have the time I would suggest glancing over this project report that the KadOH guys wrote ( I think it gives a nice high-level overview of DHTs and kademlia etc.
+Gary Hodgson that's the approach I've taken so far to "bridge" content "served" via DHT network into the traditional HTTP realm.

I'm excited to see what you've got so far, I played with Kadoh a bit and I see how to use it as a K:V store, but I still don't completely have my head around how to do the "file transfer" part yet, without leaning on an existing torrent client (I'm sure it can be done, but I'm getting tangled up in the details).

In any event I have a dozen additional applications for the tech, so I'm very interested in developing it, let me know if you can use some help :)
Add a comment...