Profile

Cover photo
Tatsuhiro Tsujikawa
Works at Hitachi GP
Attended Osaka University
Lived in Kanagawa, Japan
68 followers|28,802 views
AboutPostsPhotosVideos

Stream

Tatsuhiro Tsujikawa

Shared publicly  - 
 
I wanted HTTP/2 and SPDY benchmarking tool, but could not find anything for that purpose. spdycat and nghttp make some load but they only produce one session. I want massive amount of concurrent sessions going on. So finally I wrote the tool called h2load http://tatsuhiro-t.github.io/nghttp2/package_README.html#benchmarking-tool. It is included in nghttp2 project, and if it is built with spdylay library, it supports SPDY protocol (including spdy/2 to latest spdy/3.1). The h2load UI is inspired by weighttp, and they look very similar. The command line options are similar too. Here is a sample benchmarking result:


$ src/h2load -n100000 -c1000 -m100 https://127.0.0.1:8443/
starting benchmark...
progress: 10% done
progress: 20% done
progress: 30% done
progress: 40% done
progress: 50% done
progress: 60% done
progress: 70% done
progress: 80% done
progress: 90% done
progress: 100% done

finished in 11 sec, 939 millisec and 837 microsec, 8375 req/s, 8931 kbytes/s
requests: 100000 total, 100000 started, 100000 done, 100000 succeeded, 0 failed, 0 errored
status codes: 100000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 111773815 bytes total, 599031 bytes headers, 108600000 bytes data

" -n100000 -c1000 -m100" means that generate total 100,000 requests, using 1,000 concurrent clients (HTTP/2 or SPDY sessions), with 100 max concurrent streams.

At the moment, h2load is built on top of the nghttp2 library and its version is now draft-10, which means it cannot be used against other implementations, which are still draft-09. But after IETF meeting, httpbis will publish next implementation draft-11, then we can use this tool for testing. I fixed several bugs with this stress tool, so it is worth trying out.
4
4
Rekha Das's profile photoRoberto Peon's profile photo
Add a comment...

Tatsuhiro Tsujikawa

Shared publicly  - 
 
Today I built debian's new xserver-xorg-video-radeon 7.3.0 with libglamor 0.6 from ubuntu. Now xv is back to work again. Also gl output with vblank_mode=3, tearing is mostly gone. As a side note, I have to remove glapi-link.patch from ubnutu glamor because it causes linker error.
1
Add a comment...

Tatsuhiro Tsujikawa

Shared publicly  - 
 
HTTP/2 work is in progress. Now we are in 7th draft (http://tools.ietf.org/html/draft-ietf-httpbis-http2-07).
Although it is not marked as implementation, some of the developers start implementing it. We nghttp2  project (https://github.com/tatsuhiro-t/nghttp2) is one of them. We noticed that Firefox (https://wiki.mozilla.org/Networking/http2) is also implementing draft-07. We have made some conversation and fixed several issues (all related to header compression) and now Firefox and nghttp2 work very well. Interop FTW.
3
6
Khaldi Ahmed's profile photoDaniel Stenberg's profile photoBrian Kardell's profile photoRoberto Peon's profile photo
4 comments
 
node-http2 is now up to draft-07, too. Seems to be working fine with nghttp and Firefox, but didn't do extensive testing yet. I've also had the same compression bug :P
Add a comment...

Tatsuhiro Tsujikawa

Shared publicly  - 
 
In C++11, suppose following class A:

struct A {
  A() {};
  A(const A& a)
  {
    std::cout << "copy ctor" << std::endl;
  }
  A(A&& a)
  {
    std::cout << "move ctor" << std::endl;
  }
};

Then,

  auto v2 = std::vector<std::pair<int, A>>{};
  v2.push_back({1,A()});

outputs:

move ctor
move ctor

While,

  auto v = std::map<int, A>{};
  v.insert({1, A()});

outputs:

move ctor
copy ctor

So why copy is invoked on map::insert case?
If we change the code to

  auto v = std::map<int, A>{};
  v.insert(std::make_pair(1, A()));

or

  auto v = std::map<int, A>{};
  v.insert(std::pair<int, A>{1, A()});

then they output:

move ctor
move ctor

Used  compiler: clang-3.2 and gcc-4.7.3
1
Add a comment...

Tatsuhiro Tsujikawa

Shared publicly  - 
 
http://aria2.sourceforge.net/manual/en/html/libaria2.html

The latest release of aria2 finally includes the libaria2 C++ library API preview. The aim is to provide almost all the functionality of aria2c to the application code. There are several limitations of this library at the moment, the biggest one is only one Session object per process. This is because the aria2 code base uses static variables in many places (and we have not enough time to fix them all). 
Other than that, the library takes care of all networking and download stuff, the usage of the library is very straight forward. You can see the tutorial code in the above link using libaria2. With relatively small lines of code, your application can download HTTP(S)/FTP/BitTorrent with DHT. Of course, it works with Metalink document as well.
1
Add a comment...

Tatsuhiro Tsujikawa

Shared publicly  - 
1
Fool Saint's profile photoTatsuhiro Tsujikawa's profile photoJack loner's profile photo
4 comments
 
hi,
i'm an aria2 user from China,aria2 is best download software i think.
My computer OS is Ubuntu12.04LTS.and,i use aria2 with uGet.but,problem come.
uGet can't connect aria2,

argument:able-rpc=true
--rpc-allow-origin-all=true--rpc-listen-all=true(i copy form internet)

then,i google .
but,question don't change.
question one:
how to build aira2.conf?
question two:
how to build aira2.session?
please,help me,i spend a week on it.
thinks.
Add a comment...
In his circles
26 people
Have him in circles
68 people
Kenneth Ho's profile photo

Tatsuhiro Tsujikawa

Shared publicly  - 
 
 
So if like me
- you happen to have bad sectors on your disk
- those bad sectors propagated in the middle of the MBR
- this is a 1TB HD with 10 years of photography
- the HD won't even show up after fdisk -l

It looks like you're screwed, but ... hold on.

0. Prerequisites
- Read all the post first and all the links.
- Don't try to repair your disk with e2fsck, if it has bad sectors you will destroy data even more.
- All the backup data you extract should be stored on a reliable disk (FS with data integrity and redundancy).

1. First of all, do a full copy of the disk with gnu ddrescue
yes you can backup a disk that don't even show up in fdisk
Recover a data disk with ddrescue

- Make sure to get the most out of your disk before it dies completely, I used retries = 3. A single byte in the partition table can make the difference later on.
- Keeping the disk cool actually helps, I took a reusable ice pack I put on the disk ( with a rag to prevent condensation )

2. Do a backup of this file in case you have to modify the image to repair the filesystem. You don't want to regrab that file again, the disk might be dead by that time.

3. Create a virtual device with this image
> losetup -v -f /path/to/image.img

This will probably create /dev/loop0.

If you're lucky you'll be able to mount the partition of your choice with mount, if so go to 5.
> mount -t ext4 /dev/loop0p1 /backup/recovered_root

If like me your partition table is so defunct only loop0 exists (no loop0p0, loop0p1, ...), there are several methods to find at which sector on the disk (ie. within the image) the partition begins.
I used the following line
> fdisk -l /path/to/image.img
(yes you can actually use fdisk directly on an image)

People also report you can get these by simulating the creation of the original partition ( e2fsck with the -n flag) on the defunct disk - although this is more risky.

When done with the partition geometry you can actually instruct losetup to do is magic and create a virtual device with the partition
> losetup -v -f --offset 32256 --sizelimit 250048495616 /path/to/image.img

offset = partition_offset_in_sectors * sector_size
sizelimit = partition_length_in_sectors * sector_size

(usually first partition is at sector 63 and sector size is 512B)
Then you can mount the device (=partition)
> sudo mount -v /dev/loop0 /backup/recovered_root

4. Repair the filesystem
Chances are that mount will complain about a corrupted filesystem when mounting, of course bad sectors are pure evil. So you can try to repair the filesystem in the image, I used e2fsck since it was ext4
> e2fsck -y -v -f /dev/loop0
y : answer yes automatically
v : verbose
f : force checking

Note: for e2fsck to work I had to remove the sizelimit parameter, it would complain ext4 and disk geometry disagrees on partition size. I guess ext4 was right because fdisk actually reported me bad data : 250048495616 sectors is 232GB, but it supposed to be a 1TB partition.

After that I had a bunch of dead files but nothing too important ( I had only 1.3MB of bad sectors on a 1TB HD after all )

5. Copy your precious files to a safer place : )

If anything with restoring the filesystem failed you can still use data carving programs like foremost or PhotoRec or try even harder to repair the fs with TestDisk or exundelete. You can try different techniques since you're working on a copy of the image !

Conclusion
This is a rather lengthy post but I hope it will be useful to other.
Since then I had a real strong look at the filesystems with builtin data integrity support such as ZFS and BTRFS. It's been an interesting journey with a happy end : )

I'm currently using BTRFS with zlib compression and mirroring on two disks (RAID1 for data and metadata). 'til then it's been a real joy to use : ) Keeping my fingers crossed !

More on filesystem with data integrity support
http://en.wikipedia.org/wiki/ZFS
http://en.wikipedia.org/wiki/Btrfs
http://arstechnica.com/information-technology/2014/01/bitrot-and-atomic-cows-inside-next-gen-filesystems/
https://blogs.oracle.com/elowe/entry/zfs_saves_the_day_ta old but cool

Here a few links which helped me in my quest to rescue Princess Data
https://wiki.archlinux.org/index.php/File_Recovery
https://help.ubuntu.com/community/DataRecovery
http://ubuntuforums.org/showthread.php?t=1192598
http://askubuntu.com/questions/48717/how-to-manually-fix-a-partition-table
https://www.suse.com/communities/conversations/accessing-file-systems-disk-block-image-files/
1
Add a comment...

Tatsuhiro Tsujikawa

Shared publicly  - 
 
Building a program using different toolchain and/or on different platform may discover a bug which is a bit hard to find out without checking code carefully.

For nghttp2 project, I uses Debian and gcc toolchain. Valgrind is also used to check simple memory corruptions in easy way. We had no indication of bugs with them, until then..
I found a buffer overrun bug on freebsd 9.1 with gcc. Another bug, which was out-of-bound read, was found on Debian with clang. Both bugs were in the test code. Maybe it is a good practice to automate to build a program with difference set of toolchains even for the relatively small open source projects.
1
Add a comment...

Tatsuhiro Tsujikawa

Shared publicly  - 
 
I spent sometime to implement HTTP2.0 draft-04 based on spdylay codebase. It is not yet good enough for interop, but the basic block is working fine now. The header compression is also implemented. It is very funny to see header size is 0 if you send the same request/response twice.. Anyway, the code is here: https://github.com/tatsuhiro-t/nghttp2

The missing parts are PRIORITY frame support, client connection header and server-push thingy. Well, they will be added if I have time and energy.

spdycat and spdyd are revised to work with HTTP/2.0 protocol.
These names are now odd for HTTP/2.0 for now. I have a plan to rewrite them and give them proper names. Here is the output from spdycat:

    $ src/spdycat https://localhost:3000/COPYING http://localhost:3000/AUTHORS  -nv --no-tls
    [  0.000] Handshake complete
    [  0.000] recv SETTINGS frame <length=8, flags=0, stream_id=0>
              (niv=1)
              [4:100]
    [  0.000] send HEADERS frame <length=72, flags=5, stream_id=1>
              ; END_STREAM | END_HEADERS
              ; Open new stream
              :host: localhost:3000
              :method: GET
              :path: /COPYING
              :scheme: https
              accept: /
              accept-encoding: gzip, deflate
              user-agent: nghttp2/0.1.0-DEV
    [  0.000] send HEADERS frame <length=14, flags=5, stream_id=3>
              ; END_STREAM | END_HEADERS
              ; Open new stream
              :host: localhost:3000
              :method: GET
              :path: /AUTHORS
              :scheme: http
              accept: /
              accept-encoding: gzip, deflate
              user-agent: nghttp2/0.1.0-DEV
    [  0.001] recv HEADERS frame <length=121, flags=4, stream_id=1>
              ; END_HEADERS
              ; First response header
              :status: 200 OK
              cache-control: max-age=3600
              content-length: 1080
              date: Fri, 19 Jul 2013 17:02:21 GMT
              last-modified: Fri, 12 Jul 2013 14:55:22 GMT
              server: spdyd nghttp2/0.1.0-DEV
    [  0.001] recv DATA frame (length=1080, flags=0, stream_id=1)
    [  0.001] recv DATA frame (length=0, flags=1, stream_id=1)
    [  0.001] recv HEADERS frame <length=6, flags=4, stream_id=3>
              ; END_HEADERS
              ; First response header
              :status: 200 OK
              cache-control: max-age=3600
              content-length: 66
              date: Fri, 19 Jul 2013 17:02:21 GMT
              last-modified: Fri, 12 Jul 2013 14:55:22 GMT
              server: spdyd nghttp2/0.1.0-DEV
    [  0.001] recv DATA frame (length=66, flags=0, stream_id=3)
    [  0.001] recv DATA frame (length=0, flags=1, stream_id=3)
    [  0.001] send GOAWAY frame <length=8, flags=0, stream_id=0>
              (last_stream_id=0, error_code=NO_ERROR(0), opaque_data=)

spdycat requested 2 resources in one connection (that's first 2 HEADERS frame). And the server, spdyd, returns the 2 responses (they are also HEADERS, the last 2).
4
4
Tatsuhiro Tsujikawa's profile photoSebastien WILLEMIJNS's profile photoPierre Joye's profile photoDaniel Stenberg's profile photo
4 comments
 
Yes, it is removal, actually.. just use previous header sets. But this compression ratio could not be achieved by gzip anyway, so it is truly awesome.
Add a comment...

Tatsuhiro Tsujikawa

Shared publicly  - 
1
Add a comment...
People
In his circles
26 people
Have him in circles
68 people
Kenneth Ho's profile photo
Work
Occupation
Software Engineer
Employment
  • Hitachi GP
    Software Engineer, 2001 - present
Places
Map of the places this user has livedMap of the places this user has livedMap of the places this user has lived
Previously
Kanagawa, Japan
Links
Contributor to
Story
Tagline
Car lover, free software developer.
Introduction
A free software developer living in Japan. https://github.com/tatsuhiro-t
Education
  • Osaka University
    Computer Science, 1996 - 2001
Basic Information
Gender
Male