Comments for Eric Day - Blog http://oddments.org Thoughts, code, and other oddments. Sat, 04 Sep 2010 11:14:17 +0000 hourly 1 http://wordpress.org/?v=3.0 Comment on C vs C++ by C versus C++ « Made with Tea http://oddments.org/?p=79&cpage=1#comment-2156 C versus C++ « Made with Tea Sat, 04 Sep 2010 11:14:17 +0000 http://www.oddments.org/?p=79#comment-2156 [...] just read an article on C versus C++, you can check it out here. It’s an overview of his programming experiences with C and C++. On many aspects of the [...] [...] just read an article on C versus C++, you can check it out here. It’s an overview of his programming experiences with C and C++. On many aspects of the [...]

]]> Comment on Scale Stack vs node.js vs Twisted vs Eventlet by Eric Day http://oddments.org/?p=494&cpage=1#comment-2155 Eric Day Thu, 02 Sep 2010 05:37:31 +0000 http://oddments.org/?p=494#comment-2155 It turns out the default socket buffer sizes in Erlang are only 1500 bytes (MTU size). After putting these up to 16k like in the other servers, Erlang came in at 61.65 seconds for the 32k connection/512k payload test, just behind the C++ server and right there with Go. Memory consumption with the Erlang server was also fairly low (peak around 400M, usually around 150M). It turns out the default socket buffer sizes in Erlang are only 1500 bytes (MTU size). After putting these up to 16k like in the other servers, Erlang came in at 61.65 seconds for the 32k connection/512k payload test, just behind the C++ server and right there with Go. Memory consumption with the Erlang server was also fairly low (peak around 400M, usually around 150M).

]]> Comment on libdrizzle + SQLite hack by lee http://oddments.org/?p=40&cpage=1#comment-2132 lee Tue, 17 Aug 2010 22:52:26 +0000 http://www.oddments.org/?p=40#comment-2132 Is this such a bad idea after all? What would be required to make it useful for both MySQL & SQLite developers? Perhaps with some irony Oracle has gone & tacked SQLite onto BerkeleyDB w/ a drop-in replacement lib for API calls. In the interim of MySQL dropping BDB support, I think BDB has gained row-level (range-offset) locking of its tables. Seems to me this'd be an interesting project, the upshot being that terabytes (maybe petabytes...?) of very fast hash can be accessed via SQL. There are a handful of SQLite Server projects but I haven't eval'd those yet. What would libdrizzle bring to the table? What would it take to turn this into a project, say a libdrizzle-sqlite-bdb data stack? Maybe put it on a ceph-btrfs-drbd stack & you've got a monster. Or worse yet, a Kerrighed or OpenSSI cluster ='s a linux equiv. of a sysplex... ;-) ? Is this such a bad idea after all?

What would be required to make it useful for both MySQL & SQLite developers? Perhaps with some irony Oracle has gone & tacked SQLite onto BerkeleyDB w/ a drop-in replacement lib for API calls. In the interim of MySQL dropping BDB support, I think BDB has gained row-level (range-offset) locking of its tables.

Seems to me this’d be an interesting project, the upshot being that terabytes (maybe petabytes…?) of very fast hash can be accessed via SQL. There are a handful of SQLite Server projects but I haven’t eval’d those yet.

What would libdrizzle bring to the table? What would it take to turn this into a project, say a libdrizzle-sqlite-bdb data stack?

Maybe put it on a ceph-btrfs-drbd stack & you’ve got a monster. Or worse yet, a Kerrighed or OpenSSI cluster =’s a linux equiv. of a sysplex… ;-)

?

]]> Comment on Scale Stack vs node.js vs Twisted vs Eventlet by Eric Day http://oddments.org/?p=494&cpage=1#comment-2116 Eric Day Fri, 06 Aug 2010 07:51:14 +0000 http://oddments.org/?p=494#comment-2116 I decided to run a few more versions for just the 32k connection, 512k data test. Below are the repeated times for the original four, plus Erlang, regular Python threads, and the two versions of the Go server that were left in comments. Scale Stack 50.25 node.js 117.04 Twisted 89.46 Eventlet 77.80 Erlang 108.94 Python threads 111.04 (lots of memory even with minimal stack size) Go v1 62.95 Go v2 59.73 The Go version is very impressive, almost as fast as the C++ version. Of course these last four you get SMP without any extra work, which is a bonus. I decided to run a few more versions for just the 32k connection, 512k data test. Below are the repeated times for the original four, plus Erlang, regular Python threads, and the two versions of the Go server that were left in comments.

Scale Stack 50.25
node.js 117.04
Twisted 89.46
Eventlet 77.80
Erlang 108.94
Python threads 111.04 (lots of memory even with minimal stack size)
Go v1 62.95
Go v2 59.73

The Go version is very impressive, almost as fast as the C++ version. Of course these last four you get SMP without any extra work, which is a bonus.

]]> Comment on Scale Stack vs node.js vs Twisted vs Eventlet by Stephen Major http://oddments.org/?p=494&cpage=1#comment-2114 Stephen Major Thu, 05 Aug 2010 22:30:21 +0000 http://oddments.org/?p=494#comment-2114 I have to second the others who have mentioned the Google go lang, it compiles and is powerful like C++ but is a breeze to develop like python. Could you please perform a benchmark against GO pretty pretty please, I am really interested as my core CP for the cloud is being designed in GO and it would be sweetness if it turned out to be hand in hand. I have to second the others who have mentioned the Google go lang, it compiles and is powerful like C++ but is a breeze to develop like python.

Could you please perform a benchmark against GO pretty pretty please, I am really interested as my core CP for the cloud is being designed in GO and it would be sweetness if it turned out to be hand in hand.

]]> Comment on Scale Stack vs node.js vs Twisted vs Eventlet by Paul Nakata http://oddments.org/?p=494&cpage=1#comment-2113 Paul Nakata Thu, 05 Aug 2010 18:53:06 +0000 http://oddments.org/?p=494#comment-2113 Err, that link should be: http://github.com/fiorix/cyclone Err, that link should be: http://github.com/fiorix/cyclone

]]> Comment on Scale Stack vs node.js vs Twisted vs Eventlet by Paul Nakata http://oddments.org/?p=494&cpage=1#comment-2112 Paul Nakata Thu, 05 Aug 2010 18:50:55 +0000 http://oddments.org/?p=494#comment-2112 I'm a little late to the party, but have you checked out Cyclone (http://github.com/fiorix/cyclone)? It's Tornado ported to run on top of Twisted. If you already have a codebase that's built on Tornado it could give you the best of both worlds. I’m a little late to the party, but have you checked out Cyclone (http://github.com/fiorix/cyclone)?

It’s Tornado ported to run on top of Twisted. If you already have a codebase that’s built on Tornado it could give you the best of both worlds.

]]> Comment on Scale Stack vs node.js vs Twisted vs Eventlet by Uriel Katz http://oddments.org/?p=494&cpage=1#comment-2109 Uriel Katz Wed, 04 Aug 2010 14:24:35 +0000 http://oddments.org/?p=494#comment-2109 Gevent is faster than eventlet due to the use of non-python core,while eventlet uses a python hub. also if you don`t need to do streaming there is the libevent http server which is really really fast! I have created a proxy for Rackspace cloud files that does authentication and stream the result to the client. On my computer it handles 500 concurrent downloads of a 700MB using 100% of one core and 45MB of memory all using gevent :) Gevent is faster than eventlet due to the use of non-python core,while eventlet uses a python hub.
also if you don`t need to do streaming there is the libevent http server which is really really fast!
I have created a proxy for Rackspace cloud files that does authentication and stream the result to the client.
On my computer it handles 500 concurrent downloads of a 700MB using 100% of one core and 45MB of memory all using gevent :)

]]> Comment on Scale Stack vs node.js vs Twisted vs Eventlet by Denis http://oddments.org/?p=494&cpage=1#comment-2086 Denis Sat, 31 Jul 2010 04:54:23 +0000 http://oddments.org/?p=494#comment-2086 I'd also like to see gevent included in the tests :o) Regarding Ryan's misunderstanding of what exactly is more clear about gevent's API - one aspect is duplicating functionality. For example, in eventlet there's eventlet.green.threading.Event and eventlet.event.Event. Both classes have the same purpose but different interfaces. In gevent there's just gevent.event.Event which has the interface of threading.Event. Here's more info on the differences between the two libraries: http://blog.gevent.org/2010/02/27/why-gevent/ Also, eventlet does not truly support libevent. There's pyevent-based hub but a few quite important parts of eventlet become broken when it's selected (e.g. thread pool). I'm pretty sure everyone who wanted to use libevent with eventlet is already using gevent. I’d also like to see gevent included in the tests :o)

Regarding Ryan’s misunderstanding of what exactly is more clear about gevent’s API – one aspect is duplicating functionality. For example, in eventlet there’s eventlet.green.threading.Event and eventlet.event.Event. Both classes have the same purpose but different interfaces. In gevent there’s just gevent.event.Event which has the interface of threading.Event.

Here’s more info on the differences between the two libraries: http://blog.gevent.org/2010/02/27/why-gevent/

Also, eventlet does not truly support libevent. There’s pyevent-based hub but a few quite important parts of eventlet become broken when it’s selected (e.g. thread pool). I’m pretty sure everyone who wanted to use libevent with eventlet is already using gevent.

]]> Comment on Scale Stack vs node.js vs Twisted vs Eventlet by Ryan Williams http://oddments.org/?p=494&cpage=1#comment-2082 Ryan Williams Fri, 30 Jul 2010 16:56:16 +0000 http://oddments.org/?p=494#comment-2082 Not really sure what you mean, Keith. Eventlet replicates the multithreading interfaces exactly. We had a little session at pycon where a bunch of folks brought in their multithreaded apps and ran them on Eventlet without any code changes. :) You can even mix real threads and greenthreads, if you must, like Spawning does. Not really sure what you mean, Keith. Eventlet replicates the multithreading interfaces exactly. We had a little session at pycon where a bunch of folks brought in their multithreaded apps and ran them on Eventlet without any code changes. :) You can even mix real threads and greenthreads, if you must, like Spawning does.

]]>