dev on rails – hyperco.de blog
the official blog of hyperco.de web services and its cio
ruby performance marshalling JSON YAML
In: rails tricks
10 Aug 2011While we are using memcached to keep a lot of things available for our app, and now we switched to Rails 3 and for memcached we are using the faster Dalli Client, I was kind of interested about the marshalling performance of different methods.
For this performance test I have made a copy of a quite big object from one of our production apps. If this object is converted plain to YAML it takes about of 3.7 MB, and in byte stream ruby marshalling about 1.8 MB. It consists mainly of big hashes, whereas the hashes hold strings as keys and integers as values. Summed all values together in that object that might be around 200k objects (basically short strings and DWORD integers)
Using ruby 1.8.7 converting that object to YAML is quite a pain. Not using only over 200 MB of RAM after calling the .to_yaml method but also taking 25.6 seconds to fullfill that request on a small Debian 6 system.
The .to_json performs an edge faster with 3.8 seconds and using only about ~ 30 MB of RAM usage for the ruby process.
Much better performance showed the built-in marshalling method. The ruby built-in Marshal.dump resp. Marshal.load methods did not show a significant difference. Dumping and loading take each approximately 90 ms.
Finally retrieving that object from memcached, whose sub objects are kept in 5 chunks to not excess the 1 MB limit takes about 40 ms. And in this time not only the bytestream is being converted to an object, but also the whole data is transmitted via TCP sockets.
| time | stream length | performance | |
| YAML* | 25.6 s | 3.7 MB | 0.351 % |
| JSON | 3.8 s | 2.8 MB | 2.36 % |
| marshal | 0.09 s | 1.8 MB | 100% |
| memcached Dalli.get | 0.04 s | n.a. | 225 % |
*excessive RAM usage when streaming to YAML.
Findings
We can conclude ruby marshalling in this scenario is about 40 times faster than converting the object to JSON and almost 300 times faster than YAML. All measured with ruby 1.8.7. But the crown goes to memcached and its Dalli gem, delivering even more than 2 times of rubys built-in marshalling performance.
Finally I hope there is going to be some major improvements to the built-in JSON and YAML performance to come with future ruby and rails versions. Enough yaml libraries out there ...
Internet Explorer 9 beta released
In: random stuff
8 Oct 2010Styling web sites for different browsers has always been a hassle. Naturally web standards are some more or less common to one or another browser. Internet Explorer has always been one of the major browsers worldwide, but had its very own interpretation of "web standards". So when a new realease of the IE hits the beta state I always was very curious about how much less work I will have to do when styling future websites while adapting them to the extraordinary behaviour of microsofts browser.
Internet Explorer 7 had after several years of waiting PNG's with alpha channel supported. Internet Explorer 9, which is now out for beta comes with rounded corners (CSS3 border-radius), currently passes 95% of the ACID3-Test and utilizes Direct2D which gives very smooth browsing experience on websites were you zoom and drag a lot like bing maps or google maps.
Interestingly due to the new Direct2D implementation, google maps is even faster in IE 9 as in Google's Chrome 5 itself, which is still my favourite browser - but at least microsoft did a big step towards the future. Let's hope that the little style bugs disappear before the final release.
Links
hyperco.de NoSQL cms online now!
In: random stuff
7 Sep 2010We are proud to announce that our new CMS is online now. It is a self linking, smart navigating, and easy to administer content management system, based fully on open source software, Rails 2.3.81, Memcached2 and CouchDB3. It makes heavy use of json, and jQuery4, relies on ajax technologies and runs on all major browsers.
Till 2011 our new CMS will be exclusively available for asiacomfort.com and is being developed steadily.
- Ruby on Rails - the agile and fast web framework
- memcached - unix native, distributed, high performance, memory object caching system
- CouchDB - document oriented, distributed and scalable database system beyond SQL
- jQuery - the all in one javascript and ajax framework
rails: fast mysql table export to csv
In: random stuff
14 Dec 2009
Sometimes it is necessary to have a function for easily downloading a SQL table exported to a plain csv file for further external processing or manual data lookup. Normally people in forums on the net recommend to use phpmyadmin for tasks like this, but on a real production server phpmyadmin is of course no option security wise.
Another approach often recommended is to use FasterCSV to build the csv-file. FasterCSV is a nice tool when you want to freely create custom csv files, but performance wise this approach takes some time if you are really exporting large tables (>10000 rows).
Much faster is to just use MySQL's built-in method for exporting a table directly into csv format.
Just put this into the model you want to export as csv.
# model
# function returns table as csv string
def self.to_csv
db_name = self.configurations[Rails.env]['database']
# pay attention to the backticks ``
return `echo "use #{db_name};
select * from #{self.table_name};"
| mysql -u monitor_user --password="1234"`
end
Ensure that you have a seperate mysql user with only select priviliges on your database to get this working safely - in this example the user name is 'monitor_user' with password '1234'.
Of course if you are planning to implement this functionality in several models, you could put this into a module into ./lib and just include the module in each desired model class.
To provide the generated csv string from the model for download, you could use some code like this in your controller.
# controller
def download_csv
send_data(MyModel.to_csv, {
:type => 'text/csv',
:filename => "my_model_#{Date.today.to_s}.csv" } )
end
Beneficial over the phpmyadmin solution,
you define the security rules directly in your application,
no need to rely on mysql priviliges and
most important, you don't expose a common interface to the database server which is permanently under fire from automated attack bots1.
- for those of you who are using phpmyadmin just a few words ... even we don't have it running on our server - we are getting more than 1000 requests per day just on one domain, trying to gain access to several possible phpmyadmin paths and underlying scripts - and as I already said, it is not there, all those bots fail - so imagine for a second, how those attack bots would try to penetrate your phpmyadmin installation, once they found it
display upgrade S-PVA, E-IPS vs TN
In: random stuff
25 Nov 2009
For 2-odd years now I'm very lucky with my Eizo S2411W. Most of the time I'm surfing the web, writing code or do photoshopping. Usually at the end of the year you try to spend some money, and as I feel that displays have become real keen now (as opposed to RAM, which where half the price a year ago) I decided to get me a smaller additional pivot display, especially nice for reading or writing large text and code.
Samsung, Eizo, Dell
I ordered a Samsung SyncMaster 2243BW, as this display was one of the lowest priced models for around EUR 180 with 22 inch and pivot stand. Shortly after the arrival and setup of the display I noticed that these so popular TN-Panels had much worse viewing angle properties than I was expecting. Of course I knew about the properties of TN panels, that's why I was buying the Eizo 2 years ago having a S-PVA panel, offering the best color reproduction, highest font detail, constant color and contrast over the full viewing angle compared to any other LCD technology. My disappointment over the extraordinary nasty color shifts occurring on little viewing angle changes leaded me to the decision to get another display for comparison. I have heard IPS technology made some advantages in the past, and as you can get displays with this type of panel for little extra money compared to TN panels I decided to give them a try. The second display I ordered was a Dell 2209WA having an E-IPS panel as this was nearly the lowest price 22 inch model with pivot stand and E-IPS panel for around EUR 250.
Comparison
Dell
|
Samsung
|
On the following figure you can see the difference in color and contrast reproduction on a tilted viewing angle. On the left, the Samsung, its fairly recognizable that light tones are shifted dramatically to green, red, grey, and even white is shifted to blue, wheres blue is shifted to green. And the more darker mid-tones become very vibrant, loosing a lot of tonal detail. Depending on the viewing angle this changes all the time. So if you are looking for wicked spectral effects this display might be interesting for you, although I prefer the reproduction quality can be seen on the second photograph - the IPS panel of the Dell display reveals nearly no color shifts even on highly tilted viewing angles. One downside even for Dells IPS' - the black parts get clearly brighter on larger angles, but this effect is at least in daylight situation negligible.
Both shots are taken and processed with totally equal settings.
Enable Wordpress login with yourblog.de/login
In: random stuff
2 Nov 2009
I thought it would be cool to not have the meta-widget on my wordpress blog enabled as there is probably little need for visitors to have a login-link visible on the start page.
If you are used to simple naming conventions from rails you expect to have the login in my case on http://blog.hyperco.de/login - but the wordpress login page sits in wp-login.php - that is wordpress' general design rule. All of the pages in the wordpress root start with 'wp-' and end with '.php' - whether you like it, or not.
So I thought, wouldn't it be nice to have apache redirect all requests to files not found on the disk to the wordpress style named files?
I assume you are using apache web server and you have the RewriteEngine aka mod_rewrite enabled, then this will do the trick:
RewriteEngine on
RewriteCond %{LA-U:REQUEST_FILENAME} !-f
RewriteCond %{LA-U:REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_URI} !^(.*)\.php$
RewriteRule ^/(.*)$ /wp-$1.php [R=307]
This peace of code you can easily be pasted into your vhost config or .htaccess file or whatever else config file you are using.
- in line one we enable the RewriteEngine
- apply rule only if no file on disk with this name exists
- apply rule only if no directory on disk with this name exists
- apply rule only if REQUEST_URI does NOT end on .php
- rewrite (redirect) by regular expression takes place in the last line after all prior conditions are met, so in our example redirect /login to /wp-login.php
Explanation - line by line
- No Comments
- Tags: apache, apache2, mod_rewrite, php, rewrite, rewrite_cond, rewrite_rule, wordpress
Simple inline code in your rails templates
In: frontend
26 Oct 2009when I write views and templates sometimes it occurs that I want to use some code like
<% if c = @customer[:image] %>;
<img src='...' />;
<% end %>
but obviously this code would look much more nice with short tags enabled, so I could use code like
% if c = @customer[:image]
<img src='...' />
% endbut poorely, in a standard rails env this trim mode for ERb is not enabled by default.
For Rails 2.1.2, I've made it working by
# environment.rb:
# ALLOW open ruby code lines inside templates
ActionView::Base.erb_trim_mode = '-%'
At least for Rails 2.3.4 this does not works as a class is frozen error is raised. I'll post an update here, when I found out a simple solution to get this working in rails 2.3.x without the need to install the trim mode plugin.
hyperco.de blog
you are watching the official blog of hyperco.de web services
Photostream
Categories
- rails tricks (2)
- frontend (1)
- random stuff (5)
- No comments