In this post I’d like to describe my findings and motivations.
To support subdomains during development is frustrating. By default you can’t have some subdomain.localhost to just bind to localhost.
So you have to edit your /etc/hosts
file to add some fake subdomain and when you have to test another subdomain, you’d had to edit the file again and again and… well you get my point.
I’ve stubbled on the magic domain http://lvh.me which redirects all request (including subdomains) to 127.0.0.1
. This means http://some-subdomain.lvh.me:3000 redirects to 127.0.0.1
on port 3000
and let request.subdomain
to return some-subdomain
within your Rails controller.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
|
This week I started on a side project, requiring subdomain support. I remembered that I could use http://lvh.me but when I browsed to the url it just said ERR_CONNECTION_REFUSED
After some time I remembered Rails binds the development
environment by default to localhost
which is != 127.0.0.1
where http://lvh.me is redirecting to..
So after fixing this, I remembered to have some code, that checks if you are browsing to http://localhost:3000 to redirect to http://lvh.me:3000. Just to help remember and enforce request for subdomains to go through http://lvh.me
As many developers, I don’t like to repeat myself, and decided to make the above two steps possible within a gem.
The easy part was creating a module that adds a before_filter
to the controller where the module LocalSubdomain
is included. The action would check if the request is from lvh.me
and if not, redirect to it. Nothing special.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
|
This part was kind of tricky because we have to inject our code before the boot
of Rails is triggered, to enforce the binding address localhost
to be changed to 0.0.0.0
.
After some reading about the rails initialization process and experimenting I noticed that the gems are required (and if so) executed before the Rack::Handler
is being called, which will trigger a Ruby server to be executed (WEBrick by default).
While looking at the rack/handler.rb I noticed that the method self.default
returns a Ruby server handler. Based on which Ruby server is being used, this will return a different handler.
After inspecting WEBrick’s and Puma’s rack handlers, I noticed, they both have the self.run
method with the options
argument, containing the Host
which indicates the binding address.
Here’s a stipped part of the code I used in the gem, showing how to intercept the used Ruby server (Puma, WEBrick, …) handler and extend that handler, with a custom run
method to be able to modify the options
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 |
|
After finishing these two pieces, I’ve bundled them into a Gem called local-subdomain, so I don’t have to add the before_filter
and add -b 0.0.0.0
to my rails s
command.
However I’ve ended up with a failing Google Chrome browser which I use like always!
It crashes directly after launching a page! Arghh!
After some googling I stumbled upon a solution where you have to set “Show scroll bars” to “Always”. This seemed to work at first, but still it seems to fail at some sites. However it’s beter than crashing directly.
Goto Settings -> General -> Show scroll bars -> Always
As mentioned this fix seems to fail some times. I’ve had some troubles viewing some pull request on github but most of the time it seems to work.
If anybody finds a beter way to fix this please leave a comment :)
]]>A while ago I tried several things getting a fast Rails experience on Windows, but it failed. It was just too slow … until now! In this post I’d like to share my current setup which I plan to use as long as it pleases me.
We are going to install Vagrant which uses VirtualBox to build a complete development environment. Also we will create and use some recipes for Chef to automatically install and configure the development environment. Think of installing Ruby, PostgreSQL, and MySQL etc. automatically.
First let’s setup our Windows machine with the needed software.
c:\Program Files\Oracle\VirtualBox
)vagrant box add precise64 http://files.vagrantup.com/precise64.box
Next we have to configure our project to create a box from the base box and make it install all our goodies we need to run our Rails application.
I’m using a gem called librarian-chef which actually is a bundler for your Chef recipes.
To install run the following from the root of you project:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
|
I know we are trying to create a non-Ruby Windows solution but we do need this gem to work on Windows to complete our setup with Chef. If you really don’t want Ruby being installed on Windows, you should do the above steps on UNIX or OSX and also have to commit the cookbooks. This means we’re back at 1900 where everybody stuffed there vendor folder with code they found… If you do want to perform this on Windows and haven’t already installed Ruby, you should do this now (I suggest RubyInstaller).
The command librarian-chef init
created the file chef/Cheffile
where we can add our cookbooks. Mine contains the following:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
|
As you can see I’ve setup Git, Ruby with rbenv, SQLite, MySQL, PostgreSQL and node.js for the asset pipeline. The apt
cookbook is for running apt-get dist-upgrade
and the cookbook database
will be used later to create users to access the databases.
You could skip one or two of the database servers if you like, but it can’t harm if you do install them all and don’t use them.
Note: the database
cookbook is provided with additional actions to set superuser privileges to PostgreSQL and MySQL.
Now that we have defined our cookbooks we need to run our final command so that Chef downloads them. Just as you would do with bundler we now run the following command from our chef/
folder
1
|
|
I had some difficulties with running bundle install
because the Vagrant user didn’t have permissions on the rbenv gem folder, so I had to add a custom recipe for this. Also I’ve added a recipe to modify the pg_hba.conf
file of PostgreSQL so that we can connect with the virtual machine. Finally I’ve created a file to add our database users with full permissions. Here are the 3 files I added:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 |
|
1 2 3 |
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
|
Next we have to create a role that defines what recipes should be executed in what order. Create the file chef/roles/rails-dev.rb
containing the following content:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 |
|
If you removed some cookbooks from the chef/Cheffile
you should also remove them here.
Next we need to create a Vagrantfile
that contains the configurations for creating a box from the base box. Create the config file by running from the root of you project:
1
|
|
Note: precise64
is the name we used with the vagrant box add
command from step 4 above.
Also we want to add the virtual machine folder to our .gitignore
file:
1
|
|
Here’s the stripped version of how my file looks like:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 |
|
Alright we’re ready. We now can boot up Vagrant which will take some time the first time because it will create a new box from the base box and install all the cookbooks. From then you’re able to connect with the box and use it just like you would do on UNIX or OSX.
1 2 3 4 5 6 7 8 9 10 11 12 |
|
A VirtualBox with Ubuntu precise 64 installed and containing the following software/configurations:
rails
with a empty passwordrails
with a empty passwordlocalhost:3000
goes to vagrant-box:3000
localhost:3306
goes to vagrant-box:3306
localhost:5432
goes to vagrant-box:5432
If you have to run specific Ruby (on Rails) tasks like rspec
or bundle exec rails g migration
you should do this within the ssh session of your box.
Editing files can be done on your Windows machine, because the root of you project is directly binded to the /vagrant/
folder on the box.
belongs_to
associate to an has_and_belongs_to_many
in a Rails project I’m currently working on. Not that hard you would say, but there were some catches for deploying/migrating these changes to the production environment.
In this post I’d like to explain how to achieve this without having to do multiple deployments of your model to maintain a consistent database schema and model.
Here you can find the final migration. Note that this example could also be solved with plain sql instead of using ActiveRecord, but there might be cases were you depend on the ActiveRecord associates. In these cases this is a great workaround
Let’s say we have a Article that belongs to a Category. Our model would look like something like this:
1 2 3 4 |
|
And the executed migration would look like this:
1 2 3 4 5 6 7 8 9 |
|
First let’s create a new table that can hold the associate between multiple Articles and Categories. The migration would look like this:
1 2 3 4 5 6 7 8 9 10 11 |
|
At this point all is fine and we can migrate the database without any problems. Only this doesn’t add the current category to the new collection of categories.
Here’s were the problem actually starts, because to move the category to the categories collection for the articles, we have to define the belongs_to
but also the has_and_belongs_to_many
associate on the Article
model.
I don’t like this approach because we’ll have to define both these associates and the belongs_to
should be removed in the next release. So how to deal with this?
The fix is quite simple. We remove the belongs_to
associate from the model and only define the new has_and_belongs_to_many
associate and within the migration we extend the model with the “old” belongs_to
associate so we can use this within the migration.
So our files will look like this
1 2 3 4 |
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 |
|
Because the Nano instance has some connection limitation you have to make some config changes so you won’t get ERR max number of clients reached
error messages.
Today I’ve been struggling a lot with getting the Sidekiq to work probably with Redis to Go Nano on Heroku. The main problem was I was having difficulties with the amount of connection’s being created to the Redis server. Because the Nano variant is free but large enough for handling normal sized queues of work, we have to face the limitation of 10 connections.
If you just want to know what you need to change/setup you can go directly to a small tool I’ve built to calculate the number of connections/concurrencies need for a number of workers/web workers etc.
The error Error fetching message: ERR max number of clients reached
, is quite clear isn’t it? The actual question is how we can reduce the connections being opened to match the 10 connection limit given by the Nano instance.
After some research I found the factors that you have to tweak in order to reach the magic number of 10.
The answer for all our problems could be described in the following sum:
max connections = (Heroku worker count * (concurrency + 2 reserved connections)) + (web dyno count * (client connection size * unicorn worker_process size))
If you have a small amount of Redis connections we need to make some modifications in order to get all our processes happy and not throwing connection errors around.
At the bottom of this post I’ll put my final configuration files.
The first thing I changed was the Redis connection size for the client. By default Sidekiq takes 5 connections per client. Because my applications only queries Redis for adding tasks to the Sidekiq queue, one connection should be more than enough.
One mistake I made was forgetting about the number of dynos and Unicorn worker_process size. In my app/config/unicorn.rb
I had worker_processes 3
defined which actually means 3 workers * dyno count = redis_connections taken by the Client.
In my case where I’ve set the connection size to one we could have the following examples
If you up the client size to, for example 3 you should also multiply the Redis connections by this number (so for example 3 you’ll have 20 * 3 = 60 connections).
When calculating the number of connections the Sidekiq server needs, we need to modify the number of currencies it initializes on launch. This number represents the number of threads created by the Sidekiq server which will perform queued tasks. Each concurrency/thread takes up 1 Redis connection.
When tweaking this number I found out the Sidekiq server took 2 additional connections upon the concurrency number. This seemed to be default behavior becase Sidekiq server uses these two for the Fetcher and Retrier commands.
The default concurrency size of Sidekiq is set to 25 which mean that without modifications you need at least 27 Redis connections for the Sidekiq server.
1 2 3 4 5 6 7 |
|
1 2 3 4 5 6 7 8 9 |
|
1
|
|
1 2 |
|
1
|
|
Our goal is to start a worker when a task is added to the queue and to destroy the worker after its done processing the queue tasks. This will result in a much lower bill at the end of the month because the worker doesn’t have to be up the whole month.
First let’s take a look at a default configuration of Sidekiq within a Rails project.
First we need to add Sidekiq to our Gemfile and run bundle install
1
|
|
The worker is very basic. It just performs some heavy task and sends an email after it finished.
1 2 3 4 5 6 7 8 9 |
|
In our Procfile we have to define the worker
line to start the Sidekiq server on a Heroku worker dyno.
1 2 |
|
On the controller we probably have a line calling object.generate_download
which takes to much time. We’ll change this line to execute the MyWorker
instead:
1 2 3 4 5 6 7 |
|
NOTE: It’s recommended not to add the whole object into the worker because this is stored in the Redis database. Also the object might be changed before it’s being processed by Sidekiq. Adding the id
and fetching the object in the worker is a better approach.
At this point we have configured our Rails project to add tasks to Redis and have Sidekiq perform executing the worker when a task is added. If we deploy our Rails app to Heroku we need to add the Redis To Go addon and add a Heroku worker dyno to our application.
To test this locally we need:
bundle exec rails server
)bundle exec sidekiq
)At this point the Heroku worker dyno has to be started, to pickup tasks from the queue. It’s likely you do not need this worker to run the whole day because the queue might be empty most of the time. Here’s where the autoscaler gem come’s in handy.
This gem acts as Middleware for Sidekiq and performs the following tasks:
First we need to add the autoscaler gem to our Gemfile and run bundle install
1
|
|
The gem requires two environment variables to be set on your Heroku application. The HEROKU_API_KEY
is required to perform creating and removing a Heroku worker dyno and the HEROKU_APP
to know on which application it has to create/destroy the worker dyno on.
1 2 3 |
|
Because this gem acts as Middleware we need to create a sidekiq.rb
in our initializers folder. This file will check if we are running on Heroku and if so, activates the autoscaler as Middleware.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 |
|
At this point we’re ready to deploy our application to Heroku and let the autscaler automatically create and destroy a worker dyno whenever it needs to process tasks from the Sidekiq queue.
]]>Travis CI is a hosted continuous integration service for open source projects and Yeoman is a robust and opinionated set of tools, libraries, and a workflow that can help developers quickly build beautiful, compelling web apps.
When Yeoman was released I’ve tried to add Travis CI support to a test project. After setting everything up there seemed to be a problem with the update check being fired when running tests. This was annoying because Travis CI was waiting for input from the user.
So I’ve fixed the issue and created a pull request that was merged very quick thanks to Addy Osmani.
Anyway today I saw that there was a new version released which includes my fix and I like to share how you can make your Yeoman project be tested within Travis CI.
See this commit with all the changes you have to make to your existing Yeoman project in order to get Travis CI integration.
Or just browse my test project here: yeoman-travis-ci
First we have to add the yeoman package as a dependency to our package.json
file because we can’t install yeoman globally on Travis CI.
1 2 3 |
|
After adding the dependency and running npm install
you’ll see that yeoman is installed in the node_modules
folder of your project.
Travis CI runs the npm test
command after fetching the project and installing the dependencies. To make npm test run the Yeoman test task, we need to add the following line into our package.json
file.
1 2 3 |
|
I’ve added the --verbose
argument because I’d like to see detailed information when my tests fail, but this is an optional argument.
At this point you should be able to run npm test
and see Yeoman running your tests.
Finally we have to add a .travis.yml
file to let Travis CI know what language our project is and on which version(s) of NodeJS it should use to test our project against (note: Yeoman requires >= 0.8).
1 2 3 4 |
|
The final step is to add the Travis CI hook to you project in GitHub and push your project.
Happy coding!
]]>Of course you already know the stuff described below so go to the final query.
For retrieving almost any information about your database you can query the pg_class
see Postgres documentation.
relkind
is set to r
to query only ordinary table information.pg_namespace
table to filter out the pg_catalog
and information_schema
tablesAs result we’ll have on column called tableName
with all tables from the database
1 2 3 4 5 6 7 8 9 |
|
Adding the total number of rows was quite simple after reading the Postgres documentation. It seems that the pg_class
contains a column called reltuples
that holds the number of rows in the table.
So adding this column to the select seems to do the trick:
1 2 3 4 5 6 7 8 9 10 |
|
With this query it was easy to transform it into a SUM for all rowCount’s we retrieve. Here’s the final query to retrieve the total number of rows from all tables of a Postgres database
Because I’ve just finished the latest functionality of the plugin I wanted to add the project to Travis CI, just like I do with all my other projects (the ones with unit tests).
For the development of the plugin I used Grunt which is a task-based command line build tool for JavaScript projects. Starting a new project is as easy as running grunt init:jquery
and you have a entire generated jQuery plugin project.
By default this command configures a jQuery plugin, QUnit test suite, sample plugin, sample tests and some grunt tasks you can use. Running the grunt
command in your project folder will execute the default task meaning:
Really easy to get started coding instead of setting up your environment :)
Ok so at this point I want to add my project to Travis CI so that it tests my suite when pushing new branches/commits to Github. To do this we have to make a few changes.
Here’s the travis ci integration commit from my plugin with all the steps described below.
I’d like to keep the Grunt tasks organized so I register a new task called travis
below the default
task.
1 2 3 4 5 |
|
As you can see I only added lint
and qunit
because we don’t have to concatenate or minify a new build of our plugin. At this point you are able to run grunt travis
from the command line.
Now we have to add Grunt as a dependency to our package.json
file so that npm running on Travis CI knows it has to install Grunt. Mine dependencies block looks like this:
1 2 3 4 5 6 |
|
Travis CI runs npm test
after it fetched your project and installed the dependencies, so we need to add this task to the package.json
file.
1 2 3 |
|
I’ve added the --verbose
option, so we’ll see more output of what is going on.
Every Travis CI project needs to have a .travis.yml
file in the root of the project folder, so it know what platform and version it should use to build/test your project. Here’s the one I used:
1 2 3 4 5 |
|
Alright after these changes your project is ready to be continuously builded with Travis CI. But don’t forget to setup the Service Hook on Github!
Thanks to Ryan Seddon and Ariya Hidayat for noticing me that the before_script
section for running PhantomJS on the headless server on Travis CI isn’t needed. See this link
After viewing there product overview video I got excited and wanted to take this product for a spin.
First I needed a project for testing the performance on. At my work we developed a Rails application that needs to validate a permit request with rules, questions, filters, answers depending on a region etc. etc. This process of validating a whole permit request takes some time because it has to perform 6850 queries.
At this moment the application runs on a PostgreSQL database so lets see what the performance is at this point:
1 2 3 4 |
|
The average result is that it takes 14.312ms for 6850 queries on PostgreSQL.
After installing MemSQL and starting it running on port 3307, I created a new database added the existing data and changed some connection strings for the Rails app.
Because MemSQL is built on top of MySQL you can use all the MySQL client tools to perform operations on MemSQL.
The MemSQL documentation recommended using the memsql2 gem but to do this I had to install MySQL first on my machine.
After installing MySQL and configured the Rails application I got some strange error when starting the application.
1 2 3 4 |
|
After some googling I found out that on OSX you might need to symlink the dylib manually. NOTE Because I’ve installed a newer version of MySQL I have a libmysqlclient.18.dylib
instead of the required libmysqlclient.16.dylib
. Symlinking this file from libmysqlclient.18.dylib to libmysqlclient.16.dylib seemed to work for me.
1
|
|
At this point I am able to connect to the database and perform the benchmark, but I got the next error:
1
|
|
Apparently I’ve EXPLAIN queries on in my config and this isn’t supported for JOIN queries as MemSQL tells us.
To resolve this you just have to modify/add this line in your config/environment/development.rb
1
|
|
So after resolving all these issues I was able to run the benchmark, but the first run took 3 min and 05 seconds! First I thought that this can not be right, but I forgot the fact that it generate’s c code from the queries.
On the MemSQL server I got the following output:
1 2 3 4 5 6 |
|
As you can see, generating the compiled version for every query took about 7 seconds.
So below again the same table with results. The first line is the time with generating all the compiled queries:
1 2 3 4 |
|
The performance boost is quite huge! The average result is 6.635ms for 6850 queries if we are skipping the first result. This means it’s like 7.677ms faster!
I’ve also tried the performance when I restart the database server. I just wanted to know if it keeps the data in memory and would take longer to run after a restart.
The result:
1 2 3 4 |
|
The first time it took just a bit longer, but with an average of 7.044ms for 6850 queries this is still faster than running it on PostgreSQL!
Here’s a list with some pros and cons I noticed while experimenting with MemSQL:
plancache
folder which is 5.5GB. So performance comes with a price I think.The error Eclipse returned was
1 2 3 4 |
|
I installed Eclipse and the Android SDK tested the Android SDK by creating an AVD.
After running Eclipse I tried to add the https://dl-ssl.google.com/android/eclipse/
url to my Available Software Sites to install the Eclipse package. After selecting all the Developer Tools in the filtered list for this Software Site and hitting Next I received the error as I added above
After some googling I found a tip that I needed to add another Software Site to Eclipse.
After adding http://download.eclipse.org/releases/indigo
and retrying to install the Android Developer Tools all went well.
bundle install
on one of our projects.The following error was raised when I tried to install capybara-webkit
1 2 3 4 5 6 7 8 9 10 |
|
I first thought there were some problems with make
or cmake
because that’s what the error says. qmake: not found
, but that didn’t seem to be te real problem.
After some googlefoo I found out that I was missing the libqtwebkit-dev
package. So the problem in this case was easily resolved by running:
1
|
|
After a long search I wasn’t able to find a stable, well tested gem, I could use in our Rails project, so I developed the ideal-mollie gem. After I finishing this gem, there was only one problem.
When you want to test a iDeal transaction with the Mollie test bank, you MUST do some routing, so that Mollie can send you a request if the payment was paid or not, to your local machine from the internet. So I made a alternative Mollie test bank you can use for what ever programming language you use for making Mollie iDeal payments without the routing problems.
I’ve came op with a gem that runs a small Sinatra application that does exactly the same as the Mollie test bank, only it runs on your local machine, which means you can redirect to localhost
and more important, it can perform the payment check.
Here you can find the mollie-bank gem, and the source code
To use mollie-bank just install the gem by running
1
|
|
and then run it
1
|
|
At this point, you can go to http://localhost:4567/ and you’ll see an intro page.
Alright so now you only have to make sure that when you’re developing on your local machine, you don’t make the requests to Mollie but to your locally running Mollie Bank.
To do this you just have to change https://secure.mollie.nl
into http://localhost:4567
in you code.
At this point I’ve only wrote a sample implementation for Ruby on Rails on the Implement into existing modules Wiki page, but soon there will be more examples.
]]>So I started coding and managed to create a Rake task that will do all of the steps I described in the article in just a few seconds!
Today a found a project that does exactly the samething as the below rake task, but with support for more db providers. Heroku uses this project for retrieving and pushing you database to the heroku database server.
The project is called Taps and can be found here: http://adam.heroku.com/past/2009/2/11/taps_for_easy_database_transfers
tmp/dump.sql
To use the Rake task, you have to add 2 additional fields (ssh_user
and ssh_host
) to your database.yml
file. These fields are used to create an ssh connection for retrieving the PostgreSQL dump.
Here’s example of a modified config/database.yml
:
Put the following code into libs/tasks/database.rake
1
|
|
If you want to skip the step where the rake tasks get’s the dump using ssh, you have to copy the dump.sql into the tmp folder by yourself (note that the name must be dump.sql
)
After you copied it, you should execute the following commands, to generate the development database with the dumped sql.
1 2 |
|
You could generate a public/private key pair with RSA and append that key to the production server so you don’t have to enter the password over and over again to connect with ssh.
First check if you haven’t already generated a id_rsa
file in your $HOME/.shh
folder. If you already have a id_rsa
file continue with step 2.
Run the following command on you local machine and accept the defaults.
1
|
|
This command creates an RSA public/private key pair in your $HOME/.ssh
directory. The private key is ~/.ssh/id_rsa
and the public key is ~/.ssh/id_rsa.pub
Now you can copy the public key to the remote machine by executing the following command:
1
|
|
This command will ask you to enter the ssh password for the ssh user “root” for the hostname “productionserver.com”.
After you’ve enter the password (for the last time) you can create a ssh connection without entering the password by executing:
1
|
|
Today I’d like to share the steps I take when I need to convert a PostgreSQL database into a SQLite database.
Commonly I have to do this when a Ruby on Rails application is in production and I have to check some issues with the production data. In the production environment we usually use a PostgreSQL database and for developing I use a SQLite database, so we need some conversion.
ssh -C username@hostname.com pg_dump --data-only --inserts YOUR_DB_NAME > dump.sql
SET
SELECT pg_catalog.setval
BEGIN;
as first line and END;
as last linebundle exec rake db:migrate
1 2 3 |
|
So basically you can do the following 4 major steps to convert the PostgreSQL database into a SQLite database.
First we have to create a sql dump on the production server. I use the following command that results in a dump.sql
file in the current folder:
1
|
|
I use the --data-only
option, so it doesn’t generate the schema. Converting the pg_dump generate schema to a valid SQLite schema gave me a lot of difficulties so I chose to generate the schema with the rake db
task (we’ll discuss this in the next step).
After you created the dump, you have to download/transfer/mail/etc. that file so you have local access to it.
If you have ssh access, you can also run the following command, which will output the file directly on you local drive
1
|
|
There are a few manual find/replace and delete action’s you have to perform on the dump.sql
file by hand.
SET
statements at the topYou will see some statements at the top of the file like SET statement_timeout = 0;
and SET client_encoding = 'SQL_ASCII';
etc. Remove all of these lines that start with SET
, because SQLite doesn’t use these.
Under the SET
queries you’ll see some queries to set the correct sequence for auto incrementing the id’s. SQLite doesn’t keep these value’s in a catalog and must be removed to prevent errors.
Remove all the line’s that look like SELECT pg_catalog.setval('MY_OBJECT_id_seq', 10, true);
The pg_dump
generate’s true
and false
as value’s for the INSERT INTO
statements. If we want to import these to SQLite we have to replace these to ‘t’ and ‘f’.
1 2 3 4 |
|
The first time I imported the dump (that was 2 mb) it took like 12 minutes to complete! After some googling I found out that SQLite’s default behavior is putting each statement into a transaction, which seems to be the time waster (after the fix it toke 12 seconds).
To prevent this behavior you can run the script within 1 transaction by specifying BEGIN;
at the top of the dump.sql
and END;
at the end of the file.
So you would have:
1 2 3 |
|
So now we have fetched the production data from the PostgreSQL database, we need to recreate the development.sqlite3
database.
Make a backup and run the migration task
1 2 |
|
You must run the migration until the migrated version that is active on the production database. If not, you could have the situation where you have dropped a column and can’t import the dump because the data depends on that column.
Check the dump.sql
for the latest version number in the schema_migrations
table and migrate to that version.
For example for the version 20121701120000
you would do:
1
|
|
The final step is importing the dump file. To do this we have to execute the following command within a terminal:
1 2 3 |
|
As you can see we first remove the records from the schema_migrations
table, because these are also included in the dump.sql
. Of course you could also remove the lines from the file, but I prefer this way.
The .read
command just execute’s all the lines within the specified file.
And that’s it! You now have a stuffed development.sqlite3
database with all the production data out of the PostgreSQL database.
Today I’ve had some difficulties with a Rails migration, that took ages to complete. After some debugging I figured out that the issues was because of the before_save
and after_save
callbacks.
Of course we could remove the before and after save callbacks to speed up the process, but I don’t like the idea of releasing a model that misses these behavior. I mean they’re not there for no reason right?
After some googling I found a quick and easy way to disable the callbacks without modifying the model. Here’s an simple example of a model class and a migration the disable the callbacks only within the migration.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
|
In the above example I show only two rails callbacks. Here’s a list of the supported callbacks you also can disable with it.
:after_initialize
:after_find
:after_touch
:before_validation
:after_validation
:before_save
:around_save
:after_save
:before_create
:around_create
:after_create
:before_update
:around_update
:after_update
:before_destroy
:around_destroy
:after_destroy
:after_commit
:after_rollback
Hope these could help you out some day =)
]]>In the past we’ve made some decisions that we probably wouldn’t take again. We for example, we decided to store the gem files in the vendor folder and having the rails code included in our repository. Our goal then was, that releasing the project with Capistrano would be easier and faster, because it didn’t have to download all the gem’s.
Anyway at this point we don’t have the gem’s stored anymore in the vendor folder. Since tools like RVM and rbenv we store these within the ruby/rails version of that specific project.
Still we have the problem that the repository contains files etc. we want to remove to shrink the size of it.
First we have to track down what files are making our repository so large. I’ve found a perl script that scans your repository for files of a specific size. Here’s the code:
So I’ve placed this file within my repository and executed the below command to see all files, from all commits, that are bigger then 500 Kb
1
|
|
This give’s you an output like:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
|
As you can see the following folders contain data that we don’t need anymore and fills our repository with unused data:
To remove the folders specified above I used the commands specified by the Remove sensitive data post from GitHub Help
The only thing I changed was adding -rf
to the git rm
command to recursively force remove the files because I am dealing with multiple files/folders within the target folder.
The final command I used was:
1
|
|
Mind the vendor/cache/* vendor/bundle/* vendor/rails/* log/development.log
part. You can provide multiple path’s
When the command is finished, the history has been rewritten, but still the size of the repository hasn’t changed at this point.
You have to execute the following commands to also remove the files from you local repository.
1 2 3 |
|
Now we can force push our repository so that others can enjoy our effort.
1
|
|
After following the above steps our repository was shrinked by 98,55 %!. First the repository was 180 Mb and now it is 2.6 Mb
Ofcourse we had this numbers because there were alot of gem files within our repository that were updated frequently and pushed over and over again to the master branch.
I hope this post will help other’s to track large files within there git repository and how they can remove them to shrink the size of there repostiories.
]]>All the plugins and choice’s I had to made, and the maintenance of all these, just kept me from doing what I supposed to do with it, blogging.
So after some googling what a good alternative would be for Wordpress I found Octopress A blogging framework for hackers. Alright +1 for the title
So basically Octopress generates (bakes) HTML content from the Markdown content you write. I like this simplicity and I love the syntax.
Octopress give’s you the functionality to host it on your own server, but also to host it on Heroku or Github. Beside the Git approach (push your source and deploy the generated code to the master branch) you can use RSync to sync files to your server. I use Github to store my source and generated html.
Enough talking about Octopress for now. I’ll hope to post my next post soon.
]]>