Problems with Mongoid and Sidekiq- Brainstorming30 Jan 2013
Posts on how awesome the experience was and how much better Sidekiq is from Resque coming soon, though with all the good came some bad.
Summary of the solution
With Sidekiq, we are processing around 25X more jobs than what we were doing with Resque, processing around 15,000,000 jobs per day, at paces of over 1K per second at times (at peak we go up well past that)
This is how many jobs we processed today…
And this is a snapshot of our realtime graph
On the MongoDB side we are working with Mongoid and we have a shared environment, 9 shards with 3 replicas in each shard, all running through 2 routers.
Our production mongoid config looks like this
production: op_timeout: 3 connection_timeout: 3 sessions: default: hosts: - HOST_NAME:27017 #Single router R0 username: USER_NAME password: PASSWORD database: DATABASE_NAME options: consistency: :eventual
We are using latest versions of all relevant gems (Sidekiq, Mongoid, Moped, Redis)
All seems fine right? What’s the problem?
The problem is that we have too many connections opening and closing to our mongo instances. (~25-40 new connections per second).
Each time a job is picked up, a connection to Mongo is opened and when the job is done, this connection is closed (using Kiqstand middleware).
This is causing huge loads on our router server, and causing mongo to run out of file descriptors at times.
More then anything, this post is a callout for discussion with anyone using similar solution with similar scale and can assist, I know I would love to brainstorm on how to solve this problem.