Topics

Blog

My own topics

Workaround for the slow pagination using skip in MongoDB

Recently I came across a big MongoDB database which needed pagination. But it was surprising how getting to page number 2000 ended up in a timeout. A quick research led me to MongoDB documentation:

The cursor.skip() method is often expensive because it requires the server to walk from the beginning of the collection or index to get the offset or skip position before beginning to return results. As the offset (e.g. pageNumber above) increases, cursor.skip() will become slower and more CPU intensive. With larger collections, cursor.skip() may become IO bound.

So “skip” clearly won’t work for a big set which led to the sort and gt solution.

Sorting and $gt:

Simply sorting by a created_at field then continuing from the last known created_at date by finding the next set would solve the problem.

ids = db.c.find({}, {_id:1}).map(function(item){ return item._id; });
docs = db.c.find( { _id: { $in: ids.slice(2000,2050) } } );
Learn More

Flash an alert when Jenkins build fails

Jenkins Alert Development teams need to keep track of builds on Jenkins and sometimes an email alert is not flashy enough. Using this simple javascript on your team dashboard you can easily track failed builds and urge developers to fix it.

The script uses Jenkins JSON API with JSONP method to request the latest failed builds with author name and last commit that caused the failure. You will need to add the domain of the page where the request is sent to your “Jenkins” “Configure Global Security” “Domains from which to allow requests” configuration. Also the cookie from Jenkins is needed on the same browser so you will need to login to Jenkins from the same browser before running the script or proxy the request and use Basic Authorization using your API token to do the request.

Demo

Learn More

AWS PHP Service Discovery

Service Discovery is a simple PHP command to collect and store AWS information such as EC2s and RDSs in the current region and save them with their credentials into an encrypted JSON file on S3. The script later notifies each service via SSH and executes the service discovery client on each instance. Each client downloads the JSON file and uses it to configure different applications. It can easily be automated through Rundeck or Jenkins to be executed after each deploy.

Service Discovery is part of AWS PHP Commands.

Usage:

> php console.php aws:services:discover -h
Usage:
  aws:services:discover [options]

Options:
  -f, --forceNotify[=FORCENOTIFY]          Force Notify [default: false]
  -e, --notifyOnly[=NOTIFYONLY]            Notify only one of dev,prod [default: false]
  -c, --continueOnError[=CONTINUEONERROR]  Continue to next EC2 on client failure [default: false]
  -h, --help                               Display this help message
  -q, --quiet                              Do not output any message
  -V, --version                            Display this application version
      --ansi                               Force ANSI output
      --no-ansi                            Disable ANSI output
  -n, --no-interaction                     Do not ask any interactive question
  -v|vv|vvv, --verbose                     Increase the verbosity of messages: 1 for normal output, 2 for more verbose output and 3 for debug

Help:
 Discovers services information and credentials.
Learn More

AWS PHP Modify Security Groups Command

Modify Security Groups Command is an easy to use command that you can add to your DevOps to allow adding/Removing IPs or CIDRs to AWS security groups for all protocol and ports. The command is part of AWS PHP Commands.

Usage:

> php console.php aws:security-groups:modify -h
Usage:
  aws:security-groups:modify [options]

Options:
  -c, --cidr=CIDR            CIDR ex: 64.18.0.0/20 [default: false]
  -o, --operation=OPERATION  Operation to perform, one of add or remove [default: "add"]
  -e, --env[=ENV]            Which security groups this should run on. One of prod, dev [default: "dev"]
  -h, --help                 Display this help message
  -q, --quiet                Do not output any message
  -V, --version              Display this application version
      --ansi                 Force ANSI output
      --no-ansi              Disable ANSI output
  -n, --no-interaction       Do not ask any interactive question
  -v|vv|vvv, --verbose       Increase the verbosity of messages: 1 for normal output, 2 for more verbose output and 3 for debug

Help:
 Adds/removes CIDRs to security groups.
Learn More

Simple MySQL migration bash script 

#!/usr/bin/env bash

set -e


mkdir -p /root/tmp

#migrate "original db params" "new db params" "new db name"
function migrate(){
    mysqldump --skip-lock-tables --single-transaction --add-drop-table $1 > /root/tmp/${3}.sql
    echo "CREATE DATABASE IF NOT EXISTS ${3};" | mysql $2
    mysql --max_allowed_packet=1000M $2  $3 < /root/tmp/${3}.sql
    rm -f /root/tmp/${3}.sql
}


migrate "-uolduser -poldpassword -h oldhost olddbname" "-unewuser -pnewpassword -h newhost" "newdbname"

#more migrates here ...

Fork here

Learn More