When test under ngnix (1 worker).
TCP socket level is persistent:
tcp 0 0 ip-10-114-149-254.ec2:37426 unknown.servercentral:10044 ESTABLISHED 31867/php-fpm
tcp 0 0 ip-10-114-149-254.ec2:56806 ge0-41.mongo101.iad1.:10044 ESTABLISHED 31867/php-fpm

but sent 4 auth cmd to mongo (3 master 2 secondary)

Output:

array(2) {
  [0]=>
  array(3) {
    ["hash"]=>
    string(129) "xxx"
    ["server"]=>
    array(7) {
      ["host"]=>
      string(30) "xx"
      ["port"]=>
      int(10044)
      ["repl_set_name"]=>
      string(28) "xx"
      ["database"]=>
      string(16) "mixerbox_new_ssd"
      ["username"]=>
      string(7) "xx"
      ["auth_hash"]=>
      string(32) "xx"
      ["pid"]=>
      int(32492)
    }
    ["connection"]=>
    array(8) {
      ["last_ping"]=>
      int(1375308556)
      ["last_ismaster"]=>
      int(1375308557)
      ["ping_ms"]=>
      int(1)
      ["connection_type"]=>
      int(2)
      ["connection_type_desc"]=>
      string(7) "PRIMARY"
      ["max_bson_size"]=>
      int(16777216)
      ["tag_count"]=>
      int(0)
      ["tags"]=>
      array(0) {
      }
    }
  }
  [1]=>
  array(3) {
    ["hash"]=>
    string(129) "xx"
    ["server"]=>
    array(7) {
      ["host"]=>
      string(30) "xx"
      ["port"]=>
      int(10044)
      ["repl_set_name"]=>
      string(28) "xx"
      ["database"]=>
      string(16) "xx"
      ["username"]=>
      string(7) "xx"
      ["auth_hash"]=>
      string(32) "xx"
      ["pid"]=>
      int(32492)
    }
    ["connection"]=>
    array(8) {
      ["last_ping"]=>
      int(1375308556)
      ["last_ismaster"]=>
      int(1375308557)
      ["ping_ms"]=>
      int(3)
      ["connection_type"]=>
      int(4)
      ["connection_type_desc"]=>
      string(9) "SECONDARY"
      ["max_bson_size"]=>
      int(16777216)
      ["tag_count"]=>
      int(0)
      ["tags"]=>
      array(0) {
      }
    }
  }
}
===================================SECOND MONGO CONN REQUEST ===================================
array(2) {
  [0]=>
  array(3) {
    ["hash"]=>
    string(129) "xx"
    ["server"]=>
    array(7) {
      ["host"]=>
      string(30) "xx"
      ["port"]=>
      int(10044)
      ["repl_set_name"]=>
      string(28) "xx"
      ["database"]=>
      string(16) "xx"
      ["username"]=>
      string(7) "ilivapp"
      ["auth_hash"]=>
      string(32) "xx"
      ["pid"]=>
      int(32492)
    }
    ["connection"]=>
    array(8) {
      ["last_ping"]=>
      int(1375308556)
      ["last_ismaster"]=>
      int(1375308557)
      ["ping_ms"]=>
      int(1)
      ["connection_type"]=>
      int(2)
      ["connection_type_desc"]=>
      string(7) "PRIMARY"
      ["max_bson_size"]=>
      int(16777216)
      ["tag_count"]=>
      int(0)
      ["tags"]=>
      array(0) {
      }
    }
  }
  [1]=>
  array(3) {
    ["hash"]=>
    string(129) "xxx"
    ["server"]=>
    array(7) {
      ["host"]=>
      string(30) "xx"
      ["port"]=>
      int(10044)
      ["repl_set_name"]=>
      string(28) "set-xx"
      ["database"]=>
      string(16) "xx"
      ["username"]=>
      string(7) "xx"
      ["auth_hash"]=>
      string(32) "xx"
      ["pid"]=>
      int(32492)
    }
    ["connection"]=>
    array(8) {
      ["last_ping"]=>
      int(1375308556)
      ["last_ismaster"]=>
      int(1375308557)
      ["ping_ms"]=>
      int(3)
      ["connection_type"]=>
      int(4)
      ["connection_type_desc"]=>
      string(9) "SECONDARY"
      ["max_bson_size"]=>
      int(16777216)
      ["tag_count"]=>
      int(0)
      ["tags"]=>
      array(0) {
      }
    }
  }
}

Each connection has Two Servers, 1 to Primary and 1 To Secondary.
ALl connections have ["pid"]=> int(32492) ( I guess it is process Id)

It seems, alone with 2 connections (one to primary and one to secondary), there are TWO topology connections built, one for primary and one to secondary) - seems additional overheads for me.

Particularly this worries me:
WARN: discover_topology: ismaster worked, but we need to remove the seed host's connection ....

[31-Jul-2013 22:32:07 UTC] PHP Notice: CON INFO: ismaster: last ran at 1375309927 in /var/app/current/testmongo.php on line 47
[31-Jul-2013 22:32:07 UTC] PHP Notice: CON WARN: discover_topology: ismaster worked, but we need to remove the seed host's connection in /var/app/current/testmongo.php on line 47
[31-Jul-2013 22:32:07 UTC] PHP Notice: CON FINE: mongo_connection_destroy: Destroying connection object for $MASKED$;3367 in /var/app/current/testmongo.php on line 47
[31-Jul-2013 22:32:07 UTC] PHP Notice: CON FINE: mongo_connection_destroy: Closing socket for $MASKED$;3367. in /var/app/current/testmongo.php on line 47
[31-Jul-2013 22:32:07 UTC] PHP Notice: CON FINE: discover_topology: ismaster worked in /var/app/current/testmongo.php on line 47

@leozc: Please format the var_dump output with monospace formatting (three ` characters on a line before and after will do it). It's a bit hard to read in the current state.

Re:jmlkola.

@leozc: The error you're seeing is because the connection string hostname doesn't match what the servers themselves identify as. Take a look at mongo_connection_ismaster and look for the retval = 3 case.

Very nice hint! It ends up it is a domain name alias problem, let me investigate further...

Take a look at mongo_connection_ismaster and look for the retval = 3 case. helps