ssh tunnel not working

Pavel S shared this problem 2 years ago
Not a Problem

4.2.0 client for macos

connection with ssh server - ok

no connection with mongodb with ip and port on replica set.

this look like what not use ssh tunnel.

first try after run nosqlbooster:

{

"message" : "failed to connect to server [10.0.1.206:27017] on first connect [MongoError: connection 5 to 10.0.1.206:27017 timed out]",

"stack" : "MongoError: failed to connect to server [10.0.1.206:27017] on first connect [MongoError: connection 5 to 10.0.1.206:27017 timed out]" +

"at Pool.<anonymous> (/Applications/NoSQLBooster for MongoDB.app/Contents/Resources/app.asar/node_modules/mongodb-core/lib/topologies/server.js:336:35)" +

"at emitOne (events.js:96:13)" +

"at Pool.emit (events.js:191:7)" +

"at Connection.<anonymous> (/Applications/NoSQLBooster for MongoDB.app/Contents/Resources/app.asar/node_modules/mongodb-core/lib/connection/pool.js:280:12)" +

"at Object.onceWrapper (events.js:293:19)" +

"at emitTwo (events.js:106:13)" +

"at Connection.emit (events.js:194:7)" +

"at Socket.<anonymous> (/Applications/NoSQLBooster for MongoDB.app/Contents/Resources/app.asar/node_modules/mongodb-core/lib/connection/connection.js:197:10)" +

"at Object.onceWrapper (events.js:293:19)" +

"at emitNone (events.js:86:13)" +

"at Socket.emit (events.js:188:7)" +

"at Socket._onTimeout (net.js:351:8)" +

"at ontimeout (timers.js:386:14)" +

"at tryOnTimeout (timers.js:250:5)" +

"at Timer.listOnTimeout (timers.js:214:5)",

"name" : "MongoError"

}


second try and more:

{

"message" : "write EPIPE",

"stack" : "Error: write EPIPE" +

"at exports._errnoException (util.js:1050:11)" +

"at WriteWrap.afterWrite (net.js:813:14)",

"name" : "MongoError"

}

Best Answer
photo

I studied this question, the answer is likely "the behavior is by design".

1. According to MongoDB Server selection spec.

Clients use the hostnames listed in the replica set config, not the seed list

This spec requires clients to connect to the hostnames listed in the ismaster response. Furthermore, if the response is from a primary, the client MUST remove all hostnames not listed. In this case, the client disconnects from "host_alias" and tries "host1" and "host2". (See updateRSFromPrimary.)Thus, replica set members must be reachable from the client by the hostnames listed in the replica set config.

2. In MongoDB Official Issue tracking system

Issue: Connect to ReplicaSet through SSH tunneling, the issue status is closed.

if you are willing to connect directly to a single known member of the set, you can do this by using one of the MongoClient constructors that takes a single ServerAddress instead of a List<ServerAddress>. These constructors make direct connections to servers and don't attempt any discovery of the other replica set members, which is what's getting you into trouble.

Another option, of course, is to set up a VPN, but I assume that's not an option for you.

3. MongoDB NodeJS driver

MongoDB Nodejs 2.1 or below uses the seed list, you can simply connect MongoDB Replicaset over SSH like this.


//localhost:30001 local forward to the remote server  REMOTE_HOST:20017
MongoClient.connect("mongodb://localhost:30001/test?replicaSet=rs", function(err, db) {
console.log("db connected")
});
Run this sample with driver 2.1, I got "db connected".

But If run the same code with MongoDB Nodejs Driver 2.2 or plus. I got the following error:

connect error { MongoError: failed to connect to server [127.0.0.1:27017] on first connect [MongoNetworkError: connect ETIMEDOUT ]

NoSQLBooster is depending on mongodb nodejs driver 2.2, we plan to upgrade to 3.0 once the official 3.0 driver is released.

Comments (8)

photo
1

Thank you for your feedback.

If possible, could you please give me an accessible test MongoDB URI and SSH connection to allow me to recall the issue quickly?

photo
1

We just released 3.5.2. The version upgrades the Node.js MongoDB driver to the latest. Could you please download and give it a try?


https://www.mongobooster.com/downloads

photo
1

I have the same issue to connect to MongoDB Atlas replica set via SSH tunnel.

My mongobooster version is v3.5.6 free edition and MongoDB is Version 3.4.4

you may apply a free account in MongoDB Atlas to reproduce.

Here is the error message:


{
	"message" : "no primary found in replicaset",
	"stack" : "MongoError: no primary found in replicaset" +
              "at C:\\Users\\my_user\\AppData\\Local\\mongobooster\\app-3.5.6\\resources\\app.asar\\node_modules\\mongodb-core\\lib\\topologies\\replset.js:555:28" +
              "at Server.<anonymous> (C:\\Users\\my_user\\AppData\\Local\\mongobooster\\app-3.5.6\\resources\\app.asar\\node_modules\\mongodb-core\\lib\\topologies\\replset.js:312:24)" +
              "at Object.onceWrapper (events.js:290:19)" +
              "at emitOne (events.js:96:13)" +
              "at Server.emit (events.js:188:7)" +
              "at C:\\Users\\my_user\\AppData\\Local\\mongobooster\\app-3.5.6\\resources\\app.asar\\node_modules\\mongodb-core\\lib\\topologies\\server.js:299:14" +
              "at C:\\Users\\my_user\\AppData\\Local\\mongobooster\\app-3.5.6\\resources\\app.asar\\node_modules\\mongodb-core\\lib\\connection\\pool.js:469:18" +
              "at _combinedTickCallback (internal/process/next_tick.js:67:7)" +
              "at process._tickCallback (internal/process/next_tick.js:98:9)",
	"name" : "MongoError"
}

photo
photo
1

I tried it with my test replica set+SSH key, It works well.


I hope to solve this issue, but can not reproduce it. If possible, could you please give me your test MongoDB replica set and test SSH key? One or two days is enough.

photo
1

in my problem 10.0.1.206:27017 is arbiter

photo
1

Cannot Reproduce - its very easy

21adcac8fc0ffcb65746e5327f9f662c53f1d950c642e0f3f7cf12ab3a8ddf6c91d960642d7f299a598691711ae2c761

photo
1

Thank you for your feedback.

It seems that "discovery members" method is successful. Can you show me the specific connect error message?

And, Can you connect your test replica set server with mongo command-line + ssh tunnel (port forwarding . ssh -L 9000:localhost:27017user@example.com) ?

If the connection is successful, could you please show me the complete mongo command line?

photo
photo
1

i open vpn + ssh - and connection opened.

you can see - what connection go with out ssh to servers.

mongodb 3.4.4

photo
1

I believe the problem is that after doing initial discovery using SSH tunnel, mongobooster tries to connect to individual replica set members directly (i.e. not using SSH tunnel). The expected behavior is that mongobooster will use SSH tunnel to connect to discovered replica set member.


This is based on netstat logs Pavel has provided:


  • There are direct connections from local IP address (192.168.0.216) to mongodb port on private IP address of 10.0.1.206. As I understand, 10.0.1.x network is normally not accessible from outside (that is why SSH tunnel is needed) and connections work only because Pavel has set-up VPN tunnel.
  • So if mongobooster would work as expected, we would not see those connections, but instead would see another SSH connection to bastion host (54.XXX.143.166)


To reproduce the problem w/o having remote subnet, you would need following:

  • a replica set (maybe even 1-member replica set would be enough), say with IPs rs1, rs2, rs3
  • a "bastion" host with sshd (with different IP address), with IP bh
  • configure mongobooster to connect to replica set via SSH tunnel to "bastion" host
  • Connect
  • Find mongobooster PID, mpid
  • Use netstat to see which address mongobooster has used to actually connect to replica set member: netstat -p | fgrep <mpid>
  • Intended behaviour: mongobooster has connections only to "bastion" host IP bh
  • What will most likely happen: mongobooster will have direct connection to some replica set member

photo
1

Thanks for your very detailed answer, I can reproduce the issue now.

Suppose I already have a 3-nodes replica set in the remote host (e.g example.com). I can connect the replica set in the remote machine with the following mongo command line.


mongo "mongodb://127.0.0.1:20001,127.0.0.1:20002,127.0.0.1:20003/?readPreference=primary&replicaSet=rs"
Now, I want to connect it locally, I try to set up SSH tunnel like this:


ssh -L 30001:127.0.0.1:20001 -L 30002:127.0.0.1:20002 -L 30003:127.0.0.1:20003 user@example.com

And connect it use mongo command line


mongo "mongodb://127.0.0.1:30001,127.0.0.1:30002,127.0.0.1:30003/?readPreference=primary&replicaSet=rs"

But no luck, I got the following error:


2017-12-19T21:04:39.718+0800 I NETWORK [thread1] Starting new replica set monitor for rs/127.0.0.1:30001,127.0.0.1:30002

2017-12-19T21:04:39.990+0800 I NETWORK [thread1] Successfully connected to 127.0.0.1:30002 (1 connections now open to 127.0.0.1:30002 with a 5 second timeout)

2017-12-19T21:04:39.990+0800 I NETWORK [ReplicaSetMonitor-TaskExecutor-0] Successfully connected to 127.0.0.1:30001 (1 connections now open to 127.0.0.1:30001 with a 5 second timeout)

2017-12-19T21:04:40.147+0800 I NETWORK [ReplicaSetMonitor-TaskExecutor-0] changing hosts to rs/127.0.0.1:20001,127.0.0.1:20002 from rs/127.0.0.1:30001,127.0.0.1:30002

2017-12-19T21:04:45.146+0800 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] Failed to connect to 127.0.0.1:20001 after 5000ms milliseconds, giving up.

2017-12-19T21:04:45.146+0800 W NETWORK [thread1] Failed to connect to 127.0.0.1:20002 after 5000ms milliseconds, giving up.

2017-12-19T21:04:50.651+0800 W NETWORK [thread1] Failed to connect to 127.0.0.1:20001 after 5000ms milliseconds, giving up.

2017-12-19T21:04:52.418+0800 E - [thread2] Error saving history file: FileOpenFailed: Unable to fopen() file

2017-12-19T21:04:52.418+0800 I CONTROL [thread2] shutting down with code:0

I would like to ask, how to connect replica set over SSH tunnel by using common SSH command and mongo command?

Thanks a lot

photo
1

I studied this question, the answer is likely "the behavior is by design".

1. According to MongoDB Server selection spec.

Clients use the hostnames listed in the replica set config, not the seed list

This spec requires clients to connect to the hostnames listed in the ismaster response. Furthermore, if the response is from a primary, the client MUST remove all hostnames not listed. In this case, the client disconnects from "host_alias" and tries "host1" and "host2". (See updateRSFromPrimary.)Thus, replica set members must be reachable from the client by the hostnames listed in the replica set config.

2. In MongoDB Official Issue tracking system

Issue: Connect to ReplicaSet through SSH tunneling, the issue status is closed.

if you are willing to connect directly to a single known member of the set, you can do this by using one of the MongoClient constructors that takes a single ServerAddress instead of a List<ServerAddress>. These constructors make direct connections to servers and don't attempt any discovery of the other replica set members, which is what's getting you into trouble.

Another option, of course, is to set up a VPN, but I assume that's not an option for you.

3. MongoDB NodeJS driver

MongoDB Nodejs 2.1 or below uses the seed list, you can simply connect MongoDB Replicaset over SSH like this.


//localhost:30001 local forward to the remote server  REMOTE_HOST:20017
MongoClient.connect("mongodb://localhost:30001/test?replicaSet=rs", function(err, db) {
console.log("db connected")
});
Run this sample with driver 2.1, I got "db connected".

But If run the same code with MongoDB Nodejs Driver 2.2 or plus. I got the following error:

connect error { MongoError: failed to connect to server [127.0.0.1:27017] on first connect [MongoNetworkError: connect ETIMEDOUT ]

NoSQLBooster is depending on mongodb nodejs driver 2.2, we plan to upgrade to 3.0 once the official 3.0 driver is released.

photo
1

To sum it up: mongoboster relies on MongoDB Nodejs Driver to connect. While SSH tunneling works with standalone mongodb servers, with replica set after initial lookup there is another step when driver decides to which particular instance to connect based on actual replica set config (following mongo specs). Because of this, driver attempts to connect to remote system bypassing SSH tunnel.

Fixing this in mongobooster would mean whole lot of extra work to duplicate replica set connection logic with tunneling support. As such, it will not be done. Those who want to connect to replica set over SSH, can do it by connecting to individual services (that is what we have been doing for the past year anyway).

Am I right?

photo
1

Yes, you have summed up very well. Thanks.

photo
1

Vote for this feature. In the market only Studio 3T works fine with SSH tunnel.


By the way, what the "can do it by connecting to individual services" exactly means. Any instruction or article is helpful.

photo
1

It behaves exactly as MongoDB's server spec. We won't fix it. For details, please refer to our previous reply.


Although you can not connect to the entire replica set over SSH tunnel, you can still connect to one of the nodes to explore or manage your replica set. It is the same as you connect a stand-alone mongod server.


1. In the basic tab of Connection Editor. Set the Type as "Direct Connection", not "Replica Set | Shard Cluster".

2. Enter the server host and port, which is the one of the nodes of your replica set.


Once connected to any node in the RS, you can discover the RS topology via rs.config() or db.isMaster(). You could then use this information to reconnect to the primary node.

photo
1

"We won't fix it" - in free mongohub this type connecting work normal.

photo