New Topic

NoSQLBooster for MongoDB (formerly MongoBooster)Topic Question

Slow Performance using forEach

Fernando Ulisses dos Santos shared this question ago

Answered

When I run a script like this:

db.collection.find().forEach((it)=> {

db.collection.update({_id: it._id,{$set: {anyfield: "newvalue"}});

});

It runs very slow on nosqlbooster, I think that it's because it's running locally and showing every record that is updated, it normally return:

Updated 1 existing record(s) in 729ms

Updated 1 existing record(s) in 548ms

Updated 1 existing record(s) in 306ms

Updated 1 existing record(s) in 294ms

Updated 1 existing record(s) in 323ms

...

I wanna to disable these logs for this kind of command so it can run very fast, I tried disable: "Options -> Options That May Affect Performance -> Enable Mongo Script Logger", but it doesn't disable.

Ps. I'm running a remote database through a SSH tunnel.

The same question

Replies (3)

qinghai (support@nosqlbooster.com) ●

You can call MongoShell method "setVerboseShell(false)" to deactivate the verbose shell.

More about setVerboseShell: https://docs.mongodb.com/manual/reference/method/setVerboseShell/

Reply URL

Fernando Ulisses dos Santos ●

The option works well, but it still take a long time to run the same command in NoSQLBooster compared to run directly in mongo shell.

Reply URL

qinghai (support@nosqlbooster.com) ●

I tested it, and there were some performance gaps, but within the same order of magnitude （150M records）, I believe the performance gap is on mongodb's driver. How many pieces of data do you have to update, and I think that it's not appropriate to update big data one by one with forEach like this.

Reply URL

Fernando Ulisses dos Santos ●

Actually my original query is much more complex, and I use bulk update and everything, directly on Mongo Shell on the server it updates 1.000 records by second approximately, took about 20 minutes to update 1M records.

But using nosqlbooster, it's slow with 10 records using the follow scenario:

db.test.insert({nome:"Test1",age:30});

db.test.insert({nome:"Test2",age:31});

db.test.insert({nome:"Test3",age:32});

db.test.insert({nome:"Test4",age:33});

db.test.insert({nome:"Test5",age:34});

db.test.insert({nome:"Test6",age:35});

db.test.insert({nome:"Test7",age:36});

db.test.insert({nome:"Test8",age:37});

db.test.insert({nome:"Test9",age:38});

db.test.insert({nome:"Test10",age:39});

db.test.find({}); // run fast, 0.716s

setVerboseShell(false);

db.test.find({}).forEach((it)=> {

db.test.update({"_id": it._id},{$set: { zip: "13600" }})

}); // took 3.981s

db.test.find({}).forEach((it)=> {

it.newfield2 = "othervalue";

db.test.save(it);

}); // took 3.562s

Yes, I know that in MongoDb I could use a simple query to do this update in particular, but in my real scenario I'm making queries to another collections and some calculations before update, but I'm also using bulk update and indexes to speedup this update.

My MongoDB servers are hosted on a cloud server, so I need to connect through a SSH tunnel and my Internet connection isn't that much, it's only 20Mbits and I got some latency to the servers.

I observed that, when I run the script on server, the process "mongo" that is used by the client use about 30% of CPU while the "mongod" process took about 40% of CPU, if the code that is inside forEach is running in the client (in nosqlbooster), probably there is nothing that can be done, because it will took a lot of time to download the data to my computer and upload it again.

Unless if nosqlbooster can open a shell on the server to run the commands, even if there is a special option for this.

URL