New Topic

NoSQLBooster for MongoDB (formerly MongoBooster)Topic Problem

Export to JSON, BSON, CSV fails

Daniel Givoni shared this problem ago

Solved

seems as a problem with cursor instance - "cursor.size is not a function"

error:

{
	"message" : "cursor.size is not a function",
	"stack" : "script:102:25"
}

auto generated script:

import * as path from "path";
import * as fs from "fs";
const promisify = require("bluebird").promisify;
const appendFileAsync = promisify(fs.appendFile);

const BATCH_SIZE = 2000;

let filepath = "/Users/danielgivoni/Downloads/farmer_production.users.csv";
let targetPath = path.dirname(filepath);

if (!fs.existsSync(targetPath))
    require("mkdirp").sync(targetPath);

if (fs.existsSync(filepath))
    fs.unlinkSync(filepath);

const tab = "campain_email_farmers_query.js";

const queryScript = `db.getCollection("users").aggregate(
 [
	{
		"$lookup" : {
			"from" : "farmers",
			"localField" : "FarmerIds",
			"foreignField" : "uniqueId",
			"as" : "farms"
		}
	},
	{
		"$match" : {
			"$and" : [
				{
					"farms" : {
						"$ne" : [
							{
								
							}
						]
					}
				},
				{
					"farms.name" : {
						"$exists" : true
					}
				},
				{
					"farms.stores.emails" : {
						"$ne" : [ ]
					}
				},
				{
					"farms.valid" : {
						"$ne" : false
					}
				}
			]
		}
	},
	{
		"$unwind" : "$identityVerifer"
	},
	{
		"$unwind" : "$farms"
	},
	{
		"$unwind" : "$farms.stores"
	},
	{
		"$unwind" : "$farms.stores.emails"
	},
	{
		"$project" : {
			"telephone" : "$identityVerifer",
			"locale" : 1,
			"farm_name" : "$farms.name",
			"email" : "$farms.stores.emails",
			"_id" : 0,
			"currency" : "$farms.stores.currencyCode"
		}
	}
],
 {
	"$clusterTime" : {
		"clusterTime" : Timestamp(1569397676, 1),
		"signature" : {
			"hash" : BinData(0,"5fFgdo90tggglO5WlKOPjOQ4Ey8="),
			"keyId" : NumberLong("6726444850310283265")
		}
	}
})`;
let skip = 0;
let limit = Number.MAX_SAFE_INTEGER;

let connection = "d<..........>-a0.mlab.com:3<........>";
let db = "farmer_production";

let fields = [];

let isFirstRead = true;

const cursor = mb.getCursorFromQueryScript({ connection, db, queryScript, skip, limit });
const totalCount = cursor.size();
let processedCount = 0;

await(mb.batchReadFromCursor(cursor, BATCH_SIZE, (docs) => {
    return async(() => {
        let readLength = docs.length;
        if (!readLength) {
            return;
        }

        if (_.isEmpty(fields)) {
            fields = mb.tryGetFields(_.sampleSize(docs, 5));
        }

        let csvContent = mb.docsToCSV({ docs, fields, delimiter: ",", withColumnTitle: isFirstRead });
        isFirstRead = false;
        await(appendFileAsync(filepath, csvContent));

        processedCount += readLength;
        const percent = (processedCount / totalCount * 100).toFixed(1);
        console.log(`${percent}%	 ${processedCount}/${totalCount} docs exported to "${path.basename(filepath)}".`);

        sleep(10)
    })();
}));

_.delay(() => mb.openFolder(targetPath), 1000);

sleep(100);
console.log(`\nexport ${processedCount} docs from ${tab} to ${filepath} finished.`);

The same problem 0

Replies (6)

Daniel Givoni ●

other exports fail as well:

#script:62:9 +0.953s

export 0 docs from farmer_production:farmers@<........>b.com:<....>6 to /Users/danielgivoni/Downloads/farmer_production.farmers111.json finished.

Reply URL

This comment is in trash! Restore

Daniel Givoni ●

on version 5.1.2 works fine

Reply URL

This comment is in trash! Restore

qinghai (support@nosqlbooster.com) ●

Thank you for your feedback. We have worked out a test build to resolve this issue. Please download and give it a try.

MacOS: https://s3.mongobooster.com/download/releasesv5/nosqlbooster4mongo-5.2.2-beta.1.dmg

URL

This comment is in trash! Restore

Imraan Parker ●

Could you give me a Linux version, please?

URL

This comment is in trash! Restore

Daniel Givoni ●

i installed the beta version you add a link to:

the cursor error is not showing but the result of export is 0 files:

#script:131:9 +2.015s

export 0 docs from campain_email_farmers_query.js to /Users/danielgivoni/Downloads/farmer_production.users.json finished.

URL

This comment is in trash! Restore

qinghai (support@nosqlbooster.com) ●

Please give me some sample data, I need to recall this issue locally. You can send it to support@nosqlbooster.com.

URL

This comment is in trash! Restore

Jaive smare ●

Hi, any update here.

URL

This comment is in trash! Restore

qinghai (support@nosqlbooster.com) ●

Please download and give it a try.

MacOS: https://s3.mongobooster.com/download/releasesv5/nosqlbooster4mongo-5.2.2-beta.1.dmg

URL

This comment is in trash! Restore

Jaive smare ●

Still unable to export full csv report

URL

This comment is in trash! Restore

qinghai (support@nosqlbooster.com) ●

If you're reporting a /1f41e bug, please make sure you include steps to reproduce it. You can send your sample data to support@nosqlbooster.com

NoSQLBooster Version
Operating System (Windows/Mac/Ubuntu):
Expected Behavior
Actual Behavior
How to Reproduce (include your sample date)
Screenshots

URL

This comment is in trash! Restore

Jaive smare ●

Issue on version 5.2.1 including this --> https://s3.mongobooster.com/download/releasesv5/nosqlbooster4mongo-5.2.2-beta.1.dmg

447a8da1434390c5c915919af2cdcc5f cb506aab94bbaa99988147a2503ed0c8 616152bdd24fdbf17566e42f08777136 589e3a89777997c48d2915d4e79ed861 f7478a510f3018c1fd0da52315f610ca

The code for export is below

____________________________

import * as path from "path";
import * as fs from "fs";
const promisify = require("bluebird").promisify;
const appendFileAsync = promisify(fs.appendFile);

const BATCH_SIZE = 2000;

let filepath = "/Users/appleuser/Documents/CPLShowRoomBalanceDeducts.csv";
let targetPath = path.dirname(filepath);

if (!fs.existsSync(targetPath))
    require("mkdirp").sync(targetPath);

if (fs.existsSync(filepath))
    fs.unlinkSync(filepath);

const tab = "jive:balancededucts@ds139000-a0.kww98.fleet.mlab.com:39000 (1)";

const queryScript = `db.balancededucts.find({  "agentId" : ObjectId("5be27f8b21ab35652996babb") })
                 .sort({  "_id" : -1 })
                 .limit(200000)`;
let skip = 0;
let limit = 200000;

let connection = "ds139000-a0.kww98.fleet.mlab.com:39000";
let db = "jive";

let fields = [];

let isFirstRead = true;

const cursor = mb.getCursorFromQueryScript({ connection, db, queryScript, skip, limit });
const totalCount = cursor.size ? cursor.size() : cursor.count();
let processedCount = 0;

await(mb.batchReadFromCursor(cursor, BATCH_SIZE, (docs) => {
    return async(() => {
        let readLength = docs.length;
        if (!readLength) {
            return;
        }

        if (_.isEmpty(fields)) {
            fields = mb.tryGetFields(_.sampleSize(docs, 5));
        }

        let csvContent = mb.docsToCSV({ docs, fields, delimiter: ",", withColumnTitle: isFirstRead });
        isFirstRead = false;
        await(appendFileAsync(filepath, csvContent));

        processedCount += readLength;
        const percent = (processedCount / totalCount * 100).toFixed(1);
        console.log(`${percent}%	 ${processedCount}/${totalCount} docs exported to "${path.basename(filepath)}".`);

        sleep(10)
    })();
}));


_.delay(() => mb.openFolder(targetPath), 1000);

sleep(100);
console.log(`\nexport ${processedCount} docs from ${tab} to ${filepath} finished.`);

Issue on version 5.2.1 including this --> https://s3.mongobooster.com/download/releasesv5/nosqlbooster4mongo-5.2.2-beta.1.dmg

447a8da1434390c5c915919af2cdcc5f cb506aab94bbaa99988147a2503ed0c8 616152bdd24fdbf17566e42f08777136 589e3a89777997c48d2915d4e79ed861 f7478a510f3018c1fd0da52315f610ca

The code for export is below

____________________________

import * as path from "path";
import * as fs from "fs";
const promisify = require("bluebird").promisify;
const appendFileAsync = promisify(fs.appendFile);

const BATCH_SIZE = 2000;

let filepath = "/Users/appleuser/Documents/CPLShowRoomBalanceDeducts.csv";
let targetPath = path.dirname(filepath);

if (!fs.existsSync(targetPath))
    require("mkdirp").sync(targetPath);

if (fs.existsSync(filepath))
    fs.unlinkSync(filepath);

const tab = "jive:balancededucts@ds139000-a0.kww98.fleet.mlab.com:39000 (1)";

const queryScript = `db.balancededucts.find({  "agentId" : ObjectId("5be27f8b21ab35652996babb") })
                 .sort({  "_id" : -1 })
                 .limit(200000)`;
let skip = 0;
let limit = 200000;

let connection = "ds139000-a0.kww98.fleet.mlab.com:39000";
let db = "jive";

let fields = [];

let isFirstRead = true;

const cursor = mb.getCursorFromQueryScript({ connection, db, queryScript, skip, limit });
const totalCount = cursor.size ? cursor.size() : cursor.count();
let processedCount = 0;

await(mb.batchReadFromCursor(cursor, BATCH_SIZE, (docs) => {
    return async(() => {
        let readLength = docs.length;
        if (!readLength) {
            return;
        }

        if (_.isEmpty(fields)) {
            fields = mb.tryGetFields(_.sampleSize(docs, 5));
        }

        let csvContent = mb.docsToCSV({ docs, fields, delimiter: ",", withColumnTitle: isFirstRead });
        isFirstRead = false;
        await(appendFileAsync(filepath, csvContent));

        processedCount += readLength;
        const percent = (processedCount / totalCount * 100).toFixed(1);
        console.log(`${percent}%	 ${processedCount}/${totalCount} docs exported to "${path.basename(filepath)}".`);

        sleep(10)
    })();
}));


_.delay(() => mb.openFolder(targetPath), 1000);

sleep(100);
console.log(`\nexport ${processedCount} docs from ${tab} to ${filepath} finished.`);

URL

This comment is in trash! Restore

qinghai (support@nosqlbooster.com) ●

Please replace the `db.balancededucts.find(` with `db.getSiblingDB("jive").balancededucts.find(` and try it again. If it does not work, please send your sample JSON data and detailed repro steps to support@nosqlbooster.com

URL

This comment is in trash! Restore

Jaive smare ●

Please can your team address this issue. I rely heavily on your software to generate reports daily.

URL

This comment is in trash! Restore

Jaive smare ●

db.balancededucts.find(` with `db.getSiblingDB("jive").balancededucts.find( - doesn't work. It is saying balance deducts property can't be found

This problem (unable to export CSV) only occurs when we choose export option, AND export Current query result. If we export selected results, it exports sucessfully. But not for export Current query result - we get 0 export

URL

This comment is in trash! Restore

qinghai (support@nosqlbooster.com) ●

So, please change back the original code and execute it many times, Is the result of each execution the same? export 0 docs?

URL

This comment is in trash! Restore

Jaive smare ●

Please fix this issue or provide link to previous version as this problem only happened after the update

URL

This comment is in trash! Restore

qinghai (support@nosqlbooster.com) ●

I noticed that you all use mlab's MongoDB service. I tried that mlab, does have this problem, but other services such as atlas does not have this problem. I have been able to reproduce the problem and will solve it soon.

You can download 5.1.14 from the following URL:

MacOS: https://nosqlbooster.com/s3/download/releasesv5/nosqlbooster4mongo-5.1.14.dmg

URL

This comment is in trash! Restore

Imraan Parker ●

I am experiencing the same issue on 5.2.1

Reply URL

This comment is in trash! Restore

Jaive smare ●

I am experiencing the same issue on 5.2.1

Reply URL

This comment is in trash! Restore

qinghai (support@nosqlbooster.com) ●

We have worked out a new test build to resolve the export issue, please download and give it a try.

Windows: https://nosqlbooster.com/s3/download/releasesv5/nosqlbooster4mongo-5.2.3-beta.1.exe

macOS: https://nosqlbooster.com/s3/download/releasesv5/nosqlbooster4mongo-5.2.3-beta.1.dmg

Linux: https://nosqlbooster.com/s3/download/releasesv5/nosqlbooster4mongo-5.2.3-beta.1.AppImage

Thank you for your patience.

Reply URL

This comment is in trash! Restore

qinghai (support@nosqlbooster.com) ●

Resolved in 5.2.3

Reply URL

This comment is in trash! Restore