in Education by
I have a function that has more than 1400+ crypto pairs and I have to send an API against each pair and store the trades. Now each pair takes 3-4 seconds hence the whole function takes a lot of time. I am getting the pairs from my DB and I am storing trade data in my DB as well. I need to process the pairs in parallel so the trades from the pair in the beginning don't miss because the function is not processing. This is my current function: const getTrades = async () => { let page = 1; const results = await db.query("SELECT * FROM pairs;"); const pairs = results.rows; const latest = await db.query("SELECT MAX(trade_time) FROM trades"); const latestTrade = latest.rows[0].max; const coinResult = await db.query("SELECT * FROM coins"); let coinsInfo = coinResult.rows; coinsInfo = coinsInfo.flat(); for (const pair of pairs) { let biggestTrade = []; const response = await axios.get( `https://api.binance.com/api/v3/trades?symbol=${pair.pair}` ); let filtered = response.data; filtered = filtered.filter((trade) => trade.time > latestTrade); let sells = filtered.filter((trade) => trade.isBuyerMaker === true); let buys = filtered.filter((trade) => trade.isBuyerMaker === false); if (sells.length > 0) { biggestTrade.push( sells.reduce(function (prev, current) { return prev.quoteQty > current.quoteQty ? prev : current; }) ); } if (buys.length > 0) { biggestTrade.push( buys.reduce(function (prev, current) { return prev.quoteQty > current.quoteQty ? prev : current; }) ); } biggestTrade = biggestTrade.flat(); for (const trade of filtered) { let priceUSD = 0; let baseAssetIcon = "null"; for (const coin of coinsInfo) { if (coin.symbol.toUpperCase() === pair.quote_asset) { priceUSD = coin.current_price; } if (coin.symbol.toUpperCase() === pair.base_asset) { baseAssetIcon = coin.image_url; } if (priceUSD > 0 && baseAssetIcon != "null") { break; } } if (trade.quoteQty * priceUSD > 50000) { const results = db.query( "INSERT INTO trades (exchange_name, exchange_icon_url, trade_time, price_in_quote_asset,price_in_usd, trade_value, base_asset_icon, qty, quoteQty, is_buyer_maker, pair, base_asset_trade, quote_asset_trade) VALUES($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12, $13)", [ "Binance", "https://assets.coingecko.com/markets/images/52/small/binance.jpg?1519353250", trade.time, trade.price, priceUSD, trade.quoteQty * priceUSD, baseAssetIcon, trade.qty, trade.quoteQty, trade.isBuyerMaker, pair.pair, pair.base_asset, pair.quote_asset, ] ); console.log("TRADE ADDED"); } } } console.log("PAIRS ARE OVER"); }; pairs has over 1400 entries and this is the one where are looping through. JavaScript questions and answers, JavaScript questions pdf, JavaScript question bank, JavaScript questions and answers pdf, mcq on JavaScript pdf, JavaScript questions and solutions, JavaScript mcq Test , Interview JavaScript questions, JavaScript Questions for Interview, JavaScript MCQ (Multiple Choice Questions)

1 Answer

0 votes
by
depends on how many servers you are running this function on. if it's one single machine, use worker_threads, basically run the same function in separate threads to achieve parallelization, but to be honest, 1400 pairs are a lot, each for 3-4 seconds, so total around 1-2hrs per run if in serial. Depending on your machines, if you have 8 cores, it might reduce the time by 8 folds but still leave you like around 10 minutes. and cloud service usually charge a lot more for instances that have more cpu cores. if it's multiple machines, use a master and a queue to push new pairs to each worker machine and for each worker machine, you can also generate multiple threads for each machine, in that way you can scale horizontally, and it's possible to finish the run in seconds. in this situation, each machine you can get the cheap one from cloud providers. so depends on your requirements, if you wanna super fast, you gotta add more machines.

Related questions

0 votes
    This question already has answers here: How to split a string with angularJS (4 answers) Closed 5 years ... Questions for Interview, JavaScript MCQ (Multiple Choice Questions)...
asked Jul 14, 2022 in Education by JackTerrance
0 votes
    This question already has answers here: How to split a string with angularJS (4 answers) Closed 5 years ... Questions for Interview, JavaScript MCQ (Multiple Choice Questions)...
asked Jul 3, 2022 in Education by JackTerrance
0 votes
    This question already has answers here: How to split a string with angularJS (4 answers) Closed 5 years ... Questions for Interview, JavaScript MCQ (Multiple Choice Questions)...
asked Jun 19, 2022 in Education by JackTerrance
0 votes
    This question already has answers here: When to use ":"(colon) operator in javascript vs "=" operator ... Questions for Interview, JavaScript MCQ (Multiple Choice Questions)...
asked Jun 3, 2022 in Education by JackTerrance
0 votes
    This question already has answers here: When to use ":"(colon) operator in javascript vs "=" operator ... Questions for Interview, JavaScript MCQ (Multiple Choice Questions)...
asked Jun 2, 2022 in Education by JackTerrance
0 votes
    I'm trying to use the sensor's API with react and I can't seen to be able to make it work ... , JavaScript Questions for Interview, JavaScript MCQ (Multiple Choice Questions)...
asked Apr 12, 2022 in Education by JackTerrance
0 votes
    I'm trying to use the sensor's API with react and I can't seen to be able to make it work ... , JavaScript Questions for Interview, JavaScript MCQ (Multiple Choice Questions)...
asked Apr 10, 2022 in Education by JackTerrance
0 votes
    I created a raycaster that allows me to pick an object from the scene with my mouse like such: ... JavaScript Questions for Interview, JavaScript MCQ (Multiple Choice Questions)...
asked Apr 9, 2022 in Education by JackTerrance
0 votes
    I am trying to redirect to another site then run code on it. function gotonewpage() { window.location. ... Questions for Interview, JavaScript MCQ (Multiple Choice Questions)...
asked Apr 16, 2022 in Education by JackTerrance
0 votes
    I am trying to redirect to another site then run code on it. function gotonewpage() { window.location. ... Questions for Interview, JavaScript MCQ (Multiple Choice Questions)...
asked Apr 12, 2022 in Education by JackTerrance
0 votes
    I want to make a new span class called "button". When the "button" is clicked, it should run ... JavaScript Questions for Interview, JavaScript MCQ (Multiple Choice Questions)...
asked Mar 4, 2022 in Education by JackTerrance
0 votes
    I am trying to return an javascript value with Awesome lib in Vb.net / C#. Now I know how to ... , JavaScript Questions for Interview, JavaScript MCQ (Multiple Choice Questions)...
asked Feb 13, 2022 in Education by JackTerrance
0 votes
    so i'm reviewing and practicing making rest api with node mongoose and express. I'm having problem making ... Questions for Interview, JavaScript MCQ (Multiple Choice Questions)...
asked Jul 14, 2022 in Education by JackTerrance
0 votes
    I'm trying to debug my nodejs app using node-inspector. But Google Chrome doesn't show the code. ... JavaScript Questions for Interview, JavaScript MCQ (Multiple Choice Questions)...
asked Jul 11, 2022 in Education by JackTerrance
0 votes
    I have been looking at using the ssh2 module for sftp shipping of logs. However the cloud service hosting ... Questions for Interview, JavaScript MCQ (Multiple Choice Questions)...
asked Jun 18, 2022 in Education by JackTerrance
...