133 lines
		
	
	
		
			3.3 KiB
		
	
	
	
		
			Markdown
		
	
	
	
	
	
		
		
			
		
	
	
			133 lines
		
	
	
		
			3.3 KiB
		
	
	
	
		
			Markdown
		
	
	
	
	
	
| 
								 | 
							
								# Async-Limiter
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								A module for limiting concurrent asynchronous actions in flight. Forked from [queue](https://github.com/jessetane/queue).
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								[](http://www.npmjs.org/async-limiter)
							 | 
						||
| 
								 | 
							
								[](https://travis-ci.org/STRML/async-limiter)
							 | 
						||
| 
								 | 
							
								[](https://coveralls.io/r/STRML/async-limiter)
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								This module exports a class `Limiter` that implements some of the `Array` API.
							 | 
						||
| 
								 | 
							
								Pass async functions (ones that accept a callback or return a promise) to an instance's additive array methods.
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								## Motivation
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								Certain functions, like `zlib`, have [undesirable behavior](https://github.com/nodejs/node/issues/8871#issuecomment-250915913) when
							 | 
						||
| 
								 | 
							
								run at infinite concurrency.
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								In this case, it is actually faster, and takes far less memory, to limit concurrency.
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								This module should do the absolute minimum work necessary to queue up functions. PRs are welcome that would
							 | 
						||
| 
								 | 
							
								make this module faster or lighter, but new functionality is not desired.
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								Style should confirm to nodejs/node style.
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								## Example
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								``` javascript
							 | 
						||
| 
								 | 
							
								var Limiter = require('async-limiter')
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								var t = new Limiter({concurrency: 2});
							 | 
						||
| 
								 | 
							
								var results = []
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								// add jobs using the familiar Array API
							 | 
						||
| 
								 | 
							
								t.push(function (cb) {
							 | 
						||
| 
								 | 
							
								  results.push('two')
							 | 
						||
| 
								 | 
							
								  cb()
							 | 
						||
| 
								 | 
							
								})
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								t.push(
							 | 
						||
| 
								 | 
							
								  function (cb) {
							 | 
						||
| 
								 | 
							
								    results.push('four')
							 | 
						||
| 
								 | 
							
								    cb()
							 | 
						||
| 
								 | 
							
								  },
							 | 
						||
| 
								 | 
							
								  function (cb) {
							 | 
						||
| 
								 | 
							
								    results.push('five')
							 | 
						||
| 
								 | 
							
								    cb()
							 | 
						||
| 
								 | 
							
								  }
							 | 
						||
| 
								 | 
							
								)
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								t.unshift(function (cb) {
							 | 
						||
| 
								 | 
							
								  results.push('one')
							 | 
						||
| 
								 | 
							
								  cb()
							 | 
						||
| 
								 | 
							
								})
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								t.splice(2, 0, function (cb) {
							 | 
						||
| 
								 | 
							
								  results.push('three')
							 | 
						||
| 
								 | 
							
								  cb()
							 | 
						||
| 
								 | 
							
								})
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								// Jobs run automatically. If you want a callback when all are done,
							 | 
						||
| 
								 | 
							
								// call 'onDone()'.
							 | 
						||
| 
								 | 
							
								t.onDone(function () {
							 | 
						||
| 
								 | 
							
								  console.log('all done:', results)
							 | 
						||
| 
								 | 
							
								})
							 | 
						||
| 
								 | 
							
								```
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								## Zlib Example
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								```js
							 | 
						||
| 
								 | 
							
								const zlib = require('zlib');
							 | 
						||
| 
								 | 
							
								const Limiter = require('async-limiter');
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								const message = {some: "data"};
							 | 
						||
| 
								 | 
							
								const payload = new Buffer(JSON.stringify(message));
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								// Try with different concurrency values to see how this actually
							 | 
						||
| 
								 | 
							
								// slows significantly with higher concurrency!
							 | 
						||
| 
								 | 
							
								//
							 | 
						||
| 
								 | 
							
								// 5:        1398.607ms
							 | 
						||
| 
								 | 
							
								// 10:       1375.668ms
							 | 
						||
| 
								 | 
							
								// Infinity: 4423.300ms
							 | 
						||
| 
								 | 
							
								//
							 | 
						||
| 
								 | 
							
								const t = new Limiter({concurrency: 5});
							 | 
						||
| 
								 | 
							
								function deflate(payload, cb) {
							 | 
						||
| 
								 | 
							
								  t.push(function(done) {
							 | 
						||
| 
								 | 
							
								    zlib.deflate(payload, function(err, buffer) {
							 | 
						||
| 
								 | 
							
								      done();
							 | 
						||
| 
								 | 
							
								      cb(err, buffer);
							 | 
						||
| 
								 | 
							
								    });
							 | 
						||
| 
								 | 
							
								  });
							 | 
						||
| 
								 | 
							
								}
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								console.time('deflate');
							 | 
						||
| 
								 | 
							
								for(let i = 0; i < 30000; ++i) {
							 | 
						||
| 
								 | 
							
								  deflate(payload, function (err, buffer) {});
							 | 
						||
| 
								 | 
							
								}
							 | 
						||
| 
								 | 
							
								q.onDone(function() {
							 | 
						||
| 
								 | 
							
								  console.timeEnd('deflate');
							 | 
						||
| 
								 | 
							
								});
							 | 
						||
| 
								 | 
							
								```
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								## Install
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								`npm install async-limiter`
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								## Test
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								`npm test`
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								## API
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								### `var t = new Limiter([opts])`
							 | 
						||
| 
								 | 
							
								Constructor. `opts` may contain inital values for:
							 | 
						||
| 
								 | 
							
								* `q.concurrency`
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								## Instance methods
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								### `q.onDone(fn)`
							 | 
						||
| 
								 | 
							
								`fn` will be called once and only once, when the queue is empty.
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								## Instance methods mixed in from `Array`
							 | 
						||
| 
								 | 
							
								Mozilla has docs on how these methods work [here](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array).
							 | 
						||
| 
								 | 
							
								### `q.push(element1, ..., elementN)`
							 | 
						||
| 
								 | 
							
								### `q.unshift(element1, ..., elementN)`
							 | 
						||
| 
								 | 
							
								### `q.splice(index , howMany[, element1[, ...[, elementN]]])`
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								## Properties
							 | 
						||
| 
								 | 
							
								### `q.concurrency`
							 | 
						||
| 
								 | 
							
								Max number of jobs the queue should process concurrently, defaults to `Infinity`.
							 | 
						||
| 
								 | 
							
								
							 | 
						||
| 
								 | 
							
								### `q.length`
							 | 
						||
| 
								 | 
							
								Jobs pending + jobs to process (readonly).
							 | 
						||
| 
								 | 
							
								
							 |