Merge branch 'release/0.10.4'

-

README.md

@@ -38,7 +38,7 @@ By using a small tool it is possible to it's possible to open streams directly i
 ## Motivation
 
 The plugin is not meant to be used as a right-click to stream thing. The idea is to
-make Deluge an abstraction layer for the [TidalStream](http://www.tidalstream.org/) project, i.e. torrents to http on demand.
+make Deluge an abstraction layer for the [Tidalstream](http://www.tidalstream.org/) project, i.e. torrents to http on demand.
 
 The _allow remote_ option is to allow remote add and stream of torrents.
 
@@ -48,9 +48,75 @@ The _allow remote_ option is to allow remote add and stream of torrents.
 * [ ] Better feedback in interface about streams
 * [ ] Better feedback when using API
 * [x] Reverse proxy improvement (e.g. port different than bind port)
+* [ ] Fix problems when removing torrent from Deluge (sea of errors)
+
+# HTTP API Usage
+
+## Prerequisite
+
+Install and enable the plugin. Afterwards, head into Streaming settings and enable "Allow remote control".
+The URL found in the "Remote control url" field is where the API can be reached. The auth used is Basic Auth.
+
+## Usage
+
+There is only one end-point and that is where a torrent stream can be requested.
+
+Both return the same responses and all responses are JSON encoded.
+All successfully authenticated responses have status code 200.
+
+## POST /streaming/stream
+
+POST body must be the raw torrent you want to stream. No form formatting or anything can be used.
+
+List of URL GET Arguments
+
+* **path**: Path inside the torrent file to either a folder or a file you want to stream. The plugin will try to guess the best one. **Optional**. **Default**: '' (i.e. find the best file in the whole torrent)
+* **infohash**: Infohash of the torrent you want to stream, can make it a bit faster as it can avoid reading POST body. **Optional**.
+* **label**: If label plugin is enabled and the torrent is actually added then give the torrent this label. **Optional**. **Default**: ''
+* **wait_for_end_pieces**: Wait for the first and last piece in the streamed file to be fully downloaded. Can be necessary for some video players. It also enforces that the torrent can be actually downloaded. If the key exist with any (even empty) value, the feature is enabled. **Optional**. **Default**: false
+
+## GET /streaming/stream
+
+* **infohash**: Does the same as when POSTed. **Mandatory**.
+* **path**: Does the same as when POSTed. **Optional**.
+* **wait_for_end_pieces**: Does the same as when POSTed. **Optional**.
+
+## Success Response
+
+```json
+{
+    "status": "success", # Always equals this
+    "filename" "horse.mkv", # Filename of the streamed torrent
+    "url": "http://example.com/" # URL where the file can be reached by e.g. a media player
+}
+```
+
+## Error Response
+
+```json
+{
+    "status": "error", # Always equals this
+    "message" "Torrent failed" # description for why it failed
+}
+```
 
 # Version Info
 
+## Version 0.10.4
+* Trying to set max priority less as it destroys performance
+
+## Version 0.10.3
+* Added label support
+* Reverse proxy config / replace URL config
+* Ensure internal Deluge state is updated before trying to use it
+
+## Version 0.10.2
+* Busting cache when waiting for piece
+* Math error in calculating size of readable bytes
+
+## Version 0.10.1
+* Small bugfixes related to priorities, should actually make sequential download work.
+
 ## Version 0.10.0
 * Rewrote large parts of the code
 * Now using [thomas](https://github.com/JohnDoee/thomas) as file-reading core - this adds support for multi-rar streaming.

examples/cli-stream/README.md

@@ -0,0 +1,23 @@
+# Commandline Tool to stream
+
+Stream from the commandline.
+
+## Requirements
+
+* Python
+* deluge_client python package
+
+## Installation example
+
+```bash
+virtualenv cli-example
+cli-example/bin/pip install deluge_client
+```
+
+## Usage
+
+Open a torrent directly in VLC on Linux or OSX.
+
+```bash
+vlc `cli-example/bin/python stream-cli.py username password my_video.torrent`
+```

examples/cli-stream/stream-cli.py

@@ -0,0 +1,28 @@
+import argparse
+import urllib
+
+from deluge_client import DelugeRPCClient
+
+
+if __name__ == '__main__':
+    parser = argparse.ArgumentParser(description='Stream something.')
+    parser.add_argument('username', type=str, help='Deluge username')
+    parser.add_argument('password', type=str, help='Deluge password')
+    parser.add_argument('path_or_url', type=str, help='Path or URL to torrent')
+
+    parser.add_argument('--hostname', '-o', type=str, default='localhost', help='Deluge daemon hostname or ip')
+    parser.add_argument('--port', '-p', type=int, default=58846, help='Deluge daemon port')
+
+    args = parser.parse_args()
+
+    if args.path_or_url.startswith('http'):
+        filedata = urllib.urlopen(args.path_or_url).read()
+    else:
+        with open(args.path_or_url, 'rb') as f:
+            filedata = f.read()
+
+    client = DelugeRPCClient(args.hostname, args.port, args.username, args.password)
+    client.connect()
+
+    result = client.streaming.stream_torrent(None, None, filedata, None, None, True)
+    print(result['url'])

examples/http-api/README.md

@@ -0,0 +1,33 @@
+# HTTP API
+
+Stream using the HTTP API built into Deluge Streaming.
+
+## Requirements
+
+* Python
+* requests package
+
+## Config
+
+You need to enable HTTP API in Deluge Streaming config.
+
+## Example
+
+```python
+from streamtorrent import stream_torrent
+
+if __name__ == '__main__':
+    with open('TPB.AFK.2013.1080p.h264-SimonKlose', 'rb') as f:
+        torrent_data = f.read()
+
+    # Stream 1080p TPB AFK using infohash to avoid posting the torrent
+    # if it already exist.
+    url = stream_torrent(
+        'http://stream:password@127.0.0.1:46123/streaming/stream',
+        infohash='411a7a164505636ab1a8276395b375a3a30bff32',
+        torrent_body=torrent_data,
+        label='tpbafk'
+    )
+    print('we can stream %s' % (url, ))
+```
+

examples/http-api/streamtorrent.py

@@ -0,0 +1,57 @@
+import requests
+
+class FailedToStreamException(Exception):
+    pass
+
+
+def stream_torrent(remote_control_url, infohash=None, path=None, wait_for_end_pieces=True, label=None, torrent_body=None):
+    """
+    Add a torrent to deluge, stream it and return a URL to where it can be watched.
+
+    All optional parameters are optional but you will need to at least provide an infohash (if the torrent is already added)
+    or a torrent_body (if you want the torrent added).
+
+    remote_control_url - The URL found in Deluge Streaming config
+    infohash - Torrent infohash, makes it faster if the torrent is already added
+    path - path inside the torrent you want to stream
+    wait_for_end_pieces - make sure the first and last piece are downloaded before returning url.
+                          This might be necessary for some players
+    label - Label to set in deluge
+    torrent_body - The content of the .torrent file you want to stream
+    """
+    first_part, second_part = remote_control_url.split('@')
+    username, password = first_part.split('/')[2].split(':')
+    url = '/'.join(first_part.split('/')[:2]) + '/' + second_part
+
+    params = {}
+    if infohash:
+        params['infohash'] = infohash
+
+    if wait_for_end_pieces:
+        params['wait_for_end_pieces'] = wait_for_end_pieces
+
+    if path:
+        params['path'] = path
+
+    if label:
+        params['label'] = label
+
+    if infohash: # try to stream it without posting torrent body first
+        r = requests.get(url, auth=(username, password), params=params)
+        if r.status_code != 200:
+            raise FailedToStreamException('Got non-200 error code from Deluge')
+
+        data = r.json()
+        if data['status'] == 'success':
+            return data['url']
+
+    if torrent_body:
+        r = requests.post(url, auth=(username, password), params=params, data=torrent_body)
+        if r.status_code != 200:
+            raise FailedToStreamException('Got non-200 error code from Deluge')
+
+        data = r.json()
+        if data['status'] == 'success':
+            return data['url']
+
+    raise FailedToStreamException('Streaming was never successful')

setup.py

@@ -42,7 +42,7 @@ from setuptools import setup, find_packages
 __plugin_name__ = "Streaming"
 __author__ = "Anders Jensen"
 __author_email__ = "johndoee@tidalstream.org"
-__version__ = "0.10.0"
+__version__ = "0.10.4"
 __url__ = "https://github.com/JohnDoee/deluge-streaming"
 __license__ = "GPLv3"
 __description__ = "Enables streaming of files while downloading them."

streaming/core.py

@@ -70,6 +70,13 @@ VIDEO_STREAMABLE_EXTENSIONS = ['mkv', 'mp4', 'iso', 'ogg', 'ogm', 'm4v']
 AUDIO_STREAMABLE_EXTENSIONS = ['flac', 'mp3', 'oga']
 STREAMABLE_EXTENSIONS = set(VIDEO_STREAMABLE_EXTENSIONS + AUDIO_STREAMABLE_EXTENSIONS)
 TORRENT_CLEANUP_INTERVAL = timedelta(minutes=30)
+MAX_FILE_PRIORITY = 2
+MAX_PIECE_PRIORITY = 7
+MIN_WAIT_PIECE_PRIORITY_DELAY = timedelta(seconds=5)
+WITHIN_CHAIN_PERCENTAGE = 0.10
+MIN_PIECE_COUNT_FOR_CHAIN_CONSIDERATION = 40
+MIN_CHAIN_WAIT_DELAY = timedelta(seconds=8)
+
 
 DEFAULT_PREFS = {
     'ip': '127.0.0.1',
@@ -127,6 +134,7 @@ class Torrent(object):
         self.readers = {}
         self.cycle_lock = defer.DeferredLock()
         self.last_activity = datetime.now()
+        self.waited_pieces = set()
 
         self.torrent = get_torrent(infohash)
         status = self.torrent.get_status(['piece_length'])
@@ -151,6 +159,7 @@ class Torrent(object):
     def can_read(self, from_byte):
         self.ensure_started()
 
+        status = self.torrent.get_status(['pieces'])
         needed_piece, rest = divmod(from_byte, self.piece_length)
         last_available_piece = None
         for piece, status in enumerate(self.torrent.status.pieces[needed_piece:], needed_piece):
@@ -159,27 +168,56 @@ class Torrent(object):
             last_available_piece = piece
 
         if last_available_piece is None:
-            logger.debug('Since we are waiting for a piece, setting priority for %s to max' % (needed_piece, ))
-            self.torrent.handle.set_piece_deadline(needed_piece, 0)
-            self.torrent.handle.piece_priority(needed_piece, 7)
+            logger.debug('Since we are waiting for a piece, we need to check if we should set piece %s to max' % (needed_piece, ))
 
+            is_next_in_chain = False
             f = self.get_file_from_offset(from_byte)
-            logger.debug('Also setting file to max %r' % (f, ))
-            file_priorities = self.torrent.get_file_priorities()
-            file_priorities[f['index']] = 7
-            self.torrent.set_file_priorities(file_priorities)
+            file_piece_count = (f['size'] // self.piece_length) + 1
 
-            for _ in range(300):
+            if file_piece_count <= MIN_PIECE_COUNT_FOR_CHAIN_CONSIDERATION:
+                is_next_in_chain = True
+            else:
+                best_reader_from_byte = max(reader[1] for reader in self.readers.values() if reader[1] <= from_byte)
+                best_reader_piece = best_reader_from_byte // self.piece_length
+                downloading_pieces = self.get_currently_downloading()
+                for unfinished_piece, status in enumerate(self.torrent.status.pieces[best_reader_piece:], best_reader_piece):
+                    if not status and unfinished_piece not in downloading_pieces:
+                        break
+
+                piece_diff = best_reader_piece - unfinished_piece - 1
+                if unfinished_piece >= best_reader_piece or piece_diff / file_piece_count <= WITHIN_CHAIN_PERCENTAGE:
+                    is_next_in_chain = True
+
+            if not is_next_in_chain:
+                logger.debug('Not a next-in-chain piece, setting priority now')
+                self.torrent.handle.set_piece_deadline(needed_piece, 0)
+                self.torrent.handle.piece_priority(needed_piece, MAX_PIECE_PRIORITY)
+
+            file_priorities = self.torrent.get_file_priorities()
+            if file_priorities[f['index']] != MAX_FILE_PRIORITY:
+                logger.debug('Also setting file to max %r' % (f, ))
+                file_priorities[f['index']] = MAX_FILE_PRIORITY
+                self.torrent.set_file_priorities(file_priorities)
+
+            for i in range(300):
                 if self.torrent.status.pieces[needed_piece]:
                     break
+
                 if not reactor.running:
                     return
+
+                if is_next_in_chain and i == MIN_CHAIN_WAIT_DELAY.total_seconds() * 5 and needed_piece not in self.get_currently_downloading():
+                    logger.debug('Next in chain waiting failed, setting priority')
+                    self.torrent.handle.set_piece_deadline(needed_piece, 0)
+                    self.torrent.handle.piece_priority(needed_piece, MAX_PIECE_PRIORITY)
+
                 time.sleep(0.2)
+                status = self.torrent.get_status(['pieces'])
 
             logger.debug('Calling read again to get the real number')
             return self.can_read(from_byte)
         else:
-            return ((last_available_piece - needed_piece) * self.piece_length) + rest + self.piece_length
+            return ((last_available_piece - needed_piece) * self.piece_length) + self.piece_length - rest
 
     def is_idle(self):
         return not self.readers and self.last_activity + TORRENT_CLEANUP_INTERVAL < datetime.now()
@@ -239,7 +277,7 @@ class Torrent(object):
 
                 if f['path'] in must_whitelist:
                     if f['path'] in first_files:
-                        file_priorities[i] = 7
+                        file_priorities[i] = MAX_FILE_PRIORITY
                     else:
                         file_priorities[i] = 1
                 elif f['path'] not in cannot_blacklist:
@@ -248,7 +286,7 @@ class Torrent(object):
             self.torrent.set_file_priorities(file_priorities)
 
         if self.readers:
-            status = self.torrent.get_status(['files', 'file_progress'])
+            status = self.torrent.get_status(['files', 'file_progress', 'pieces'])
             file_ranges = {}
             fileset_ranges = {}
             for path, from_byte, to_byte in self.readers.values():
@@ -258,6 +296,10 @@ class Torrent(object):
                 else:
                     file_ranges[path] = from_byte
 
+                reader_piece = from_byte // self.piece_length
+                self.torrent.handle.set_piece_deadline(reader_piece, 0)
+                self.torrent.handle.piece_priority(reader_piece, MAX_PIECE_PRIORITY)
+
                 for fileset_hash, fileset in self.filesets.items():
                     if path in fileset['files']:
                         if fileset_hash in fileset_ranges:
@@ -265,6 +307,23 @@ class Torrent(object):
                         else:
                             fileset_ranges[fileset_hash] = fileset['files'].index(path)
 
+            file_priorities = self.torrent.get_file_priorities()
+            logger.debug('Fileset heads: %r' % (fileset_ranges, ))
+            for fileset_hash, first_file in fileset_ranges.items():
+                fileset = self.filesets[fileset_hash]
+                logger.debug('From index %s' % (first_file, ))
+                file_mapping = {f['path']: f['index'] for f in status['files']}
+                for i, f in enumerate(fileset['files']):
+                    index = file_mapping[f]
+                    if i < first_file:
+                        file_priorities[index] = 0
+                    elif i == first_file:
+                        file_priorities[index] = MAX_FILE_PRIORITY
+                    else:
+                        file_priorities[index] = 1
+
+            self.torrent.set_file_priorities(file_priorities)
+
             currently_downloading = self.get_currently_downloading()
             logger.debug('File heads: %r' % (file_ranges, ))
             for f, progress in zip(status['files'], status['file_progress']):
@@ -291,28 +350,9 @@ class Torrent(object):
 
                     if piece < current_piece:
                         self.torrent.handle.piece_priority(piece, 0)
-                    elif piece == current_piece:
-                        self.torrent.handle.piece_priority(piece, 7)
                     else:
                         self.torrent.handle.piece_priority(piece, 1)
 
-            file_priorities = self.torrent.get_file_priorities()
-            logger.debug('Fileset heads: %r' % (fileset_ranges, ))
-            for fileset_hash, first_file in fileset_ranges.items():
-                fileset = self.filesets[fileset_hash]
-                logger.debug('From index %s' % (first_file, ))
-                file_mapping = {f['path']: f['index'] for f in status['files']}
-                for i, f in enumerate(fileset['files']):
-                    index = file_mapping[f]
-                    if i < first_file:
-                        file_priorities[index] = 0
-                    elif i == first_file:
-                        file_priorities[index] = 7
-                    else:
-                        file_priorities[index] = 1
-
-            self.torrent.set_file_priorities(file_priorities)
-
     def get_currently_downloading(self):
         currently_downloading = set()
         for peer in self.torrent.handle.get_peer_info():
@@ -322,6 +362,7 @@ class Torrent(object):
         return currently_downloading
 
     def reset_priorities(self):
+        status = self.torrent.get_status(['pieces'])
         for piece in range(len(self.torrent.status.pieces)):
             self.torrent.handle.piece_priority(piece, 1)
 
@@ -388,8 +429,7 @@ class TorrentHandler(object):
 
     def get_filesystem(self, infohash):
         torrent = get_torrent(infohash)
-        status = torrent.get_status(['piece_length', 'files', 'file_progress', 'save_path'])
-        self.piece_length = status['piece_length']
+        status = torrent.get_status(['files', 'file_progress', 'save_path'])
         save_path = status['save_path']
 
         found_rar = False
@@ -453,12 +493,21 @@ class TorrentHandler(object):
     @defer.inlineCallbacks
     def stream(self, infohash, path, wait_for_end_pieces=False):
         logger.debug('Trying to get path:%s from infohash:%s' % (path, infohash))
-        local_torrent = self.get_torrent(infohash)
         torrent = get_torrent(infohash)
 
+        for _ in range(10):
+            status = torrent.get_status(['piece_length'])
+            if status['piece_length'] > 0:
+                break
+            yield sleep(0.2)
+
+        local_torrent = self.get_torrent(infohash)
+
         filesystem = self.get_filesystem(infohash)
         if path:
             stream_item = filesystem.get_item_from_path(path)
+            if stream_item and filesystem == stream_item and path != stream_item.id:
+                stream_item = filesystem.get_item_from_path('%s/%s' % (filesystem.id, path))
         else:
             stream_item = filesystem
 
@@ -509,11 +558,13 @@ class TorrentHandler(object):
                         wait_for_pieces.append(piece - 1)
 
             logger.debug('We want first and last piece first, these are the pieces: %r' % (wait_for_pieces, ))
-            for piece in wait_for_pieces:
-                torrent.handle.set_piece_deadline(piece, 0)
-                torrent.handle.piece_priority(piece, 7)
+            if wait_for_pieces:
+                for piece in wait_for_pieces:
+                    torrent.handle.set_piece_deadline(piece, 0)
+                    torrent.handle.piece_priority(piece, MAX_PIECE_PRIORITY)
 
             for _ in range(220):
+                status = torrent.get_status(['pieces'])
                 for piece in wait_for_pieces:
                     if not torrent.status.pieces[piece]:
                         break
@@ -563,6 +614,7 @@ class StreamResource(Resource):
         infohash = request.args.get('infohash')
         path = request.args.get('path')
         wait_for_end_pieces = bool(request.args.get('wait_for_end_pieces'))
+        label = request.args.get('label')
 
         if path:
             path = path[0]
@@ -574,11 +626,16 @@ class StreamResource(Resource):
         else:
             infohash = infohash
 
+        if label:
+            label = label[0]
+        else:
+            label = None
+
         payload = request.content.read()
         if not payload:
             defer.returnValue(json.dumps({'status': 'error', 'message': 'invalid torrent'}))
 
-        result = yield self.client.stream_torrent(infohash=infohash, filedump=payload, filepath_or_index=path, wait_for_end_pieces=wait_for_end_pieces)
+        result = yield self.client.stream_torrent(infohash=infohash, filedump=payload, filepath_or_index=path, wait_for_end_pieces=wait_for_end_pieces, label=label)
         defer.returnValue(json.dumps(result))
 
     @defer.inlineCallbacks
@@ -752,7 +809,7 @@ class Core(CorePluginBase):
 
     @export
     @defer.inlineCallbacks
-    def stream_torrent(self, infohash=None, url=None, filedump=None, filepath_or_index=None, includes_name=False, wait_for_end_pieces=False):
+    def stream_torrent(self, infohash=None, url=None, filedump=None, filepath_or_index=None, includes_name=False, wait_for_end_pieces=False, label=None):
         logger.debug('Trying to stream infohash:%s, url:%s, filepath_or_index:%s' % (infohash, url, filepath_or_index))
         torrent = get_torrent(infohash)
 
@@ -771,6 +828,15 @@ class Core(CorePluginBase):
             core = component.get("Core")
             try:
                 yield core.add_torrent_file('file.torrent', filedump.encode('base64'), {'add_paused': True})
+                if label and 'Label' in component.get('CorePluginManager').get_enabled_plugins():
+                    label_plugin = component.get('CorePlugin.Label')
+                    if label not in label_plugin.get_labels():
+                        label_plugin.add(label)
+
+                    try:
+                        label_plugin.set_torrent(infohash, label)
+                    except:
+                        logger.exception('Failed to set label')
             except:
                 logger.exception('Failed to add torrent')
                 defer.returnValue({'status': 'error', 'message': 'failed to add torrent'})

streaming/torrentfile.py

@@ -50,6 +50,9 @@ class DelugeTorrentInput(InputBase.find_plugin('file')):
         if self.can_read_to <= tell or self.can_read_to is None:
             self.can_read_to = self.torrent.can_read(self.offset + tell) + tell
 
+            if self._open_file:
+                self._open_file.seek(tell)
+
         real_num = min(num, self.can_read_to - tell)
         if num != real_num:
             logger.info('The real number we can read to is %s and not %s at position %s' % (real_num, num, tell))