diff --git a/DESIGN.md b/DESIGN.md new file mode 100644 index 0000000..3b63a91 --- /dev/null +++ b/DESIGN.md @@ -0,0 +1,51 @@ +## Document persistence + +A Document object contains the metadata for an editable collaborative session. +For example: the title, 'live' status, creation/modification dates, and most importantly an array of 'chunks'. + +A DocumentChunk object contains a 'snapshot' of the session, which is represented by the `fileId` of an ODT file stored in GridFS, and an array of WebODF 'operations' that are to be executed on that snapshot. + +As a session progresses, the last chunk is loaded into memory and more operations are appended to it. Once the session is closed or someone issues a 'save' action, an ODT snapshot is made - and with it, a new chunk is created. This way, when a new person joins a session, they only have to access the last chunk their browser does not need to 'replay' the entire operation history of the document. + +## Storage adapters + +A storage adapter is a way to get documents in and out of Manticore with special overrides of the document API's controller methods. + +In the default case where Manticore is a standalone installation, the `local` storage adapter is used. +However, two other storage adapters are implemented: `webdav` and `chwala`. + +### `webdav` + +This can be used with any authentication strategy. You can set the `WEBDAV_SERVER` and `WEBDAV_PATH` config variables to point to the server of your choice. There are many different implementations of WebDAV out there, and Manticore's adapter has been seen to work with Kolab's iRony, OwnCloud, and Box.net. + +There may not be sufficient uniqueness guarantees for documents in a WebDAV server, so our adapter tries to build a unique `contentId` by combining the `etag` and `last-modified` values into one. When a new document (available to the logged-in user) is found on the WebDAV server, a `Document` entry for it is created in the DB and its `webdav.contentId` field is set to the aforementioned value. + +The uniqueness of the `contentId` allows us to reasonably detect when a document has been moved, modified, or deleted on the server. + +When a document is 'opened' for editing (with a `GET`, for example), if a chunk isn't available with Manticore, the file is downloaded by Manticore from the WebDAV server and the first chunk is made. +When the 'save' action is done in an ongoing editing session, a new chunk+snapshot is created and the new ODT file is persisted back to the server, and the WebDAV/Manticore `etag`s are made consistent. + +### `chwala` + +This is only usable with the LDAP auth strategy. Manticore is meant to be used (only the editor part) via an `iframe` within the Roundcube UI. + +`Document`s are created in Manticore when the storage adapter gets an authenticated (as the 'creating' user) `POST` request from chwala, containing the Chwala `id` of the file, a `title` string, and an access permissions list. +When the first chunk has to be created, Manticore will issue an authenticated `GET` to Chwala to retrieve the actual ODT file. + +For more info, see `README-roundcube.md`. + +## Editing + +The client-side and server-side talk to each other through 'adaptors'. + +## Managing sessions (rooms) + +1. `components/adaptor/index.js` contains the logic for receiving requests from users to join an editing 'room' addressed by the document Id, and determining whether to create a 'room' if none exists. +2. `components/adaptor/room.js` handles every websocket-communicated event and action that can happen in an editing session in terms of operations and status messages, relays WebODF operations between clients, adds cleanup operations in case clients drop out, and persists these operations to the latest chunk. Each document gets it's own room. +3. `components/adaptor/recorder.js` contains two separate kinds of code that are run: + - In the Manticore nodejs process, to issue instructions to the serverside phantomjs session used for maintaining the live DOM version of a session + - In the phantomjs-contained webpage, that takes care to service requests from the node process and generates zipped-up ODT snapshots when needed. + + The live ODT DOM is generated whenever a fresh session is created, so that clients can open collaborative documents almost instantly without replaying anything. + +Every incoming operation by a client is relayed to the webodf instance maintaining the live DOM in `recorder.js`, and only when everything executes fine without errors is the parent `Room` informed of it, at which point the new operations are appended to the DB entry and passed on to other clients. diff --git a/server/api/document/storage/chwala/index.js b/server/api/document/storage/chwala/index.js index ac81afe..0165018 100644 --- a/server/api/document/storage/chwala/index.js +++ b/server/api/document/storage/chwala/index.js @@ -1,209 +1,251 @@ 'use strict'; var _ = require('lodash'); var async = require('async'); var mongoose = require('mongoose'); var https = require('https'); var fs = require('fs'); var url = require('url'); var path = require('path'); var Grid = require('gridfs-stream'); var multer = require('multer'); var request = require('request'); var querystring = require('querystring'); var crypto = require('crypto'); var config = require('../../../../config/environment'); var User = require('../../../user/user.model'); var Document = require('../../document.model').Document; var DocumentChunk = require('../../document.model').DocumentChunk; var Template = require('../../../template/template.model'); var gfs = Grid(mongoose.connection.db, mongoose.mongo); var serverUrl = config.storage.chwala.server; +/** + * Uses the auth encryption key key to decipher ldap password + */ function decrypt(password) { var decipher = crypto.createDecipher('aes-256-cbc', config.auth.ldap.key); return decipher.update(password, 'base64', 'utf8') + decipher.final('utf8'); } +/** + * Sends an authenticated request as a user (with their LDAP password), for a given + * Chwala ID (uuid), and downloads that to a given GridFS file Id. + */ function downloadToGridFS(user, uuid, fileId, cb) { var file = gfs.createWriteStream({ _id: fileId, filename: uuid, mode: 'w', chunkSize: 1024 * 4, content_type: 'application/vnd.oasis.opendocument.text', root: 'fs' }); request.get({ url: serverUrl + '/' + uuid, auth: { user: user.ldap.username, pass: decrypt(user.ldap.password) } }) .on('error', function (err) { cb(err); }) .pipe(file); file.on('finish', cb); } +/** + * Sends an authenticated PUT request to Chwala to upload a file to a given Chwala ID. + * The file is read from readStream. + */ + function uploadToServer(user, uuid, readStream, cb) { readStream.pipe(request.put({ url: serverUrl + '/' + uuid, auth: { user: user.ldap.username, pass: decrypt(user.ldap.password) }, headers: { 'Content-Type': 'application/vnd.oasis.opendocument.text' } }, cb)); } +/** + * Creates the first chunk for a Document by retrieving the ODT from Chwala into GridFS. + */ function createFirstChunk(user, uuid, cb) { var chunkId = new mongoose.Types.ObjectId(), fileId = new mongoose.Types.ObjectId(); var firstChunk = new DocumentChunk({ _id: chunkId, snapshot: { fileId: fileId } }); downloadToGridFS(user, uuid, fileId, function(err) { if (err) { return cb(err); } firstChunk.save(function (err) { cb(err, firstChunk); }); }); } +/** + * Lists the available documents that have the user as either the creator or an editor + */ exports.index = function (req, res) { var userId = req.user._id; Document.find({ '$or': [ { 'creator': userId }, { 'editors': { '$in': [userId] } } ] }) .populate('creator', 'name email') .populate('editors', 'name email') .exec(function (err, documents) { if(err) { return handleError(res, err); } return res.json(200, documents); }); }; +/** + * Return the metadata of a Document with a given ID, + * with the creators and editors' names/emails populated + */ exports.show = function(req, res) { Document.findById(req.params.id) .populate('creator', 'name email') .populate('editors', 'name email') .exec(function (err, document) { if(err) { return handleError(res, err); } if(!document) { return res.send(404); } return res.json(document); }); }; +/** + * Creating a document from a template is not supported by the Chwala adapter, + * as this is something handled on the Roundcube/Chwala side. + */ exports.createFromTemplate = function (req, res) { return res.send(405); }; +/** + * Handles an incoming POST request from Chwala to create a new document with + * a given Chwala ID, title, and access list. If no such document exists already, + * a first chunk for it is created by retrieving the ODT file from Chwala. + */ exports.upload = function (req, res, next) { var id = req.body.id, title = req.body.title, access = req.body.access; Document.findById(id, function (err, document) { if (err) { return handleError(res, err); } if (document) { return res.json(422, document); } createFirstChunk(req.user, id, function (err, firstChunk) { if (err) { return handleError(res, err); } Document.create({ _id: id, title: title, creator: req.user._id, chunks: [firstChunk._id], access: access }, function (err, document) { if (err) { return handleError(res, err); } next(); }); }); }); }; +/** + * When a PUT request comes with a given Chwala ID, it means the intention is to + * override the Manticore copy with a completely new version of the document, because + * it was perhaps modified by other means unkown to Manticore. So a new first chunk is + * created and we stop tracking the old chunks. + */ exports.overwrite = function (req, res) { Document.findById(req.params.id, function (err, document) { if (err) { return handleError(res, err); } if (!document) { return res.json(404, document); } createFirstChunk(req.user, req.params.id, function (err, newFirstChunk) { if (err) { return handleError(res, err); } document.date = Date.now(); document.chunks = [newFirstChunk] document.markModified('chunks'); document.save(function (err, document) { if (err) { return handleError(res, err); } return res.send(200); }); }); }); } +/** + * Creates a new chunk from an in-memory document snapshot (most likely the current one), + * writes it to GridFS, streams it back to Chwala via a PUT request, and once that succeeds, + * creates a new chunk to track the file and appends it to the DB entry. + */ exports.createChunkFromSnapshot = function (document, snapshot, cb) { var chunkId = new mongoose.Types.ObjectId(), fileId = new mongoose.Types.ObjectId(); var writeStream = gfs.createWriteStream({ _id: fileId, filename: document.title + '_' + Date.now(), mode: 'w', chunkSize: 1024 * 4, content_type: 'application/vnd.oasis.opendocument.text', root: 'fs' }); writeStream.end(new Buffer(snapshot.data), function () { User.findById(document.creator._id, function (err, user) { if (err) { return cb(err); } uploadToServer( user, document._id, gfs.createReadStream({ _id: fileId }), function (err, response) { if (err) { return cb(err); } var chunk = new DocumentChunk({ _id: chunkId, sequence: snapshot.sequence, snapshot: { fileId: fileId, operations: snapshot.operations } }); chunk.save(function (err) { if (err) { return cb(err); } document.chunks.push(chunkId); document.markModified('chunks'); cb(null, chunk); }); } ); }); }); }; function handleError(res, err) { console.log(err); return res.send(500, err); } diff --git a/server/api/document/storage/local/index.js b/server/api/document/storage/local/index.js index 3839a7f..73ced95 100644 --- a/server/api/document/storage/local/index.js +++ b/server/api/document/storage/local/index.js @@ -1,147 +1,164 @@ 'use strict'; var multer = require('multer'); var mongoose = require('mongoose'); var Grid = require('gridfs-stream'); var Document = require('../../document.model').Document; var DocumentChunk = require('../../document.model').DocumentChunk; var Template = require('../../../template/template.model'); var gfs = Grid(mongoose.connection.db, mongoose.mongo); +/** + * Lists the available documents that have the user as either the creator or an editor + */ exports.index = function (req, res) { var userId = req.user._id; Document.find({ '$or': [ { 'creator': userId }, { 'editors': { '$in': [userId] } } ] }).populate('creator', 'name email').exec(function (err, documents) { if(err) { return handleError(res, err); } return res.json(200, documents); }); }; +/** + * Returns the document metadata for a given ID + */ exports.show = function(req, res) { Document.findById(req.params.id, function (err, document) { if(err) { return handleError(res, err); } if(!document) { return res.send(404); } return res.json(document); }); }; +/** + * Handles a multipart upload of one or more files, streams them into GridFS, + * creates chunks to track them, and creates Documents with the filenames as titles. + */ exports.upload = function (req, res, next) { multer({ upload: null, limits: { fileSize: 1024 * 1024 * 20, // 20 Megabytes files: 5 }, onFileUploadStart: function (file) { var chunkId = new mongoose.Types.ObjectId(), fileId = new mongoose.Types.ObjectId(); var firstChunk = new DocumentChunk({ _id: chunkId, snapshot: { fileId: fileId } }); var newDocument = new Document({ title: file.originalname, creator: req.user._id, chunks: [chunkId] }); this.upload = gfs.createWriteStream({ _id: fileId, filename: file.originalname, mode: 'w', chunkSize: 1024 * 4, content_type: file.mimetype, root: 'fs' }); this.upload.on('finish', function () { firstChunk.save(function (err) { if (!err) { newDocument.save(); } }); }); }, onFileUploadData: function (file, data) { this.upload.write(data); }, onFileUploadComplete: function (file) { this.upload.end(); } })(req, res, next); }; - +/** + * Creates a first-chunk whose snapshot data points to a template's GridFS ID, + * for a new Document with the same title as the template. + */ exports.createFromTemplate = function (req, res) { Template.findById(req.params.id, function (err, template) { if (err) { return handleError(res, err); } if (!template) { return res.send(404); } var chunkId = new mongoose.Types.ObjectId(); var firstChunk = new DocumentChunk({ _id: chunkId, snapshot: { fileId: template.fileId } }); var newDocument = new Document({ title: template.title, creator: req.user._id, chunks: [chunkId] }); firstChunk.save(function (err) { if (!err) { newDocument.save(function (err) { return res.json(201, newDocument); }); } }) }); }; +/** + * Takes an in-memory snapshot of the document and streams the ODT into GridFS. + * Creates a chunk to represent the new file and appends it to the DB entry. + */ exports.createChunkFromSnapshot = function (document, snapshot, cb) { var chunkId = new mongoose.Types.ObjectId(), fileId = new mongoose.Types.ObjectId(); var writeStream = gfs.createWriteStream({ _id: fileId, filename: document.title + '_' + Date.now(), mode: 'w', chunkSize: 1024 * 4, content_type: 'application/vnd.oasis.opendocument.text', root: 'fs' }); writeStream.write(new Buffer(snapshot.data), function () { writeStream.end(function () { var chunk = new DocumentChunk({ _id: chunkId, sequence: snapshot.sequence, snapshot: { fileId: fileId, operations: snapshot.operations } }); chunk.save(function (err) { if (err) { return cb(err); } document.chunks.push(chunkId); document.markModified('chunks'); cb(null, chunk); }); }); }); }; function handleError(res, err) { return res.send(500, err); } diff --git a/server/api/document/storage/webdav/index.js b/server/api/document/storage/webdav/index.js index 174a790..a43fa49 100644 --- a/server/api/document/storage/webdav/index.js +++ b/server/api/document/storage/webdav/index.js @@ -1,459 +1,515 @@ 'use strict'; var _ = require('lodash'); var dav = require('dav'); var async = require('async'); var mongoose = require('mongoose'); var https = require('https'); var fs = require('fs'); var url = require('url'); var path = require('path'); var Grid = require('gridfs-stream'); var multer = require('multer'); var request = require('request'); var querystring = require('querystring'); var crypto = require('crypto'); var config = require('../../../../config/environment'); var User = require('../../../user/user.model'); var Document = require('../../document.model').Document; var DocumentChunk = require('../../document.model').DocumentChunk; var Template = require('../../../template/template.model'); var gfs = Grid(mongoose.connection.db, mongoose.mongo); var serverUrl = config.storage.webdav.server, serverPath = config.storage.webdav.path; +/** + * Uses the auth encryption key key to decipher webdav password + */ function decrypt(password) { var decipher = crypto.createDecipher('aes-256-cbc', config.storage.webdav.key); return decipher.update(password, 'base64', 'utf8') + decipher.final('utf8'); } +/** + * Creates a new dav "client" which will use the users's credentials + * as authentication headers to talk to the WebDAV server. + */ function makeDavClient (user) { return new dav.Client( new dav.transport.Basic(new dav.Credentials({ username: user.webdav.username, password: decrypt(user.webdav.password) })), { baseUrl: serverUrl } ); } +/** + * GETs the file with the relevant href from the WebDAV server, + * authenticated as the user, and streams the retrieved file into GridFS, + * addresed by the specified fileId + */ function saveToGridFS(user, href, fileId, cb) { var file = gfs.createWriteStream({ _id: fileId, filename: href.split('/').pop(), mode: 'w', chunkSize: 1024 * 4, content_type: 'application/vnd.oasis.opendocument.text', root: 'fs' }); + // The dav library's API seems insufficient to retrieve files in this manner, + // so we just use the `request` API. request.get({ url: serverUrl + href, auth: { user: user.webdav.username, pass: decrypt(user.webdav.password) } }) .on('error', function (err) { cb(err); }) .pipe(file); file.on('finish', cb); } +/** + * The only way we have of making sure that a file is the same one we're referring to, + * both content-wise, is to take both the etag and last-modified metadata + * into account. We refer to this as a `contentId`. + */ function makeContentId(webdavDoc) { return webdavDoc.props.getetag + webdavDoc.props.getlastmodified; } // The WebDAV server is always authoritative. function synchronizeUserFilesToDB(user, webdavDocuments, objectCache, cb) { Document.find({ '$or': [ { 'creator': user._id }, { 'editors': { '$in': [user._id] } } ] }) .populate('creator', 'name email').exec(function (err, dbDocuments) { // Since we're doing things that could possibly invalidate DB documents, // we have to work with the live cache var persistenceQueue = []; var finalDocuments = []; dbDocuments.forEach(function (dbDoc) { var trackedDoc; if (dbDoc.creator._id.equals(user._id)) { trackedDoc = objectCache.getTrackedObject(dbDoc); finalDocuments.push(trackedDoc); } else { finalDocuments.push(dbDoc); return; } var sameHrefDoc = _.find(webdavDocuments, function (doc) { return doc.href === trackedDoc.webdav.href; }); var sameContentDoc = _.find(webdavDocuments, function (doc) { return makeContentId(doc) === trackedDoc.webdav.contentId; }); if (!sameContentDoc) { if (!sameHrefDoc) { // Document has been effectively deleted, take it offline and remove trackedDoc.live = false; persistenceQueue.push(function (cb) { trackedDoc.remove(cb); }); finalDocuments.pop(); } else { // Document has been modified, take it offline, throw away local changes, and update content ID trackedDoc.live = false; trackedDoc.chunks.length = 0; trackedDoc.markModified('chunks'); trackedDoc.webdav.contentId = makeContentId(sameHrefDoc); trackedDoc.date = Date.parse(sameHrefDoc.props.getlastmodified); trackedDoc.markModified('webdav'); persistenceQueue.push(function (cb) { trackedDoc.save(cb); }); } } else { if (!sameHrefDoc) { // Document has been moved, just update href and save without interrupting anything trackedDoc.webdav.href = sameContentDoc.href; trackedDoc.markModified('webdav'); persistenceQueue.push(function (cb) { trackedDoc.save(cb); }); } else { if (makeContentId(sameHrefDoc) === makeContentId(sameContentDoc)) { // nothing changed } else { // Document moved to sameContentDoc.props.href and there is a new doc at sameHrefDoc.href trackedDoc.webdav.href = sameContentDoc.href; trackedDoc.markModified('webdav'); persistenceQueue.push(function (cb) { trackedDoc.save(cb); }); } } } }); // Once modified documents are persisted, purge non-live documents from the cache async.parallel(persistenceQueue, function (err) { if (err) return cb(err); dbDocuments.forEach(function (dbDoc) { if (objectCache.isTracked(dbDoc) && !dbDoc.live) { objectCache.forgetTrackedObject(dbDoc); } }); return cb(null, finalDocuments); }); }); } +/** + * Return a de-duplicated flat list of files available on the WebDAV server + * and Documents that are available locally + */ exports.index = function (req, res) { var davClient = makeDavClient(req.user); // Retrieve a list of files and filter by mimetype davClient.send(dav.request.propfind({ depth: 'infinity', props: [ { name: 'getcontenttype', namespace: dav.ns.DAV }, { name: 'getetag', namespace: dav.ns.DAV }, { name: 'getlastmodified', namespace: dav.ns.DAV } ] }), serverPath) .then( function success(response) { var webdavDocuments = _(response) .filter(function (item) { return item.props.getcontenttype === 'application/vnd.oasis.opendocument.text'; }) .map(function (item) { item.href = querystring.unescape(item.href); return item; }) .value(); synchronizeUserFilesToDB(req.user, webdavDocuments, req.app.get('objectCache'), function (err, updatedDocuments) { if (err) return handleError(res, err); // Transform the webdavDocuments array into something more usable // The fake ID of this document is represented as a combination of it's href and content id webdavDocuments = webdavDocuments.map(function (doc) { return { _id: new Buffer(doc.href + '__' + makeContentId(doc)).toString('base64'), title: doc.href.split('/').pop(), creator: req.user.profile, date: Date.parse(doc.props.getlastmodified), webdav: { href: doc.href, contentId: makeContentId(doc) } }; }); // Merge the two document types, such that hrefs are unique in the resulting list. DB docs have priority here var mergedDocuments = _.uniq(updatedDocuments.concat(webdavDocuments), function (doc) { return (doc.webdav && doc.webdav.contentId) || makeContentId(doc); }); return res.json(200, mergedDocuments); }); }, function failure(error) { handleError(res, error); }); }; +/** + * Creates the first chunk for a Document by retrieving it from the WebDAV server + * and saving into GridFS. + */ function createFirstChunk(user, href, cb) { var chunkId = new mongoose.Types.ObjectId(), fileId = new mongoose.Types.ObjectId(); var firstChunk = new DocumentChunk({ _id: chunkId, snapshot: { fileId: fileId } }); saveToGridFS(user, href, fileId, function(err) { if (err) { return cb(err); } firstChunk.save(function (err) { cb(err, firstChunk); }); }); } +/** + * When a document is requested by and ID: + * 1. If the ID exists in the DB, return that. + * - If the ID does exist, but has no chunks, it had been invalidated. So, create a first-chunk. + * 2. If the ID parameter is a combination of `href` and `contentId`, return a + * document with the same `webdav.contentId` value in the DB. + * - If such a document is not found, retrieve the document metadata from the WebDAV + * server, create an entry for it in the DB, and then return it. + */ exports.show = function(req, res) { if (mongoose.Types.ObjectId.isValid(req.params.id)) { Document.findById(req.params.id, function (err, document) { if(err) { return handleError(res, err); } if(!document) { return res.send(404); } if (document.chunks.length) { return res.json(200, document); } // If the document had been invalidated, it has no chunks, so generate one createFirstChunk(req.user, document.webdav.href, function (err, firstChunk) { if (err) { return handleError(res, err); } document.chunks.push(firstChunk); document.save(function () { res.json(200, document); }) }); }); } else { var identifier = new Buffer(req.params.id, 'base64').toString('ascii').split('__'), href = identifier[0], contentId = identifier[1]; Document.findOne({ 'webdav.contentId': contentId, }, function (err, document) { if (err) { return handleError(res, err); } if (document) { return res.json(200, document); } else { var davClient = makeDavClient(req.user); davClient.send(dav.request.propfind({ props: [] }), href) .then( function success(response) { var webdavDoc = response[0]; createFirstChunk(req.user, href, function(err, firstChunk) { if (err) { return handleError(res, err); } Document.create({ title: href.split('/').pop(), creator: req.user._id, created: Date.parse(webdavDoc.props.getlastmodified), date: Date.parse(webdavDoc.props.getlastmodified), chunks: [firstChunk._id], webdav: { href: href, contentId: makeContentId(webdavDoc) } }, function (err, document) { if (err) { return handleError(res, err); } res.json(201, document); }); }) }, function failure(error) { res.send(404, error); }); } }); } }; +/** + * Authenticated as a user, upload a document to the WebDAV server, with + * an option to replace/overwrite it. + */ function uploadToServer(user, readStream, href, replace, cb) { var nonConflictingPath; function upload () { readStream.pipe(request.put({ url: nonConflictingPath, auth: { user: user.webdav.username, pass: decrypt(user.webdav.password) }, headers: { 'Content-Type': 'application/vnd.oasis.opendocument.text' } }, cb)); } if (replace) { nonConflictingPath = serverUrl + href; upload(); } else { + // Check for the existence of a conflicting path makeDavClient(user).send(dav.request.propfind({ depth: 1, props: [ { name: 'getcontenttype', namespace: dav.ns.DAV }, { name: 'getetag', namespace: dav.ns.DAV }, { name: 'getlastmodified', namespace: dav.ns.DAV } ] }), path.dirname(href)) .then( function success(response) { function makePath(dir, basename, ext) { return dir + '/' + basename + ext; } var files = _(response) .filter(function (item) { return item.props.getcontenttype === 'application/vnd.oasis.opendocument.text'; }).map(function (item) { item.href = querystring.unescape(item.href); return item; }).value(); + // Generate a new href for storage of the file, so that it does not + // conflict with an existing file. var iteration = 0, extension = path.extname(href), basename = path.basename(href, extension), basename_i = basename, dirname = path.dirname(href); for (var i = 0; i < files.length; i++) { if (files[i].href === makePath(dirname, basename_i, extension)) { iteration++; basename_i = basename + ' (' + iteration + ')'; i = -1; } } nonConflictingPath = serverUrl + makePath(dirname, basename_i, extension); upload(); }, function failure(err) { cb(err); }); } } +/** + * Handle multipart file uploads done through the Manticore UI. These files are + * immediately uploaded to the WebDAV server's `serverPath` directory. + */ exports.upload = function (req, res, next) { multer({ upload: null, limits: { fileSize: 1024 * 1024 * 20, // 20 Megabytes files: 5 }, onFileUploadComplete: function (file) { uploadToServer(req.user, fs.createReadStream(file.path), serverPath + '/' + file.originalname, false, function (err) { if (err) { console.log (err); } } ); } })(req, res, next); }; +/** + * Create a new Document from a registered template, and upload it to the WebDAV + * server, without replacing/overwriting any file of the same name. + */ exports.createFromTemplate = function (req, res) { Template.findById(req.params.id, function (err, template) { if (err) { return handleError(res, err); } if (!template) { return res.send(404); } var chunkId = new mongoose.Types.ObjectId(); var firstChunk = new DocumentChunk({ _id: chunkId, snapshot: { fileId: template.fileId } }); var newDocument = new Document({ title: template.title, creator: req.user._id, chunks: [chunkId] }); uploadToServer( req.user, gfs.createReadStream({ _id: template.fileId }), serverPath + '/' + template.title + '.odt', false, function (err, response) { if (err) { return handleError(res.err); } newDocument.webdav = { href: response.request.uri.path, contentId: response.headers.etag + response.headers.date }; firstChunk.save(function (err) { if (!err) { newDocument.save(function (err) { return res.json(201, newDocument); }); } }); } ); }); }; +/** + * Take an ODT snapshot, write it to GridFS, and upload it to the WebDAV server, + * overwriting any exisiting file at the same href. Creates a new chunk. + * This is intended to be used when the 'save' action is invoked. + */ exports.createChunkFromSnapshot = function (document, snapshot, cb) { var chunkId = new mongoose.Types.ObjectId(), fileId = new mongoose.Types.ObjectId(); var writeStream = gfs.createWriteStream({ _id: fileId, filename: document.title + '_' + Date.now(), mode: 'w', chunkSize: 1024 * 4, content_type: 'application/vnd.oasis.opendocument.text', root: 'fs' }); writeStream.end(new Buffer(snapshot.data), function () { User.findById(document.creator._id, function (err, user) { if (err) { return cb(err); } uploadToServer( user, gfs.createReadStream({ _id: fileId }), document.webdav.href, true, function (err, response) { if (err) { return cb(err); } var chunk = new DocumentChunk({ _id: chunkId, sequence: snapshot.sequence, snapshot: { fileId: fileId, operations: snapshot.operations } }); chunk.save(function (err) { if (err) { return cb(err); } document.webdav.contentId = response.headers.etag + response.headers.date; document.chunks.push(chunkId); document.markModified('chunks'); cb(null, chunk); }); }); }); }); }; function handleError(res, err) { console.log(err); return res.send(500, err); } diff --git a/server/components/adaptor/recorder.js b/server/components/adaptor/recorder.js index e10304c..3be7ccc 100644 --- a/server/components/adaptor/recorder.js +++ b/server/components/adaptor/recorder.js @@ -1,145 +1,148 @@ var phantom = require('phantom'); var fs = require('fs'); var EventEmitter = require('events').EventEmitter; var config = require('../../config/environment'); var Recorder = function (lastChunk, cb) { var self = this, manticoreHost = 'http://' + (config.ip || 'localhost') + ':' + config.port, baseSnapshotUrl = manticoreHost + '/api/documents/snapshot/' + lastChunk.snapshot.fileId, emitter = new EventEmitter(), snapshotReadyCb = function () {}, ph, page; function phantomCallback(data) { function documentLoaded(data) { cb(); } switch(data.event) { case 'log': console.log('PhantomJS Log :', data.message); break; case 'documentLoaded': documentLoaded(data); break; default: emitter.emit(data.event, data); } } + /** + * The following block is meant to run in the Phantom webpage context + */ /* jshint ignore:start */ function loadDocument(url, sequence, operations) { window.console.log = function (message) { window.callPhantom({event: 'log', message: message}); }; var odfElement = document.getElementById('odf'); document.odfCanvas = new odf.OdfCanvas(odfElement); document.odfCanvas.addListener('statereadychange', function () { // The "sequence" is the number of ops executed so far in the canonical document history window.sequence = sequence; document.session = new ops.Session(document.odfCanvas); document.odtDocument = document.session.getOdtDocument(); var operationRouter = new ops.OperationRouter(); operationRouter.push = function (opspecs) { var op, i; if (!opspecs.length) { return; } for (i = 0; i < opspecs.length; i++) { op = document.session.getOperationFactory().create(opspecs[i]); if (op && op.execute(document.session.getOdtDocument())) { window.sequence++; // console.log('Just executed op ' + opspecs[i].optype + 'by ' + opspecs[i].memberid); } else { break; } } }; document.session.setOperationRouter(operationRouter); document.session.enqueue(operations); window.callPhantom({event: 'documentLoaded'}); }); document.odfCanvas.load(url); } /* jshint ignore:end */ this.push = function (operations, cb) { page.evaluate(function (opspecs) { document.session.enqueue(opspecs); }, function () { cb(); }, operations); }; this.getSnapshot = function (cb) { var eventId = Date.now(); emitter.once('snapshotReady' + eventId, function (data) { cb(data); }); page.evaluate(function (eventId) { var doc = document.odtDocument, ops = []; doc.getMemberIds().forEach(function (memberId) { ops.push({ optype: 'AddMember', timestamp: Date.now(), memberid: memberId, setProperties: doc.getMember(memberId).getProperties() }); var cursor = doc.getCursor(memberId); if (cursor) { var selection = doc.getCursorSelection(memberId); ops.push({ optype: 'AddCursor', timestamp: Date.now(), memberid: memberId }); ops.push({ optype: 'MoveCursor', timestamp: Date.now(), memberid: memberId, position: selection.position, length: selection.length, selectionType: cursor.getSelectionType() }); } }); document.odfCanvas.odfContainer().createByteArray(function (data) { window.callPhantom({ event: 'snapshotReady' + eventId, operations: ops, data: data, sequence: window.sequence }); }); }, function () {}, eventId); }; this.destroy = function (cb) { ph.exit(); cb(); }; function init() { phantom.create('--web-security=no', function (instance) { ph = instance; ph.createPage(function (p) { p.open('file://' + __dirname + '/odf.html?host=' + manticoreHost, function (status) { if (status === 'success') { page = p; page.set('onCallback', phantomCallback); page.evaluate(loadDocument, function (){}, baseSnapshotUrl, lastChunk.sequence - lastChunk.snapshot.operations.length, lastChunk.snapshot.operations.concat(lastChunk.operations)); } else { return cb(new Error('Could not initialize recorder module.')); } }); }); }); } init(); }; module.exports = Recorder; diff --git a/server/components/adaptor/room.js b/server/components/adaptor/room.js index cb3c9df..d83b9ee 100644 --- a/server/components/adaptor/room.js +++ b/server/components/adaptor/room.js @@ -1,484 +1,558 @@ /* * Copyright (C) 2015 KO GmbH * * @licstart * This file is part of Kotype. * * Kotype is free software: you can redistribute it and/or modify it * under the terms of the GNU Affero General Public License (GNU AGPL) * as published by the Free Software Foundation, either version 3 of * the License, or (at your option) any later version. * * Kotype is distributed in the hope that it will be useful, but * WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU Affero General Public License for more details. * * You should have received a copy of the GNU Affero General Public License * along with Kotype. If not, see . * @licend * * @source: https://github.com/kogmbh/Kotype/ */ var async = require("async"); var _ = require("lodash"); var RColor = require('../colors'); var DocumentChunk = require("../../api/document/document.model").DocumentChunk; var DocumentController = require("../../api/document/document.controller"); var Recorder = require('./recorder'); +/** + * Handles the adding/removal of members in an editing session, relays operations, + * performs cleanup after dropouts, and talks to the client with socketio messages + */ var Room = function (app, document, objectCache, cb) { var self = this; + /** + * Handles chunking of operations to a document - including retrieval + * and appending of new ops across several chunks + */ var ChunkManager = function (seedChunk) { var serverSeq, chunks = []; + /** + * Starts at `seq` sequence number and returns an array of all operations + * after that, spanning one or more chunks + */ function getOperationsAfter(seq) { var ops = []; for (var i = chunks.length - 1; i >= 0; i--) { if (chunks[i].sequence >= seq) { ops = chunks[i].operations.concat(ops); } else { var basedOn = seq - chunks[i].sequence; ops = chunks[i].operations.slice(basedOn).concat(ops); break; } } return ops; } this.getOperationsAfter = getOperationsAfter; function appendOperations(ops) { var lastChunk = getLastChunk(); lastChunk.operations = lastChunk.operations.concat(ops); serverSeq += ops.length; } this.appendOperations = appendOperations; function appendChunk(chunk) { var trackedChunk = objectCache.getTrackedObject(chunk); chunks.push(trackedChunk); serverSeq = trackedChunk.sequence + trackedChunk.operations.length; } this.appendChunk = appendChunk; function getLastChunk() { return _.last(chunks); } this.getLastChunk = getLastChunk; this.getServerSequence = function () { return serverSeq; }; appendChunk(seedChunk); }; var chunkManager, recorder, hasCursor = {}, sockets = [], userColorMap = {}, randomColor = new RColor(), saveInProgress = false, isAvailable = false; + /** + * Synchronizes the title of the Document in the DB and any changes to the ODF + * metadata title + */ function trackTitle(ops) { var newTitle, i; for (i = 0; i < ops.length; i += 1) { if (ops[i].optype === "UpdateMetadata" && ops[i].setProperties["dc:title"] !== undefined) { newTitle = ops[i].setProperties["dc:title"]; } } if (newTitle !== undefined) { if (newTitle.length === 0) { newTitle = "Untitled Document"; } } if (newTitle) { document.title = newTitle; } } + /** + * Synchronizes the list of people who have joined the session to The + * `editors` field of the Document entry in the DB + */ function trackEditors() { // TODO: rather track by ops, to decouple from socket implementation sockets.forEach(function (socket) { var _id = socket.user._id; if (document.editors.indexOf(_id) === -1) { document.editors.push(_id); } }); } + /** + * Inspects the array of operations and updates the hasCursor fields for each + * user + */ function trackCursors(ops) { var i; for (i = 0; i < ops.length; i += 1) { if (ops[i].optype === "AddCursor") { hasCursor[ops[i].memberid] = true; } if (ops[i].optype === "RemoveCursor") { hasCursor[ops[i].memberid] = false; } } } - // Removes all cursors and members in the correct order within the last chunk + /** + * Removes all cursors and members in the correct order within the last chunk + * It may have happen that a user drops out due to a connection loss, error, or + * if Manticore is uncleanly terminated. + * In that case, this function will make sure to remove their cursors from the + * document and then remove the "member" via the relevant operations. + */ function sanitizeDocument() { var chunk = chunkManager.getLastChunk(), ops = chunk.snapshot.operations.concat(chunk.operations), unbalancedCursors = {}, unbalancedMembers = {}, lastAccessDate = document.date, newOps = [], i; for (i = 0; i < ops.length; i += 1) { if (ops[i].optype === "AddCursor") { unbalancedCursors[ops[i].memberid] = true; } else if (ops[i].optype === "RemoveCursor") { unbalancedCursors[ops[i].memberid] = false; } else if (ops[i].optype === "AddMember") { unbalancedMembers[ops[i].memberid] = true; } else if (ops[i].optype === "RemoveMember") { unbalancedMembers[ops[i].memberid] = false; } } Object.keys(unbalancedCursors).forEach(function (memberId) { if (unbalancedCursors[memberId]) { newOps.push({ optype: "RemoveCursor", memberid: memberId, timestamp: lastAccessDate }); } }); Object.keys(unbalancedMembers).forEach(function (memberId) { if (unbalancedMembers[memberId]) { newOps.push({ optype: "RemoveMember", memberid: memberId, timestamp: lastAccessDate }); } }); if (newOps.length) { // Update op stack chunkManager.appendOperations(newOps); } } + /** + * Broadcast a socketio message to all clients + */ function broadcastMessage(message, data) { sockets.forEach(function (peerSocket) { peerSocket.emit(message, data) }); } + /** + * Send an array of operations to a given member, referred to by it's socket + */ function sendOpsToMember(socket, ops) { socket.emit("new_ops", { head: chunkManager.getServerSequence(), ops: ops }); } + /** + * Sends an array of operations to a given member, which are necessary to + * bring the snapshot provided to them to the latest state of the document + */ function setupMemberSnapshot(socket, snapshot) { socket.emit("replay", { head: chunkManager.getServerSequence(), ops: snapshot.operations.concat(chunkManager.getOperationsAfter(snapshot.sequence)) }); } + /** + * Send ops authored by a given member to all other members except that one + */ function broadcastOpsByMember(socket, ops) { if (!ops.length) { return; } sockets.forEach(function (peerSocket) { if (peerSocket.memberId !== socket.memberId) { sendOpsToMember(peerSocket, ops); } }); } + /** + * Takes the array of incoming ops, plays them on the serverside live document, + * and if the execution succeeds without errors, adds them to the document's latest chunk, + * while making sure to update the relevant Document metadata based on the ops content + */ function writeOpsToDocument(ops, cb) { if (!ops.length || !document.live) { cb(); } recorder.push(ops, function () { trackTitle(ops); trackEditors(); // Update op stack chunkManager.appendOperations(ops); // Update modified date document.date = new Date(); cb(); }); } + /** + * Assigns a unique WebODF memberId to a new user, and adds them to the document. + * This lets the same user join the same session several times, for example through + * different devices. + * A random color is assigned to the member, and we try to make the color + * sufficiently distinguishable from all the other colors present in the session + */ function addMember(user, cb) { var memberId, op, timestamp = Date.now(), color = userColorMap[user._id]; memberId = user.name + "_" + timestamp.toString(); // Let user colors persist in a Room even after they've // left and joined. if (!color) { userColorMap[user._id] = color = randomColor.get(true, 0.7); } op = { optype: "AddMember", memberid: memberId, timestamp: timestamp, setProperties: { fullName: user.name, email: user.email, color: color } }; writeOpsToDocument([op], function () { cb(memberId, [op]); }); } + /** + * Removes a user instance from the sessions, by first issuing an op + * to remove their cursor, and then their member. + */ function removeMember(memberId, cb) { var ops = [], timestamp = Date.now(); if (hasCursor[memberId]) { ops.push({ optype: "RemoveCursor", memberid: memberId, timestamp: timestamp }); } ops.push({ optype: "RemoveMember", memberid: memberId, timestamp: timestamp }); writeOpsToDocument(ops, function () { cb(ops); }); } this.socketCount = function () { return sockets.length; }; + /** + * Runs everything that's required bring a new socket up to speed on the + * latest state of the document, byt adding their member to the session, + * then extracting the latest snapshot from the PhantomJS recorder, + * and serving that snapshot out as the editor's Genesis URL. + */ this.attachSocket = function (socket) { // Add the socket to the room and give the // client it's unique memberId addMember(socket.user, function (memberId, ops) { socket.memberId = memberId; sockets.push(socket); broadcastOpsByMember(socket, ops); // Generate genesis URL with the latest document version's snapshot // We use a two-time URL because WebODF makes two identical GET requests (!) var genesisUrl = '/genesis/' + Date.now().toString(), usages = 0; recorder.getSnapshot(function (snapshot) { var buffer = new Buffer(snapshot.data); app.get(genesisUrl, function (req, res) { usages++; res.set('Content-Type', 'application/vnd.oasis.opendocument.text'); res.attachment(document.title); res.send(buffer); var routes = app._router.stack; if (usages === 2) { buffer = null; for (var i = 0; i < routes.length; i++) { if (routes[i].path === genesisUrl) { routes.splice(i, 1); break; } } } }); + /** + * Inform the client that the socket has joined the session and + * with the returned permission type, so they have enough info + * on how to initialize the editor + */ socket.emit("join_success", { memberId: memberId, genesisUrl: genesisUrl, permission: document.getAccessType(socket.user.email) }); // Service replay requests socket.on("replay", function () { setupMemberSnapshot(socket, snapshot); }); // Store, analyze, and broadcast incoming commits socket.on("commit_ops", function (data, cb) { var clientSeq = data.head, ops = data.ops; if (clientSeq === chunkManager.getServerSequence()) { writeOpsToDocument(ops, function () { cb({ conflict: false, head: chunkManager.getServerSequence() }); trackCursors(ops); broadcastOpsByMember(socket, data.ops); }); } else { cb({ conflict: true }); } }); // Service save requests. A save is a commit + socket.on("save", function (cb) { // Saves are blocking inside the phantomjs process, and they affect everyone, // therefore use a lock. if (saveInProgress) { var checkIfSaved = setInterval(function () { if (!saveInProgress) { clearInterval(checkIfSaved); cb(); } }, 1000); } else { saveInProgress = true; recorder.getSnapshot(function (snapshot) { DocumentController.createChunkFromSnapshot(document, snapshot, function (err, chunk) { saveInProgress = false; if (err) { return cb(err); } chunkManager.appendChunk(chunk); cb(); }); }); } }); // Service various requests socket.on("access_get", function (data, cb) { cb({ access: document.isPublic ? "public" : "normal" }); }); if (socket.user.identity !== "guest") { socket.on("access_change", function (data) { document.isPublic = data.access === "public"; broadcastMessage("access_changed", { access: data.access === "public" ? "public" : "normal" }); if (data.access !== "public") { sockets.forEach(function (peerSocket) { if (peerSocket.user.identity === "guest") { console.log(peerSocket.user.name); removeSocket(peerSocket); } }); } }); } }); // Handle dropout events socket.on("leave", function () { removeSocket(socket); }); socket.on("disconnect", function () { removeSocket(socket); }); }); }; function detachSocket(socket, callback) { removeMember(socket.memberId, function (ops) { broadcastOpsByMember(socket, ops); socket.removeAllListeners(); function lastCB() { socket.removeAllListeners(); if (callback) { callback(); } } // If a socket that is already connected is being // removed, this means that this is a deliberate // kicking-out, and not a natural event that could // result in a reconnection later. Therefore, clean // up. if (socket.connected) { console.log(socket.user.name + " is connected, removing"); socket.on('disconnect', lastCB); socket.emit("kick"); socket.emit("disconnect"); } else { console.log(socket.user.name + " is not connected, removing"); lastCB(); } }); } + /** + * Detach a connected socket and then remove it from the list of known sockets, + * so a reconnect is never attempted again + */ function removeSocket(socket) { var index = sockets.indexOf(socket); detachSocket(socket); if (index !== -1) { sockets.splice(index, 1); } } this.getDocument = function () { return document; }; this.isAvailable = function () { return isAvailable; }; this.destroy = function (callback) { async.each(sockets, function (socket, cb) { detachSocket(socket, cb); }, function () { //objectCache.forgetTrackedObject(chunk); delete app.get('roomCache')[document._id]; document.live = false; sockets.length = 0; recorder.destroy(callback); }); }; function init() { // Setup caching DocumentChunk.findById(_.last(document.chunks), function (err, lastChunk) { chunkManager = new ChunkManager(lastChunk); // Sanitize leftovers from previous session, if any sanitizeDocument(); recorder = new Recorder(chunkManager.getLastChunk(), function () { isAvailable = true; app.get('roomCache')[document._id] = self; cb(); }); }); } init(); }; module.exports = Room; diff --git a/server/config/local.env.sample.js b/server/config/local.env.sample.js index 17adeb1..9f5e3f7 100644 --- a/server/config/local.env.sample.js +++ b/server/config/local.env.sample.js @@ -1,73 +1,73 @@ 'use strict'; // Use local.env.js for environment variables that grunt will set when the server starts locally. // Use for your api keys, secrets, etc. This file should not be tracked by git. // // You will need to set these on the server you deploy to. module.exports = { DOMAIN: 'http://localhost:9000', SESSION_SECRET: 'manticore-secret', // Control debug level for modules using visionmedia/debug DEBUG: '', /* * Default access permissions for documents. * If a user has a link to a session and tries to open it, this represents the * access type they have if no permission has been explicitly set for them. * Possible values: 'write', 'read', 'deny'. * By default, is set to 'write' for testing purposes. * If completely outsourcing access control to a third party service (like Kolab), set it to 'deny'. * If left blank, defaults to 'deny'. */ - DEFAULT_ACCESS: 'allow', + DEFAULT_ACCESS: 'write', /* * Supported authentication strategies. * 1. 'local' for using Manticore's built-in accounts system. Allow signups. * 2. 'webdav' for authenticating against a WebDAV server. Only login, no signups. * 3. 'ldap' for authenticating against an LDAP service. Only login, no signups. */ AUTH: 'local', /* * Supported storage backends. * 1. 'local' for storing everything in Mongo/GridFS. The fastest and most reliable way. * Can be used with any AUTH strategy. * 2. 'webdav' for two-way synchronizing of documents with a WebDAV server. * Can be used if AUTH is 'ldap' or 'webdav'; those credentials are used to talk to the storage server. * 3. 'chwala' can be used for integrating with Kolab. */ STORAGE: 'local', /* * WebDAV server config, only if AUTH or STORAGE is 'webdav'. */ WEBDAV_SERVER: 'https://demo.owncloud.org', WEBDAV_PATH: '/remote.php/webdav', /* * When using Chwala storage, it is expected that Manticore will be embedded within Roundcube, * so make sure you provide the host for the Roundcube server. This is intended for safe * cross-origin communication. */ CHWALA_SERVER: 'http://172.17.0.12', ROUNDCUBE_SERVER: 'http://172.17.0.12', /* * Make sure you provide an encryption key to protect users' auth credentials. * This is necessary because the storage server may not support authentication tokens. */ AUTH_ENCRYPTION_KEY: 'suchauth123muchkey456', // LDAP server config, only if AUTH is 'ldap' LDAP_SERVER: 'ldap://172.17.0.12', LDAP_BASE: 'ou=People,dc=example,dc=org', LDAP_FILTER: '(&(objectclass=person)(|(uid={{username}})(mail={{username}})))', LDAP_BIND_DN: 'uid=binderservice,ou=Special Users,dc=example,dc=org', LDAP_BIND_PW: 'binderpass', // locodoc Server config LOCODOC_SERVER: 'http://localhost:3030' };