diff --git a/source/developer-guide/process.rst b/source/developer-guide/process.rst new file mode 100644 --- /dev/null +++ b/source/developer-guide/process.rst @@ -0,0 +1,500 @@ +.. _developer-guide-process: + +=================== +Development Process +=================== + +.. todo:: + + Describe continuous integration + +.. todo:: + + Describe the difference between build, unit, functional, install, + integration, performance and acceptance tests. + +.. todo:: + + Describe continuous deployment + +.. todo:: + + Describe how to create a ticket + +.. todo:: + + Describe how to submit a patch + +.. todo:: + + Describe how to review and accept or decline a patch + +.. todo:: + + Describe how to formulate a feature request + +Kolab Groupware development largely follows an iterative and incremental +agile software development metholodogy also known as `Scrum`_. + +In summary, the Kolab Groupware development process is divided into +stages, with different teams participating in getting tasks to the next +stage of the process. + +Kolab Groupware is a collaboration suite consisting of many components, +each of them separate software development projects, some of them +dependent on third-party software development projects, and some of them +separate altogether. This creates some discrepancies in applying vanilla +Scrum practices to all projects in the Kolab universe. + +Furthermore, Kolab Groupware is a community software development +project, and not all participants in the community can be expected to be +familiar with (otherwise professional) software development processes. + +It is therefore that we document here how we work with `Scrum`_ using +`Phabricator`_. + +Benefits of Phabricator, Scrum and Test-Driven Development +========================================================== + +A plethora of benefits exist in using Phabricator and Scrum over the +current processes (which do not fall under any known nomenclature), and +Test-Driven Development (TDD). + +* The processes and facilities are focused on structured software + development. + +* Software development stands in stark contrast with longer-term + commitments such as support. + +* A developer is supposed to work on tomorrow's version of the + software, and not yesterday's, as every distraction with yesterday's + software stands in the way of the focus and dedication needed to + provide tomorrow's innovative and competitive software. + +* And every distraction stands in the way of the personal and + professional development of the individual developer. + +* We aim to pull product support away from software development. + +* A Bugzilla is not suitable for agile software development (even with + tooling such as Scrumbugz), and the former point may have + illustrated we wish software development to be agile. + +* Various factors in the world of IT as well as common software + development practices predicate a shift toward continuous deployment + (at one cost) over sometimes overly precautious long-term support + methodologies (at another cost). + +* The dynamics of these costs work in favor of a level of agility in + software development as well as operations, especially with smaller + teams. + +Getting Acquainted with Phabricator +=================================== + +We use `Projects`_ in `Phabricator`_ as a means to group people and +assign them roles, rights and responsibilities in a broader process. + +Tickets are created as `Maniphest`_ tasks, and roll on forward through +the process from thereon. + +Software Development Projects +----------------------------- + +The first type of project is a **Software Development Project**. These +are generally associated with one or more source code management +repositories. + +Each software development project is responsible for a single software +component included in Kolab Groupware, such as **chwala** or +**pykolab**. + +Because the Kolab community is an open community, membership to these +software development projects is *open* -- meaning registered users can +join and leave as they like. + +Authorization Groups (Developer Teams) +-------------------------------------- + +Each software development project that operates source code management +repositories also has a group associated with it that is used for +authorization to push to the associated source code management +repositories. + +For the **PyKolab** software development project for example, a group +**PyKolab Developers** has access to push to the ``pykolab`` GIT +repository. + +These groups are mostly self-regulated, in that members of the group can +add other members to the group. + +Teams and Roles +--------------- + +Teams and roles that are not software development teams include teams +such as `Product Owners`_, `Scrum Masters`_ and `Quality Assurance`_. + +Sprints +------- + +`Sprints`_ are what is in the name. They are a time-limited series of +events resulting in deliverables being emitted at the end of the sprint. + +This series of events consists of: + +**Planning Meeting** + + Held at the beginning of each sprint, the `Product Owners`_ and the + developer team negotiate which tasks on the Sprint's backlog they + will attempt to complete. + + The `Product Owners`_ determine the priority of tasks and declare + which (of all tasks) render the most business value. + + The development team determines the amount of work they feel they + can achieve during the sprint without incurring technical debt. + + .. NOTE:: + + The `Product Owners`_ use tasks related to individual software + development projects and associate them with the **next** + sprint in order to build an initial sprint backlog. + + Working the other way around, where the team in collaboration + with the `Product Owners`_ determine which tasks to put on the + sprint backlog doesn't scale to the number of individual + software development projects for Kolab Groupware as a whole. + +**Daily Scrums** + + Short, daily stand-up meetings are used to let team members + summarize yesterday's work, today's work and talk about any + impediments encountered. + + This ensures team members communicate early and often about their + work, problems they find, and facilitate the fast exchange of + knowledge and ideas or call for help. + +**Review Meeting** + + + +**Retrospective Meeting** + +Process Stages +============== + +The stages of the development process largely align with a manufacturing +process of, say, cars. + +.. _developer-guide-process-stage-concept: + +Concept +------- + +This stage is an almost completely iterative process; Ideas are +generated, possibly prototyped and thrown away on a regular basis. + +As such, concepts may or may not make it to production. In keeping with +the analogy of a car manufacturer, this may be a design drawing for a +new car (up to and including showing off a prototype on a trade show), +or some new technology (stage 2 artificial intelligence). + +For Kolab Groupware, this stage mostly involves research and +development not necessarily based on consumer input. + +The output of this process (if at all "valid") is additional input to +the next stage, :ref:`developer-guide-process-stage-pre-production`. + +.. _developer-guide-process-stage-pre-production: + +Pre-Production +-------------- + +The pre-production stage is an interaction between consumers of the +product, `Product Owners`_, `Architecture & Design`_ and +`Quality Assurance`_. + +:ref:`developer-guide-process-roles-consumers` of the product request +features or report issues, that end up on the *Product Backlog* [#]_. +`Product Owners`_ prioritize these tasks. + +Depending on the complexity of the task, and with the existing need to +facilitate the smoothest possible execution of the **Production** phase, +`Product Owners`_ and `Scrum Masters`_ request a review to be aided by +the `Architecture & Design`_ and `Quality Assurance`_ teams. + +The input for this stage may be anything from UI elements with the wrong +teint (easy to forward to **Production** as-is) to complex enhancements +to Kolab Groupware as a whole, requiring design and possibly touching +the codebase of multiple software projects (such as the implementation +of Federated Central Authentication Services with Two-Factor +Authentication support). + +Regardless, the effort at this stage involves breaking down tasks in to +digestible sub-tasks ready for **Production**. + +Production +---------- + +Production is the stage where development work takes place. The form in +which is this happens is through `Sprints`_, which are limited chunks of +time in which several events take place that shape the development +process for :ref:`developer-guide-process-roles-developers`. + +The input for this stage has initially been composed during the previous +sprint, with `Product Owners`_ putting tasks from dozens of software +development project backlogs on to the backlog for the current sprint. + +Note that at this point, most facets of tasks have already been +negotiated -- but only to a certain extent. + +If a relatively small task involves changes in one software project +dependent on changes in another software project, then this should have +been settled either before the individual sub-tasks end up on the sprint +backlog, or happens during the sprint planning meeting. + +Post-Production +--------------- + +Roles +===== + +.. _developer-guide-process-roles-stakeholders: + +Stakeholders +------------ + +Stakeholders are parties interested in the product, and are responsible +for the creation of feature implementation requests, bug reports and +acceptance testing. + +As such, stakeholders include customers (of Kolab Systems and its +partners), but actually include all consumers of Kolab. In other words, +regardless of affiliation or status, while you are reading this you +are probably a stakeholder. + +However, some stakeholders' wishes have more weight than others. This +is likely reflected (at most) in the priority of the task(s) at hand, +which dictates the order in which tasks are worked on. + +.. _developer-guide-process-role-product-owner: + +Product Owner +------------- + +The role of a product owner (usually a singular entity) is to determine +the value of goals (set out as epics and tasks in `Maniphest`_) and +prioritize the associated epics and tasks accordingly. + +When the product owner does so, he or she determines the value of goals +on behalf of the stakeholders, and in light of an overall vision on +where the product should go. + +Within the Kolab Groupware universe, where the *product* is the Kolab +release, and dozens of individual software development projects make up +the *product*, several aspects make the `Product Owners`_ a team of +individuals rather than a single person: + +* Not all software development projects are related to the product. + + Such may include work on Phabricator or the Sprint extension to + Phabricator (i.e. facilitatory third-party software). + +* Not all software development projects are under the control of the + business. + + This may include third-party software development projects we + contribute to (i.e. Roundcube). + +However in the context of `Scrum`_ and particularly the `Sprints`_, +there is only one stakeholder representative -- **the** Product Owner. + +.. _developer-guide-process-roles-scrum-masters: + +Scrum Masters +------------- + +`Scrum Masters`_ facilitate the Scrum process ... + +.. rubric:: Checklist for the Beginning and End of a Sprint + +#. Remove the hash-tag `sprint_server_current` from the past sprint, +#. Remove the hash-tag `sprint_server_next` from the current sprint, +#. Add the hash-tag `sprint_server_current` to the current sprint, +#. Create a new sprint, +#. Set the start and end dates for the new sprint, +#. Add the hash-tag `sprint_server_next` to the next sprint. + +.. _developer-guide-process-roles-architecture-and-design: + +Architecture & Design +--------------------- + +.. _developer-guide-process-roles-quality-assurance: + +Quality Assurance +----------------- + +.. _developer-guide-process-roles-developers: + +Developers +---------- + +.. _developer-guide-process-roles-consumers: + +Consumers +--------- + +* A user or customer requests the resolution of a certain issue. + +* The product owner(s) + +Continuous Integration, Delivery & Deployment +============================================= + +.. graphviz:: + + digraph { + rankdir = LR; + splines = true; + overlab = prism; + + edge [color=gray50, fontname=Calibri, fontsize=11]; + node [shape=record, fontname=Calibri, fontsize=11]; + + "contributor" -> "Phabricator"; + + "Phabricator" -> "Build Host"; + + "Build Host" -> "Docker Container(s)"; + + } + +#. A *contributor* contributes code via `Differential`_ to create or + update a code review proposal, or `Diffusion`_ to commit directly + to the GIT repository for the Software Development Project. + +#. `Herald`_, the `Phabricator`_ rule engine, is used to trigger the + execution of a `Harbormaster`_ build plan [#]_. + +#. `Harbormaster`_ allocates a `Drydock`_ host to execute a script on. + +#. The `Drydock`_ host's script is supplied with the repository and + the commit in question. + +#. The `Drydock`_ hosts used are `Atomic`_ hosts, meaning they run + `Docker`_ containers to execute Continuous Integration. + +#. The so-called entrypoint for the Docker images (executed when a + container is started from the image) is designed to execute + Continuous Integration and Continuous Delivery. + +Continuous Integration +---------------------- + +Continuous Integration is performed using the following stages; + +* A *build* stage in which the software build dependencies are + installed in to the Docker container, and the software is built. + + In the case of software that does not need to be compiled, such as + PHP and/or JavaScript, the *build* stage executes syntax checks. + +* A *unit testing* stage, which is limited to testing individual + functions of the software, such as ``sum(a, b)`` with ``a=1; b=2`` + indeed resulting in ``3``, but also does not yield unexpected + behaviour: + + * ``sum(a, b)`` over ``a=1`` correctly raises an argument + exception (if expected to), or returns ``1`` (if expected to), + + * ``sum(a, b)`` over ``a="Hello"`` and ``b="world!"``; + + * does not unintentionally result in ``"Hello world!"`` nor + ``"Helloworld!"``, + + * raises the correct exception, or returns false, none or the + equivalent of ``NaN``, + + * ``sum(a, b)`` over ``a=1; b="apple"`` does not result in ``1`` + (unless expected) and raises the correct exception (if expected + to). + + * *etcetera*, *etcetera*. + + Unit testing further includes testing if underlying software + behaves in a way that the software can handle correctly. This may + include tests that determine whether stripping invalid characters + from a string (by **iconv**) yields the expected result, and + whether the version of the API provided by underlying software is + sufficient. + + .. NOTE:: + + In cases where function calls require infrastructure (such as a + SQL server) with :term:`fixtures`, unit tests have to replace + the code that would interact with the infrastructure with code + that returns data as if the infrastructure existed. This is + called *mocking up*. + + Tests that do indeed require infrastructure, and/or cannot be + *mocked up*, are called *functional tests*. + +* A *functional testing* stage, in which functions of the software + are tested which require a minimal amount of infrastructure. + + This may include functions that imply a caching layer such as + **memcached** is available (populating a cache with fixtures, + marking it as dirty, and expecting retrieval to return no data). + + .. NOTE:: + + It should be noted that the distinction between *unit* and + *functional* testing is largely a matter of job control. + + It should also be noted that functional tests are executed + against a *minimal* infrastructure -- the keyword being + minimal. This usually precludes infrastructure needed to + authenticate users, and usually precludes testing functions + against various implementations of a piece of functionality + (such as an IMAP server needed by Roundcube testing possibly + being either of UW-IMAP, Courier, Cyrus IMAP, Dovecot, ...). + +* A *packaging* stage, in which the software is packaged up for + distribution. + +* An *installation testing* stage, in which the result of the + *packaging* is installed. + +* An *upgrade testing* stage, in which any number of former release's + fixtures are put in place, and the upgrade path is supposed to be + completed successfully automatically (such as a + :command:`rake db:migrate`). + +* An *integration testing* stage + +* An *acceptance testing* stage, in which human beings experience + the implementation. + +Continuous Delivery +------------------- + +.. _developer-guide-using-arcanist: + +Using Arcanist +============== + +.. rubric:: Footnotes + +.. [#] + + Kolab Groupware consists of dozens of individual product backlogs. + +.. [#] + + The sources for this script live in the Stick repository at + `drydocker.sh`_. + +.. _Scrum: http://en.wikipedia.org/wiki/Scrum_%28software_development%29 +.. _drydocker.sh: https://git.kolab.org/diffusion/QA/browse/master/drydocker.sh