Page MenuHomePhorge
Differential D89 Diff 198 source/developer-guide/process.rst

Changeset View

Standalone View

View Options

source/developer-guide/process.rst

  • This file was added.
.. _developer-guide-process:
===================
Development Process
===================
.. todo::
Describe continuous integration
.. todo::
Describe the difference between build, unit, functional, install,
integration, performance and acceptance tests.
.. todo::
Describe continuous deployment
.. todo::
Describe how to create a ticket
.. todo::
Describe how to submit a patch
.. todo::
Describe how to review and accept or decline a patch
.. todo::
Describe how to formulate a feature request
Kolab Groupware development largely follows an iterative and incremental
agile software development metholodogy also known as `Scrum`_.
In summary, the Kolab Groupware development process is divided into
stages, with different teams participating in getting tasks to the next
stage of the process.
Kolab Groupware is a collaboration suite consisting of many components,
each of them separate software development projects, some of them
dependent on third-party software development projects, and some of them
separate altogether. This creates some discrepancies in applying vanilla
Scrum practices to all projects in the Kolab universe.
Furthermore, Kolab Groupware is a community software development
project, and not all participants in the community can be expected to be
familiar with (otherwise professional) software development processes.
It is therefore that we document here how we work with `Scrum`_ using
`Phabricator`_.
Benefits of Phabricator, Scrum and Test-Driven Development
==========================================================
A plethora of benefits exist in using Phabricator and Scrum over the
current processes (which do not fall under any known nomenclature), and
Test-Driven Development (TDD).
* The processes and facilities are focused on structured software
development.
* Software development stands in stark contrast with longer-term
commitments such as support.
* A developer is supposed to work on tomorrow's version of the
software, and not yesterday's, as every distraction with yesterday's
software stands in the way of the focus and dedication needed to
provide tomorrow's innovative and competitive software.
* And every distraction stands in the way of the personal and
professional development of the individual developer.
* We aim to pull product support away from software development.
* A Bugzilla is not suitable for agile software development (even with
tooling such as Scrumbugz), and the former point may have
illustrated we wish software development to be agile.
* Various factors in the world of IT as well as common software
development practices predicate a shift toward continuous deployment
(at one cost) over sometimes overly precautious long-term support
methodologies (at another cost).
* The dynamics of these costs work in favor of a level of agility in
software development as well as operations, especially with smaller
teams.
Getting Acquainted with Phabricator
===================================
We use `Projects`_ in `Phabricator`_ as a means to group people and
assign them roles, rights and responsibilities in a broader process.
Tickets are created as `Maniphest`_ tasks, and roll on forward through
the process from thereon.
Software Development Projects
-----------------------------
The first type of project is a **Software Development Project**. These
are generally associated with one or more source code management
repositories.
Each software development project is responsible for a single software
component included in Kolab Groupware, such as **chwala** or
**pykolab**.
Because the Kolab community is an open community, membership to these
software development projects is *open* -- meaning registered users can
join and leave as they like.
Authorization Groups (Developer Teams)
--------------------------------------
Each software development project that operates source code management
repositories also has a group associated with it that is used for
authorization to push to the associated source code management
repositories.
For the **PyKolab** software development project for example, a group
**PyKolab Developers** has access to push to the ``pykolab`` GIT
repository.
These groups are mostly self-regulated, in that members of the group can
add other members to the group.
Teams and Roles
---------------
Teams and roles that are not software development teams include teams
such as `Product Owners`_, `Scrum Masters`_ and `Quality Assurance`_.
Sprints
-------
`Sprints`_ are what is in the name. They are a time-limited series of
events resulting in deliverables being emitted at the end of the sprint.
This series of events consists of:
**Planning Meeting**
Held at the beginning of each sprint, the `Product Owners`_ and the
developer team negotiate which tasks on the Sprint's backlog they
will attempt to complete.
The `Product Owners`_ determine the priority of tasks and declare
which (of all tasks) render the most business value.
The development team determines the amount of work they feel they
can achieve during the sprint without incurring technical debt.
.. NOTE::
The `Product Owners`_ use tasks related to individual software
development projects and associate them with the **next**
sprint in order to build an initial sprint backlog.
Working the other way around, where the team in collaboration
with the `Product Owners`_ determine which tasks to put on the
sprint backlog doesn't scale to the number of individual
software development projects for Kolab Groupware as a whole.
**Daily Scrums**
Short, daily stand-up meetings are used to let team members
summarize yesterday's work, today's work and talk about any
impediments encountered.
This ensures team members communicate early and often about their
work, problems they find, and facilitate the fast exchange of
knowledge and ideas or call for help.
**Review Meeting**
**Retrospective Meeting**
Process Stages
==============
The stages of the development process largely align with a manufacturing
process of, say, cars.
.. _developer-guide-process-stage-concept:
Concept
-------
This stage is an almost completely iterative process; Ideas are
generated, possibly prototyped and thrown away on a regular basis.
As such, concepts may or may not make it to production. In keeping with
the analogy of a car manufacturer, this may be a design drawing for a
new car (up to and including showing off a prototype on a trade show),
or some new technology (stage 2 artificial intelligence).
For Kolab Groupware, this stage mostly involves research and
development not necessarily based on consumer input.
The output of this process (if at all "valid") is additional input to
the next stage, :ref:`developer-guide-process-stage-pre-production`.
.. _developer-guide-process-stage-pre-production:
Pre-Production
--------------
The pre-production stage is an interaction between consumers of the
product, `Product Owners`_, `Architecture & Design`_ and
`Quality Assurance`_.
:ref:`developer-guide-process-roles-consumers` of the product request
features or report issues, that end up on the *Product Backlog* [#]_.
`Product Owners`_ prioritize these tasks.
Depending on the complexity of the task, and with the existing need to
facilitate the smoothest possible execution of the **Production** phase,
`Product Owners`_ and `Scrum Masters`_ request a review to be aided by
the `Architecture & Design`_ and `Quality Assurance`_ teams.
The input for this stage may be anything from UI elements with the wrong
teint (easy to forward to **Production** as-is) to complex enhancements
to Kolab Groupware as a whole, requiring design and possibly touching
the codebase of multiple software projects (such as the implementation
of Federated Central Authentication Services with Two-Factor
Authentication support).
Regardless, the effort at this stage involves breaking down tasks in to
digestible sub-tasks ready for **Production**.
Production
----------
Production is the stage where development work takes place. The form in
which is this happens is through `Sprints`_, which are limited chunks of
time in which several events take place that shape the development
process for :ref:`developer-guide-process-roles-developers`.
The input for this stage has initially been composed during the previous
sprint, with `Product Owners`_ putting tasks from dozens of software
development project backlogs on to the backlog for the current sprint.
Note that at this point, most facets of tasks have already been
negotiated -- but only to a certain extent.
If a relatively small task involves changes in one software project
dependent on changes in another software project, then this should have
been settled either before the individual sub-tasks end up on the sprint
backlog, or happens during the sprint planning meeting.
Post-Production
---------------
Roles
=====
.. _developer-guide-process-roles-stakeholders:
Stakeholders
------------
Stakeholders are parties interested in the product, and are responsible
for the creation of feature implementation requests, bug reports and
acceptance testing.
As such, stakeholders include customers (of Kolab Systems and its
partners), but actually include all consumers of Kolab. In other words,
regardless of affiliation or status, while you are reading this you
are probably a stakeholder.
However, some stakeholders' wishes have more weight than others. This
is likely reflected (at most) in the priority of the task(s) at hand,
which dictates the order in which tasks are worked on.
.. _developer-guide-process-role-product-owner:
Product Owner
-------------
The role of a product owner (usually a singular entity) is to determine
the value of goals (set out as epics and tasks in `Maniphest`_) and
prioritize the associated epics and tasks accordingly.
When the product owner does so, he or she determines the value of goals
on behalf of the stakeholders, and in light of an overall vision on
where the product should go.
Within the Kolab Groupware universe, where the *product* is the Kolab
release, and dozens of individual software development projects make up
the *product*, several aspects make the `Product Owners`_ a team of
individuals rather than a single person:
* Not all software development projects are related to the product.
Such may include work on Phabricator or the Sprint extension to
Phabricator (i.e. facilitatory third-party software).
* Not all software development projects are under the control of the
business.
This may include third-party software development projects we
contribute to (i.e. Roundcube).
However in the context of `Scrum`_ and particularly the `Sprints`_,
there is only one stakeholder representative -- **the** Product Owner.
.. _developer-guide-process-roles-scrum-masters:
Scrum Masters
-------------
`Scrum Masters`_ facilitate the Scrum process ...
.. rubric:: Checklist for the Beginning and End of a Sprint
#. Remove the hash-tag `sprint_server_current` from the past sprint,
#. Remove the hash-tag `sprint_server_next` from the current sprint,
#. Add the hash-tag `sprint_server_current` to the current sprint,
#. Create a new sprint,
#. Set the start and end dates for the new sprint,
#. Add the hash-tag `sprint_server_next` to the next sprint.
.. _developer-guide-process-roles-architecture-and-design:
Architecture & Design
---------------------
.. _developer-guide-process-roles-quality-assurance:
Quality Assurance
-----------------
.. _developer-guide-process-roles-developers:
Developers
----------
.. _developer-guide-process-roles-consumers:
Consumers
---------
* A user or customer requests the resolution of a certain issue.
* The product owner(s)
Continuous Integration, Delivery & Deployment
=============================================
.. graphviz::
digraph {
rankdir = LR;
splines = true;
overlab = prism;
edge [color=gray50, fontname=Calibri, fontsize=11];
node [shape=record, fontname=Calibri, fontsize=11];
"contributor" -> "Phabricator";
"Phabricator" -> "Build Host";
"Build Host" -> "Docker Container(s)";
}
#. A *contributor* contributes code via `Differential`_ to create or
update a code review proposal, or `Diffusion`_ to commit directly
to the GIT repository for the Software Development Project.
#. `Herald`_, the `Phabricator`_ rule engine, is used to trigger the
execution of a `Harbormaster`_ build plan [#]_.
#. `Harbormaster`_ allocates a `Drydock`_ host to execute a script on.
#. The `Drydock`_ host's script is supplied with the repository and
the commit in question.
#. The `Drydock`_ hosts used are `Atomic`_ hosts, meaning they run
`Docker`_ containers to execute Continuous Integration.
#. The so-called entrypoint for the Docker images (executed when a
container is started from the image) is designed to execute
Continuous Integration and Continuous Delivery.
Continuous Integration
----------------------
Continuous Integration is performed using the following stages;
* A *build* stage in which the software build dependencies are
installed in to the Docker container, and the software is built.
In the case of software that does not need to be compiled, such as
PHP and/or JavaScript, the *build* stage executes syntax checks.
* A *unit testing* stage, which is limited to testing individual
functions of the software, such as ``sum(a, b)`` with ``a=1; b=2``
indeed resulting in ``3``, but also does not yield unexpected
behaviour:
* ``sum(a, b)`` over ``a=1`` correctly raises an argument
exception (if expected to), or returns ``1`` (if expected to),
* ``sum(a, b)`` over ``a="Hello"`` and ``b="world!"``;
* does not unintentionally result in ``"Hello world!"`` nor
``"Helloworld!"``,
* raises the correct exception, or returns false, none or the
equivalent of ``NaN``,
* ``sum(a, b)`` over ``a=1; b="apple"`` does not result in ``1``
(unless expected) and raises the correct exception (if expected
to).
* *etcetera*, *etcetera*.
Unit testing further includes testing if underlying software
behaves in a way that the software can handle correctly. This may
include tests that determine whether stripping invalid characters
from a string (by **iconv**) yields the expected result, and
whether the version of the API provided by underlying software is
sufficient.
.. NOTE::
In cases where function calls require infrastructure (such as a
SQL server) with :term:`fixtures`, unit tests have to replace
the code that would interact with the infrastructure with code
that returns data as if the infrastructure existed. This is
called *mocking up*.
Tests that do indeed require infrastructure, and/or cannot be
*mocked up*, are called *functional tests*.
* A *functional testing* stage, in which functions of the software
are tested which require a minimal amount of infrastructure.
This may include functions that imply a caching layer such as
**memcached** is available (populating a cache with fixtures,
marking it as dirty, and expecting retrieval to return no data).
.. NOTE::
It should be noted that the distinction between *unit* and
*functional* testing is largely a matter of job control.
It should also be noted that functional tests are executed
against a *minimal* infrastructure -- the keyword being
minimal. This usually precludes infrastructure needed to
authenticate users, and usually precludes testing functions
against various implementations of a piece of functionality
(such as an IMAP server needed by Roundcube testing possibly
being either of UW-IMAP, Courier, Cyrus IMAP, Dovecot, ...).
* A *packaging* stage, in which the software is packaged up for
distribution.
* An *installation testing* stage, in which the result of the
*packaging* is installed.
* An *upgrade testing* stage, in which any number of former release's
fixtures are put in place, and the upgrade path is supposed to be
completed successfully automatically (such as a
:command:`rake db:migrate`).
* An *integration testing* stage
* An *acceptance testing* stage, in which human beings experience
the implementation.
Continuous Delivery
-------------------
.. _developer-guide-using-arcanist:
Using Arcanist
==============
.. rubric:: Footnotes
.. [#]
Kolab Groupware consists of dozens of individual product backlogs.
.. [#]
The sources for this script live in the Stick repository at
`drydocker.sh`_.
.. _Scrum: http://en.wikipedia.org/wiki/Scrum_%28software_development%29
.. _drydocker.sh: https://git.kolab.org/diffusion/QA/browse/master/drydocker.sh