The "problem" is with "when it is time to generate the invoice". So, should we first detect if profile changed in ldap and create an invoice then or we should not and wait with hkccp subscriptions update until the new period starts or there's any other reason to create a new invoice? I suppose, we should detect changes and force new invoice.

In T1125#16619, @machniak wrote:

The "problem" is with "when it is time to generate the invoice". So, should we first detect if profile changed in ldap and create an invoice then or we should not and wait with hkccp subscriptions update until the new period starts or there's any other reason to create a new invoice? I suppose, we should detect changes and force new invoice.

OK, two different stages here;

1. A daily cronjob is triggered to potentially generate any new invoices required,
2. A new invoice may need to be triggered.

For 1), I understand the entire routine may only ever examine the database-level entries, such that no LDAP changes would be included in it's considerations on whether or not to trigger a new invoice. Please confirm/deny that/whether amending this routine to include a pull from current LDAP entries would resolve the ambiguity.

For 2), I reckon the #HKCCP will not learn about any changes having occurred unless it is an addition or deletion of a member account.

1. It could be resolved in the invoicing routine, but on a higher level than subscriptions() method. However, I think that a separate cronjob would be better. Less chances we'll break something else ;)

2. user addition/deletion, but also any subscription change (e.g. quota change on an existing user).

In T1125#16653, @machniak wrote:

1. It could be resolved in the invoicing routine, but on a higher level than subscriptions() method. However, I think that a separate cronjob would be better. Less chances we'll break something else ;)

I would consider syncing up the #HKCCP local database copy of what subscriptions an account may have held at some point in the past is probably the most sustainable solution.

I'm afraid a separate cronjob still puts some probably-out-of-date database in the lead, whereas LDAP must be.

What would the cronjob look like? A proverbial iteration of for user in $list; do bin/backend.php sync-user $user; done?

2. user addition/deletion, but also any subscription change (e.g. quota change on an existing user).

However, note that the subscription changes in the case of T1124: Plugin to change limited user options would go straight to LDAP, do not pass Go and do not get $20k -- the #HKCCP won't learn about the changes unless it pulls/resyncs data from the canonical source of authorizations, namely LDAP.

In T1125#16655, @vanmeeuwen wrote:

I would consider syncing up the #HKCCP local database copy of what subscriptions an account may have held at some point in the past is probably the most sustainable solution.

I'm afraid a separate cronjob still puts some probably-out-of-date database in the lead, whereas LDAP must be.

Well, yes. Until the cronjob do the sync.

What would the cronjob look like? A proverbial iteration of for user in $list; do bin/backend.php sync-user $user; done?

It would be something similiar to what invoicing already does. So for every account, and for every user get ldap attributes and compare with subscriptions, then sync differences. Which would create new entries in subscriptions history, and this would be normally catched by invoicing script as subscription change and need for new invoice.

2. user addition/deletion, but also any subscription change (e.g. quota change on an existing user).

However, note that the subscription changes in the case of T1124: Plugin to change limited user options would go straight to LDAP, do not pass Go and do not get $20k -- the #HKCCP won't learn about the changes unless it pulls/resyncs data from the canonical source of authorizations, namely LDAP.

That's why we're here, right? I just responded to your question about what HKCCP does now.

In T1125#16657, @machniak wrote:

In T1125#16655, @vanmeeuwen wrote:

I would consider syncing up the #HKCCP local database copy of what subscriptions an account may have held at some point in the past is probably the most sustainable solution.

I'm afraid a separate cronjob still puts some probably-out-of-date database in the lead, whereas LDAP must be.

Well, yes. Until the cronjob do the sync.

Right, so as a result, we run this particular cronjob after the current cronjob.

This leaves a time-window of up to just short of a full day in which all sorts of mutations could occur, for the invoicing to still respond to out-dated information -- subscription changes among others.

What would the cronjob look like? A proverbial iteration of for user in $list; do bin/backend.php sync-user $user; done?

It would be something similiar to what invoicing already does. So for every account, and for every user get ldap attributes and compare with subscriptions, then sync differences. Which would create new entries in subscriptions history, and this would be normally catched by invoicing script as subscription change and need for new invoice.

The sheer volume of users justifies delaying invoices (by hours if needed), but does not justify issuing invoices on information already out-dated.

Running two separate cronjobs creates the potential for dealing on the basis of out-dated information, as well as the race-conditional of which finishes first and/or in time for the other to kick off.

2. user addition/deletion, but also any subscription change (e.g. quota change on an existing user).

However, note that the subscription changes in the case of T1124: Plugin to change limited user options would go straight to LDAP, do not pass Go and do not get $20k -- the #HKCCP won't learn about the changes unless it pulls/resyncs data from the canonical source of authorizations, namely LDAP.

That's why we're here, right? I just responded to your question about what HKCCP does now.

I'm attempting to articulate this over and over again so that we can all understand we're on the same page, and avoid ambiguity and misunderstandings.

Yes, this is why we're here.

I believe we're deliberating whether or not the #HKCCP should synchronize subscriptions from LDAP in order to issue then-correct invoices (i.e. "inline", blocking each line-item, incurring massive amounts of processing), or whether it could act on ever-so-slightly outdated information that is updated separately -- so that the massive amounts of processing can be made to "unblock" the invoice generation.

So, as I understand, it's better to integrate this with the invoicing process. It will be slower, because it will need to touch every user record. Now it fetches only the account-owner-user from ldap and if there's no changes in SQL database skips account users processing. Now it would need to fetch all users ldap records + SELECT their subscriptions to detect and apply changes. At least I can implement this in a way it won't need to fetch/SELECT twice per user.