GGCE Server issueshttps://gitlab.croptrust.org/grin-global/grin-global-server/-/issues2020-06-16T18:31:13+02:00https://gitlab.croptrust.org/grin-global/grin-global-server/-/issues/42Genebank reporting2020-06-16T18:31:13+02:00Nora Castaneda-AlvarezGenebank reportingWe want to enable partners to report on several genebank management aspects, using information recorded in GG. Partners should be able to report on:
* Availability of selected crop collections - accession level.
* Security (safety dupl...We want to enable partners to report on several genebank management aspects, using information recorded in GG. Partners should be able to report on:
* Availability of selected crop collections - accession level.
* Security (safety duplication) - accession level.
* Data availability - accession level.
* Distribution - accession level.
Some considerations to look upon are:
1. Availability is estimated on four variables:
* health - conducted at inventory level.
* viability - conducted at inventory level.
* quantity sufficiency - conducted at inventory level.
* legal status - defined at accession level.
2. Distribution: we also ask genebanks to report on:
* type of recipient
* country of recipient
* distinguish between internal vs. external recipients (same institution or not)
* exclude material sent to safety duplication
* type of germplasm (SAMPSTAT)
* distribution of subsets
* purpose of the requestor
3. Storage: we ask genebanks about the storage of inventory/accessions.
4. Security: we ask genebanks to report on the places where they have sent material for safety duplication.
How to summarize inventory level to report at accession level?
5. We also want to know how many viability tests are conducting per year.
6. We want to know the number of accessions/inventory (?) regenerated and characterized per year.
7. We want to know the number of requests received annually.
8. We want to know the total number of accessions and number of unique accessions distributed annually.
Here is the table of indicators and targets applicable to Seeds for Resilience (indicators 1,2,3,5,11 are potentially estimated through GG):
| | **Indicator** | **Target at year 5** |
|----|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:----------------------------------------------------------------------------------------------------------------------------------------------------:|
| 1 | **Availability:** % collection which is clean of pathogens of quarantine risk, viable, and in sufficient quantity to be immediately available for international distribution from medium-term storage (or local distribution for some tree species). | 90% accessions available |
| 2 | **Security:** For seed crops: % collection held in long-term storage at two locations and also in Svalbard Global Seed Vault (except for tree species). For clonal crops: % of the collection held in long-term storage or cryopreservation at two locations; % of the collection held in slow growth conditions in vitro at two locations | 90% seed accessions safety duplicated 50% clonal accessions in cryo-preservation; Intermediate target 90% accessions duplicated in in vitro |
| 3 | **Data availability** | Passport: PDCI >6 Characterization: 100% accessions with characterization data |
| 5 | **Distribution:** % of annual requests for seed samples from users successfully serviced | 80% relevant requests serviced annually |
| 11 | Number of user requests for seed samples | 100 |Matija ObrezaMatija Obrezahttps://gitlab.croptrust.org/grin-global/grin-global-server/-/issues/64Acquisition: Checking for Duplicates2020-07-29T03:58:54+02:00Grace CapilitAcquisition: Checking for DuplicatesAfter documenting incoming materials to the Genebank, we want to check whether we already have a copy of these newly received materials in the Genebank, either as registered seeds (accession) or unregistered samples (from another incomin...After documenting incoming materials to the Genebank, we want to check whether we already have a copy of these newly received materials in the Genebank, either as registered seeds (accession) or unregistered samples (from another incoming batch with TemporaryID only)
This function will search the database for similar entries by name (cultivar name, DonorID, CollNo, etc) using algorithms for
* exact match
* edistance
* soundex
Checking for duplicates can be done
1) by batch where the user can either specify/choose the BatchID of the incoming materials to be compared or upload list from a file or
2) by entry where a name is entered in the search boxGrace CapilitGrace Capilithttps://gitlab.croptrust.org/grin-global/grin-global-server/-/issues/62Acquisition: Evaluating Incoming samples for Inclusion in the Processing2020-08-05T09:15:30+02:00Grace CapilitAcquisition: Evaluating Incoming samples for Inclusion in the ProcessingThis will enable Genebank managers to document evaluation of incoming materials that will guide succeeding actions on the sample.
Will use accession_action to record
1) the incoming seed status
| Code | Description ...This will enable Genebank managers to document evaluation of incoming materials that will guide succeeding actions on the sample.
Will use accession_action to record
1) the incoming seed status
| Code | Description |
|------|---------------------------------------------------|
| REP | REPLACEMENT |
| MIS | MISSING SAMPLE/PACKET |
| HLD | HOLD |
| DUPY | PROBABLE DUPLICATE (INCLUDE IN SEED PROCESSING) |
| SHP | WITH SEED HEALTH PROBLEM |
| EXC | TO BE EXCLUDED IN SEED PROCESSING/PLANTING |
| INC | TO BE INCLUDED IN SEED PROCESSING/PLANTING |
| SUF | SUFFICIENT, FOR DIRECT ACCNO ASSIGNMENT |
| DUPN | EXACT DUPLICATE (NOT INCLUDED IN SEED PROCESSING) |
| MIX | MIXED SAMPLE (INSUFFICIENT AMOUNT) |
| MXS | MIXED SAMPLE (SUFFICIENT AMOUNT) |
2) viability (non-viable, weak, viable)
3) planting material availability (sufficient, insufficient)
4) remnant status (sufficient, insufficient, no remnants)
quantitative data
5) population structure (1, 2, 3, etc)
-indicates the number of population, >1 when mixed
6) number of rows that can be plantedGrace CapilitGrace Capilithttps://gitlab.croptrust.org/grin-global/grin-global-server/-/issues/76Acquisition: Selection of Planting Materials2020-09-02T07:40:42+02:00Grace CapilitAcquisition: Selection of Planting MaterialsIn this function, we should be able to select incoming materials for planting. These are the entries whose status afer the evaluation allows them to be part of the initial seed increase, like:
1) INC- Homogeneous/pure seeds but with i...In this function, we should be able to select incoming materials for planting. These are the entries whose status afer the evaluation allows them to be part of the initial seed increase, like:
1) INC- Homogeneous/pure seeds but with insufficient amount for storage
2) DUPY - probable duplicate that needs further testing
3) MIX - mixed sample with insufficient amount
4) REP - replacement of an old accession which is already dead or no longer available
But there are 2 categories for the selection
A) initial seed increase of incoming materials that are not yet planted ever since (first time to plant)
B) incoming materials that were already planted but harvest is not successful (replanting)
For both selection, species and type of material are required.
At IRRI, species is grouped as O. sativa, O. glaberrima and wild. Type of material is either Regular Genebank materials or Genetic Stocks. We need to add this function of identifying type of material after assigning TempID.
The cropyear and selection method should be indicated. These will be used to indicate the new inventory group for the entries that will be selected. (Example: 2020DS, ISI)
Incoming Batch(es) (accession_inv_group) will be picked and records with status sited above will be retrieved and tagged as selected materials for planting.
The selected materials can be be further reviewed and user can still delete entries that will not be included in the planting.Grace CapilitGrace Capilithttps://gitlab.croptrust.org/grin-global/grin-global-server/-/issues/232API for Citations and Literature2021-10-04T11:18:35+02:00Maxym BorodenkoAPI for Citations and LiteratureImplement API for https://gitlab.croptrust.org/grin-global/grin-global-ui/-/issues/330Implement API for https://gitlab.croptrust.org/grin-global/grin-global-ui/-/issues/330Matija ObrezaMatija Obrezahttps://gitlab.croptrust.org/grin-global/grin-global-server/-/issues/91Add pending actions for new inventories2021-10-04T11:21:55+02:00Matija ObrezaAdd pending actions for new inventoriesWhen a new `Inventory` is added to GG-CE, the system must automatically register two **PENDING** `InventoryAction` records:
1. `QUANTITYSET`: *Quantity update* -- The actual amount of material is inspected.
1. `LOCATIONSET`: *Location u...When a new `Inventory` is added to GG-CE, the system must automatically register two **PENDING** `InventoryAction` records:
1. `QUANTITYSET`: *Quantity update* -- The actual amount of material is inspected.
1. `LOCATIONSET`: *Location update* -- The location of the inventory is set.
These two actions are automatically closed when:
1. `QUANTITYSET` is closed when `Inventory#quantity` is updated
1. `LOCATIONSET` is closed when `Inventory#storageLocation1..4` is updated
The codes need to be registered as `CommunityCodeValues`.
It may be sensible to add an aspect on the `#save` method(s) of `InventoryRepository`, check if `Inventory` is not yet persisted and add these actions in the aspect.
Also the `InventoryService#addInventoryAction` should first try to close an existing pending action before adding a new one. It should be renamed to `#closeInventoryAction`.Matija ObrezaMatija Obrezahttps://gitlab.croptrust.org/grin-global/grin-global-server/-/issues/298Firehose notification system2022-06-23T12:32:52+02:00Matija ObrezaFirehose notification systemWe introduced the Firehose to have information about what data was committed to the database. Firehose emits notifications about entity changes.
This ticket is to design API support that allows users to:
1. Subscribe to specific Fireho...We introduced the Firehose to have information about what data was committed to the database. Firehose emits notifications about entity changes.
This ticket is to design API support that allows users to:
1. Subscribe to specific Firehose event types
1. Receive email notifications when those events happen
Involve @ahrybeniuk, @crabil, @jalarcon and @adovhopolenko in the user requirements and UI design.
## Ideas to consider
1. Alert me when `Accession.accessionNumber` is modified
1. Alert me whenever a new `Inventory` is registered
1. Alert me when an action of type XX is completed
## Message delivery
1. Alert me immediately
1. Daily update
1. Weekly update?Artem HrybeniukArtem Hrybeniukhttps://gitlab.croptrust.org/grin-global/grin-global-server/-/issues/234blaze-persistence?2022-08-17T11:32:47+02:00Matija Obrezablaze-persistence?Should we look what https://github.com/Blazebit/blaze-persistence brings to the table?
I spotted the URL to blaze-persistence in the now obsolete `.fetchCount()` method.Should we look what https://github.com/Blazebit/blaze-persistence brings to the table?
I spotted the URL to blaze-persistence in the now obsolete `.fetchCount()` method.Matija ObrezaMatija Obrezahttps://gitlab.croptrust.org/grin-global/grin-global-server/-/issues/362Strong-typed permission checks2022-09-27T13:07:08+02:00Matija ObrezaStrong-typed permission checksAny suggestion how to make the String in `@PreAuthorize("@ggceSec.actionAllowed('ViabilityTest', 'ADMINISTRATION')")` expression strongly typed so it references `SecurityAction` enum and perhaps even the enums of `BasePermission` (`CREAT...Any suggestion how to make the String in `@PreAuthorize("@ggceSec.actionAllowed('ViabilityTest', 'ADMINISTRATION')")` expression strongly typed so it references `SecurityAction` enum and perhaps even the enums of `BasePermission` (`CREATE`, `ADMINISTRATION`, ...)?Artem HrybeniukArtem Hrybeniukhttps://gitlab.croptrust.org/grin-global/grin-global-server/-/issues/323SysUser with mandatory unique Cooperator2023-01-29T17:28:05+01:00Matija ObrezaSysUser with mandatory unique CooperatorEach `SysUser` record in GGCE should have a corresponding `Cooperator` record (1-1). Please update it to `nullable = false` and update liqibase.
Before adding the not-null constraint, liquibase needs to have a precondition check that al...Each `SysUser` record in GGCE should have a corresponding `Cooperator` record (1-1). Please update it to `nullable = false` and update liqibase.
Before adding the not-null constraint, liquibase needs to have a precondition check that all `select count(sys_user_id) from sys_user where cooperator_id is null` equals `0`.
Also add a unique index on `sys_user.cooperator_id`, ensuring that each entry has its own `Cooperator`. Precondition `select count(cooperator_id) from sys_user group by cooperator_id having count(cooperator_id) > 1` equals `0`.
When this is ready, UI needs updating so that the list of users includes the link to `Cooperator` and the cooperator name.Artem HrybeniukArtem Hrybeniukhttps://gitlab.croptrust.org/grin-global/grin-global-server/-/issues/332Jasper: Font extensions2023-01-29T17:28:05+01:00Matija ObrezaJasper: Font extensionsTo support various fonts in PDFs the fonts need to be loaded using a **custom classloader**. The current thread's classloader needs to be saved:
```java
final ClassLoader contextClassLoader = Thread.currentThread().getContextClassLoader...To support various fonts in PDFs the fonts need to be loaded using a **custom classloader**. The current thread's classloader needs to be saved:
```java
final ClassLoader contextClassLoader = Thread.currentThread().getContextClassLoader();
try {
.. do things with a custom classloader
} finally {
Thread.currentThread().setContextClassLoader(contextClassLoader);
}
```
The fonts can be exported as `jar` by users and uploaded to file repository (e.g. `/fonts` folder ?) to be embedded in PDF. This requires a `net.sf.jasperreports.engine.fonts.SimpleFontExtensionsRegistryFactory`
- https://community.jaspersoft.com/questions/534323/reportclassloader-and-font-extensions-issue
- http://jasperreports.sourceforge.net/sample.reference/fonts/
Note: for the moment I updated Docker to include MS core fonts (Arial). That needs to be removed when font extensions are working.Artem HrybeniukArtem Hrybeniukhttps://gitlab.croptrust.org/grin-global/grin-global-server/-/issues/393Remove sys_table*, sys_dataview* from initial data2023-07-26T08:57:22+02:00Matija ObrezaRemove sys_table*, sys_dataview* from initial dataWith 2023.3 we will no longer provide the original GG dataviews with a fresh install of GGCE.
In this issue we scrub the rows from CSV files in `liquibase/initial_data`.With 2023.3 we will no longer provide the original GG dataviews with a fresh install of GGCE.
In this issue we scrub the rows from CSV files in `liquibase/initial_data`.Matija ObrezaMatija Obrezahttps://gitlab.croptrust.org/grin-global/grin-global-server/-/issues/459Switch to OpenSearch2024-02-19T10:26:42+01:00Matija ObrezaSwitch to OpenSearchWe are using ES6 which is the last version before ES changed it's licensing model. The open source version was forked into OpenSearch.
In this ticket we replace ES with OS.
I think it makes sense to first upgrade ES libs in our project...We are using ES6 which is the last version before ES changed it's licensing model. The open source version was forked into OpenSearch.
In this ticket we replace ES with OS.
I think it makes sense to first upgrade ES libs in our project to the version mentioned in OS migration guide at https://opensearch.org/docs/latest/upgrade-to/upgrade-to/ and then switch over to OS.
The reason for first upgrading ES and then switching to OpenSearch in stages is that we can also slowly migrate our ES cluster.
We now have some custom code for serializing entities for indexing. That code will eventually be removed and replaced with DTO-style mapping to a distinct type before sending it over to ES/OS.Artem HrybeniukArtem Hrybeniukhttps://gitlab.croptrust.org/grin-global/grin-global-server/-/issues/465Slow performance in inventory_viability_data dataview2024-03-14T20:00:02+01:00Antonio RiveraSlow performance in inventory_viability_data dataviewSince we upgrade to the 2024.1.2 versión the server performance for the _inventory_viability_data_ dataview
We send request that has a length of only 13, 000 records. The request lasted 2 hours before ending
![image.png](/uploads/efb8...Since we upgrade to the 2024.1.2 versión the server performance for the _inventory_viability_data_ dataview
We send request that has a length of only 13, 000 records. The request lasted 2 hours before ending
![image.png](/uploads/efb85ecccfffdc814e0802ff5ad015bf/image.png)
We check the server memory. It looks fine.
![image.png](/uploads/9d7059045513de0ad48da0dff9d4a266/image.png)Matija ObrezaMatija Obrezahttps://gitlab.croptrust.org/grin-global/grin-global-server/-/issues/471Action notifications2024-03-15T10:24:40+01:00Matija ObrezaAction notificationsThis ticket adds new notification generators `org.gringlobal.notification.AccessionActionNotifications`, `org.gringlobal.notification.InventoryActionNotifications` and `org.gringlobal.notification.OrderRequestNotifications` that send the...This ticket adds new notification generators `org.gringlobal.notification.AccessionActionNotifications`, `org.gringlobal.notification.InventoryActionNotifications` and `org.gringlobal.notification.OrderRequestNotifications` that send the relevant action schedules to the subscribers. The scheduling is described in #470 and subscriber list in #472.
`AccessionActionNotifications` will use `accession-schedule.mustache` template and look like:
```java
@ScheduledNotification(value = "sendWeeklyAccessionSchedule")
@Transactional(readOnly = true)
public void generateWeeklyAccessionSchedule() {
// Load AccessionActions for next 7 days (from today at 00:00 - to today+8days at 00:00)
// Use accession-schedule.mustache to generate the body
}
@ScheduledNotification(value = "sendMonthlyAccessionSchedule")
@Transactional(readOnly = true)
public void generateMonthlyAccessionSchedule() {
// Load AccessionActions for this month (from 1st of this month at 00:00 - to 1st of next month at 00:00)
// Use accession-schedule.mustache to generate the body
}
```
The default templates should generate a table of actions similar to that in the UI:
> ## Accession action schedule {{fromInstant}} to {{toInstant}}
>
> View it in GGCE +Link to GGCE UI
>
> ![image](/uploads/c1474b01658b5580bf2adf17e5283458/image.png)
>
Template customization follows the logic in #4692024.3Artem HrybeniukArtem Hrybeniukhttps://gitlab.croptrust.org/grin-global/grin-global-server/-/issues/446Add Production Location Geography to "Get Invetory" Dataview2024-03-15T10:51:46+01:00Antonio RiveraAdd Production Location Geography to "Get Invetory" DataviewWe need to display the Production Geography Location in the inventory dataview (as the barcode column). We stored here the increment site value after migrate to GGCE.
![image](/uploads/83e240dd038a48b136a380e924a96037/image.png)
Locati...We need to display the Production Geography Location in the inventory dataview (as the barcode column). We stored here the increment site value after migrate to GGCE.
![image](/uploads/83e240dd038a48b136a380e924a96037/image.png)
Location value in the column "adm1" from the "Geography" table
![image](/uploads/137cfa5d211503b4756d0723a6c87a0d/image.png)2024.2Matija ObrezaMatija Obrezahttps://gitlab.croptrust.org/grin-global/grin-global-server/-/issues/436Improved auto-numbering of accessions and inventories2024-03-15T10:58:45+01:00Matija ObrezaImproved auto-numbering of accessions and inventoriesWe use `-1` in the numerical part of accession and inventory numbers (`_number_part2`) to automatically assign the next available value.
The current implementation does not take into account numbers that were registered in the past and ...We use `-1` in the numerical part of accession and inventory numbers (`_number_part2`) to automatically assign the next available value.
The current implementation does not take into account numbers that were registered in the past and may assign the same number when the right conditions are met. Example: Accession `INTRO 13` is renamed to `ACC 9999`. If we now create a new `INTRO -1` it will may assign `INTRO 13` because it is not currently used.
We "log" `inventoryNumber` and `accessionNumber` as AccessionInvName with category `SITE_ID`. The improved auto-numbering mechanism should generate a candidate number (same as current implementation), then check if there already exists a matching `AccessionInvName` in category `SITE_ID`. If so, then it should use the next available number.
## For consideration
- We can't differentiate between accession and inventory in `AccessionInvName` without `join Inventory i` and inspecting `i.formTypeCode`.
- [Performance] We may need to check many `AccessionInvName` records to determine a "free number". What optimizations can we use?Artem HrybeniukArtem Hrybeniukhttps://gitlab.croptrust.org/grin-global/grin-global-server/-/issues/470Custom schedule of email notifications2024-03-15T11:29:02+01:00Matija ObrezaCustom schedule of email notificationsThis ticket adds support for **user-defined schedule** for running `KPINotifications`. Instead of fixed `@Scheduled` on the method in code, we allow the *Administrator* to define a custom schedule.
The custom schedule is stored in `@Tab...This ticket adds support for **user-defined schedule** for running `KPINotifications`. Instead of fixed `@Scheduled` on the method in code, we allow the *Administrator* to define a custom schedule.
The custom schedule is stored in `@Table("notification_schedule") public class NotificationSchedule extends VersionedModel` entity. It's properties include:
- the cron expression `String cron`
- the bean `String generator` and the method in the bean execute `String method`
- The combination of `cron + generator + method` forms a unique constraint
GGCE will inspect this table on startup and add `active = true` entries to its Spring scheduler.
On any change to `NotificationSchedule` the Spring scheduler will be updated, removing the task when deleted or `active = false`, and adjusting the `cron` if `active = true`. This can be done in the `update/create` methods in the `NotificationScheduleServiceImpl`.
The API controller is at */api/v2/admin/schedule* and follows `FilteredCRUDService2Impl` (with DTO, etc).
The API needs to be able to inform the UI which methods are available for scheduling. A new annotation `@ScheduledNotification` is added to `KPINotifications` and this annotation determines which methods can be included in the custom schedule.
```java
@ScheduledNotification(value = "sendAllKPIs")
@Transactional(readOnly = true)
public void generateLastKPIExecutions() {
for (var execution : executionRepository.listActiveExecutions()) {
notifyLastKPIExecution(execution.getName())
}
}
```
We convert from `@Scheduled` to a new `NotificationSchedule` record in `ApplicationStartup` using a programmable liquibase change.
## UI
In GGCE Web *Admin tools* a new section (menu) **Scheduler** is added. It allows the Adminstrator to manage `NotificationSchedule` records.
Multiple schedules can exist for the same `@ScheduledNotification`: for example one that runs on Mondays, and one on Fridays.2024.3Artem HrybeniukArtem Hrybeniukhttps://gitlab.croptrust.org/grin-global/grin-global-server/-/issues/473Assigning actions2024-03-22T01:21:59+01:00Matija ObrezaAssigning actionsThis ticket adds support for assigning `...Action` to a specific user (`SysUser`), user group (`SysGroup`) or any other SID for that matter.
```java
public abstract class AbstractAction ... {
@ManyToOne
@JoinColumn(nullable = true, .....This ticket adds support for assigning `...Action` to a specific user (`SysUser`), user group (`SysGroup`) or any other SID for that matter.
```java
public abstract class AbstractAction ... {
@ManyToOne
@JoinColumn(nullable = true, ...)
private AclSid assignee;
...
}
```
`ActionFilter` should allow for filtering by `assignee`.
## UI
Include **Assignee** in all action tables, forms and search filters. Filtering uses the same approach as `modifiedBy`:
![image](/uploads/779c8cbf517a5b03279479b9cfe5a497/image.png)
Add support to select multiple actions and "Assign selected actions" to some `SysUser` or `SysGroup`.2024.4Artem HrybeniukArtem Hrybeniukhttps://gitlab.croptrust.org/grin-global/grin-global-server/-/issues/476Error handling for Genesys upload2024-03-25T12:18:48+01:00Matija ObrezaError handling for Genesys uploadWhen uploading passport data to Genesys the code now stops the upload when there are more then 10 errors reported by Genesys API.
```java
// If more than 10 errors, stop uploading, report error
if (opErrors.size() > 10) {
```
If there ...When uploading passport data to Genesys the code now stops the upload when there are more then 10 errors reported by Genesys API.
```java
// If more than 10 errors, stop uploading, report error
if (opErrors.size() > 10) {
```
If there are less than 10 errors then those errors are not reported to the client, giving the impression that upload was successful. The code should still stop the upload at 10 errors, but it should collect error messages reported by Genesys API and make them available to the client as part of `UploadState.errors`:
```java
@Data
class UploadState
...
/** Errors reported by Genesys API */
private Map<AccessionInfo, String> errors;
```
## UI update
The upload progress dialog should include display of the number of errors found (in red) and display the error table linking to accessions:
|Accession|Error|
|--|--|
|[TMe-419](https://demo.ggce.genesys-pgr.org)|*Error message here...*|2024.2Artem HrybeniukArtem Hrybeniuk