public:smart2b:home [OdinS Wiki]

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
# Smart2B T3.1 Lead by ODINS

{{tag> smart2b}}

## Issue report form

Please, before reporting your issue, make sure you have reviewed the following sections of this page, as they may contain information about it and potentially how to solve, or the ongoing process in solving it.

Afterwards, if your issue has not been addressed in the following sections, please copy the following issue report form into an email, fill it out and send it to jsanchez@odins.es

0. Email subject: `[Smart2B] Issue Report - <topic>`

1. Reported by:
2. Email:
3. Tasks involved/Partners involved:
4. Broker Entities in particular involved:
5. Severity (select from Minor, Moderate, Major and Critical):
6. Summary:
7. Description:
8. Error messages obtained:
9. Requests that produce the errors (so that we can reproduce the issue):
10. Time interval (UTC time) during which these errors have occurred:

We will try to resolve your issue as soon as possible. Thank you very much and sorry for the inconvenience.

## Ongoing Issues known by the ODINS team

- WP5 app is having issues in order to plot and get all the data that must be shown in the browser - https://wiki.odins.es/public/smart2b/home#wp5_integration_ongoing_issues

## Entity Info Spreadsheet

- Always make sure that you are using the latest release, an announce is emailed to the partners participating in the WP2345 Alignment meeting and also a copy is placed in the Sharepoint folder.
- Please **DO NOT** place copies of the spreadsheet yourself in the Sharepoint or circulate it among partners, only ODINS is in charge of releasing the spreadsheet and all the management is done bilaterally.
- The Entity Info Spreadsheet is under `WP3 / Entity Information Spreadsheet`

## Authorization Access to smart2b.odins.es resources

- Each and every partner should have different credentials to access the Smart2B Platform. These credentials are mandatory and required for each request.

- The access policy is bottom down. I.e. initially partners have access to nothing and are granted permissions due to request and only if it makes sense.

Here are the current permissions allowed:

- The following credentials have READ access to **ALL** entities of **ALL TYPES** in the NGSI-LD Broker and Storage component.
    - certh@smart2b.com
    - edpsel@smart2b.com
    - enerbrain@smart2b.com
    - fcid@smart2b.com
    - nordomatic@smart2b.com
    - xtel@smart2b.com
    - vito@smart2b.com
    - rwth@smart2b.com
    - tug@smart2b.com


- For any entity **OF ANY TYPE** that start with the pattern `SCML-.*` these accounts have WRITE permissions.
    - edpsel@smart2b.com
    - xtel@smart2b.com

- For any entity **OF ANY TYPE** that start with the pattern `AirBnB-.*` these accounts have WRITE permissions.
    - edpsel@smart2b.com
    - fcid@smart2b.com
    - xtel@smart2b.com

- For any entity **OF ANY TYPE** that start with the pattern `AOC-.*` these accounts have WRITE permissions.
    - enerbrain@smart2b.com

- For any entity **OF ANY TYPE** that start with the pattern `ABL-.*` these accounts have WRITE permissions.
    - nordomatic@smart2b.com
	 - edpsel@smart2b.com

- For any entity **OF ANY TYPE** that start with the pattern `ARA-.*` these accounts have WRITE permissions.
    - nordomatic@smart2b.com
    - edpsel@smart2b.com
    - xtel@smart2b.com

- For any entity **OF ANY TYPE** that start with the pattern `CJC-.*` these accounts have WRITE permissions.
    - enerbrain@smart2b.com

- For all the entities of **Type DeviceMeasurement** the following accounts have WRITE Access:
    - certh@smart2b.com
    - **NOTE** this is due to how the T3.2 components led by CERTH (Data Filtering, Semmantic Annotation) work.

- For all the entities of **Type Device** the following accounts have WRITE Access:
    - enerbrain@smart2b.com
    - **NOTE** this is due to how the T3.3 components led by EB (Translator) work.


## Disaggregation of SCML entities

The SCML entities were disaggregated by petition of Miguel Brito in the following form during October 2023.

Please, always check that you're working with the most up to date version of the spreadsheet since those entities changed names:

At the moment of writing this message, the one currently is the v2.1 of the spreadsheet.

https://edponcloud.sharepoint.com/:f:/r/teams/O365_H2020Smart2B/Shared%20Documents/General/WP3%20-%20Development%20of%20SMART2B%20Platform%20%26%20APIs/Entity%20Information%20Spreadsheet?csf=1&web=1&e=TuC7UB


```
        ------------------------------------------------------------------------------------------
        AirBnB-Mezaninne: (First Left:58, First Right:54, Ground Right:35)

        AirBnB-Mezaninne_FirstFloor-Right              --> AirBnB-MezaninneFirstRight_FirstFloor-Right
        AirBnB-Mezaninne_FirstFloor-Left              --> AirBnB-MezaninneFirstLeft_FirstFloor-Left
        AirBnB-Mezaninne_GroundFloor-Right              --> AirBnB-MezaninneGroundRight_GroundFloor-Right
        ------------------------------------------------------------------------------------------


        ------------------------------------------------------------------------------------------
        SCML-LargoOttolini

        SCML-LargoOttolini_FirstFloor-Right --> SCML-LargoOttoliniFirst_FirstFloor-Right
        SCML-LargoOttolini_GroundFloor-Right --> SCML-LargoOttoliniGround_GroundFloor-Right
        SCML-LargoOttolini_SecondFloor-Left --> SCML-LargoOttoliniSecond_SecondFloor-Left

        ------------------------------------------------------------------------------------------

        SCML-MarquesadeAlorna: 132 (ground:63, second:69)

        SCML-MarquesadeAlorna_GroundFloor-Right --> SCML-MarquesadeAlornaGround_GroundFloor-Right
        SCML-MarquesadeAlorna_SecondFloor-Right --> SCML-MarquesadeAlornaSecond_SecondFloor-Right

        ------------------------------------------------------------------------------------------

        SCML-RuadaVinha:240 (ground:78, first:78, second: 84)

        SCML-RuadaVinha_FirstFloor --> SCML-RuadaVinhaFirst_FirstFloor
        SCML-RuadaVinha_SecondFloor --> SCML-RuadaVinhaSecond_SecondFloor
        SCML-RuadaVinha_ThirdFloor --> SCML-RuadaVinhaThird_ThirdFloor

        ------------------------------------------------------------------------------------------
```

## Integrating Partner FAQ

- What is said here must be complemented with the contents of [Smart2B Deliverable D3.1 Section 8](https://edponcloud.sharepoint.com/:b:/r/teams/O365_H2020Smart2B/Shared%20Documents/General/4.%20Deliverables%20repository/D3.1-%20FIWARE-based%20harmonization,%20data%20annotations%20and%20filtering_storage.pdf?csf=1&web=1&e=wp9i3p), where all the steps to perform operations against the NGSI-LD Broker and Storage component are thoroughly documented.

- Each integrating partner is supposed to only perform PATCH operations against the **DeviceMeasurement** entities that are relevant to their pilots (see list above). If an integrating partner requests an authorization token to entities that are not within their administrative domain - I.e. belong to a different pilot - the request to generate an authorization request will return an error.

- For the most part, integrating partners (I.e. XTEL, EDP SEL, EB, Nordomatic) might forget about performing either GET or PATCH operations against **Building** and **Zone** entities. Those are required for other WP4 services but not really relevant otherwise for building equipment integration.

- Due to several circumstances, **The name of the entities in the broker may change**. Although we, as a consortium, try our best to avoid this matter, it is unavoidable. Hence, Make sure that you're using the entities from the latest release of the [Entity Info Collection Spreadsheet](https://edponcloud.sharepoint.com/:f:/r/teams/O365_H2020Smart2B/Shared%20Documents/General/WP3%20-%20Development%20of%20SMART2B%20Platform%20%26%20APIs/Entity%20Information%20Spreadsheet?csf=1&web=1&e=ObB9AU), located in the WP3 Folder in the Sharepoint Folder.
    - Otherwise, if you employ names not available in the Broker, the behaviour is undefined, and most likely not error-prone.

- Make sure that when you request the authorization token, the URL does contain the right pilot prefix:

E.g. 

If a partner wishes to perform a PATCH request against the following entity:

```
ABL-RetirementHome_ThirdFloor-NA-BuildingP-Z3050Z1_Nordomatic-BMS-NA-RT01_Temperature
```

Note that the entity starts with `ABL-`, hence, the right prefix to request an authorization token would be `ABL-.*` as shown below:


```
{"token": "{{idmToken}}","ac": "PATCH", "de": "https://smart2b.odins.es:1027", "re": "/ngsi-ld/v1/entities/urn:ngsi-ld:DeviceMeasurement:ABL-.*/attrs"}
```

- Note the `ABL-.*` at the end of the URL. As a consequence, this authorization token will be valid only to perform the `PATCH` operation to those entities. If the developer wishes to perform PATCH operations against entities that have a different prefix (e.g. `ARA-`), then the authorization token must be adjusted.

### Verification of Context Source Registrations

After registering the Context Source it can be checked if the registrations process was successful --- i.e. the integrating partner software component that receives the messages from the NGSI-LD Broker in WP3 Smart2B Platform is supposed to be registered before actuation can carry over from the WP3 Smart2B platform all the way down to the building equipment.

The following POSTMAN collection can be employed in order to verify if the Context Source Registration took place successfully.

{{:public:smart2b:smart2b-security-partners-allcontextproviders.postman_collection.json.zip}}

**IMPORTANT**

There is a limitation of `1000` items per query, specified as an URL option like this:

```
https://smart2b.odins.es:1027/ngsi-ld/v1/csourceRegistrations/?limit=1000&offset=0
```

In order to request the following 1000 items, the `offset` variable must be set like this:

```
https://smart2b.odins.es:1027/ngsi-ld/v1/csourceRegistrations/?limit=1000&offset=1000
```

- For more information about the high-level architecture and flow of Actuation, please refer to [Low level interaction for actuation with NGSI-LD Broker](#low_level_interaction_for_actuation_with_ngsi-ld_broker)

## Smart2B platform resource monitoring processes

Currently we have a series of processes that are executed automatically with certain periodicity and that are responsible for monitoring the status of the different resources of the machine where the Smart2B services are hosted.

These processes notify us by email in case that the percentage of disk, RAM or CPU usage of the machine exceeds certain values, so that we can act as soon as possible and prevent this from becoming a performance problem for Smart2B services.

In addition, we have another process that is responsible for periodically recording in a log the percentage of disk, RAM and CPU usage of the machine, so that we can know the status of the machine at all times and thus have more information in order to deal with possible incidents related to the operation of Smart2B services.



## Smart2B services monitoring and restart processes

We have a series of processes that run automatically and are responsible for monitoring the correct functioning of the Smart2B platform services, and also for restarting those services whose functioning is not correct.

### Scheduled restart of security components

We have an automatic process that runs every 4 hours and restarts the security components of the Smart2B platform: Keyrock, Capability Manager, PEP-Proxy, Historical PEP-Proxy and XACML.

The reason for these scheduled reboots is that they help us automatically refresh the security certificates (which even if we renew them on the Smart2B machine, they are not updated until the security components are restarted), and we also discovered that these reboots are good for prevent specific blockages of security components.

### Monitoring of security components

We have other automatic processes that run every minute and check that the APIs of the security components respond correctly, leaving a timeout of 10 seconds. If after this time the API does not respond, or if it does not respond correctly, these automatic processes will restart the corresponding components and will record these restarts in a log file.

The scheduled reboots mentioned above are not recorded in this log file.

### Scheduled restart of the Elasticsearch historical service

We have an automatic process that runs daily and restarts the Elasticsearch historical service of the Smart2B platform.

### Orion NGSI-LD broker monitoring

We also have an automatic process that runs every minute and checks if the Orion NGSI-LD broker service of the Smart2B platform is stopped or restarting, and if so, recreates the container of said service (without affecting the database).



## Study of the viability of calculating the price of the generated energy and integrating it into the Smart2B platform

The objective of this study was to verify if it is possible to obtain the price of the generated energy daily and by country, in an analogous way to how we already obtain the price of the consumed energy, and if possible, to make an assessment of the complexity of implement the necessary logic and integrate it into the services of the Smart2B platform.

Properties that have photovoltaic installations continue to be connected to the electrical grid, since normally these installations do not produce enough energy to supply the occupants indefinitely. In these cases, the energy produced by the photovoltaic installations is subtracted from the energy consumed from the electrical grid, so that the occupants end up paying only for the difference between the two. However, we think that there could be the case in which the energy produced by these photovoltaic installations exceeds the energy consumed from the electrical network for a certain property, and at this point, we think it would be interesting to try to find out if there are some way to calculate the price that the corresponding energy entity would pay for this energy surplus that would pass into the electrical grid. Furthermore, even if energy surpluses were not produced, by having the price of this generated energy it would be easy to calculate the money that the occupants of a property save on their bill thanks to the photovoltaic installations.

First of all, we remember that currently we already obtain the price of the energy consumed from the electrical grid daily and by country, thanks to ENTSO-E and a development of integration of this service with the Smart2B platform. In this way, every day we consult the ENTSO-E to obtain the kWh prices in each country and by time slot, and we register this information in a series of entities in our broker Orion NGSI-LD, so that they can be consulted using the services of the Smart2B platform.

Therefore, initially we tried to find out if it was possible to also obtain the prices of the generated energy in each country through ENTSO-E, and integrate this information with the services of the Smart2B platform in a similar way as we did for the prices of the consumed energy, but unfortunately we found that ENTSO-E does not have this information.

Then we began to look for another service from which we could obtain information related to the prices of the generated energy in each country, so that, if we found it, we could then assess the complexity of implementing a development that integrates said information with the services of the Smart2B platform.

Finally, and after investing considerable time in the search for a service from which to obtain the prices of the generated energy without success, we decided to abandon the search, and opted for the option of asking partners in each country to contact the corresponding energy entities to provide them with a fixed price for the generated kWh, in order to manually integrate this information with the services of the Smart2B platform.



## WP5 Integration Ongoing issues

{{:public:smart2b:wp3-5-arch.svg}}

- It takes a lot of time to get the data for the WP5 app whenever it queries the QuantumLeap + CrateDB for data to plot.
- This is very likely related with the complete chain that provides data to the QuantumLeap & CrateBD components in T3.2
- Aliki posts data each 60 minutes back to the QuantumLeap.


- [X] (ODINS) review the last emails with the exchanges to catch-up.
- [ ] (Jesús ODINS) Retrieve the queries that WP5 Manuel Fonseca FC.ID is using to query data from the platform
    - [X] (Sergio ODINS) Alternatively, use the credentials provided by FC.ID in order to try the requests interacting with the website to tray to accelerate working in this.
- [X] Nuno Matheus (EDP) reports that the platform is not working properly
    - [X] Assess exactly what components are not working.
    - [X] (Sergio ODINS) Aliki's email 2023-12-05 gives further information on why there was missing data from the QL.
- [ ] OdinS has in place several different service monitoring processes running in the background that did not raise any alarm. Nevertheless, this monitoring processes did not raise any alarm - look into it.
    - [X] (Sergio ODINS) - briefly document in maybe half a screen the service monitoring processes that we have in the background - what they monitor an the frequency to document the issues.
- [ ] No data has been found for November 2023 for:
    - SCML SantaTeresinha
    - [X] (Sergio ODINS) Aliki's email 2023-12-05 gives further information on why there was missing data from the QL.
- [ ] Assess in which step of the chain the issue might be happening: Sensors - Integratio - Platform - Data Filter - QuantumLeap
    - [X] (Sergio ODINS) Aliki's email 2023-12-05 gives further information on why there was missing data from the QL.
- [X] (Sergio ODINS) Nuno Dionisio (FC.ID) asks what the `Index` from the requests to the QuantumLeap are.
- [ ] (Jesús ODINS) The energy consumption data published against the Broker might be different.
    - (Jesús ODINS) In some cases it might be a cumulative counter that never stops increasing.
    - (Jesús ODINS) In other cases, the sensor reports each X period of minutes with the power consumed in that particular period of time.
    - [ ] (Jesús ODINS) Clarify which cases do we have in the project and set in place a mechanism that distinguishes among them to avoid misinterpretation by the APP.
- [ ] (FOR LATER) One of the biggest concerns for FC.ID is that there is a summary panel with all the information about a whole building in one single screen. This summary takes too much time to process since the queries are too slow. Hence, the matter should be reviewed after addressing the previous points so that the queries do not take so long and the summary screen can be shown in an acceptable rate.
- [ ] (FOR LATER) *NICE TO HAVE* - FC.ID could set some kind of cache memory in ephemeral storage for the data that was queried previously in case that the data cannot be retrieved from the platform at an acceptable rate.


```
Dear Nuno, Aliki, and Jesus,

We (WP5) have been connecting the WebApp with the API to access the data about consumption, temperature, CO2, etc., and we would like to report some situations and make some requests.

## Situations (assuming we are doing our calls correctly)
- Currently, the application takes a lot of time:
  - login, which includes getting the list of buildings belonging to the user: 1m23s
  - Receiving the data for the Details: 1m45s
  - Last Month's details: 2m30s

- Missing data:
  - There is only data until the end of October for temperature, CO2, and humidity (See "This Year" in the Time Interval)
  - Only consumption information is available for "Today"

Please test it yourself to determine where things are taking too long and if we are doing something wrong during the requests.
  - WebApp: https://project-smart2b.lasige.di.fc.ul.pt/
  - username/passwd: <REDACTED>


## Requests
- Data sent on each request
  - Currently, you are sending the average, maximum, and minimum values for each request (regardless of the type of information being requested)
  - However, it should be different for consumption/production and temperature/CO2/humidity
  - In the case of consumption/production, the value per hour/day/month should be the overall energy of that period, while for temperature/CO2/humidity it should be the average during the period
  - Thus, to simplify, when we request values, you just need to send one value (we will compute the average, maximum, and minimum for the presented chart).
  - Case 1: If we ask for the consumption/production in the "Last Month", we need/want to receive the total consumption/production for each day of the month.
  - Case 2: If we ask for the temperature/CO2/humidity in the "Last Month", we need/want to receive the average temperature/CO2/humidity for each day of the month.
  - In terms of the JSON sent for each request, it can be similar to this:
{
"id": "AOC-ForumComplex_Temperature",
"numValue": [
{
"value": 26.330648104349773,
"value_eur": 0.0,
"observedAt": "2023-10-01T00:00:00.000+00:00"
}
]
}

- Data for the overview screen of a building
  - In the overview screen, we show consumption, production, comfort, Status, and well-being data. As it is, to show all the values, we need to perform several requests to the API (which will be even slower). To optimize access to this information, we suggest the creation of a new end-point, which will provide all the information in a single request/JSON.
  - The idea is that these values will be computed automatically every hour (or another time interval more suitable) and stored in the platform, so when the WebApp performs a request all the values are already computed and ready to send in a single JSON
  - We need the following values for consumption/production:
    - Total consumption/production of the current day
    - Total consumption/production of the current month
    - Total consumption/production of the current year
  - We need the following values for Comfort
    - Average temperature/Humidity/CO2 in the last hour (or another time interval more suitable)


Please let me know if I have made myself clear or if you have any doubts.
If you consider it useful, we can arrange a meeting to discuss and clarify some of these points.

```


**Email by Aliki 2023-12-05 about lack of data**

```
Hello all,

Following a thorough investigation into the data posting within the aggregation module, I'm pleased to inform you that we've successfully identified and resolved the root cause behind some entities lacking data or experiencing gaps after the end of October. The issue, traced to a low-level programming aspect which I'm describing below in case you're curious, has been addressed on our end.
I would appreciate it if you could take a moment to test the aggregated data on your side to ensure everything is now functioning as expected. If not, please inform me at your earliest convenience.


The Issue causing the missing data and data gaps:

It appears that an issue arose concerning the data type when submitting information back to the QuantumLeap component. Please refer to the provided examples for clarification:

{"data": [{"id": "AOC-ForumComplex_FirstFloor-NA-NA-Elevators_OdinS-NA-MD0101E20010700105-TH_Humidity_Humidity",
          "type": "DeviceMeasurement",
               "measurement_type": {"type": "Text", "value": "Humidity"},
           "min_value": {"type": "Number", "value": "39.48"},
           "max_value": {"type": "Number", "value": "41.15"},
               "average": {"type": "Number", "value": "39.874285714285726"},
           "total_cost_eur": {"type": "Number", "value": "0.0"},
           "kwh_per_square_meter": {"type": "Number", "value": "nan"},
           "timestamp_start": {"type": "DateTime", "value": "2023-12-05T12:00:00"},
            "timestamp_end": {"type": "DateTime", "value": "2023-12-05T12:59:59"}}]}'

{"data": [{"id": "AOC-ForumComplex_FirstFloor-NA-NA-Elevators_OdinS-NA-MD0101E20010700105-TH_Humidity_Humidity",
          "type": "DeviceMeasurement",
               "measurement_type": {"type": "Text", "value": "Humidity"},
           "min_value": {"type": "Number", "value": "39.48"},
           "max_value": {"type": "Number", "value": "41.15"},
               "average": {"type": "Number", "value": "39.874285714285726"},
           "total_cost_eur": {"type": "Number", "value": "0.0"},
           "kwh_per_square_meter": {"type": "Number", "value": "None"},
           "timestamp_start": {"type": "DateTime", "value": "2023-12-05T12:00:00"},
            "timestamp_end": {"type": "DateTime", "value": "2023-12-05T12:59:59"}}]}'

The only difference between these examples lies in the value of "kwh_per_square_meter", where the first instance contains "nan" and the second instance contains "None". Upon attempting to post this data back to the QuantumLeap component, it was observed that when any attribute had a "nan" value, the data did not appear in the broker. Despite receiving a valid response confirming successful posting, the data was not reflected in the broker.

According to QuantumLeap documentation (https://quantumleap.readthedocs.io/en/latest/user/using/#data-casting), if an entity attribute was in origin received as a Number, following insert changing the same attribute to Text would fail. To mitigate this failures, QuantumLeap attempts data casting, if casting is not possible, values are replaced with None, to ensure that the insert in the database is not failing totally for the received entity.


As a result, the internal QuantumLeap process and our solution unfolds as follows:

1. Handling "nan" Values: When QuantumLeap detects "nan," it attempts to internally cast this value to a numeric type, specifically a "Number". If the internal casting operation fails, QuantumLeap takes a fallback approach and replaces the problematic "nan" value with "None."

2. Possible Internal Processing Issues: Even though a valid response is received, there remains uncertainty about the success of the internal data saving process within QuantumLeap. Due to limitations on our end, it's challenging to verify whether QuantumLeap encounters internal issues  preventing the successful storage of data containing "nan."

3. Resolution with "None" Value: Interestingly, changing the value from "nan" to "None" appears to circumvent potential issues during the data insertion process. The use of "None" as a replacement value seems to ensure a smoother operation, possibly overcoming internal challenges associated with handling "nan."

This issue remained unnoticed in numerous entities primarily because a substantial portion of them lacked "nan" values. However, entities housing comfortability data, encompassing temperature, humidity, and CO2, consistently exhibited this problem as their "kwh_per_square_meter" and "total_cost_eur" values were consistently set to "nan".

```

**Email Manuel FC.ID 2023-12-05**

```
Hi Aliki,

Now things seem faster, but we are still not getting any information about comfort.

You can try and see for yourself what is being provided by the API to the application.
By default when we enter (after selecting a building) the application shows the values for "Today". Then we can change the time interval by clicking on the icon on the top right. 

```


**Retrieve the queries that WP5 Manuel Fonseca FC.ID is using to query data from the platform. Alternatively, use the credentials provided by FC.ID in order to try the requests interacting with the website to tray to accelerate working in this**

```
We have been monitoring the logs of the QuantumLeap service and also the logs of the 2 PEP-Proxy services of the Smart2B platform, while we were trying to interact with the website "https://project-smart2b.lasige.di.fc.ul.pt/", but no matter what we did, the website always ended up hanging (it seems to be loading information, but it never finishes loading), and none of these interactions with the website have served to register any request to our QuantumLeap service. We have tried to launch some GET requests to QuantumLeap from Postman and these have generated logs, so we deduce that the website hangs before even making any request to QuantumLeap. However, if partners insist that we analyze the response times of our QuantumLeap service, we would need them to tell us what requests they send to our service from the web in order to reproduce it.
```


**Aliki's email 2023-12-05 gives further information on why there was missing data from the QL**

```
Aliki comments that when they sent information to QuantumLeap, when trying to write a null value on an attribute of a given entity, they did so by setting the "value" of said attribute to "nan". They noticed that when they launched a write request against our QuantumLeap, if the body of this request contained some "nan", even if QuantumLeap responded with a "200 - OK - Notification successfully processed", then they checked that it seemed as if the information contained in this write request would never have reached QuantumLeap. And now they have discovered that apparently the problem was that QuantumLeap does not process the value "nan" well, and that after systematically replacing it with "None" in its write requests, now all the information reaches QuantumLeap and can be consulted correctly.

We have tried launching a write request on a test entity in our QuantumLeap service whose body contained a "nan", and we have verified that although it got a "200 - OK - Notification successfully processed" in response, when later querying said entity it was as if the writing had never been done. Next, we tried to do the same but replacing the "nan" with "None", and although QuantumLeap's response is the same, this time we were able to consult the new information registered for this entity. Therefore, everything seems to indicate that Aliki's hypothesis is correct, and that the problem they reported due to lost data in QuantumLeap would already be solved.
```


**Nuno Dionisio (FC.ID) asks what the Index from the requests to the QuantumLeap are**

```
The only "Index" that we have found is the one in the header "--header 'Fiware-TimeIndex-Attribute: timestamp_start'" that is used in QuantumLeap write requests, and this is the information we have found about it in the Smart2B Wiki page regarding QuantumLeap:

"This argument is essential due to how CrateDB functions internally. It indicates that the entries registered in the data base are time-indexed by the timestamp_start property. Hence, following queries can specify time periods and those will be filtered internally by QuantumLeap using the timestamp_start property."
```


**[Smart2B] Issue report request - no data after October 2nd**

```
After reviewing the file "get_data_example.py" we have verified that the requests they are launching are against the endpoint "https://project-smart2b.lasige.di.fc.ul.pt/api", that is, they are not launching directly against our Smart2B platform, so we cannot know for sure if these issues are occurring before reaching our platform services. To be sure, we would need them to provide us with the exact requests they make against the services of our Smart2B platform.

However, we have also reviewed the "cjc_indoor_comfort_service.json" file and have launched a series of requests to obtain information about the entities listed in our Orion NGSI-LD broker, Elasticsearch and QuantumLeap services. Of the 14 entities listed, the first has not been modified in either the broker or Elasticsearch since the date "2023-10-02T20:36:19.297Z", and this entity does not appear to exist in QuantumLeap. However, the other 13 entities are being updated today in both the broker and Elasticsearch (and quite frequently), and all have historical records in QuantumLeap to date "2023-12-13T13:00:00.000+00:00".

Knowing this, we would ask Brida to contact the partners in charge of the endpoint "https://project-smart2b.lasige.di.fc.ul.pt/api" to try to find out why they are not recovering the expected data, and if they insist that it is a problem with our Smart2B platform, we will need them to provide us with the requests that they make strictly against the services of our platform, through the domain "smart2b.odins.es".
```

## Actuation

### Low level interaction for actuation with NGSI-LD Broker

**IMPORTANT** please note that in this scenario, the Sequence starter would be the T3.3 Translator developed by Andrea EB.
{{:public:smart2b:actuation.svg}}


{{:public:smart2b:actuator2.svg}}

### Procedure to check actuation issues.

- First of all, check that the entity is created in the broker, using the Entity Info Collection Spreadsheet. It must be in the `Actuators` tab of the spreadsheet.
- Check that the query is using the right attribute --- Column `J` in the spreadsheet.
- Check the status of the **Context Source registration** - **TODO**
    - E.g. Query with a GET request the same ACTUATOR entity and look at the `command` attributes and their info.
    - Query the broker itself to get relevant information about the Context Source registration for its entities --- should get the partner IP address and TCP port.
- Check that a Context Source Registration exists for the particular entity in the Broker. For this, follow [these instructions](#verification_of_context_source_registrations) --- otherwise no actuation will happen when the command reaches the broker.
- If the context source registration is not propertly done --- get in contact with the relevant pilot owner. (For SCML it would be EDP SEL, for AOC its either ODINS or EB, for ABL its Martin Kaufman, for CJC is EB, for ARA is EDP Miguel)
- If the context source is successfully registered, when the commands are not reaching the building equipment, get in contact with the relevant pilot owner.


## Get a complete list of Smart2B platform entities by type

Below is attached a JSON file that contains a Postman collection with the necessary requests to recover all the information corresponding to the entities of each type that are registered on the Smart2B platform:

{{:public:smart2b:smart2b-security-partners-allentities.postman_collection.zip}}

To import a collection of requests into Postman we must go to our Workspace, press the "Import" button and then select the corresponding JSON file.

This Postman collection contains the requests necessary to request an authentication token, an authorization token, and then retrieve the information for all entities of a specific type. To launch these last requests, we must always keep the "limit" header with a value of "1000", and vary the value of the "offset" header to recover the entities in packages of 1000 units. That is, the "offset" header must take the values "0", "1000", "2000", ... until no more entities are recovered.


## Healthcheck and Gotify monitoring

{{:public:smart2b:healthcheck_gotify.svg}}

Due to the recent issues due to services being down in pilot tests, we will integrate a simple watchdog mechanism.

The intention of this mechanism is to have a common notification platform (Gotify.odins.es) where all the relevant partners (specially the pilot owners) will be notified within minutes if a component is down due to some unknown reason.

### Steps to follow by non-technical partners (Project coordination or Pilot Leaders)

- Gotify is the chosen tool to notify relevant partners (coordination, pilot leaders, etc) that a service might be down or malfunctioning. When this happens, experiments or pilot visits must be discussed with the relevant technical partners to assess if it makes sense to cancel the visit/experiment ahead of time.

- In your favourite device, go to https://gotify.odins.es and use the following credentials.

- user: `smart2b`
- pass: `52MTf4hxcYnNJCrM`

Alternatively you might install the official Gotify SmartPhone Tools

- https://play.google.com/store/apps/details?id=com.github.gotify&hl=en_US

Check periodically this notification web when an experiment is near. If there are recent notifications, chances are that there is something not working as expected. Try to get in contact with the relevant partner ASAP and assess the situation in case that visits to the pilot sites are scheduled.

### Steps to implement by Integrating/Technical Partners

#### Gotify.odins.es

- Gotify is the chosen tool to notify other partners (coordination, pilot leaders, etc) that a service might be down or malfunctioning.

- ODINS is running our own instance of Gotify at https://gotify.odins.es

- Got there and use the following credentials:

- user: `smart2b`
- pass: `52MTf4hxcYnNJCrM`

- Next, create your own APP at the top titled after your partner --- here we have relative freedom to pick anything that makes sense to you, do not constrain yourself with the details.

{{:public:smart2b:20240228-152605.png}}

- **NOTE** you might disregard the `Priority`, it is something that has not been discussed yet at a technical level and do whatever. For us all notifications are to be attended so priority is not a criteria to pick which one to attend.

- Choose your created APP and get your authorization token:

{{:public:smart2b:20240228-152829.png}}

- This token is the one that must be employed by the next section.

- Besides, the gotify procedure might be employed in case that something is going wrong or an issue has been detected proactively.

- https://gotify.net/docs/more-pushmsg contains the instructions that you might employ in case you can go around `healthchecks.io`




#### Healtchecks.io

- Each partner must create their own account in the free service https://healthchecks.io/

- Next create a project within the application

{{:public:smart2b:20240228-150847.png}}

- Next create a check with the most frequency possible, adapted to your needs, but we are talking a matter of minutes.

{{:public:smart2b:20240228-151016.png}}

- Click the three-dot menu to setup the check

{{:public:smart2b:20240228-151106.png}}

- Click on the Use examples in order to know what queries must be run from the watched component towards healthcheck

{{:public:smart2b:20240228-151220.png}}

**NOTE** that many different languages/apps are available in order to perform the regular query. Pick whichever option you prefer.

- Next, click the `INTEGRATIONS` tab and choose your preferred methods, among them email is suggested.

- Next, click the `Gotify` integration, which is the notification portal that will be employed within **Smart2B**.

- In the `Gotify Server URL` type `https://gotify.odins.es`

- In the `Application Token` paste the APP token that you grabbed from `gotify.odins.es`.

{{:public:smart2b:20240228-151638.png}}

It should look like this.

{{:public:smart2b:20240228-151738.png}}

- It is recommended that you try to test the integration and see if the notification reaches Gotify

{{:public:smart2b:20240228-151958.png}}

- Henceforth, the monitored component must ping the `healthchecks.io` server within the specified period of time. Otherwise, it will post a notification towards Gotify, so that all the partners are aware that maybe something is wrong and can react accordingly.