public:smart2b:home

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
public:smart2b:home [2023/12/20 13:23]
sergio.lujan_um.es [WP5 Integration Ongoing issues] 		public:smart2b:home [2024/10/09 08:53] (current)
Line 163: Line 163:
 - Note the `ABL-.*` at the end of the URL. As a consequence, this authorization token will be valid only to perform the `PATCH` operation to those entities. If the developer wishes to perform PATCH operations against entities that have a different prefix (e.g. `ARA-`), then the authorization token must be adjusted. - Note the `ABL-.*` at the end of the URL. As a consequence, this authorization token will be valid only to perform the `PATCH` operation to those entities. If the developer wishes to perform PATCH operations against entities that have a different prefix (e.g. `ARA-`), then the authorization token must be adjusted.
  
 +### Verification of Context Source Registrations
  
 +After registering the Context Source it can be checked if the registrations process was successful --- i.e. the integrating partner software component that receives the messages from the NGSI-LD Broker in WP3 Smart2B Platform is supposed to be registered before actuation can carry over from the WP3 Smart2B platform all the way down to the building equipment.
 +
 +The following POSTMAN collection can be employed in order to verify if the Context Source Registration took place successfully.
 +
 +{{:public:smart2b:smart2b-security-partners-allcontextproviders.postman_collection.json.zip}}
 +
 +**IMPORTANT**
 +
 +There is a limitation of `1000` items per query, specified as an URL option like this:
 +
 +```
 +https://smart2b.odins.es:1027/ngsi-ld/v1/csourceRegistrations/?limit=1000&offset=0
 +```
 +
 +In order to request the following 1000 items, the `offset` variable must be set like this:
 +
 +```
 +https://smart2b.odins.es:1027/ngsi-ld/v1/csourceRegistrations/?limit=1000&offset=1000
 +```
 +
 +- For more information about the high-level architecture and flow of Actuation, please refer to [Low level interaction for actuation with NGSI-LD Broker](#low_level_interaction_for_actuation_with_ngsi-ld_broker)
  
 ## Smart2B platform resource monitoring processes ## Smart2B platform resource monitoring processes
Line 198: Line 220:
  
 We also have an automatic process that runs every minute and checks if the Orion NGSI-LD broker service of the Smart2B platform is stopped or restarting, and if so, recreates the container of said service (without affecting the database). We also have an automatic process that runs every minute and checks if the Orion NGSI-LD broker service of the Smart2B platform is stopped or restarting, and if so, recreates the container of said service (without affecting the database).
 +
 +
 +
 +## Study of the viability of calculating the price of the generated energy and integrating it into the Smart2B platform
 +
 +The objective of this study was to verify if it is possible to obtain the price of the generated energy daily and by country, in an analogous way to how we already obtain the price of the consumed energy, and if possible, to make an assessment of the complexity of implement the necessary logic and integrate it into the services of the Smart2B platform.
 +
 +Properties that have photovoltaic installations continue to be connected to the electrical grid, since normally these installations do not produce enough energy to supply the occupants indefinitely. In these cases, the energy produced by the photovoltaic installations is subtracted from the energy consumed from the electrical grid, so that the occupants end up paying only for the difference between the two. However, we think that there could be the case in which the energy produced by these photovoltaic installations exceeds the energy consumed from the electrical network for a certain property, and at this point, we think it would be interesting to try to find out if there are some way to calculate the price that the corresponding energy entity would pay for this energy surplus that would pass into the electrical grid. Furthermore, even if energy surpluses were not produced, by having the price of this generated energy it would be easy to calculate the money that the occupants of a property save on their bill thanks to the photovoltaic installations.
 +
 +First of all, we remember that currently we already obtain the price of the energy consumed from the electrical grid daily and by country, thanks to ENTSO-E and a development of integration of this service with the Smart2B platform. In this way, every day we consult the ENTSO-E to obtain the kWh prices in each country and by time slot, and we register this information in a series of entities in our broker Orion NGSI-LD, so that they can be consulted using the services of the Smart2B platform.
 +
 +Therefore, initially we tried to find out if it was possible to also obtain the prices of the generated energy in each country through ENTSO-E, and integrate this information with the services of the Smart2B platform in a similar way as we did for the prices of the consumed energy, but unfortunately we found that ENTSO-E does not have this information.
 +
 +Then we began to look for another service from which we could obtain information related to the prices of the generated energy in each country, so that, if we found it, we could then assess the complexity of implementing a development that integrates said information with the services of the Smart2B platform.
 +
 +Finally, and after investing considerable time in the search for a service from which to obtain the prices of the generated energy without success, we decided to abandon the search, and opted for the option of asking partners in each country to contact the corresponding energy entities to provide them with a fixed price for the generated kWh, in order to manually integrate this information with the services of the Smart2B platform.
  
  
Line 359: Line 397:
 We have been monitoring the logs of the QuantumLeap service and also the logs of the 2 PEP-Proxy services of the Smart2B platform, while we were trying to interact with the website "https://project-smart2b.lasige.di.fc.ul.pt/", but no matter what we did, the website always ended up hanging (it seems to be loading information, but it never finishes loading), and none of these interactions with the website have served to register any request to our QuantumLeap service. We have tried to launch some GET requests to QuantumLeap from Postman and these have generated logs, so we deduce that the website hangs before even making any request to QuantumLeap. However, if partners insist that we analyze the response times of our QuantumLeap service, we would need them to tell us what requests they send to our service from the web in order to reproduce it. We have been monitoring the logs of the QuantumLeap service and also the logs of the 2 PEP-Proxy services of the Smart2B platform, while we were trying to interact with the website "https://project-smart2b.lasige.di.fc.ul.pt/", but no matter what we did, the website always ended up hanging (it seems to be loading information, but it never finishes loading), and none of these interactions with the website have served to register any request to our QuantumLeap service. We have tried to launch some GET requests to QuantumLeap from Postman and these have generated logs, so we deduce that the website hangs before even making any request to QuantumLeap. However, if partners insist that we analyze the response times of our QuantumLeap service, we would need them to tell us what requests they send to our service from the web in order to reproduce it.
 ``` ```
 +
 +
 +**Aliki's email 2023-12-05 gives further information on why there was missing data from the QL**
 +
 +```
 +Aliki comments that when they sent information to QuantumLeap, when trying to write a null value on an attribute of a given entity, they did so by setting the "value" of said attribute to "nan". They noticed that when they launched a write request against our QuantumLeap, if the body of this request contained some "nan", even if QuantumLeap responded with a "200 - OK - Notification successfully processed", then they checked that it seemed as if the information contained in this write request would never have reached QuantumLeap. And now they have discovered that apparently the problem was that QuantumLeap does not process the value "nan" well, and that after systematically replacing it with "None" in its write requests, now all the information reaches QuantumLeap and can be consulted correctly.
 +
 +We have tried launching a write request on a test entity in our QuantumLeap service whose body contained a "nan", and we have verified that although it got a "200 - OK - Notification successfully processed" in response, when later querying said entity it was as if the writing had never been done. Next, we tried to do the same but replacing the "nan" with "None", and although QuantumLeap's response is the same, this time we were able to consult the new information registered for this entity. Therefore, everything seems to indicate that Aliki's hypothesis is correct, and that the problem they reported due to lost data in QuantumLeap would already be solved.
 +```
 +
 +
 +**Nuno Dionisio (FC.ID) asks what the Index from the requests to the QuantumLeap are**
 +
 +```
 +The only "Index" that we have found is the one in the header "--header 'Fiware-TimeIndex-Attribute: timestamp_start'" that is used in QuantumLeap write requests, and this is the information we have found about it in the Smart2B Wiki page regarding QuantumLeap:
 +
 +"This argument is essential due to how CrateDB functions internally. It indicates that the entries registered in the data base are time-indexed by the timestamp_start property. Hence, following queries can specify time periods and those will be filtered internally by QuantumLeap using the timestamp_start property."
 +```
 +
 +
 +**[Smart2B] Issue report request - no data after October 2nd**
 +
 +```
 +After reviewing the file "get_data_example.py" we have verified that the requests they are launching are against the endpoint "https://project-smart2b.lasige.di.fc.ul.pt/api", that is, they are not launching directly against our Smart2B platform, so we cannot know for sure if these issues are occurring before reaching our platform services. To be sure, we would need them to provide us with the exact requests they make against the services of our Smart2B platform.
 +
 +However, we have also reviewed the "cjc_indoor_comfort_service.json" file and have launched a series of requests to obtain information about the entities listed in our Orion NGSI-LD broker, Elasticsearch and QuantumLeap services. Of the 14 entities listed, the first has not been modified in either the broker or Elasticsearch since the date "2023-10-02T20:36:19.297Z", and this entity does not appear to exist in QuantumLeap. However, the other 13 entities are being updated today in both the broker and Elasticsearch (and quite frequently), and all have historical records in QuantumLeap to date "2023-12-13T13:00:00.000+00:00".
 +
 +Knowing this, we would ask Brida to contact the partners in charge of the endpoint "https://project-smart2b.lasige.di.fc.ul.pt/api" to try to find out why they are not recovering the expected data, and if they insist that it is a problem with our Smart2B platform, we will need them to provide us with the requests that they make strictly against the services of our platform, through the domain "smart2b.odins.es".
 +```
 +
 +## Actuation
 +
 +### Low level interaction for actuation with NGSI-LD Broker
 +
 +**IMPORTANT** please note that in this scenario, the Sequence starter would be the T3.3 Translator developed by Andrea EB.
 +{{:public:smart2b:actuation.svg}}
 +
 +
 +{{:public:smart2b:actuator2.svg}}
 +
 +### Procedure to check actuation issues.
 +
 +- First of all, check that the entity is created in the broker, using the Entity Info Collection Spreadsheet. It must be in the `Actuators` tab of the spreadsheet.
 +- Check that the query is using the right attribute --- Column `J` in the spreadsheet.
 +- Check the status of the **Context Source registration** - **TODO**
 +    - E.g. Query with a GET request the same ACTUATOR entity and look at the `command` attributes and their info.
 +    - Query the broker itself to get relevant information about the Context Source registration for its entities --- should get the partner IP address and TCP port.
 +- Check that a Context Source Registration exists for the particular entity in the Broker. For this, follow [these instructions](#verification_of_context_source_registrations) --- otherwise no actuation will happen when the command reaches the broker.
 +- If the context source registration is not propertly done --- get in contact with the relevant pilot owner. (For SCML it would be EDP SEL, for AOC its either ODINS or EB, for ABL its Martin Kaufman, for CJC is EB, for ARA is EDP Miguel)
 +- If the context source is successfully registered, when the commands are not reaching the building equipment, get in contact with the relevant pilot owner.
 +
 +
 +## Get a complete list of Smart2B platform entities by type
 +
 +Below is attached a JSON file that contains a Postman collection with the necessary requests to recover all the information corresponding to the entities of each type that are registered on the Smart2B platform:
 +
 +{{:public:smart2b:smart2b-security-partners-allentities.postman_collection.zip}}
 +
 +To import a collection of requests into Postman we must go to our Workspace, press the "Import" button and then select the corresponding JSON file.
 +
 +This Postman collection contains the requests necessary to request an authentication token, an authorization token, and then retrieve the information for all entities of a specific type. To launch these last requests, we must always keep the "limit" header with a value of "1000", and vary the value of the "offset" header to recover the entities in packages of 1000 units. That is, the "offset" header must take the values "0", "1000", "2000", ... until no more entities are recovered.
 +
 +
 +## Healthcheck and Gotify monitoring
 +
 +{{:public:smart2b:healthcheck_gotify.svg}}
 +
 +Due to the recent issues due to services being down in pilot tests, we will integrate a simple watchdog mechanism.
 +
 +The intention of this mechanism is to have a common notification platform (Gotify.odins.es) where all the relevant partners (specially the pilot owners) will be notified within minutes if a component is down due to some unknown reason.
 +
 +### Steps to follow by non-technical partners (Project coordination or Pilot Leaders)
 +
 +- Gotify is the chosen tool to notify relevant partners (coordination, pilot leaders, etc) that a service might be down or malfunctioning. When this happens, experiments or pilot visits must be discussed with the relevant technical partners to assess if it makes sense to cancel the visit/experiment ahead of time.
 +
 +- In your favourite device, go to https://gotify.odins.es and use the following credentials.
 +
 +- user: `smart2b`
 +- pass: `52MTf4hxcYnNJCrM`
 +
 +Alternatively you might install the official Gotify SmartPhone Tools
 +
 +- https://play.google.com/store/apps/details?id=com.github.gotify&hl=en_US
 +
 +Check periodically this notification web when an experiment is near. If there are recent notifications, chances are that there is something not working as expected. Try to get in contact with the relevant partner ASAP and assess the situation in case that visits to the pilot sites are scheduled.
 +
 +### Steps to implement by Integrating/Technical Partners
 +
 +#### Gotify.odins.es
 +
 +- Gotify is the chosen tool to notify other partners (coordination, pilot leaders, etc) that a service might be down or malfunctioning.
 +
 +- ODINS is running our own instance of Gotify at https://gotify.odins.es
 +
 +- Got there and use the following credentials:
 +
 +- user: `smart2b`
 +- pass: `52MTf4hxcYnNJCrM`
 +
 +- Next, create your own APP at the top titled after your partner --- here we have relative freedom to pick anything that makes sense to you, do not constrain yourself with the details.
 +
 +{{:public:smart2b:20240228-152605.png}}
 +
 +- **NOTE** you might disregard the `Priority`, it is something that has not been discussed yet at a technical level and do whatever. For us all notifications are to be attended so priority is not a criteria to pick which one to attend.
 +
 +- Choose your created APP and get your authorization token:
 +
 +{{:public:smart2b:20240228-152829.png}}
 +
 +- This token is the one that must be employed by the next section.
 +
 +- Besides, the gotify procedure might be employed in case that something is going wrong or an issue has been detected proactively.
 +
 +- https://gotify.net/docs/more-pushmsg contains the instructions that you might employ in case you can go around `healthchecks.io`
 +
 +
 +
 +
 +#### Healtchecks.io
 +
 +- Each partner must create their own account in the free service https://healthchecks.io/
 +
 +- Next create a project within the application
 +
 +{{:public:smart2b:20240228-150847.png}}
 +
 +- Next create a check with the most frequency possible, adapted to your needs, but we are talking a matter of minutes.
 +
 +{{:public:smart2b:20240228-151016.png}}
 +
 +- Click the three-dot menu to setup the check
 +
 +{{:public:smart2b:20240228-151106.png}}
 +
 +- Click on the Use examples in order to know what queries must be run from the watched component towards healthcheck
 +
 +{{:public:smart2b:20240228-151220.png}}
 +
 +**NOTE** that many different languages/apps are available in order to perform the regular query. Pick whichever option you prefer.
 +
 +- Next, click the `INTEGRATIONS` tab and choose your preferred methods, among them email is suggested.
 +
 +- Next, click the `Gotify` integration, which is the notification portal that will be employed within **Smart2B**.
 +
 +- In the `Gotify Server URL` type `https://gotify.odins.es`
 +
 +- In the `Application Token` paste the APP token that you grabbed from `gotify.odins.es`.
 +
 +{{:public:smart2b:20240228-151638.png}}
 +
 +It should look like this.
 +
 +{{:public:smart2b:20240228-151738.png}}
 +
 +- It is recommended that you try to test the integration and see if the notification reaches Gotify
 +
 +{{:public:smart2b:20240228-151958.png}}
 +
 +- Henceforth, the monitored component must ping the `healthchecks.io` server within the specified period of time. Otherwise, it will post a notification towards Gotify, so that all the partners are aware that maybe something is wrong and can react accordingly.
 +
 +
 +
  
  • public/smart2b/home.1703075027.txt.gz
  • Last modified: 2024/10/09 08:35
  • (external edit)