All-sky search in API

Helpdesk + feature requests
1

Dear Astro-COLIBRI Team - first of all thanks for your hard work!

On the web-interface it is quite easy to get a list of all (e.g.) GRB events found during a set timeframe - I would love to get this list from the API aswell, for further processing. Mid-december I figured I could do this using the cone-search endpoint, and it worked quite well, just setting the radius to a high number. Now, however, I get timeouts when using a radius > ca. 20. Is there a way to circumvent that and still get my required all-sky list?

The code Iā€™ve been using is exactly from the notebook under ā€œUse the cone search endpoint with custom filtersā€ and setting radius to e.g. 100.

Thank you in advance!

2

Hi!

Thanks a lot for reporting this!
We can indeed reproduce your error. At first sight it looks like it is due to an inconsistency in one of our databases that cause the request to fail. Weā€™ll have a closer look (and hopefully a solution) soon.

Btw. we are also already working on implementing a dedicated endpoint for these searches. It will most likely be called /latest_transients and will allow to retrieve the full list of transients fitting a given filter settings in a given time window. This will be accessible from the API but also directly from our interfaces. It is schedule to become available with the next major release.

1 Like
3

The release v2.9.0 (January 15, 2024) introduces the latest_transients endpoint that can be used from the API but also from the graphical interfaces to download selected events. Note that timeouts can still occur for large requests (i.e. covering very long time ranges) due to limited bandwith to our underlying database.

2 Likes
4

Hi all!
the latest_transients endpoint documentation refers to ā€œThe filter structure (see dedicated documentation)ā€ . Where can i find the documentation?
Is it possible possible to refer to my profile filtering in the request?

Thank you in advance!

5

Thanks a lot for the feedback!
Indeed, the statement is not very clear and weā€™ll update it. What it should point to is the interactive notebook available on Google Colab

This should give you an introduction to the definition of the event filters. Note that you can download the notebook as jupyiter notebook, copy it to your own GoogleColab space, etc. in order to retain the changes you may want to do to it.

With ā€œprofile filteringā€ you probably mean the filters set up in the graphical interfaces? Two point related to that:

  • At this stage you can not use your settings in the interface in a request to the API. But you could use the interfaces directly to download the events fullfilling your selection criteria. There is a ā€œdownloadā€ button in the top row of the web-interface and at the bottom of the filtering menu in the app: Capture dā€™eĢcran 2024-03-02 aĢ€ 10.34.27
  • Let me use this opportunity to report on a known issue with the filter settings: they are currently not stored correctly in your user account. This means that the default filters are re-activated when reloading the web-interface and that the synchronization between the web and app interfaces is not working as expected. We are working towards a solutionā€¦
1 Like
6

Hi all; Iā€™m writing a desktop program (with Delphi) to download selected events.
Therefore, Iā€™m setting up a Json Filter to collect events (according to ā€˜typeā€™ i,e. SN,OT,Unclassified OT).
Assuming that it is possible with a spec ā€˜fieldspecificationā€™ , What should be the related ā€˜fieldā€™ and ā€˜valueā€™ keywords ?
Any extended documentation about the possibilities of the spec ā€˜fieldspecificationā€™ ?
Thanks.

1 Like
7

Hi Jean-Paul,

The keyword field refers to the fieldā€™s name in the database on which to apply the filter. We do not yet have detailed documentation on this point, but as it is the same as the keyword used in the JSON returned, everything can be inferred from the API endpointā€™s answer.

The keyword value refers to the value to which the data will be compared. The exact impact will depend on the operation chosen (with the keyword operation).

As example :

  • In this first example, the system will compare the data stored in the DB field observatory and keep only entries with data identical to the given value, aka icecube.
    { 'type': 'FieldSpecification', 'field': 'observatory', 'value': 'icecube', 'operation': '==', 'typeField': 'string' }

  • In this second example, the system will compare the value stored in the DB field magnitude and keep the one below the value of 18.
    { 'type': 'FieldSpecification', 'field': 'magnitude', 'value': 18, 'operation': '<', 'typeField': 'float' }

I hope it is clearer now; otherwise, let me know. I could help you to construct the first requests.

1 Like
8

Hi Mathieu,
Iā€™m still exploring some issues with /latest_transientsā€¦
the system will compare the value stored in the DB field magnitude and keep the one below the value of 18. The field ā€˜magnitudeā€™ does not appears in the VoEvents output; Using the notebook, changing the magnitude limit value has no effect on number of results returned (391).

Concerning the type of events, the documentation page is not strict about the desination to use for a query SN SNe OT OTs.
Thanks for your precious help.
JPG

9

Hi Jean-Paul,

Thanks for your feedback. Iā€™ve noticed there were a mistake in the notebook. The magnitude field should have been named transient_flux. For optical transients, the value correspond the magnitude at detection. If you look at the notebook you should see the correct version now, sorry for the confusion caused by the mistake.

You could found below a filter to gather all supernovae, with a magnitude below 18.

my_filters = {
    'type': 'AndSpecification',
    'specs': [
      {
        'type': 'FieldSpecification',
        'field': 'transient_flux',
        'value': 18,
        'operation': '<',
        'typeField': 'float'
      },
      {
        'type': 'FieldSpecification',
        'field': 'type',
        'value': 'ot_sn',
        'operation': '==',
        'typeField': 'string'
      }
    ],
}

If you are looking for a specific type of supernovae, you could look at the classification field in the returned data and add a filter on this field.

If you are looking at different kind of optical transient, instead for type to search the value ot_sn, you could keep the values ot_other for all the other types of classified transients or ot for all the non classified transients.

2 Likes
10

Hi Mathieu,
I have currently problems with my filter which stopped working with ā€˜error 400 Bad requestā€™.
I tried with yours (perhaps outof date) and got the same message.
Do we have to change anything with the new version?
Thanks for your follow up.

1 Like
11

Hi Jean-Paul,

thanks for reporting this!
There was indeed a bug in the latest version that prevented queries like yours. I just launched a new instance of the API that should fix this issue. Can you give it another try and let us know?

12

Thanks Fabian. Itā€™s now running fine.

1 Like
13

Hi Fabian,
Return of the problem with ā€˜error 400 Bad requestā€™ .
How to run the API with the filters selected and kept for my profile?

14

That is very strange as nothing has changed (at first sight) on our end. Can you send me the filter settings you are using so that we can run some tests?
We havenā€™t (yet) a way to use the filters stored in the user profiles for API calls. But this is something weā€™ll be working on soon.

15

Hi Fabian,
Sorry for this false alert: using an invalid module late in the night.

1 Like