Group discussion > Tools who can help converting spreadsheets to DwC measurement and facts standard

Tools who can help converting spreadsheets to DwC measurement and facts standard

David Valentim Dias
125 days ago

Hi group,

There is a long time since we didn't bring any discussion at this group.

I'm curious about how other nodes are solving the problem of converting spreadsheets to DwC meansurements and facts extension.

Here at SiBBr we're using the mycenaConverte (source and description https://github.com/sibbr/tableconverter and in production https://ferramentas.sibbr.gov.br/mycena/). mycenaConverte asks for spreadsheets and columns you want to transpose to measurements and facts extensions.

At GB23 iirc Taiwan presented some scripts for this goal too.

I think each country is replicating this function. Can you share you solution?


Dag Endresen
20 days ago

OBIS has presented an interesting "Extended Measurement Or Fact" extension for reporting experimental data (Darwin Core: MeasurementOrFact) with an Event Core dataset:
 http://rs.gbif.org/extension/obis/extended_measurement_or_fact.xml (2016-09-05)

See also:
https://www.slideshare.net/OBIS-IOC/expanding-obis-beyond-species-occurrence-data-including-sampling-event-facts-and-measurements-related-to-the-environment (2016-06-08)

De Pooter D, Appeltans W, Bailly N, Bristol S, Deneudt K, Eliezer M, Fujioka E, Giorgetti A, Goldstein P, Lewis M, Lipizer M, Mackay K, Marin M, Moncoiffé G, Nikolopoulou S, Provoost P, Rauch S, Roubicek A, Torres C, van de Putte A, Vandepitte L, Vanhoorne B, Vinci M, Wambiji N, Watts D, Klein Salas E, Hernandez F (2017) Toward a new data standard for combined marine biological and environmental datasets - expanding OBIS beyond species occurrences. Biodiversity Data Journal 5: e10989. https://doi.org/10.3897/BDJ.5.e10989

David Valentim Dias
20 days ago

Thanks Dag,

Here at SiBBr since our first course in sampling event data we're recommending OBIS extension. The need to publish two resources for meansures about event and occurrence seemed a bit silly.

My real concearn is the inequality of tecnology between nodes. I hope soon SiBBr will be realeasing source code of our tools.

Some of our less known tools functionalities are:

  • Automatic conversion to .dbf from Brahms -> DwC spreadsheet. With this automation we published 1 million records in period of four months
  • Proxy API to automatic: publish, delete, change visibility, check existence of resources at IPT, generate automatic report (record count, publishing date etc) of all resources in any IPT we want
  • Create meta.xml from occurrence.txt and eml.xml, combined with above solution we can create and export DwC-A without any need of IPT
  • Harvester: based on work at lontra-harvester we recreated the "indexing" process, with little capacitacion any biologist can change or include filters (like data quality). Currently under testing
  • Others tools are made public at GB23 SiBBr presentation

Some of these tools can improve the process of publication across GBIF network.