Title: | Access Data from the European Tracking Network |
---|---|
Description: | Package with functions to access and process data from the European Tracking Network hosted by VLIZ. |
Authors: | Pieter Huybrechts [aut, cre] , Peter Desmet [aut] , Damiano Oldoni [aut] , Stijn Van Hoey [aut] , Research Institute for Nature and Forest (INBO) [cph] (https://www.vlaanderen.be/inbo/en-gb/), Research Foundation - Flanders [fnd] (https://lifewatch.be) |
Maintainer: | Pieter Huybrechts <[email protected]> |
License: | MIT + file LICENSE |
Version: | 2.2.1 |
Built: | 2024-10-28 06:37:22 UTC |
Source: | https://github.com/inbo/etn |
Connect to the ETN database using username and password.
connect_to_etn(username = Sys.getenv("userid"), password = Sys.getenv("pwd"))
connect_to_etn(username = Sys.getenv("userid"), password = Sys.getenv("pwd"))
username |
Character. Username to use for the connection. |
password |
Character. Password to use for the connection. |
ODBC connection to ETN database.
## Not run: # Connect to the ETN database using your rstudio.lifewatch.be username and # password, and save as the default connection variable "con" con <- connect_to_etn() # Connect to the ETN database using non-default username and password con <- connect_to_etn(username = "my_username", password = "my_password") ## End(Not run)
## Not run: # Connect to the ETN database using your rstudio.lifewatch.be username and # password, and save as the default connection variable "con" con <- connect_to_etn() # Connect to the ETN database using non-default username and password con <- connect_to_etn(username = "my_username", password = "my_password") ## End(Not run)
Download all acoustic data related to an animal project as a data package that can be deposited in a research data repository. Includes option to filter on scientific names.
download_acoustic_dataset( connection = con, animal_project_code, scientific_name = NULL, directory = animal_project_code )
download_acoustic_dataset( connection = con, animal_project_code, scientific_name = NULL, directory = animal_project_code )
connection |
A connection to the ETN database. Defaults to |
animal_project_code |
Character. Animal project you want to download data for. Required. |
scientific_name |
Character (vector). One or more scientific names. Defaults to no all (all scientific names, include "Sync tag", etc.). |
directory |
Character. Relative path to local download directory. Defaults to creating a directory named after animal project code. Existing files of the same name will be overwritten. |
The data are downloaded as a Frictionless Data Package containing:
file | description |
animals.csv |
Animals related to an animal_project_code , optionally filtered on scientific_name (s), as returned by get_animals() . |
tags.csv |
Tags associated with the selected animals, as returned by get_tags() . |
detections.csv |
Acoustic detections for the selected animals, as returned by get_acoustic_detections() . |
deployments.csv |
Acoustic deployments for the acoustic_project_code (s) found in detections, as returned by get_acoustic_deployments() . This allows users to see when receivers were deployed, even if these did not detect the selected animals. |
receivers.csv |
Acoustic receivers for the selected deployments, as returned by get_acoustic_receivers() . |
datapackage.json |
A Frictionless Table Schema metadata file describing the fields and relations of the above csv files. This file is copied from here and can be used to validate the data package. |
The function will report the number of records per csv file, as well as the included scientific names and acoustic projects. Warnings will be raised for:
Animals with multiple tags
Tags associated with multiple animals
Deployments without acoustic project: these deployments will not be listed
in deployments.csv
and will therefore raise a foreign key validation
error.
Duplicate detections: detections with the duplicate detection_id
. These
are removed by the function in detections.csv
.
Important: The data are downloaded as is from the database, i.e. no
quality or consistency checks are performed by this function. We therefore
recommend to verify the data before publication. A consistency check can be
performed by validation tools of the Frictionless Framework, e.g.
frictionless validate datapackage.json
on the command line using
frictionless-py.
## Not run: # Set default connection variable con <- connect_to_etn() # Download data for the 2012_leopoldkanaal animal project (all scientific names) download_acoustic_dataset(animal_project_code = "2012_leopoldkanaal") #> Downloading data to directory `2012_leopoldkanaal`: #> * (1/6): downloading animals.csv #> * (2/6): downloading tags.csv #> * (3/6): downloading detections.csv #> * (4/6): downloading deployments.csv #> * (5/6): downloading receivers.csv #> * (6/6): adding datapackage.json as file metadata #> #> Summary statistics for dataset `2012_leopoldkanaal`: #> * number of animals: 104 #> * number of tags: 103 #> * number of detections: 2215243 #> * number of deployments: 1968 #> * number of receivers: 454 #> * first date of detection: 2012-07-04 #> * last date of detection: 2021-09-02 #> * included scientific names: Anguilla anguilla #> * included acoustic projects: albert, Apelafico, bpns, JJ_Belwind, leopold, MOBEIA, pc4c, SPAWNSEIS, ws2, zeeschelde #> #> Warning message: #> In download_acoustic_dataset(animal_project_code = "2012_leopoldkanaal") : #> Found tags associated with multiple animals: 1145373 ## End(Not run)
## Not run: # Set default connection variable con <- connect_to_etn() # Download data for the 2012_leopoldkanaal animal project (all scientific names) download_acoustic_dataset(animal_project_code = "2012_leopoldkanaal") #> Downloading data to directory `2012_leopoldkanaal`: #> * (1/6): downloading animals.csv #> * (2/6): downloading tags.csv #> * (3/6): downloading detections.csv #> * (4/6): downloading deployments.csv #> * (5/6): downloading receivers.csv #> * (6/6): adding datapackage.json as file metadata #> #> Summary statistics for dataset `2012_leopoldkanaal`: #> * number of animals: 104 #> * number of tags: 103 #> * number of detections: 2215243 #> * number of deployments: 1968 #> * number of receivers: 454 #> * first date of detection: 2012-07-04 #> * last date of detection: 2021-09-02 #> * included scientific names: Anguilla anguilla #> * included acoustic projects: albert, Apelafico, bpns, JJ_Belwind, leopold, MOBEIA, pc4c, SPAWNSEIS, ws2, zeeschelde #> #> Warning message: #> In download_acoustic_dataset(animal_project_code = "2012_leopoldkanaal") : #> Found tags associated with multiple animals: 1145373 ## End(Not run)
Get data for deployments of acoustic receivers, with options to filter results.
get_acoustic_deployments( connection = con, deployment_id = NULL, receiver_id = NULL, acoustic_project_code = NULL, station_name = NULL, open_only = FALSE )
get_acoustic_deployments( connection = con, deployment_id = NULL, receiver_id = NULL, acoustic_project_code = NULL, station_name = NULL, open_only = FALSE )
connection |
A connection to the ETN database. Defaults to |
deployment_id |
Integer (vector). One or more deployment identifiers. |
receiver_id |
Character (vector). One or more receiver identifiers. |
acoustic_project_code |
Character (vector). One or more acoustic project codes. Case-insensitive. |
station_name |
Character (vector). One or more deployment station names. |
open_only |
Logical. Restrict deployments to those that are currently
open (i.e. no end date defined). Defaults to |
A tibble with acoustic deployment data, sorted by
acoustic_project_code
, station_name
and deploy_date_time
. See also
field definitions.
# Set default connection variable con <- connect_to_etn() # Get all acoustic deployments get_acoustic_deployments(con) # Get specific acoustic deployment get_acoustic_deployments(con, deployment_id = 1437) # Get acoustic deployments for a specific receiver get_acoustic_deployments(con, receiver_id = "VR2W-124070") # Get open acoustic deployments for a specific receiver get_acoustic_deployments(con, receiver_id = "VR2W-124070", open_only = TRUE) # Get acoustic deployments for a specific acoustic project get_acoustic_deployments(con, acoustic_project_code = "demer") # Get acoustic deployments for two specific stations get_acoustic_deployments(con, station_name = c("de-9", "de-10"))
# Set default connection variable con <- connect_to_etn() # Get all acoustic deployments get_acoustic_deployments(con) # Get specific acoustic deployment get_acoustic_deployments(con, deployment_id = 1437) # Get acoustic deployments for a specific receiver get_acoustic_deployments(con, receiver_id = "VR2W-124070") # Get open acoustic deployments for a specific receiver get_acoustic_deployments(con, receiver_id = "VR2W-124070", open_only = TRUE) # Get acoustic deployments for a specific acoustic project get_acoustic_deployments(con, acoustic_project_code = "demer") # Get acoustic deployments for two specific stations get_acoustic_deployments(con, station_name = c("de-9", "de-10"))
Get data for acoustic detections, with options to filter results. Use
limit
to limit the number of returned records.
get_acoustic_detections( connection = con, start_date = NULL, end_date = NULL, acoustic_tag_id = NULL, animal_project_code = NULL, scientific_name = NULL, acoustic_project_code = NULL, receiver_id = NULL, station_name = NULL, limit = FALSE )
get_acoustic_detections( connection = con, start_date = NULL, end_date = NULL, acoustic_tag_id = NULL, animal_project_code = NULL, scientific_name = NULL, acoustic_project_code = NULL, receiver_id = NULL, station_name = NULL, limit = FALSE )
connection |
A connection to the ETN database. Defaults to |
start_date |
Character. Start date (inclusive) in ISO 8601 format (
|
end_date |
Character. End date (exclusive) in ISO 8601 format (
|
acoustic_tag_id |
Character (vector). One or more acoustic tag ids. |
animal_project_code |
Character (vector). One or more animal project codes. Case-insensitive. |
scientific_name |
Character (vector). One or more scientific names. |
acoustic_project_code |
Character (vector). One or more acoustic project codes. Case-insensitive. |
receiver_id |
Character (vector). One or more receiver identifiers. |
station_name |
Character (vector). One or more deployment station names. |
limit |
Logical. Limit the number of returned records to 100 (useful
for testing purposes). Defaults to |
A tibble with acoustic detections data, sorted by acoustic_tag_id
and date_time
. See also
field definitions.
# Set default connection variable con <- connect_to_etn() # Get limited sample of acoustic detections get_acoustic_detections(con, limit = TRUE) # Get all acoustic detections from a specific animal project get_acoustic_detections(con, animal_project_code = "2014_demer") # Get 2015 acoustic detections from that animal project get_acoustic_detections( con, animal_project_code = "2014_demer", start_date = "2015", end_date = "2016", ) # Get April 2015 acoustic detections from that animal project get_acoustic_detections( con, animal_project_code = "2014_demer", start_date = "2015-04", end_date = "2015-05", ) # Get April 24, 2015 acoustic detections from that animal project get_acoustic_detections( con, animal_project_code = "2014_demer", start_date = "2015-04-24", end_date = "2015-04-25", ) # Get acoustic detections for a specific tag at two specific stations get_acoustic_detections( con, acoustic_tag_id = "A69-1601-16130", station_name = c("de-9", "de-10") ) # Get acoustic detections for a specific species, receiver and acoustic project get_acoustic_detections( con, scientific_name = "Rutilus rutilus", receiver_id = "VR2W-124070", acoustic_project_code = "demer" )
# Set default connection variable con <- connect_to_etn() # Get limited sample of acoustic detections get_acoustic_detections(con, limit = TRUE) # Get all acoustic detections from a specific animal project get_acoustic_detections(con, animal_project_code = "2014_demer") # Get 2015 acoustic detections from that animal project get_acoustic_detections( con, animal_project_code = "2014_demer", start_date = "2015", end_date = "2016", ) # Get April 2015 acoustic detections from that animal project get_acoustic_detections( con, animal_project_code = "2014_demer", start_date = "2015-04", end_date = "2015-05", ) # Get April 24, 2015 acoustic detections from that animal project get_acoustic_detections( con, animal_project_code = "2014_demer", start_date = "2015-04-24", end_date = "2015-04-25", ) # Get acoustic detections for a specific tag at two specific stations get_acoustic_detections( con, acoustic_tag_id = "A69-1601-16130", station_name = c("de-9", "de-10") ) # Get acoustic detections for a specific species, receiver and acoustic project get_acoustic_detections( con, scientific_name = "Rutilus rutilus", receiver_id = "VR2W-124070", acoustic_project_code = "demer" )
Get data for acoustic projects, with options to filter results.
get_acoustic_projects(connection = con, acoustic_project_code = NULL)
get_acoustic_projects(connection = con, acoustic_project_code = NULL)
connection |
A connection to the ETN database. Defaults to |
acoustic_project_code |
Character (vector). One or more acoustic project codes. Case-insensitive. |
A tibble with acoustic project data, sorted by project_code
. See
also
field definitions.
# Set default connection variable con <- connect_to_etn() # Get all acoustic projects get_acoustic_projects(con) # Get a specific acoustic project get_acoustic_projects(con, acoustic_project_code = "demer")
# Set default connection variable con <- connect_to_etn() # Get all acoustic projects get_acoustic_projects(con) # Get a specific acoustic project get_acoustic_projects(con, acoustic_project_code = "demer")
Get data for acoustic receivers, with options to filter results.
get_acoustic_receivers(connection = con, receiver_id = NULL, status = NULL)
get_acoustic_receivers(connection = con, receiver_id = NULL, status = NULL)
connection |
A connection to the ETN database. Defaults to |
receiver_id |
Character (vector). One or more receiver identifiers. |
status |
Character. One or more statuses, e.g. |
A tibble with acoustic receiver data, sorted by receiver_id
. See
also
field definitions.
Values for owner_organization
will only be visible if you are member of
the group.
# Set default connection variable con <- connect_to_etn() # Get all acoustic receivers get_acoustic_receivers(con) # Get lost and broken acoustic receivers get_acoustic_receivers(con, status = c("lost", "broken")) # Get a specific acoustic receiver get_acoustic_receivers(con, receiver_id = "VR2W-124070")
# Set default connection variable con <- connect_to_etn() # Get all acoustic receivers get_acoustic_receivers(con) # Get lost and broken acoustic receivers get_acoustic_receivers(con, status = c("lost", "broken")) # Get a specific acoustic receiver get_acoustic_receivers(con, receiver_id = "VR2W-124070")
Get data for animal projects, with options to filter results.
get_animal_projects(connection = con, animal_project_code = NULL)
get_animal_projects(connection = con, animal_project_code = NULL)
connection |
A connection to the ETN database. Defaults to |
animal_project_code |
Character (vector). One or more animal project codes. Case-insensitive. |
A tibble with animal project data, sorted by project_code
. See
also
field definitions.
# Set default connection variable con <- connect_to_etn() # Get all animal projects get_animal_projects(con) # Get a specific animal project get_animal_projects(con, animal_project_code = "2014_demer")
# Set default connection variable con <- connect_to_etn() # Get all animal projects get_animal_projects(con) # Get a specific animal project get_animal_projects(con, animal_project_code = "2014_demer")
Get data for animals, with options to filter results. Associated tag
information is available in columns starting with tag
and
acoustic_tag_id
. If multiple tags are associated with a single animal,
the information is comma-separated.
get_animals( connection = con, animal_id = NULL, tag_serial_number = NULL, animal_project_code = NULL, scientific_name = NULL )
get_animals( connection = con, animal_id = NULL, tag_serial_number = NULL, animal_project_code = NULL, scientific_name = NULL )
connection |
A connection to the ETN database. Defaults to |
animal_id |
Integer (vector). One or more animal identifiers. |
tag_serial_number |
Character (vector). One or more tag serial numbers. |
animal_project_code |
Character (vector). One or more animal project codes. Case-insensitive. |
scientific_name |
Character (vector). One or more scientific names. |
A tibble with animals data, sorted by animal_project_code
,
release_date_time
and tag_serial_number
. See also
field definitions.
# Set default connection variable con <- connect_to_etn() # Get all animals get_animals(con) # Get specific animals get_animals(con, animal_id = 305) # Or string value "305" get_animals(con, animal_id = c(304, 305, 2827)) # Get animals from specific animal project(s) get_animals(con, animal_project_code = "2014_demer") get_animals(con, animal_project_code = c("2014_demer", "2015_dijle")) # Get animals associated with a specific tag_serial_number get_animals(con, tag_serial_number = "1187450") # Get animals of specific species (across all projects) get_animals(con, scientific_name = c("Rutilus rutilus", "Silurus glanis")) # Get animals of a specific species from a specific project get_animals(con, animal_project_code = "2014_demer", scientific_name = "Rutilus rutilus")
# Set default connection variable con <- connect_to_etn() # Get all animals get_animals(con) # Get specific animals get_animals(con, animal_id = 305) # Or string value "305" get_animals(con, animal_id = c(304, 305, 2827)) # Get animals from specific animal project(s) get_animals(con, animal_project_code = "2014_demer") get_animals(con, animal_project_code = c("2014_demer", "2015_dijle")) # Get animals associated with a specific tag_serial_number get_animals(con, tag_serial_number = "1187450") # Get animals of specific species (across all projects) get_animals(con, scientific_name = c("Rutilus rutilus", "Silurus glanis")) # Get animals of a specific species from a specific project get_animals(con, animal_project_code = "2014_demer", scientific_name = "Rutilus rutilus")
Get data for cpod projects, with options to filter results.
get_cpod_projects(connection = con, cpod_project_code = NULL)
get_cpod_projects(connection = con, cpod_project_code = NULL)
connection |
A connection to the ETN database. Defaults to |
cpod_project_code |
Character (vector). One or more cpod project codes. Case-insensitive. |
A tibble with animal project data, sorted by project_code
. See
also
field definitions.
# Set default connection variable con <- connect_to_etn() # Get all animal projects get_cpod_projects(con) # Get a specific animal project get_cpod_projects(con, cpod_project_code = "cpod-lifewatch")
# Set default connection variable con <- connect_to_etn() # Get all animal projects get_cpod_projects(con) # Get a specific animal project get_cpod_projects(con, cpod_project_code = "cpod-lifewatch")
Get data for tags, with options to filter results. Note that there
can be multiple records (acoustic_tag_id
) per tag device
(tag_serial_number
).
get_tags( connection = con, tag_type = NULL, tag_subtype = NULL, tag_serial_number = NULL, acoustic_tag_id = NULL )
get_tags( connection = con, tag_type = NULL, tag_subtype = NULL, tag_serial_number = NULL, acoustic_tag_id = NULL )
connection |
A connection to the ETN database. Defaults to |
tag_type |
Character (vector). |
tag_subtype |
Character (vector). |
tag_serial_number |
Character (vector). One or more tag serial numbers. |
acoustic_tag_id |
Character (vector). One or more acoustic tag
identifiers, i.e. identifiers found in |
A tibble with tags data, sorted by tag_serial_number
. See also
field definitions.
Values for owner_organization
and owner_pi
will only be visible if you
are member of the group.
# Set default connection variable con <- connect_to_etn() # Get all tags get_tags(con) # Get archival tags, including acoustic-archival get_tags(con, tag_type = c("archival", "acoustic-archival")) # Get tags of specific subtype get_tags(con, tag_subtype = c("built-in", "range")) # Get specific tags (note that these can return multiple records) get_tags(con, tag_serial_number = "1187450") get_tags(con, acoustic_tag_id = "A69-1601-16130") get_tags(con, acoustic_tag_id = c("A69-1601-16129", "A69-1601-16130"))
# Set default connection variable con <- connect_to_etn() # Get all tags get_tags(con) # Get archival tags, including acoustic-archival get_tags(con, tag_type = c("archival", "acoustic-archival")) # Get tags of specific subtype get_tags(con, tag_subtype = c("built-in", "range")) # Get specific tags (note that these can return multiple records) get_tags(con, tag_serial_number = "1187450") get_tags(con, acoustic_tag_id = "A69-1601-16130") get_tags(con, acoustic_tag_id = c("A69-1601-16129", "A69-1601-16130"))
List all available acoustic project codes
list_acoustic_project_codes(connection = con)
list_acoustic_project_codes(connection = con)
connection |
A connection to the ETN database. Defaults to |
A vector of all unique project_code
of type = "acoustic"
in
project.sql
.
List all available acoustic tag ids
list_acoustic_tag_ids(connection = con)
list_acoustic_tag_ids(connection = con)
connection |
A connection to the ETN database. Defaults to |
A vector of all unique acoustic_tag_id
in acoustic_tag_id.sql
.
List all available animal ids
list_animal_ids(connection = con)
list_animal_ids(connection = con)
connection |
A connection to the ETN database. Defaults to |
A vector of all unique id_pk
present in common.animal_release
.
List all available animal project codes
list_animal_project_codes(connection = con)
list_animal_project_codes(connection = con)
connection |
A connection to the ETN database. Defaults to |
A vector of all unique project_code
of type = "animal"
in
project.sql
.
List all available cpod project codes
list_cpod_project_codes(connection = con)
list_cpod_project_codes(connection = con)
connection |
A connection to the ETN database. Defaults to |
A vector of all unique project_code
of type = "cpod"
in
project.sql
.
List all available receiver ids
list_deployment_ids(connection = con)
list_deployment_ids(connection = con)
connection |
A connection to the ETN database. Defaults to |
A vector of all unique id_pk
present in acoustic.deployments
.
List all available receiver ids
list_receiver_ids(connection = con)
list_receiver_ids(connection = con)
connection |
A connection to the ETN database. Defaults to |
A vector of all unique receiver
present in acoustic.receivers
.
List all available scientific names
list_scientific_names(connection = con)
list_scientific_names(connection = con)
connection |
A connection to the ETN database. Defaults to |
A vector of all unique scientific_name
present in
common.animal_release
.
List all available station names
list_station_names(connection = con)
list_station_names(connection = con)
connection |
A connection to the ETN database. Defaults to |
A vector of all unique station_name
present in
acoustic.deployments
.
List all available tag serial numbers
list_tag_serial_numbers(connection = con)
list_tag_serial_numbers(connection = con)
connection |
A connection to the ETN database. Defaults to |
A vector of all unique tag_serial_numbers
present in
common.tag_device
.
Get a vector with all unique values found in a given column of a data.frame.
Concatenated values (A,B
) in the column can be returned as single values
(A
and B
).
list_values(.data, column, split = ",")
list_values(.data, column, split = ",")
.data |
Data frame. Data.frame to select column from. |
column |
Character or integer. Quoted or unqoted column name or column position. |
split |
Character (vector). Character or regular expression(s) passed
to |
A vector of the same type as the given column.
# Set default connection variable con <- connect_to_etn() library(dplyr) # For %>% # List unique scientific_name from a dataframe containing animal information df <- get_animals(con, animal_project_code = "2014_demer") list_values(df, "scientific_name") # Or using pipe and unquoted column name df %>% list_values(scientific_name) # Or using column position df %>% list_values(8) # tag_serial_number can contain comma-separated values df <- get_animals(con, animal_id = 5841) df$tag_serial_number # list_values() will split those and return unique values list_values(df, tag_serial_number) # Another expression can be defined to split values (here ".") list_values(df, tag_serial_number, split = "\\.")
# Set default connection variable con <- connect_to_etn() library(dplyr) # For %>% # List unique scientific_name from a dataframe containing animal information df <- get_animals(con, animal_project_code = "2014_demer") list_values(df, "scientific_name") # Or using pipe and unquoted column name df %>% list_values(scientific_name) # Or using column position df %>% list_values(8) # tag_serial_number can contain comma-separated values df <- get_animals(con, animal_id = 5841) df$tag_serial_number # list_values() will split those and return unique values list_values(df, tag_serial_number) # Another expression can be defined to split values (here ".") list_values(df, tag_serial_number, split = "\\.")
Transforms and downloads data from a European Tracking Network
animal project to Darwin Core.
The resulting CSV file(s) can be uploaded to an IPT for publication to OBIS and/or GBIF.
A meta.xml
or eml.xml
file are not created.
write_dwc( connection = con, animal_project_code, directory = ".", rights_holder = NULL, license = "CC-BY" )
write_dwc( connection = con, animal_project_code, directory = ".", rights_holder = NULL, license = "CC-BY" )
connection |
Connection to the ETN database. |
animal_project_code |
Animal project code. |
directory |
Path to local directory to write file(s) to.
If |
rights_holder |
Acronym of the organization owning or managing the rights over the data. |
license |
Identifier of the license under which the data will be published. |
CSV file(s) written to disk or list of data frames when
directory = NULL
.
Data are transformed into an Occurrence core. This follows recommendations discussed and created by Peter Desmet, Jonas Mortelmans, Jonathan Pye, John Wieczorek and others. See the SQL file(s) used by this function for details.
Key features of the Darwin Core transformation:
Deployments (animal+tag associations) are parent events, with capture,
surgery, release, recapture (human observations) and acoustic detections
(machine observations) as child events.
No information about the parent event is provided other than its ID,
meaning that data can be expressed in an Occurrence Core with one row per
observation and parentEventID
shared by all occurrences in a deployment.
The release event often contains metadata about the animal (sex, lifestage, comments) and deployment as a whole.
Acoustic detections are downsampled to the first detection per hour, to reduce the size of high-frequency data. Duplicate detections (same animal, tag and timestamp) are excluded. It is possible for a deployment to contain no detections, e.g. if the tag malfunctioned right after deployment.