... | ... | @@ -6,13 +6,13 @@ The use of command line utilities such as `pepcli` is subject to details of the |
|
|
|
|
|
The `pepcli` utility must be invoked from a command line, with parameters telling it what to do. The general form of invocation is
|
|
|
|
|
|
```
|
|
|
```plaintext
|
|
|
pepcli [general flags] <COMMAND> [flags] [parameters...]
|
|
|
```
|
|
|
|
|
|
The various [commands are documented](#commands) in some detail on this page. The general flags are [documented separately](#general-flags). Some commands have subcommands; some subcommands have sub-sub-commands, and so on:
|
|
|
|
|
|
```
|
|
|
```plaintext
|
|
|
pepcli [general flags] <COMMAND> <SUBCOMMAND> [flags] [parameters...]
|
|
|
pepcli [general flags] <COMMAND> <SUBCOMMAND> <SUBSUBCOMMAND> [flags] [parameters...]
|
|
|
```
|
... | ... | @@ -23,7 +23,7 @@ The abilities and options of underlying commands are documented with the parent |
|
|
|
|
|
The `pepcli` application provides command line help if it is invoked without parameters, or with the `--help` switch:
|
|
|
|
|
|
```
|
|
|
```plaintext
|
|
|
pepcli # Produces command line help
|
|
|
pepcli --help # Produces command line help
|
|
|
```
|
... | ... | @@ -31,13 +31,10 @@ pepcli --help # Produces command line help |
|
|
The `--help` switch is also supported by most (or all?) of `pepcli`'s commands and subcommands. This can be used to "drill down" through command line help to construct an appropriate command line, e.g. by sequentially invoking:
|
|
|
|
|
|
<pre>
|
|
|
pepcli --help # Output includes the <em>ama</em> command, so we then issue:
|
|
|
pepcli ama --help # Output includes the <em>query</em> subcommand, so we then issue:
|
|
|
pepcli ama query --help # Output mentions the <em>--column-group</em> switch, so we then issue:
|
|
|
pepcli ama query --column-group ShortPseudonyms # The completed command line
|
|
|
</pre>
|
|
|
|
|
|
## Enrollment
|
|
|
pepcli --help # Output includes the _ama_ command, so we then issue: pepcli ama --help # Output includes the _query_ subcommand, so we then issue: pepcli ama query --help # Output mentions the _\--column-group_ switch, so we then issue: pepcli ama query --column-group ShortPseudonyms # The completed command line
|
|
|
|
|
|
</pre>## Enrollment
|
|
|
|
|
|
Most of `pepcli`'s commands will connect to one or more of the PEP servers, and most server requests will require the user to be [enrolled](Access-control#enrollment). There are two primary methods get enrolled for `pepcli` usage:
|
|
|
|
... | ... | @@ -56,7 +53,7 @@ The `pepcli` utility's general flags can be used to indicate how to connect to a |
|
|
|
|
|
For brevity, these general flags will not be mentioned in the documentation of individual commands, or in the examples below. But they may be included with any command issued to `pepcli`, e.g.:
|
|
|
|
|
|
```
|
|
|
```plaintext
|
|
|
pepcli --oauth-token /PATH/TO/OAuthToken.json --client-working-directory /PATH/TO/config-directory list -C \* -P \*
|
|
|
```
|
|
|
|
... | ... | @@ -69,9 +66,9 @@ The `pepcli` utility supports commands for various tasks, aimed at different typ |
|
|
General purpose:
|
|
|
|
|
|
- [`query`](#query) provides information on your access to the PEP environment.
|
|
|
- [`query column-access`](#query-column-access) lists the columns and column groups accessible to the enrolled user.
|
|
|
- [`query participant-group-access`](#query-participant-group-access) lists the participant groups accessible to the enrolled user.
|
|
|
- [`query enrollment`](#query-enrollment) tells users how they're enrolled.
|
|
|
- [`query column-access`](#query-column-access) lists the columns and column groups accessible to the enrolled user.
|
|
|
- [`query participant-group-access`](#query-participant-group-access) lists the participant groups accessible to the enrolled user.
|
|
|
- [`query enrollment`](#query-enrollment) tells users how they're enrolled.
|
|
|
|
|
|
Data storage and retrieval:
|
|
|
|
... | ... | @@ -83,19 +80,20 @@ Data storage and retrieval: |
|
|
Administrative tasks:
|
|
|
|
|
|
- [`ama`](#ama) provides subcommands to perform administrative tasks related to the Access Manager service.
|
|
|
- [`ama query`](ama-query) summarizes the current data structure and access rules.
|
|
|
- For users enrolled as a `Data Administrator`:
|
|
|
- [`ama column`](ama-column) can be used to create and remove columns, and to group and un-group them.
|
|
|
- [`ama columnGroup`](ama-columngroup) can be used to create and remove column groups.
|
|
|
- For users enrolled as an `Access Administrator`:
|
|
|
- [`ama cgar`](ama-cgar) can be used to manage the type(s) of access that access groups have to column groups.
|
|
|
|
|
|
- [`ama query`](ama-query) summarizes the current data structure and access rules.
|
|
|
- For users enrolled as a `Data Administrator`:
|
|
|
- [`ama column`](ama-column) can be used to create and remove columns, and to group and un-group them.
|
|
|
- [`ama columnGroup`](ama-columngroup) can be used to create and remove column groups.
|
|
|
- [`ama group`](ama-group) can be used to create and remove participant groups, and add participants to them
|
|
|
- [`ama group auto-assign`](ama-group-auto-assign) can be used to automatically fill participant groups based on study contexts.
|
|
|
- For users enrolled as an `Access Administrator`:
|
|
|
- [`ama cgar`](ama-cgar) can be used to manage the type(s) of access that access groups have to column groups.
|
|
|
- [`castor`](#castor) provides subcommands to perform administrative tasks related to PEP's [Castor integration](Castor-integration) functionality.
|
|
|
- [`castor export`](#castor-export) @@@ TODO @@@
|
|
|
- [`castor list-sp-columns`](#castor-list-sp-columns) lists short pseudonym columns that are bound to Castor studies.
|
|
|
- [`castor list-import-columns`](#castor-list-import-columns) lists columns required by PEP's [Castor import](Castor-integration#import).
|
|
|
- [`castor create-import-columns`](#castor-create-import-columns) creates columns required by PEP's [Castor import](Castor-integration#import).
|
|
|
- [`castor column-name-mapping`](#castor-column-name-mapping) allows a Data Administrator to configure [column name mappings](Castor-integration#column-name-mappings) to have PEP [import Castor data](Castor-integration#import) into appropriately named columns.
|
|
|
- [`castor export`](#castor-export) @@@ TODO @@@
|
|
|
- [`castor list-sp-columns`](#castor-list-sp-columns) lists short pseudonym columns that are bound to Castor studies.
|
|
|
- [`castor list-import-columns`](#castor-list-import-columns) lists columns required by PEP's [Castor import](Castor-integration#import).
|
|
|
- [`castor create-import-columns`](#castor-create-import-columns) creates columns required by PEP's [Castor import](Castor-integration#import).
|
|
|
- [`castor column-name-mapping`](#castor-column-name-mapping) allows a Data Administrator to configure [column name mappings](Castor-integration#column-name-mappings) to have PEP [import Castor data](Castor-integration#import) into appropriately named columns.
|
|
|
|
|
|
## `ama`
|
|
|
|
... | ... | @@ -105,7 +103,7 @@ The `ama` command's various sub-commands can be used to perform administrative t |
|
|
|
|
|
The `cgar` subcommand is short for "column group access rule". It allows `Access Administrator` to determine the types of access that access groups have to column groups:
|
|
|
|
|
|
```
|
|
|
```plaintext
|
|
|
pepcli ama cgar create <column group name> <access group name> <mode>
|
|
|
pepcli ama cgar remove <column group name> <access group name> <mode>
|
|
|
```
|
... | ... | @@ -118,7 +116,7 @@ After using the `cgar` subcommand, the rule immediately takes effect. Users enro |
|
|
|
|
|
The `column` subcommand allows `Data Administrator` to create and remove columns:
|
|
|
|
|
|
```
|
|
|
```plaintext
|
|
|
pepcli ama column create <column name>
|
|
|
pepcli ama column remove <column name>
|
|
|
```
|
... | ... | @@ -132,20 +130,20 @@ Note that column removal will not discard data present in those columns; it will |
|
|
|
|
|
The `column` subcommand can also be used to group and un-group columns into column groups:
|
|
|
|
|
|
```
|
|
|
```plaintext
|
|
|
pepcli ama column addTo <column name> <column group name>
|
|
|
pepcli ama column removeFrom <column name> <column group name>
|
|
|
```
|
|
|
|
|
|
When columns are added to a column group, those columns immediately become available to users who can access the column group.
|
|
|
|
|
|
Column groups can be created and removed using [the `pepcli ama columnGroup`](#ama-columngroup) subcommand. `Access Administrator` can [grant access to column groups](Access-control#access-rules) using the `pepcli ama cgar` subcommand.
|
|
|
Column groups can be created and removed using [the `pepcli ama columnGroup`](#ama-columngroup) subcommand. `Access Administrator` can [grant access to column groups](Access-control#access-rules) using the `pepcli ama cgar` subcommand.
|
|
|
|
|
|
### `ama columnGroup`
|
|
|
|
|
|
The `columnGroup` subcommand allows `Data Administrator` to create and remove column groups:
|
|
|
|
|
|
```
|
|
|
```plaintext
|
|
|
pepcli ama columnGroup create <column group name>
|
|
|
pepcli ama columnGroup remove <column group name>
|
|
|
```
|
... | ... | @@ -155,30 +153,42 @@ Because of technical limitations, column group names may contain only [printable |
|
|
Once a column group has been created, use the [`pepcli ama column`](#ama-column) subcommand to determine which columns are included in the group. `Access Administrator` can [grant access to column groups](Access-control#access-rules) using the `pepcli ama cgar` subcommand.
|
|
|
|
|
|
### `ama group`
|
|
|
|
|
|
The `group` subcommand allows `Data Administrator` to create and remove participant groups:
|
|
|
|
|
|
```
|
|
|
```plaintext
|
|
|
pepcli ama group create <participant group name>
|
|
|
pepcli ama group remove <participant group name>
|
|
|
```
|
|
|
|
|
|
The `group` subcommand also allows to add and remove participants to and from participant groups:
|
|
|
|
|
|
```
|
|
|
```plaintext
|
|
|
pepcli ama group addTo <participant group name> <participant>
|
|
|
pepcli ama group removeFrom <participant group name> <participant>
|
|
|
```
|
|
|
|
|
|
There are multiple ways to specify `<participant>`:
|
|
|
|
|
|
- A PEP-id
|
|
|
- A _Local Pseudonym_. This is the pseudonym that is used as directory name, when doing `pepcli pull` (this will be replaced by the shorter _Participant Alias_ with the pep release planned in Februari or March 2023)
|
|
|
- A _Participant Alias_ . This is a shorter version, derived from (cropping the) _Local Pseudonym_ . (formerly called _user pseudonym_)
|
|
|
|
|
|
### `ama group auto-assign`
|
|
|
|
|
|
Participants can be part of one or more _contexts_. This is administered in the pepAssessor GUI, during the data collection phase. Participants can also be marked as being test participants. PEP can automatically derive participant groups based on this information. This can be done with:
|
|
|
|
|
|
```
|
|
|
pepcli ama group auto-assign
|
|
|
```
|
|
|
|
|
|
For every existing study context, this command will add participants that are in that context to the participant group with the name `all-[STUDY_CONTEXT]`. Test participants are not added. If the group does not exist, it will be created.
|
|
|
|
|
|
### `ama query`
|
|
|
|
|
|
The `query` subcommand summarizes the current state of PEP's [data structure](Data-structure) and [access rules](Access-control#access-rules). Both the `Access Administrator` and `Data Administrator` roles can invoke:
|
|
|
|
|
|
```
|
|
|
```plaintext
|
|
|
pepcli ama query
|
|
|
```
|
|
|
|
... | ... | @@ -211,7 +221,7 @@ Manipulation of column name mappings requires enrollment as a `Data Administrato |
|
|
|
|
|
This command ensures that all columns exist that will be needed when the specified Castor data are [imported into PEP](https://gitlab.pep.cs.ru.nl/pep-public/user-docs/-/wikis/Castor-integration#import). Since the import process cannot create columns itself, Data Administrator can use this command to create them beforehand. When invoked without further parameters, the command creates missing columns for all Castor data that will be imported. Each created column is also automatically added to the `Castor` column group, ensuring that the column is writable to the import process:
|
|
|
|
|
|
```
|
|
|
```plaintext
|
|
|
pepcli castor create-import-columns
|
|
|
```
|
|
|
|
... | ... | @@ -222,7 +232,7 @@ The above switch-less invocation may lack information required to create import |
|
|
|
|
|
E.g. when a Castor study is bound to short pseudonym column `ShortPseudonym.Covid.Castor.CovidQuestionnaires`, and that study contains survey packages that are answered up to 30 times:
|
|
|
|
|
|
```
|
|
|
```plaintext
|
|
|
pepcli castor create-import-columns --sp-column ShortPseudonym.Covid.Castor.CovidQuestionnaires --answer-set-count 30
|
|
|
```
|
|
|
|
... | ... | @@ -234,7 +244,7 @@ pepcli castor create-import-columns --sp-column ShortPseudonym.Covid.Castor.Covi |
|
|
|
|
|
This command lists the columns that are needed when the specified Castor data are [imported into PEP](https://gitlab.pep.cs.ru.nl/pep-public/user-docs/-/wikis/Castor-integration#import). Since the import process cannot create columns itself, Data Administrator must create required columns beforehand. (S)he then uses this command to find out which columns are needed. The command must be invoked with the `--sp-column` switch, specifying the column containing short pseudonym values that correspond with the Castor study's record IDs. E.g.:
|
|
|
|
|
|
```
|
|
|
```plaintext
|
|
|
pepcli castor list-import-columns --sp-column ShortPseudonym.Visit1.Castor.HomeQuestionnaires`
|
|
|
```
|
|
|
|
... | ... | @@ -254,7 +264,7 @@ The command accepts an optional `--imported-only` switch. If specified, the comm |
|
|
|
|
|
When the [`pepcli list`](#list) command has produced IDs referring to data, the associated data can be retrieved using the `pepcli get` command:
|
|
|
|
|
|
```
|
|
|
```plaintext
|
|
|
pepcli get -t <ticket file> -i <identifier> -o <output file>
|
|
|
```
|
|
|
|
... | ... | @@ -270,11 +280,11 @@ Note that the `pepcli get` command is not capable of data re-pseudonymization. D |
|
|
|
|
|
Use the `pepcli list` command to determine which data is available in PEP:
|
|
|
|
|
|
```
|
|
|
```plaintext
|
|
|
pepcli list -C <column group> -P <participant group> -T <ticket out file>
|
|
|
```
|
|
|
|
|
|
The command outputs its information as a JSON array with one entry per subject. Every such entry contains the subject's polymorphic pseudonym, plus an array listing the columns in which data is stored for the subject. Depending on switches passed to the command, small data may be *inlined*, i.e. included in the command's output. For larger entries, the output will include an ID instead of the data itself. Such IDs can then be passed to the [`pepcli get` command](#get) to retrieve the associated data.
|
|
|
The command outputs its information as a JSON array with one entry per subject. Every such entry contains the subject's polymorphic pseudonym, plus an array listing the columns in which data is stored for the subject. Depending on switches passed to the command, small data may be _inlined_, i.e. included in the command's output. For larger entries, the output will include an ID instead of the data itself. Such IDs can then be passed to the [`pepcli get` command](#get) to retrieve the associated data.
|
|
|
|
|
|
Important flags are:
|
|
|
|
... | ... | @@ -283,7 +293,7 @@ Important flags are: |
|
|
- `-P` Participant group to list data for. Can be repeated if you want data for more than one participant group. There is a special participant group `*` that contains all participants.
|
|
|
- `-p` Specific participant to list data for. Can be repeated, and combined with `-P` if you want multiple participants and participant groups
|
|
|
- `-l` Include the local pseudonyms in the output. By default pepcli will only show polymorphic pseudonyms (PP). These are not constant, and cannot be used to see whether data belongs to the same participant. You need the local pseudonyms (LP) for that.
|
|
|
- `-T` The first thing PEP does when you interact with it, is checking whether you have access to the participant(group)s and column(group)s you request. If you do have access, it will hand out a ticket. You can store this ticket with the `-T` flag, to use it for later actions.
|
|
|
- `-T` The first thing PEP does when you interact with it, is checking whether you have access to the participant(group)s and column(group)s you request. If you do have access, it will hand out a ticket. You can store this ticket with the `-T` flag, to use it for later actions.
|
|
|
- `-t` You can pass a ticket from an earlier request with the `-t` flag. The column(group)s and participant(group)s of this request must be a subset of the earlier request.
|
|
|
- `-s` The size limit (in bytes) for data that should be inlined, i.e. be included the the `list` command's output. Currently defaults to 1000. Setting this to 0 means that data will ALWAYS be inlined
|
|
|
- `--no-inline-data` Never inline data
|
... | ... | @@ -295,7 +305,7 @@ Note that the `pepcli list` command is not capable of data re-pseudonymization. |
|
|
|
|
|
The `pull` command downloads a data set from PEP and stores the data in files. If you need more fine-grained control, use the [`list`](#list) and [`get`](#get) commands instead.
|
|
|
|
|
|
```
|
|
|
```plaintext
|
|
|
pepcli pull --all-accessible
|
|
|
pepcli pull -C <column group> -P <participant group>
|
|
|
```
|
... | ... | @@ -326,17 +336,17 @@ Use the `pepcli query` command to retrieve information about your access to the |
|
|
|
|
|
The `column-access` subcommand lists the columns and column groups that are accessible to you.
|
|
|
|
|
|
```
|
|
|
```plaintext
|
|
|
pepcli query column-access
|
|
|
```
|
|
|
|
|
|
The output will include all accessible columns and column groups, as well as whether you have read and/or write access.
|
|
|
The output will include all accessible columns and column groups, as well as whether you have read and/or write access.
|
|
|
|
|
|
### `query participant-group-access`
|
|
|
|
|
|
The `participant-group-access` subcommand lists the columns and column groups that are accessible to you.
|
|
|
|
|
|
```
|
|
|
```plaintext
|
|
|
pepcli query participant-group-access
|
|
|
```
|
|
|
|
... | ... | @@ -345,7 +355,7 @@ The output will include whether you have `access` and/or `enumerate` privileges |
|
|
- the `access` privilege allows you to retrieve data from rows included in the group.
|
|
|
- the `enumerate` privilege allows you to list the rows included in the group.
|
|
|
|
|
|
Most actions require the user to hold both privileges. E.g. when specifying a `-P groupname` to [`pepcli pull`](#pull), you'll want to list the rows included in the group *and* retrieve data from them.
|
|
|
Most actions require the user to hold both privileges. E.g. when specifying a `-P groupname` to [`pepcli pull`](#pull), you'll want to list the rows included in the group _and_ retrieve data from them.
|
|
|
|
|
|
Members of the `Data Administrator` user group automatically have full access to all participant groups.
|
|
|
|
... | ... | @@ -353,7 +363,7 @@ Members of the `Data Administrator` user group automatically have full access to |
|
|
|
|
|
Use the `enrollment` subcommand to find out how you're [enrolled](#enrollment) into the PEP system.
|
|
|
|
|
|
```
|
|
|
```plaintext
|
|
|
pepcli query enrollment
|
|
|
```
|
|
|
|
... | ... | @@ -362,42 +372,48 @@ The output will include your user name (ID) and the user group to which you belo |
|
|
## `store`
|
|
|
|
|
|
You can store data with this command:
|
|
|
```
|
|
|
|
|
|
```plaintext
|
|
|
pepcli store -c <column name> -p <participant> -i /PATH/TO/DATA/FILE
|
|
|
```
|
|
|
This will output the identifier of the stored entry.
|
|
|
|
|
|
This will output the identifier of the stored entry.
|
|
|
|
|
|
The flags are:
|
|
|
|
|
|
- `-c` The column to store the data in
|
|
|
- `-p` This is either the participant identifer, or the polymorphic pseudonym to store data for. PPs can be obtained with `pepcli list`.
|
|
|
- `-p` This is either the participant identifer, or the polymorphic pseudonym to store data for. PPs can be obtained with `pepcli list`.
|
|
|
- `-i` Path to the file to store. `-` means stdin, and is the default.
|
|
|
- `-d` Data to store. Use either this or `-i`
|
|
|
- `-T` By default, pepcli will request a write-only ticket. You can use `-T` and give a path to store the ticket in. If you use this flag, pepcli will also request read access to the entry that is stored. You can then use the ID in the output, together with this ticket for `pepcli get`. This way you can check whether the data was stored correctly. Note that pepcli also performs its own checks to see whether the data was stored correctly.
|
|
|
|
|
|
# Specific usage scenarios
|
|
|
|
|
|
## Uploading and downloading data
|
|
|
|
|
|
See [Uploading and downloading data](Uploading-and-downloading-data)
|
|
|
|
|
|
## Data administration
|
|
|
|
|
|
### Creating participant groups, based on certain attributes
|
|
|
|
|
|
Lets say we want to create a participant group `males`, which contains all male participants. The sex of a participant can be found in a column `Castor.GeneralInfo`.
|
|
|
|
|
|
1. We start by creating the participant group:
|
|
|
```
|
|
|
|
|
|
```plaintext
|
|
|
pepcli ama group create males
|
|
|
```
|
|
|
|
|
|
2. We then download the data for the column `Castor.GeneralInfo`, for all participants:
|
|
|
```
|
|
|
|
|
|
```plaintext
|
|
|
pepcli pull -c "Castor.GeneralInfo" -P "*"
|
|
|
```
|
|
|
|
|
|
3. The data administrator now filters the downloaded data: They take note of the directory names of those participants that, according to the downloaded data, are male. How this is done exactly, is outside of the scope of PEP. The result is a list of directory names. These directory names are the local pseudonyms of the participants.
|
|
|
|
|
|
3. The data administrator now filters the downloaded data: They take note of the directory names of those participants that, according to the downloaded data, are male. How this is done exactly, is outside of the scope of PEP. The result is a list of directory names. These directory names are the local pseudonyms of the participants.
|
|
|
4. We can now use this list to add participants to the participant group:
|
|
|
|
|
|
```
|
|
|
```plaintext
|
|
|
pepcli ama group addTo males <local pseudonym>
|
|
|
```
|
|
|
For each `<local pseudonym>` from the list of step 3. |
|
|
|
|
|
For each `<local pseudonym>` from the list of step 3. |
|
|
\ No newline at end of file |