|
|
|
The `pepcli` application is the primary command line interface (CLI) to access the PEP system. It is available for multiple platforms, and is included in PEP's Docker images @@@ which one(s)? @@@ and in the Windows client software installer.
|
|
|
|
|
|
|
|
The use of command line utilities such as `pepcli` is subject to details of the system on which it is run. For example, a literal `*` (asterisk) parameter value must be escaped to `\*` on Linux to prevent [shell expansion](https://www.gnu.org/software/bash/manual/html_node/Shell-Expansions.html). Such details are not covered in this documentation. Users are expected to be knowledgeable enough about their platforms to perform basic tasks and avoid common pitfalls.
|
|
|
|
|
|
|
|
# Authentication
|
|
|
|
If you are working in a desktop environment (e.g. on Windows machine), you can authenticate to PEP with:
|
|
|
|
|
|
|
|
```
|
|
|
|
pepLogon
|
|
|
|
```
|
|
|
|
|
|
|
|
This will open a browser window in which you can authenticate. After that, you will remain authenticated for one day.
|
|
|
|
|
|
|
|
If you are working on a server environment, you will receive an oauth token from us. This can be passed to `pepcli`, as explained below.
|
|
|
|
|
|
|
|
# General usage
|
|
|
|
Most `pepcli` command have the following form:
|
|
|
|
```
|
|
|
|
pepcli [general flags] <COMMAND> [command flags] [parameter 1] [parameter 2] [...]
|
|
|
|
```
|
|
|
|
Some commands have subcommands:
|
|
|
|
```
|
|
|
|
pepcli [general flags] <COMMAND> <SUBCOMMAND> [subcommand flags] [parameter 1] [parameter 2] [...]
|
|
|
|
```
|
|
|
|
|
|
|
|
Some of the general flags are the following:
|
|
|
|
|
|
|
|
- `--oauth-token` You need this flag if you cannot use pepLogon. You will receive a token file that authenticates you to PEP. This flag takes the path to that file. After using this flag once you will remain authenticated for a day. So it's not necessary to pass this flag again. It's not a problem if you do pass it again, so when writing code that runs pepcli it's probably easiest to always pass this flag.
|
|
|
|
- `--client-working-directory` pepcli comes with a set of configuration files. By default it looks for those files in the directory where the pepcli binary itself is located. This can be overridden with this flag.
|
|
|
|
|
|
|
|
For brevity, these general flags will not be shown in the examples below. A full command looks like this:
|
|
|
|
|
|
|
|
```
|
|
|
|
pepcli --oauth-token /PATH/TO/OAuthToken.json --client-working-directory /PATH/TO/config-directory list -C \* -P \*
|
|
|
|
```
|
|
|
|
|
|
|
|
# PEP Commands
|
|
|
|
## See which columns you have access to
|
|
|
|
```
|
|
|
|
pepcli query column-access
|
|
|
|
```
|
|
|
|
This will list the column groups and the columns you have access to, as well as whether you have read and/or write access
|
|
|
|
|
|
|
|
## List data
|
|
|
|
You can list data e.g. as follows:
|
|
|
|
```
|
|
|
|
pepcli list -C <column group> -P <participant group> -T <ticket out file>
|
|
|
|
```
|
|
|
|
|
|
|
|
This will list the data that is in PEP, in json format. If a data entry is short enough, it will be displayed directly in the output. For larger entries it will display an id, which can be used with the `pepcli get` command. There are flags to change this behaviour.
|
|
|
|
|
|
|
|
Important flags are:
|
|
|
|
|
|
|
|
- `-C` Column group to list data for. Can be repeated if you want data for more than one column group. There is a special column group `*` that contains all columns.
|
|
|
|
- `-c` Specific column to list data for. Can be repeated, and combined with `-C` if you want multiple columns and column groups
|
|
|
|
- `-P` Participant group to list data for. Can be repeated if you want data for more than one participant group. There is a special participant group `*` that contains all participants.
|
|
|
|
- `-p` Specific participant to list data for. Can be repeated, and combined with `-P` if you want mulitple participants and participant groups
|
|
|
|
- `-l` Include the local pseudonyms in the output. By default pepcli will only show polymorphic pseudonyms (PP). These are not constant, and cannot be used to see whether data belongs to the same participant. You need the local pseudonyms (LP) for that.
|
|
|
|
- `-T` The first thing PEP does when you interact with it, is checking whether you have access to the partipant(group)s and column(group)s you request. If you do have access, it will hand out a ticket. You can store this ticket with the `-T` flag, to use it for later actions.
|
|
|
|
- `-t` You can pass a ticket from an earlier request with the `-t` flag. The column(group)s and participant(group)s of this request must be a subset of the earlier request.
|
|
|
|
- `-s` The size limit (in bytes) for data that should be inlined. Currently defaults to 1000. Setting this to 0 means that data wil ALWAYS be inlined
|
|
|
|
- `--no-inline-data` Never inline data
|
|
|
|
- `-g` Data MAY show up grouped, when it belongs to the same participant. By default this depends on the order in which data comes in, so this grouping is not guaranteed. Use `-g` to force grouping of data. This may impact performance.
|
|
|
|
|
|
|
|
Note: Shells use `*` for globbing. We do not want this behaviour, so make sure you escape it with a backslash or double quotes when invoking pepcli from e.g. bash.
|
|
|
|
|
|
|
|
## Get specific entries
|
|
|
|
After you have done a list, depending on whether data was inlined or not, you have IDs for data entries. You can retrieve the data as follows:
|
|
|
|
```
|
|
|
|
pepcli get -t <ticket file> -i <identifier> -o <output file>
|
|
|
|
```
|
|
|
|
|
|
|
|
The flags are:
|
|
|
|
|
|
|
|
- `-t` The ticket you stored with the `-T` flag of `pepcli list`
|
|
|
|
- `-i` The identifier you got from `pepcli list`
|
|
|
|
- `-o` The file to write the output to. `-` indicates stdout. This is the default.
|
|
|
|
- `-m` Also retrieve the metadata and write it to the given file. `-` indicates stdout. Default is to not retrieve metadata. If you do not know what you would need metadata for, then you probably don't need it.
|
|
|
|
|
|
|
|
|
|
|
|
## Store data
|
|
|
|
You can store data with this command:
|
|
|
|
```
|
|
|
|
pepcli store -c <column name> -p <participant> -i /PATH/TO/DATA/FILE
|
|
|
|
```
|
|
|
|
This will output the identifier of the stored entry.
|
|
|
|
|
|
|
|
|
|
|
|
The flags are:
|
|
|
|
|
|
|
|
- `-c` The column to store the data in
|
|
|
|
- `-p` This is either the participant identifer, or the polymorphic pseudonym to store data for. PPs can be obtained with `pepcli list`.
|
|
|
|
- `-i` Path to the file to store. `-` means stdin, and is the default.
|
|
|
|
- `-d` Data to store. Use either this or `-i`
|
|
|
|
- `-T` By default, pepcli will request a write-only ticket. You can use `-T` and give a path to store the ticket in. If you use this flag, pepcli will also request read access to the entry that is stored. You can then use the ID in the output, together with this ticket for `pepcli get`. This way you can check whether the data was stored correctly. Note that pepcli also performs its own checks to see whether the data was stored correctly.
|
|
|
|
|
|
|
|
## Pull data
|
|
|
|
The above commands are low level PEP commands. But there is also a command that makes downloading from PEP more convenient:
|
|
|
|
|
|
|
|
```
|
|
|
|
pepcli pull -C <column group> -P <participant group>
|
|
|
|
```
|
|
|
|
|
|
|
|
This will by default store the data to the directory `pulled-data`.
|
|
|
|
|
|
|
|
|
|
|
|
Important flags are:
|
|
|
|
|
|
|
|
- `-C` Column group to list data for. Can be repeated if you want data for more than one column group. There is a special column group `*` that contains all columns.
|
|
|
|
- `-c` Specific column to list data for. Can be repeated, and combined with `-C` if you want multiple columns and column groups
|
|
|
|
- `-P` Participant group to list data for. Can be repeated if you want data for more than one participant group. There is a special participant group `*` that contains all partipants.
|
|
|
|
- `-o` Directory to write files to. Default is `pulled-data`.
|
|
|
|
- `-f` Overwrite or remove existing data in output directory
|
|
|
|
- `-r` Resume an interupted download
|
|
|
|
- `-u` Updates an existing output directory, e.g. when new data is available. This will use the same participant(group)s and column(group)s as the original download, so `-c`, `-C`, `-p` and `-P` are not allowed with this flag
|
|
|
|
|
|
|
|
|
|
|
|
# Specific examples
|
|
|
|
Let's say we have the following column groups, with the listed columns:
|
|
|
|
|
|
|
|
- Example
|
|
|
|
* ExampleColumn1
|
|
|
|
* ExampleColumn2
|
|
|
|
* ExampleColumn3
|
|
|
|
* ExampleColumn4
|
|
|
|
- SomeData
|
|
|
|
* SomeData.ColumnA
|
|
|
|
* SomeData.ColumnB
|
|
|
|
* SomeData.ColumnC
|
|
|
|
|
|
|
|
Each participant must have a participant identifier, e.g. `CP1234567890123`, provided by you. This will be the seed of the pseudonymization performed by PEP. These identifiers are very sensitive, and may NOT be sent to others. Instead, they must use the pseudonyms created by PEP.
|
|
|
|
|
|
|
|
To store data from the file `data.txt`, for participant identified by `CP1234567890123` in column `ExampleColumn1`:
|
|
|
|
|
|
|
|
```
|
|
|
|
pepcli store -c ExampleColumn1 -p CP1234567890123 -i data.txt
|
|
|
|
```
|
|
|
|
|
|
|
|
To store the string `Hello world` in column `SomeData.ColumnA`:
|
|
|
|
```
|
|
|
|
pepcli store -c SomeData.ColumnA -p CP1234567890123 -d "Hello World"
|
|
|
|
```
|
|
|
|
|
|
|
|
To list the data for column groups `Example` and `SomeData`, and store the ticket in the file `ticket.out`:
|
|
|
|
```
|
|
|
|
pepcli list -C Example -C SomeData -P \* -T ticket.out
|
|
|
|
```
|
|
|
|
|
|
|
|
If you executed the previous examples, and `data.txt` was sufficiently large (>1000 bytes), this will display an ID for the column `Example1`. You can get the data (printed to stdout) with:
|
|
|
|
```
|
|
|
|
pepcli get -t ticket.out -i <IDENTIFIER FROM LIST>
|
|
|
|
``` |
|
|
|
\ No newline at end of file |